Evaluating the Performance of a Vehicle Pose Measurement

System

Harry Scott

Sandor Szabo

National Institute of Standards and Technology

Abstract

A method is presented for evaluating the

performance of a vehicle pose measurement

system (e.g., GPS, inertial sensors, etc.). The

method supports evaluations of the system on a

vehicle moving at high speeds. An example is

provided to illustrate the method.

Keywords

Vehicle, pose, position, measurement, accuracy,

uncertainty, GPS, IMU

1 Introduction

Precise knowledge of the pose of a moving

ground vehicle is important in such applications

as navigation, safety, robotics, metrology and

surveying. Vehicle pose measurement (VPM)

system manufacturers typically state the

accuracy of their systems under static conditions,

that is, with the system stationary over time.

Latency and synchronization errors are difficult

to detect under static conditions. In this paper, a

method is described for evaluating the

performance of a VPM's ability to measure the

position of a moving vehicle. The method

requires that the VPM have a synchronization

interface, which we call a sync pulse interface.

This interface is often incorporated in a VPM to

support photogrammetry applications where a

camera's vertical sync latches a pose

measurement. In the method presented in this

paper, a photo emitter/detector sensor that senses

energy reflected off a reflective target generates

a sync pulse. By mounting the photo sensor on

the vehicle and placing a reflective target on the

ground, pulses are generated whenever the

vehicle drives directly over the target. Repeated

measurements of the target as the vehicle drives

by are used to determine the repeatability in the

VPM's measurement of the target location.

The National Institute of Standards and

Technology (NIST) Intelligent Systems Division

is pursuing work in several areas of intelligent

vehicles, including research in autonomous

mobility and development of performance

measurement techniques. Various testbed

vehicles are used in support of several projects,

and each has been, or will be, outfitted with a

VPM system. These systems are used in at least

two distinctive ways. For autonomous vehicles,

the VPM is a principal component of a real-time

navigation system, providing necessary

information for the vehicle to move through the

environment. A somewhat different need is met

by the ability of the VPM system to capture

appropriate information in real-time that can be

combined with other sources of information after

the fact to generate more precise pose

information than is available in real-time. This

meets a metrology need, providing improved

accuracy to evaluate performance of the vehicle

itself and its various sensors.

This paper describes a method to determine or

confirm the performance of an implemented

VPM on a testbed vehicle. Since the VPM

examined in this paper is intended for metrology

applications, steps are included in the method to

post process the data to obtain the highest

possible accuracy. The method described could

also be used to obtain real-time performance

information. This is accomplished by analyzing

the real-time solutions as opposed to the postprocessed solutions. The method consists of a

way to precisely trigger vehicle position

measurements, a procedure for collecting data

and a way to analyze the repeatability in the

vehicle position measurements. Although VPMs

are capable of providing a full position and

orientation solution, this current effort addresses

only the position measurement capability. In

the following sections, the method is described

in the context of evaluating a specific VPM,

though the described method is applicable to any

VPM that supports a trigger mechanism.
Figure 1 Note the downward looking photo sensor, the retroreflector on the ground and the vertical

rod to guide the driver.

2 Test Procedures

The vehicle used in the evaluation is a full size

passenger sedan. The installed VPM is a system

that integrates a dual frequency carrier phase

Global Positioning System (GPS), a secondary

GPS, an inertial measurement unit (IMU)

consisting of accelerometers and fiber optic

gyros, a wheel encoder, and a system control and

data collection computer. This system is

configured to capture and provide solutions at a

200 Hz data rate. In addition, it is configured to

capture all data required for later post

processing.

The system is augmented with an external photo

sensor connected to the sync pulse interface.

When triggered, the result is the notation in a file

of an event occurrence, along with the time of

the event with microsecond time precision.

Since triggers will be caused during vehicle

motion, latency in the assertion of this signal will

be reflected in position errors in the resulting

data. For this reason, a low latency photo sensor

(300 ms) was employed. During post

processing, pose solutions are determined for

selected events of interest. The sensor was

mounted slightly in front of the vehicle bumper,

in line with the vehicle driver position. A

vertical rod above the sensor assists the driver in

steering the sensor over a target while driving.

The target is a simple retroreflective surface,

constructed by affixing a layer of retroreflector

material (available in sheets) to an aluminum

disk. A disk of 15.24 cm (6 in) diameter is used.

This size was selected as large enough to enable

the driver to successfully steer the vehicle over it

at speeds of interest most of the time, even in a

grassy and somewhat bumpy field, but small

enough to keep the target detection points close

to a surveyed point. In this test a National

Geodetic Survey (NGS) survey marker, flush

with the ground, is used. The retroreflector is

simply centered on top of it.

The configuration of the photo sensor, its

mounting, and target is shown in Figure 1.

To evaluate the VPM's measurement

performance, the vehicle is driven over the target

from a variety of directions while collecting all

data necessary to compute, via post processing,

the position of the sensor at the time of target

crossing. In the analysis phase, the repeatability

of the measurements of the target's radius and the

location of the target with respect to the survey

marker produce a measure of performance of the

VPM system.

The position solutions are computed for the

location of the sensor (actually for a point on the

ground directly below the sensor). This requires

a transformation of coordinate frames between

the VPM system and the position of interest

below the sensor. This transformation depends

on knowledge of the translations and rotations

between the two reference frames. In the case of

this implementation, translation and rotation

measurements between the IMU reference frame

and the sensor reference frame are needed.

These measurements are entered into the VPM

system. Determination of the translation and

rotation parameters is performed in two steps.
The first is a best-effort measurement of the (x,

y, z) distances between the IMU and the point

below the sensor. Performing these

measurements is physically awkward because of

the location of the IMU in the vehicle trunk and

the sensor location at the front bumper. An

attempt was made to mount the IMU to align

with the major vehicle axes as well.

Since these measurements are difficult and

somewhat prone to error, data is also collected

during the test run which is used to calibrate the

system by adjusting the translation and rotation

parameters as required. Several events are

triggered by driving the vehicle sensor over the

target as slowly ("creep") as possible and as well

centered laterally as possible. These creep

events are collected for approaches to the target

from four directions. The slow speed is intended

to eliminate any significant data latency errors.

Positions of events collected in this manner

should conform to the edge of the target, and the

calibration parameters are adjusted slightly to

yield solutions consistent with the known target

diameter. This “creep” data also helps detect any

movement of the sensor that may have occurred

between tests.

After collecting the initial creep calibration data,

the vehicle was driven over the target at a

number of speeds and from a number of

directions. Speeds were limited to about 10.6

m/s ( 20 mph) due to the roughness of the grassy

field where these tests were performed. (The

sensor/retroreflector system has been tested

successfully at highway speeds on a roadway as

well.) Approximately 40 events and their

associated data were collected in this way,

including a second set of creep points after

completion of the vehicle at-speed runs.

3 Post Processing

Since the purpose of this test was to determine

the maximum accuracy possible from the VPM,

post processing was conducted on the data. If

the purpose of a test is to evaluate the real-time

performance of a VPM, then this step would not

be performed.

The data logged on the vehicle during the test is

retrieved, and the real-time navigation solution,

though not the subject of this paper, is examined

to confirm the existence of good data depicting

an appropriate vehicle trajectory. The raw data

is then post processed to obtain the more precise

solution for the events of interest.

To enhance the quality of the solution,

differential GPS post processing is used. A

detailed explanation of this processing is beyond

the scope of this paper, but the approach

essentially makes use of information collected

from another nearby GPS receiver (base station)

at an accurately known location to remove

(during post processing) certain kinds of errors

from the reported position of the rover (our

vehicle).

For these tests, we used a National Geodetic

Survey (NGS) Continuously Operating

Reference Station (CORS) located in

Gaithersburg, MD known as GAIT.

The NGS, an office of NOAA's National Ocean

Service, coordinates a network of continuously

operating reference stations (CORS) that provide

GPS carrier phase and code range measurements

in support of 3-dimensional positioning activities

throughout the United States and its territories.

(See http://www.ngs.noaa.gov/CORS/)

The reference station data for station GAIT is

downloaded from the NGS for the appropriate

period during which we performed the test, and

used by post processing software to enhance the

GPS solution. Further post processing is

performed to integrate this GPS solution with the

raw data from the VPM system sensors (IMU

and wheel encoder). Smoothing algorithms are

executed, and interpolation of navigation

solutions for the recorded events is performed.

The result is a file with a full navigation solution

for each event. The solution information in this

file is analyzed below.

4 Analysis

Two types of VPM measurement errors are

estimated: target (i.e., reflector) location and

target radius. The estimated location error is

derived as the difference between the surveyed

(static) location of the target and VPM measured

(dynamic) location of the target. The estimated

radius error is the difference between the known

radius of the target and VPM measured radius of

the target. The location error may be left

uncalculated if survey data is not available. The

following process is used to compute

uncertainties.
1. Put event points in a convenient local

coordinate system. First, the event points (VPMmeasured coordinates of target's edge) are

transformed from latitude and longitude

coordinates into UTM coordinates (see

velvet.tec.army.mil/access/milgov/fact_sheet/geo

trans.html) so that errors may be expressed in

meters. Second, if available, the known location

of the target is subtracted from each event point.

This places the event points in a coordinate

system whose origin is the surveyed location of

the target.

O

Figure 2. Example measurement points

surrounding the target's edge.

2. Determine the target location error. A circle

is fit to the event points. The fit produces an

offset vector, O, indicating the center of the

circle. Figure 2 shows an example set of data

points with the offset vector to the center of the

circle. If the origin of the coordinate system is

the surveyed location of the target, then the

magnitude of O, el, describes the estimated target

location error.

3. Determine the target radius error and

uncertainty. First, the offset vector of the circle

is subtracted from each event point. This places

the event points into a coordinate system with

the center of the circle at the origin (see Figure

3). Then the distance of each event point from

the origin, di, is computed. These distances are

the measured radii of the target. The mean and

standard deviation of the radius measurements,

r and Sr

, are computed. The estimated target

radius error, el, is the difference between the

mean of the target radius measurements and the

target radius measured by hand. Choosing a 95

% level of confidence, the component of the

expanded uncertainty due to data scatter in the

radius measurement is:

rr u =kS (1)

Where k = 2 for N 30 and k equal to a t-factor

obtained from a t-distribution for N < 30.

di

Figure 3. Measurements points

translated such that center of circle is at

origin. The distance of each point is the

measured radius of the target

Figure 4 shows a plot of the event points

collected when the vehicle traveled over a

circular target with a radius of 7.62 cm (3 in).

The origin of the plot coincides with the survey

marker's coordinates, in this case, the coordinates

of the NGS survey marker where the target was

placed. The large circle is the target drawn to

scale. Each event point is plotted as a small

circle with an attached velocity vector (origin of

vector is inside of small circle and points away

from circle in direction of travel). The vectors

are scaled to fit to the plot. The largest velocity

vector is labeled 10.6 m/s to provide a scale for

comparison (label is at end of vector opposite

small circle shown on left side of plot). The

results of the analysis of these measurements

indicate a location error of 0.3 cm, an estimated

radius error of 0.2 cm and a radius uncertainty of

1.3 cm (95 % level of confidence).
-0.081

-0.060-0.040-0.0200.0000.0200.0400.0600.082

-0.084

-0.075

-0.070

-0.065

-0.060

-0.055

-0.050

-0.045

-0.040

-0.035

-0.030

-0.025

-0.020

-0.015

-0.010

-0.005

0.000

0.005

0.010

0.015

0.020

0.025

0.030

0.035

0.040

0.045

0.050

0.055

0.060

0.065

0.070

0.079

10.6 m/s

Easting (m)

el = 0.3 cm 

er = 0.2 cm 

ur = 1.3 cm 

k = 2.0

c = 95.0 %

V = 40

C:\downloads\nbs3a_8_2\sbet_nbs3a_all_adj2.out (8/6

Figure 4 Plot of event points (shown as small circles) triggered when vehicle is driven over the

edge of a circular target (large circle) with a radius of 7.62 cm. Vectors at each event point are

vehicle velocity at event point. The largest velocity was 10.6 m/s (left side of target).

5 Conclusions

Further work is needed in several areas. This

effort, an initial step in developing a

performance measurement capability, focused

only on vehicle position information in x and y.

Height (z) data was collected but the data has not

yet been analyzed. The testing methodology

needs to be expanded to include orientation (roll,

pitch, yaw) capabilities of VPM systems as well.

That information, along with position

information, is critical in registering the data

received from intelligent vehicle sensors such as

cameras and laser scanners, and directly affects

the correctness of the model of the world being

maintained by the vehicle computers. In

addition, the scope of the work here included

differential post processing of GPS data and

further post processing of all the navigation

sensor data. This is appropriate for metrology

purposes. For navigation of intelligent vehicles,

the real-time solution is important, and methods

for determining the quality of the real-time

navigation solution should be explored. Further,

a real-time navigation solution may make use of

real-time differential GPS corrections received

by the vehicle during operation. The method

described here can be used to examine position

determination performance for those types of
systems as well, and should be extended to

characterize orientation measurement

performance. Further, these tests were

conducted with good GPS satellite coverage.

While gaps in satellite coverage occurred, a test

specifically excluding satellite info for

prescribed periods would enable performance

analysis of the VPM systems when they rely

more on their inertial components.

We wish to acknowledge the National Highway

Traffic and Safety Administration's Office of

Vehicle Safety Research and the U.S. Army

Research Laboratories' Robotics Program Office

for their support of this work.