The AUTONAV / DOT Project:

Baseline Measurement System

For Evaluation of

Roadway Departure Warning Systems

Submitted to

U.S. Department of Transportation

National Highway Traffic Safety Administration

Office of Collision Avoidance Research

Washington, D.C. 20590

Submitted by

Sandor Szabo, Karl Murphy and Maris Juberts

National Institute of Standards and Technology

Intelligent Systems Division

Building 220, Rm. B127

Gaithersburg, MD 20899

Army Research Laboratory

Aberdeen Proving Grounds

Aberdeen, MD

January 27, 1999

- i -

Table of Contents 

EXECUTIVE SUMMARY .......................................................................................................... 1

1 INTRODUCTION................................................................................................................. 2

2 SCENARIO FOR A ROADWAY DEPARTURE SYSTEM EVALUATION ................ 4

2.1 Vehicle Instrumentation................................................................................................................................5

2.2 Collecting Roadway Departure Incident Data ............................................................................................6

2.3 Performance Analysis....................................................................................................................................7

3 BASELINE MEASUREMENT SYSTEM........................................................................ 10

3.1 Inertial Navigation System..........................................................................................................................10

3.2 GPS...............................................................................................................................................................11

3.3 Kalman Filter...............................................................................................................................................12

3.4 Camera Calibration.....................................................................................................................................15

3.4.1 Local Road-Surface Calibration...........................................................................................................16

3.4.2 Camera to GPS Calibration..................................................................................................................17

3.5 Video Digital Recording System.................................................................................................................22

4 EVALUATION OF BASELINE MEASUREMENT SYSTEM ..................................... 25

4.1 Verification of Camera Calibration ...........................................................................................................27

4.2 Static Measurements ...................................................................................................................................28

4.3 Dynamic Measurements and Latency Calculation ...................................................................................30

4.4 Affects of GPS Dropout...............................................................................................................................32

4.5 Lane Position Measurements......................................................................................................................38

5 CONCLUSION ................................................................................................................... 39

APPENDIX A: AUTONAV PROGRAM DESCRIPTION..................................................... 40

UBM 4D Perception and Control System ..........................................................................................................40

DSRC Vision Front End.....................................................................................................................................40

EBK LADAR.....................................................................................................................................................40

REFERENCES............................................................................................................................ 42
- ii -

List of Tables Table 1. The possible outcomes of a warning system evaluation at each instance in time...........................................8

Table 2. Camera calibration attributes. .......................................................................................................................17

Table 3. Video files recorded for test..........................................................................................................................26

Table 4. Navigation data files corresponding to video files in Table 3. ......................................................................26

Table 5. Values of visually acquired nails and their distance from the origin (i.e., the known survey location of the

nail). ...................................................................................................................................................................28

Table 6. Visual-based measurements of nail (see Figure 18) with distance from manual surveyed coordinates of the

nail (origin). .......................................................................................................................................................30

Table 7. Elapsed time between navigation system mode changes. .............................................................................33
- iii -

List of Figures Figure 1. Vehicle instrumentation for evaluating performance of a roadway departure warning system. ....................5

Figure 2. Comparison between inertial sensor drift and GPS. ....................................................................................11

Figure 3. Components of the GPS and INS. ...............................................................................................................12

Figure 4. A complementary filter is used to estimate the true position of the vehicle. ...............................................13

Figure 5. The Kalman filter computes estimated drift and noise in the Northing direction. .......................................14

Figure 6. Picture of downward pointing camera mounted on side of vehicle. ............................................................15

Figure 7. Calibration target is used to define a local road-surface origin (x,y) at top left control point......................16

Figure 8. Vehicle coordinate systems describing camera-to-GPS calibration.............................................................18

Figure 9. Derivation of position transformation between the local road surface and the GPS coordinate systems.....19

Figure 10. Two points are required to derive the rotation transformation between the local road surface and the GPS

coordinate systems. ............................................................................................................................................20

Figure 11. Derivation of rotation transformation between camera and GPS coordinate systems................................21

Figure 12. Components of the Video Digital Recording System. ...............................................................................23

Figure 13. Picture of the BMS Video/Data Digital Recording System. ......................................................................23

Figure 14. A single frame recorded by BMS showing two survey nails highlighted using black and white paint......26

Figure 15. Vehicle driven along a half figure-eight trajectory along with survey nails acquired using video at

beginning and end (see also Figure 16). ............................................................................................................27

Figure 16. Filtered and unfiltered (raw) GPS coordinates of nail at beginning and end of half figure 8 trajectory. ...28

Figure 17. Trajectory of vehicle during test to acquire nail at beginning and end of run............................................29

Figure 18. Results of visually acquiring survey nail at the beginning and end of several runs. ..................................30

Figure 19. The error due to latency is shown in the discrepancy between the coordinates of the surveyed nail

(surveyedNail) and the visually acquired nail (videoNail) as the vehicle (filtered_gps) drives by at 10.75 m/s.31

Figure 20. A run out the NIST gate onto Muddy Branch Rd and then back to the start showing mode changes in the

navigation system (DR = DeadReckoning, DGPS = Differential GPS, cpDGPS = carrier phase DGPS)..........32

Figure 21. Vehicle position updates during section of trajectory labeled “a” in Figure 20. ........................................34

Figure 22. Vehicle position updates during section of trajectory labeled “b” in Figure 20.........................................35

Figure 23. Vehicle position updates during section of trajectory labeled “c” in Figure 20. ........................................36

Figure 24. Nail locations, derived from navigation system (filtered_gps) and raw GPS (unfiltered_gps) data, at the

beginning and end of run in Figure 20................................................................................................................37

Figure 25. Trajectory of vehicle and location of outside lane marker shown with minimum and maximum distance

between vehicle and lane labeled (min/max: distance from lane in meters, GPS record number). ....................38
- 1 -

Executive Summaryi 

This report describes a set of performance metrics and a performance measurement system

developed for the evaluation of roadway departure warning systems. The work is part of an

effort by the Department of Transportation’s National Highway Traffic and Safety

Administration (NHTSA) to develop quantitative measures of performance for the evaluation of

highway safety systems. The work is also part of an effort to promote dual-use technologies

between the Department of Transportation and the Department of Defense. The Department of

Commerce’s National Institute of Standards and Technology undertook the development of the

performance measurement system.

The approach for quantitative evaluation rests on establishing a baseline measurement of

roadway departure that is of higher accuracy than the warning system under test. The output of

the warning system is then compared to the baseline measurement to produce a quantitative

measure of performance. The baseline measurement system is installed on a vehicle with the

warning system. The baseline system achieves high accuracy by aiming a calibrated camera onto

an area of the road adjacent to the vehicle. As the vehicle is driven and roadway departure

maneuvers are performed, the output of the video camera, the vehicle’s position (from a global

positioning system and inertial sensors), and the output of the warning system are synchronized

and digitally recorded. After a vehicle run is complete, a suite of software is used to

automatically analyze the baseline video data to determine roadway departure. Since the video

and other data remain available after the vehicle run, an operator (to establish the credibility of

the baseline measurement) can verify a roadway departure. Also, the data can be analyzed to

provide insight into the cause of a warning system failure (e.g., insufficient roadway markings,

high lateral velocity, etc.). The time of departure is compared with the warning system output to

determine the reliability, failure and false alarm rates of the warning system.

This report describes the performance metrics and the baseline measurement system. It also

describes efforts to determine the accuracy of the baseline system in determining the time and

position of roadway departures. In addition, the precise, synchronized recording of video and

GPS data proved useful in determining the accuracy and the delays of GPS when measuring the

position of a moving vehicle. The baseline measurement system is now sufficiently mature to be

used to evaluate roadway departure warning systems. The measurement system also serves as a

promising platform to perform a wide range of other evaluations.

 i

 This paper was prepared by U.S. Government employees and is not subject to copyright. The use of commercial

equipment or products is for demonstration only and does not imply recommendation or endorsement by NIST.
- 2 -

1 Introduction

One of the goals of the Department of Transportation, National Highway Traffic and Safety

Administration (NHTSA) is to improve safety on U.S. highways. An avenue for improved safety

is the use of driver warning systems for collision avoidance and roadway departure. Warning

systems for roadway departure rely on advanced sensors and processing algorithms to warn the

driver when the vehicle is leaving the road unintentionally. The U.S. Department of Defense,

under the AUTONAV (Autonomous Vehicle Navigation Control System) project, is looking at

similar sensors and algorithms for controlling autonomous ground vehicles. The basis for the

interaction between DOT and DOD is to promote dual-use technologies for the improvement of

surface transportation safety. The Army Research Lab (ARL) has provided the National Institute

of Standards and Technology (NIST), the primary research participant, with a vehicle and

various sensing and control components (see Appendix A: Autonav Program Description) to

serve as the test-bed to evaluate warning systems and to develop methods for performance

measurement. This report describes a set of performance metrics and a performance

measurement system developed for the evaluation of roadway departure warning systems.

A primary function of a roadway departure warning system is to detect the location of the

roadway boundaries. The location of the roadway must be detected sufficiently forward of the

vehicle to provide time for both the warning system to determine whether the vehicle is on a

course to leave the lane, and for the driver to take the appropriate action. The warning system

must operate on a wide variety of roads, day or night, and under a wide range of weather

conditions. The main effort in the AUTONAV/DOT project was to develop a method for

quantitatively measuring the performance of the warning system under these varied conditions.

The primary measure of performance is the reliability of the warning system to detect roadway

departure incidents and to reject false alarms. The method uses a baseline measurement system

(BMS) to obtain a baseline of how a warning system is expected to perform. The baseline is

compared to the warning system’s actual performance in order to determine a measure of

reliability. Section 2 presents a scenario of how a warning system is evaluated using the BMS.

The components of the BMS are described in section 3. Several tests were conducted to validate

the measurement capabilities of the BMS. The results of the BMS tests are presented in section

4.

The BMS was developed based on NIST’s experience1

 with systems that have demonstrated the

capability to perform lane tracking and that could serve as a potential warning system. During

the time of the work, there were no commercially available systems, but recent announcements

have been made that products will soon be available. The final method for evaluation of these

warning systems will depend on the goals of the manufacturer and of the evaluators. For

example, the evaluation may also seek to obtain the response of the driver to the warning

system’s operation. The intent of the efforts described in this report is to build an understanding

of the types of measurements that can be performed and to demonstrate a method of evaluation.

More work needs to be done to refine the measures of performance, to increase the number of

measurement capabilities, and to improve the accuracy of the measurements. The authors
- 3 -

express gratitude to Alhad Chande and August Burgett of the National Highway Traffic Safety

Administration’s Office of Crash Avoidance Research for their insights, perseverance and

contributions toward the completion of this project and the report
- 4 -

2 Scenario for a Roadway Departure System Evaluation

Several types of evaluations can be conducted to obtain an overall performance evaluation of a

roadway departure warning system. A preliminary evaluation is usually conducted prior to a

field operational test in order to determine the safety and reliability of the warning system under

a controlled set of circumstances. The results of a preliminary evaluation are used to design and

implement a full-scale field operational test involving a large number of drivers operating under

normal everyday driving conditions. A scenario for a preliminary evaluation of a roadway

departure warning system is described in this section. This scenario drove the design and

implementation of a performance measurement system called the baseline measurement system

(BMS). The following list highlights some general attributes of a preliminary evaluation.

1. Ability to generate a large number of roadway departure incidents.

2. Ability to record the response of the warning system to each incident.

3. Ability to establish a baseline, that is, the correct operation expected from the warning

system, and to compare the output of the warning system against the baseline in a

quantitative manner.

4. Ability to perform evaluations on real roads – not just on a closed test tracks.

5. Ability to perform evaluations in wide variety of environmental conditions: lighting, weather,

etc.

6. Ability to perform the evaluations in a cost-effective manner.

The first step of an evaluation is to instrument the vehicle with the warning system and the BMS.

An overview of the vehicle instrumentation is given in section 2.1. Once the instrumentation is

in place and calibrated, the remaining procedures in an evaluation plan consist of a series of testdrives and data collections described in section 2.2. Section 2.3 describes how the data is

analyzed to produce various measures of performance that describes the reliability of the warning

system.
- 5 -

downward-looking

camera

Roadway

Departure

Warning System

Video/Data

Digital

Recording

System

DGPS/INS

System

Baseline Measurement System

Figure 1. Vehicle instrumentation for evaluating performance of a roadway departure

warning system.

2.1 Vehicle Instrumentation

The first step to a preliminary evaluation of a roadway departure warning system is to install the

warning system and the baseline measurement system on the vehicle. For scenario purposes, it is

assumed that the warning system is vision-based, and the manufacturer installs the system on the

vehicle and performs any required calibrations. Figure 1 shows the components of the BMS that

are installed on the vehicle. The Video/Digital Recording subsystem consists of a calibrated

camera and a digital recorder. The camera is used to measure the position of the vehicle relative

to a lane marker. It is mounted on the vehicle and points to the side and downward as close to

perpendicular as possible. Currently, only a single camera is used to capture either the inside or

outside lane marker. There are several methods to extend the BMS to multiple cameras so that

both markers can be detected. The camera is calibrated so that image coordinates in pixels can

be transformed into road-surface coordinates in millimeters.

The GPS/INS subsystem measures vehicle position and velocity using a carrier phase differential

GPS augmented with an inertial navigation system. When GPS/INS data is available, the camera

can be calibrated further so that image coordinates can be transformed into GPS coordinates

(northing and easting). It should be noted that a warning system can be evaluated without the

GPS/INS subsystem. The camera system data provides the distance to the lane marker, which is

sufficient for determining when a vehicle departs a lane. The GPS/INS subsystem is necessary

when the evaluation also requires where the vehicle departs the lane. Both video and navigation

data are stored digitally on an array of hard disks. Digital storage on disk enables

synchronization of all the data. In addition, the data can be examined immediately to ensure its

quality. Finally, the analysis of the data is not limited by the onboard processor’s requirement to

respond to real-time events.
- 6 -

The third component shown in Figure 1 is the warning system under evaluation. An interface is

developed between the BMS and the warning system so that roadway departure warnings

generated by the system can be recorded along with the BMS video and GPS/INS data. Other

types of warning system data may be recorded, including:

1. Loss of lane marker tracking. This may result when the lane markers fade or disappear.

2. Time-to-departure from lane. The estimated time until the vehicle crosses the lane marker.

3. Distance from lane marker and closing rate to lane marker.

4. Lane curvature.

2.2 Collecting Roadway Departure Incident Data

The purpose of the data collection is to obtain data describing the performance of the warning

system under a wide variety of operational conditions. The conditions are selected to closely

mimic real driving environments. The data provides insight into the accuracy of the system in

detecting a roadway departure and in rejecting false alarms. Generating incidents for a rear-end

collision avoidance system is difficult, but this is not the case for a roadway departure warming

system. Driving the vehicle over various roads and intentionally crossing over the lane markers

is sufficient for a preliminary evaluation.

The operational parameters for data collection are the various conditions that affect the

performance of the warning system. Prior to the evaluation, each parameter is identified and the

space the parameter spans is defined. For example, lighting is one parameter and it spans from

full daylight to nighttime. When possible, the parameter is measured in order to quantify under

what conditions the data was collected. The following is a brief list of some parameters that may

affect system performance.

1. Lane proximity – distance between the vehicle and lane marker.

2. Lane departure rate – rate at which vehicle can approach lane marker.

3. Lane departure frequency – amount of time between lane crossings.

4. Road surface types – asphalt, concrete, etc.

5. Road surface quality – cracks, potholes, etc.

6. Lane marker contrast – bright, faded, etc.

7. Lane marker type – solid, stripped, etc.

8. Lane types – merge, split, intersection, etc.

9. Lane curvature – curvature of road the system can monitor.

10. Light intensity – direct sunlight, clouds, dusk, night, etc.

11. Weather condition. Dry, rain, snow, etc.

Once the BMS is installed on the vehicle and calibrated, the process of collecting data is

straightforward. First, a time and place is selected based on the type of data needed, i.e.,

nighttime, rainy weather on interstate. The driver turns on the BMS and starts driving. The

driver performs maneuvers that test the response of the warning system to both lane crossings

and non-lane crossings. The data is collected in relatively short segments of 30 to 60 minutes

each. During each data collection, the driving is divided into two phases. In the first phase, no

lane crossings are performed in order to gather data characterizing the false alarm rate of the
- 7 -

warning system. In the second phase, a variety of maneuvers are performed that cause the

vehicle to cross a lane marker. This can be done in a safe manner by performing the maneuver on

sections of road where no other vehicles are close by. Also, run-off-road maneuvers can be

performed along roads where sufficient asphalt surface exists on the other side of the lane marker

to permit safe control of the vehicle after the lane marker is crossed.

The steps performed by the driver to collect data are summarized in the following list.

1. Select time and place for data collection in order to satisfy desired operational parameters.

2. Turn on BMS recording and verify data quality.

3. Drive down center of lane for 10 minutes.

4. Decrease distance to lane marker in increments of 15 cm (6 inches) and maintain distance for

1 minute. Stop decreasing distance prior to contact with lane marker. Return to center and

perform same steps toward opposite lane marker (assumes BMS records lane position on

both sides). Return to lane center when done.

5. Slowly oscillate position of vehicle within lane without touching lane marker. In safe

manner, slowly increase frequency of oscillation.

6. Select section of road where lane crossings can be performed.

7. Perform several lane changes in both directions.

8. Slowly oscillate position of vehicle so as to cross lane marker repeatedly. In safe manner,

slowly increase frequency of oscillation.

9. Turn off BMS recording and verify data quality.

10. Make entries describing conditions during data collection not automatically recorded by the

BMS (weather, road quality, etc.).

After the collection run is complete, the data is transferred from the vehicle to a workstation

where warning system performance is evaluated using BMS analysis software.

2.3 Performance Analysis

The performance of the warning system is judged by its reliability, that is, the ability to warn

when a lane departure takes place and to reject false alarms. A quantitative measure of

performance (MOP) for reliability of the warning system is defined as:

100 ⋅100 

 





+ + +

+ ⋅ = 













 = TP TN FP FN

TP TN

all possible outcomes

successful outcomes general reliability (2-1)

The general reliability of the warning system is obtained by comparing the output of the warning

system against the data collected by the BMS. In each video frame, both the BMS and the

warning system measure the distance between the vehicle and the lane marker. For a basic

system that warns when the lane edge is crossed, if the distance goes to zero, then the vehicle has

departed the road and the warning system should signal a roadway departure incident. Column 1

in Table 1 labels the possible responses of the warning system. The remaining entries in the table

describe the possible outcomes of a comparison between the warning system and the BMS data.

A true positive (TP) outcome indicates that both the warning system and the BMS data indicate a

roadway departure incident. A false positive (FP) outcome, or a false alarm, occurs when the
- 8 -

warning system issues a warning and none is detected in the BMS data. A false negative

outcome, potentially the most catastrophic result, occurs when the warning system completely

misses a roadway departure. Finally, a true negative indicates that the warning system is not

responding and none exists.

Warning System Response BMS indicates roadway

departure

BMS indicates no roadway

departure

warning system warns of

roadway departure

True Positive (TP)

Correctly warns of roadway

departure

False Positive (FP)

False alarm

warning system does not warn

of roadway departure

False Negative (FN)

Misses roadway departure

True Negative (TN)

Correctly does not warn of

roadway departure.

Table 1. The possible outcomes of a warning system evaluation at each instance in time.

The general reliability metric rewards a warning system for correctly not warning of a roadway

departure (true negative). This leads to a problem with the metric because the majority of the

data will produce true negative outcomes. This bias may produce a high reliability value even

when the warning system misses all roadway departures. For example, consider the following

case where the warning system misses all 10 lane departures (TP + FN defines the total number

of lane departures) and yet scores a general reliability of 99%:

100 99%

0 1000 0 10

0 1000

10

0

1000

0

⋅ = 

 





+ + +

+ =

=

=

=

=

general reliability

FN

FP

TN

TP

An alternative metric focuses on the reliability of the warning system in critical situations when

the vehicle departs the road.

⋅100 

 





+ = TP FN

TP critical reliability (2-2)

A critical reliability of 100% indicates that the warning system never failed to detect a roadway

departure when one occurred. Based on the example above, the critical reliability would be 0%

and not 99%. The failure rate of the warning system can be derived from the warning system’s

critical reliability.

1 ⋅100 

 





+ = − = TP FN

FN failure rate critical reliability (2-3)
- 9 -

The false alarm rate of the warning system is the number of false positives over some number of

possible outcomes. The question is over what range of possible outcomes is the rate calculated?

⋅100 

 





+ + + = TP TN FP FN

FP false alarm rate (2-4)

Again, including a large number of true negative outcomes in the denominator will bias the false

alarm rate toward a smaller value. Since the false alarm rate is more of an annoyance than a

failure, and false alarms can occur at any time (not just during critical moments), the rate is

defined over all possible outcomes.

A report is generated that summarizes the results of the analysis for all sets of data. The report

specifies the state of the operational parameters listed in section 2.2 and the metrics described

above. The metrics will vary for each set of data. For example, the reliability may be 98%

during daylight hours and 88% at night. A cumulative reliability is computed based on the

results under all the conditions.

The preceding scenario provides one example of how the BMS is used to evaluate a roadway

departure warning system. Reliability is the primary measure of performance but additional

measures may be appropriate. These measures will be developed as the goals of the evaluation

of a specific warning system are finalized. The BMS is a general-purpose system and can be

programmed to meet a wide range of performance analysis requirements. The remaining

sections in this report provide details of the design, integration, and capability of the BMS.
- 10 -

3 Baseline Measurement System

The BMS is primarily a data collection system designed to capture video, GPS data, and the

output from a countermeasure system under evaluation. A digital recording system is used to

capture and synchronize all the data more easily. Recording the data, as opposed to performing

the evaluation in real-time, has several advantages. First, and foremost, is the ability to analyze

the data in order to reduce the uncertainty of the reference. Without the ability to analyze the

data, it is difficult to make the assertion that the BMS serves as the reference for other

measurements. As an example, suppose a roadway departure system under evaluation reports

that the vehicle crossed a lane marker at a particular position or instance in time. The BMS, on

the other hand, reports that the vehicle did not cross a lane marker at that point. Which system is

correct? Recording the entire event enables the evaluator to examine the data and judge which is

more accurate. With the visual data recorded, additional methods can be used to improve the

uncertainty. For example, survey markers can be used as a reference to improve position

measurements. Another advantage of recording the data is the ability to conduct a variety of

analyses that may not be possible under real-time constraints.

The BMS consists of two distinct subsystems. The GPS/INS subsystem is responsible for

measuring vehicle position in GPS coordinates. The next section covers the Inertial Navigation

System and the Differential GPS separately. The focus of the descriptions is to present the

operating characteristics and the accuracy of each component as if they were used individually.

The section on the Kalman filter describes how the INS and GPS measurements are combined

together to achieve a measurement system that exhibits the best characteristics of each.

The second subsystem of the BMS is the video/data digital recording system (VDDRS). The

VDDRS is based on a ruggedized, off-the-shelf, Silicon Graphics Inc. O2 computer system. The

O2 records camera input and data from the GPS/INS subsystem onto an array of disks. The

camera is calibrated so that points in the image can be transformed into GPS coordinates. A

variety of analysis software exists that automatically locates a lane marker in a video frame,

detects lane crossings, and plots the results of an evaluation. The camera calibration and the

digital recorder are covered in the remaining portions of this section.

3.1 Inertial Navigation System

The U.S. Army’s Modular Azimuth and Positioning System, MAPS2

, is an inertial navigation

system containing three ring laser gyros, three accelerometers, and a rear axle odometer. MAPS

can supply orientation and translation data at 5 Hz. A Navigation Interface Unit (NIU),

previously developed by Alliant Tech, is used to increase the update rate. The NIU requests

orientation data from the MAPS, which is available by itself at 25 Hz, and taps into the odometer

to measure the distance traveled. It combines the odometer and heading data to generate vehicle

position data at 25 Hz.

The MAPS/NIU position measurement drifts as the vehicle moves, due mainly to wheel slippage.

Our estimate of the drift rate of the system on pavement and in dry grass is slightly more than

0.1% of the distance traveled. The estimate was obtained by first marking on the pavement the
- 11 -

initial location of the tires and recording the vehicle position reported by the system. The vehicle

was driven along a course and returned as close as possible to the initial tire locations. The

output of the system was then compared to the initial vehicle position and the difference was

calculated to be 0.1% of the distance traveled. The runs were not repeated enough to obtain a

useful standard deviation of the estimate. Because the system uses odometry obtained from the

wheels, the system drifts more when driving on slippery surfaces and steep slopes.

Figure 2 shows the inertial dead reckoning system drifting as the vehicle is driven around a

square track. The inertial system drifts to the north and to the west by a similar amount each

pass. In this example, the vehicle was driven through patches of snow causing the inertial dead

reckoning system to drift at 0.4% of distance traveled.

Inertial

Differential GPS

-20 0 20 40 60 80 100 120

 Easting (m)

120

100

80

60

40

20

0

-20

Northing (m)

Figure 2. Comparison between inertial sensor drift and GPS.

3.2 GPS

The satellite based Global Positioning System, GPS, has a stated accuracyii of 100 m in the standalone mode for civilian receivers and around 10 m to 30 m for stand alone military receivers. To

improve accuracy, differential GPS is used. Figure 3 shows the components of the differential

GPS for the NIST vehicle. A stationary receiver at a known location, in this case, located on top

of NIST Administration Building, transmits perceived position errors over a data radio to the

 ii These GPS accuracy figures are from the 1994 Federal Radionavigation Plan. The figures express the value of two

standard deviations of radial error from the actual antenna position to an ensemble of position estimates made under

specified satellite conditions.
- 12 -

moving GPS receiver onboard the vehicle. The vehicle receiver uses this data to improve its

position estimate. Using this technique, accuracies from several meters down to sub-meter can

be obtained. Even higher accuracy can be obtained by including carrier phase measurements in

the transmissions. Using long-term averaging and post processing, carrier phase differential GPS

can achieve accuracies of a centimeter or better while the receiver is stationary. An important

feature of carrier phase differential GPS, available within the last two years on commercial

systems, is the ability to resolve the cycle integer ambiguities of the carrier phase while the

remote receiver is moving. This allows the vehicle to lose satellite lock (caused by driving under

a bridge, near buildings, through foliage, etc.) and then reacquire phase lock on the fly. When

there are not enough satellites visible for GPS to calculate an accurate position, the GPS/INS

system relies solely on the inertial dead reckoning of the MAPS and NIU.

DGPS/INS 

System 

(VME)

MAPS 

INS

gyros 

odometry

data radio

DGPS 

Rx

gps antenna

data antenna

data radio

DGPS 

Rx

gps antenna

data antenna

Figure 3. Components of the GPS and INS.

3.3 Kalman Filter

Both the INS and the GPS systems have advantages and disadvantages. INS has a higher data

rate of 25 Hz, but it drifts. The GPS system does not drift significantly over time, but it has a

noise component, a slower data rate of 2 Hz, and is not always available. To get the best of both

systems, their outputs are combined using a complementary filter, shown in Figure 4.
- 13 -

Inertial 

Dead 

Reckoning

Differential 

GPS

Truth + Noise

Truth + Drift

+

–

Measured Drift = 

Drift – Noise Kalman 

Filter

Estimated 

Drift

Estimated 

Truth + –

2 Hz

25 Hz

2 Hz

25 Hz

Figure 4. A complementary filter is used to estimate the true position of the vehicle.

The vehicle is at some position we will call truth. Both navigation sensors try to measure this

but have some inherent error. The inertial dead reckoning system reports truth with an error we

will call drift, and the GPS also reports truth but with an error we will call GPS noise. Taking

the difference between the output of the two sensors cancels the true position and results in the

difference between the drift and the GPS noise. The Kalman filter tries to remove the noise and

estimate the drift. Subtracting this drift estimate from the inertial dead reckoning system output

yields an estimate of the true vehicle position. The Kalman filter runs once for each GPS update,

producing drift estimates at 2 Hz. However, truth is estimated at 25 Hz using the current inertial

dead reckoning data and latest drift estimate. In general, drift changes slowly, allowing the

calculation of position between GPS updates and during short GPS dropouts.

The Kalman filter uses a simple model for drift that assumes the drifts in Northing and in Easting

are independent of each other. This results in two separate scalar filters. Recall that the filter is

estimating drift and not position. The state and output model is

d

n +1 = dn

+ w

n

z

n = dn

+ v

n

( 3-1 )

where

dn is the drift at time n,

wn is the system noise with covariance qn,

zn is the measured drift (zn = Inertial − GPS), and

vn is the measurement noise with covariance r

n.

The system noise wn models wheel slippage and other factors that cause the drift to change. As

drift tends to increase with the distance traveled, the covariance of the system noise, qn, is

modeled as being proportional to the distance traveled since the last GPS update. The

measurement noise vn is the GPS noise. Its covariance, rn

, is modeled as a constant.
- 14 -

The Kalman filter reduces to the following scalar equations (one set for Northing and an identical

set for Easting)

Kn = pn pn + r

n ( )

ˆ d n +1 = ˆ d n + Kn zn − ˆ d ( n )

pn+1 = pn ( ) 1− Kn + qn

( 3-2)

where

is the estimated drift with an error covariance . ˆ

is the Kalman filter gain, and

n n

n

d p

K

The recursive filter is started with an initial guess for ˆ d o and po.

The position of the vehicle is then estimated as

n n dn Truth Inertial ˆ ~ = − ( 3-3 )

Measured Drift

Estimated Drift

0 100 200 300 time (s)

6

5

4

3

2

1

0

-1

Drift & Noise (m)

Estimated GPS Noise

Figure 5. The Kalman filter computes estimated drift and noise in the Northing direction.

Figure 5 show the results of the Kalman filter for the first three laps of the square course shown

in Figure 2. The GPS receiver is operating in the standard (non-carrier phase) differential mode,
- 15 -

which provides position data to 1 meter (as claimed by the manufacturer). Both the measured

drift, z, and the estimated drift, ˆ d , increase in magnitude with time. During each lap the vehicle

drifts in a similar direction at the same location producing a repetitive pattern about every 100 s.

The sensor noise, i.e., the GPS noise, cannot be determined exactly, but it can be estimated as (z

– ˆ d ). Note the peaks at 50 s, 150 s, and 250 s. These were caused when the vehicle slipped on a

snow patch causing a large change in drift that the filter erroneously attributed to GPS noise.

This mistake is corrected in a few seconds as further GPS updates corroborate the large change in

drift.

3.4 Camera Calibration

The BMS camera is used to make measurements of features on the road. The camera calibration

is the process of establishing the transformation of points in the image into the corresponding

points on the road surface. In the general case, camera calibration is a difficult problem. The

calibration process can be greatly simplified for the case where the camera is fixed on the vehicle

and is pointed relatively straight down on the road surface. The picture in Figure 6 shows how

the camera is mounted to the vehicle for the tests conducted in this report.

camera in housing

Figure 6. Picture of downward pointing camera mounted on side of vehicle.

The calibration process is divided into two phases. In the first phase, a look-up table is built that

defines a transformation from points in the image plane, expressed in pixel coordinates, into
- 16 -

points on a local coordinate system on the road surface, expressed in millimeters. In the second

phase of calibration, the transformation between the local road-surface coordinate system and the

GPS coordinates of the vehicle is established. Once the calibration is completed, a routine can be

called which transforms (i,j) camera points into (North, East) GPS points.

3.4.1 Local Road-Surface Calibration

The calibration for the local road surface establishes a transformation between image points and

road-surface points. A simple calibration is a scale factor that converts pixels into millimeters.

A single scale factor does not accurately model the transformation from image coordinates to

world coordinates. A lookup table calibration is used in the BMS to store scale factors

throughout the image plane. The calibration is good for a single camera pose (position and

orientation) relative to the road surface. The accuracy of the calibration depends on the fact that

the camera pose and the road-surface orientation do not vary significantly. Besides camera pose,

other variables affect the calibration process. One variable is the lens. A wide-angle lens will

provide a greater coverage but will decrease the resolution, thereby changing the number of

millimeters per pixel (i.e., the scale factor). A narrow-angle lens has the opposite effect.

Another variable is the image format. When collecting data with the digital recorder, the user

can select the number of rows or columns of an image to save. For example, in order to save

disk space, every other row of the image may be skipped. In a full image, the 480 pixels in a

column may span approximately 1.25 m on the road surface. Skipping every other row results in

240 pixels spanning 1.25 m, thereby changing the calibration. In summary, the calibration must

be performed whenever the camera pose, lens, or image digitization format is changed.

Figure 7 shows a picture of the target used to calibrate the side-looking camera. The target

consists of a large, flat sheet that can be rolled out to accommodate varying fields-of-view. Since

the sheet is very thin, the surface beneath is also effectively calibrated once the calibration is

complete. The calibration target has a constant offset between control points, but the camera

pose, lens, and image format distorts the offset. An image processing program, written in

Khoros3

, computes the location of the centroids of each control point in the image. Figure 7

shows the output of the program. A white boundary around the perimeter of a control point

indicates the centroid was successfully detected. The output of the Khoros program is an ASCII

file containing the locations of the centroid of each control point.

x

y

j

i

Figure 7. Calibration target is used to define a local road-surface origin (x,y) at top left

control point.

The next step is to identify the control points in the image that correspond with the control points

on the calibration target. A C program is used for this step. Figure 7 shows the two coordinate

systems used for this step. The origin of the image frame is at the top, right portion of the image
- 17 -

(arrows labeled i and j). The origin of a local road-surface coordinate system is at the top, right

control point (arrows labeled x and y). The program automatically matches centroids in the

image with control points on the calibration target. It then assigns offsets in the x and y

directions (relative to the local road-surface coordinate system) to each centroid based on the

known offsets of the calibration target.

The last step in the local road-surface calibration is to build a lookup table that is used to

transform pixels points into road-surface points. The lookup table, a C data structure, is created

each time the system is initialized. Each node in the data structure contains the pixel coordinates

of a control point in the image, the (x,y) coordinates of the corresponding control point in the

local road surface coordinate system, and the coefficients for interpolating between pixels in the

horizontal and vertical directions. Once the lookup table is created, a lookup routine is called

whenever there is a need to transform an image point into a road-surface point. The lookup

routine searches the data structure to isolate the bounding region containing the pixel and then

uses linear interpolation to transform the point onto the road surface.

The camera calibration technique was evaluated to determine the FOV and the resolution in

transforming image points to road-surface points. Table 2 summarizes certain attributes of the

camera calibration performed for this report. Cam Ht is the height of camera (in meters) above

the road (measured by hand). Lens is the lens focal length (in mm) obtained from the lens body.

Lat (perpendicular to vehicle travel) and Long (parallel to vehicle travel) FOV are the maximum

coverage of the camera field-of-view expressed in meters. This is obtained using the calibration

lookup table. Max and Min Res are the distances (in mm) on the road surface subtended by a

pixel in the lateral direction (along the x axis in Figure 7). This varies with pixel location. This

is obtained using the lookup table.

Cam Ht m Lens mm Lat FOV m Long FOV m Max Res mm Min Res mm

1.375 6 1.7 1.25 1.86 3.28

Table 2. Camera calibration attributes.

3.4.2 Camera to GPS Calibration

To complete the camera calibration, it is necessary to derive the transformation from the local

road-surface coordinate system into the GPS coordinate system. Figure 8 shows some relevant

coordinate systems used for this camera calibration.
- 18 -

Forward x

y z

x

y

z

x

y

z

x

y

z

veh

niu

gps_ant

cam local

road

surface

north

east down

gps_world

niu, veh, cam and gps_ant are rigidly attached to vehicle.

niu and veh xy planes are coplanar with axles.

cam xy plane is coplanar with road.

vehicle

Veh-to-cam transform

Figure 8. Vehicle coordinate systems describing camera-to-GPS calibration.

The BMS obtains the location of the vehicle whenever a camera image is taken. The vehicle

location, shown as the coordinate system (CS) labeled veh, is relative to the gps_world CS. The

camera-to-GPS calibration determines the transformation from the veh CS to the cam CS

(labeled veh-to-cam transform in Figure 8). The x and y axes of the cam CS are identical to the x

and y axes of the local road-surface CS shown in Figure 7. The veh-to-cam transformation

depends on the position of the camera on the vehicle and the orientation of the camera with

respect to the vehicle forward axis. Note that the transformation does not define the position and

orientation of the camera on the vehicle. Once the veh-to-cam transformation is known, any

pixel in the camera can be transformed back to the vehicle and then into GPS coordinates.

Computing the veh-to-cam transformation requires that two points on the road be surveyed

manually using a GPS antenna. The position portion of the transformation only requires one

point, f, shown in Figure 9.
- 19 -

gps

veh

cam

f

g_f

c_f v_c

g_v

N

θ

φ

θ

E

y

y

x

x

Figure 9. Derivation of position transformation between the local road surface and the

GPS coordinate systems.

Figure 9 shows a top view of the coordinate systems shown in Figure 8. Below are the

definitions of the systems.

1. gps - the world CS as measured by the orbiting GPS satellites.

2. veh - a CS centered within the vehicle. The output of the GPS is the location of the veh CS

with respect to the GPS CS.

3. cam - a CS in the camera field-of-view whose origin is the local road-surface origin shown in

Figure 7 rotated so the x vector in the vehicle forward direction. The output of the lookup

function is expressed as a vector in the cam CS.

The following vectors are shown in Figure 9.

1. g_v - the position of the vehicle in GPS coordinates. This is given by the GPS.

2. c_f - the position of the point f in the cam CS. c_f = lookup(i,j) where (i,j) is the pixel

coordinates of point f lookup is a routine that returns the GPS coordinates of an image point

(i,j). The point is selected manually while viewing the image on a screen.

3. g_f - the position of point f derived by manually surveying the point using a GPS antenna

above the point.

4. v_c - The position of the cam CS with respect to the veh CS. This vector is the translation

portion of the GPS calibration, which is unknown and is the value to derive.

The following angles are also shown in Figure 9:

1. θ - the heading of the vehicle. This is derived from the GPS system and is with respect to

North.

2. φ - the angle of the cam CS with respect to the veh x axis. This angle is the orientation

portion of the GPS calibration, which is unknown and will be derived later.
- 20 -

The solution for the veh-to-cam transformation does not require the roll and pitch angles of the

vehicle. This is acceptable since the roll and pitch of the cam CS (i.e., the local road surface)

closely parallels the veh CS.

As can be seen from Figure 9, the vector g_f can be written as:

g_f = trans(g_v) + rotz( ) trans(v_c) + rotz(θ + φ) trans(c_f) ( 3-4 )

 [Note: The function rotz (angle) defines a rotation of angle about the z axis (positive direction

of z is out from the page) and the function trans(vector) defines a translation along the vector (the

magnitude of the translation is equal to the magnitude of the vector).] The translation vector v_c

can be derived from ( 3-4 ) as follows:

v_c = rotz(-θ )[ g_f - rotz(θ + φ) trans(c_f) - trans(g_v)] ( 3-5 )

The orientation portion of the camera calibration does not require the roll and pitch, only the

angle between the forward axes of the cam CS and the veh CS. The camera orientation can be

derived using points b and f in Figure 10.

gps

veh

cam

f

g_f

c_f

α

θ

N b

b_f

g_b

c_b

E

y

y

x

x

Figure 10. Two points are required to derive the rotation transformation between the local

road surface and the GPS coordinate systems.

The new vectors in Figure 10 are:
- 21 -

1. c_b - the position of the point b in the cam CS. c_b = lookup(i.j) where (i,j) is the pixel

coordinates of point b. The point is selected manually while viewing the image on a screen.

2. g_b - the position of point b derived by manually surveying the point using a GPS antenna

above the point.

3. b_f - the vector from point b to point f. The vector can be derived in either the GPS or cam

coordinate systems.

The new angle in Figure 10 is:

1. α - the angle of the cam CS with respect to North. This angle, as shown in Figure 9, equals θ

plus φ.

This leads to the following equation to be solved:

φ = α −θ ( 3-6 )

Figure 11 shows how the difference in the vectors as expressed in each coordinate system is used

to derive the rotation portion of the calibration.

gps

α cam

N

λ

γ

b_f gps

b_f cam

Figure 11. Derivation of rotation transformation between camera and GPS coordinate

systems.

The following vectors shown in Figure 11 can be derived using vector addition (see also Figure

10):

b_f g_f g_b

b_f c_f c_b

gps

cam

= −

= − ( 3-7 )

The equations in ( 3-7 ) can be used to define two additional angles:
- 22 -

1. γ - the angle of b_fgps with respect to North.

2. λ - the angle of b_fcam with respect to the forward axis of the cam CS.

These angles are derived by:

2 arctan( )

2 arctan( )

cam

gps

b_f

b_f

= −

= −

λ π

γ π

( 3-8 )

As can be seen in Figure 11, the angle λ represents the angle of b_fcam as seen through the

camera. The fact this differs from γ, the angle of the b_fgps on the road, is the result of the

rotation of the cam coordinate system by the angle α. Therefore,

γ = α + λ ( 3-9 )

Given the known values γ, λ, θ and equation ( 3-6 ) yields:

θ

φ γ λ θ

= − −

= − −

arctan( ) arctan( ) b_f cam b_f gps

( 3-10 )

Once the calibration has produced the position and orientation parameters, any point c_f =

lookup(i,j), can be transformed into Northing and Easting using equation ( 3-4 ).

3.5 Video Digital Recording System

The components of the Video Digital Recording System are shown in Figure 12. They consist of

a camera, a Silicon Graphics Inc. O2 computer, and a RAID (Redundant Array of Inexpensive

Disks) storage device. A picture of the system is shown in Figure 13. The O2 is used to digitize

video, read GPS/INS data, read data from a system under evaluation, and store all data onto the

RAID. The RAID consists of three 9-MB hard disks that operate in parallel but appear as a

single volume in the file system. The O2 is ruggedized to withstand shock and vibration

disturbances when the vehicle is driven off-road.
- 23 -

Host 

Processor

data 

storage 

(3 Disk RAID)

Video/Data Digital Recording System (SGI O2)

Ethernet/serial Ethernet

SCSI

System to 

evaluate 

(e.g., Autonav)

DGPS/INS 

System 

(VME)

side-looking 

camera

Figure 12. Components of the Video Digital Recording System.

Removable disk drives

Figure 13. Picture of the BMS Video/Data Digital Recording System.
- 24 -

Data collection consists of recording video and GPS/INS data simultaneously. A single C

routine is used to initialize the O2 video input, to read in each image, to read in the current

GPS/INS data, to format the image, and to write the video and GPS/INS data to disk.

The maximum recording bandwidth of the O2 is achieved by routing the video directly to a disk

buffer without accessing individual pixels. Any processor access to video memory, for example

to examine some pixel value, significantly reduces the recording bandwidth. The O2 is able to

record color video at 640x480 pixels at 30 frames per second, or at 320x240 pixels at 60 fields

per second. Occasional dropouts may occur. The color format uses 2 bytes per pixel. The

resultant bandwidth is (640x480x2x30/1024^2) • 17.6 Mb/s.
- 25 -

4 Evaluation of Baseline Measurement System

Several tests were conducted to evaluate the BMS and to show how the BMS might be used to

measure the performance of a countermeasure system. Below is a summary of some factors that

affect the accuracy of the BMS. Later in this section are the results of several tests conducted to

validate the BMS and to evaluate its performance.

GPS Receiver

Radio link dropout, signal multi-path, satellite geometry, and satellite-lock (i.e., the satellite is no

longer visible due to trees, building, etc.) all affect GPS accuracy. During dynamic

measurements (i.e., while the vehicle is moving), delays in processing and communications are a

significant source of error. During these tests, we relied on the real-time capabilities of the GPS

without any corrections or post-processing of the data.

Inertial Navigation System

In order to increase the update rate of the vehicle position measurement, only the INS gyros and

odometry are used. The drift in this mode of operation is primarily due to wheel slippage and is

generally a function of distance traveled. The drift is reduced whenever the GPS updates are

received. Thus, the predominate effects of INS drift are confined to the periods between the half

second GPS updates and when the GPS loses lock with the satellites.

Kalman Filter Algorithm

The Kalman Filter combines data between the GPS and the INS, so uncertainties will depend to a

large part on the uncertainties contributed by these components. In addition, a small error has

been identified due to the differences in the latencies of the GPS and INS. An algorithm has

been developed that appears to correct for these errors but the algorithm was not completely

implemented in time for these tests.

Camera Calibration

The camera calibration transforms pixel coordinates into GPS coordinates. There are two parts

to the transformation. The first part transforms pixels coordinates into x,y coordinates (in mm)

in a local road-surface coordinate system (i.e., the road in the camera FOV). The uncertainties in

this part of the calibration are due to the resolution of the camera, the distortion of the lens, the

camera orientation with respect to the road, and the density of the calibration target. The

resolution of the camera falls in the 2 mm to 3 mm range. The accuracy of this transformation is

difficult to isolate due to the large number of factors, but it can be assumed the uncertainty is

bounded from below by the resolution. The second part of the transformation converts local

road-surface coordinates into GPS coordinates. Since the GPS receivers are used to establish this

transformation, the accuracy of this transformation depends on the uncertainties of the GPS.

Test Procedures

The tests consist of running the vehicle over various parts of the NIST grounds and collecting

data using the BMS. The data for each run consists of video recorded at 60 Hz. and, for each

video frame, the corresponding data from the GPS, INS and the Kalman filter. The video data

and the position (or navigation) data are stored in individual files. A naming convention
- 26 -

simplifies keeping track of the video and data. Table 3 shows the video file names. The file

name contains the date and time the video was collected, the format (in this case 60 Hz field

video) and the dimensions of each frame. The names of the position data files corresponding to

the video are shown in Table 4. By looking at the “time” field of the file name, one can see that

the total time required to collect six runs of data was approximately 13 min (from hr-mm-ss of

10-39-30 to 10-52-56). This illustrates the ease in which data can be collected using the BMS.

Size (bytes) File Name (_date_time_hzxformat_colsxrow)

196,455,600 _04-03-98_10-39-30_60xFI_720x121.raw

647,388,720 _04-03-98_10-41-40_60xFI_720x121.raw

621,078,480 _04-03-98_10-44-28_60xFI_720x121.raw

664,638,480 _04-03-98_10-47-46_60xFI_720x121.raw

616,809,600 _04-03-98_10-50-17_60xFI_720x121.raw

1,259,493,840 _04-03-98_10-52-56_60xFI_720x121.raw

Table 3. Video files recorded for test.

Size (bytes) File Name (_date_time_hzxformat_colsxrow)

505120 _04-03-98_10-39-30.nav

1664544 _04-03-98_10-41-40.nav

1596896 _04-03-98_10-44-28.nav

1708896 _04-03-98_10-47-46.nav

1585920 _04-03-98_10-50-17.nav

3238368 _04-03-98_10-52-56.nav

Table 4. Navigation data files corresponding to video files in Table 3.

A reference for comparison was established by inserting nails into the road asphalt. The nails

were surveyed manually using the GPS. The survey involved long-term averaging and the use of

carrier phase corrections. The uncertainties are believed to lie in the 1 cm to 2 cm range. The

nails were highlighted, see Figure 14, with black and white paint. The highlighted nails make it

easy for the analyst to select the nail in the image. The nails provide a reference point for

evaluating the BMS. At any video frame where the nail is visible, the nail can be selected and

transformed into GPS space. The coordinates of the nail acquired visually can be compared to

the manual survey measurements to determine uncertainties in the BMS. The remainder of this

section covers the tests and the results.

survey nail location

survey nail location

Figure 14. A single frame recorded by BMS showing two survey nails highlighted using

black and white paint.
- 27 -

4.1 Verification of Camera Calibration

The goal of this test was to establish the correctness of the camera calibration software. The

vehicle was driven to a point where a survey nail was visible in the video. The BMS recorder

was turned on and the vehicle was driven through a half figure eight and stopped facing in the

opposite direction with the survey nail again visible in the video. Figure 15 shows the trajectory

of the vehicle and the locations of the nail acquired visually using the BMS. The unfiltered_gps

line is the raw GPS data at half-second updates, the filtered_gps is the output of the Kalman

filter, and the ins is the output of the MAPS.

start

unfiltered_gps

filtered_gps

ins

filNail1

filNail2

unfilNail1

unfilNail2

Half Figure 8 - verify nail locations 

 (script: /home/timon/proj/4-3-98/graphHalfFig8_1.tcl)

 (data: _04-03-98_10-39-30.nav)

e (meters)

-10 0 10

n (meters) -20

-10

0

Figure 15. Vehicle driven along a half figure-eight trajectory along with survey nails

acquired using video at beginning and end (see also Figure 16).

Acquiring the nail in both direction exercises the calibration transformations from the camera’s

FOV to GPS space. If the calibration parameters and the transformations are correct, the nail

should align at the beginning and at the end, and should coincide with the surveyed coordinates

of the nail. Figure 16, a zoomed in view of the nails in Figure 15, shows the coordinates of the

nail acquired in the first (legend titles ending in 1) and last (legend titles ending in 2) video

frame. The coordinates of the nail are calculated using the vehicle’s filtered and unfiltered GPS

coordinates. The origin of the graph is the surveyed location of the nail. Two of the points align
- 28 -

very closely with the origin. The locations of the nails and their distance from the surveyed nails

are listed in Table 5.

filNail1

filNail2

unfilNail1

unfilNail2

Half Figure 8 - nails only

 (script: /home/timon/proj/4-3-98/graphHalfFig8_2.tcl)

 (data: _04-03-98_10-39-30.nav)

e (meters)

-0.1 0 0.1

n (meters) -0.2

-0.1

0

Figure 16. Filtered and unfiltered (raw) GPS coordinates of nail at beginning and end of

half figure 8 trajectory.

Data Point Name North (m) East (m) Distance From Origin (cm)

filNailN1 -0.096 0.033 10.2

filNailN2 0.005 -0.023 2.4

unfilNailN1 0.021 -0.011 2.4

unfilNailN2 -0.187 -0.023 18.8

Table 5. Values of visually acquired nails and their distance from the origin (i.e., the

known survey location of the nail).

4.2 Static Measurements

This test seeks to establish the uncertainty of the BMS in measuring the position of a survey nail

using the video system and the GPS while the vehicle is static (i.e., stationary). Several runs

were made on the NIST grounds. Figure 17 shows the typical path in GPS space the vehicle took

during a run.
- 29 -

vehicle

n_vs_e

e (meters)

308600 308800 309000

n (meters) 4333200

4333400

4333600

start

East Drive

South Drive

Gate F

Figure 17. Trajectory of vehicle during test to acquire nail at beginning and end of run.

The survey nail was acquired visually at the beginning and end of each run. The location of the

nail was obtained after the vehicle was stopped and in only one video frame. Figure 18 and

Table 6 show the measured locations of the nail. Only runs where satellite-lock was maintained

are included in this test. Again, the origin is the surveyed location of the nail. The coordinates

of the visually surveyed nail are computed using the unfiltered (raw) GPS data in order to isolate

the errors due to the INS and the Kalman filter. The majority of the points are close to the origin.

It is unknown what caused one nail to appear 53.905 cm from the origin. The standard

uncertainty (i.e., the standard deviation) over all the measurements in Table 6 is +/- 18 cm. The

expanded uncertainty4

 (based upon a t-distribution and 95% confidence level) is +/- 42 cm. This

is within the stated specifications of the manufacturer5
- 30 -

u_04_03_98_10_39_30_0

u_04_03_98_10_39_30_2254

u_04_03_98_10_41_40_0

u_04_03_98_10_41_40_7430

u_04_03_98_10_47_46_0

u_04_03_98_10_47_46_7628

u_04_03_98_10_50_17_0

u_04_03_98_10_50_17_7079

Start/End Nail Location using straight GPS without dropout

 (script: /home/timon/proj/4-3-98/graphAllNailsOK.tcl)

e (meters)

-0.4 -0.2 0 0.2 0.4

n (meters) -0.6

-0.4

-0.2

0

Figure 18. Results of visually acquiring survey nail at the beginning and end of several

runs.

Data Point Name North (m) East (m) Distance From Origin (cm)

u_04_03_98_10_39_30_0 0.012 -0.014 1.822

u_04_03_98_10_39_30_2254 -0.187 -0.024 18.887

u_04_03_98_10_41_40_0 -0.537 0.052 53.905

u_04_03_98_10_41_40_7430 0.026 -0.013 2.870

u_04_03_98_10_47_46_0 0.050 -0.025 5.593

u_04_03_98_10_47_46_7628 0.041 -0.028 4.946

u_04_03_98_10_50_17_0 0.040 -0.029 4.905

u_04_03_98_10_50_17_7079 0.030 -0.033 4.481

Table 6. Visual-based measurements of nail (see Figure 18) with distance from manual

surveyed coordinates of the nail (origin).

4.3 Dynamic Measurements and Latency Calculation

A dynamic GPS measurement, as opposed to a static measurement, is obtained while the vehicle

is moving. Latency in obtaining the GPS measurement from the receiver is one source of

uncertainty when making dynamic measurements. An approach to determine latency is to

compare the coordinates of a survey nail obtained visually while driving against the coordinates

of the survey nail obtained manually by survey.

To obtain a dynamic measurement, a survey nail was visually captured using the camera while

the vehicle was driven along a road. Figure 19 shows only the section of the vehicle’s trajectory

as it drives by the surveyed nail. The vehicle’s trajectory, the circles connected by lines, is
- 31 -

labeled as filtered_gps. Each circle represents a Kalman filter update. The manually surveyed

coordinates of the nail, the plus sign labeled surveyedNail, is shown at the origin. The nail

acquired visually using the camera and GPS filtered data is shown as the circle labeled

videoNail.

start

filtered_gps

surveyedNail

videoNail

Half Figure 8 - verify nail locations 

 (script: /home/timon/proj/4-3-98/graphNailDelay.tcl)

 (data: _04-03-98_10-41-40.nav)

e (meters)

-1 0 1 2 3

n (meters) - 4

- 3

- 2

- 1

0

Figure 19. The error due to latency is shown in the discrepancy between the coordinates of

the surveyed nail (surveyedNail) and the visually acquired nail (videoNail) as the vehicle

(filtered_gps) drives by at 10.75 m/s.

Two methods are used to calculate the latency. One method uses the relationship velocity =

distance / time. Let distance equal the difference between the nail’s surveyed coordinates and the

nail’s coordinates measured visually. Let velocity equal the speed given by the odometer (10.75

m/s). Solving for time in the velocity relationship yields a latency of 316 ms.

To calculate the latency in a different manner, each point along the vehicle’s trajectory is

examined and the point falling closest to the survey nail is identified. For this data set, the

vehicle passed closest to the nail’s coordinates at GPS data record 837 (this data record

corresponds with video frame 837, which is labeled “start” in Figure 19). The nail was visually

located in video frame 852 (this corresponds to GPS data record 852). Each record is time

stamped and subtracting the differences between the time stamps yields a latency of 300 ms.

This compares closely with the previously mentioned latency estimate of 316 ms. These

methods show how a latency calculation might be made but additional measurements need to be

made prior to obtaining an uncertainty value for the estimate.

The GPS receiver manufacturer claims their fast algorithm for solving carrier phase GPS location

produces a solution in less than 250 ms. The additional latency measured also takes into account
- 32 -

the delays in communications between the BMS and the GPS receiver, and any internal

processing in the BMS.

A 300 ms latency increases the uncertainty in the vehicle’s position proportional to velocity. For

example, at 30 m/s (roughly 60 miles/hr), the latency increases the uncertainty by 9 meters. For

longitudinal measurements, this uncertainty is significant. For lateral measurements, since the

velocity approaching the lane marker is small, the uncertainty is smaller. One approach to reduce

the uncertainty, not examined for this report, is by extrapolating vehicle position based on its

previous trajectory. This partially depends on a small variance in latency, which again suggests

additional measurements need to be made.

4.4 Affects of GPS Dropout

One of the main sources of GPS errors is the loss of satellite-lock. During two of the test runs,

satellite loss occurred and the BMS relied on the INS for position measurement. Figure 20

shows a run starting at the intersection of East and South Dr. on the NIST grounds. The vehicle

was driven onto Muddy Branch Rd., back into NIST, along Center Dr., and returned to the start

location. To simplify reading the axis scales, the origin of the graph is offset from absolute GPS

coordinates to the start location of the vehicle. The run took approximately 4.5 min and covered

a distance of 2885 m.

0:cpDGPS

9159:DR

9495:DGPS

9813:DR

10632:DGPS 13139:cpDGPS

unfiltered_gps

filtered_gps

ins

0

9159

9495

9813

10632

13139

Example of gps dropout 

 (script: /home/timon/proj/4-3-98/graphDropout3.tcl)

 (data: _04-03-98_10-52-56.nav)

e (meters)

-500 0 500

n (meters) -1000

-500

0

Muddy Branch Rd

East Dr

Center Dr

South Dr

a

b c

Figure 20. A run out the NIST gate onto Muddy Branch Rd and then back to the start

showing mode changes in the navigation system (DR = DeadReckoning, DGPS =

Differential GPS, cpDGPS = carrier phase DGPS).
- 33 -

The circles indicate where the navigation system changed modes. Each circle is labeled with the

new mode and the video frame number where the mode change occurred. At the start of the run,

the navigation system was in carrier phase DGPS mode. This mode is sustained until the vehicle

was driven beneath trees (at frame 9159) and satellite-lock was lost. At this point, the navigation

system mode changed to dead reckoning (DR) and the system relies on the INS for position

measurement. Satellite lock was obtained a short distance later (frame 9495) but lost again

(frame 9813). At frame 10632 the satellites were reacquired and carrier phase was established

near the end of the run at frame 13139. The plot and mode changes in Figure 20 were generated

automatically using the BMS analysis software.

Table 7 shows the elapsed time between navigation system mode changes. It took 5 seconds and

14 seconds to acquire satellites (time elapsed between DR and DGPS modes). The longer time

reflects the additional time required to clear trees and regain satellite line-of-site. Once satellitelock was acquired, it took 44 seconds (time elapsed between DGPS and cpDGPS modes) to

resolve the cycle integer ambiguity required for carrier phase operation. This agrees with the

manufacturer’s claim of less than a minute.

elapsed (sec) new mode frames

0 cpDGPS 0

163 DR 9159

5 DGPS 9495

5 DR 9813

14 DGPS 10632

44 cpDGPS 13139

Table 7. Elapsed time between navigation system mode changes.
- 34 -

Figure 21 is an enlarged view of the vehicle trajectory during section “a” in Figure 20. The

trajectory appears distorted because the axes scales are different in order to maximize the range

of the plot. The figure shows how the output of the Kalman filter tracks the INS, offset by the

previous drift, when GPS lock is lost. The drift value remains constant while the GPS lock is

lost. Each circle in the figure represents a GPS update.

9

9159:

9495:DGPS

9813:DR

10632:DGPS unfiltered_gps

filtered_gps

ins

9094

9159

9495

9813

10632

Example of gps dropout 

 (script: /home/timon/proj/4-3-98/graphDropout4.tcl)

 (data: _04-03-98_10-52-56.nav)

e (meters)

-470 -460 -450 -440 -430

n (meters) -300

-200

-100

0

Figure 21. Vehicle position updates during section of trajectory labeled “a” in Figure 20.
- 35 -

Figure 22 is an enlarged view of the vehicle trajectory during section “b” in Figure 20. The

figure shows how the output of the Kalman filter converges to the GPS data once GPS satellite

lock is obtained.

10632:DGPS

unfiltered_gps

filtered_gps

ins

10632

Example of gps dropout 

 (script: /home/timon/proj/4-3-98/graphDropout5.tcl)

 (data: _04-03-98_10-52-56.nav)

e (meters)

-480 -460 -440 -420 -400

n (meters)

0

2 0

4 0

6 0

Figure 22. Vehicle position updates during section of trajectory labeled “b” in Figure 20.
- 36 -

Figure 23 is an enlarged view of the vehicle trajectory during section “c” in Figure 20. The

figure shows the position along the trajectory when the receiver achieved carrier phase lock. The

Kalman filter converges even further when the vehicle slows to a stop.

13083:DGPS13139:cpDGPS

unfiltered_gps

filtered_gps

ins

13083

13139

Example of gps dropout 

 (script: /home/timon/proj/4-3-98/graphDropout6.tcl)

 (data: _04-03-98_10-52-56.nav)

e (meters)

-30 -20 -10 0

n (meters) -10

0

1 0

2 0

3 0

Figure 23. Vehicle position updates during section of trajectory labeled “c” in Figure 20.
- 37 -

Finally, in Figure 24, the locations of the nail have been picked out from the video and

transformed into GPS space. The origin of the graph is the manually surveyed location of the

nail. The circles are the locations of the nail in the first and last (14456) video frame using the

unfiltered (raw) GPS data. The pluses are the nails based on Kalman filtered output. Both sets

of data are within 10 centimeters of the nail.

f_04_03_98_10_52_56_0

u_04_03_98_10_52_56_0

f_04_03_98_10_52_56_14456

u_04_03_98_10_52_56_14456

All Nails

 (script: /home/timon/proj/4-3-98/graph10-52-56Nails.tcl)

e (meters)

-0.1 0

n (meters) -0.1

0

Figure 24. Nail locations, derived from navigation system (filtered_gps) and raw GPS

(unfiltered_gps) data, at the beginning and end of run in Figure 20
- 38 -

4.5 Lane Position Measurements

This test shows how the BMS can measure the location of the passenger-side lane marker while

the vehicle is moving. A sequence of the recorded video data is processed and the lane location

in each frame is extracted automatically using an adjustable template correlation function

developed at Carnegie Mellon University6

. The lane location is transformed into absolute GPS

space. The lane and the vehicle locations are shown in Figure 25. The analysis software

automatically labels the location of the minimum and maximum distances between the vehicle

and the lane. The software can be extended to also flag when the vehicle crosses the lane.

max: 1.095, 3667

min: 0.070, 4070

lane

vehicle

Track Lane 

 (script: /home/timon/proj/4-3-98/graphTrackLane.tcl)

 (data: )

e (meters)

308740 308760 308780 308800

n (meters) 4333180

4333200

4333220

4333240

Figure 25. Trajectory of vehicle and location of outside lane marker shown with minimum

and maximum distance between vehicle and lane labeled (min/max: distance from lane in

meters, GPS record number).
- 39 -

5 Conclusion

The thrust of the AUTONAV / DOT project has been to develop a quantitative means for

measuring the performance of roadway departure warning systems. A series of performance

measures describing the reliability of a warning system has been defined. A baseline

measurement system (BMS) has been developed to gather, process and analyze data that is then

used to compute the performance measures.

The BMS uses a carrier phase differential GPS receiver and an inertial navigation system (ring

laser gyros and odometry) to measure vehicle position and a calibrated camera system to measure

the vehicle’s position relative to a lane marker. An interface is developed between the BMS and

the warning system so that the output of the warning system can also be recorded. The output of

the warning system, the camera and GPS is synchronized and stored digitally onto a large array

of disks in real-time (60 Hz) during normal driving scenarios. Afterwards, the data is analyzed to

determine when the vehicle departs a lane. The time of departure is compared against the output

of the warning system. True positive and true negative outcomes indicate reliable operation,

while false positive and false negative outcomes indicate unreliable operation. The tally of the

outcomes is used to produce reliability, failure and false alarm rates, which are the primary

performance metrics.
- 40 -

Appendix A: Autonav Program Description 

The goal of the U.S. Department of Defense, AUTONAV (Autonomous Vehicle Navigation

Control System) project is to develop a control system and sensor suite that when mounted on a

vehicle will enable the vehicle to autonomously navigate. The vehicles will be used to perform

scouting and reconnaissance missions without putting the soldier in danger. The project team is

integrating and evaluating several sensor and control system components that may be of use to

the Department of Transportation. Below is a summary of the main components.

UBM 4D Perception and Control System

The UBM (Universität der Bundeswehr Munchen) 4D Perception and Control System7

 is a

system capable of a wide range of autonomous driving functions. The UBM system has

demonstrated the ability to drive autonomously along a highway, to perceive traffic in front and

to the sides, to maintain speed and spacing with a vehicle in front, and to pass vehicles when

appropriate. A core component of the UBM system is the Road Detection and Tracking (RDT)

module. The RDT is capable of looking forward, detecting lane markers, tracking the lane

markers, and computing lane curvatures. The system is currently undergoing conversion to run

on standard PC hardware and will be integrated on the NIST vehicle and evaluated in the 1999

time frame.

DSRC Vision Front End

The David Sarnoff Research Center VFE8

 is a special-purpose vision processor that can perform

complex operations on video in real-time. Two of the primary functions of the VFE for

AUTONAV is image stabilization and stereo processing. Image stabilization is done

electronically by detecting global motion in the image and then warping the image to subtract out

the motion. Stabilization is an important preprocessing step to any countermeasure system based

on video. Stabilization helps simplify image processing algorithms and improves accuracy.

Stereo processing takes images from two cameras and produces a single range image. Each pixel

in the image is the range along the two cameras’ view axis. Range images form the basis for

collision detection and countermeasure systems. The system will be integrated on the vehicle

and tested in the 1999 time frame.

EBK LADAR

EBK9

 (Entfernungs-Bild-Kamera) LADAR, manufactured by Dornier, uses a laser and a two

degree-of-freedom scanning mechanism to collect range measurements. The ladar produces

128x64 (horizontal x vertical pixels) range image once each second, though each pixel is

available immediately upon measurement. The maximum range is 50 m and the range resolution

is 6 cm. The ladar produces range images similar to a stereo system but with different

characteristics. The primary drawback is the time it takes to scan the laser mechanically and to

make the measurements. The advantage of ladar over stereo is the ability to produce more robust

readings under a wider variety of conditions. Tests are now being conducted in the AUTONAV

project to determine the performance of stereo systems versus ladar. NIST is also looking at

flash ladar systems that do not require a mechanically scanned laser. A flash ladar illuminates
- 41 -

the entire scene and measures range simultaneously throughout the image. Typical flash ladars

currently under development have the ability to produce up to a 320x240 range image at 10 Hz.

The range images from a ladar form the basis for collision detection and collision avoidance

countermeasure systems.
- 42 -

References 

 1

 K. Murphy, et al., “Ground Vehicle Control at NIST: from Teleoperation to Autonomy”, Seventh Annual Space

Operations, Applications, and Research Symposium, Houston, TX, August 3-5, 1993.

2

 H-726 Modular Azimuth Position System Technical Description, December 1992, Honeywell Avionics Division,

St. Petersburg, FL.

3

 K. Konstatinides and J. Rasure, "The Khoros Software Development Environment", Journal of Visual Languages

and Computing, pp. 217-246, 1991.

4

 B. N. Taylor and C. E. Kuyatt, “Guidelines for Evaluating and Expressing the Uncertainty of NIST Measurement

Results”, NIST Technical Note 1297, 1994 Edition. http://physics.nist.gov/cuu/Uncertainty/index.html.

5

 Ashtech Supplement to Z-12 Receiver Operating Manual Covering RTZ Functions, March 1995. Phone 1-800-229-

2400.

6

 M. Chen, D. Pomerleau, T. Jochem, “Aurora: A Vision-Based Roadway Departure Warning System ”, Proceedings

of IEEE International Conference on Intelligent Robots and Systems, Pittsburgh, PA, August 1995. Also appears in

“Run-Off-Road Collision Avoidance Using IVHS Countermeasures, Task 3, Volume 1”, Contract No. DTNH22-93-

C-07023, USDOT National Highway Traffic Safety Administration, Office of Collision Avoidance Research, August

1995, pg. 14.

7

 E.D. Dickmanns, et al.,"The Seeing Passenger Car 'VaMoRs-P'", International Symposium on Intelligent Vehicles

'94, October 24-26, 1994, Paris.

8

 P.J. Burt, P. Anandan and K.J. Hanna, "An Electronic Front End Processor for Active Vision", SPIE Conference on

Intelligent Robotics, November 1992.

9

 Dornier GmbH, Box 1420, 7990 Friedrichshafen 1, Germany. Phone: 07545-89969.