# ![](media/image1.png){width="1.6618110236220471in" height="1.3929133858267717in"}![](media/image2.jpeg){width="3.1992125984251967in" height="2.011023622047244in"}![](media/image3.jpeg){width="3.1992125984251967in" height="2.098031496062992in"}![](media/image4.jpeg){width="3.1992125984251967in" height="2.042913385826772in"}  {#section .TOC .unnumbered}

#   {#section-1 .TOC}

Form DOT F 1700.7 (8-72)

+-----------------------+-----------+-----------+-----------+----------+
| **1. Report No.**     | **2.      |           | **3.      |          |
|                       | G         |           | Re        |          |
| FHWA-HOP-07-123       | overnment |           | cipient's |          |
|                       | Accession |           | Catalog   |          |
|                       | No.**     |           | No.**     |          |
+-----------------------+-----------+-----------+-----------+----------+
| **4. Title and        |           |           | **5.      |          |
| Subtitle**            |           |           | Report    |          |
|                       |           |           | Date**    |          |
| Transportation        |           |           |           |          |
| Management System     |           |           | September |          |
| Performance           |           |           | 2005      |          |
| Monitoring,           |           |           |           |          |
| Evaluation, and       |           |           |           |          |
| Reporting --- A       |           |           |           |          |
| Primer for            |           |           |           |          |
| Practitioners         |           |           |           |          |
+-----------------------+-----------+-----------+-----------+----------+
|                       |           |           | **6.      |          |
|                       |           |           | P         |          |
|                       |           |           | erforming |          |
|                       |           |           | Org       |          |
|                       |           |           | anization |          |
|                       |           |           | Code**    |          |
+-----------------------+-----------+-----------+-----------+----------+
| **7. Authors**        |           |           | **8.      |          |
|                       |           |           | P         |          |
| Byungkyu (Brian) Park |           |           | erforming |          |
|                       |           |           | Org       |          |
|                       |           |           | anization |          |
|                       |           |           | Report    |          |
|                       |           |           | No.**     |          |
+-----------------------+-----------+-----------+-----------+----------+
| **9. Performing       |           |           | **10.     |          |
| Organization Name and |           |           | Work Unit |          |
| Address**             |           |           | No.       |          |
|                       |           |           | (TRAIS)** |          |
| University of         |           |           |           |          |
| Virginia              |           |           | 11. *     |          |
|                       |           |           | *Contract |          |
| Department of Civil   |           |           |     or    |          |
| Engineering           |           |           |     Grant |          |
|                       |           |           |     No.** |          |
| PO Box 400742         |           |           |           |          |
|                       |           |           | DTFH61-0  |          |
| Charlottesville, VA   |           |           | 1-C-00180 |          |
| 22904-4742            |           |           |           |          |
+-----------------------+-----------+-----------+-----------+----------+
| **12. Sponsoring      |           |           | **13.     |          |
| Agency Name and       |           |           | Type of   |          |
| Address**             |           |           | Report    |          |
|                       |           |           | and       |          |
| Federal Highway       |           |           | Period    |          |
| Administration        |           |           | Covered** |          |
|                       |           |           |           |          |
| U.S. Department of    |           |           | Primer    |          |
| Transportation        |           |           |           |          |
|                       |           |           | **14.     |          |
| 1200 New Jersey Ave., |           |           | S         |          |
| N.E. (HOP)            |           |           | ponsoring |          |
|                       |           |           | Agency    |          |
| Washington, DC 20590  |           |           | Code**    |          |
|                       |           |           |           |          |
|                       |           |           | HOP       |          |
+-----------------------+-----------+-----------+-----------+----------+
| **15. Supplementary   |           |           |           |          |
| Notes**               |           |           |           |          |
|                       |           |           |           |          |
| Raj Ghaman (COTM)     |           |           |           |          |
+-----------------------+-----------+-----------+-----------+----------+
| **16. Abstract**      |           |           |           |          |
|                       |           |           |           |          |
| This primer provides  |           |           |           |          |
| an overview of TMS    |           |           |           |          |
| performance           |           |           |           |          |
| measurement and       |           |           |           |          |
| introduces the reader |           |           |           |          |
| to the idea of        |           |           |           |          |
| establishing a        |           |           |           |          |
| performance           |           |           |           |          |
| measurement program;  |           |           |           |          |
| developing individual |           |           |           |          |
| and detailed          |           |           |           |          |
| performance measures  |           |           |           |          |
| for various functions |           |           |           |          |
| and sub-functions of  |           |           |           |          |
| a TMS; data           |           |           |           |          |
| collection,           |           |           |           |          |
| screening, archiving  |           |           |           |          |
| and processing        |           |           |           |          |
| efforts; and          |           |           |           |          |
| performance           |           |           |           |          |
| monitoring,           |           |           |           |          |
| evaluation and        |           |           |           |          |
| reporting. This       |           |           |           |          |
| primer principally    |           |           |           |          |
| addresses the         |           |           |           |          |
| equipment, systems,   |           |           |           |          |
| and programs involved |           |           |           |          |
| in a TMS that can be  |           |           |           |          |
| used to obtain the    |           |           |           |          |
| most efficient and    |           |           |           |          |
| desirable outputs and |           |           |           |          |
| outcomes.             |           |           |           |          |
+-----------------------+-----------+-----------+-----------+----------+
| **17. Key Words**     |           | **18.     |           |          |
|                       |           | Dis       |           |          |
| Transportation        |           | tribution |           |          |
| management system,    |           | St        |           |          |
| performance measures, |           | atement** |           |          |
| data collection, data |           |           |           |          |
| screening, archiving, |           | No        |           |          |
| data processing,      |           | rest      |           |          |
| performance           |           | rictions. |           |          |
| monitoring,           |           | This      |           |          |
| performance           |           | document  |           |          |
| evaluation,           |           | is        |           |          |
| reporting**\          |           | available |           |          |
| **                    |           | to the    |           |          |
|                       |           | public    |           |          |
|                       |           | from: The |           |          |
|                       |           | National  |           |          |
|                       |           | Technical |           |          |
|                       |           | In        |           |          |
|                       |           | formation |           |          |
|                       |           | Service,  |           |          |
|                       |           | Spr       |           |          |
|                       |           | ingfield, |           |          |
|                       |           | VA 22161. |           |          |
+-----------------------+-----------+-----------+-----------+----------+
| **19. Security        | **20.     |           | **21. No  | **22.    |
| Classif. (of this     | Security  |           | of        | Price**  |
| report)**             | Classif.  |           | Pages**   |          |
|                       | (of this  |           |           | N/A      |
| Unclassified          | page)**   |           | 25        |          |
|                       |           |           |           |          |
|                       | Unc       |           |           |          |
|                       | lassified |           |           |          |
+-----------------------+-----------+-----------+-----------+----------+
| **Form DOT F 1700.7   |           |           |           |          |
| (8-72)**              |           |           |           |          |
| **Reproduction of     |           |           |           |          |
| completed page        |           |           |           |          |
| authorized.**         |           |           |           |          |
+-----------------------+-----------+-----------+-----------+----------+

# Table of Contents {#table-of-contents .TOC}

[Introduction to Transportation Management Systems
1](#introduction-to-transportation-management-systems)

[Performance Monitoring, Evaluation, and Reporting
1](#performance-monitoring-evaluation-and-reporting)

[Performance Measurement Program 2](#performance-measurement-program)

[Purpose & Benefits 2](#purpose-benefits)

[Steps to Establish a Performance Measurement Program
4](#steps-to-establish-a-performance-measurement-program)

[Performance Measures 4](#performance-measures)

[Stakeholders 6](#stakeholders)

[TMS Performance Monitoring, Evaluation, & Reporting
8](#tms-performance-monitoring-evaluation-reporting)

[Evaluation 11](#evaluation)

[Root-Cause Analysis 13](#root-cause-analysis)

[Benefit Analysis 13](#benefit-analysis)

[Reporting 14](#reporting)

[Best Practice: WSDOT 16](#best-practice-wsdot)

[Best Practice: VDOT 18](#best-practice-vdot)

[References 19](#references)

## Introduction to Transportation Management Systems

Transportation engineering has evolved significantly in the recent past.
Along with the traditional solution of capacity enhancements and
construction of new roadway facilities, improving the operational
efficiency of the existing facilities is increasingly being pursued
throughout the nation and the world. The technology components used for
such operational improvements are usually referred to as Intelligent
Transportation Systems (ITS). Various ITS elements are often grouped
together to provide viable solutions to local transportation problems
and help build the visions of the communities. A Transportation
Management System (TMS) provides a broader context, and may be loosely
defined as the deployed form of ITS, which, along with the human
resource factor, contributes to transportation management. In
particular, TMS includes computer hardware, software, communications,
and surveillance technologies that service freeway and arterial systems.
A traffic management center (TMC) is the physical facility that houses
central equipment, software and personnel to monitor, control, and
operate one or more TMSs.

TMSs are categorized based on the type of functions they perform. The
TMSs controlled by transportation agencies at all levels (states,
metropolitan regions, counties, and cities) frequently focus on managing
three particular systems -- the freeways, the arterials, and transit.
Other specific types of TMSs include emergency management, toll ways,
etc. Each TMS is further composed of several different functions. For
example, a freeway TMS could include functions such as incident
management, special event management, ramp metering, work zone
management, etc. Depending on the size of the region under the scope of
a TMS, the number/type of activities and personnel involved in the
traffic management functions could be quite complex. There are mainly
three stakeholders extremely and directly interested in the degree of
performance of a TMS: the TMS manager, the traveling public, and the
elected officials.

## Performance Monitoring, Evaluation, and Reporting

Three important outputs for the successful implementation of a
performance measurement program are performance monitoring, evaluation,
and reporting. Performance monitoring is defined as a regular review of
metrics and budgets against the projections plan. Another way to define
performance monitoring is as an ongoing internal process of examining
the actual system condition through observed data. Performance
monitoring often involves the following three functions:

-   Examines the incoming streams of data.

-   Compares the incoming data to the expected data and the existing
    goals and policies.

-   Flags any outliers for further study (detailed evaluation).

An operator is likely to monitor traffic on some sections or corridors
on a minute-by-minute basis. A supervisor is likely to monitor traffic
at the network or regional level regularly, with emphasis on specific
sections on an ad hoc basis. A TMC manager is likely to review the
system- and policy-level attributes of the region on daily basis,
emphasising a specific day when necessary.

The next step in the performance measurement program is evaluation.
Evaluation is the process in which the collected data are analyzed and
the results are compared to benchmark performance measures. The purpose
of performance evaluation is to determine how well goals are met so that
appropriate changes can be made to the TMC operations (such as policy,
planning, maintenance, etc.). The evaluation process also is helpful in
selecting alternative procedures and refining management techniques.

The performance measures being monitored can raise three significant
concerns to an agency:

-   Were the correct investments made?

-   Are demands of the public and state legislature being met?

-   Are agency goals and objectives being met?

This step in the process also identifies areas for system improvement
and provides information that later will be reported to decision makers
and the public. An example is the Highway Performance Monitoring System
sponsored by Caltrans. The data collected from this system are used to
determine:

-   Allocation of funds to the policies.

-   Travel trends and future transportation forecasts.

-   Environmental Protection Agency (EPA) air quality conformity
    tracking.

Once the best procedures are determined from the evaluation process, the
results need to be reported. Reporting provides visual information about
the data. The frequency of reporting (i.e., weekly, quarterly, or
annually) varies from agency to agency. The report gives feedback on the
planning and decision-making process, on trends in system performance,
accomplishments, and areas of needed improvement. Using this
information, the analyst can recommend changes to policy goals and
objectives, performance targets, and performance measures.

## Performance Measurement Program

### Purpose & Benefits

All organizations, whether public or private, are interested in
developing and implementing effective performance measurement programs,
since it is only through such programs that organizations can maintain
efficiency. In a broad context, performance measurement is the use of
quantifiable indicators of program effectiveness and efficiency to
determine progress toward specific, predefined organizational goals and
objectives. Performance measures related to transportation systems can
be grouped into three categories: input, output, and outcome measures.
Input measures address the supply of resources available to implement a
program; output measures quantitatively address the delivery of
transportation programs, projects and services; and outcome measures
address the degree to which the transportation system meets policy goals
and objectives. A specific example is shown in Table 1.

**Table 1: Examples of Types of Measures (FHWA 2004)**

  -------------- --------------------------- --------------------------------
  **Measures**   **Traditional Capacity**    **Maintenance and Operations
                                             Oriented**

  **Input**      Capital projects budget     No. of incident response patrols

  **Output**     Miles of roadway built      Response time to incidents

  **Outcome**    Reduced miles of congestion Change in incident-related delay
  -------------- --------------------------- --------------------------------

Performance management encompasses setting the agency/program goals
appropriately and revising them as needed. Figure 1 illustrates some
important aspects of performance measurement and their relation to
performance management.

![](media/image5.wmf){width="4.947916666666667in"
height="3.8229166666666665in"}

**Figure 1: Linking Performance Management to Performance Measurement
(ICMA 2005)**

A TMS needs a performance measurement program for many reasons. Federal
and state statutes require the implementation of a performance
measurement program. Stakeholders such as legislative decision makers
and taxpayers are rightly inquisitive about how their tax dollars are
spent in the transportation domain. A performance measurement program
also is needed to provide performance trends over time and, ultimately,
strategic feedback for decision makers. Box 1 highlights the potential
benefits from a performance measurement program.

**Box 1: Potential Benefits of a Performance Measurement Program (NCHRP
Synthesis 311)**

The performance measurement process starts by defining the services that
the organization promises to provide, including the quality or level of
service (e.g., timeliness, reliability, etc.) that is to be delivered.
Performance measures are then used to prioritize projects, provide
feedback on the effectiveness of long-term strategies, refine goals and
objectives, and improve processes for the delivery of transportation
services. They reflect the concerns of the transportation service user
and system operator and provide useful information to managers and
decision makers on how well the system is performing.

### Steps to Establish a Performance Measurement Program

Several key steps must be completed to completely establish a
performance measurement program. These include:

-   Identify the vision, goals, and objectives of the agency. Involve
    stakeholders in defining these three items.

-   Identify intended uses and audiences.

-   Develop TMS performance measures and relate to respective programs.

-   Identify performance benchmarks.

-   Collect complete, accurate and consistent data and monitor this data
    in a way that supports decision making.

-   Analyze and evaluate data.

-   Report data to stakeholders in a useful manner.

-   Identify action areas and communicate them to stakeholders.

### Performance Measures

The range of needs and uses of potential performance measures must be
well understood before an agency can determine which performance
measures to implement. A good, worthwhile measure is defined clearly and
is directly related to predefined goals and objectives. It is
understandable, logical, and allows for repeatability. It also allows
for data collection and shows trends. Further, these measures should be
reviewed according to the criteria in Box 2.

**Box 2: Criteria for Defining Performance Measures (Hack, 2005)**

The following illustrate some examples of common TMS performance
measures:

#### Incident Detection Rate and False Alarm Rate (FAR)

These metrics are used to measure the performance of incident detection
algorithms. The detection rate can be taken as the percentage of
incidents detected by the software versus the number of incidents that
occur. The FAR can be taken as the percentage of false alarms versus the
number of tests run by the software. Factors that may affect the
performance of an incident detection algorithm include: the operating
conditions of the roadway (at or below capacity), the duration and
severity of the incident, the geometric characteristics of the roadway
(grade, change in the number of lanes, presence of ramps), weather
(including the condition of the road surface as wet or dry), detector
spacing, the location of the incident with respect to a detector, and
the diversity of the traveling vehicles.

#### Delay

This traffic performance metric measures added travel time caused by
congestion. It can be calculated as:

$$\begin{array}{r}
\text{Total Segment Delay }(\text{veh-min}) = \lbrack\text{Actual Acceptable Travel Time }(\text{min}) - \\
\text{Actual Travel Time }(\text{min})\rbrack \times \text{Volume }(\text{veh}),
\end{array}$$

$$\text{Total Delay }(\text{veh-min}) = \sum_{i = 1}^{n}\text{Segment Delay}_{i}$$

Equation 1

Acceptable travel time for expected conditions is generally based on the
posted speed limit but may "be calculated using a congestion threshold
speed established from local performance goals for mobility."
"Acceptable travel conditions" are usually free-flow (Federal Highway
Administration 2002).

Another method is to measure the divergence of the actual travel time
from the expected travel time. Equation 2 can be used to calculate delay
over a set of links assuming free-flow conditions.

$$D = \sum_{i = 1}^{n}{L_{i} \times F_{i}(t) \times \left\lbrack \frac{1}{V_{i}} - \frac{1}{f_{i}} \right\rbrack}$$

Equation 2

Where,

*L~i~* = The length of the *i*th segment holding the *i*th TMS, which
can be derived from adjacent TMS locations marked by milepost value.

*F~i~(t)* = The total volume at the ith TMS site for the specified
period t.

*f~i~* = The free-flow speed at the ith segment.

### Stakeholders

The Victoria Transport Policy Institute defines stakeholders as
"individuals or groups that are affected by a decision and have an
interest in its outcome." With respect to a TMS, stakeholders are
interest groups who benefit from, or are otherwise impacted by, a TMS
and its operations. Stakeholders include the following:

-   Agency management and staff.

-   Transportation professionals:

```{=html}
<!-- -->
```
-   Transportation providers.

-   Transportation system users.

```{=html}
<!-- -->
```
-   Citizens.

-   Elected officials.

-   Policy makers:

```{=html}
<!-- -->
```
-   FHWA.

-   State DOTs.

-   Metropolitan Planning Organization (MPO).

-   Municipalities.

-   Emergency Responders and Management.

Stakeholders are interested in performance measures and the associated
monitoring, evaluation and reporting processes for the following
reasons:

-   Improving transportation to serve people and commerce.

-   Improving management access to relevant performance data.

-   Improving agency efficiency and effectiveness in terms.

-   Returning on investment in transportation.

-   Efficient allocation of investment in transportation.

-   Accountability of the agency.

When establishing performance measures, it is imperative to involve
stakeholders such as those involved in freeway and signal systems,
planning operations, emergency management, and departments of
transportation. The stakeholders should be involved in each phase of a
performance measurement program, including the processes of defining
performance measures and how they are to be used. Stakeholder support is
critical for initial acceptance and continued success of the performance
measures. Without stakeholders considering the determined measures
appropriate, it is "impossible to use the results of the analysis
process to report performance and negotiate the changes needed to
improve it. Those who are expected to use the process to shape and make
decisions should be allowed to influence the design of the program from
the beginning." Those persons accountable for results but who are not
necessarily decision makers, such as data collectors, should be
involved. Their involvement is necessary to gain their support so that
they do not circumvent the process or its intended outcome.

A very important group of stakeholders are the citizens, or
transportation system users. Accordingly, agencies need to focus on
measuring citizen satisfaction and communicating these results to the
group. For example, customer satisfaction may be measured from the
results of focus groups and surveys. The city of Phoenix, Arizona, has
shown great success in involving stakeholders, decision makers and
citizens in its performance measurement program. Box 3 highlights this
best practice. Its tactics and strategies can easily be transferred to a
TMS performance measurement program.

**Box 3: Phoenix, AZ, Best Practice of Stakeholder, Decision Maker, and
Public Involvement**

## TMS Performance Monitoring, Evaluation, & Reporting

Performance monitoring is primarily an operational task that allows
real-time (or immediate) decisions to be made based on the up-to-date
information produced by the system. There are multiple purposes for
monitoring this information, such as:

-   Identifying transportation systems or corridors with poor
    performance.

-   Calculating the degree to which transportation facilities are
    meeting goals and objectives established for those facilities.

-   Determining specific areas of management programs or systems that
    require improvements.

An example of traffic performance monitoring comes from the Archived
Data Management System (ADMS) web site, which stores and allows access
to Virginia traffic data. Figure 2 shows graphs from the ADMS Daily
Report, which will be implemented in an upcoming build of ADMS. This
report provides a way for State transportation officials to monitor the
TMS status. The left graph, for instance, gives the speed index value,
the average percentage of the speed limit traveled on area freeways for
the previous week. The middle graph provides an updated incident count,
and the right graph shows the percentage of stations available to
collect data. Using these three performance measures, TMC officials can
monitor system mobility, safety, and the effectiveness of field
equipment.

![](media/image6.png)

Figure 2: ADMS Daily Report Graph Examples

Currently, TMC operators as well as the public can monitor the
performance of corridors and freeways through images obtained via
traffic cameras, where they are available over the Internet, cable TV or
other medium. For example, Figure 3 shows a screen shot of the New York
City TMC's Advanced Traveler Information System. The web site
([http://www.nytmc.org](http://www.nytmc.org/)) allows the user to view
streaming video or a still image from a number of New York area
intersections, providing the public with valuable, real-time traffic
conditions at points around New York City.

![](media/image7.png){width="5.375694444444444in" height="3.0in"}

Figure 3: Real-time Image of an Intersection from the ATIS of the NYC
TMC

The San Diego TMC displays another type of real-time information to both
the public and TMC operators. Figure 4 exhibits the TMC's real-time map,
which reports the current speed on any given section of highway or
freeway.

![](media/image8.png){width="5.749305555555556in"
height="3.1034722222222224in"}

Figure 4: San Diego TMC Real-time Map

In this example, the menu located on the left allows the user to select
a specific freeway and direction. Based on this selection, the current
traveling speeds at various points on the corridor are displayed on the
right side of the screen. A large, speed-based, color-coded map of the
area also is displayed in the center of the screen, illustrating where
construction will soon occur. In addition, the color-coded map can help
operators identify segments where sensors are not working properly or
extreme congestion is present.

For TMS or TMC managers, performance monitoring can be accomplished via
a regularly released report. Such reports update transportation
officials on the condition of specific TMS components (e.g., traffic
sensors, signals, etc.) and overall system performance. For instance,
officials in Northern Virginia are currently working to produce a daily
report on the condition of the high-occupancy vehicle (HOV) facilities
in the area, along I-95 and I-395. The report displays the previous
weekday's speed and volume data for both morning and afternoon peak
periods in the HOV lanes compared to the average speed and volume from
the previous month. Figure 5 shows a typical data table from this HOV
daily report.

![](media/image9.png){width="5.995138888888889in"
height="4.476388888888889in"}

Figure 5: Analysis for HOV 3+ Restrictions During Morning Peak in
Northern Virginia

### Evaluation

Performance evaluation is the analysis and manipulation of data to
determine the conditions and effectiveness of the TMS. Different
techniques, such as before-and-after comparisons and trend analyses, can
help TMCs assess their performance and the ultimate results of their
work.

Data analysis methods are an important part of performance evaluation.
Agencies should consider the following criteria when selecting an
analytic tool to evaluate their systems:

-   Identification of the analysis context for the task at hand (i.e.,
    planning, design or operations/construction).

-   Determination of the appropriate geographic scope or study area for
    the analysis, including isolated intersection, single roadway,
    corridor or network.

-   Capability of modeling various facility types, such as freeways,
    high-occupancy vehicle (HOV) lanes, ramps, arterials, toll plaza,
    etc.

-   Ability to analyze various travel modes, such as single-occupancy
    vehicle (SOV), HOV, bus, train, truck, bicycle, and pedestrian
    traffic.

-   Ability to analyze various traffic management strategies and
    applications such as ramp metering, signal coordination, incident
    management, etc.

-   Capability of estimating traveler responses to traffic management
    strategies including route diversion, departure time choice, mode
    shift, destination choice and induced/foregone demand.

-   Ability to produce direct output performance measures such as safety
    (crashes, fatalities), efficiency (throughput, volumes,
    vehicle-miles of travel (VMT)), mobility (travel time, speed,
    vehicle-hours of travel (VHT)), productivity (cost savings) and
    environmental (emissions, fuel consumption, noise).

-   Tool/cost effectiveness for the task at hand, mainly from a
    management or operational perspective. Parameters influencing
    cost-effectiveness include tool capital cost, level of effort
    required, ease of use, hardware requirements, data requirements,
    animation, etc.

Although there are numerous methods to analyze the data, this subsection
focuses on the more successful and frequently practiced techniques.
These techniques include before-and-after evaluations, benefit-cost
evaluations, analysis of trends, and comparison group evaluations.

#### Before-and-After

The most common method to evaluate the effectiveness is the
before-and-after evaluation. This methodology studies the transportation
network before and after the implementation of the new strategy or
system. The same performance measures are used in the "before" and
"after" conditions. An example of this type of evaluation is a study on
the use of a strobe light in the red lens of a traffic signal. The
purpose of the strobe light in this system is to draw the driver's
attention to the traffic signal. The before-and-after study helped
determine if this new technology prevents accidents. The *Freeway
Management and Operations Handbook* identifies several limitations to
this evaluation method. These limitations include:

-   Difficulty in distinguishing the effects of an individual
    improvement when multiple improvements were made at one time.

-   Time required for drivers to adjust their travel behavior after the
    system or strategy is implemented. Thus, the true effects of the
    changes may not be measured if the "after" data are collected too
    soon.

-   Susceptibility to errors caused by time-related factors because of
    the often long time lag between the "before" and "after" condition.

-   Fluctuation of a performance measure over time until an
    extraordinary value is observed, which causes the performance
    measure to return more typical values. This tendency is called
    regression to the mean. If the "before" or "after" condition exudes
    this tendency, it hides the true performance of the system.

#### Trend Analysis

A prerequisite to trend analysis is that the data must be archived. As a
result, trends can be identified over time by graphical means or other
statistical functions. An example is the INFORM system in Long Island,
New York, and the way it tracks the percentage of devices online and
maintains a trend analysis. This type of analysis is also beneficial
because it indicates which aspects of the problem are benefiting from
the investments made in the system.

#### Comparison Group Evaluation

This type of evaluation creates a comparison group with untreated sites,
making it a control for other factors in the evaluation. This method
makes it easy to see how effective the strategy or new technology is at
accomplishing the objective. Often, this comparison group is then
applied to the before-and-after evaluation.

### Root-Cause Analysis

Root-cause analysis is performed after an error occurs while collecting
data, such as a loop detector producing incorrect readings. RCA
determines the data collection problem so that it can be corrected.
Prior to performing this type of analysis, it should be confirmed as
being cost beneficial. It is ineffective to determine indefinitely the
root cause of the occurrence. General causes, like operator error,
should not be investigated, however, as the purpose of this analysis is
to fix the problem. There are four major steps to this analysis, as
shown in Box 4.

**Box 4: Steps for Root-cause Analysis (Rooney and Heuvel 2004)**

### Benefit Analysis

Benefit analysis uses statistics to determine whether and how the
implemented project positively contributes to the intended audience and
the overall system. The benefit-cost analysis is the recommended
practice to describe the system benefits to the public and decision
makers. At the regional level, the benefit-cost analysis will allow the
system to be evaluated against traditional transportation program needs.
The most practiced benefit analysis is benefit-cost evaluation.

This technique is the most widely accepted methodology for evaluating
transportation improvement alternatives. The analyst must assign values
to possible benefits and disadvantages of the system (such as shorter
travel time or increased congestion). The analyst should consult an
operations practitioner to ensure that the full range of benefits is
captured. By analyzing the alternatives with respect to system costs,
the analyst can determine objectively which offers the best benefit-cost
ratio. The formula given by the *Freeway Management and Operations
Handbook* is:

B/C = (benefit of alternative i)/(cost of alternative i)

If the benefit of the alternative is greater than the cost, then the
improvement in the system is economically justified. This ratio provides
a convenient basis for comparison of each alternative.

An incremental benefit-cost analysis should be used if the cost,
quantities, and complexities of the alternatives' components build upon
each other. For this approach the benefits and costs should be analyzed
in terms of additional benefits achieved and costs incurred over the
next expensive alternative. Doing so determines whether an investment
necessary to achieve the next incremental step in the system can be
justified in terms of the incremental benefits that would be achieved.

The downside of this method, however, is that not all benefits are
easily quantified and not all quantifiable benefits can be converted
into monetary value. One solution to this problem is to use utility-cost
analysis. The utility-cost analysis assigns a weight to each goal and
subgoal. Then, each alternative is rated based on the utility of each
alternative in satisfying each goal and subgoal. Then, by applying the
following formula, the utility can be calculated:

Utility = Σ Weight of goal \*rate of goal

The utility-cost ratio can be determined with the following formula:

U/C = (Utility of alternative i)/(cost of alternative i)

### Reporting

Reporting allows for communicating valuable information about the TMS
with the stakeholders, decision makers, and the public. These
stakeholders can include (but are not limited to) government officials,
agency management, and agency staff. This communication link is achieved
by analyzing and interpreting the meaning and significance of the
information into terms that are understandable by the audience. Good
performance reporting focuses on a few critical aspects of the
performance of the system and explains why these attributes of the
performance were chosen to report.

#### Audience

Reporting needs for various stakeholder groups are often different, so
they should be linked to previously established goals and objectives.
Stating goals and performance expectations show the relation of the
results through either visual or written information.

For those stakeholders in management or government positions, the report
should communicate the current program status, future plans, and ways
for the program to proceed. The public, however, is more interested in
areas such as the acquisition and use of resources, service efforts, and
accomplishments. One way to illustrate these accomplishments is to
relate the results to the capacity to meet or exceed the current
performance expectations. The public is also interested in any risks
that it may be susceptible to as a user of the system. Thus, it is good
practice to explain what the key risks are, the level of the risks, and
how the risks influenced any choices made in relation to policy, goals,
and performance expectations.

One way to communicate transportation information to the public is
through the media. Releasing the results of traffic and other related
studies to the press has proven an effective way to increase public
awareness. For example, the National Transportation Operations Coalition
(NTOC) released its first National Traffic Signal Report Card in April
2005. This study used the results of a traffic signal system survey to
give national grades from A to F in six distinct categories related to
traffic signals. To raise awareness about the results---traffic signals
are not being used to their full potential---the NTOC released the study
findings through various media channels and held a national press
conference in Washington, DC. By disseminating information through the
media, officials can thus reach many more people with important new
developments in transportation.

In addition to disseminating information to the public, reporting is
also important to several activities within a TMS agency. These
activities include planning, designing, operations and enforcement. The
report related to any of these specific activities provides crucial
information that could help improve the quality of the activities
themselves.

The people responsible for reporting usually spend a considerable amount
of time on structuring---formatting and publishing performance
measurement results in the form of written and electronic reports. A
major resource in planning a system is the information gained from
long-term travel trends and infrastructure projects. They utilize some
of the aforementioned analysis techniques, such as benefit-cost
evaluation, to determine the appropriate applications that should be
implemented in the system.

One primary purpose of performance reporting is to help manage
operations. Most performance measures capture information related to
everyday operations (MTG 2004). This information usually includes
information on traffic data, such as traffic congestion (Kwon 2004).
Reporting information about these operations relays information to the
audience on what is currently being done by the audience and how well it
is doing.

Law enforcement is an integral part of any transportation system. The
enforcement agencies often are considered stakeholders for a system.
They help promote safety within the transportation system. Thus,
numerous performance measures relate to the operations of these
enforcement agencies (such as response time). By reporting the results
from these performance measures, these agencies can determine what areas
under their purview need improvement.

#### Content and Frequency of Reporting

There are two types of reports that TMSs use to communicate information:
internal reports and external reports. Internal reports stay within the
agency and communicate information to different staff members. External
reports, on the other hand, convey to audiences outside the agency how
successful the agency is at accomplishing its mission, goals, and
objectives in the context of "potential significant decision making or
accountability implications."

There are several trends among agencies on how to report information.
One trend is to post the report on their intranet sites. These types of
reports are disseminated on a more frequent basis, such as weekly,
monthly, or continually. To make some information accessible to a
broader public, many agencies also choose to post data via the Internet.
One example is the Washington DOT's *Gray Notebook*. These types of
reports tend to be generated either monthly or quarterly. Agencies also
tend to generate more formal reports biannually or annually for their
government and business stakeholders. These formal reports are those
most commonly used among agencies. These types of reports include annual
reports, business plans and other bounded reports. To keep managers and
CEOs knowledgeable about the system, many agencies produce executive and
mid-management reports. These reports can be produced in printed or
electronic form. They do not need to be created with any particular
frequency, but rather, depend on how often the executive members would
like them. They vary from weekly to annually. "Notebooks" are another
trend in reporting. The purpose of a notebook is to ensure that key
decision makers are up-to-date on the goals of the program and its
progress. Notebooks tend to be updated every month or quarter.

### Best Practice: WSDOT 

Washington Department of Transportation is one of the leading agencies
in terms of public communication. Their quarterly performance report is
called *Measures, Markers, and Mileposts,* also known as the "Gray
Notebook." The Gray Notebook explains the agency's planning process and
the rationale behind different actions. It also assesses the
effectiveness of the statewide system. It tracks a variety of
performance and accountability measures for routine review by the
Transportation Commission. The Gray Notebook also is continually
evolving and has become an important source of information about
department performance for the CEO, state legislators and other agency
stakeholders. These reports engage the reader and make data more readily
accessible to the audience. There are several criteria that this report
abides by, which are:

-   Avoid colors; make the chart work in black and white.

-   Use plain English and avoid jargon.

-   Show only relevant data and remove "chart junk" (outline boxes,
    lines, colors).

-   Cite data sources.

-   Eliminate legend boxes and use pointers to label data.

-   Drop extra grid lines and numbers and lighten line values.

-   Avoid 3Ds; don't do multidimensional graphs.

-   Use clear chart title and subtitles to explain the x-axis and
    y-axis, content, and purpose.

The Gray Notebook contains an array of information about the agency. The
report is divided into two sections: the *Beige Pages* and the *White
Pages*. The *Beige Pages* is a project delivery performance report that
summarizes the project and the associated financial information. The
*White Pages* gives three types of updates: annual performance topics,
quarterly performance topics and special topics. Annual performance
topics include pavement conditions, congestion and bridge conditions.
The specific topics relevant to TMSs include:

-   Traffic Fatalities:

```{=html}
<!-- -->
```
-   Comparing Fatal and Disabling Crashes and Vehicle Miles Traveled
    (VMT).

-   Fatality Rate per Capita.

-   Fatality Rates Compared to National Average.

-   Seatbelt Use.

```{=html}
<!-- -->
```
-   Pavement Assessment:

```{=html}
<!-- -->
```
-   Pavement Condition Rating.

-   Washington Pavement Roughness vs. Other States.

```{=html}
<!-- -->
```
-   Highway Maintenance.

-   Incident Response:

```{=html}
<!-- -->
```
-   Total Number of Responses by Month.

-   Number of Responses to All Incident by Time of Day.

-   Clearance Time by Response Mode.

-   Training Incident Responders.

Quarterly performance topics include highway construction, worker
safety, incident response, Washington State ferries and Amtrak cascades.
The specific include:

-   Washington State Ferries:

```{=html}
<!-- -->
```
-   Total Number of Complaints per 100,000 customers.

-   Common Complaints Rate per 100,000 customers.

-   Trip Reliability.

-   On-time Performance.

-   Ridership and Revenue.

-   Fare Box Recovery.

-   Terminal and Vessel Preservation Performance.

-   Capital Expenditure Performance.

```{=html}
<!-- -->
```
-   State-supported Amtrak Cascade:

```{=html}
<!-- -->
```
-   Monthly and Annual Ridership.

-   On-time Performance.

-   Fare box Recovery.

-   Grain Train Carload.

Special topics include special events and innovations. Specific topics
found in their 2004 report included:

-   Oversize and Overweight Permits:

```{=html}
<!-- -->
```
-   Non-electric Permits Turn Around Time.

-   Motor Vehicle Permit Revenue.

-   Pre-audit of Projects:

```{=html}
<!-- -->
```
-   Highway and Ferry Programs.

-   Capital Management Project.

-   Environmental Programs.

### Best Practice: VDOT 

As a part of an upcoming build of the Virginia DOT's Archived Data
Management System (ADMS) Web site, a daily report will provide
transportation officials with a summary of the previous day's freeway
traffic conditions in a metropolitan region (Evanchik 2005). For
instance, the report gives updates on freeway mobility, number of
incidents and traffic sensor availability from the previous day using
graphs and maps. Below (Figure 6) is an example of the first page of the
report.

![](media/image10.png)

Figure 6: Example Report Page from ADMS Web Site (Evanchik 2005)

Due to the popularity of the Internet, most reports are available
online. Reports also are commonly published electronically on a CD-ROM.
A key factor for reporting these results is that the information must be
presented in a manner for the audience to understand and interpret.
NCHRP Synthesis 311 advises that reports combine written text (9
percent), tables (37 percent), charts (24 percent) and maps (24
percent).

## References

Artrip, Diane, *Case Study: City of Phoenix, Arizona in Use And The
Effects Of Using Performance Measures For Budgeting, Management, And
Reporting*, January 2004.

Evanchik, Daniel M. (2005). Using a Speed Index and Visual Data Display
to Analyze Freeway Traffic Data. M.S. Thesis, University of Virginia..

Federal Highway Administration, *Freeway Management and Operations
Handbook, Final Report*, September 2002

Federal Highway Administration, *Linking Planning and Operations,*
September 2004

Hack, Becki, *Designing Performance Measures and Metrics,* Accessed:
February 2005,
<http://www.bettermanagement.com/library/library.aspx?libraryid=7078>

International City/County Management Association (ICMA), Accessed:
February 2005, [http://www.icma.org](http://www.icma.org/)

Kwon, Taek M. (2004). Annual Report Transportation Data Laboratory.
<http://www.d.umn.edu/natsrl/Publications/FY2004reports/TDRL_2004.pdf>

MTG Management Consultants. (June 2004). *Best Practices In
Transportation Department Performance Measurement Structures*.

Rooney, James J. and Lee N. Vanden Heuvel. (July 2004). *Root Cause for
Beginners*.
<http://www.asq.org/pub/qualityprogress/past/0704/qp0704rooney.pdf>

Transportation Research Board, *NCHRP Synthesis 311,* Washington D.C.,
2003.

![](media/image11.png){width="1.0527777777777778in"
height="1.0840277777777778in"}