# Presentation: 823795

## TMS Performance Monitoring, Evaluation & Reporting Handbook

- University of Virginia
- and
- SAIC

- TMC Pooled Fund Study Annual Meeting
- June 14, 2005

## Overview

- Project team / management
- Background / purpose / objectives
- Intended audience
- Key topics / milestones
- Progress report
- Preliminary survey results
- Handbook at-a-glance
- Contents of the handbook (highlights)
- Next steps
- Project contacts

## Project Team

- FHWA
- Mr. **Raj Ghaman**, COTM
- SAIC
- Ms. **Rebecca Barnes**, TOM
- Indiana DOT
- Mr. **Mark Newland**, Panel Chair
- University of Virginia
- Dr. **Byungkyu (Brian) Park**, PI
- Dr. **Brian Smith**, Co-PI
- Dr. **Mike Demetsky**, Co-PI
- Mr. **Ramkumar (Ram) Venkatanarayana**

## Project Management

- University of Virginia
- Monthly teleconference meeting
- Handbook and other documents development
- SAIC
- Document mockups
- Outreach material – Handbook, Primer, Tri-fold brochure, Questions & Answers, etc.

## Background

- Lack of standard technical references and/or tools for assessing the operations of Traffic Management Centers (TMC) and Transportation Management Systems (TMS)
- While performing TMS performance monitoring, there exist (1) a disconnect between the goals and monitoring efforts, (2) inconsistency in data collection and analysis and (3) lack of information sharing

## Purpose

- To develop a handbook that provides guidance and recommended practices for initiating, maintaining and utilizing information derived from TMS performance monitoring, evaluating and reporting

## Objectives

- Provide an overview of TMS performance monitoring, evaluation, and reporting
- Discuss the basics and importance of a performance measurement program
- Provide examples of agency goals and performance measures
- Describe common data requirements, collection, and archiving techniques
- Explain performance monitoring, evaluation, and reporting
- Provide a Self-Assessment Questionnaire

## Intended Audience

- Primary audience: various agency personnel who uses performance measures obtained from a TMS including operators and managers who manage TMC facilities and TMS
- anyone else who is interested in TMS performance monitoring, evaluation, and reporting.  This could include transportation engineers and researchers, Federal/State/Local agencies, and roadway maintenance personnel

## Key Issues & Topics

- Performance measurement program
- Agency goals and performance measures
- Data requirement, collection and archiving
- TMS performance monitoring, evaluation and reporting
- Self assessment toolkit

## Key Milestones

- Kickoff meeting – August 19, 2004
- Final Annotated Outline – December 20, 2004
- Final Draft Handbook – March, 2005 (Delivered Chapters 2, 3, 5 and 7)
- Final Handbook for Printing – August 15, 2005
- Final Outreach Materials – September 12, 2005

## Progress Report

- Completed preliminary survey on the use of performance measures by TMS/TMC
- Completed the following draft chapters
- Chapters 2, 3, 5, and 7 (overview of TMS performance monitoring, evaluation and reporting; performance measurement program; data requirement, collection and archiving; and self-assessment)
- Under development on Chapters 1, 4, and 6

## Preliminary Survey

- Purpose
- Understand the state of the practices on the use of performance measures
- Identify candidate TMS/TMC sites for self-assessment case studies
- Web-based survey
- Consisted of 24 questions regarding their operations and performance measures
- 80 TMC/TMS personnel were invited
- 53 visited website (74%)
- 28 completed (35%)

## Preliminary Survey Results

**Question 11** – What data do you collect from your traffic monitoring procedures? Check all that apply.
- Average Motorist Speed – 71%
- Recurring Delay – 14%
- Travel Time – 21%
- Traffic Volume – 75%
- Incident Delay – 29%
- Emergency Management Response Times – 25%
- Weather Information – 25%
- None of the Above – 7%

## Preliminary Survey Results (cont’d)

**Question 12** -- With whom do you share these traffic data? Check all that apply.
- State DOT – 63%
- Other TMCs – 63%
- Emergency Personnel (Fire, Rescue, Police, Etc.) – 48%
- Local Media – 48%
- Transit Agencies – 15%
- Bridge/Tunnel Authorities –11%
- None of the Above – 19%
- Other responses include public, FHWA, local universities, MPOs and cities.

## Preliminary Survey Results (cont’d)

**Question 13 -- Does your center archive data?**
- Yes – 89%
- No – 11%
**Question 14** -- Has your TMC published any information regarding ITS performance measures?
- Yes – 14%
- No – 64%
- Not Yet, But Will in the Future – 21%

## Preliminary Survey Results (cont’d)

**Question 17** -- Which of the following performance measures does your TMC analyze? Check all that apply.
- Benefit-Cost Analysis – 57%
- Incident Delay Analysis – 62%
- Travel Time – 29%
- Crash and Fatality Reduction – 38%
- Emissions and Fuel Consumption – 24%
- Dispatcher Evaluations – 19%
- Comment Cards – 19%
- Motorist Phone Calls – 14%
- Website Surveys – 24%
- Website Hits – 38%
- None of the Above – 5%

## Preliminary Survey Results (cont’d)

**Question 18** – Does your TMC publish performance evaluations reports periodically?
- Yes – 21%
- No – 79%
**Question 19** -- In what format are the reports published?
- Newsletter – 0%
- Formal Report – 63%
- Website Presentation – 0%
- None, No Reports Published – 25%

## Survey Findings

- Most TMCs archive data (89%)
- Not many TMCs periodically publish performance evaluation report (only 21%)
- Lack of data sharing (19% of TMCs does not share their data with other agencies)

## Handbook At-A-Glance

**Chapter 1 – ****Introduction**.  This chapter defines the background, purpose, and scope of the handbook as well as the intended audience.
**Chapter 2 – ****Overview of TMS Performance Monitoring, Evaluation and Reporting**.  This chapter provides a high level overview of TMS performance monitoring, evaluation, and reporting, and how it relates to TMS.
**Chapter 3 – ****Performance Measurement Program**.  This chapter discusses the purpose, need, and importance for a TMS performance measurement program.

## Handbook At-A-Glance (cont’d)

**Chapter 4 – ****Agency** **Goals and Performance Measures**.  This chapter presents typical performance measurement goals of TMS related agencies. This chapter also presents high-level performance measures by TMS functions and calculation methods of such performance measures.
**Chapter 5 – ****Data Requirements, Collection and Archiving****.  **This chapter provides performance measure data requirements and best practices for data collection, evaluation, and reporting.

## Handbook At-A-Glance (cont’d)

**Chapter 6 – ****Performance Monitoring, Evaluation and Reporting**.  This chapter provides various evaluation methodologies and processes related to TMS performance.  It also discusses recommended reporting techniques, formats, and frequencies for reporting TMS performance.
**Chapter 7 – ****Self- Assessment****.**** **This self-assessment will be in the form of a checklist of questions drawn from case studies.  This checklist can be used by TMCs to assess the status and performance of the TMS.
**Appendix A – Survey Questionnaire**
**Appendix B – Contact List of Traffic Management Centers**

## Highlights of Handbook Contents

- Performance measurement program
- TMS functions and their performance measures
- Data and key ‘data’ elements in TMS/TMC
- TMS/TMC performance  monitoring, evaluation and reporting

## Performance Measurement Program

## TMS/TMC Types and their Functions

## TMS Functions by System

|                          System
Function | Freeway | Signal | Transit
(Bus) | Transit
(Subway/Light Rail) |
| --- | --- | --- | --- | --- |
| Special event management | x | x | x | x |
| Incident management | x | x | x | x |
| Coordination with emergency agencies | x | x | x | x |
| Surveillance | x | x | x | x |
| Planned lane closure | x | x | x |  |
| Public and private information dissemination | x |  | x |  |
| Emergency management | x |  | x | x |
| Interagency information sharing | x |  | x | x |
| Traffic/track signal control |  | x | x |  |
| HAZMAT |  |  | x |  |
| Data fusion |  |  | x |  |

## Performance Measures

- Developed high-level performance measures by TMS/TMC functions
- Freeway system
- Signal system
- Transit system
- Distinguished output vs. outcome measures
- Provided calculation methods of proposed performance measures

## Signal System Example

## Key “Data” Elements in TMS/TMC

## Slide 29

|                         Performance
                  Measure/Program

Data | Mobility-travel time (Congestion) | VMT | Safety-incidents | Environmental Analysis | Information Dissemination | HOV study  | Incident Management | Ramp Metering | Arterial Mobility | Other |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Traffic volumes | X | X | X | X |  | X | X | X | X |  |
| Speeds | X |  |  | X |  | X |  | X | X |  |
| Vehicle Classification | X | X | X | X |  | X |  |  |  |  |
| Probe data (AVL, GIS, AVI) | X |  |  | X | X |  |  |  | X |  |
| Incident/Event log (VMS, HAR, 511, HOV/RHOV open time logs) | X |  | X | X | X | X | X | X | X |  |
| Travel time | X |  |  | X | X |  | X |  | X |  |
| Maintenance log | X | X | X |  | X |  | X | X |  |  |
| Signal data (light times, offsets, queues) |  |  |  |  |  |  | X | X | X |  |
| Weather data | X |  | X | X | X |  | X |  |  |  |
| Video or image stream |  |  |  |  | X |  | X |  |  |  |
| Air/Water/Noise pollution |  |  |  | X | X |  |  |  |  |  |
| Metadata (location of probe, area coverage, weather, other events etc.) | X | X | X |  | X | X | X |  |  | X |

## Performance Monitoring, Evaluation and Reporting

**Performance monitoring: **  Examines the actual system conditions through observed data
**Evaluation**: Analyzes the collected data and compares the results to benchmark performance measures
**Reporting**: Provides visual information and  an interpretation of these? graphs to decision makers and the public
**Highlighted Best Practices **

## Next Steps 

- Complete draft handbook
- Revise the handbook based on panel comments/suggestions
- Conduct case studies using self-assessment toolkit
- Develop outreach materials

## Project Contacts

**Raj Ghaman**
- Contract Office Task Manager
- Federal Highway Administration
- Phone: 202-493-3270
- Email: _[Raj.Ghaman@fhwa.dot.gov](mailto:Raj.Ghaman@fhwa.dot.gov)_
**Rebecca Barnes**
- Task Order Manager
- Science Applications International Corporation
- Phone: 703-676-6783
- Email: _[rebecca.w.barnes@saic.com](mailto:rebecca.w.barnes@saic.com)_

**Byungkyu (Brian) Park**
- Project Manager
- University of Virginia
- Phone: 434-924-6347
- Email: _[bpark@virginia.edu](mailto:bpark@virginia.edu)_