Surface Transportation: DOT Is Progressing toward a Performance-Based Approach, but States and Grantees Report Potential Implementation Challenges
Published: 2015-01-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In fiscal year 2013, DOT provided about $50 billion to states and other grantees (such as metropolitan planning organizations and transit agencies) to support highway and transit infrastructure and safety. However, it is not clear if this funding has improved system performance because, in part, these programs have lacked links to performance and national goals. Since 2008, GAO has highlighted the need to link surface transportation programs to performance. MAP-21 transformed surface transportation programs by including provisions for both DOT and its grantees to move toward a national performance-based approach, in many cases holding grantees accountable for performance for the first time. This approach includes DOT establishing national performance measures, and states and other grantees setting targets related to these measures, and reporting progress.
GAO examined (1) the progress that DOT has made in developing a national performance-based approach to surface transportation and (2) the challenges states and other grantees report facing in implementing this approach. GAO reviewed proposed rules and DOT information on rulemaking milestones and interviewed officials from DOT, national transportation organizations, and a non-generalizable sample of 29 grantees in 13 states, selected on the basis of state population growth, performance management experience, and other factors. DOT reviewed a draft of this report and provided technical comments, which were incorporated as appropriate.
section_title: What GAO Found
paragraphs: The Department of Transportation (DOT) has made progress moving toward a national performance-based approach since the Moving Ahead for Progress in the 21st Century Act (MAP-21) was enacted in July 2012. DOT has undertaken extensive outreach efforts with states and other grantees and is developing nine rules that, among other things, are to establish national performance measures, and target-setting and reporting requirements for grantees. Five of the rules have been released for public comment, and none has been finalized. (See figure.) DOT has missed five of six deadlines established by MAP-21. However, given the extent to which MAP-21 transformed surface transportation programs and the length and complexity of the rulemaking process, these deadlines may have been ambitious. DOT's progress has also been affected by the varying experiences within its operating administrations in implementing a performance-based approach. For example, the National Highway Traffic Safety Administration has been implementing such an approach in its grant programs since 2008, while other DOT operating administrations started more recently.
States and other grantees reported potential data and cost challenges with implementing a national performance-based approach. According to some grantees and national organizations GAO contacted, data may not be available for measuring performance in some areas, such as congestion, and collecting data that are available may prove difficult. For example, DOT has proposed states collect traffic data on all public roads, and some state officials told GAO this step would be a substantial undertaking; New York officials told GAO that the state has adequate data for only about 37 percent of its public roads. Further, some grantees noted they may lack the staff or resources to analyze data, and some raised concerns that implementation costs may be substantial. DOT officials acknowledged these challenges and are working with grantees to help overcome obstacles. For example, DOT is working with states to share data, best practices, and templates for reporting.

section_title: Letter
paragraphs: The U.S. Department of Transportation (DOT) annually provides billions  of dollars to states and other grantees to improve the nation’s highway  and transit infrastructure and safety. DOT provides federal funding to  these grantees, including states, transit agencies, and metropolitan  planning organizations (MPO), primarily through programs administered  by three of its operating administrations: The Federal Highway  Administration (FHWA), which administers the federal-aid highway  program that provides billions to states for highway and bridge projects;  the National Highway Traffic Safety Administration (NHTSA), which  provides grant funding to support state highway safety programs; and the  Federal Transit Administration (FTA), which administers a variety of grant  programs that provide funding for the development and expansion of bus,  rail and other transit projects, among other things. In fiscal year 2013,  DOT provided approximately $50 billion to grantees for surface  transportation projects through the programs administered by FHWA,  FTA, and NHTSA.
However, it is not clear whether the federal funding provided to states and  other grantees through these surface transportation programs has  improved the Nation’s system performance because, among other things,  these programs have lacked links to performance and national goals.  Since 2008, we have highlighted the need to link surface transportation  programs to performance and target surface transportation resources  more effectively. In particular, we have noted that tracking specific  outcomes that are clearly linked to program goals could provide a strong  foundation for holding grant recipients responsible for achieving federal  objectives and measuring program performance. Accordingly, we  recommended that Congress consider reexamining and refocusing  surface transportation programs to, among other things, make grantees  more accountable through the use of more performance-based links  between funding and program outcomes. Funding the nation’s surface  transportation system is on GAO’s High Risk List.
In July 2012, the President signed into law the Moving Ahead for  Progress in the 21st Century Act (MAP-21) that included provisions for  both DOT and its grantees to move toward a national performance-based  approach for surface transportation. Among other things, the act  amended the United States Code to establish national surface  transportation performance goals, and to provide a three-stage process  for a national performance-based approach for surface transportation  through which (1) DOT defines performance measures in support of the  national goals and develops an approach for measuring grantee  performance; (2) states and other grantees set and maintain targets  based on the DOT-defined performance measures, collect data, and  report to DOT their progress in meeting their targets; and (3) DOT  evaluates grantee performance and reports to Congress on the extent to  which grantees have met their targets. If progress toward meeting some  targets is insufficient, states must report action they will undertake to  achieve their targets. In addition, MAP-21 links funding to performance by  requiring states to take corrective action if progress toward certain targets  is insufficient, and to spend a specified portion of their annual federal  funding to improve National Highway System bridges and Interstate- system pavement should conditions fall below minimum standards set by  MAP-21 and the Secretary of Transportation, respectively. MAP-21 was  originally enacted as a 2-year authorization for surface transportation  programs through September 30, 2014. In August 2014, legislation was  enacted that extended programs and funding authorized under MAP-21  until May 31, 2015.
This report examines (1) the progress that DOT has made in developing a  national performance-based approach to surface transportation and (2)  the challenges states and other grantees report facing in implementing  this approach. We prepared this report under the authority of the U.S.  Comptroller General to conduct evaluations in light of broad  congressional interest in this area and the relationship of this topic to our  High-Risk area on funding the nation’s surface transportation system.
To address these objectives, we reviewed academic literature from 2011  through 2014—representing the time period just before and since MAP- 21 was enacted—as well as our body of work on performance  management in the federal government and transportation programs. To  describe the progress that DOT has made in developing a national  performance-based approach for surface transportation programs, we  reviewed related requirements and timetables in MAP-21 for FHWA, FTA,  and NHTSA as well as interim and proposed rulemakings developed by  these operating administrations in response to these requirements. We  did not evaluate the potential effectiveness of the proposed rules because  the content may change before rules are finalized in response to  comments submitted by grantees and other parties. We also analyzed  DOT documents, guidance, and other information provided to grantees  and analyzed DOT information on estimated and completed rulemaking  milestones to understand the progress FHWA, FTA, and NHTSA have  made developing the performance-based approach. We interviewed  FHWA, FTA, and NHTSA officials to understand the factors affecting  DOT’s progress in developing the performance-based approach.
To understand the challenges states and other grantees face in  implementing a performance-based approach to surface transportation,  we analyzed comments submitted to DOT by—and interviewed officials  from—national transportation organizations and 29 grantees in 13 states.  The grantees comprised a sample of state transportation agencies, state  highway safety offices, metropolitan and rural planning organizations, and  transit agencies, selected to include grantees in states with varying  population growth trends, population size, geographic location, transit  infrastructure age, experiences managing and measuring transportation  project performance, and incorporating recommendations from DOT and  national transportation organizations. Based on these criteria, in four  states, we interviewed officials from the state department of  transportation, one or more metropolitan planning organizations, local  transit providers, and DOT regional oversight offices; in the remaining  nine states, we interviewed officials from at least one state transportation  agency, planning organization or transit agency in that state. The  information gathered from these 29 grantees cannot be generalized to all  federal surface transportation grantees; however, they provided insight  into the types of challenges that some grantees anticipate in  implementing DOT’s future rulemaking and overall performance based  approach, as well as, perspectives on DOT’s outreach and rulemaking  progress. Additional information on our scope and methodology can be  found in appendix I.
We conducted this performance audit from May 2014 to January 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: Most of the federal funding for highway and transit infrastructure and  safety improvements is provided by FHWA, FTA, and NHTSA.
FHWA provides about $38 billion each year, or nearly 93 percent of its  overall budget, to states to design, construct and maintain the nation’s  roadway and bridge infrastructure through the federal-aid highway  program. These funds are distributed through annual  apportionments established by statutory formulas. FHWA oversees  the federal-aid highway program, primarily through its 52 division  offices located in each state, the District of Columbia, and Puerto  Rico. FHWA division offices have 10 to 61 staff each, depending on  the size of the state’s highway program and other factors. As of  October 2014, FHWA had approximately 2,900 staff—about two-thirds  in the field and the remaining third at FHWA headquarters in  Washington, D.C.
FTA provides approximately $10.5 billion each year for transit project  grants to states, transit agencies, and MPOs. The grant funding  provided by FTA represents 98 percent of its overall $10.7 billion  budget; the remaining portion of the budget funds the administration  and oversight of its various grant programs by about 500 federal  employees in its Washington, D.C., headquarters and 10 regional  offices.
NHTSA provides about $500 million per year, or about 60 percent of  its annual budget of approximately $800 million, to state highway  safety offices through grants. NHTSA awards these grants to state  highway safety offices to support projects that focus on mitigating  behavioral factors associated with crashes and improving traffic safety  information systems, among other things. The remaining third of  NHTSA’s budget supports various highway and vehicle safety  research and enforcement programs. NHTSA employs about 600  employees in its Washington, D.C. headquarters; its 10 regional  offices; and its Vehicle Research and Test Center.
States develop a variety of plans as part of their overall transportation  programs. Each state, as well as the District of Columbia and Puerto  Rico, is required by statute to carry out a continuing, cooperative, and  comprehensive statewide transportation planning process that requires  periodic updates of plans. The statewide transportation planning  process addresses both urbanized and nonmetropolitan areas of the state  and includes both highway and transit needs. As part of the statewide  planning process, states must develop a long-range statewide  transportation plan and a state transportation improvement program  (STIP). The long-range statewide transportation plan establishes a state’s  strategic vision and direction for its transportation investments for at least  a 20-year period. The STIP is the state program of transportation projects  covering at least a 4-year period that are to be supported with federal  surface transportation funds, as well as regionally significant projects  requiring an action by FHWA or FTA, whether or not the projects are  federally funded. At least every 4 years, state DOTs are required by  statute to submit an updated STIP to FHWA and FTA for review and  approval, in which the state certifies that the transportation planning  process has been carried out in accordance with federal planning  requirements. In urbanized areas, MPOs are also required to produce a  long-range transportation plan, referred to as a metropolitan  transportation plan, and a transportation improvement program (TIP). We  have previously found that the cycle and frequency of updates of these  plans vary across states and MPOs. In addition, each state is required  to develop a strategic highway safety plan to, among other things, provide  for performance management and identify strategies that cover all  aspects of highway safety, which include (1) infrastructure improvements;  (2) behavioral approaches, such as education and enforcement efforts  meant to change drivers’ behavior; and (3) emergency medical services.  Furthermore, each year each state is required to develop a highway  safety plan, describing the state’s highway safety program and the  activities they plan to undertake. According to NHTSA, states have  voluntarily included measurable goals and objectives in their highway  safety plans since 2010, enabling states to track the status of highway  safety implementation efforts and monitor progress.
Prior to enactment of MAP-21 in 2012, DOT’s surface transportation  funding programs and statewide planning generally were not linked to  performance-based outcomes, although some states used performance  measures, most notably in the safety area. Historically, most federal  surface transportation funds have been distributed through formulas that  have only an indirect relationship to infrastructure needs, and many have  no relationship to outcomes or grantees’ performance. FTA had  incorporated performance-based principles into some of its formula grant  programs to allocate funding. We previously found that about 5 percent of  fiscal year 2010 funding for six formula grant programs we reviewed, on  average, was required to be awarded based on transit agency  performance. In a 2010 report, we noted that DOT’s oversight of the  statewide planning process focused on state DOTs’ compliance with  planning process requirements and did not consider transportation  planning outcomes. At the state level, we found that states and MPOs  have incorporated some performance measures and targets into their  plans, though the use of measures and targets varied depending on the  area. For example, we found nearly all state DOTs had safety  performance measures and quantifiable targets related to these  measures, due, in part, to the federal requirement that state DOTs  develop a strategic highway safety plan that establishes statewide goals  and objectives to reduce highway fatalities and serious injuries. In other  areas, such as measuring truck freight traffic, vehicle congestion, and  emissions, we noted states generally did not incorporate performance  measures into their plans. However, even in areas such as safety where  states had used performance measures, we reported that only a few  states made significant attempts to integrate performance measurement  into their statewide planning process to inform investment decisions.
Since 2008, we have recommended that Congress consider reexamining  and refocusing surface transportation programs to include performance- based principles. In particular, we have noted that tracking specific  outcomes that are clearly linked to program goals could provide a strong  foundation for holding grant recipients responsible for achieving federal  objectives and measuring overall program performance. Furthermore,  we have previously found that data-driven decision-making leads to better  results, and that if agencies do not use performance measures and  performance information to track progress to track progress toward goals,  they may be at risk of failing to achieve their goals. In addition, we found  through our prior work that performance information can be used across a  range of management functions to improve results, from setting program  priorities and allocating resources to taking corrective action to solve  program problems.
MAP-21 included a number of statutory requirements related to  transforming the surface transportation system to a performance-based  approach, for the operating administrations, as well as for states and  other grantees, in many cases holding grantees accountable for the first  time. For instance, MAP-21 directed the Secretary of Transportation to  establish performance measures related to highway safety, asset  condition, and highway system performance, among other things. In  addition, states and other grantees are required to establish targets  related to the performance measures established by DOT and to collect  data, and track progress against those targets. States and other grantees  are also required to report on progress toward achieving their targets.  For example, beginning in 2016, states are required to report every 2  years to DOT on the progress they have made in achieving their FHWA  targets. Further, in 2017, 5 years after the passage of MAP-21, DOT is  required to report to Congress on progress made by states and other  grantees in achieving their targets. In addition to establishing a  performance-based approach for existing programs under NHTSA,  FHWA, and FTA, MAP-21 expanded FTA’s safety oversight role and  directed it to establish performance criteria for its new safety  responsibilities, establish performance measures for state of good repair,  develop a national transit safety plan, and develop requirements for local  agency safety plans, among other things.
To implement statutory requirements, such as those contained in MAP- 21, Congress has provided regulatory agencies, such as DOT, with the  authority and responsibility for developing and issuing regulations. The  basic process by which all federal agencies develop and issue regulations  is spelled out in the Administrative Procedures Act (APA). This act  establishes procedures and broadly applicable federal requirements for  rulemaking, including informal rulemaking, also known as notice and  comment rulemaking. (See fig .1) Among other things, the act generally  requires agencies to publish a notice of proposed rulemaking in the  Federal Register. After giving interested persons an opportunity to  comment on the proposed rule by providing “written data, views, or  arguments,” the agency may then publish the final rule. In addition to the  requirements under the act, proposed rules are often subject to internal  and external deliberations and reviews, including official management  approval within an agency, and to review and approval by the Office of  Management and Budget (OMB).
section_title: DOT Has Made Progress in Moving toward a National Performance-Based System, but Multiple Factors Affect Progress
subsections: section_title: DOT Has Made Extensive Outreach Efforts to Communicate with States and Other Grantees
paragraphs: Since MAP-21 was enacted in July 2012, DOT has conducted an  extensive outreach effort to prepare states and grantees for a national  performance-based approach to surface transportation. DOT worked  closely with national transportation organizations and grantees to share  information and collect input about the potential performance-based  requirements. DOT has:
Established a performance working group in August 2012, comprised  of FHWA, FTA, and other DOT officials as well as representatives  from nine national transportation organizations. According to DOT  officials, the working group, which meets on a quarterly basis, has  served as a forum for DOT to coordinate outreach efforts with national  organizations, and also provided national organizations with a better  understanding of the potential performance-based requirements and  implementation schedule.
Conducted webinars, organized regional workshops, and facilitated  peer exchanges and national organization conferences to share  information with grantees. For instance, FHWA and FTA sponsored  a roundtable on MAP-21 during an Association of Metropolitan  Planning Organizations (AMPO) conference and also participated in  Transportation Research Board annual meetings.
Begun development of a technical assistance program and training  curriculum on transportation performance management to assist  grantees with the transition to a performance-based approach. For  example, according to FHWA officials, FHWA has begun developing a  technical assistance program that will include resources such as the  development of guidebooks and evaluation tools, and identification of  best practices. Once FHWA finalizes the rulemakings, FHWA will  provide on-site assistance to its grantees through the technical  assistance program. Additionally, FHWA has begun developing a  training curriculum for states and MPOs that will be available once the  performance-based approach is formally established, according to  FHWA officials. FHWA also conducted a national assessment of state  readiness to help inform its technical assistance program and training  curriculum.
Established guidance to help communicate how certain MAP-21  performance requirements will be implemented. FHWA and FTA, for  instance, jointly issued policy guidance to clarify the requirement  that public transportation providers must be represented on the board  of MPOs that serve as transportation management areas.
Sought formal public input on potential approaches for implementing  the MAP-21 requirements. For example, FTA issued an advance  notice of proposed rulemaking (ANPRM), for its rules related to transit  safety plans and transit asset management to solicit feedback and  obtain input from grantees on the content of potential performance- based approaches to transit.
A majority of grantee officials we spoke to stated that DOT’s outreach  efforts have helped them understand the potential requirements and  mechanics of implementing a national performance-based system. For  example, officials from the New York State Association of Metropolitan  Planning Organizations noted that DOT officials have provided helpful  and extensive information, including responses to frequently asked  questions on the FHWA website to help support the development of the  performance-based approach and keep grantees informed. Similarly,  officials from Washington state noted that as a result of DOT’s outreach  efforts, the rules that have been released for public comment have  contained few unanticipated requirements.
section_title: Rules to Guide the Performance-Based Approach Are under Development
paragraphs: DOT is formally establishing requirements for states and other grantees  to implement the performance-based approach envisioned by MAP-21  through the rulemaking process. (See table 1.) Specifically, DOT plans to  establish performance measures, as well as target-setting and reporting  requirements for grantees for a range of transportation activities. In some  cases, MAP-21 required DOT to use the rulemaking process to  implement performance-based requirements and established certain  deadlines; however, in other cases, MAP-21 did not establish deadlines  for implementing performance-based requirements. For example, while  MAP-21 required FTA to issue a rule for target-setting and reporting  requirements in transit agency safety plans, it did not establish a deadline  for the issuance of a final rule. In addition, DOT officials stated that they  may provide additional guidance to states and other grantees outside of  the rulemaking process. For example, FTA officials stated that FTA plans  to establish a National Transit Safety plan and establish minimum safety  performance criteria for transit agencies as required by MAP-21, through  a policy document, rather than through the formal rulemaking process.
DOT’s operating administrations have made varying degrees of progress  developing their associated performance rules. (See fig. 2.) As of January  5, 2015, five out of nine rules have been available for public comment,  and the remaining four rules are still under development.
NHTSA issued an interim final rule for public comment in January  2013 but has not established a timeline for issuing a final rule.  According to NHTSA officials, the agency is awaiting enactment of a  new surface transportation reauthorization act before deciding on a  timeframe since the timing and content of such a statute could require  NHTSA to revise its interim rule.
FHWA has released three of its five proposed rules for public  comment. FHWA released two proposed rules on safety for public  comment in March 2014 and is currently evaluating the comments it  received. In addition, on January 5, 2015, FHWA released a notice  of proposed rulemaking on pavement and bridge performance  measures for public comment. FHWA plans to release its remaining  two rules for public comment by April 2015.
FTA’s planned rules related to transit asset management and transit  agency safety plans remain under development. FTA has not  released either of its rules for public comment, although FTA did issue  an ANPRM to solicit public input about the contents of its proposed  rules. FTA officials said it plans to release proposed rules for transit  asset management for public comment in June 2015, and a proposed  rule for transit agency safety plans in August 2015.
FTA and FHWA have jointly released for public comment a notice of  proposed rulemaking related to performance-based metropolitan and  statewide planning. The public comment period for this proposed  rule closed in October 2014, and FHWA and FTA are currently  evaluating the comments received on this proposed rule.
In developing these rules, DOT has missed five out of six performance- related rulemaking deadlines established in MAP-21. However, DOT  officials declined to provide specific dates for when some of the rules will  be finalized. FHWA plans to finalize the two safety rules and the  metropolitan and statewide planning rule developed with FTA by  September 2015. However, FHWA has not established when its  remaining rules would be finalized. FTA officials stated that it has not  determined when its two rules would be finalized. FHWA and FTA officials  we spoke with stated they could not provide a more specific date when  the rulemaking process would be completed for the rules that have not  yet been released for comment, noting the challenges of predicting the  effect of the public comments submitted in response to its proposed  rulemaking. As part of finalizing the rules, DOT plans to establish specific  deadlines for grantees’ implementation of the MAP-21 performance  target- setting and reporting requirements. NHTSA is awaiting enactment  of a new surface transportation reauthorization to see if it includes any  additional requirements pertaining to its grants programs before deciding  on a timeframe.
section_title: Multiple Factors Affect DOT’s Progress in Developing a National Performance-Based Approach
paragraphs: DOT’s progress in developing a national performance-based approach  has been affected by the extent to which MAP-21 transformed surface  transportation programs, the length and complexity of the rulemaking  process, and DOT’s varying levels of experience implementing a  performance-based approach.
MAP-21 began the transformation of the nature and structure of  surface transportation programs by establishing a comprehensive  performance-based approach for NHTSA, FHWA, and FTA, in many  cases, holding states and other grantees accountable for performance  for the first time. Historically, most federal surface transportation funds  have been distributed through formulas that have only an indirect  relationship to infrastructure needs, and many have no relationship to  outcomes or grantees’ performance. In general, funds have been  distributed without regard to performance, and therefore, it has been  difficult to determine whether federal spending has improved the  performance of the nation’s surface transportation infrastructure. As  part of this transformational effort to shift from a process-based  approach to a performance-based approach, MAP-21 required DOT  to establish national performance measures that will be used by  states and other grantees to set targets and report on progress in  meeting those targets. However, state and grantee officials told us  that in some instances, there is no broad consensus among DOT,  states, MPOs, and transit grantees on how to implement a  performance-based approach. While some states and grantees have  been measuring performance for years, state and grantee officials we  spoke with told us that their approaches have been tailored to their  state and local interests and priorities, as opposed to national goals.
In addition to moving toward a performance-based approach to  surface transportation, MAP-21 created a number of new  responsibilities for DOT. For example, every four years, DOT is  required to review and certify the process used by each state to  develop its asset management plan, which identifies strategies that  help the state make progress toward its targets for asset condition,  among other things. In addition, FHWA and NHTSA are responsible  for evaluating and approving the process by which initial targets will  be set by states in their state highway safety plans. FTA has  additional responsibilities for overseeing the safety of the nation’s  transit system. As part of these responsibilities, FTA plans to develop  an oversight approach concurrently with the development of  performance measures to evaluate federal transit safety. While FTA  does not have a formal role in evaluating targets related to the MAP- 21 performance measures, grantees must report to FTA on their  progress meeting targets, and identify the following year’s targets.
The rulemaking process is lengthy and complex. Given the length of  time a rule typically requires to be developed under the APA and  extent to which MAP-21 transformed surface transportation programs,  the rulemaking deadlines established by MAP-21 may have been  ambitious. In our prior work, we found that rulemakings can range  from 1 to nearly 14 years to complete, depending on a number of  factors. More specifically, we found that a rulemaking takes an  average of four years to complete, based on a sample of 16  rulemakings from four federal agencies, including DOT. Within this  sample, these four federal agencies took an average of about 2 ½  years to develop and issue a proposed rule. In contrast, the timeline  for MAP-21 generally gave DOT approximately 1 to 1 ½ years to  develop and issue some of the proposed rules.
The number and extent of comments received may affect the duration  and complexity of the rulemaking process. Specifically, agencies are  required to respond to comments that have been received in response  to a notice of proposed rulemaking. We have previously found that  when agencies did respond to public comments, they often made  changes to the rules. FHWA officials stated that they received a  substantial number of comments on their safety performance  measures rule, and it will take time to address these comments and  revise the rule if necessary. Similarly, FTA officials stated that they  received over 2500 pages of comments on FTA’s ANPRM, and are  reviewing and using the comments to develop FTA’s transit safety and  asset management plan rules. In addition, after public release of four  of the five proposed rules published thus far, DOT extended the public  comment period, which may delay the remaining milestones in the  rulemaking process up through the publication of the final rule.
Furthermore, DOT’s rules are interrelated, so delays in establishing  performance measures may cause delays in completing the process  for moving to a performance-based approach. Until DOT finalizes the  performance measures required by MAP-21, states and grantees will  be unable to establish targets and report on progress made in relation  to those performance measures. For example, three FHWA rules  develop performance measures, which directly affect state target- setting and reporting requirements for grantees. (See fig. 3.) Delays in  establishing performance measures may also affect information on  grantee progress reported to Congress. For example, in MAP-21,  Congress directed states to report biennially to DOT, beginning in  October 2016, on the progress made in achieving the targets set  under DOT’s performance-based approach. MAP-21 also directed  DOT to report to Congress in October 2017 on the progress states  and MPOs have made in achieving their targets. However, DOT  officials acknowledged that the reports due to Congress in October  2017 may contain limited information depending on the effective dates  of the MAP-21 performance requirements.
There are varying levels of experience with performance  measurement across DOT’s operating administrations. For example,  NHTSA has been collaborating with grantees and the Governors  Highway Safety Association (GHSA) to develop performance  measures in its highway safety grant programs since 2008.  According to NHTSA officials, in 2010, states voluntarily began  reporting performance measures, targets, and progress meeting these  targets in their highway safety plans. NHTSA uses these measures as  a basis for reporting to Congress about highway safety. In contrast,  FHWA and FTA have had more limited experience in applying a  performance-based approach to its grant programs. For example,  federal-aid highway program funding has historically been distributed  through a complicated process based on statutory formulas. Prior to  MAP-21, FTA had incorporated performance-based principles into  some of its formula grant programs to allocate funding. We previously  found that about 5 percent of fiscal year 2010 funding for six formula  grant programs we reviewed, on average, was awarded based on  transit agency performance. We found that, after these grants were  awarded, FTA was required to collect performance data, but FTA  typically did not use the performance data to evaluate the  effectiveness of its formula grant programs.
Notwithstanding the factors affecting DOT’s effort to implement the  performance-based approach, most states and grantees we spoke with  continue to take steps to implement performance-based principles into  their transportation programs, though some of these grantees are  awaiting further guidance from DOT. Notably, in 2014, FHWA, through its  division offices, made an assessment of states’ readiness to implement  MAP-21 requirements and, in a draft summary report, stated that most  states are actively involved and taking steps to prepare for MAP-21  implementation. State DOTs have also worked with national  transportation organizations to develop criteria and broad guidelines for  performance measures and to evaluate how they can be applied by  members at the state or local levels. For example, officials from the  American Association of State Highway and Transportation Officials  (AASHTO) and state DOTs have been active in the development and  deployment of transportation performance management guidelines. This  resulted in six overarching principles on how national-level performance  measures should be developed and implemented, which guided AASHTO  recommendations on these topics that were submitted to FHWA in  November 2012. However, a few of the grantees we spoke with stated  that they are waiting for final rulemakings and rollout of national  performance measures prior to implementing a performance-based  approach. These officials noted that due to the uncertainty about the  content of the transit-related rules, they are prioritizing local requirements  over the potential MAP-21 requirements.
section_title: States and Other Grantees Reported Potential Data and Cost Challenges to Implementing DOT’s Performance-Based Approach
paragraphs: Officials from states and other grantee agencies and representatives from  national organizations we interviewed reported a variety of challenges to  implementing a performance-based approach, of which data and cost  challenges were most frequently cited. Specifically, officials and  representatives indicated data availability, collection, and analysis would  be significant challenges for their agencies. Officials and representatives  also frequently cited implementation costs, some associated with  addressing the data challenges, although they acknowledged that they  had not or could not estimate the financial impact because DOT’s  rulemakings have not been finalized.
subsections: section_title: States and Other Grantees Reported Potential Data Availability, Collection and Analysis Challenges
paragraphs: Officials and representatives we interviewed anticipated facing challenges  related to the need for data, including their availability, collection and  analysis. We have previously found that implementing a performance  accountability framework necessitates agencies having to collect and  analyze data that are useful and relevant. In describing these  challenges, state and other grantee officials we interviewed  acknowledged that since DOT’s rulemakings had not been finalized yet, it  was unclear the extent to which DOT’s requirements may exceed their  current data and performance-based efforts.
Representatives from 5 national organizations and officials from 18 state  and other grantee agencies we interviewed noted that data may not be  available to measure progress, in part because there is a lack of  consensus on what or how to measure certain performance areas. For  example, state and grantee officials told us they were concerned that data  may not be available to measure congestion, in part, because there is a  lack of consensus among states and grantees regarding how to measure  it. In the absence of consensus, states and other grantees have used a  variety of metrics to measure congestion, which could complicate the  development of national congestion measures. Furthermore, while there  has been a heightened importance placed on congestion data in rural  areas due to the MAP-21 requirements and the increased freight traffic in  rural areas resulting from oil and gas development, most congestion data  have been historically focused on urban areas. Accordingly, a number of  grantees we spoke with told us that available data sets may not reflect  congestion in rural areas. In addition, some safety data may not be  available. For example, while a national database for transportation  fatalities exists, no national data source currently exists for serious  injuries. Although NHTSA and GHSA developed and issued guidelines for  defining serious injuries in 2012, states currently define and collect crash  data differently which makes it hard to aggregate data. For example,  Florida DOT officials told us their definition of a serious injury varied from  other states, making their data inconsistent with other states’. In its  proposed rulemaking on safety performance measures, FHWA  acknowledged the need for a consistent definition of serious injuries, and  proposed steps—including a system for classifying serious injuries—to  address this challenge.
Data collection also poses a challenge for states and grantees. Officials  from 19 states and other grantees, and representatives from 6 of the  national organizations we interviewed reported the following types of  potential challenges associated with the data collection:
States may have to collect additional data. For example, MAP-21  expanded the definition of the National Highway System, adding  50,000 miles including many roads owned and operated by local  agencies; officials from some states noted that they have not  previously collected data on these roads. In addition, FHWA has  proposed that states collect data on the average annual daily traffic on  all public roads. According to some state officials we spoke with,  however, this would be a substantial undertaking. For example, New  York DOT officials told us the state currently has adequate data for  only about 37 percent of its public roads. Officials from some state  DOTs also stated that they may have to collect additional data on  local roads, based on their review of the requirements in FHWA’s  proposed rulemaking on the Highway Safety Improvement Program.  For example, officials from the Montana, North Dakota, South Dakota,  and Wyoming DOTs stated the requirements in FHWA’s proposed  rulemakings may require the collection of additional data on low- volume roads, such as all gravel and other unpaved roads in their  states.
Collecting and standardizing performance data may require additional  collaboration among state and local transportation providers. For  example, as part of standardizing the approach to collecting data,  grantee officials from Michigan reported that it would be difficult to  obtain consistent data from rural bus agencies on the condition of  their assets. Similarly, officials from one Florida MPO anticipated  challenges in obtaining data from transit providers since the MPO has  limited means for ensuring transit agencies collect and report data to  them. These officials stated they are working with these transit  operators to improve cooperation. Also, in order to standardize their  approach to collecting data, grantees stated that they may need to  revise their collection protocols. However, officials from some states  suggested that this could pose a significant challenge because such  protocols necessarily involve new training for agencies, such as police  departments, to ensure that data is collected uniformly or could  involve establishing or revising some thresholds.
Some performance data may be proprietary and expensive to obtain  from private transportation providers. We previously found that state  and local officials had limited data to quantify rail-related freight  impacts (such as traffic delay times or costs) on local communities.  State and grantee officials we interviewed suggested that they may  need to obtain data from private sources, which in some cases may  prove challenging. For example, according to MPO officials in Florida,  freight data necessary to measure congestion (specifically on the  volume of rail transport through local areas) are proprietary, and it  may be difficult or expensive to access this data set from operators.  Similarly, officials from the New York State Association of MPOs also  raised challenges about collecting performance data on freight,  because of their expense and propriety nature. We have previously  found that private railroads are often unwilling to provide detailed  information on train counts, lengths, speed, and times to local  communities to help them quantify traffic impacts at highway-grade  rail crossings.
Moreover, representatives from four national transportation organizations  and officials from 10 states and other grantees we interviewed stated that  even when data can be collected, current staffing levels, areas of  expertise and data systems may not support the analyses that may be  required. For example, representatives from the National Association for  Regional Councils (NARC) told us that ensuring MPOs have sufficient  staff capable of monitoring program performance would be a challenge.   We previously found that some MPOs have limited expertise or staff to  analyze performance data. Furthermore, during the course of our  current review, officials from several MPOs told us that they rely heavily  on their state DOTs to analyze data. Additionally, some state officials  noted that their data systems may not be set up to report or break out  analyses according to DOT’s anticipated requirements. For example,  South Carolina officials stated that, to conduct the analysis potentially  required by DOT’s rulemaking, they may need to revise their existing  legacy database systems or invest in a new database system. FHWA  officials have acknowledged that the data analysis capabilities of some  states and grantees are limited, noting in FHWA’s assessment of state  readiness that 35 states may face data integration challenges and that 28  states may face data processing challenges.
While several state and other grant recipient officials we spoke with  reported anticipating few or no data-related challenges, many of these  officials raised concerns about how data and performance measurement  would be ultimately used by DOT. Some state officials expressed concern  that DOT’s approach will direct states to develop targets based on  outdated national data, as opposed to more current data available to the  states. For example, NHTSA requires that states develop targets for  highway safety based on data collected and submitted into FARS.  However, the lag between when these data are collected and submitted  by states and finalized by NHTSA is approximately 24 months. As a  result, officials from several states, as well as AASHTO and GHSA  representatives, expressed concern that states may be required to set  targets based on outdated FARS data, rather than focus on setting  targets based on the more current data collected by states, and that DOT  would be using outdated data to determine whether states are making  significant progress in meeting their targets. Officials and representatives  also questioned if performance measurement should or would be tied to  DOT’s funding decisions, which some of them opposed, because they  reported that it would be difficult to compare the transportation and transit  priorities and operating environments of different states and transit  agencies. Officials from several states further underscored to us that  performance was a function of available funding and investments, which  has not kept pace with the level of deteriorating infrastructure. Moreover,  officials from some states and other grantees raised concerns that DOT’s  requirements might impede or hinder their approaches that have evolved  over time to meet state and local needs. For example, Washington state  officials anticipated that many of their existing performance management  programs may be broader in scope than MAP-21 performance measures  and provide more detailed information necessary to support state  investment decision processes. The officials noted that the effort required  to comply with the DOT national performance requirements in MAP-21  may affect the state’s ability to support its own state performance  measurement and investment decision process. These officials reported  being concerned with how DOT will integrate legacy performance  programs in states like Washington into the new framework.
DOT officials acknowledged that states and grantees will face some data  challenges as part of the transition to the performance-based approach,  though DOT officials noted they have taken some steps to anticipate  these challenges. For example, DOT has encouraged and worked with  states to share data, best practices, and templates for reporting. In  addition, DOT officials stated that their overall approach has emphasized  the use of existing data sets and plans to further detail its approach to  ensuring data availability through its upcoming rulemakings. In addition,  FHWA has used information gathered through its assessment of state  readiness to help inform its technical assistance program, which DOT  anticipates will address some of the above mentioned challenges.
section_title: Implementation Costs Could Vary and Be a Significant Challenge for Some States and Grantees
paragraphs: Representatives from 5 national organizations and officials from 22 of the  29 states and other grantees we spoke to reported potential challenges  with the costs of implementing a national performance measurement  system. According to FHWA officials, the total implementation costs of  any one of its proposed rulemakings to states and grantees in any one  year will not exceed $100 million for any one rule. For example, in March  2014, FHWA estimated the 10-year costs associated with the eight new  requirements proposed in its Safety Performance Measure rulemaking  (see table 1) were approximately $67 million. About $26 million of this is  estimated to be one-time costs incurred by state DOTs and MPOs to  standardize and update data collection and reporting systems, as well as  the increase in level of effort from labor to establish and report targets. In  calculating this cost, DOT assumed that a portion of MPOs,  approximately half, would establish their own targets and that a portion  would adopt state DOT targets. FHWA acknowledged that larger MPOs  may have more resources available to develop performance targets and  that it believed smaller MPOs (those with populations under 200,000)  would have no incremental costs as they would adopt state-established  targets. DOT’s estimates for other rulemakings reflect the associated  costs of activities (e.g., planning and data collection) beyond what is  currently performed by states, MPOs, and providers of public  transportation.
On a per-state or agency basis, the cost of implementation may vary  depending on the characteristics of each grantee, and some states  reported that these costs may exceed the average cost estimated by  DOT. One MPO official reported that her organization believed DOT’s  estimate that the cost of administering oversight of transportation  planning would increase no more than 15 percent was low. According to  officials from some of the national organizations and officials from about  one-third of states and grantees we interviewed, implementation costs to  hire additional staff or allow for staff time and travel may be burdensome,  particularly for smaller or more rural states and grantees. Officials from  the New York State Association of MPOs explained that hiring and  training new staff to collect and analyze performance data would require  additional funding and that smaller MPOs would be more acutely affected  than the larger MPOs, particularly those that do not currently conduct  sophisticated analysis with the performance data. One rural transit  provider we interviewed also relayed concerns about having to potentially  hire an additional staff person to serve as a chief safety officer in order to  meet this requirement being considered by FTA. Officials from some  planning organizations we interviewed underscored their lack of planning  professionals and how the few individuals employed by their agencies  were already responsible for fulfilling multiple roles.
Implementation costs may also vary by grantees’ experience in different  surface transportation areas. In summer 2014, FHWA queried its division  offices about the readiness of states to implement performance-based  approaches to transportation projects in their states. According to the  division offices, in the areas of congestion, air quality and freight, only 20  to 40 percent of states had adopted performance management practices,  and only about 10 to 20 percent of states are very highly prepared to  carry out performance-based programs in these areas. For example, in  Nevada, FHWA officials reported that state officials may need some  additional assistance to integrate data and develop metrics for  congestion, freight, and air quality. On the other hand, state DOTs and  other grantees may be better positioned to meet DOT’s performance  measurement requirements in other areas, such as safety, pavement,  and bridge condition. According to FHWA’s assessment, about 40 to 50  percent of the states are ready to carry out a performance-based program  in some areas, such as safety, pavement, and bridge condition, where,  historically, measures and targets have been used by states to evaluate  program performance. According to the division offices, in these areas  about 60 to 70 percent of the states integrate performance into their long- range plans, develop quantifiable targets to guide investment decision  making, and develop plans that document their strategies to achieve  targets.
According to AASHTO, some of its members have concerns that  collecting data to support these rules may reduce the portion of their  federal highway grants available to fund transportation projects over the  long term. While these costs are eligible for federal planning funds,  officials from several grantee agencies said they do not expect additional  federal funding for implementation. Representatives from several national  organizations and officials from some states and other grantees we  interviewed emphasized that DOT should, whenever possible, leverage  existing resources or data or procure data sources itself. Representatives  from the American Public Transportation Association suggested that DOT  consider the data that it will be asking transit agencies to collect, pointing  to recent FTA-sponsored asset management pilot projects that  demonstrated the data collection process can be burdensome and should  be based on the types of information that states and local transit agencies  currently collect. Similarly, representatives from the National Association  of Development Organizations also underscored the cost of acquiring and  analyzing data and the need to structure data collection around existing  sources. DOT officials agreed that data availability, collection, analysis  and associated costs present potential challenges, and stated that in  order to address this situation, DOT has emphasized the use of existing  data sets and intends to further detail its approach to ensuring data  availability through its upcoming rulemakings.
section_title: Concluding Observations
paragraphs: DOT’s effort to develop a performance-based approach for its various  surface transportation grant recipients represents a significant shift for the  Department, transforming the federal involvement in surface  transportation to a more outcome-oriented approach, as we have  recommended since 2008. In many cases, the shift to a performance- based approach will hold states and other grantees accountable for  performance for the first time. Such an approach could provide a strong  foundation for holding grant recipients responsible for achieving federal  objectives and measuring overall program performance. However, the  development and implementation of a performance-based approach is  not without challenges. Notably, as NHTSA’s implementation of a  performance-based approach has demonstrated, this broader effort of  incorporating performance-based principles into the overall surface  transportation program may take longer than Congress originally  anticipated. In addition, while many grantees have been evaluating the  performance of transportation projects for many years, experiences with  performance measurement varies and even the most experienced  grantees may face challenges aligning their individual approaches with  DOT’s national approach. Nevertheless, given the billions of dollars  provided annually in federal funding to improve the nation’s highway and  transit infrastructure and safety—and competing demands for that  funding—the ability to assess the outcomes of federal surface  transportation programs and hold grant recipients accountable for results  could help all parties more efficiently allocate federal funds to address  specific challenges and areas of concern.
section_title: Agency Comments
paragraphs: We provided a draft of this report to DOT for review and comment. DOT  provided technical comments, which we incorporated as appropriate. We  are sending copies of this report to interested congressional committees  and the Secretary of Transportation. In addition, this report will be   available at no charge on GAO’s website at http://www.gao.gov. If you or  your staff have any questions about this report, please contact me at  (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. GAO staff who made major contributions to this report are  listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of our review were to determine: (1) the progress that DOT  has made in developing a national performance-based approach to  surface transportation and (2) the challenges states and other grantees  report facing in implementing this approach.
To address these objectives, we reviewed scholarly/peer-reviewed,  government, trade, and non-profit literature from 2011 through 2014, the  time period just before and since MAP-21 was enacted. The reviewed  literature included those published by the National Cooperative Highway  Research Program, which is sponsored by FHWA and American  Association of State Highway and Transportation Officials (AASHTO) and  presented at the Transportation Research Board annual conference. The  literature examined local and national transportation performance based  approaches. We also reviewed DOT-conducted performance  management case studies and workshop summaries and reports that  were publicly available, as well as, our body of work on performance  management in the federal government and transportation programs.
To examine the progress that DOT has made in developing a national  performance-based approach for surface transportation programs, we  reviewed related requirements and timetables in MAP-21 for FHWA, FTA,  and NHTSA as well as interim and proposed rulemakings developed by  these operating administrations in response to these requirements. We  did not evaluate the potential effectiveness of the proposed rules because  the content may change before rules are finalized in response to  comments submitted by grantees and other parties. We also analyzed  DOT documents, information obtained through interviews with DOT  officials, guidance issued through the rulemaking process, and  information from grantee interviews. We also analyzed information from  the Report on DOT Significant Rulemakings, issued monthly, which  provides the original, actual, and projected dates for the major milestones  in the rulemaking process to understand the progress FHWA, FTA, and  NHTSA have made developing the performance-based approach. In the  event that actual dates were unavailable, projected dates, obtained  directly from the Report on DOT Significant Rulemakings were used to  calculate the length of milestones. We interviewed FHWA, FTA, and  NHTSA officials to understand the factors affecting DOT’s progress in  developing the performance-based approach.
To understand the challenges states and other grantees face in  implementing a performance-based approach to surface transportation,  we analyzed comments submitted to DOT by and interviewed  representatives from seven national transportation organizations (such as  AASHTO, Association of Metropolitan Planning Organizations, and  American Public Transportation Association), which broadly represent  state and local transportation, transit, and planning interests and  perspectives, both in urban and rural communities.
We also reviewed the comments submitted to DOT and conducted semi- structured interviews with state and local officials from 29 grantee  agencies in 13 states—Florida, Maryland, Michigan, Montana, New York,  North Dakota, Oregon, South Carolina, South Dakota, Virginia,  Washington, Wyoming, and Utah. We developed criteria—population  growth trends, population size, geographic location, transit infrastructure  age, and experiences managing and measuring transportation project  performance—which we used to select a cross-section of grantee  agencies (i.e., state transportation agencies, state highway safety offices,  metropolitan and rural planning organizations, and transit agencies). We  also relied on suggestions from trade organizations and DOT to identify  agencies that would meet our criteria and to identify specific state and  local officials. Based on these criteria, in four states, we interviewed  officials from the state department of transportation, one or more  metropolitan planning organizations, local transit providers, and DOT  regional oversight offices; in the remaining nine states, we interviewed  officials from at least one state transportation agency, planning  organization or transit agency in that state. In addition to discussing  anticipated challenges, state and grantee officials also provided  information about state and local transportation planning processes, the  experience and extent of available performance management data, and  DOT’s progress to date with its outreach and rulemaking. The results of  interviews with state and local officials from these 29 grantee agencies  cannot be generalized to all federal transportation grantees; however,  they provided insight into the types of challenges that some grantees  anticipate in implementing DOT’s future rulemaking and overall  performance based approach, as well as, perspectives on DOT’s  outreach and rulemaking progress.
To further understand the challenges that states and other grantees may  face, we also reviewed the draft results of a questionnaire DOT  administered to its 52 FHWA Division Offices assessing states’ readiness  to implement a performance-based approach. This questionnaire was  administered from July through August 2014, and its purpose was to  assist FHWA in how to best design program support for states in meeting  requirements under DOT’s future performance-based approach.
We conducted this performance audit from May 2014 to January 2015 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Department of Transportation Rulemakings for Selected MAP-21 Performance Requirements
paragraphs: Appendix II: Department of Transportation  Rulemakings for Selected MAP-21  Performance Requirements The Statewide, Metropolitan, and Non-Metropolitan Planning rulemaking has been developed jointly  with the Federal Transit Administration.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Steve Cohen (Assistant  Director), Sara Vermillion (Assistant Director), Timothy Bober, Melissa  Bodeau, Elizabeth Curda, Bert Japikse, Terence Lam, Sara Ann  Moessbauer, Josh Ormond, Amy Suntoke, Friendly Vang-Johnson, and  Matt Voit made key contributions to this report.