Public Transportation: New Starts Program Challenges and Preliminary Observations on Expediting Project Development
Released: 2009-06-03T00:00:00

Letter

Background

FTA generally funds New Starts projects through FFGAs, which are required by statute to establish the terms and conditions for federal participation in a New Starts project. FFGAs also define a project’s scope, including the length of the system and the number of stations; its schedule, including the date when the system is expected to open for service; and its cost. For projects to obtain FFGAs, New Starts projects must emerge from a regional, multimodal transportation planning process. The first two phases of the New Starts process—systems planning and alternatives analysis—address this requirement. The systems planning phase identifies the transportation needs of a region, while the alternatives analysis phase provides information on the benefits, costs, and impacts of different options, such as rail lines or bus routes, in a specific corridor versus a region. The alternatives analysis phase results in the selection of a locally preferred alternative, which is the New Starts project that FTA evaluates for funding. After a locally preferred alternative is selected, the project sponsor submits an application to FTA for the project to enter the preliminary engineering phase. When this phase is completed and federal environmental requirements under the National Environmental Policy Act are satisfied, FTA may approve the project’s advancement into final design, after which FTA may recommend the project for an FFGA and advance the project into construction. FTA oversees grantees’ management of projects from the preliminary engineering phase through the construction phase.

To help inform administration and congressional decision makers about which projects should receive federal funds, FTA currently distinguishes among proposed projects by evaluating and assigning ratings to various statutory evaluation criteria—including both local financial commitment and project justification criteria—and then assigning an overall project rating. (See fig. 1.) These evaluation criteria reflect a range of benefits and effects of the proposed project, such as cost-effectiveness, as well as the ability of the project sponsor to fund the project and finance the continued operation of its transit system. FTA has developed specific measures for each of the criteria outlined in the statute. On the basis of these measures, FTA assigns the proposed project a rating for each criterion and then assigns a summary rating for local financial commitment and project justification. These two ratings are averaged together to create an overall rating, which is used in conjunction with a determination of the project’s “readiness” for construction to determine what projects are recommended for funding. Projects are rated at several points during the New Starts process—as part of the evaluation for entry into the preliminary engineering and the final design phases, and yearly for inclusion in the New Starts Annual Report. As required by statute, the administration uses the FTA evaluation and rating process, along with the development phases of New Starts projects, to decide which projects to recommend to Congress for funding.

Previous GAO Work Has Identified Key Challenges in Managing New Starts Program

Frequent Changes to the New Starts Program Have Sometimes Led to Confusion and Delays

Numerous changes have been made to the New Starts program over the last decade. These changes include statutory, regulatory, and administrative changes to the program. For example, we reported in 2005 that FTA had implemented 16 changes to the New Starts application, evaluation, rating, and project development oversight process since the fiscal year 2001 evaluation cycle. Additional changes have been made to the program since 2005. Examples of these changes made to the program over the last 10 years, in chronological order, include the following.

New data collection requirements: Starting with the fiscal year 2004 evaluation cycle, FTA required project sponsors seeking an FFGA to submit a plan for the collection and analysis of information to determine the impacts of the project and the accuracy of the forecasts that were prepared during project planning and development. SAFETEA-LU subsequently codified this “before and after” study requirement.

Evaluation measures revised: FTA revised its cost-effectiveness and mobility improvements criteria by adopting the Transportation System User Benefits (TSUB) measure that includes benefits for both new and existing transit system riders. Although project sponsors generally view the new cost-effectiveness measure of cost per hour of TSUB as an improvement over the previous measure of cost per new rider, we have reported that some project sponsors have had difficulties correctly calculating the TSUB value for their projects, resulting in delays and additional costs as they conduct multiple iterations of the TSUB measure.

New analysis requirement added: Starting with the fiscal year 2005 evaluation cycle, FTA required project sponsors to complete risk assessments. Since implementation, the form and timing of the risk assessments have evolved since 2003, but the intent of the assessments remains to identify the issues that could affect the project’s schedule or cost.

Policy on funding recommendations changed: In 2005, the administration informed the transit community that it would target its funding recommendations to projects that achieve a cost-effectiveness rating of medium or higher. Previously, the administration had recommended projects for funding that had lower cost-effectiveness ratings, if they met all other criteria.

New programs established: SAFETEA-LU established the Small Starts program, a new capital investment grant program, simplifying the requirements imposed for those seeking funding for lower-cost projects. This program is intended to advance smaller-scale projects through an expedited and streamlined evaluation and rating process. FTA subsequently introduced a separate eligibility category within the Small Starts program for Very Small Starts projects. Very Small Starts projects qualify for an even simpler evaluation and rating process.

New evaluation criteria introduced: Given past concerns that the evaluation process did not account for a project’s impact on economic development, SAFETEA-LU added economic development to the list of project justification criteria that FTA must use to evaluate and rate New Starts projects.

Although the impetus for each change varied, FTA officials stated that, in general, all of the changes the agency has initiated were intended to make the process more rigorous, systematic, and transparent. This increased rigor, in turn, helps FTA and project sponsors deliver more New Starts projects within budget and on time, according to FTA.

However, frequent changes to the New Starts program create challenges for project sponsors. For example, we have previously reported that some project sponsors told us that FTA did not create clear expectations or provide sufficient guidance about certain changes. In addition, we reported that project sponsors said some changes made the application process more expensive and required them to spend significantly more time to complete the application. We have heard similar concerns from project sponsors during our ongoing review. Specifically, some project sponsors we interviewed told us that they have had to redo completed analyses because FTA applies regulatory and administrative changes to projects in the pipeline. In general, according to project sponsors and other stakeholders we have spoken to, this rework adds time and costs to completing the New Starts project development process.

FTA’s Project Evaluation Process Does Not Currently Capture All Project Benefits

FTA currently assigns a 50 percent weight to both the cost-effectiveness and the land use criteria when developing the project justification summary rating. The other project justification criteria are not weighted, although the mobility improvements criterion is used as a “tiebreaker.” FTA officials have told us that they do not currently use the environmental benefits and operating efficiencies criteria in determining the project justification summary rating because the measures do not, as currently structured, provide meaningful distinctions among competing New Starts projects. FTA does not use the economic development criterion because of difficultly developing a measure that is separate and distinct from the land use criterion. We have found in the past that many project sponsors had similar views, noting that individual projects are too small to have much impact, in terms of, for example, air quality, on the whole region or the whole transit system. In contrast, FTA officials have told us that the cost-effectiveness and land use measures help to make meaningful distinctions among projects. For example, according to FTA, existing transit supportive land use plans and policies demonstrate an area’s commitment to transit and are a strong indicator of a project’s future success. Furthermore, according to many FTA officials, experts, and the literature we have consulted, FTA’s cost-effectiveness measure accounts for most secondary project benefits, including economic development, because these benefits are typically derived from mobility improvements that reduce users’ travel times. Therefore, developing new measures for these other criteria may result in the double-counting of certain project benefits.

However, in 2008, we reported that FTA’s evaluation measures could be underestimating total project benefits. FTA’s measure of cost- effectiveness, for instance, considers how the mobility improvements from a proposed project will reduce users’ travel times. Although this measure can capture most secondary project benefits, it does not account for benefits for non-transit users (e.g., highway travel time savings) or capture any economic development benefits that are not directly correlated to mobility improvements. The omission of these benefits means proposed projects that convey significant travel time savings for motorists, for example, are not recognized in the selection process. Beyond the cost- effectiveness measure, we reported that project sponsors and experts expressed frustration that FTA does not include certain criteria in the calculation of project ratings, such as economic development and environmental benefits. They noted that this practice limits the information captured on projects, particularly since these are important benefits of transit projects at the local level. As a result, FTA may be underestimating projects’ total benefits, particularly in areas looking to use these projects as a way to relieve congestion or promote more high- density development. In these cases, however, the extent to which FTA’s current approach to estimating benefits affects how projects are ranked in FTA’s evaluation and ratings process is unclear.

FTA officials have acknowledged these limitations, but noted that improvements in local travel models are needed to resolve some of these issues. In particular, many local models used to estimate future travel demand for New Starts are incapable of reliably estimating highway travel time savings as a result of the proposed project, according to FTA officials.

There is great variation in the models local transportation planning agencies use to develop travel forecasts (which underlie many of the New Starts measures), producing significant variation in forecast quality and limiting the ability to assess quality against the general state of practice. In 2008, we made a series of recommendations designed to address the limitations of FTA’s current evaluation process, including recommending that (1) the Secretary of Transportation seek additional resources to improve local travel models in the next authorizing legislation to improve the New Starts evaluation process and the measures of project benefits; (2) FTA establish a timeline for issuing, awarding, and implementing the result of its request for proposals on short- and long-term approaches to measuring highway user benefits from transit improvements; (3) the Administrators of FTA and Federal Highway Administration collaborate in efforts to improve the consistency and reliability of local travel models; and (4) the Administrator of FTA establish a timeline for initiating and completing its longer-term effort to develop more robust measures of transit projects’ environmental benefits. FTA is working to address these recommendations. For instance, FTA conducted a colloquium on environmental benefits of transit projects in October 2008, which resulted in a discussion paper on the evaluation of economic development. Further, in a Federal Register Notice published on January 26, 2009, FTA issued and sought comments on a discussion paper on new ways of evaluating economic development effects. FTA is now reviewing comments on that paper.

In May 2009, FTA also took steps to address concerns about the exclusion of some project justification criteria from the evaluation process. In a Notice of Availability for New Starts and Small Starts Policies and Procedures and Requests for Comments in the Federal Register, FTA proposed changing the weights assigned for the project justification criteria for New Starts projects. Specifically, FTA proposes to set the weights at 20 percent each for the mobility, cost-effectiveness, land use, and economic development criteria, and 10 percent each for operating efficiencies and environmental benefits. According to FTA, these changes reflect statutory direction that project justification criteria should be given “comparable, but not necessarily equal, numerical weight” in calculating the overall project rating. FTA is currently soliciting public comments on these proposed changes.

Striking Appropriate Balance between Maintaining a Robust Evaluation Process and Minimizing Complexity Is Challenging

We reported in 2008 that experts and some project sponsors we spoke with generally support FTA’s quantitatively rigorous process for evaluating proposed transit projects but are concerned that the process has become too burdensome and complex, and as noted earlier, may underestimate certain project benefits. For example, several experts and transportation consultants told us that although it is appropriate to measure the extent to which transit projects create primary and secondary benefits, such as mobility improvements and economic development, it is difficult to quantify all of these projected benefits. Additionally, several project sponsors noted that the complexity of the evaluation process can necessitate hiring consultants to handle the data requests and navigate the application process—which could increase the project’s costs. Our previous reviews of the New Starts program have noted similar concerns from project sponsors. For example, in 2007, we reported that a majority of project sponsors told us that the complexity of the requirements—such as the analysis and modeling required for travel forecasts—creates disincentives for entering the New Starts pipeline. Sponsors also said that the expense involved in fulfilling the application requirements, including the costs of hiring additional staff and consultants, discourages agencies with less resources from applying for this funding.

In response to such concerns, FTA has tried to simplify the evaluation process in several ways. For example, as previously mentioned, FTA established the Very Small Starts eligibility category within the Small Starts program for projects less than $50 million in total cost. This program further simplifies the application requirements in place for the Small Starts program, which funds lower-cost projects, such as bus rapid transit and streetcar projects. Additionally, in its New Starts program, FTA no longer rates projects on the operating efficiencies criterion because, according to FTA, operating efficiencies are already sufficiently captured in FTA’s cost-effectiveness measures, and the measure did not adequately distinguish among projects. Thus, projects no longer have to submit information on operating efficiencies. Likewise, FTA no longer requires project sponsors to submit information on environmental benefits because it found that the information gathered did not adequately distinguish among projects and that EPA’s ambient air quality rating was sufficient. FTA also commissioned a study by Deloitte in June 2006 to review the project development process and identify opportunities for streamlining or simplifying the process. This study identified a number of ways that FTA’s project development process could be streamlined, including revising the policy review and issuance cycle to minimize major policy and guidance changes to every 2 years and conducting a human capital assessment to identify skill gaps and opportunities for reallocating resources in order to enhance FTA’s ability to review and assist New Starts projects in a timely and efficient manner. According to FTA, the agency has implemented 75 percent of Deloitte’s recommendations; some of the other recommendations are on hold pending the upcoming reauthorization of the surface transportation program, including the New Starts program.

Potential Options to Expedite Project Development in the New Starts Program

As part of our ongoing work, we are reviewing existing research, including past GAO reports, analyzing data on the length of time it takes for projects to complete the New Starts process, and interviewing project sponsors, industry stakeholders and consultants, and transportation experts to identify options to expedite project development in the New Starts program. Using these sources, we have preliminarily identified the following options. While each option could help expedite project development, each option has advantages and disadvantages to consider and some options could require legislative changes. In addition, each option would likely require certain trade-offs, namely reducing the level of rigor in the evaluation process in exchange for a more streamlined process. The discussion that follows is not intended to endorse any potential option, but instead to describe some potential options for expediting project development. We will continue to work with FTA and other stakeholders to identify other options as well as examine the merits and challenges of all identified options for inclusion in our report later this summer.

Tailor the New Starts evaluation process to risks posed by the projects: Project sponsors, consultants, and experts we interviewed suggested that FTA adopt a more risk-based evaluation process for New Starts projects based on the project’s costs or complexity, the federal share of the project’s costs, or the project sponsor’s New Starts experience. For example, FTA could align the level of oversight with the proposed federal share of the project—that is, the greater the financial exposure for the federal government, the greater the level of oversight. Similarly, FTA could reduce or eliminate certain reviews for project sponsors who have successfully developed New Starts projects in the past, while applying greater oversight to project sponsors who have no experience with the New Starts process. We have noted the value in using risk-based approaches to oversight. For example, we have previously reported that assessing risks can help agencies allocate finite resources and help policy makers make informed decisions. By adopting a more risk-based approach, FTA could allow select projects to move more quickly through the New Starts process and more efficiently use its scarce resources. However, the trade-off of not applying all evaluation measures to every project is that FTA could miss the opportunity to detect problems early in the project’s development.

Consider greater use of letters of intent and early systems work agreements: The linear, phased evaluation process of the New Starts program hampers project sponsors’ ability to utilize alternative project delivery methods, such as design-build, according to project sponsors. These alternative project delivery methods have the potential to develop a project cheaper and quicker than traditional project delivery methods can. However, project sponsors told us it is difficult to attract private sector interest early enough in the process to use alternative project delivery methods because there is no guarantee that the project will ultimately receive federal funding through the New Starts program. The Deloitte study also noted that New Starts project sponsors miss the opportunity to use alternative project delivery methods because of the lack of early commitment of federal funding for the projects. To encourage the private sector involvement needed, project sponsors, consultants, and experts we interviewed suggested that FTA use letters of intent or early systems work agreements. Through a letter of intent, FTA announces its intention to obligate an amount from future available budget authority to a project. A challenge of using letters of intent is that they can be misinterpreted as an obligation of federal funds, when in fact they only signal FTA’s intention to obligate future funds should the project meet all New Starts criteria and requirements. In contrast, an early systems work agreement obligates an amount of available budget authority to a project. The challenge of using an early systems work agreement is that FTA can only use these agreements with projects that will be granted an FFGA, thus limiting FTA’s ability to use these agreements for projects in the pipeline.

Consistently use road maps or similar project schedules: Project sponsors said that FTA should more consistently use road maps or similar tools to define the project sponsor’s and FTA’s expectations and responsibilities for moving the project forward. Without establishing these expectations, project sponsors have little information about how long it will take FTA to review its request to move from alternatives analysis to preliminary engineering, for example. This lack of information makes it difficult for the project sponsor to effectively manage the project. Given the benefits of clearly setting expectations, Deloitte recommended that FTA use road maps for all projects. FTA has used road maps for select projects, but the agency does not consistently use them for all projects. A limitation of using road maps is that expected time frames are subject to change—that is, project schedules often change as a project evolves throughout the development process. Furthermore, every project is unique, making it difficult to set a realistic time frame for each phase of development. Consequently, the road maps can provide only rough estimates of expected time frames.

Combine two or more project development phases: Project sponsors and consultants told us that waiting for FTA’s approval to enter preliminary engineering, final design, and construction can cause delays. While FTA determines whether a project can advance to the next project development phase, work on the project essentially stops. Project sponsors can advance the project at their own risk, meaning they could have to redo the work if FTA does not subsequently approve an aspect of the project. The amount of time it takes for FTA to determine whether a project can advance can be significant. For example, one project sponsor told us that FTA’s review of its application to advance from alternatives analysis to preliminary engineering took 8 months; about the same amount of time it took the project sponsor to complete alternatives analysis. FTA officials told us the length of time for reviews depends on a number of factors, most importantly the completeness and accuracy of the project sponsor’s submissions. To reduce the “start/stop” phenomena project sponsors described, FTA could seek a legislative change to combine two or more of the statutorily required project development phases—for example, combining the preliminary engineering and final design phases. The Deloitte study also recommended that FTA redefine or more clearly define the project phases to more accurately reflect FTA’s current requirements and to better accommodate alternative delivery methods.

Apply changes only to future projects: Project sponsors told us that the frequent changes to the New Starts program can result in additional costs and delays as project sponsors are required to redo analyses to reflect the changes. In an attempt to create a process that provides more stability for project sponsors, in May 2006, FTA modified its policy to allow a project that has been approved for entry into final design not be subject to changes in the New Starts policy and guidance. However, this policy change does not apply to projects approved for entry into preliminary engineering, which is the New Starts project development phase that has the most requirements for project sponsors and the phase where project sponsors told us that frequent changes to the project by sponsors and to the New Starts process by FTA result in additional costs and delays. Furthermore, another project sponsor noted that new requirements cause delays because each element of a proposed project is interrelated, so changing one requirement can stop momentum on a project. To avoid this rework, some project sponsors, consultants, and experts we interviewed suggested that FTA apply changes only to future projects, not projects currently in preliminary engineering. However, by not applying changes to projects in preliminary engineering, FTA could miss the opportunity to enhance its oversight of these projects.

Mr. Chairman, this concludes my statement. I would be pleased to answer any questions that you or other members of the subcommittee may have at this time.

GAO Contact and Staff Acknowledgments

For further information on this testimony, please contact A. Nicole Clowers, Acting Director, Physical Infrastructure Issues, at (202) 512-2834, or clowersa@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this statement. Key contributors to this testimony were Kyle Browning, Gary Guggolz, Raymond Sendejas, and Carrie Wilks.

Related GAO Products

Public Transportation: Improvements Are Needed to More Fully Assess Predicted Impacts of New Starts Projects. GAO-08-844. Washington, D.C.: July 25, 2008.

Public Transportation: Future Demand Is Likely for New Starts and Small Starts Programs, but Improvements Needed to the Small Starts Application Process. GAO-07-917. Washington, D.C.: July 27, 2007.

Public Transportation: New Starts Program Is in a Period of Transition. GAO-06-819. Washington, D.C.: August 30, 2006.

Public Transportation: Preliminary Information on FTA’s Implementation of SAFETEA-LU Changes. GAO-06-910T. Washington, D.C.: June 27, 2006.

Opportunities Exist to Improve the Communication and Transparency of Changes Made to the New Starts Program. GAO-05-674. Washington, D.C.: June 28, 2005.

Mass Transit: FTA Needs to Better Define and Assess Impact of Certain Policies on New Starts Program. GAO-04-748. Washington, D.C.: June 25, 2004.

Mass Transit: FTA Needs to Provide Clear Information and Additional Guidance on the New Starts Ratings Process. GAO-03-701. Washington, D.C.: June 23, 2003.

Mass Transit: FTA’s New Starts Commitments for Fiscal Year 2003. GAO-02-603. Washington, D.C.: April 30, 2002.

Mass Transit: FTA Could Relieve New Starts Program Funding Constraints. GAO-01-987. Washington, D.C.: August 15, 2001.

Mass Transit: Implementation of FTA’s New Starts Evaluation Process and FY 2001 Funding Proposals. GAO/RCED-00-149. Washington, D.C.: April 28, 2000.

Mass Transit: Status of New Starts Transit Projects With Full Funding Grant Agreements. GAO/RCED-99-240. Washington, D.C.: August 19, 1999.

Mass Transit: FTA’s Progress in Developing and Implementing a New Starts Evaluation Process. GAO/RCED-99-113. Washington, D.C.: April 26, 1999.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.