Public Transportation: Improvements Are Needed to More Fully Assess Predicted Impacts of New Starts Projects
Released: 2008-07-25T00:00:00

Letter

Background

FTA generally funds New Starts projects through FFGAs, which are required by statute to establish the terms and conditions for federal participation in a New Starts project. FFGAs may also define a project’s scope, including the length of the system and the number of stations; its schedule, including the date when the system is expected to open for service; and its cost. For projects to obtain FFGAs, New Starts projects must emerge from a regional, multimodal transportation planning process. The first two phases of the New Starts process—systems planning and alternatives analysis—address this requirement. The systems planning phase identifies the transportation needs of a region, while the alternatives analysis phase provides information on the benefits, costs, and impacts of different options, such as rail lines or bus routes, in a specific corridor versus a region. The alternatives analysis phase results in the selection of a locally preferred alternative, which is the New Starts project that FTA evaluates for funding. After a locally preferred alternative is selected, the project sponsor submits an application to FTA for the project to enter the preliminary engineering phase. When this phase is completed and federal environmental requirements are satisfied, FTA may approve the project’s advancement into final design, after which FTA may approve the project for an FFGA and proceed to construction. FTA oversees grantees’ management of projects from the preliminary engineering phase through the construction phase.

To help inform administration and congressional decisions about which projects should receive federal funds, FTA currently distinguishes between proposed projects by evaluating and assigning ratings to various statutory evaluation criteria—including both project justification and local financial commitment criteria—and then assigning an overall project rating. (See fig. 1.) These evaluation criteria reflect a broad range of benefits and effects of the proposed project, such as cost-effectiveness, as well as the ability of the project sponsor to fund the project and finance the continued operation of its transit system. FTA has developed specific measures for each of the criteria outlined in the statute. On the basis of these measures, FTA assigns the proposed project a rating for each criterion and then assigns a summary rating for local financial commitment and project justification. These two ratings are averaged together, and then FTA assigns projects a “high,” “medium-high,” “medium,” “medium-low,” or “low” overall rating, which is used to rank projects and determine what projects are recommended for funding. Projects are rated at several points during the New Starts process—as part of the evaluation for entry into the preliminary engineering and the final design phases, and yearly for inclusion in the New Starts Annual Report. As required by SAFETEA-LU, the administration uses the FTA evaluation and rating process, along with the phase of development of New Starts projects, to decide which projects to recommend to Congress for funding. Although many projects receive a summary rating that would make them eligible for an FFGA, only a few are proposed for an FFGA in a given fiscal year. FTA proposes FFGAs for those projects that are projected to meet the following conditions during the fiscal year for which funding is proposed: All nonfederal project funding must be committed and available for the project.

The project must be in the final design phase and have progressed far enough for uncertainties about costs, benefits, and impacts (e.g., financial or environmental) to be minimized.

The project must meet FTA’s tests for readiness and technical capacity, which confirm that there are no remaining cost, project scope, or local financial commitment issues.

SAFETEA-LU introduced a number of changes to the New Starts program, including some that affect the evaluation and rating process. For example, given past concerns that the evaluation process did not account for a project’s impact on economic development and FTA’s lack of communication to sponsors about upcoming changes, the statute added economic development to the list of project justification criteria that FTA must use to evaluate and rate New Starts projects, and requires FTA to issue notice and guidance each time significant changes are made to the program. SAFETEA-LU also established the Small Starts program, a new capital investment grant program, simplifying the requirements imposed for those seeking funding for lower-cost projects such as bus rapid transit, streetcar, and commuter rail projects. This program is intended to advance smaller-scale projects through an expedited and streamlined evaluation and rating process. FTA also subsequently introduced a separate eligibility category within the Small Starts program for “Very Small Starts” projects. Small Starts projects that qualify as Very Small Starts are simple, low-cost projects that FTA has determined qualify for a simplified evaluation and rating process.

In addition to implementing the Small Starts program, FTA has taken other steps to implement SAFETEA-LU changes to the New Starts evaluation process. For example, FTA incorporated economic development into the existing evaluation framework by considering the information provided by project sponsors as an “other factor.” FTA also sought public comments on different proposals for revising the evaluation process to better reflect the statute through the Advanced Notice of Proposed Rulemaking (ANPRM) and the final NPRM for the New Starts and Small Starts programs. However, following concerns voiced by Members of Congress and the transit industry about the weights placed on different project benefits, FTA was prohibited from using funds to proceed with the rulemaking process, with the exception of reviewing comments, under the fiscal year Consolidated Appropriations Act of 2008. Figure 2 shows a timeline of FTA’s efforts to date to implement SAFETEA-LU changes to the New Starts evaluation and ratings process.

FTA’s Project Evaluation Measures Include a Range of Information, but Not All Project Benefits Are Fully Captured

FTA primarily uses the cost-effectiveness and land use criteria to evaluate New Starts projects, but concerns have been raised about the extent to which the measures for these criteria capture total project benefits. Specifically, FTA’s TSUB measure considers how the mobility improvements from a proposed project will reduce users’ travel times. According to FTA officials, experts, and the literature we consulted, the TSUB measure accounts for most secondary project benefits, including economic development, because these benefits are typically derived from mobility improvements that reduce users’ travel times. However, project sponsors and experts raised concerns about how FTA currently measures and weights different project justification criteria, noting that these practices may underestimate some project benefits. For example, some experts and project sponsors we spoke to said that the TSUB measure does not account for benefits for nontransit users or capture any economic development benefits that are not directly correlated to mobility improvements. As a result, FTA may be underestimating projects’ total benefits, particularly in areas looking to use these projects as a way to relieve congestion or promote more high-density development. In these cases, it is unclear the extent to which FTA’s current approach to estimating benefits impacts how projects are ranked in FTA’s evaluation and ratings process. FTA officials acknowledged these limitations, but noted that improvements in local travel models are needed to resolve some of these issues.

FTA Emphasizes Cost- Effectiveness and Land Use in Developing Project Justification Ratings

FTA currently relies on the cost-effectiveness and land use criteria to evaluate and rate New Starts projects. Specifically, FTA assigns a weight of 50 percent to both the cost-effectiveness and land use criteria when developing project justification ratings. Table 1 provides a summary of all project justification criteria that FTA is required to review, the measures it uses to evaluate these criteria, and how this information is used to rate projects.

To evaluate the land use criterion, FTA has developed and uses three qualitative land use measures: land use in the project area, the extent to which the area has transit supportive plans and policies, and the performance and impacts of these policies. For example, to determine whether a project’s surrounding area has transit supportive plans and policies, FTA examines whether there are growth management strategies and transit supportive corridor policies in place, the extent to which zoning regulations near stations are transit supportive, and the tools available to implement land use policies.

To evaluate cost-effectiveness, FTA relies on the TSUB measure and costs. The TSUB measure captures predicted improvements in mobility caused by the implementation of a project. In particular, TSUB captures transit users’ cost and travel time savings, as well as improvements in comfort, convenience, and reliability of travel. Project sponsors use local travel models to forecast ridership and simulate trips taken in 2030, the forecast year used to estimate savings over time for two alternatives. To evaluate the benefits for these two alternatives, FTA uses the outputs from these models to consider and weigh a range of attributes, such as time spent waiting at and walking to the transit station, and calculates the perceived level of time savings associated with a given project. The first alternative, known as the baseline alternative, assumes low-cost improvements to the project area’s current transportation network, while the second alternative-—the “build alternative”—assumes the proposed New Starts transit project is constructed. As outlined in figure 3, FTA uses the forecasts for these two alternatives to calculate the predicted TSUB value for the proposed project. To determine a project’s final cost-effectiveness rating, FTA divides the project’s annual capital and operating costs by its predicted TSUB value and compares the computed figure to established cost-effectiveness breakpoints.

FTA officials that we interviewed noted that the TSUB measure used to assess the cost-effectiveness criterion in the New Starts evaluation framework emphasizes predicted mobility improvements because most project benefits are realized only when transit users perceive that their time and cost of travel has been reduced. For example, the introduction of new transit service may reduce users’ overall travel time to a given destination. These reductions in travel time usually occur because a project offers faster travel times as a result of travel on the project’s fixed guideway, which does not incur the degree of congestion faced by buses operating in mixed travel. According to FTA, such transit user benefits are the distinct and primary benefit of transit investments. Most other benefits of transit projects, such as economic development, are considered secondary benefits because they are still directly related to mobility improvements. For example, transportation investments that improve the accessibility and attractiveness of certain locations can result in higher property values in those areas, which can affect the type and density of development that occurs in the area of the investment. The transportation literature and different experts we consulted agreed that such increases in property values are generally the result of mobility improvements. As such, they noted that conducting a separate evaluation of secondary benefits, such as economic development, may be inappropriate because it can result in double counting certain project impacts. For example, in a 2002 report, the Transportation Research Board (TRB) reported that secondary benefits like economic development “are double counts” of mobility improvements and must be carefully measured and presented “in such a way that decision makers are aware of the potential for double counting.”

FTA also considers information on environmental benefits, mobility improvements, and other factors (including economic development), but these criteria are not weighted in the current evaluation framework. As a result, they are not used to calculate the project justification rating, except under certain circumstances. For example, FTA currently evaluates information on mobility improvements, but this criterion is not used in determining the project justification rating, except in certain cases as a tiebreaker when the average of the cost-effectiveness and land use ratings falls equally between two categories.

Experts and Other Program Stakeholders Expressed Concerns That FTA’s Current Evaluation Measures Could Be Underestimating Total Project Benefits

Project sponsors and experts we interviewed raised concerns about how FTA uses and measures different New Starts project justification criteria in the evaluation framework, which could potentially result in certain project benefits being underestimated. Some project sponsors we spoke with expressed frustration that FTA does not include certain criteria in the initial calculation of project ratings, such as economic development and environmental benefits. They noted that this practice limits the information captured on projects, particularly since these are important benefits of transit projects at the local level and were required to be evaluated under SAFETEA-LU. In addition to these concerns, we have previously reported that FTA’s reliance on two evaluation criteria to calculate a project’s overall rating is not aligned with the multiple-measure evaluation and rating process outlined in statute and current New Starts regulations. As a result, we recommended that FTA improve the measures used to evaluate New Starts projects or provide a crosswalk in the regulations showing clear linkages between the criteria in the statute and the criteria used in the evaluation process. FTA’s current guidance on the New Starts evaluation process states that environmental benefits are not weighted presently because the current measure does not meaningfully distinguish among projects. Furthermore, FTA officials we interviewed told us that they had not yet developed a reliable way to incorporate economic development into the framework, had not received any reasonable suggestions for measuring this criterion, or had project sponsors submit information demonstrating the impacts of their projects on economic development. Despite these issues, however, they acknowledged that the current approach for evaluating projects does not align with SAFETEA-LU and noted that the revised evaluation process described in the NPRM and proposed policy guidance was developed to meet these requirements.

Different experts and project sponsors we interviewed also disagreed with FTA’s emphasis on mobility in the cost-effectiveness measure, noting that it does not account for other important project benefits. Specifically, experts and project sponsors, as well as members of the transit industry and DOT officials, stated that FTA’s TSUB measure does not capture the benefits that accrue to highway users as user benefits when more people switch to the improved transit service and highway congestion decreases. The omission of these nontransit user benefits means that the benefits accruing to motorists are not accounted for in the evaluation process. In cases where a project’s predicted impact on congestion is significant, this omission may lead FTA to underestimate a project’s total user benefits. Given FTA’s focus on cost-effectiveness in the evaluation process, underestimating user benefits for certain projects could impact the overall project ratings and change the relative ranking of proposed transit projects.

In response to this issue, FTA officials told us that although the TSUB measure and existing software have the capacity to capture highway user benefits, they do not currently accept estimates of nontransit user benefits because local travel models do not reliably predict changes in travel speeds resulting from transit investments. Instead, FTA currently adjusts the cost-effectiveness breakpoints upward, which has the effect of giving all projects the same credit for highway travel time savings. As a result, some projects are being credited with achieving these benefits, even when the project has no impact at all on highway travel time savings, while other projects may not be receiving enough credit for their impact on highway travel time savings. FTA officials noted that they would prefer to estimate the predicted impact of projects on highway congestion rather than using a rough proxy for these benefits, particularly since their current approach does not distinguish among projects in a meaningful way. Officials at FTA and the Office of the Secretary of Transportation also told us that they are conducting research on ways to improve the estimation of highway speeds (and thus, the calculation of nontransit user benefits) by local travel models, but a significant investment of resources by different levels of government will likely be required to do so.

A few experts we spoke with also commented that FTA’s cost- effectiveness measure does not capture any project benefits, such as economic development effects, that are unrelated to mobility improvements. As noted earlier, FTA contends that its emphasis on mobility improvements is appropriate, since most secondary project benefits—including economic development—are derived from this measure. Although our work, the transportation literature we reviewed, and experts we consulted generally support this contention, these sources also indicated that some secondary project benefits, namely certain economic development effects, may not always accrue in direct proportion to mobility improvements. Some studies we reviewed and experts we spoke with noted that property value increases near a project may occur due to option value or agglomeration effects, both of which are indirect results of transit investments and not explicitly related to mobility improvements. In such cases, FTA’s existing TSUB measure would understate the total benefits that result from providing enhanced access to a dense urban core, rather than transporting commuters from longer distances (e.g., light or heavy rail) due to its emphasis on travel time savings. Furthermore, our previous work on measuring costs and benefits of transportation investments has stated that there could be some residual benefit from these indirect effects that is not accounted for in travel time benefits or other direct impacts. This lack of accounting for certain secondary benefits in the TSUB measure may prevent FTA from capturing all project benefits and developing accurate project rankings.

In interviews with FTA officials about this issue, they acknowledged that some benefits may accrue in varying proportions to mobility improvements—that is, certain benefits may not be directly related to changes in mobility improvements. In such cases, the current evaluation process may not favor certain types of projects—such as streetcars—that are not designed to create travel time savings, but rather create other benefits. Such benefits could include changes in land use that are not captured by the TSUB measure. In the future, FTA officials told us that they would prefer to improve local models, so that they can consistently and reliably assess projects’ impact on nontransit users and economic development.

Finally, some project sponsors also expressed concern about FTA’s requirement to use fixed land use assumptions when estimating the predicted user benefits resulting from the implementation of a proposed project. According to sponsors, this practice prevents FTA from explicitly counting some future benefits that may arise due to an area’s increased accessibility. For example, some transit projects’ primary goal is to change land use around transit stations in order to capitalize on the area’s enhanced accessibility. Such changes could also lead to increases in future transit ridership, resulting in higher user benefits for the project. Furthermore, a recent panel of experts convened by FTA noted that it was unrealistic to evaluate only the incremental impacts of the proposed transit project, since local governments often find it difficult to justify high-density, mixed-use zoning in the absence of transit. Thus, by assuming that no such land use changes will occur, FTA may be underestimating projects’ predicted user benefits.

FTA officials told us they have two reasons for fixing land use assumptions when calculating user benefits. First, it is difficult to determine the magnitude of the additional land use changes, including economic development that will result from a project. Most localities do not have analytical methods for these projections, and the methods that do exist are often more unreliable than the local models used to forecast travel demand. Second, even with a reasonable estimate of additional development, it is difficult to value the benefits of the additional development. Officials from FTA told us that significant changes to local travel models would be required before they could allow project sponsors to vary their assumptions about future land use when estimating user benefits.

FTA Faces Several Systemic Challenges to Improving the New Starts Program

FTA faces several systemic challenges to improving the New Starts program, including addressing multiple program goals, limitations of local travel models, the need to maintain the rigor while minimizing the complexity of the evaluation process, and developing clear and consistent guidance for incorporating qualitative information into the evaluation process. FTA and project sponsors we spoke with have interpreted the emphasis of the New Starts program differently because the evaluation criteria, which have been delineated in previous and existing transportation legislation, establish multiple goals for the program. Additionally, models used to generate local travel demand forecasts have limited capabilities and may not provide all of the information needed to properly evaluate transit projects. FTA has taken some steps to mitigate the modeling limitations but faces challenges in doing so, including a lack of resources to invest in local travel model improvements. Finally, experts, transportation consultants, and some project sponsors we spoke with support FTA’s rigorous process for evaluating proposed transit projects but are concerned that the process has become too burdensome and complex. FTA has taken some steps to streamline its evaluation process and incorporate qualitative information into the assessment, but project sponsors we spoke to emphasized the continued need for clear, consistent guidance on how such qualitative information will be used.

FTA and Project Sponsors Have Interpreted Emphasis of New Starts Program Differently

FTA and project sponsors we spoke with have interpreted the emphasis of the New Starts program differently. Although the goals have not been explicitly articulated in legislation, the evaluation criteria outlined within the law express various goals of the New Starts program. These include mobility improvements, environmental benefits, operating efficiencies, cost-effectiveness, economic development, and land use. The presence of multiple program goals within the statute, as articulated by the evaluation criteria, has led to different interpretations by FTA and project sponsors about what project benefits should be emphasized in the New Starts evaluation process. As noted earlier, FTA focuses on mobility improvements in its evaluation process because it contends that those benefits are a critical goal of all transit projects and that most secondary project benefits, including economic development, are derived from improvements that reduce users’ travel times. Many of the experts and some of the project sponsors we spoke to agreed that transit projects can work toward a number of different goals, including mobility improvements, though some project sponsors told us that creating nontransportation benefits, such as generating local economic development, can be the primary goal of a project. In the latter case, the primary goal of a project is not to create significant mobility improvements, but rather to stimulate high-density development and change land use patterns around a transit station. Accordingly, such projects may not generate the mobility improvements needed to qualify for New Starts funding under the current New Starts evaluation process. Some project sponsors, therefore, could devote substantial resources to apply for New Starts funding for projects that are incompatible with FTA’s emphasis on mobility improvements.

Local Modeling Limitations Prevent Full Evaluation of Project Impacts

The models used to generate local travel forecasts are limited and may not provide sufficient or reliable information to properly evaluate transit projects. According to a recent report by TRB, the demands on local models have grown significantly in recent years as a result of new policy concerns, such as the need to estimate motor vehicle emissions and evaluate alternative land use policies, and existing models are inadequate to address many of these new concerns. The current models used by most MPOs are generally able to represent aggregate and corridor-level travel demand, but they are not dynamic. That is, they are based on average travel speeds over discrete areas and cannot represent the conditions that would be expected by an individual traveler choosing how, when, and where to travel. This limitation affects a model’s ability to accurately represent travel behavior, nonauto (e.g., walking or biking) or transit travel, and transit’s impacts on highway congestion, thereby limiting a model’s ability to provide all of the information needed to properly evaluate transit projects.

Some of the experts, as well as FTA and Office of the Secretary officials we interviewed, agreed that local modeling capacity is limited and should be updated to better reflect travel behavior. For example, one expert maintained that transit projects’ estimated impacts on all travel in the region can be tested with estimates that are “sensitive” enough to pick up projects’ impacts, but noted that most MPOs do not have the capacity to generate such estimates. In addition, the TRB report and some experts we spoke with have expressed concerns that many MPOs have inadequate traffic and household data to validate their models and provide information on the travel behavior of different populations. Our past work has also cited the difficulties of accurately predicting changes in traveler behavior, land use, or usage of highways resulting from a transit project with current travel models, as well as concerns about the quality of data inputs into local travel models.

FTA has taken some steps to mitigate the modeling limitations—which TRB recognized in its report on the state of the practice—but faces challenges in doing so. As previously discussed, FTA has developed proxy measures to account for certain project benefits that cannot be accurately modeled at the present time, such as projects’ impacts on highway congestion. FTA officials told us that they would prefer to improve local models so that they can consistently and reliably assess projects’ impacts on nontransit users and economic development. To that end, FTA has recently developed a request for proposals to seek approaches for predicting changes in highway user benefits that can be used in the short- term (within 5 years). However, the request for proposals has not yet been issued or awarded, and there is no timeline for doing so. Additionally, according to officials from FTA and the Office of the Secretary, FTA approached FHWA to help with this effort, but FHWA declined to be involved because it deemed the issue to be only relevant to transit. As a result, the Office of the Secretary provided the other half of the funding for the request for proposals. Officials from FTA and the Office of the Secretary stated that the improvements to travel models would affect the way all planning is done and, thus, have impacts on numerous local, state, and federal programs, including highway programs.

Officials from FTA and the Office of the Secretary emphasized that the request for proposals is just a small step forward to improve modeling. In the long-term, larger, more fundamental changes are needed to create dynamic travel models. For example, current models would need to be adjusted to capture the movement of individuals rather than parts of the transportation system, such as a highway segment. Additionally, models need to be altered so that they produce second-by-second results rather than results by groups of hours. These long-term improvements would allow for reliable and accurate estimates of highway user benefits resulting from transit-related mobility improvements and would also improve travel speed estimates at both the regional and micro levels. Like the efforts to improve approaches for predicting changes in highway user benefits, FTA and Office of the Secretary officials said that these long-term changes in modeling will benefit many transportation programs beyond the New Starts program.

However, FTA and Office of the Secretary officials told us that a significant investment of resources by all levels of government will likely be required to overcome current modeling limitations. In its 2007 report, TRB called for $20 million annually to update local travel models across the country. Currently, DOT invests about $2.4 million annually to improve modeling capabilities. Approximately $500,000 per year is allocated to DOT’s Travel Model Improvement Program, which is designed to assist MPO model development efforts, and another $1.9 million is set aside annually through SAFETEA-LU for the development of TRANSIMS. TRB also reported that MPOs face similar challenges. Specifically, MPO budgets for model development have not grown commensurately with travel modeling and forecasting requirements at the federal level, and staffing levels often limit the extent to which MPOs can focus on improvements to travel models in addition to their typical obligations.

Striking Appropriate Balance between Maintaining a Robust Evaluation Process and Minimizing the Complexity Is Challenging

Experts and some project sponsors we spoke with generally support FTA’s quantitatively rigorous process for evaluating proposed transit projects but are concerned that the process has become too burdensome and complex, and as noted earlier, may underestimate certain project benefits. For example, several experts and transportation consultants told us that although it is appropriate to measure the extent to which transit projects create primary and secondary benefits, such as mobility improvements and economic development, it is difficult to quantify all of these projected benefits. Additionally, several project sponsors noted that the complexity of the evaluation process can necessitate hiring consultants to handle the data requests and navigate the application process—which could increase the project’s costs. Our previous reviews of the New Starts program have noted similar concerns from project sponsors. For example, in 2007, we reported that a majority of project sponsors told us that the complexity of the requirements—such as the analysis and modeling required for travel forecasts—creates disincentives for entering the New Starts pipeline. Sponsors also said that the expense involved in fulfilling the application requirements, including the costs of hiring additional staff and consultants, discourages agencies with fewer resources from applying for this funding.

In response to such concerns, FTA has tried to simplify the evaluation process in several ways. For example, following SAFETEA-LU’s passage, FTA established the Very Small Starts eligibility category within the Small Starts program for projects less than $50 million in total cost. This program further simplifies the application requirements in place for the Small Start program, which funds lower-cost projects, such as bus rapid transit, streetcar, and commuter rail projects. Additionally, in its New Starts program, FTA no longer rates projects on the operating efficiencies criterion because, according to FTA, operating efficiencies are already sufficiently captured in FTA’s cost-effectiveness measures, and the measure did not adequately distinguish among projects. Thus, projects no longer have to submit information on operating efficiencies. Likewise, FTA no longer requires project sponsors to submit information on environmental benefits because it found that the information gathered did not adequately distinguish among projects and that EPA’s ambient air quality rating was sufficient. FTA also commissioned a study by Deloitte in June 2006 to review the project development process and identify opportunities for streamlining or simplifying the process. This study identified a number of ways that FTA’s project development process could be streamlined, including revising the policy review and issuance cycle to minimize major policy and guidance changes to every 2 years and conducting a human capital assessment to identify skill gaps and opportunities for reallocating resources in order to enhance FTA’s ability to review and assist New Starts projects in a timely and efficient manner. FTA is working to implement these recommendations.

Incorporating Qualitative Information into the Evaluation Process Is Challenging

Incorporating qualitative information into the New Starts evaluation process can provide a more balanced approach to evaluating transit projects, but developing clear and consistent guidance for incorporating qualitative information can be challenging. Though a quantitative evaluation process can be both rigorous and transparent, it does have limitations. Our past work and some experts and project sponsors we interviewed expressed concern about using a strictly quantitative process when evaluating proposed transportation investments because, as discussed above, certain benefits cannot be easily quantified. For example, some project sponsors and experts said that because certain impacts, such as economic development, cannot be easily quantified, a qualitative approach is needed to ensure that those project impacts are included in the New Starts evaluation process. Additionally, experts and project sponsors we spoke with raised concerns about FTA’s heavy reliance on quantitative measures in the New Starts evaluation process, noting that it can be very costly to run multiple iterations of travel models (which a quantitative-focused evaluation process requires) and that some transit agencies do not have the expertise to refine their models to FTA’s specifications.

In recognition of the limitations of a quantitative analysis, FTA has integrated some qualitative information into its current evaluation process. For example, FTA currently uses three qualitative land use measures to evaluate a transit project’s potential land use impacts. The NPRM also proposes to incorporate some qualitative information into the evaluation process, including measures of a transit project’s impact on economic development. Additionally, FTA incorporated the make-the-case document into its evaluation process in 2003, which allows project sponsors to submit an essay that justifies why the New Starts project is the best possible alternative and why it is needed. Although the fiscal year 2009 rating cycle was the first time that FTA planned to rate the make-the-case documents for the evaluation process, it ultimately decided not to because agency officials were generally dissatisfied with the quality of the make- the-case documents submitted. FTA officials attributed the overall unsatisfactory quality of the make-the-case documents to insufficient guidance about what information to include in the document and how this information would be evaluated. FTA told us that they are working to improve the guidance for the next rating cycle. According to a few project sponsors we spoke to, FTA’s recent experience with the make-the-case document illustrates the need for consistent, transparent guidance for using qualitative information in its evaluation process. To help FTA incorporate qualitative information into the evaluation and rating process in a transparent and consistent manner, a few experts we spoke with suggested that FTA convene an external panel of transportation experts to rate qualitative information, such as the make-the-case document and the economic development criterion.

Different Options for Evaluating Proposed New Starts Projects Exist, but All Have Limitations

Different options for evaluating proposed transit projects exist. However, all have limitations and are impacted to varying degrees by the systemic challenges previously identified, including local modeling limitations and the need to balance the rigor of the evaluation process with an interest in minimizing complexity. One option is to revise the current evaluation process as proposed by FTA in the August 2007 NPRM and proposed policy guidance. A second option is to use benefit-cost analysis as the evaluation framework for projects. A third option is to use evaluation frameworks that vary by project goal in order to better support local transit priorities. A fourth option is to eliminate the federal evaluation process and devolve these responsibilities to the state level by making New Starts a formula grant program.

FTA’s Proposed Revisions to Existing Evaluation Process Address Some Concerns but May Continue to Inaccurately Estimate Total Project Benefits

One option to evaluate proposed transit projects is to revise the existing New Starts evaluation process, as proposed by FTA. In response to provisions in SAFETEA-LU and to improve the New Starts program, FTA proposed to revise the current process by introducing new evaluation measures and weights, as described in its August 2007 NPRM and proposed policy guidance. The proposed process revises the current evaluation process to reflect the multiple measure approach to evaluating transit projects described in SAFETEA-LU. As in the current process, FTA’s proposed evaluation process assigns ratings to projects on the basis of various evaluation criteria to determine summary ratings for both local financial commitment and project justification (see fig. 4). In contrast to the current process, however, the proposed process places weights on measures that were previously not used to calculate initial project justification ratings, including environmental benefits, economic development, and mobility improvements.

Under the proposed evaluation process, project justification criteria are grouped into categories of “cost-effectiveness” and “effectiveness.” The cost-effectiveness category accounts for 50 percent of the overall project justification rating and is based on the current measure of cost- effectiveness with no proposed changes. The effectiveness category accounts for the other 50 percent of the project justification rating and is based on measures of (1) mobility improvements, (2) economic development and land use, and (3) environmental benefits. See table 2 for descriptions of all the proposed project justification measures.

Although experts and project sponsors had differing opinions, many experts we spoke to generally thought that the weights proposed for the project justification criteria were appropriate. In particular, many said it was appropriate that FTA retained its emphasis on mobility improvements in the proposed evaluation framework by weighting the cost-effectiveness criterion heavily. They generally agreed with FTA’s assumption that societal benefits from transit projects generally result from user benefits— that is, reductions in the real and perceived cost of travel. As such, FTA’s measure of predicted user benefits accounts for many project benefits. Under the proposed process, FTA would measure different dimensions of user benefits as part of its cost-effectiveness, mobility improvements, and economic development criteria. In addition, as called for by many project sponsors and experts we spoke to, the proposed framework places weights on measures of economic development, environmental benefits, and other factors, such as congestion impacts. Many of those experts said that the weights placed on economic development and environmental benefits are appropriate. In particular, the experts said that the relatively low weight placed on the measures of economic development is appropriate because transit-related development benefits are generally transfers of economic activity from one area to another and not net benefits to a region. They also said that many economic development benefits result from user benefits, and as such, they are captured in the cost-effectiveness criterion. As we have reported in the past, these benefits represent real benefits for the jurisdiction making the transportation improvement but are considered transfers and not real economic benefits from a regional or national perspective.

Further, although SAFETEA-LU lists economic development effects and transit supportive land use as separate project justification criteria, most of the experts we spoke to agreed with FTA that combining measures of economic development and land use into a single evaluation criterion is appropriate because the two criteria are strongly related. Although many experts generally agreed with the weights proposed, some project sponsors we spoke to disagreed with the weights placed on the evaluation criteria. In particular, they told us that transit user benefits, as measured under the cost-effectiveness and mobility improvements criteria, continue to be weighted too heavily under the proposed evaluation process. They stated that mobility improvements are emphasized at the expense of other project benefits, such as economic development. A provision in the SAFETEA-LU Technical Corrections Act of 2008 amended the language of 49 U.S.C.§ 5309 to require that FTA give comparable, but not necessarily equal, numerical weight to each project justification criteria in calculating the overall project rating. This provision could potentially address the foregoing concerns, as FTA is now required to capture project benefits in a comparable manner. However, an FTA official told us that the evaluation process proposed in their August 2007 NPRM and proposed policy guidance would have made the change now expressed in law by proposing to weight each of the different criteria included in the statute.

Furthermore, according to experts and project sponsors we spoke with, the proposed revisions to the current evaluation process preserve the rigor of FTA’s existing evaluation framework. Unlike the Federal Aid Highway Program, in which funds are automatically distributed to states via formulas, the New Starts program’s evaluation process requires local transit agencies to compete for project funds based on specific financial and project justification criteria. As noted by some experts we spoke with and in our past work, the use of such a rigorous and systematic evaluation process helps to properly distinguish among different projects and could serve as a model for other transportation programs. Further, some project sponsors also noted that use of the make-the-case document, as proposed under the “other factors” criterion, could be an effective way to incorporate additional qualitative information into the evaluation process.

Proposed Revisions May Still Inaccurately Estimate Total Project Benefits because of Modeling Limitations and Use of Proxy Measures

Although experts and project sponsors had differing opinions, many experts and project sponsors noted that the revised process may still inaccurately estimate total project benefits because of how certain benefits are measured. As a result, without improvements to the way FTA measures certain project benefits, it risks ranking proposed projects inaccurately. In particular, some experts and project sponsors we spoke with expressed continued concern about how FTA measures user benefits for the purposes of rating projects’ cost-effectiveness, noting the lack of accounting for nontransit user benefits, such as highway users, and the use of fixed land use assumptions when calculating transit user benefits. As previously discussed, FTA maintains that its measure of transit user benefits is the best that can be done given local modeling limitations and recognizes that these limitations may impact the relative ranking of proposed projects. Many project sponsors and experts we spoke to also expressed concern about how FTA measures project costs when determining the cost-effectiveness rating. As required by FTA, the cost used for this rating must include “all essential project elements necessary for completion of the project.” According to FTA, there has been much discussion in the past as to what constitutes an essential element of the project versus a project “betterment.” In its August 2007 NPRM, FTA sought industry comment on how the concept of essential project elements should be addressed in the evaluation process. Many of the stakeholders we consulted, as well as comments submitted to FTA’s docket, said that betterments should be excluded from the project cost when calculating cost-effectiveness. This could result in better cost- effectiveness scores for some proposed projects, according to FTA. Some stakeholders we spoke to also noted that defining what an essential project element is can be difficult.

Although many experts we spoke to agreed with the weight placed on cost-effectiveness in the evaluation process, some also said that FTA should not rely solely on the TSUB measure as a proxy for all other benefits, which they maintained is the practical effect of both the current and proposed evaluation processes. Some benefits, such as economic development unrelated to mobility improvements, are not captured by the TSUB measure or the proposed new measures of project benefits, according to many experts we spoke to. FTA’s continued emphasis on its measures of mobility in the revised evaluation process may lead to underestimating projects’ total benefits and, thus, inappropriately ranking proposed projects. FTA acknowledged this concern in its August 2007 Proposed Policy Guidance, noting that not all transit-related economic development is the result of improvements in mobility. FTA is currently studying the magnitude of benefits unrelated to mobility improvements that result from projects and told us that local modeling limitations have made it difficult to estimate projects’ land use impacts. In particular, FTA convened an expert panel on October 17, 2007, to discuss methods for evaluating the economic development benefits of transit projects. FTA’s intended objective is to develop, to the extent possible, a standardized, empirically based, and rational method for evaluating the potential economic development benefits of New Starts projects. (See table 3 for more information on the proposed evaluation measures.)

Some experts and project sponsors also expressed concern that the proposed evaluation process introduces evaluation measures that will not appropriately distinguish among projects. In particular, they said that FTA’s proposed measures of economic development, congestion, and environmental benefits are crude proxy measures of the real benefits and will not meaningfully distinguish among projects. FTA officials acknowledged that the proposed measures of environmental benefits are imperfect proxies but said that they are the most appropriate measures available to distinguish among projects, given the difficulties in forecasting the impact of projects on the environment. Further, they said that they decided not to propose measures of the predicted impact of projects on the environment, including air quality and greenhouse gas emissions, in order to avoid placing additional burden on project sponsors. The officials also said that they are conducting research to identify other technically appropriate measures. In particular, FTA’s August 2007 Proposed Policy Guidance states that the agency is initiating a long-term effort, in consultation with the transit community and environmental experts, to develop more robust environmental measures that will be effective at distinguishing among candidate projects. However, FTA has not established a timeline for this effort and, according to transit associations we spoke with, has not contacted them to publicize this long-term project. FTA officials also acknowledged that the proposed measure of congestion impacts, as part of the mobility improvements criterion, is an imperfect proxy, but is appropriate given difficulties in forecasting the impact of projects on nontransit users. Also, as noted earlier, FTA is collaborating with the Office of the Secretary to develop methods of measuring transit’s impact on highway users. Given local travel modeling limitations and SAFETEA-LU provisions, FTA officials told us that their proposed measures of congestion and environmental benefits are appropriate, respond to the intent of SAFETEA-LU, and minimize the burden on project sponsors. However, some experts and project sponsors told us that these proxy measures make the evaluation process more complicated without improving the relative ranking of projects. To appropriately balance the rigorous evaluation of projects with the complexity of the process, many experts and project sponsors said that FTA should include only those evaluation measures that help properly distinguish among projects.

Furthermore, some experts and project sponsors we spoke with said FTA’s proposed measures of economic development are not appropriate because they will not capture projects’ impacts on local development patterns. They noted that the measures should be of predicted impacts and not of current conditions. Because local models do not reliably predict the complex interaction between transit projects and land use, some experts and project sponsors we spoke to said that FTA should rely on both quantitative and qualitative measures to evaluate projects’ predicted economic development impacts. For example, a project sponsor told us that local economic models along with surveys of local real estate experts can be used to help assess the future impact of a transit project on a corridor’s development. FTA officials told us that the proposed measures of economic development and land use are drawn from research identifying the causal factors for economic development and therefore are the most appropriate and reliable measures available given difficulties in forecasting the impact of transit projects on economic development and land use. FTA officials also noted that they have solicited feedback about measuring these benefits in the past and have not received any practical or appropriate suggestions.

Benefit-Cost Analysis Is Another Evaluation Option, though Implementation Challenges Exist

A second option to evaluate proposed transit projects is benefit-cost analysis. Benefit-cost analysis, a process that attempts to quantify and monetize benefits and costs accruing to society from an investment, can be used to identify investment alternatives with the greatest net benefit to the locality, region, or nation. This analysis examines the immediate and long-term effects of the investment for both users and nonusers. Because benefit-cost analysis can be used to systematically assess proposed investments, it may be a useful tool for evaluating New Starts projects. Although using this approach to evaluate other federal investments is commonly advocated, FTA is currently prohibited from considering the dollar value of mobility improvements in evaluating projects, developing regulations, or carrying out any other duties. This prohibition has the practical effect of precluding FTA from conducting benefit-cost analysis of proposed transit projects.

Despite this prohibition, benefit-cost analysis could help FTA better organize and evaluate information about proposed transit projects. Some experts we spoke to said that benefit-cost analysis, in conjunction with other qualitative evaluation measures, would be an ideal framework for evaluating New Starts projects. Most experts we spoke to agreed that, conceptually, benefit-cost analysis offers a full comparison of transit projects’ benefits and costs. One expert said that it is appropriate to have an evaluation process that produces detailed estimates of all benefits and costs so that projects with the highest net benefits can be identified and funded because the New Starts’ program budget is limited. In the past, we have encouraged the use of benefit-cost analysis in other areas, such as freight transportation, and noted the usefulness of the analysis for federal transportation decision makers. Some experts also maintained that most of the information necessary for benefit-cost analysis is already produced or available to project sponsors. Most experts we spoke to who advocated using benefit-cost analysis, however, maintained that the quantitative results of the analysis should be used in concert with qualitative measures to account for those factors that cannot be monetized. We have noted in the past that guidance on benefit-cost analysis advises decision makers to augment the results of the analysis with consideration of other factors, such as the equitable distribution of benefits. Executive Order 12893 directs agencies to assess benefits and costs of proposed infrastructure investments. In addition, we and others, including the Office of Management and Budget and DOT, have also identified benefit-cost analysis as a useful tool for integrating the social, environmental, economic, and other effects of investment alternatives and for helping transportation decision makers identify projects with the greatest net benefits. In this way, benefit-cost analysis could provide FTA with a systematic and comprehensive assessment of proposed projects’ impacts.

In addition to the legal prohibition on FTA monetizing certain project benefits, there are many short-term challenges to implementing benefit- cost analysis. First, according to some experts we spoke to and our previous work, because local travel models produce outputs that become inputs for benefit-cost analysis, this approach to evaluating projects is limited by the previously mentioned limitations of local travel models. Accordingly, some experts we spoke to maintained that the results of benefit-cost analysis would not be reliable. FTA officials also told us that many project sponsors do not have the technical capacity to conduct benefit-cost analysis. A second challenge identified by many experts and project sponsors is the difficulty of monetizing certain project benefits and considering the distribution of predicted benefits. For example, determining how to quantify and monetize reductions in emissions and travel time can be challenging. Although agency guidance exists, researchers do not always agree on the appropriate methods for valuing these impacts. Additionally, while benefit-cost analysis attempts to determine the net benefits of projects, it does not usually consider the distribution of those benefits across locations or populations or other equity concerns that may exist. As two experts told us, and as we have noted in the past, these distributional issues could be addressed within benefit-cost analysis by, for example, weighting the benefits and costs to a disadvantaged group differently than those to other segments of the population. However, it can be difficult in practice to determine the appropriate weights to assign to particular groups. Some experts and project sponsors said that FTA should not adopt this approach to evaluating projects because of these particular weaknesses.

An FTA official told us that they do not support using benefit-cost analysis because of the challenges associated with monetizing benefits. FTA officials also maintained that their current evaluation process captures information similar to a formal benefit-cost analysis. They also said that their current process is appropriate because the goal of the New Starts evaluation process, given funding constraints, is to produce a relative ranking of proposed projects, not to identify all projects with positive net benefits. As we have previously stated, FTA’s emphasis on mobility improvements and reliance on certain proxy measures in the current and proposed evaluation processes may underestimate total project benefits, thereby impacting the relative ranking of projects. In contrast, benefit-cost analysis would attempt to monetize all benefits and costs, which experts told us would be a more comprehensive approach to evaluating projects. Finally, an FTA official we spoke with also noted that the statutory prohibition on monetizing mobility improvements when evaluating projects prevents FTA from using benefit-cost analysis for the New Starts program.

Evaluation Process Could Differ by Project Goal, but This Option May Not Support National Priorities

A third option to evaluate proposed transit projects is to evaluate them differently based on their primary goal. Experts and projects sponsors told us that transit projects have different and multiple goals, from improving mobility to reducing greenhouse gas emissions. (See figure 5 for examples of transit project goals.) Some experts and project sponsors said that the New Starts program could focus more on facilitating local transit goals, such as economic development, by using different evaluation processes for projects with different goals. They advocated for options that would emphasize local goals because they said the practical effect of FTA’s current evaluation process is the exclusion of certain transit projects from funding consideration. More specifically, projects with the goal of fostering high-density development through the construction of transit stations often cannot achieve a successful ranking under the New Starts process because they generally are not predicted to create significant transit user benefits. According to one expert we spoke to, this goal- focused option could either involve different evaluation criteria for different types of projects or consistent criteria but different weights for the criteria based on the goal of the project. For example, projects with the primary goal of catalyzing and managing local economic development could be evaluated mainly on the basis of predicted economic development effects and the extent of transit-supportive policies and characteristics in the project corridor.

Experts and project sponsors we spoke to said the main weakness of using different evaluation frameworks is that federal transit spending should reflect national priorities. More specifically, they said that because the New Starts program is funded by the federal government, projects should go through a national evaluation process designed to support those projects that serve particular national goals. One expert in particular said that FTA should retain its primary focus on funding projects that improve mobility and not on those designed to change the structure of cities. FTA officials also maintained that projects should not be evaluated differently because the New Starts program is a national program and, as such, should have an evaluation process that reflects national priorities and is consistently applied to all projects. Additionally, some experts we spoke to said that establishing defensible and appropriate measures for different evaluation processes could be difficult. Some experts also said that it may be hard to separate projects into different categories, given the fact that most projects have overlapping goals. Finally, some experts expressed concern that project sponsors would self-select into the evaluation process under which they score best. Such self-selection could increase the total number of projects qualifying for New Starts funding, while potentially decreasing the rigor of the selection process. FTA officials also expressed this concern because potential measures associated with certain goals, such as economic development, are relatively subjective. The officials maintained that it would be difficult to develop appropriate and defensible metrics to assess projects with goals other than mobility improvements.

Evaluation Process Could Be Devolved to the States under Formula Grant Program but Could Lack Federal Accountability

According to some experts we spoke to, a fourth option is to eliminate the evaluation process at the federal level and devolve this responsibility to the states. In particular, these experts suggested using a formula grant program to distribute New Starts funds, noting that this option would result in projects that better reflect local transit priorities. One expert we spoke to maintained that most transit projects only have local or regional benefits and no national impacts, and thus, should be controlled by states. A formula grant program in particular, according to some of those experts, could ensure that local areas build projects that meet their needs, as opposed to those that meet FTA’s expectations.

According to experts we spoke to, shifting the federal investment in fixed guideway transit from a discretionary grant program to a formula grant program would devolve the evaluation of projects to the state or local levels. Formula grant programs allocate funds to states or their subdivisions in accordance with a distribution formula prescribed in law or regulation. Grant recipients may then allocate these funds to specific projects based on program eligibility guidelines. One expert we spoke to also suggested developing a large-scale transportation formula grant program that would include money for New Starts projects. Such a program could use performance-based indicators to make state allocations.

Other experts we spoke to, however, said that establishing accountability mechanisms for project performance under a formula program could be difficult. Formula grant programs lodge decision power, and thus accountability, at the state and local levels to varying degrees and with varying constraints. The practical result of this, as we have noted in our past work, is often that program-specific performance information is collected through program operations, which limits the ability of the federal government to hold grantees accountable. Some formula grant programs’ designs inherently limit the prospect of collecting program-wide performance data through program operations. As we have also previously reported, many current surface transportation projects funded through formula grant programs are not effective at addressing key transportation challenges. They generally do not address these challenges because the federal role is unclear and programs lack links to needs or performance. Furthermore, devolving the evaluation process for proposed transit projects would also eliminate the rigorous, national, evaluation process FTA has developed—through the New Starts program—which we have previously recognized as a model for other programs. More specifically, we have noted that while the New Starts program requires project sponsors to justify their proposed transit projects on the basis of cost- effectiveness and other criteria, there are no similar federal requirements for analyses of highway project benefits because those projects are funded under a formula program.

Conclusions

FTA’s New Starts program is often cited as a model for other federal transportation programs. FTA’s recommendations for funding are based on a rigorous examination of the benefits and costs of proposed projects, and Congress has generally followed FTA’s funding recommendations. However, there is growing lack of confidence among Members of Congress and the transit industry about the process and the results it produces. For instance, FTA may be underestimating projects’ benefits because existing and proposed evaluation measures do not fully capture all potential benefits, such as benefits to highway users and environmental benefits. Capturing these other benefits potentially could change the relative rankings of proposed projects and FTA’s funding recommendations. According to FTA officials and some experts we interviewed, local models must be improved in order to develop and employ better measures of project impacts. These models produce the data necessary to measure potential benefits of transit projects, such as the projects’ impacts on highway congestion. However, due to technical limitations, current models cannot be counted on to accurately and reliably produce this information. Without improvements to these models, FTA will have to continue using proxies for certain benefits—which could lead to inaccurate assessments of projects’ benefits. Improving these models is a complex and costly endeavor—and will likely require support from all levels of the government. However, given that New Starts projects cost hundreds of millions of dollars, it seems prudent that FTA and other federal, state, and local agencies take steps to improve the models used to provide critical information to policymakers about the merits of the projects and ultimately, whether the projects should be implemented. Furthermore, the benefits of improving local travel models would extend beyond transit projects, as data from these models are used to inform regional transportation planning for other modes, as well. The upcoming reauthorization of all transportation programs, including the New Starts program, provides an opportunity to seek additional resources to improve local travel models.

FTA is working to improve the New Starts evaluation process and, in particular, address the limitations associated with its current measures. For example, FTA has issued a request for proposals to develop approaches for predicting changes in highway user benefits, which could help eliminate the need to use crude proxies in the evaluation process and, therefore, more accurately measure project benefits. However, FTA has not established a timeline for completing this effort. Furthermore, FHWA has declined to participate in this effort, even though the results could benefit all kinds of transportation planning. In addition, although FTA has committed to work with environmental experts to improve the environmental benefits measures, FTA has not begun this effort, or established time frames for initiating or completing this effort. Given that there is general consensus that FTA’s existing and proposed environmental benefits measures do not meaningfully distinguish among projects, FTA should work expeditiously to improve these measures before having project sponsors develop and submit information that is not useful for evaluation and rating purposes. In addition, FTA has worked to incorporate qualitative information about certain project benefits in the evaluation process, which can help ensure that all project benefits are fully considered. However, the inclusion of qualitative information in the evaluation process does not negate the need for FTA to work to improve existing or develop new quantitative measures for the different evaluation criteria.

There are a number of alternatives FTA can consider as it explores options for revamping the New Starts program. The NRPM presents one way to modify the existing evaluation framework, but there are also several different options that could serve as a means to determine which transit projects should receive New Starts funding. In particular, our past work and some of the experts we spoke to identified benefit-cost analysis as a viable tool that could provide a comprehensive analysis of projects’ costs and benefits over time. However, FTA’s ability to consider this approach is constrained by the current prohibition on placing dollar values on mobility improvements. Going forward, it is important that FTA have the flexibility to consider a wide range of approaches for evaluating transit projects, including benefit-cost analysis, as it seeks to improve the New Starts program.

Recommendations for Executive Action

To improve the New Starts evaluation process and the measures of project benefits, which could change the relative ranking of projects, we recommend that the Secretary of Transportation take the following five actions: (1) Seek additional resources to improve local travel models in the next authorizing legislation; (2) Seek a legislative change to allow FTA to consider the dollar value of mobility improvements in evaluating projects, developing regulations, or carrying out any other duties; (3) Direct the Administrator of FTA to establish a timeline for issuing, awarding, and implementing the result of its request for proposals on short- and long-term approaches to measuring highway user benefits from transit improvements; (4) Direct the Administrator of FTA to establish a timeline for initiating and completing its longer-term effort to develop more robust measures of transit projects’ environmental benefits that are practically useful in distinguishing among proposed projects, including consultation with the transit community, and; (5) Direct the Administrators of FTA and FHWA to collaborate in efforts to improve the consistency and reliability of local travel models, including the aforementioned request for proposals on approaches to measuring highway user benefits.

Agency Comments and Our Evaluation

We provided a draft of this report to DOT for review and comment. DOT generally agreed with the findings and recommendations in this report, and provided clarifying comments and technical corrections, which we incorporated, as appropriate.

We are sending copies of this report to DOT and appropriate congressional committees. We will also make copies available to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov. If you have any questions about this report, please contact me at siggerudk@gao.gov or (202) 512-2834. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix IV.

Appendix I: Summary of New Starts and Small Starts Projects Evaluated, Rated, and Recommended for Funding for FY 2009

Administration Requests $1.62 Billion for New Starts and Small Starts Projects for Fiscal Year 2009

The Federal Transit Administration (FTA) evaluated and rated 29 New Starts, Small Starts, and Very Small Starts projects for funding during the fiscal year 2009 evaluation cycle. FTA evaluated and rated 13 New Starts projects, 2 of which had pending full funding grant agreements (FFGA) and were recommended for funding. FTA did not recommend any new New Starts projects for funding this year. FTA also evaluated and rated 16 Small Starts and Very Small Starts projects and recommended 13 of these projects for funding. The fiscal year 2009 President’s budget requests $1.62 billion in New Starts funding, the majority of which is for 15 projects with existing FFGAs.

FTA Evaluated and Rated 13 New Starts Projects but Did Not Recommend Any New Projects for Funding

FTA identified 16 New Starts projects during the fiscal year 2009 cycle, including 2 projects with pending FFGAs and 14 projects in preliminary engineering and final design. (See table 4 for a full list of these projects.) Of the 16 total projects, 13 projects were evaluated and rated using the newly instituted five-level scale, and 3 projects were statutorily exempt from being rated.

Although they evaluated and rated fewer New Starts projects during the fiscal year 2009 cycle than in previous years, FTA officials told us that this decrease does not indicate that there are fewer projects in the pipeline. They stated that the Annual Report only provides a snapshot of the total portfolio of projects in development or under construction. As a result, projects that have existing FFGAs or those that are currently in alternatives analysis are not included in this list. Since last year’s New Starts evaluation and rating cycle, four projects in the pipeline “graduated” from final design and received FFGAs, and one sponsor withdrew two projects from the process after changing the project type in both corridors from bus rapid transit to light rail rapid transit. FTA expects that the revised projects will return to the pipeline and progress toward an FFGA in the future. FTA officials also anticipate that several other projects that are currently in alternatives analysis will move into preliminary engineering at some point in the near future, at which point they will be evaluated and rated.

FTA did not recommend any new projects for funding in the current evaluation cycle but did recommend funding for two projects with pending FFGAs: the West Corridor Light Rail Transit (LRT) in Denver and the University Link LRT Extension in Seattle. In its Annual Report, FTA states that both of these projects meet the New Starts criteria, are at an advanced stage of development with few remaining uncertainties, and are expected to be ready for an FFGA prior to or during fiscal year 2009. The total capital cost of these two projects is estimated to be $2.46 billion, with the total federal New Starts share for the West Corridor LRT at 44 percent and the University Link LRT extension at 42 percent of the total cost, respectively. FTA also recommended reserving $78 million in New Starts funding for final design activities for projects that will reach final design prior to the development of the fiscal year 2009 appropriations bill. Unlike in previous years, FTA has not specified which projects will be eligible for this funding or allocated a particular amount for any given project. According to the Annual Report and officials we spoke to at FTA, this approach will allow the agency to make “real time” funding recommendations as project uncertainties are mitigated and Congress makes final appropriations decisions. FTA does not expect that all of the projects in preliminary engineering will advance to final design in fiscal year 2009 (see table 4).

FTA Evaluated and Rated 16 Small Starts and Very Small Starts Projects and Recommended Funding for 13 Projects

FTA evaluated and rated 16 eligible Small Starts and Very Small Starts projects, including 12 projects that were advanced into project development during this cycle and 4 existing Small Starts projects that were not fully funded in fiscal year 2008. Ten projects received a “medium” rating and 6 projects received a “medium-high” rating. FTA recommended 13 of these 16 projects for funding. (See table 5 for a list of FTA’s funding recommendations for fiscal year 2009.) The total capital cost of the 13 projects that FTA recommended for funding is estimated to be $771.6 million, and the total Small Starts, including Very Small Starts, share is expected be about $451.6 million. Most of these projects are proposed to be funded under a multiyear Project Construction Grant Agreement. However, three projects, which have requested less than $25 million in total Small Starts funding, are proposed in this budget to be funded under one-year capital grants.

Administration’s Fiscal Year 2009 Budget Recommends $1.62 Billion for the New Starts Program

The administration’s fiscal year 2009 budget proposal recommends that $1.62 billion be made available for the New Starts program. This amount is $51.7 million more than the program’s fiscal year 2008 appropriation. Figure 6 illustrates the planned uses of the administration’s proposed request for the New Starts fiscal year 2009 budget, including the following: $1,146.62 million would be allocated among the 15 projects with existing $160 million would be allocated among 2 projects with pending FFGAs; $78 million would be allocated to projects that will reach final design before the end of this fiscal year; $200 million would be allocated for Small Starts projects; $20 million for ferry capital projects (Alaska and Hawaii) and Denali $16.2 million for oversight activities.

Appendix II: Scope and Methodology

To address our objectives, we reviewed previous GAO reports, FTA’s existing and proposed New Starts policy guidance, FTA’s August 2007 Notice of Proposed Rulemaking (NPRM) for New Starts, and the provisions of SAFETEA-LU that address the New Starts program to identify the information captured by the current and proposed New Starts project justification criteria. We also reviewed various pieces of legislation, including SAFETEA-LU and New Starts authorizing legislation, along with legislative history, to determine the extent to which New Starts program goals have been expressed or defined in law. Furthermore, we reviewed FTA’s Annual Report on New Starts for fiscal year 2009 to determine the number of projects evaluated, rated, and recommended for funding, the amount of funding requested for these projects, and the total costs of proposed projects.

We also examined a sample of public comments submitted in response to the proposed revisions to FTA’s current evaluation process, as described in the NPRM. First, we reviewed all 104 comments submitted to the docket to understand the range of perspectives on the proposed revisions described in the NPRM. Second, following this review, we conducted a more in-depth review of 13 comments submitted by (1) project sponsors we interviewed; (2) professional and advocacy groups we interviewed; and (3) organizations submitting extensive and relevant comments, as determined by team members. Third, upon completion of this analysis, we also reviewed 27 of the remaining 91 comments. After sorting the remaining comments, we randomly selected comments in proportion to the total number of comments received by (1) geographic diversity; (2) relevance of comment to FTA’s proposals; and (3) diversity of opinion. We categorized and analyzed comments to determine the frequency of particular perspectives and opinions about FTA’s proposed revisions, as well as other options for evaluating projects. Because the comments were selected as a nonprobability sample, the results cannot be generalized to all comments.

We interviewed FTA and transit industry officials to get an in-depth assessment of the information captured by the current and proposed New Starts project justification measures as well as how FTA’s current evaluation process influences projects’ cost, schedule, and design. We also interviewed FTA officials to discuss how the design and use of these measures impacts the calculation of project benefits, how the proposed revisions respond to SAFETEA-LU and past concerns voiced by the transit industry, and what other options they have considered to measure different project justification criteria. To learn more about the ongoing rulemaking process, we also attended New Starts Listening Sessions in Washington, D.C., and Charlotte, North Carolina, in October 2007. We also attended FTA’s expert panel discussion to identify approaches for incorporating land use and economic development into the New Starts evaluation framework. In addition, we interviewed three industry associations (that represent project sponsors) that participate closely in these programs: the American Public Transportation Association, New Starts Working Group, and Reconnecting America.

We also interviewed 11 project sponsors, including both Small Starts projects in the project development phase and New Starts projects in the preliminary engineering or final design stages for the fiscal year 2009 evaluation cycle. We conducted semistructured interviews with the project sponsors to gather additional information on FTA’s current evaluation process; how FTA’s evaluation measures influence projects’ cost, schedule, and design; and other options for evaluating proposed transit projects. We selected these projects based on the following criteria: (1) projects seeking different types of funding (e.g., New Starts or Small Starts); (2) projects involving different modes of transit (e.g., rail, light rail, or bus); (3) projects in different stages of project development (e.g., preliminary engineering or final design); (4) projects of different sizes (based on the total capital cost and ridership projections); and (5) projects from different geographic areas. Because the 11 projects were selected as a nonprobability sample, the results cannot be generalized to all projects. Table 6 lists the New Starts and Small Starts project sponsors we interviewed for our review.

To further address our objectives, we interviewed a variety of transportation experts and consultants to obtain their perspectives on FTA’s current evaluation process and other options for evaluating proposed transit projects. We used a semistructured interview guide and followed up by e-mail to collect comparable information from all experts. We selected an initial group of transportation experts to interview based on their past participation in GAO and FTA expert panels on similar topics and their research on transit issues, including the New Starts program. During these initial interviews, we solicited recommendations of other experts we should interview. Using this snowballing technique, we selected the most frequently recommended experts for interviews, as well as those with the most relevant expertise. Table 7 lists the experts we interviewed.

Following the interviews, team members categorized and analyzed the experts’ comments to determine the frequency of particular perspectives about FTA’s current evaluation process and other options for evaluating projects. To supplement the perspectives of these experts, we also interviewed other scholars and consultants with specific knowledge of the New Starts project evaluation process, including Don Emerson, Principal Consultant, Parsons Brinckerhoff Consulting; Laurie Hussey, Consultant, Cambridge Systematics, Inc.; Terry Moore, Planning Director, Land-Use and Transportation Planning, ECONorthwest; Kenneth Orski, Editor and Publisher, Innovation Briefs; Randy Pozdena, Senior Economist, Monetary Policy and Industrial Organization, ECONorthwest; Michael Replogle, Transportation Director, Environmental Defense; and Ronald Utt, Herbert and Joyce Morgan Senior Research Fellow, Heritage Foundation.

We also reviewed academic and professional literature about the impact of public transit on mobility, economic development, and the environment. The purpose of our literature review was to assess the accuracy of particular assertions made by experts, project sponsors, and government officials we interviewed. Our literature review included articles identified through searches of research databases and the Internet, as well as suggestions of experts we interviewed. Team members analyzed and summarized the evidence from these articles in consultation with a GAO methodologist and economist.

We conducted this performance audit from October 2007 to June 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix III: Explanation of FTA’s Calculation of Transportation System User Benefits

The transportation system user benefits (TSUB) measure is intended to capture all the significant user benefits of a proposed transit project. The measure includes predicted travel time savings and accounts for other benefits by quantifying the effect of nontravel time factors that influence travel behavior. The unit of the TSUB measure is equivalent to minutes of in-vehicle travel time.

Project sponsors use local travel demand models to forecast ridership and simulate trips taken in 2030, which is the forecast year used for estimating benefits over time, for two alternatives. The baseline alternative assumes low-cost improvements to the transportation network, while the second alternative (the “build alternative”) assumes the proposed New Starts transit project (e.g., fixed guideway transit infrastructure investment) is constructed.

Travel time savings from a proposed transit project can result from a shorter wait, a shorter walk, or shorter in-vehicle times. To adequately account for the time saved for each of these, the predicted travel time savings for wait and walk times are weighted by a factor of two or three, compared to in-vehicle time savings, because behavioral surveys have shown that travelers perceive these out-of-vehicle times as more onerous. The exact weighting factor is usually derived from local travel models calibrated based on local travel surveys.

Other factors beyond travel time—namely, travel time reliability and the convenience and comfort of the travel mode—are also incorporated into the measure of user benefits through what is commonly referred to as a modal constant. The modal constant varies by locality based on the results of the model’s calibration. Local models are generally calibrated by adjusting the modal constant until the model accurately predicts current travel patterns. Once a model is calibrated with a particular constant, it is used to forecast future travel times, and thus travel time savings, for the baseline and build alternatives. These travel time savings, reflecting both actual time savings and nontravel time factors, are referred to as user benefits.

The TSUB measure values user benefits differently for different individuals. More specifically, it values the benefits of predicted users of the project differently based on the travel mode they are switching from (e.g., automobile or transit). Behavioral surveys have shown that automobile users react differently to the user benefits created by a transit project. Some require very small reductions in transit travel time to change their travel mode from automobile to transit (i.e., the build alternative) because they are relatively indifferent between the existing transit option and automobile travel. These travelers receive benefits, which economists call gains in consumer surplus, because the reduction in transit travel times is greater than what is required to induce their change in travel mode. Others require the transit project’s full measure of time savings before they perceive any advantage to transit and change their mode. These travelers, even though they choose to switch modes, receive little gain in consumer surplus. In between these two kinds of travelers are those with a range of preferences. Accordingly, the “average” traveler that changes to the proposed transit project from automobile travel requires half of the time savings created by the project to change, and thus receives half of the project’s benefits as a gain in consumer surplus. For example, if a transit project is introduced that makes travel in a particular corridor 10 minutes faster than driving an automobile, the average benefit to an automobile user switching to transit will be 5 minutes because some will require time savings of less than 5 minutes to change modes and some will require more. To account for this variation, FTA divides the total predicted time savings for new transit riders by two when calculating user benefits because, on average, only half of the benefits are received by those travelers as gains in consumer surplus while the other half of the benefits are needed to induce the change in mode and do not represent a net benefit gain. Alternatively, individuals who switch transit modes— from bus in the baseline alternative to a new light rail, for example—would get the full 10 minute benefit of the switch because no benefit is needed to induce a mode shift since they are already transit users. These transit users take advantage of the full travel time savings.

Transit projects can also create benefits for those who do not choose to use them. For example, a transit project that reduces the number of automobile travelers may reduce overall highway congestion. FTA does not currently credit proposed projects with predicted benefits to highway users because (1) FTA has found that most travel models around the country do not predict plausible changes in highway speeds resulting from transit improvements and (2) the absence of a consistent method for highway speed prediction leads directly to potentially large differences in the predicted benefits of transit projects with similar impacts. To account for benefits to highway users, such as reduced congestion as the result of more transit users, FTA raises the breakpoints for the cost-effectiveness criterion by 20 percent, since they are only using the transit user benefits as the denominator of cost-effectiveness.

After accounting for factors that influence travel behavior as noted above, travel times are compared between the baseline alternative and build alternatives to produce the estimate of user benefits. That measure of user benefits, TSUB, becomes the denominator in the calculation of FTA’s cost- effectiveness criterion.

Appendix IV: GAO Contact and Staff Acknowledgments

Staff Acknowledgments

In addition to the individual named above, Nikki Clowers, Assistant Director; Vidhya Ananthakrishnan; Kyle Browning; Lauren Calhoun; Jay Cherlow; David Hooper; Delwen Jones; Sara Ann Moessbauer; Josh Ormond; and Susan Zimmerman made key contributions to this report.