Public Transportation: Improvements Are Needed to More Fully Assess Predicted Impacts of New Starts Projects
Published: 2008-07-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Through the New Starts program, the Federal Transit Administration (FTA) evaluates and recommends new fixed guideway transit projects for funding using the evaluation criteria identified in law. In August 2007, FTA issued a Notice of Proposed Rulemaking (NPRM), in part, to incorporate certain provisions within the Safe, Accountable, Flexible, Efficient, Transportation Equity Act: A Legacy for Users (SAFETEA-LU) into the evaluation process. SAFETEA-LU requires GAO to annually review FTA's New Starts process. This report discusses (1) the information captured by New Starts project justification criteria, (2) challenges FTA faces as it works to improve the New Starts program, and (3) options for evaluating New Starts projects. To address these objectives, GAO reviewed statutes, FTA guidance and regulations governing the New Starts program, and interviewed experts, project sponsors, and Department of Transportation (DOT) officials.
section_title: What GAO Found
paragraphs: FTA primarily uses cost-effectiveness and land use criteria to evaluate New Starts projects, but concerns have been raised about the extent to which the measures for these criteria capture total project benefits. FTA's current transportation system user benefits measure, which assesses a project's cost effectiveness, focuses on how proposed projects will improve mobility by reducing the real and perceived cost of travel. FTA told GAO that such mobility improvements are a critical goal of all transit projects. While the literature and most experts that GAO consulted with generally agree with this assertion, they also raised concerns that certain benefits are not captured. As a result, FTA may be underestimating transit projects' total benefits, but it is unclear the extent to which this impacts FTA's evaluation and rating process. FTA officials acknowledged many of these limitations but noted that resolving these issues would be difficult without a substantial investment of resources by all levels of government to improve and update local travel models. FTA faces several systemic challenges to improving the New Starts program, including addressing multiple program goals, limitations in local travel models, the need to maintain the rigor while minimizing the complexity of the evaluation process, and developing clear and consistent guidance for incorporating qualitative information. The evaluation criteria identified in the law reflect multiple goals for the program, which has led to varying expectations between FTA and project sponsors about what types of projects should be funded. Also, models that generate local travel demand forecasts are limited and may not provide all of the information needed to properly evaluate transit projects. FTA has taken steps to mitigate the modeling limitations, such as incorporating proxy measures to account for certain project impacts and developing a request for proposals to improve local travel models so that they can better predict changes in highway user benefits. However, according to FTA officials, the request for proposals is only a first step in improving local travel models, and additional resources are needed. Experts and project sponsors GAO interviewed discussed different options for evaluating proposed transit projects but identified significant limitations of each option. One option is to revise the current New Starts evaluation process as proposed by FTA in the August 2007 NPRM. While some experts GAO spoke to appreciated the rigor of the current evaluation process, others noted that the NPRM may still underestimate total project benefits. For example, FTA's measure of mobility improvements does not account for benefits accruing to highway users, and its measures of environmental benefits may not properly distinguish among projects. Experts also discussed other options for evaluating proposed transit projects, including benefit-cost analysis. Unlike FTA's current evaluation process, benefit-cost analysis would attempt to monetize all benefits and costs, which experts told GAO would be a more comprehensive approach to evaluating projects. FTA is currently prohibited by statute from considering the dollar value of mobility improvements in evaluating projects.

section_title: Letter
paragraphs: Since the early 1970s, a significant portion of the federal government’s  share of new capital investment in mass transportation has come through  the Federal Transit Administration’s (FTA) New Starts program. Through  this program, FTA identifies and recommends new fixed-guideway transit  projects for grants, typically through full funding grant agreements  (FFGA). Over the last decade, the New Starts program has provided state  and local agencies with over $10 billion to help design and construct  transit projects throughout the country.
The Safe, Accountable, Flexible, Efficient Transportation Equity Act: A  Legacy for Users (SAFETEA-LU) authorized the New Starts program  through fiscal year 2009. Although SAFETEA-LU maintained a number of  program requirements imposed by previous authorizing legislation, it also  made some changes to the program. For example, FTA must continue to  prioritize projects for funding by evaluating, rating, and recommending  potential projects on the basis of specific local financial commitment and  project justification criteria, including cost-effectiveness, operating  efficiencies, land use, mobility improvements, and environmental benefits.  SAFETEA-LU, however, also added economic development as a project  justification criterion.
We have previously identified FTA’s use of a rigorous and systematic  evaluation process to distinguish among proposed New Starts investments  as a model for other transportation programs. However, we and others  have also identified challenges facing the New Starts program. For  example, our past reviews found that many program stakeholders thought  that FTA’s process for evaluating New Starts projects was too complex  and costly and did not effectively use all of the criteria outlined in  SAFETEA-LU and previous legislation to account for different project  benefits, such as economic development. This latter issue is of particular  concern, given that FTA’s evaluation process is intended to provide a  meaningful and transparent approach for distinguishing between proposed  projects by assessing a range of project benefits. As a result, by not  measuring or underestimating certain benefits, the relative rankings of  proposed projects could change and subsequently impact FTA’s funding  recommendations.
In August 2007, FTA issued a Notice of Proposed Rulemaking (NPRM) to  implement SAFETEA-LU provisions into the evaluation process and make  additional changes that FTA believes will improve the New Starts  program. However, FTA’s proposed changes to the current evaluation  framework were not well received by Members of Congress and the transit  industry, and the Consolidated Appropriations Act of 2008 prohibited FTA  from spending money to issue the final rule this fiscal year. These issues  and the upcoming reauthorization of all surface transportation programs,  including the New Starts program, have led stakeholders and  policymakers to re-examine the existing evaluation process and consider  potential modifications and other options for evaluating New Starts  projects in the future.
We are required by SAFETEA-LU to report each year on FTA’s processes  and procedures for evaluating, rating, and recommending New Starts  projects for federal funding and on FTA’s implementation of these  processes and procedures. This report discusses the (1) information  captured by New Starts project justification criteria, (2) challenges FTA  faces as it works to improve the New Starts program, and (3) options for  evaluating New Starts projects. In addition, appendix I contains an  overview of FTA’s fiscal year 2009 New Starts Annual Report and budget  request. To address these objectives, we reviewed SAFETEA-LU, FTA  guidance and regulations governing the New Starts program and other  FTA documents, including the annual New Starts report; reviewed and  summarized research about the impacts of transit projects; attended New  Starts Listening Sessions in Washington, D.C. and Charlotte, N.C. to learn  more about the NPRM; interviewed experts, consultants, project sponsors,  industry associations, and Department of Transportation (DOT) officials  about the current and proposed New Starts evaluation frameworks, as  well as other options for evaluating projects; and analyzed a sample of  comments to FTA’s docket on the NPRM for New Starts and Small Starts.  Appendix II contains additional information about our scope and  methodology. We conducted this performance audit from October 2007 to  June 2008 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: FTA generally funds New Starts projects through FFGAs, which are  required by statute to establish the terms and conditions for federal  participation in a New Starts project. FFGAs may also define a project’s  scope, including the length of the system and the number of stations; its  schedule, including the date when the system is expected to open for  service; and its cost. For projects to obtain FFGAs, New Starts projects  must emerge from a regional, multimodal transportation planning process.  The first two phases of the New Starts process—systems planning and  alternatives analysis—address this requirement. The systems planning  phase identifies the transportation needs of a region, while the alternatives  analysis phase provides information on the benefits, costs, and impacts of  different options, such as rail lines or bus routes, in a specific corridor  versus a region. The alternatives analysis phase results in the selection of a  locally preferred alternative, which is the New Starts project that FTA  evaluates for funding. After a locally preferred alternative is selected, the  project sponsor submits an application to FTA for the project to enter the  preliminary engineering phase. When this phase is completed and federal  environmental requirements are satisfied, FTA may approve the project’s  advancement into final design, after which FTA may approve the project  for an FFGA and proceed to construction. FTA oversees grantees’  management of projects from the preliminary engineering phase through  the construction phase.
To help inform administration and congressional decisions about which  projects should receive federal funds, FTA currently distinguishes  between proposed projects by evaluating and assigning ratings to various  statutory evaluation criteria—including both project justification and local  financial commitment criteria—and then assigning an overall project  rating. (See fig. 1.) These evaluation criteria reflect a broad range of  benefits and effects of the proposed project, such as cost-effectiveness, as  well as the ability of the project sponsor to fund the project and finance  the continued operation of its transit system. FTA has developed specific  measures for each of the criteria outlined in the statute. On the basis of  these measures, FTA assigns the proposed project a rating for each  criterion and then assigns a summary rating for local financial  commitment and project justification. These two ratings are averaged  together, and then FTA assigns projects a “high,” “medium-high,”  “medium,” “medium-low,” or “low” overall rating, which is used to rank  projects and determine what projects are recommended for funding.  Projects are rated at several points during the New Starts process—as part  of the evaluation for entry into the preliminary engineering and the final  design phases, and yearly for inclusion in the New Starts Annual Report.  As required by SAFETEA-LU, the administration uses the FTA evaluation  and rating process, along with the phase of development of New Starts  projects, to decide which projects to recommend to Congress for  funding. Although many projects receive a summary rating that would  make them eligible for an FFGA, only a few are proposed for an FFGA in a  given fiscal year. FTA proposes FFGAs for those projects that are  projected to meet the following conditions during the fiscal year for which  funding is proposed:    All nonfederal project funding must be committed and available for the  project.
The project must be in the final design phase and have progressed far  enough for uncertainties about costs, benefits, and impacts (e.g., financial  or environmental) to be minimized.
The project must meet FTA’s tests for readiness and technical capacity,  which confirm that there are no remaining cost, project scope, or local  financial commitment issues.
SAFETEA-LU introduced a number of changes to the New Starts program,  including some that affect the evaluation and rating process. For  example, given past concerns that the evaluation process did not account  for a project’s impact on economic development and FTA’s lack of  communication to sponsors about upcoming changes, the statute added  economic development to the list of project justification criteria that FTA  must use to evaluate and rate New Starts projects, and requires FTA to  issue notice and guidance each time significant changes are made to the  program. SAFETEA-LU also established the Small Starts program, a new  capital investment grant program, simplifying the requirements imposed  for those seeking funding for lower-cost projects such as bus rapid transit,  streetcar, and commuter rail projects. This program is intended to  advance smaller-scale projects through an expedited and streamlined  evaluation and rating process. FTA also subsequently introduced a  separate eligibility category within the Small Starts program for “Very  Small Starts” projects. Small Starts projects that qualify as Very Small  Starts are simple, low-cost projects that FTA has determined qualify for a  simplified evaluation and rating process.
In addition to implementing the Small Starts program, FTA has taken other  steps to implement SAFETEA-LU changes to the New Starts evaluation  process. For example, FTA incorporated economic development into the  existing evaluation framework by considering the information provided by  project sponsors as an “other factor.” FTA also sought public comments  on different proposals for revising the evaluation process to better reflect  the statute through the Advanced Notice of Proposed Rulemaking  (ANPRM) and the final NPRM for the New Starts and Small Starts  programs. However, following concerns voiced by Members of Congress  and the transit industry about the weights placed on different project  benefits, FTA was prohibited from using funds to proceed with the  rulemaking process, with the exception of reviewing comments, under the  fiscal year Consolidated Appropriations Act of 2008. Figure 2 shows a  timeline of FTA’s efforts to date to implement SAFETEA-LU changes to  the New Starts evaluation and ratings process.
section_title: FTA’s Project Evaluation Measures Include a Range of Information, but Not All Project Benefits Are Fully Captured
paragraphs: FTA primarily uses the cost-effectiveness and land use criteria to evaluate  New Starts projects, but concerns have been raised about the extent to  which the measures for these criteria capture total project benefits.  Specifically, FTA’s TSUB measure considers how the mobility  improvements from a proposed project will reduce users’ travel times.  According to FTA officials, experts, and the literature we consulted, the  TSUB measure accounts for most secondary project benefits, including  economic development, because these benefits are typically derived from  mobility improvements that reduce users’ travel times. However, project  sponsors and experts raised concerns about how FTA currently measures  and weights different project justification criteria, noting that these  practices may underestimate some project benefits. For example, some  experts and project sponsors we spoke to said that the TSUB measure  does not account for benefits for nontransit users or capture any  economic development benefits that are not directly correlated to mobility  improvements. As a result, FTA may be underestimating projects’ total  benefits, particularly in areas looking to use these projects as a way to  relieve congestion or promote more high-density development. In these  cases, it is unclear the extent to which FTA’s current approach to  estimating benefits impacts how projects are ranked in FTA’s evaluation  and ratings process. FTA officials acknowledged these limitations, but  noted that improvements in local travel models are needed to resolve  some of these issues.
subsections: section_title: FTA Emphasizes Cost- Effectiveness and Land Use in Developing Project Justification Ratings
paragraphs: FTA currently relies on the cost-effectiveness and land use criteria to  evaluate and rate New Starts projects. Specifically, FTA assigns a weight  of 50 percent to both the cost-effectiveness and land use criteria when  developing project justification ratings. Table 1 provides a summary of all  project justification criteria that FTA is required to review, the measures it  uses to evaluate these criteria, and how this information is used to rate  projects.
To evaluate the land use criterion, FTA has developed and uses three  qualitative land use measures: land use in the project area, the extent to  which the area has transit supportive plans and policies, and the  performance and impacts of these policies. For example, to determine  whether a project’s surrounding area has transit supportive plans and  policies, FTA examines whether there are growth management strategies  and transit supportive corridor policies in place, the extent to which  zoning regulations near stations are transit supportive, and the tools  available to implement land use policies.
To evaluate cost-effectiveness, FTA relies on the TSUB measure and costs.  The TSUB measure captures predicted improvements in mobility caused  by the implementation of a project. In particular, TSUB captures transit  users’ cost and travel time savings, as well as improvements in comfort,  convenience, and reliability of travel. Project sponsors use local travel  models to forecast ridership and simulate trips taken in 2030, the forecast  year used to estimate savings over time for two alternatives. To evaluate  the benefits for these two alternatives, FTA uses the outputs from these  models to consider and weigh a range of attributes, such as time spent  waiting at and walking to the transit station, and calculates the perceived  level of time savings associated with a given project. The first alternative,  known as the baseline alternative, assumes low-cost improvements to the  project area’s current transportation network, while the second  alternative-—the “build alternative”—assumes the proposed New Starts  transit project is constructed. As outlined in figure 3, FTA uses the  forecasts for these two alternatives to calculate the predicted TSUB value  for the proposed project. To determine a project’s final cost-effectiveness  rating, FTA divides the project’s annual capital and operating costs by its  predicted TSUB value and compares the computed figure to established  cost-effectiveness breakpoints.
FTA officials that we interviewed noted that the TSUB measure used to  assess the cost-effectiveness criterion in the New Starts evaluation  framework emphasizes predicted mobility improvements because most  project benefits are realized only when transit users perceive that their  time and cost of travel has been reduced. For example, the introduction of  new transit service may reduce users’ overall travel time to a given  destination. These reductions in travel time usually occur because a  project offers faster travel times as a result of travel on the project’s fixed  guideway, which does not incur the degree of congestion faced by buses  operating in mixed travel. According to FTA, such transit user benefits are  the distinct and primary benefit of transit investments. Most other benefits  of transit projects, such as economic development, are considered  secondary benefits because they are still directly related to mobility  improvements. For example, transportation investments that improve the  accessibility and attractiveness of certain locations can result in higher  property values in those areas, which can affect the type and density of  development that occurs in the area of the investment. The transportation  literature and different experts we consulted agreed that such increases in  property values are generally the result of mobility improvements. As  such, they noted that conducting a separate evaluation of secondary  benefits, such as economic development, may be inappropriate because it  can result in double counting certain project impacts. For example, in a  2002 report, the Transportation Research Board (TRB) reported that  secondary benefits like economic development “are double counts” of  mobility improvements and must be carefully measured and presented “in  such a way that decision makers are aware of the potential for double  counting.”
FTA also considers information on environmental benefits, mobility  improvements, and other factors (including economic development), but  these criteria are not weighted in the current evaluation framework. As a  result, they are not used to calculate the project justification rating, except  under certain circumstances. For example, FTA currently evaluates  information on mobility improvements, but this criterion is not used in  determining the project justification rating, except in certain cases as a  tiebreaker when the average of the cost-effectiveness and land use ratings  falls equally between two categories.
section_title: Experts and Other Program Stakeholders Expressed Concerns That FTA’s Current Evaluation Measures Could Be Underestimating Total Project Benefits
paragraphs: Project sponsors and experts we interviewed raised concerns about how  FTA uses and measures different New Starts project justification criteria  in the evaluation framework, which could potentially result in certain  project benefits being underestimated. Some project sponsors we spoke  with expressed frustration that FTA does not include certain criteria in the  initial calculation of project ratings, such as economic development and  environmental benefits. They noted that this practice limits the  information captured on projects, particularly since these are important  benefits of transit projects at the local level and were required to be  evaluated under SAFETEA-LU. In addition to these concerns, we have  previously reported that FTA’s reliance on two evaluation criteria to  calculate a project’s overall rating is not aligned with the multiple-measure  evaluation and rating process outlined in statute and current New Starts  regulations. As a result, we recommended that FTA improve the  measures used to evaluate New Starts projects or provide a crosswalk in  the regulations showing clear linkages between the criteria in the statute  and the criteria used in the evaluation process. FTA’s current guidance on  the New Starts evaluation process states that environmental benefits are  not weighted presently because the current measure does not  meaningfully distinguish among projects. Furthermore, FTA officials we  interviewed told us that they had not yet developed a reliable way to  incorporate economic development into the framework, had not received  any reasonable suggestions for measuring this criterion, or had project  sponsors submit information demonstrating the impacts of their projects  on economic development. Despite these issues, however, they  acknowledged that the current approach for evaluating projects does not  align with SAFETEA-LU and noted that the revised evaluation process  described in the NPRM and proposed policy guidance was developed to  meet these requirements.
Different experts and project sponsors we interviewed also disagreed with  FTA’s emphasis on mobility in the cost-effectiveness measure, noting that  it does not account for other important project benefits. Specifically,  experts and project sponsors, as well as members of the transit industry  and DOT officials, stated that FTA’s TSUB measure does not capture the  benefits that accrue to highway users as user benefits when more people  switch to the improved transit service and highway congestion decreases.  The omission of these nontransit user benefits means that the benefits  accruing to motorists are not accounted for in the evaluation process. In  cases where a project’s predicted impact on congestion is significant, this  omission may lead FTA to underestimate a project’s total user benefits.  Given FTA’s focus on cost-effectiveness in the evaluation process,  underestimating user benefits for certain projects could impact the overall  project ratings and change the relative ranking of proposed transit  projects.
In response to this issue, FTA officials told us that although the TSUB  measure and existing software have the capacity to capture highway user  benefits, they do not currently accept estimates of nontransit user benefits  because local travel models do not reliably predict changes in travel  speeds resulting from transit investments. Instead, FTA currently adjusts  the cost-effectiveness breakpoints upward, which has the effect of giving  all projects the same credit for highway travel time savings. As a result,  some projects are being credited with achieving these benefits, even when  the project has no impact at all on highway travel time savings, while other  projects may not be receiving enough credit for their impact on highway  travel time savings. FTA officials noted that they would prefer to estimate  the predicted impact of projects on highway congestion rather than using  a rough proxy for these benefits, particularly since their current approach  does not distinguish among projects in a meaningful way. Officials at FTA  and the Office of the Secretary of Transportation also told us that they are  conducting research on ways to improve the estimation of highway speeds  (and thus, the calculation of nontransit user benefits) by local travel  models, but a significant investment of resources by different levels of  government will likely be required to do so.
A few experts we spoke with also commented that FTA’s cost- effectiveness measure does not capture any project benefits, such as  economic development effects, that are unrelated to mobility  improvements. As noted earlier, FTA contends that its emphasis on  mobility improvements is appropriate, since most secondary project  benefits—including economic development—are derived from this  measure. Although our work, the transportation literature we reviewed,  and experts we consulted generally support this contention, these sources  also indicated that some secondary project benefits, namely certain  economic development effects, may not always accrue in direct  proportion to mobility improvements. Some studies we reviewed and  experts we spoke with noted that property value increases near a project  may occur due to option value or agglomeration effects, both of which are  indirect results of transit investments and not explicitly related to mobility  improvements. In such cases, FTA’s existing TSUB measure would  understate the total benefits that result from providing enhanced access to  a dense urban core, rather than transporting commuters from longer  distances (e.g., light or heavy rail) due to its emphasis on travel time  savings. Furthermore, our previous work on measuring costs and benefits  of transportation investments has stated that there could be some residual  benefit from these indirect effects that is not accounted for in travel time  benefits or other direct impacts. This lack of accounting for certain  secondary benefits in the TSUB measure may prevent FTA from capturing  all project benefits and developing accurate project rankings.
In interviews with FTA officials about this issue, they acknowledged that  some benefits may accrue in varying proportions to mobility  improvements—that is, certain benefits may not be directly related to  changes in mobility improvements. In such cases, the current evaluation  process may not favor certain types of projects—such as streetcars—that  are not designed to create travel time savings, but rather create other  benefits. Such benefits could include changes in land use that are not  captured by the TSUB measure. In the future, FTA officials told us that  they would prefer to improve local models, so that they can consistently  and reliably assess projects’ impact on nontransit users and economic  development.
Finally, some project sponsors also expressed concern about FTA’s  requirement to use fixed land use assumptions when estimating the  predicted user benefits resulting from the implementation of a proposed  project. According to sponsors, this practice prevents FTA from explicitly  counting some future benefits that may arise due to an area’s increased  accessibility. For example, some transit projects’ primary goal is to change  land use around transit stations in order to capitalize on the area’s  enhanced accessibility. Such changes could also lead to increases in future  transit ridership, resulting in higher user benefits for the project.  Furthermore, a recent panel of experts convened by FTA noted that it was  unrealistic to evaluate only the incremental impacts of the proposed  transit project, since local governments often find it difficult to justify  high-density, mixed-use zoning in the absence of transit. Thus, by  assuming that no such land use changes will occur, FTA may be  underestimating projects’ predicted user benefits.
FTA officials told us they have two reasons for fixing land use  assumptions when calculating user benefits. First, it is difficult to  determine the magnitude of the additional land use changes, including  economic development that will result from a project. Most localities do  not have analytical methods for these projections, and the methods that do  exist are often more unreliable than the local models used to forecast  travel demand. Second, even with a reasonable estimate of additional  development, it is difficult to value the benefits of the additional  development. Officials from FTA told us that significant changes to local  travel models would be required before they could allow project sponsors  to vary their assumptions about future land use when estimating user  benefits.
section_title: FTA Faces Several Systemic Challenges to Improving the New Starts Program
paragraphs: FTA faces several systemic challenges to improving the New Starts  program, including addressing multiple program goals, limitations of local  travel models, the need to maintain the rigor while minimizing the  complexity of the evaluation process, and developing clear and consistent  guidance for incorporating qualitative information into the evaluation  process. FTA and project sponsors we spoke with have interpreted the  emphasis of the New Starts program differently because the evaluation  criteria, which have been delineated in previous and existing  transportation legislation, establish multiple goals for the program.  Additionally, models used to generate local travel demand forecasts have  limited capabilities and may not provide all of the information needed to  properly evaluate transit projects. FTA has taken some steps to mitigate  the modeling limitations but faces challenges in doing so, including a lack  of resources to invest in local travel model improvements. Finally, experts,  transportation consultants, and some project sponsors we spoke with  support FTA’s rigorous process for evaluating proposed transit projects  but are concerned that the process has become too burdensome and  complex. FTA has taken some steps to streamline its evaluation process  and incorporate qualitative information into the assessment, but project  sponsors we spoke to emphasized the continued need for clear, consistent  guidance on how such qualitative information will be used.
subsections: section_title: FTA and Project Sponsors Have Interpreted Emphasis of New Starts Program Differently
paragraphs: FTA and project sponsors we spoke with have interpreted the emphasis of  the New Starts program differently. Although the goals have not been  explicitly articulated in legislation, the evaluation criteria outlined within  the law express various goals of the New Starts program. These include  mobility improvements, environmental benefits, operating efficiencies,  cost-effectiveness, economic development, and land use. The presence of  multiple program goals within the statute, as articulated by the evaluation  criteria, has led to different interpretations by FTA and project sponsors  about what project benefits should be emphasized in the New Starts  evaluation process. As noted earlier, FTA focuses on mobility  improvements in its evaluation process because it contends that those  benefits are a critical goal of all transit projects and that most secondary  project benefits, including economic development, are derived from  improvements that reduce users’ travel times. Many of the experts and  some of the project sponsors we spoke to agreed that transit projects can  work toward a number of different goals, including mobility  improvements, though some project sponsors told us that creating  nontransportation benefits, such as generating local economic  development, can be the primary goal of a project. In the latter case, the  primary goal of a project is not to create significant mobility  improvements, but rather to stimulate high-density development and  change land use patterns around a transit station. Accordingly, such  projects may not generate the mobility improvements needed to qualify for  New Starts funding under the current New Starts evaluation process.  Some project sponsors, therefore, could devote substantial resources to  apply for New Starts funding for projects that are incompatible with FTA’s  emphasis on mobility improvements.
section_title: Local Modeling Limitations Prevent Full Evaluation of Project Impacts
paragraphs: The models used to generate local travel forecasts are limited and may not  provide sufficient or reliable information to properly evaluate transit  projects. According to a recent report by TRB, the demands on local  models have grown significantly in recent years as a result of new policy  concerns, such as the need to estimate motor vehicle emissions and  evaluate alternative land use policies, and existing models are inadequate  to address many of these new concerns. The current models used by  most MPOs are generally able to represent aggregate and corridor-level  travel demand, but they are not dynamic. That is, they are based on  average travel speeds over discrete areas and cannot represent the  conditions that would be expected by an individual traveler choosing how,  when, and where to travel. This limitation affects a model’s ability to  accurately represent travel behavior, nonauto (e.g., walking or biking) or  transit travel, and transit’s impacts on highway congestion, thereby  limiting a model’s ability to provide all of the information needed to  properly evaluate transit projects.
Some of the experts, as well as FTA and Office of the Secretary officials  we interviewed, agreed that local modeling capacity is limited and should  be updated to better reflect travel behavior. For example, one expert  maintained that transit projects’ estimated impacts on all travel in the  region can be tested with estimates that are “sensitive” enough to pick up  projects’ impacts, but noted that most MPOs do not have the capacity to  generate such estimates. In addition, the TRB report and some experts we  spoke with have expressed concerns that many MPOs have inadequate  traffic and household data to validate their models and provide  information on the travel behavior of different populations. Our past work  has also cited the difficulties of accurately predicting changes in traveler  behavior, land use, or usage of highways resulting from a transit project  with current travel models, as well as concerns about the quality of data  inputs into local travel models.
FTA has taken some steps to mitigate the modeling limitations—which  TRB recognized in its report on the state of the practice—but faces  challenges in doing so. As previously discussed, FTA has developed proxy  measures to account for certain project benefits that cannot be accurately  modeled at the present time, such as projects’ impacts on highway  congestion. FTA officials told us that they would prefer to improve local  models so that they can consistently and reliably assess projects’ impacts  on nontransit users and economic development. To that end, FTA has  recently developed a request for proposals to seek approaches for  predicting changes in highway user benefits that can be used in the short- term (within 5 years). However, the request for proposals has not yet been  issued or awarded, and there is no timeline for doing so. Additionally,  according to officials from FTA and the Office of the Secretary, FTA  approached FHWA to help with this effort, but FHWA declined to be  involved because it deemed the issue to be only relevant to transit. As a  result, the Office of the Secretary provided the other half of the funding for  the request for proposals. Officials from FTA and the Office of the  Secretary stated that the improvements to travel models would affect the  way all planning is done and, thus, have impacts on numerous local, state,  and federal programs, including highway programs.
Officials from FTA and the Office of the Secretary emphasized that the  request for proposals is just a small step forward to improve modeling. In  the long-term, larger, more fundamental changes are needed to create  dynamic travel models. For example, current models would need to be  adjusted to capture the movement of individuals rather than parts of the  transportation system, such as a highway segment. Additionally, models  need to be altered so that they produce second-by-second results rather  than results by groups of hours. These long-term improvements would  allow for reliable and accurate estimates of highway user benefits  resulting from transit-related mobility improvements and would also  improve travel speed estimates at both the regional and micro levels. Like  the efforts to improve approaches for predicting changes in highway user  benefits, FTA and Office of the Secretary officials said that these long-term  changes in modeling will benefit many transportation programs beyond  the New Starts program.
However, FTA and Office of the Secretary officials told us that a  significant investment of resources by all levels of government will likely  be required to overcome current modeling limitations. In its 2007 report,  TRB called for $20 million annually to update local travel models across  the country. Currently, DOT invests about $2.4 million annually to improve  modeling capabilities. Approximately $500,000 per year is allocated to  DOT’s Travel Model Improvement Program, which is designed to assist  MPO model development efforts, and another $1.9 million is set aside  annually through SAFETEA-LU for the development of TRANSIMS. TRB  also reported that MPOs face similar challenges. Specifically, MPO  budgets for model development have not grown commensurately with  travel modeling and forecasting requirements at the federal level, and  staffing levels often limit the extent to which MPOs can focus on  improvements to travel models in addition to their typical obligations.
section_title: Striking Appropriate Balance between Maintaining a Robust Evaluation Process and Minimizing the Complexity Is Challenging
paragraphs: Experts and some project sponsors we spoke with generally support  FTA’s quantitatively rigorous process for evaluating proposed transit  projects but are concerned that the process has become too burdensome  and complex, and as noted earlier, may underestimate certain project  benefits. For example, several experts and transportation consultants told  us that although it is appropriate to measure the extent to which transit  projects create primary and secondary benefits, such as mobility  improvements and economic development, it is difficult to quantify all of  these projected benefits. Additionally, several project sponsors noted that  the complexity of the evaluation process can necessitate hiring  consultants to handle the data requests and navigate the application  process—which could increase the project’s costs. Our previous reviews  of the New Starts program have noted similar concerns from project  sponsors. For example, in 2007, we reported that a majority of project  sponsors told us that the complexity of the requirements—such as the  analysis and modeling required for travel forecasts—creates disincentives  for entering the New Starts pipeline. Sponsors also said that the expense  involved in fulfilling the application requirements, including the costs of  hiring additional staff and consultants, discourages agencies with fewer  resources from applying for this funding.
In response to such concerns, FTA has tried to simplify the evaluation  process in several ways. For example, following SAFETEA-LU’s passage,  FTA established the Very Small Starts eligibility category within the Small  Starts program for projects less than $50 million in total cost. This  program further simplifies the application requirements in place for the  Small Start program, which funds lower-cost projects, such as bus rapid  transit, streetcar, and commuter rail projects. Additionally, in its New  Starts program, FTA no longer rates projects on the operating efficiencies  criterion because, according to FTA, operating efficiencies are already  sufficiently captured in FTA’s cost-effectiveness measures, and the  measure did not adequately distinguish among projects. Thus, projects no  longer have to submit information on operating efficiencies. Likewise, FTA  no longer requires project sponsors to submit information on  environmental benefits because it found that the information gathered did  not adequately distinguish among projects and that EPA’s ambient air  quality rating was sufficient. FTA also commissioned a study by Deloitte in  June 2006 to review the project development process and identify  opportunities for streamlining or simplifying the process. This study  identified a number of ways that FTA’s project development process could  be streamlined, including revising the policy review and issuance cycle to  minimize major policy and guidance changes to every 2 years and  conducting a human capital assessment to identify skill gaps and  opportunities for reallocating resources in order to enhance FTA’s ability  to review and assist New Starts projects in a timely and efficient manner.  FTA is working to implement these recommendations.
section_title: Incorporating Qualitative Information into the Evaluation Process Is Challenging
paragraphs: Incorporating qualitative information into the New Starts evaluation  process can provide a more balanced approach to evaluating transit  projects, but developing clear and consistent guidance for incorporating  qualitative information can be challenging. Though a quantitative  evaluation process can be both rigorous and transparent, it does have  limitations. Our past work and some experts and project sponsors we  interviewed expressed concern about using a strictly quantitative process  when evaluating proposed transportation investments because, as  discussed above, certain benefits cannot be easily quantified. For example,  some project sponsors and experts said that because certain impacts, such  as economic development, cannot be easily quantified, a qualitative  approach is needed to ensure that those project impacts are included in  the New Starts evaluation process. Additionally, experts and project  sponsors we spoke with raised concerns about FTA’s heavy reliance on  quantitative measures in the New Starts evaluation process, noting that it  can be very costly to run multiple iterations of travel models (which a  quantitative-focused evaluation process requires) and that some transit  agencies do not have the expertise to refine their models to FTA’s  specifications.
In recognition of the limitations of a quantitative analysis, FTA has  integrated some qualitative information into its current evaluation process.  For example, FTA currently uses three qualitative land use measures to  evaluate a transit project’s potential land use impacts. The NPRM also  proposes to incorporate some qualitative information into the evaluation  process, including measures of a transit project’s impact on economic  development. Additionally, FTA incorporated the make-the-case document  into its evaluation process in 2003, which allows project sponsors to  submit an essay that justifies why the New Starts project is the best  possible alternative and why it is needed. Although the fiscal year 2009  rating cycle was the first time that FTA planned to rate the make-the-case  documents for the evaluation process, it ultimately decided not to because  agency officials were generally dissatisfied with the quality of the make- the-case documents submitted. FTA officials attributed the overall  unsatisfactory quality of the make-the-case documents to insufficient  guidance about what information to include in the document and how this  information would be evaluated. FTA told us that they are working to  improve the guidance for the next rating cycle. According to a few project  sponsors we spoke to, FTA’s recent experience with the make-the-case  document illustrates the need for consistent, transparent guidance for  using qualitative information in its evaluation process. To help FTA  incorporate qualitative information into the evaluation and rating process  in a transparent and consistent manner, a few experts we spoke with  suggested that FTA convene an external panel of transportation experts to  rate qualitative information, such as the make-the-case document and the  economic development criterion.
section_title: Different Options for Evaluating Proposed New Starts Projects Exist, but All Have Limitations
paragraphs: Different options for evaluating proposed transit projects exist. However,  all have limitations and are impacted to varying degrees by the systemic  challenges previously identified, including local modeling limitations and  the need to balance the rigor of the evaluation process with an interest in  minimizing complexity. One option is to revise the current evaluation  process as proposed by FTA in the August 2007 NPRM and proposed  policy guidance. A second option is to use benefit-cost analysis as the  evaluation framework for projects. A third option is to use evaluation  frameworks that vary by project goal in order to better support local  transit priorities. A fourth option is to eliminate the federal evaluation  process and devolve these responsibilities to the state level by making  New Starts a formula grant program.
subsections: section_title: FTA’s Proposed Revisions to Existing Evaluation Process Address Some Concerns but May Continue to Inaccurately Estimate Total Project Benefits
paragraphs: One option to evaluate proposed transit projects is to revise the existing  New Starts evaluation process, as proposed by FTA. In response to  provisions in SAFETEA-LU and to improve the New Starts program, FTA  proposed to revise the current process by introducing new evaluation  measures and weights, as described in its August 2007 NPRM and  proposed policy guidance. The proposed process revises the current  evaluation process to reflect the multiple measure approach to evaluating  transit projects described in SAFETEA-LU. As in the current process,  FTA’s proposed evaluation process assigns ratings to projects on the basis  of various evaluation criteria to determine summary ratings for both local  financial commitment and project justification (see fig. 4). In contrast to  the current process, however, the proposed process places weights on  measures that were previously not used to calculate initial project  justification ratings, including environmental benefits, economic  development, and mobility improvements.
Under the proposed evaluation process, project justification criteria are  grouped into categories of “cost-effectiveness” and “effectiveness.” The  cost-effectiveness category accounts for 50 percent of the overall project  justification rating and is based on the current measure of cost-  effectiveness with no proposed changes. The effectiveness category  accounts for the other 50 percent of the project justification rating and is  based on measures of (1) mobility improvements, (2) economic  development and land use, and (3) environmental benefits. See table 2 for  descriptions of all the proposed project justification measures.
Although experts and project sponsors had differing opinions, many  experts we spoke to generally thought that the weights proposed for the  project justification criteria were appropriate. In particular, many said it  was appropriate that FTA retained its emphasis on mobility improvements  in the proposed evaluation framework by weighting the cost-effectiveness  criterion heavily. They generally agreed with FTA’s assumption that  societal benefits from transit projects generally result from user benefits— that is, reductions in the real and perceived cost of travel. As such, FTA’s  measure of predicted user benefits accounts for many project benefits.  Under the proposed process, FTA would measure different dimensions of  user benefits as part of its cost-effectiveness, mobility improvements, and  economic development criteria. In addition, as called for by many project  sponsors and experts we spoke to, the proposed framework places  weights on measures of economic development, environmental benefits,  and other factors, such as congestion impacts. Many of those experts said  that the weights placed on economic development and environmental  benefits are appropriate. In particular, the experts said that the relatively  low weight placed on the measures of economic development is  appropriate because transit-related development benefits are generally  transfers of economic activity from one area to another and not net  benefits to a region. They also said that many economic development  benefits result from user benefits, and as such, they are captured in the  cost-effectiveness criterion. As we have reported in the past, these benefits  represent real benefits for the jurisdiction making the transportation  improvement but are considered transfers and not real economic benefits  from a regional or national perspective.
Further, although SAFETEA-LU lists economic development effects and  transit supportive land use as separate project justification criteria, most  of the experts we spoke to agreed with FTA that combining measures of  economic development and land use into a single evaluation criterion is  appropriate because the two criteria are strongly related. Although many  experts generally agreed with the weights proposed, some project  sponsors we spoke to disagreed with the weights placed on the evaluation  criteria. In particular, they told us that transit user benefits, as measured  under the cost-effectiveness and mobility improvements criteria, continue  to be weighted too heavily under the proposed evaluation process. They  stated that mobility improvements are emphasized at the expense of other  project benefits, such as economic development. A provision in the  SAFETEA-LU Technical Corrections Act of 2008 amended the language  of 49 U.S.C.§ 5309 to require that FTA give comparable, but not necessarily  equal, numerical weight to each project justification criteria in calculating  the overall project rating. This provision could potentially address the  foregoing concerns, as FTA is now required to capture project benefits in a  comparable manner. However, an FTA official told us that the evaluation  process proposed in their August 2007 NPRM and proposed policy  guidance would have made the change now expressed in law by proposing  to weight each of the different criteria included in the statute.
Furthermore, according to experts and project sponsors we spoke with,  the proposed revisions to the current evaluation process preserve the rigor  of FTA’s existing evaluation framework. Unlike the Federal Aid Highway  Program, in which funds are automatically distributed to states via  formulas, the New Starts program’s evaluation process requires local  transit agencies to compete for project funds based on specific financial  and project justification criteria. As noted by some experts we spoke with  and in our past work, the use of such a rigorous and systematic evaluation  process helps to properly distinguish among different projects and could  serve as a model for other transportation programs. Further, some  project sponsors also noted that use of the make-the-case document, as  proposed under the “other factors” criterion, could be an effective way to  incorporate additional qualitative information into the evaluation process.
subsections: section_title: Proposed Revisions May Still Inaccurately Estimate Total Project Benefits because of Modeling Limitations and Use of Proxy Measures
paragraphs: Although experts and project sponsors had differing opinions, many  experts and project sponsors noted that the revised process may still  inaccurately estimate total project benefits because of how certain  benefits are measured. As a result, without improvements to the way FTA  measures certain project benefits, it risks ranking proposed projects  inaccurately. In particular, some experts and project sponsors we spoke  with expressed continued concern about how FTA measures user benefits  for the purposes of rating projects’ cost-effectiveness, noting the lack of  accounting for nontransit user benefits, such as highway users, and the  use of fixed land use assumptions when calculating transit user benefits.  As previously discussed, FTA maintains that its measure of transit user  benefits is the best that can be done given local modeling limitations and  recognizes that these limitations may impact the relative ranking of  proposed projects. Many project sponsors and experts we spoke to also  expressed concern about how FTA measures project costs when  determining the cost-effectiveness rating. As required by FTA, the cost  used for this rating must include “all essential project elements necessary  for completion of the project.” According to FTA, there has been much  discussion in the past as to what constitutes an essential element of the  project versus a project “betterment.” In its August 2007 NPRM, FTA  sought industry comment on how the concept of essential project  elements should be addressed in the evaluation process. Many of the  stakeholders we consulted, as well as comments submitted to FTA’s  docket, said that betterments should be excluded from the project cost  when calculating cost-effectiveness. This could result in better cost-  effectiveness scores for some proposed projects, according to FTA. Some  stakeholders we spoke to also noted that defining what an essential  project element is can be difficult.
Although many experts we spoke to agreed with the weight placed on  cost-effectiveness in the evaluation process, some also said that FTA  should not rely solely on the TSUB measure as a proxy for all other  benefits, which they maintained is the practical effect of both the current  and proposed evaluation processes. Some benefits, such as economic  development unrelated to mobility improvements, are not captured by the  TSUB measure or the proposed new measures of project benefits,  according to many experts we spoke to. FTA’s continued emphasis on its  measures of mobility in the revised evaluation process may lead to  underestimating projects’ total benefits and, thus, inappropriately ranking  proposed projects. FTA acknowledged this concern in its August 2007  Proposed Policy Guidance, noting that not all transit-related economic  development is the result of improvements in mobility. FTA is currently  studying the magnitude of benefits unrelated to mobility improvements  that result from projects and told us that local modeling limitations have  made it difficult to estimate projects’ land use impacts. In particular, FTA  convened an expert panel on October 17, 2007, to discuss methods for  evaluating the economic development benefits of transit projects. FTA’s  intended objective is to develop, to the extent possible, a standardized,  empirically based, and rational method for evaluating the potential  economic development benefits of New Starts projects. (See table 3 for  more information on the proposed evaluation measures.)
Some experts and project sponsors also expressed concern that the  proposed evaluation process introduces evaluation measures that will not  appropriately distinguish among projects. In particular, they said that  FTA’s proposed measures of economic development, congestion, and  environmental benefits are crude proxy measures of the real benefits and  will not meaningfully distinguish among projects. FTA officials  acknowledged that the proposed measures of environmental benefits are  imperfect proxies but said that they are the most appropriate measures  available to distinguish among projects, given the difficulties in forecasting  the impact of projects on the environment. Further, they said that they  decided not to propose measures of the predicted impact of projects on  the environment, including air quality and greenhouse gas emissions, in  order to avoid placing additional burden on project sponsors. The officials  also said that they are conducting research to identify other technically  appropriate measures. In particular, FTA’s August 2007 Proposed Policy  Guidance states that the agency is initiating a long-term effort, in  consultation with the transit community and environmental experts, to  develop more robust environmental measures that will be effective at  distinguishing among candidate projects. However, FTA has not  established a timeline for this effort and, according to transit associations  we spoke with, has not contacted them to publicize this long-term project.  FTA officials also acknowledged that the proposed measure of congestion  impacts, as part of the mobility improvements criterion, is an imperfect  proxy, but is appropriate given difficulties in forecasting the impact of  projects on nontransit users. Also, as noted earlier, FTA is collaborating  with the Office of the Secretary to develop methods of measuring transit’s  impact on highway users. Given local travel modeling limitations and  SAFETEA-LU provisions, FTA officials told us that their proposed  measures of congestion and environmental benefits are appropriate,  respond to the intent of SAFETEA-LU, and minimize the burden on project  sponsors. However, some experts and project sponsors told us that these  proxy measures make the evaluation process more complicated without  improving the relative ranking of projects. To appropriately balance the  rigorous evaluation of projects with the complexity of the process, many  experts and project sponsors said that FTA should include only those  evaluation measures that help properly distinguish among projects.
Furthermore, some experts and project sponsors we spoke with said  FTA’s proposed measures of economic development are not appropriate  because they will not capture projects’ impacts on local development  patterns. They noted that the measures should be of predicted impacts and  not of current conditions. Because local models do not reliably predict the  complex interaction between transit projects and land use, some experts  and project sponsors we spoke to said that FTA should rely on both  quantitative and qualitative measures to evaluate projects’ predicted  economic development impacts. For example, a project sponsor told us  that local economic models along with surveys of local real estate experts  can be used to help assess the future impact of a transit project on a  corridor’s development. FTA officials told us that the proposed measures  of economic development and land use are drawn from research  identifying the causal factors for economic development and therefore are  the most appropriate and reliable measures available given difficulties in  forecasting the impact of transit projects on economic development and  land use. FTA officials also noted that they have solicited feedback about  measuring these benefits in the past and have not received any practical or  appropriate suggestions.
section_title: Benefit-Cost Analysis Is Another Evaluation Option, though Implementation Challenges Exist
paragraphs: A second option to evaluate proposed transit projects is benefit-cost  analysis. Benefit-cost analysis, a process that attempts to quantify and  monetize benefits and costs accruing to society from an investment, can  be used to identify investment alternatives with the greatest net benefit to  the locality, region, or nation. This analysis examines the immediate and  long-term effects of the investment for both users and nonusers. Because  benefit-cost analysis can be used to systematically assess proposed  investments, it may be a useful tool for evaluating New Starts projects.  Although using this approach to evaluate other federal investments is  commonly advocated, FTA is currently prohibited from considering the  dollar value of mobility improvements in evaluating projects, developing  regulations, or carrying out any other duties. This prohibition has the  practical effect of precluding FTA from conducting benefit-cost analysis of  proposed transit projects.
Despite this prohibition, benefit-cost analysis could help FTA better  organize and evaluate information about proposed transit projects. Some  experts we spoke to said that benefit-cost analysis, in conjunction with  other qualitative evaluation measures, would be an ideal framework for  evaluating New Starts projects. Most experts we spoke to agreed that,  conceptually, benefit-cost analysis offers a full comparison of transit  projects’ benefits and costs. One expert said that it is appropriate to have  an evaluation process that produces detailed estimates of all benefits and  costs so that projects with the highest net benefits can be identified and  funded because the New Starts’ program budget is limited. In the past, we  have encouraged the use of benefit-cost analysis in other areas, such as  freight transportation, and noted the usefulness of the analysis for federal  transportation decision makers. Some experts also maintained that most  of the information necessary for benefit-cost analysis is already produced  or available to project sponsors. Most experts we spoke to who advocated  using benefit-cost analysis, however, maintained that the quantitative  results of the analysis should be used in concert with qualitative measures  to account for those factors that cannot be monetized. We have noted in  the past that guidance on benefit-cost analysis advises decision makers to  augment the results of the analysis with consideration of other factors,  such as the equitable distribution of benefits. Executive Order 12893  directs agencies to assess benefits and costs of proposed infrastructure  investments. In addition, we and others, including the Office of  Management and Budget and DOT, have also identified benefit-cost  analysis as a useful tool for integrating the social, environmental,  economic, and other effects of investment alternatives and for helping  transportation decision makers identify projects with the greatest net  benefits. In this way, benefit-cost analysis could provide FTA with a  systematic and comprehensive assessment of proposed projects’ impacts.
In addition to the legal prohibition on FTA monetizing certain project  benefits, there are many short-term challenges to implementing benefit- cost analysis. First, according to some experts we spoke to and our  previous work, because local travel models produce outputs that become  inputs for benefit-cost analysis, this approach to evaluating projects is  limited by the previously mentioned limitations of local travel models.  Accordingly, some experts we spoke to maintained that the results of  benefit-cost analysis would not be reliable. FTA officials also told us that  many project sponsors do not have the technical capacity to conduct  benefit-cost analysis. A second challenge identified by many experts and  project sponsors is the difficulty of monetizing certain project benefits and  considering the distribution of predicted benefits. For example,  determining how to quantify and monetize reductions in emissions and  travel time can be challenging. Although agency guidance exists,  researchers do not always agree on the appropriate methods for valuing  these impacts. Additionally, while benefit-cost analysis attempts to  determine the net benefits of projects, it does not usually consider the  distribution of those benefits across locations or populations or other  equity concerns that may exist. As two experts told us, and as we have  noted in the past, these distributional issues could be addressed within  benefit-cost analysis by, for example, weighting the benefits and costs to a  disadvantaged group differently than those to other segments of the  population. However, it can be difficult in practice to determine the  appropriate weights to assign to particular groups. Some experts and  project sponsors said that FTA should not adopt this approach to  evaluating projects because of these particular weaknesses.
An FTA official told us that they do not support using benefit-cost analysis  because of the challenges associated with monetizing benefits. FTA  officials also maintained that their current evaluation process captures  information similar to a formal benefit-cost analysis. They also said that  their current process is appropriate because the goal of the New Starts  evaluation process, given funding constraints, is to produce a relative  ranking of proposed projects, not to identify all projects with positive net  benefits. As we have previously stated, FTA’s emphasis on mobility  improvements and reliance on certain proxy measures in the current and  proposed evaluation processes may underestimate total project benefits,  thereby impacting the relative ranking of projects. In contrast, benefit-cost  analysis would attempt to monetize all benefits and costs, which experts  told us would be a more comprehensive approach to evaluating projects.  Finally, an FTA official we spoke with also noted that the statutory  prohibition on monetizing mobility improvements when evaluating  projects prevents FTA from using benefit-cost analysis for the New Starts  program.
section_title: Evaluation Process Could Differ by Project Goal, but This Option May Not Support National Priorities
paragraphs: A third option to evaluate proposed transit projects is to evaluate them  differently based on their primary goal. Experts and projects sponsors told  us that transit projects have different and multiple goals, from improving  mobility to reducing greenhouse gas emissions. (See figure 5 for examples  of transit project goals.) Some experts and project sponsors said that the  New Starts program could focus more on facilitating local transit goals,  such as economic development, by using different evaluation processes  for projects with different goals. They advocated for options that would  emphasize local goals because they said the practical effect of FTA’s  current evaluation process is the exclusion of certain transit projects from  funding consideration. More specifically, projects with the goal of  fostering high-density development through the construction of transit  stations often cannot achieve a successful ranking under the New Starts  process because they generally are not predicted to create significant  transit user benefits. According to one expert we spoke to, this goal- focused option could either involve different evaluation criteria for  different types of projects or consistent criteria but different weights for  the criteria based on the goal of the project. For example, projects with  the primary goal of catalyzing and managing local economic development  could be evaluated mainly on the basis of predicted economic  development effects and the extent of transit-supportive policies and  characteristics in the project corridor.
Experts and project sponsors we spoke to said the main weakness of  using different evaluation frameworks is that federal transit spending  should reflect national priorities. More specifically, they said that because  the New Starts program is funded by the federal government, projects  should go through a national evaluation process designed to support those  projects that serve particular national goals. One expert in particular said  that FTA should retain its primary focus on funding projects that improve  mobility and not on those designed to change the structure of cities. FTA  officials also maintained that projects should not be evaluated differently  because the New Starts program is a national program and, as such,  should have an evaluation process that reflects national priorities and is  consistently applied to all projects. Additionally, some experts we spoke  to said that establishing defensible and appropriate measures for different  evaluation processes could be difficult. Some experts also said that it may  be hard to separate projects into different categories, given the fact that  most projects have overlapping goals. Finally, some experts expressed  concern that project sponsors would self-select into the evaluation  process under which they score best. Such self-selection could increase  the total number of projects qualifying for New Starts funding, while  potentially decreasing the rigor of the selection process. FTA officials also  expressed this concern because potential measures associated with  certain goals, such as economic development, are relatively subjective.  The officials maintained that it would be difficult to develop appropriate  and defensible metrics to assess projects with goals other than mobility  improvements.
section_title: Evaluation Process Could Be Devolved to the States under Formula Grant Program but Could Lack Federal Accountability
paragraphs: According to some experts we spoke to, a fourth option is to eliminate the  evaluation process at the federal level and devolve this responsibility to  the states. In particular, these experts suggested using a formula grant  program to distribute New Starts funds, noting that this option would  result in projects that better reflect local transit priorities. One expert we  spoke to maintained that most transit projects only have local or regional  benefits and no national impacts, and thus, should be controlled by states.  A formula grant program in particular, according to some of those experts,  could ensure that local areas build projects that meet their needs, as  opposed to those that meet FTA’s expectations.
According to experts we spoke to, shifting the federal investment in fixed  guideway transit from a discretionary grant program to a formula grant  program would devolve the evaluation of projects to the state or local  levels. Formula grant programs allocate funds to states or their  subdivisions in accordance with a distribution formula prescribed in law  or regulation. Grant recipients may then allocate these funds to specific  projects based on program eligibility guidelines. One expert we spoke to  also suggested developing a large-scale transportation formula grant  program that would include money for New Starts projects. Such a  program could use performance-based indicators to make state  allocations.
Other experts we spoke to, however, said that establishing accountability  mechanisms for project performance under a formula program could be  difficult. Formula grant programs lodge decision power, and thus  accountability, at the state and local levels to varying degrees and with  varying constraints. The practical result of this, as we have noted in our  past work, is often that program-specific performance information is  collected through program operations, which limits the ability of the  federal government to hold grantees accountable. Some formula grant  programs’ designs inherently limit the prospect of collecting program-wide  performance data through program operations. As we have also previously  reported, many current surface transportation projects funded through  formula grant programs are not effective at addressing key transportation  challenges. They generally do not address these challenges because the  federal role is unclear and programs lack links to needs or performance.  Furthermore, devolving the evaluation process for proposed transit  projects would also eliminate the rigorous, national, evaluation process  FTA has developed—through the New Starts program—which we have  previously recognized as a model for other programs. More specifically,  we have noted that while the New Starts program requires project  sponsors to justify their proposed transit projects on the basis of cost- effectiveness and other criteria, there are no similar federal requirements  for analyses of highway project benefits because those projects are funded  under a formula program.
section_title: Conclusions
paragraphs: FTA’s New Starts program is often cited as a model for other federal  transportation programs. FTA’s recommendations for funding are based  on a rigorous examination of the benefits and costs of proposed projects,  and Congress has generally followed FTA’s funding recommendations.  However, there is growing lack of confidence among Members of  Congress and the transit industry about the process and the results it  produces. For instance, FTA may be underestimating projects’ benefits  because existing and proposed evaluation measures do not fully capture  all potential benefits, such as benefits to highway users and environmental  benefits. Capturing these other benefits potentially could change the  relative rankings of proposed projects and FTA’s funding  recommendations. According to FTA officials and some experts we  interviewed, local models must be improved in order to develop and  employ better measures of project impacts. These models produce the  data necessary to measure potential benefits of transit projects, such as  the projects’ impacts on highway congestion. However, due to technical  limitations, current models cannot be counted on to accurately and  reliably produce this information. Without improvements to these models,  FTA will have to continue using proxies for certain benefits—which could  lead to inaccurate assessments of projects’ benefits. Improving these  models is a complex and costly endeavor—and will likely require support  from all levels of the government. However, given that New Starts projects  cost hundreds of millions of dollars, it seems prudent that FTA and other  federal, state, and local agencies take steps to improve the models used to  provide critical information to policymakers about the merits of the  projects and ultimately, whether the projects should be implemented.  Furthermore, the benefits of improving local travel models would extend  beyond transit projects, as data from these models are used to inform  regional transportation planning for other modes, as well. The upcoming  reauthorization of all transportation programs, including the New Starts  program, provides an opportunity to seek additional resources to improve  local travel models.
FTA is working to improve the New Starts evaluation process and, in  particular, address the limitations associated with its current measures.  For example, FTA has issued a request for proposals to develop  approaches for predicting changes in highway user benefits, which could  help eliminate the need to use crude proxies in the evaluation process and,  therefore, more accurately measure project benefits. However, FTA has  not established a timeline for completing this effort. Furthermore, FHWA  has declined to participate in this effort, even though the results could  benefit all kinds of transportation planning. In addition, although FTA has  committed to work with environmental experts to improve the  environmental benefits measures, FTA has not begun this effort, or  established time frames for initiating or completing this effort. Given that  there is general consensus that FTA’s existing and proposed  environmental benefits measures do not meaningfully distinguish among  projects, FTA should work expeditiously to improve these measures  before having project sponsors develop and submit information that is not  useful for evaluation and rating purposes. In addition, FTA has worked to  incorporate qualitative information about certain project benefits in the  evaluation process, which can help ensure that all project benefits are  fully considered. However, the inclusion of qualitative information in the  evaluation process does not negate the need for FTA to work to improve  existing or develop new quantitative measures for the different evaluation  criteria.
There are a number of alternatives FTA can consider as it explores options  for revamping the New Starts program. The NRPM presents one way to  modify the existing evaluation framework, but there are also several  different options that could serve as a means to determine which transit  projects should receive New Starts funding. In particular, our past work  and some of the experts we spoke to identified benefit-cost analysis as a  viable tool that could provide a comprehensive analysis of projects’ costs  and benefits over time. However, FTA’s ability to consider this approach is  constrained by the current prohibition on placing dollar values on mobility  improvements. Going forward, it is important that FTA have the flexibility  to consider a wide range of approaches for evaluating transit projects,  including benefit-cost analysis, as it seeks to improve the New Starts  program.
section_title: Recommendations for Executive Action
paragraphs: To improve the New Starts evaluation process and the measures of project  benefits, which could change the relative ranking of projects, we  recommend that the Secretary of Transportation take the following five  actions:  (1) Seek additional resources to improve local travel models in the next  authorizing legislation;  (2) Seek a legislative change to allow FTA to consider the dollar value of  mobility improvements in evaluating projects, developing regulations, or  carrying out any other duties;  (3) Direct the Administrator of FTA to establish a timeline for issuing,  awarding, and implementing the result of its request for proposals on  short- and long-term approaches to measuring highway user benefits from  transit improvements;  (4) Direct the Administrator of FTA to establish a timeline for initiating  and completing its longer-term effort to develop more robust measures of  transit projects’ environmental benefits that are practically useful in  distinguishing among proposed projects, including consultation with the  transit community, and;  (5) Direct the Administrators of FTA and FHWA to collaborate in efforts to  improve the consistency and reliability of local travel models, including  the aforementioned request for proposals on approaches to measuring  highway user benefits.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOT for review and comment. DOT  generally agreed with the findings and recommendations in this report,  and provided clarifying comments and technical corrections, which we  incorporated, as appropriate.
We are sending copies of this report to DOT and appropriate congressional  committees. We will also make copies available to others upon request. In  addition, the report will be available at no charge on the GAO Web site at  http://www.gao.gov. If you have any questions about this report, please  contact me at siggerudk@gao.gov or (202) 512-2834. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Summary of New Starts and Small Starts Projects Evaluated, Rated, and Recommended for Funding for FY 2009
subsections: section_title: Administration Requests $1.62 Billion for New Starts and Small Starts Projects for Fiscal Year 2009
paragraphs: The Federal Transit Administration (FTA) evaluated and rated 29 New  Starts, Small Starts, and Very Small Starts projects for funding during the  fiscal year 2009 evaluation cycle. FTA evaluated and rated 13 New Starts  projects, 2 of which had pending full funding grant agreements (FFGA)  and were recommended for funding. FTA did not recommend any new  New Starts projects for funding this year. FTA also evaluated and rated 16  Small Starts and Very Small Starts projects and recommended 13 of these  projects for funding. The fiscal year 2009 President’s budget requests $1.62  billion in New Starts funding, the majority of which is for 15 projects with  existing FFGAs.
subsections: section_title: FTA Evaluated and Rated 13 New Starts Projects but Did Not Recommend Any New Projects for Funding
paragraphs: FTA identified 16 New Starts projects during the fiscal year 2009 cycle,  including 2 projects with pending FFGAs and 14 projects in preliminary  engineering and final design. (See table 4 for a full list of these projects.)  Of the 16 total projects, 13 projects were evaluated and rated using the  newly instituted five-level scale, and 3 projects were statutorily exempt  from being rated.
Although they evaluated and rated fewer New Starts projects during the  fiscal year 2009 cycle than in previous years, FTA officials told us that this  decrease does not indicate that there are fewer projects in the pipeline.  They stated that the Annual Report only provides a snapshot of the total  portfolio of projects in development or under construction. As a result,  projects that have existing FFGAs or those that are currently in  alternatives analysis are not included in this list. Since last year’s New  Starts evaluation and rating cycle, four projects in the pipeline “graduated”  from final design and received FFGAs, and one sponsor withdrew two  projects from the process after changing the project type in both corridors  from bus rapid transit to light rail rapid transit. FTA expects that the  revised projects will return to the pipeline and progress toward an FFGA  in the future. FTA officials also anticipate that several other projects that  are currently in alternatives analysis will move into preliminary  engineering at some point in the near future, at which point they will be  evaluated and rated.
FTA did not recommend any new projects for funding in the current  evaluation cycle but did recommend funding for two projects with pending  FFGAs: the West Corridor Light Rail Transit (LRT) in Denver and the  University Link LRT Extension in Seattle. In its Annual Report, FTA states  that both of these projects meet the New Starts criteria, are at an advanced  stage of development with few remaining uncertainties, and are expected  to be ready for an FFGA prior to or during fiscal year 2009. The total  capital cost of these two projects is estimated to be $2.46 billion, with the  total federal New Starts share for the West Corridor LRT at 44 percent and  the University Link LRT extension at 42 percent of the total cost,  respectively. FTA also recommended reserving $78 million in New Starts  funding for final design activities for projects that will reach final design  prior to the development of the fiscal year 2009 appropriations bill. Unlike  in previous years, FTA has not specified which projects will be eligible for  this funding or allocated a particular amount for any given project.  According to the Annual Report and officials we spoke to at FTA, this  approach will allow the agency to make “real time” funding  recommendations as project uncertainties are mitigated and Congress  makes final appropriations decisions. FTA does not expect that all of the  projects in preliminary engineering will advance to final design in fiscal  year 2009 (see table 4).
section_title: FTA Evaluated and Rated 16 Small Starts and Very Small Starts Projects and Recommended Funding for 13 Projects
paragraphs: FTA evaluated and rated 16 eligible Small Starts and Very Small Starts  projects, including 12 projects that were advanced into project  development during this cycle and 4 existing Small Starts projects that  were not fully funded in fiscal year 2008. Ten projects received a  “medium” rating and 6 projects received a “medium-high” rating. FTA  recommended 13 of these 16 projects for funding. (See table 5 for a list of  FTA’s funding recommendations for fiscal year 2009.) The total capital  cost of the 13 projects that FTA recommended for funding is estimated to  be $771.6 million, and the total Small Starts, including Very Small Starts,  share is expected be about $451.6 million. Most of these projects are  proposed to be funded under a multiyear Project Construction Grant  Agreement. However, three projects, which have requested less than $25  million in total Small Starts funding, are proposed in this budget to be  funded under one-year capital grants.
section_title: Administration’s Fiscal Year 2009 Budget Recommends $1.62 Billion for the New Starts Program
paragraphs: The administration’s fiscal year 2009 budget proposal recommends that  $1.62 billion be made available for the New Starts program. This amount is  $51.7 million more than the program’s fiscal year 2008 appropriation.  Figure 6 illustrates the planned uses of the administration’s proposed  request for the New Starts fiscal year 2009 budget, including the following:    $1,146.62 million would be allocated among the 15 projects with existing    $160 million would be allocated among 2 projects with pending FFGAs;    $78 million would be allocated to projects that will reach final design  before the end of this fiscal year;    $200 million would be allocated for Small Starts projects;    $20 million for ferry capital projects (Alaska and Hawaii) and Denali    $16.2 million for oversight activities.
section_title: Appendix II: Scope and Methodology
paragraphs: To address our objectives, we reviewed previous GAO reports, FTA’s  existing and proposed New Starts policy guidance, FTA’s August 2007  Notice of Proposed Rulemaking (NPRM) for New Starts, and the  provisions of SAFETEA-LU that address the New Starts program to  identify the information captured by the current and proposed New Starts  project justification criteria. We also reviewed various pieces of  legislation, including SAFETEA-LU and New Starts authorizing legislation,  along with legislative history, to determine the extent to which New Starts  program goals have been expressed or defined in law. Furthermore, we  reviewed FTA’s Annual Report on New Starts for fiscal year 2009 to  determine the number of projects evaluated, rated, and recommended for  funding, the amount of funding requested for these projects, and the total  costs of proposed projects.
We also examined a sample of public comments submitted in response to  the proposed revisions to FTA’s current evaluation process, as described  in the NPRM. First, we reviewed all 104 comments submitted to the  docket to understand the range of perspectives on the proposed revisions  described in the NPRM. Second, following this review, we conducted a  more in-depth review of 13 comments submitted by (1) project sponsors  we interviewed; (2) professional and advocacy groups we interviewed; and  (3) organizations submitting extensive and relevant comments, as  determined by team members. Third, upon completion of this analysis, we  also reviewed 27 of the remaining 91 comments. After sorting the  remaining comments, we randomly selected comments in proportion to  the total number of comments received by (1) geographic diversity; (2)  relevance of comment to FTA’s proposals; and (3) diversity of opinion. We  categorized and analyzed comments to determine the frequency of  particular perspectives and opinions about FTA’s proposed revisions, as  well as other options for evaluating projects. Because the comments were  selected as a nonprobability sample, the results cannot be generalized to  all comments.
We interviewed FTA and transit industry officials to get an in-depth  assessment of the information captured by the current and proposed New  Starts project justification measures as well as how FTA’s current  evaluation process influences projects’ cost, schedule, and design. We also  interviewed FTA officials to discuss how the design and use of these  measures impacts the calculation of project benefits, how the proposed  revisions respond to SAFETEA-LU and past concerns voiced by the transit  industry, and what other options they have considered to measure  different project justification criteria. To learn more about the ongoing  rulemaking process, we also attended New Starts Listening Sessions in  Washington, D.C., and Charlotte, North Carolina, in October 2007. We also  attended FTA’s expert panel discussion to identify approaches for  incorporating land use and economic development into the New Starts  evaluation framework. In addition, we interviewed three industry  associations (that represent project sponsors) that participate closely in  these programs: the American Public Transportation Association, New  Starts Working Group, and Reconnecting America.
We also interviewed 11 project sponsors, including both Small Starts  projects in the project development phase and New Starts projects in the  preliminary engineering or final design stages for the fiscal year 2009  evaluation cycle. We conducted semistructured interviews with the project  sponsors to gather additional information on FTA’s current evaluation  process; how FTA’s evaluation measures influence projects’ cost,  schedule, and design; and other options for evaluating proposed transit  projects. We selected these projects based on the following criteria: (1)  projects seeking different types of funding (e.g., New Starts or Small  Starts); (2) projects involving different modes of transit (e.g., rail, light rail,  or bus); (3) projects in different stages of project development (e.g.,  preliminary engineering or final design); (4) projects of different sizes  (based on the total capital cost and ridership projections); and (5) projects  from different geographic areas. Because the 11 projects were selected as  a nonprobability sample, the results cannot be generalized to all projects.  Table 6 lists the New Starts and Small Starts project sponsors we  interviewed for our review.
To further address our objectives, we interviewed a variety of  transportation experts and consultants to obtain their perspectives on  FTA’s current evaluation process and other options for evaluating  proposed transit projects. We used a semistructured interview guide and  followed up by e-mail to collect comparable information from all experts.  We selected an initial group of transportation experts to interview based  on their past participation in GAO and FTA expert panels on similar topics  and their research on transit issues, including the New Starts program.  During these initial interviews, we solicited recommendations of other  experts we should interview. Using this snowballing technique, we  selected the most frequently recommended experts for interviews, as well  as those with the most relevant expertise. Table 7 lists the experts we  interviewed.
Following the interviews, team members categorized and analyzed the  experts’ comments to determine the frequency of particular perspectives  about FTA’s current evaluation process and other options for evaluating  projects. To supplement the perspectives of these experts, we also  interviewed other scholars and consultants with specific knowledge of the  New Starts project evaluation process, including Don Emerson, Principal  Consultant, Parsons Brinckerhoff Consulting; Laurie Hussey, Consultant,  Cambridge Systematics, Inc.; Terry Moore, Planning Director, Land-Use  and Transportation Planning, ECONorthwest; Kenneth Orski, Editor and  Publisher, Innovation Briefs; Randy Pozdena, Senior Economist, Monetary  Policy and Industrial Organization, ECONorthwest; Michael Replogle,  Transportation Director, Environmental Defense; and Ronald Utt, Herbert  and Joyce Morgan Senior Research Fellow, Heritage Foundation.
We also reviewed academic and professional literature about the impact of  public transit on mobility, economic development, and the environment.  The purpose of our literature review was to assess the accuracy of  particular assertions made by experts, project sponsors, and government  officials we interviewed. Our literature review included articles identified  through searches of research databases and the Internet, as well as  suggestions of experts we interviewed. Team members analyzed and  summarized the evidence from these articles in consultation with a GAO  methodologist and economist.
We conducted this performance audit from October 2007 to June 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix III: Explanation of FTA’s Calculation of Transportation System User Benefits
paragraphs: The transportation system user benefits (TSUB) measure is intended to  capture all the significant user benefits of a proposed transit project. The  measure includes predicted travel time savings and accounts for other  benefits by quantifying the effect of nontravel time factors that influence  travel behavior. The unit of the TSUB measure is equivalent to minutes of  in-vehicle travel time.
Project sponsors use local travel demand models to forecast ridership and  simulate trips taken in 2030, which is the forecast year used for estimating  benefits over time, for two alternatives. The baseline alternative assumes  low-cost improvements to the transportation network, while the second  alternative (the “build alternative”) assumes the proposed New Starts  transit project (e.g., fixed guideway transit infrastructure investment) is  constructed.
Travel time savings from a proposed transit project can result from a  shorter wait, a shorter walk, or shorter in-vehicle times. To adequately  account for the time saved for each of these, the predicted travel time  savings for wait and walk times are weighted by a factor of two or three,  compared to in-vehicle time savings, because behavioral surveys have  shown that travelers perceive these out-of-vehicle times as more onerous.  The exact weighting factor is usually derived from local travel models  calibrated based on local travel surveys.
Other factors beyond travel time—namely, travel time reliability and the  convenience and comfort of the travel mode—are also incorporated into  the measure of user benefits through what is commonly referred to as a  modal constant. The modal constant varies by locality based on the results  of the model’s calibration. Local models are generally calibrated by  adjusting the modal constant until the model accurately predicts current  travel patterns. Once a model is calibrated with a particular constant, it is  used to forecast future travel times, and thus travel time savings, for the  baseline and build alternatives. These travel time savings, reflecting both  actual time savings and nontravel time factors, are referred to as user  benefits.
The TSUB measure values user benefits differently for different  individuals. More specifically, it values the benefits of predicted users of  the project differently based on the travel mode they are switching from  (e.g., automobile or transit). Behavioral surveys have shown that  automobile users react differently to the user benefits created by a transit  project. Some require very small reductions in transit travel time to change  their travel mode from automobile to transit (i.e., the build alternative)  because they are relatively indifferent between the existing transit option  and automobile travel. These travelers receive benefits, which economists  call gains in consumer surplus, because the reduction in transit travel  times is greater than what is required to induce their change in travel  mode. Others require the transit project’s full measure of time savings  before they perceive any advantage to transit and change their mode.  These travelers, even though they choose to switch modes, receive little  gain in consumer surplus. In between these two kinds of travelers are  those with a range of preferences. Accordingly, the “average” traveler that  changes to the proposed transit project from automobile travel requires  half of the time savings created by the project to change, and thus receives  half of the project’s benefits as a gain in consumer surplus. For example, if  a transit project is introduced that makes travel in a particular corridor 10  minutes faster than driving an automobile, the average benefit to an  automobile user switching to transit will be 5 minutes because some will  require time savings of less than 5 minutes to change modes and some will  require more. To account for this variation, FTA divides the total predicted  time savings for new transit riders by two when calculating user benefits  because, on average, only half of the benefits are received by those  travelers as gains in consumer surplus while the other half of the benefits  are needed to induce the change in mode and do not represent a net  benefit gain. Alternatively, individuals who switch transit modes— from  bus in the baseline alternative to a new light rail, for example—would get  the full 10 minute benefit of the switch because no benefit is needed to  induce a mode shift since they are already transit users. These transit  users take advantage of the full travel time savings.
Transit projects can also create benefits for those who do not choose to  use them. For example, a transit project that reduces the number of  automobile travelers may reduce overall highway congestion. FTA does  not currently credit proposed projects with predicted benefits to highway  users because (1) FTA has found that most travel models around the  country do not predict plausible changes in highway speeds resulting from  transit improvements and (2) the absence of a consistent method for  highway speed prediction leads directly to potentially large differences in  the predicted benefits of transit projects with similar impacts. To account  for benefits to highway users, such as reduced congestion as the result of  more transit users, FTA raises the breakpoints for the cost-effectiveness  criterion by 20 percent, since they are only using the transit user benefits  as the denominator of cost-effectiveness.
After accounting for factors that influence travel behavior as noted above,  travel times are compared between the baseline alternative and build  alternatives to produce the estimate of user benefits. That measure of user  benefits, TSUB, becomes the denominator in the calculation of FTA’s cost- effectiveness criterion.
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
paragraphs: In addition to the individual named above, Nikki Clowers, Assistant  Director; Vidhya Ananthakrishnan; Kyle Browning; Lauren Calhoun; Jay  Cherlow; David Hooper; Delwen Jones; Sara Ann Moessbauer; Josh  Ormond; and Susan Zimmerman made key contributions to this report.