Intercity Passenger Rail: Recording Clearer Reasons for Awards Decisions Would Improve Otherwise Good Grantmaking Practices
Published: 2011-03-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The American Recovery and Reinvestment Act of 2009 (Recovery Act) appropriated $8 billion for high and conventional speed passenger rail. The Federal Railroad Administration (FRA), within the Department of Transportation (the department), was responsible for soliciting applications, evaluating them to determine program eligibility and technical merits, and selecting awards, which were announced in January 2010. This report examines the extent to which FRA (1) applied its established criteria to select projects, (2) followed recommended practices for awarding discretionary grants, and (3) communicated outcomes to the public, compared with selected other Recovery Act competitive grant programs. To address these topics GAO reviewed federal legislation, FRA documents, and guidance for other competitive grant programs using Recovery Act funds. GAO also analyzed data resulting from the evaluation and selection process and interviewed a cross-section of FRA officials and applicants.
section_title: What GAO Found
paragraphs: FRA applied its established criteria during the eligibility and technical reviews, but GAO could not verify whether it applied its final selection criteria because the documented rationales for selecting projects were typically vague. Specifically, FRA used worksheets and guidebooks that included the criteria outlined in the funding announcement to aid in assessing the eligibility and technical merit of applications. FRA also recorded general reasons for selecting applications and publicly posted broad descriptions of the selected projects. However, the documented reasons for these selection decisions were typically vague or restated the criteria listed in the funding announcement. In addition, there were only general reasons given for the applications not selected or for adjusting applicants' requested funding amounts. FRA subsequently provided GAO with more detailed reasons for several of its selection decisions, but this information was not included in the department's record of its decisions. Documentation on the rationales for selection decisions is a key part of ensuring accountability and is recommended by the department as well as other federal agencies. Without a detailed record of selection decisions, FRA leaves itself vulnerable to criticism over the integrity of those decisions--an important consideration, given that passenger rail investments have a very public profile. FRA also substantially followed recommended practices when awarding grants, including communicating key information to applicants prior to the competition, planning for the competition, using a merit review panel with certain characteristics, assessing whether applicants were likely to be able to account for grant funds, notifying applicants of awards decisions, and documenting the rationale for awards decisions (albeit generally). For example, FRA issued a funding announcement that communicated key pieces of information, such as eligibility, technical review, and selection criteria. FRA officials also conducted extensive outreach to potential applicants, including participating in biweekly conference calls, providing several public presentations on the program, and conducting one-on-one site visits with potential applicants. According to FRA, officials used lessons from a number of other grant programs when developing its approach to reviewing and selecting projects. FRA publicly communicated outcome information similar to other Recovery Act competitive grant programs we examined, including projects selected, how much money they were to receive, and a general description of projects and their intended benefits. Only one of the programs GAO examined communicated more outcome information on technical scores and comments; however, this program used a much different approach to select awards than FRA used to select intercity passenger rail awards. According to officials, FRA did not disclose outcome information from the technical reviews because officials were concerned that releasing reviewers' names and associated scores could discourage them from participating in future grant application reviews.
section_title: What GAO Recommends
paragraphs: GAO recommends that FRA create additional records to document the substantive reasons behind award decisions to better ensure accountability for its use of federal funds. In commenting on a draft of this report, the department agreed to consider our recommendation. The department also provided technical comments, which were incorporated as appropriate.

section_title: Letter
paragraphs: A recent influx of federal funds has breathed new life into the prospect of  developing an expanded national passenger rail network in the United  States. Specifically, the American Recovery and Reinvestment Act of 2009  (Recovery Act) appropriated $8 billion—significantly more than Congress  provided in recent years—to develop high speed and intercity passenger  rail service. Interest in these funds was high, and in January 2010 the  Federal Railroad Administration (FRA)—an agency within the Department  of Transportation (the department)—selected 62 applications in 23 states  and the District of Columbia to receive the money. The vast majority  (almost 90 percent) of the $8 billion awarded went to develop new or  substantially improved passenger rail corridor projects, which, in several  cases, expect to deliver high speed rail service reaching speeds of more  than 150 miles per hour. The remaining funding generally went to projects  focusing on upgrades and improvements to existing rail service (typically  up to 79 miles per hour).
With the Recovery Act funding, FRA recognized that it needed to  transform itself from essentially a rail safety organization to one that can  make and oversee multibillion dollar investment choices. This report  assesses how FRA made the first of those choices and ensured that  national investment goals are being met. It focuses on the extent to which  FRA (1) applied its established criteria to select projects; (2) followed  recommended practices for awarding discretionary grants; and (3)  communicated outcomes to the public, compared with selected other  Recovery Act competitive grant programs. These topics are the main focus  of the report. In addition, we are also reporting on the extent to which  selected projects align with legislative and federal goals. (See app. I.)
Our overall approach to addressing these topics was to (1) review publicly  available information, such as federal legislation, plans, and other  guidance, about the high speed intercity passenger rail program’s  evaluation, selection and communication approach, and compare it to  practices used by other competitive grant programs; (2) review documents  that FRA used in reviewing applications and selecting awardees to  determine the extent to which FRA applied its established criteria; (3)  analyze FRA data on technical review scores to determine the statistical  relationship between some of FRA’s published criteria and the selection  decisions; and (4) interview a cross-section of officials from 12 of the 40  states and the District of Columbia which submitted either a  preapplication or an application for Recovery Act funding (selected to  reflect a range of application outcomes, award amounts, number of  applications, and geographic location), a random sample of 18 of the 44  department reviewers which included at least one person from each  applicant review panel, and other FRA officials who oversaw the  evaluation and selected awards. We focused our review on projects  selected by FRA in January 2010 and funded through the Recovery Act,  which included applications submitted for ready-to-go projects (called  “track 1a”), the completion of environmental and preliminary engineering  requirements necessary to prepare projects for future funding (called  “track 1b”), and projects to develop new high speed rail or intercity  passenger services or substantially upgrade existing corridor service  (called “track 2”). We assessed the reliability of FRA’s scoring data by  conducting a series of data tests, reviewing documents and reports about  FRA’s data systems, and speaking with officials familiar with the data. We  determined that these data are sufficiently reliable for our reporting  purposes. (Additional information on our scope and methodology is  contained in app. II.)
We conducted this performance audit from April 2010 to March 2011 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
subsections: section_title: Background
paragraphs: FRA is the primary federal agency responsible for overseeing safety in the  railroad industry, as well as for distributing federal funds for intercity  passenger rail service. FRA also administers federal operating and capital  grants to the National Railroad Passenger Corporation (known as  Amtrak), which have averaged between $1 billion and $1.3 billion per year  since fiscal year 2003. FRA also approves Railroad Rehabilitation and  Improvement Financing loans and Rail Line Relocation and Improvement  Capital grants, and is the granting agency for the $120 million in fiscal year  2008 and fiscal year 2009 capital funds for intercity passenger rail projects.
Recent legislation has vastly increased the federal role in and federal funds  for developing intercity passenger rail service. The Passenger Rail  Investment and Improvement Act of 2008 (PRIIA), enacted in October  2008, authorized more than $3.7 billion for three different federal  programs for high speed rail, intercity passenger rail, and congestion  reduction grants. PRIIA also called for FRA to create a preliminary  national rail plan within 1 year after passage of the act as well as a long- range national rail plan that promotes an integrated and efficient national  rail system. FRA released a preliminary national rail plan in October 2009  and a subsequent progress report in September 2010.
The Recovery Act, enacted in February 2009, appropriated $8 billion for  the three PRIIA-established intercity passenger rail programs. Unlike  PRIIA, which authorized an 80 percent federal share for a project’s capital  costs, the Recovery Act provided up to 100 percent federal funding  available for obligation through fiscal year 2012 and expenditure through  fiscal year 2017. The Recovery Act required that the department develop  a strategic plan to use these funds. In April 2009, FRA released its strategic  plan for developing high speed rail in America and distributing federal  funds.
PRIIA and the Recovery Act created new responsibilities for FRA to plan,  award, and oversee the use of new federal funds for intercity passenger  rail. In response, FRA launched the high speed intercity passenger rail  (HSIPR) program in June 2009 by issuing a funding announcement and  interim guidance, which outlined the requirements and procedures for  obtaining federal funds. FRA further outlined the vision and goals of the  program through a number of outreach events and meetings, including  seven regional workshops and more than 25 one-on-one site visits and  conference calls with potential state applicants. States expressed a great  deal of enthusiasm for the new program, requesting $102 billion across 278  preapplications, which FRA used to gauge initial interest and anticipate its  staffing needs to manage the program. States, including the District of  Columbia, ultimately submitted 229 applications for $57.8 billion in  Recovery Act funds.
FRA asked applicants to submit Recovery Act project applications under  three tracks: 1a, 1b, and 2. Track 1 was intended to primarily address  economic recovery goals, and could either focus on ready-to-go projects  (track 1a) or the completion of environmental and preliminary engineering  requirements necessary to prepare projects for future funding (track 1b).  Track 2 focused on much larger, long-term projects to develop new high  speed rail services or substantially upgrade existing corridor service.  While track 1 and track 2 applications were submitted and reviewed at  different times, FRA used a similar approach to assess them, and applied  the same criteria during three independent steps: eligibility determination,  technical review, and selection. (See fig. 1.)
The eligibility determination was conducted by panels of officials  experienced in environmental requirements or passenger and commuter  rail. These officials used worksheets to aid in assessing application  completeness and determining whether applicants and proposed projects  were eligible to receive funds. Eligibility panels also made preliminary  determinations as to whether applicants had substantially completed  environmental requirements and whether the projects they submitted were  ready to begin. For the most part, applications deemed not yet ready or  ineligible were not forwarded for technical review; because the track 1  eligibility and technical review periods overlapped, there were two  Recovery Act applications that received technical review scores and were  later deemed not yet ready or ineligible and removed from award  consideration.
The technical review was conducted by panels of officials with experience  in several fields, such as passenger and commuter rail, grants  management, and environmental requirements. The technical review  differed slightly for track 1 applications, which were reviewed by 12  panels composed of three reviewers, and track 2 applications, which were  reviewed by a single panel of eight reviewers. For both tracks, reviewers  used guidebooks to assess applications against six technical review  criteria: (1) transportation benefits, (2) economic recovery benefits, (3)  other public benefits (e.g., environmental quality and energy efficiency),  (4) project management approach, (5) sustainability of benefits, and (6)  timeliness of project completion. (See table 1.) The guidebooks provided  detailed descriptions of what was included within each of these criterion,  as well as step-by-step instructions on reviewing applications that included  a suggested scoring method using a scale from one (lowest) to five  (highest). For example, the track 2 guidebook suggested applications that  included more than one major weakness, were nonresponsive, or failed to  address a particular criterion be given a technical review score of a one for  that criterion. Applications that technical panelists determined were  responsive, and included major and minor strengths and no major or very  few minor weaknesses in a particular criterion, were to be given a  technical review score of a five for that criterion. After completing an  individual evaluation of each application, reviewers convened within their  panel to discuss their overall thoughts on the application and technical  review scores for each criterion, which they could revise based on input  from other panelists. To arrive at a final score for each application, FRA  officials used a formula that averaged individual scores and weighted the  scores based on established priorities identified in the funding  announcement. In addition, program officials standardized track 1  application scores to correct for potential inconsistencies across review  panels.
After the technical review, senior department and FRA officials—Deputy  Secretary, Under Secretary for Policy, FRA Administrator, and FRA  Deputy Administrator, among others—selected projects to recommend to  the Secretary of Transportation. They considered the technical review  scores along with four additional pre-established selection criteria  identified in the funding announcement: (1) region and location, (2)  innovation, (3) partnerships, and (4) track type and funding round. (See  table 2.) HSIPR program officials gave five briefings to senior officials on  the results of the technical review and possible factors to consider in  making award decisions, such as potential project cost, service speed,  shared benefits, and readiness. Program officials also provided additional  information, including funding scenarios, facts sheets on individual  applications, and corridor maps upon request. According to FRA, senior  officials considered this information when making their recommendations,  but did not numerically score or rank applications.
On January 27, 2010, the FRA Administrator recommended 62 applications  for funding and the Secretary of Transportation concurred with these  recommendations. On January 28, 2010, DOT announced the selections.  The selections were spread across several types of intercity passenger rail,  including projects for emerging high speed rail (operating at speeds up to  90 miles per hour), regional corridors (operating at speeds between 90 and  124 miles per hour), and core express corridors (operating at speeds  between 125 and 250 miles per hour or more). For example, the  department selected one project to receive $35 million to rehabilitate track  and provide service from Portland to Brunswick, Maine at speeds up to 70  miles per hour. Another project was to receive more than $50 million to  construct 11 miles of dedicated passenger rail track near Rochester, New  York, which will allow for service speeds up to 110 miles per hour. A third  project was selected to receive almost $2.3 billion to initiate the first part  of the California’s high speed rail system, which will allow for more than  200 miles per hour service between Los Angeles, San Francisco and the  Central Valley, and eventually, San Diego. These selections were  consistent with the criteria in PRIIA, the Recovery Act, and FRA’s strategic  plan, which included broad goals that gave FRA discretion in developing a  national passenger rail system. (Additional information on the legislative  and program goals, and how the selected projects fit into them, is  contained in app. I.)
section_title: FRA Applied Its Established Criteria to Determine Eligibility and Assess Technical Merit, but Selection Rationales Were Typically Too Vague to Assess
paragraphs: FRA applied its established criteria to determine eligibility and assess  applications’ technical merit. However, its rationales for selecting projects  were typically too general to determine how it applied the additional  selection criteria. When asked for more information on certain  applications, FRA provided specific reasons for its selection decisions,  but, in our opinion, creating a detailed, comprehensive record alongside  the final selections is preferable. Officials reported that they used the  technical review scores as a starting point from which to apply each of the  four selection criteria, which is partially supported by our analysis of FRA  data. For example, we found that applications receiving a higher technical  review score were about seven to eight times more likely to be selected for  an award compared to those receiving a lower technical review score.
subsections: section_title: FRA Applied Its Established Criteria to Determine Eligibility and Assess Technical Merit
paragraphs: We found that FRA applied eligibility criteria established in its funding  announcement when determining whether applications were eligible.  Specifically, eligibility criteria listed in the funding announcement aligned  with criteria outlined in the worksheets used by the panelists to verify that  applications were eligible. Panelists were given separate worksheets to  conduct the track 1 and track 2 eligibility reviews, and each of these  worksheets included eligibility criteria listed in the funding  announcement. For instance, as outlined in the funding announcement,  the track 1 worksheet required eligibility panelists to indicate if the  application was submitted on-time, by an eligible applicant, and with all of  the required supporting documents. Similarly, the track 2 worksheet  included questions regarding applicant eligibility, qualifications, and  construction grant prerequisites which aligned with the eligibility criteria  listed in the funding announcement.
FRA also applied the established technical review criteria communicated  in its funding announcement by including these criteria in the guidebooks  provided to technical panelists to assess the technical merits of each  application. Specifically, the guidebooks FRA provided to panelists for  reviewing track 1a, 1b, and 2 applications were divided into six sections  that aligned with each of the six technical review criteria listed in the  funding announcement. Moreover, the criteria within these sections of the  guidebook often matched the criteria in the funding announcement very  closely and, in some cases, word-for-word. For example, the funding  announcement stated that an applicant’s experience administering similar  projects would be considered under the technical review criteria of  project management approach, which was included word-for-word in the  project management approach section of the guidebooks. We spoke to at  least one representative from each technical review panel; these  representatives confirmed that panelists used the criteria listed in the  guidebooks and did not use other criteria during their evaluation of  individual applications.
section_title: Officials Reported Using Technical Review Panel Results and Selection Criteria to Make Awards Recommendations; Decision Rationales Provided Little Insight into Selections
paragraphs: Senior department and FRA officials recommended to the Secretary of  Transportation applications to receive awards and the proposed amounts  of the awards. When deciding which applications to recommend for  awards, senior FRA officials told us that they used the results of the  technical review panels and the four selection criteria. These four criteria  were described in FRA’s June 2009 funding announcement as: (1) region  and location (e.g., ensuring geographic balance, integration into the  nationwide transportation network, and assistance to economically  distressed regions), (2) innovation (e.g., pursuing new technology with a  favorable public return, promoting domestic manufacturing, and  developing human capital capacity for sustainable rail development), (3)  partnerships (e.g., multi-state planning and investment and workforce  diversity), and (4) tracks and round timing (e.g., longer-term track 2  corridor development balanced with ready-to-go track 1 investments). For  example, officials stated that they used the innovation criterion to select  applications with higher proposed speeds of service. In particular, senior  officials reported using this criterion to reinforce the selection of the  California and Florida intercity passenger rail projects, which were the  only eligible projects with the potential for service above 150 miles per  hour. In another example, officials reported that they applied the  partnership criterion by assessing applicants’ track record with  implementing large transportation projects as well as demonstrated  relationships with key stakeholders, such as private railroads.
Senior FRA officials stated they developed their funding amount  recommendations based on their professional judgment and national high  speed intercity passenger rail program goals. Officials told us they  accounted for the risks related to the total cost of the project during  selection discussions and weighed them against the overall policy goals of  developing a national high speed passenger rail network. Officials also  stated that they used their professional judgment about rail systems to  recommend the award amounts for each application, paying particular  attention to the amounts distributed to the large, track 2 projects, and that  they are continuing to assess the effect of changes to the requested  funding amounts during the scope of work negotiations with awardees.
According to FRA, its rationales for selecting applications are recorded in  a recommendation from the FRA Administrator to the Secretary of  Transportation and in a memorandum from the Secretary to the  Administrator concurring on the recommendations and specifying  potential funding amounts. The rationales stated in these memorandums  were typically vague, such as “aligns well with FRA’s published evaluation  criteria” and “will result in significant transportation benefits   preserve and create jobs.” These rationales most often restated the criteria  listed in the funding announcement generally (e.g., result in significant  transportation benefits) rather than providing insight into why the  department viewed projects as meritorious.
In addition, the memorandums did not provide any information on why  other applications were not recommended for selection, which prevents  us from assessing how the department viewed the merits of successful  applications over unsuccessful ones. For example, we found several  instances in which, without documentation, it was difficult to determine  the reasons why some projects were selected and others were not.  Specifically, FRA decided not to select six track 1a applications from New  York that received higher technical review panel scores, and selected a  lower scoring track 1a application from the same applicant. FRA officials  subsequently told us that the lower scoring application was selected for a  number of reasons, including improving the reliability of the passenger  trains on the rail line, ensuring that the project will become part of the  infrastructure of any significant improvements to passenger rail service  west of Albany, and improving the fluidity of both passenger and freight  rail operations on this heavily used rail route. Similarly, FRA selected a  lower scoring track 1a application from Illinois, but not a relatively high  scoring one. FRA officials subsequently told us that they selected this  application because it is an essential part of a long-standing program of  projects to improve the fluidity of rail traffic in the highly congested  Chicago area. FRA officials also told us that the scope of the relatively  high scoring track 1a application was included in Illinois’ selected track 2  application. This level of information, which provides some insight into  the merits of projects, was not included in the department’s record of its  decisions.
In addition to the memorandums, FRA posted descriptions on its Web site  of the selected projects, their expected benefits, and prospective award  amounts. However, these descriptions are not particularly useful in  understanding why these projects were selected because the cited  benefits—such as reducing travel times, increasing travel speed and  ridership, providing attractive transportation alternatives, and creating  jobs—were supposed to be integral to all projects. For example, FRA’s  Web site describes one project as increasing on-time performance and  ultimately allowing speeds of up to 110 miles per hour on its segment, but  does not give any indication why this project was meritorious. Other  descriptions were similar.
FRA also sent letters to individual applicants regarding its decision, and, if  the application was not selected, a brief explanation as to why it was not  selected. For example, a number of these letters explained that  applications were not selected because they did not meet a prerequisite,  had application materials that did not provide sufficient support for the  proposed activities, or did not submit all application materials necessary  to adequately evaluate the project. However, these letters did not provide  further details on how the proposed projects did not meet the prerequisite,  how the application materials were insufficient, or which application  materials were not received. Other decision letters provided applicants  with similarly broad explanations. FRA officials also told us that they  called all applicants, as well as their state secretaries of transportation and  state governors, to inform them of FRA’s decisions. Several of the states  that we contacted reported that the primary purpose of these calls was for  FRA to provide feedback on their individual projects and, when requested,  give explanations for why projects were not selected. While applicants  stated that this information will be helpful during future application  rounds, there is no required written record of these conversations and,  therefore, they do not provide others with insight on why selection  decisions were made.
Documentation of agency activities is a key part of accountability for  decisions. The department has a financial assistance guidance manual to  assist agencies with administering awards competitions and which FRA  officials told us that they used to develop the competition framework.  The manual recommends that all discretionary project selections, such as  the intercity passenger awards, include an explanation of how the projects  were selected based on the established funding priorities, but does not lay  out expectations for the level of explanation. In particular, the manual  recommends that officials document decisions if projects with the highest  priority are not funded.
While the department documented its decisions, as required by its  financial assistance manual guidance, the absence of an insightful internal  record of the reasons behind award recommendations, and the final  selections where they differ, can give rise to challenges to the integrity of  the decisions made. While FRA was able to provide us with specific  reasons on a case-by-case basis for why projects were selected, almost a  year after these decisions were made, we believe creating a sufficiently  detailed record has increased relevance in high-stakes, high-profile  decisions, such as the intercity passenger rail awards competition in which  there are vocal critics and ardent supporters of the program.
Similar arguments apply for creating an internal record for amounts  recommended for awards. FRA officials understood that the available  Recovery Act funds were not sufficient to fully fund a number of the  projects and sought to fund projects or portions of projects that could  provide transportation benefits if no additional federal funds were  available. For these decisions, FRA proposed awarding 10 states  (including the District of Columbia) all (100 percent) of the funds they  applied for, 8 states nearly all (91–99 percent) of the funds they applied  for, 5 states some (47–86 percent) of the funds they applied for, and one  state with slightly more (104 percent) than it applied for. (See fig. 2. See  also app. III for dollar amounts associated with fig. 2.) The applicant  notification letters did not offer an explanation for why FRA proposed  award amounts that differed from requests, and applicants we spoke with  did not report that FRA had provided such information to them. Given that  infrastructure projects have an inclination for cost growth, developing a  record that explains why the recommended costs are appropriate for the  proposed project provides integrity to the final decisions.
The current economic climate has also increased the importance of  providing an internal rationale for large differences between requested  funds and proposed award amounts. Many states have faced large budget  deficits in 2010 that will require them to make difficult budget decisions  about the future use of state funds, particularly where the Recovery Act  awards will not provide all the funding expected to be needed to complete  a project. For example, as of June 2010, Florida had made $3 billion in  budget cuts to close its budget deficit. For its high speed rail award,  Florida is slated to receive less than half of what it said is needed to  complete the proposed Tampa to Orlando High Speed Rail Express  project. An official from the Florida Department of Transportation is  hopeful that Florida will receive additional federal grants, but is unsure  where the remaining funds will come from otherwise. Additionally,  Washington state applied for 16 separate projects totaling $976 million and  was selected to receive a composite award of $590 million. Washington  state officials acknowledged that the award amount will not fund all 16 of  the projects, and have since reduced the scope of the application to the 11  projects that could be completed with the awarded amount while still  providing the maximum benefit to the corridor. FRA officials stated they  awarded amounts that differed from those requested in applications as a  result of their recognition that many of the projects were based on  preliminary work that was not well-refined, and that states differed in their  ability to accurately estimate costs.  In contrast, North Carolina received  4 percent more funding than originally requested. According to FRA, the  additional funding was allocated to North Carolina for possibly adding  additional train frequencies for a Recovery Act project. While we  recognize that FRA may have developed these proposed award amounts  for good reasons, without a written record of the department’s rationale  for these adjustments, after the fact reconstructions of funding amount  decisions invite outside criticism of the decisions.
section_title: Applications with Higher Technical Review Scores Were Typically Chosen Over Those with Lower Scores
paragraphs: One of your interests was in how the results of technical review panels  aligned with final award decisions. As discussed earlier, while FRA  considered the technical review panels to be an important part of its  decision making, they were not the sole basis for selecting projects. This  was detailed in FRA’s funding announcement, which described how  applications were first to be assessed against six technical review criteria  and then final recommendations would be made using the technical  review results and four selection criteria.
While the technical review panel evaluations alone were not meant to  designate final selections, we found that of 179 eligible Recovery Act  applications, senior management recommended 92 percent (57 of 62) of  higher scoring applications for funding; that is they received review panel  scores of 3 or higher out of 5 possible points. (See fig. 3.) Within these  recommended applications, most received a technical review score of 3 or  4, and three of the five applications that received a technical review score  of 5 were recommended for selection. One of the two applications that  scored a 5 and was not selected for funding was included in a selected  track 2 application.
In a few cases though, senior officials recommended applications that  received a lower technical review score (i.e., a score of 2) because,  according to FRA, they believed these projects included freight and  commuter rail service partners that were willing to make cost  contributions in line with their potential benefit share, were strategically  important to other selected applications, or helped to achieve regional  balance. These considerations were included in the four selection criteria  senior department and FRA officials said they used to evaluate  applications. For example, one of the two applications from Rhode Island  requested $1.2 million to complete preliminary engineering and  environmental reviews and received a lower overall technical review  score, in part because technical reviewers did not believe the applicant  had sufficiently quantified the transportation and economic recovery  benefits. This application was later recommended for selection. According  to FRA, senior officials recommended applications receiving lower  technical review scores, such as this Rhode Island application, in part to  achieve greater regional balance. Additionally, FRA indicated this  particular application was one of the few applications proposed for the  Northeast Corridor, which further supported the region/location selection  criteria. In another instance, senior officials selected a track 2 application  from California that requested $194 million for preliminary engineering  and environmental requirements for a large corridor application that  received a lower technical review panel score. According to FRA, senior  officials recommended some applications receiving lower technical review  scores due to the projects’ strategic importance to other selected  applications. Officials stated that they recommended the track 2 California  application because the completion of preliminary engineering and  environmental requirements were necessary to move forward on several  other large California projects also recommended for an award.
Officials also told us that some applications receiving a higher technical  review score (i.e., 3, 4, or 5) were not selected in order to ensure regional  balance, especially when an applicant had already been selected for other  large awards. For example, a track 1a application from North Carolina  received a higher technical review panel score due, in part, to the  anticipated transportation benefits of increased ridership and on-time- performance, and the applicant’s estimates that the project would create  more than 400 new jobs. Most of the projects that North Carolina applied  for under this application were also included as part of a larger, intercity  passenger rail application that was later recommended for selection, and  the state was awarded an estimated total of $545 million for high and  conventional speed rail projects. Department and senior FRA officials  reported that higher evaluated applications were not selected if the  proposed project was already included in larger selected projects, to avoid  duplicative selections. Another example was FRA’s decision not to select a  higher scoring track 1a application from Florida that requested $270  million to acquire 61 miles of right-of-way. This application was scored  highly due in part to its immediate benefits and substantial contribution of  state funds but, similar to North Carolina, Florida had already been  awarded $1.25 billion for a separate large, track 2 corridor project. In  addition, almost 90 percent of the applications that scored a 4 and were  not selected were submitted by applicants that had either already received  a large award or had submitted a relatively high number of applications.
To provide further insight into the attributes that were consistent with  being selected for Recovery Act awards, we examined technical review  score and application data using a statistical model and found that two out  of four variables we included in our model, technical review scores and  the number of applications submitted per state, were significantly related  to the likelihood of an application being selected for an award.  Applications with higher scores (i.e., scores of 3, 4, or 5) were about seven  to eight times more likely to be selected than those with scores of 1 or 2.  For example, an application receiving a technical review score of 5, the  highest possible score, was more than nine times more likely to be  selected for an award as an application receiving a technical review score  of a 1 or 2. This analysis supports statements from senior department and  FRA officials indicating that the technical review scores were largely the  basis for their selection deliberations.
Additionally, we found that states submitting fewer applications (i.e.,  between one and three) were more than three times more likely to have  their application selected than states submitting higher numbers of  applications (i.e., between four and nine). This result suggests that  selection officials attempted to spread the awards across different  applicants, which is consistent with FRA’s reported efforts to attain  geographic distribution. However, the results differed somewhat for the  four states that submitted 10 or more applications. In this case two of the  states had a lower likelihood of being selected for an award than states  submitting fewer than 10 applications, while one state had a higher  likelihood of being selected. One additional state had about the same  likelihood of being selected as states submitting between four and nine  applications. When asked about these differences across states, FRA  officials said that the number of applications submitted did not affect their  selection decisions.
section_title: FRA Substantially Met Recommended Practices for Awarding Discretionary Grants
paragraphs: We identified six recommended practices used across the federal  government to ensure a fair and objective evaluation and selection of  discretionary grant awards. These practices are based on policies and  guidance used by the Office of Management and Budget and other federal  agencies—including the department, and our work. FRA substantially  followed these practices, including communicating key information to  applicants, planning for the competition, using a technical merit review  panel with desirable characteristics, assessing applicants’ ability to  account for funds, and notifying applicants of awards decisions. (See table  3.) In our opinion, FRA partially met one recommended practice:  documenting the rationale for funding decisions. As discussed previously,  we believe it would have been beneficial to provide more detail about the  rationales for these decisions. According to FRA officials, the methods  they used to evaluate and select applications were based on best practices  collected from several other federal government agencies, which we  believe likely helped them meet a number of the recommended practices  we identified.
Communicate with potential applicants prior to the competition. FRA  issued a funding announcement that included information on the $8 billion  in available funding, key dates, the competition rules, the funding  priorities and relative importance for each one, and the types of projects  FRA would consider for federal grants. Applicants we spoke with praised  FRA’s communication and stated that FRA officials did a good job  providing information and answering questions during the period leading  up to the preapplication and application deadlines. For example, officials  from several states indicated that FRA officials participated in biweekly  conference calls, which were helpful in understanding the technical  aspects of how to apply. Applicants also indicated that the outreach  events, particularly the site visits, helped them refine their applications  and ensure projects met program requirements.
Plan for administering the technical review. FRA developed two plans  for determining technical merit: (1) the track 1 technical review used 12  panels each comprised of three reviewers and (2) the track 2 technical  review used one panel of eight reviewers. Track 1 applications were  randomly assigned across the panels, while the track 2 panel reviewed all  of the eligible applications. FRA identified and asked for volunteers to  participate in the technical reviews from within FRA and across several  other agencies within the department. FRA officials also provided  reviewers with guidebooks to document their application assessments and  instructed them to input the results, including scores and comments, into a  centralized database. FRA standardized final track 1 application scores to  account for any unintentional differences in the way panels assessed and  scored applications, but did not need to standardize track 2 scores  because the review was conducted by a single panel. Finally, according to  officials, FRA oversaw the review by examining technical review scores  and comments, and conducting daily meetings with representatives from  each panel to ensure panelists were consistently applying the criteria.
Develop a technical review panel with certain characteristics. FRA  compiled technical review panels that included staff with background in  several relevant fields, such as grants management, passenger and  commuter rail, and environmental requirements, and made other  knowledgeable staff available if panelists had questions. FRA officials  stated that panelists were also required to sign or submit a previously  completed conflict of interest form to attest to their independence. In  addition, panelists were given guidebooks to assess applications that  included the technical review criteria and were told by FRA program  officials to apply only these criteria during their efforts. FRA also trained  panelists during a 1-day orientation session.
Assess applicants’ capabilities to account for funds. FRA required  applicants to provide information on their ability to account for funds.  Specifically, applicants were asked to describe their experience, if any,  managing rail investment projects. If applicants reported that they did not  have experience on projects similar to the one they were proposing, FRA  instead asked applicants to include a plan for building the capacity to  manage the project. The application also required applicants to provide  information on their financial management capability, including previous  audit results, and the applicants’ ability to manage potential cost overruns  and financial shortfalls. In addition, FRA required applicants to submit  supplemental materials such as a detailed capital cost budget, which  provided a breakdown of the activities included in each application and  their anticipated cost. These pieces of information were assessed by FRA  through an eligibility panel, to ensure the application was complete, and a  technical review panel, to evaluate the applicants’ overall ability to  manage the project.
Notify applicants of awards decisions. FRA officials provided each  applicant with a letter indicating which applications were selected and a  general reason why individual applications were not selected. While FRA  did not include estimated award amounts in these notification letters, this  information was made publicly available on the department’s Web site and  distributed through a press release. In addition, most of the applicants we  spoke with indicated that FRA provided informal feedback on applications  via telephone calls shortly after the awards were announced. For example,  an official from one applicant stated that FRA provided information on  ways to improve applications that were not selected, which the applicant  used when applying for funds in future rounds.
Document rationale for awards decisions. According to the guidance  from the department, Department of Commerce, the Department of  Education, and our work, agencies should document their rationale for  award decisions. As stated previously, FRA documented how it applied the  technical criteria for selected projects, and provided applicants with a  general explanation for selecting or rejecting individual projects. However,  as discussed in a previous section, in our view FRA typically did not  clearly document specific reasons for selecting individual projects,  reasons for not selecting other projects, or how changes made to  requested funding amounts might affect applicants’ ability to achieve  project goals.
According to FRA, officials used lessons from a number of other  government programs when developing the method for evaluating and  selecting projects. For example, one of the officials responsible for  developing the funding announcement, technical review guidebooks, and  the format of the technical review panels stated that he relied on his  experience working with large transit grants to create a review that was  both quantifiable and allowed for subjective professional judgment. In  addition, this official noted that FRA examined the methods used by other  agencies, such as the Department of Health and Human Services, the  Department of Justice, and the Federal Transit Administration, to develop  and implement a list of best practices for awarding discretionary grants.
section_title: FRA Publicly Communicated at Least as Much Outcome Information as Other Competitively Awarded Recovery Act Grant Programs
paragraphs: FRA publicly communicated outcome information, such as a list of awards  and the award amounts, at a level similar to or greater than most other  Recovery Act competitive grant programs that we examined. Specifically,  FRA communicated information on award decisions to the public, but did  not communicate the results of the technical review that had contributed  to these decisions. Only one of the programs that we examined—the  Department of Education’s State Innovation grants (known as Race to the  Top)—publicly communicated the results of its technical review, which  include technical scores and comments; however, this program used a  much different approach for selecting awardees than the HSIPR program.
Members of Congress and the President have emphasized the need for  accountability, efficiency, and transparency in the expenditure of  Recovery Act funds and have made it a central principle of the act.  However, the act did not define the attributes of transparency or how  deeply an agency’s actions should be transparent. We also did not find  any non-Recovery Act requirement or guidance instructing federal  programs to publicly disclose the reasons for their selection decisions. To  assess the extent to which FRA publicly communicated outcome  information, we compared the HSIPR program to 21 other Recovery Act  competitive grant programs, including Race to the Top. (See fig. 4.) We  selected 20 of these programs randomly from a list of almost 200  competitively awarded grant programs that distributed Recovery Act  funds. We included the 21st program, Race to the Top, because it was of  interest to you.
FRA publicly communicated at least as much outcome information as all  but one Recovery Act competitive grant programs we reviewed. Specifically,  FRA publicly communicated through its Web site the selection decisions,  including the amount of funds requested, general benefits from the project,  and the potential award amounts for the 62 Recovery Act applications that it  selected. It did not communicate the results of the technical review. Out of  the other 21 competitively awarded Recovery Act programs we examined,  13 communicated selection information similar to FRA, including awards  and award amounts, but not the results of the technical review. For  example, the Department of Health and Human Services’ National Institutes  of Health published a list of 21,581 award winners for nearly $9 billion, but,  similar to the HSIPR program, did not report the results of the technical  review. Eight other programs conveyed less information than FRA and did  not publicly communicate the results of the technical review or the awards  and award amounts.
Race to the Top was the only program we examined that publicly provided  the results of its technical review. These results, which were posted on the  Department of Education’s Web site, included scores and comments from  reviewers for each applicant, but were not connected to individual  reviewers by name. According to its Web site, the Department of Education  decided to release this level of detailed information because the $4 billion  Race to the Top program was larger than any other discretionary program  the Department of Education had previously administered, and officials  sought to ensure the highest level of integrity and transparency. Unlike the  HSIPR program, however, Race to the Top used these scores as the sole  basis for selecting awards and only chose applicants receiving the highest  scores. As described in the previous section, the technical review scores  were an important component for making HSIPR selection decisions, but  did not include consideration of additional pre-established selection criteria  designed to ensure long-term success and sustainability of the program. As  such, publishing them without additional decision making information on  the specific reasons for selecting and not selecting individual applications  could lead to erroneous conclusions about FRA’s decisions.
According to FRA officials, the results of the technical review were not  communicated because department officials were concerned that  associating technical review scores and comments with a specific  reviewer could discourage reviewers from participating in future  department competitive grant evaluations. Furthermore, in their view, this  might also prevent reviewers in future funding rounds from providing  candid evaluations. However, as the Race to the Top program  demonstrated, it would be possible for FRA to present overall technical  panel review assessments or their individual comments without linking  individuals’ names to comments, if it chooses to do so. FRA officials stated  that the anonymous disclosure of technical scores and comments would  still prevent FRA and department leadership and staff from frankly  expressing their individual judgments, as they might still be concerned  over how theses opinions would reflect on the FRA and HSIPR program if  they were made public.
section_title: Conclusions
paragraphs: The $8 billion appropriated by the Recovery Act for the High Speed  Intercity Passenger Rail program represents a large investment in the  development of a national passenger rail network. FRA established a fair  and objective approach for distributing these funds and substantially  followed recommended discretionary grant award practices used  throughout the government. The exception is what we view as incomplete  documentation of why some applications were chosen and not others, and  how FRA decided to distribute the funds at the time those decisions were  made. This incomplete documentation is notable given the robust  documentation of the other steps used to determine eligibility and assess  technical merit. We believe that establishing a record that provides insight  into why decisions were made, rather than merely restating general  technical review and selection criteria, including amounts to be provided,  would enhance the credibility of FRA’s awards decisions to the extent that  this record confirms that selected projects aligned with established  criteria and goals. By not establishing this record, FRA invites skepticism  about the overall fairness of its decisions, even if they are sound, and  hinders meaningful disclosure of how it made its decisions, if it chooses to  do so.
section_title: Recommendation for Executive Action
paragraphs: To help ensure accountability over federal funds, we recommend that the  Secretary of Transportation direct the Administrator of the Federal  Railroad Administration to create additional records that document the  rationales for award decisions in future HSIPR funding rounds, including  substantive reasons (1) why individual projects are selected or not  selected and (2) for changes made to requested funding amounts.
section_title: Agency Comments
paragraphs: We provided a draft of this report to the Department of Transportation for  its review and comment. The department told us that it carefully  constructed the grant processes for the HSIPR program based on  extensive review and consideration of best practices both within and  outside the agency with the intent of providing a comprehensive and  transparent process. The department indicated that its overall intent was  to select the best projects that offered the greatest available and  achievable benefit to the nation. The department told us that it would  carefully consider our recommendation to determine if there are means to  further enhance the transparency of its grant selection process with  additional documentation, without creating a process that is unduly  burdensome to administer. The department also offered technical  comments which we incorporated as appropriate.
As agreed with your office, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. We are sending copies of this report to congressional  subcommittees with responsibilities for surface transportation issues; the  Director, Office of Management and Budget; the Secretary of  Transportation; and the Administrator of the Federal Railroad  Administration. In addition, this report will be available at no charge on  GAO’s Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. GAO staff who made key contributions to this  report are Owen Bruce, Matthew Cook, Colin Fallon, Michele Fejfar, Maria  Gaona, Grant Mallie, James Ratzenberger, Douglas Sloane, Matthew Voit,  and Crystal Wesco.
section_title: Appendix I: Extent to Which Recovery Act Projects Align with Statutory and Other Goals
paragraphs: We examined the extent to which American Recovery and Reinvestment  Act of 2009 (Recovery Act) projects selected by the Federal Railroad  Administration (FRA) align with legislative and the administration’s goals  to develop high speed and conventional rail networks.
subsections: section_title: Congress Provided Broad Goals with a Priority for High Speed Systems
paragraphs: Congress provided its most recent expectations for high and conventional  speed rail in the Passenger Rail Investment and Improvement Act of 2008  (PRIIA) and the Recovery Act. In this regard, PRIIA speaks generally about  supporting improvements to high and conventional speed rail and does not  set out any expectations for relative attention to high and conventional  speed passenger rail improvements. The Recovery Act appropriated $8  billion for both forms of rail service broadly. However, it required that  FRA give priority to projects that support the development of intercity  high speed service. Further, the act required that FRA develop a strategic  plan that describes how FRA will use Recovery Act funding to improve  and deploy high speed systems. FRA had wide latitude to achieve goals  laid out in its strategic plan.
section_title: FRA’s Vision Describes Broad Goals for High Speed Rail, but Provides Limited Detail on How Goals Will Be Achieved
paragraphs: FRA has outlined its vision for developing intercity passenger rail service  in its strategic plan, as required by the Recovery Act, and in both its  preliminary national rail plan issued in 2009 and the plan’s update nearly a  year later. FRA’s vision documents—the strategic plan and its updated  national rail plan—described broad goals, such as for transportation,  safety, and economic competitiveness, and established categories for the  type of high speed rail projects it intends to support. For example, the  strategic plan notes the high speed rail program aims to generate  construction and operating jobs, while providing a steady market for  various industries producing rail, control systems, locomotives, and  passenger cars. In addition, the plan notes that investments in high speed  rail can result in competitive trip times and rail transport can also result in  higher-density development as compared to other modes of  transportation. Similarly, the updated national rail plan sets a goal of  connecting communities through high speed rail while, among other  things, reducing congestion, boosting economic growth, and promoting  economic sustainability.
However, these vision documents provide limited details on goals for the  high speed rail program. For example, while the strategic plan emphasizes  investments that will yield tangible benefits to rail performance and  improve connections between different modes of transportation, it does  not describe how and when FRA intends to realize these benefits. In  addition, as we reported last June, the preliminary rail plan did not offer  specific recommendations for future action and was designed to serve as a  springboard for further discussion with states and freight railroads. While  the update to this plan included improving rail performance as a goal and  provided some measurements for high speed rail performance, such as  competitive trip times, it did not provide any specific targets for these  metrics, or any time line showing when FRA hopes to attain these  improvements.
section_title: FRA’s Application Selection Is Consistent with the Recovery Act’s Priority for High Speed Rail Service
paragraphs: Consistent with the Recovery Act’s direction to give priority for high speed  rail service, about half (45 percent) of the applications selected were for  core express corridors (high speed service of 125–250 miles per hour or  more) or regional corridors (higher-speed service of 90–124 miles per  hour) using categories of service similar to those FRA established in its  vision documents. (See table 4.) FRA did not establish specific targets for  the number of each type of project it intended to support.
section_title: Selected Applications Reflect Short-Term Economic Recovery and Long-Term Infrastructure Investment Goals
paragraphs: In addition to providing priority for high speed projects, the applications  that FRA selected were consistent with near-term economic recovery  goals and its long-term development goals. While most selected projects  are short-term in nature and are intended to support economic recovery  goals established by the Recovery Act, most funding was provided to  several long-term, high speed corridor projects. Specifically, we found that  48 of the 62 applications selected were track 1 applications, which are  smaller projects designed to be completed within 2 years. (See table 5.)  These projects represent about 11 percent of the funding provided for high  speed rail and intercity passenger rail through the Recovery Act. The  remaining 89 percent of funding was provided for 14 track 2 applications,  which are primarily long-term, corridor projects. The funding allocation  aligns with FRA’s focus on long-term investments that will support  development of a high speed passenger rail network as described in the  funding announcement.
While FRA announced in January 2010 awards of nearly $8 billion in grants  for the program, many of these projects have only recently begun. As of  December 31, 2010, FRA had obligated $4.2 billion, or about 54 percent of  the funding awarded in January, and about $50 million has been spent for  projects selected under track 2.
In May 2009, FRA issued a plan for spending Recovery Act funds, which it  updated in July 2010. FRA missed its May 2009 targets for obligations and  spending through 2010 estimate because it had planned to announce  awards—and begin obligating funds—in the autumn of 2009. However,  FRA did not make those announcements until January 2010 and did not  begin to obligate funds until May 2010. FRA then revised its estimates in  July 2010. FRA surpassed the calendar year 2010 goals for obligating and  spending funds in the July 2010 plan. (See table 6.) During calendar year  2010, FRA obligated about 11 times as much as anticipated in the July 2010  plan, while awardees have spent about 7 times as much as planned over  the same time period. The Recovery Act authorized obligation of funds  through September 30, 2012, and FRA intends to obligate all funds by this  date.
Passenger rail investments are often long-term efforts that must be carried  out in partnership between the state and others, notably private railroads.  For example, in order to begin design and construction on many of these  projects, grant recipients must negotiate and secure agreements with  private freight railroads to use their tracks for passenger rail trains.  However, officials from these railroads are concerned that sharing tracks  would create safety risks and liability concerns, prevent freight expansion,  and cause rail congestion. Some of the states have experienced delays  finalizing these agreements with the railroads and, accordingly, have not  completed agreements with FRA to obligate awarded funding.
section_title: Appendix II: Scope and Methodology
subsections: section_title: Criteria Used to Select Projects
paragraphs: To determine the extent to which FRA applied its established criteria to  select projects, we identified the criteria that it planned to use from its  June 23, 2009, funding announcement outlining its evaluation and selection  approach. We then compared these criteria to the worksheets and  guidebooks that FRA used to determine eligibility and assess technical  merit. Finally, we interviewed FRA officials who participated in evaluating  and selecting projects to obtain information on whether and how they  applied the established criteria. Specifically, we randomly selected 1  technical reviewer from each of the 12 track 1, 3, and 4 panels (12 out of 36  reviewers), and 6 of the 8 reviewers from the track 2 panel. In addition, we  interviewed senior FRA officials to further understand how senior  Department of Transportation (the department) and FRA officials applied  the selection criteria, selected projects, and determined the amount of  funding provided for each project. We also asked FRA officials to provide  reasons for why several lower scored applications were selected, while  other higher scored applications were not.
We conducted semi-structured interviews with officials from 10 of the 40  states and the District of Columbia that submitted a preapplication or an  application for track 1 and track 2 funding about how FRA communicated  its approach to reviewing applications and award results. We selected  these states on the basis of four characteristics: (1) the extent to which  applicants progressed through the preapplication, application, evaluation,  and selection stages; (2) geographic regions; (3) the number of  applications submitted; and (4) the amount of funding. We also contacted  officials in two additional states (Ohio and Washington) to understand the  effect of FRA’s funding decisions on the scope of these states’ proposed  rail program. Our efforts were limited to applications requesting funding  under track 1 and track 2 of the High Speed Intercity Passenger Rail  Program (HSIPR) in August 2009 and October 2009 and awarded Recovery  Act funding for projects in January 2010. We did not review FRA’s  rationale for its decision in December 2010 to redistribute $1.195 billion  from two projects in Ohio and Wisconsin to on-going high speed rail  projects in 13 states.
We also assessed whether FRA’s approach to calculating reviewers’  individual scores and compiling them for an overall panel score reflected  the criteria and weights for each criterion as published in the funding  announcement as well as the overall reliability of the data used to make  these calculations. To do this, we reviewed documentation about the  system used to collect the information and spoke with officials  knowledgeable about the data. We found some inaccuracies in how FRA  calculated the technical review scores. Specifically, we found that some  standardized scores were incorrect due to the inclusion of three duplicate  records and three applications deemed not yet ready or ineligible. In  addition, we noted FRA incorrectly weighted some technical evaluation  scores for applications submitted under track 1b. However, we determined  that these errors would not materially affect our findings and for the  purposes of examining the effect of the scores on application selection, we  found the data to be sufficiently reliable. FRA officials said that they  would correct their calculations for future rounds of rail funding.
Further we performed tests to determine the variables (e.g., technical  review scores and number of applications submitted) that had a significant  statistical relationship with being selected for an award. Our approach is  described in app. IV.
section_title: Following Recommended Practices for Discretionary Grant Awards
paragraphs: To determine the extent to which FRA used recommended practices for  awarding discretionary grants, we examined Office of Management and  Budget guidance, guidance from several federal agencies, and our reports  on this issue. (See table 7.) We identified key grant practices  recommended across executive branch agencies and compared them to  practices analyzed in our prior work. Specifically, we identified six  recommended practices relating to (1) communicating with potential  applicants prior to the competition, (2) planning for administering the  review of applications, (3) developing a technical review panel with  certain characteristics, (4) assessing applicants’ abilities to manage grant  funds, (5) notifying applicants of decisions, and (6) documenting reasons  for award decisions. We compared these practices to information from the  2009 funding announcement, guidance to applicant reviewers, and to  statements made by FRA officials regarding their implementation of their  grants award program. For this effort, one analyst carried out the  comparison and a second analyst verified the comparison results. Where  differences existed, the two analysts discussed them and reached  agreement. We also discussed the extent of FRA’s use of several of these  practices with the officials from our sample of 10 states.
section_title: Communication of Selection Results
paragraphs: To determine the extent FRA publicly communicated information about  the results of its award competition, we compared the information it  communicated to the public about its awards to the types of information  communicated by a random sample of 20 other competitively awarded  Recovery Act programs. (See table 8.) We selected the sample from 193  Recovery Act programs identified in the Catalog of Federal Domestic  Assistance as competitive grant programs using Recovery Act funds. In  addition, we compared the information communicated about FRA’s  awards to the information communicated by the Innovation Grants  program (Race to the Top)—a discretionary grant program run by the  Department of Education. We included the Race to the Top program  because you expressed interest in it.
We first reviewed materials on FRA’s Web site and other public releases,  such as press releases and outreach presentations to determine what FRA  publicly communicated. We then discussed these results with FRA  officials to confirm our results.
For each of the 21 other Recovery Act programs, we reviewed three public  information sources: (1) the program’s Catalog of Federal Domestic  Assistance award announcement, (2) internet search results, and (3)  Grants.gov, which provides information on more than 1,000 grant  programs. For each program, we searched these sources for information  about final award results (project description, why the project was  selected, and award amount) and for information that demonstrated how  applications fared at different states of the process (eligibility  determination and internal reviews, such as technical review panels). We  defined the results of any technical review as either scores or comments,  and when at least one of these elements was listed in at least one of the  three sources of information, we concluded that technical review  information was publicly communicated about the program. In carrying  out this assessment, one analyst carried out the work and a second analyst  independently performed the same tasks. The two analysts then compared  their results and resolved any differences. The results of our comparison  to a sample of other Recovery Act programs are not generalizable across  all Recovery Act programs.
section_title: Alignment with Statutory and Other Goals
paragraphs: To determine the extent to which HSIPR applications align with statutory  and other goals, we reviewed federal laws, including the Recovery Act and  PRIIA. We analyzed FRA’s Federal Register notice describing its approach  for selecting applications, its strategic vision for high speed rail, and its  preliminary national rail plan and its subsequent update to gather  information on any goals the agency has established for high speed rail  networks and conventional service and the types of projects it seeks to  support. We did not assess whether the applications selected by FRA will  achieve the stated benefits or costs.
We reviewed information submitted by applicants, namely the type of  project proposed, the funding requested and awarded, and the estimated  future speed of the projects. We used this data to sort projects into three  categories developed by FRA: core express corridors, regional corridors,  and emerging high speed rail routes. FRA’s definition of top speeds within  these categories overlap, which we modified slightly to provide discrete  endpoints.
Of the 62 applications selected by FRA, 13 did not provide data on  anticipated top speed after project completion. These 13 applications  include a variety of improvements, including station rehabilitations, the  reconfiguration of rolling stock, and existing tracks and grade crossings  upgrades for which one would not expect top speed information. We used  these data as background on selected applications and did not assess them  for reliability. We also reviewed FRA’s Recovery Act plans and compared  FRA goals for obligating and spending awarded funds to its actual rates of  obligating and spending from January 2010 through December 2010. After  reviewing a Department of Transportation Inspector General audit report  on its financial management system and speaking with department  officials familiar with the system, we determined that these data were  sufficiently reliable.
section_title: Appendix III: Difference Between the Amounts Requested and Estimated Awards by State
paragraphs: In January 2010, FRA proposed to provide 18 of the 24 states, including the  District of Columbia, selected for awards all or nearly all (91 percent or  more) of the money that they requested. (See table 9.) The agency  proposed to provide one state, North Carolina, with slightly more than it  requested and the remaining five states with amounts varying from 47  percent to 86 percent of the amounts requested.
In December 2010, nearly a year after making these proposals, FRA  announced that $1.195 billion in Recovery Act funds for high speed rail— representing most of the $810 million for Wisconsin’s Milwaukee-Madison  corridor and $385 million for Ohio’s Cincinnati-Columbus-Cleveland “3C”  route, originally designated for these states in January 2010—would be  redirected to high speed rail projects already underway in 13 other states.  In making these changes, FRA noted that Wisconsin has suspended work  under its existing high speed rail agreement and the incoming governors in  Wisconsin and Ohio have both indicated that they will not move forward  to use high speed rail money received under Recovery Act. The adjusted  amounts resulted in FRA proposing to provide all or nearly all of the  original request amounts (91 percent or more) for one additional state  (Oregon). (See table 10.) While most of the funding was redistributed to  three states (California, Florida, and Washington), the total funding  awarded to these three states was less than 80 percent of their original  requests.
section_title: Appendix IV: Additional Results from Our Statistical Analysis of Award Decisions
paragraphs: This appendix contains information related to our statistical analyses of  FRA and application data to examine possible relationships between  several variables and FRA’s selection decisions.
subsections: section_title: Overview of the Data
paragraphs: We obtained the data for our analysis from the Application Review Module  of GrantSolutions, the database FRA used to store application information  and technical review scores. Our analysis examined all 206 out of 259  submitted applications which FRA deemed eligible and ready to receive  federal funds. Eligible applications included those requesting Recovery  Act funds, tracks 1 and 2, as well as those requesting annual  appropriations, tracks 3 and 4. We included track 3 and 4 applications in  our analysis because FRA reviewed, weighted, and calculated the results  for tracks 1, 3, and 4 applications as a group rather than by distinct tracks.
To assess the reliability of the data in Application Review Module, we  reviewed database user manuals, spoke with officials knowledgeable  about the data, and conducted a series of data tests. We found some  inaccuracies in how FRA calculated the technical review scores.  Specifically, we found that some final technical review scores were  incorrect due to the inclusion of three duplicate technical review scores  and three applications later determined to be not yet ready or ineligible.  In addition, FRA had mistakenly applied incorrect weights to the track 1b  application technical review scores, which resulted in 15 final scores that  were one point higher than they should have been and another 5 final  scores that were one point lower than they should have been. We  determined that these errors would not materially affect our findings and  for the purposes of examining the effect of scores on application selection,  found these data are sufficiently reliable.
section_title: Methodology
paragraphs: To determine the extent to which specific variables were related to the  department’s selection of applications, we considered a set of bivariate  tables and conducted a series of bivariate and multivariate regression  analyses. From the tables and regression analyses we were interested in  determining how the department’s decision to select an application for an  award was affected by four variables: (1) the technical review scores, (2)  application track, (3) the requested funding amount, and (4) the number of  applications submitted by state or groups of states.
Our analyses provide us with estimates, called odds ratios, which indicate  the differences in the odds of applications being selected for an award  across certain categories of the different variables we examined. An odds  ratio of 1.0 would indicate that applications in different categories were  equally likely to be selected for an award. An odds ratio of less than 1.0  implies that applications in the category to which the odds ratio applies  were less likely to be selected relative to those they are being compared to  (known as the “reference” category). For example, if applications  receiving a technical review score of a 3 had an odds ratio of 0.5 it would  indicate that they were half as likely to be selected for an award as  applications that received a score of 1 or 2 (the reference category).  Inversely, an odds ratio greater than 1.0 suggests that applications with  that characteristic were more likely to be selected. For example, if  applications receiving a technical review score of a 5 had an odds ratio of  3.0, we would conclude that applications receiving that score were three  times more likely to be selected relative to the reference category. The  primary reason for preferring odds ratios to describe the relationships  across variables is because the significance of the differences between  specific odds ratios can be easily tested and the ratios can be re-estimated  after considering other variables.
section_title: Technical Review Score Affected the Selection of Applications
paragraphs: We first examined the effect of technical review scores on the likelihood  of being selected for an award, and found that the odds of being selected  for an award were in general greater for applications receiving higher  technical review scores than for applications receiving lower ones. For the  purposes of our analyses we combined scores of 1 and 2, as there were  only five applications that had received a score of 1 and there was no  evidence that they were significantly different from applications that had  been assigned scores of 2, in terms of being selected for funding. A smaller  percentage of the applications that received scores of 1 or 2 were selected  for funding (16 percent selected) than applications that had received a  score of 3 (46 percent selected) or 4 (42 percent selected), and  applications that received a score of 5 had the highest percentage of being  selected for funding (69 percent selected). In addition, the odds ratios of  4.30, 3.67, and 11.57 indicate that applications receiving a higher technical  review score (3, 4, or 5, respectively) were at least three times more likely  to be selected for an award than those receiving a lower technical review  score (i.e., 1 or 2). (See table 11.)
We followed several steps to calculate the odds ratios of 4.30, 3.67, and  11.57. First, we derived the odds that applications with certain technical  review scores would be selected for an award. For example, to determine  the selection odds for applications receiving a score of 1 or 2, we divided  the number of applications receiving a score of 1 or 2 that were selected  by the number applications receiving those scores that were not selected.  Seven were selected for awards, whereas 36 were not; the resulting odds  (7/36) equal 0.19. This means that 19 applications receiving a score of 1 or  2 would be selected for an award for every 100 that were not. By  comparison, the odds on being selected for applications receiving a  technical review score of 3 were 41/49, or 0.84, which indicates that 84  applications receiving a score of 3 would be selected for an award for  every 100 that were not. The odds ratio comparing these two odds is  0.84/0.19 equal to 4.30. This odds ratio suggests that the odds of being  selected for an award are more than four times greater for applications  receiving a technical review score of 3 than for applications receiving a  score of 1 or 2.
section_title: Application Track Affected the Likelihood of Applications Being Selected for Award
paragraphs: We also found that there were sizable differences in the likelihood of  applications submitted under different tracks being selected for an award.  While only between one-quarter and one-third of the applications in tracks  1a and 1b were selected, 61 percent of track 2 applications were selected,  as were nearly three-fourths of the applications in tracks 3 and 4. These  differences are also apparent from looking at the odds and odds ratios in  table 12. The odds on being selected for funding were slightly lower (by a  factor of 0.77) for track 1b applications than for track 1a, but they were  more than three times greater for track 2 applications than for track 1a  applications, and more than five times greater for track 3 and track 4  applications than for track 1a applications. (See table 12.)
section_title: Amount Requested Had an Effect on Likelihood of Selection
paragraphs: There were also sizable differences in the likelihood of applications being  selected based on the amount requested. Slightly more than half of the  applications that requested less than $1 million were funded, as were  exactly half of the applications that requested $50 million or more. At the  same time, roughly 40 percent of applications requesting between $1  million and $10 million were funded, and less than one-fourth of the  applications requesting $10 to $50 million were funded. The odds and odds  ratios indicate that applications requesting the lowest amounts were the  most likely to be selected and that applications requesting the highest  amounts were almost as likely as those requesting the lowest amounts to  be selected. Applications requesting more than $1 million but less than $50  million were somewhat less likely to be selected than applications  requesting less than $1 million or more than $50 million. (See table 13.)
section_title: Number of Applications per State Had Sizable Effect on the Likelihood of Application Selection
paragraphs: We examined the differences of applications being selected by each state  and, ultimately, by the number of applications submitted in many of the  states. We first considered the number of applications selected for each of  the 34 states that submitted eligible applications. Of these 34, 4 states  submitted more than 10 applications, 22 states submitted 3 or fewer  applications, and 9 states submitted a single application. We found a  statistically significant relationship indicating there was a much greater  tendency for applications to be selected when they came from states in  which a maximum of three applications were submitted.
Given the low number of applications submitted by many of the states,  however, we could not control for all of the differences between states in  a multivariate analysis in which the effects of the other variables are  estimated simultaneously. Therefore, we combined the states with smaller  numbers of applications into two groups: one group contained states  which submitted one to three applications and the other group contained  states submitting four to nine applications. These groupings did not result  in the loss of any significant information with respect to differences in the  likelihood of applications being selected across states. The results of  these state groupings indicate that the percentage of applications selected  for funding from states with one to three applications (70 percent  selected) were considerably higher than the percentage of funded  applications from states with four to nine applications (40 percent  selected). In addition, those states that submitted more than nine  applications showed considerable differences in the percent of  applications selected. California (37 percent selected) and Missouri (75  percent selected) had a relatively high percentage of applications selected,  and New York (18 percent selected) and Washington state (5 percent  selected) had a relatively low percentages of applications selected.
As in the previous tables, the odds and odds ratios give us the same sense  of the association that the percentages reveal; the odds on being selected  for funding were more than three times higher for applications from states  submitting one to three applications than for applications from California  or from states submitting four to nine applications (2.33/0.68, which equals  3.43). In addition, New York and Washington state were much less likely to  likely to have an application selected than California and Missouri. (See  table 14.)
section_title: Regression Analyses Show that Technical Review Score and Number of Applications Submitted Significantly Affected the Likelihood of Selection
paragraphs: We also examined the data using bivariate and multivariate logistic  regression models to estimate the effects of these different variables on  the likelihood of applications being selected for funding. The bivariate  models examine the effects of the technical review score, track, amount  requested, and state or group of states from which the application arose,  one at a time. The odds ratios from these models are the same as those  produced from the observed frequencies in the different two-way tables  described above. From these models, however, we obtain specific tests of  the significance of the differences between each variable category to  determine more generally whether there are differences between any of  the categories. In the bivariate regression model we find that many, but  not all, of the odds ratios describing these differences are significant. (See  table 15.)
The multivariate model estimates the net effects of these different  variables on the likelihood of applications being selected for funding, or  the effects of each variable when the effects of other variables are  considered simultaneously, rather than one at a time. Our results indicate  that the differences between tracks and amount requested categories are  rendered insignificant when technical review scores and state and state  group are taken into account, while the effect of technical review score  and the differences between state and state group remain sizable and in  most cases significant. Specifically, we found that, when we accounted for  all four variables, applications receiving a technical review score of 3, 4, or  5 were about seven to eight times more likely to be selected for funding  than applications which scored 1 or 2. In addition, applications from states  that submitted one to three applications, and applications from Missouri,  were three and nine times as likely, respectively, to be selected as those  from California, while those from Washington state were less than one- tenth as likely as those from California to be selected. The remaining  variable categories were not significant in the multivariate model and,  therefore, do not provide a statistical explanation for why applications  were more or less likely to be selected for an award.