Intercity Passenger Rail: Recording Clearer Reasons for Awards Decisions Would Improve Otherwise Good Grantmaking Practices
Released: 2011-04-11T00:00:00

Letter

Background

FRA is the primary federal agency responsible for overseeing safety in the railroad industry, as well as for distributing federal funds for intercity passenger rail service. FRA also administers federal operating and capital grants to the National Railroad Passenger Corporation (known as Amtrak), which have averaged between $1 billion and $1.3 billion per year since fiscal year 2003. FRA also approves Railroad Rehabilitation and Improvement Financing loans and Rail Line Relocation and Improvement Capital grants, and is the granting agency for the $120 million in fiscal year 2008 and fiscal year 2009 capital funds for intercity passenger rail projects.

Recent legislation has vastly increased the federal role in and federal funds for developing intercity passenger rail service. The Passenger Rail Investment and Improvement Act of 2008 (PRIIA), enacted in October 2008, authorized more than $3.7 billion for three different federal programs for high speed rail, intercity passenger rail, and congestion reduction grants. PRIIA also called for FRA to create a preliminary national rail plan within 1 year after passage of the act as well as a long- range national rail plan that promotes an integrated and efficient national rail system. FRA released a preliminary national rail plan in October 2009 and a subsequent progress report in September 2010.

The Recovery Act, enacted in February 2009, appropriated $8 billion for the three PRIIA-established intercity passenger rail programs. Unlike PRIIA, which authorized an 80 percent federal share for a project’s capital costs, the Recovery Act provided up to 100 percent federal funding available for obligation through fiscal year 2012 and expenditure through fiscal year 2017. The Recovery Act required that the department develop a strategic plan to use these funds. In April 2009, FRA released its strategic plan for developing high speed rail in America and distributing federal funds.

PRIIA and the Recovery Act created new responsibilities for FRA to plan, award, and oversee the use of new federal funds for intercity passenger rail. In response, FRA launched the high speed intercity passenger rail (HSIPR) program in June 2009 by issuing a funding announcement and interim guidance, which outlined the requirements and procedures for obtaining federal funds. FRA further outlined the vision and goals of the program through a number of outreach events and meetings, including seven regional workshops and more than 25 one-on-one site visits and conference calls with potential state applicants. States expressed a great deal of enthusiasm for the new program, requesting $102 billion across 278 preapplications, which FRA used to gauge initial interest and anticipate its staffing needs to manage the program. States, including the District of Columbia, ultimately submitted 229 applications for $57.8 billion in Recovery Act funds.

FRA asked applicants to submit Recovery Act project applications under three tracks: 1a, 1b, and 2. Track 1 was intended to primarily address economic recovery goals, and could either focus on ready-to-go projects (track 1a) or the completion of environmental and preliminary engineering requirements necessary to prepare projects for future funding (track 1b). Track 2 focused on much larger, long-term projects to develop new high speed rail services or substantially upgrade existing corridor service. While track 1 and track 2 applications were submitted and reviewed at different times, FRA used a similar approach to assess them, and applied the same criteria during three independent steps: eligibility determination, technical review, and selection. (See fig. 1.)

The eligibility determination was conducted by panels of officials experienced in environmental requirements or passenger and commuter rail. These officials used worksheets to aid in assessing application completeness and determining whether applicants and proposed projects were eligible to receive funds. Eligibility panels also made preliminary determinations as to whether applicants had substantially completed environmental requirements and whether the projects they submitted were ready to begin. For the most part, applications deemed not yet ready or ineligible were not forwarded for technical review; because the track 1 eligibility and technical review periods overlapped, there were two Recovery Act applications that received technical review scores and were later deemed not yet ready or ineligible and removed from award consideration.

The technical review was conducted by panels of officials with experience in several fields, such as passenger and commuter rail, grants management, and environmental requirements. The technical review differed slightly for track 1 applications, which were reviewed by 12 panels composed of three reviewers, and track 2 applications, which were reviewed by a single panel of eight reviewers. For both tracks, reviewers used guidebooks to assess applications against six technical review criteria: (1) transportation benefits, (2) economic recovery benefits, (3) other public benefits (e.g., environmental quality and energy efficiency), (4) project management approach, (5) sustainability of benefits, and (6) timeliness of project completion. (See table 1.) The guidebooks provided detailed descriptions of what was included within each of these criterion, as well as step-by-step instructions on reviewing applications that included a suggested scoring method using a scale from one (lowest) to five (highest). For example, the track 2 guidebook suggested applications that included more than one major weakness, were nonresponsive, or failed to address a particular criterion be given a technical review score of a one for that criterion. Applications that technical panelists determined were responsive, and included major and minor strengths and no major or very few minor weaknesses in a particular criterion, were to be given a technical review score of a five for that criterion. After completing an individual evaluation of each application, reviewers convened within their panel to discuss their overall thoughts on the application and technical review scores for each criterion, which they could revise based on input from other panelists. To arrive at a final score for each application, FRA officials used a formula that averaged individual scores and weighted the scores based on established priorities identified in the funding announcement. In addition, program officials standardized track 1 application scores to correct for potential inconsistencies across review panels.

After the technical review, senior department and FRA officials—Deputy Secretary, Under Secretary for Policy, FRA Administrator, and FRA Deputy Administrator, among others—selected projects to recommend to the Secretary of Transportation. They considered the technical review scores along with four additional pre-established selection criteria identified in the funding announcement: (1) region and location, (2) innovation, (3) partnerships, and (4) track type and funding round. (See table 2.) HSIPR program officials gave five briefings to senior officials on the results of the technical review and possible factors to consider in making award decisions, such as potential project cost, service speed, shared benefits, and readiness. Program officials also provided additional information, including funding scenarios, facts sheets on individual applications, and corridor maps upon request. According to FRA, senior officials considered this information when making their recommendations, but did not numerically score or rank applications.

On January 27, 2010, the FRA Administrator recommended 62 applications for funding and the Secretary of Transportation concurred with these recommendations. On January 28, 2010, DOT announced the selections. The selections were spread across several types of intercity passenger rail, including projects for emerging high speed rail (operating at speeds up to 90 miles per hour), regional corridors (operating at speeds between 90 and 124 miles per hour), and core express corridors (operating at speeds between 125 and 250 miles per hour or more). For example, the department selected one project to receive $35 million to rehabilitate track and provide service from Portland to Brunswick, Maine at speeds up to 70 miles per hour. Another project was to receive more than $50 million to construct 11 miles of dedicated passenger rail track near Rochester, New York, which will allow for service speeds up to 110 miles per hour. A third project was selected to receive almost $2.3 billion to initiate the first part of the California’s high speed rail system, which will allow for more than 200 miles per hour service between Los Angeles, San Francisco and the Central Valley, and eventually, San Diego. These selections were consistent with the criteria in PRIIA, the Recovery Act, and FRA’s strategic plan, which included broad goals that gave FRA discretion in developing a national passenger rail system. (Additional information on the legislative and program goals, and how the selected projects fit into them, is contained in app. I.)

FRA Applied Its Established Criteria to Determine Eligibility and Assess Technical Merit, but Selection Rationales Were Typically Too Vague to Assess

FRA applied its established criteria to determine eligibility and assess applications’ technical merit. However, its rationales for selecting projects were typically too general to determine how it applied the additional selection criteria. When asked for more information on certain applications, FRA provided specific reasons for its selection decisions, but, in our opinion, creating a detailed, comprehensive record alongside the final selections is preferable. Officials reported that they used the technical review scores as a starting point from which to apply each of the four selection criteria, which is partially supported by our analysis of FRA data. For example, we found that applications receiving a higher technical review score were about seven to eight times more likely to be selected for an award compared to those receiving a lower technical review score.

FRA Applied Its Established Criteria to Determine Eligibility and Assess Technical Merit

We found that FRA applied eligibility criteria established in its funding announcement when determining whether applications were eligible. Specifically, eligibility criteria listed in the funding announcement aligned with criteria outlined in the worksheets used by the panelists to verify that applications were eligible. Panelists were given separate worksheets to conduct the track 1 and track 2 eligibility reviews, and each of these worksheets included eligibility criteria listed in the funding announcement. For instance, as outlined in the funding announcement, the track 1 worksheet required eligibility panelists to indicate if the application was submitted on-time, by an eligible applicant, and with all of the required supporting documents. Similarly, the track 2 worksheet included questions regarding applicant eligibility, qualifications, and construction grant prerequisites which aligned with the eligibility criteria listed in the funding announcement.

FRA also applied the established technical review criteria communicated in its funding announcement by including these criteria in the guidebooks provided to technical panelists to assess the technical merits of each application. Specifically, the guidebooks FRA provided to panelists for reviewing track 1a, 1b, and 2 applications were divided into six sections that aligned with each of the six technical review criteria listed in the funding announcement. Moreover, the criteria within these sections of the guidebook often matched the criteria in the funding announcement very closely and, in some cases, word-for-word. For example, the funding announcement stated that an applicant’s experience administering similar projects would be considered under the technical review criteria of project management approach, which was included word-for-word in the project management approach section of the guidebooks. We spoke to at least one representative from each technical review panel; these representatives confirmed that panelists used the criteria listed in the guidebooks and did not use other criteria during their evaluation of individual applications.

Officials Reported Using Technical Review Panel Results and Selection Criteria to Make Awards Recommendations; Decision Rationales Provided Little Insight into Selections

Senior department and FRA officials recommended to the Secretary of Transportation applications to receive awards and the proposed amounts of the awards. When deciding which applications to recommend for awards, senior FRA officials told us that they used the results of the technical review panels and the four selection criteria. These four criteria were described in FRA’s June 2009 funding announcement as: (1) region and location (e.g., ensuring geographic balance, integration into the nationwide transportation network, and assistance to economically distressed regions), (2) innovation (e.g., pursuing new technology with a favorable public return, promoting domestic manufacturing, and developing human capital capacity for sustainable rail development), (3) partnerships (e.g., multi-state planning and investment and workforce diversity), and (4) tracks and round timing (e.g., longer-term track 2 corridor development balanced with ready-to-go track 1 investments). For example, officials stated that they used the innovation criterion to select applications with higher proposed speeds of service. In particular, senior officials reported using this criterion to reinforce the selection of the California and Florida intercity passenger rail projects, which were the only eligible projects with the potential for service above 150 miles per hour. In another example, officials reported that they applied the partnership criterion by assessing applicants’ track record with implementing large transportation projects as well as demonstrated relationships with key stakeholders, such as private railroads.

Senior FRA officials stated they developed their funding amount recommendations based on their professional judgment and national high speed intercity passenger rail program goals. Officials told us they accounted for the risks related to the total cost of the project during selection discussions and weighed them against the overall policy goals of developing a national high speed passenger rail network. Officials also stated that they used their professional judgment about rail systems to recommend the award amounts for each application, paying particular attention to the amounts distributed to the large, track 2 projects, and that they are continuing to assess the effect of changes to the requested funding amounts during the scope of work negotiations with awardees.

According to FRA, its rationales for selecting applications are recorded in a recommendation from the FRA Administrator to the Secretary of Transportation and in a memorandum from the Secretary to the Administrator concurring on the recommendations and specifying potential funding amounts. The rationales stated in these memorandums were typically vague, such as “aligns well with FRA’s published evaluation criteria” and “will result in significant transportation benefits preserve and create jobs.” These rationales most often restated the criteria listed in the funding announcement generally (e.g., result in significant transportation benefits) rather than providing insight into why the department viewed projects as meritorious.

In addition, the memorandums did not provide any information on why other applications were not recommended for selection, which prevents us from assessing how the department viewed the merits of successful applications over unsuccessful ones. For example, we found several instances in which, without documentation, it was difficult to determine the reasons why some projects were selected and others were not. Specifically, FRA decided not to select six track 1a applications from New York that received higher technical review panel scores, and selected a lower scoring track 1a application from the same applicant. FRA officials subsequently told us that the lower scoring application was selected for a number of reasons, including improving the reliability of the passenger trains on the rail line, ensuring that the project will become part of the infrastructure of any significant improvements to passenger rail service west of Albany, and improving the fluidity of both passenger and freight rail operations on this heavily used rail route. Similarly, FRA selected a lower scoring track 1a application from Illinois, but not a relatively high scoring one. FRA officials subsequently told us that they selected this application because it is an essential part of a long-standing program of projects to improve the fluidity of rail traffic in the highly congested Chicago area. FRA officials also told us that the scope of the relatively high scoring track 1a application was included in Illinois’ selected track 2 application. This level of information, which provides some insight into the merits of projects, was not included in the department’s record of its decisions.

In addition to the memorandums, FRA posted descriptions on its Web site of the selected projects, their expected benefits, and prospective award amounts. However, these descriptions are not particularly useful in understanding why these projects were selected because the cited benefits—such as reducing travel times, increasing travel speed and ridership, providing attractive transportation alternatives, and creating jobs—were supposed to be integral to all projects. For example, FRA’s Web site describes one project as increasing on-time performance and ultimately allowing speeds of up to 110 miles per hour on its segment, but does not give any indication why this project was meritorious. Other descriptions were similar.

FRA also sent letters to individual applicants regarding its decision, and, if the application was not selected, a brief explanation as to why it was not selected. For example, a number of these letters explained that applications were not selected because they did not meet a prerequisite, had application materials that did not provide sufficient support for the proposed activities, or did not submit all application materials necessary to adequately evaluate the project. However, these letters did not provide further details on how the proposed projects did not meet the prerequisite, how the application materials were insufficient, or which application materials were not received. Other decision letters provided applicants with similarly broad explanations. FRA officials also told us that they called all applicants, as well as their state secretaries of transportation and state governors, to inform them of FRA’s decisions. Several of the states that we contacted reported that the primary purpose of these calls was for FRA to provide feedback on their individual projects and, when requested, give explanations for why projects were not selected. While applicants stated that this information will be helpful during future application rounds, there is no required written record of these conversations and, therefore, they do not provide others with insight on why selection decisions were made.

Documentation of agency activities is a key part of accountability for decisions. The department has a financial assistance guidance manual to assist agencies with administering awards competitions and which FRA officials told us that they used to develop the competition framework. The manual recommends that all discretionary project selections, such as the intercity passenger awards, include an explanation of how the projects were selected based on the established funding priorities, but does not lay out expectations for the level of explanation. In particular, the manual recommends that officials document decisions if projects with the highest priority are not funded.

While the department documented its decisions, as required by its financial assistance manual guidance, the absence of an insightful internal record of the reasons behind award recommendations, and the final selections where they differ, can give rise to challenges to the integrity of the decisions made. While FRA was able to provide us with specific reasons on a case-by-case basis for why projects were selected, almost a year after these decisions were made, we believe creating a sufficiently detailed record has increased relevance in high-stakes, high-profile decisions, such as the intercity passenger rail awards competition in which there are vocal critics and ardent supporters of the program.

Similar arguments apply for creating an internal record for amounts recommended for awards. FRA officials understood that the available Recovery Act funds were not sufficient to fully fund a number of the projects and sought to fund projects or portions of projects that could provide transportation benefits if no additional federal funds were available. For these decisions, FRA proposed awarding 10 states (including the District of Columbia) all (100 percent) of the funds they applied for, 8 states nearly all (91–99 percent) of the funds they applied for, 5 states some (47–86 percent) of the funds they applied for, and one state with slightly more (104 percent) than it applied for. (See fig. 2. See also app. III for dollar amounts associated with fig. 2.) The applicant notification letters did not offer an explanation for why FRA proposed award amounts that differed from requests, and applicants we spoke with did not report that FRA had provided such information to them. Given that infrastructure projects have an inclination for cost growth, developing a record that explains why the recommended costs are appropriate for the proposed project provides integrity to the final decisions.

The current economic climate has also increased the importance of providing an internal rationale for large differences between requested funds and proposed award amounts. Many states have faced large budget deficits in 2010 that will require them to make difficult budget decisions about the future use of state funds, particularly where the Recovery Act awards will not provide all the funding expected to be needed to complete a project. For example, as of June 2010, Florida had made $3 billion in budget cuts to close its budget deficit. For its high speed rail award, Florida is slated to receive less than half of what it said is needed to complete the proposed Tampa to Orlando High Speed Rail Express project. An official from the Florida Department of Transportation is hopeful that Florida will receive additional federal grants, but is unsure where the remaining funds will come from otherwise. Additionally, Washington state applied for 16 separate projects totaling $976 million and was selected to receive a composite award of $590 million. Washington state officials acknowledged that the award amount will not fund all 16 of the projects, and have since reduced the scope of the application to the 11 projects that could be completed with the awarded amount while still providing the maximum benefit to the corridor. FRA officials stated they awarded amounts that differed from those requested in applications as a result of their recognition that many of the projects were based on preliminary work that was not well-refined, and that states differed in their ability to accurately estimate costs. In contrast, North Carolina received 4 percent more funding than originally requested. According to FRA, the additional funding was allocated to North Carolina for possibly adding additional train frequencies for a Recovery Act project. While we recognize that FRA may have developed these proposed award amounts for good reasons, without a written record of the department’s rationale for these adjustments, after the fact reconstructions of funding amount decisions invite outside criticism of the decisions.

Applications with Higher Technical Review Scores Were Typically Chosen Over Those with Lower Scores

One of your interests was in how the results of technical review panels aligned with final award decisions. As discussed earlier, while FRA considered the technical review panels to be an important part of its decision making, they were not the sole basis for selecting projects. This was detailed in FRA’s funding announcement, which described how applications were first to be assessed against six technical review criteria and then final recommendations would be made using the technical review results and four selection criteria.

While the technical review panel evaluations alone were not meant to designate final selections, we found that of 179 eligible Recovery Act applications, senior management recommended 92 percent (57 of 62) of higher scoring applications for funding; that is they received review panel scores of 3 or higher out of 5 possible points. (See fig. 3.) Within these recommended applications, most received a technical review score of 3 or 4, and three of the five applications that received a technical review score of 5 were recommended for selection. One of the two applications that scored a 5 and was not selected for funding was included in a selected track 2 application.

In a few cases though, senior officials recommended applications that received a lower technical review score (i.e., a score of 2) because, according to FRA, they believed these projects included freight and commuter rail service partners that were willing to make cost contributions in line with their potential benefit share, were strategically important to other selected applications, or helped to achieve regional balance. These considerations were included in the four selection criteria senior department and FRA officials said they used to evaluate applications. For example, one of the two applications from Rhode Island requested $1.2 million to complete preliminary engineering and environmental reviews and received a lower overall technical review score, in part because technical reviewers did not believe the applicant had sufficiently quantified the transportation and economic recovery benefits. This application was later recommended for selection. According to FRA, senior officials recommended applications receiving lower technical review scores, such as this Rhode Island application, in part to achieve greater regional balance. Additionally, FRA indicated this particular application was one of the few applications proposed for the Northeast Corridor, which further supported the region/location selection criteria. In another instance, senior officials selected a track 2 application from California that requested $194 million for preliminary engineering and environmental requirements for a large corridor application that received a lower technical review panel score. According to FRA, senior officials recommended some applications receiving lower technical review scores due to the projects’ strategic importance to other selected applications. Officials stated that they recommended the track 2 California application because the completion of preliminary engineering and environmental requirements were necessary to move forward on several other large California projects also recommended for an award.

Officials also told us that some applications receiving a higher technical review score (i.e., 3, 4, or 5) were not selected in order to ensure regional balance, especially when an applicant had already been selected for other large awards. For example, a track 1a application from North Carolina received a higher technical review panel score due, in part, to the anticipated transportation benefits of increased ridership and on-time- performance, and the applicant’s estimates that the project would create more than 400 new jobs. Most of the projects that North Carolina applied for under this application were also included as part of a larger, intercity passenger rail application that was later recommended for selection, and the state was awarded an estimated total of $545 million for high and conventional speed rail projects. Department and senior FRA officials reported that higher evaluated applications were not selected if the proposed project was already included in larger selected projects, to avoid duplicative selections. Another example was FRA’s decision not to select a higher scoring track 1a application from Florida that requested $270 million to acquire 61 miles of right-of-way. This application was scored highly due in part to its immediate benefits and substantial contribution of state funds but, similar to North Carolina, Florida had already been awarded $1.25 billion for a separate large, track 2 corridor project. In addition, almost 90 percent of the applications that scored a 4 and were not selected were submitted by applicants that had either already received a large award or had submitted a relatively high number of applications.

To provide further insight into the attributes that were consistent with being selected for Recovery Act awards, we examined technical review score and application data using a statistical model and found that two out of four variables we included in our model, technical review scores and the number of applications submitted per state, were significantly related to the likelihood of an application being selected for an award. Applications with higher scores (i.e., scores of 3, 4, or 5) were about seven to eight times more likely to be selected than those with scores of 1 or 2. For example, an application receiving a technical review score of 5, the highest possible score, was more than nine times more likely to be selected for an award as an application receiving a technical review score of a 1 or 2. This analysis supports statements from senior department and FRA officials indicating that the technical review scores were largely the basis for their selection deliberations.

Additionally, we found that states submitting fewer applications (i.e., between one and three) were more than three times more likely to have their application selected than states submitting higher numbers of applications (i.e., between four and nine). This result suggests that selection officials attempted to spread the awards across different applicants, which is consistent with FRA’s reported efforts to attain geographic distribution. However, the results differed somewhat for the four states that submitted 10 or more applications. In this case two of the states had a lower likelihood of being selected for an award than states submitting fewer than 10 applications, while one state had a higher likelihood of being selected. One additional state had about the same likelihood of being selected as states submitting between four and nine applications. When asked about these differences across states, FRA officials said that the number of applications submitted did not affect their selection decisions.

FRA Substantially Met Recommended Practices for Awarding Discretionary Grants

We identified six recommended practices used across the federal government to ensure a fair and objective evaluation and selection of discretionary grant awards. These practices are based on policies and guidance used by the Office of Management and Budget and other federal agencies—including the department, and our work. FRA substantially followed these practices, including communicating key information to applicants, planning for the competition, using a technical merit review panel with desirable characteristics, assessing applicants’ ability to account for funds, and notifying applicants of awards decisions. (See table 3.) In our opinion, FRA partially met one recommended practice: documenting the rationale for funding decisions. As discussed previously, we believe it would have been beneficial to provide more detail about the rationales for these decisions. According to FRA officials, the methods they used to evaluate and select applications were based on best practices collected from several other federal government agencies, which we believe likely helped them meet a number of the recommended practices we identified.

Communicate with potential applicants prior to the competition. FRA issued a funding announcement that included information on the $8 billion in available funding, key dates, the competition rules, the funding priorities and relative importance for each one, and the types of projects FRA would consider for federal grants. Applicants we spoke with praised FRA’s communication and stated that FRA officials did a good job providing information and answering questions during the period leading up to the preapplication and application deadlines. For example, officials from several states indicated that FRA officials participated in biweekly conference calls, which were helpful in understanding the technical aspects of how to apply. Applicants also indicated that the outreach events, particularly the site visits, helped them refine their applications and ensure projects met program requirements.

Plan for administering the technical review. FRA developed two plans for determining technical merit: (1) the track 1 technical review used 12 panels each comprised of three reviewers and (2) the track 2 technical review used one panel of eight reviewers. Track 1 applications were randomly assigned across the panels, while the track 2 panel reviewed all of the eligible applications. FRA identified and asked for volunteers to participate in the technical reviews from within FRA and across several other agencies within the department. FRA officials also provided reviewers with guidebooks to document their application assessments and instructed them to input the results, including scores and comments, into a centralized database. FRA standardized final track 1 application scores to account for any unintentional differences in the way panels assessed and scored applications, but did not need to standardize track 2 scores because the review was conducted by a single panel. Finally, according to officials, FRA oversaw the review by examining technical review scores and comments, and conducting daily meetings with representatives from each panel to ensure panelists were consistently applying the criteria.

Develop a technical review panel with certain characteristics. FRA compiled technical review panels that included staff with background in several relevant fields, such as grants management, passenger and commuter rail, and environmental requirements, and made other knowledgeable staff available if panelists had questions. FRA officials stated that panelists were also required to sign or submit a previously completed conflict of interest form to attest to their independence. In addition, panelists were given guidebooks to assess applications that included the technical review criteria and were told by FRA program officials to apply only these criteria during their efforts. FRA also trained panelists during a 1-day orientation session.

Assess applicants’ capabilities to account for funds. FRA required applicants to provide information on their ability to account for funds. Specifically, applicants were asked to describe their experience, if any, managing rail investment projects. If applicants reported that they did not have experience on projects similar to the one they were proposing, FRA instead asked applicants to include a plan for building the capacity to manage the project. The application also required applicants to provide information on their financial management capability, including previous audit results, and the applicants’ ability to manage potential cost overruns and financial shortfalls. In addition, FRA required applicants to submit supplemental materials such as a detailed capital cost budget, which provided a breakdown of the activities included in each application and their anticipated cost. These pieces of information were assessed by FRA through an eligibility panel, to ensure the application was complete, and a technical review panel, to evaluate the applicants’ overall ability to manage the project.

Notify applicants of awards decisions. FRA officials provided each applicant with a letter indicating which applications were selected and a general reason why individual applications were not selected. While FRA did not include estimated award amounts in these notification letters, this information was made publicly available on the department’s Web site and distributed through a press release. In addition, most of the applicants we spoke with indicated that FRA provided informal feedback on applications via telephone calls shortly after the awards were announced. For example, an official from one applicant stated that FRA provided information on ways to improve applications that were not selected, which the applicant used when applying for funds in future rounds.

Document rationale for awards decisions. According to the guidance from the department, Department of Commerce, the Department of Education, and our work, agencies should document their rationale for award decisions. As stated previously, FRA documented how it applied the technical criteria for selected projects, and provided applicants with a general explanation for selecting or rejecting individual projects. However, as discussed in a previous section, in our view FRA typically did not clearly document specific reasons for selecting individual projects, reasons for not selecting other projects, or how changes made to requested funding amounts might affect applicants’ ability to achieve project goals.

According to FRA, officials used lessons from a number of other government programs when developing the method for evaluating and selecting projects. For example, one of the officials responsible for developing the funding announcement, technical review guidebooks, and the format of the technical review panels stated that he relied on his experience working with large transit grants to create a review that was both quantifiable and allowed for subjective professional judgment. In addition, this official noted that FRA examined the methods used by other agencies, such as the Department of Health and Human Services, the Department of Justice, and the Federal Transit Administration, to develop and implement a list of best practices for awarding discretionary grants.

FRA Publicly Communicated at Least as Much Outcome Information as Other Competitively Awarded Recovery Act Grant Programs

FRA publicly communicated outcome information, such as a list of awards and the award amounts, at a level similar to or greater than most other Recovery Act competitive grant programs that we examined. Specifically, FRA communicated information on award decisions to the public, but did not communicate the results of the technical review that had contributed to these decisions. Only one of the programs that we examined—the Department of Education’s State Innovation grants (known as Race to the Top)—publicly communicated the results of its technical review, which include technical scores and comments; however, this program used a much different approach for selecting awardees than the HSIPR program.

Members of Congress and the President have emphasized the need for accountability, efficiency, and transparency in the expenditure of Recovery Act funds and have made it a central principle of the act. However, the act did not define the attributes of transparency or how deeply an agency’s actions should be transparent. We also did not find any non-Recovery Act requirement or guidance instructing federal programs to publicly disclose the reasons for their selection decisions. To assess the extent to which FRA publicly communicated outcome information, we compared the HSIPR program to 21 other Recovery Act competitive grant programs, including Race to the Top. (See fig. 4.) We selected 20 of these programs randomly from a list of almost 200 competitively awarded grant programs that distributed Recovery Act funds. We included the 21st program, Race to the Top, because it was of interest to you.

FRA publicly communicated at least as much outcome information as all but one Recovery Act competitive grant programs we reviewed. Specifically, FRA publicly communicated through its Web site the selection decisions, including the amount of funds requested, general benefits from the project, and the potential award amounts for the 62 Recovery Act applications that it selected. It did not communicate the results of the technical review. Out of the other 21 competitively awarded Recovery Act programs we examined, 13 communicated selection information similar to FRA, including awards and award amounts, but not the results of the technical review. For example, the Department of Health and Human Services’ National Institutes of Health published a list of 21,581 award winners for nearly $9 billion, but, similar to the HSIPR program, did not report the results of the technical review. Eight other programs conveyed less information than FRA and did not publicly communicate the results of the technical review or the awards and award amounts.

Race to the Top was the only program we examined that publicly provided the results of its technical review. These results, which were posted on the Department of Education’s Web site, included scores and comments from reviewers for each applicant, but were not connected to individual reviewers by name. According to its Web site, the Department of Education decided to release this level of detailed information because the $4 billion Race to the Top program was larger than any other discretionary program the Department of Education had previously administered, and officials sought to ensure the highest level of integrity and transparency. Unlike the HSIPR program, however, Race to the Top used these scores as the sole basis for selecting awards and only chose applicants receiving the highest scores. As described in the previous section, the technical review scores were an important component for making HSIPR selection decisions, but did not include consideration of additional pre-established selection criteria designed to ensure long-term success and sustainability of the program. As such, publishing them without additional decision making information on the specific reasons for selecting and not selecting individual applications could lead to erroneous conclusions about FRA’s decisions.

According to FRA officials, the results of the technical review were not communicated because department officials were concerned that associating technical review scores and comments with a specific reviewer could discourage reviewers from participating in future department competitive grant evaluations. Furthermore, in their view, this might also prevent reviewers in future funding rounds from providing candid evaluations. However, as the Race to the Top program demonstrated, it would be possible for FRA to present overall technical panel review assessments or their individual comments without linking individuals’ names to comments, if it chooses to do so. FRA officials stated that the anonymous disclosure of technical scores and comments would still prevent FRA and department leadership and staff from frankly expressing their individual judgments, as they might still be concerned over how theses opinions would reflect on the FRA and HSIPR program if they were made public.

Conclusions

The $8 billion appropriated by the Recovery Act for the High Speed Intercity Passenger Rail program represents a large investment in the development of a national passenger rail network. FRA established a fair and objective approach for distributing these funds and substantially followed recommended discretionary grant award practices used throughout the government. The exception is what we view as incomplete documentation of why some applications were chosen and not others, and how FRA decided to distribute the funds at the time those decisions were made. This incomplete documentation is notable given the robust documentation of the other steps used to determine eligibility and assess technical merit. We believe that establishing a record that provides insight into why decisions were made, rather than merely restating general technical review and selection criteria, including amounts to be provided, would enhance the credibility of FRA’s awards decisions to the extent that this record confirms that selected projects aligned with established criteria and goals. By not establishing this record, FRA invites skepticism about the overall fairness of its decisions, even if they are sound, and hinders meaningful disclosure of how it made its decisions, if it chooses to do so.

Recommendation for Executive Action

To help ensure accountability over federal funds, we recommend that the Secretary of Transportation direct the Administrator of the Federal Railroad Administration to create additional records that document the rationales for award decisions in future HSIPR funding rounds, including substantive reasons (1) why individual projects are selected or not selected and (2) for changes made to requested funding amounts.

Agency Comments

We provided a draft of this report to the Department of Transportation for its review and comment. The department told us that it carefully constructed the grant processes for the HSIPR program based on extensive review and consideration of best practices both within and outside the agency with the intent of providing a comprehensive and transparent process. The department indicated that its overall intent was to select the best projects that offered the greatest available and achievable benefit to the nation. The department told us that it would carefully consider our recommendation to determine if there are means to further enhance the transparency of its grant selection process with additional documentation, without creating a process that is unduly burdensome to administer. The department also offered technical comments which we incorporated as appropriate.

As agreed with your office, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. We are sending copies of this report to congressional subcommittees with responsibilities for surface transportation issues; the Director, Office of Management and Budget; the Secretary of Transportation; and the Administrator of the Federal Railroad Administration. In addition, this report will be available at no charge on GAO’s Web site at http://www.gao.gov.

If you or your staff have any questions regarding this report, please contact me at (202) 512-2834 or flemings@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are Owen Bruce, Matthew Cook, Colin Fallon, Michele Fejfar, Maria Gaona, Grant Mallie, James Ratzenberger, Douglas Sloane, Matthew Voit, and Crystal Wesco.

Appendix I: Extent to Which Recovery Act Projects Align with Statutory and Other Goals

We examined the extent to which American Recovery and Reinvestment Act of 2009 (Recovery Act) projects selected by the Federal Railroad Administration (FRA) align with legislative and the administration’s goals to develop high speed and conventional rail networks.

Congress Provided Broad Goals with a Priority for High Speed Systems

Congress provided its most recent expectations for high and conventional speed rail in the Passenger Rail Investment and Improvement Act of 2008 (PRIIA) and the Recovery Act. In this regard, PRIIA speaks generally about supporting improvements to high and conventional speed rail and does not set out any expectations for relative attention to high and conventional speed passenger rail improvements. The Recovery Act appropriated $8 billion for both forms of rail service broadly. However, it required that FRA give priority to projects that support the development of intercity high speed service. Further, the act required that FRA develop a strategic plan that describes how FRA will use Recovery Act funding to improve and deploy high speed systems. FRA had wide latitude to achieve goals laid out in its strategic plan.

FRA’s Vision Describes Broad Goals for High Speed Rail, but Provides Limited Detail on How Goals Will Be Achieved

FRA has outlined its vision for developing intercity passenger rail service in its strategic plan, as required by the Recovery Act, and in both its preliminary national rail plan issued in 2009 and the plan’s update nearly a year later. FRA’s vision documents—the strategic plan and its updated national rail plan—described broad goals, such as for transportation, safety, and economic competitiveness, and established categories for the type of high speed rail projects it intends to support. For example, the strategic plan notes the high speed rail program aims to generate construction and operating jobs, while providing a steady market for various industries producing rail, control systems, locomotives, and passenger cars. In addition, the plan notes that investments in high speed rail can result in competitive trip times and rail transport can also result in higher-density development as compared to other modes of transportation. Similarly, the updated national rail plan sets a goal of connecting communities through high speed rail while, among other things, reducing congestion, boosting economic growth, and promoting economic sustainability.

However, these vision documents provide limited details on goals for the high speed rail program. For example, while the strategic plan emphasizes investments that will yield tangible benefits to rail performance and improve connections between different modes of transportation, it does not describe how and when FRA intends to realize these benefits. In addition, as we reported last June, the preliminary rail plan did not offer specific recommendations for future action and was designed to serve as a springboard for further discussion with states and freight railroads. While the update to this plan included improving rail performance as a goal and provided some measurements for high speed rail performance, such as competitive trip times, it did not provide any specific targets for these metrics, or any time line showing when FRA hopes to attain these improvements.

FRA’s Application Selection Is Consistent with the Recovery Act’s Priority for High Speed Rail Service

Consistent with the Recovery Act’s direction to give priority for high speed rail service, about half (45 percent) of the applications selected were for core express corridors (high speed service of 125–250 miles per hour or more) or regional corridors (higher-speed service of 90–124 miles per hour) using categories of service similar to those FRA established in its vision documents. (See table 4.) FRA did not establish specific targets for the number of each type of project it intended to support.

Selected Applications Reflect Short-Term Economic Recovery and Long-Term Infrastructure Investment Goals

In addition to providing priority for high speed projects, the applications that FRA selected were consistent with near-term economic recovery goals and its long-term development goals. While most selected projects are short-term in nature and are intended to support economic recovery goals established by the Recovery Act, most funding was provided to several long-term, high speed corridor projects. Specifically, we found that 48 of the 62 applications selected were track 1 applications, which are smaller projects designed to be completed within 2 years. (See table 5.) These projects represent about 11 percent of the funding provided for high speed rail and intercity passenger rail through the Recovery Act. The remaining 89 percent of funding was provided for 14 track 2 applications, which are primarily long-term, corridor projects. The funding allocation aligns with FRA’s focus on long-term investments that will support development of a high speed passenger rail network as described in the funding announcement.

While FRA announced in January 2010 awards of nearly $8 billion in grants for the program, many of these projects have only recently begun. As of December 31, 2010, FRA had obligated $4.2 billion, or about 54 percent of the funding awarded in January, and about $50 million has been spent for projects selected under track 2.

In May 2009, FRA issued a plan for spending Recovery Act funds, which it updated in July 2010. FRA missed its May 2009 targets for obligations and spending through 2010 estimate because it had planned to announce awards—and begin obligating funds—in the autumn of 2009. However, FRA did not make those announcements until January 2010 and did not begin to obligate funds until May 2010. FRA then revised its estimates in July 2010. FRA surpassed the calendar year 2010 goals for obligating and spending funds in the July 2010 plan. (See table 6.) During calendar year 2010, FRA obligated about 11 times as much as anticipated in the July 2010 plan, while awardees have spent about 7 times as much as planned over the same time period. The Recovery Act authorized obligation of funds through September 30, 2012, and FRA intends to obligate all funds by this date.

Passenger rail investments are often long-term efforts that must be carried out in partnership between the state and others, notably private railroads. For example, in order to begin design and construction on many of these projects, grant recipients must negotiate and secure agreements with private freight railroads to use their tracks for passenger rail trains. However, officials from these railroads are concerned that sharing tracks would create safety risks and liability concerns, prevent freight expansion, and cause rail congestion. Some of the states have experienced delays finalizing these agreements with the railroads and, accordingly, have not completed agreements with FRA to obligate awarded funding.

Appendix II: Scope and Methodology

Criteria Used to Select Projects

To determine the extent to which FRA applied its established criteria to select projects, we identified the criteria that it planned to use from its June 23, 2009, funding announcement outlining its evaluation and selection approach. We then compared these criteria to the worksheets and guidebooks that FRA used to determine eligibility and assess technical merit. Finally, we interviewed FRA officials who participated in evaluating and selecting projects to obtain information on whether and how they applied the established criteria. Specifically, we randomly selected 1 technical reviewer from each of the 12 track 1, 3, and 4 panels (12 out of 36 reviewers), and 6 of the 8 reviewers from the track 2 panel. In addition, we interviewed senior FRA officials to further understand how senior Department of Transportation (the department) and FRA officials applied the selection criteria, selected projects, and determined the amount of funding provided for each project. We also asked FRA officials to provide reasons for why several lower scored applications were selected, while other higher scored applications were not.

We conducted semi-structured interviews with officials from 10 of the 40 states and the District of Columbia that submitted a preapplication or an application for track 1 and track 2 funding about how FRA communicated its approach to reviewing applications and award results. We selected these states on the basis of four characteristics: (1) the extent to which applicants progressed through the preapplication, application, evaluation, and selection stages; (2) geographic regions; (3) the number of applications submitted; and (4) the amount of funding. We also contacted officials in two additional states (Ohio and Washington) to understand the effect of FRA’s funding decisions on the scope of these states’ proposed rail program. Our efforts were limited to applications requesting funding under track 1 and track 2 of the High Speed Intercity Passenger Rail Program (HSIPR) in August 2009 and October 2009 and awarded Recovery Act funding for projects in January 2010. We did not review FRA’s rationale for its decision in December 2010 to redistribute $1.195 billion from two projects in Ohio and Wisconsin to on-going high speed rail projects in 13 states.

We also assessed whether FRA’s approach to calculating reviewers’ individual scores and compiling them for an overall panel score reflected the criteria and weights for each criterion as published in the funding announcement as well as the overall reliability of the data used to make these calculations. To do this, we reviewed documentation about the system used to collect the information and spoke with officials knowledgeable about the data. We found some inaccuracies in how FRA calculated the technical review scores. Specifically, we found that some standardized scores were incorrect due to the inclusion of three duplicate records and three applications deemed not yet ready or ineligible. In addition, we noted FRA incorrectly weighted some technical evaluation scores for applications submitted under track 1b. However, we determined that these errors would not materially affect our findings and for the purposes of examining the effect of the scores on application selection, we found the data to be sufficiently reliable. FRA officials said that they would correct their calculations for future rounds of rail funding.

Further we performed tests to determine the variables (e.g., technical review scores and number of applications submitted) that had a significant statistical relationship with being selected for an award. Our approach is described in app. IV.

Following Recommended Practices for Discretionary Grant Awards

To determine the extent to which FRA used recommended practices for awarding discretionary grants, we examined Office of Management and Budget guidance, guidance from several federal agencies, and our reports on this issue. (See table 7.) We identified key grant practices recommended across executive branch agencies and compared them to practices analyzed in our prior work. Specifically, we identified six recommended practices relating to (1) communicating with potential applicants prior to the competition, (2) planning for administering the review of applications, (3) developing a technical review panel with certain characteristics, (4) assessing applicants’ abilities to manage grant funds, (5) notifying applicants of decisions, and (6) documenting reasons for award decisions. We compared these practices to information from the 2009 funding announcement, guidance to applicant reviewers, and to statements made by FRA officials regarding their implementation of their grants award program. For this effort, one analyst carried out the comparison and a second analyst verified the comparison results. Where differences existed, the two analysts discussed them and reached agreement. We also discussed the extent of FRA’s use of several of these practices with the officials from our sample of 10 states.

Communication of Selection Results

To determine the extent FRA publicly communicated information about the results of its award competition, we compared the information it communicated to the public about its awards to the types of information communicated by a random sample of 20 other competitively awarded Recovery Act programs. (See table 8.) We selected the sample from 193 Recovery Act programs identified in the Catalog of Federal Domestic Assistance as competitive grant programs using Recovery Act funds. In addition, we compared the information communicated about FRA’s awards to the information communicated by the Innovation Grants program (Race to the Top)—a discretionary grant program run by the Department of Education. We included the Race to the Top program because you expressed interest in it.

We first reviewed materials on FRA’s Web site and other public releases, such as press releases and outreach presentations to determine what FRA publicly communicated. We then discussed these results with FRA officials to confirm our results.

For each of the 21 other Recovery Act programs, we reviewed three public information sources: (1) the program’s Catalog of Federal Domestic Assistance award announcement, (2) internet search results, and (3) Grants.gov, which provides information on more than 1,000 grant programs. For each program, we searched these sources for information about final award results (project description, why the project was selected, and award amount) and for information that demonstrated how applications fared at different states of the process (eligibility determination and internal reviews, such as technical review panels). We defined the results of any technical review as either scores or comments, and when at least one of these elements was listed in at least one of the three sources of information, we concluded that technical review information was publicly communicated about the program. In carrying out this assessment, one analyst carried out the work and a second analyst independently performed the same tasks. The two analysts then compared their results and resolved any differences. The results of our comparison to a sample of other Recovery Act programs are not generalizable across all Recovery Act programs.

Alignment with Statutory and Other Goals

To determine the extent to which HSIPR applications align with statutory and other goals, we reviewed federal laws, including the Recovery Act and PRIIA. We analyzed FRA’s Federal Register notice describing its approach for selecting applications, its strategic vision for high speed rail, and its preliminary national rail plan and its subsequent update to gather information on any goals the agency has established for high speed rail networks and conventional service and the types of projects it seeks to support. We did not assess whether the applications selected by FRA will achieve the stated benefits or costs.

We reviewed information submitted by applicants, namely the type of project proposed, the funding requested and awarded, and the estimated future speed of the projects. We used this data to sort projects into three categories developed by FRA: core express corridors, regional corridors, and emerging high speed rail routes. FRA’s definition of top speeds within these categories overlap, which we modified slightly to provide discrete endpoints.

Of the 62 applications selected by FRA, 13 did not provide data on anticipated top speed after project completion. These 13 applications include a variety of improvements, including station rehabilitations, the reconfiguration of rolling stock, and existing tracks and grade crossings upgrades for which one would not expect top speed information. We used these data as background on selected applications and did not assess them for reliability. We also reviewed FRA’s Recovery Act plans and compared FRA goals for obligating and spending awarded funds to its actual rates of obligating and spending from January 2010 through December 2010. After reviewing a Department of Transportation Inspector General audit report on its financial management system and speaking with department officials familiar with the system, we determined that these data were sufficiently reliable.

Appendix III: Difference Between the Amounts Requested and Estimated Awards by State

In January 2010, FRA proposed to provide 18 of the 24 states, including the District of Columbia, selected for awards all or nearly all (91 percent or more) of the money that they requested. (See table 9.) The agency proposed to provide one state, North Carolina, with slightly more than it requested and the remaining five states with amounts varying from 47 percent to 86 percent of the amounts requested.

In December 2010, nearly a year after making these proposals, FRA announced that $1.195 billion in Recovery Act funds for high speed rail— representing most of the $810 million for Wisconsin’s Milwaukee-Madison corridor and $385 million for Ohio’s Cincinnati-Columbus-Cleveland “3C” route, originally designated for these states in January 2010—would be redirected to high speed rail projects already underway in 13 other states. In making these changes, FRA noted that Wisconsin has suspended work under its existing high speed rail agreement and the incoming governors in Wisconsin and Ohio have both indicated that they will not move forward to use high speed rail money received under Recovery Act. The adjusted amounts resulted in FRA proposing to provide all or nearly all of the original request amounts (91 percent or more) for one additional state (Oregon). (See table 10.) While most of the funding was redistributed to three states (California, Florida, and Washington), the total funding awarded to these three states was less than 80 percent of their original requests.

Appendix IV: Additional Results from Our Statistical Analysis of Award Decisions

This appendix contains information related to our statistical analyses of FRA and application data to examine possible relationships between several variables and FRA’s selection decisions.

Overview of the Data

We obtained the data for our analysis from the Application Review Module of GrantSolutions, the database FRA used to store application information and technical review scores. Our analysis examined all 206 out of 259 submitted applications which FRA deemed eligible and ready to receive federal funds. Eligible applications included those requesting Recovery Act funds, tracks 1 and 2, as well as those requesting annual appropriations, tracks 3 and 4. We included track 3 and 4 applications in our analysis because FRA reviewed, weighted, and calculated the results for tracks 1, 3, and 4 applications as a group rather than by distinct tracks.

To assess the reliability of the data in Application Review Module, we reviewed database user manuals, spoke with officials knowledgeable about the data, and conducted a series of data tests. We found some inaccuracies in how FRA calculated the technical review scores. Specifically, we found that some final technical review scores were incorrect due to the inclusion of three duplicate technical review scores and three applications later determined to be not yet ready or ineligible. In addition, FRA had mistakenly applied incorrect weights to the track 1b application technical review scores, which resulted in 15 final scores that were one point higher than they should have been and another 5 final scores that were one point lower than they should have been. We determined that these errors would not materially affect our findings and for the purposes of examining the effect of scores on application selection, found these data are sufficiently reliable.

Methodology

To determine the extent to which specific variables were related to the department’s selection of applications, we considered a set of bivariate tables and conducted a series of bivariate and multivariate regression analyses. From the tables and regression analyses we were interested in determining how the department’s decision to select an application for an award was affected by four variables: (1) the technical review scores, (2) application track, (3) the requested funding amount, and (4) the number of applications submitted by state or groups of states.

Our analyses provide us with estimates, called odds ratios, which indicate the differences in the odds of applications being selected for an award across certain categories of the different variables we examined. An odds ratio of 1.0 would indicate that applications in different categories were equally likely to be selected for an award. An odds ratio of less than 1.0 implies that applications in the category to which the odds ratio applies were less likely to be selected relative to those they are being compared to (known as the “reference” category). For example, if applications receiving a technical review score of a 3 had an odds ratio of 0.5 it would indicate that they were half as likely to be selected for an award as applications that received a score of 1 or 2 (the reference category). Inversely, an odds ratio greater than 1.0 suggests that applications with that characteristic were more likely to be selected. For example, if applications receiving a technical review score of a 5 had an odds ratio of 3.0, we would conclude that applications receiving that score were three times more likely to be selected relative to the reference category. The primary reason for preferring odds ratios to describe the relationships across variables is because the significance of the differences between specific odds ratios can be easily tested and the ratios can be re-estimated after considering other variables.

Technical Review Score Affected the Selection of Applications

We first examined the effect of technical review scores on the likelihood of being selected for an award, and found that the odds of being selected for an award were in general greater for applications receiving higher technical review scores than for applications receiving lower ones. For the purposes of our analyses we combined scores of 1 and 2, as there were only five applications that had received a score of 1 and there was no evidence that they were significantly different from applications that had been assigned scores of 2, in terms of being selected for funding. A smaller percentage of the applications that received scores of 1 or 2 were selected for funding (16 percent selected) than applications that had received a score of 3 (46 percent selected) or 4 (42 percent selected), and applications that received a score of 5 had the highest percentage of being selected for funding (69 percent selected). In addition, the odds ratios of 4.30, 3.67, and 11.57 indicate that applications receiving a higher technical review score (3, 4, or 5, respectively) were at least three times more likely to be selected for an award than those receiving a lower technical review score (i.e., 1 or 2). (See table 11.)

We followed several steps to calculate the odds ratios of 4.30, 3.67, and 11.57. First, we derived the odds that applications with certain technical review scores would be selected for an award. For example, to determine the selection odds for applications receiving a score of 1 or 2, we divided the number of applications receiving a score of 1 or 2 that were selected by the number applications receiving those scores that were not selected. Seven were selected for awards, whereas 36 were not; the resulting odds (7/36) equal 0.19. This means that 19 applications receiving a score of 1 or 2 would be selected for an award for every 100 that were not. By comparison, the odds on being selected for applications receiving a technical review score of 3 were 41/49, or 0.84, which indicates that 84 applications receiving a score of 3 would be selected for an award for every 100 that were not. The odds ratio comparing these two odds is 0.84/0.19 equal to 4.30. This odds ratio suggests that the odds of being selected for an award are more than four times greater for applications receiving a technical review score of 3 than for applications receiving a score of 1 or 2.

Application Track Affected the Likelihood of Applications Being Selected for Award

We also found that there were sizable differences in the likelihood of applications submitted under different tracks being selected for an award. While only between one-quarter and one-third of the applications in tracks 1a and 1b were selected, 61 percent of track 2 applications were selected, as were nearly three-fourths of the applications in tracks 3 and 4. These differences are also apparent from looking at the odds and odds ratios in table 12. The odds on being selected for funding were slightly lower (by a factor of 0.77) for track 1b applications than for track 1a, but they were more than three times greater for track 2 applications than for track 1a applications, and more than five times greater for track 3 and track 4 applications than for track 1a applications. (See table 12.)

Amount Requested Had an Effect on Likelihood of Selection

There were also sizable differences in the likelihood of applications being selected based on the amount requested. Slightly more than half of the applications that requested less than $1 million were funded, as were exactly half of the applications that requested $50 million or more. At the same time, roughly 40 percent of applications requesting between $1 million and $10 million were funded, and less than one-fourth of the applications requesting $10 to $50 million were funded. The odds and odds ratios indicate that applications requesting the lowest amounts were the most likely to be selected and that applications requesting the highest amounts were almost as likely as those requesting the lowest amounts to be selected. Applications requesting more than $1 million but less than $50 million were somewhat less likely to be selected than applications requesting less than $1 million or more than $50 million. (See table 13.)

Number of Applications per State Had Sizable Effect on the Likelihood of Application Selection

We examined the differences of applications being selected by each state and, ultimately, by the number of applications submitted in many of the states. We first considered the number of applications selected for each of the 34 states that submitted eligible applications. Of these 34, 4 states submitted more than 10 applications, 22 states submitted 3 or fewer applications, and 9 states submitted a single application. We found a statistically significant relationship indicating there was a much greater tendency for applications to be selected when they came from states in which a maximum of three applications were submitted.

Given the low number of applications submitted by many of the states, however, we could not control for all of the differences between states in a multivariate analysis in which the effects of the other variables are estimated simultaneously. Therefore, we combined the states with smaller numbers of applications into two groups: one group contained states which submitted one to three applications and the other group contained states submitting four to nine applications. These groupings did not result in the loss of any significant information with respect to differences in the likelihood of applications being selected across states. The results of these state groupings indicate that the percentage of applications selected for funding from states with one to three applications (70 percent selected) were considerably higher than the percentage of funded applications from states with four to nine applications (40 percent selected). In addition, those states that submitted more than nine applications showed considerable differences in the percent of applications selected. California (37 percent selected) and Missouri (75 percent selected) had a relatively high percentage of applications selected, and New York (18 percent selected) and Washington state (5 percent selected) had a relatively low percentages of applications selected.

As in the previous tables, the odds and odds ratios give us the same sense of the association that the percentages reveal; the odds on being selected for funding were more than three times higher for applications from states submitting one to three applications than for applications from California or from states submitting four to nine applications (2.33/0.68, which equals 3.43). In addition, New York and Washington state were much less likely to likely to have an application selected than California and Missouri. (See table 14.)

Regression Analyses Show that Technical Review Score and Number of Applications Submitted Significantly Affected the Likelihood of Selection

We also examined the data using bivariate and multivariate logistic regression models to estimate the effects of these different variables on the likelihood of applications being selected for funding. The bivariate models examine the effects of the technical review score, track, amount requested, and state or group of states from which the application arose, one at a time. The odds ratios from these models are the same as those produced from the observed frequencies in the different two-way tables described above. From these models, however, we obtain specific tests of the significance of the differences between each variable category to determine more generally whether there are differences between any of the categories. In the bivariate regression model we find that many, but not all, of the odds ratios describing these differences are significant. (See table 15.)

The multivariate model estimates the net effects of these different variables on the likelihood of applications being selected for funding, or the effects of each variable when the effects of other variables are considered simultaneously, rather than one at a time. Our results indicate that the differences between tracks and amount requested categories are rendered insignificant when technical review scores and state and state group are taken into account, while the effect of technical review score and the differences between state and state group remain sizable and in most cases significant. Specifically, we found that, when we accounted for all four variables, applications receiving a technical review score of 3, 4, or 5 were about seven to eight times more likely to be selected for funding than applications which scored 1 or 2. In addition, applications from states that submitted one to three applications, and applications from Missouri, were three and nine times as likely, respectively, to be selected as those from California, while those from Washington state were less than one- tenth as likely as those from California to be selected. The remaining variable categories were not significant in the multivariate model and, therefore, do not provide a statistical explanation for why applications were more or less likely to be selected for an award.