Highway Research: Systematic Selection and Evaluation Processes Needed for Research Program
Released: 2002-05-24T00:00:00

Letter

Background

FHWA is the DOT agency responsible for federal highway programs— including distributing billions of dollars in federal highway funds to the states—and developing federal policy regarding the nation’s highways. The agency provides technical assistance to improve the quality of the transportation network, conducts transportation research, and disseminates research results throughout the country. FHWA’s business units conduct these activities through its research and technology program, which includes “research” (conducting research activities), “development” (developing practical applications or prototypes of research findings), and “technology” (communicating research and development knowledge and products to users). FHWA maintains a highway research facility in McLean, Virginia. This facility, known as the Turner-Fairbank Highway Research Center, has over 24 indoor and outdoor laboratories and support facilities. Approximately 300 federal employees, on-site contract employees, and students are currently engaged in transportation research at the center.

According to FHWA officials, the agency’s research and technology program is oriented to supporting the agency’s and DOT’s strategic goals for the nation’s transportation system, including to promote public health and safety by working toward the elimination of transportation-related deaths and injuries; to provide an accessible, affordable, and reliable transportation system for all people, goods, and regions; to support a transportation system that sustains the nation’s economic growth; to protect and enhance communities and the natural environment affected by transportation; and to ensure the security of the transportation system for the movement of people and goods, and to support the national security strategy.

The research and technology program is generally a component of broader agency programs directed toward the achievement of these strategic goals.

For example, in a recent report the Transportation Research Board’s Research and Technology Coordinating Committee (RTCC) stated that most of FHWA’s research and technology program’s projects are aimed at incremental improvements to lower highway construction and maintenance costs, improve highway system performance, increase highway capacity, reduce highway fatalities and injuries, reduce adverse environmental impacts, and provide a variety of benefits such as improved travel times and fewer hazards for highway users.

Concerned about the strategic focus of surface transportation research and technology activities, Congress required DOT to establish a strategic planning process to identify national priorities related to research and technology for surface transportation when it passed the Transportation Equity Act for the 21st Century in 1998. This process was to result in a strategic plan that included, among other things, performance goals, resources needed to achieve those goals, and performance indicators for the succeeding 5 years for each area of research and technology deployment. The plan was also to be developed with comments from external stakeholders. In response to this requirement, FHWA contributed to the development of a research, development, and technology strategic plan for all of DOT. DOT’s plan identifies formal research, development, and technology strategies to support each of DOT’s strategic goals. The plan is not focused solely on surface transportation research but applies to all modes, including examples of research activities undertaken by FHWA in support of the agency’s strategic goals. Congress also required that a group established by the National Research Council review DOT’s plan, and this has taken place for several years. Separately, in 1998 FHWA developed a 10-year strategic plan for the agency as a whole, stating that research is a strategy for achieving the plan’s objectives. The Research, Development, and Technology business unit has developed performance plans that support some of FHWA’s research efforts.

Funding mechanisms for this program’s activities have varied in recent years. Prior to fiscal year 1992, they were wholly funded from FHWA’s administrative and operating funds. From fiscal years 1992 through 1997, the program was supported by a mix of operating funds and funds made available for specific types of research. For fiscal years 1998 through 2003, the Transportation Equity Act for the 21st Century authorized funding for the following seven research activities: surface transportation research, technology deployment, training and education, intelligent transportation systems, intelligent transportation systems deployment, university transportation centers, and the Bureau of Transportation Statistics. Since 1998, FHWA has generally not used administrative funds for research activities. A portion of the funds for the research and technology program are designated for or directed to particular research programs and recipients, either in the authorization or appropriations legislation or in committee reports. Although FHWA technical staff set priorities for the research and technology program, its activities are carried out through a combination of federal employees, private contractors and grantees, and university researchers. During the past decade, the use of contract employees instead of federal employees to conduct research has increased. Because the program’s authorizing legislation is scheduled to expire in fiscal year 2003, to continue it Congress will have to reauthorize the program and determine how it will be funded.

FHWA’s Surface Transportation Research and Technology Program’s Organization Is Complex and Decentralized

Since 1998, individual business units within FHWA have directed and carried out the activities of FHWA’s research and technology program that fall under the surface transportation research and technology deployment areas. (See app. II for agency organization charts.) Under the current organization, directors of these business units (Federal Lands Highway; Infrastructure; Operations; Planning and Environment; Policy; Research, Development, and Technology; and Safety) work collaboratively to provide leadership for the program’s activities (see table 1).

The program’s management is complex because these business units are individually responsible, among other things, for identifying research needs, developing strategies to address transportation problems, and managing research and technology activities that support the agency’s strategic goals. In some cases, the business units conduct their own research. However, the Research, Development, and Technology business unit, located at the Turner-Fairbank Highway Research Center, conducts research for the Infrastructure, Operations, and Safety business units. The Research, Development, and Technology business unit also works with the other business units to prepare materials to support the program’s overall budget, and it serves as FHWA’s liaison to other organizations that advise FHWA on research or conduct highway-related research. The agency’s leadership team, composed of the business unit directors, field service directors, a division administrator, the FHWA administrator, and the FHWA executive director, meets periodically to advise the business units on research and technology program priorities, budgets, and milestones.

FHWA’s leadership team advises the business units on how funds should be distributed by considering designations in statutes and committee reports and the stated needs of individual business units. The Office of the Administrator approves final budgets for the business units. In fiscal year 2002, the business unit responsible for the largest percentage of surface transportation research and technology deployment funds was the Infrastructure business unit (see fig. 1).

Prior to the agencywide restructuring in 1998, research activities were managed throughout the organization, including at the Office of the Associate Administrator for Research and Development and the Office of Technology Applications. Decisions related to developing research and technology projects, budgets, and acquisition plans were made by the Research and Technology Executive Board. Chaired by the executive director, the board’s membership included all agency associate administrators, the director of the Intelligent Transportation Systems Joint Program Office, and one regional administrator. The board met periodically to obtain information from working groups composed of representatives from across the agency, the National Highway Institute, and other DOT agencies. FHWA has recently assessed the effects of its 1998 agencywide restructuring and has drafted 13 recommendations to address the limitations of the new organization. Two of these recommendations specifically address the agency’s research and technology program, identifying the need to raise its stature in FHWA. The agency has created and filled the position of assistant director for Research, Technology, and Innovation Deployment as a response to this recommendation. This new position will also be responsible for implementing recent recommendations made by the RTCC for improving FHWA’s program.

FHWA Collaborates with Other DOT Offices for Research Efforts

In addition to its own research projects, FHWA collaborates with other DOT agencies to conduct research. For example, FHWA works with DOT’s Research and Special Programs Administration to coordinate efforts to support key research identified in the department’s strategic plan. In fiscal year 2001, FHWA and the Research and Special Programs Administration contributed an estimated $15.2 million and $3.5 million, respectively, for these collaborative, “intermodal” research and technology efforts. Examples of FHWA’s research with other transportation modes include: an ongoing study with DOT’s National Highway Traffic Safety Administration, through the Georgia Institute of Technology, to investigate the relationship between vehicle speed and crash risk under various demographic, environmental, and physical conditions. Funds from FHWA were spent to compare the speeds of drivers involved in crashes with the prevailing speeds of other drivers at the time and location of the crashes; and a study at the Center for Climate Change and Environmental Forecasting, with the collaboration of several other agencies, including DOT’s Maritime Administration, Federal Railroad Administration, and National Highway Traffic Safety Administration. This study examined the potential effects on transportation infrastructure of such climate change phenomena as rising sea levels, increasing frequency of severe weather events, and changing precipitation levels.

Other Organizations Have a Significant Role in Research and Technology Efforts

Several other entities and organizations, detailed below, conduct surface transportation research that can be related to FHWA’s research and technology program. FHWA officials told us that the agency has both formal and informal means for coordination with some of these other organizations.

Each of the 50 states, Washington, D. C., and the Commonwealth of Puerto Rico have an independent highway research program. In general, state programs address technical questions associated with the planning, design, construction, rehabilitation, and maintenance of highways. State highway research projects usually reflect local concerns. According to an official at the Transportation Research Board, 47 states indicated that they spent approximately $322 million in 1999 on such research. State research programs are generally funded through federal funds set aside from the federal highway aid apportioned to the states. FHWA division administrators in each state approve the state’s annual or biennial research program, funded by a subset of federal funds. The national association that represents state departments of transportation, the American Association of State Highway and Transportation Officials, also plays a key role in highway research. This association has a standing committee on research that develops voluntary standards and guidelines. The National Cooperative Highway Research Program conducts research on acute problems related to highway planning, design, construction, operation, and maintenance that are common to most states. Typically, its research projects are problem-oriented and designed to produce results that have an immediate application. As voluntary program members, state departments of transportation approve research projects and agree to provide financial support. Each member state provides an amount equal to 5.5 percent of its state planning and research funds. Program funding for fiscal year 2001 was $30.6 million. FHWA formally coordinates with members of this program and the American Association of State Highway and Transportation Officials to review proposed projects. FHWA also participates in selecting projects that complement the agency’s defined program, reducing duplication and leveraging limited funding.

The private sector conducts or sponsors individual programs. Private organizations include companies that design and construct highways and supply highway-related products, national associations of industry components, and engineering associations active in construction and highway transportation. Funding information for private-sector highway research is generally proprietary in nature, although an official of the Transportation Research Board estimated that the total funding for this research ranged from $75 million to $150 million annually.

Universities receive funding for research on surface transportation from FHWA, the states, and the private sector. For example, since 1988 DOT has awarded grants under its University Transportation Center program to universities throughout the nation to support education, research, and technology deployment. Each grantee is called a University Transportation Center, whether working alone or as the lead of a consortium of universities. Some have formed centers for research, education, and training in specialty areas related to highway transportation. Thirty-three centers currently exist; they were either selected competitively or specified in legislation. The Office of Innovation, Research, and Education within the department’s Research and Special Programs Administration manages the program; funding provided for the 33 centers in fiscal year 2001 from FHWA’s research and technology program amounted to $23.9 million.

FHWA Processes for Developing Research Agendas and Evaluating Research Outcomes Do Not Always Follow Best Practices for Federal Research Programs

Leading organizations that conduct scientific and engineering research, other federal agencies with research programs, and experts in research and technology have identified and use best practices for developing research agendas and evaluating research outcomes. Although the uncertain nature of research outcomes over time makes it difficult to set specific, measurable program goals and evaluate results, the best practices we identified are designed to ensure that the research objectives are related to the areas of greatest interest and concern to research users and that research is evaluated according to these objectives. These practices include: Developing research agendas through the involvement of external stakeholders: External stakeholder involvement and merit review are particularly important for FHWA because its research is expected to improve the construction, safety, and operation of transportation systems that are primarily managed by others, such as state departments of transportation. According to RTCC, research has to be closely connected to its stakeholders to help ensure relevance and program support, and stakeholders are more likely to promote the use of research results if they are involved in the research process from the start. The committee also identified merit review of research proposals based on technical criteria by independent technical experts as being necessary to help ensure the most effective use of federal research funds. In 1999, we reported that other federal science agencies—such as the Environmental Protection Agency and the National Science Foundation—used such reviews to varying degrees to assess the merits of competitive and noncompetitive research proposals.

Evaluation of research using expert review of the quality of research outcomes or other best practices: A form of expert review called peer review is a process that includes an independent assessment of the technical and scientific merit or quality of research by peers with essential subject area expertise and perspective equal to that of the researchers. Peer review does not require that the final impact of the research be known. In 1999, we reported that federal agencies, such as the Department of Agriculture, the National Institutes of Health, and the Department of Energy, use peer review to help them (1) determine whether to continue or renew research projects, (2) evaluate the results of research prior to publication of those results, and (3) evaluate the performance of programs and scientists. In its 1999 report, the Committee on Science, Engineering, and Public Policy also stated that expert review is widely used to evaluate three aspects of the Government Performance and Results Act:(1) the quality of current research as compared with other work being conducted in the field, (2) the relevance of research to the agency’s goals and mission, and (3) whether the research is at the “cutting edge.”

External Stakeholders’ Involvement in Developing the Program’s Research Agendas Has Been Limited

Although FHWA engages external stakeholders in elements of its research and technology program, the agency currently does not follow the best practice of engaging external stakeholders on a sustained basis. The agency expects each business unit to determine how or whether to involve external stakeholders in the research process. As a result, this approach is used inconsistently. Prior to its 1998 restructuring, FHWA worked with some external stakeholders to initiate “roadmapping” activities for each of its key research areas that would have resulted in research agendas for these areas. To prepare individual roadmaps, the agency’s working groups collaborated across agency office boundaries and with members of the RTCC. However, before the roadmapping had been completed for all research areas, FHWA changed its approach to managing research because of the agency’s reorganization, and RTCC’s involvement with roadmapping ceased.

FHWA acknowledges that its approach to preparing research agendas is inconsistent and that the directors of FHWA’s business units primarily use input from the agency’s business units, resource centers, and division offices. Although agency officials told us that resource center and division office staff provide the business unit directors with input based on their interactions with external stakeholders, external stakeholder input into developing research agendas is usually ad hoc, provided through technical committees and professional societies. For example, the agency’s agenda for environmental research was developed with input from both internal sources (including DOT’s and FHWA’s strategic plans and staff) and external sources (including the Transportation Research Board’s reports on environmental research needs and clean air, environmental justice leaders, planners, civil rights advocates, and legal experts). Similarly, the agency uses external stakeholders to provide merit review of research projects on an ad hoc basis. For example, to prepare its “Conditions and Performance Report”, the Policy business unit used a peer review group to provide input into the Highway Economic Requirements System (an economic model that uses marginal cost-benefit analysis to optimize highway investment).

FHWA acknowledges that the agency lacks a consistent, transparent, and systematic approach for engaging stakeholders in setting research agendas. However, FHWA has recently taken several steps to increase the involvement of external stakeholders in developing research agendas. First, FHWA’s work with RTCC has resulted in the agency’s obtaining occasional external guidance for its overall program since 1991. The committee points out, however, that it cannot provide broad-based input from stakeholders on the full range of potential highway research topics or specific projects on a continuing basis because its membership is not representative of all the disciplines included in FHWA’s research and technology program. In its 2001 report, the committee recommended that decisions about FHWA research topics should balance stakeholders’ concerns against experts’ external reviews and recommendations as to which research areas hold promise for significant breakthroughs. According to the draft response to the recommendation, FHWA plans to develop such a process by June 30, 2002. In addition, in 1998, FHWA helped organize a National Highway Research and Technology Partnership Forum to identify national highway research and technology needs using input from external stakeholders. Although the forum identified research needs and priorities for FHWA’s consideration in its draft report of August 2001, its long-term role remains to be seen.

FHWA officials told us that their ability to develop their research agendas using best practices is also affected by funding designations contained in statutes and committee reports. These designations take a variety of forms, including requiring FHWA to initiate or maintain specific research efforts and specifying dollar amounts for particular recipients. According to agency officials, the designations made by the Transportation Equity Act for the 21st Century and conference reports accompanying recent appropriations acts have represented significant proportions of the agency’s research budget. Using agency data, we calculated that 44 percent of authorized surface transportation research and technology deployment funds in fiscal year 2000, 48 percent in fiscal year 2001, and 44 percent in fiscal year 2002 were designated (see app. I, tables 4, 5, and 6).Agency officials acknowledged that these funding designations reflect congressional interests and priorities but also stated that without these designations, FHWA would have an enhanced opportunity to consistently plan its research agendas and select researchers for its projects according to accepted best practices.

FHWA Lacks a Systematic Process to Evaluate Research Outcomes

In 1999, the Committee on Science, Engineering, and Public Policy reported that federal agencies that support research in science and engineering have been challenged to find the most useful and effective ways to evaluate the performance and results of the research programs they support. However, the committee found that research programs, no matter what their character and goals, can be evaluated meaningfully on a regular basis and in accordance with the Government Performance and Results Act. The committee emphasized that the evaluation methods must match the type of research and its objectives, and it concluded that expert or peer review is a particularly effective means to evaluate federally funded research. The peer review process includes an independent assessment of the technical and scientific merits of research by those with knowledge and expertise equal to that of the researchers whose work they review.

According to FHWA officials, the agency does not have an agencywide systematic process to evaluate whether its research projects are achieving intended results and does not generally use a peer review approach. Although the agency’s business units may use such various methods as obtaining feedback from customers and evaluating outputs or outcomes versus milestones, they all use success stories as the primary method to evaluate research outcomes. According to agency officials, success stories are examples of research results adopted or implemented by such stakeholders as state departments of transportation. Although agency officials told us that peer reviews are useful to assess research quality, relevance, and technical breakthroughs, success stories can document the financial returns on investment and nonmonetary benefits of research and technology efforts. FHWA officials provided us with the following examples of success stories: Research conducted by the Infrastructure business unit produced a specification guide on how to mitigate earthquake damage to structures. The guide was adopted by the American Association of State Highway Transportation Officials for inclusion in its guidance to state departments of transportation.

The operations research and technology group developed the 511 traveler telephone number that replaced 300 different traveler information telephone numbers nationwide. This single, three-digit number is currently being used in the states of Utah and Nebraska and in parts of Virginia, Kentucky, and Ohio to provide motorists with timely local travel information to help relieve traffic congestion.

To respond to one of FHWA’s priority safety emphases, the safety research and technology group developed rumble strips to warn drivers who are driving their vehicles off the road. According to agency officials, in the eight states surveyed that have used rumble strips, crash reduction has ranged from 18 to 72 percent, and the cost-benefit ratio has ranged from 30:1 to as high as 60:1.

Research on long-term pavement performance is significantly improving the pavement-engineering process nationwide. Engineers are using a software tool known as a long-term pavement performance bind to more accurately determine the asphalt binder grade needed for specific environmental conditions. This software tool has helped highway agencies to save at least $50 million each year by reducing the application of unnecessary substances that increase the costs of highway construction.

In 2001, RTCC also concluded that peer or expert review is an appropriate way to evaluate FHWA’s surface transportation research and technology program. Therefore, the committee recommended a variety of actions, including a systematic evaluation of outcomes by panels of external stakeholders and technical experts to help ensure the maximum return on investment in research. Agency officials told us that increased stakeholder involvement and peer review will require significant additional expenditures for the program. However, a Transportation Research Board official told us that the cost of obtaining expert assistance could be relatively low because the time needed to provide input would be minimal and could be provided by such inexpensive methods as electronic mail. As a partial response to RTCC’s recommendation, FHWA has established a laboratory assessment process that will be used to conduct regular reviews of the Turner-Fairbank Highway Research Center. These reviews will be conducted by panels of external technical experts and will include such issues as technical excellence and quality of lab activities. FHWA’s draft response to this recommendation indicates that it plans to initiate an evaluation process by June 30, 2002.

Conclusions

With millions of dollars for its research, FHWA’s research and technology program has the potential to significantly improve the nation’s highway system. FHWA has described several success stories to us but, because its decisions about selecting research and identifying priorities are uneven in the extent to which they use best practices such as seeking external input, it is unclear whether the agency is selecting the most important and relevant research. In addition, because FHWA does not systematically evaluate its research and technology program, it is unclear whether the research is having the intended results or whether some refocusing of the research would be justified. Therefore, we agree with several of the recent recommendations from the Transportation Research Board’s Research and Technology Coordinating Committee, which were designed to remedy these limitations of FHWA’s program. In its draft response to these recommendations, FHWA has indicated that it will take action on most of them. The cost of making such improvements in FHWA’s research and technology program is unknown and will influence the extent to which FHWA can adopt certain best practices. Because Congress has been concerned about the strategic focus of FHWA’s research and technology program and will soon have to make decisions about the nature of the program and the level of resources to devote to it, information generated by FHWA’s potentially improved processes for developing research agendas and evaluating research outcomes, as well as information about the cost of such changes, will also be useful to Congress.

Recommendations for Executive Action

To help ensure that FHWA’s research agenda and approach to evaluation are identifying research with the highest value to the surface transportation community and monitoring the outcomes of that research, we are recommending that the secretary of transportation direct the FHWA administrator to develop a systematic approach for obtaining input from external stakeholders in determining the research and technology program’s agendas; develop a systematic process for evaluating significant ongoing and completed research that incorporates peer review or other best practices in use at federal agencies that conduct research; and develop specific plans for implementing these recommendations, including time frames and estimates of their cost.

Agency Comments and Our Evaluation

We obtained oral comments on a draft of this report from FHWA officials, including the director of Research, Development, and Technology and the director of the Office of Program Development and Evaluation. These officials indicated that they were pleased that the draft report had recognized some of the FHWA research and technology program’s accomplishments to date, along with its potential to significantly improve the nation’s highway system. They also indicated general agreement with the draft report’s overall assessment of the program and the draft report’s recommendations.

The FHWA officials told us that they have been working with both internal and external groups to assess the processes used to plan the research and technology program and to evaluate its results. These officials maintain that the program is essentially sound and pursues worthy research in an effective manner with key program stakeholders. Nonetheless, the agency officials agreed that improvements are possible in the methods used to select research and technology projects and to evaluate program results.

They told us that FHWA had recently taken steps to make research a higher priority for the agency by investing in research to meet stakeholders’ needs, improving delivery of innovations to potential users, and improving business processes in the research and technology program. As a result of a major restructuring assessment, FHWA officials told us that the agency has also committed to making research and technology more prominent as a strategy for achieving FHWA’s mission. With regard to project planning and selection, FHWA officials explained that they are examining ways to improve existing methods for incorporating stakeholder input and seeking means to further ensure that stakeholder perspectives are fully and effectively considered. Finally, with regard to evaluating program results, FHWA officials told us that although there are merits to current methods, more extensive and consistent use of best practices such as peer review could benefit the program.

We acknowledge that FHWA recently has planned or put into place several initiatives designed to improve its research and technology program, and we describe these actions in this report. Nevertheless, we continue to believe that additional actions in response to our recommendations are warranted to improve FHWA’s processes for setting research agendas and evaluating research efforts.

We are sending copies of this report to congressional committees and subcommittees with responsibilities for transportation, the secretary of transportation, the Federal Highway Administration administrator, and the director of the Office of Management and Budget. We will make copies available to others upon request.

If you or your staff have any questions about this report, please call me at (202) 512-2834. Key contributors to this report were Sharon Dyer, Sally Gilley, Octavia Parks, Deena Richart, and Kate Siggerud.

Appendix I: Funding Information for FHWA’s Research and Technology Program

In fiscal year 1992 (the first year in which FHWA’s research and technology program was authorized under the Intermodal Surface Transportation Efficiency Act of 1991), authorized funding for the entire program increased almost fivefold, from approximately $88.6 million in fiscal year 1991 to $442.4 million. Since that time, authorized funding for FHWA’s research and technology program has remained relatively flat; from fiscal year 1992 through fiscal year 2001, authorized funding for the program went from $442.4 million to $437.3 million. However, since fiscal year 1998 these authorized funds have been subject to an obligation limitation that has reduced amounts available for research purposes an average of about 11 percent a year below authorized funding levels (see fig. 2).

The areas of research funded from fiscal years 1992 through 2001 have varied based on authorizing legislation. From fiscal year 1992 through fiscal year 1997, the majority of FHWA’s entire surface transportation research and technology funding went to support the Intelligent Vehicle Highway Systems program. The remainder of funds primarily supported the agency’s highway research, development, and technology program and applied research and technology program. Since fiscal year 1998, the majority of the agency’s research and technology program funds have continued to support the intelligent transportation systems program as well as the surface transportation research program. (See tables 2 and 3 for funding allocations by program area for fiscal years 1992 through 2001.)

These funds were subject to designations in statutes and committee reports, with the Infrastructure business unit being the most affected (see tables 4, 5, and 6 for designations by business unit for fiscal years 2000 through 2002). In fiscal year 2002, approximately 80 percent of the surface transportation research and technology deployment funds provided to the Infrastructure business unit were designated.

Appendix II: Federal Highway Administration Organization Charts