Highway Research: DOT's Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes
Released: 2003-04-10T00:00:00

Letter

Background

FHWA is the DOT agency responsible for federal highway programs— including distributing billions of dollars in federal highway funds to the states—and developing federal policy regarding the nation’s highways. The agency provides technical assistance to improve the quality of the transportation network, conducts transportation research, and disseminates research results throughout the country. FHWA’s program offices conduct these activities through its Research and Technology Program, which includes “research” (conducting research activities), “development” (developing practical applications or prototypes of research findings), and “technology” (communicating research and development knowledge and products to users). FHWA maintains a highway research facility in McLean, Virginia. This facility, known as the Turner-Fairbank Highway Research Center, has over 24 indoor and outdoor laboratories and support facilities. Approximately 300 federal employees, on-site contract employees, and students are currently engaged in transportation research at the center.

FHWA’s research and technology program is based on the research and technology needs of each of its program offices such as the Offices of Infrastructure, Safety, or Policy. Each of the program offices is responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals for research and technology activities that support the agency’s strategic goals. (See Appendix I for examples of research that these offices undertake.) One program office that is located at FHWA’s research facility provides support for administering the overall program and conducts some of the research. The agency’s leadership team, consisting of the associate administrators of the program offices and other FHWA offices, provides periodic oversight of the overall program. In 2002 FHWA appointed the Director of its Office of Research, Development, and Technology as the focal point for achieving the agency’s national performance objective of increasing the effectiveness of all FHWA program offices, as well as its partners and stakeholders, in determining research priorities and deploying technologies and innovation.

In addition to the research activities within FHWA, the agency collaborates with other DOT agencies to conduct research and technology activities. For example, FHWA works with DOT’s Research and Special Programs Administration to coordinate efforts to support key research identified in the department’s strategic plan. Other nonfederal research and technology organizations also conduct research funded by FHWA related to highways and bridges. Among these are state research and technology programs that address technical questions associated with the planning, design, construction, rehabilitation, and maintenance of highways. In addition, the National Cooperative Highway Research Program conducts research on acute problems related to highway planning, design, construction, operation, and maintenance that are common to most states. Private organizations, including companies that design and construct highways and supply highway-related products, national associations of industry components, and engineering associations active in construction and highway transportation, also conduct or sponsor individual programs. Universities receive funding for research on surface transportation from FHWA, the states, and the private sector.

Research Community Promotes Use of Best Practices for Developing Research Agendas and Evaluating Research Outcomes

Leading organizations that conduct scientific and engineering research, other federal agencies with research programs, and experts in research and technology have identified and use best practices for developing research agendas and evaluating research outcomes. Although the uncertain nature of research outcomes over time makes it difficult to set specific, measurable program goals and evaluate results, the best practices we identified are designed to ensure that the research objectives are related to the areas of greatest interest and concern to research users and that research is evaluated according to these objectives. These practices include (1) developing research agendas through the involvement of external stakeholders and (2) evaluation of research using techniques such as expert review of the quality of research outcomes.

Developing Research Agendas Through the Involvement of External Stakeholders

External stakeholder involvement is particularly important for FHWA because its research is expected to improve the construction, safety, and operation of transportation systems that are primarily managed by others, such as state departments of transportation. According to the Transportation Research Board’s Research and Technology Coordinating Committee, research has to be closely connected to its stakeholders to help ensure relevance and program support, and stakeholders are more likely to promote the use of research results if they are involved in the research process from the start. The committee also identified merit review of research proposals by independent technical experts based on technical criteria as being necessary to help ensure the most effective use of federal research funds. In 1999, we reported that other federal science agencies—such as the Environmental Protection Agency and the National Science Foundation—used such reviews to varying degrees to assess the merits of competitive and noncompetitive research proposals. In April 2002, the Office of Management and Budget issued investment criteria for federal research and technology program budgets that urge these agencies to put into place processes to assure the relevance, quality and performance of their programs. For example, the guidance requires these programs to have agendas that are assessed prospectively and retrospectively through external review to ensure that funds are being expended on quality research efforts.

Evaluation of Research Using Systematic Approach to Review the Quality of Research Outcomes

The Committee on Science, Engineering, and Public Policy reported in 1999 that federal agencies that support research in science and engineering have been challenged to find the most useful and effective ways to evaluate the performance and results of the research programs they support. Nevertheless, the committee found that research programs, no matter what their character and goals, can be evaluated meaningfully on a regular basis and in accordance with the Government Performance and Results Act. Similarly, in April 2002 the Office of Management and Budget issued investment criteria for federal research and technology program budgets that require these programs to define appropriate outcome measures and milestones that can be used to track progress toward goals and assess whether funding should be enhanced or redirected. In addition, program quality should be assessed periodically in relation to these criteria through retrospective expert review. The Committee on Science, Engineering, and Public Policy also emphasized that the evaluation methods must match the type of research and its objectives, and it concluded that expert or peer review is a particularly effective means to evaluate federally funded research.

Peer review is a process that includes an independent assessment of the technical and scientific merit or quality of research by peers with essential subject area expertise and perspective equal to that of the researchers. Peer review does not require that the final impact of the research be known. In 1999, we reported that federal agencies, such as the Department of Agriculture, the National Institutes of Health, and the Department of Energy, use peer review to help them (1) determine whether to continue or renew research projects, (2) evaluate the results of research prior to publication of those results, and (3) evaluate the performance of programs and scientists. In its 1999 report, the Committee on Science, Engineering, and Public Policy also stated that expert review is widely used to evaluate: (1) the quality of current research as compared with other work being conducted in the field, (2) the relevance of research to the agency’s goals and mission, and (3) whether the research is at the “cutting edge.”

External Stakeholders’ Involvement in Developing FHWA’s Research Agendas Has Been Limited

Although FHWA engages external stakeholders in elements of its research and technology program, the agency currently does not follow the best practice of engaging external stakeholders on a consistent and transparent basis in setting its research agendas. The agency expects each program office to determine how or whether to involve external stakeholders in the agenda setting process. As we reported in May 2002, FHWA acknowledges that its approach to preparing research agendas is inconsistent and that the associate administrators of FHWA’s program offices primarily use input from the agency’s program offices, resource centers, and division offices. Although agency officials told us that resource center and division office staff provide the associate administrators with input based on their interactions with external stakeholders, to the extent that external stakeholder input into developing research agendas occurs, it is usually ad hoc and provided through technical committees and professional societies. For example, the agency’s agenda for environmental research was developed with input from both internal sources (including DOT’s and FHWA’s strategic plans and staff) and external sources (including the Transportation Research Board’s reports on environmental research needs and clean air, environmental justice leaders, planners, civil rights advocates, and legal experts).

In our May 2002 report we recommended that FHWA develop a systematic approach for obtaining input from external stakeholders in determining its research and technology program’s agendas. FHWA concurred with our recommendation and has taken steps to develop such an approach. FHWA formed a planning group consisting of internal stakeholders as well as representatives from the Research and Special Programs Administration and the Pennsylvania Department of Transportation to determine how to implement our recommendation. This planning group prepared a report analyzing the approaches that four other federal agencies are taking to involve external stakeholders in setting their research and technology program agendas. Using the lessons learned from reviewing these other agencies’ activities, FHWA has drafted a Corporate Master Plan for Research and Deployment of Technology & Innovation. Under the draft plan, the agency would be required to establish specific steps for including external stakeholders in the agenda setting process for all areas of research throughout the agency’s research and technology program by fiscal year 2004. In drafting this plan, FHWA officials obtained input from internal stakeholders as well as external stakeholders, including state departments of transportation, academia, consultants, and members of the Transportation Research Board. It appears that FHWA has committed to taking the necessary steps to adopt the best practice of developing a systematic process for involving external stakeholders in the agenda setting process. The draft plan invites external stakeholders to assist FHWA with such activities as providing focus and direction to the research and technology program and setting the program’s agendas and priorities. However, because FHWA’s plan has not been finalized, we cannot comment on its potential effectiveness in involving external stakeholders.

FHWA Lacks a Systematic Approach to Evaluating Research Outcomes

As we reported last year, FHWA does not have an agency wide systematic process to evaluate whether its research projects are achieving intended results that uses such techniques as peer review. Although the agency’s program offices may use methods such as obtaining feedback from customers and evaluating outputs or outcomes versus milestones, they all use success stories as the primary method to evaluate and communicate research outcomes. According to agency officials, success stories are examples of research results adopted or implemented by such stakeholders as state departments of transportation. These officials told us that success stories can document the financial returns on investment and nonmonetary benefits of research and technology efforts. However, we raised concerns that success stories are selective and do not cover the breadth of FHWA’s research and technology program.

In 2001, the Transportation Research Board’s Research and Technology Coordinating Committee concluded that peer or expert review is an appropriate way to evaluate FHWA’s surface transportation research and technology program. Therefore, the committee recommended a variety of actions, including a systematic evaluation of outcomes by panels of external stakeholders and technical experts to help ensure the maximum return on investment in research. Agency officials told us that increased stakeholder involvement and peer review will require significant additional expenditures for the program. However, a Transportation Research Board official told us that the cost of obtaining expert assistance could be relatively low because the time needed to provide input would be minimal and could be provided by such inexpensive methods as electronic mail.

In our May 2002 report, we recommended that FHWA develop a systematic process for evaluating significant ongoing and completed research that incorporates peer review or other best practices in use at federal agencies that conduct research. While FHWA has concurred that the agency must measure the performance of its research and technology program, it has not developed, defined or adopted a framework for measuring performance. FHWA’s report on efforts of other federal agencies that conduct research, discussed above, analyzed the approaches that four other federal agencies are taking to evaluate their research and technology programs using these best practices. According to FHWA’s assistant director for Research, Technology, and Innovation Deployment, the agency is using the results of this report to develop its own systematic approach for evaluating its research and technology program. However, this official noted that FHWA has been challenged to find the most useful and effective ways to evaluate the performance and results of the agency’s research and technology program. According to FHWA’s draft Corporate Master Plan for Research and Deployment of Technology & Innovation, FHWA is committed to developing a systematic method of evaluating its research and technology program that includes the use of a merit review panel. This panel would conduct evaluations and reviews in collaboration with representatives from FHWA staff, technical experts, peers, special interest groups, senior management, and contracting officers. According to the draft plan, these merit reviews would be conducted on a periodic basis for program-level and agency-level evaluations, while merit reviews at the project level would depend on the project’s size and complexity. FHWA is still in the process of developing, defining, and adopting a framework for measuring performance. Therefore, we cannot yet comment on how well FHWA’s efforts to evaluate research outcomes will follow established best practices.

Mr. Chairman, this concludes my prepared statement. I would be pleased to answer any questions that you or Members of the Committee may have.

Contacts and Acknowledgments

For further information on this testimony, please contact Katherine Siggerud at (202) 512-2834 or siggerudk@gao.gov. Deena Richart made key contributions to this testimony.

Appendix I: Roles of Program Offices in FHWA’s Research and Technology Program

FHWA’s research and technology program is based on the research and technology needs of each of its program offices such as the Offices of Infrastructure, Safety, and Policy. Each of the program offices is responsible for identifying research needs, formulating strategies to address transportation problems, and setting goals for research and technology activities that support the agency’s strategic goals. (See table 1.)