Highway Research: DOT's Actions to Implement Best Practices for Setting Research Agendas and Evaluating Outcomes
Published: 2003-04-10T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Improvement and innovation based on highway research have long been important to the highway system. The Federal Highway Administration (FHWA) is the primary federal agency involved in highway research. Throughout the past decade, FHWA received hundreds of millions of dollars for its surface transportation research program, including nearly half of the Department of Transportation's approximate $1 billion budget for research in fiscal year 2002. Given the expectations of highway research and the level of resources dedicated to it, it is important to know that FHWA is conducting high quality research that is relevant and useful. In May 2002, GAO issued a report on these issues and made recommendations to FHWA, which the agency agreed with, aimed at improving its processes for setting research agendas and evaluating its research efforts. GAO was asked to testify on (1) best practices for developing research agendas and evaluating research outcomes for federal research programs; (2) how FHWA's processes for developing research agendas align with these best practices; and (3) how FHWA's processes for evaluating research outcomes align with these best practices.
section_title: What GAO Found
paragraphs: Leading organizations, federal agencies, and experts that conduct scientific and engineering research use best practices designed to ensure that research objectives are related to the areas of greatest interest to research users and that research is evaluated according to these objectives. Of the specific best practices recommended by experts--such as the Committee on Science, Engineering, and Public Policy and the National Science Foundation--GAO identified the following practices as particularly relevant for FHWA: (1) developing research agendas in consultation with external stakeholders to identify high-value research and (2) using a systematic approach to evaluate research through such techniques as peer review. FHWA's processes for developing its research agendas do not always consistently include stakeholder involvement. External stakeholder involvement is important for FHWA because its research is to be used by others that manage and construct transportation systems. FHWA acknowledges that its approach for developing research agendas lacks a systematic process to ensure that external stakeholders are involved. In response to GAO's recommendation, FHWA has drafted plans that take the necessary steps toward developing a systematic process for involving external stakeholders. While the plans appear responsive to GAO's recommendation, GAO cannot evaluate their effectiveness until they are implemented. FHWA does not have a systematic process that incorporates techniques such as peer review for evaluating research outcomes. Instead, the agency primarily uses a "success story" approach to communicate about those research projects that have positive impacts. As a result, it is unclear the extent to which all research projects have achieved their objectives. FHWA acknowledges that it must do more to measure the performance of its research program, however, it is still in the process of developing a framework for this purpose. While FHWA's initial plans appear responsive to GAO's recommendation, GAO cannot evaluate their effectiveness until they are implemented.

section_title: Letter
paragraphs: Improvement and innovation based  on highway research have long  been important to the highway  system.  The Federal Highway  Administration (FHWA)  is the  primary federal agency involved in  highway research.  Throughout the  past decade, FHWA received  hundreds of millions of dollars for  its surface transportation research  program, including nearly half of  the Department of Transportation’s  approximate $1 billion budget for  research in fiscal year 2002.  Given  the expectations of highway  research and the level of resources  dedicated to it, it is important to  know that FHWA is conducting  high quality research that is  relevant and useful.  In May 2002,  GAO issued a report on these  issues and made recommendations  to FHWA,  which the agency agreed  with, aimed at improving its  processes for setting research  agendas and evaluating its research  efforts.
Leading organizations, federal agencies, and experts that conduct scientific  and engineering research use best practices designed to ensure that research  objectives are related to the areas of greatest interest to research users and  that research is evaluated according to these objectives.  Of the specific best  practices recommended by experts—such as the Committee on Science,  Engineering, and Public Policy and the National Science Foundation—GAO  identified the following practices as particularly relevant for FHWA: (1)  developing research agendas in consultation with external stakeholders to  identify high-value research and (2) using a systematic approach to evaluate  research through such techniques as peer review.
FHWA’s processes for developing its research agendas do not always  consistently include stakeholder involvement.  External stakeholder  involvement is important for FHWA because its research is to be used by  others that manage and construct transportation systems.  FHWA  acknowledges that its approach for developing research agendas lacks a  systematic process to ensure that external stakeholders are involved.  In  response to GAO’s recommendation, FHWA has drafted plans that take the  necessary steps toward developing a systematic process for involving  external stakeholders.  While the plans appear responsive to GAO’s  recommendation, as shown in the table below, GAO cannot evaluate their  effectiveness until they are implemented.
GAO was asked to testify on (1)  best practices for developing  research agendas and evaluating  research outcomes for federal  research programs; (2) how  FHWA's processes for developing  research agendas align with these  best practices; and (3) how  FHWA's processes for evaluating  research outcomes align with these  best practices.
FHWA does not have a systematic process that incorporates techniques such  as peer review for evaluating research outcomes.  Instead, the agency  primarily uses a “success story” approach to communicate about those  research projects that have positive impacts. As a result, it is unclear the  extent to which all research projects have achieved their objectives.  FHWA  acknowledges that it must do more to measure the performance of its  research program, however, it is still in the process of developing a  framework for this purpose.  While FHWA’s initial plans appear responsive  to GAO’s recommendation, GAO cannot evaluate their effectiveness until  they are implemented.
Does FHWA have plans  that are responsive to  the recommendation?
Has FHWA  implemented the  recommendation?
GAO’s Recommendation  Develop a systematic approach for obtaining  input from external stakeholders in  determining the research program’s agendas.   Develop a systematic process for evaluating  significant ongoing and completed research  that includes peer review or other best  practices in use at federal research agencies.  Develop specific plans for implementing these  recommendations, including time frames and  estimates of their costs.
Mr. Chairman and Members of the Committee:  We appreciate the opportunity to testify today on the Federal Highway  Administration’s (FHWA) surface transportation research and technology  program. Change, improvement, and innovation based on highway  research have long been important to the highway system. While this  research is a shared responsibility among FHWA, state departments of  transportation, and private organizations, we focused on FHWA’s  important leadership role as the primary federal agency involved in  highway research. Throughout the past decade, FHWA has received  hundreds of millions of dollars for its surface transportation research and  technology program, including nearly half of the Department of  Transportation’s (DOT) approximate $1 billion budget for research,  development, and technology in fiscal year 2002. Given the important  expectations of highway research and the significant level of resources  dedicated to it, it is important for the Congress and the American people  to know that the agency is conducting research that is relevant and useful  to stakeholders and that is of high quality. In May 2002 we issued a report  on these issues and made recommendations to FHWA, which the agency  agreed with, aimed at improving its processes for setting research agendas  and evaluating its research efforts. As it considers reauthorizing FHWA’s  research and technology program, Congress will be making decisions  about the structure of the program. Accordingly, my testimony today will  discuss (1) best practices for developing research agendas and evaluating  research outcomes for federal research programs; (2) the extent to which  FHWA’s processes for developing research agendas align with the best  practices for similar federal research programs; and (3) the extent to  which FHWA’s processes for evaluating research outcomes align with  these best practices.
My statement is based in part on our May 2002 report, which focused  primarily on those activities funded by the surface transportation research  and technology deployment funding categories identified in the  Transportation Equity Act for the 21st Century. In developing that report,  we held discussions with FHWA officials and reviewed relevant program  documents, legislation, and publications on best practices in federal  research from the Transportation Research Board, the Committee on  Science, Engineering, and Public Policy and others. In preparing for this  hearing, we also updated FHWA’s activities in response to our findings and  recommendations.
subsections: section_title: Background
paragraphs: FHWA is the DOT agency responsible for federal highway programs— including distributing billions of dollars in federal highway funds to the  states—and developing federal policy regarding the nation’s highways. The  agency provides technical assistance to improve the quality of the  transportation network, conducts transportation research, and  disseminates research results throughout the country. FHWA’s program  offices conduct these activities through its Research and Technology  Program, which includes “research” (conducting research activities),  “development” (developing practical applications or prototypes of  research findings), and “technology” (communicating research and  development knowledge and products to users). FHWA maintains a  highway research facility in McLean, Virginia. This facility, known as the  Turner-Fairbank Highway Research Center, has over 24 indoor and  outdoor laboratories and support facilities. Approximately 300 federal  employees, on-site contract employees, and students are currently  engaged in transportation research at the center.
FHWA’s research and technology program is based on the research and  technology needs of each of its program offices such as the Offices of  Infrastructure, Safety, or Policy. Each of the program offices is responsible  for identifying research needs, formulating strategies to address  transportation problems, and setting goals for research and technology  activities that support the agency’s strategic goals. (See Appendix I for  examples of research that these offices undertake.) One program office  that is located at FHWA’s research facility provides support for  administering the overall program and conducts some of the research. The  agency’s leadership team, consisting of the associate administrators of the  program offices and other FHWA offices, provides periodic oversight of  the overall program. In 2002 FHWA appointed the Director of its Office of  Research, Development, and Technology as the focal point for achieving  the agency’s national performance objective of increasing the  effectiveness of all FHWA program offices, as well as its partners and  stakeholders, in determining research priorities and deploying  technologies and innovation.
In addition to the research activities within FHWA, the agency  collaborates with other DOT agencies to conduct research and technology  activities. For example, FHWA works with DOT’s Research and Special  Programs Administration to coordinate efforts to support key research  identified in the department’s strategic plan. Other nonfederal research  and technology organizations also conduct research funded by FHWA  related to highways and bridges. Among these are state research and  technology programs that address technical questions associated with the  planning, design, construction, rehabilitation, and maintenance of  highways. In addition, the National Cooperative Highway Research  Program conducts research on acute problems related to highway  planning, design, construction, operation, and maintenance that are  common to most states. Private organizations, including companies that  design and construct highways and supply highway-related products,  national associations of industry components, and engineering  associations active in construction and highway transportation, also  conduct or sponsor individual programs. Universities receive funding for  research on surface transportation from FHWA, the states, and the private  sector.
section_title: Research Community Promotes Use of Best Practices for Developing Research Agendas and Evaluating Research Outcomes
paragraphs: Leading organizations that conduct scientific and engineering research,  other federal agencies with research programs, and experts in research  and technology have identified and use best practices for developing  research agendas and evaluating research outcomes. Although the  uncertain nature of research outcomes over time makes it difficult to set  specific, measurable program goals and evaluate results, the best practices  we identified are designed to ensure that the research objectives are  related to the areas of greatest interest and concern to research users and  that research is evaluated according to these objectives. These practices  include (1) developing research agendas through the involvement of  external stakeholders and (2) evaluation of research using techniques such  as expert review of the quality of research outcomes.
subsections: section_title: Developing Research Agendas Through the Involvement of External Stakeholders
paragraphs: External stakeholder involvement is particularly important for FHWA  because its research is expected to improve the construction, safety, and  operation of transportation systems that are primarily managed by others,  such as state departments of transportation. According to the  Transportation Research Board’s Research and Technology Coordinating  Committee, research has to be closely connected to its stakeholders to  help ensure relevance and program support, and stakeholders are more  likely to promote the use of research results if they are involved in the  research process from the start. The committee also identified merit  review of research proposals by independent technical experts based on  technical criteria as being necessary to help ensure the most effective use  of federal research funds. In 1999, we reported that other federal science  agencies—such as the Environmental Protection Agency and the National  Science Foundation—used such reviews to varying degrees to assess the  merits of competitive and noncompetitive research proposals. In April  2002, the Office of Management and Budget issued investment criteria for  federal research and technology program budgets that urge these agencies  to put into place processes to assure the relevance, quality and  performance of their programs. For example, the guidance requires these  programs to have agendas that are assessed prospectively and  retrospectively through external review to ensure that funds are being  expended on quality research efforts.
section_title: Evaluation of Research Using Systematic Approach to Review the Quality of Research Outcomes
paragraphs: The Committee on Science, Engineering, and Public Policy reported in  1999 that federal agencies that support research in science and  engineering have been challenged to find the most useful and effective  ways to evaluate the performance and results of the research programs  they support. Nevertheless, the committee found that research programs,  no matter what their character and goals, can be evaluated meaningfully  on a regular basis and in accordance with the Government Performance  and Results Act. Similarly, in April 2002 the Office of Management and  Budget  issued investment criteria for federal research and technology  program budgets that require these programs to define appropriate  outcome measures and milestones that can be used to track progress  toward goals and assess whether funding should be enhanced or  redirected. In addition, program quality should be assessed periodically in  relation to these criteria through retrospective expert review. The  Committee on Science, Engineering, and Public Policy also emphasized  that the evaluation methods must match the type of research and its  objectives, and it concluded that expert or peer review is a particularly  effective means to evaluate federally funded research.
Peer review is a process that includes an independent assessment of the  technical and scientific merit or quality of research by peers with essential  subject area expertise and perspective equal to that of the researchers.  Peer review does not require that the final impact of the research be  known. In 1999, we reported that federal agencies, such as the Department  of Agriculture, the National Institutes of Health, and the Department of  Energy, use peer review to help them (1) determine whether to continue  or renew research projects, (2) evaluate the results of research prior to  publication of those results, and (3) evaluate the performance of programs  and scientists. In its 1999 report, the Committee on Science, Engineering,  and Public Policy also stated that expert review is widely used to evaluate:  (1) the quality of current research as compared with other work being  conducted in the field, (2) the relevance of research to the agency’s goals  and mission, and (3) whether the research is at the “cutting edge.”
section_title: External Stakeholders’ Involvement in Developing FHWA’s Research Agendas Has Been Limited
paragraphs: Although FHWA engages external stakeholders in elements of its research  and technology program, the agency currently does not follow the best  practice of engaging external stakeholders on a consistent and transparent  basis in setting its research agendas. The agency expects each program  office to determine how or whether to involve external stakeholders in the  agenda setting process. As we reported in May 2002, FHWA acknowledges  that its approach to preparing research agendas is inconsistent and that  the associate administrators of FHWA’s program offices primarily use  input from the agency’s program offices, resource centers, and division  offices. Although agency officials told us that resource center and division  office staff provide the associate administrators with input based on their  interactions with external stakeholders, to the extent that external  stakeholder input into developing research agendas occurs, it is usually ad  hoc and provided through technical committees and professional  societies. For example, the agency’s agenda for environmental research  was developed with input from both internal sources (including DOT’s and  FHWA’s strategic plans and staff) and external sources (including the  Transportation Research Board’s reports on environmental research needs  and clean air, environmental justice leaders, planners, civil rights  advocates, and legal experts).
In our May 2002 report we recommended that FHWA develop a systematic  approach for obtaining input from external stakeholders in determining its  research and technology program’s agendas. FHWA concurred with our  recommendation and has taken steps to develop such an approach. FHWA  formed a planning group consisting of internal stakeholders as well as  representatives from the Research and Special Programs Administration  and the Pennsylvania Department of Transportation to determine how to  implement our recommendation. This planning group prepared a report  analyzing the approaches that four other federal agencies are taking to  involve external stakeholders in setting their research and technology  program agendas. Using the lessons learned from reviewing these other  agencies’ activities, FHWA has drafted a Corporate Master Plan for  Research and Deployment of Technology & Innovation. Under the draft  plan, the agency would be required to establish specific steps for including  external stakeholders in the agenda setting process for all areas of  research throughout the agency’s research and technology program by  fiscal year 2004. In drafting this plan, FHWA officials obtained input from  internal stakeholders as well as external stakeholders, including state  departments of transportation, academia, consultants, and members of the  Transportation Research Board. It appears that FHWA has committed to  taking the necessary steps to adopt the best practice of developing a  systematic process for involving external stakeholders in the agenda  setting process. The draft plan invites external stakeholders to assist  FHWA with such activities as providing focus and direction to the research  and technology program and setting the program’s agendas and priorities.  However, because FHWA’s plan has not been finalized, we cannot  comment on its potential effectiveness in involving external stakeholders.
section_title: FHWA Lacks a Systematic Approach to Evaluating Research Outcomes
paragraphs: As we reported last year, FHWA does not have an agency wide systematic  process to evaluate whether its research projects are achieving intended  results that uses such techniques as peer review. Although the agency’s  program offices may use methods such as obtaining feedback from  customers and evaluating outputs or outcomes versus milestones, they all  use success stories as the primary method to evaluate and communicate  research outcomes. According to agency officials, success stories are  examples of research results adopted or implemented by such  stakeholders as state departments of transportation. These officials told us  that success stories can document the financial returns on investment and  nonmonetary benefits of research and technology efforts. However, we  raised concerns that success stories are selective and do not cover the  breadth of FHWA’s research and technology program.
In 2001, the Transportation Research Board’s Research and Technology  Coordinating Committee concluded that peer or expert review is an  appropriate way to evaluate FHWA’s surface transportation research and  technology program. Therefore, the committee recommended a variety of  actions, including a systematic evaluation of outcomes by panels of  external stakeholders and technical experts to help ensure the maximum  return on investment in research. Agency officials told us that increased  stakeholder involvement and peer review will require significant  additional expenditures for the program. However, a Transportation  Research Board official told us that the cost of obtaining expert assistance  could be relatively low because the time needed to provide input would be  minimal and could be provided by such inexpensive methods as electronic  mail.
In our May 2002 report, we recommended that FHWA develop a systematic  process for evaluating significant ongoing and completed research that  incorporates peer review or other best practices in use at federal agencies  that conduct research. While FHWA has concurred that the agency must  measure the performance of its research and technology program, it has  not developed, defined or adopted a framework for measuring  performance. FHWA’s report on efforts of other federal agencies that  conduct research, discussed above, analyzed the approaches that four  other federal agencies are taking to evaluate their research and technology  programs using these best practices. According to FHWA’s assistant  director for Research, Technology, and Innovation Deployment, the  agency is using the results of this report to develop its own systematic  approach for evaluating its research and technology program. However,  this official noted that FHWA has been challenged to find the most useful  and effective ways to evaluate the performance and results of the agency’s  research and technology program. According to FHWA’s draft Corporate  Master Plan for Research and Deployment of Technology & Innovation,  FHWA is committed to developing a systematic method of evaluating its  research and technology program that includes the use of a merit review  panel. This panel would conduct evaluations and reviews in collaboration  with representatives from FHWA staff, technical experts, peers, special  interest groups, senior management, and contracting officers. According  to the draft plan, these merit reviews would be conducted on a periodic  basis for program-level and agency-level evaluations, while merit reviews  at the project level would depend on the project’s size and complexity.  FHWA is still in the process of developing, defining, and adopting a  framework for measuring performance. Therefore, we cannot yet  comment on how well FHWA’s efforts to evaluate research outcomes will  follow established best practices.
Mr. Chairman, this concludes my prepared statement. I would be pleased  to answer any questions that you or Members of the Committee may have.
section_title: Contacts and Acknowledgments
paragraphs: For further information on this testimony, please contact Katherine  Siggerud at (202) 512-2834 or siggerudk@gao.gov. Deena Richart made key  contributions to this testimony.
section_title: Appendix I: Roles of Program Offices in FHWA’s Research and Technology Program
paragraphs: FHWA’s research and technology program is based on the research and  technology needs of each of its program offices such as the Offices of  Infrastructure, Safety, and Policy. Each of the program offices is  responsible for identifying research needs, formulating strategies to  address transportation problems, and setting goals for research and  technology activities that support the agency’s strategic goals. (See table  1.)