Transportation Research: The Department of Transportation Has Made Progress in Coordinating and Reviewing Its Research Activities
Released: 2009-02-12T00:00:00

Letter

Coordination and Evaluation of RD&T Activities Help Promote Efficient and Effective Use of Federal Research Funds

Coordinating and evaluating research are important elements in ensuring federal dollars are used efficiently and effectively. RITA is responsible for coordinating and reviewing the DOT operating administrations’ RD&T activities so that (1) no unnecessary duplication takes place and (2) the activities have been evaluated in accordance with best practices. The Committee on Science, Engineering, and Public Policy—a joint committee of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine—has emphasized the importance of careful coordination and focused evaluation of federal research and developed principles to help agencies evaluate their research programs. The committee recommended establishing a formal process to coordinate research across agencies. While this recommendation is focused on cross-agency research, the goals—enhancing collaboration, ensuring that questions are explored, and reducing inefficiencies—are important and applicable within agencies as well. Coordination of research ensures that information is shared so that, if necessary, research can be adjusted to ensure a field is appropriately covered and understood. In addition, the committee noted that evaluating research against established performance measures in agency strategic plans, developing measures that are appropriate for the type of research being developed, and using expert reviews aid in assessing the quality of the research.

Relatedly, the Government Performance and Results Act of 1993 (GPRA) requires federal agencies to set performance goals and measure performance against those goals to ensure the effectiveness of federal investments. GPRA’s emphasis on results implies that federal programs contributing to the same or similar outcomes should be closely coordinated to ensure that goals are consistent and complementary, and that program efforts are mutually reinforcing.

Making appropriate and cost-effective investment choices is an essential aspect of responsible fiscal stewardship. Such choices are even more important in today’s climate of expected trillion-dollar deficits. Careful decisions will need to be made to ensure that RD&T activities achieve their intended (or other) purposes and do so efficiently and economically.

RITA Has Made Progress in Improving Its Coordination, Review, and Performance Measurement of DOT’s RD&T Programs

In 2006, we made seven recommendations to enhance RITA’s ability to manage and ensure the effectiveness of RD&T activities, including developing strategies for coordinating and reviewing RD&T activities and developing performance goals and measures. (See table 1.) RITA has implemented five of our recommendations and is making progress on implementing the remaining two.

RITA Implemented a Coordination and Review Strategy, Developed a DOT-wide Database of RD&T Activities, and Communicated Results of Evaluations

Preventing duplication of effort. In response to our recommendation, RITA developed a strategy to ensure that no unnecessary duplication of research programs occurs within the department, incorporated the results into various high-level DOT planning documents, and reported the results in its strategic plan. RITA’s strategy consists of ongoing internal reviews of all of DOT’s research programs. These reviews are conducted by (1) convening meetings in which officials from each of the operating administrations share information about areas of ongoing and planned research, seeking opportunities for joint effort, and (2) conducting annual reviews of each operating administration’s research plans, looking for research duplication, among other things. In addition, RITA has formed eight working groups, in concert with DOT’s operating administrations, to foster collaboration on cross-modal issues. According to a RITA official, results of these reviews have identified several areas for cross-modal collaboration, including climate change, freight capacity, security, alternative energy technologies, and advanced materials and sensors. According to RITA officials, as a result of these actions, RITA is better able to meet legislative and DOT requirements for coordinating its research, leverage resources for cross-modal research initiatives, and prevent unnecessary research duplication.

Following best practices. RITA also developed a strategy to ensure that the results of all DOT’s research activities are evaluated according to established best practices. The strategy includes three primary mechanisms: (1) ensuring systematic application of the Office of Management and Budget’s Research and Development Investment Criteria (relevance, quality, and performance) and the Program Assessment Rating Tool by the operating administrations; (2) annual internal program reviews with self-reporting by the operating administrations; and (3) documenting the operating administrations’ external stakeholder coordination and review. According to RITA, reviews conducted in fiscal years 2007 and 2008 focused on how well the operating administrations are implementing best practices, including external stakeholder involvement, merit review of competitive proposals, independent expert review, research performance measures, and external research coordination. RITA reports the results of its reviews to the department’s RD&T Planning Council, which consists of administrators from each of the operating administrations, including RITA, and officials from DOT’s Office of the Secretary. According to RITA officials, as a result of these efforts, RITA is better able to determine the quality and effectiveness of its research activities and investments and determine whether they are achieving their intended (or other) goals.

Establishing RD&T project databases. RITA created two database systems to inventory and track all of DOT’s research activities and provide tools for querying and searching individual projects to identify potential duplication and areas where operating administrations could collaborate. The first database, the RITA Research Notification System, captures research investments at the transaction level, allowing users to search by activity, contracts and grants, and contractor names, enabling identification of funded programs for coordination, collaboration and review. The second database is part of the annual Research Planning and Investment Coordination (RPIC) process, which captures research at the budget request level, allowing for departmentwide transparency and coordination of proposed programs and projects. According to a RITA official, eventual combination of the two databases will offer a mechanism for measuring and tracking investments from request through funding and execution.

Communicating evaluation efforts. To communicate its efforts in evaluating DOT’s research to Congress, senior DOT officials, and the transportation community, RITA and its predecessor organization published a summary of all research program evaluations for 2004 through 2006 and included that summary in a high-level DOT planning document and in a report to Congress. First, RITA’s predecessor published what was essentially a summary of all research program evaluations conducted in fiscal year 2004—in the form of a summary of the results of its review of the operating administrations’ application of the Office of Management and Budget’s Research and Development Investment Criteria—in its 2005 annual RD&T plan. Secondly, RITA developed a summary of the results of its fiscal year 2005 and 2006 research program reviews, and a schedule of RITA’s planned fiscal year 2007 reviews, and included it in DOT’s “Research, Development and Technology Annual Funding Fiscal Years 2006-2008, A Report to Congress.” This report also includes summaries of research program evaluations conducted by modal research advisory committees, the Transportation Research Board, and key modal stakeholders in fiscal years 2006 and 2007. According to RITA officials, as a result of this reporting, RITA has provided better continuity and context to Congress and the transportation community about the results of its research evaluations.

Documenting processes. RITA has also acted to document its process for systematically evaluating the results of its own multimodal research programs, such as the Hydrogen Safety Program and various grant programs. RITA evaluates the results of its RD&T activities by ensuring they align with DOT goals, meet the research and development investment criteria, and are subject to an annual peer review process. RITA has documented this process in its strategic plan.

RITA Has Not Yet Developed an Overall Implementing Strategy, Evaluation Plan, or Performance Measures

Establishing performance goals. In 2006, we found that RITA lacked performance goals and an implementing strategy and evaluation plan to delineate how the activities and results of its coordination, facilitation, and review practices will further DOT’s mission and ensure the effectiveness of the department’s RD&T investment. RITA has partially implemented our recommendation that it develop these elements. Setting meaningful goals for performance, and using performance information to measure performance against those goals, is consistent with requirements in GPRA. Developing an evaluation plan and analyzing performance information against set goals for its own coordination, facilitation, and review practices could assist RITA in identifying any problem areas and taking corrective actions. Linking performance goals with the planning and budget process, such as DOT’s annual budget process, can also help RITA determine where to target its resources to improve performance. Guidance provided by the Committee on Science, Engineering, and Public Policy notes that evaluating the performance of research in the context of the strategic planning process ensures the research is relevant to the agency’s mission. Without such goals and an evaluation plan, it is difficult for RITA to determine its success in overseeing the effectiveness of DOT’s RD&T activities.

According to RITA officials, while an overall implementing strategy and evaluation plan has not yet been established, RITA has created performance goals. A RITA official told us that the RPIC process—a relatively new process that integrates the budget and strategic planning processes—will help in creating an implementing strategy. The RPIC process is meant to provide information to the Planning Council and Planning Team, which is responsible for defining the department’s overall RD&T strategic objectives. The RPIC process assesses the department’s RD&T activities in terms of the following performance goals: (1) balanced portfolio (e.g., mix of basic, applied, developmental, and high risk RD&T), (2) alignment of RD&T programs with DOT goals and each operating administration’s mission, and (3) return on investment. The RPIC process has been in place only for fiscal year 2009, and as a result, the Planning Council does not yet have the information needed to make decisions about a strategy. In addition, RITA does not yet have an evaluation plan to monitor and evaluate whether it is achieving its goals. A RITA official told us that the RPIC process needs to be in place for 2 or 3 fiscal years before it can provide enough information for RITA to establish a strategy or evaluation plan.

Developing performance measures. In 2006, we also found that RITA did not work with the operating administrations to develop common performance measures for DOT’s RD&T activities. According to RITA officials, RITA has partially implemented our recommendation that it do so. Without common performance measures for the RD&T activities of the operating administrations, RITA and the operating administrations lack the means to monitor and evaluate the collective results of those activities and determine that they are achieving their intended (or other) results and furthering DOT-wide priorities. In response to our recommendation, RITA officials told us that they are working with the operating administrations through the RD&T Planning Team—made up of senior officials in RITA and each of the operating administrations. During Planning Team meetings, representatives from each of the operating administrations share information about how RD&T projects are measured and prioritized. For example, according to a RITA official, the Federal Railroad Administration measures how frequently its RD&T projects are used in real-world applications. Once representatives from each operating administration have had the chance to share information, RITA officials will then look for commonalities and determine whether any of the measures could be adopted for the department’s RD&T activities.

In closing, since it became operational in 2005, RITA has taken a number of positive steps to meet its vision of becoming a DOT-wide resource for managing and ensuring the effectiveness of RD&T activities. While we have not assessed the effectiveness of these efforts since our 2006 report, we believe that RITA has made progress. We will continue to monitor RITA’s performance in implementing our recommendations. As reauthorization approaches, we look forward to assisting Congress as it considers RITA’s management of DOT’s research program, to better ensure that taxpayers receive the maximum value for DOT’s RD&T investment.

Mr. Chairman, this concludes my prepared statement. I would be pleased to respond to any questions that you or other members of the subcommittee might have.

Contact and Acknowledgments

For further information regarding this statement, please contact David Wise at (202) 512-2834 or wised@gao.gov. Individuals who made key contributions to this statement are Michelle Everett, Colin Fallon, Erin Henderson, and James Ratzenberger.

This is a work of the U.S. government and is not subject to copyright protection in the United States. The published product may be reproduced and distributed in its entirety without further permission from GAO. However, because this work may contain copyrighted images or other material, permission from the copyright holder may be necessary if you wish to reproduce this material separately.