Transportation Research: The Department of Transportation Has Made Progress in Coordinating and Reviewing Its Research Activities
Published: 2009-02-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Research, development, and technology (RD&T) activities are vital to meeting the Department of Transportation's (DOT) priorities, such as increasing safety, enhancing mobility, and supporting the nation's economic growth. In fiscal year 2008, the department's RD&T budget totaled over $1.1 billion, primarily for highway and aviation projects. Over the years, concerns have been raised about DOT's capabilities to improve RD&T coordination and evaluation efforts across the agency. In 2004, Congress created DOT's Research and Innovative Technology Administration (RITA) to coordinate and review the department's RD&T programs and activities for the purposes of reducing research duplication, enhancing opportunities for joint efforts, and ensuring RD&T activities are meeting goals. In 2006 GAO reported that RITA had made progress toward these ends, but needed to do more. GAO's testimony focuses on (1) the importance of coordinating and evaluating RD&T activities and (2) RITA's progress in implementing GAO's 2006 recommendations. GAO's statement is based on its 2006 report, a review of best practices for coordination and evaluation, and follow-up discussions with RITA officials on actions to implement GAO's recommendations. GAO did not assess whether RITA's actions have improved the effectiveness of the department's RD&T investment.
section_title: What GAO Found
paragraphs: Coordinating and evaluating research are important elements in ensuring that federal dollars are used efficiently and effectively. Coordinating research enhances collaboration, ensures that questions are explored, and reduces inefficiencies, such as from duplication of research. Evaluating research activities entails comparing research with established performance measures in agency strategic plans and using expert reviews to assess the quality of the research. With DOT's large RD&T budget--over $1.1 billion--coordination and evaluation are critical to making cost-effective investment choices in today's climate of expected trillion-dollar deficits. RITA has fully implemented five recommendations that GAO made in 2006 aimed at enhancing RITA's ability to manage and determine the effectiveness of RD&T activities, and partially implemented the remaining two. (See table below.) Regarding implemented recommendations, most notably, RITA has implemented a strategy to coordinate RD&T activities and look for areas where joint efforts would be appropriate. Results of its coordination efforts have identified a number of areas for cross-modal collaboration, including the areas of climate change and freight capacity. RITA has also developed a strategy to ensure that the results of DOT's research activities are evaluated against best practices, using governmentwide guidance and external stakeholder reviews. Regarding partially implemented recommendations, RITA has not yet developed an overall strategy, evaluation plan, or performance measures that delineate how its activities ensure the effectiveness of the department's RD&T investment. However, it has developed a process for doing so. In this regard, RITA plans to use an existing departmentwide strategic planning and budget process and collaborative meetings to develop an overall strategy and performance measures. RITA officials expect that it will fully implement activities related to this recommendation by 2012. GAO will continue to monitor RITA's activities.

section_title: Letter
paragraphs: We appreciate the opportunity to participate in this hearing on the  Department of Transportation’s (DOT) research, development, and  technology (RD&T) activities. RD&T activities are vital to meeting DOT’s  transportation priorities, such as increasing safety, enhancing mobility,  and supporting the nation’s economic growth. In fiscal year 2008, the  department’s RD&T budget totaled over $1.1 billion, primarily for projects  undertaken by DOT’s Federal Highway Administration and Federal  Aviation Administration. Coordinating RD&T throughout DOT and  reviewing it to make sure that it is evaluated is important to ensure the  efficiency and effectiveness of RD&T investment.
Over the years, we and others have raised concerns about DOT’s  capabilities for improving RD&T coordination and evaluation across the  agency. In part to ameliorate those concerns, in 2004 Congress created the  Research and Innovative Technology Administration (RITA). RITA is  responsible for coordinating, facilitating, and reviewing the department’s  RD&T programs and activities to identify research duplication and  opportunities for joint efforts and to ensure RD&T activities are meeting  intended or other goals. These include activities conducted by DOT’s  operating administrations as well as other RD&T and statistical programs  managed by RITA (e.g., the Bureau of Transportation Statistics and  University Transportation Centers). RITA carries out its responsibilities  through multiple groups and actions, including its two coordinating  bodies—the RD&T Planning Council and Planning Team—and budget  reviews. While RITA has DOT-wide responsibilities, it does not have the  authority to direct changes in the operating administrations’ RD&T  activities. In 2006, we reported on RITA’s progress in overseeing RD&T  activities and made seven recommendations to enhance RITA’s ability to  manage and ensure the effectiveness of these activities.
My testimony today addresses the importance of coordinating and  evaluating RD&T so that federal dollars are used efficiently and  effectively, as well as RITA’s progress in implementing our 2006  recommendations. It is based primarily on our 2006 report, a review of  best practices for coordination and evaluation, and follow-up discussions  with RITA officials on actions taken on our recommendations. We have  not assessed whether RITA’s actions have improved the effectiveness of  the department’s RD&T investment since our 2006 report. We conducted  this work in January and February 2009 in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives.
subsections: section_title: Coordination and Evaluation of RD&T Activities Help Promote Efficient and Effective Use of Federal Research Funds
paragraphs: Coordinating and evaluating research are important elements in ensuring  federal dollars are used efficiently and effectively. RITA is responsible for  coordinating and reviewing the DOT operating administrations’ RD&T  activities so that (1) no unnecessary duplication takes place and (2) the  activities have been evaluated in accordance with best practices. The  Committee on Science, Engineering, and Public Policy—a joint committee  of the National Academy of Sciences, the National Academy of  Engineering, and the Institute of Medicine—has emphasized the  importance of careful coordination and focused evaluation of federal  research and developed principles to help agencies evaluate their research  programs. The committee recommended establishing a formal process to  coordinate research across agencies. While this recommendation is  focused on cross-agency research, the goals—enhancing collaboration,  ensuring that questions are explored, and reducing inefficiencies—are  important and applicable within agencies as well. Coordination of  research ensures that information is shared so that, if necessary, research  can be adjusted to ensure a field is appropriately covered and understood.  In addition, the committee noted that evaluating research against  established performance measures in agency strategic plans, developing  measures that are appropriate for the type of research being developed,  and using expert reviews aid in assessing the quality of the research.
Relatedly, the Government Performance and Results Act of 1993 (GPRA)  requires federal agencies to set performance goals and measure  performance against those goals to ensure the effectiveness of federal  investments. GPRA’s emphasis on results implies that federal programs  contributing to the same or similar outcomes should be closely  coordinated to ensure that goals are consistent and complementary, and  that program efforts are mutually reinforcing.
Making appropriate and cost-effective investment choices is an essential  aspect of responsible fiscal stewardship. Such choices are even more  important in today’s climate of expected trillion-dollar deficits. Careful  decisions will need to be made to ensure that RD&T activities achieve  their intended (or other) purposes and do so efficiently and economically.
section_title: RITA Has Made Progress in Improving Its Coordination, Review, and Performance Measurement of DOT’s RD&T Programs
paragraphs: In 2006, we made seven recommendations to enhance RITA’s ability to  manage and ensure the effectiveness of RD&T activities, including  developing strategies for coordinating and reviewing RD&T activities and  developing performance goals and measures. (See table 1.) RITA has  implemented five of our recommendations and is making progress on  implementing the remaining two.
subsections: section_title: RITA Implemented a Coordination and Review Strategy, Developed a DOT-wide Database of RD&T Activities, and Communicated Results of Evaluations
paragraphs: Preventing duplication of effort. In response to our recommendation,  RITA developed a strategy to ensure that no unnecessary duplication of  research programs occurs within the department, incorporated the results  into various high-level DOT planning documents, and reported the results  in its strategic plan. RITA’s strategy consists of ongoing internal reviews of  all of DOT’s research programs. These reviews are conducted by (1)  convening meetings in which officials from each of the operating  administrations share information about areas of ongoing and planned  research, seeking opportunities for joint effort, and (2) conducting annual  reviews of each operating administration’s research plans, looking for  research duplication, among other things. In addition, RITA has formed  eight working groups, in concert with DOT’s operating administrations, to  foster collaboration on cross-modal issues. According to a RITA official,  results of these reviews have identified several areas for cross-modal  collaboration, including climate change, freight capacity, security,  alternative energy technologies, and advanced materials and sensors.  According to RITA officials, as a result of these actions, RITA is better able  to meet legislative and DOT requirements for coordinating its research,  leverage resources for cross-modal research initiatives, and prevent  unnecessary research duplication.
Following best practices. RITA also developed a strategy to ensure that  the results of all DOT’s research activities are evaluated according to  established best practices. The strategy includes three primary  mechanisms: (1) ensuring systematic application of the Office of  Management and Budget’s Research and Development Investment Criteria  (relevance, quality, and performance) and the Program Assessment Rating  Tool by the operating administrations; (2) annual internal program  reviews with self-reporting by the operating administrations; and (3)  documenting the operating administrations’ external stakeholder  coordination and review. According to RITA, reviews conducted in fiscal  years 2007 and 2008 focused on how well the operating administrations  are implementing best practices, including external stakeholder  involvement, merit review of competitive proposals, independent expert  review, research performance measures, and external research  coordination. RITA reports the results of its reviews to the department’s  RD&T Planning Council, which consists of administrators from each of the  operating administrations, including RITA, and officials from DOT’s Office  of the Secretary. According to RITA officials, as a result of these efforts,  RITA is better able to determine the quality and effectiveness of its  research activities and investments and determine whether they are  achieving their intended (or other) goals.
Establishing RD&T project databases. RITA created two database  systems to inventory and track all of DOT’s research activities and provide  tools for querying and searching individual projects to identify potential  duplication and areas where operating administrations could collaborate.  The first database, the RITA Research Notification System, captures  research investments at the transaction level, allowing users to search by  activity, contracts and grants, and contractor names, enabling  identification of funded programs for coordination, collaboration and  review. The second database is part of the annual Research Planning and  Investment Coordination (RPIC) process, which captures research at the  budget request level, allowing for departmentwide transparency and  coordination of proposed programs and projects. According to a RITA  official, eventual combination of the two databases will offer a mechanism  for measuring and tracking investments from request through funding and  execution.
Communicating evaluation efforts. To communicate its efforts in  evaluating DOT’s research to Congress, senior DOT officials, and the  transportation community, RITA and its predecessor organization  published a summary of all research program evaluations for 2004 through  2006 and included that summary in a high-level DOT planning document  and in a report to Congress. First, RITA’s predecessor published what was  essentially a summary of all research program evaluations conducted in  fiscal year 2004—in the form of a summary of the results of its review of  the operating administrations’ application of the Office of Management  and Budget’s Research and Development Investment Criteria—in its 2005  annual RD&T plan. Secondly, RITA developed a summary of the results of  its fiscal year 2005 and 2006 research program reviews, and a schedule of  RITA’s planned fiscal year 2007 reviews, and included it in DOT’s  “Research, Development and Technology Annual Funding Fiscal Years  2006-2008, A Report to Congress.” This report also includes summaries of  research program evaluations conducted by modal research advisory  committees, the Transportation Research Board, and key modal  stakeholders in fiscal years 2006 and 2007. According to RITA officials, as  a result of this reporting, RITA has provided better continuity and context  to Congress and the transportation community about the results of its  research evaluations.
Documenting processes. RITA has also acted to document its process  for systematically evaluating the results of its own multimodal research  programs, such as the Hydrogen Safety Program and various grant  programs. RITA evaluates the results of its RD&T activities by ensuring  they align with DOT goals, meet the research and development investment  criteria, and are subject to an annual peer review process. RITA has  documented this process in its strategic plan.
section_title: RITA Has Not Yet Developed an Overall Implementing Strategy, Evaluation Plan, or Performance Measures
paragraphs: Establishing performance goals. In 2006, we found that RITA lacked  performance goals and an implementing strategy and evaluation plan to  delineate how the activities and results of its coordination, facilitation, and  review practices will further DOT’s mission and ensure the effectiveness  of the department’s RD&T investment. RITA has partially implemented our  recommendation that it develop these elements. Setting meaningful goals  for performance, and using performance information to measure  performance against those goals, is consistent with requirements in GPRA.  Developing an evaluation plan and analyzing performance information  against set goals for its own coordination, facilitation, and review  practices could assist RITA in identifying any problem areas and taking  corrective actions. Linking performance goals with the planning and  budget process, such as DOT’s annual budget process, can also help RITA  determine where to target its resources to improve performance.  Guidance provided by the Committee on Science, Engineering, and Public  Policy notes that evaluating the performance of research in the context of  the strategic planning process ensures the research is relevant to the  agency’s mission. Without such goals and an evaluation plan, it is difficult  for RITA to determine its success in overseeing the effectiveness of DOT’s  RD&T activities.
According to RITA officials, while an overall implementing strategy and  evaluation plan has not yet been established, RITA has created  performance goals. A RITA official told us that the RPIC process—a  relatively new process that integrates the budget and strategic planning  processes—will help in creating an implementing strategy. The RPIC  process is meant to provide information to the Planning Council and  Planning Team, which is responsible for defining the department’s overall  RD&T strategic objectives. The RPIC process assesses the department’s  RD&T activities in terms of the following performance goals: (1) balanced  portfolio (e.g., mix of basic, applied, developmental, and high risk RD&T),  (2) alignment of RD&T programs with DOT goals and each operating  administration’s mission, and (3) return on investment. The RPIC process  has been in place only for fiscal year 2009, and as a result, the Planning  Council does not yet have the information needed to make decisions about  a strategy. In addition, RITA does not yet have an evaluation plan to  monitor and evaluate whether it is achieving its goals. A RITA official told  us that the RPIC process needs to be in place for 2 or 3 fiscal years before  it can provide enough information for RITA to establish a strategy or  evaluation plan.
Developing performance measures. In 2006, we also found that RITA  did not work with the operating administrations to develop common  performance measures for DOT’s RD&T activities. According to RITA  officials, RITA has partially implemented our recommendation that it do  so. Without common performance measures for the RD&T activities of the  operating administrations, RITA and the operating administrations lack  the means to monitor and evaluate the collective results of those activities  and determine that they are achieving their intended (or other) results and  furthering DOT-wide priorities. In response to our recommendation, RITA  officials told us that they are working with the operating administrations  through the RD&T Planning Team—made up of senior officials in RITA  and each of the operating administrations. During Planning Team  meetings, representatives from each of the operating administrations  share information about how RD&T projects are measured and prioritized.  For example, according to a RITA official, the Federal Railroad  Administration measures how frequently its RD&T projects are used in  real-world applications. Once representatives from each operating  administration have had the chance to share information, RITA officials  will then look for commonalities and determine whether any of the  measures could be adopted for the department’s RD&T activities.
In closing, since it became operational in 2005, RITA has taken a number  of positive steps to meet its vision of becoming a DOT-wide resource for  managing and ensuring the effectiveness of RD&T activities. While we  have not assessed the effectiveness of these efforts since our 2006 report,  we believe that RITA has made progress. We will continue to monitor  RITA’s performance in implementing our recommendations. As  reauthorization approaches, we look forward to assisting Congress as it  considers RITA’s management of DOT’s research program, to better  ensure that taxpayers receive the maximum value for DOT’s RD&T  investment.
Mr. Chairman, this concludes my prepared statement. I would be pleased  to respond to any questions that you or other members of the  subcommittee might have.
section_title: Contact and Acknowledgments
paragraphs: For further information regarding this statement, please contact David  Wise at (202) 512-2834 or wised@gao.gov. Individuals who made key  contributions to this statement are Michelle Everett, Colin Fallon, Erin  Henderson, and James Ratzenberger.
This is a work of the U.S. government and is not subject to copyright protection in the  United States. The published product may be reproduced and distributed in its entirety  without further permission from GAO. However, because this work may contain  copyrighted images or other material, permission from the copyright holder may be  necessary if you wish to reproduce this material separately.