Transportation Research: Actions Needed to Improve Coordination and Evaluation of Research
Published: 2003-05-01T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Research and Special Programs Administration (RSPA) within the Department of Transportation (DOT) is responsible for coordinating and ensuring the evaluation of DOT research programs to promote the efficient use of departmental research funds, which in fiscal year 2002 totaled over $1 billion. RSPA is also responsible for conducting multimodal research that cuts across different modes of transportation. The House Committee on Appropriations directed GAO to examine RSPA's coordination and evaluation of research within DOT and the status of its own multimodal research.
section_title: What GAO Found
paragraphs: RSPA has met some, but not all, legislative and DOT requirements pertaining to the coordination of departmental research efforts. For example, while RSPA develops an annual plan and meets monthly with other DOT research officials, RSPA does not review the status of all DOT research activities. Thus, it cannot determine whether duplication of research efforts within DOT does or does not occur. Additionally, RSPA has not developed standards against which to measure its performance in coordinating research within DOT. Moreover, RSPA has not fully met all legislative and DOT requirements to measure research results and oversee research evaluations across DOT. RSPA officials cited a lack of ready information on DOT research activities budget constraints and a lack of authority over other DOT agencies as reasons why they served primarily an information-sharing role, rather than as an overseer and manager of the coordination and evaluation processes. Since 1999, RSPA has budgeted $37 million to conduct four major research programs with applicability to more than one mode of transportation--for example, using technology to improve energy efficiency and reduce emissions and transportation dependence on petroleum. According to the Associate Administrator for Innovation, Research, and Education, RSPA's current multimodal research programs are scheduled for completion by the end of fiscal year 2004 and have had a variety of positive results. However, RSPA does not have an evaluation process to systematically evaluate the results of its multimodal research programs.

section_title: Letter
paragraphs: In fiscal year 2002, the Department of Transportation’s (DOT) research and  development budget totaled more than $1 billion. This sum supported the  many individual projects undertaken by the Federal Highway  Administration, the Federal Aviation Administration, and the other  operating administrations that constitute DOT. This research is vital to  meeting the department’s key transportation priorities, such as increasing  transportation safety, enhancing mobility for all Americans, supporting the  nation’s economic growth, and protecting the environment. The Congress  has recognized the importance of coordinating and evaluating research  throughout DOT and established requirements in the Transportation Equity  Act for the 21st Century to ensure that those tasks are accomplished. In  turn, the department has given responsibility to the Research and Special  Programs Administration’s Office of Innovation, Research, and Education  (hereafter referred to as RSPA) for coordinating, and ensuring the  evaluation of, DOT research programs to promote the efficient use of  research funds. RSPA is additionally responsible for conducting  multimodal research—research that applies to more than one mode of  transportation—for the department that contributes to the safe, effective,  and efficient transportation of people and goods.
In House Report 107-722, accompanying the DOT and Related Agencies  Appropriations Bill for Fiscal Year 2003, the House Committee on  Appropriations, directed us to examine RSPA’s role in coordinating  research activities and conducting multimodal research throughout the  department. Specifically, in subsequent discussions with Committee staff,  we agreed to address the following questions:  (1) To what extent has RSPA  fulfilled requirements for coordinating DOT research efforts? (2) To what  extent has RSPA fulfilled requirements for evaluating research within DOT?  and (3) What types of multimodal research has RSPA conducted since 1999,  and what have been the results?
To address questions regarding RSPA’s efforts to coordinate and evaluate  the department’s research efforts, we examined pertinent legislation, DOT  policy guidance, and DOT performance reports and plans as well as reports  and documents provided by RSPA, including the department’s Research,  Development, and Technology Plan. Although we did not review the  individual research programs and agendas of each DOT modal  administration, we reviewed external assessments conducted by us and the  National Research Council’s Transportation Research Board regarding  RSPA’s role and efforts in coordinating DOT research. Further, we  interviewed RSPA officials, including RSPA’s Associate Administrator for  Innovation, Research, and Education (hereafter Associate Administrator);  officials from the Volpe National Transportation Systems Center in  Cambridge, Massachusetts; and all of the members of the department’s  Research and Technology Coordinating Council (hereafter Coordinating  Council), to discuss research coordination efforts and identify potential  improvements. At the time of our review, the council was made up of 15  members representing the department’s Office of the Assistant Secretary  for Transportation Policy, Office of Intelligence and Security, Office of the  Assistant Secretary for Budget, Office of Intermodalism, United States  Coast Guard, Federal Aviation Administration, Federal Highway  Administration, Federal Railroad Administration, National Highway Traffic  Safety Administration, Federal Transit Administration, Maritime  Administration, RSPA, Bureau of Transportation Statistics, Federal Motor  Carrier Safety Administration, and Transportation Security  Administration.  In addition, to determine the types and status of  multimodal research that RSPA conducted, we (1) reviewed and analyzed  RSPA budget data from fiscal years 1999 through 2003 and (2) reviewed  RSPA’s multimodal project plans agreements and published project results  for the same period. (One of RSPA’s multimodal research programs—the  Transportation Infrastructure Assurance Program—is also the subject of a  separate GAO review.)
We conducted our review from September 2002 through February 2003 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: RSPA has both legislative and departmental responsibilities for  coordinating and evaluating DOT’s research and development programs,  which, in fiscal year 2002, amounted to about $1 billion. The Transportation  Equity Act for the 21st Century (TEA-21) made DOT responsible for  establishing a strategic plan for surface transportation research. The plan is  to include a discussion of efforts to coordinate federal surface  transportation research and technology development activities to avoid  unnecessary duplication of effort. It also is to contain a description of  program evaluations and a schedule for future evaluations of DOT research  projects, among other things. DOT policy delegates the responsibility for  meeting this and other legislative mandates related to research and  development activities to RSPA’s Associate Administrator as chair of the  department’s Coordinating Council. Moreover, DOT policy requires RSPA’s  Associate Administrator, through the Coordinating Council, to coordinate  all research activities by developing processes for sharing information  about research and technology and reviewing the status of all research and  technology projects throughout DOT.
RSPA’s Office of Innovation, Research, and Education had nine full-time  employees and a budget of about $560,000 for fiscal year 2003. In addition  to coordination and evaluation duties, RSPA manages and supports a  variety of other programs for the department, including its Technology  Transfer and Technology Sharing Programs, Small Business Innovation  Research Program, and University Transportation Centers Program.  RSPA  also conducts research on multimodal issues that affect the U.S.  transportation system as a whole in contrast to the other operating  administrations within DOT that focus on specific sectors of the U.S.  transportation system. In fiscal year 2003, RSPA conducted and managed  four major multimodal research programs. Participants in RSPA’s  multimodal research programs include stakeholders from the departments’  operating administrations; other federal departments and agencies; state  departments of transportation; private and state universities; private-sector  partners; and various consortia. (See app. I for a listing of project  stakeholders and researchers involved in RSPA’s multimodal research  programs.)
section_title: RSPA Has Met Some, but Not All, Legislative or DOT Requirements for Coordinating DOT Research Efforts
paragraphs: Although RSPA has developed an annual plan and taken other steps to  facilitate research coordination, it has not fully met legislative and DOT  requirements for coordinating departmental research. Figure 1 summarizes  these requirements and the extent to which RSPA has met them.
subsections: section_title: RSPA Facilitates Research Coordination by Developing an Annual Plan and Conducting Monthly Meetings
paragraphs: RSPA has met its legislative requirement under TEA-21 to develop an  annual surface transportation research and technology development plan  to coordinate and document research efforts. The plan covers not only  surface transportation but also air and maritime. Now in its fourth edition,  the plan discusses state and local transportation research activities;  describes each operating administration’s mission; and conveys priorities  for the department’s research activities, including identifying examples of  research programs that are necessary to achieve the department’s strategic  goals.  According to the Associate Administrator, the plan is an important  resource for the department’s budget and program deployment processes  and helps to avoid unnecessary duplication of research among the  operating administrations. In March 2000, the National Research Council,  acting through the Transportation Research Board (TRB), assessed DOT’s  strategic planning process, including the first edition of its research plan,  and generally commended RSPA’s efforts in coordinating the department’s research activities.  In September 2001, we reported that the department’s  research plan achieved a number of important strategic functions,  including supporting the department's budget and program development  process, conveying priorities, linking research and development initiatives  occurring throughout the department to specific strategic goals, and  focusing on research and technology needs that concern the department as  a whole. However, upon reviewing the fiscal year 2003 research plan, we  found that it does not summarize results of the previous year’s surface  transportation research programs. Such a summary is also absent from the  department’s overall performance plan and reports, where it is required by  TEA-21.  Since the explicit intent of the research plan is to focus the  department’s research efforts, and the department is required to gather  summary information on the previous years’ research results, this  information could provide continuity and context for observations about  planned research for future years.
The research plan is the culmination of monthly Coordinating Council  meetings in which the Associate Administrator meets with members from  each of DOT’s operating administrations as well as representatives from  DOT’s Office of Policy, Office of Budget, Office of Intelligence and Security,  and Office of Intermodalism. These meetings also serve as a means to fulfill  the legislative requirement to coordinate surface transportation research  and technology development and the DOT requirement to coordinate  research across all modes within DOT. Agendas and minutes from these  meetings indicate that the Coordinating Council’s discussions involve  research processes, procedures, policy, and presentations from council  members and guest speakers. According to the Associate Administrator,  these meetings further provide a forum for networking among DOT  researchers, a venue for DOT operating administrations to learn about each  other’s research, and an opportunity for information sharing and  technology transfer. In addition, in instances where research has  multimodal applicability, the Associate Administrator said RSPA staff and  representatives from the other operating administrations have been active  in sharing information through working groups, such as the Human Factors  Coordinating Committee, which shares information on research conducted  by each of the operating administrations focusing on identifying ways to  better manage human operator fatigue.
section_title: RSPA Has Not Reviewed All DOT Research Projects to Identify Unnecessary Duplication
paragraphs: RSPA does not fully meet its legislative requirement to coordinate surface  transportation and technology development activities because it does not  review all surface transportation research projects to determine whether  surface transportation researchers within DOT are unnecessarily  duplicating research efforts. A similar DOT requirement broadens this  responsibility to make RSPA responsible for reviewing the status of all  research projects throughout DOT for the same purpose. In its March 2000  report, TRB reported the absence of information on the current status of all  DOT research programs in the department’s annual research plan.  We  discovered that this information also remained absent from DOT’s fiscal  year 2003 research plan. The Associate Administrator said that RSPA  lacked (1) readily accessible data on the research activities of other parts  of DOT and (2) the staff and resources to review all research projects  across the department to, at a minimum, identify and report on the extent  of unnecessary duplication, if any, across the department. He also said that  even if RSPA reviewed all of the department’s research activities and  identified any unnecessary duplication of effort, RSPA does not have the  program and budget authority to direct changes in other operating  administrations’ research activities.
RSPA’s Associate Administrator explained that RSPA does not have readily  accessible data on the research activities of other parts of DOT because  efforts to develop a computer-based tracking system to share DOT research  program and budget information have stalled. RSPA’s 1998 strategic plan  discussed the need to create and deploy such a system to meet its strategic  goal for coordinating research and technology. In fiscal year 1999, RSPA  planned to allocate $200,000 annually for 4 years to develop and implement  a system. According to the Associate Administrator, as of January 2003,  RSPA had spent about $500,000 of its allocation and completed  development of a prototype database. He said that implementing the  centralized information system would (1) make basic project information  (such as project methodologies, funding levels and sources, schedules, and  planned products) across the department more accessible; (2) provide  greater levels of organization and clarity on historical research; (3)  facilitate strategic planning and coordination; and (4) improve the  department’s annual research plan by providing decision-makers with more  complete, accurate, and timely information on all DOT research activities.  According to the Associate Administrator and Coordinating Council  members, some operating administrations do not support implementation  of the system because they believe that the system would put additional  demands on limited resources and would produce little in terms of tangible  results. Coordinating Council members also said it would duplicate  existing information systems already in place at some operating  administrations and the new system would not be integrated into their  other, modal-unique information systems (such as budget and accounting  information systems). According to the RSPA official in charge of  developing the database, each operating administration would require up  to approximately 2 full-time employees for up to 1 year to input the  historical research project data going back 5 years, and an additional ½ to 1  full-time employees per year to manage and update the database.
The Associate Administrator also said that RSPA did not review the status  of all operating administrations’ research projects to identify any  unnecessary duplication because his office lacks sufficient staff and  resources to do so. He noted, for example, that RSPA’s total research and  technology budget for fiscal year 1999 was about $3.7 million—of which  $2.2 million was allocated for research and development planning and  management activities—and 13 full-time employees. However, in fiscal  year 2003, this decreased to a total budget of about $2.9 million—of which  $560,000 was allocated for research and development planning and  management activities—and 9 full-time employees.  The Associate  Administrator said that the decline in RSPA’s staff and resources—the only  such staff and resources in the department for conducting long-term  transportation research planning and coordinating research plans and  programs—has also severely limited RSPA’s efforts to coordinate with  transportation research stakeholders outside of DOT, such as state, local,  and other federal agencies. Figure 2 shows RSPA’s funding levels for its  research and development planning and management activities from fiscal  years 1999 through 2003.
RSPA’s Associate Administrator said he believes that little or no duplication  of research activities occurs. He said that, because the monthly  Coordinating Council meetings provided a forum for discussing ongoing  and planned research, unnecessary duplication of research efforts would  be identified. Also, he said that most DOT operating administrations have  discrete research programs and budgets that support their mode-specific  regulatory and safety mandates. For example, the Federal Highway  Administration research focuses on public roads and highways, and its  primary users are state and local transportation departments that seek  better ways to repair the public infrastructure and find improved materials  for pavements. Similarly, the Federal Railroad Administration focuses on  the rail industry’s privately owned infrastructure and these owners—freight  railroads, Amtrak, commuter railroads, and shippers—look to Federal  Railroad Administration to conduct research that will reduce track failure,  equipment failure, and human error.
According to members of the Coordinating Council, apparent duplication  in research programs might reflect a number of conditions—for example,  research that was intended to validate previous research results, expand  research applications, and address different needs (such as pavement  research for airport runways and highways)—or an effort to explore  alternative approaches before selecting one for further development. None  of the council members, however, could provide us with specific examples  of research projects that reflected these conditions.
Other Coordinating Council members with whom we spoke, however, said  that the council should take further steps to more effectively coordinate  DOT research. For example, one member said the Coordinating Council  should review all of the current projects across the department to improve  the level and quality of the department’s research coordination efforts.  Another member said that the primary functions of the Coordinating  Council should be to universally review DOT research projects to eliminate  unnecessary duplication of effort and to provide opportunities for joint  research partnerships on similar work. Such opportunities might include  research focusing on safety, environmental, training, and human factor  issues. However, Coordinating Council members said that DOT support for  the council and research coordination in general had declined in recent  years (e.g., lower ranking members of the operating administrations attend  the meetings, instead of the more senior personnel that had once attended,  and meetings were shortened from 2 hours to 1 hour per month), and that  greater departmental support for this effort was warranted.
The Associate Administrator said that RSPA did not have the program and  budget authority over the department’s operating administrations’ research  activities to direct changes in research programs, even if opportunities for  greater joint efforts or elimination of unnecessary duplicative research  were found. In a 1996 report examining the status of the department’s  coordination of surface transportation research, we identified RSPA’s lack  of internal clout within the department as an obstacle to its ability to  function effectively as its research strategic planner because it had no  direct program or budget authority over the department’s operating  administrations’ research activities. Although DOT proposed the creation  of such an organization to have budgeting and program authority for DOT  research in its fiscal year 1996 budget submission, the Congress did not  approve the agency’s proposal. According to the Associate Administrator,  there are no current legislative or budget initiatives to pursue this proposal.
Although DOT’s earlier effort to overcome RSPA’s lack of internal authority  was not implemented, neither the department nor RSPA has developed  alternative approaches to overcome this obstacle as well as to address the  information and resource limitations that continue to hinder RSPA’s efforts  to meet legislative and DOT requirements for coordinating departmental  research. Developing a strategy that incorporates information about the  costs involved in reviewing research projects throughout DOT to ensure  that unnecessary duplication does not occur, and that determines whether  finalizing the development and implementation of the DOT-wide research  tracking system database could serve this purpose, is an important first  step for RSPA to meet the legislative and DOT requirements entrusted to it.  A strategy is also critical for communicating to the Congress and the  Secretary of Transportation the challenges RSPA faces, and the specific  actions it can take, in meeting the requirements with the resources it  possesses.
section_title: RSPA Has Not Developed Performance Standards Against Which to Measure Its Coordination Efforts
paragraphs: RSPA has not established performance measures to systematically  document the results and benefits of coordinating DOT research  activities. In the absence of systematically gathered data on research  activities across the department and associated performance measures, it  is difficult to determine RSPA’s overall success in coordinating DOT’s  billion-dollar research program. Demonstrated successes could garner  greater departmental support for RSPA’s research coordination efforts. In  its fiscal year 2003 budget submission, RSPA cited the difficulty in defining  and measuring the effectiveness of research coordination activities.  According to RSPA, it is because of this difficulty that it relies upon  external program assessments to provide independent evaluation of its  research and coordination activities.
Although we support the use of external assessments, we have reported  that quantifiable measures are necessary to assess agency performance to  meet the intent of the Government Performance and Results Act of 1993  (GPRA). Among the stated purposes of GPRA is the improvement of  federal program effectiveness and public accountability. For agencies to  successfully become high-performing organizations, their leaders need to  foster performance-based cultures, find ways to measure performance, and  use performance information to make decisions. A fundamental element in  an organization’s efforts to manage for results is its ability to set meaningful  goals for performance and, using performance information, measure  performance against those goals. High-performing, results-oriented  organizations establish a set of measures to gauge progress over various  dimensions of performance.
In crafting GPRA, the Congress expressed its interest in American  taxpayers’ getting quality results from the programs they pay for as well as  its concern about waste and inefficiency in federal programs. The  fundamental reason for collecting information on a program’s performance  is to take action in managing the program on the basis of that information.  By using performance information to assess the way a program is  conducted, managers can consider alternative approaches and processes in  areas where goals are not being met and enhance the use of program  approaches and processes that are working well. Performance information  also allows program managers to compare their programs’ results with  goals and thus determine where to target program resources to improve  performance. When managers are forced to reduce their resources, the  same analysis can help them target the reductions to minimize the impact  on program results.
GPRA’s emphasis on results implies that federal programs contributing to  the same or similar outcomes should be closely coordinated to ensure that  goals are consistent and complementary, and that program efforts are  mutually reinforcing. Thus, measuring the effectiveness of RSPA’s  coordination of DOT research is a critical element of fulfilling its legislative  and departmental coordination responsibilities—an element RSPA has not  yet addressed.
section_title: RSPA Has Met Some, but Not All, Legislative or DOT Requirements to Evaluate DOT Research
paragraphs: RSPA has not fully met all legislative and DOT requirements for evaluating  research within the department. RSPA does not meet a legislative  requirement for measuring the results of federal surface transportation  research and partially meets a related DOT policy requirement to oversee  and develop ways to improve research evaluations throughout the  department. Figure 3 summarizes these requirements and the extent to  which RSPA has met them.
Although the department has delegated to RSPA the responsibility for  meeting legislative and DOT requirements for evaluating research projects  throughout the department, RSPA’s oversight of DOT research evaluation is  limited to facilitating exchange of information on evaluation approaches.  As delegated by the Secretary of Transportation, RSPA is responsible for  measuring the results of federal surface transportation research activities  and how these results impact the performance of the surface  transportation systems of the United States, as stated in TEA-21. Also,  TEA-21 calls for a strategic planning process that includes information on  research program evaluations conducted and a schedule of future  evaluations. RSPA has not taken steps to meet these legislative  responsibilities for measuring the results of DOT surface transportation  research, describing research program evaluations, and establishing a  schedule for future evaluations.
In terms of DOT policy, RSPA is responsible for overseeing and developing  more efficient, effective, and participative ways to evaluate and measure  research program effectiveness and progress across all operating  administrations. RSPA has taken steps to develop and communicate more  effective means of evaluation by discussing this issue at monthly  Coordinating Council meetings. For example, in October 2002, the council  provided a forum for discussing four different agency approaches to  research evaluation. In addition, according to RPSA officials, the council  has discussed criteria established by the Office of Management and Budget  for federal investment in research and how these criteria can have an  impact on performance evaluation. RSPA, however, does not oversee  operating administrations’ research evaluation efforts and therefore cannot  ensure that steps are being consistently taken to improve evaluation  approaches.
The Associate Administrator said that RSPA does not measure the results  of federal transportation research activities or provide oversight of the  operating administrations’ research program evaluation processes for the  following two reasons: (1) the operating administrations have  responsibility for performing and measuring their own research programs  and (2) the resource constraints that have limited RSPA’s ability to  coordinate DOT-wide research also limit the agency’s ability to oversee  research program evaluations across the department.
Coordinating Council members said that increased oversight of DOT-wide  research programs would be beneficial. Our previous work examining  DOT’s research activities also indicates that such oversight is  warranted. For example, in reviewing the Federal Highway  Administration’s research program, which accounts for almost half of  DOT’s research budget, we found that the Federal Highway Administration  lacked a systematic process for conducting research evaluations, and that  the processes it used were not always consistent with federal research best  practices because it generally did not use a peer review approach.Thus,  without oversight, RSPA and the department have no assurance that, at a  minimum, operating administration research programs are routinely  evaluated or that approaches to evaluations are consistent with established  best practices.
DOT and RSPA also have not developed a strategy to meet the requirement  to measure the results of federal transportation research activities and how  they impact the performance of the surface transportation systems of the  United States. Developing such a strategy that incorporates information  about the costs involved in ensuring that evaluations are completed and  performed according to best practices is an important first step for DOT  and RSPA to meet the requirement. A strategy also is critical for  communicating to the Congress and the Secretary of Transportation (1) the  challenges that RSPA and the department face and (2) the specific actions  that can be taken to meet this requirement given available resources. After  we raised these issues to senior RSPA officials as a result of our review,  they developed a proposed model for reorganizing the Coordinating  Council to provide an opportunity for RSPA to meet legislative and  departmental requirements to oversee DOT research evaluation. They said  they were still considering the proposal when we completed our review.
section_title: RSPA Has Conducted Multimodal Research in Four Areas but Does Not Have a Process to Systematically Evaluate Program Results
paragraphs: Since 1999, RSPA has conducted four multimodal research programs— advanced vehicle technologies, commercial remote sensing and spatial  information, operator fatigue management, and transportation  infrastructure assurance. According to RSPA’s Associate Administrator,  these four programs have resulted or will result in significant contributions  and improvements to the transportation industry. For example, he said that  research aimed at advanced vehicle technologies has resulted in the testing  and development of components currently used in the production of  commercially available hybrid electrical vehicles. Nonetheless, RSPA does  not have a process to systematically evaluate the results of all its  multimodal research programs. In the absence of such a process, RSPA  manages its multimodal research programs by monitoring research  contract agreements and using expert or peer review panels to assess the  quality and relevance of ongoing research. By not systematically evaluating  program results, however, RSPA is limited in its ability to determine the  extent to which its multimodal research programs are achieving their  intended goals.
subsections: section_title: Status of RSPA’s Multimodal Research Programs
paragraphs: Since 1999, RSPA has conducted four multimodal research programs, of  which two were congressionally mandated. Specifically, TEA-21 required  DOT to conduct research on using (1) technology to improve energy  efficiency, and reduce emissions and transportation dependence on  petroleum, and (2) satellite images to improve transportation safety and  disaster planning. Transportation research experts within DOT developed a  third RSPA research program to develop more effective means to increase  the endurance and reduce fatigue of drivers and pilots. Finally, the catalyst  for research in a fourth area that assesses key transportation system  vulnerabilities to damage from disasters or terrorist threats came from the  National Research Council’s TRB.
According to the Associate Administrator, RSPA identified and selected  individual projects for these multimodal research programs by obtaining  input from experts within and outside DOT. For example, RSPA, in  conjunction with other DOT operating administrations, published a plan in  June 1999 to guide the selection of human fatique-related projects. In  addition, in April 2000, RSPA issued a strategic multimodal research and  development program plan to help focus advanced vehicle technology  research. Also, in December 2000, RSPA and TRB held a conference on  remote sensing and spatial information research to, among other things,  discuss and define issues and possible research needs with representatives  from academia, transportation agencies, remote sensing businesses,  consulting firms, and other groups. (See app. I for a list of project  stakeholders and researchers involved in RSPA’s multimodal research  programs.)  Table 1 provides summary data concerning the scope, funding,  and status of RSPA’s four multimodal research programs.
The Advanced Vehicle Technologies Program was mandated in 1998  under section 5111 of TEA-21. This program combines transportation  technologies and innovative program elements to produce new vehicles,  components, and infrastructure for medium- and heavy-duty  transportation needs. Since 1999, approximately $15 million has been  allocated toward 55 separate research projects, 43 of which have been  completed, with the goal of improving energy efficiency and U.S.  competitiveness while reducing emissions and transportation  dependence on petroleum. According to the Associate Administrator,  the program has resulted in (1) the testing of components that are being  used in the development and production of commercially available  hybrid electrical vehicles and (2) the development and implementation  of electric vehicle recharging stations in Hawaii. A picture of an electric  vehicle at a recharging station is shown in figure 4. RSPA plans to have  all of the projects completed by the end of 2004.
The Commercial Remote Sensing and Spatial Information  Technologies Program was mandated in 1998 under section 5113 of  TEA-21. The joint program between RSPA and the National Aeronautics  and Space Administration (NASA) began in 1999. This research program  focuses in part on using satellite images to assess transportation  hazards and improve disaster recovery; provide opportunities to  monitor and evaluate regional traffic flow, including the movement of  freight; plan for improvements in the maintenance and security of  transportation infrastructures; and aid in transportation corridor  planning. Figure 5 shows an example of satellite-based photography of  interstate highways. According to the Associate Administrator, RSPA  has also supported transportation security technology project activities  in the aftermath of the terrorist attacks of September 11, 2001. For  instance, according to RSPA officials, the Remote Sensing Program  reoriented two unmanned aerial vehicle projects toward monitoring for  security as well as traditional transportation applications. RSPA has  allocated $18 million to this research program since it began in 1999 and  has disseminated program information; results to date have been  disseminated through Web sites, publications, workshops, and  conferences. Eighteen separate research projects constitute the  program; 4 have been completed, and RSPA plans to complete the  remaining 14 projects by the end of 2003.
The Human Centered Systems: Operator Fatigue Management  Program was conceived by DOT researchers and experts with the  primary goal to develop techniques that transportation operating  companies can employ to ensure endurance and fatigue-free  performance of their workforces. Since fiscal year 1999, approximately  $1 million has been allocated to the Human Centered Systems: Operator  Fatigue Management Program. According to the Associate  Administrator, this program has resulted in significant benefit to the  varied DOT transportation community stakeholders. For example, in  January 2003, the program resulted in the production of the Commercial  Transportation Operator Alertness Management Handbook, which  describes measures to better manage driver and pilot fatigue. According  to RSPA’s Associate Administrator, this handbook has been in high  demand by the U.S. Coast Guard, the Maritime Administration, and the  Federal Transit Administration.  Two of the four projects being  conducted are ongoing. RSPA plans to complete the program by the end  of 2004. In figure 6, airline pilots participate in NASA research in this  area.
The Transportation Infrastructure Assurance Program consists of  assessments on four separate transportation vulnerabilities. These  include assessing the (1) relationship between the safety and security of  the nation’s transportation infrastructure and some of the nation’s other  critical infrastructures, such as energy, e-commerce, banking and  finance, and telecommunications; (2) transportation and logistical  requirements for emergency response teams in response to terrorist  attacks using biochemical, nuclear, and explosive weapons of mass  destruction; (3) costs, benefits, and practicality of alternative backup  systems for the global positioning system; and (4) trade-offs between  the different modes of transportation and security for hazardous  materials. Since fiscal year 2001, the Congress has appropriated $3  million to conduct these assessments. In total, RSPA plans to publish 11  formal reports on the four vulnerabilities being assessed and develop a  series of presentations and workshops to further disseminate the  information. Figure 7 shows a picture of a global positioning satellite.
From fiscal years 1999 through 2003, RSPA budgeted about $37 million to  conduct these four major multimodal research programs. Of this $37  million, about 9 percent, or $3.3 million, came to RSPA from direct  congressional appropriations. For example, in fiscal year 2001, the  Congress appropriated $1 million for the Transportation Infrastructure  Assurance Program and $300,000 for the Human Centered Systems:  Operator Fatigue Management Program; in fiscal years 2002 and 2003,  RSPA received an additional $1 million for the Transportation  Infrastructure Assurance Program. The remaining $33.7 million for these  programs was provided through reimbursable funding from other DOT  administrations. Figure 8 summarizes RSPA’s annual budget for  multimodal research from fiscal years 1999 to 2003.
According to the Associate Administrator, RSPA’s current multimodal  research programs are scheduled for completion by the end of fiscal year  2004. He added that RSPA has made budgetary and legislative proposals to  undertake future multimodal research to, among other things, further  examine applications of unmanned aerial vehicles for commercial remote  sensing or examine infrastructure safety issues in hydrogen energy  systems. He noted, however, that there are no approved plans for future  multimodal research, pending the President’s budget proposal for the  department and the reauthorization of TEA-21, which might affect RSPA’s  multimodal research roles and responsibilities.
section_title: RSPA Oversees Research Contracts and Assesses the Status of Its Ongoing Research but Lacks a Systematic Process for Evaluating the Results of Its Multimodal Research
paragraphs: RSPA oversees its multimodal research programs by monitoring research  contract agreements. Specifically, the Associate Administrator said that  RSPA assesses project progress against contractual milestones to ensure  that the research is being completed on time and within cost, while meeting  research objectives. He added that researchers must meet or exceed  contractual expectations, or corrective actions are taken. These actions  may include project cancellation. RSPA provided a recent example of the  impact of its monitoring efforts that dealt with a project being conducted  under its Commercial Remote Sensing and Spatial Information  Technologies Program. The evaluation focused on an unmanned aerial  vehicle application to real-time traffic flow monitoring, a demonstration  project with the Ohio Department of Transportation and several  universities and industry partners. Due to a technical problem, the  scheduled demonstration could not be conducted. The Remote Sensing  Program Manager, working with the Federal Highway Administration  Project Manager assigned to the project, pulled together a technical team  to assess the state of the project. The technical team became convinced  that the contractor could remedy the technical situation, allowed 3  additional months to make the milestone, and successfully encouraged the  project partners to cover the costs of the delay. According to the Associate  Administrator, these actions resulted in a demonstration to a state partner  with no cost increase to RSPA.
In addition to providing this contractual oversight, RSPA uses the  principles of expert or peer review through the use of multimodal and  multiagency program oversight panels to assess the status, quality, and  relevance of its ongoing multimodal research programs, according to the  Associate Administrator. For example, in December 2000, TRB and RSPA  held a conference on remote sensing and spatial information research with  representatives from academia, transportation agencies, remote sensing  businesses, consulting firms, and other groups. During the conference,  participants met in breakout sessions to discuss and assess research  progress and interim results. As we have reported, expert or peer review is a particularly effective means to evaluate federally funded research.  The  Committee on Science, Engineering, and Public Policy has also reported  that expert review is widely used to evaluate the quality of current research  as compared with other work being conducted in the field and with the  relevance of research to the agency's goals and mission. However, RSPA  has not established a process or policies for systematically using peer or  expert reviews to evaluate the results of all its multimodal research. For  example, RSPA has no plans to evaluate the results of its Transportation  Infrastructure Assurance Program.
We, among others, recognize that federal agencies that support research in  science and engineering have been challenged to find the most useful and  effective ways to evaluate the performance and results of the research  programs. For example, since GPRA was passed in 1993, some questions  have been raised about its applicability to the research activities of  government agencies. Because the process required by GPRA is based on a  5-year strategic planning horizon, concerns exist that GPRA constrains, and  perhaps prohibits, the long-term thinking and planning that characterize  the federal role in research. This concern is particularly relevant for basic  research, but even successes from highly applied research (the type  sponsored by DOT) can require 5 to 10 years before achieving widespread  recognition. Nonetheless, as we noted in our report examining DOT  highway research, without systematic program evaluation, it is unclear as  to whether research efforts are having the intended results. Such a  systematic approach to evaluation, according to best practices used in  other federal research programs, includes review of all ongoing and  completed research on a regular basis and in accordance with GPRA  principles.
RSPA’s Associate Administrator acknowledged that a documented process  for systematically evaluating the results of its multimodal research  programs would be beneficial, but that the process should be tailored to  match the type of research and its objectives. He added that RSPA had not  developed and implemented a process for systematically evaluating the  results of its multimodal research because of a lack of funding and staffing  resources. For example, he estimated that an external evaluation to assess  the results of its multimodal programs could cost as much as $100,000 for  each program. Nonetheless, without establishing and implementing a  process for systematically evaluating the results of its research, RSPA  cannot ensure that its multimodal research programs are achieving their  intended goals.
section_title: Conclusions
paragraphs: To its credit, RSPA has taken steps in recent years to meet its legislative  and department responsibilities for coordinating and overseeing the  evaluation of the department’s transportation research activities.  Nevertheless, ensuring that no unnecessary duplication of research  programs occurs and that research programs—including the ones that  RSPA conducts—are evaluated for results are critical responsibilities,  given the importance of, and amount of money spent on, DOT research.  Without a strategy to meet legislative and DOT requirements to coordinate  and oversee evaluation of departmental research, RSPA may not be able to  meet these responsibilities, particularly given its lack of authority within  the department and resource limitations. In addition, a lack of performance  standards against which to measure coordination efforts limits RSPA’s  ability to identify areas where coordination is working effectively and areas  that could be improved upon. With a strategy and performance measures in  place, however, RSPA and DOT should be in a better position to assure the  Congress that the department is making the most of its significant research  dollars through effective coordination and evaluation of its research  programs.
section_title: Recommendations for Executive Action
paragraphs: To better meet legislative and DOT requirements for coordinating and  evaluating transportation research within the department, we recommend  the Secretary, in conjunction with RSPA’s Administrator, work with DOT  operating administrations to:   Develop a strategy for reviewing all DOT research projects to identify  areas of unnecessary research duplication, overlap, and opportunities  for joint efforts. The strategy should address time frames for  implementing this review as well as discuss the extent to which  finalizing the development and implementation of a DOT-wide research  tracking system database could serve to facilitate this process. Once this  strategy has been developed and implemented, the results of this effort  should be incorporated in the department’s annual research plan and  reported to the Congress on an annual basis.
Develop and apply quantifiable performance measures to assess the  effectiveness of research coordination efforts (once a strategy for  review has been developed and implemented), and document the results  of these efforts in the department’s annual research plan. These  measures could include the number of research projects identified as  possible candidates for joint effort or elimination and/or the associated  reduction in the department’s research spending.
Develop a strategy to ensure that the results of all DOT’s transportation  research activities are evaluated according to established best practices.  This strategy should include estimates of the costs for ensuring that  evaluations are completed. Once the strategy has been developed and  implemented, the results of these efforts should be incorporated in the  department’s annual research plan and reported to the Congress on an  annual basis.
Include in the department’s annual research plan a summary of all  research program evaluations conducted and a schedule of future  evaluations.
In addition, we recommend that the Secretary direct RSPA’s Administrator  to document RSPA’s process for systematically evaluating the results of its  own multimodal research programs, and apply this process to any future  multimodal research programs that RSPA conducts.
section_title: Agency Comments and Our Evaluation
paragraphs: We obtained oral comments on a draft of this report from RSPA officials,  including the Associate Administrator for Innovation, Research, and  Education. These officials generally agreed with the contents of the draft  report but did not comment specifically on the report’s recommendations.  They also provided technical comments that we incorporated as  appropriate.
Regarding RSPA’s evaluation of its own multimodal research, the officials  said that RSPA had conducted evaluation activities—peer and expert  reviews of the progress of three of its four multimodal research programs  and had no plans to evaluate the fourth. We acknowledge that RSPA has  used peer and expert review to evaluate the status of at least one of its  ongoing multimodal research programs—commercial remote sensing and  spatial information technology—and we describe this example in this  report. Nevertheless, we continue to believe that our recommendation for  documenting and applying a process for systematically evaluating the  results of any future multimodal research programs conducted by RSPA is  warranted to ensure that such evaluations are consistently conducted in  accordance with established best practices.
We are sending copies of this report to congressional committees and  subcommittees with responsibilities for transportation, the Secretary of  Transportation, the Research and Special Programs Administration  Administrator, and the Director of the Office of Management and Budget.  We will make copies available to others upon request. In addition, the  report will be available at no charge on the GAO Web site at  http://www.gao.gov.
If you have questions about this report, please contact me at (202) 512-2834  or siggerudk@gao.gov. Other key contributors were Colin Fallon,  Christopher Keisling, Bert Japikse, Steve Morris, and Jason Schwartz.