Pipeline Safety: Systematic Process Needed to Evaluate Outcomes of Research and Development Program
Released: 2003-06-30T00:00:00

Letter

Background

Three primary types of pipelines form a 2.2 million-mile network across the nation.

Natural gas transmission pipelines transport natural gas over long distances from sources to communities.

Natural gas distribution pipelines continue to transport natural gas from transmission lines to consumers.

Hazardous liquid pipelines transport crude oil to refineries and refined oil products, such as gasoline, to product terminals.

OPS, within DOT’s Research and Special Programs Administration (RSPA), is responsible for enhancing the safety of and reducing the potential environmental impacts of transporting natural gas and hazardous liquids through pipelines. The agency primarily carries out this responsibility through regulation, oversight, enforcement, and R&D. OPS sets and enforces regulations that pipeline operators must follow in designing, constructing, maintaining, and operating pipelines. State agencies responsible for overseeing pipeline safety help OPS to enforce its regulations. In December 2000, it began implementing a new risk-based regulatory approach, called “integrity management.” Under this approach, operators are required, in addition to meeting minimum safety standards, to better protect pipeline segments where a leak or rupture could have significant consequences, such as near highly populated areas, by conducting new tests of these segments, completing repairs according to specified schedules, and developing comprehensive plans for addressing the range of risks facing these segments. The agency’s R&D program is aimed at advancing the most promising technologies for ensuring the safe operations of pipelines. For example, current R&D projects seek to develop new and improved techniques for assessing the condition of pipelines and detecting anomalies—such as leaks, corrosion, and damage from excavators—that can lead to pipeline accidents. From 1998 through 2002, a total of 1,770 pipeline accidents occurred, resulting in 100 fatalities and $621 million in property damage.

OPS’s R&D program has undergone major changes in the last several years. In particular, the agency has developed a new agenda for its R&D program, using the input of key experts and stakeholders, and has received significant increases in funding for this program.

Until 2001, most of the research funded by OPS was aimed at helping the agency perform its regulatory function or was in response to an accident investigation or congressional direction. In November 2001, the agency held an R&D planning workshop to gain the perspectives of a variety of experts and stakeholders on areas of R&D that have the most potential for enhancing pipeline safety. Attendees included representatives of federal and state agencies, research organizations, industry groups, pipeline companies, and technical organizations that set industry safety standards. OPS used the R&D priorities identified in this workshop to develop a new agenda for its R&D program, focusing on three main areas: (1) developing new technologies for preventing damage and detecting leaks, (2) improving technologies for operating, controlling, and monitoring the condition of pipelines, and (3) improving pipeline materials. From March through December 2002, the agency issued announcements requesting project proposals in these areas, asking that prospective funding recipients provide at least 50 percent of the proposed project’s cost. As of May 2003, it had funded 10 R&D proposals it received in response to these announcements. In addition, after its November 2001 R&D workshop, OPS established a Web site on its R&D program in order to improve communications with experts, stakeholders, and the public about its R&D agenda and activities.

OPS’s budget for its R&D program has risen more than sevenfold since fiscal year 1998, with the most significant increases occurring since fiscal year 2001. Figure 1 shows the agency’s budgeted amounts for R&D from fiscal years 1998 through 2003. OPS’s budget for R&D rose steadily from fiscal year 1998 to fiscal year 2001, from $1.3 million to $2.8 million. In fiscal year 2002, the agency received $4.8 million for its R&D program, which was $2 million more than RSPA had requested for the program. Agency officials attribute this funding increase to increased concerns for pipeline safety within Congress following the tragic pipeline accidents in Bellingham, Washington (1999), and Carlsbad, New Mexico (2000), which together caused 15 fatalities. For fiscal year 2003, RSPA requested and received about $4 million in additional funding for the program, for a total of $8.7 million. OPS officials told us that this requested increase was a response to heightened congressional interest in achieving technological solutions to pipeline safety, as evidenced by legislative proposals that called for increased attention to this area. RSPA is proposing funding for OPS’s R&D program of $9.2 million in fiscal year 2004, an increase of about $0.5 million above the fiscal year 2003 amount. OPS officials explained that they intend to use most of this increase for a study, required by the Pipeline Safety Improvement Act of 2002, to assess the performance of controllers who monitor pipeline operations. Overall, agency officials also attribute recent increases in funding for OPS’s pipeline safety R&D program to a recognition of the challenges posed by the agency’s new integrity management regulatory approach and the criticality of the nation’s pipeline infrastructure, in the aftermath of the terrorist attacks of September 11, 2001.

OPS’s pipeline safety R&D program is continuing to evolve in response to new directives in the Pipeline Safety Improvement Act of 2002 for the planning and reporting of federal pipeline R&D efforts. The act, which became law in December 2002, assigned the Secretary of Transportation responsibility for developing a 5-year plan for pipeline R&D and transmitting the plan to Congress by December 2003, in coordination with DOE and the National Institute of Standards and Technology. (OPS officials told us that the Secretary has delegated this responsibility to OPS.)

DOE operates an R&D program that is focused on developing future technologies to improve the integrity, reliability, and security of the natural gas infrastructure, including pipelines and storage facilities. In comparison with OPS’s R&D program, which focuses on the development of quick-to-market technologies that could become available in the short term (1-3 years) or midterm (3-5 years), DOE’s program focuses on technologies that could become available in the midterm (3-5 years) or longer term (5-8 years). The National Institute of Standards and Technology does not operate an R&D program focused on pipelines, but, reflecting its expertise in materials research, the act assigns it a key role in planning future pipeline R&D.

The Department of the Interior’s Minerals Management Service (MMS), although not assigned an R&D planning role in the act, funds pipeline R&D, including research on offshore pipeline safety. Consequently, OPS plans to include that agency in efforts to develop a 5-year plan for pipeline R&D. The act requires the heads of DOT, DOE, and the National Institute of Standards and Technology to jointly report annually to Congress, beginning in December 2003, on the status and results of implementation of the plan.

OPS’s R&D Funding Is Aligned with Its Mission and Pipeline Safety Goals

Since fiscal year 2001, OPS has allocated its rising R&D funding to three main areas of pipeline safety R&D that were identified at its 2001 workshop: (1) developing new technologies for preventing damage to pipelines and detecting leaks, (2) improving technologies for operating, controlling, and monitoring the condition of pipelines, and (3) improving the performance of pipeline materials. The agency has also allocated some R&D funding to a fourth area, efforts to improve the agency’s mapping and information systems.

On the basis of our work, we believe that the agency’s R&D funding is generally aligned with its mission and pipeline safety goals. The agency has obtained the views of external experts and stakeholders in determining what types of R&D are aligned with its mission of ensuring the safe, reliable, and environmentally sound operation of the nation’s pipeline transportation system. The agency has also recently improved coordination with other federal agencies that fund pipeline R&D in order to avoid overlap between their R&D programs. Both of these practices have been recommended by leading organizations that conduct scientific and engineering research. OPS has also linked its R&D efforts with its performance goals of reducing the impacts of pipeline incidents, including fatalities and injuries, and reducing spills of hazardous material. In its plans, the agency has described how new and improved technologies resulting from its R&D funding can help achieve these performance goals. Finally, a number of key experts and stakeholders told us that, in their view, the agency has chosen appropriate R&D areas to fund.

OPS Allocates Pipeline Safety R&D Funding to Four Major Areas

OPS allocates its R&D budget to three major areas involving the research and development of pipeline safety technologies as well as to a fourth area—efforts to improve the agency’s pipeline mapping and information systems—that does not involve such research and development. Figure 2 shows how the agency plans to distribute its fiscal year 2003 R&D budget of $8.7 million among these areas.

OPS plans to spend the largest share of its R&D budget, 46 percent, or $4.0 million, on the area of Damage Prevention and Leak Detection, which includes the development of new technologies to prevent damage to pipelines, detect pipeline defects, and quickly and accurately locate and control pipeline leaks. Damage to pipelines from “third parties,” such as companies performing excavation work, is the leading cause of pipeline failures and can lead to property damage and injuries or fatalities.

OPS plans to allocate 21 percent of its R&D budget, $1.9 million, to the area of Enhanced Operations, Controls, and Monitoring, which includes improvements in technologies for operating, controlling, and monitoring the integrity of pipelines to help identify and prioritize pipeline safety problems and solutions.

The agency intends to spend a slightly lesser amount, 19 percent of its R&D budget, or $1.7 million, on the area of Improved Materials Performance, which includes improvements in pipeline materials in order to extend the integrity and lifetime of installed pipelines and their various components.

Finally, the agency plans to allocate the smallest portion of its R&D budget, 14 percent, or $1.2 million, to the area of Mapping and Information Systems, which includes efforts to improve the collection, integration, and analysis of data on the location and safety performance of pipelines. These efforts make pipeline mapping information available to federal, state, and local officials and support pipeline inspection activities of OPS and its state partners.

Since fiscal year 2001, OPS’s allocation of funding to each of the three main areas of pipeline safety R&D—Damage Prevention and Leak Detection; Enhanced Operations, Controls, and Monitoring; and Improved Materials Performance—has risen significantly, while its allocation to Mapping and Information Systems efforts has remained level. The tripling of the agency’s R&D budget—from $2.8 million in fiscal year 2001 to $8.7 million in fiscal year 2003—has enabled it to increase funding for these three R&D areas. Specifically, OPS has increased funding for R&D efforts in Damage Prevention and Leak Detection from $1.3 million in fiscal year 2001 to $4.0 million in fiscal year 2003, an increase of over 200 percent. The agency has increased funding for Enhanced Operations, Controls, and Monitoring from $309,000 in fiscal year 2001 to $1.9 million in fiscal year 2003, an increase of more than 500 percent. OPS started funding Improved Materials Performance research in fiscal year 2002, increasing funding in this area to a level of $1.7 million in fiscal year 2003.

Agency officials explained to us that they allocated funding to these three R&D areas in fiscal years 2002 and 2003 based on the results of their 2001 R&D planning workshop. For example, they added Improved Materials Performance to their R&D agenda because it was identified as a priority area at the workshop. They have also considered other factors in deciding how to allocate funding. For example, the agency significantly increased funding for R&D in the areas of Damage Prevention and Leak Detection and Enhanced Operations, Controls, and Monitoring because of a great need for improved performance in these areas. OPS officials explained that, because the agency’s new risk-based regulatory approach requires pipeline operators to assess and mitigate risks to pipeline segments where a leak or rupture could have significant consequences, these operators need better tools and methods for monitoring pipelines and making necessary repairs. They also noted that OPS’s R&D results assist in the creation of industry standards on the appropriate use of new technologies. In addition, officials explained that they decided to allocate a significant portion of their R&D budget to the area of Improved Materials Performance because, on the basis of current information on the development of pipeline technologies, they believed that advances in this area held much promise for improving pipeline safety.

Finally, OPS has allocated about $1.2 million per year to the Mapping and Information Systems area since fiscal year 2001 in order to maintain efforts to improve these systems. (See fig. 3.)

OPS has provided $3.0 million in funding to 10 projects related to Damage Prevention and Leak Detection since fiscal year 2001. Examples of funded projects include the following: OPS provided $0.6 million in funding to five projects focused on improving in-line inspection techniques, including “smart pigs” and other technologies, for detecting damage and defects in pipe walls. Such improved techniques can help to prevent pipeline leaks or ruptures by making possible the early detection and repair of damage and defects.

In partnership with the U.S. Air Force, OPS provided $1.2 million in funding to a project focused on developing an approach for detecting pipeline leaks using an airborne laser system that measures levels of chemicals in the atmosphere just above the earth’s surface.

OPS has provided $0.9 million in funding to six projects related to Enhanced Pipeline Operations, Controls, and Monitoring since fiscal year 2001. Most of this funding—$0.6 million—has been allocated to two projects to improve alternative inspection techniques, called direct assessment, for identifying internal and external corrosion and other defects in pipelines that cannot accommodate smart pigs. This is a significant issue for natural gas pipelines. One industry association estimates that only about 35 percent of the total natural gas pipeline mileage can accommodate smart pigs, which are typically used to assess the condition of liquid pipelines. OPS officials told us that they are planning to fund three additional R&D projects in this area in June 2003.

As of May 2003, OPS has provided $0.1 million in funding to one project in the area of Improved Materials Performance. This project seeks to develop a “smart” composite pipe that will allow for real-time monitoring of the condition of the pipe through a remote monitoring system. The agency requested proposals in this area in December 2002 and expects to start funding some of these proposals in the summer of 2003. Among the types of proposals that OPS has requested are proposals to develop materials that better withstand third-party damage, corrosion, and higher grade/strength steels; and materials that facilitate the operation of pipelines at higher design pressures.

Finally, of the roughly $1.2 million that OPS has allocated each year since fiscal year 2001 to the Mapping and Information Systems area, it spent or plans to spend about $800,000 each year for efforts to improve the National Pipeline Mapping System, which depicts the location of pipelines in relation to areas that are populated or environmentally sensitive, and about $400,000 each year for efforts to integrate information systems the agency uses in overseeing pipeline safety in cooperation with the states.

The agency expects to continue funding this area at this level for the foreseeable future in order to improve and update these systems continually. OPS officials explained that these mapping and information systems assist OPS inspectors and state and local officials in their efforts to oversee pipelines and protect the community and environment from pipeline leaks or ruptures.

Expert Review and Coordination Help OPS Align Its R&D Funding with Its Mission and Goals

OPS’s mission is to ensure the safe, reliable, and environmentally sound operation of the nation’s pipeline transportation system. It has indicated in its budget and plans that its R&D program supports this broad mission as well as the following more specific performance goals: (1) to reduce deaths, injuries, property damage, and economic disruptions resulting from pipeline incidents and (2) to reduce the amount of oil and other hazardous liquids spilled from pipelines. The agency has described how new and improved technologies resulting from its R&D funding can help achieve these performance goals. For example, the number of pipeline incidents and the amount of hazardous material spilled could be reduced through the use of improved technologies for detecting third-party damage, corrosion, and defects and the use of improved pipeline materials that can better withstand damage and corrosion.

The Committee on Science, Engineering, and Public Policy—a joint committee of the National Academy of Sciences, the National Academy of Engineering, and the Institute of Medicine—has recommended the use of expert review to determine whether a research program is focused on the subjects most relevant to an agency’s mission. Under this form of review, experts in related fields as well as potential users of the research evaluate the relevance of research to an agency’s mission and goals and its potential value to intended users.

OPS has used expert review to help it develop a research agenda that is aligned with its mission and goals. At its November 2001 R&D planning workshop, it asked a variety of experts as well as potential users of research to identify the types of R&D that would be most likely to enhance pipeline safety. Participants included representatives from federal and state agencies with pipeline responsibilities, pipeline companies and their associations, research groups, and technical organizations that set industry safety standards for pipelines. The agency subsequently used the results of this workshop in developing its research agenda, guided by an R&D planning panel composed of key experts from such groups.

OPS has also used peer review, a form of expert review, in deciding which R&D proposals to fund, a practice that is recommended by the Committee on Science, Engineering, and Public Policy. OPS’s review panels have included representatives from other federal agencies that conduct pipeline R&D, industry associations, and associations of state agencies with pipeline safety responsibilities.

The Pipeline Safety Improvement Act, enacted in December 2002, requires that the Secretary of Transportation consult with a variety of groups in preparing a 5-year plan for pipeline safety R&D, which must be provided to Congress by December 2003. In response, OPS is continuing to involve various experts and stakeholders in its R&D planning. Agency officials have told us that, in preparation for developing this 5-year plan, they are in the process of obtaining updated external views in order to reassess research priorities. This has involved participating in the pipeline R&D planning efforts of industry associations and research organizations, discussing R&D priorities with state agency officials, and reconvening their R&D planning panel of outside experts. In developing the plan, agency officials also plan to consult with OPS’s two technical advisory committees. Finally, OPS plans to hold another R&D workshop during the winter of 2003-04.

Coordination among federal agencies that conduct related research helps to avoid duplication and ensure that each agency performs research that is aligned with its particular mission and goals. The Committee on Science, Engineering, and Public Policy has recommended that agencies establish a formal process for coordinating similar fields of research, in order to improve collaboration, help keep important questions from being overlooked, and avoid duplication of effort. Since 2001, OPS has increased efforts to coordinate pipeline R&D with DOE and the Department of the Interior’s MMS, both of which also conduct research related to pipelines. This increased coordination has taken the form of mutual participation in panels that review R&D proposals and workshops to plan R&D activities. According to OPS officials, officials of these agencies have used these opportunities to communicate about their respective pipeline R&D efforts and avoid duplication. However, these agencies have not had a formal mechanism in place that defines each agency’s responsibilities for pipeline R&D.

The Pipeline Safety Improvement Act requires that the heads of DOT, DOE, and the National Institute of Standards and Technology develop a memorandum of understanding to formally coordinate pipeline R&D efforts. (Although the institute does not operate an R&D program focused on pipelines, the act assigned it a key role in pipeline R&D based on its expertise in materials research.) In response, OPS, DOE, and the institute have developed such a memorandum and are in the process of finalizing it. The Pipeline Safety Improvement Act also requires that DOT coordinate with DOE and the National Institute of Standards and Technology in developing a 5-year plan for pipeline R&D. In response, OPS is involving DOE and the institute, as well as MMS, in efforts to develop such a plan. These agencies are also considering holding joint workshops on pipeline R&D in the future. In addition, OPS and the National Institute of Standards and Technology have started to participate in each others’ proposal review panels and are discussing entering into an agreement to have the institute conduct some research on pipeline materials.

We asked a number of key experts and stakeholders for their views on the extent to which OPS’s R&D agenda is aligned with its mission and goals. These individuals included officials in DOE and MMS, representatives of four industry associations, a former head of a state agency that regulates gas pipelines, the heads of two leading pipeline R&D organizations, two foremost technical experts in pipeline safety, and an environmentalist active in pipeline safety. Six of these individuals have been or are members of OPS advisory committees or R&D planning or review panels. They generally told us that, in their view, the agency has chosen to fund appropriate areas.

Experts Generally Support OPS’s R&D Priorities

The pipeline safety R&D priorities of the experts who completed our questionnaire are generally consistent with OPS’s R&D priorities. Of the three main R&D areas that OPS is currently funding, Damage Prevention and Leak Detection received the most scores of high or very high funding priority; Enhanced Operations, Controls, and Monitoring received the second highest number of such scores; and Improved Materials Performance received the third highest number. This ranking corresponds to the relative levels of funding OPS has assigned to these areas, as described in the previous section. However, the experts’ level of support for Improved Materials Performance was much lower than that for the other two main R&D areas that OPS is funding. OPS officials told us that they are currently updating their research agenda, using the input of experts and stakeholders, and that they will consider our questionnaire results in this process.

To obtain the views of experts on pipeline safety R&D priorities, we asked 55 experts to complete a questionnaire indicating the funding priority they would assign to various types of pipeline safety R&D, using categories identified as part of OPS’s 2001 R&D planning workshop. Table 1 provides a description of the main categories of R&D we asked experts to prioritize. The first three categories correspond to the main areas of R&D that OPS is currently funding. Although the fourth category—Arctic and Offshore Technologies—was identified as a main area of pipeline R&D at its workshop, OPS decided not to include it as a main area in its R&D agenda.

Agency officials told us that they made this decision because R&D related to Arctic and Offshore Technologies was not considered to be a high priority by participants at its workshop and because MMS funds some R&D in this area and is the primary offshore regulator. We did not include Mapping and Information Systems—an area that OPS is currently funding from its R&D budget—as a category for the experts to rate because it was not identified as a main category of R&D at the 2001 workshop.

Figure 4 shows how the 49 experts who completed our questionnaire rated the four categories of pipeline safety R&D. We also asked experts to rate specific types of R&D within each category. (See app. I for how the experts rated specific types of R&D within these main categories and for information on the agency’s funding of these specific types of R&D. See app. II for information on our methodology for selecting experts and obtaining their views.)

The experts who completed our questionnaire strongly supported the Damage Prevention and Leak Detection and Enhanced Operations, Controls, and Monitoring categories of R&D as important areas for OPS to fund. Ninety-two percent of the experts (45 of 49) indicated that the Damage Prevention and Leak Detection category should receive high or very high funding priority. Within this category, experts assigned the most scores of high or very high funding priority to the following types of R&D: improvements in the ability of in-line inspection tools, such as “smart pigs,” to detect damage and defects (39 of 49), and the development of new technologies, such as the innovative application of ultrasonics, that can be used for inspecting pipelines (38 of 49). Several experts we interviewed highlighted the need to improve methods for detecting damage to pipelines, citing the fact that third-party damage is the leading cause of pipeline accidents. According to both liquid and gas pipeline associations, current inspection tools cannot reliably detect such damage to pipelines.

Eighty percent of the experts (39 of 49) indicated that the Enhanced Operations, Controls, and Monitoring category should receive high or very high funding priority. Within this category, the type of R&D that received the most scores of high or very high funding priority (37 of 49) was the improvement of alternative inspection techniques, called direct assessment, to identify corrosion and other defects in pipelines that cannot accommodate in-line inspection devices known as smart pigs. This is a significant issue for natural gas pipelines because the majority of these pipelines cannot currently accommodate smart pigs, which are typically used to assess the condition of liquid pipelines.

In contrast to the experts’ views on the importance of these first two categories, less than one-third of the experts considered the remaining two categories of R&D, Improved Materials Performance and Arctic and Offshore Technologies, to be a high priority for OPS to fund. Thirty-one percent of the experts (15 of 49) assigned scores of high or very high funding priority to the Improved Materials Performance category, and 20 percent (10 of 49) assigned such scores to the Arctic and Offshore Technologies category. However, within the category of Improved Materials Performance, about half (25 of 49) of the experts indicated that the type of R&D aimed at developing damage- and defect-resistant materials should receive high or very high funding priority. Such materials could be used in the replacement of existing pipe or in the installation of new pipe. One researcher we interviewed noted that such materials are particularly important for the gas pipeline industry, which is expanding its infrastructure in response to increased demands for natural gas. One industry association estimates that the natural gas industry will need to install about 49,500 miles of transmission pipeline from 2001 through 2015 to meet increased demand in the United States.

Some differences exist in the views of experts from the following three subgroups: (1) federal and state government and public interest organizations, (2) pipeline industry and technical and consulting organizations, and (3) research organizations. As shown in table 2, experts from all three subgroups generally gave the category of Damage Prevention and Leak Detection the highest ranking, followed by the category of Enhanced Operations, Controls, and Monitoring. However, experts from research organizations considered the categories of Improved Materials Performance and Arctic and Offshore Technologies to be more important for OPS to fund than did experts from the other two subgroups. For example, 70 percent of experts from research organizations (7 of 10) rated Improved Materials Performance as a high or very high priority compared with 19 percent of experts from government and public interest organizations (3 of 16) and 22 percent of experts from pipeline industry and technical and consulting organizations (5 of 23). In addition, 60 percent of the researchers (6 of 10) rated Arctic and Offshore Technologies as a high or very high priority for OPS compared with 19 percent of experts from government and public interest organizations (3 of 16) and only 4 percent of experts from pipeline industry and technical and consulting organizations (1 of 23).

An OPS official told us that he believed that researchers rated the Improved Materials Performance category more highly than other experts did because researchers have the best and most current information about the “state of the art” in technology development and are more aware of opportunities in this area. A leading expert from a pipeline research organization noted that the foundation of pipeline R&D has been the development of defect-resistant steels and that, as a consequence, researchers in this area are very interested in R&D that will lead to further improvements in the performance of pipeline materials. He also explained that researchers may have rated the Arctic and Offshore Technologies category more highly than the other types of experts who completed our questionnaire because researchers may be more aware of the need for such R&D to support the construction of new pipelines in these areas in order to reach new energy supplies.

OPS Lacks a Systematic Process for Evaluating R&D Outcomes

Although OPS has received significant increases in funding for its R&D program in recent years, the agency has not developed a systematic process for evaluating the effectiveness of its R&D program. For example, the agency tracks and disseminates information on the progress of individual R&D projects but has not developed a process for assessing and reporting on the results of its R&D program as a whole. Such a process is needed to demonstrate the program’s progress toward achieving its objectives, such as the development and use of new technologies that can improve pipeline safety. OPS has taken some preliminary steps toward developing an evaluation process for its R&D program and could benefit from adopting identified best practices for systematically evaluating the outcomes of federal R&D programs. Leading research organizations, the Office of Management and Budget, and GAO have identified a number of such practices, including setting clear R&D goals and measuring progress toward these goals, using expert review to evaluate the quality of research outcomes, and reporting periodically on evaluation results. The results of evaluations can be used to refocus the direction of R&D programs periodically, as necessary, to ensure that resources are most effectively utilized.

OPS’s Efforts to Evaluate Research Outcomes Have Been Limited

Although OPS has funded R&D to develop pipeline safety technologies since the mid-1990s, the agency’s efforts to evaluate the outcomes of this R&D have been limited and have focused on individual projects. OPS’s R&D contracts define project goals and require research performers to meet specific milestones for the development of a technology. Contracts also require research performers to report quarterly and at the end of the project on results, including milestones achieved and patents applied for and received. OPS has made some efforts to disseminate the results to date of individual R&D projects. For example, it has started to put “success stories” on its Web site that describe achievements in ongoing projects, such as the development of product prototypes. These success stories help to communicate the results of individual projects to industry and other interested parties.

At the program level, OPS has not yet established specific quantifiable goals for its R&D program or a method for measuring progress toward these goals. OPS has indicated, in various planning documents, that its R&D program will help achieve its performance goals of reducing the impacts of pipeline incidents, including fatalities and injuries, and reducing spills of hazardous material. However, agency officials have acknowledged that it is difficult to show the effect of the R&D program on these performance goals. A more immediate objective of the program, according to agency plans, is to promote the transfer of new and improved pipeline safety technologies to the market in the near term.

In deciding which R&D proposals to fund, OPS gives preference to those that plan to bring a new product to market within 5 years. In addition, agency officials told us that OPS plans to promote the use of new technologies by providing information to potential users and its state partners about them and, when appropriate, by encouraging their use through regulation. Agency officials told us that the R&D program aims to have 80 percent of its projects result in products on the market within 5 years. Such an objective is specific and measurable, but OPS has not formally established it as a goal in any plan or developed a method for measuring progress toward achieving it. Furthermore, since the agency has not yet established specific goals or outcome measures for its R&D program, it does not have a process for documenting and reporting on the extent to which this program is achieving its goals.

OPS officials explained that they have not yet developed a process for evaluating the outcomes of the agency’s R&D program because, prior to 2001, the program’s budget was relatively low and, since restructuring the program in 2001, they have focused program efforts on building a process for setting research priorities. However, officials do recognize the need for evaluating R&D outcomes and have taken some preliminary steps toward developing an evaluation process for their R&D program.

OPS is considering some possible measures of the outcomes of its R&D program as a whole, such as the number of new patents resulting from R&D efforts. In addition, agency officials told us that, although tracking the transfer to the market of new pipeline safety technologies can be challenging, OPS intends to track the use of new technologies in the future through its process for inspecting operators’ “integrity management” programs. For example, OPS inspectors could document the use of new or improved technologies by companies to evaluate the condition of their pipelines. Agency officials noted that the agency will develop inspection protocols that require inspectors to collect data on the use of new technologies after their proposed integrity management rule for natural gas transmission pipelines is finalized.

OPS is also considering the number of documented R&D “success stories”—summaries of the accomplishments of individual R&D projects— as a possible measure of program results. However, in previous reviews of R&D programs operated by other federal agencies, we have found that the success story approach is selective and does not adequately assess programwide performance.

In early June 2003, OPS presented a potential set of performance measures for its R&D program to its R&D planning panel of outside experts in order to obtain their views on these measures. This panel includes representatives of DOE, MMS, the National Institute of Standards and Technology, pipeline industry associations, state agencies with pipeline responsibilities, and a key pipeline research organization. OPS intends to refine its set of measures based on comments received from this panel and to continue obtaining the views of this panel as it moves forward in developing an evaluation process for its R&D program.

Finally, OPS officials also told us that the agency intends to obtain the views of experts on its R&D outcomes as well as on its future R&D priorities at its next R&D workshop, scheduled for the winter of 2003-04. However, OPS is in the beginning stages of planning this workshop and has not defined a process for using experts’ views to evaluate the outcomes of its R&D program.

OPS officials told us that they are considering including information on the effectiveness of the agency’s R&D program in the annual reports to Congress on pipeline R&D that the agency is required to submit, starting in December 2003. The Pipeline Safety Improvement Act requires that DOT, DOE, and the National Institute of Standards and Technology jointly provide these annual reports to Congress, but does not fully specify what types of information should be included in these reports.

Best Practices Help Agencies Systematically Evaluate Research Outcomes

Since OPS is in the beginning stages of developing an evaluation process for its R&D program, it could benefit from adopting best practices for systematically evaluating federal R&D programs. Leading organizations that conduct scientific and engineering research, the Office of Management and Budget, and GAO have identified a number of these best practices. Although the uncertain nature of research outcomes over time can make it challenging to demonstrate the results of such R&D programs, these practices are designed to enable agencies to systematically assess and report on these results regularly in accordance with the Government Performance and Results Act of 1993. These assessments can be used to refocus the direction of R&D programs periodically, as necessary, to ensure that resources are most effectively utilized. Identified best practices are discussed in the following sections.

Setting Clear, Quantifiable Goals and Measuring Progress toward These Goals

We have previously reported that, to be effective, any R&D program must be directed toward a clear, measurable goal. Such goals help ensure a direct linkage between R&D program efforts and an agency’s overall performance goals and mission. Applied research programs, such as OPS’s R&D program, are directed toward achieving specific useful outcomes, such as the development of new technologies, which can help accomplish agency performance goals. The Committee on Science, Engineering, and Public Policy recommended in a 1999 report that agencies operating applied research programs measure progress toward practical outcomes and noted that such measurement can usually be performed annually using milestones.

Similarly, in May 2002 the Office of Management and Budget established investment criteria for federal R&D programs that require these programs to clearly define goals and track progress toward these goals using appropriate outcome measures and interim milestones. Indicators that have been used to measure the outcomes of R&D include the achievement of specific targets for developing new or improved technologies and patent applications filed and granted. However, measuring research outcomes can be challenging. For example, outcomes may not occur for a number of years and may be difficult to track.

Using Expert Review to Evaluate the Quality of Research Outcomes

In its 1999 report and again in 2001, the Committee on Science, Engineering, and Public Policy recommended the use of expert review, supplemented by quantitative methods, to evaluate research regularly. Expert review can be a useful addition to performance measures because of the value of the reviewers’ deep knowledge of the field. Such review can be performed on a somewhat longer term basis, rather than annually, and does not require that the final impact of the research be known. Peer review, a form of expert review, includes an independent assessment of the technical and scientific merit or quality of research by peers with essential subject matter expertise and perspective equal to that of the researchers. In 1999, we reported that some federal agencies, such as the Department of Agriculture, the National Institutes of Health (NIH), and DOE, use peer review to help them evaluate the performance of programs and determine whether to continue or renew research projects.

The Committee on Science, Engineering, and Public Policy reported in 2001 on the use of expert review, including peer review, by NIH, DOE, the National Science Foundation, the Department of Defense, and the National Aeronautics and Space Administration to evaluate the quality of their research programs. These agencies used varying methods for carrying out this review, including convening panels of experts who use defined evaluation processes and obtaining the views of external advisory committees. The Committee on Science, Engineering, and Public Policy has also noted that expert evaluation of applied research programs requires the input of potential users of the results of the research, since the ultimate usability of these results is an important factor in determining the worth of the research. Similarly, key experts and stakeholders we interviewed noted that the degree to which new technologies are actually used would be a good indication of the effectiveness of OPS’s R&D program. One industry association representative we interviewed noted that a “constant theme” raised by pipeline companies is the need for R&D efforts to produce new technologies that they can actually use in operating their pipelines.

Reporting Periodically on Evaluation Results

Periodic reporting by applied research programs on results can help keep key stakeholders—including oversight organizations and potential users of new technologies—up-to-date on program accomplishments. According to the Committee on Science, Engineering, and Public Policy, applied research programs can usually report annually on progress in meeting milestones. In addition, a retrospective analysis over several years is needed to evaluate outcomes that take more than 1 year to emerge. The committee also has recommended that agencies demonstrate the value of their review processes by publicly describing them to oversight groups, the potential users of research results, and the general public. One expert we interviewed stressed the importance of periodic public reporting by OPS on research goals and outcomes and on the method for evaluating outcomes, in order to disseminate research results and build support for its R&D program.

Conclusions

OPS has made significant progress in establishing a pipeline safety research agenda that is aligned with its mission and goals and that incorporates the views of experts and stakeholders. However, without a systematic process for evaluating the outcomes of its R&D program, the agency is not able to demonstrate that it is effectively using its increased resources for R&D to foster new and improved technologies that can enhance pipeline safety. Identified best practices for evaluating federal R&D programs—including setting clear quantifiable R&D goals and measuring progress toward these goals, using expert review to evaluate the quality of research outcomes, and reporting periodically on evaluation results—can guide OPS as it moves forward in developing an evaluation process for its program. By following such practices, the agency can help ensure that it develops a systematic evaluation process that will enable it to determine and demonstrate the results of its investment in pipeline safety R&D. OPS could use such an evaluation process to periodically refocus the direction of its program in order to make the most effective use of resources.

Furthermore, although the Pipeline Safety Improvement Act’s requirement for annual reports on pipeline R&D, starting in December 2003, does not specify in detail what information should be included in these reports, this requirement provides an opportunity for the agency to keep Congress informed about the results of evaluations of its R&D program. In addition, such reporting, along with other communication methods already in use by the agency, can keep other interested parties—including the pipeline industry, state pipeline safety agencies, pipeline safety advocates, and researchers—up-to-date on the program’s progress in advancing the most promising pipeline safety technologies.

Recommendations for Executive Action

To improve OPS’s ability to demonstrate the effectiveness of its R&D program and make the most effective use of program funds, we recommend that the Secretary of Transportation direct OPS to develop a systematic process for evaluating the outcomes of its R&D program that incorporates identified best practices and include in the annual reports to Congress, which are required by the Pipeline Safety Improvement Act, information on the results of R&D evaluations.

Agency Comments

We provided DOT with a draft of this report for review and comment. DOT officials, including OPS’s Director of Program Development, provided oral comments on the draft on June 13, 2003. The officials generally agreed with the report’s findings and conclusions. They emphasized that they are starting to develop a framework for evaluating the effectiveness of their pipeline safety R&D program and that they intend to finalize this framework by December 2003 by documenting it in the 5-year plan and first annual report on pipeline R&D that DOT is required to submit to Congress, jointly with DOE and the National Institute of Standards and Technology. They also noted that they agree with and intend to implement our recommendations and provided some technical clarifications, which we have incorporated as appropriate.

We are sending copies of this report to the Secretary of Transportation, the Administrator of RSPA, RSPA’s Associate Administrator for Pipeline Safety, the Director of the Office of Management and Budget, and appropriate congressional committees. We will make copies available to others upon request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-2834 or guerrerop@gao.gov. Individuals making key contributions to this report are listed in appendix III.

Experts’ Views on R&D Priorities and OPS’s R&D Funding, by Type of R&D

We asked selected experts to review the following descriptions of specific types of pipeline safety research and development (R&D) and assign a funding priority to each, based on its importance in achieving the Office of Pipeline Safety’s (OPS) mission of ensuring the safe, reliable, and environmentally sound operation of the nation’s pipeline transportation system. Experts used the following scale: 1=little or no funding, 2=some funding priority, 3=moderate funding priority, 4=high funding priority, and 5=very high funding priority. Experts could also indicate that they did not know or had no basis to judge the funding priority for a particular type of R&D. The following table shows, for each type of R&D, the number of experts who assigned it a high or very high funding priority and OPS’s current and planned allocation of funding to it. A total of 49 experts completed our questionnaire.

Allocated $592,500 to five projects in November 2002 for periods of 9 to 24 months.

Allocated $500,000 to one project in November 2002 for a period of 24 months.

Allocated $182,000 to one project in April 2001 for period of 12 months. Requested proposals in March 2002 but did not fund any of those received. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Allocated $534,521 to two projects in July 2002 for periods of 23 to 24 months.

Requested proposals in March 2002 but did not fund any of those received. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

OPS’s current and planned allocation of funding to this type of R&D Requested proposals in March 2002 but did not fund any of those received. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in March 2002 but did not fund any of those received. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

Allocated $600,000 to one project in April 2001 for a period of 12 months. Allocated an additional $600,000 to this project in April 2002 for an additional 12 months. Plans to allocate an additional $600,000 to this project in May 2003 for an additional 12 months.

Allocated $572,000 to two projects in January 2003 for periods of 12 to 26 months.

Allocated $297,000 to one project in January 2003 for a period of 26 months.

Allocated $275,000 to one project in January 2003 for a period of 12 months.

Allocated $675,281 to four projects in May and July 2002 for periods of 12 to 24 months.

Allocated $97,737 to three projects in May 2002 for a period of 12 months. Allocated $70,000 to an additional project in January 2003 for a period of 24 months. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in June 2002 but did not fund any of those received. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Allocated $98,680 to one project in November 2002 for a period of 6 months. Requested additional proposals in December 2002 and plans to make funding decisions in July 2003.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Allocated $7,781 to one project in May 2002 for a period of 12 months.

Allocated $50,000 to one project in May 2001 for a period of 12 months.

Allocated $59,955 to one project in May 2002 for a period of 12 months.

Requested proposals in December 2002 and plans to make funding decisions in July 2003.

Scope and Methodology

To perform our work, we reviewed Office of Pipeline Safety (OPS) documentation on its research and development (R&D) funding and analyzed this information to identify trends; reviewed pertinent legislation and agency documents pertaining to the R&D program; and interviewed OPS officials regarding their R&D funding, agenda-setting processes, and processes for evaluating the outcomes of their R&D program. We also interviewed key experts and stakeholders concerning OPS’s management of its R&D program, including the alignment of the agency’s research agenda with its mission and goals, and their views on R&D priorities and gaps. These individuals included officials of the Department of Energy (DOE) and the Department of the Interior’s Minerals Management Service (MMS) who are responsible for pipeline R&D; representatives of pipeline industry associations and leading pipeline research organizations; and several key experts in pipeline safety. Also, we identified best practices for evaluating the outcomes of federal R&D through a review of relevant literature and compared the agency’s processes with these practices.

To determine the views of experts on pipeline safety R&D priorities, we sought to identify experts considered to be very knowledgeable about the development of new pipeline safety technologies or pipeline safety issues. To identify appropriate experts, we obtained recommendations on individuals to contact from key organizations, contacted those individuals, and obtained further recommendations from them on additional individuals to contact. We identified initial individuals to contact through prior work on pipeline safety issues or through recommendations from OPS. These initial contacts included officials in DOE and MMS, representatives of four industry associations, a former head of a state agency that regulates gas pipelines, the heads of two leading pipeline R&D organizations, two technical experts in pipeline safety, and an environmentalist active in pipeline safety. Six of these individuals have been or are members of OPS advisory committees or R&D planning or review panels. We obtained recommendations from these individuals on experts who could provide us with views on pipeline safety R&D priorities.

We based our final selection of experts on the criteria of knowledge, balance, and independence. We considered indications of their extent of knowledge of pipeline safety R&D, as evidenced by the number of times they had been recommended, their participation in OPS’s R&D planning and review activities, or other relevant factors. We included individuals from a variety of groups in order to achieve a balanced representation of experts, including some who are relatively independent of OPS and the pipeline industry. We included individuals from federal and state agencies, pipeline safety advocacy groups, industry associations, pipeline companies, technical and consulting organizations, and research organizations. We also provided our list of identified experts to the National Academy of Sciences and OPS for their review and comment.

We contacted 55 individuals whom we had identified as appropriate experts for our review and asked them to complete a questionnaire indicating their views on pipeline safety R&D priorities. Forty-nine individuals responded, for an 89 percent response rate. Our results pertaining to experts’ views on R&D priorities represent the views of only the experts who responded to our questionnaire. In a number of cases, these individuals collaborated with others in their organizations in completing their questionnaires. Listed below are the organizational affiliations of experts who completed our questionnaire.

Government and Public Interest Organizations Federal Agencies Federal Energy Regulatory Commission Minerals Management Service, Department of the Interior National Institute of Standards and Technology, Department of Commerce National Transportation Safety Board Office of Fossil Energy, Department of Energy State Agencies and Associations National Association of Pipeline Safety Representatives National Association of Regulatory Utility Commissioners New York State Department of Public Service Railroad Commission of Texas Virginia State Corporation Commission Washington Utilities and Transportation Commission Pipeline Safety Advocacy Groups Common Ground Alliance Cook Inlet Keeper Safe Bellingham Pipeline Industry and Technical/Consulting Organizations Industry Associations American Gas Association American Petroleum Institute Association of Oil Pipelines Interstate Natural Gas Association of America Pipeline Companies BP Pipelines, North America ConocoPhillips CMS Panhandle Companies Duke Energy El Paso Corporation Enbridge Pipelines Enron Explorer Pipeline Company ExxonMobil Pipeline Company KeySpan Energy Peoples Energy Technical/Consulting Organizations Accufacts, Inc. Batten and Associates, Inc. Duckworth Pipeline Integrity Services, Inc. HSB Solomon Kiefner and Associates, Inc.

National Association of Corrosion Engineers Advantica, Inc. Battelle CFER Technologies Edison Welding Institute Gas Technology Institute Ohio State University, Fontana Corrosion Center Pipeline Research Council International, Inc. Southwest Research Institute Texas A&M University, Department of Mechanical Engineering University of Florida, Department of Chemical Engineering In the questionnaire, we asked respondents to review descriptions of various main categories of pipeline safety R&D as well as specific types of R&D within these main categories and indicate what funding priority they would assign to each. (See table 1 for descriptions of the main R&D categories. See app. I for descriptions of the types of R&D within these main categories.) We based the R&D categories and descriptions on materials prepared as part of an R&D planning workshop held by OPS in 2001, in which a variety of experts and stakeholders participated; on announcements the agency subsequently issued soliciting proposals for R&D in various areas; and on other OPS documents related to pipeline safety R&D.

We compiled the scores obtained from the questionnaires to produce a ranking of R&D priorities representing the views of the experts who completed our survey. We also analyzed our results to determine whether any differences existed in the responses of experts from the three subgroups: government and public interest organizations, industry and technical and consulting organizations, and research organizations. In addition, we identified organizations that had bid on R&D funding from OPS in fiscal year 2002 and conducted a separate analysis of the responses of experts from these organizations to determine how they compared with those of other experts who completed our questionnaire. Seven of the experts who completed our questionnaire are from organizations that had bid on OPS R&D funding within this time frame.

We conducted our work from January 2003 through June 2003 in accordance with generally accepted government auditing standards.

GAO Contacts and Staff Acknowledgments

GAO Contacts

Acknowledgments

In addition to those named above, Sharon Dyer, Etana Finkler, Judy Guilliams-Tapia, Brandon Haller, Bert Japikse, Nancy Kingsbury, Donna Leiss, Gary Stofko, Ron Stouffer, and Stacey Thompson made key contributions to this report.

GAO’s Mission

The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to daily E-mail alert for newly released products” under the GAO Reports heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs