United States General Accounting Office

GAO Report to Congressional Committees

September 1998 NUCLEAR WASTE

Further Actions

Needed to Increase the

Use of Innovative

Cleanup Technologies

GAO/RCED-98-249

GAO United States

General Accounting Office

Washington, D.C. 20548

Resources, Community, and

Economic Development Division

B-280628 

September 25, 1998

Congressional Committees

As requested, this report addresses the deployment rate for technologies developed by the

Office of Science and Technology in the Department of Energy’s (DOE) Environmental

Management program and compares this rate with those of other similar organizations. The

report also addresses obstacles to the use of innovative technologies at DOE’s cleanup sites and

the Environmental Management program’s efforts to overcome these obstacles.

As arranged with your offices, unless you publicly announce its contents earlier, we plan no

further distribution of this report until 15 days after the date of this letter. At that time, we will

provide copies of the report to interested congressional committees, the Secretaries of Energy

and Defense, the Administrator of the Environmental Protection Agency, the Director of the

Office of Management and Budget, and other interested parties. We will also make copies

available to others on request.

Please call me at (202) 512-8021 if you or your staff have any questions. Major contributors to

this report are listed in appendix V.

(Ms.) Gary L. Jones

Associate Director, Energy,

 Resources, and Science Issues
B-280628 

List of Committees

The Honorable Tom Bliley

Chairman

The Honorable John D. Dingell

Ranking Minority Member

Committee on Commerce

House of Representatives

The Honorable Joe Barton

Chairman

The Honorable Ron Klink

Ranking Minority Member

Subcommittee on Oversight

 and Investigations

Committee on Commerce

House of Representatives

Page 2 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
B-280628 

Page 3 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

The Department of Energy (DOE) estimates that it could save $20 billion or

more by using innovative technologies to clean up the hazardous and

radioactive contaminants resulting from the production of nuclear

weapons at its facilities. To achieve these savings, the Office of Science

and Technology (OST),1

 within DOE’s Office of Environmental Management

(EM), is developing technologies that could reduce cleanup costs,

accelerate cleanups, provide methods for cleanup activities for which

there are no existing cost-effective technologies, and/or reduce risks to

cleanup workers and the public. However, previous reports by GAO and

others have identified obstacles that make it difficult to select and use

innovative technologies at DOE sites.

Because of concern about the benefits returned from the approximately

$2.5 billion invested in OST since 1989, the House Committee on Commerce

and its Subcommittee on Oversight and Investigations requested that GAO

review EM’s efforts to deploy innovative technologies. Specifically, GAO was

asked to determine (1) to what extent innovative technologies developed

by OST have been deployed (used) at DOE sites and how this rate of

deployment compares with the rates of other government organizations

that develop environmental technologies; (2) what progress EM has made

in overcoming obstacles to deploying innovative technologies at DOE

cleanup sites; and (3) what EM is doing to increase the deployment of

innovative technologies.

Background In 1989, the Congress directed DOE to establish a research program to

develop technologies to reduce environmental hazards and restore the

environment. OST develops environmental technologies from the early

stages, which involve laboratory experimentation, through the later stages,

which involve the demonstration and testing of technologies’ performance.

OST is organized into “focus areas” that concentrate on the major cleanup

problems facing DOE sites. Among DOE’s cleanup challenges being

addressed by OST’s technology research and development are the

following: massive underground tanks that contain high-level radioactive

wastes, migrating areas of hazardous and radioactive substances in

groundwater, acres of contaminated soil, and thousands of buildings no

longer in use that require decontamination and dismantlement. DOE sites

are responsible for selecting the technologies to be used in cleanup

projects. These selections are subject to the review and approval of the

1

This office was originally named the Office of Technology Development, but it was later renamed

because basic science research for waste cleanup was added to its responsibilities.

Page 4 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

Environmental Protection Agency (EPA) and the state agencies that

regulate DOE’s cleanups.

Results in Brief The Office of Science and Technology has initiated 713 technology

development projects and has reported that 152 projects have been

deployed one or more times, for an overall deployment rate of 21 percent.

GAO found many errors in the office’s deployment data and estimates that

Environmental Management has deployed between 88 and 130 of these

projects, for an overall deployment rate of 12 to 18 percent. The Office of

Science and Technology overstated its deployment information because it

had not previously maintained comprehensive deployment data; compiled

the data rapidly in response to congressional requests; and lacked

procedures for compiling the data, such as a formal definition of what

constitutes a deployment. In comparison with the deployment rates of

other programs that demonstrate environmental technologies—the

Environmental Protection Agency’s Superfund Innovative Technology

Evaluation Program and the Department of Defense’s Environmental

Security Technology Certification Program—the Office of Science and

Technology’s deployment rate for projects at comparable stages of

development falls between the rates of these two programs. However,

comparisons of the Office of Science and Technology’s deployment rate

with the rates of other organizations’ programs must be viewed with

caution because no organization is fully comparable with the Office of

Science and Technology, and the deployment rate is not the only possible

measure of success for research and development programs.

As DOE’s Environmental Management program has matured, DOE waste

cleanup sites have made progress in overcoming some obstacles to

implementing innovative technologies, such as addressing the concerns of

regulators and public stakeholders. However, other obstacles that are

internal to the operations of the Office of Environmental Management and

its Office of Science and Technology continue to slow the use of

innovative technologies. These obstacles include the lack of

(1) involvement by technology users in the development of cleanup

technologies by the Office of Science and Technology and (2) technical

assistance by the Office of Science and Technology to help sites select and

implement technologies.

After congressional hearings in May 1997, the Office of Environmental

Management initiated changes in its organization and processes to

increase the deployment of innovative technologies. Some of these

Page 5 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

initiatives address the internal obstacles limiting deployment. For

example, one initiative provided for involving technology users in setting

overall plans and priorities for the Office of Science and Technology, such

as the general areas of research in which the office should invest.

However, the office has not yet improved developer-user cooperation in

individual technology development projects. Furthermore, the Office of

Science and Technology does not consistently and rigorously use its

existing decision-making process for managing the development of

innovative technologies. This process would require technology users to

be involved at various stages in a technology’s development. In addition,

the Office of Environmental Management has yet to determine how it will

(1) provide technical assistance to sites in selecting and implementing

innovative technologies and (2) make modifications to completed

technologies to meet sites’ specific needs and conditions. These steps

could increase the use of completed technology development projects

when they present benefits over the use of conventional approaches.

Principal Findings

Deployment Rate of

OST-Developed

Technologies

OST developed deployment data in response to a November 1996 request

from the Subcommittee on Oversight and Investigations, House

Committee on Commerce. As of January 1998, OST’s database showed that

EM had initiated 713 technology projects since OST’s inception. On the basis

of its independent verification and analysis of OST’s deployment data, GAO

estimates that EM actually has deployed from 88 to 130 projects, to achieve

an overall deployment rate of 12 to 18 percent for the 713 projects. In

contrast, OST has reported that 152 of the 713 technology projects initiated

since its inception have been deployed. Thus, according to OST’s data,

about 1 in 5 OST technologies have been deployed one or more times, for

an overall deployment rate of 21 percent. (See app. I for more detail on

GAO’s methodology.)

OST’s inaccurate deployment data resulted from several factors.

Specifically, OST had not maintained comprehensive deployment data and

compiled the data quickly in response to the congressional request. In

addition, the lack of a formal definition of deployment led to differing

understandings among the personnel responsible for compiling the data.

OST has begun to establish procedures for entering and updating project

data and, in August 1998, formalized a definition of deployment. In the

Page 6 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

interim, however, OST has continued to report inaccurate data to the

Congress. For example, OST’s latest semiannual report to the Congress,

dated April 1998, stated that, through September 1997, over 140

OST-sponsored technologies had been deployed.

GAO compared OST’s deployment rate with those of the Environmental

Protection Agency’s Superfund Innovative Technology Evaluation

Program and the Department of Defense’s Environmental Security

Technology Certification Program. (See app. II for more detail on these

comparisons.) For projects at comparable later stages of development,

OST’s deployment rate ranges from 28 to 45 percent,2 while the Superfund

program’s rate is 59 percent and the Environmental Security program’s

rate is 38 percent. However, comparisons of OST’s deployment rate with

the rates of other organizations must be viewed with caution for several

reasons. For example, GAO found few organizations that engage in the

range of environmental research OST performs, and none of the 10

organizations that GAO contacted routinely tracked deployment data on

their projects. It should also be noted that, while GAO was asked to develop

deployment information, the deployment rate is not the only performance

measure relevant to research and development programs and is more

applicable to technologies in the later stages of development.

Furthermore, developers of later-stage technologies believe that the

deployment rate is an incomplete performance measure and that cost

savings or some measure of dollar impact should also be used to evaluate

a program’s success. EM has used the completion of demonstrations as a

performance measure for OST for several years and is considering

developing additional measures more relevant to earlier stages of

technology development, as well as a performance measure addressing

cost savings resulting from using innovative, rather than conventional,

technologies.

Obstacles to Deploying

Innovative Technologies at

DOE Sites

As EM’s cleanup program has matured, it has addressed several of the

obstacles to using innovative technologies reported by GAO, EM, and others.

For example, one reported obstacle was a lack of awareness among DOE

field staff about available technologies. In visits to DOE’s waste cleanup

sites, GAO found that federal and contractor staff had become better

informed than in the past about innovative technologies that are relevant

to their cleanup needs. In addition, DOE sites and their regulators had

improved their working relationships, and, in cases where innovative

2

Since GAO used only a sample of OST’s projects to make its estimates, these estimates have a margin

of error. The range stated reflects this margin of error at the 95-percent confidence level. See app. I for

details about GAO’s methodology.

Page 7 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

technologies were selected, DOE sites had found ways to address

regulatory concerns. In several cases, sites had phased in technologies to

increase the regulators’ confidence in the technologies’ performance.

On the other hand, several obstacles that are internal to EM’s and OST’s

operations continue to hinder the deployment of OST-developed

technologies. Specifically, OST has not involved users as technologies are

being developed. OST did not comprehensively identify the sites’

technology needs until October 1996 and has not involved technology

users in the decision-making process (referred to as the gates system)

during the development of technologies. As one result of the lack of user

involvement, site staff find that some OST technologies are generic and do

not meet their specific conditions and performance requirements. For

example, officials at DOE’s Hanford site tried two OST technologies that

promised to characterize the nature and extent of contamination in soil

more quickly than conventional methods and thus enable the site to

remediate the contamination sooner. The officials, however, rejected the

technologies because they were not designed to work effectively in

Hanford’s arid soil.

Furthermore, when technologies require modification to fit a site’s specific

situation, site officials told GAO that it is not clear who should pay for these

modifications. For example, Hanford officials wanted to use OST’s

Electrical Resistance Tomography equipment to help detect leaks in waste

tanks from outside the tanks, but Hanford did not have funding to

fine-tune the equipment for its needs. While OST has data identifying

potential uses for its technologies at specific cleanup projects, EM lacks a

policy on who is responsible for paying for any necessary modifications to

adapt the OST-developed technologies for specific field uses.

Finally, OST has infrequently provided technical assistance to sites during

technology selection or implementation. GAO recommended in 1994 that EM

give OST a formal role in technology selection decisions.3 However, the

recommendation was not implemented because site personnel lack

confidence in OST’s ability to provide expert technical advice and

assistance and are therefore reluctant to allow OST a formal role in their

technology selections. In visits to five DOE sites, GAO found that the sites

infrequently sought technical assistance from OST. According to a report by

an advisory board to EM, OST staff are not always well informed about

technologies developed by organizations other than OST. OST’s managers

3

Department of Energy: Management Changes Needed to Expand Use of Innovative Cleanup

Technologies (GAO/RCED-94-205, Aug. 10, 1994).

Page 8 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

recognize that staff in its focus areas do not always have sufficient

technical expertise to assist sites with their specific cleanup problems.

EM’s Efforts to Increase

Deployment

In response to congressional hearings and concerns, the Assistant

Secretary for EM identified actions intended to increase the deployment of

innovative technologies, such as establishing the Technology Acceleration

Committee, which is composed of upper managers from EM headquarters

and field offices; identifying responsibilities and performance measures

for deployment; and requiring sites to develop deployment plans. These

actions were completed over time—some in the fall of 1997 and others in

the spring and summer of 1998. For instance, for fiscal year 1998, EM

established performance measures on technology deployment for its sites.

In addition to EM’s actions, OST is requiring multiyear plans for its five

focus areas; and in February 1998, EM established user steering

committees for the focus areas to assist with these plans and priority

setting.

However, continued attention by EM’s upper management to deployment is

not ensured because the Technology Acceleration Committee may not

remain in effect following the departure of EM’s Assistant Secretary and

because planned deployment measures for managers have not been

implemented. Although the actions initiated by the Assistant Secretary for

EM were to include deployment measures in the annual performance

expectations of EM’s upper managers, EM has not carried out this aspect of

its planned performance measures. Instead, EM managers’ involvement in

the Technology Acceleration Committee was considered to be a sufficient

means of holding the managers accountable for deployment. This

Committee has not met since January 1998, and EM is considering

establishing a new executive committee of senior managers to address EM

issues, including deployment.

Furthermore, the actions specified by the Assistant Secretary for EM do not

include user involvement in individual OST projects—the type of

involvement needed to ensure that completed technologies address sites’

requirements. Although the new user steering committees help OST in its

overall planning and priority setting, user involvement in individual

technology development projects was not addressed by the actions

specified by the Assistant Secretary for EM. Meanwhile, existing processes

that would require user involvement are not used. In 1993, OST developed

the decision-making system known as the gates system, which

(1) establishes requirements for technology projects at various stages of

Page 9 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

development and (2) identifies decision points, called gates, at which

projects are evaluated and “go/no-go” decisions are made. The gates

system includes requirements for user involvement at the various

development stages. For example, gate 2—moving from applied research

to exploratory development—requires that research be linked to the

specific needs of end-users at sites and that end-users’ performance

requirements are specified. However, OST’s use of the gates system has

been spotty, in part because a rigorous application of its requirements

might indicate that some projects should be terminated for reasons such

as the lack of an identified customer, according to several EM officials.

In addition, the actions specified by the Assistant Secretary for EM are not

designed to increase the deployment—and return on investment—of

completed OST technologies. While site officials consider OST’s

technologies too generic, the actions do not address modifying those

technologies so that they could have additional cost-effective uses. The

initiatives also do not specify policies, resources, or processes for

providing sites with technical assistance on innovative technologies.

Recommendations To ensure the deployment of future technology development projects and

increase the use of innovative technologies that OST has completed, GAO

recommends that EM (1) consistently use the gates system to make

decisions about OST’s projects and to increase the involvement of users in

technology projects, (2) use existing data to identify technologies that

could be used cost-effectively for site cleanup projects and identify funds

for modifying these technologies if needed, and (3) establish centers of

expertise for innovative technologies by using existing focus areas or

another approach if needed and require that a representative from one of

these centers participate in the technology selection process on each

cleanup project. GAO also makes several other recommendations to

improve deployment data and upper management’s attention to

technology deployment.

Agency Comments GAO provided a draft of this report to DOE for its review and comment.

Overall, DOE stated that it agreed with the findings and recommendations

of the draft report. DOE did provide specific comments and corrections,

and GAO made changes where appropriate. DOE’s comments are included as

appendix III. GAO also provided portions of the draft of this report

concerning the Environmental Security Technology Certification Program

to the Department of Defense and portions concerning the Superfund

Page 10 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Executive Summary

Innovative Technology Evaluation Program to the Environmental

Protection Agency. The Department of Defense stated that it found the

report to be correct with respect to its program and provided specific

comments and corrections that GAO incorporated where appropriate. The

Department’s comments are included as appendix IV. Similarly, the Acting

Director of the Environmental Protection Agency’s Office of Resources

Management and Administration told GAO that the information on the

Superfund Innovative Technology Evaluation Program contained in the

report was summarized and reported accurately.

Page 11 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Contents

Executive Summary 4

Chapter 1 

Introduction

14

OST Established to Develop Innovative Technologies 14

Use of Innovative Technology May Have Benefits, Including the

Reduction of Cleanup Costs

16

GAO Has Reported in the Past on Problems in Managing

Technology Development and Deployment

17

Objectives, Scope, and Methodology 18

Chapter 2 

EM Has Deployed

Less Than 20 Percent

of OST’s Projects

21

EM Has Deployed About 88 to 130 of OST’s Technology Projects 21

Several Factors Have Led to OST’s Inaccurate Deployment Data 22

Deployment Rate for OST Technologies Falls Between the Rates

of Two Other Environmental Technology Programs

24

Deployment Rate Comparisons Between OST and Other

Environmental Technology Developers May Have Limited

Usefulness

25

Chapter 3 

Some Obstacles

Continue to Slow

Deployment of

Innovative

Technologies

29

Sites Have Overcome Obstacles to Using Some Innovative

Technologies

29

OST Has Developed Innovative Technologies Without Sufficient

User Involvement

31

OST Has Not Provided Some Needed Technical Assistance 36

Chapter 4 

EM’s Efforts to

Increase Deployment

Do Not Fully Address

Cooperation Between

Technology

Developers and Users

38

EM Has Established Initial Deployment Performance Measures

for Field Sites and Is Developing Additional Measures

39

EM Is Increasing User Input to OST’s Planning and Priority

Setting

41

Continued Upper Management Attention to Deployment Is Not

Ensured

42

EM and OST Initiatives Do Not Fully Address User Input to

Technology Projects and User Need for Deployment Assistance

44

Page 12 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Contents

Chapter 5 

Conclusions and

Recommendations

48

Recommendations 49

Agency Comments and Our Evaluation 50

Appendixes Appendix I: Methodology Used to Assess the Reliability of OST’s

Data

52

Appendix II: OST’s Deployment Rate Compared With the Rates of

Two Governmental Environmental Technology Demonstration

Programs

55

Appendix III: Comments of the Department of Energy 58

Appendix IV: Comments of the Department of Defense 63

Appendix V: Major Contributors to This Report 65

Tables Table 2.1: Deployment of OST-Developed Technologies 22

Table 2.2: OST’s Deployment Rate Compared With the Rates of

Other Environmental Technology Demonstration Programs

25

Table 2.3: Environmental Technology Development Entities

Contacted

26

Figure Figure I.1: Universe and Sample of OST Projects 54

Abbreviations

ASEM Assistant Secretary for Environmental Management

ASTD Accelerated Site Technology Deployment

DOE Department of Energy

EM Environmental Management

EMAB Environmental Management Advisory Board

EPA Environmental Protection Agency

ESTCP Environmental Security Technology Certification Program

GAO General Accounting Office

OST Office of Science and Technology

R&D research and development

SITE Superfund Innovative Technology Evaluation

Page 13 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

Nearly a decade ago, the Department of Energy (DOE) embarked on a

mission to deal with the environmental legacy of the Cold War. This DOE

mission, which is expected to continue for many years, involves a number

of activities, with the most ambitious and far-ranging being the cleanup of

the hazardous and radioactive contaminants that resulted from the

production of nuclear weapons at DOE facilities. The challenges of this task

are technical, institutional, and economic. For example, thousands of tons

of radioactive waste must be treated and put into permanent storage;

contaminated soil must be stabilized; contaminated water must be treated;

and nuclear reactors and materials-processing facilities must be

decontaminated, decommissioned, and demolished. In some cases, no safe

and effective technology is currently available to address the more

complex contamination problems.

In June 1998, DOE estimated that it will be very expensive, about

$150 billion, to clean up the legacy of the Cold War.1

 However, DOE has

also stated that the cost of cleaning up its contaminated facilities and sites

can be significantly reduced through the use of innovative cleanup

technologies. It supports the development of these technologies through

its Office of Science and Technology (OST), within the Office of

Environmental Management (EM).

OST Established to

Develop Innovative

Technologies

In 1989, DOE established EM to clean up and restore its contaminated

facilities and sites in compliance with federal and state environmental

laws and regulations. The Congress also directed the Secretary of Energy

to establish “a program of research for the development of technologies

useful for (1) the reduction of environmental hazards and contamination

resulting from defense waste, and (2) environmental restoration of

inactive defense waste disposal sites.”

In response, DOE established the Office of Technology Development within

EM to develop innovative technologies to support the waste cleanup and

restoration efforts of EM’s program offices—the Offices of Waste

Management, Environmental Restoration, and Nuclear Material and

Facility Stabilization. The Office of Technology Development was renamed

the Office of Science and Technology in 1994, when basic science research

for waste cleanup was added to its responsibilities. OST’s projects are

intended to produce technologies that could accelerate cleanups, reduce

costs, enable cleanup activities for which there are no existing

cost-effective technologies, or reduce risks to cleanup workers. From

1

Accelerating Cleanup: Paths to Closure, DOE/EM-0362, June 1998.

Page 14 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

fiscal year 1990 through fiscal year 1998, the Congress appropriated

approximately $2.5 billion for OST’s development of innovative waste

cleanup technologies,2

 and OST has initiated over 700 projects. OST’s budget

for technology development activities in fiscal year 1998 is about

$220 million. OST requested a total of $180.5 million for technology

development activities for fiscal year 1999.3

OST develops technology at DOE’s national laboratories, private companies

under contract to OST, and universities. Although OST is responsible for

technology development, DOE waste sites are responsible for selecting the

technologies they will use, with the review and approval of the

Environmental Protection Agency (EPA) and state agencies that regulate

DOE’s cleanups, and with input from the public involved with the site.4

To serve sites’ needs for cleanup technology, OST is organized into five

major remediation and waste management problem areas (termed “focus

areas”). OST first established focus areas in 1994 in order to better serve

the cleanup sites by concentrating technology resources on each of the

major cleanup problems DOE faces. OST currently has the following five

focus areas:

• Mixed Waste Characterization, Treatment, and Disposal. Known as “mixed

waste,” this focus area addresses the large inventory of mixed, low-level,

and transuranic waste;5

• Radioactive Tank Waste Remediation. Known as “tanks,” this focus area

addresses the hundreds of large storage tanks containing over 100 million

gallons of radioactive waste;

• Subsurface Contaminants. This focus area addresses hazardous and

radioactive contaminants in soil and groundwater and the remediation

challenges posed by numerous DOE landfills;

2

Another $353 million was appropriated for basic science research and for activities not directly

related to technology development. While OST continues to be responsible for basic research, these

other responsibilities have since been moved to other parts of EM or DOE.

3

Figures for fiscal years 1998 and 1999 exclude funds for basic science research and funds to be

transferred to the Small Business Innovative Research Program.

4

Remediation activities at DOE’s facilities are governed by the Comprehensive Environmental

Response, Compensation, and Liability Act of 1980, as amended, and the Resource Conservation and

Recovery Act of 1976, as amended. These acts lay out the requirements for identifying waste sites,

studying the extent of their contamination and identifying possible remedies, and involving the public

in making decisions about the sites.

5

Mixed waste contains both radioactive and hazardous waste. Low-level waste contains radioactive

waste not classified as high-level waste, spent nuclear fuel, transuranic waste, or radioactive

by-product material. Transuranic waste contains radioactive elements that are above uranium in the

Periodic Table of elements—that is, elements that have an atomic number greater than 92. Transuranic

waste is produced artificially (during a man-made nuclear reaction).

Page 15 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

• Deactivation and Decommissioning. This focus area addresses the

deactivation, decommissioning, and disposal of aging and contaminated

DOE weapons complex facilities; and

• Plutonium Stabilization and Disposition. This focus area addresses the

over 20 tons of excess plutonium that must be stabilized.

OST has established a lead field office to manage each focus area. For

example, the Savannah River site manages the Subsurface Contaminants

Focus Area. EM has also established site technology coordination groups in

each of its field offices to identify sites’ technology needs, provide

information to OST and its focus areas, and communicate information

about OST’s technology development projects to the cleanup sites.

In 1994, the Assistant Secretary for EM established the Environmental

Management Advisory Board (EMAB) to provide the Assistant Secretary

with information, advice, and recommendations on issues confronting the

EM program, including advice on the development and deployment of

innovative technology for waste cleanup. EMAB has about 25 members from

industry; academia; and private, federal, tribal, state, and local

environmental groups. EMAB has been very active in studying OST and

recommending improvements in its operations.

Use of Innovative

Technology May Have

Benefits, Including the

Reduction of Cleanup

Costs

DOE believes it will be very costly and take many years to clean up its

waste sites if only conventional technology is used. For example, using the

conventional method of removing contaminants from groundwater can

involve pumping and treating the water for 30 years or more. In addition,

no technology exists to address some cleanup problems. For example, no

technology exists for some aspects of removing and treating the

radioactive waste now in large tanks at several major DOE facilities.

Furthermore, some cleanup activities could be dangerous or impossible

for cleanup workers unless innovative technologies, such as remote

robotic devices to clean inside radioactive waste tanks, are used.

Those in the Congress and in DOE who led the effort to establish OST

believed that the use of innovative technology would reduce the cost of

waste cleanup. For example, in 1995, DOE estimated that it would cost

between $200 billion and $350 billion and take another 75 years to

complete the cleanup. However, DOE also estimated that the use of new

technologies could reduce cleanup costs by a minimum of $9 billion to as

Page 16 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

much as $80 billion, depending on the cleanup scenario.6 More recently, in

1997, the Army Corps of Engineers reviewed cost savings estimates

developed by OST for 37 of its technology projects and concluded that

these 37 projects could potentially save about $20 billion over the use of

conventional technology. DOE believes that cleanup costs could

significantly exceed current estimates if innovative technology is not used.

GAO Has Reported in

the Past on Problems

in Managing

Technology

Development and

Deployment

We have issued a number of reports and testified on the operation and

management of EM’s technology program. Among other things, we have

identified obstacles to the deployment of innovative technology at DOE’s

cleanup sites. In 1992, we reported that EM had not established key

management tools, such as cost estimates and schedules, and decision

points for evaluating technology projects.7

 In January 1993, EM

implemented a management plan for the technology program that

incorporated our recommendations. The program established cost

estimates and schedules for projects. EM also developed decision points

(called gates) and related requirements for evaluating projects and making

“go/no-go” decisions.

In 1994, we reported that officials at DOE cleanup sites may not be familiar

with innovative technologies and may fear that using new technologies

may lead DOE to miss cleanup deadlines if the technology fails to perform

as expected.8

 In response to our report, OST took several actions, including

establishing the site technology coordination groups discussed earlier, to

improve communication on sites’ technology needs and the capabilities of

newly developed technologies. In addition, to help ensure that

development activities were concentrated on the most pressing cleanup

needs, EM restructured its technology development program into the focus

areas.

In 1996, we reported that EM had not coordinated technology development

to prevent duplication of effort, particularly between OST and the Office of

Waste Management, which together had 60 projects to develop equipment

6

Estimating the Cold War Mortgage: The 1995 Baseline Environmental Management Report (DOE/EM,

Mar. 1995).

7

Cleanup Technology: Better Management for DOE’s Technology Development Program

(GAO/RCED-92-145, Apr. 10, 1992).

8

Department of Energy: Management Changes Needed to Expand Use of Innovative Cleanup

Technologies (GAO/RCED-94-205, Aug. 10, 1994).

Page 17 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

to melt and immobilize waste.9 A key reason for the duplication was EM’s

lack of a comprehensive list of technology development projects. EM

subsequently developed a list of its technology projects. We also found

that more technology projects were being started at the sites where the

focus areas were physically headquartered. In following up on this

situation in 1997, we found that this concentration had decreased.

In 1997, we testified before the Subcommittee on Oversight and

Investigations, House Committee on Commerce, that OST appeared to have

made some improvements in its project management, but we had

continuing concern about the extent of use of OST-developed technologies

at DOE’s waste cleanup sites and the validity of OST data on deployments

and expected cost savings.10 OST had also proposed a new initiative,

Accelerated Site Technology Deployment (ASTD),11 to facilitate the use of

its technologies. We expressed several concerns about the likely

effectiveness of this initiative, which provides funding to DOE sites for the

first use of an innovative technology. OST provided a total of approximately

$26 million to 14 ASTD projects in fiscal year 1998.

Objectives, Scope,

and Methodology

The Chairman and Ranking Minority Member of the House Committee on

Commerce and the Chairman and Ranking Minority Member of its

Subcommittee on Oversight and Investigations asked us to review EM’s

Office of Science and Technology. Specifically, we were asked to

determine (1) to what extent innovative technologies developed by OST

have been deployed (used) at DOE sites and how this rate of deployment

compares with the rates of other government organizations that develop

environmental technologies; (2) what obstacles exist to deploying

innovative technologies at DOE sites; and (3) what EM is doing to overcome

obstacles to deploying innovative technologies.

To determine the extent to which OST-developed technologies have been

deployed at DOE sites, we obtained deployment information on OST’s

projects from an OST management information system. This information

provided project names and numerical identifiers, research stage,

deployment sites (if any), and other project information, as of

January 1998. We also obtained information about OST’s use and definition

9

Energy Management: Technology Development Program Taking Action to Address Problems

(GAO/RCED-96-184, Jul. 9, 1996).

10Cleanup Technology: DOE’s Program to Develop New Technologies for Environmental Cleanup

(GAO/T-RCED-97-161, May 7, 1997).

11Formerly called the Technology Deployment Initiative.

Page 18 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

of the term “deployment” and OST’s procedures for entering and updating

the information in this system. In order to assess the accuracy of OST’s

deployment data, we used a random sample of the projects that OST listed

as deployed, and we verified the claimed deployments with site operations

officials. Upon finding a significant error rate, we used our sample results

to estimate a range for the actual number of OST project deployments. The

methodology for our verification is described in appendix I.

To compare the rate of deployment for OST’s technologies with the rates of

other government organizations that develop environmental technologies,

we used database searches and contacts with federal agency officials to

identify federal government programs that develop environmental

technologies. We contacted the eight government programs whose

research and development work was most comparable to OST’s in mission

and scope, and two of these programs—EPA’s Superfund Innovative

Technology Evaluation Program and the Department of Defense’s

Environmental Security Technology Certification Program—were able to

provide deployment data for comparison with OST’s data. We also

contacted two private sector organizations that develop environmental

technologies but found that they did not maintain deployment data.

Because the two federal programs providing deployment data conduct

technology demonstrations but not earlier phases of research and

development, we identified OST projects that had reached a similar stage of

maturity to provide an equitable group for comparison. (See app. II for a

detailed discussion of these two programs.)

To identify obstacles that exist to deploying innovative technologies at DOE

sites, we first reviewed past reports on this subject by GAO, DOE, and

advisory groups to DOE. In order to obtain more current and specific

information about obstacles to deployment and EM’s progress in

overcoming them, we visited five DOE sites: Hanford (Washington State),

Savannah River (South Carolina), Oak Ridge (Tennessee), Fernald (Ohio),

and Lawrence Livermore National Laboratory (California). We selected

these sites to provide varied perspectives: The first three sites are among

the largest DOE cleanup sites; Fernald is far along in its cleanup efforts and

represents a medium-sized cleanup effort; and Lawrence Livermore has a

smaller cleanup effort and budget. For the site visits, we identified

OST-developed technologies that were either selected for use at the site,

considered for use but not selected, or potentially applicable to the site’s

cleanup problems. We identified technologies to discuss with site officials

from our meetings with managers of OST’s focus areas, records maintained

by EM’s Office of Environmental Restoration, and discussions with

Page 19 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 1 

Introduction

headquarters EM officials. The technologies were judgmentally selected to

provide coverage of (1) EM’s various cleanup challenges and the related

OST focus areas and (2) innovative technologies selected or not selected

for use.

We discussed with EM field personnel and contractor staff the obstacles

they faced to using particular technologies, the ways they addressed and

overcame these obstacles (for deployed technologies), and the reasons

they did not select the technologies. We discussed a total of 30

OST-developed technologies with one or more of the five sites and obtained

documentation on related selection decisions. We analyzed this

information to identify (1) commonly cited obstacles to deployment and

(2) the means by which sites overcame these obstacles in those cases in

which OST-developed technologies were selected or in use.

In order to identify EM’s actions to overcome deployment obstacles, we

reviewed a memo from the Assistant Secretary for EM that directed a

number of actions to increase deployment, and we obtained information

about the status and results of these actions. We also interviewed OST

managers to identify additional actions under way within OST and obtained

related documentation. To assess the adequacy of these actions, we

compared the EM and OST actions to the obstacles to deployment that we

had identified.

We provided a draft of this report to DOE for its review and comment, and a

draft of chapter 2 and appendix II to the Department of Defense and EPA

for their review and comment. DOE’s and the Department of Defense’s

written comments and our responses are included in appendixes III and

IV.

We performed our review from August 1997 through September 1998 in

accordance with generally accepted government auditing standards.

Page 20 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

While OST has initiated 713 technology development projects, we estimate

that EM has deployed between 88 and 130 of these projects, for an overall

deployment rate of 12 to 18 percent. In contrast, OST has reported that 152

projects have been deployed one or more times, for an overall deployment

rate of 21 percent. OST’s overstated deployment information is the result of

several factors, including its rapid compilation of deployment data in

response to congressional requests and the lack of a formal definition of

what constitutes a deployment.

Most organizations we contacted, including some private technology

developers, did not track deployment data comparable to OST’s. We

contacted eight government programs and two private sector programs

engaged in environmental technology research and found only two that

could provide data on deployment. In comparing data from these two

organizations and OST, we found that OST’s deployment rate was close to

that of the 2-year-old Environmental Security Technology Certification

Program (ESTCP) in the Department of Defense and somewhat lower than

the rate for the 12-year-old Superfund Innovative Technology Evaluation

(SITE) program in EPA. However, it is important to recognize that the value

of deployment rate comparisons with other organizations is limited. To

assess the overall performance of a research and development (R&D)

program like OST, other measures in addition to deployment would be

relevant.

EM Has Deployed

About 88 to 130 of

OST’s Technology

Projects

OST developed deployment data in response to a November 1996 request

from the Subcommittee on Oversight and Investigations, House

Committee on Commerce. As of January 1998, OST’s database showed that

EM had initiated 713 technology development projects since OST’s

inception. On the basis of our verification and analysis of these data, using

a 95-percent confidence interval, we estimate that EM actually deployed

from 88 to 130 projects, to achieve an overall deployment rate of 12 to

18 percent for the 713 projects.1

 In contrast, OST has reported that 152 of

the 713 technology projects initiated since its inception have been

deployed. Thus, according to OST’s data, about 1 in 5 OST technologies have

been deployed one or more times, for an overall deployment rate of

21 percent.

We also found that OST had overstated the number of deployment

instances reported for each technology project. OST’s database listed a

1

To verify OST’s deployment data, we contacted site personnel for a randomly selected sample of the

projects that OST claimed had been deployed. See app. I for a detailed explanation of the confidence

interval and our verification methodology.

Page 21 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

total of 283 deployment instances2 for the 152 projects claimed as

deployed. We estimate that of the 283 deployment instances claimed by

OST, only 137 to 216 have actually occurred. Table 2.1 lists OST’s data, the

error rates we found, and our estimates of actual deployments based on

the error rate found in our sample of 30 projects.

Table 2.1: Deployment of

OST-Developed Technologies

OST’s claimed deployments Error rate found by GAO

GAO’s

estimates of

deploymentsa

152 projects deployed 27 percent 88 to 130

projects

deployed

283 deployment instances 38 percent 137 to 216

deployment

instances

aSince we used only a sample of projects to make our estimates, these estimates have a margin

of error. The range shown reflects this margin of error at the 95-percent confidence level.

Several Factors Have

Led to OST’s

Inaccurate

Deployment Data

OST’s inaccurate deployment data resulted from several factors.

Specifically, OST compiled deployment data quickly, in response to a

congressional request that came 7 years after the program’s inception,

because it had not previously maintained comprehensive data. In addition,

the lack of a formal definition for deployment led to differing

understandings among the focus area personnel responsible for compiling

the data. Finally, OST has begun only recently to establish procedures for

entering and updating project data. If such procedures had been in place

early on, they would have uncovered the need to formalize the definition

of deployment.

Initial Compilation of

Deployment Data

Some inaccuracy in OST’s deployment data may have been due to the fact

that the data were compiled quickly. OST prepared deployment data in

response to a November 1996 request from the Chairman of the House

Committee on Commerce. Previously, OST had not maintained

comprehensive deployment data on its projects. Instead, OST tended to

focus its performance measures on completed demonstrations. For the

November 1996 request, OST gathered deployment data for its projects over

a period of several months and provided the information to the Chairman

2

For example, OST reported four deployment instances for the Waste Inspection Tomography

technology project: one use at Hanford, one at Idaho National Environmental Engineering Laboratory,

and two at Lawrence Livermore National Laboratory. This project thus counts as 1 of the 152 projects

OST claimed as deployed and 4 of the 283 deployment instances.

Page 22 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

in April 1997. At that time, OST reported that 150 projects had been

deployed and an additional 41 projects had been selected for use in the

future (for a total of 191 past and future deployments).

Lack of a Formal

Definition for Deployment

Another reason for OST’s inaccurate data has been the lack of a formal

definition of deployment, leading to different understandings among the

focus area personnel who collected the deployment data about what

should be counted as a deployment. According to OST managers, while

gathering data to respond to the Committee’s request, OST headquarters

officials told focus area personnel to refer to an earlier definition of

implementation for the meaning of deployment but did not distribute new

written guidance. This definition, which OST had formalized and

distributed in April 1996, defined implementation to mean that the

technology was used or selected for use to meet specified user

performance measures (e.g., completion of an assessment or treatment of

waste for disposal). However, officials of the Subsurface Contaminants

Focus Area provided us with a definition of deployment that they received

along with the instructions for responding to the Committee’s data

request. This definition stated that the number of deployments means the

number of “hot” demonstrations (that is, demonstrations in radioactive

environments) and that deployment site means the location of a hot

demonstration.

We found that OST focus area personnel entering the data frequently

regarded demonstrations as deployments. For example, OST counted as a

deployment the use of a characterization technology called Laser

Ablation/Mass Spectroscopy at the Pacific Northwest National Laboratory

in Washington State. In response to our questions, site contractor officials

stated the technology’s use at the laboratory had been a demonstration in

which data derived from the laser technology were compared with data

derived from a conventional technology. At this time, the site cannot rely

upon the laser technology to accomplish its goals for characterization.

Status of OST Efforts to

Improve Data Quality

While OST has issued a definition of deployment and is taking other steps

to improve data quality, written procedures for data verification have not

yet been developed. In August 1998, the Acting Deputy Assistant Secretary

for OST issued a memo that formally defined deployment. The definition

appropriately emphasized that deployments must accomplish site

objectives, such as the completion of assessments, cleanups, or the

treatment and disposal of wastes. The memo stated that this definition is

Page 23 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

to be used for performance measurement. OST has also completed a data

verification effort for those projects considered deployed during fiscal

year 1997. It used verification by site personnel and other data sources to

improve the accuracy of this portion of the data.

According to OST officials, the office intends to continue similar

verification efforts in the future. However, these data verification plans are

not reflected in OST’s draft procedures for its database, which do not

specify a method of data verification. The procedures, drafted in

January 1998, identify OST’s focus areas as responsible for entering data

and ensuring their quality and completeness. These procedures also

require that the data on ongoing projects be updated at least once per

quarter (every 3 months). However, the draft procedures do not identify

any means by which the data are to be verified or spot-checked for

accuracy. While site technology coordination groups can comment on the

deployments listed in the database, the procedures do not state any

requirement for data review and concurrence by these groups or other site

officials. The Acting Deputy Assistant Secretary for OST told us that OST

plans to obtain further advice about verification methods and then develop

written procedures.

OST has not yet determined whether, or to what extent, to verify data from

the years prior to 1997 because of the time and resources involved.

According to the Acting Deputy Assistant Secretary for OST, the office is

seeking clarification from the House Committee on Commerce on the

degree of accuracy or certainty needed.

Deployment Rate for

OST Technologies

Falls Between the

Rates of Two Other

Environmental

Technology Programs

We compared the deployment rate for OST’s technologies with the

deployment rates for technologies sponsored by EPA’s SITE program and

the Department of Defense’s ESTCP. The SITE program is engaged solely in

the environmental technology demonstration and implementation stages

of R&D. Similarly, ESTCP demonstrates and validates technologies and funds

environmental technologies that have progressed to the stage at which

field demonstrations are warranted. Taking into account the limitations of

this comparison, OST’s deployment rate for projects at comparable stages

of development falls between the rates of the two organizations that

provided data, as shown in table 2.2. (App. II discusses in detail how we

developed each comparison.)

Page 24 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

Table 2.2: OST’s Deployment Rate

Compared With the Rates of Other

Environmental Technology

Demonstration Programs

Other demonstration programs Comparison rate OST ratea

Superfund Innovative Technology

Evaluation program 59% 28 to 45%

Environmental Security Technology

Certification Programb 38% 28 to 40%

aOST deployment rates based on projects at comparable stages of maturity.

bESTCP data based on the program’s first 2 years of operation.

Source: GAO’s analyses of data provided by the agency programs.

Deployment Rate

Comparisons Between

OST and Other

Environmental

Technology

Developers May Have

Limited Usefulness

Comparisons of OST’s deployment rate with the rates of other

organizations must be viewed with caution when assessing how well EM is

doing in deploying OST-developed technologies. We found few

organizations that engage in the range of environmental research OST

performs, and no organization we contacted routinely tracked deployment

data on its projects. Data provided by the two organizations differed

widely in source and composition. Finally, many individuals we contacted

question whether a deployment rate is a sufficient benchmark for

successful R&D.

A Small Number of

Environmental Technology

Development Programs

Maintain Deployment Data

Most organizations we contacted, including some private technology

developers, did not track deployment data comparable to OST’s. Of the

eight government programs and two private sector programs engaged in

environmental technology research we contacted, only the SITE program

and ESTCP could provide data on deployment. Even these two programs

needed to compile their information so that it could be expressed as

deployment rates. Table 2.3 shows the entities that we contacted.

Page 25 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

Table 2.3: Environmental Technology

Development Entities Contacted Entities Program

Environmental Protection Agency Superfund Innovative Technology

Evaluation Program

Department of Defense Environmental Security Technology

Certification Program

Strategic Environmental Research and

Development Program

Chief of Naval Operations; Environmental

Research, Development, Test and

Evaluation Program

U.S. Army Corps of Engineers Waterways

Experiment Station; Environmental

Laboratory

U.S. Air Force Research Laboratory; Air

Base and Environmental Technology

Division

Air Force Center for Environmental

Excellence

U.S. Air Force; Environment, Safety and

Occupational Health—Technical Planning

Integrated Product Team

Private sector organizations Petroleum Environmental Research Forum

DuPont Corporate Remediation Group

Furthermore, we found that only one of the other government programs

listed in table 2.3 engaged in nearly the full range of environmental R&D

that OST performs. OST’s R&D includes basic science research, applied

research and engineering development, field testing and demonstration,

and implementation by the end user (commercialization). Most of the

governmental organizations we contacted performed either the early

stages of R&D or the later stages, but not both. Technology development

efforts undertaken at the early stages have more unknowns and are likely

to involve a greater risk of failure than efforts at the later stages. Since we

would expect performance results to differ for each stage, meaningful

comparisons can only be made among projects or programs that are at

similar stages of R&D maturity.

Two organizations provided us with very different types of data. EPA’s SITE

program had accumulated survey data on the number of contracts their

technology vendors had obtained over about 8 years. We agreed that a

Page 26 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

contract for use could be considered deployment of the technology. As to

be expected, the survey responses were less than 100 percent, unlike the

OST and ESTCP data, which include all of these agencies’ technology

projects. Therefore, the data from EPA’s SITE program are incomplete, and

the deployment rate for SITE could actually be higher. The Department of

Defense’s ESTCP provided a description of the transition (deployment)

status for all of its projects from the program’s first 2 years of existence.

Since ESTCP is a relatively new program, its deployment data are based on a

limited number of projects and may be less representative of the

program’s future performance. We did not verify the accuracy of these

organizations’ deployment data, but we reviewed their available project

summaries and believe the organizations’ approaches were reasonable

responses to our request. Nevertheless, differences in how the programs

defined deployment, and whether they counted incomplete projects, will

affect computed rates.

Deployment Rate May Be

an Incomplete Measure of

OST’s Performance

As we have previously reported, measuring the performance of R&D

programs is difficult.3 Performance measures used in other federal R&D

programs include the scientific peer review of projects, numbers of

patents issued, and studies of publications. Recent R&D management

literature suggests that certain measures, such as the number of patents

issued, are best suited to earlier stages of research, while outcome

measures, such as deliverables and customer satisfaction, are more

relevant for later-stage research. In this context, a deployment rate

measure would be most useful when applied to more mature projects. At

the same time, program managers need to assess how successful the

program has been at selecting early-stage projects with high potential for

future payoff.

Officials in a number of programs we contacted told us that deployment

has only recently been raised as a possible performance measure.

Furthermore, programs performing earlier stages of R&D were less likely to

have any deployment data. Developers of later-stage technologies believed

that the deployment rate is an incomplete performance measure, and that

cost savings or some measure of dollar impact should also be used to

evaluate program success. EM is considering developing a performance

measure that would assess cost savings from the use of innovative

technologies.

3

Measuring Performance: Strengths and Limitations of Research Indicators (GAO/RCED-97-91, Mar. 21,

1997).

Page 27 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 2 

EM Has Deployed Less Than 20 Percent of

OST’s Projects

EM and OST recognize that deployment is not the only relevant measure of

success in technology development. We reviewed performance measures

established for OST for fiscal years 1994 through 1997 and found that

completing demonstrations of technologies and the number of

technologies made available for use—that is, number that have completed

development—were the main performance measures used. In fiscal year

1998, OST’s performance measures are (1) demonstrate 35 new

technologies, (2) make 40 alternative technologies available for use with

cost and engineering data, and (3) perform 49 deployments of new

technologies. As described in more detail in chapter 4, performance

measures for deploying innovative technologies are also being applied to

EM’s field operations offices in fiscal year 1998, and OST is considering

developing additional performance measures for its focus areas that

address technologies in various stages of development.

Page 28 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

As EM’s cleanup program has matured, several of the obstacles to using

innovative technologies reported previously by us, EM, and others have

been addressed. For instance, DOE sites and their regulators have improved

their working relationships, and, in cases where innovative technologies

were selected, DOE sites have found ways to address regulator concerns

about whether these technologies will achieve required objectives.

However, some obstacles, internal to EM and OST program operations,

continue to slow the deployment of innovative technologies, and, in some

cases, have led OST to spend millions of dollars for technologies that the

cleanup sites do not want. The most significant and continuing of these

internal obstacles has been EM’s and OST’s failure to involve users

sufficiently in the design and development of technology targeted for use

at the cleanup sites. As a result, OST has developed generic technologies

that do not meet site-specific needs or that require modification to make

them usable by the site. However, EM has not clearly defined

responsibilities and funding sources for modifying technologies among OST

and potential technology users. Furthermore, OST still has no clearly

defined role in helping sites select the appropriate technology and

infrequently provided technical assistance in the cases we reviewed.

Several factors contribute to these problems. First, prior to 1996, OST had

not comprehensively assessed users’ technology needs and linked these

needs with technology development efforts. Second, OST has not fully

implemented its system for monitoring, and if necessary, modifying or

terminating ongoing technology development projects—a system that

would require interaction with technology users.

Sites Have Overcome

Obstacles to Using

Some Innovative

Technologies

DOE’s field and contractor staff face a number of challenges when

attempting to use an innovative cleanup technology. Past reports by us,

EM, and advisory groups have catalogued the challenges: the perceived

risks of exceeding projected costs or failing to meet time schedules; the

need to convince regulators and stakeholders of the advantages of

innovative technology; and technical problems, including the need to

modify a technology to make it fit a specific situation. However, as the EM

technology program has matured and site personnel, regulators, and

stakeholders have become more aware of the benefits of using some

innovative technologies, some obstacles have diminished in importance.

Furthermore, when the use of a new technology is clearly and significantly

advantageous, cleanup sites make a strong effort to overcome any

obstacles to its use.

Page 29 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

Specifically, when regulators and stakeholders are concerned about a

technology’s effectiveness, sites have provided additional data or testing

and, occasionally, modified technology to satisfy some concern or

implemented an innovative technology in phases to obtain performance

data. For example, according to Hanford officials, using a new technology

to encapsulate certain carbon-based waste would be much less costly than

incinerating it. State regulations, however, called for incinerating such

waste. Nonetheless, Hanford persisted and obtained a waiver from the

state to encapsulate the waste. At Oak Ridge, DOE and its contractors

wanted to use a frozen soil barrier to contain a relatively small pool of

water that had been contaminated with reactor waste. However,

regulators and stakeholders were skeptical that this innovative technology

would work and be cost effective. Oak Ridge demonstrated the technology

to obtain cost and performance data and provided this information to

regulators and stakeholders. The technology has since gained wide

acceptance by these groups.

Some technology may have to be modified to satisfy regulatory concerns.

For example, Hanford officials wanted to test an innovative technology for

cleaning up contaminated soil, which they believed was better than

current methods. However, regulators were concerned about the possible

expulsion of carbon tetrachloride contaminants into the air. Hanford

officials convinced the regulators to allow them to experiment with the

new technology by offering to add a filter to the equipment to catch any

contaminants. The modification was a low-cost and easy addition to the

equipment.

In some cases, sites implement a technology in phases to obtain

performance data and to assure themselves and convince regulators and

stakeholders of the technology’s viability. For example, OST funded the

development of a robotics device, called Houdini, that could help clean up

waste in tanks. Oak Ridge, with the help of the manufacturer, adapted

Houdini to help clean up radioactive waste stored at the bottom of the

site’s large tanks. However, because Houdini had never been used to clean

up radioactive waste, no information was available on the device’s

performance and reliability. Oak Ridge therefore had to implement

Houdini in phases using nonradioactive “cold” testing; followed by

treatability tests in a lower radiation environment; and finally, “hot” testing

on the radioactive waste in its tanks.

Field officials also told us that the projects in which OST and an EM

operating group get involved as a joint venture seem to work well. In these

Page 30 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

cases, OST provides funding and some technical assistance, and the

operating group also provides funding and implements the project. If there

are also partners from industry, they further enhance the chances for

success. For example, at Hanford, OST and EM’s environmental restoration

group are participating in a large project to demonstrate a number of

technologies that can be used to put Hanford’s old, shutdown reactors into

safe interim storage. Hanford officials were convinced that if the

demonstrated technologies were successful, the time needed to prepare

the reactors for storage could be cut by 7 years.

The demonstration project started in 1996, with contributions totaling

about $8 million from OST and about $16 million from the environmental

restoration group. However, the project did not have the extra money to

make needed refinements and modifications to technologies.

Consequently, Hanford officials suggested partnering with private

contractors who would assume the risk and cost of getting the

technologies to perform. OST’s Deactivation and Decommissioning Focus

Area, which routinely works with the private sector, helped to bring about

this partnership with private contractors. As of July 1998, the project had

successfully demonstrated 20 technologies and deployed 13 of them at

Hanford’s C Reactor, two other Hanford reactors, and a number of other

DOE reactors throughout the complex. In addition, the technologies have

been transferred to the commercial reactor sector and will be used to help

put the nuclear power plant in Chernobyl, Ukraine, into safe storage.

OST Has Developed

Innovative

Technologies Without

Sufficient User

Involvement

Despite the progress that has been made, some obstacles internal to EM

and OST operations continue to slow the deployment of innovative

technologies. In particular, OST has developed technologies that tend to be

generic solutions to cleanup problems and, if usable at all, have to be

modified to fit a site’s specific problem. These problems occur in part

because OST had not, until 1996, comprehensively assessed the technology

needs of the cleanup sites and has not involved potential technology users

in the development of technology that might be used to address specific

cleanup problems. Without user involvement, there have been no

identified customers for some of the technology that OST has sponsored.

For example, of the 107 technologies that OST has completed, 31

technologies, costing $71 million to develop, have not been used by

cleanup sites.1

1

From OST data as of Jan. 1998, unadjusted for errors.

Page 31 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

Cleanup Sites Charge That

Technology Does Not Meet

Their Needs

According to EM field and contractor personnel responsible for waste

cleanup, in many cases, OST technologies do not meet their needs. They

said that OST has many times assumed that “one-size-fits-all” and therefore

has developed generic solutions to cleanup problems. However, these

solutions either do not fit a site’s specific needs or must be modified

before they can be used. For example, Fernald workers needed portable

equipment that would allow them to characterize contamination within

buildings without climbing ladders to obtain samples from contaminated

areas. OST said that, although its laser-induced fluorescence imaging

equipment had not been field-tested, the equipment had been designed to

meet needs such as Fernald’s. However, when Fernald workers attempted

to use the equipment, they found that it was not ready for field use. It was

cumbersome (not really “portable”) and light interfered with measurement

readings. As a result, the equipment was not usable and was returned to

the manufacturer for modifications. Consequently, Fernald personnel

continued to take samples from the contaminated building areas by hand.

Although they realized that the OST equipment had not been thoroughly

tested before they tried it, Fernald officials said, they believed that if OST

had involved them in the design and development of the equipment, the

problems would have been avoided, or at least identified and corrected

earlier.

Similarly, officials at DOE’s Hanford site tried two OST technologies that

promised to support faster remediation of contaminated soil but had to

reject them because they were not designed to work in Hanford’s arid soil.

The officials said that the concept for faster remediation of contaminated

soil was attractive and probably would have been acceptable to Hanford’s

regulators, but the generic design of the technologies did not meet

Hanford’s specific needs.

Furthermore, some site officials said that they would like to use some OST

technologies, but the technologies require modification to fit the site’s

situation. They pointed out that it is not clear who should make and pay

for these modifications. For example, a project manager at DOE’s Savannah

River Site told us that he would like to use more innovative technology in

his projects, but it is unclear who is responsible for making site-specific

modifications, and his program does not have funding to make such

modifications. At Hanford, officials were interested in using OST’s

Electrical Resistance Tomography to help detect leaks in their high-level

radioactive waste tanks. (Hanford has 67 known or suspected leaking

tanks.) However, a Hanford official said that the technology needed

substantial fine-tuning to make it work on the Hanford tanks, and no

Page 32 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

funding was available to pay for this. He said that it was unclear who is

responsible for funding modifications to OST technologies.

When only minor, inexpensive modifications are required, site

representatives said that they have made and usually paid for them. But

other technologies that are of interest to sites would require more

extensive and more expensive modifications. Without a clear policy on

who is responsible for modifying the technology and paying for the

modification, the sites are likely to reject the innovative technology and

select a known alternative.

User Needs for Technology

Not Fully Assessed Until

Recently

Until its reorganization in 1994, OST did not involve the cleanup sites in

identifying technologies that need to be developed and did not conduct

comprehensive needs assessments until 1996 and 1997. Therefore, most of

the technologies developed through OST were not based on a

comprehensive assessment of the technology needs of those responsible

for cleaning up DOE waste sites. Instead, OST consulted with its developers

at the national laboratories in deciding which technologies it would

sponsor to solve sites’ cleanup problems. These technical solutions,

according to potential technology users, tend toward the “one-size-fits-all”

development philosophy.

We reported in 1994 that technology needs had not been comprehensively

identified to allow prudent research decisions nor had various

environmental program offices in headquarters and in the field worked

together effectively to identify and evaluate all of the possible technology

solutions available.2

 In 1995, and again in 1996, the Environmental

Management Advisory Board (EMAB) told the Assistant Secretary of

Environmental Management that the lack of a comprehensive assessment

linking identified needs with technology development efforts was a

“primary barrier” to technology deployment. EMAB said that technology

development and deployment must be linked together as a single system.

Site technology coordination groups, established in 1994, made early

attempts to assess the needs of potential technology users. However,

because OST considered data from these early surveys unreliable, it and the

site groups developed guidance and worksheets for a more comprehensive

assessment, which the site technology coordination groups carried out in

October 1996. In October 1997, an updated needs assessment and a

2

Department of Energy: Management Changes Needed to Expand Use of Innovative Cleanup

Technologies (GAO/RCED-94-205, Aug. 10, 1994).

Page 33 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

database that matches technology needs with appropriate existing

technology or the future efforts of technology developers was completed,

according to the director of OST’s Office of Technology Systems.

OST Has Not Involved

Users in Technology

Development

In addition to not involving the cleanup sites in identifying technology

needs, OST has not sufficiently involved users in designing technologies

and monitoring their development to help ensure that they meet users’

needs. In 1992, we recommended that EM institute a technology

development management system with explicit decision points at which

the technology would be assessed to determine whether development

should continue or be terminated. OST established its “Technology

Investment Decision Model” (called the “gates system”) to do this. The

gates system satisfies our 1992 recommendation and was intended to be “a

user-oriented decision-making process for managing technology

development and for linking technology-development activities with

cleanup operations.” However, OST has not fully implemented the gates

system and thus cannot be certain that appropriate technology is

developed to meet the needs of DOE’s cleanup sites.

Under OST’s gates system, the focus areas are to assess a technology’s

development at six stages, from basic research through implementation.

At each stage, the focus area is to make a go/no-go decision, with input

from potential users. The critical decision points include the following:

• Gate 1: Entrance Into Applied Research Stage. To pass through gate 1 and

enter this stage, a proposed technology must be shown to address national

interests and priority environmental needs. EM guidance states that if a

technology does not address a specific need, it should not pass through

gate 1.

• Gate 2: Entrance Into Exploratory Development Stage. To pass through

gate 2 and enter this stage, a technology has to be linked with the specific

needs of an identified user.

• Gate 3: Entrance Into Advanced Development Stage. To pass through gate

3 and enter this stage, the technology must be able to meet an identified

user’s specific performance requirements. In addition, it must be

documented that the research to develop the technology is expected to

produce results consistent with the user’s time frame for deployment and

implementation.

• Gate 4: Entrance Into Engineering Development Stage. To pass through

gate 4 and enter this stage, the technology must be shown to meet the

user’s specific needs in a timely manner. In addition, it must be

Page 34 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

documented that the proposed innovative technology will be more

cost-effective than current methods or other emerging technology.

• Gate 5: Entrance Into Demonstration Stage. To pass through gate 5 and

enter this stage, the identified user for the technology must make a

commitment to deploy the technology if it meets performance

requirements. In addition, the user must agree to share in the cost of and

the responsibility for demonstrating the technology.

• Gate 6: Entrance Into Implementation Stage. To pass through gate 6 into

implementation, the technology must successfully complete a “real world”

demonstration, either at a DOE site or another location, using actual waste

streams and/or anticipated operating conditions. In addition, it must be

documented that the technology has proven to be viable, cost-effective,

and applicable to the users’ needs.

As this discussion of the gates system shows, OST’s focus areas must

identify a user for the technology in the early stages of development.

Furthermore, this user must stay involved throughout the development

process to ensure that the technology will meet the needs and

implementation schedule of the user.

OST, however, has not fully implemented its gates system to involve

potential users in the assessment of technology that it is developing, and,

in some cases, OST has not identified an end user for the technology.

Furthermore, a review by EM and EMAB representatives, completed in late

1997, revealed that OST’s focus areas do not consistently use the gates

system and do not consistently involve potential technology users in

technology development decisions.

EMAB has pointed out in numerous reports that OST has failed to rigorously

apply the gates system. EMAB has stated that OST should use the gates

system to identify and terminate technologies that have no identified

customer, are not cost-effective, or have limitations that may increase the

risk of failure when used. According to the Chairman of EMAB’s Committee

on Technology Development and Transfer, OST officials told him that they

did not rigorously apply the gates system because it yielded results that

OST and technology developers at the laboratories did not like—that is,

indicating that some technology projects should be terminated.

Similarly, representatives of one of OST’s focus areas told us that OST does

not rigorously use the gates system because it would force OST to

terminate technologies that have no identified customer, do not meet

users’ needs, are technically limited, or have some other fault. The

Page 35 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

manager of the Subsurface Contaminants Focus Area told us that his focus

area had rigorously applied the gates system and terminated some

technologies, which led to a confrontation with the laboratories

developing the technologies.

The Director of OST’s Office of Technology Systems told us that the gates

system was never fully implemented because staff were confused by other

evaluations and OST’s reorganization into focus areas, which were taking

place at the time the gates system was instituted. He said that the gates

system was currently not being used but would be reinstituted in the

future. According to the Acting Deputy Assistant Secretary of OST, the

criteria of the gates system are still valid, but when focus areas tried to use

the gates system, their approach was inappropriate and did not work.

Specifically, he stated that focus areas set up panels to periodically review

their projects according to the gate criteria. Instead, the gates system was

intended to be used on an ongoing basis, so that the focus areas could

determine whether requirements for the various stages of technology

development, including user involvement, had been met. According to this

official, OST’s intent was that technology developers and technology users

have frequent interaction.

OST Has Not Provided

Some Needed

Technical Assistance

OST has not fulfilled its role of providing technology users with the

technical advice and assistance that they need to identify solutions to

cleanup problems and to help implement those solutions. Focus areas’

ability to provide technical help varies widely, although this was a

principal mission when these groups were established in 1994. Some site

officials responsible for cleanup told us that they are reluctant to try new

technologies without a reliable source for advice and assistance, but some

are reluctant to seek help from the focus areas because they do not trust

the focus areas’ abilities.

Focus Areas’ Capability to

Provide Technical

Assistance Varies

EM established the focus areas in part to provide technology users with

technical advice and assistance. However, EMAB has consistently noted the

lack of technical knowledge in some focus areas and suggested that this

problem be addressed. Similarly, we found that cleanup sites are skeptical

of the technical expertise of some focus areas and rarely call upon them

for assistance.

EMAB believes that the focus areas need to become experts not only in

OST-sponsored technology but also in other domestic and foreign

Page 36 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 3 

Some Obstacles Continue to Slow

Deployment of Innovative Technologies

technology that might help solve waste cleanup problems. EMAB reported

in January 1998 that some focus areas do not know the state-of-the-art

technology for their area. The Chairman of EMAB’s Committee on

Technology Development and Transfer told us that the Tanks Focus Area

and the Deactivation and Decommissioning Focus Area seem capable, but

he said that EMAB is concerned about the capability of the Subsurface

Contaminants Focus Area, which has the largest workload by type of

waste problem.

During our visits to five cleanup sites, we found that the sites infrequently

sought technical assistance from OST and its focus areas. Site officials said

that technical assistance would be helpful in deploying new technologies,

but some are not convinced of the focus areas’ technical expertise.

Furthermore, they preferred to go directly to a vendor for technical

assistance because the vendor was much more knowledgeable than OST.

Sites Are Reluctant to

Allow OST a Role in Major

Technology Decisions

In 1994, we recommended that OST be given a formal role in sites’

selections of technologies to solve cleanup problems. For example, OST

could formally take part in sites’ feasibility studies to identify and analyze

technologies that could potentially solve a specific waste cleanup problem

and to help a site decide which technology to use. However, some site

officials told us that OST and its focus areas are not familiar enough with

their sites’ waste cleanup problems and appropriate solutions. They said

that our recommendation was not taken because site officials are skeptical

of OST’s ability to provide quality technical advice and assistance and

therefore are reluctant to allow OST more of a role in selecting cleanup

technologies for their sites.

The Acting Deputy Assistant Secretary for OST told us that he is aware of

this problem and has directed the focus areas to become more technically

competent and supportive. He said that providing technical assistance

should be routine for the focus areas; they should be out in the field

providing this help, not waiting in the office for the sites to call them. He

emphasized that if the focus areas are not able to provide expert technical

assistance, he will look to other groups, perhaps the national laboratories,

to provide needed technical assistance.

Page 37 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

EM management devoted little attention to the deployment of innovative

technologies until a congressional oversight hearing in May 1997 criticized

EM’s performance in deploying technology. Following the hearing, the

Assistant Secretary of EM issued a memorandum in July 1997, directing OST

and other EM offices to initiate specified actions designed to facilitate

technology deployment. Some of these actions have already been

completed, and the remainder were to be completed by September 30,

1998. These actions establish responsibilities, require the development of

performance measures for technology deployment, establish the

Technology Acceleration Committee of upper-level EM and field managers,

require sites to develop deployment plans, and continue the Accelerated

Site Technology Deployment program that funds individual projects. OST

has additional initiatives under way, including establishing

technology-user steering committees and developing multiyear plans for

technology development.

However, EM’s efforts only partially address the internal obstacles limiting

deployment. On the positive side, EM has established deployment

performance measures for field sites and required sites to develop

deployment plans. Users’ involvement in developing overall plans and

priorities for OST’s work is also improving. On the other hand, although the

initiatives provided for upper management attention through the

Technology Acceleration Committee, the future of this Committee is

uncertain because of the departure of EM’s Assistant Secretary, who

established it. According to EM officials, a broader executive committee

addressing EM issues may take its place. EM did not carry out its plans to

include deployment in the annual performance expectations of its senior

managers, considering their membership in the then-active Technology

Acceleration Committee to be sufficient to hold managers accountable. In

addition, EM has not yet improved developer-user cooperation for

individual projects. Specifically, EM’s initiatives do not require OST to use

its existing decision process for technology development (the gates

system), which would require user involvement at various stages in the

development process. Furthermore, EM has yet to determine how it will

provide deployment assistance to cleanup sites to (1) more routinely

provide technical assistance in selecting and implementing innovative

technologies and (2) make modifications to completed technologies to

better meet sites’ needs when it is cost-effective to do so.

Page 38 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

EM Has Established

Initial Deployment

Performance

Measures for Field

Sites and Is

Developing Additional

Measures

In a July 1997 memo, EM’s Assistant Secretary stated that technology

deployment is the responsibility of all senior EM management, including

the managers of EM’s operating groups, OST, and field offices. EM

management had not previously emphasized technology deployment, and

this was the first formal assignment of responsibility for deployment. The

Assistant Secretary also directed that performance measures based on

technology deployment be established for those groups involved with

deployment efforts and be included in the performance expectations for

senior managers. In response, EM has instituted or is planning performance

measures addressing the deployment of innovative technologies at several

levels: (1) DOE field sites undergoing cleanup, (2) contractors that manage

the DOE field sites, and (3) OST and its focus areas. Field sites were also

required to submit deployment plans addressing both their overall

approach to utilizing innovative technologies and their plans to achieve

deployments in specific cleanup projects. EM continues to refine its

performance measures and has asked EMAB for advice about improving

performance measures at the various levels to help increase deployment.

Performance Measures and

Deployment Plans for Field

Sites

In responding to our written inquiry to EM management in March 1998, the

Acting Deputy Assistant Secretary for OST stated that “in analyzing the

most appropriate and optimum way” to accelerate technology deployment,

EM management concluded that deployment goals can best be achieved by

holding those at the point of implementation of new technology—the field

sites—responsible for deployment. EM has established two indicators to

measure field sites’ efforts to use innovative technology to clean up waste

sites: (1) the number of technologies deployed annually and (2) life-cycle

cost savings resulting from the use of innovative technology. For the

present, annual targets for the number of deployments are based on the

amount of annual EM funding a site receives. EM established a target that

requires field offices to agree to deploy one new technology for every

$100 million in annual funding that they receive. For example, DOE’s Oak

Ridge site will receive about $600 million in EM funding in fiscal year 1998

and is therefore expected to use six new technologies a year in its effort to

clean up nuclear waste. For fiscal year 1998, field sites have agreed to

deploy a total of 49 new technologies, which can be from OST or other

sources. OST believes that the majority of these new technologies will be

ones that it has sponsored.

Field sites must also submit site-specific deployment plans for innovative

technologies. The plans, most of which were submitted in May and

Page 39 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

June 1998,1 describe the sites’ overall approaches to deploying innovative

technologies, such as processes for identifying deployment opportunities

and involving regulators. The plans also specify opportunities to deploy

innovative technologies in the sites’ cleanup projects. For instance, the

plans describe the schedule for technology deployments, projected

benefits from using the technologies, and funding requirements.

In the future, EM may establish performance targets for field sites that are

based on the amount of savings that would be produced by using

innovative, rather than conventional, technology over the life of a project.

These measures were not established in fiscal year 1998 because EM lacked

a standard methodology for calculating cost savings. However, in

March 1998, EM completed a draft of a standardized process for calculating

these savings.

Performance Measures for

DOE’s Contractors

The need for contract incentives for the use of innovative technologies has

been broadly recognized by EM managers in headquarters and the field.

Each of the five sites that we visited had used performance measures

addressing deployment for the site’s management contractor. Some sites

have experimented with different approaches to determine which

measures work best. For example, at Savannah River, DOE tried

performance-based incentives for its contractor in 1995 and 1996 that were

based on the number of innovative technologies used and the associated

cost savings; then, in 1997 and 1998, it switched to incentives based on the

cost savings achieved—regardless of whether conventional or innovative

technologies were used. According to DOE’s Assistant Manager for

Environmental Quality at Savannah River, over half the cost savings that

the contractor achieved in environmental restoration in 1997 came from

the use of innovative technologies, and he believes that the cost savings

measure has worked the best in providing incentives for using innovative

technologies. At Lawrence Livermore National Laboratory, which

participates in a number of OST technology projects, the contractor’s

performance measures address both using innovative technologies in the

laboratory’s cleanup activities and supporting their use at other sites.

Performance Measures for

OST

OST’s performance will also be measured on the basis of technology

development and related deployment. For example, OST’s performance

goals for fiscal year 1998 include demonstrating 35 new technologies;

1

One of the sites required to develop a deployment plan—Savannah River—submitted a draft plan in

July 1998 and had not developed a final plan at the time of our review.

Page 40 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

finishing the development of 40 “alternative” technologies; and, along with

the cleanup sites, taking responsibility for the 49 deployments of

technology to be used in waste cleanup projects.

According to the Acting Deputy Assistant Secretary for OST, several

additional performance measures are under consideration for OST’s focus

areas to help ensure that the technologies still in development are

“deployable” when they are completed. These measures include whether

the focus areas’ projects address high-priority technology needs and

whether end users consider the technologies under development to be

viable solutions to their needs.

Further Improvements

Planned for Performance

Measures

In a June 1998 meeting, EMAB presented its analysis, prepared at EM’s

request, of how EM should improve performance measures for technology

development and deployment. Among other things, EMAB emphasized that

the use of performance measures must be supported by EM’s leadership

and that performance measures for EM’s technology research,

development, and deployment must be integrated with similar measures

for site cleanup programs. EMAB also suggested that EM’s Technology

Acceleration Committee review and improve existing research and

development performance measures. The Acting Deputy Assistant

Secretary for OST told us that EMAB’s advice would be considered in

designing additional performance measures for OST’s focus areas. As of

September 1998, EM was still in the process of identifying and improving

performance measures to help ensure that cost-effective innovative

technologies are used for waste cleanup.

EM Is Increasing User

Input to OST’s

Planning and Priority

Setting

EM has established a mechanism—a user steering committee for each of

OST’s focus areas—to engage technology users in setting overall plans and

priorities for the work of the focus areas. The committees include the

senior managers of DOE field sites (such as sites with tank waste for the

Tanks Focus Area) and headquarters officials appropriate to the focus

area. These committees are to work on budgeting, planning, and setting

directions for the R&D investments of the focus areas. The committees are

modeled after the practice of the Tanks Focus Area, which set up such a

committee in 1996. The committees for the other focus areas began

organizing in February 1998.

Among other things, user steering committees will help focus areas

develop their multiyear program plans. OST is initiating these 5-year plans

Page 41 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

to manage and measure focus areas’ performance under the requirements

of the Government Performance and Results Act of 1993.2 OST plans to

complete the first set of plans by December 31, 1998, and to develop the

plans annually for the upcoming 5 years. In addition, at their meetings in

the spring of 1998, the user steering committees provided input to the

focus areas’ proposed fiscal year 2000 budgets.

Continued Upper

Management

Attention to

Deployment Is Not

Ensured

While the EM and OST initiatives have begun to address internal barriers to

the deployment of innovative technologies, continued attention by EM’s

upper management to deployment is not ensured. The attention may not

continue because (1) the future of the Technology Acceleration

Committee is uncertain and (2) deployment measures have not been

included in the contracts of EM’s senior managers.

Upper Management

Committee’s Future Is

Uncertain

In response to the July 1997 memo by EM’s Assistant Secretary, the

Technology Acceleration Committee, composed of senior-level managers

from EM headquarters and the field, was organized and met in September

1997. This Committee’s purpose is to “provide corporate leadership to

ensure an aggressive effort to deploy alternative and more effective

technologies through full integration of the technology development and

user organizations.” According to the Committee’s draft charter, it would

meet at least once per quarter. The Committee met again in January 1998,

but has not met since. According to the Acting Deputy Assistant Secretary

of OST, the Committee has been inactive because it reported directly to

EM’s Assistant Secretary, who left the Department in January 1998.

According to the Acting Assistant Secretary for EM, EM is considering

establishing a broader executive committee of senior managers to address

EM issues, including the deployment of innovative technologies.

To date, the Technology Acceleration Committee has increased

communication among OST, EM line offices, and field offices. It has

discussed issues such as clarifying deployment responsibilities, involving

technology users throughout the technology development process, and

improving incentives for contractors. The Committee also directed the

establishment of user steering committees for focus areas. Because the

user steering committees have members from EM’s headquarters and field

offices, we believe that the existence of the Technology Acceleration

Committee facilitated this innovation. The Acting Deputy Assistant

2

This act requires executive departments and agencies to establish long-term strategic goals and

annual performance goals and measures for their programs.

Page 42 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

Secretary of OST agreed with the importance of the Committee but thought

that a broader executive committee of senior officials could address

technology deployment and other EM issues.

Even with these improvements, unresolved issues affecting technology

deployment still exist and could benefit from the attention of EM’s upper

management. As noted above, EMAB suggested that the Technology

Acceleration Committee review and improve R&D performance measures.

In addition, the site-specific deployment plans state that a number of

issues need to be resolved, such as learning the possible effects of EM’s

increased use of fixed-price contracting and private financing (referred to

as “privatization”) on the use of innovative technologies.3

For example, the deployment plan of the Ohio field office raises

privatization as a policy issue requiring guidance from headquarters,

stating that most fixed-price bidders will use technologies with which they

are familiar. As a result, the plan states, technologies that were developed

at considerable expense may not be deployed because of bidders’

reluctance to assume a risk of failure. In our visits to field sites, we

observed instances in which the use of OST-developed technologies was

uncertain because EM planned to solicit fixed-price bids for cleanup work

and the technology selected would depend on the choice of the private

firm winning the contract. For instance, the Houdini robot was designed

for retrieving radioactive wastes from silos at the Fernald site. However,

when EM decided to solicit fixed-price bids for waste retrieval from

Fernald’s silos, the Houdini robot was instead used in the radioactive

waste tanks at Oak Ridge. Fernald had not yet received bids at the time of

our visit, and environmental remediation officials told us that the

companies bidding for this work will define which waste retrieval tools

they would use—Houdini might or might not be included.

EM Has Not Established

Deployment Performance

Expectations in Its Senior

Managers’ Contracts

In his July 1997 memo, the Assistant Secretary for EM stated that,

beginning in October 1997, performance expectations for EM’s senior

managers in headquarters and the field would be developed to require the

deployment of alternative and more effective technologies. However, the

Acting Deputy Assistant Secretary for OST, in response to our written

inquiry to EM management, stated that technology-related performance

measures would not be included in senior managers’ performance

contracts and that senior managers are held responsible for technology

3

Our report entitled Department of Energy: Alternative Financing and Contracting Strategies for

Cleanup Projects (GAO/RCED-98-169, May 29, 1998) provides further details about privatization.

Page 43 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

deployment through their membership in the Technology Acceleration

Committee. However, as noted above, this Committee has not met since

January 1998, and its future is uncertain.

EM and OST

Initiatives Do Not

Fully Address User

Input to Technology

Projects and User

Need for Deployment

Assistance

EM’s and OST’s current efforts and initiatives only partially address the

internal obstacles to deployment that were discussed in chapter 3.

Specifically, the new initiatives do not reinforce the need for OST’s focus

areas to use the technology development gates system and do not provide

for OST’s deployment assistance to help sites select new waste-cleanup

technologies or modify existing technologies for site use.

Initiatives Do Not Stipulate

the Use of the Gates

System

Although EM’s initiatives involve users in setting the overall plans and

priorities of OST’s focus areas, they do not fully address the need for

detailed user input on individual technology projects. The Acting Deputy

Assistant Secretary for OST told us that the focus areas need to use OST’s

existing gates system to obtain user input into the design and development

of cleanup technology. Furthermore, he said that it is necessary to use this

system to help prevent the development of technologies that do not meet

sites’ needs, a problem discussed in chapter 3. However, in contrast to

these statements of support for the gates system, we found that EM’s new

initiatives do not require its use nor identify an alternative means to ensure

that technology developers and users communicate and cooperate about

individual technology development projects.

EM Has Not Fully

Addressed Deployment

Assistance to Cleanup

Sites

EM and OST initiatives have not fully addressed two areas that must be

considered when deploying innovative technologies: (1) providing

technical assistance to sites on innovative technologies and (2) modifying

completed technologies for use at specific sites. One potential vehicle for

providing deployment assistance—OST’s new Accelerated Site Technology

Deployment program—has not increased technical assistance in most

cases and did not have the benefit of information that EM now has that EM

could use to improve its priority setting for deployment assistance.

Technical Assistance for Sites

Is Not Yet Well-Defined

EM and OST have not yet identified sources of expertise and procedures or

developed a policy for routinely providing technical assistance on

innovative technologies to DOE sites. OST recognizes that focus areas

Page 44 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

should more frequently provide technical assistance to sites when they are

selecting and beginning to implement technologies and that this assistance

should address innovative technologies developed by other sources as

well as by OST. EMAB has questioned whether the focus areas currently have

the expertise needed to provide such assistance. The Acting Deputy

Assistant Secretary for OST acknowledged that the focus areas vary in their

degree of expertise and ability to provide technical assistance. He noted

that the Tanks Focus Area works closely with one of the national

laboratories, which can provide in-depth expertise, and stated that the

other focus areas need to develop a roster of technical experts who can be

consulted for particular site cleanup problems that the focus areas cannot

solve. Furthermore, the Acting Deputy Assistant Secretary stated that

performance measures that encourage focus areas to provide technical

assistance will be needed.

Some initial steps have been taken to involve OST in selecting technology

for environmental restoration sites. In fiscal year 1998, the Office of

Environmental Restoration began including OST in its processes for

providing sites with information and support for technology selection

decisions. OST is contributing funding and the technical support of its focus

area staff to this program. However, OST does not have a similar

involvement with EM’s Office of Waste Management or Office of Nuclear

Material and Facility Stabilization.

EM lacks a policy on whether OST should provide technical assistance for

major cleanup actions routinely or only if requested by a site. While the

management-level Technology Acceleration Committee reached an

“understanding” that the focus areas’ role should include technical support

to end users for deployment, the Committee did not identify resources,

procedures, or policies for such technical assistance. According to the

Acting Deputy Assistant Secretary for OST, policies and procedures for

providing technical assistance will be one of the elements addressed in the

business system redesign currently under way in OST, and procedures may

be completed by the end of 1998.4

EM Has Not Addressed Need to

Modify Some Completed

Technologies

The initiatives do not address a barrier to deployment that we discussed in

chapter 3—the lack of a mechanism and resources for modifying

completed technologies for use at specific sites. In fact, none of the

initiatives, action plans, or meetings of the Technology Acceleration

4

OST sought advice about its business processes from a management consulting firm, which

completed a draft study in Mar. 1998. The firm suggested ways to improve and streamline OST’s

budgeting and priority setting, roles and responsibilities for technology services, information

management, and tracking of ongoing projects.

Page 45 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

Committee even raise this issue. Officials at three of the five sites we

visited told us that OST sometimes considers its technology development

work completed before technologies are ready for specific applications in

the field. The Acting Deputy Assistant Secretary for OST agreed that this is

a problem and told us that, while the Tanks Focus Area develops

technologies fully to the point of use, technologies from the other focus

areas were not always ready for field use. For example, he stated that the

Mixed Waste Focus Area had not tested its thermal treatment technologies

on actual radioactive waste. The Acting Deputy Assistant Secretary stated

that sites and focus areas should work together to enable and jointly fund

the first use of an OST-developed technology.

While joint OST and site support for deployment has occurred for some

projects—including the reactor safe-storage project at Hanford and the

use of the Houdini robot in tanks at Oak Ridge that are described in

chapter 3—EM lacks an overall policy, procedure, and designation of

responsibilities for situations in which OST-developed technologies may

require modification for site use. Nor has EM identified resources for this

purpose, except to the extent that some projects under the Accelerated

Site Technology Deployment program may address this need. According to

the Director of OST’s Office of Technology Systems, focus areas consider

any funding needs for technology modifications when requested by sites.

He noted that such requests would compete for limited funding with the

focus areas’ technology projects.

EM has data that could be used to identify OST technologies that might have

additional cost-effective deployments. Sites’ Accelerating Cleanup plans,

issued in draft in June 1997 and most recently submitted in June 1998,

provide a comprehensive compilation of sites’ technology needs, as well as

detailed information on each cleanup project across the DOE complex.5 OST

has developed a database, called a linkage table, that identifies links

between its completed and ongoing projects and the sites’ technology

needs. EM could identify OST-developed technologies that could provide

cost-effective solutions to sites’ needs and set priorities for deployment

assistance to cleanup projects, including technical assistance and

technology modifications, if needed. OST’s Deactivation and

Decommissioning Focus Area has already used this database to contact

potential technology users at the sites and inquire whether the focus area

can provide assistance. However, OST has not required its focus areas to do

this.

5

The Assistant Secretary for EM initiated the plans—now known as Accelerating Cleanup: Paths to

Closure—in 1996 to speed up the cleanup process and identify the steps needed to complete the

cleanup at each site. EM also issues a national summary of sites’ plans.

Page 46 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 4 

EM’s Efforts to Increase Deployment Do Not

Fully Address Cooperation Between

Technology Developers and Users

OST’s Accelerated Site

Technology Deployment

Program Does Not Fully Meet

Sites’ Needs for Deployment

Assistance

One potential vehicle for deployment assistance is OST’s Accelerated Site

Technology Deployment (ASTD) program, begun in fiscal year 1998. OST

funded 14 ASTD projects at 12 sites to deploy innovative technologies in

cleanup projects. The approximately $26 million that OST provided to site

projects in fiscal year 1998 resulted in an additional investment of about

$708 million from the sites over the life of the projects.6 OST identified

potential ASTD projects through site proposals and competitively evaluated

the proposals to select projects to fund. Selection criteria included the

technical merit of the approach, interest in deploying the technologies at

multiple locations, and commitment of additional funding by the site.

While ASTD may be helping these selected projects in addressing obstacles

to deployment, the program has not fostered interaction among

technology developers and users in many instances. For example, we

found that OST’s focus areas provided technical assistance to only 5 of the

14 ASTD projects, and national laboratory personnel who had helped to

develop some of the technologies provided technical assistance to 2

additional ASTD projects. It should also be noted that technical assistance

and technology modifications on a smaller scale than the current ASTD

projects may be appropriate in some cases.

6

OST plans to provide additional funds to some of the 14 projects in fiscal year 1999 and beyond.

According to the Director of the Office of Technology Systems, OST plans to provide about $21 million

in fiscal year 1999.

Page 47 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 5 

Conclusions and Recommendations

Having spent more than $2 billion and 9 years on over 700 innovative

cleanup technology projects, EM and OST recognize that the cleanup

program can only benefit from these efforts if the innovative technologies

that have been developed are successfully deployed. To promote

deployment, EM and OST have initiated a number of actions aimed at

improving the relationship between technology developers in OST and the

users at EM’s cleanup sites. However, we are concerned that the

committees and processes that EM and OST are now creating will be

ineffective if they are not accompanied by more fundamental changes in

how EM conducts technology development and deployment.

We believe that EM and OST need to take three relatively straightforward

actions to increase the deployment of existing innovative technologies

First, OST must make sure that it has adequate technical expertise to assist

users in evaluating and implementing innovative technologies that it and

others have developed. The focus areas are the logical source for this

expertise; however, if they are unable to meet this need, other centers of

expertise, possibly in the national laboratories, need to be developed.

Second, we continue to believe that OST staff, equipped with the

appropriate expertise, need to be formally involved in evaluating and

selecting technologies for use at the cleanup sites. We believe that the

program’s experience has shown that without a specific requirement to

bridge the gap between developers and users, each party will continue to

operate in its own environment, with users deploying only those

technologies with which they are familiar, and OST developing technologies

that are generic and not designed for specific situations.

Third, existing innovative technologies could be implemented, as we found

repeatedly, if they could be modified or fine-tuned to address a specific

site cleanup problem. Information now exists from sites’ Accelerating

Cleanup plans and OST’s linkage tables to identify technologies that can be

modified to fit specific situations. However, such modification takes

money, and without specific action by EM management, neither users nor

developers are likely to provide these funds on their own. For example, if

OST uses its funds to fine-tune an existing technology, it is reducing the

funds available for its other missions. Similarly, users can logically view

the use of their funds to modify a technology as taking away resources

that they need for other cleanups. However, EM’s experience, for example,

from the project for safe storage of the C reactor at the Hanford site or

from the ASTD program, has shown that successful deployment can occur if

both parties make a financial commitment.

Page 48 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 5 

Conclusions and Recommendations

Additional technology development will be needed to address technology

problems for which no cost-effective solution exists, such as high-level

waste tanks at Hanford. To ensure the deployment of technologies that are

currently under development or will be developed, EM does not need

additional processes and procedures. Rather, it needs to rigorously and

consistently apply its current gates system. Consistent use of this system

by focus areas would help ensure that technology developers and users

communicate and cooperate throughout the development of individual

technologies, and that, if technologies are not living up to their potential or

there is not adequate commitment from users, the project can be

terminated and the funds redirected to more productive uses.

Ensuring that these actions are taken consistently will require the

commitment of top management in the EM program. The Technology

Acceleration Committee is a sound idea; however, it has already missed a

planned meeting, and we are concerned that it could easily slip into

disuse. We believe that continuing a committee of senior EM managers is a

key element in ensuring that top management is focused on formulating

policy for technology deployment. An additional important element is the

establishment of performance measures that hold EM’s top managers

accountable for technology deployment. While EM has made clear to field

managers that they are responsible for deploying innovative technologies,

this commitment needs to be reflected throughout the organization if

additional innovative technologies are to be successfully deployed.

Finally, with an increased emphasis on deployment, EM will need more

accurate data than it currently has on deployment efforts. A verification

effort similar to the one we undertook will be needed to provide valid data

on future deployments. On the other hand, we recognize that improving

data on prior deployments may not be cost-effective. Therefore, reporting

existing data as estimates could lend more credibility to the data and the

overall program. In addition, EM has recognized that deployment is not the

only relevant measure of success in technology development. EM’s recent

efforts to develop additional performance measures for the entire program

are a step in the right direction.

Recommendations To increase the deployment of existing technologies and ensure that

technologies developed in the future are used, we recommend that the

Secretary of Energy direct the Assistant Secretary for Environmental

Management to

Page 49 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 5 

Conclusions and Recommendations

• direct the Deputy Assistant Secretary for the Office of Science and

Technology to establish centers of expertise for innovative technologies

by using existing focus areas or another approach if needed and require

that a representative from one of these centers participate in the

technology selection process on each cleanup project;

• direct the cleanup programs and OST to (1) use existing data to identify

OST-developed technologies that can be cost-effectively modified to meet

sites’ needs and (2) identify funds to modify these technologies if needed;

• direct that the gates system be used rigorously and consistently as a

decision-making tool for managing technology development projects and

as a vehicle for increasing developer-user cooperation;

• use their annual performance expectations to hold EM headquarters

managers responsible for increasing the deployment of innovative

technology; and

• implement a system to verify the accuracy of future deployment data and

label any existing data that have not been verified as an estimate.

Agency Comments

and Our Evaluation

Overall, DOE agreed with the recommendations in our report. In doing so,

DOE offered information regarding actions it had taken or intended to take

that it believed were responsive to our recommendations. However, DOE’s

responses to two of the recommendations suggest that the actions

described would not be fully responsive to these recommendations. DOE’s

comments are included as appendix III.

In response to our recommendation that OST establish centers of expertise

and include a representative from one of these centers in the technology

selection process, DOE indicated a willingness to act on our

recommendation but offered few specifics, especially with respect to

involving OST in the technology selection process. In 1994, we also

recommended that OST be given a formal role in the technology selection

process. During our current review, we found that this recommendation

had not been implemented primarily because site officials were skeptical

about OST’s ability to provide quality technical advice and were therefore

reluctant to allow OST more of a role in selecting cleanup technologies. We

believe that it will take more specific actions by OST, beyond the

generalized user steering committees cited in its response, to develop

credible expertise and thus gain a role in the technology selection process.

In response to our recommendation that DOE rigorously and consistently

use the gates system as a decision-making tool for managing technology

development, DOE also agreed with the recommendation but noted that it

Page 50 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Chapter 5 

Conclusions and Recommendations

had incorporated the gates system into its system of peer review. While we

recognize the value of peer review as a mechanism for obtaining

independent technical judgments about projects OST is pursuing, we note

that peer review can occur infrequently over the life of a project and after

significant decisions are made. Therefore, we do not believe that peer

review is a substitute for focus area managers using a disciplined

decision-making system that involves users throughout the technology

development and deployment process.

Page 51 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix I 

Methodology Used to Assess the Reliability

of OST’s Data

This appendix describes the methodology we used to assess the reliability

of the Office of Science and Technology’s (OST) data in order to address

the question of the extent to which innovative technologies developed by

OST have been deployed (used) at the Department of Energy’s (DOE) waste

cleanup sites and to compare this rate of deployment with the rates of

other government organizations that develop environmental technologies.

To answer this question, we obtained an electronic file from OST

containing project data from OST’s inception in 1989 through January 1998.

OST had initially developed most of these data in response to other

congressional inquiries. The data included, among other things, each

project’s total cost to OST, research stage, number of times deployed and

associated deployment sites. We limited our reliability assessments to data

on the number of times each project was deployed and the deployment

sites identified in the database.

Methodology We used attribute sampling techniques and verified claimed technology

deployments with site operations officials or with individuals familiar with

the technology’s use at that site. We used our sample results to estimate a

range for the actual number of OST project deployments.

Universe The OST data listed 713 projects initiated since the program’s inception.

The records for 152 of these projects indicated the technology was

deployed at one or more sites. We chose to verify the deployments for this

group of projects because (1) no clear-cut way exists to verify that a

technology has not been used at any DOE site and (2) we believed that the

risk was low that OST neglected to count significant numbers of deployed

projects, thereby understating its success.

Sample and Verification We randomly selected a sample of 30 projects from the 152 projects that

OST claimed were deployed 1 or more times. For each project in our

sample, we contacted all of the deployment sites listed in OST’s database

under the project. To the extent possible, we verified project deployment

either with DOE Environmental Management (EM) or contractor officials

having operational duties at the sites. At the time of our reliability

assessment, OST lacked a formal definition of deployment. We developed

the following definition on the basis of (1) how the term was used by the

Committee requesting our review, (2) what the need was for comparability

with the two other organizations providing deployment data, and (3) what

Page 52 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix I 

Methodology Used to Assess the Reliability

of OST’s Data

technology users thought was reasonable. We considered a technology to

be deployed if, at any site listed,

• the cleanup or waste management project was completed using the

technology;

• the cleanup or waste management project is currently using the

technology;

• a contract is in place for use of the technology in the future; or

• a demonstration either accomplished the cleanup or characterization

goals, was expanded to accomplish these goals, or resulted in a contract

for the use of the technology at that site.

Estimations Using our sample results, we estimated a range for the actual number of

projects deployed 1 or more times. Because we used a probability sample

of deployed projects to estimate the number of actual deployments, this

estimate has a measurable precision, or sampling error, which may be

expressed as a plus/minus figure. The sampling error indicates how closely

we can reproduce from a sample the results that we would obtain if we

were to take a complete count of the universe, using the same verification

methods. By adding the sampling error to and subtracting it from the

estimate, we can develop upper and lower bounds for each estimate. The

resulting range is called a confidence interval. Sampling errors and

confidence intervals are stated at a certain confidence level—in this case,

95 percent. The 95-percent confidence level means that in 95 out of 100

instances, the sampling procedure we used would produce a confidence

interval containing the universe value we are estimating.

As a result, we estimate that a total of 88 to 130 OST projects have been

deployed 1 or more times. We used this estimated range of actual

deployments to compute the estimated range of OST’s deployment rate. The

deployment rate represents the proportion of deployed projects to the

total number of technology development projects started by OST. For our

estimate of the deployment rate, we substituted the number of deployed

projects reported by OST with the range we estimated above. As a result,

we computed that OST actually deployed from 88 to 130 of the 713 projects

reported from 1989 to January 1998, or 12 to 18 percent.

Owing to multiple deployments for some projects, the OST database

reported 283 deployment instances (locations) for the 152 deployed

projects, naming a site for most instances. For example, the Waste

Inspection Tomography project was listed as deployed once at Idaho

Page 53 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix I 

Methodology Used to Assess the Reliability

of OST’s Data

National Environmental Engineering Laboratory, once at Hanford, and

twice at Lawrence Livermore National Laboratory, for a total of four

deployment instances.

Using the same verification sample, we performed cluster sampling

analysis to estimate a range for the actual total number of deployment

instances for all OST projects. From this analysis, we estimate that the

number of deployment instances ranges from 137 to 216. Figure I.1

illustrates how we used the same universe and sample to verify both

project deployments and the total number of deployment instances

claimed.

Figure I.1: Universe and Sample of

OST Projects

Universe

Sample

713

projects in

OST database

56

deployments

283

deployments

No

deployments

561

projects

152

projects

30

projects

Page 54 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix II 

OST’s Deployment Rate Compared With the

Rates of Two Governmental Environmental

Technology Demonstration Programs

Two government programs that demonstrate innovative environmental

technologies provided deployment data for comparison with OST’s data:

the Environmental Protection Agency’s (EPA) Superfund Innovative

Technology Evaluation (SITE) program and the Department of Defense’s

Environmental Security Technology Certification Program (ESTCP). As

discussed in chapter 2, we found that any deployment rate comparisons

have limited usefulness. Nevertheless, deployment data from these

programs provide some evidence that OST has deployed its technologies at

rates lower than or close to rates experienced by other governmental

programs in this area of research and development. This appendix

describes the two demonstration programs and the comparisons we made.

Unlike OST, which develops technologies beginning with basic research

and progressing through their demonstration and implementation, the SITE

program and ESTCP only demonstrate and implement existing technologies.

In order to provide equitable comparisons, we identified those OST projects

at stages of maturity similar to SITE and ESTCP projects, resulting in two

subgroups of OST projects to use for comparison. Furthermore, we

adjusted OST’s deployment rates for the two subgroups to reflect the errors

that we found in OST’s deployment data for each group. The methodology

used for these adjustments is described in appendix I. We then compared

the adjusted deployment rates for the two subgroups of OST projects with

the deployment rates for SITE and ESTCP.

Superfund Innovative

Technology

Evaluation

The SITE program has been demonstrating innovative environmental

technologies for about 12 years. EPA initiated the program in response to

the 1986 Superfund Amendments and Reauthorization Act. The program’s

mission is to encourage the development and implementation of

(1) innovative treatment technologies for remediating hazardous waste

sites and (2) characterization and monitoring technologies for evaluating

the nature and extent of contamination at hazardous waste sites. The SITE

program does not perform earlier stages of technology development but

demonstrates technologies in order to evaluate their effectiveness. It

provides methodologies for demonstrations and funds associated sampling

and testing. The technology vendors generally pay for all technology

equipment. According to a program official, the SITE program’s fiscal year

1997 budget was $6.8 million.

SITE published its demonstration results for all projects completed through

December 1996 in EPA SITE Technology Profiles, Ninth Edition. To gather

information for this publication, the SITE program had surveyed its

Page 55 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix II 

OST’s Deployment Rate Compared With the

Rates of Two Governmental Environmental

Technology Demonstration Programs

vendors, asking them, for example, to report how many contracts or jobs

they have obtained to use the SITE-demonstrated technologies. Although

they had not previously analyzed the survey data in this manner, SITE

officials agreed to compute a deployment rate for SITE technologies by

counting a technology project as deployed if its vendor reported one or

more contracts or jobs to use the technology. According to SITE officials,

because up to 10 years had elapsed since the first demonstrations and

some vendors had changed management or gone out of business, about

64 percent of vendors responded to the surveys. SITE computed a

deployment rate for all 80 completed demonstrations, whether or not the

vendor responded to the surveys. As a result, the SITE deployment rate may

be based on incomplete reporting of deployments and could actually be

higher.

The SITE program reported that 59 percent of its demonstration projects

were deployed at least once.1

 For projects at a similar stage of maturity,

we estimate that OST deployed from 28 to 45 percent. We computed this

OST deployment rate for projects OST identified as either having ended the

demonstration stage of development or having entered the implementation

stage. OST listed 191 projects in this subgroup and claimed that 94 were

deployed 1 or more times. From our verification sample of 30 projects, 18

projects fell into this subgroup, and we found four errors. Using the same

statistical methods described in appendix I, we estimate that OST actually

deployed from 53 to 86 of these relatively mature projects, to produce an

estimated deployment rate of 28 to 45 percent, which is lower than the rate

for the SITE program.

Environmental

Security Technology

Certification Program

The Department of Defense’s ESTCP demonstrates and validates

technologies and funds environmental technologies that have progressed

to the stage where field demonstrations are warranted. ESTCP has existed

only since fiscal year 1995. Since the program is relatively new, its

deployment data are based on a limited number of projects and may be

less representative of the program’s future performance. Like the SITE

program, ESTCP demonstrates technologies that other research

organizations have brought to the demonstration stage. ESTCP also

encourages investment by the technology user or vendor. ESTCP operated

on a budget of about $22 million in fiscal year 1997.

1

We did not test the reliability of SITE or ESTCP data, but we believe the data to be reasonable on the

basis of our review of project descriptions.

Page 56 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix II 

OST’s Deployment Rate Compared With the

Rates of Two Governmental Environmental

Technology Demonstration Programs

ESTCP provided a description of the transition status (the technology’s

actual use or selection for future use) for 32 ESTCP projects started in fiscal

years 1995 or 1996.2

 Because the number of projects was small, we

decided to use a deployment rate for comparison that was based upon the

transition status of all projects ESTCP began in its first 2 years of

existence—regardless of whether they had been completed.

ESTCP reported that, for projects started in the program’s first 2 years of

existence, 38 percent were deployed at least once. For projects at a similar

stage of maturity, we estimated that OST deployed from 28 to 40 percent.

Our estimate was based on OST projects that had entered demonstration or

later stages, regardless of whether OST had ended the project.3 OST listed

290 projects in this subgroup and said that 129 were deployed 1 or more

times. From our verification sample of 27 projects that fell into this

subgroup, we found six errors. Using the same statistical methods as for

the SITE program comparison, we estimated that OST actually deployed

from 80 to 115 of these projects, to produce an estimated deployment rate

of 28 to 40 percent for all of its projects reaching demonstration or later

stages. This rate is similar to that achieved by ESTCP in its first 2 years of

existence.

2

In fiscal years 1995 and 1996, ESTCP started 33 projects; however, we removed 1 project from the

analysis because its objective was not technology deployment.

3

The subgroups of OST projects that we used for comparison differ in that the SITE program’s

comparison subgroup included all OST projects that had ended the demonstration stage and/or

entered into the implementation stage, while the comparison subgroup for ESTCP included all OST

projects that had at least entered the demonstration or implementation stage of development.

Page 57 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix III 

Comments of the Department of Energy

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

Page 58 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix III 

Comments of the Department of Energy

Now on pp. 3 and 15.

See comment 1.

Now on pp. 7 and 33-34.

See comment 1.

See comment 1.

Now on p. 15.

See comment 1.

Now on pp. 15 and 16.

See comment 1.

Now on p. 17.

See comment 1.

Now on p. 24.

See comment 1.

Now on p. 32.

See comment 1.

Now on p. 40.

See comment 1.

Page 59 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix III 

Comments of the Department of Energy

Now on p. 43.

See comment 1.

See comment 2.

Page 60 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix III 

Comments of the Department of Energy

See comment 3.

Page 61 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix III 

Comments of the Department of Energy

The following are GAO’s comments on DOE’s letter dated August 19, 1998.

GAO’s Comments 1. We concur with this comment and have made changes to the report

where appropriate.

2. See chapter 5 for our reply to DOE’s response to our recommendation

that OST establish centers of expertise and include a representative from

one of these centers in the technology selection process.

3. See chapter 5 for our reply to DOE’s response to our recommendation

that DOE rigorously and consistently use the gates system as a

decision-making tool for managing technology development.

Page 62 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix IV 

Comments of the Department of Defense

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

See comment 1.

Page 63 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix IV 

Comments of the Department of Defense

The following is GAO’s comment on the letter from the Department of

Defense, dated August 28, 1998.

GAO’s Comment 1. We concur with this comment and have made changes to the report

where appropriate.

Page 64 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Appendix V 

Major Contributors to This Report

Resources,

Community, and

Economic

Development Division

James Noël, Assistant Director

Rachel J. Hesselink, Evaluator-In-Charge

Jack Paul, Senior Evaluator

Ilene M. Pollack, Senior Evaluator

Seattle Field Office Patricia J. Rennie, Senior Evaluator

Evan Stoll, Programmer Analyst

Office of General

Counsel

Susan W. Irwin, Senior Attorney

(141094) Page 65 GAO/RCED-98-249 DOE’s Innovative Cleanup Technologies
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100