United States

Environmental Protection

Agency

Office of Research and

Development

Washington DC 20460

EPA/600/K-96/003

February 1997

Environmental Technology

Verification Program

Verification Strategy

Updated September 1997
 Verification Strategy i

Environmental Technology

Verification Program

Verification Strategy

Office of Research and Development

U.S. Environmental Protection Agency

Washington, DC 20460

EPA/600/K-96/003

February 1997

Updated September 1997

Printed on Recycled Paper
 Verification Strategy ii
 Verification Strategy iii

Foreword

The U.S. Environmental Protection Agency’s (EPA) Office of

Research and Development (ORD) is publishing this document,

Verification Strategy, to report the goals, operating principles, and future

plans of the newly created Environmental Technology Verification

Program (ETV). ETV is intended to expand the environmental technology

choices of public and private decision makers, both in our country and

abroad. The use of new technologies will allow faster and more thorough

pollution prevention, control, cleanup, and monitoring, while bringing the

overall cost of regulatory compliance down.

Evaluations of promising new technologies by ETV should be

useful to a broad array of organizations:

• Buyers of technology, and the consulting engineering community

that advises them, who need an objective source of data to make

informed purchasing decisions.

• Permitters at state, local and federal level who must decide on

which technologies to allow into use.

• The financial community that must determine the level of risk

involved in supporting innovative technology developers.

• Vendors who want to market their products on a level

playing field of objectively acquired data.

The challenges of the next three years will be substantial as we

produce and broadly distribute credible data that are products of ETV.

Meeting these challenges will ensure that the environment, the economy,

and the marketplace benefit from the steady stream of scientific and

technological innovation this program will encourage.

Mr. Henry L. Longest II

Acting Assistant Administrator

Office of Research and Development

U.S. Environmental Protection Agency
 Verification Strategy iv

Contents

Page

Foreword ................................................................................................... iii

Background ............................................................................................... 1

Important Definitions.................................................................................. 2

Operating Principles .................................................................................. 3

ETV Pilot Process ..................................................................................... 7

Selection Criteria for ETV Pilot Projects .................................................... 9

2001 Report to Congress ........................................................................ 11

ETV Program Vision and Projected Development Scenario ................... 11

ETV Web Site Information ....................................................................... 14
 Verification Strategy 1

Background

Throughout its history, the U.S. Environmental Protection Agency

(EPA) has evaluated technologies to determine their effectiveness in

monitoring, preventing, controlling, and cleaning up pollution. Since the

early 1990s, however, numerous government and private groups have

identified the lack of an organized and ongoing program to produce

independent, credible performance data as a major impediment to the

development and use of innovative environmental technology. Such data

are needed by technology buyers and permitters both at home and abroad

to make informed technology decisions. Because of this broad input, the

President’s environmental technology strategy, Bridge to a Sustainable

Future, and the Vice President’s National Performance Review, contain

initiatives for an EPA program to accelerate the development of

environmental technology through objective verification and reporting of

technology performance. In 1994, EPA’s Office of Research and Development formed a workgroup to plan the implementation of the Environmental

Technology Verification Program (ETV). The workgroup produced a

Verification White Paper that guided the initial stages of the program. This

document, Verification Strategy, updates the earlier paper based upon the

evolution of the program over the last two years. It outlines the operating

principles and implementation activities that are shaping the program, as

well as the challenges that are emerging and the decisions that must be

addressed in the future. ETV will continue to be modified through input

from all parties having a stake in environmental technology, through further

operational experience, and through formal evaluation of the program.

The goal of ETV, which remains unchanged, is to verify the

environmental performance characteristics of commercial-ready technology through the evaluation of objective and quality assured data, so that

potential purchasers and permitters are provided with an independent and

credible assessment of what they are buying and permitting.

Environmental Technology Verification Program

Verification Strategy
2 Verification Strategy

Important Definitions

A clear definition of the words “evaluate” and “verify”, along with the

word “certify”, is important to establish at the outset of the program. The

technology development community, the regulated community, and those

charged with executing environmental standards at all levels of

government require a precise understanding of what EPA means and does

not mean by the activities to be undertaken through ETV.

EPA intends to sponsor the evaluation of environmental technologies

through adequate testing and verify that they perform at the levels reported.

By evaluate and verify we mean:

Evaluate / Evaluation

To carefully examine and judge the efficacy of a technology; to

submit technologies for testing under conditions of observation and

analysis; syn., measure, estimate, classify, test.

Verify / Verification

To establish or prove the truth of the performance of a technology

under specific, predetermined criteria or protocols and adequate

data quality assurance procedures; syn., confirm, corroborate,

substantiate, validate.

EPA does not intend to certify that a technology will always, or

under circumstances other than those used in testing, operate at the levels

verified. By certify we mean:

Certify / Certification

To guarantee a technology as meeting a standard or performance

criteria into the future; syn., ensure, warrant, guarantee.

EPA understands that the word certify can have a variety of

meanings, but the Agency believes that the above definition is the one

most commonly understood. Misuse of the term could cause confusion

among the public.
 Verification Strategy 3

Operating Principles

Several important operating principles have defined the basic ETV

program structure and remain fundamental to its operation. These are

briefly outlined below.

1. Performance Evaluation Goal

Under ETV, environmental technologies are evaluated to ascertain and

report their performance characteristics. EPA and its partners will not

seek to determine regulatory compliance; will not rank technologies or

compare their performance; will not label or list technologies as

acceptable or unacceptable; and will not seek to determine “best

available technology” in any form. In general, the Agency will avoid all

potential pathways to picking “winners and losers”. The goal of the

program is to make objective performance information available to all of

the actors in the environmental marketplace for their consideration and

decision making.

2. Commercial-Ready Technologies

The ETV program is a service of EPA to the domestic and international

marketplace in order to encourage rapid acceptance and

implementation of improved environmental technology. ETV, therefore,

focuses its resources on technologies that are either in, or ready for,

full-scale commercialization. The program does not evaluate

technologies at the pilot or bench scale and does not conduct or

support research. Participation in ETV is completely voluntary.

3. Third-Party Verification Organizations

ETV leverages the capacity, expertise, and existing facilities of others

through third-party partnerships in order to achieve universal coverage

for all technology types as rapidly as possible. Third-party verification

organizations are chosen from the both the public and private sector,
4 Verification Strategy

including states, universities, associations, business consortia, private

testing firms, and federal laboratories. EPA designs and conducts

auditing and oversight procedures of these organizations, as

appropriate, to assure the credibility of the process and data. In order to

determine if EPA participation is important to the commercialization

process, ETV is testing the option of one totally unstructured and

independent, private sector pilot in which EPA’s role will be solely

fiduciary. In addition, the Agency will continue to publish the results of

commercial-ready technology evaluations that it conducts in the normal

course of its business.

4. Pilot Phase

The program will begin with a five year pilot phase to test a wide range

of partner and procedural alternatives, as well as the true market

demand for and response to such a program. Throughout the pilot

period, EPA and its partners will operate in a flexible and creative

manner in order to identify new and efficient methods to verify

environmental technologies, while maintaining the highest credibility

standards. The operational objective will be to actively look for ways to

optimize procedures without compromising quality. The ultimate

objective of the pilot phase is to design and implement a permanent

verification capacity and program within EPA, should the evaluation of

the effectiveness of the program warrant it.

5. Pilot Technology Areas

ETV has begun with pilots in narrow technology areas in each of the

major environmental media and will expand as appropriate, based on

market forces, availability of resources, and the willingness of the

marketplace to pay for third-party verification. For example, the drinking

water technology pilot has started with a focus on microbial and

particulate contaminants, and disinfection byproducts in small systems

(less than 3300 users), an obvious and very large domestic and

international market with pressing environmental problems. In future

years, the program will be expanded to the wider area of nitrates and

synthetic organic chemicals and pesticides in all drinking water

systems. Success in particular technology areas will allow the program

to have a “pump-priming” effect to bring new technologies to the

marketplace. Selection criteria for ETV pilot programs and other

verification focus areas are discussed in a subsequent section of this

paper.
 Verification Strategy 5

6. Stakeholder Groups

ETV is guided and shaped by using the expertise of appropriate

stakeholder groups in all aspects of the program. These groups consist

of representatives of all verification customer groups: buyers and users

of technology, developers and vendors, and, most importantly,

technology “enablers”, i.e., the consulting engineering community that

recommends technology alternatives to purchasers, and the state

permitters and regulators who allow it to be used. Stakeholder groups

must be unique to each technology area in order to capture the

important individual aspects of the different environmental media and to

get buy-in from affected groups. For example, state drinking water

permitters are necessary to participate in development of testing

protocols for cryptosporidium; air pollution regulators are needed to

evaluate innovative compliance monitoring devices; metal production

parts manufacturers need to help design testing procedures for new

coating compounds. In general, the role of stakeholders will be to assist

in the development of procedures and protocols, prioritize types of

technologies to be verified, review all important documents emerging

from the pilot, assist in defining and conducting outreach activities

appropriate to the particular area, and, finally, to serve as information

conduits to the particular constituencies that they represent. As of

September 1997, over 180 individuals are serving in the six

stakeholder groups formed to date.

7. Private Sector Funding

Over the five year pilot phase of the program, the costs of verifying

technologies in many pilots will move from a primarily government

funded effort to a primarily private sector funded effort. At least two

pilots will be vendor supported from the beginning. The original goal, as

articulated in the 1994 strategy, called for complete private sector

sponsorship within three years. A recent review (1995) of the program

by a distinguished panel of outside experts convened by the EPA

Science Advisory Board (SAB) concluded that such a goal was

probably not achievable in so short a time-frame (they suggested five to

eight years) and that some level of government support (10 to 20% of

ongoing costs) would remain necessary to keep the activity viable.

Conclusions on this issue will be reached as data emerge on the

economic value-added of the program and the level of cost that the

private sector is willing to bear in the various technology sectors.
6 Verification Strategy

8. Pilot Evaluation and Program Decisions

The Agency will collect data on operational parameters, e.g., number of

participants; cost and time required to perform tests and report results,

and on outcomes, e.g., use of data by the states and public; sales

reported by vendors, in order to evaluate all aspects of the program.

EPA will use this information to make long-term recommendations to

the Congress on the future and shape of the program in 2001. Among

the choices at that time will be the formulation of a permanent, broad

scale program; the narrowing of efforts to certain areas in which ETV

appears to be effective; or the discontinuance of verification efforts. The

latter conclusion could be reached either because state regulators/

permit writers and the technology innovation industry are not assisted

by ETV or because the cost of verification proves to be prohibitive.

9. Outreach and Information Diffusion

As was pointed out by the SAB in its 1995 review of ETV, verification

alone will not move better, cheaper, faster technologies to success in

the marketplace. In particular, substantive and substantial interface with

the permitters of environmental technology (primarily at the state level)

will be necessary to have any chance of rapidly implementing

innovative approaches. To date, the outreach activities of the program

have been limited to assuring substantial state representation on the

Stakeholder Groups that are designing the protocols and procedures

for each pilot; developing informational fact sheets about the program;

and placing a Web page on the Internet. In 1998, the Agency intends to

develop an overarching outreach strategy through consultation with all

affected groups and with the help of major organizations in the

technology area, e.g., National Governors Association, Western

Governors Association, Environmental Council of the States, National

Pollution Prevention Roundtable, appropriate corporations, and others.

10.Market Gap Definition

Lastly, EPA will track applications and expressions of interest on the

part of technology developers who come to all parts of the Agency that

do not fit into the present suite of verification activities. This universe

will be characterized during the initial stages of the pilot period and a

strategy to address gaps will be developed.
 Verification Strategy 7

ETV Pilot Process

Although a wide degree of flexibility will characterize the pilot

projects (see above), each ETV pilot generally will go through two periods

of development: an organizational phase and an operational phase (see

Figure 1).

Figure 1. ETV Pilot Process

Organization Phase Operations Phase

Verification Organization (VO)

Selected

(EPA)

Stakeholder Groups Formulated

(EPA/VO)

Technologies Prioritized

and

Generic Protocols Developed

(EPA/VO/Stakeholders)

Open Solicitation

 (VO)

Generic/Tailored Test Plans

(Developer/VO)

Testing by VO

OR

Testing by Developer

(With VO Oversight)

Verification Report/Statement

(EPA/VO)

Information Diffusion

(EPA/Stakeholders)

QA Evaluation

(EPA/VO)
8 Verification Strategy

Organizational Phase

During the organizational phase, EPA will select one or more partner

organizations to oversee and conduct verification activities. This

important step will usually occur through an open solicitation process,

although some exceptions will be appropriate. All partner proposals will

be peer reviewed. EPA and its partner(s) will then select approximately

25 appropriate participants for the Stakeholder Group (see #6) that will

guide the progress of the program. EPA may formulate the

Stakeholders Group prior to selecting the partner organization if the

procurement process is protracted. Stakeholder Groups will then begin

the important process of establishing priorities and defining procedures

and protocols appropriate to that particular type of environmental

technology and customer group.

Operational Phase

Once the basic building blocks of the program are in place, actual

verification activities begin (these steps will not always be strictly

sequential). Verification activities in each technology area will be

announced in the Commerce Business Daily and other appropriate

publications to encourage maximal participation by technology

developers, and to assure a level playing field. Test plans will be

developed with the participation of developers and tests conducted by

independent third parties (either the verification organization or other

testing organizations approved by the verification organization).

Appropriate quality assurance procedures will be incorporated into all

aspects of the project and reports will be peer reviewed. Verification

statements of three to five pages, based on the performance data

contained in the reports, will be issued by EPA and appear on the ETV

Internet Web page. Other outreach activities, as defined by the

Stakeholder Group, such as state permitter training, will be conducted.
 Verification Strategy 9

Selection Criteria for ETV Pilot Projects

The appropriate selection of verification pilot programs and other

ETV verification activities to be carried out under the program is critical to

its ultimate success. The ETV pilot programs are designed to meet the

needs of the many stakeholders in environmental technology, while

allowing EPA to experiment with a variety of procedures and partnership

arrangements to determine the optimal form of program implementation.

The following criteria have been used to choose ETV verification pilots

and other verification activities. The first three criteria are applied in

cascading order, i.e., each pilot passes the previous criterion in order to go

on to the next.

1. Address Important Environmental Needs

All programs conducted by the EPA have environmental improvement

and protection as their ultimate objective. It is particularly important

that this program, which is designed to assist the environmental

technology industry, select pilot verification activities that have clear

and positive environmental benefits. Such benefits may improve the

environment by achieving higher levels of pollution reduction or by

accelerating the rate of technology implementation through lower cost

or simplified operation.

2. Present Substantial Business Opportunities for the Private

Sector

Pilot areas selected have clear market niches, both domestically and

internationally, that present the potential for a substantial increase in

technology sales and use. Although voluntary environmental

improvement technologies, e.g., indoor air filtration systems, and

existing regulatory program technologies, e.g., hazardous waste

monitoring devices, are of interest to the program, particular attention

will be paid to technology areas that stress the Agency’s commitment to

pollution prevention or address upcoming regulatory requirements and
10 Verification Strategy

deadlines that present obvious and major opportunities for innovation

and increased economic activity.

3. Involve Multiple Developers and Vendors

To clearly demonstrate the potential of a successful verification

program, the ETV pilots must benefit the largest possible number of

technology developers and users. Pilot programs will focus on areas in

which a number of technologies and companies are active.

The last two criteria are programmatic and allow the Agency to test

operating program parameters in the pilot phase.

4. Address the Full Range of Environmental Media

Assuring that all major environmental media and program areas are

included in the pilots will allow ETV to begin interaction with interested

parties, e.g., technology developers, technology buyers, regulation

writers and state permitters, across a broad range of environmental

areas. This will assist the program to identify particularly fruitful

technology focus areas for the future, spread the word about the EPA

verification function widely, identify regulatory barriers that inhibit the

use of verified technologies, and assist developers across a wide

spectrum of technology.

5. Test a Variety of Verification Organization Types

As described above, EPA is interested in evaluating all possible

verification organization alternatives, including federal government

laboratories, state verification programs, universities, industry

associations, independent testing organizations, and developer

conducted testing (with verification organization or EPA oversight).

Pilot programs selected are expected to cover all of these alternatives.

Having applied the above criteria to a wide variety of potential pilot

candidates, the Agency has selected twelve categories as ETV pilots (see

Table 1).
 Verification Strategy 11

1. Site Characterization & Monitoring Technologies

2. P2, Recycling & Waste Treatment Technologies

3. Drinking Water Systems

4. Indoor Air Products

5. P2 Innovative Coatings and Coating Equipment

6. EvTEC (any technology area)

7. Advanced Monitoring Systems (Air and Water)

8. Air Pollution Control Technology

9. Greenhouse Gas Technology

10. Wet Weather Flow Technologies

11. Source Water Protection Technologies

12. P2 Metal Finishing Technologies

2001 Report to Congress

ETV will prepare a Report to Congress, to be sumitted in 2001. The

questions that will be addressed in the Report include:

1. Does the environmental marketplace need/value an EPA

verification program?

2. Is the need different from one technology area to another?

3. Can we design a credible program that the developer community

can afford?

4. Where is value added from ETV:

• Protocol development?

• Third party QA oversight?

• EPA Verification Statement?

ETV Program Vision and Projected Development

Scenario

Figure 2 lays out the projected ten year development scenario for

the ETV Program in which assumptions are made concerning: (1) the

amount of funding received by the program; (2) the number of viable

technology areas (defined as pilots in the early stages) needed; (3) the

number of EPA staff assigned to quality assurance and technical oversight;

and (4) the number of new technologies that come to the program for

verification. All of these assumptions are projections based upon the

Table 1. ETV Pilots
12 Verification Strategy

experience of technology verification programs in other fields and the level

of resources needed to accomplish the work projected. Changes to any of

the assumptions would, of course, change all of the projections.

1. Federal Funding

The projected budget moves from $7M in 1995 to $10M for three years

of development and building market acceptance. It then begins a

gradual decline as the private sector pays more and more of the costs.

By 2005 the steady-state cost of the program for all environmental

areas is projected to be $2M.

0

5

10

15

20

25

30

35

40

45

50

1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005

Federal $M Per Year Technology Areas

# Verified Technologies Per Year FTE

Figure 2. ETV Program Scenario
 Verification Strategy 13

2 EPA personnel are assigned to ETV from appropriate media areas (e.g., drinking

water, advanced air monitors) within ORD’s national laboratories and centers

based upon programmatic requirements. Six EPA staff are assigned to ETV

directly for management coordination, information dissemination, and oversight.

2. Technology Areas

Starting in 1995, pilot programs gradually expand to cover all

appropriate environmental areas (areas defined primarily by customer

groups). By 1998, twelve areas are defined and implemented and

remain in place throughout the program, although their technology

focus can shift over time, based on market forces.

3. EPA Personnel2

EPA staff necessary to oversee third-party verifiers and maintain the

quality and credibility of the program move from 10 in 1995 to as many

as 30 while the program is in its formative stages. As procedures and

quality assurance measurers become routinized, staff demands

gradually lower to a steady state of 10 FTE by 2003.

4. Verified Technologies

Due to a demand backlog, the number of technologies verified rapidly

rises from five in 1996 to 50 in 2001, and then declines to a steady state

of about 30 to 35 technologies each year.

If executed as projected in this scenario, the Environmental

Technology Verification Program could reasonably be expected to verify

the performance of approximately 300 innovative technologies in a

decade.
14 Verification Strategy

Accessing the Latest ETV Information

Ongoing information about the progress of the ETV Program,

including all published documents and a list of upcoming events, can

be downloaded from the ETV Web Site at http://www.epa.gov/etv

ETV Web Site Information