United States General Accounting Office

GAO Testimony

Before the House Subcommittee on Energy and

Environment, Committee on Science, House of

Representatives

For Release

on Delivery

Expected at

1 p.m. EST

Tuesday

March 11, 1997

PEER REVIEW

EPA’s Implementation

Remains Uneven

Statement by Stanley J. Czerwinski, Associate Director,

Environmental Protection Issues,

Resources, Community, and Economic

Development Division

GAO/T-RCED-97-95

Mr. Chairman and Members of the Subcommittee:

We are pleased to be here today to discuss our recent report on the

Environmental Protection Agency’s (EPA) implementation of its peer

review policy.1

 As you know, peer review is the critical evaluation of

scientific and technical work products by independent experts to enhance

the products’ quality, credibility, and acceptability. Furthermore, peer

review can actually help avoid costly and time-consuming delays by

helping to steer product development along the most efficient, effective

course. The EPA’s current peer review policy—updated in 1994 and

currently under evaluation by the agency in the light of our report’s

findings—stresses the importance of such reviews and calls for the peer

review of all major scientific and technical work products that may

eventually play an important role in key agency decisions. In the light of

the critical role that peer review plays in supporting the agency’s

important decisions, we assessed EPA’s (1) progress in implementing its

peer review policy and (2) efforts to improve the peer review process.

In summary, we found that:

• Despite some recent progress, peer review continues to be implemented

unevenly. Although we found some cases in which EPA’s peer review policy

was properly followed, we also found cases in which key aspects of the

policy were not followed or in which peer review was not conducted at all.

We believe that two of the primary reasons for this uneven implementation

are (1) inadequate accountability and oversight to ensure that all relevant

products are properly peer reviewed and (2) confusion among EPA’s staff

and management about what peer review is, its importance and benefits,

and how and when it should be conducted.

• EPA officials readily acknowledge this uneven implementation and, during

the course of our work, had a number of efforts under way to improve the

peer review process. Although we found these efforts to be steps in the

right direction, we concluded that they were not addressing the underlying

problems that we had identified. Accordingly, we recommended that EPA

ensure that (1) upper-level managers have the information they need to

know whether or not all relevant products have been considered for peer

review and (2) staff and managers are educated about the need for and

benefits of peer review and their specific responsibilities in implementing

the policy. EPA agreed with our recommendations and has several efforts

under way to implement them. For example, EPA plans to initiate a peer

review training program for its managers and staff in June 1997. While it is

1

Peer Review: EPA’s Implementation Remains Uneven (GAO/RCED-96-236, Sept. 24, 1996).

Page 1 GAO/T-RCED-97-95
still too early to be certain if these efforts will be fully successful, we are

encouraged by the high-level attention being paid to this very important

process.

Background Peer review is well established as a mechanism for assuring the quality,

credibility, and acceptability of individual and institutional work products.

This assurance is accomplished by having the products undergo an

objective, critical review by independent reviewers. Peer review has long

been used by academia, professional organizations, industry, and

government. Within EPA, peer review has taken many different forms,

depending upon the nature of the work product, the relevant statutory

requirements, and office-specific practices and needs.

In keeping with scientific custom and/or congressional mandates, several

offices within EPA have used peer review for many years to enhance the

quality of science within the agency. In response to a panel of outside

academicians’ recommendations in 1992,2 EPA issued a policy statement in

1993 calling for peer review of the major scientific and technical work

products used to support the agency’s rulemaking and other decisions.

However, the Congress, GAO,

3

 and others subsequently raised concerns

that the policy was not being implemented consistently across the agency.

In response to these concerns, in 1994 EPA reaffirmed the central role that

peer review plays in ensuring that the agency’s decisions are based on

sound science and credible data and revised its 1993 policy.

The new policy, while retaining the essence of the prior one, was intended

to expand and improve the use of peer review throughout EPA. The 1994

policy continued to stress that major products should normally be peer

reviewed, but it also recognized that statutory and court-ordered

deadlines, resource limitations, and other constraints might limit or even

preclude the use of peer review. The policy applied to major work

products that are primarily scientific or technical in nature and that may

contribute to the basis for policy or regulatory decisions. In contrast, other

products used in decision-making are not covered by the policy, nor are

the ultimate decisions themselves. While peer review can take place at

several different points along a product’s development, such as during the

planning stage, it should be applied to a relatively well-developed product.

2

Safeguarding the Future: Credible Science, Credible Decisions (EPA/600/9-91/050, Mar. 1992).

3

Peer Review: EPA Needs Implementation Procedures and Additional Controls (GAO/RCED-94-89,

Feb. 22, 1994).

Page 2 GAO/T-RCED-97-95
The 1994 policy also clarified that peer review is not the same thing as the

peer input, stakeholders’ involvement, or public comment—mechanisms

used by EPA to develop products, to obtain the views of interested and

affected parties, and/or to build consensus among the regulated

community. While each of these mechanisms serves a useful purpose, the

policy points out that they are not a substitute for peer review because

they do not necessarily solicit the same unbiased, expert views that are

obtained through peer review.

EPA’s policy assigned responsibility to each Assistant and Regional

Administrator to develop standard operating procedures and to ensure

their use. To help facilitate consistent EPA-wide implementation, EPA’s

Science Policy Council—chaired by EPA’s Deputy Administrator—was

directed to help the offices and regions develop their procedures and

identify products that should be peer reviewed. The Council was also

given the responsibility for assessing agencywide progress and developing

any needed changes to the policy. However, the ultimate responsibility for

implementing the policy was placed with the Assistant and Regional

Administrators.

Implementation

Remains Uneven

We found that—2 years after EPA established its peer review policy—

implementation was still uneven. We concluded that EPA’s uneven

implementation was primarily due to (1) inadequate accountability and

oversight to ensure that all products are properly peer reviewed by

program and regional offices and (2) confusion among agency staff and

management about what peer review is, what its significance and benefits

are, and when and how it should be conducted.

According to the Executive Director of the Science Policy Council, the

unevenness could be attributed to a number of factors. First, while some

offices within EPA—such as the Office of Research and Development

(ORD)—have historically used peer review for many years, other program

offices and regions have had little prior experience. In addition, the

Director and other EPA officials told us that statutory and court-ordered

deadlines, budget constraints, and problems in finding and obtaining

qualified, independent peer reviewers also contributed to the problem.

Inadequate Oversight to

Ensure Consistent

Implementation

EPA’s oversight primarily consisted of a two-part reporting scheme that

called for each office and region to annually list (1) the candidate products

nominated for peer review during the upcoming year and (2) the status of

Page 3 GAO/T-RCED-97-95
the products previously nominated. If a candidate product was no longer

scheduled for peer review, the list had to note this and explain why peer

review was no longer planned.

Although we found this to be an adequate oversight tool for tracking the

status of previously nominated products, we pointed out that it does not

provide upper-level managers with sufficient information to ensure that all

products warranting peer review have been identified. This fact, together

with the misperceptions about what peer review is and the deadlines and

budget constraints that project officers often operate under, has meant

that the peer review program to date has largely been one of

self-identification, allowing some important work products to go unlisted.

According to the Science Policy Council, reviewing officials would be

much better positioned to determine if the peer review policy and

procedures are being properly and consistently implemented if, instead,

EPA’s list contained all major products along with what peer review is

planned and, if none, the reasons why not.

We noted that the need for more comprehensive oversight is especially

important given the policy’s wide latitude in allowing peer review to be

forgone in cases facing time and/or resource constraints. As explained by

the Executive Director of EPA’s Science Policy Council, because so much

of the work that EPA performs is in response to either statutory or

court-ordered mandates and the agency frequently faces budget

uncertainties or limitations, an office under pressure might argue for

nearly any given product that peer review is a luxury the office cannot

afford in the circumstances.

However, as the Executive Director of the Science Advisory Board (SAB)4

told us, not conducting peer review can sometimes be more costly to the

agency in terms of time and resources. He told us of a recent Office of

Solid Waste rulemaking concerning a new methodology for delisting

hazardous wastes in which the Office’s failure to have the methodology

appropriately peer reviewed resulted in important omissions, errors, and

flawed approaches in the methodology; these problems will now take from

1 to 2 years to correct. The SAB also noted that further peer review of the

individual elements of the proposed methodology is essential before the

scientific basis for this rulemaking can be established.

4

The SAB is a legislatively established body of independent experts that provides advice to the EPA

Administrator on scientific and engineering issues.

Page 4 GAO/T-RCED-97-95
Peer Review Policy and

Procedures Not Well

Understood

Although EPA’s policy and procedures provide substantial information

about what peer review entails, we found that some EPA staff and

managers had misperceptions about what peer review is, what its

significance and benefits are, and when and how it should be conducted.

Several cases we reviewed illustrate this lack of understanding about what

peer review entails. Officials from EPA’s Office of Mobile Sources (OMS)

told the House Commerce Committee in August 1995 that they had not had

any version of the mobile model5

 peer reviewed. Subsequently, in

April 1996, OMS officials told us they recognize that external peer review is

needed and that EPA planned to have the next iteration of the model so

reviewed.

We found similar misunderstandings in several other cases we reviewed.

EPA regional officials who produced a technical product that assessed the

environmental impacts of tributyl tin6

 told us that the contractor-prepared

product had been peer reviewed. While we found that the draft product

did receive some internal review by EPA staff and external review by

contributing authors, stakeholders, and the public, it was not reviewed by

experts independent of the product itself or of its potential regulatory

ramifications. When we pointed out that—according to EPA’s policy and

the region’s own peer review procedures—these reviews are not a

substitute for peer review, the project director said that she was not aware

of these requirements.

In two other cases we reviewed, there were misunderstandings about the

components of a product that should be peer reviewed. For example, in

the Great Waters study—an assessment of the impact of atmospheric

pollutants in significant water bodies—the scientific data were subjected

to external peer review, but the study’s conclusions that were based on

these data were not. Similarly, in the reassessment of dioxin—an

examination of the health risks posed by dioxin—the final chapter

summarizing and characterizing dioxin’s risks was not as thoroughly peer

reviewed.7

 In both cases, the project officers did not have the conclusions

peer reviewed because they believed that the development of conclusions

is an inherently governmental function that should be performed

5

The mobile model is one of the primary tools used by EPA, states, and localities to calculate the

estimated emissions reduction benefits of pollution control activities called for in state

implementation plans.

6

Tributyl tin is a compound that has been used since the 1960s in antifouling paints for boats and large

ships.

7

Although the entire product was reviewed by EPA’s Science Advisory Board, the Board expressed

dissatisfaction that the risk characterization chapter did not receive prior peer review.

Page 5 GAO/T-RCED-97-95
exclusively by EPA staff. However, some EPA officials with expertise in

conducting peer reviews disagreed, maintaining that it is important to have

peer reviewers comment on whether or not EPA has properly interpreted

the results of the underlying scientific and technical data. EPA’s quality

assurance requirements also state that conclusions should be peer

reviewed.8

EPA’s Actions to

Improve the Peer

Review Process

During our review, we found that EPA had recently taken a number of steps

to improve the peer review process. Although we believed that these steps

should prove helpful, we concluded that they did not fully address the

previously-discussed underlying problems and made some

recommendations for improvement. EPA agreed with our findings and

recommendations and has recently undertaken steps to implement them.

While it is too early to gauge the effectiveness of these efforts, we are

encouraged by the attention peer review is receiving by the agency’s

upper-level management.

EPA’s Past Efforts Near the completion of our review, in June 1996, EPA’s Deputy

Administrator directed the Science Policy Council’s Peer Review Advisory

Group and ORD’s National Center for Environmental Research and Quality

Assurance to develop an annual peer review self-assessment and

verification process to be conducted by each office and region. The

self-assessment was to include information on each peer review

completed during the prior year as well as feedback on the effectiveness of

the overall process. The verification would consist of the signature of

headquarters, laboratory, or regional directors to certify that the peer

reviews were conducted in accordance with the agency’s policy and

procedures. If the peer review did not fully conform to the policy, the

division director or the line manager must explain significant variances

and actions needed to limit future significant departures from the policy.

The self-assessments and verifications were to be submitted and reviewed

by the Peer Review Advisory Group to aid in its oversight responsibilities.

According to the Deputy Administrator, this expanded assessment and

verification process would help build accountability and demonstrate EPA’s

commitment to the independent review of the scientific analyses

underlying the agency’s decisions to protect public health and the

environment.

8

EPA Requirements for Quality Management Plans (EPA QA/R-2, Aug. 1994). This document

establishes the criteria and mandatory specifications for quality assurance and quality control

activities.

Page 6 GAO/T-RCED-97-95
During our review, we also found a number of efforts under way within

individual offices and regions to improve their implementation of peer

review. For example, the Office of Water drafted additional guidance to

further clarify the need for, use of, and ways to conduct peer review. The

Office of Solid Waste and Emergency Response formed a team to help

strengthen the office’s implementation of peer review by identifying ways

to facilitate good peer review and addressing barriers to its successful use.

Additionally, EPA’s Region 10 formed a Peer Review Group with the

responsibility for overseeing the region’s reviews.

We concluded that the above efforts should help address the problems we

found. However, we also concluded that the efforts aimed at improving the

oversight of peer review fell short by not ensuring that all relevant

products had been considered for peer review and did not require

documenting the reasons why products were not selected. Similarly, we

noted that the efforts aimed at better informing staff about the benefits

and use of peer review would be more effective if they were done

consistently throughout the agency.

EPA’s Current Plans EPA agreed with our findings and conclusions and has recently undertaken

a number of steps to implement our recommendations. On November 5,

1996, the Deputy Administrator asked ORD’s Assistant Administrator, in

consultation with the other Assistant Administrators, to develop proposals

to strengthen the peer review process. In response, ORD’s Assistant

Administrator proposed a three-pronged approach consisting of (1) audits

of a select number of work products to determine how well the peer

review policy was followed; (2) a series of interviews with office and

regional staff involved with peer review to determine the processes used

to implement the policy; and (3) training to educate and provide help to

individuals to improve the implementation of the peer review policy.

Significantly, the Deputy Administrator has echoed our message that EPA

needs to improve its oversight to ensure that all appropriate products are

peer reviewed. In a January 14, 1997, memorandum to the Assistant and

Regional Administrators, the Deputy stated, “I want you to ensure that

your lists of candidates for peer review are complete.” To help accomplish

this goal, each organization is directed to use, among other things, EPA’s

regulatory agenda and budget planning documents to help identify

potential candidates for peer review.

Page 7 GAO/T-RCED-97-95
While we agree that this should prove to be a useful tool, we continue to

encourage EPA to expand its existing candidate list to include all major

work products, along with explanations of why individual products are not

nominated for peer review. An all-inclusive list such as this will be

extremely useful to those overseeing the peer review process to determine

whether or not all products have been appropriately considered for peer

review.

In summary, peer review is critical for improving the quality of scientific

and technical products and for enhancing the credibility and acceptability

of EPA’s decisions that are based on these products. We are encouraged by

the renewed attention EPA is giving to improving the peer review process.

Although it is too early for us to gauge the success of these efforts, the

involvement of the agency’s upper-level management should go a long way

to ensure that the problems we identified are resolved. Mr. Chairman, this

concludes my prepared statement. I will be happy to respond to your

questions or the questions of Subcommittee members.

(160384) Page 8 GAO/T-RCED-97-95
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100