Peer Review: EPA's Implementation Remains Uneven
Released: 1997-03-11T00:00:00

Letter

Background

Peer review is well established as a mechanism for assuring the quality, credibility, and acceptability of individual and institutional work products. This assurance is accomplished by having the products undergo an objective, critical review by independent reviewers. Peer review has long been used by academia, professional organizations, industry, and government. Within EPA, peer review has taken many different forms, depending upon the nature of the work product, the relevant statutory requirements, and office-specific practices and needs.

In keeping with scientific custom and/or congressional mandates, several offices within EPA have used peer review for many years to enhance the quality of science within the agency. In response to a panel of outside academicians’ recommendations in 1992, EPA issued a policy statement in 1993 calling for peer review of the major scientific and technical work products used to support the agency’s rulemaking and other decisions. However, the Congress, GAO, and others subsequently raised concerns that the policy was not being implemented consistently across the agency. In response to these concerns, in 1994 EPA reaffirmed the central role that peer review plays in ensuring that the agency’s decisions are based on sound science and credible data and revised its 1993 policy.

The new policy, while retaining the essence of the prior one, was intended to expand and improve the use of peer review throughout EPA. The 1994 policy continued to stress that major products should normally be peer reviewed, but it also recognized that statutory and court-ordered deadlines, resource limitations, and other constraints might limit or even preclude the use of peer review. The policy applied to major work products that are primarily scientific or technical in nature and that may contribute to the basis for policy or regulatory decisions. In contrast, other products used in decision-making are not covered by the policy, nor are the ultimate decisions themselves. While peer review can take place at several different points along a product’s development, such as during the planning stage, it should be applied to a relatively well-developed product.

The 1994 policy also clarified that peer review is not the same thing as the peer input, stakeholders’ involvement, or public comment—mechanisms used by EPA to develop products, to obtain the views of interested and affected parties, and/or to build consensus among the regulated community. While each of these mechanisms serves a useful purpose, the policy points out that they are not a substitute for peer review because they do not necessarily solicit the same unbiased, expert views that are obtained through peer review.

EPA’s policy assigned responsibility to each Assistant and Regional Administrator to develop standard operating procedures and to ensure their use. To help facilitate consistent EPA-wide implementation, EPA’s Science Policy Council—chaired by EPA’s Deputy Administrator—was directed to help the offices and regions develop their procedures and identify products that should be peer reviewed. The Council was also given the responsibility for assessing agencywide progress and developing any needed changes to the policy. However, the ultimate responsibility for implementing the policy was placed with the Assistant and Regional Administrators.

Implementation Remains Uneven

We found that—2 years after EPA established its peer review policy— implementation was still uneven. We concluded that EPA’s uneven implementation was primarily due to (1) inadequate accountability and oversight to ensure that all products are properly peer reviewed by program and regional offices and (2) confusion among agency staff and management about what peer review is, what its significance and benefits are, and when and how it should be conducted.

According to the Executive Director of the Science Policy Council, the unevenness could be attributed to a number of factors. First, while some offices within EPA—such as the Office of Research and Development (ORD)—have historically used peer review for many years, other program offices and regions have had little prior experience. In addition, the Director and other EPA officials told us that statutory and court-ordered deadlines, budget constraints, and problems in finding and obtaining qualified, independent peer reviewers also contributed to the problem.

Inadequate Oversight to Ensure Consistent Implementation

EPA’s oversight primarily consisted of a two-part reporting scheme that called for each office and region to annually list (1) the candidate products nominated for peer review during the upcoming year and (2) the status of the products previously nominated. If a candidate product was no longer scheduled for peer review, the list had to note this and explain why peer review was no longer planned.

Although we found this to be an adequate oversight tool for tracking the status of previously nominated products, we pointed out that it does not provide upper-level managers with sufficient information to ensure that all products warranting peer review have been identified. This fact, together with the misperceptions about what peer review is and the deadlines and budget constraints that project officers often operate under, has meant that the peer review program to date has largely been one of self-identification, allowing some important work products to go unlisted. According to the Science Policy Council, reviewing officials would be much better positioned to determine if the peer review policy and procedures are being properly and consistently implemented if, instead, EPA’s list contained all major products along with what peer review is planned and, if none, the reasons why not.

We noted that the need for more comprehensive oversight is especially important given the policy’s wide latitude in allowing peer review to be forgone in cases facing time and/or resource constraints. As explained by the Executive Director of EPA’s Science Policy Council, because so much of the work that EPA performs is in response to either statutory or court-ordered mandates and the agency frequently faces budget uncertainties or limitations, an office under pressure might argue for nearly any given product that peer review is a luxury the office cannot afford in the circumstances.

However, as the Executive Director of the Science Advisory Board (SAB)told us, not conducting peer review can sometimes be more costly to the agency in terms of time and resources. He told us of a recent Office of Solid Waste rulemaking concerning a new methodology for delisting hazardous wastes in which the Office’s failure to have the methodology appropriately peer reviewed resulted in important omissions, errors, and flawed approaches in the methodology; these problems will now take from 1 to 2 years to correct. The SAB also noted that further peer review of the individual elements of the proposed methodology is essential before the scientific basis for this rulemaking can be established.

Peer Review Policy and Procedures Not Well Understood

Although EPA’s policy and procedures provide substantial information about what peer review entails, we found that some EPA staff and managers had misperceptions about what peer review is, what its significance and benefits are, and when and how it should be conducted. Several cases we reviewed illustrate this lack of understanding about what peer review entails. Officials from EPA’s Office of Mobile Sources (OMS) told the House Commerce Committee in August 1995 that they had not had any version of the mobile model peer reviewed. Subsequently, in April 1996, OMS officials told us they recognize that external peer review is needed and that EPA planned to have the next iteration of the model so reviewed.

We found similar misunderstandings in several other cases we reviewed. EPA regional officials who produced a technical product that assessed the environmental impacts of tributyl tin told us that the contractor-prepared product had been peer reviewed. While we found that the draft product did receive some internal review by EPA staff and external review by contributing authors, stakeholders, and the public, it was not reviewed by experts independent of the product itself or of its potential regulatory ramifications. When we pointed out that—according to EPA’s policy and the region’s own peer review procedures—these reviews are not a substitute for peer review, the project director said that she was not aware of these requirements.

In two other cases we reviewed, there were misunderstandings about the components of a product that should be peer reviewed. For example, in the Great Waters study—an assessment of the impact of atmospheric pollutants in significant water bodies—the scientific data were subjected to external peer review, but the study’s conclusions that were based on these data were not. Similarly, in the reassessment of dioxin—an examination of the health risks posed by dioxin—the final chapter summarizing and characterizing dioxin’s risks was not as thoroughly peer reviewed. In both cases, the project officers did not have the conclusions peer reviewed because they believed that the development of conclusions is an inherently governmental function that should be performed exclusively by EPA staff. However, some EPA officials with expertise in conducting peer reviews disagreed, maintaining that it is important to have peer reviewers comment on whether or not EPA has properly interpreted the results of the underlying scientific and technical data. EPA’s quality assurance requirements also state that conclusions should be peer reviewed.

EPA’s Actions to Improve the Peer Review Process

During our review, we found that EPA had recently taken a number of steps to improve the peer review process. Although we believed that these steps should prove helpful, we concluded that they did not fully address the previously-discussed underlying problems and made some recommendations for improvement. EPA agreed with our findings and recommendations and has recently undertaken steps to implement them. While it is too early to gauge the effectiveness of these efforts, we are encouraged by the attention peer review is receiving by the agency’s upper-level management.

EPA’s Past Efforts

Near the completion of our review, in June 1996, EPA’s Deputy Administrator directed the Science Policy Council’s Peer Review Advisory Group and ORD’s National Center for Environmental Research and Quality Assurance to develop an annual peer review self-assessment and verification process to be conducted by each office and region. The self-assessment was to include information on each peer review completed during the prior year as well as feedback on the effectiveness of the overall process. The verification would consist of the signature of headquarters, laboratory, or regional directors to certify that the peer reviews were conducted in accordance with the agency’s policy and procedures. If the peer review did not fully conform to the policy, the division director or the line manager must explain significant variances and actions needed to limit future significant departures from the policy. The self-assessments and verifications were to be submitted and reviewed by the Peer Review Advisory Group to aid in its oversight responsibilities. According to the Deputy Administrator, this expanded assessment and verification process would help build accountability and demonstrate EPA’s commitment to the independent review of the scientific analyses underlying the agency’s decisions to protect public health and the environment.

During our review, we also found a number of efforts under way within individual offices and regions to improve their implementation of peer review. For example, the Office of Water drafted additional guidance to further clarify the need for, use of, and ways to conduct peer review. The Office of Solid Waste and Emergency Response formed a team to help strengthen the office’s implementation of peer review by identifying ways to facilitate good peer review and addressing barriers to its successful use. Additionally, EPA’s Region 10 formed a Peer Review Group with the responsibility for overseeing the region’s reviews.

We concluded that the above efforts should help address the problems we found. However, we also concluded that the efforts aimed at improving the oversight of peer review fell short by not ensuring that all relevant products had been considered for peer review and did not require documenting the reasons why products were not selected. Similarly, we noted that the efforts aimed at better informing staff about the benefits and use of peer review would be more effective if they were done consistently throughout the agency.

EPA’s Current Plans

EPA agreed with our findings and conclusions and has recently undertaken a number of steps to implement our recommendations. On November 5, 1996, the Deputy Administrator asked ORD’s Assistant Administrator, in consultation with the other Assistant Administrators, to develop proposals to strengthen the peer review process. In response, ORD’s Assistant Administrator proposed a three-pronged approach consisting of (1) audits of a select number of work products to determine how well the peer review policy was followed; (2) a series of interviews with office and regional staff involved with peer review to determine the processes used to implement the policy; and (3) training to educate and provide help to individuals to improve the implementation of the peer review policy.

Significantly, the Deputy Administrator has echoed our message that EPA needs to improve its oversight to ensure that all appropriate products are peer reviewed. In a January 14, 1997, memorandum to the Assistant and Regional Administrators, the Deputy stated, “I want you to ensure that your lists of candidates for peer review are complete.” To help accomplish this goal, each organization is directed to use, among other things, EPA’s regulatory agenda and budget planning documents to help identify potential candidates for peer review.

While we agree that this should prove to be a useful tool, we continue to encourage EPA to expand its existing candidate list to include all major work products, along with explanations of why individual products are not nominated for peer review. An all-inclusive list such as this will be extremely useful to those overseeing the peer review process to determine whether or not all products have been appropriately considered for peer review.

In summary, peer review is critical for improving the quality of scientific and technical products and for enhancing the credibility and acceptability of EPA’s decisions that are based on these products. We are encouraged by the renewed attention EPA is giving to improving the peer review process. Although it is too early for us to gauge the success of these efforts, the involvement of the agency’s upper-level management should go a long way to ensure that the problems we identified are resolved. Mr. Chairman, this concludes my prepared statement. I will be happy to respond to your questions or the questions of Subcommittee members.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.