Peer Review: EPA's Implementation Remains Uneven
Published: 1996-09-24T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a congressional request, GAO reviewed the Environmental Protection Agency's (EPA): (1) progress in implementing its peer review policy; and (2) efforts to improve the peer review process.
section_title: What GAO Found
paragraphs: GAO found that: (1) although EPA has made progress in implementing its peer review policy, after nearly 2 years, implementation remains uneven; (2) while GAO found cases in which the peer review policy was followed, GAO also found cases in which important aspects of the policy were not followed or peer review was not conducted at all; (3) two primary reasons for this unevenness are: (a) confusion among agency staff and management about what peer review is, what its significance and benefits are, and how and when it should be conducted; and (b) inadequate accountability and oversight mechanisms to ensure that all relevant products are properly peer reviewed; (4) EPA officials readily acknowledged this uneven implementation and identified several of the agency's efforts to improve the peer review process; (5) because of concern about the effectiveness of the existing accountability and oversight mechanisms for ensuring proper peer review, EPA's Deputy Administrator recently established procedures to help build accountability and demonstrate EPA's commitment to the independent review of the scientific analyses underlying the agency's decisions; (6) these efforts are steps in the right direction; however, educating all staff about the merits of and procedures for conducting peer review would increase the likelihood that peer review is properly implemented agencywide; and (7) furthermore, by ensuring that all relevant products have been considered for peer review and that the reasons for those not selected have been documented, EPA's upper-level managers will have the necessary information to ensure that the policy is properly implemented.

section_title: Letter
paragraphs: Peer review is the critical evaluation of scientific and technical work products by independent experts. Within the Environmental Protection Agency (EPA), the agency’s senior leadership has stated that peer review is an important mechanism for enhancing the quality, credibility, and acceptability of products that may ultimately form the basis of regulations and other key decisions by the agency. Properly implemented, peer review can also conserve resources by steering product development along the most efficient, effective course, thereby avoiding costly and time-consuming delays. EPA’s current policy, issued in June 1994, expands on the agency’s prior policy and practices. The new policy continues to emphasize that major scientific and technical work products should normally be peer reviewed. In light of the key role that peer review plays in developing regulations, you asked us to assess EPA’s (1) progress in implementing its peer review policy and (2) efforts to improve the peer review process.
subsections: section_title: Background
paragraphs: In accordance with scientific custom and/or statutory mandates, several offices within EPA have used peer review for many years to enhance the quality of science within the agency. In May 1991, the EPA Administrator established a panel of outside academicians to, among other things, enhance the stature of science at EPA and determine how the agency can best ensure that sound science is the foundation for the agency’s regulatory and decision-making processes. In March 1992, the expert panelrecommended that, among other things, EPA establish a uniform peer review process for all scientific and technical products used to support EPA’s guidance and regulations. In response, EPA issued a policy statement in January 1993 calling for peer review of the major scientific and technical work products used to support the agency’s rulemaking and other decisions. However, the Congress, GAO, and others subsequently raised concerns that the policy was not being consistently implemented throughout EPA. The congressional concern resulted in several proposed pieces of legislation that included prescriptive requirements for peer reviews.
Subsequently, in June 1994 the EPA Administrator reaffirmed the central role of peer review in the agency’s efforts to ensure that its decisions rest on sound science and credible data by directing that the agency’s 1993 peer review policy be revised. The new policy retained the essence of the prior policy and was intended to expand and improve the use of peer review throughout EPA. Although the policy continued to emphasize that major scientific and technical products should normally be peer reviewed, it also recognized that statutory and court-ordered deadlines, resource constraints, and other constraints may limit or preclude the use of peer review. According to the Executive Director of the Science Policy Council, one of the most significant new features of the 1994 action was the Administrator’s directive to the agency’s Science Policy Council to organize and guide an agencywide program for implementing the policy.
The policy and procedures emphasize that peer review is not the same thing as other mechanisms that EPA often uses to obtain the views of interested and affected parties and/or to build consensus among the regulated community. More specifically, EPA’s policy and procedures state that peer review is not peer input, which is advice or assistance from experts during the development of a product; stakeholders’ involvement, which is comments from those people or organizations (stakeholders) that have significant financial, political, or other interests in the outcome of a rulemaking or other decision by EPA; or public comment, which is comments obtained from the general public on a proposed rulemaking and may or may not include the comments of independent experts.
While each of these activities serves a useful purpose, the policy and procedures point out that they are not a substitute for peer review. For example, as noted in EPA’s Standard Operating Procedures, public comments on a rulemaking do not necessarily solicit the same unbiased, expert views as are obtained through peer review.
In order to accommodate the differences in EPA’s program and regional offices, the policy assigned responsibility to each program and regional office to develop standard operating procedures and to ensure their use. To help facilitate agencywide implementation, EPA’s Science Policy Council was assigned the responsibility of assisting the offices and regions in developing their procedures and identifying products that should be considered for peer review. The Council was also given the responsibility for overseeing the agencywide implementation of the policy by promoting consistent interpretation, assessing agencywide progress, and developing revisions to the policy, if warranted. However, EPA’s policy specifies that the Assistant and Regional Administrators for each office are ultimately responsible for implementing the policy, including developing operating procedures, identifying work products subject to peer review, determining the type and timing of such reviews, and documenting the process and outcome of each peer review conducted.
Our objectives, scope, and methodology are fully described in appendix I.
section_title: Implementation Remains Uneven
paragraphs: Two years after EPA established its peer review policy, implementation is still uneven. EPA acknowledges this problem and provided us with a number of examples to illustrate the uneven implementation. At our request, the Science Policy Council obtained information from EPA program and regional offices and provided us with examples in which, in their opinion, peer review was properly conducted; cases in which it was conducted but not fully in accordance with the policy; and cases in which peer review was not conducted at all. The following table briefly summarizes the cases they selected; additional information on these nine cases is provided in appendix II.
According to the Executive Director of the Science Policy Council, this unevenness can be attributed to several factors. First, some offices within EPA have historically used peer review, while others’ experience is limited to the 2 years since the policy was issued. For example, in accordance with scientific custom, the Office of Research and Development (ORD) has used peer review for obtaining critical evaluations of certain work products for more than 20 years. Additionally, statutes require that certain work products developed by EPA be peer reviewed by legislatively established bodies. For example, criteria documents developed by ORD for the National Ambient Air Quality Standards must receive peer review from EPA’s Science Advisory Board (SAB), and pesticide documents must receive peer review from the Scientific Advisory Panel. In contrast, some EPA regional offices and areas within some EPA program offices have had little prior experience with peer review. In addition to these offices’ varying levels of experience with peer review, the Science Policy Council’s Executive Director and other EPA officials said that statutory and court-ordered deadlines, budget constraints, and difficulties associated with finding and obtaining the services of qualified, independent peer reviewers have also contributed to peer review not being consistently practiced agencywide. A report by the National Academy of Public Administration confirmed that EPA frequently faces court-ordered deadlines. According to the Academy, since 1993 the courts have issued an additional 131 deadlines that EPA must comply with or face judicial sanctions. Also, as explained to us by officials from EPA’s Office of Air and Radiation (OAR), just about everything EPA does in some program areas, such as Clean Air Act implementation, is to address either legislative or court-ordered mandates.
Others have attributed EPA’s problems with implementing peer review in the decision-making process to other factors. For example, in its March 1995 interim report on EPA’s research and peer review program within the Office of Research and Development, the National Academy of Sciences’ National Research Council noted that, even in EPA’s research community, knowledge about peer review could be improved. The Council’s interim report pointed out that “although peer review is widely used and highly regarded, it is poorly understood by many, and it has come under serious study only in recent years.” Although we agree that the issues EPA and others have raised may warrant further consideration, we believe that EPA’s uneven implementation is primarily due to (1) confusion among agency staff and management about what peer review is, what its significance and benefits are, and when and how it should be conducted and (2) ineffective accountability and oversight mechanisms to ensure that all products are properly peer reviewed by program and regional offices.
subsections: section_title: Peer Review Policy and Procedures Not Well Understood
paragraphs: Although the policy and procedures provide substantial information about what peer review entails, we found that some EPA staff and managers had misperceptions about what peer review is, what its significance and benefits are, and when and how it should be conducted. For example, officials from EPA’s Office of Mobile Sources (OMS) told the House Commerce Committee in August 1995 that they had not had any version of the mobile model peer reviewed. Subsequently, in April 1996, OMS officials told us they recognize that external peer review is needed and that EPA plans to have the next iteration of the model peer reviewed. However, when asked how the peer review would be conducted, OMS officials said they plan to use the public comments on the revised model they receive as the peer review. As EPA’s policy makes clear, public comments are not the same as nor are they a substitute for peer review.
We found a similar misunderstanding about what peer review entails in a regional office we visited. The region prepared a product that assesses the impacts of tributyl tin—a compound used since the 1960s in antifouling paints for boats and large ships. Although regional staff told us that this contractor-prepared product had been peer reviewed, we found that the reviews were not in accordance with EPA’s peer review policy. The draft product received some internal review by EPA staff and external review by contributing authors, stakeholders, and the public; however, it was not reviewed by experts previously uninvolved with the product’s development nor by those unaffected by its potential regulatory ramifications. When we pointed out that—according to EPA’s policy and the region’s own peer review procedures—these reviews are not a substitute for peer review, the project director said that she was not aware of these requirements.
In two other cases we reviewed, there was misunderstanding about the components of a product that should be peer reviewed. For example, in the Great Waters study—an assessment of the impact of atmospheric pollutants in significant water bodies—the scientific data were subjected to external peer review, but the study’s conclusions that were based on these data were not. Similarly, in the reassessment of dioxin—a reexamination of the health risks posed by dioxin—the final chapter summarizing and characterizing dioxin’s risks was not as thoroughly peer reviewed. More than any other, this chapter indicated EPA’s conclusions based on its reassessment of the dioxin issue. In both cases, the project officers did not have these chapters peer reviewed because they believed that the development of conclusions is an inherently governmental function that should be performed exclusively by EPA staff. However, some EPA officials with expertise in conducting peer reviews disagreed, maintaining that it is important to have peer reviewers comment on whether or not EPA has properly interpreted the results of the underlying scientific and technical data.
section_title: Inadequate Mechanisms to Ensure Consistent Implementation
paragraphs: In addition to the uncertainty surrounding the peer review policy, we also noted problems with EPA’s accountability and oversight mechanisms. EPA’s current oversight mechanism primarily consists of a two-part reporting scheme: Each office and region annually lists (1) the candidate products nominated for peer review during the upcoming year and (2) the status of products previously nominated. If a candidate product is no longer scheduled for peer review, the list must note this and explain why peer review is no longer planned. Agency officials said this was the most extensive level of oversight to which all program and regional offices could agree when the peer review procedures were developed.
Although this is an adequate oversight mechanism for tracking the status of previously nominated products, it does not provide upper-level managers with sufficient information to ensure that all products warranting peer review have been identified. This, when taken together with the misperceptions about what peer review is and with the deadlines and budget constraints that project officers often operate under, has meant that the peer review program to date has largely been one of self-identification, allowing some important work products to go unlisted. According to the Science Policy Council’s Executive Director, reviewing officials would be much better positioned to determine if the peer review policy and procedures are being properly and consistently implemented if, instead, EPA’s list contained all major products along with what peer review is planned and, if none, the reasons why not.
The need for more comprehensive accountability and oversight mechanisms is especially important given the policy’s wide latitude in allowing peer review to be forgone in cases facing time and/or resource constraints. As explained by EPA’s Science Policy Council’s Executive Director, because so much of the work that EPA performs is in response to either statutory or court-ordered mandates and the agency frequently faces budget uncertainties or limitations, an office under pressure might argue for nearly any given product that peer review is a luxury the office cannot afford in the circumstances.
However, as the Executive Director of EPA’s Science Advisory Board told us, not conducting peer review can sometimes be more costly to the agency in terms of time and resources. He told us of a recent rulemaking by the Office of Solid Waste concerning a new methodology for delisting hazardous wastes in which the program office’s failure to have the methodology appropriately peer reviewed resulted in important omissions, errors, and flawed approaches in the methodology, which will now take from 1 to 2 years to correct. The SAB also noted that further peer review of individual elements of the proposed methodology is essential before the scientific basis for this rulemaking can be established.
section_title: EPA’s Actions to Improve the Peer Review Process
paragraphs: EPA has recently taken a number of steps to improve the peer review process. Although these steps should prove helpful, they do not fully address the underlying problems discussed above.
subsections: section_title: Agencywide Efforts
paragraphs: In June 1996, EPA’s Deputy Administrator directed the Science Policy Council’s Peer Review Advisory Group and ORD’s National Center for Environmental Research and Quality Assurance to develop an annual peer review self-assessment and verification process to be conducted by each office and region. The self-assessment will include information on each peer review completed during the prior year as well as feedback on the effectiveness of the overall process. The verification will consist of the signature of headquarters, laboratory, or regional directors to certify that the peer reviews were conducted in accordance with the agency’s policy and procedures. If the peer review did not fully conform to the policy, the division director or the line manager will explain significant variances and actions needed to limit future significant departures from the policy. The self-assessments and verifications will be submitted and reviewed by the Peer Review Advisory Group to aid in its oversight responsibilities. According to the Deputy Administrator, this expanded assessment and verification process will help build accountability and demonstrate EPA’s commitment to the independent review of the scientific analyses underlying the agency’s decisions to protect public health and the environment. These new accountability and oversight processes should take full effect in October 1996.
ORD’s National Center for Environmental Research and Quality Assurance has also agreed to play an expanded assistance and oversight role in the peer review process. Although the details had not been completed, the Center’s Director told us that his staff will be available to assist others in conducting peer reviews and will try to anticipate and flag the problems that they observe. In addition, the Center recently developed an automated Peer Review Panelist Information System—a registry with information on identifying and contacting potential reviewers according to their areas of expertise. Although the system was designed to identify potential reviewers of applications for EPA grants, cooperative agreements, and fellowships, the Center’s Director stated that the registry (or similarly designed ones) could also be used to identify potential peer reviewers for EPA’s technical and scientific work products.
section_title: Office-Specific Efforts
paragraphs: Recognizing that confusion remains about what peer review entails, the Office of Water recently drafted additional guidance that further clarifies the need for, use of, and ways to conduct peer review. The Office has also asked the Water Environment Federation to examine its current peer review process and to provide recommendations on how to improve it. The Federation has identified the following areas of concern, among others, where the program should be improved: (1) the types of, levels of, and methodologies for peer review; (2) the sources and selection of reviewers; (3) the funding/resources for peer review; and (4) the follow-up to, and accountability for, peer review. Similarly, OAR’s Office of Mobile Sources proposed a Peer Review/Scientific Presence Team in March 1996 to help OMS personnel better understand the principles and definitions involved in the peer review process. In addition to promoting greater understanding, this team would also help identify products and plan for peer review, as well as facilitate and oversee the conduct of peer reviews for OMS’ scientific and technical work products.
The Office of Solid Waste and Emergency Response recently formed a team to support the Administrator’s goal of sound science through peer review. The team was charged with strengthening the program office’s implementation of peer review by identifying ways to facilitate good peer review and addressing barriers to its successful use. In May 1996, the team developed an implementation plan with a series of recommendations that fall into the following broad categories: (1) strengthening early peer review planning; (2) improving the ability of the Assistant Administrator to manage peer review activities; (3) providing guidance and examples to support the staff’s implementation of peer review; and (4) developing mechanisms to facilitate the conduct of peer reviews.
EPA’s Region 10 formed a Peer Review Group with the responsibility for overseeing the region’s reviews. In March 1996, the group had a meeting with the region’s senior management, where it was decided to later brief mid-level managers on the importance of peer review and their peer review responsibilities. Agreement was also reached to have each of the region’s offices appoint a peer review contact who will receive training from the Peer Review Group and be responsible for managing some peer reviews and for coordinating other major peer review projects.
The above agencywide and office-specific efforts should help address the confusion about peer review and the accountability and oversight problems we identified. However, the efforts aimed at better informing staff about the benefits and use of peer review are not being done fully in all offices and would be more effective if done consistently throughout the agency. Similarly, the efforts aimed at improving the accountability and oversight of peer review fall short in that they do not ensure that each office and region has considered all relevant products for peer review and that the reasons are documented when products are not selected.
section_title: Conclusions
paragraphs: Despite some progress, EPA’s implementation of its peer review policy remains uneven 2 years after it became effective. Confusion remains about what peer review entails and how it differs from the mechanisms that EPA uses to obtain the views of interested and affected parties. Furthermore, the agency’s accountability and oversight mechanism provides too much leeway for managers to opt out of conducting peer reviews without having to justify or document such decisions. The annual listing of only those products that have been selected for peer review has not enabled upper-level managers to see what products have not been nominated for peer review nor the reasons for their exclusion.
A more useful tool would be to have the list contain all planned major products with detailed information about the managers’ decisions about peer review. For example, if peer review is planned, the list would contain—as the current procedures already require—information on the type and timing of it. More significantly, if the managers elect to not conduct peer review on individual products, the list would provide an explanation of why the products are not being nominated. This process would provide upper-level managers with the necessary information to determine whether or not all products have been appropriately considered for peer review.
We acknowledge that there are other difficulties in properly conducting peer reviews. However, we believe that as EPA strengthens the implementation of its peer review policy and gains more widespread experience with the process, the agency will be better positioned to address these other issues.
section_title: Recommendations
paragraphs: To enhance the quality and credibility of its decision-making through the more widespread and consistent implementation of its peer review policy, we recommend that the Administrator, EPA, do the following:
Ensure that staff and managers are educated about the need for and benefits of peer review; the difference between peer review and other forms of comments, such as peer input, stakeholders’ involvement, and public comment; and their specific responsibilities in implementing the policy.
Expand the current list of products nominated for peer review to include all major products, along with explanations of why individual products are not nominated for peer review.
section_title: Agency Comments
paragraphs: We provided copies of a draft of this report to the Administrator of EPA for review and comment. In responding to the draft, EPA officials stated that the report was clear, instructive, and fair. The officials also provided us with some technical and presentational comments that we have incorporated as appropriate.
We conducted our review from February 1996 through August 1996 in accordance with generally accepted government auditing standards. A detailed discussion of our scope and methodology appears in appendix I.
As arranged with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 15 days after the date of this letter. At that time, we will send copies to the Administrator of EPA and other interested parties. We will also make copies available to others upon request.
Please call me at (202) 512-6111 if you or your staff have any questions. Major contributors to this report are listed in appendix III.
section_title: Objectives, Scope, and Methodology
paragraphs: The Chairmen of the Senate Small Business Committee; the Subcommittee on Clean Air, Wetlands, Private Property, and Nuclear Safety, Senate Committee on Environment and Public Works; and the Subcommittee on Energy Production and Regulation, Senate Committee on Energy and Natural Resources, asked us to assess the Environmental Protection Agency’s (EPA) (1) progress in implementing its peer review policy and (2) efforts to improve the peer review process.
To assess the status of EPA’s implementation of its peer review policy, we reviewed relevant documents and discussed the agency’s use of peer review with officials from EPA’s Science Policy Council; Office of Air and Radiation (Washington, DC, Durham, NC, and Ann Arbor, MI); Office of Water; Office of Program Planning and Evaluation; Office of Solid Waste and Emergency Response; and Office of Prevention, Pesticides, and Toxic Substances (Washington, DC); Office of Research and Development (Washington, DC and Research Triangle Park, NC); and EPA Region 10 (Seattle, WA).
We also interviewed and obtained documents from officials with the National Academy of Sciences; the Water Environment Federation; the National Environmental Policy Institute; and the Natural Resources Defense Council.
We reviewed a selection of scientific and technical products to obtain examples of how EPA’s program and regional offices were implementing the peer review policy. We asked officials from EPA’s Science Policy Council and Science Advisory Board to identify products that, in their opinion, fell into the following categories: (1) those that fully complied with the policy; (2) those that received some level of peer review but did not fully comply with the policy; and (3) those that should have received but did not receive peer review. We then interviewed the officials responsible for the products to determine how decisions were made about the products’ peer review.
To assess EPA’s efforts to improve the peer review process, we reviewed relevant documents and discussed the agency’s recent, ongoing, and planned improvements with officials from EPA’s Science Policy Council; Science Advisory Board; and the program and regional offices identified above.
We conducted our review from February though August 1996 in accordance with generally accepted government auditing standards.
section_title: Peer Review Activities for Nine Scientific and Technical Work Products
paragraphs: At our request, the Science Policy Council obtained information from EPA program and regional offices and provided us with examples illustrating the current uneven implementation of EPA’s peer review policy. This list was further augmented by the Executive Director of the Science Advisory Board. Although these products are not necessarily a representative sample, the Executive Director of EPA’s Science Policy Council stated that these cases provide good illustrations of how the level of peer review within EPA remains uneven. We have grouped the cases below according to whether (1) EPA’s peer review policy was followed, (2) the policy was not fully followed, or (3) a peer review was not conducted but should have been.
subsections: section_title: Peer Review Policy Followed
subsections: section_title: Eastern Columbia Plateau Aquifer System
paragraphs: In January 1993, EPA Region 10 received a petition from a local environmental group to designate the Eastern Columbia Plateau Aquifer System as a “Sole-Source Aquifer” under the Safe Drinking Water Act. The technical work product was entitled Support Document for Sole Source Aquifer Designation of the Eastern Columbia Plateau Aquifer System. Under the act, EPA may make this designation if it determines that the aquifer is the principal or sole source for the area’s drinking water. Once so designated, EPA would then review federally assisted projects in the area to determine if these activities could contaminate the aquifer.
In August 1994, EPA prepared a draft document that presented the technical basis for the designation. Technical questions were raised by commentors that prompted EPA to convene a panel of experts to review the document. The panel was given a list of specific technical issues to address, the draft document, and the supporting materials. The peer review panel convened July 26-27, 1995, to discuss their views.
The peer reviewers were chosen by asking several “stakeholder” organizations, including local governments, an environmental organization, and the United States Geological Survey, to nominate respected scientists with expertise in areas such as hydrogeology. From more than 15 nominees, a selection committee of EPA staff from outside Region 10 chose 6 peer review panel members. Although one stakeholder group expressed dissatisfaction that their candidate was not chosen for the panel, they eventually agreed that the panel fairly and objectively reviewed the support document.
In July 1995, EPA received the peer review panel’s report and is still in the process of responding to the panel’s comments and those received from the public.
section_title: Waste Technologies Industries Incinerator
paragraphs: Waste Technologies Industries (WTI) began limited operation of a hazardous waste incinerator in East Liverpool, Ohio, in April 1993. Although permitted for operation under the Clean Air Act, the Clean Water Act, and the Resource Conservation and Recovery Act, the facility became the focus of national attention and controversy due to several concerns. For example, it was being built near populated areas and an elementary school, and the public was skeptical about industries’ management of commercial incinerators, the ability of government agencies to regulate them, and whether the existing laws and regulations are sufficient to protect public health and the environment. The WTI site was chosen, in part, because of its proximity to steel mills, chemical plants, and other industries generating hazardous waste suitable for incineration. When fully operational, this site will incinerate over 100,000 tons of hazardous wastes annually.
The original permit for WTI had been based solely on the modeled effects of direct inhalation exposures and had not included other exposure scenarios, such as indirect exposure through the food chain. Because of such risk assessment omissions and the controversy associated with the facility, EPA decided to conduct an on-site risk assessment of the cumulative human health and ecological risks associated with the operations of this facility, as well as such risks from accidents at the facility, and to publish its findings prior to the full operation of the WTI site.
According to the Senior Science Advisor for the Office of Solid Waste and Emergency Response, peer review was envisioned early in the process and occurred at several stages, including peer review of the agency’s approach to addressing these issues and peer review of the entire report, including the conclusions and recommendations. She also said that about $120,000, or nearly 20 percent of all extramural funds that EPA spent on this over 3-year effort, went to cover peer review costs.
section_title: Peer Review Policy Not Fully Followed
subsections: section_title: Dioxin Reassessment
paragraphs: EPA began to assess the risks of dioxin in the early 1980s, resulting in a 1985 risk assessment that classified the chemical as a probable human carcinogen, primarily on the basis of animal studies available at that time. The implications of additional advances in the early 1990s were uncertain: some maintained that dioxin’s risks were not as great as earlier believed, while others made the opposite argument. Given the growing controversy, in April 1991 EPA decided to work closely with the broader scientific community to reassess the full range of dioxin risks. The draft product, which was released for public comment in September 1994, contained an exposure document and a health effects document. The last chapter of the health effects document characterized the risks posed from dioxin by integrating the findings of the other chapters. “The importance of this . . . demands that the highest standards of peer review extend to the risk characterization itself. Although it can be argued that this is in fact being carried out by this SAB  Committee, submitting the risk characterization chapter for external peer review prior to final review by the SAB would serve to strengthen the document, and assure a greater likelihood of its acceptance by the scientific community-at-large. It is recommended strongly that: a) the risk characterization chapter undergo major revision; and b) the revised document be peer reviewed by a group of preeminent scientists, including some researchers from outside the dioxin “community” before returning to the SAB.”
Members of Congress also criticized EPA’s risk characterization document and its lack of peer review. In the House and Senate reports on the fiscal year 1996 appropriations bill for EPA, concerns were raised that the draft document “does not accurately reflect the science on exposures to dioxins and their potential health effects . . . EPA selected and presented scientific data and interpretations . . . dependent upon assumptions and hypotheses that deserve careful scrutiny . . . and inaccuracies and omissions . . . were the result of the Agency’s failure to consult with and utilize the assistance of the outside scientific community . . .”
The committees directed EPA to respond to the SAB’s concerns and consult with scientists in other agencies in rewriting the risk characterization chapter. The House committee also restricted EPA from developing any new rules that raise or lower dioxin limits on the basis of the risk reassessment.
As of July 1996, EPA was in the process of responding to the committees’, SAB’s, and the public’s comments. The risk characterization chapter is being subjected to a major revision and will be peer reviewed by external scientific experts prior to referral back to the SAB. The SAB will then be asked to evaluate EPA’s response to their suggestions and the adequacy of the additional peer review conducted on the draft report.
section_title: Great Waters Program
paragraphs: Section 112(m) of the Clean Air Act Amendments of 1990 required EPA to determine if atmospheric inputs of pollutants into the Great Waters warrants further reductions of atmospheric releases and to report the agency’s findings to the Congress 3 years after the act’s enactment. The Great Waters program includes the Great Lakes, Lake Champlain, Chesapeake Bay, and the coastal waters. EPA made its first report to the Congress in May 1994.
The scientific and technical data in this report, Deposition of Air Pollutants to the Great Waters: First Report to Congress, were peer reviewed by 63 reviewers. The reviewers represented a number of different perspectives, including academia, industry, environmental groups, EPA offices, other federal and state agencies, and Canadian entities. According to the Great Waters Program Coordinator, the reviewers were given copies of all the report chapters, except the conclusions and recommendation chapter, so that they could prepare for a peer review workshop. The reviewers then met to discuss the report and provide EPA with their views.
EPA expended a great deal of effort to ensure that the science in the report was peer reviewed; however, the program coordinator said the agency did not have the conclusions and recommendations chapter peer reviewed. The decision not to peer review this chapter was based on the belief by those directing the program that these were the agency’s opinions based on the information presented and thus an inherently governmental function not subject to peer review. However, others within EPA believe that nothing should be withheld from peer review and said that the conclusions should have been peer reviewed to ensure that they were indeed consistent with the scientific content.
section_title: Unit Pricing Report
paragraphs: Residential unit pricing programs involve charging households according to the amount, or number of units, of garbage that they produce. In accordance with the principle that the polluter pays, unit pricing provides a financial incentive for reducing municipal waste generation and enhancing recycling. EPA’s Office of Policy, Planning and Evaluation (OPPE) used a cooperative agreement to have an assessment prepared of the most significant literature on unit pricing programs to determine the degree to which unit pricing programs meet their stated goals. The paper, which was completed in March 1996, highlights those areas where analysts generally agree on the outcomes associated with unit pricing, as well as those areas where substantial controversy remains. Unit pricing is still voluntary in the United States, according to the project officer; however, he said EPA believes that the more information that municipalities have readily available as they make long-term solid waste landfill decisions, the more likely these local governments are to employ some form of unit pricing as a disincentive to the continued unrestrained filling of landfills.
The OPPE project director had the report internally peer reviewed by three EPA staff knowledgeable about unit pricing. The report was not externally peer reviewed, he said, because it is designed to be used only as a reference guide by communities that are considering implementing some type of unit pricing program to reduce waste, and because EPA does not intend to use the report to support any regulatory actions.
section_title: Technical Analysis of AJ Mine’s Tailing Impoundment
paragraphs: The Alaska Juneau (AJ) Gold Mine project was a proposal by the Echo Bay, Alaska, company to reopen the former mine near Juneau. The proposal entailed mining approximately 22,500 tons of ore per day and, after crushing and grinding the ore, recovering gold through the froth flotation and carbon-in-leach (also called cyanide leach) processes. After the destruction of residual cyanide, the mine tailings would be discharged in a slurry form to an impoundment that would be created in Sheep Creek Valley, four miles south of downtown Juneau.
An environmental impact statement was prepared on the proposal in 1992. Because the project would require permits for fill materials and discharging wastewater into surface waters, EPA’s regional staff developed a model to predict the environmental ramifications of the proposal. According to regional staff, a careful analysis of the proposal was important because the issues in this proposal could potentially set a precedent for similar future proposals.
EPA went through three iterations of the model. The first model was presented in a report entitled A Simple Model for Metals in the Proposed AJ Mine Tailings Pond. The report was reviewed by an engineer in EPA’s Environmental Research Laboratory and a firm that worked for the City and Borough of Juneau. The second model was a customized version of one developed by EPA’s Research Laboratory. After receiving comments from the firm representing Echo Bay, ORD laboratories, the Corps of Engineers, and others, EPA decided to also use another model to evaluate the proposal’s potential environmental effects. In 1994, EPA prepared a technical analysis report on the proposal. The report received peer review by several of the same individuals who commented on the models, as well as others. Although the reviewers had expertise in the subject matter, several were not independent of the product’s development or its regulatory and/or financial ramifications.
Based partially on the model’s predictions, it became evident that EPA would withhold permit approval for the project. Accordingly, Echo Bay developed an alternative design for its project. In May 1995, EPA hired a contractor to prepare a supplemental environmental impact statement that will assess the revised project’s ecological effects. The agency plans to have the impact statement peer reviewed.
section_title: Methodology for Establishing Hazardous Waste Exit Criteria
paragraphs: Under the Resource Conservation and Recovery Act (RCRA), EPA is not only responsible for controlling hazardous wastes but also for establishing procedures for determining when hazardous wastes are no longer a health and/or ecological concern. As such, EPA’s Office of Solid Waste (OSW) developed a new methodology for establishing the conditions under which wastes listed as hazardous may be delisted. This methodology was presented in an OSW report, Development of Human Health Based and Ecologically Based Exit Criteria for the Hazardous Waste Identification Project (March 3, 1995), which was intended to support the Hazardous Waste Identification Rule. The intent of this rule is to establish human health-based and ecologically based waste constituent concentrations—known as exit criteria—for constituents in wastes below which listed hazardous wastes would be reclassified and become delisted as a hazardous waste. Such wastes could then be handled as a nonhazardous solid waste under other provisions of RCRA. OSW’s support document describes a proposed methodology for calculating the exit concentrations of 192 chemicals for humans and about 50 chemicals of ecological concern for five types of hazardous waste sources; numerous release, transport, and exposure pathways; and for biological effects information. “The Subcommittee is seriously concerned about the level of scientific input and the degree of professional judgment that, to date, have been incorporated into the methodology development. It was clear to the Subcommittee that there has been inadequate attention given to the state-of-the-science for human and ecological risk assessment that exists within EPA, let alone in the broader scientific community, in the development of the overall methodology, the identification of individual equations and associated parameters, the selection of models and their applicability, and the continual need for sound scientific judgment.”
The SAB also noted that further peer review of individual elements of the proposed methodology is essential before the scientific basis can be established. The SAB concluded that the methodology at present lacks the scientific defensibility for its intended regulatory use.
According to SAB’s Executive Director, this is a case where the program office’s decision to not conduct a peer review of the key supporting elements of a larger project resulted in extra cost and time to the agency, as well as missed deadlines. He pointed out that the experience on this one effort had now, he believed, caused a cultural change in the Office of Solid Waste, to the extent that they now plan to have peer consultation with the SAB on several upcoming lines of effort.
section_title: Peer Review Not Conducted
subsections: section_title: Mobile 5a Model
paragraphs: Mobile 5A, also known as the mobile source emissions factor model, is a computer program that estimates the emissions of hydrocarbons, carbon monoxide, and nitrogen oxide for eight different types of gasoline-fueled and diesel highway motor vehicles. The first mobile model, made available for use in 1978, provided emissions estimates only for tailpipe exhaust emissions from passenger cars. Since that time, major updates and improvements to the mobile model have resulted in the addition of emissions estimates for evaporative (nontailpipe exhaust) emissions and for uncorrected in-use deterioration due to tampering or poor maintenance, according to the OMS Emission Inventory Group Manager. Also, other categories of vehicles, such as light-duty trucks and motorcycles, have been added over the years, she said.
The development of the next generation model, Mobile 6, is currently under way. As with other models, the mobile model exists because precise information about the emissions behavior of the approximately 200 million vehicles in use in the United States is not known, according to the Group Manager. The primary use of the mobile model is in calculating the estimated emissions reductions benefits of various actions when applied to the mobile sources in an area. For example, the mobile model can estimate the impact of participating in a reformulated gasoline program, or of using oxygenated fuels in an area, or of requiring periodic inspection and maintenance of selected vehicle categories. In essence, the mobile model is one of the primary tools that EPA, states, and localities use to measure the estimated emissions reduction effectiveness of the pollution control activities called for in State Implementation Plans.
None of the previous mobile models has been peer reviewed. However, EPA has obtained external views on the model through stakeholders’ workshops and experts’ meetings; one of the largest of these meetings involved over 200 stakeholders, according to OMS officials. The agency recognizes that these workshops and meetings are not a substitute for peer review and, in a reversal of the agency’s views of 10 months ago, EPA now plans to have Mobile 6 peer reviewed, they said. Several constraints, such as the limited number of unbiased experts available to do peer review in some fields and the resources for compensating reviewers, still have to be overcome, they added.
section_title: Tributyl Tin Impacts
paragraphs: Tributyl tin (TBT) is a compound used since the 1960s as an antifouling ingredient for marine paints. In the 1970s, antifouling paints were found to adversely affect the environment. Although restrictions were placed on TBT by the United States and a number of other countries in the 1980s, elevated levels of TBT continue to be found in marine ecosystems. In light of the uncertain human health and environmental effects of TBT, an interagency group consisting of EPA Region 10 officials, the Washington State Departments of Ecology and Natural Resources, the National Oceanographic and Atmospheric Administration, the U.S. Army Corps of Engineers, and others was formed to derive a marine/estuarine sediment effects-based cleanup level (or screening level) for TBT. In April 1996, a contractor-prepared report was issued with recommended screening levels; EPA regional staff served as the project managers and made significant contributions to the revisions to and final production of the report.
Although an EPA project manager maintains that the report was peer reviewed, the reviews did not meet the requirements of EPA’s peer review policy nor the region’s standard operating procedures for conducting peer reviews. While the report was reviewed by members of the interagency group, other experts who provided input to the report, the affected regulated community, and the general public, there was not an independent review by experts not associated with preparing the report or by those without a stake in its conclusions and recommendations. When we explained to the project manager why EPA’s Science Policy Council characterized the report as not having received peer review, the project manager acknowledged that she was not familiar with either EPA’s peer review policy or the region’s standard operating procedures. EPA is currently in the process of responding to the comments it has received.
section_title: Major Contributors to This Report
subsections: section_title: Resources, Community, and Economic Development Division, Washington, D.C.
section_title: Norfolk Regional Office
paragraphs: James R. Beusse, Senior Evaluator Philip L. Bartholomew, Staff Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.
U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000  or by using fax number (301) 258-4066, or TDD (301) 413-0006.
Each day, GAO issues a list of newly available reports and testimony.  To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone.  A recorded menu will provide information on how to obtain these lists.