Peer Review: OMB's Proposed, Revised, and Final Bulletins
Released: 2005-02-03T00:00:00

Peer Review and Regulatory Policy

There is no official, government-wide definition of "peer review," even in the OMBbulletins. (5) Agency-specific definitions and descriptions of the term differ, but all of them contain the basicconcept of a review of scientific or technical merit by individuals with both sufficient technicalcompetence and no unresolved conflicts of interest. In its final bulletin, OMB states that peer review"typically evaluates the clarity of hypotheses, the validity of research design, the quality of the datacollection procedures, the robustness of the methods employed, the appropriateness of the methodsfor the hypotheses being tested, the extent to which the conclusions follow from the analysis, andthe strengths and limitations of the overall product." (6)

Support for Peer Review

Some type of peer review has been used for centuries within the scientific community tojudge the quality of science. Peer review can take many different forms, and is used for a variety ofpurposes. For example, it is used commonly by federal agencies to evaluate research proposals, andplays a major role in funding decisions. In these cases it is often part of a broader category ofevaluation known as merit review. Peer review is also the usual method by which the editors ofscientific publications evaluate proposed research reports. It even plays an important role in manyresearch institutions, including many government agencies, in decisions about retention andpromotion for individual scientists and in reviewing research programs. A February 1999 reportfrom the Committee on Science, Engineering, and Public Policy concluded that expert review (themost common form of which is peer review) is the most effective means of evaluating federallyfunded research programs. (7)

Peer review is also used for scientific and technical products relating to policies, includingregulations, to determine whether the underlying scientific findings are well supported. Forexample, such peer review was established as Environmental Protection Agency (EPA) policy in1993. (8) In 1996, a panelof leading economists concluded that peer review should be used for economic analyses supportingregulations with a potentially large impact on the economy (e.g., those whose annual economic costsexceed $1 billion). (9) Thepanel also indicated that reviewers should be selected based on their demonstrated expertise andreputations.

In its peer review bulletins, OMB recognized the variety of ways that peer review is used.OMB said that independent peer review is especially important in the regulatory arena becausefederal agencies often develop or fund the science that underlies their regulations, and then overseethe peer review of those scientific studies -- thereby creating the appearance of a conflict of interest.

Legislative Proposals. A number of statutes andlegislative proposals have advocated the use of peer review in a regulatory context for particularissues. For example, the Safe Drinking Water Act (42 U.S.C. 300g-1(b)(3)(A)) requires EPA, whentaking action under the act based on science, to use the "best available, peer reviewed science andsupporting studies." In the 108th Congress, the Water Resources Development Act of 2003( H.R. 2557 ) would have generally required project studies to be subject to peer reviewby an independent panel of experts if the project has an estimated total cost of more than $50 million. Similarly, the Sound Science for Endangered Species Act Planning Act of 2003 ( H.R. 1662 , 108th Congress) would have directed the Secretary of the Interior, in making decisions aboutspecies protection, to give greater weight to certain kinds of data that had been peer reviewed byqualified individuals as defined in the bill.

There have also been legislative efforts to require peer review more broadly. For example,in the 106th Congress, the Senate considered but did not enact bipartisan legislation ( S. 746 ) that would have required virtually all agencies to provide for an independent peer review of anyrequired risk assessments and cost-benefit analyses of major rules that the agencies or the OMBDirector reasonably anticipated were likely to have a $500 million effect on the economy. The billwould have required that peer reviews be conducted through panels that were "broadlyrepresentative" and involved participants with relevant expertise who were "independent of theagency." However, if an agency certified that adequate peer review had already been conducted, andthe OMB Director agreed, no further peer review would have been required. In its comments on thislegislation, the General Accounting Office (GAO) (now the Government Accountability Office)generally supported the use of peer review in this context, noting that "the rigorous, independentreview of economic analyses should help enhance the quality, credibility, and acceptability ofagencies' decisionmaking." (10) However, GAO cautioned that (given the number of reviewscontemplated) agencies would need to plan carefully for the reviews, and that the panels would needto reflect all points of view.

Presidential Support. Recent presidentialadministrations have also supported peer review as a preferred means of assessing scientific research,both prospectively and retrospectively. For example, beginning with the FY1996 budget cycle,OMB and OSTP have jointly provided annual direction to agencies, encouraging them to emphasizethe funding of peer-reviewed research over nonpeer-reviewed research for most scientific activities. In 1997, the then-Administrator of OMB's Office of Information and Regulatory Affairs (OIRA)testified that the Clinton Administration supported peer review, but also said the administrationrecognized that it is not cost-free in terms of agencies' resources or time. (11)

On September 20, 2001, the new OIRA Administrator for the George W. BushAdministration issued a memorandum for the President's Management Council recommending(among other things) that agencies subject regulatory impact analyses and supporting technicaldocuments for "economically significant" and "major" rules (e.g., those with a $100 million annualeffect on the economy) to independent, external peer review. (12) The OIRA Administratoralso recommended certain criteria for peer review (e.g., disclosure by peer reviewers of priortechnical or policy positions on the issues at hand and their sources of personal and institutionalfunding), and said OIRA would give agency analyses that had undergone such a review "a measureof deference" during its reviews of their regulatory proposals under Executive Order 12866. (13)

Cautionary Notes About Peer Review

Although the concept of peer review has generally received wide support, some observershave also raised cautionary notes or mentioned certain limitations to the approach in particularsituations. For example, as OMB indicated in its 1997 testimony, peer review costs can besignificant for agencies in terms of both time and agency resources. (14) To pay for peer reviewprocedures, agencies may have to divert resources from other areas (e.g., regulatory enforcement orstandards development). Scientists from academic institutions who perform peer reviews on avoluntary basis may also incur opportunity costs with respect to other activities such as teaching andresearch. Other concerns have focused on how peer reviews have been implemented. For example,GAO noted that peer review methods varied within and among federal agencies, (15) and that agencies'economic analyses of major rules were often not peer reviewed. (16) A number of observershave expressed concerns about the impact of peer review requirements on the pace of regulatoryactivity, with additional requirements exacerbating what is already often regarded as an "ossified"rulemaking process. (17) In fact, some critics have suggested that regulatory relief and delay is the primary purpose of peerreview proposals in this context.

Still other concerns about peer review have centered on issues of bias and balance. Expertsagree that effective peer review panels must be (and must be perceived to be) free from anysignificant conflict of interest and properly balanced, allowing for a wide range of views andappropriate expertise. However, in June 2001, GAO reported that the policies and proceduresdeveloped by EPA's Science Advisory Board to ensure balance and independence of the Board's peerreviewers had limitations that reduced their effectiveness. (18) For example, GAO said that the staff office did not routinelyensure that panel members' financial disclosures were complete or that they contained enoughinformation to determine whether a conflict of interest existed. Also, panel members reportedly didnot have to disclose information regarding previous positions on the matter being reviewed until thepanel's first meeting, thereby making it difficult to determine the independence and balance of thepanel members until after they had been selected. GAO also reported that the panel's policies andprocedures did not adequately inform the public about the points of view represented on the panels. GAO made several recommendations in this report that were designed to better ensure that theScience Advisory Board's peer review panels are independent and balanced.

Procedural Flexibility or Uniformity. As notedpreviously, GAO reported that federal agencies' peer review practices are not consistent. (19) However, several studiesof peer review in the federal government have suggested that rigidly uniform peer review proceduresmay not be desirable. For example, a 1996 National Science and Technology Council reportindicated that peer review implementation should be flexible and "appropriate to the nature ofscientific processes." (20) Similarly, in its 1999 report on peer review, GAO reported that OSTP believed agencies' peer reviewpractices should be "flexible and tailored to agency missions and type of research, and that specificuniform practices should not be dictated for every agency or all federally funded research." (21) GAO also said thatagencies viewed a variety of peer review methods as "both appropriate and essential, reflecting thevarying nature of the research and its purposes, the differences in research timelines, the broadspectrum of [research and development] performers, and the varying funding mechanisms, such asgrants, contracts, and cooperative agreements." Notably, although GAO pointed out that peer reviewpractices varied across and within agencies, it did not recommend greater uniformity in thosepractices.

Peer Review and the Information Quality Act

OMB said that the final peer review bulletin was being issued "under the Information QualityAct [IQA] and OMB's general authorities to oversee the quality of agency information, analyses, andregulatory actions." (22) The IQA (also known as the "Data Quality Act") was enacted in December 2000 as Section 515 ofthe Treasury and General Government Appropriations Act for Fiscal Year 2001 ( P.L. 106-554 ). Theact supplemented the Paperwork Reduction Act (PRA), which already required OMB to "developand oversee the implementation of policies, principles, standards, and guidelines to apply to federalagency dissemination of public information." (23) The PRA also required agencies to manage their informationresources to "improve the integrity, quality, and utility of information to all users within and outsidethe agency." (24)

Although little noticed at the time it was enacted, the IQA has subsequently been the subjectof significant debate and controversy. (25) The act required OMB to issue guidance to federal agenciesdesigned to ensure the "quality, objectivity, utility, and integrity" of information disseminated to thepublic. It also required agencies to issue their own information quality guidelines, and to establishadministrative mechanisms that allow affected persons to seek correction of information maintainedand disseminated by the agencies that does not comply with the OMB guidance. However, it did notrequire either OMB or the agencies to issue peer review bulletins.

OMB published its final government-wide IQA guidelines in the Federal Register inFebruary 2002. (26) Inthem, OMB encouraged (but did not require) the use of peer reviews in the development ofagency-disseminated information. Specifically, OMB indicated that agencies can presume that dataare sufficiently "objective" if they have been subject to an independent peer review process (e.g., asused by scientific journals), but said a member of the public could rebut this presumption "based ona persuasive showing by the petitioner in a particular instance." OMB also indicated that journalpeer review may not be sufficient for information likely to have an important public policy or privatesector impact. Finally, the IQA guidelines set minimum standards for the transparency ofagency-sponsored peer review (referencing the standards in the above-mentioned September 20,2001, memorandum issued by the OIRA Administrator -- e.g., disclosure of reviewers' priortechnical or policy positions and sources of funding).

The final peer review bulletin also cited Executive Order 12866 and OMB's authority to"manage the agencies under the purview of the President's Constitutional authority to supervise theunitary Executive Branch" as other authorities supporting the bulletin's issuance. (27) E.O. 12866 says thatOIRA within OMB is the "repository of expertise concerning regulatory issues," and says thatagencies should base their decisions on "the best reasonably obtainable scientific, technical,economic, or other information concerning the need for, and consequences of the intendedregulation." (28) However, the executive order does not specifically mention peer review.

OMB's Proposed Peer Review Bulletin

When it published its proposed peer review bulletin in the Federal Register in September2003, OMB noted in the preamble to the bulletin the importance of peer review and the sometimesinconsistent nature of its application. For these reasons, the office proposed requiring peer reviewfor "the most important scientific and technical information relevant to regulatory policies that[agencies] disseminate to the public." (29) OMB said the new requirements would supplement, but notsupplant, both its own information quality guidelines and any agency-specific peer reviewrequirements for regulatory information. Although OMB noted that its bulletins "are not laws orbinding regulations," as a practical matter, OMB's instructions to the agencies carry substantialweight because of its authority to review agencies' budget proposals and most significant regulationsbefore they are published.

In essence, OMB's bulletin proposed that each covered federal agency take three actions: (1)have all "significant regulatory information" that the agencies intend to disseminate peer reviewed;(2) have "especially significant regulatory information" peer reviewed according to even higherstandards; and (3) provide OMB with information at least once each year about upcoming scientificstudies that might support significant regulatory disseminations, and the agencies' plans forconducting peer reviews of those studies. The proposed bulletin also contained several other, relatedrequirements. The bulletin defined "agency" broadly, including cabinet departments and agencies,independent agencies (e.g., EPA), and independent regulatory agencies (e.g., the FederalCommunications Commission and the Nuclear Regulatory Commission).

Peer Review of Significant Regulatory Information

The proposed bulletin defined "significant regulatory information" as scientific or technicalinformation that qualifies as "influential" under OMB's information quality guidelines and is"relevant to regulatory policies." OMB's information quality guidelines define "influentialinformation" as information that the agency can "reasonably determine" will have a "clear andsubstantial impact on important public policies or important private sector decisions" when it isdisseminated to the public. The proposed peer review bulletin said information is "relevant toregulatory policies" if it "might be used by local, state, regional, federal and/or internationalregulatory bodies." The bulletin indicated that the peer review for this type of information shouldbe "appropriate" and "scientifically rigorous," but also said the appropriate peer review mechanismwould depend on "the novelty and complexity of the science to be reviewed, the benefit and costimplications, and any controversy regarding the science."

For information at the lower end of those dimensions, the proposed bulletin indicated thata review by qualified specialists within a separate part of the agency disseminating the informationcould satisfy the peer review requirement. Also (as in the information quality guidelines), agencieswere allowed to presume that any studies that had already been independently peer reviewed did nothave to be reexamined, but the bulletin said this presumption "is rebuttable based on a persuasiveshowing in a particular instance." Finally, the bulletin said peer review was not needed for significantregulatory information related to national defense or foreign affairs, or information that isdisseminated in the course of an individual agency adjudication or proceeding on a permitapplication.

Review of Especially Significant Regulatory Information

The proposed bulletin indicated that regulatory information should be considered "especiallysignificant" (and therefore subject to more specific peer review requirements) if (a) the agencyintends to disseminate it in support of an economically significant regulatory action (e.g., with a$100 million annual impact on the economy), (30) (b) its dissemination would otherwise have a clear and substantialeffect on important public policies or important private sector decisions (with an impact of $100million in any year), or (c) OMB determines it is of "significant interagency interest" or "is relevantto an Administration policy priority." The bulletin laid out a series of proposed requirements forthese types of peer reviews. For example, it said agencies must

select peer reviewers primarily based on their scientific and technical expertiseand "strive" to appoint reviewers who are independent of the agency and do not have any "real orperceived conflicts of interest" (e.g., have no financial interest in the issue, do not seek or receivesubstantial funding from the agency, and have not conducted a peer review for the agency oradvocated a position in recent years on the specific topic being examined). If obtaining the requisiteexpertise requires the selection of a panel member with a perceived bias, the bulletin said agenciesmust "ensure that another reviewer with a contrary bias is appointed to balance the panel";

provide peer reviewers with an explicit statement describing the purpose andscope of the review, and sufficient information to understand the data, methods, and conclusions ofthe material being reviewed;

permit other interested agencies and persons to submit comments and providethose comments to peer reviewers for their consideration; and

direct peer reviewers to issue a final report describing the nature of theirreview and their findings and conclusions, including any dissenting opinions. The report must alsodescribe the names, affiliations, and qualifications of all peer reviewers and any prior affiliationswith the agency.

Also, agencies were instructed to consult with OMB and OSTP regarding the "sufficiency" of theirplanned peer review policies and, on request, to discuss with OMB the sufficiency of their plans toreview specific documents.

OMB Notification of Upcoming Peer Reviews

The proposed bulletin indicated that, at least once each year, each agency must provide OMBwith a summary description of any "existing, ongoing, or contemplated" studies that might supportsignificant regulatory information that the agency intends to disseminate within the next year. Eachagency was also to provide OMB with the agency's plan for conducting peer reviews of such studiesunder the bulletin's requirements, including an agency contact person. OMB indicated that, to reducepaperwork, this information was to be included in other reports being submitted to OMB underExecutive Order 12866 or the Information Quality Act.

Other Provisions

The proposed bulletin also contained several other peer review requirements or provisions,including the following:

Agencies that are likely to disseminate "significant" or "especially significant"regulatory information were required to supplement or amend their information quality guidelinesto incorporate the requirements of the proposed peer review bulletin for "significant" and "especiallysignificant" information. These agencies' supplements or amendments were also to address (a) whatfactors could preclude an individual's participation as a peer reviewer, (b) how confidential businessinformation and other confidential information would be protected, and (c) what information aboutpeer reviewers should be disclosed (e.g., sources of funding that could suggest a conflict ofinterest).

OMB could waive the requirements for peer review if an agency made "acompelling case" that a waiver was necessary (e.g., an imminent health hazard or homeland securitythreat).

Agencies were permitted to retain an outside firm to oversee the peer reviewprocess to select and supervise the peer review panels, which the bulletin indicated would allow theagencies to avoid the requirements of the Federal Advisory Committee Act (FACA). (31)

OMB initially said these peer review requirements would apply to information disseminated on orafter January 1, 2004. OMB initially requested public comments on the proposed bulletin byOctober 28, 2003, but later extended the comment period to December 15, 2003, and then to January16, 2004.

Comments on the Proposed Peer Review Bulletin and OMB's Responses

OMB said that it received 187 comments on the proposed bulletin on peer review, and postedthose comments on its website. (32) In addition to this formal commenting process, OMB and OSTPsponsored an open workshop on the peer review requirements in November 2003. (33) The workshop was hostedby the National Academy of Sciences (NAS), which subsequently endorsed the bulletin's objectivesbut expressed several concerns. OMB said the comments that it received varied substantially, withsome highly favorable and others suggesting that the bulletin be withdrawn and reconsidered. Ingeneral, supporters of the proposed bulletin reportedly indicated that peer review standards acrossthe government were inconsistent, and that more consistent use of peer review could increase thetechnical quality and credibility of regulatory science. They also reportedly asserted that peer reviewcould protect science-based regulations from political criticism and litigation. For example, the U.S.Chamber of Commerce said that OMB had "provided a Bulletin of high quality and soundjudgment," and strongly supported "OMB's efforts to improve the quality, objectivity, utility, andintegrity of information disseminated by the federal government to the public." (34) Similarly, the NationalPetrochemical & Refiners Association strongly supported OMB's proposed bulletin, saying that itwould "correct the current variability in federal agency implementation of peer reviews by providingthe needed guidance on process requirements." (35)

OMB said that many critics of the proposed bulletin viewed it as an effort to inject politicalconsiderations into the world of science, and to use the uncertainty that inevitably surrounds scienceas an excuse to delay new rules that could be costly to regulated entities. They also reportedly saidthat the bulletin appeared more concerned about peer reviewers' possible conflicts of interest withagencies than their potential conflicts with regulated industries.

In its summary of the public comments, OMB identified a number of broad themes and notedhow it responded to those comments. (36) Specifically, OMB said the comments centered on such issuesas the need for the bulletin, its scope and applicability, the prescriptive nature of the review processfor "especially significant" information, and the costs of implementing the bulletin.

Need for the Bulletin

OMB said many commenters indicated that it was unclear why the bulletin was needed. Forexample, the American Public Health Association said it was "unaware of any evidence that thecurrent system is not working or any examples of inappropriate or flawed federal regulations beingpromulgated as a result of failure to peer review." (37) In response, OMB said it would provide additional informationin the revised bulletin on the importance of peer review and the variability of agencies' peer reviewpolicies.

Clarifications of Scope and Applicability

A number of commenters reportedly said the scope and applicability of the bulletin wasunclear.

Some commenters questioned the focus on "regulatory" information, and saidit would be difficult to know in advance whether a particular study might be used in support of futureregulatory action. In its response to the comments, OMB said it would revise the bulletin to focuson "influential" scientific information instead of the potential regulatory impact of the informationdisseminated. As noted previously, the IQA guidelines state that information should be considered"influential" if "the agency can reasonably determine that dissemination of the information will haveor does have a clear and substantial impact on important public policies or important privatedecisions." (38)

Some commenters expressed concern that the criteria for requiring the morestringent peer review requirements for highly significant information were too broad. In response,OMB said it would narrow the applicability of those requirements to cover only "scientificassessments" that either have a $500 million annual effect on important public policies or privatesector decisions or involve novel, complex, or precedent-setting approaches or significantinteragency interest.

OMB said it agreed with comments questioning the applicability of thebulletin's requirements to agencies' regulatory impact analyses, and said it would more clearly statethat those analyses would not be covered by the revised bulletin. (39) However, OMB went onto say that the bulletin's requirements would cover the models and data underlying those analyses,and encouraged agencies to have their regulatory impact analyses peer reviewed by "governmentanalysts."

OMB also said it would clarify that the bulletin did not apply to productsreleased by government-funded scientists that were not represented as the views of the agencysupporting the research.

Finally, in response to several concerns, OMB said it would exempt from thebulletin's coverage any time-sensitive medical, public health, and safety disseminations, and allowagency heads to waive or defer the bulletin's requirements "where warranted by a compellingrationale."

Agency Discretion and Peer Review Procedures

Commenters also said that the proposed bulletin was too prescriptive in the proceduresapplicable to "especially significant" regulatory information, and said extensive peer reviews wouldbe costly and could lead to delays in regulatory decisions. In response, OMB said that it would giveagencies additional discretion in the revised bulletin to determine the level of peer review requiredfor any particular document, and to use alternative procedures if they could demonstrate that thoseprocedures would meet the requirements in the government-wide and agency-specific IQAguidelines. The response included significant discussion of the current and potential role of NASin conducting peer reviews. In its comments, NAS had pointed to the role its committees haveplayed in performing peer reviews of various agency programs, activities, policies, and proposals. In determining the appropriate level of review, OMB said agencies should consider the costs andbenefits of peer review. OMB also emphasized that its role is not to veto studies, but rather to ensurethat they have the appropriate level of review before implementation.

Selection of Reviewers

OMB also said it was incorporating suggestions regarding the selection of peer reviewers,including that agencies solicit reviewers from the public and from scientific and professionalsocieties. One major concern of the commenters was that the criteria outlined in the proposedbulletin (e.g., discouraging participation of reviewers who were seeking or receiving funding viaresearch grants from an agency) could result in the most qualified reviewers being excluded fromconsideration. OMB also said that the revised bulletin would make clear that the receipt of researchgrants through peer-reviewed competitions does not disqualify a potential reviewer.

Public Participation

According to OMB, many of the commenters raised concerns about public participation inthe peer review process, questioning (among other things) whether that participation would be usefuland raising logistical considerations. As a result, OMB said it had decided to allow the agencies todecide whether public participation would be allowed and in what forms. However, OMB said itwould provide for public comment on the agencies' peer review plans for upcoming documents.

Cost of Implementing the Bulletin

Finally, OMB said some of the commenters requested that OMB provide an estimate of thecosts associated with implementation of the peer review bulletin. For example, in a March 2004letter to OMB, Senators Joseph Lieberman and Richard Durbin said it was "curious that OIRA -- anoffice known for insisting on cost-benefit analysis -- provides no estimate of costs, nordocumentation of the benefits gained or specific harms that will be reduced by the proposal. Wesuggest that the proposed rules do not pass OMB's own test of good rulemaking." (40)

OMB estimated that there would be about 125 "influential" scientific documents coveredeach year that are related to rulemaking, and at least 1,250 other documents each year that areunrelated to rulemaking. OMB said some of these documents may be exempt or otherwise notrequire peer reviews (e.g., they had already been adequately peer reviewed), but did not quantify thatassertion. OMB said that a "high-quality individual letter review" would cost an agency about $5,000per document. Therefore, assuming that 1,000 of the 1,250 documents required peer review andcould be reviewed by letter, OMB said the cumulative cost of these reviews would be $5 million.

OMB said "one or two dozen" of these documents could be considered "highly influentialscientific assessments" requiring more intense peer review. OMB estimated that an average peerreview of these documents would cost $50,000. Therefore, if 20 of these "highly influential"assessments were peer reviewed, the cost would be $1 million. OMB said these costs are "likelylarge overestimates of the incremental costs of the Bulletin because many agencies already engagein peer review practices." The response also included an analysis that provided a methodology foragencies to use in deciding when a more expensive peer-review study by NAS would becost-effective. (41)

Other Comments

Several other types of comments were offered on the proposed guidelines that were notincluded in OMB's summary. For example:

OMB Watch questioned OMB's authority to issue a bulletin on peer review,noting that the PRA and the IQA do not explicitly provide the office with that responsibility, andthat Congress had previously considered and rejected several proposals to mandate peer review. OMB Watch said that if OMB believes government-wide peer review requirements are needed, "itshould instead submit its ideas to Congress for consideration and possible authorization." (42)

The American Bar Association (ABA), noting that the proposed bulletinrequired peer review whenever the impact of the regulatory action would have exceeded $100million, recommended that OMB limit the bulletin's categorical requirement for peer review to"those situations in which the scientific information is at least arguably complex, novel, orcontroversial." (43) TheABA said peer review "is simply not the correct mechanism to address significant use of routine,established, or accepted scientific information."

The ABA also said it agreed that one factor in determining whether anindividual should be a peer reviewer is whether that individual is receiving or seeking substantialfunding from the agency involved. However, the ABA also said that an issue of equal concern iswhether a potential reviewer is receiving or seeking funding from a private company or industry orcitizen group with a stake in the information involved.

The American Association for the Advancement of Science (AAAS) said itwas concerned that one of the bulletin's factors in selecting reviewers -- whether the prospectivereviewer had advocated a position on the matter at issue -- could exclude a qualified scientist whohad simply made statements based on a "preponderance of peer reviewed scientific evidence" (e.g.,that particulate matter of a certain size had been shown to cause asthma in children). (44) AAAS said suchstatements should not be considered "advocacy." The organization also said that OMB should makeclear that individual comments in agencies' peer review reports would not be associated withindividual reviewers. In March 2004, the AAAS Council adopted a resolution strongly opposing theproposed bulletin and requesting that it be withdrawn. (45)

The AEI-Brookings Joint Center for Regulatory Studies recommended (amongother things) that OMB or some other entity conduct an evaluation of the peer review program aftera certain amount of time "to determine whether peer review actually led to a material improvementin the quality of the regulatory analysis." (46) AEI-Brookings said if OMB could not develop a reasonableapproach to evaluation, "it probably should not proceed with this effort."

OMB's Revised Peer Review Bulletin

The comments that OMB received on the proposed bulletin led the office to release what itdescribed as a "substantially revised" bulletin in April 2004. The changes were characterized in themedia as a "partial retreat" in OMB's initiative. (47) OMB said the revised bulletin provided a more extensivediscussion of why government-wide peer review guidance was needed, gave agencies morediscretion in determining the appropriate type of peer review for specific products, and more clearlyindicated that the guidance does not create any new rights for litigation. Also, the revised bulletinpermitted an agency (under certain circumstances) to use a scientist as a peer reviewer who had received research grants from the agency -- a practice that was discouraged in the proposed bulletin. The table appended to this report provides a side-by-side comparison of the proposed and revisedbulletins. The sections below discuss similarities and differences in the documents.

Changes in Scope and Application

Among the more notable changes made to the bulletin were revisions to the scope andapplicability of the peer review requirements. In general, the scope of the revised bulletin wasarguably much broader than the proposed bulletin. For example, whereas the proposed bulletinapplied to "significant regulatory information," the revised bulletin applied to "influential scientific information" -- a scope that included, but was not limited to, regulatory information. (48) However, the revisedbulletin also made it clear that certain types of information within this broad category did not haveto be peer reviewed, including:

information distributed within the government or to contractors orgrantees;

responses to requests under the Freedom of Information Act (FOIA), thePrivacy Act, or FACA; (49)

correspondence with individuals;

press releases;

and distributions made to peer reviewers under the bulletin, provided theycontain a disclaimer. (50)

The revised bulletin also explicitly exempted from its requirements information that (1) wasrelated to national security, foreign affairs, or international trade or treaties where compliance withthe bulletin "would interfere with the need for secrecy or promptness"; (51) (2) was produced bygovernment-funded scientists that was not represented as the views of the agency; (3) was an agencyregulatory impact analysis or regulatory flexibility analysis under Executive Order 12866; (52) or (4) was medical, health,or safety information that was time sensitive or based on a recent clinical trial that had already beenadequately peer reviewed. The bulletin also allowed agency heads to waive or defer the bulletinrequirements "where warranted by a compelling rationale." (53)

In structure, the revised peer review bulletin was similar to the proposed bulletin in that itstill essentially required agencies to take three actions (to the extent permitted by law):

have a peer review conducted on all "influential scientific information" thatthe agency intends to disseminate,

have all "highly influential scientific assessments" peer reviewed according tomore specific and demanding standards, and

indicate what "influential" and "highly influential" information the agencyplans to peer review in the future.

Each of these required actions is described more fully in the sections below.

Influential Scientific Information

The revised peer review bulletin defined the term "influential scientific information" asinformation the agency "reasonably can determine that the dissemination of which will have or doeshave a clear and substantial impact on important public policies or private sector decisions." Therefore, in contrast to the proposed bulletin, the peer review requirements in the revised bulletinwere not limited to regulatory policies.

In addition to the limitations and exemptions discussed above, the revised bulletin said thatagencies were also not required to conduct a peer review of influential information that had alreadyhad an "adequate" peer review. The bulletin gave agencies substantial discretion in determiningwhether a prior review was "adequate," specifically stating that the earlier review "need not complywith all of the requirements of this bulletin." It also said that the sufficiency of the prior reviewdepended on the "novelty and complexity of the science to be reviewed and the benefit and costimplications." Language in the proposed bulletin about rebutting an agency's presumption regardingwhether earlier peer reviews were sufficient was not included in the revised bulletin.

The revised bulletin (like the proposed bulletin) also gave agencies discretion in determiningthe appropriate peer review mechanism for any information not previously reviewed, with the levelof rigor required again dependent on the "novelty and complexity of the science to be reviewed andthe benefit and cost implications." The bulletin stated that "appropriate" reviews could range fromexaminations by other federal specialists to formal panels composed of independent experts. Thepreamble to the bulletin indicated that some peer reviews could be done by letter (e.g., when a draftdocument covers only one discipline), but said panels were preferable when time and resourcespermitted their use. Peer reviewers were to be selected based on their relevant expertise, and werenot to have participated in developing the product being reviewed. The revised bulletin alsocontained requirements to handle participation by scientists with conflicts of interest, (54) and said that the commentsof peer reviewers or a summary of those comments should be made available to the public.

When Peer Review Is Required. In response toconcerns that the trigger for peer reviews was too vague, OMB indicated in the revised bulletin thatthe reviews were needed whenever an agency "reasonably determines" that scientific information"will have or does have a clear and substantial impact on important public policies or private sectordecisions."

This change did not appear to address one of the underlying concerns voiced in the commentson the proposed bulletin. The definition of "influential" information is essentially the same as theone used to define "significant" information in the proposed bulletin; the only difference is thebroader applicability to non-regulatory information. Therefore, agencies still may find it difficultto determine whether a particular study will, at some point in the future, have a "clear and substantialimpact on important public policies," much less "important private sector decisions." Two agenciesin analogous situations may reach different conclusions regarding the need for peer review. As aresult, the overall purpose of the guidelines -- a more consistent application of peer review amongfederal agencies -- may not be achieved.

Highly Influential Scientific Assessments

The revised bulletin contained a number of additional peer review requirements for"scientific assessments" that are "highly influential." These requirements were much more specificthan those placed on "influential" scientific assessments, differed in some respects from therequirements in the proposed bulletin, and appeared to give OMB and the agencies significantauthority to determine when the requirements are applicable.

When Enhanced Peer Reviews Are Required. The revised bulletin defined a "scientific assessment" as "an evaluation of a body of scientific ortechnical knowledge which typically synthesizes multiple factual inputs, data, models, assumptionsand/or applies best professional judgement to bridge uncertainties in the available information." Theterm "highly influential scientific assessment" was defined in the revised bulletin as "influential"scientific information (as defined in the previous section of this report) that either the disseminatingagency or OMB determined

met the above-mentioned definition of a "scientific assessment";and

could have a "clear and substantial impact on important public policies ... orprivate sector decisions with a potential effect of more than $500 million in any year," or involved"precedent setting, novel, and complex approaches, or significant interagencyinterest."

As discussed previously, in the proposed bulletin enhanced peer review requirements appliedto "especially significant" influential information expected to be disseminated in support of aregulatory action that could have a $100 million impact in any year. Therefore, although the dollarthreshold for when enhanced peer review is required was higher in the revised bulletin ($500 millioninstead of $100 million), the peer review requirements were more broadly applicable in that theyapplied to more than just regulations. Also, the language regarding "precedent setting, novel, andcomplex approaches" replaced criteria in the proposed bulletin in which OMB alone could decidethat certain information required enhanced peer review because it was of "significant interagencyinterest" or "relevant to an Administration policy priority."

Selection of Reviewers. The revised bulletindescribed in some detail how peer reviewers of highly influential scientific assessments should beselected, but still gave agencies substantial discretion in making the final decision. For example, thebulletin said that reviewers must be selected primarily on the basis of "necessary" expertise,experience, and skills, and must be diverse enough to "fairly" represent different perspectives. Therevised bulletin also said that agencies must consider requesting the public to nominate reviewers,and must generally "bar participation by scientists employed by the agency sponsoring the review." (As noted previously, the proposed bulletin said agencies should "strive" to select peer reviewerswho were independent of the agency.) In addition, as in the proposed bulletin, the revised bulletinsaid agencies should try to avoid repeated use of the same reviewer on multiple assignments.

However, in a significant departure from the proposed bulletin, the revised bulletin also saidthat research grants awarded to a scientist through a competitive, peer-reviewed process did notdisqualify that scientist from serving on a peer review panel. (The proposed bulletin indicated thatpeer reviewers who were "receiving or seeking substantial funding from the agency through acontract or research grant" should not be selected as peer reviewers.) Also, whereas the proposedbulletin advocated balancing panel members' biases, the revised bulletin contained no language tothat effect. Finally, the revised bulletin said that agencies could commission entities independent ofthe agency to select peer reviewers and manage the process.

Public Comments. The revised bulletin also gaveagencies discretion in the area of public comments. As noted previously, the proposed peer reviewbulletin instructed agencies to permit interested agencies and persons to submit comments on theinformation being reviewed, and to provide those comments to the peer reviewers for theirconsideration. In contrast, the revised bulletin said that agencies could decide on their own whetherto make the information at issue available for public comment, but said agencies should "consider"having a public comment period and public meetings before the reviewers "when there is sufficientpublic interest." The revised bulletin did not indicate how agencies are to make this determination,or how much public interest is "sufficient" to trigger the need for public comments. The bulletin alsosaid that if an agency decides to make a draft assessment available to the public, it must, "wheneverpracticable," provide peer reviewers with a summary or compilation of the comments received.

Other Requirements. The revised bulletincontained several other types of requirements that were also in the proposed bulletin. For example,the revised bulletin required agencies to provide reviewers with "sufficient information" to allowthem to understand the data, assumptions, and analytic procedures used to support key findings orconclusions. Also, agencies were required to instruct peer reviewers to prepare a report summarizingtheir findings and conclusions. The report must contain the names, affiliations, and credentials ofthe reviewers. Agencies were, in turn, required to prepare a written response to the report, and boththe report and the response must be posted on the agency's website.

Alternative Procedures. The revised peer reviewbulletin said that an agency can (1) rely on scientific information produced by NAS, (2) commissionNAS to peer review the information, or (3) use an alternative procedure "specifically approved bythe [OIRA] Administrator, in consultation with OSTP." As noted previously, the bulletin permittedthe agency head to waive or defer some or all of the peer review requirements for "influentialscientific assessments" and "highly influential scientific assessments" "where warranted by acompelling rationale." (55) If the requirements are waived, however, the bulletin said peer review should be conducted "as soonas practicable thereafter." The proposed bulletin did not provide for alternative procedures.

OMB Authority. As is apparent from thepreceding discussion, the revised peer review bulletin (like the proposed bulletin) gave OMBsignificant authority to decide when agencies should use the more rigorous form of peer review. Specifically, the revised bulletin indicated that these procedures should be used when OMB (or theagency) determines that a scientific assessment (1) "could have a clear and substantial impact onimportant public policies ... or private sector decisions with a potential effect of more than $500million in any year," or (2) involves "precedent setting, novel, and complex approaches, orsignificant interagency interest." Therefore, under this definition, OMB could unilaterally determinethat certain information that an agency intends to disseminate is sufficiently "novel" or "complex"to trigger these "highly influential" additional peer review requirements. (56)

Also, whereas the revised bulletin defined "influential"scientific information as informationthat the agency can reasonable determine "will have or does have" a clear and substantial impact onimportant public policies or private sector decisions, the bulletin said that information should beconsidered "highly influential" if either the agency or OMB determines it "could" have such animpact -- conceptually, a lower threshold than for information that is only "influential."

OMB retained significant authority over the peer review process in other ways. For example,OMB could determine that an agency's alternative peer review procedures are sufficiently rigorousto replace the requirements in the revised bulletin.

Upcoming Peer Reviews

The proposed bulletin required agencies to notify OMB at least once each year of any existingor upcoming studies that might trigger the peer review requirements within the next year, and of theagencies' plans for conducting those peer reviews. In contrast, the revised bulletin required eachcovered agency to post an "agenda" on its website every six months delineating any informationdisseminations subject to peer review. The revised bulletin was also much more specific about thesenotices than the proposed bulletin, requiring each entry on the agenda to contain a detaileddescription of the "peer review plan," including:

the title, subject and purpose of the plan, with an agencycontact,

how the review will be conducted (panel or individualletters),

the anticipated number of reviewers, and

a "succinct description of the primary disciplines or types of expertise neededin the review."

Each peer review plan was also required to describe how reviewers will be selected, whether thepublic will be asked to nominate peer reviewers, whether there will be opportunities for the publicto comment on the information being reviewed, and whether the agency will provide reviewers withcopies of any public comments before beginning their work. The revised bulletin required agenciesto establish a mechanism to allow the public to comment on the adequacy of the peer review plans,and must consider those comments.

Comments on the Revised Bulletin

OMB said that 57 individuals and organizations commented on the April 2004 revised peerreview bulletin. (57) About a dozen of the comments essentially asked OMB to allow more time for public comments andinteragency review. Most of the substantive comments provided were supportive of at least someof the changes that OMB made to the bulletin. However, some commenters believed the changeshad weakened the bulletin to such an extent that they withdrew their initial support, while othersbelieved the changes had not gone far enough. Some of the individuals' and organizations'comments are summarized below.

National Academy of Sciences

The presidents of the National Academy of Sciences and its two affiliated institutions, theNational Academy of Engineering and the Institute of Medicine, released a statement on April 15,2004 that generally praised the revision, arguing that it would "improve the quality of thegovernment's scientific assessments and ... decision-making," and "better accommodate the diversecircumstances of ... federal agencies." (58) The statement suggested, but did not state, that furtherimprovements were nevertheless desirable. (59)

OMB Watch

In May 2004, OMB Watch filed a comment on the revised peer review bulletin, stating thatalthough OMB had made "significant revisions" to the bulletin, it nevertheless had "fail[ed] tocorrect some of the most fundamental complaints. (60) Specifically, OMB Watch said that:

OMB lacked the authority to issue the guidelines, given (1) the lack of specificauthority to do so in the statutes and orders cited and (2) that Congress had not enacted legislationdesigned to require peer review.

OMB had not established that there was a government-wide peer review"problem" in need of a government-wide solution.

OMB retained too much control over federal peer review practices (e.g., indetermining when certain information requires a more rigorous peer review and in approvingalternative peer review plans), and recommends that some of those responsibilities be given to theNational Academy of Sciences.

the "independence" standard in the bulletin would diminish the ability ofgovernment scientists to participate as peer reviewers, while not affecting scientists fromindustry.

the revised bulletin would likely significantly delay the regulatory process atagencies by adding "several new layers of bureaucracy" (e.g., public comments on peer reviews andpeer review plans).

In its conclusion, OMB Watch recommended that OMB withdraw the bulletin and instead convenean interagency committee to investigate peer review techniques and flaws, or make changes in thebulletin to address their concerns.

Chamber of Commerce

The U.S. Chamber of Commerce submitted comments on the revised bulletin in May 2004,stating that the revised bulletin has "fundamental problems that must be resolved," and that it couldnot support the issuance of a final bulletin until those problems are addressed. (61) The Chamber said it had"deep concerns" about (among other things):

the blanket exemption given to NAS peer reviews;

the "excessive" discretion given to agencies in the implementation of thebulletin;

the absence of "provisions allowing affected parties to contest any agencydetermination of applicability, peer review type, panel selection, charge, or other peer reviewprogram element as it applies to a specific case;" and

insufficient limitations on the use of proprietary information andmodels.

In summary, the Chamber said that the changes OMB made in issuing the revised bulletin "are sosevere and debilitating as to eliminate the public benefit of having a common, government-wideminimum standard for peer review."

Members of Congress

In May 2004, twelve Members of Congress provided OMB with comments on the revisedbulletin, stating that the revision did not address previously expressed concerns that the proposal was"unjustified, overly broad, burdensome, and did not appropriately guard against appointment ofreviewers with conflicts of interest," or that it provided OMB with "excessive authority over theproduction and dissemination of government information. (62) The Members said thatthe need for the bulletin remained unjustified, that the exemptions in the revised bulletin resulted inan unbalanced approach, and that it created "considerable new burdens on agencies." They also saidthat "safeguards to ensure the integrity of the process for selection of reviewers have not beenincluded in the Bulletin." In particular, they said there was "no requirement that agencies disclosepublicly whether any panel members with a conflict were selected." In conclusion, the Memberscharacterized the bulletin as "a wolf in sheep's clothing," and said it was "designed to prevent thedissemination of government information, stifle public debate, and delay legitimate governmentregulatory action." They also said that the agencies, not OMB, are "the repository of scientific andtechnical expertise," and said they should be given "full authority to govern their own peer reviewprocedures."

AAAS

AAAS filed a comment with OMB in June 2004, saying that the "revised version is muchimproved" but raising several issues. (63) For example, AAAS:

questioned why OMB said scientists seeking an exemption to the peer reviewrequirements for information produced by government-funded scientists were only "advised" toinclude a disclaimer rather than required to do so;

suggested that agencies be required to make public the criteria for determiningwhen outside entities would be commissioned to select peer reviewers or manage the peer reviewprocess, and how these entities would be selected and overseen by the agencies; and

endorsed the idea of an interagency working group on peer review, andrecommended that the group be required to report to the public annually on its deliberations andfindings, and to conduct studies and hold meetings regarding the impact of the guidelines on scienceand the quality of the information produced.

The editor of AAAS's flagship publication, Science, praised OMB for its efforts to address theconcerns of the scientific community, saying that most of the concerns regarding the proposedbulletin were answered in the revision, and that OMB "deserves credit for being a goodlistener." (64)

Other Scientific Societies

A number of other scientific societies also commented on the revised peer review bulletin. For example, an official in the Association of American Medical Colleges was quoted in April 2004as saying that OMB had been "very responsive," and said the revised document "is much, much moreflexible." (65) In July2004 (after the comment period for the revised bulletin closed), a consortium of 13 scientificsocieties, mostly in the life sciences, released a general statement on the role of scientific peer reviewin policy development. (66) The statement cited peer reviews as "critically important toolsfor policy makers" and laid out a list of eight "considerations" the societies believe will help ensurethat agency peer reviews are well designed and performed. While several of the points were alsomade by other organizations, some provided a somewhat different emphasis. For example, thesocieties:

stressed that for scientific peer review to be effective, it "should be insulatedfrom politics as much as possible."

said peer review is not a way to get the "right" answer for policy; in manyinstances, even the best data are subject to alternative interpretations, and there is no "best"answer.

said that, while avoiding and managing conflicts of interests is important, mostscientific peer review rests on the presumed integrity of the reviewers, stemming from professionalstandards of conduct to which scientists subscribe.

said it is critical to acknowledge the cultural differences among scientists,policymakers, and the public in developing and implementing an appropriate model for peer review. For example, science is inherently uncertain, and scientists must acknowledge that in their work,including peer review. Managers, in contrast, often are not in a position to embrace that uncertaintybut must make decisions in the face of it.

Final Peer Review Bulletin

As noted previously, on December 15, 2004, OMB released its final bulletin on peer review. OMB said that it had made "minor revisions" to the April version of the bulletin that were"responsive to the public's comments." (67) Perhaps most notably, OMB said that the final bulletin:

requires that agencies disclose to the public the names of peer reviewers forboth "influential scientific information" and "highly influential scientific assessments." However,the bulletin does not require agencies to disclose which reviewers made whichcomments.

adds an annual reporting requirement to address concerns about the lack of anenforcement mechanism, thereby allowing OMB to "track how agencies are using the Bulletin,including provisions for waivers and exceptions." OMB also said it expects the public to monitoragencies' actions under the bulletin.

requires agencies to designate at the time they plan a peer review whether itshould be considered "influential scientific information" or a "highly influential scientificassessment."

provides additional criteria for what constitutes a "highly influential scientificassessment." Specifically, the term now includes influential assessments that are "novel,controversial, or precedent setting or have significant interagency interest." OMB also clarified thatthe $500 million impact test covers all impacts (public or private sector) that occur in any year.

broadened the applicability of the exemption for time-sensitive health or safetyinformation, noting that it is not restricted to medical data from clinical trials that were subject toadequate peer review before the start of the trial.

requires agencies to adopt or adapt the National Academy of Sciences policyfor committee selection with respect to evaluating conflicts of interests of peer reviewers who arenot federal employees.

OMB also reportedly made a number of what appear to be less significant changes to the finalbulletin. For example, OMB said that the final version of the bulletin:

retains a substantial degree of discretion for the agencies, but also clarifies theintent of its language concerning the "adequacy of prior peer review" and the use of deferrals andwaivers to address concerns that the revised draft was too discretionary.

encourages agencies to provide public participation opportunities for reviewsof "highly influential assessments," but avoids a strict mandate for public participation and addslanguage "to stress that agencies should avoid open-ended comment periods."

"reinstate[s]" instructions that peer reviewers are to comment on scientific andtechnical questions, and not to provide opinions on the policy implications.

provides a "rare exception" to allow a premier government scientist from thedepartment or agency that prepared an assessment to participate in a peer review (under certaincircumstances).

Concluding Observations

OMB's peer review bulletin is likely to have a significant effect on federal rulemaking andother forms of information dissemination and public policy. That effect is likely to be both direct(through agencies' and OMB's enforcement of the bulletin's requirements) and indirect (throughreferences to the bulletin by others). For example, Section 402 of the Specialty CropsCompetitiveness Act of 2004 ( P.L. 108-465 , signed by the President on December 21, 2004)indicated that a required peer review of the procedures and standards governing the considerationof certain import and export requests "shall be consistent with the guidance by the Office ofManagement and Budget pertaining to peer review and information quality."

Some of the initial issues and concerns raised by commenters on the proposed peer reviewbulletin were clarified or otherwise addressed in the revised and final versions of the bulletin. Perhaps most notably, the bulletin now makes it clear that scientists are not prohibited from servingas peer reviewers if they receive research grants from the agency based on investigator-initiated,competitive, peer reviewed proposals. The bulletin also provides exemptions from the peer reviewrequirements for time-sensitive medical, public health, and safety information, and other compellingcircumstances. However, a number of other issues remain unclear, including (1) the amount ofdiscretion that agencies will actually have in carrying out the bulletin's requirements, (2) the degreeto which the bulletin will accomplish the stated goal of making federal peer review practices moreconsistent, and (3) the effects of the bulletin on agencies and the federal rulemaking process.

Agency Discretion

The final peer review bulletin appears to give federal agencies substantial discretion indetermining whether peer review is required for specific products and, if so, what type of peer reviewmechanism is appropriate and who should serve as peer reviewers. For example, the bulletin saysagencies need not have peer review conducted on influential scientific information that had alreadybeen subject to "adequate" peer review. To determine whether a prior review was "adequate,"agencies are directed to consider (among other things) whether the science is "novel" or "complex,"and whether it is "important" to decision making. (Notably, though, the final bulletin no longerindicates that an "adequate" peer review "need not comply with" all of the requirements of thebulletin.) Also, the final bulletin says that information should be considered "influential scientificinformation" if the agency "reasonably can determine" that it will have or does have a "clear andsubstantial" impact on "important" public policies or private sector decisions. The bulletin says thatpeer reviewers should be selected to provide the necessary expertise, experience, and skills, and thegroup of reviewers should be "sufficiently" broad and diverse to "fairly" represent the "relevant"scientific perspectives and knowledge.

On the other hand, the final bulletin also gives OMB substantial discretion in certain areas. For example, the bulletin indicates that OMB can require agencies to use the more exactingprocedures for "highly influential scientific assessments" if OMB determines the information is a"scientific assessment" that "could" have a substantial impact on public policies or private sectordecisions with a "potential impact" of more than $500 million in any year, or is "novel, controversial,or precedent setting," or has "significant" interagency interest. Also, OMB can unilaterally approveagencies' use of alternative peer review procedures.

The amount of discretion that agencies will actually have in carrying out their peer reviewprograms (or, conversely, the amount of control that OMB will retain) will be apparent only throughthe bulletin's implementation. The amount of agency discretion (or OMB control) could varysubstantially from one administration to another.

Consistency

OMB indicated that stronger peer review policies were needed because of the importance ofthe issue and, citing a 1999 GAO report, because of the "variability in both the definition andimplementation of peer review across agencies." OMB went on to say that, prior to the developmentof the bulletin, "there were no government-wide standards concerning when peer review is requiredand, if required, what type of peer review processes are appropriate." Therefore, OMB said that thebulletin "establishes minimum standards for when peer review is required for scientific information,and the types of peer review that should be considered by agencies in different circumstances."

However, the final bulletin may not provide the desired consistency in peer review definitionor implementation. As indicated above, the bulletin leaves many key terms undefined or subject tointerpretation, and gives the agencies substantial discretion regarding when certain actions shouldbe taken (e.g., when previous peer reviews are "adequate") and which reviewers should be selected(i.e., those with the "necessary" expertise, experience, and skills). To the extent that agencies areallowed to exercise discretion in these areas, consistency may be forfeited. On the other hand, strictenforcement of uniform procedures established by OMB carries with it a different set of concernsabout aggregation of power within the Executive Office of the President, and may be resisted byfederal agencies. Also, as noted previously, although GAO reported that agencies' peer reviewpractices were inconsistent, it did not recommend greater consistency, and some view variation inthose practices as appropriate and desirable.

Effects on Agencies and Rulemaking

A number of commenters expressed concerns regarding the effect that adding new peerreview requirements would have on what is already viewed by some observers as a lengthy,"ossified" federal rulemaking process. Somewhat related concerns have been voiced regarding thecost of the requirements to federal agencies, with Senators Lieberman and Durbin suggesting thatthe requirements "do not pass OMB's own [cost-benefit] test of good rulemaking." In response,OMB provided estimates in the preamble to the revised bulletin indicating that it did not believe thecosts would be prohibitive to the agencies. As noted previously, though, in 1997, OMB indicatedthat peer review costs could be significant in terms of both time and agency resources. (68) Little empirical data arecurrently available regarding the cost of peer reviews, how they affect the federal rulemakingprocess, or their effect on the quality of the information being reviewed. It is even less clear howthe peer review requirements suggested by OMB will affect those factors, and there appears to beno mechanism in place for collecting such data. There are some indications, however, that therequirements could delay regulatory action. (69)

The AEI-Brookings Joint Center for Regulatory Studies recommended that OMB or someother entity build into the peer review program an evaluation to determine its effect on the qualityof regulatory analyses. Any such evaluation could, at least conceptually, include an examination ofthe cost of the new peer review requirements to federal agencies and its effect on the pace ofrulemaking. To determine the effect of the peer review bulletin on these or other factors, baselineinformation regarding the current state of the art would need to be gathered before the bulletin'simplementation.

Even if the data indicate that peer review adds time to the early stages of the rulemakingprocess, that time may be worth the investment if doing so reduces the likelihood of subsequentjudicial challenges to the rules. Peer review may also provide agencies with a preview of likelyobjections to a rule during the notice and comment phase, thereby allowing them to minimize anyweaknesses and respond quickly to adverse comments. All of these factors would have to beconsidered in any evaluation of the effect of peer review on the regulatory process.

Table 1. Comparison of Proposed, Revised, and Final Peer Review Bulletins