Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

OFFICE OF MANAGEMENT AND
BUDGET

Guidelines for Ensuring and Maximizing the Quality,Objectivity, Utility,and Integrity of Information Disseminated by Federal Agencies; Republication

EditorialNote: Due tonumerous errors, this document isbeing reprinted inits entirety.Itwas originally printedin the Federal Register on Thursday,January 3, 2002 at 67 FR 369-378 and was corrected on Tuesday, February 5,2002 at 67 FR 5365.
AGENCY: Office ofManagement and Budget, Executive Office ofthe President.
ACTION: Final guidelines.

SUMMARY: These final guidelines implement section 515 ofthe Treasury and General Government Appropriations Act for Fiscal Year 2001 (PublicLaw 106-554; H.R. 5658). Section 515 directs the Office of Management and Budget (OMB) to issue government-wide guidelines that "provide policy andprocedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies." By October 1, 2002, agencies
must issue their own implementing guidelines thatinclude "administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminatedby the agency" that does
not comply with the OMB guidelines. These final guidelines also reflect the changes OMB made to the guidelines issued September 28, 2001, as a result ofreceiving additional comment on the "capable ofbeing substantially reproduced" standard (paragraphs V.3.B, V.9, and V.10), which OMB previously issued on September 28, 2001, on an interim final basis. DATES: EffectiveDate:January 3, 2002.
FOR FURTHER INFORMATION CONTACT: Brooke J.Dickson, Office ofInformation and Regulatory Affairs, Office of Management and Budget, Washington, DC 20503. Telephone (202) 395-3785 or by e-mail to informationquality@omb.eop.gov.

SUPPLEMENTARY INFORMATION: In section 515(a) ofthe Treasury and General Government Appropriations Act for Fiscal Year 2001 (PublicLaw 106-554; H.R. 5658), Congress directed the Office ofManagement and Budget (OMB) to issue, by September 30, 2001, government-wide guidelines that "provide policy andprocedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, and integrity of information (including statistical information) disseminated by Federal agencies * * *"Section 515(b) goes on to state that the OMB guidelines shall: "(1) apply tothe sharing byFederal agencies of,and access to, information disseminated byFederal agencies; and

"(2) require that each Federal agency to which the guidelines apply- "(A) issue guidelines ensuring and maximizing the quality, objectivity, utility, and integrity ofinformation (including statistical information) disseminated bythe agency, by not later than 1year after the date ofissuance of the guidelines under subsection (a); "(B) establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated bythe agency that does
not comply with the guidelines issued under subsection (a); and "(C) report periodically to the Director- "(i) the number and nature of complaints received by the agency regarding the accuracy ofinformation disseminated bythe agency and;

"(ii) how such complaints were handled bythe agency."

Proposed guidelines were published in the FederalRegister on June 28, 2001 (66 FR 34489). Final guidelines were published in the Federal Register on September 28, 2001 (66 FR 49718). The Supplementary Information tothe final guidelines published in September 2001 provides background, the underlying principles OMB followed in issuing the final guidelines, and statements of intent concerning detailed provisions in the final guidelines.

In the final guidelilnes published in September 2001, OMB also requested additional comment onthe "capable of being substantially reproduced" standard and the related definition of "influential scientific or statistical information" (paragraphs V.3.B, V.9, andV.10), which were issued onan interim final basis. The final guidelines published today discuss the public
comments OMBreceived, the OMB response, and amendments to the final guidelines published in September 2001.

In developing agency-specific guidelines, agencies should refer both to the Supplementary Information to the final guidelines published in the Federal Register onSeptember 28, 2001 (66 FR 49718), and also to the Supplementary Information published today. We stress thatthe three "Underlying Principles" that OMB followed in drafting the guidelines that

we published on September 28, 2001 (66 FR 49719), are also applicable to the amended guidelines that we publish today.

In accordance with section 515, OMB has designed the guidelines to help agencies ensure and maximize the quality,utility, objectivity and integrity ofthe information thatthey disseminate (meaning to share with, orgive access to, the public). It is crucial that information Federal agencies disseminate meets these guidelines. In this respect, the fact that the Internet enables agencies to communicate information quickly and easily to awide audience not only offers great benefits to society,but alsoincreases the potential harm that can result from the dissemination ofinformation that does
notmeet basic information quality guidelines. Recognizing the wide variety ofinformation Federal agencies disseminate andthe wide variety of dissemination practices that agencies have, OMB developed the guidelines with several principles in mind.

First, OMB designed the guidelines to apply to a wide variety ofgovernment information dissemination activities thatmay range in importance and scope. OMB also designed the guidelines tobe generic enough to fit all media,be they printed, electronic, or in other form. OMB sought to avoid the problems that wouldbe inherent in developing detailed, prescriptive, "one-size-fits-all" government-wide guidelines that would artificially require different types of dissemination activities to be treated in the same manner. Through this flexibility, each agency willbe able to incorporate therequirements ofthese OMB guidelines into the agency's own information resource management and administrative practices.

Second, OMB designed the guidelines

so that agencies will meetbasic information quality standards. Giventhe administrative mechanisms required by section 515 as well as the standards set forth in the PaperworkReduction Act, it is clear that agencies shouldnot disseminate substantive information that does not meet a basic level of quality.We recognize that some government information mayneed to meethigher or more specific information quality standards than those that would apply to other types of government information. The more important the information, the higher the quality standards to which it should be held, for example, in those situations involving "influential scientific, financial, or statistical information" (a phrase defined in these guidelines). The guidelines recognize, however, that 8452Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

information quality comes at a cost. Accordingly, the agencies should weigh the costs (for example, including costs attributable to agency processing effort, respondent burden, maintenance of needed privacy, and assurances of suitable confidentiality) and thebenefits ofhigher information quality in the development ofinformation, andthe level ofquality to which the information disseminated willbe held.

Third, OMB designed the guidelines

so thatagencies can apply them in a

common-sense and workable manner. It is important that these guidelines do not impose unnecessary administrative burdens that would inhibit agencies from continuing totake advantage ofthe Internet and other technologies to disseminate information that canbe of greatbenefit andvalue tothe public. In this regard, OMB encourages agencies to incorporate the standards and procedures required by these guidelines into their existing information resources management and administrative practices rather than create new and potentially duplicative orcontradictory
processes. The primary example ofthis is thatthe guidelines recognize that, in accordance with OMB Circular A-130, agencies already have in place wellestablished information quality standards and administrative mechanisms that allow persons to seek and obtain correction ofinformation that is maintained and disseminated by the agency. Under the OMB guidelines, agencies need only ensure that their

own guidelines are consistent with these OMB guidelines, and then ensure that their administrative mechanisms satisfy the standards and procedural requirements in the new agency guidelines. Similarly, agencies may rely

ontheir implementation ofthe Federal Government's computer security laws (formerly, the Computer Security Act, and now the computer security provisions ofthe Paperwork Reduction Act) to establish appropriate security safeguards for ensuring the "integrity" ofthe information thatthe agencies disseminate.

In addition,in response to concerns expressed by some ofthe agencies, we want to emphasize that OMB recognizes that Federal agencies provide a wide variety ofdata and information. Accordingly, OMB understands thatthe guidelines discussed below cannot be implemented in the same way by each agency. In some cases, for example, the data disseminatedby an agency are not collected by that agency; rather, the information the agency must provide in

atimely manner is compiled from a variety ofsources that are constantly updated and revised and may be confidential. In such cases, while agencies' implementation ofthe guidelines may differ, the essence ofthe guidelines will apply. That is, these agencies must make their methods transparent byproviding documentation, ensure quality by reviewing the underlying methods used in developing the data and consulting (as appropriate) with experts and users, andkeep users informed about corrections and revisions. Summary ofOMB Guidelines

These guidelines apply toFederal agencies subject to the Paperwork Reduction Act (44 U.S.C. chapter 35). Agencies are directed to develop information resources management procedures for reviewing and substantiating (by documentation or othermeans selectedby the agency)the quality (includingthe objectivity, utility, and integrity) ofinformation before it is disseminated. In addition, agencies are to establish administrative mechanisms allowing affected persons to seek and obtain, where appropriate, correction ofinformation disseminated by the agency that does not comply with the OMB or agency guidelines. Consistent with the underlying principles described above, these guidelines stress the importance of having agencies apply these standards and develop their administrative mechanisms so they canbe implemented in a common sense and workable manner. Moreover, agencies
must applythese standards flexibly, and in amanner appropriate tothe nature andtimeliness ofthe information to be disseminated, and incorporate them into existing agency information resources management and administrative practices. Section 515 denotes four substantive
terms regarding information disseminated byFederal agencies: quality,utility, objectivity, and integrity. It is not always clear how each substantive term relates-or how the four terms in aggregate relate-to the widely divergent types ofinformation that agencies disseminate. The guidelines provide definitions that attemptto establish a clear meaning so that both the agency andthe public can readily judge whether aparticular type ofinformation to be disseminated does

or does not meet these attributes.

In the guidelines, OMB defines "quality" as the encompassing term, of which "utility," "objectivity," and "integrity" are the constituents. "Utility" refers to the usefulness ofthe information tothe intended users. "Objectivity" focuses onwhether the disseminated information is being presented in an accurate, clear, complete, and unbiased manner, and as

a matter ofsubstance, is accurate, reliable, and unbiased. "Integrity" refers
to security-the protection of information from unauthorized access

or revision, to ensure that the information is not compromised through corruption or falsification. OMB modeled the definitions of "information," "government information," "information dissemination product," and "dissemination" on the longstanding definitions ofthose terms in OMB Circular A-130, but tailored them to fit into the context ofthese guidelines.

In addition, Section 515 imposes two reporting requirements on the agencies. The first report, to be promulgated no laterthan October 1,2002, must provide the agency's information quality guidelines that describe administrative mechanisms allowing affected persons to seek and obtain, where appropriate, correction ofdisseminated information that does not comply withthe OMB and agency guidelines. The second report is

an annual fiscal year report to OMB (to be first submitted onJanuary 1,2004) providing information (bothquantitative and qualitative, where appropriate) on the number, nature, and resolution of complaints received by the agency regarding its perceived or confirmed failure to comply with these OMB and
agency guidelines.

Public Comments andOMB Response

ApplicabilityofGuidelines.Some
comments raised concerns about the applicability ofthese guidelines, particularly in the context of scientific research conducted by Federally employed scientists or Federal grantees who publish and communicate their research findings in the same manner as their academic colleagues. OMB believes thatinformation generated and disseminated in these contexts isnot covered by these guidelines unless the agency represents the information as, or

uses the information in support of, an official position ofthe agency.

As a general matter, these guidelines apply to "information" thatis "disseminated" by agencies subject to the Paperwork Reduction Act (44 U.S.C. 3502(1)). See paragraphs II, V.5 and V.8. The definitions of"information" and "dissemination" establish the scope of the applicability ofthese guidelines. "Information" means "any communication or representation of knowledge such as facts or data * * " This definition ofinformation in paragraph V.5 does "not include opinions, where the agency's presentation makes it clear thatwhat is 8453Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

being offered is someone's opinion rather than fact or the agency's views."

"Dissemination" is defined tomean "agency initiated or sponsored distribution ofinformation tothe public." As used in paragraph V.8, "agency INITIATED * * * distribution ofinformation to thepublic" refers to information thatthe agency disseminates, e.g., a risk assessment prepared by the agency to inform the agency's formulation ofpossible regulatory or other action. In addition, ifan agency, as an institution, disseminates information prepared by

an outside party in a manner that reasonably suggests thatthe agency agrees with the information, this appearance ofhaving the information represent agency views makes agency dissemination ofthe information subject
to these guidelines. By contrast, an agency does not "initiate" the dissemination ofinformation when a Federally employed scientist orFederal grantee or contractor publishes and communicates his or her research findings in the same manner as his or her academic colleagues, even ifthe Federal agency retains ownership or otherintellectual property rights because the Federal government paid for the research. To avoid confusion regarding whether the agency agrees with the information (and istherefore disseminating itthrough the employee

orgrantee), the researcher should include an appropriate disclaimer in the publication or speech to the effect that the "views are mine, and do not necessarily reflect the view" ofthe

agency.

Similarly, as used in paragraph V.8., 

### "agency * * * SPONSORED distribution ofinformation tothe public" refers to situations where an agency has directed a third-party to disseminate information, or where the
agency has the authority to review and approve the information before release. Therefore, for example, ifan agency through a procurement contract or a grant provides for a person to conduct research, and then the agency directs the person to disseminate theresults (or the agency reviews and approves the results before they may be disseminated), then the agency has "sponsored" the dissemination ofthis information. By contrast, ifthe agency simplyprovides funding to support research, and itthe researcher (notthe agency)who decides whether to disseminate the results and-ifthe results are to be released-who determines the content and presentation ofthe dissemination, then the agency has not "sponsored" the dissemination

eventhough ithas funded the research and evenifthe Federal agency retains ownership or other intellectual property rights because the Federal government paid for the research. To avoid confusion regarding whether the agency is sponsoring the dissemination, the researcher shouldinclude an appropriate disclaimer in the publication or speech to the effect that the "views aremine, and donot necessarily reflect the view" ofthe
agency. On the otherhand, subsequent
agency dissemination ofsuch information requires thatthe information adhere to the agency's information quality guidelines. In sum, these guidelines govern an agency's dissemination ofinformation, but generally do not govern athird-party's dissemination ofinformation (the exception being where the agency is essentially using the third-party to disseminate information on the agency's behalf). Agencies, particularly those that fund scientific research, are encouraged
to clarify the applicability ofthese guidelines tothe various types of information they andtheir employees and grantees disseminate.

Paragraph V.8 also states that the definition of "dissemination" does not include" * *distribution limited to correspondence with individuals or persons, press releases, archival records, public filings, subpoenas or adjudicative processes." The exemption from the definition of "dissemination" for "adjudicative processes" isintended to exclude, from the scope ofthese guidelines, the findings and determinations that an agency makes in the course ofadjudications involving specific parties. There are wellestablished procedural safeguards and rights to address the quality of adjudicatory decisions andto provide
persons with an opportunity to contest decisions. These guidelines do not impose any additional requirements on agencies during adjudicative proceedings and do not provide parties
to such adjudicative proceedings any additional rights ofchallenge or appeal.

ThePresumption FavoringPeerReviewedInformation.As ageneral matter, in the scientific and research context, we regard technical information that has been subjected to formal, independent, external peerreview as presumptively objective. As the guidelines state in paragraph V.3.b.i: "If data and analytic results have been subjected to formal, independent, external peer review, the information
may generally be presumed to be of acceptable objectivity." An example ofa formal, independent, external peer review is the review process used by scientific journals. Most comments approved ofthe prominentrole thatpeer review plays in the OMB guidelines. Some comments contended that peer review was not accepted as auniversal standard that incorporates an established, practiced, and sufficient level ofobjectivity. Other
comments stated thatthe guidelines wouldbe better clarified by making peer review one ofseveral factors that an agency should consider in assessing the objectivity (and quality in general) of original research. In addition, several
comments noted thatpeer review does
not establish whether analytic results

are capable ofbeing substantially reproduced. In light ofthe comments, the final guidelines in new paragraph V.3.b.i qualify the presumption infavor ofpeer-reviewed information as follows: "However, this presumption is rebuttable based on apersuasive showing by the petitioner in a particular instance."

We believe that transparency is important for peer review, andthese guidelines set minimum standards for the transparency of agency-sponsored
peer review. As we state in new paragraph V.3.b.i: "Ifdata and analytic results havebeen subjected to formal, independent, external peer review,the information may generally be presumed
to be of acceptable objectivity. However, this presumption isrebuttable based on

a persuasive showing by the petitioner in a particular instance. Ifagencysponsored peer review is employed to help satisfy the objectivity standard, the review process employed shall meet the general criteria for competent and credible peer reviewrecommended by OMB-OIRA to the President's Management Council (9/20/01) (http:// www.whitehouse.gov/omb/inforeg/ oira review-process.html), namely, 'that (a) peer reviewers be selectedprimarily

on the basis ofnecessary technical expertise, (b)peer reviewers be expected
to disclose to agencies prior technical/ policy positions they may have taken on the issues athand, (c) peer reviewers be expected to disclose to agencies their

sources ofpersonal andinstitutional funding (private or public sector), and (d) peer reviews be conducted in an open andrigorous manner.'"

The importance ofthese general criteria for competent and credible peer review hasbeen supported by anumber ofexpert bodies. For example, "the work offully competent peer-review panels can be undermined by allegations ofconflict ofinterest and bias. Therefore, the best interests ofthe Board are served by effective policies and procedures regarding potential conflicts ofinterest, impartiality, and panel balance." (EPA'sScienceAdvisory 

### 8454Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

BoardPanels:ImprovedPoliciesand Procedures NeededtoEnsure IndependenceandBalance,GAO-01- 536, General Accounting Office, Washington, DC, June 2001, page 19.) As another example, "risk analyses shouldbe peer-reviewed and accessible-both physically and intellectually-so that decision-makers
atall levels willbe able to respond critically torisk characterizations. The intensity ofthe peer reviews should be
commensurate with the significance of the risk or its management implications." (SettingPriorities, GettingResults:A NewDirection for EPA, Summary Report, National Academy ofPublic Administration, Washington, DC, April 1995, page 23.)

These criteria for peer reviewers are generally consistent with thepractices

now followedby the National Research Council ofthe National Academy of Sciences. In considering these criteria for peer reviewers, we note thatthere

are many types ofpeer reviews and that
agency guidelines concerning the use of
peer review should tailor the rigor of
peer review to the importance ofthe information involved. More generally, agencies should define their peer-review standards in appropriate ways, given the
nature and importance ofthe information they disseminate.

IsJournal PeerReviewAlways Sufficient?Some comments argued that journal peer review should be adequate
to demonstrate quality, even for influential information that canbe expected tohave major effects on public policy. OMB believes that this position
overstates the effectiveness ofjournal peer review as aquality-control mechanism.

Although journal peerreview is clearly valuable, there are cases where flawed science hasbeen published in respected journals. For example, the NIH Office ofResearch Integrity recently reported the following case regarding environmental health research:

"Based on the report ofan investigation conducted by [XX] University, datedJuly 16, 1999, and additional analysis conductedby ORI in its oversightreview, the US Public Health Service found that Dr. [X] engaged in scientific misconduct. Dr. [X] committed scientific misconduct by intentionally falsifying theresearch results published in the journal SCIENCE andbyproviding falsified and fabricated materials to investigating officials at [XX] University in response to arequest for original datato support the researchresults and conclusions report inthe SCIENCE paper.In addition, PHS findsthat there isno original dataor other corroborating evidence to support the research results and conclusions reportedin the SCIENCE paper asawhole." (66 FR 52137, October 12, 2001). Although such cases offalsification

are presumably rare, there is a significant scholarly literature documenting quality problems with articles published in peer-reviewed research. "In a [peer-reviewed] metaanalysis that surprised many-and some doubt-researchers found little evidence that peer review actually improves the quality ofresearch papers." (See, e.g., Science,Vol. 293, page 2187 (September 21, 2001.)) In part for this reason, many agencies have already adopted peer review and science advisory practices that go beyond journal peerreview. See,
e.g., Sheila Jasanoff, The FifthBranch: ScienceAdvisersasPolicyMakers, Cambridge, MA, Harvard University Press, 1990; Mark R. Powell, Scienceat EPA:Informationin theRegulatory Process.Resources for the Future, Washington, DC., 1999, pages 138-139; 151-153; Implementationofthe EnvironmentalProtectionAgency's Peer ReviewProgram: An SAB Evaluationof ThreeReviews, EPA-SAB-RSAC-01- 009, A Review ofthe Research Strategies Advisory Committee (RSAC) ofthe EPA Science Advisory Board (SAB), Washington, DC., September 26, 2001. For information likely to have an important public policy orprivate sector impact, OMB believes that additional quality checks beyond peer review are appropriate.

Definitionof "Influential".OMB guidelines apply stricter quality standards to the dissemination of information thatis considered "influential." Comments noted that the breadth ofthe definition of"influential" in interim final paragraph V.9 requires much speculation on thepart of agencies.

We believe that this criticism has merit and have therefore narrowed the definition. In this narrower definition, "influential", when used in the phrase "influential scientific, financial, or statistical information", is amended to

mean that "the agency canreasonably determine that dissemination ofthe information will have or does have a clear and substantial impact on important public policies or important private sector decisions." The intent of the new phrase "clear and substantial" is toreduce the need for speculation on the part of agencies. We added the present tense-"or does have"-to this

narrower definition because on occasion, an information dissemination
may occur simultaneously with a particular policy change. In response to

apublic comment, we added an explicit reference to "financial" information as consistent with our original intent.

Given the differences in the many Federal agencies covered by these guidelines, and the differences in the
nature ofthe information they disseminate, we alsobelieve it will be helpful ifagencies elaborate on this definition of "influential" in the context oftheir missions and duties, with due consideration ofthe nature ofthe information they disseminate. As we state in amended paragraph V.9, "Each
agency is authorized to define 'influential' in ways appropriate for it given the nature and multiplicity of issues for which the agency is responsible."

Reproducibility.As we state in new paragraph V.3.b.ii: "If an agency is responsible for disseminating influential scientific, financial, or statistical information, agency guidelines shall include ahigh degree oftransparency about data and methods to facilitate the reproducibility ofsuch information by qualified third parties." OMB believes that a reproducibility standard is practical and appropriate for information thatis considered "influential", as defined in paragraph V.9-that "will have or does have a clear and substantial impact on important public policies or important private sector decisions." The reproducibility standard applicable to influential scientific, financial, or statistical information is intended to

ensure thatinformation disseminated by agencies is sufficiently transparent in
terms ofdata and methods ofanalysis thatitwould be feasible for areplication
to be conducted. The fact that the use oforiginal and supporting data and analytic results have been deemed "defensible" by peer-review procedures does not necessarily implythat the results are transparent and replicable.

ReproducibilityofOriginal and Supporting Data.Several ofthe
comments objected to the exclusion of original and supporting data from the reproducibility requirements. Comments instead suggested that OMB should apply the reproducibility standard to original data, andthat OMB should provide flexibility to the agencies in determining what constitutes "original and supporting" data. OMB agrees and asks that agencies consider, in developing their own guidelines, which categories oforiginal and supporting data should be subject to the reproducibility standard andwhich should not.To help in resolving this issue, we also ask agencies to consult directly with relevant scientific and technical communities on the feasibility ofhaving the selected categories of original and supporting data subject to the reproducibility standard. Agencies

are encouraged to address ethical, feasibility, and confidentiality issues 

### 8455Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

with care. As we state in new paragraph V.3.b.ii.A, "Agencies may identify, in consultation with the relevant scientific andtechnical communities, those particular types ofdata that can practicably be subjected to a reproducibility requirement, given ethical, feasibility, orconfidentiality constraints." Further, as we state in our expanded definition of "reproducibility" in paragraph V.10, "If agencies apply the reproducibility test to specific types oforiginal or supporting data, the associated guidelines shall provide relevant definitions ofreproducibility (e.g., standards for replication oflaboratory data)." OMB urges caution in the
treatment oforiginal and supporting databecause it may often be impractical

oreven impermissible or unethical to apply the reproducibility standard to such data. Forexample, it may notbe ethical to repeat a "negative" (ineffective)clinical (therapeutic) experiment and itmay not be feasible to replicate the radiation exposures studied after the Chernobyl accident. When agencies submit their draft agency guidelines for OMB review, agencies shouldinclude a description ofthe
extent to which the reproducibility standard is applicable and reflect consultations with relevant scientific andtechnical communities that were used in developing guidelines related to applicability ofthereproducibility standard to original and supporting data.

It is also important to emphasize that the reproducibility standard does not apply to all original and supporting data disseminatedby agencies. As we state in

new paragraph V.3.b.ii.A, "With regard
to original and supporting data related [to influential scientific, financial, or statistical information], agency guidelines shall not require that all disseminated databe subjected to a reproducibility requirement." In addition, we encourage agencies to address how greater transparency can be achieved regarding original and supporting data. As we also state in new paragraph V.3.b.ii.A, "It is understood that reproducibility ofdata is an indication oftransparency about research design and methods and thus

areplication exercise (i.e.,a new experiment, test, or sample) shall not be required prior to each dissemination." Agency guidelines need to achieve a high degree oftransparency about data

even when reproducibility isnot required.

Reproducibilityof AnalyticResults. Many public comments were critical of the reproducibility standard and expressed concern that agencies would be required to reproduce each analytical result before it is disseminated. While several comments commended OMBfor establishing an appropriate balance in the "capable ofbeing substantially reproduced" standard, others considered this standard to be inherently subjective. There were also
comments that suggested the standard would cause more burden for agencies.

It is not OMB's intent that each
agency must reproduce each analytic result before it is disseminated. The
purpose ofthereproducibility standard is to cultivate a consistent agency commitment to transparency about how analytic results are generated: the specific data used, the various assumptions employed, the specific analytic methods applied, andthe statistical procedures employed. If sufficient transparency is achieved on each ofthese matters, then an analytic result shouldmeet the "capable ofbeing substantially reproduced" standard.

While there is much variation in types ofanalytic results, OMBbelieves that reproducibility is a practical standard to apply to most types of analytic results. As we state in new paragraph V.3.b.ii.B, "With regard to analytic results related [to influential scientific, financial, or statistical information], agency guidelines shall generally require sufficient transparency about data and methods that an independent reanalysis couldbe undertaken by a qualified member ofthepublic. These transparency standards apply to agency analysis ofdata from a single study as well as to analyses thatcombine information from multiple studies." We elaborate upon this principle in our expanded definition of "reproducibility" in paragraph V.i0: "With respect to analytic results, 'capable ofbeing substantially reproduced' means thatindependent analysis ofthe original or supporting datausing identical methods would generate similar analytic results, subject
to an acceptable degree ofimprecision

or error."

Even in a situation where the original and supporting data are protected by confidentiality concerns, or the analytic computer models or other research methods may be kept confidential to protect intellectual property, itmay still be feasible to have the analytic results subject to the reproducibility standard. For example, a qualified party, operating under the same confidentiality protections as the original analysts, may be askedto use the same data, computer model or statistical methods to replicate the analytic results reported in the original study. See, e.g., "Reanalysis ofthe Harvard SixCities Study andthe American Cancer Society Study of Particulate AirPollution and Mortality," ASpecial Report ofthe Health Effects Institute's Particle Epidemiology Reanalysis Project, Cambridge, MA, 2000.

The primary benefit ofpublic transparency isnot necessarily that

errors in analytic results willbe detected, although error correction is clearly valuable. The more important benefit oftransparency is that thepublic willbe able to assess howmuch an agency's analytic resulthinges on the specific analytic choices made bythe
agency. Concreteness about analytic choices allows, for example, the implications of alternative technical choices tobe readily assessed. This type ofsensitivity analysis is widely regarded as an essential feature ofhighquality analysis, yet sensitivity analysis
cannot be undertaken by outside parties unless a high degree oftransparency is achieved. The OMB guidelines donot compel such sensitivity analysis as a necessary dimension ofquality, but the transparency achieved by reproducibility will allow the public to undertake sensitivity studies ofinterest.

We acknowledge that confidentiality

concerns will sometimes preclude public access as an approach to reproducibility. In response to public comment, we have clarified that such

concerns do include interests in "intellectual property." To ensure that the OMB guidelines have sufficient flexibility with regard to analytic transparency, OMB has, in new paragraph V.3.b.ii.B.i, provided agencies

an alternative approach for classes or types of analytic results that cannot practically be subject to the reproducibility standard. "[In those situations involving influential scientific, financial, or statistical information * * *

### ] making the data and methods publicly available will assist in determining whether analytic results are reproducible. However, the objectivity standard does not override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections. Specifically, in cases where reproducibility will not occur due to other compelling interests, we expect agencies (1) to perform robustness checks appropriate to the importance of the information involved, e.g., determining whether a specific statistic is sensitive to the choice ofanalytic method, and, accompanying the information disseminated, to document their efforts to assure the needed robustness in information quality, and (2) address in their guidelines the 

### 8456Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

degree to which they anticipate the opportunity for reproducibility tobe limited bythe confidentiality of underlying data. As we state in new paragraph V.3.b.ii.B.ii, "In situations where public access to data and methods will not occur due to other compelling interests, agencies shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken. Agency guidelines shall, however, in allcases, require adisclosure ofthe specific data

sources thathave been used and the specific quantitative methods and assumptions thathave been employed."

Given the differences in the many Federal agencies covered by these guidelines, andthe differences in robustness checks and the level ofdetail for documentation thereof thatmight be appropriate for different agencies, we alsobelieve it willbe helpful ifagencies elaborate on these matters in the context oftheir missions and duties, with due consideration ofthe nature ofthe information they disseminate. As we state in new paragraph V.3.b.ii.B.ii, "Each agency is authorized to define the type ofrobustness checks, and the level ofdetail for documentation thereof, in
ways appropriate for it given the nature and multiplicity ofissues for which the agency is responsible."

We leave the determination ofthe appropriate degree ofrigor tothe discretion ofagencies andthe relevant scientific and technical communities that work with the agencies. We do, however, establish a general standard for the appropriate degree ofrigor in our expanded definition of "reproducibility" in paragraph V.10: "'Reproducibility' means that the information is capable ofbeing substantially reproduced, subject to an acceptable degree ofimprecision. For information judged to have more (less) important impacts, the degree of imprecision that istolerated is reduced (increased)." OMB will review each agency's treatment ofthis issue when reviewing the agency guidelines as a whole.

Comments also expressed concerns regarding interim final paragraph V.3.B.iii, "making the data and models publicly available will assist in determining whether analytic results are capable ofbeing substantially reproduced," and whether it could be interpreted to constitute public dissemination ofthese materials, rendering mootthe reproducibility test. (Forthe equivalent provision, see new paragraph V.3.b.ii.B.i.) The OMB guidelines do not require agencies to reproduce each disseminated analytic result by independent reanalysis. Thus, public dissemination ofdata and models perse does not mean that the analytic result has been reproduced. It

means onlythat the result should be CAPABLE ofbeing reproduced. The transparency associated with this capability ofreproduction is what the OMB guidelines are designed to achieve.

We also want to build on a general observation that we made in our final guidelines published in September 2001. In those guidelines we stated: in those situations involving influential scientific[, financial,] or statistical information, the substantial reproducibility standard is added as a quality standard above and beyond

somepeer review quality standards" (66
FR 49722 (September 28, 2001)). A hypothetical example may serve to illustrate this point. Assume that two Federal agencies initiated or sponsored the dissemination offive scientific studies after October 1, 2002 (see paragraph 111.4) that were, before dissemination, subjected to formal, independent, external peerreview, i.e., that met the presumptive standard for "objectivity" under paragraph V.3.b.i. Further assume, atthe time of dissemination, thatneither agency reasonably expected thatthe dissemination ofany ofthese studies would have "a clear and substantial impact" onimportant public policies, i.e., thatthese studies were not considered "influential" under paragraph V.9, andthus not subject to the reproducibility standards in paragraphs V.3.b.ii.A orB. Then

assume, two years later, in 2005, that

one ofthe agencies decides to issue an important and far-reaching regulation based clearly and substantially onthe agency's evaluation ofthe analytic results setforth in these five studies and that such agency reliance onthese five studies aspublished in the agency's notice ofproposed rulemaking would constitute dissemination ofthese five studies. These guidelines would require the rulemaking agency, prior to publishing the notice ofproposed rulemaking, to evaluate these five studies to determine ifthe analytic results statedtherein would meet the "capable ofbeing substantially reproduced" standards in paragraph V.3.b.ii.B and, ifnecessary, related standards governing original and supporting data in paragraph V.3.b.ii.A. Ifthe agency were to decide that any of the five studies would not meet the reproducibility standard, the agency may still rely onthem but only ifthey satisfy the transparency standard andas applicable-the disclosure of robustness checks required by these guidelines. Otherwise, the agency should not disseminate any ofthe studies that did notmeet the applicable standards in the guidelines at the time itpublishes the notice ofproposed rulemaking.

Some comments suggested that OMB consider replacing the reproducibility standard with a standard concerning "confirmation" ofresults for influential scientific and statistical information. Although we encourage agencies to consider "confirmation" as a relevant standard-at leastin some cases-for assessing the objectivity oforiginal and supporting data, we believe that "confirmation" is too stringent a standard to apply to analytic results. Often the regulatory impact analysis prepared by an agency for a major rule, for example, willbe the only formal analysis ofan important subject. It wouldbe unlikely thatthe results ofthe regulatory impact analysis had already been confirmed by other analyses. The "capable ofbeing substantially reproduced" standard is less stringent than a "confirmation" standard because it simply requires that an agency's analysis be sufficiently transparent that another qualified party could replicate it through reanalysis.

Health,Safety,andEnvironmental Information.We note, in the scientific context, that in 1996 the Congress, for health decisions under the Safe DrinkingWater Act, adopted a basic standard ofquality for the use of science in agency decisionmaking. Under 42 U.S.C. 300g-l(b)(3)(A), an agency is directed, "to the degree that an Agency action is based on science," to use "(i) the best available, peer-reviewed science and supporting studies conducted in accordance with sound and objective scientific practices; and (ii)data collected by accepted methods

orbest available methods (ifthe reliability ofthe method and the nature ofthe decisionjustifies use ofthe data)."

We further note thatin the 1996 amendments to the Safe Drinking Water Act, Congress adopted abasic quality standard for the dissemination ofpublic information about risks ofadverse health effects. Under 42 U.S.C. 300g1(b)(3)(B), the agency is directed, "to

ensure thatthe presentation of information [risk] effects is comprehensive, informative, and understandable." The agency is further directed, "in a document made available
to the public in support ofaregulation [to] specify, to the extentpracticable- (i) each population addressed by any estimate [ofapplicable risk effects]; (ii) the expected risk or central estimate of 

### 8457Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

risk for the specific populations [affected]; (iii) each appropriate upperbound or lower-bound estimate ofrisk; (iv)each significant uncertainty identified in the process ofthe
assessment of [risk] effects and the studies that would assist in resolving the uncertainty; and (v)peer-reviewed studies known to the [agency] that support, are directly relevant to, or fail
to support any estimate of [risk] effects andthe methodology used to reconcile inconsistencies in the scientific data."

As suggested in several comments, we have included these congressional standards directly in new paragraph V.3.b.ii.C, and made them applicable to the information disseminatedby all the agencies subject to these guidelines: "With regard to analysis ofrisks to human health, safety and the environment maintained or disseminatedby the agencies, agencies shall either adopt or adapt the quality principles applied by Congress torisk information used and disseminated pursuant tothe Safe Drinking Water Act Amendments of 1996 (42 U.S.C. 300g1(b)(3)(A) &(B))." The word "adapt" is intended toprovide agencies flexibility in applying these principles to various types ofrisk assessment.

Comments also argued that the continued flow ofvital information from agencies responsible for disseminating health and medical information to medical providers, patients, and the public may be disrupted due tothese
peer review and reproducibility standards. OMB responded by adding to

new paragraph V.3.b.ii.C: "Agencies responsible for dissemination ofvital health and medical information shall interpret the reproducibility and peerreview standards in a manner appropriate to assuringthe timely flow ofvital information from agencies to medical providers, patients, health agencies, and the public. Information quality standards may be waived temporarily by agencies under urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude specified in agency-specific guidelines."

AdministrativeCorrection Mechanisms.In addition to commenting

onthe substantive standards in these guidelines, many ofthe comments noted that the OMB guidelines on the administrative correction ofinformation donot specify a time period inwhich the agency investigation and response must be made. OMB has added the following new paragraph III.3.i to direct agencies to specify appropriate time periods in which the investigation and response need to be made. "Agencies shall specify appropriate time periods for agency decisions on whether and how to correct the information, and agencies shall notify the affected
persons ofthe corrections made." Several comments statedthat the
OMB guidelines needed to direct agencies to consider incorporating an administrative appeal process into their administrative mechanisms for the correction ofinformation. OMB agreed, and added the following new paragraph III.3.ii: "If the person who requested the correction does not agree with the agency's decision (includingthe corrective action, ifany), the person may file for reconsideration within the
agency. The agency shall establish an administrative appeal process to review the agency's initial decision, and specify appropriate time limits in which to resolve suchrequests for reconsideration." Recognizing that
many agencies already have aprocess in place to respond to public concerns, it is not necessarily OMB's intent to require these agencies to establish anew

or different process. Rather, our intent is
to ensure that agency guidelines specify

an objective administrative appeal
process that, upon furthercomplaint by the affected person, reviews an agency's decision to disagree with the correction request. An objective process will

ensure that the office that originally disseminates the information does not have responsibility for both the initial response andresolution ofa disagreement. In addition,the agency guidelines should specify thatifthe agencybelieves other agencies may have

an interest in the resolution ofany administrative appeal, the agency should consult with those other agencies about their possible interest.

Overall, OMB does not envision administrative mechanisms that would burden agencies with frivolous claims. Instead, the correction process should

serve to address the genuine and valid needs ofthe agency and its constituents without disrupting agency processes. Agencies, in making their determination ofwhether ornot to correct information,
may reject claims made in bad faith or withoutjustification, and are required to undertake onlythe degree ofcorrection that they conclude is appropriate for the
nature andtimeliness ofthe information involved, and explain suchpractices in their annual fiscal year reports to OMB. OMB's issuance ofthese final guidelines isthe beginning ofan evolutionary process thatwill include draft agency guidelines, public comment, final agency guidelines, development ofexperience with OMB and agency guidelines, and continued refinement ofboth OMB and agency guidelines. Just as OMBrequested public commentbefore issuingthese final guidelines, OMB will refine these guidelines as experience develops and further public comment is obtained.

Dated:December 21, 2001. JohnD.Graham, Administrator,Officeof Informationand Regula toryAffairs.

Guidelines forEnsuringand Maximizing the Quality, Objectivity, Utility, andIntegrity ofInformation Disseminated by FederalAgencies

L OMBResponsibilities

Section 515 ofthe Treasury and General Government Appropriations Act for FY2001 (PublicLaw 106-554) directs the Office ofManagement and Budget to issue government-wide guidelines thatprovide policy and procedural guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility, andintegrity ofinformation, including statistical information, disseminated byFederal agencies.

II.Agency Responsibilities

Section 515 directs agencies subject to the Paperwork Reduction Act (44 U.S.C. 3502(1)) to1. Issue their own information quality guidelines ensuring and maximizing the quality, objectivity, utility, andintegrity ofinformation, including statistical information, disseminated bythe agency

no laterthan one year after the date of issuance ofthe OMB guidelines;

2. Establish administrative mechanisms allowing affected persons to seek and obtain correction of information maintained and disseminated bythe agency that does
not comply with these OMB guidelines; and

3. Report to the Director ofOMB the number and nature ofcomplaints received by the agency regarding agency compliance with these OMB guidelines concerning the quality, objectivity, utility, and integrity ofinformation and how such complaints were resolved.

IlL.Guidelines forEnsuringand Maximizingthe Quality,Objectivity, Utility,andIntegrityofInformation Disseminated byFederal Agencies
1. Overall, agencies shall adopt a basic standard ofquality (including objectivity, utility, and integrity) as a performance goal and shouldtake appropriate steps to incorporate information quality criteria into agency information dissemination practices. Quality is to be ensured and established
atlevels appropriate to the nature and timeliness ofthe information to be disseminated. Agencies shall adopt 

### 8458Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

specific standards ofquality that are appropriate for the various categories of information they disseminate. 2.As a matter ofgood and effective
agency information resources management, agencies shall develop a process for reviewing the quality (includingthe objectivity, utility, and integrity) ofinformation before itis disseminated. Agencies shall treat information quality as integral to every step ofan agency's development of information, including creation, collection, maintenance, and dissemination. This process shall enable the agency to substantiate the quality of the information it has disseminated through documentation or other means appropriate tothe information.

3.To facilitate public review, agencies shall establish administrative mechanisms allowing affected persons to seek and obtain, where appropriate, timely correction ofinformation maintained and disseminated by the
agency that does not comply with OMB

oragency guidelines. These administrative mechanisms shall be flexible, appropriate to the nature and timeliness ofthe disseminated information, and incorporated into
agency information resources management and administrative practices.

i. Agencies shall specify appropriate time periods for agency decisions on whether and how to correct the information, and agencies shall notify the affected persons ofthe corrections made.

ii. Iftheperson who requested the correction does not agree with the agency's decision (includingthe corrective action, ifany), the person may file for reconsideration within the
agency. The agency shall establish an administrative appeal process to review the agency's initial decision, and specify appropriate time limits in which to resolve suchrequests for reconsideration.

4. The agency's pre-dissemination review, under paragraph 111.2, shall apply to information that the agency first disseminates on or after October 1, 2002. The agency's administrative mechanisms, under paragraph 111.3., shall apply to information thatthe
agency disseminates on or after October 1,2002, regardless ofwhen the agency first disseminated the information.

IV.AgencyReportingRequirements
1.Agencies must designate the Chief Information Officer or another official to be responsible for agency compliance with these guidelines. 2.The agency shall respond to complaints in a manner appropriate to the nature and extent ofthe complaint. Examples of appropriate responses include personal contacts via letter or telephone, form letters, press releases or

mass mailings that correct a widely disseminated error or address a frequently raised complaint. 3.Each agency must prepare a draft report, no later than April 1,2002, providing the agency's information quality guidelines and explaining how such guidelines will ensure and maximize the quality, objectivity, utility, and integrity ofinformation, including statistical information, disseminated bythe agency. This report must also detail the administrative mechanisms developed bythat agency to allow affected persons to seek and obtain appropriate correction of information maintained and disseminated bythe agency that does
not comply with the OMB or the agency guidelines.

4. The agency must publish a notice ofavailability ofthis draftreport in the Federal Register, and post this report on the agency's website, toprovide an opportunity for public comment.

5.Upon consideration ofpublic
comment and after appropriate revision, the agency must submit this draft report to OMB for review regarding consistency with these OMB guidelines

no later than July 1, 2002. Upon completion ofthat OMB review and completion ofthis report, agencies must publish notice ofthe availability ofthis report in its final form in the Federal Register, andpost this report on the agency's web siteno later than October 1, 2002.

6. On an annual fiscal-year basis, each
agency must submit areport tothe Director ofOMB providing information (both quantitative and qualitative, where appropriate) on the number and
nature ofcomplaints received bythe
agency regarding agency compliance with these OMB guidelines and how such complaints were resolved. Agencies must submitthese reports no later than January 1ofeach following
year, withthe first report dueJanuary 1, 2004.

V. Definitions

1. "Quality" is an encompassing term comprising utility, objectivity, and integrity. Therefore, the guidelines sometimes refer to these four statutory terms, collectively, as "quality." 2. "Utility" refers to the usefulness of the information to its intended users, including the public. In assessing the usefulness ofinformation that the
agency disseminates to the public, the agency needs to consider the uses ofthe information not only from the perspective ofthe agencybut also from the perspective ofthe public. As a result, when transparency of information is relevant for assessing the information's usefulness from the public's perspective, the agency must take care to ensure that transparency has been addressed inits review ofthe information.

3. "Objectivity" involves two distinct elements, presentation and substance.

a. "Objectivity" includes whether disseminated information isbeing presented in an accurate, clear, complete, and unbiased manner. This involves whether the information is presented within a proper context. Sometimes, in disseminating certain types ofinformation to the public, other information must also be disseminated in order to ensure an accurate, clear, complete, and unbiasedpresentation. Also, the agency needs to identify the

sources ofthe disseminated information (tothe extentpossible, consistent with confidentiality protections) and, in a scientific, financial, or statistical context, the supporting data and models, so thatthe public can assess for itselfwhether there may be somereason to question the objectivity ofthe

sources. Where appropriate, data should have full, accurate, transparent documentation, and error sources affecting data quality should be identified and disclosed to users.

b. In addition, "objectivity" involves

a focus on ensuring accurate, reliable, and unbiased information. In a scientific, financial, or statistical context, the original and supporting data shall be generated, and the analytic results shall be developed, using sound statistical and research methods.

i. Ifdata and analytic results have been subjected to formal, independent, external peer review, the information
may generally be presumed to be of acceptable objectivity. However, this presumption isrebuttable based on a persuasive showing by the petitioner in

a particular instance. Ifagencysponsored peer review is employed to help satisfy the objectivity standard, the review process employed shall meet the general criteria for competent and credible peer reviewrecommended by OMB-OIRA to the President's Management Council (9/20/01) (http:// www.whitehouse.gov/omb/inforeg/ oira review-process.html), namely, "that (a)peer reviewers be selected primarily on the basis ofnecessary technical expertise, (b)peer reviewers be expected to disclose to agencies prior technical/policy positions they may havetaken on the issues athand, (c)
peer reviewers be expected to disclose
to agencies their sources ofpersonal and 

### 8459Federal Register/Vol. 67, No. 36/Friday, February 22, 2002/Notices

institutional funding (private or public sector), and (d) peer reviews be conducted in an open and rigorous manner."

ii. Ifan agency is responsible for disseminating influential scientific, financial, or statistical information,
agency guidelines shall include a high degree oftransparency about data and methods to facilitate the reproducibility ofsuch information by qualified third parties.

A. With regard to original and supporting data related thereto, agency guidelines shall not require that all disseminated databe subjected to a reproducibility requirement. Agencies
may identify, in consultation with the relevant scientific and technical communities, those particular types of datathat can practicable be subjected to

areproducibility requirement, given ethical, feasibility, orconfidentiality constraints. Itis understood that reproducibility ofdata is an indication oftransparency about research design and methods andthus a replication exercise (i.e., a new experiment, test,or sample) shall not be required prior to each dissemination.

B. With regard to analytic results related thereto, agency guidelines shall generally require sufficient transparency about data and methods that an independent reanalysis could be undertaken by aqualified member ofthe public. These transparency standards apply to agency analysis ofdata from a single study as well as to analyses that combine information from multiple studies.

i. Making the data and methods publicly available will assist in determining whether analytic results are reproducible. However, the objectivity standard does not override other compelling interests such as privacy, trade secrets, intellectual property, and other confidentiality protections.

ii. In situations where public access to data andmethods will not occur due to other compelling interests, agencies shall apply especially rigorous robustness checks to analytic results and document what checks were undertaken. Agency guidelines shall, however, in all cases, require a disclosure ofthe specific data sources that havebeen used and the specific quantitative methods and assumptions that havebeen employed. Each agency is authorized to define the type of robustness checks, and the level of detail for documentation thereof, in
ways appropriate for it given the nature and multiplicity ofissues for which the
agency is responsible.

C. With regard to analysis ofrisks to human health, safety andthe environment maintained or disseminated bythe agencies, agencies shall either adopt or adapt the quality principles applied by Congress torisk information used and disseminated pursuant tothe Safe Drinking Water Act Amendments of1996 (42 U.S.C. 300g1(b)(3)(A) &(B)). Agencies responsible for dissemination ofvital health and medical information shall interpret the reproducibility andpeer-review standards in a manner appropriate to assuringthe timely flow ofvital information from agencies to medical providers, patients, health agencies, and the public. Information quality standards may be waived temporarily by agencies under urgent situations (e.g., imminent threats to public health or homeland security) in accordance with the latitude specified in agency-specific guidelines.

4. "Integrity" refers to the security of information-protection ofthe information from unauthorized access

orrevision, to ensure that the information isnot compromised through corruption or falsification. 5. "Information" means any communication or representation of knowledge such as facts or data, in any medium orform, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from aweb page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agency's presentation makes it clear that what is being offered is someone's opinion rather than fact or the agency's views. 6. "Government information" means information created, collected, processed, disseminated, or disposed of by or for the Federal Government. 7. "Information dissemination product" means any books, paper, map, machine-readable material, audiovisual production, or other documentary material, regardless ofphysical form or characteristic, an agency disseminates to the public. This definition includes any electronic document, CD-ROM, orweb
page. 8. "Dissemination" means agency initiated or sponsored distribution of information to the public (see 5 CFR 1320.3(d) (definition of"Conduct or Sponsor")). Dissemination does not include distribution limited to government employees or agency contractors or grantees; intra- or interagency use or sharing ofgovernment information; and responses to requests for agency records underthe Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act or other similar law. This definition also does not include distribution limited to correspondence with individuals or persons, press releases, archival records, public filings, subpoenas or adjudicative

processes.

9. "Influential", when usedin the phrase "influential scientific, financial,

or statistical information", means that the agency can reasonably determine that dissemination ofthe information will have or does have a clear and substantial impact on important public policies orimportant private sector decisions. Each agency is authorized to define "influential" in ways appropriate for itgiven the nature and multiplicity ofissues forwhich the agency is responsible.

10. "Reproducibility" means thatthe information is capable ofbeing substantially reproduced, subject to an acceptable degree ofimprecision. For informationjudged to have more (less) important impacts, the degree of imprecision that is toleratedis reduced (increased). Ifagencies apply the reproducibility testto specific types of original or supporting data, the associated guidelines shall provide relevant definitions ofreproducibility (e.g., standards for replication of laboratory data). With respect to analytic results, "capable ofbeing substantially reproduced" means that independent analysis ofthe original or supporting data using identical methods would generate similar analytic results, subject to an acceptable degree of imprecision or error.
[FR Doc. 02-59 Filed 1-2-02; 1:36 pm]

BILLING CODE 3110-01-M

Editorial Note:Due to numerous errors, this document is beingreprinted in its entirety. It was originally printed in the Federal Register on Thursday, January 3, 2002 at 67FR 369-378 and was corrected on Tuesday, February 5, 2002 at 67 FR 5365.

[FR Doc. R2-59 Filed 2-21-02; 8:45 am]

BILLING CODE 1505-01-D 

### 8460