Friday,


February 22, 2002


Part IX 

Office of 

Management and 

Budget 

Guidelines for Ensuring and Maximizing 

the Quality, Objectivity, Utility, and 

Integrity of Information Disseminated by 

Federal Agencies; Notice; Republication 
8452 Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 

OFFICE OF MANAGEMENT AND 

BUDGET 

Guidelines for Ensuring and 

Maximizing the Quality, Objectivity, 

Utility, and Integrity of Information 

Disseminated by Federal Agencies; 

Republication 

Editorial Note: Due to numerous errors, 

this document is being reprinted in its 

entirety. It was originally printed in the 

Federal Register on Thursday, January 3, 

2002 at 67 FR 369–378 and was corrected on 

Tuesday, February 5, 2002 at 67 FR 5365. 

AGENCY: Office of Management and 

Budget, Executive Office of the 

President. 

ACTION: Final guidelines. 

SUMMARY: These final guidelines 

implement section 515 of the Treasury 

and General Government 

Appropriations Act for Fiscal Year 2001 

(Public Law 106–554; H.R. 5658). 

Section 515 directs the Office of 

Management and Budget (OMB) to issue 

government-wide guidelines that 

‘‘provide policy and procedural 

guidance to Federal agencies for 

ensuring and maximizing the quality, 

objectivity, utility, and integrity of 

information (including statistical 

information) disseminated by Federal 

agencies.’’ By October 1, 2002, agencies 

must issue their own implementing 

guidelines that include ‘‘administrative 

mechanisms allowing affected persons 

to seek and obtain correction of 

information maintained and 

disseminated by the agency’’ that does 

not comply with the OMB guidelines. 

These final guidelines also reflect the 

changes OMB made to the guidelines 

issued September 28, 2001, as a result 

of receiving additional comment on the 

‘‘capable of being substantially 

reproduced’’ standard (paragraphs 

V.3.B, V.9, and V.10), which OMB 

previously issued on September 28, 

2001, on an interim final basis. 

DATES: Effective Date: January 3, 2002. 

FOR FURTHER INFORMATION CONTACT: 

Brooke J. Dickson, Office of Information 

and Regulatory Affairs, Office of 

Management and Budget, Washington, 

DC 20503. Telephone (202) 395–3785 or 

by e-mail to 

informationquality@omb.eop.gov. 

SUPPLEMENTARY INFORMATION: In section 

515(a) of the Treasury and General 

Government Appropriations Act for 

Fiscal Year 2001 (Public Law 106–554; 

H.R. 5658), Congress directed the Office 

of Management and Budget (OMB) to 

issue, by September 30, 2001, 

government-wide guidelines that 

‘‘provide policy and procedural 

guidance to Federal agencies for 

ensuring and maximizing the quality, 

objectivity, utility, and integrity of 

information (including statistical 

information) disseminated by Federal 

agencies * * *’’ Section 515(b) goes on 

to state that the OMB guidelines shall: 

‘‘(1) apply to the sharing by Federal 

agencies of, and access to, information 

disseminated by Federal agencies; and 

‘‘(2) require that each Federal agency 

to which the guidelines apply— 

‘‘(A) issue guidelines ensuring and 

maximizing the quality, objectivity, 

utility, and integrity of information 

(including statistical information) 

disseminated by the agency, by not later 

than 1 year after the date of issuance of 

the guidelines under subsection (a); 

‘‘(B) establish administrative 

mechanisms allowing affected persons 

to seek and obtain correction of 

information maintained and 

disseminated by the agency that does 

not comply with the guidelines issued 

under subsection (a); and 

‘‘(C) report periodically to the 

Director— 

‘‘(i) the number and nature of 

complaints received by the agency 

regarding the accuracy of information 

disseminated by the agency and; 

‘‘(ii) how such complaints were 

handled by the agency.’’ 

Proposed guidelines were published 

in the Federal Register on June 28, 2001 

(66 FR 34489). Final guidelines were 

published in the Federal Register on 

September 28, 2001 (66 FR 49718). The 

Supplementary Information to the final 

guidelines published in September 2001 

provides background, the underlying 

principles OMB followed in issuing the 

final guidelines, and statements of 

intent concerning detailed provisions in 

the final guidelines. 

In the final guidelilnes published in 

September 2001, OMB also requested 

additional comment on the ‘‘capable of 

being substantially reproduced’’ 

standard and the related definition of 

‘‘influential scientific or statistical 

information’’ (paragraphs V.3.B, V.9, 

and V.10), which were issued on an 

interim final basis. The final guidelines 

published today discuss the public 

comments OMB received, the OMB 

response, and amendments to the final 

guidelines published in September 

2001. 

In developing agency-specific 

guidelines, agencies should refer both to 

the Supplementary Information to the 

final guidelines published in the 

Federal Register on September 28, 2001 

(66 FR 49718), and also to the 

Supplementary Information published 

today. We stress that the three 

‘‘Underlying Principles’’ that OMB 

followed in drafting the guidelines that 

we published on September 28, 2001 

(66 FR 49719), are also applicable to the 

amended guidelines that we publish 

today. 

In accordance with section 515, OMB 

has designed the guidelines to help 

agencies ensure and maximize the 

quality, utility, objectivity and integrity 

of the information that they disseminate 

(meaning to share with, or give access 

to, the public). It is crucial that 

information Federal agencies 

disseminate meets these guidelines. In 

this respect, the fact that the Internet 

enables agencies to communicate 

information quickly and easily to a wide 

audience not only offers great benefits to 

society, but also increases the potential 

harm that can result from the 

dissemination of information that does 

not meet basic information quality 

guidelines. Recognizing the wide variety 

of information Federal agencies 

disseminate and the wide variety of 

dissemination practices that agencies 

have, OMB developed the guidelines 

with several principles in mind. 

First, OMB designed the guidelines to 

apply to a wide variety of government 

information dissemination activities 

that may range in importance and scope. 

OMB also designed the guidelines to be 

generic enough to fit all media, be they 

printed, electronic, or in other form. 

OMB sought to avoid the problems that 

would be inherent in developing 

detailed, prescriptive, ‘‘one-size-fits-all’’ 

government-wide guidelines that would 

artificially require different types of 

dissemination activities to be treated in 

the same manner. Through this 

flexibility, each agency will be able to 

incorporate the requirements of these 

OMB guidelines into the agency’s own 

information resource management and 

administrative practices. 

Second, OMB designed the guidelines 

so that agencies will meet basic 

information quality standards. Given the 

administrative mechanisms required by 

section 515 as well as the standards set 

forth in the Paperwork Reduction Act, it 

is clear that agencies should not 

disseminate substantive information 

that does not meet a basic level of 

quality. We recognize that some 

government information may need to 

meet higher or more specific 

information quality standards than 

those that would apply to other types of 

government information. The more 

important the information, the higher 

the quality standards to which it should 

be held, for example, in those situations 

involving ‘‘influential scientific, 

financial, or statistical information’’ (a 

phrase defined in these guidelines). The 

guidelines recognize, however, that 
Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 8453 

information quality comes at a cost. 

Accordingly, the agencies should weigh 

the costs (for example, including costs 

attributable to agency processing effort, 

respondent burden, maintenance of 

needed privacy, and assurances of 

suitable confidentiality) and the benefits 

of higher information quality in the 

development of information, and the 

level of quality to which the information 

disseminated will be held. 

Third, OMB designed the guidelines 

so that agencies can apply them in a 

common-sense and workable manner. It 

is important that these guidelines do not 

impose unnecessary administrative 

burdens that would inhibit agencies 

from continuing to take advantage of the 

Internet and other technologies to 

disseminate information that can be of 

great benefit and value to the public. In 

this regard, OMB encourages agencies to 

incorporate the standards and 

procedures required by these guidelines 

into their existing information resources 

management and administrative 

practices rather than create new and 

potentially duplicative or contradictory 

processes. The primary example of this 

is that the guidelines recognize that, in 

accordance with OMB Circular A–130, 

agencies already have in place wellestablished information quality 

standards and administrative 

mechanisms that allow persons to seek 

and obtain correction of information 

that is maintained and disseminated by 

the agency. Under the OMB guidelines, 

agencies need only ensure that their 

own guidelines are consistent with 

these OMB guidelines, and then ensure 

that their administrative mechanisms 

satisfy the standards and procedural 

requirements in the new agency 

guidelines. Similarly, agencies may rely 

on their implementation of the Federal 

Government’s computer security laws 

(formerly, the Computer Security Act, 

and now the computer security 

provisions of the Paperwork Reduction 

Act) to establish appropriate security 

safeguards for ensuring the ‘‘integrity’’ 

of the information that the agencies 

disseminate. 

In addition, in response to concerns 

expressed by some of the agencies, we 

want to emphasize that OMB recognizes 

that Federal agencies provide a wide 

variety of data and information. 

Accordingly, OMB understands that the 

guidelines discussed below cannot be 

implemented in the same way by each 

agency. In some cases, for example, the 

data disseminated by an agency are not 

collected by that agency; rather, the 

information the agency must provide in 

a timely manner is compiled from a 

variety of sources that are constantly 

updated and revised and may be 

confidential. In such cases, while 

agencies’ implementation of the 

guidelines may differ, the essence of the 

guidelines will apply. That is, these 

agencies must make their methods 

transparent by providing 

documentation, ensure quality by 

reviewing the underlying methods used 

in developing the data and consulting 

(as appropriate) with experts and users, 

and keep users informed about 

corrections and revisions. 

Summary of OMB Guidelines 

These guidelines apply to Federal 

agencies subject to the Paperwork 

Reduction Act (44 U.S.C. chapter 35). 

Agencies are directed to develop 

information resources management 

procedures for reviewing and 

substantiating (by documentation or 

other means selected by the agency) the 

quality (including the objectivity, 

utility, and integrity) of information 

before it is disseminated. In addition, 

agencies are to establish administrative 

mechanisms allowing affected persons 

to seek and obtain, where appropriate, 

correction of information disseminated 

by the agency that does not comply with 

the OMB or agency guidelines. 

Consistent with the underlying 

principles described above, these 

guidelines stress the importance of 

having agencies apply these standards 

and develop their administrative 

mechanisms so they can be 

implemented in a common sense and 

workable manner. Moreover, agencies 

must apply these standards flexibly, and 

in a manner appropriate to the nature 

and timeliness of the information to be 

disseminated, and incorporate them into 

existing agency information resources 

management and administrative 

practices. 

Section 515 denotes four substantive 

terms regarding information 

disseminated by Federal agencies: 

quality, utility, objectivity, and 

integrity. It is not always clear how each 

substantive term relates—or how the 

four terms in aggregate relate—to the 

widely divergent types of information 

that agencies disseminate. The 

guidelines provide definitions that 

attempt to establish a clear meaning so 

that both the agency and the public can 

readily judge whether a particular type 

of information to be disseminated does 

or does not meet these attributes. 

In the guidelines, OMB defines 

‘‘quality’’ as the encompassing term, of 

which ‘‘utility,’’ ‘‘objectivity,’’ and 

‘‘integrity’’ are the constituents. 

‘‘Utility’’ refers to the usefulness of the 

information to the intended users. 

‘‘Objectivity’’ focuses on whether the 

disseminated information is being 

presented in an accurate, clear, 

complete, and unbiased manner, and as 

a matter of substance, is accurate, 

reliable, and unbiased. ‘‘Integrity’’ refers 

to security—the protection of 

information from unauthorized access 

or revision, to ensure that the 

information is not compromised 

through corruption or falsification. OMB 

modeled the definitions of 

‘‘information,’’ ‘‘government 

information,’’ ‘‘information 

dissemination product,’’ and 

‘‘dissemination’’ on the longstanding 

definitions of those terms in OMB 

Circular A–130, but tailored them to fit 

into the context of these guidelines. 

In addition, Section 515 imposes two 

reporting requirements on the agencies. 

The first report, to be promulgated no 

later than October 1, 2002, must provide 

the agency’s information quality 

guidelines that describe administrative 

mechanisms allowing affected persons 

to seek and obtain, where appropriate, 

correction of disseminated information 

that does not comply with the OMB and 

agency guidelines. The second report is 

an annual fiscal year report to OMB (to 

be first submitted on January 1, 2004) 

providing information (both quantitative 

and qualitative, where appropriate) on 

the number, nature, and resolution of 

complaints received by the agency 

regarding its perceived or confirmed 

failure to comply with these OMB and 

agency guidelines. 

Public Comments and OMB Response 

Applicability of Guidelines. Some 

comments raised concerns about the 

applicability of these guidelines, 

particularly in the context of scientific 

research conducted by Federally 

employed scientists or Federal grantees 

who publish and communicate their 

research findings in the same manner as 

their academic colleagues. OMB 

believes that information generated and 

disseminated in these contexts is not 

covered by these guidelines unless the 

agency represents the information as, or 

uses the information in support of, an 

official position of the agency. 

As a general matter, these guidelines 

apply to ‘‘information’’ that is 

‘‘disseminated’’ by agencies subject to 

the Paperwork Reduction Act (44 U.S.C. 

3502(1)). See paragraphs II, V.5 and V.8. 

The definitions of ‘‘information’’ and 

‘‘dissemination’’ establish the scope of 

the applicability of these guidelines. 

‘‘Information’’ means ‘‘any 

communication or representation of 

knowledge such as facts or data * * *’’ 

This definition of information in 

paragraph V.5 does ‘‘not include 

opinions, where the agency’s 

presentation makes it clear that what is 
8454 Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 

being offered is someone’s opinion 

rather than fact or the agency’s views.’’ 

‘‘Dissemination’’ is defined to mean 

‘‘agency initiated or sponsored 

distribution of information to the 

public.’’ As used in paragraph V.8, 

‘‘agency INITIATED * * * distribution 

of information to the public’’ refers to 

information that the agency 

disseminates, e.g., a risk assessment 

prepared by the agency to inform the 

agency’s formulation of possible 

regulatory or other action. In addition, 

if an agency, as an institution, 

disseminates information prepared by 

an outside party in a manner that 

reasonably suggests that the agency 

agrees with the information, this 

appearance of having the information 

represent agency views makes agency 

dissemination of the information subject 

to these guidelines. By contrast, an 

agency does not ‘‘initiate’’ the 

dissemination of information when a 

Federally employed scientist or Federal 

grantee or contractor publishes and 

communicates his or her research 

findings in the same manner as his or 

her academic colleagues, even if the 

Federal agency retains ownership or 

other intellectual property rights 

because the Federal government paid for 

the research. To avoid confusion 

regarding whether the agency agrees 

with the information (and is therefore 

disseminating it through the employee 

or grantee), the researcher should 

include an appropriate disclaimer in the 

publication or speech to the effect that 

the ‘‘views are mine, and do not 

necessarily reflect the view’’ of the 

agency. 

Similarly, as used in paragraph V.8., 

‘‘agency * * * SPONSORED 

distribution of information to the 

public’’ refers to situations where an 

agency has directed a third-party to 

disseminate information, or where the 

agency has the authority to review and 

approve the information before release. 

Therefore, for example, if an agency 

through a procurement contract or a 

grant provides for a person to conduct 

research, and then the agency directs 

the person to disseminate the results (or 

the agency reviews and approves the 

results before they may be 

disseminated), then the agency has 

‘‘sponsored’’ the dissemination of this 

information. By contrast, if the agency 

simply provides funding to support 

research, and it the researcher (not the 

agency) who decides whether to 

disseminate the results and—if the 

results are to be released—who 

determines the content and presentation 

of the dissemination, then the agency 

has not ‘‘sponsored’’ the dissemination 

even though it has funded the research 

and even if the Federal agency retains 

ownership or other intellectual property 

rights because the Federal government 

paid for the research. To avoid 

confusion regarding whether the agency 

is sponsoring the dissemination, the 

researcher should include an 

appropriate disclaimer in the 

publication or speech to the effect that 

the ‘‘views are mine, and do not 

necessarily reflect the view’’ of the 

agency. On the other hand, subsequent 

agency dissemination of such 

information requires that the 

information adhere to the agency’s 

information quality guidelines. In sum, 

these guidelines govern an agency’s 

dissemination of information, but 

generally do not govern a third-party’s 

dissemination of information (the 

exception being where the agency is 

essentially using the third-party to 

disseminate information on the agency’s 

behalf). Agencies, particularly those that 

fund scientific research, are encouraged 

to clarify the applicability of these 

guidelines to the various types of 

information they and their employees 

and grantees disseminate. 

Paragraph V.8 also states that the 

definition of ‘‘dissemination’’ does not 

include ‘‘* * * distribution limited to 

correspondence with individuals or 

persons, press releases, archival records, 

public filings, subpoenas or adjudicative 

processes.’’ The exemption from the 

definition of ‘‘dissemination’’ for 

‘‘adjudicative processes’’ is intended to 

exclude, from the scope of these 

guidelines, the findings and 

determinations that an agency makes in 

the course of adjudications involving 

specific parties. There are wellestablished procedural safeguards and 

rights to address the quality of 

adjudicatory decisions and to provide 

persons with an opportunity to contest 

decisions. These guidelines do not 

impose any additional requirements on 

agencies during adjudicative 

proceedings and do not provide parties 

to such adjudicative proceedings any 

additional rights of challenge or appeal. 

The Presumption Favoring PeerReviewed Information.As a general 

matter, in the scientific and research 

context, we regard technical information 

that has been subjected to formal, 

independent, external peer review as 

presumptively objective. As the 

guidelines state in paragraph V.3.b.i: ‘‘If 

data and analytic results have been 

subjected to formal, independent, 

external peer review, the information 

may generally be presumed to be of 

acceptable objectivity.’’ An example of a 

formal, independent, external peer 

review is the review process used by 

scientific journals. 

Most comments approved of the 

prominent role that peer review plays in 

the OMB guidelines. Some comments 

contended that peer review was not 

accepted as a universal standard that 

incorporates an established, practiced, 

and sufficient level of objectivity. Other 

comments stated that the guidelines 

would be better clarified by making peer 

review one of several factors that an 

agency should consider in assessing the 

objectivity (and quality in general) of 

original research. In addition, several 

comments noted that peer review does 

not establish whether analytic results 

are capable of being substantially 

reproduced. In light of the comments, 

the final guidelines in new paragraph 

V.3.b.i qualify the presumption in favor 

of peer-reviewed information as follows: 

‘‘However, this presumption is 

rebuttable based on a persuasive 

showing by the petitioner in a particular 

instance.’’ 

We believe that transparency is 

important for peer review, and these 

guidelines set minimum standards for 

the transparency of agency-sponsored 

peer review. As we state in new 

paragraph V.3.b.i: ‘‘If data and analytic 

results have been subjected to formal, 

independent, external peer review, the 

information may generally be presumed 

to be of acceptable objectivity. However, 

this presumption is rebuttable based on 

a persuasive showing by the petitioner 

in a particular instance. If agencysponsored peer review is employed to 

help satisfy the objectivity standard, the 

review process employed shall meet the 

general criteria for competent and 

credible peer review recommended by 

OMB–OIRA to the President’s 

Management Council (9/20/01) (http:// 

www.whitehouse.gov/omb/inforeg/ 

oira_review-process.html), namely, ‘that 

(a) peer reviewers be selected primarily 

on the basis of necessary technical 

expertise, (b) peer reviewers be expected 

to disclose to agencies prior technical/ 

policy positions they may have taken on 

the issues at hand, (c) peer reviewers be 

expected to disclose to agencies their 

sources of personal and institutional 

funding (private or public sector), and 

(d) peer reviews be conducted in an 

open and rigorous manner.’ ’’ 

The importance of these general 

criteria for competent and credible peer 

review has been supported by a number 

of expert bodies. For example, ‘‘the 

work of fully competent peer-review 

panels can be undermined by 

allegations of conflict of interest and 

bias. Therefore, the best interests of the 

Board are served by effective policies 

and procedures regarding potential 

conflicts of interest, impartiality, and 

panel balance.’’ (EPA’s Science Advisory 
Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 8455 

Board Panels: Improved Policies and 

Procedures Needed to Ensure 

Independence and Balance, GAO–01– 

536, General Accounting Office, 

Washington, DC, June 2001, page 19.) 

As another example, ‘‘risk analyses 

should be peer-reviewed and 

accessible—both physically and 

intellectually—so that decision-makers 

at all levels will be able to respond 

critically to risk characterizations. The 

intensity of the peer reviews should be 

commensurate with the significance of 

the risk or its management 

implications.’’ (Setting Priorities, 

Getting Results: A New Direction for 

EPA, Summary Report, National 

Academy of Public Administration, 

Washington, DC, April 1995, page 23.) 

These criteria for peer reviewers are 

generally consistent with the practices 

now followed by the National Research 

Council of the National Academy of 

Sciences. In considering these criteria 

for peer reviewers, we note that there 

are many types of peer reviews and that 

agency guidelines concerning the use of 

peer review should tailor the rigor of 

peer review to the importance of the 

information involved. More generally, 

agencies should define their peer-review 

standards in appropriate ways, given the 

nature and importance of the 

information they disseminate. 

Is Journal Peer Review Always 

Sufficient? Some comments argued that 

journal peer review should be adequate 

to demonstrate quality, even for 

influential information that can be 

expected to have major effects on public 

policy. OMB believes that this position 

overstates the effectiveness of journal 

peer review as a quality-control 

mechanism. 

Although journal peer review is 

clearly valuable, there are cases where 

flawed science has been published in 

respected journals. For example, the 

NIH Office of Research Integrity recently 

reported the following case regarding 

environmental health research: 

‘‘Based on the report of an investigation 

conducted by [XX] University, dated July 16, 

1999, and additional analysis conducted by 

ORI in its oversight review, the US Public 

Health Service found that Dr. [X] engaged in 

scientific misconduct. Dr. [X] committed 

scientific misconduct by intentionally 

falsifying the research results published in 

the journal SCIENCE and by providing 

falsified and fabricated materials to 

investigating officials at [XX] University in 

response to a request for original data to 

support the research results and conclusions 

report in the SCIENCE paper. In addition, 

PHS finds that there is no original data or 

other corroborating evidence to support the 

research results and conclusions reported in 

the SCIENCE paper as a whole.’’ (66 FR 

52137, October 12, 2001). 

Although such cases of falsification 

are presumably rare, there is a 

significant scholarly literature 

documenting quality problems with 

articles published in peer-reviewed 

research. ‘‘In a [peer-reviewed] metaanalysis that surprised many—and some 

doubt—researchers found little evidence 

that peer review actually improves the 

quality of research papers.’’ (See, e.g., 

Science, Vol. 293, page 2187 (September 

21, 2001.)) In part for this reason, many 

agencies have already adopted peer 

review and science advisory practices 

that go beyond journal peer review. See, 

e.g., Sheila Jasanoff, The Fifth Branch: 

Science Advisers as Policy Makers, 

Cambridge, MA, Harvard University 

Press, 1990; Mark R. Powell, Science at 

EPA: Information in the Regulatory 

Process. Resources for the Future, 

Washington, DC., 1999, pages 138–139; 

151–153; Implementation of the 

Environmental Protection Agency’s Peer 

Review Program: An SAB Evaluation of 

Three Reviews, EPA–SAB–RSAC–01– 

009, A Review of the Research Strategies 

Advisory Committee (RSAC) of the EPA 

Science Advisory Board (SAB), 

Washington, DC., September 26, 2001. 

For information likely to have an 

important public policy or private sector 

impact, OMB believes that additional 

quality checks beyond peer review are 

appropriate. 

Definition of ‘‘Influential’’. OMB 

guidelines apply stricter quality 

standards to the dissemination of 

information that is considered 

‘‘influential.’’ Comments noted that the 

breadth of the definition of ‘‘influential’’ 

in interim final paragraph V.9 requires 

much speculation on the part of 

agencies. 

We believe that this criticism has 

merit and have therefore narrowed the 

definition. In this narrower definition, 

‘‘influential’’, when used in the phrase 

‘‘influential scientific, financial, or 

statistical information’’, is amended to 

mean that ‘‘the agency can reasonably 

determine that dissemination of the 

information will have or does have a 

clear and substantial impact on 

important public policies or important 

private sector decisions.’’ The intent of 

the new phrase ‘‘clear and substantial’’ 

is to reduce the need for speculation on 

the part of agencies. We added the 

present tense—‘‘or does have’’—to this 

narrower definition because on 

occasion, an information dissemination 

may occur simultaneously with a 

particular policy change. In response to 

a public comment, we added an explicit 

reference to ‘‘financial’’ information as 

consistent with our original intent. 

Given the differences in the many 

Federal agencies covered by these 

guidelines, and the differences in the 

nature of the information they 

disseminate, we also believe it will be 

helpful if agencies elaborate on this 

definition of ‘‘influential’’ in the context 

of their missions and duties, with due 

consideration of the nature of the 

information they disseminate. As we 

state in amended paragraph V.9, ‘‘Each 

agency is authorized to define 

‘influential’ in ways appropriate for it 

given the nature and multiplicity of 

issues for which the agency is 

responsible.’’ 

Reproducibility. As we state in new 

paragraph V.3.b.ii: ‘‘If an agency is 

responsible for disseminating influential 

scientific, financial, or statistical 

information, agency guidelines shall 

include a high degree of transparency 

about data and methods to facilitate the 

reproducibility of such information by 

qualified third parties.’’ OMB believes 

that a reproducibility standard is 

practical and appropriate for 

information that is considered 

‘‘influential’’, as defined in paragraph 

V.9—that ‘‘will have or does have a 

clear and substantial impact on 

important public policies or important 

private sector decisions.’’ The 

reproducibility standard applicable to 

influential scientific, financial, or 

statistical information is intended to 

ensure that information disseminated by 

agencies is sufficiently transparent in 

terms of data and methods of analysis 

that it would be feasible for a replication 

to be conducted. The fact that the use 

of original and supporting data and 

analytic results have been deemed 

‘‘defensible’’ by peer-review procedures 

does not necessarily imply that the 

results are transparent and replicable. 

Reproducibility of Original and 

Supporting Data. Several of the 

comments objected to the exclusion of 

original and supporting data from the 

reproducibility requirements. 

Comments instead suggested that OMB 

should apply the reproducibility 

standard to original data, and that OMB 

should provide flexibility to the 

agencies in determining what 

constitutes ‘‘original and supporting’’ 

data. OMB agrees and asks that agencies 

consider, in developing their own 

guidelines, which categories of original 

and supporting data should be subject to 

the reproducibility standard and which 

should not. To help in resolving this 

issue, we also ask agencies to consult 

directly with relevant scientific and 

technical communities on the feasibility 

of having the selected categories of 

original and supporting data subject to 

the reproducibility standard. Agencies 

are encouraged to address ethical, 

feasibility, and confidentiality issues 
8456 Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 

with care. As we state in new paragraph 

V.3.b.ii.A, ‘‘Agencies may identify, in 

consultation with the relevant scientific 

and technical communities, those 

particular types of data that can 

practicably be subjected to a 

reproducibility requirement, given 

ethical, feasibility, or confidentiality 

constraints.’’ Further, as we state in our 

expanded definition of 

‘‘reproducibility’’ in paragraph V.10, ‘‘If 

agencies apply the reproducibility test 

to specific types of original or 

supporting data, the associated 

guidelines shall provide relevant 

definitions of reproducibility (e.g., 

standards for replication of laboratory 

data).’’ OMB urges caution in the 

treatment of original and supporting 

data because it may often be impractical 

or even impermissible or unethical to 

apply the reproducibility standard to 

such data. For example, it may not be 

ethical to repeat a ‘‘negative’’ 

(ineffective) clinical (therapeutic) 

experiment and it may not be feasible to 

replicate the radiation exposures 

studied after the Chernobyl accident. 

When agencies submit their draft agency 

guidelines for OMB review, agencies 

should include a description of the 

extent to which the reproducibility 

standard is applicable and reflect 

consultations with relevant scientific 

and technical communities that were 

used in developing guidelines related to 

applicability of the reproducibility 

standard to original and supporting 

data. 

It is also important to emphasize that 

the reproducibility standard does not 

apply to all original and supporting data 

disseminated by agencies. As we state in 

new paragraph V.3.b.ii.A, ‘‘With regard 

to original and supporting data related 

[to influential scientific, financial, or 

statistical information], agency 

guidelines shall not require that all 

disseminated data be subjected to a 

reproducibility requirement.’’ In 

addition, we encourage agencies to 

address how greater transparency can be 

achieved regarding original and 

supporting data. As we also state in new 

paragraph V.3.b.ii.A, ‘‘It is understood 

that reproducibility of data is an 

indication of transparency about 

research design and methods and thus 

a replication exercise (i.e., a new 

experiment, test, or sample) shall not be 

required prior to each dissemination.’’ 

Agency guidelines need to achieve a 

high degree of transparency about data 

even when reproducibility is not 

required. 

Reproducibility of Analytic Results. 

Many public comments were critical of 

the reproducibility standard and 

expressed concern that agencies would 

be required to reproduce each analytical 

result before it is disseminated. While 

several comments commended OMB for 

establishing an appropriate balance in 

the ‘‘capable of being substantially 

reproduced’’ standard, others 

considered this standard to be 

inherently subjective. There were also 

comments that suggested the standard 

would cause more burden for agencies. 

It is not OMB’s intent that each 

agency must reproduce each analytic 

result before it is disseminated. The 

purpose of the reproducibility standard 

is to cultivate a consistent agency 

commitment to transparency about how 

analytic results are generated: the 

specific data used, the various 

assumptions employed, the specific 

analytic methods applied, and the 

statistical procedures employed. If 

sufficient transparency is achieved on 

each of these matters, then an analytic 

result should meet the ‘‘capable of being 

substantially reproduced’’ standard. 

While there is much variation in types 

of analytic results, OMB believes that 

reproducibility is a practical standard to 

apply to most types of analytic results. 

As we state in new paragraph V.3.b.ii.B, 

‘‘With regard to analytic results related 

[to influential scientific, financial, or 

statistical information], agency 

guidelines shall generally require 

sufficient transparency about data and 

methods that an independent reanalysis 

could be undertaken by a qualified 

member of the public. These 

transparency standards apply to agency 

analysis of data from a single study as 

well as to analyses that combine 

information from multiple studies.’’ We 

elaborate upon this principle in our 

expanded definition of 

‘‘reproducibility’’ in paragraph V.10: 

‘‘With respect to analytic results, 

‘capable of being substantially 

reproduced’ means that independent 

analysis of the original or supporting 

data using identical methods would 

generate similar analytic results, subject 

to an acceptable degree of imprecision 

or error.’’ 

Even in a situation where the original 

and supporting data are protected by 

confidentiality concerns, or the analytic 

computer models or other research 

methods may be kept confidential to 

protect intellectual property, it may still 

be feasible to have the analytic results 

subject to the reproducibility standard. 

For example, a qualified party, 

operating under the same 

confidentiality protections as the 

original analysts, may be asked to use 

the same data, computer model or 

statistical methods to replicate the 

analytic results reported in the original 

study. See, e.g., ‘‘Reanalysis of the 

Harvard Six Cities Study and the 

American Cancer Society Study of 

Particulate Air Pollution and Mortality,’’ 

A Special Report of the Health Effects 

Institute’s Particle Epidemiology 

Reanalysis Project, Cambridge, MA, 

2000. 

The primary benefit of public 

transparency is not necessarily that 

errors in analytic results will be 

detected, although error correction is 

clearly valuable. The more important 

benefit of transparency is that the public 

will be able to assess how much an 

agency’s analytic result hinges on the 

specific analytic choices made by the 

agency. Concreteness about analytic 

choices allows, for example, the 

implications of alternative technical 

choices to be readily assessed. This type 

of sensitivity analysis is widely 

regarded as an essential feature of highquality analysis, yet sensitivity analysis 

cannot be undertaken by outside parties 

unless a high degree of transparency is 

achieved. The OMB guidelines do not 

compel such sensitivity analysis as a 

necessary dimension of quality, but the 

transparency achieved by 

reproducibility will allow the public to 

undertake sensitivity studies of interest. 

We acknowledge that confidentiality 

concerns will sometimes preclude 

public access as an approach to 

reproducibility. In response to public 

comment, we have clarified that such 

concerns do include interests in 

‘‘intellectual property.’’ To ensure that 

the OMB guidelines have sufficient 

flexibility with regard to analytic 

transparency, OMB has, in new 

paragraph V.3.b.ii.B.i, provided agencies 

an alternative approach for classes or 

types of analytic results that cannot 

practically be subject to the 

reproducibility standard. ‘‘[In those 

situations involving influential 

scientific, financial, or statistical 

information * * * ] making the data and 

methods publicly available will assist in 

determining whether analytic results are 

reproducible. However, the objectivity 

standard does not override other 

compelling interests such as privacy, 

trade secrets, intellectual property, and 

other confidentiality protections. ’’ 

Specifically, in cases where 

reproducibility will not occur due to 

other compelling interests, we expect 

agencies (1) to perform robustness 

checks appropriate to the importance of 

the information involved, e.g., 

determining whether a specific statistic 

is sensitive to the choice of analytic 

method, and, accompanying the 

information disseminated, to document 

their efforts to assure the needed 

robustness in information quality, and 

(2) address in their guidelines the 
Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 8457 

degree to which they anticipate the 

opportunity for reproducibility to be 

limited by the confidentiality of 

underlying data. As we state in new 

paragraph V.3.b.ii.B.ii, ‘‘In situations 

where public access to data and 

methods will not occur due to other 

compelling interests, agencies shall 

apply especially rigorous robustness 

checks to analytic results and document 

what checks were undertaken. Agency 

guidelines shall, however, in all cases, 

require a disclosure of the specific data 

sources that have been used and the 

specific quantitative methods and 

assumptions that have been employed.’’ 

Given the differences in the many 

Federal agencies covered by these 

guidelines, and the differences in 

robustness checks and the level of detail 

for documentation thereof that might be 

appropriate for different agencies, we 

also believe it will be helpful if agencies 

elaborate on these matters in the context 

of their missions and duties, with due 

consideration of the nature of the 

information they disseminate. As we 

state in new paragraph V.3.b.ii.B.ii, 

‘‘Each agency is authorized to define the 

type of robustness checks, and the level 

of detail for documentation thereof, in 

ways appropriate for it given the nature 

and multiplicity of issues for which the 

agency is responsible.’’ 

We leave the determination of the 

appropriate degree of rigor to the 

discretion of agencies and the relevant 

scientific and technical communities 

that work with the agencies. We do, 

however, establish a general standard 

for the appropriate degree of rigor in our 

expanded definition of 

‘‘reproducibility’’ in paragraph V.10: 

‘‘ ‘Reproducibility’ means that the 

information is capable of being 

substantially reproduced, subject to an 

acceptable degree of imprecision. For 

information judged to have more (less) 

important impacts, the degree of 

imprecision that is tolerated is reduced 

(increased).’’ OMB will review each 

agency’s treatment of this issue when 

reviewing the agency guidelines as a 

whole. 

Comments also expressed concerns 

regarding interim final paragraph 

V.3.B.iii, ‘‘making the data and models 

publicly available will assist in 

determining whether analytic results are 

capable of being substantially 

reproduced,’’ and whether it could be 

interpreted to constitute public 

dissemination of these materials, 

rendering moot the reproducibility test. 

(For the equivalent provision, see new 

paragraph V.3.b.ii.B.i.) The OMB 

guidelines do not require agencies to 

reproduce each disseminated analytic 

result by independent reanalysis. Thus, 

public dissemination of data and 

models per se does not mean that the 

analytic result has been reproduced. It 

means only that the result should be 

CAPABLE of being reproduced. The 

transparency associated with this 

capability of reproduction is what the 

OMB guidelines are designed to 

achieve. 

We also want to build on a general 

observation that we made in our final 

guidelines published in September 

2001. In those guidelines we stated: ‘‘... 

in those situations involving influential 

scientific[, financial,] or statistical 

information, the substantial 

reproducibility standard is added as a 

quality standard above and beyond 

some peer review quality standards’’ (66 

FR 49722 (September 28, 2001)). A 

hypothetical example may serve to 

illustrate this point. Assume that two 

Federal agencies initiated or sponsored 

the dissemination of five scientific 

studies after October 1, 2002 (see 

paragraph III.4) that were, before 

dissemination, subjected to formal, 

independent, external peer review, i.e., 

that met the presumptive standard for 

‘‘objectivity’’ under paragraph V.3.b.i. 

Further assume, at the time of 

dissemination, that neither agency 

reasonably expected that the 

dissemination of any of these studies 

would have ‘‘a clear and substantial 

impact’’ on important public policies, 

i.e., that these studies were not 

considered ‘‘influential’’ under 

paragraph V.9, and thus not subject to 

the reproducibility standards in 

paragraphs V.3.b.ii.A or B. Then 

assume, two years later, in 2005, that 

one of the agencies decides to issue an 

important and far-reaching regulation 

based clearly and substantially on the 

agency’s evaluation of the analytic 

results set forth in these five studies and 

that such agency reliance on these five 

studies as published in the agency’s 

notice of proposed rulemaking would 

constitute dissemination of these five 

studies. These guidelines would require 

the rulemaking agency, prior to 

publishing the notice of proposed 

rulemaking, to evaluate these five 

studies to determine if the analytic 

results stated therein would meet the 

‘‘capable of being substantially 

reproduced’’ standards in paragraph 

V.3.b.ii.B and, if necessary, related 

standards governing original and 

supporting data in paragraph V.3.b.ii.A. 

If the agency were to decide that any of 

the five studies would not meet the 

reproducibility standard, the agency 

may still rely on them but only if they 

satisfy the transparency standard and— 

as applicable—the disclosure of 

robustness checks required by these 

guidelines. Otherwise, the agency 

should not disseminate any of the 

studies that did not meet the applicable 

standards in the guidelines at the time 

it publishes the notice of proposed 

rulemaking. 

Some comments suggested that OMB 

consider replacing the reproducibility 

standard with a standard concerning 

‘‘confirmation’’ of results for influential 

scientific and statistical information. 

Although we encourage agencies to 

consider ‘‘confirmation’’ as a relevant 

standard—at least in some cases—for 

assessing the objectivity of original and 

supporting data, we believe that 

‘‘confirmation’’ is too stringent a 

standard to apply to analytic results. 

Often the regulatory impact analysis 

prepared by an agency for a major rule, 

for example, will be the only formal 

analysis of an important subject. It 

would be unlikely that the results of the 

regulatory impact analysis had already 

been confirmed by other analyses. The 

‘‘capable of being substantially 

reproduced’’ standard is less stringent 

than a ‘‘confirmation’’ standard because 

it simply requires that an agency’s 

analysis be sufficiently transparent that 

another qualified party could replicate it 

through reanalysis. 

Health, Safety, and Environmental 

Information. We note, in the scientific 

context, that in 1996 the Congress, for 

health decisions under the Safe 

Drinking Water Act, adopted a basic 

standard of quality for the use of science 

in agency decisionmaking. Under 42 

U.S.C. 300g–1(b)(3)(A), an agency is 

directed, ‘‘to the degree that an Agency 

action is based on science,’’ to use ‘‘(i) 

the best available, peer-reviewed 

science and supporting studies 

conducted in accordance with sound 

and objective scientific practices; and 

(ii) data collected by accepted methods 

or best available methods (if the 

reliability of the method and the nature 

of the decision justifies use of the 

data).’’ 

We further note that in the 1996 

amendments to the Safe Drinking Water 

Act, Congress adopted a basic quality 

standard for the dissemination of public 

information about risks of adverse 

health effects. Under 42 U.S.C. 300g– 

1(b)(3)(B), the agency is directed, ‘‘to 

ensure that the presentation of 

information [risk] effects is 

comprehensive, informative, and 

understandable.’’ The agency is further 

directed, ‘‘in a document made available 

to the public in support of a regulation 

[to] specify, to the extent practicable— 

(i) each population addressed by any 

estimate [of applicable risk effects]; (ii) 

the expected risk or central estimate of 
8458 Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 

risk for the specific populations 

[affected]; (iii) each appropriate upperbound or lower-bound estimate of risk; 

(iv) each significant uncertainty 

identified in the process of the 

assessment of [risk] effects and the 

studies that would assist in resolving 

the uncertainty; and (v) peer-reviewed 

studies known to the [agency] that 

support, are directly relevant to, or fail 

to support any estimate of [risk] effects 

and the methodology used to reconcile 

inconsistencies in the scientific data.’’ 

As suggested in several comments, we 

have included these congressional 

standards directly in new paragraph 

V.3.b.ii.C, and made them applicable to 

the information disseminated by all the 

agencies subject to these guidelines: 

‘‘With regard to analysis of risks to 

human health, safety and the 

environment maintained or 

disseminated by the agencies, agencies 

shall either adopt or adapt the quality 

principles applied by Congress to risk 

information used and disseminated 

pursuant to the Safe Drinking Water Act 

Amendments of 1996 (42 U.S.C. 300g– 

1(b)(3)(A) & (B)).’’ The word ‘‘adapt’’ is 

intended to provide agencies flexibility 

in applying these principles to various 

types of risk assessment. 

Comments also argued that the 

continued flow of vital information from 

agencies responsible for disseminating 

health and medical information to 

medical providers, patients, and the 

public may be disrupted due to these 

peer review and reproducibility 

standards. OMB responded by adding to 

new paragraph V.3.b.ii.C: ‘‘Agencies 

responsible for dissemination of vital 

health and medical information shall 

interpret the reproducibility and peerreview standards in a manner 

appropriate to assuring the timely flow 

of vital information from agencies to 

medical providers, patients, health 

agencies, and the public. Information 

quality standards may be waived 

temporarily by agencies under urgent 

situations (e.g., imminent threats to 

public health or homeland security) in 

accordance with the latitude specified 

in agency-specific guidelines.’’ 

Administrative Correction 

Mechanisms. In addition to commenting 

on the substantive standards in these 

guidelines, many of the comments noted 

that the OMB guidelines on the 

administrative correction of information 

do not specify a time period in which 

the agency investigation and response 

must be made. OMB has added the 

following new paragraph III.3.i to direct 

agencies to specify appropriate time 

periods in which the investigation and 

response need to be made. ‘‘Agencies 

shall specify appropriate time periods 

for agency decisions on whether and 

how to correct the information, and 

agencies shall notify the affected 

persons of the corrections made.’’ 

Several comments stated that the 

OMB guidelines needed to direct 

agencies to consider incorporating an 

administrative appeal process into their 

administrative mechanisms for the 

correction of information. OMB agreed, 

and added the following new paragraph 

III.3.ii: ‘‘If the person who requested the 

correction does not agree with the 

agency’s decision (including the 

corrective action, if any), the person 

may file for reconsideration within the 

agency. The agency shall establish an 

administrative appeal process to review 

the agency’s initial decision, and specify 

appropriate time limits in which to 

resolve such requests for 

reconsideration.’’ Recognizing that 

many agencies already have a process in 

place to respond to public concerns, it 

is not necessarily OMB’s intent to 

require these agencies to establish a new 

or different process. Rather, our intent is 

to ensure that agency guidelines specify 

an objective administrative appeal 

process that, upon furthercomplaint by 

the affected person, reviews an agency’s 

decision to disagree with the correction 

request. An objective process will 

ensure that the office that originally 

disseminates the information does not 

have responsibility for both the initial 

response and resolution of a 

disagreement. In addition, the agency 

guidelines should specify that if the 

agency believes other agencies may have 

an interest in the resolution of any 

administrative appeal, the agency 

should consult with those other 

agencies about their possible interest. 

Overall, OMB does not envision 

administrative mechanisms that would 

burden agencies with frivolous claims. 

Instead, the correction process should 

serve to address the genuine and valid 

needs of the agency and its constituents 

without disrupting agency processes. 

Agencies, in making their determination 

of whether or not to correct information, 

may reject claims made in bad faith or 

without justification, and are required to 

undertake only the degree of correction 

that they conclude is appropriate for the 

nature and timeliness of the information 

involved, and explain such practices in 

their annual fiscal year reports to OMB. 

OMB’s issuance of these final 

guidelines is the beginning of an 

evolutionary process that will include 

draft agency guidelines, public 

comment, final agency guidelines, 

development of experience with OMB 

and agency guidelines, and continued 

refinement of both OMB and agency 

guidelines. Just as OMB requested 

public comment before issuing these 

final guidelines, OMB will refine these 

guidelines as experience develops and 

further public comment is obtained. 

Dated: December 21, 2001. 

John D. Graham, 

Administrator, Office of Information and 

Regulatory Affairs. 

Guidelines for Ensuring and 

Maximizing the Quality, Objectivity, 

Utility, and Integrity of Information 

Disseminated by Federal Agencies 

I. OMB Responsibilities 

Section 515 of the Treasury and 

General Government Appropriations 

Act for FY2001 (Public Law 106–554) 

directs the Office of Management and 

Budget to issue government-wide 

guidelines that provide policy and 

procedural guidance to Federal agencies 

for ensuring and maximizing the 

quality, objectivity, utility, and integrity 

of information, including statistical 

information, disseminated by Federal 

agencies. 

II. Agency Responsibilities 

Section 515 directs agencies subject to 

the Paperwork Reduction Act (44 U.S.C. 

3502(1)) to— 

1. Issue their own information quality 

guidelines ensuring and maximizing the 

quality, objectivity, utility, and integrity 

of information, including statistical 

information, disseminated by the agency 

no later than one year after the date of 

issuance of the OMB guidelines; 

2. Establish administrative 

mechanisms allowing affected persons 

to seek and obtain correction of 

information maintained and 

disseminated by the agency that does 

not comply with these OMB guidelines; 

and 

3. Report to the Director of OMB the 

number and nature of complaints 

received by the agency regarding agency 

compliance with these OMB guidelines 

concerning the quality, objectivity, 

utility, and integrity of information and 

how such complaints were resolved. 

III. Guidelines for Ensuring and 

Maximizing the Quality, Objectivity, 

Utility, and Integrity of Information 

Disseminated by Federal Agencies 

1. Overall, agencies shall adopt a 

basic standard of quality (including 

objectivity, utility, and integrity) as a 

performance goal and should take 

appropriate steps to incorporate 

information quality criteria into agency 

information dissemination practices. 

Quality is to be ensured and established 

at levels appropriate to the nature and 

timeliness of the information to be 

disseminated. Agencies shall adopt 
Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 8459 

specific standards of quality that are 

appropriate for the various categories of 

information they disseminate. 

2. As a matter of good and effective 

agency information resources 

management, agencies shall develop a 

process for reviewing the quality 

(including the objectivity, utility, and 

integrity) of information before it is 

disseminated. Agencies shall treat 

information quality as integral to every 

step of an agency’s development of 

information, including creation, 

collection, maintenance, and 

dissemination. This process shall enable 

the agency to substantiate the quality of 

the information it has disseminated 

through documentation or other means 

appropriate to the information. 

3. To facilitate public review, agencies 

shall establish administrative 

mechanisms allowing affected persons 

to seek and obtain, where appropriate, 

timely correction of information 

maintained and disseminated by the 

agency that does not comply with OMB 

or agency guidelines. These 

administrative mechanisms shall be 

flexible, appropriate to the nature and 

timeliness of the disseminated 

information, and incorporated into 

agency information resources 

management and administrative 

practices. 

i. Agencies shall specify appropriate 

time periods for agency decisions on 

whether and how to correct the 

information, and agencies shall notify 

the affected persons of the corrections 

made. 

ii. If the person who requested the 

correction does not agree with the 

agency’s decision (including the 

corrective action, if any), the person 

may file for reconsideration within the 

agency. The agency shall establish an 

administrative appeal process to review 

the agency’s initial decision, and specify 

appropriate time limits in which to 

resolve such requests for 

reconsideration. 

4. The agency’s pre-dissemination 

review, under paragraph III.2, shall 

apply to information that the agency 

first disseminates on or after October 1, 

2002. The agency’s administrative 

mechanisms, under paragraph III.3., 

shall apply to information that the 

agency disseminates on or after October 

1, 2002, regardless of when the agency 

first disseminated the information. 

IV. Agency Reporting Requirements 

1. Agencies must designate the Chief 

Information Officer or another official to 

be responsible for agency compliance 

with these guidelines. 

2. The agency shall respond to 

complaints in a manner appropriate to 

the nature and extent of the complaint. 

Examples of appropriate responses 

include personal contacts via letter or 

telephone, form letters, press releases or 

mass mailings that correct a widely 

disseminated error or address a 

frequently raised complaint. 

3. Each agency must prepare a draft 

report, no later than April 1, 2002, 

providing the agency’s information 

quality guidelines and explaining how 

such guidelines will ensure and 

maximize the quality, objectivity, 

utility, and integrity of information, 

including statistical information, 

disseminated by the agency. This report 

must also detail the administrative 

mechanisms developed by that agency 

to allow affected persons to seek and 

obtain appropriate correction of 

information maintained and 

disseminated by the agency that does 

not comply with the OMB or the agency 

guidelines. 

4. The agency must publish a notice 

of availability of this draft report in the 

Federal Register, and post this report on 

the agency’s website, to provide an 

opportunity for public comment. 

5. Upon consideration of public 

comment and after appropriate revision, 

the agency must submit this draft report 

to OMB for review regarding 

consistency with these OMB guidelines 

no later than July 1, 2002. Upon 

completion of that OMB review and 

completion of this report, agencies must 

publish notice of the availability of this 

report in its final form in the Federal 

Register, and post this report on the 

agency’s web site no later than October 

1, 2002. 

6. On an annual fiscal-year basis, each 

agency must submit a report to the 

Director of OMB providing information 

(both quantitative and qualitative, 

where appropriate) on the number and 

nature of complaints received by the 

agency regarding agency compliance 

with these OMB guidelines and how 

such complaints were resolved. 

Agencies must submit these reports no 

later than January 1 of each following 

year, with the first report due January 1, 

2004. 

V. Definitions 

1. ‘‘Quality’’ is an encompassing term 

comprising utility, objectivity, and 

integrity. Therefore, the guidelines 

sometimes refer to these four statutory 

terms, collectively, as ‘‘quality.’’ 

2. ‘‘Utility’’ refers to the usefulness of 

the information to its intended users, 

including the public. In assessing the 

usefulness of information that the 

agency disseminates to the public, the 

agency needs to consider the uses of the 

information not only from the 

perspective of the agency but also from 

the perspective of the public. As a 

result, when transparency of 

information is relevant for assessing the 

information’s usefulness from the 

public’s perspective, the agency must 

take care to ensure that transparency has 

been addressed in its review of the 

information. 

3. ‘‘Objectivity’’ involves two distinct 

elements, presentation and substance. 

a. ‘‘Objectivity’’ includes whether 

disseminated information is being 

presented in an accurate, clear, 

complete, and unbiased manner. This 

involves whether the information is 

presented within a proper context. 

Sometimes, in disseminating certain 

types of information to the public, other 

information must also be disseminated 

in order to ensure an accurate, clear, 

complete, and unbiased presentation. 

Also, the agency needs to identify the 

sources of the disseminated information 

(to the extent possible, consistent with 

confidentiality protections) and, in a 

scientific, financial, or statistical 

context, the supporting data and 

models, so that the public can assess for 

itself whether there may be some reason 

to question the objectivity of the 

sources. Where appropriate, data should 

have full, accurate, transparent 

documentation, and error sources 

affecting data quality should be 

identified and disclosed to users. 

b. In addition, ‘‘objectivity’’ involves 

a focus on ensuring accurate, reliable, 

and unbiased information. In a 

scientific, financial, or statistical 

context, the original and supporting 

data shall be generated, and the analytic 

results shall be developed, using sound 

statistical and research methods. 

i. If data and analytic results have 

been subjected to formal, independent, 

external peer review, the information 

may generally be presumed to be of 

acceptable objectivity. However, this 

presumption is rebuttable based on a 

persuasive showing by the petitioner in 

a particular instance. If agencysponsored peer review is employed to 

help satisfy the objectivity standard, the 

review process employed shall meet the 

general criteria for competent and 

credible peer review recommended by 

OMB–OIRA to the President’s 

Management Council (9/20/01) (http:// 

www.whitehouse.gov/omb/inforeg/ 

oira_review-process.html), namely, 

‘‘that (a) peer reviewers be selected 

primarily on the basis of necessary 

technical expertise, (b) peer reviewers 

be expected to disclose to agencies prior 

technical/policy positions they may 

have taken on the issues at hand, (c) 

peer reviewers be expected to disclose 

to agencies their sources of personal and 
8460 Federal Register / Vol. 67, No. 36 / Friday, February 22, 2002 / Notices 

institutional funding (private or public 

sector), and (d) peer reviews be 

conducted in an open and rigorous 

manner.’’ 

ii. If an agency is responsible for 

disseminating influential scientific, 

financial, or statistical information, 

agency guidelines shall include a high 

degree of transparency about data and 

methods to facilitate the reproducibility 

of such information by qualified third 

parties. 

A. With regard to original and 

supporting data related thereto, agency 

guidelines shall not require that all 

disseminated data be subjected to a 

reproducibility requirement. Agencies 

may identify, in consultation with the 

relevant scientific and technical 

communities, those particular types of 

data that can practicable be subjected to 

a reproducibility requirement, given 

ethical, feasibility, or confidentiality 

constraints. It is understood that 

reproducibility of data is an indication 

of transparency about research design 

and methods and thus a replication 

exercise (i.e., a new experiment, test, or 

sample) shall not be required prior to 

each dissemination. 

B. With regard to analytic results 

related thereto, agency guidelines shall 

generally require sufficient transparency 

about data and methods that an 

independent reanalysis could be 

undertaken by a qualified member of the 

public. These transparency standards 

apply to agency analysis of data from a 

single study as well as to analyses that 

combine information from multiple 

studies. 

i. Making the data and methods 

publicly available will assist in 

determining whether analytic results are 

reproducible. However, the objectivity 

standard does not override other 

compelling interests such as privacy, 

trade secrets, intellectual property, and 

other confidentiality protections. 

ii. In situations where public access to 

data and methods will not occur due to 

other compelling interests, agencies 

shall apply especially rigorous 

robustness checks to analytic results 

and document what checks were 

undertaken. Agency guidelines shall, 

however, in all cases, require a 

disclosure of the specific data sources 

that have been used and the specific 

quantitative methods and assumptions 

that have been employed. Each agency 

is authorized to define the type of 

robustness checks, and the level of 

detail for documentation thereof, in 

ways appropriate for it given the nature 

and multiplicity of issues for which the 

agency is responsible. 

C. With regard to analysis of risks to 

human health, safety and the 

environment maintained or 

disseminated by the agencies, agencies 

shall either adopt or adapt the quality 

principles applied by Congress to risk 

information used and disseminated 

pursuant to the Safe Drinking Water Act 

Amendments of 1996 (42 U.S.C. 300g– 

1(b)(3)(A) & (B)). Agencies responsible 

for dissemination of vital health and 

medical information shall interpret the 

reproducibility and peer-review 

standards in a manner appropriate to 

assuring the timely flow of vital 

information from agencies to medical 

providers, patients, health agencies, and 

the public. Information quality 

standards may be waived temporarily by 

agencies under urgent situations (e.g., 

imminent threats to public health or 

homeland security) in accordance with 

the latitude specified in agency-specific 

guidelines. 

4. ‘‘Integrity’’ refers to the security of 

information—protection of the 

information from unauthorized access 

or revision, to ensure that the 

information is not compromised 

through corruption or falsification. 

5. ‘‘Information’’ means any 

communication or representation of 

knowledge such as facts or data, in any 

medium or form, including textual, 

numerical, graphic, cartographic, 

narrative, or audiovisual forms. This 

definition includes information that an 

agency disseminates from a web page, 

but does not include the provision of 

hyperlinks to information that others 

disseminate. This definition does not 

include opinions, where the agency’s 

presentation makes it clear that what is 

being offered is someone’s opinion 

rather than fact or the agency’s views. 

6. ‘‘Government information’’ means 

information created, collected, 

processed, disseminated, or disposed of 

by or for the Federal Government. 

7. ‘‘Information dissemination 

product’’ means any books, paper, map, 

machine-readable material, audiovisual 

production, or other documentary 

material, regardless of physical form or 

characteristic, an agency disseminates to 

the public. This definition includes any 

electronic document, CD–ROM, or web 

page. 

8. ‘‘Dissemination’’ means agency 

initiated or sponsored distribution of 

information to the public (see 5 CFR 

1320.3(d) (definition of ‘‘Conduct or 

Sponsor’’)). Dissemination does not 

include distribution limited to 

government employees or agency 

contractors or grantees; intra- or interagency use or sharing of government 

information; and responses to requests 

for agency records under the Freedom of 

Information Act, the Privacy Act, the 

Federal Advisory Committee Act or 

other similar law. This definition also 

does not include distribution limited to 

correspondence with individuals or 

persons, press releases, archival records, 

public filings, subpoenas or adjudicative 

processes. 

9. ‘‘Influential’’, when used in the 

phrase ‘‘influential scientific, financial, 

or statistical information’’, means that 

the agency can reasonably determine 

that dissemination of the information 

will have or does have a clear and 

substantial impact on important public 

policies or important private sector 

decisions. Each agency is authorized to 

define ‘‘influential’’ in ways appropriate 

for it given the nature and multiplicity 

of issues for which the agency is 

responsible. 

10. ‘‘Reproducibility’’ means that the 

information is capable of being 

substantially reproduced, subject to an 

acceptable degree of imprecision. For 

information judged to have more (less) 

important impacts, the degree of 

imprecision that is tolerated is reduced 

(increased). If agencies apply the 

reproducibility test to specific types of 

original or supporting data, the 

associated guidelines shall provide 

relevant definitions of reproducibility 

(e.g., standards for replication of 

laboratory data). With respect to 

analytic results, ‘‘capable of being 

substantially reproduced’’ means that 

independent analysis of the original or 

supporting data using identical methods 

would generate similar analytic results, 

subject to an acceptable degree of 

imprecision or error. 

[FR Doc. 02–59 Filed 1–2–02; 1:36 pm] 

BILLING CODE 3110–01–M 

Editorial Note: Due to numerous errors, 

this document is being reprinted in its 

entirety. It was originally printed in the 

Federal Register on Thursday, January 3, 

2002 at 67 FR 369–378 and was corrected on 

Tuesday, February 5, 2002 at 67 FR 5365. 

[FR Doc. R2–59 Filed 2–21–02; 8:45 am] 

BILLING CODE 1505–01–D