EPA/100/B-06/002 

U.S. Environmental Protection Agency

Peer Review Handbook

3rd Edition

Prepared for the U.S. Environmental Protection Agency

by Members of the Peer Review Advisory Group,

for EPA’s Science Policy Council

 

Science Policy Council

U.S. Environmental Protection Agency

Washington, DC 20460
This report is dedicated to the memory of our friend and colleague, 

Nash Gerald. Nash was a wonderful colleague who thoughtfully 

strived to perfect the use of peer review at EPA. 

Nash joined EPA in 1989. In addition to his duties coordinating 

science/policy and research for the Office of Air Quality Planning 

and Standards, he consistently contributed to considerations of the 

Agency's peer review practices. Nash’s courteous and 

indefatigable approach to resolving details greatly enhanced the 

revision of this Handbook. He will be missed. 
 Peer Review Handbook Page iii 

Disclaimer

This Handbook was developed by EPA to provide guidance to EPA staff and managers who are 

planning and conducting peer reviews. This Handbook is intended to improve the internal 

management of EPA by providing recommended procedures and approaches for EPA staff and 

managers. This Handbook is a guidance manual and not a rule or regulation. The Handbook 

does not replace existing laws or regulation, does not change or substitute for any legal 

requirement, and is not legally enforceable. This Handbook does not create or confer legal rights 

or impose any legally binding requirements on EPA or any party. The use of non-mandatory 

language such as "may," "can" or "should" in this Handbook does not connote a requirement but 

does indicate EPA’s strongly preferred approach to ensure the quality of peer reviews conducted 

or initiated by EPA. Mention of trade names or commercial products does not constitute 

endorsement or recommendation for use. 
 Peer Review Handbook Page iv 

The Peer Review Advisory Group

Connie Bosma ORD 

Jennifer Bowen OPEI 

Michele Burgess OSWER (now at USDA) 

Mary E Clark OAR 

 Kerry Dearfield ORD (now at USDA) 

 Patrick Easter OSWER

Daniel Fort SAB 

Nash Gerald OAR 

Roland Hemmett Region 2 

 Vincia Holloman OEI 

 Joyce Jatko ORD (now at Navy) 

Barbara Klieforth ORD 

Steven Knott OPPTS 

Andrew McIntosh AO (now at GAO) 

John E Miller ORD 

Manisha Patel OGC 

Mary Reiley OW 

Nancy Wentworth OEI
 Peer Review Handbook Page v 

Table Of Contents

DISCLAIMER ...............................................................................................................................iii 

PEER REVIEW ADVISORY GROUP iv

FOREWORD ...............................................................................................................................xiii 

LIST OF ACRONYMS ................................................................................................................ xv 

QUICK START GUIDE TO THE PEER REVIEW PROCESS................................................... 1 

Figure 1 – Flowchart for Planning a Peer Review...................................................................... 2 

Figure 2 - Flowchart for Conducting a Peer Review .................................................................. 3 

Figure 3 - Flowchart for Completing a Peer Review.................................................................. 4 

Manager’s Planning Checklist for Peer Review ......................................................................... 5 

Regulatory Action Development Checklist for Workgroups...................................................... 6 

PEER REVIEW GUIDANCE ........................................................................................................ 9 

1. THE NEED FOR PEER REVIEW.......................................................................................... 11 

1.1 Overview Statement...................................................................................................... 11 

1.2 Understanding Peer Review.......................................................................................... 11 

1.2.1 Why Use Peer Review? ........................................................................................ 11 

1.2.2 What is Peer Involvement? ................................................................................... 12 

1.2.3 What is Peer Review? ........................................................................................... 12 

1.2.4 What is Peer Input?............................................................................................... 12 

1.2.5 How is Peer Review Different from Peer Input?.................................................. 13 

1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer 

 Reviewer for the Same Work Product Later in the Process?................................ 13 

1.2.7 Can the Same Peer Reviewer be Used More Than Once if a Product Will Be 

 Peer Reviewed More Than Once, and Can the Same Peer Reviewer be Used 

 Again and Again for Different Products? ............................................................. 13 

1.2.8 How is Peer Review Different from Public Comment?........................................ 14 

1.2.9 How is Peer Review Different from Stakeholder Involvement? .......................... 14 

1.2.10 What Role Does Peer Review Have in Regulatory Development? ...................... 14 

1.2.11 How Does the Tiering Process Affect Peer Review? ........................................... 15 

1.2.12 Should You Discuss Peer Review in the Analytic Blueprint for Your 

 Regulation? ........................................................................................................... 15 
 Peer Review Handbook Page vi 

1.2.13 What Role does Peer Review Have in Regulatory Negotiations? ........................ 16 

1.2.14 Should You Address Peer Review in the Preamble of a Regulation? .................. 16 

1.2.15 How is Peer Review Documented in the Action Memo for Regulations? ........... 16 

1.3. Peer Review and Information Quality .......................................................................... 16 

1.3.1 Overview Statement.............................................................................................. 16 

1.3.2 How Does Peer Review Ensure the Quality of Information EPA Disseminates? 17 

1.3.3 How Do You Ensure the Quality of Data Supporting EPA’s Information 

 Products?............................................................................................................... 17 

1.3.4 Are Work Products Undergoing Peer Review Subject to the Guidelines?........... 18 

1.4 Annual Agency Reporting ............................................................................................ 19 

1.4.1 What Are EPA’s Reporting Practices and What Information Is Publicly 

 Available? ............................................................................................................. 19 

1.4.2 What Information Is Provided in the Science Inventory?..................................... 19

1.4.3 How Will the Public Be Allowed to Comment on the Peer Review Plans 

 for Influential Scientific Information and Highly Influential Scientific 

 Assessments? ........................................................................................................ 21 

1.4.4 What Is ORD’s Role in Reviewing the Science Inventory and How Will This 

 Review Be Conducted?......................................................................................... 21 

1.4.5 How Will EPA Submit an Annual Report to OMB? ............................................ 21 

1.5 The Roles of People and Organizations in Peer Review .............................................. 22 

1.5.1 Who is Ultimately Accountable for Peer Review?............................................... 22 

1.5.2 Who are the Agency Staff involved in Peer Review?........................................... 22 

1.5.3 Who are the Decision Makers and What are Their Responsibilities? .................. 22 

1.5.4 Who are the Peer Review Coordinators and What are Their Responsibilities? ... 23 

1.5.5 Who are the Peer Review Leaders and What are Their Responsibilities?............ 25 

1.5.6 Who are the Peer Reviewers? ............................................................................... 26 

1.5.7 What are the Responsibilities of Peer Reviewers? ............................................... 26 

1.5.8 What is an Independent Peer Reviewer? .............................................................. 26 

1.5.9 When does an Agency Internal Peer Reviewer Qualify as Independent? ............ 27 

1.5.10 What is a Peer Review Panel? .............................................................................. 27 

1.5.11 What is a Subject Matter Expert? ......................................................................... 27 

1.5.12 What is the Role of the Science Policy Council (SPC)?....................................... 27 

1.5.13 What is the Role of the Peer Review Advisory Group (PRAG)?......................... 28 

1.5.14 What is the Role of the Office of Research and Development (ORD)? ............... 28 
 Peer Review Handbook Page vii 

2. PLANNING A PEER REVIEW.............................................................................................. 29 

2.1 Overview Statement...................................................................................................... 29 

2.2 Determining Which Work Products to Peer Review .................................................... 29 

2.2.1 What are Scientific and/or Technical Work Products?......................................... 29 

2.2.2 What Scientific and/or Technical Work Products Need Peer Review?................ 30 

2.2.3 How Does One Determine Whether a Scientific and/or Technical Work Product 

 is Influential Scientific Information?.................................................................... 30 

2.2.4 How Does One Determine Whether Influential Scientific Information is a 

 Highly Influential Scientific Assessment?............................................................ 32 

2.2.5 What Economic Work Products Need Peer Review?........................................... 34 

2.2.6 Should Economic Work Products Prepared in Support of Regulations that are 

 Classified as Major or Economically Significant be Peer Reviewed?.................. 35 

2.2.7 What Other Economic Work Products Might Benefit from Peer Review? .......... 36 

2.2.8 What Other Social Science Work Products Need Peer Review?.......................... 36 

2.2.9 How Should Peer Review be Handled for Products Developed Under an 

 Interagency Agreement (IAG)? ............................................................................ 37 

2.2.10 Should Products from Contracts, Grants, and Cooperative Agreements 

 Receive Peer Review?........................................................................................... 37 

2.2.11 How Does Peer Review Apply to Products Generated Through EPA Grants 

 or Cooperative Agreements?................................................................................. 38 

2.2.12 Can the Recipient of a Grant or Cooperative Agreement Use Agreement Funds 

 to Pay Peer Reviewers of their Work Products?................................................... 39 

2.2.13 Do Products Generated Under EPA Grants or Cooperative Agreements Need 

 to be Reported in the Science Inventory............................................................... 40 

2.2.14 Should Site-Specific Decisions be Subject to Peer Review?................................ 40 

2.2.15 Should NEPA Products be Subject to Peer Review?............................................ 40 

2.2.16 Should Environmental Regulatory Models be Peer Reviewed?........................... 41 

2.2.17 Should Another Organization’s Work Products That Have Been Submitted to 

 EPA for Use in Decision Making be Peer Reviewed?.......................................... 41 

2.2.18 Can Work Products That are Not Determined to be Influential Scientific 

 Information or Highly Influential Scientific Assessments Still be 

 Peer Reviewed?..................................................................................................... 42 

2.3 Determining Which Work Products Do Not Receive Peer Review ............................. 42 

2.3.1 What Information Is Exempt from the OMB Bulletin Provisions? ...................... 42 

2.3.2 Are There Other Circumstances Where Peer Review Is Not Necessary? ............ 43 

2.3.3 For Information That Is Not Exempt, Are There Circumstances Where the 

 Peer Review Provisions of the OMB Bulletin Can Be Waived or Deferred? ...... 43 

2.3.4 Does Publication in a Refereed Scientific Journal Mean That Adequate Peer 
 Peer Review Handbook Page viii 

 Review Has Been Performed? .............................................................................. 43 

2.3.5 Do Voluntary Consensus Standards Undergo Peer Review? ............................... 44 

2.4 Choosing a Peer Review Mechanism ........................................................................... 44 

2.4.1 How Do You Determine the Appropriate Peer Review Mechanism? .................. 44 

2.4.2 What are Some Examples of Internal Peer Review? ............................................ 46 

2.4.3 What are Some Examples of External Peer Review? ........................................... 46 

2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?.................... 47 

2.4.5 Do Agency Work Products Become Candidates for Peer Review when Peer 

 Reviewed Journal Articles are Used in Support of that Work Product?............... 47 

2.4.6 When and How Often Should Peer Review Occur? ............................................. 48 

2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review? ........ 49 

2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the 

 Peer Review? ........................................................................................................ 49 

2.5 Creating the Peer Review Record and Peer Review Reports ....................................... 49 

2.5.1 What is the Peer Review Record?......................................................................... 49 

2.5.2 How Can the Peer Review Record Improve the Peer Review Process?............... 50 

2.5.3 What Should Be in the Peer Review Record? ...................................................... 50 

2.5.4 How and When Should You Develop and Post Peer Review Reports and 

 Agency Responses for Influential Scientific Information and Highly Influential 

 Scientific Assessments?........................................................................................ 51 

2.5.5 What Should You Do with a Peer Review Record That Pertains to a 

 Rulemaking Action? ............................................................................................. 51 

2.5.6 When Should the Peer Review Record Building Process Begin? ........................ 51 

2.5.7 What are the Differences in Record Keeping for a Review by Individuals 

 Compared to Review by a Panel? ......................................................................... 52 

2.5.8 Where Should the Peer Review Record be Kept and For How Long?................. 52 

2.5.9 Are Internal Peer Review Comments Included in the Peer Review Record?....... 52 

2.6 Budget Planning............................................................................................................ 53 

2.6.1 What Budgetary Factors Should You Consider in a Peer Review?...................... 53 

2.6.2 What Input is Needed for the Annual Budget Formulation and Budget 

 Execution Process? ............................................................................................... 53 

2.7 Legal Considerations .................................................................................................... 53 

2.7.1 What Are the Legal Ramifications of the Peer Review Policy?........................... 53 

2.7.2 Is Legal Advice Needed?...................................................................................... 54 

2.8 Federal Advisory Committee Act (FACA) Considerations.......................................... 54 

2.8.1 When Do FACA Requirements Apply to EPA-Run Peer Reviews?.................... 54 
 Peer Review Handbook Page ix 

2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? ................................. 55 

2.8.3 How Do You Ensure that a Contractor-Run Peer Review Does Not Become 

 Subject to FACA?................................................................................................. 55 

3. CONDUCTING A PEER REVIEW........................................................................................ 57 

3.1 Overview Statement...................................................................................................... 57 

3.2 Charge to the Peer Reviewers....................................................................................... 57 

3.2.1 What is a Charge? ................................................................................................. 57 

3.2.2 What are the Essential Elements of a Charge? ..................................................... 58 

3.2.3 Where Can You Get an Example of a Charge? .................................................... 58 

3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?.............. 58 

3.2.5 Who Writes the Charge When You Hire a Contractor to Conduct the 

 Peer Review? ........................................................................................................ 59 

3.2.6 Is it Okay to Ask a Contractor to Develop the Charge to the Peer Reviewers? ... 59 

3.3 Time Line...................................................................................................................... 59 

3.3.1 What are the Factors in Scheduling a Peer Review? ............................................ 59 

3.4 Selection of Peer Reviewers ......................................................................................... 60 

3.4.1 What are Considerations for Selecting Peer Reviewers?...................................... 60 

3.4.2 Where Do You Find Peer Reviewers?.................................................................. 60 

3.4.3 Are External or Internal Peer Reviewers Preferred?............................................. 62 

3.4.4 What is Important in the Mix of a Peer Review Panel?........................................ 62 

3.4.5 Ethical Standards .................................................................................................. 63 

3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of 

 Conflicts of Interest?............................................................................................. 68 

3.4.7 Are There Constraints to Selecting Peer Reviewers? ........................................... 71 

3.4.8 If State Employees Are Used as Peer Reviewers, Can EPA Pay Them 

 for This Service?................................................................................................... 72 

3.4.9 Does the Agency Recognize Peer Reviews That External Parties Conduct 

 and/or Pay For?..................................................................................................... 72 

3.5 Materials for Peer Reviewers........................................................................................ 73 

3.5.1 What Instructions Do You Give Peer Reviewers?................................................ 73 

3.5.2 What Materials Should be Sent to Peer Reviewers?............................................. 73 

3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?.......... 75 

3.6 Peer Review Services.................................................................................................... 75 

3.6.1 What are Gratuitous Services for Peer Review?................................................... 75 
 Peer Review Handbook Page x 

3.6.2 Can You Use a Contract to Obtain Peer Review Services?.................................. 76 

3.6.3 How Do You Write a Statement of Work for Contracts?..................................... 76 

3.6.4 What are Some Management Controls for Contracts? ......................................... 77 

3.6.5 Can You Identify and/or Select Peer Reviewers When Using a Contract? .......... 80 

3.6.6 How is Travel Handled with Contracts or Purchase Orders? ............................... 80 

3.6.7 How is Travel Handled with Special Government Employees?........................... 81 

4. COMPLETING A PEER REVIEW......................................................................................... 82 

4.1 Overview....................................................................................................................... 82 

4.2 Final Work Product....................................................................................................... 82

4.2.1 How Do You Incorporate Peer Review Comments into the Final 

 Work Product? ...................................................................................................... 82 

4.2.2 How Might Peer Review Comments Impact Your Work Product?...................... 83 

4.2.3 What Should the Final Work Product Say About the Peer Review Process?....... 84 

4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous?................................... 84 

4.3 Completing the Peer Review Record............................................................................ 85 

4.3.1 How Do You Complete the Peer Review Record?............................................... 85 

4.3.2 Where Should the Peer Review Records be Kept, and for How Long? ............... 85 

4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA?......... 86 

APPENDIX A EPA’S PEER REVIEW POLICY ................................................................... A-1 

APPENDIX B OMB INFORMATION QUALITY BULLETIN FOR PEER REVIEW AND 

PREAMBLE ..................................................................................................... B-1

APPENDIX C SOUND SCIENCE AND PEER REVIEW IN RULEMAKING...................... C-2 

PEER REVIEW (CONDITIONAL TEMPLATE) ................................................................. C-4 

FRAMEWORK FOR ACTION MEMORANDA.................................................................. C-5 

APPENDIX D EXAMPLES OF PEER REVIEW CHARGES................................................. D-1 

CHARGE EXAMPLE 1 - US EPA Science Advisory Board (SAB) Consultation on 

Suspended and Bedded Sediments.......................................................................................... D-3 

CHARGE EXAMPLE 2 - Economics - Benefits Transfer from Adults to Children ............ D-4 

CHARGE EXAMPLE 3 - Economics - Study of Municipal Government Costs and Financial 

Impacts from Environmental Regulations .............................................................................. D-5 
 Peer Review Handbook Page xi 

CHARGE EXAMPLE 4 - External Peer Review of the Design of the Targeted National 

Sewage Sludge Survey for Pollutants in Sewage Sludge ....................................................... D-7 

CHARGE EXAMPLE 5 - External Peer Review of the Draft National Strategy for the 

Development of Water Quality Criteria for Suspended and Bedded Sediments (SABS) ...... D-9 

CHARGE EXAMPLE 6 - Clean Air Scientific Advisory Committee Review of the 

Agency’s National Ambient Air Monitoring Strategy.......................................................... D-12 

CHARGE EXAMPLE 7 - The Multimedia, Multipathway, and Multireceptor 

Risk Assessment (3MRA) Model ......................................................................................... D-15 

APPENDIX E GUIDANCE ON REQUESTING A REVIEW BY THE SCIENCE 

ADVISORY BOARD (SAB) ......................................................................................................E-1 

TABLE 1 – PROCESS FOR SELECTING SCIENCE ADVISORY BOARD (SAB) 

PROJECTS ..............................................................................................................................E-5 

APPENDIX F EXAMPLE STATEMENTS OF WORK FOR CONTRACTS..........................F-8 

STATEMENT OF WORK - EXAMPLE 1 - Peer Review of Prioritization Tool Report ....F-10 

STATEMENT OF WORK - EXAMPLE 2 External Peer Review of Protozoa Method 

Development Criteria Document ...........................................................................................F-17 

STATEMENT OF WORK - EXAMPLE 3 - Technical Support for Peer Review Services.F-19 
 Peer Review Handbook Page xii 
 Peer Review Handbook Page xiii 

Foreword

Science is the foundation that supports all of our work here at EPA. Strong, independent science 

is of paramount importance to our environmental policies. The quality of science that underlies 

our regulations is vital to the credibility of EPA’s decisions and ultimately the Agency’s 

effectiveness in protecting human health and the environment. One important way to ensure 

decisions are based on defensible science is to have an open and transparent peer review process. 

Consistent Agency-wide implementation of peer review has been an EPA priority for many 

years. Since issuing the Peer Review Policy in 1993, EPA has taken many steps to support and 

strengthen the policy. In January 1998, EPA issued the 1st edition of the Peer Review Handbook 

to provide guidance about peer review and peer review processes. The Handbook was updated 

in December 2000 to reflect feedback from EPA’s science community on improvements to the 

usability of the Handbook. The science community found that the Handbook contributed greatly 

to the Agency goal of sound science and substantially improved EPA’s peer review process. 

In December 2004, the Office of Management and Budget (OMB) issued its “Final Information 

Quality Bulletin for Peer Review” (See Appendix B). This OMB Bulletin establishes 

government-wide guidance aimed at enhancing the practice of peer review of government 

science documents. The Bulletin includes guidance for all Federal agencies on what information 

should be subject to peer review, for selecting peer reviewers and for providing opportunities for 

public participation. It also defines a peer review planning process to allow the public to 

comment on peer review plans. While incorporating the provisions of the OMB Bulletin, EPA 

updated its Peer Review Policy to reiterate the Agency's commitment to peer review. This 

policy encourages and expects peer review of all scientific and technical information that is 

intended to inform or support Agency decisions and notes that influential scientific information, 

including highly influential scientific assessments, should be peer reviewed in accordance with 

this Handbook. 
 Peer Review Handbook Page xiv
 Peer Review Handbook Page xv 

Acronyms

AA Assistant Administrator 

AAS Advisory and Assistance Services 

ADP Action Development Process 

ANPR Advance Notice of Proposed Rulemaking 

ASTM American Society of Testing and Materials 

BOSC Board of Scientific Counselors 

CBI Confidential Business Information 

CCMP Comprehensive Conservation and Management Plan 

CENR Committee on the Environment and Natural Resources 

CFR Code of Federal Regulations 

CMM Contracts Management Manual 

CO Contract(ing) Officer 

COI Conflict of Interest 

COR Contracting Officer Representative 

CREM Council for Regulatory Environmental Modeling 

CTGs Control Techniques Guidelines 

DEO Deputy Ethics Official 

DFO Designated Federal Officer 

DRA Deputy Regional Administrator 

EA Economic Analysis (previously known as Regulatory Impact Analysis) 

EC Executive Council of the SAB 

ECOC Ecological Chemicals of Concern 

EEAC Environmental Economics Advisory Committee 

EIMS Environmental Information Management System 

EIS Environmental Impact Statement 

ELG Effluent Limitation Guidelines 

EMAP Environment Monitoring and Assessment Program 

EO Executive Order 

EPA Environmental Protection Agency 

EPAAR EPA Acquisition Regulations 

FACA Federal Advisory Committee Act 

FAR Federal Acquisition Regulations 

FIFRA Federal Insecticide, Fungicide and Rodenticide Act 

FIRMR Federal Information Resource Management Regulation 

FOIA Freedom of Information Act 

FR Federal Register 

FTE Full Time Equivalent 

GSA General Services Administration 

HHS Health and Human Services 
 Peer Review Handbook Page xvi 

HISA Highly Influential Scientific Assessment 

IDLH Immediately Dangerous to Life or Health Concentration 

IAG Interagency Agreement 

IGA Inherently Governmental Activities 

IGF Inherently Governmental Function 

IQGs Information Quality Guidelines 

IRIS Integrated Risk Information System 

ISC-COMPDEP Industrial Source Complex - Complex Terrain Deposition 

ISI Influential Scientific Information 

ISO International Organization for Standardization 

LAN Local Area Network 

LOC Level of Concern 

LOE Level of Effort 

NAS National Academy of Sciences 

NEPA National Environmental Policy Act 

NOAA National Oceanic and Atmospheric Administration 

NODA Notice of Data Availability 

NPDES National Pollutant Discharge Elimination System 

NPS National Park Service 

NRC National Research Council 

NRMRL National Risk Management Research Laboratory 

NTTAA National Technology Transfer and Advancement Act of 1995 

OAM Office of Acquisition Management 

OAP Office of Atmospheric Programs 

OAQPS Office of Air Quality Planning and Standards 

OAR Office of Air and Radiation 

OARM Office of Administration and Resources Management 

OCHP Office of Children’s Health Protection 

OEI Office of Environmental Information 

OGC Office of General Counsel 

OGE Office of Government Ethics 

OIA Office of International Affairs 

OIG Office of the Inspector General 

OIRA OMB's Office of Information and Regulatory Affairs 

OMB White House Office of Management and Budget 

ORC Office of Regional Counsel 

ORD Office of Research and Development 

ORIA Office of Radiation and Indoor Air 

ORMI Office of Regulatory Management and Information 

OSA Office of the Science Advisor 

OSP Office of Science Policy 
 Peer Review Handbook Page xvii 

OST Office of Science and Technology 

OSW Office of Solid Waste 

OTAQ Office of Transportation and Air Quality 

PAHs Polynuclear aromatic hydrocarbons 

PCB Polychlorinated biphenyl 

PCDD Polychlorinated dibenzo(p)dioxin 

PCDF Polychlorinated dibenzofuran 

PEP Peconic Estuary Program 

PL Professional Level 

PMCR Preliminary Model Calibration Report 

PO Purchase Order 

PR Procurement Request 

PPA Pollution Prevention Act 

PRA Paperwork Reduction Act 

PRAG Peer Review Advisory Group 

PRPT Peer Review Product Tracking Database 

Q&A Question and Answer 

QRT Quick Response Task 

RA Regional Administrator 

RC Regional Counsel 

RCRA Resource Conservation and Recovery Act 

RFA Regulatory Flexibility Act 

RfC Reference Concentration (Inhalation) 

RFC Request for Correction 

RfD Reference Dose (Oral) 

RFR Request for Reconsideration 

RGE Regular Government Employee 

RIA Regulatory Impact Analysis (now known as Economic Analysis) 

RI/FS Remedial Investigation/Feasibility Study 

ROD Record of Decision 

RSAC Research Strategies Advisory Committee 

SAB Science Advisory Board 

SAME Simplified Acquisition Made Easy 

SAP Scientific Advisory Panel 

SBREFA Small Business Regulatory Fairness Act 

SERA Screening level Ecological Risk Assessment 

SGE Special Government Employee 

SI Science Inventory 

SOPs Standard Operating Procedures 

SOW Statement of Work 

SPC Science Policy Council 
 Peer Review Handbook Page xviii

SPC-SC Science Policy Council-Steering Committee 

SRA Science Review Administrator 

STAR Science To Achieve Results 

TAMS Toxics Air Monitoring System 

TD Technical Directive 

TQM Total Quality Management 

UCL Upper Confidence Limits 

URL Uniform Resource Locator 

US United States 

USC United States Code 

USEPA United States Environmental Protection Agency 

USGS United States Geological Survey 

VSCF Value of a Statistical Cancer Fatality 

VSL Value of Statistical Life 

WAM Work Assignment Manager 

WMB Waste Minimization Branch 

WTI Waste Technologies Industries 
 Peer Review Handbook Page 1 

Quick Start Guide To 

The Peer Review Process

This Section provides flowcharts describing the major steps in conducting a peer review and 

checklists to help plan a successful peer review. Cross references to the appropriate sections in 

the Peer Review Handbook are shown in parentheses and bolded. The flowcharts and checklists 

included are: 

$ Figure 1 - Flowchart for Planning a Peer Review

$ Figure 2 - Flowchart for Conducting a Peer Review

$ Figure 3 - Flowchart for Completing a Peer Review

$ Manager’s Planning Checklist for Peer Review

$ Regulatory Action Development Checklist for Workgroups
 Peer Review Handbook Page 2 

Figure 1 – Flowchart for Planning a Peer Review

Continue with Conducting a Peer Review

Work Product Should be Peer 

Reviewed

-Identify Basis for Charge

-Identify Key Staff

-Create Peer Review Record

-Ensure Availability of Funds

-Determine Overall Time Frame

No

Yes

No

Does Work Product 

Consist Only of 

Science Previously 

PeerReviewed?

Is Peer Review

Exempt or Waived?

Yes

Peer Review

Not Needed

Yes

Peer Review

Not Needed

Would the Work 

Product Still Warrant 

Peer Review?

Is the Work 

Product ISI/HISA ? 

Identify Work Product 

and Include in 

Science Inventory

No

No

Yes

1. Determine if the work product:

 – Is a scientific, engineering, economic, 

social science, or statistical document 

(§2.2.1) 

– Is influential scientific information (ISI) or

a highly influential scientific assessment 

(HISA) (§§2.2.3, 2.2.4, 2.2.5) 

2. Peer Review typically needed for: 

– ISI, including HISA (§2.2.2)

– Other work products unless peer review is

determined not to be warranted

3. Peer Review typically not needed if: 

 – ISI/HISA consists only of science 

previously peer reviewed and the review is

deemed adequate under the Agency’s 

Policy (§2.3.2)

– meets criteria for exemption (§2.3.1)

– receives waiver (§2.3.3)

– otherwise determined not to be warranted

4. If a work product is subject to peer review: 

– Identify criteria/basis for charge (§3.2.1) 

– Identify key staff (§1.5)

– Create a peer review record (§2.5)

– Ensure source of funding for the peer 

review (§2.6) 

– Determine/estimate overall time frame for 

peer review (§3.3) 
 Peer Review Handbook Page 3 

Figure 2 - Flowchart for Conducting a Peer Review

1. Develop the charge (§3.2)

– Determine which key issues to address 

Work Product

(Ready for Peer Review)

Develop the Charge

Select the Peer Review 

Mechanism

Set the Timelines, 

Including Deadlines

Select Peer Reviewers

Send Materials to Peer 

Reviewers & Conduct 

Review

Complete Peer Review 

and Obtain Comments 

of Peer Reviewers

Continue With 

Completing a Peer 

Review

Add to Peer Review 

Record

Work Product

(Ready for Peer Review)

Develop the Charge

Select the Peer Review 

Mechanism

Set the Timelines, 

Including Deadlines

Select Peer Reviewers

Send Materials to Peer 

Reviewers & Conduct 

Review

Complete Peer Review 

and Obtain Comments 

of Peer Reviewers

Continue With 

Completing a Peer 

Review

Set the Timelines, 

Including Deadlines

Select Peer Reviewers

Send Materials to Peer 

Reviewers & Conduct 

Review

Complete Peer Review 

and Obtain Comments 

of Peer Reviewers

Continue With 

Completing a Peer 

Review

Add to Peer Review 

Record

– Include in peer review record (§2.5.3) 

2. Select a peer review mechanism (§2.4): 

 – Internal (§2.4.2) 

 – External (§2.4.3) 

– Mail (e.g., Letter) review (§2.4.3) 

– Face to face meeting (§2.4.3)

– One time or multiple meetings (§2.4.6)

– Include logistical information in the peer review 

record (§2.5.3)

3. Determine the specific timeline(§3.3): 

– When will the review be started

– What are the intermediate check points 

– What is the deadline for completion

4. Select peer reviewers (§3.4): 

– Determine expertise needed (§§3.4.1, 3.4.4) 

– Determine sources of peer reviewers (§3.4.2) 

– If ISI/HISA, consider asking public to nominate peer

reviewers (§3.4.2) 

– Consider and address the balance of the panel (§3.4.4) 

– Consider conflicts of interest (§§3.4.5, 3.4.6)

– Include documentation in peer review record (§2.5)

5. Materials for the peer review (§3.5): 

– Obtain materials from Program for review 

 – Prepare instructions for peer reviews (§3.5.1)

– Forward materials to peer reviewers (§3.5.2) 

– Include copy of materials in peer review record (§2.5)

6. Conduct the peer review: 

– If ISI/HISA, ask reviewers to prepare peer review 

report (§2.5.4) 

– If HISA, consider seeking public comment on 

work product and allowing the public to present to 

the peer reviewers (§§1.4.2, 1.4.3, 1.5.3, 2.4.7) 

– If HISA, and if seeking public comment, provide peer 

reviewers with significant public comments

– Include in peer review record (§2.5)

Note: Some of these steps may occur concurrently.
 Peer Review Handbook Page 4 

Figure 3 - Flowchart for Completing a Peer Review

YES NO

Document How 

Used and Place

In Peer Review Record

Incorporate

Comments Into

Work Product

Document Why Not Used 

and Place in Peer Review 

Record

Finalize the Work Product

Update the Peer Review Record

Done

Brief Decision Maker on 

Approach to Handling 

Comments

Are Peer Reviewers

Comments Used in 

Work Product?

Brief Decision Maker, 

if appropriate

Receive Comments from Peer 

Reviewers

YES NO

Document How 

Used and Place

In Peer Review Record

Incorporate

Comments Into

Work Product

Document Why Not Used 

and Place in Peer Review 

Record

Finalize the Work Product

Update the Peer Review Record

Done

Brief Decision Maker on 

Approach to Handling 

Comments

YES

Document How 

Used and Place

In Peer Review Record

Incorporate

Comments Into

Work Product

Document Why Not Used 

and Place in Peer Review 

Record

Finalize the Work Product

Update the Peer Review Record

Done

Brief Decision Maker on 

Approach to Handling 

Comments

Document How 

Used and Place

In Peer Review Record

Incorporate

Comments Into

Work Product

Document Why Not Used 

and Place in Peer Review 

Record

Finalize the Work Product

Update the Peer Review Record

Done

Brief Decision Maker on 

Approach to Handling 

Comments

Are Peer Reviewers

Comments Used in 

Work Product?

Brief Decision Maker, 

if appropriate

Receive Comments from Peer 

Reviewers

Are Peer Reviewers

Comments Used in 

Work Product?

Brief Decision Maker, 

if appropriate

Receive Comments from Peer 

1. Evaluate comments from peer reviewers Reviewers

(§4.2.1) 

– Consider and respond to comments 

– Obtain clarification, if needed

– Prepare Agency response (§§4.3.1, 4.3.1.1)

– Include comments in peer review record

2. Brief your Decision Maker; obtain written

management approval of response to comments

(§§1.5.3, 4.2.1)

3. Comments that are considered, but not used

– Document why not used (§4.3.1)

– Include documentation in peer review

record (§§2.5.3, 4.3.1) 

4. Comments that are used (§§2.5.3, 4.2, 4.3)

– Revise the work product by incorporating

comments

– Send revised work product back to peer

reviewers, if necessary 

5. Finalize work product (§4.3.1)

– Include in peer review record (§§2.5.3, 

4.3.1) 

– Post peer review report and related 

materials (e.g., charge, Agency response) 

on the internet (§4.3.1)

– For ISI/HISA that support rulemaking 

– include peer review discussion and 

– certification in preamble to the rule (§§ 

1.2.14, 2.5.5, 4.2.3)
 Peer Review Handbook Page 5 

Manager’s Planning Checklist for Peer Review

1) Title of Work Product: ___________________________________________________ 

2) What Decision/Rule/Regulation/Action Does this Work Product Support: ________ 

______________________________________________________________________________ 

3) Designation of Scientific and Technical Work Products

G Is the work product scientific or technical __yes __no? 

G Is the work product __influential scientific information (ISI), __highly influential scientific 

 assessment (HISA), or __other? (See Section 2.2.3 and 2.2.4 of the Peer Review Handbook 

 for an explanation of these terms). 

4) Determining What Peer Review is Needed

G If ISI or HISA, peer review is needed. 

G If not influential, is peer review still needed? 

G What peer review mechanism is needed (internal and/or external)? 

G When does the review need to be done? 

G How much time will be needed to conduct/complete the review? 

G Are there court ordered deadlines or other constraints? 

G Has senior management (AA/RA/others) been informed of progress/problems? 

G What would constitute success for this review? 

5) Determining the Resources for Peer Review

G What is the priority of this project relative to other projects in the same office? 

G What resources are needed to conduct the review? 

G What are the impacts of the review on personnel? 

G Who will lead the peer review? 

G Who will conduct the peer review? 

G Who will maintain the peer review record? 

G Where will the peer review record be kept? 

G What mechanism will be used for the peer review? 

G Has the charge been developed? 

G Has internal and external coordination been initiated/completed? 

G Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes 

 to the final work product) been made? 

G How will results of the review be presented and addressed in the final work product (e.g., in a 

 preamble, in an accompanying appendix – as well as changes in the work product itself)? 

G Has the work product been entered into the Science Inventory? 

6) Comments: _____________________________________________________________
 Peer Review Handbook Page 6 

Regulatory Action Development Checklist for Workgroups

This checklist will help workgroups plan for peer review in the larger context of regulatory 

development. Each numbered section corresponds to a time period in the regulatory 

development process. 

1. Peer Review Prior to Proposal 

Tier 1 or Tier 2 Rule1

G Is the peer review schedule incorporated into the Analytic Blueprint? 

G Does this rule rely upon influential scientific information or a highly influential scientific 

assessment? 

G Will the work product be reviewed using external peer review? 

Tier 3 Rule

G Is the peer review schedule incorporated in the plans (e.g., the analytic blueprint) for 

producing the action? 

G Does this rule rely upon influential scientific information or a highly influential scientific 

assessment? 

G If an internal mechanism will be used for peer review, is it acceptable according to Section 

2.4 of the Peer Review Handbook? 

2. Sending a Proposed Rule Forward for the Administrator’s Signature 

G Has peer review been completed? 

G Does the action memo indicate whether the rule relies upon influential scientific 

information or a highly influential scientific assessment? 

G If the proposed rule relies on influential scientific information or a highly influential 

scientific assessment, is there a discussion of the peer review in the preamble of the rule? 

3. Before the Proposed Rule Publishes 

G Were the peer review report and any relevant materials included in the docket for this 

rulemaking?

4. Peer Review Prior to Finalization 

G Is a new peer review plan necessary as a result of new regulatory options?

 

1

For further information on tiering and criteria used to determine the appropriate tier for an action, see 

http://intranet.epa.gov/adplibrary/adp/tiering/step2.htm
 Peer Review Handbook Page 7 

5. Sending a Final Rule Forward for the Administrator’s Signature 

G Has any new peer review been completed? 

G Does the action memo indicate whether the rule relies upon influential scientific information or 

a highly influential scientific assessment? 

G If the final rule relies on influential scientific information or a highly influential scientific 

assessment, is there a discussion of the peer review in the preamble of the rule? 

6. Before the Final Rule Publishes 

G Were the peer review report and any relevant materials included in the docket for this 

rulemaking? 
 Peer Review Handbook Page 8 
 Peer Review Handbook Page 9 

U.S. Environmental Protection Agency 

PEER REVIEW GUIDANCE
 Peer Review Handbook Page 10 
 Peer Review Handbook Page 11 

1. The Need For Peer Review

1.1 Overview Statement

Peer review at the U.S. Environmental Protection Agency (EPA) takes many different forms 

depending on the nature of the work product, relevant statutory requirements, and office-specific 

policies and practices. In January 1993, responding to recommendations in the report 

Safeguarding the Future: Credible Science, Credible Decisions

2

, EPA issued an Agency-wide 

policy for peer review. In 1994 EPA reaffirmed the central role of peer review and instituted an 

Agency-wide implementation program. In 1998, a Peer Review Handbook was created as a 

single, centralized form of implementation guidance for Agency staff and managers. The Peer 

Review Handbook was revised and reissued in December 2000. The newly updated Peer 

Review Policy and this 3rd Edition of the Handbook incorporate insights from the use of the 2nd

Edition as well as the provisions of the OMB Final Information Quality Bulletin for Peer Review

that was issued to help agencies enhance their peer review transparency and accountability. 

Among the changes introduced by the OMB Bulletin is the use of the terms “influential scientific 

information” and “highly influential scientific assessments” (see Sections 2.2.3 and 2.2.4 for 

explanations of these terms). 

Peer review is not free; 

however, not doing peer

review can be costly.

1.2 Understanding Peer Review

1.2.1 Why Use Peer Review?

Peer review is intended to uncover any technical problems 

or unresolved issues in a preliminary (or draft) work product through the use of independent 

experts. This information is then used to revise that draft product so that the final work product 

will reflect sound technical information and analyses. Peer review is a process for enhancing a 

scientific or technical work product so that the decision or position taken by the Agency, based 

on that product, has a sound, credible basis. To be most effective, peer review of a scientific 

and/or technical work product should be incorporated into the up-front planning of any action 

based on the work product - this includes obtaining the proper resource commitments (people 

and money) and establishing realistic schedules.

Peer review of scientific and/or technical work products should not be looked upon as another 

“hurdle” in the Agency decision making processes. Although conducting a peer review means 

that time and resources have to be included in the decision making process, the benefits justify 

the added cost. Peer review enhances the credibility and acceptance of the decision based on the 

work product. By ensuring a sound basis for decisions, greater cost savings are realized since 

 

2

EPA/600/9-91/050, March 1992
 Peer Review Handbook Page 12 

decisions will not be challenged as often and extra effort will not be expended to go back and 

redo the work product. So while peer review is not free, the cost of not doing peer review is 

usually much more expensive. Furthermore, not conducting a peer review may potentially place 

the Agency in the position of having to decide whether to try to defend a scientifically 

questionable position – which can be very costly in terms of both resources, and, more 

importantly, credibility. 

1.2.2 What is Peer Involvement?

Peer involvement is the process whereby Agency staff involve subject-matter experts from 

outside their program in one or more aspects of the development of work products. Peer 

involvement, therefore, constitutes active outreach to and participation by the broad scientific, 

engineering, public health, economics and other social science communities beyond the Agency 

(external), as well as within the Agency (internal). Typically, peer involvement takes two 

general forms: peer input (ongoing discussions during the development of the work product) and 

peer review (an evaluation of a work plan, preliminary draft or the like, or the final objective 

expert evaluation of the work product). 

1.2.3 What is Peer Review?

Peer review is a documented critical review of a specific Agency scientific and/or technical work 

product. Peer review is conducted by qualified individuals (or organizations) who are 

independent of those who performed the work, and who are collectively equivalent in technical 

expertise (i.e., peers) to those who performed the original work. Peer review is conducted to 

ensure that activities are technically supportable, competently performed, properly documented, 

and consistent with established quality criteria. Peer review is an in-depth assessment of the 

assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance 

criteria, and conclusions pertaining to the specific major scientific and/or technical work product 

and of the documentation that supports them. Peer review may provide an evaluation of a 

subject where quantitative methods of analysis or measures of success are unavailable or 

undefined such as research and development. Peer review is usually characterized by a one-time 

interaction or a limited number of interactions by independent peer reviewers. Peer review is 

encouraged during the early stages of the project or methods selection, and/or as part of the 

culmination of the work product, as appropriate. Regardless of the timing of peer review, the 

goal is ensuring that the final product is technically sound. 

1.2.4 What is Peer Input?

Many Agency work products are developed with the input of various scientific and technical 

experts inside and outside the Agency. Like the contribution made by peer reviewers, peer input 

is valuable and enhances the scientific or technical basis of the products. Peer input, sometimes 

referred to as peer consultation, generally connotes an interaction during the development of an 

evolving Agency work product, providing an open exchange of data, insights, and ideas. Peer 
 Peer Review Handbook Page 13 

input may be characterized by a continued and iterative interaction with scientific experts during 

work product development. A common example of peer input is the input received from 

workgroup members during development of a product. Many Agency products are developed 

through the efforts of a workgroup, which may include external experts, such as State and Tribal 

representatives. These workgroup members have an active, ongoing participation in developing 

the work product. Another example of obtaining peer input is that of an Agency Office sending 

a draft work product to a list of stakeholder representatives for general comments (stakeholder 

representatives often include experts who could be considered “peers”). 

1.2.5 How is Peer Review Different from Peer Input?

The key distinctions between peer input as described above and formal peer review are the 

independence of the peer reviewers and their level of involvement. The goal of peer review is to 

obtain an independent, third-party review of the product from experts who have not substantially 

contributed to its development. When experts have a material stake in the outcome of the peer 

review (such as a regulated party) or have participated substantially in the development of the 

product (such as a workgroup member), those experts’ reviews may not qualify as unbiased, 

independent peer review and may be better characterized as peer input. 

It is clear that peer input provides valuable contributions to the 

development of the work product. However, peer input does 

not substitute for peer review. In other words, one cannot 

argue that a peer review is not necessary if a work product has 

received “enough” peer input. 

Peer Input is not a 

substitute for Peer Review

1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer 

Reviewer for the Same Work Product Later in the Process?

Generally, the answer is no as that expert is no longer independent, but rather a contributor to the 

work product. There may be special circumstances where the expertise is so narrow that another 

peer reviewer is not available. The Peer Review Leader (see Section 1.5.5) will normally be 

responsible for making this determination and documenting the decision in the peer review 

record. 

1.2.7 Can the Same Peer Reviewer be Used More Than Once if a Product Will Be 

Peer Reviewed More Than Once, and Can the Same Peer Reviewer be Used 

Again and Again for Different Products?

There is no prohibition on using the same peer reviewer more than once on the same product or 

for multiple products of the same office. However, it is preferable to use different people each 

time to provide a broader perspective. When using a contractor to provide peer review services, 

you should recognize that contractors may have a “pool” of reviewers that they use regularly. If 

the same peer reviewers are used repeatedly, they may lose their independence (or the 
 Peer Review Handbook Page 14 

appearance of independence) from the work product(s). If a peer reviewer is asked to participate 

in multiple reviews of the same product it should be noted in the peer review record. 

1.2.8 How is Peer Review Different from Public Comment?

Peer review and public comment are not the same. Public comment solicited from the general 

public through the Federal Register or by other means is often required by the Administrative 

Procedure Act, other relevant statutes or both. Public comment may also be solicited for policy 

purposes. The Agency takes public comment on some strictly scientific products and almost all 

regulatory decisions. Public commenters usually include a broad array of people with an interest 

in the technical analysis or the regulatory decision; some are scientific experts (who may provide 

some peer input), some are experts in other areas, and some are interested non-experts. The 

critical distinction is that public comment does not necessarily draw the kind of independent, 

expert information and in-depth analyses expected from the peer review process. Public 

comment is open to all issues, whereas the peer 

review process is limited to consideration of 

specified technical issues. While it may be an 

important component of EPA’s decision making 

process, public comment does not substitute for peer 

review. 

Public comment does not substitute

for peer review.

1.2.9 How is Peer Review Different from Stakeholder Involvement?

Stakeholder involvement occurs when the Agency works with external interest groups that have 

some stake in or concerns about the outcome of the technical work product or regulatory 

position. This is an interactive process which usually involves other agencies, industry groups, 

regulated-community experts, environmental groups, other interest groups that represent a broad 

spectrum of the regulated community, etc., and usually strives for a consensus approach. The 

goal of peer review, on the other hand, is to obtain an 

independent, third-party review for ensuring scientific 

quality and technical credibility of the work product that 

supports a policy or decision. 

Stakeholder involvement is not 

a peer review mechanism.

1.2.10 What Role Does Peer Review Have in 

Regulatory Development? 

Peer review of scientific and/or technical work products that support regulations is an important, 

fundamental step in the policy setting and regulatory development processes and affirms the 

credibility of the Agency. Because the work products supporting regulations often are subject to 

intense scrutiny by the general public and the stakeholders involved, you should plan ahead and 

document the peer review process for such work products. 
 Peer Review Handbook Page 15 

In general, peer review should be completed prior to issuance of the proposed regulation. In 

some cases, work products that support final regulations may need an additional peer review if 

those scientific and/or technical work products change significantly after the public comment 

period. 

A regulation itself is not subject to the Peer Review Policy. However, if a regulation is 

supported by influential scientific information or a highly influential scientific assessment, the 

underlying work product should be peer reviewed before EPA issues the proposed regulation. 

See Sections 2.2.3 and 2.2.4 for a more complete explanation of what is considered “influential 

scientific information” or a “highly influential scientific assessment” and for guidance in 

determining whether a work product contains this type of information. Remember, soliciting 

comments from stakeholders and the general public after a rule is proposed does not constitute 

peer review. 

The original data and formal analytic models used in Economic Analyses (EAs) are subject to 

the OMB Bulletin if they constitute influential scientific information or highly influential 

scientific assessments. However, the straightforward application of data, models, and scientific 

and economic analyses used in previously peer-reviewed EAs and economic assessments or 

other peer-reviewed products are not subject to additional formal peer review. 

 1.2.11 How Does the Rulemaking Tiering Process Affect Peer Review? 

Tier 1 and Tier 2 rulemakings are, by definition, important Agency rulemakings3

. Therefore, you 

should carefully scrutinize work products supporting Tier 1 and Tier 2 rules to determine 

whether they should undergo peer review. In almost all cases you should use external peer 

review for a work product that is intended to support a Tier 1 or Tier 2 rulemaking. Although 

acceptable in certain circumstances for influential scientific information, the use of an internal 

peer review mechanism for such work products should be avoided. 

Work products supporting Tier 3 rulemakings may also benefit from peer review. For work 

products supporting a Tier 3 rule, internal or external peer review may be appropriate depending 

on the nature of the product and other factors. You can find more information on the differences 

between internal and external peer review in Section 2.4. 

 1.2.12 Should You Discuss Peer Review in the Analytic Blueprint for Your 

Regulation?

Workgroups should specifically address peer review in each analytic blueprint. For peer review 

purposes, development of the analytic blueprint is the process whereby the workgroup identifies 

 

3 For further information on tiering and criteria used to determine the appropriate tier for an action, see 

http://intranet.epa.gov/adplibrary/adp/tiering/step2.htm
 Peer Review Handbook Page 16 

the supporting scientific and/or technical work products and recommends what kind of peer 

review is needed. The analytic blueprint should show the schedule of the peer review in the 

context of the schedule for the overall rulemaking. 

 1.2.13 What Role does Peer Review Have in Regulatory Negotiations? 

Regulatory negotiations are not generally subject to peer review; however, to ensure final 

decisions are based on sound and credible science, where possible, work products that support 

the negotiation should be subjected to peer review before the negotiation takes place. 

 1.2.14 Should You Address Peer Review in the Preamble of a Regulation? 

Generally, yes. For proposed and final regulations that rely on influential scientific information 

or a highly influential scientific assessment, you should discuss the peer review report in the 

preamble, as described in the OMB Bulletin. If you are not the rulewriter, take steps to ensure 

the regulatory workgroup is aware of this provision of the OMB Bulletin. See Appendix C – 

Sound Science and Peer Review in Rulemaking, for template information. 

 1.2.15 How is Peer Review Documented in the Action Memo for Regulations?

For all rules requiring the Administrator's signature (proposed and final), indicate in the action 

memo what kind of peer review took place. See Appendix C for more information. 

1.3. Peer Review and Information Quality 

 1.3.1 Overview Statement

EPA is committed to providing quality environmental information to its partners and the public. 

This commitment is integral to its mission to protect human health and the environment. One of 

its goals is that all parts of society have access to accurate information sufficient to effectively 

participate in managing human health and environmental risks. To fulfill this and other 

important goals, EPA strives to rely upon information of appropriate quality for each decision 

made. In early 2002, pursuant to the Information Quality Act4

, the OMB developed governmentwide guidance for “ensuring and maximizing” the quality of information Federal agencies 

disseminate to the public.5

 In late 2002, EPA developed and issued its Information Quality 

 

4

The Information Quality Act, also known as the Data Quality Act, was issued by Congress, under Section 

515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554; 

H.R. 5658)

5

Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information 

Disseminated by Federal Agencies, OMB, 2002 (67 FR 8452). 
 Peer Review Handbook Page 17 

Guidelines6 for ensuring the information that supports EPA’s mission is reliable and accurate 

(objectivity); is appropriate for the intended use (utility); and, is protected from compromise 

(integrity). Peer review, when appropriate, enhances the quality of the information EPA 

disseminates. 

1.3.2 How Does Peer Review Ensure the Quality of Information EPA 

Disseminates?

The Agency recognizes peer review as a component of pre-dissemination review that 

complements and enhances the “objectivity” and “utility” of EPA’s information products. The 

Agency recommends that offices conduct pre-dissemination reviews of information to ensure 

that the information is of appropriate quality before it is disseminated to the public. Predissemination review is especially important for influential scientific information and highly 

influential scientific assessments. 

1.3.3 How Do You Ensure the Quality of Data Supporting EPA’s Information 

Products?

EPA’s Quality System7

 provides guidance on systematic planning for work products that involve 

the collection of new environmental data or the use of existing environmental data. The 

Agency’s policies for planning for the appropriate level of quality assurance and peer review 

ensure that the information the public receives is reliable and defensible. Documentation of this 

planning occurs at various stages in development of the final product: 

a) A Quality Assurance Project Plan or similar documentation is developed and approved 

prior to the collection of new data, or use of existing data. Systematic planning helps 

ensure that the data and information are of the right type, quality and quantity for the 

conclusions reached in the final work product. 

b) EPA’s Action Development Process (ADP)8

 is another mechanism that assists the 

Agency in achieving the objectivity and transparency of information used in developing 

 

6

Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information 

Disseminated by the Environmental Protection Agency, EPA/260R-02-008 (October 2002), 

(http://www.epa.gov/quality/informationguidelines) 

7

EPA’s Quality System provides guidance on systematic planning for the management, collection and use 

of all environmental data to ensure the data are of the right type, quantity and quality for the intended use. EPA 

Quality Manual for Environmental Programs 5360 A1. (May 2000), Section 1.3.1, 

http://www.epa.gov/QUALITY/qs-docs/5360.pdf

8

EPA's Action Development Process: Guidance for EPA Staff on Developing Quality Actions, EPA June 

2004, http://intranet.epa.gov/adplibrary/index.htm
 Peer Review Handbook Page 18 

regulations, economic and policy decisions. The stakeholder involvement and 

consultation that are part of the ADP enhance EPA’s efforts to disseminate quality 

information that is useful to the public, and are a valuable component of predissemination review. 

c) The guidance entitled General Assessment Factors for Evaluating the Quality of 

Scientific and Technical Information9

 is an additional resource for EPA’s staff as they 

evaluate the quality and utility of information, regardless of the source. The document 

complements EPA’s Information Quality Guidelines by raising the awareness of the 

information-generating public about EPA’s ongoing interest in ensuring and enhancing 

the quality of information available for Agency use. 

d) Peer review may be an important assessment tool. For example, peer review may be used 

by researchers to confirm the robustness of a theory or design before proceeding to the 

next level, such as field sampling or fabrication of new equipment. This is particularly 

helpful in "cutting edge" situations where technical standards or other measures of 

validation may not yet exist. 

1.3.4 Are Work Products Undergoing Peer Review Subject to the Guidelines?

Products that are undergoing peer review are not considered to be disseminated under EPA’s 

Information Quality Guidelines because they are dynamic documents that are subject to change 

and therefore, do not represent EPA’s final decision or position. These products should contain 

the following disclaimer: 

This information is distributed solely for the purpose of pre-dissemination peer 

review under applicable information quality guidelines. It has not been formally 

disseminated by EPA. It does not represent and should not be construed to 

represent any Agency determination or policy.

 

In cases where the information is highly relevant to specific policy or regulatory deliberations, 

this disclaimer should appear on each page of the work product. Peer review products that are 

disseminated are subject to EPA’s Information Quality Guidelines. 

 

9 General Assessment Factors for Evaluating the Quality of Scientific and Technical Information, EPA 

(June 2003), http://www.epa.gov/osa/spc/pdfs/assess2.pdf
 Peer Review Handbook Page 19 

1.4 Annual Agency Reporting

1.4.1 What Are EPA’s Reporting Practices and What Information Is Publicly 

Available?

EPA maintains a database, called the Science Inventory (http://cfpub.epa.gov/si), which is 

designed to track and report science activities across the Agency, including their status and peer 

review plans (See section 1.4.2). Offices are expected to keep this information current; agenda 

entries for the highly influential scientific assessments and influential scientific information 

products should be updated at least every six months. Real-time updates may occasionally be 

necessary, for instance if there is an imminent change in the timing for the peer review of a high 

visibility study or a change in the timing of the public availability of a draft of a highly 

influential assessment. 

EPA will make its peer review plans for highly influential scientific assessments and influential 

scientific information available for public comment through the Science Inventory. Offices are 

to consider any comments that are submitted on the peer review plans for which they are 

responsible. The OMB Bulletin also calls for final peer review reports for the highly influential 

scientific assessments and influential scientific information to be developed and made available 

to the public. EPA will make these reports available on the Science Inventory web site, which 

will directly link to the peer review agenda entry for that item. Additional information on the 

content of these reports is provided in Section 2.5. 

The Deputy Administrator has designated the Office of Research and Development (ORD) to 

conduct an annual review of the peer review plans. As called for in OMB's Peer Review 

Bulletin, ORD expects to submit a report to OMB by December 15 of each year. This report will 

include information concerning the peer reviews conducted on the highly influential scientific 

assessments and influential scientific information during the previous fiscal year. ORD will 

generate this report from the information in the Science Inventory database. 

1.4.2 What Information Is Provided in the Science Inventory? 

The Science Inventory database is designed to track and report peer review and other science 

activities across the Agency. The database is the single repository for product-specific peer 

review reporting and tracking and uses a common reporting form. Peer review work products 

are divided into three categories: highly influential scientific assessments, influential scientific 

information, and other products. 

Each AA/RA will designate a Peer Review Coordinator (See section 1.5.4) to ensure that their 

science activities and peer review products are fully represented, accurate, and complete. 

Specific information is included for each work product, including a summary or abstract of the 

product, what the product will be used for, and whether the product will undergo peer review. If 

the product will undergo peer review, additional information regarding the peer review plans is 
 Peer Review Handbook Page 20 

provided. If the activity or product is considered to be influential scientific information or a 

highly influential scientific assessment, the following information should be provided to address 

the provision of the OMB Bulletin that allows the public to view and comment on the Agency’s 

peer review plans for these activities or products: 

a) A paragraph including the title, subject and purpose of the activity or product; 

b) An Agency contact to whom inquiries may be directed to learn the specifics of the peer 

review plan; 

c) Whether the dissemination is likely to be influential scientific information or a highly 

influential scientific assessment; 

d) The timing of the review (including deferrals); 

e) Whether the review will be conducted through a panel or individual letters (or whether an 

alternative procedure will be employed); 

f) Whether there will be opportunities for the public to comment on the work product to be 

peer reviewed and, if so, how and when these opportunities will be provided; 

g) Whether EPA will provide significant and relevant public comments to the peer 

reviewers before they conduct their review; 

h) The anticipated number of reviewers (3 or fewer; 4-10; or more than 10); 

i) A succinct description of the primary disciplines or expertise needed in the review; 

j) Whether reviewers will be selected by EPA or by a designated outside organization; 

k) Whether the public, including scientific or professional societies, will be asked to 

nominate peer reviewers. 

If the product will not undergo peer review, Offices are asked to provide a statement on why peer 

review is not necessary (see Section 2.3 for further details). 

Each record should be electronically signed by the Peer Review Coordinator. By signing it, the 

Peer Review Coordinator indicates that it is ready for ORD review and, in the case of highly 

influential scientific assessments and influential scientific information, is ready to be made 

available for public comment. 
 Peer Review Handbook Page 21 

1.4.3 How Will the Public Be Allowed to Comment on the Peer Review Plans for 

Influential Scientific Information and Highly Influential Scientific 

Assessments?

The OMB Bulletin directs Federal agencies to make the peer review plans for influential 

scientific information and highly influential scientific assessments publicly available for 

comment. EPA will provide for this by making electronic files of the peer review plans available 

on the Science Inventory. The public can then comment on these plans and comments will be 

directed to the peer review contact identified for the product. Offices are to consider the 

comments submitted by the public on the peer review plans for highly influential scientific 

assessments and influential scientific information, in accordance with the OMB Bulletin. 

1.4.4 What Is ORD’s Role in Reviewing the Science Inventory and How Will This 

Review Be Conducted?

ORD conducts an annual review of the peer review products in the database. The purpose of this 

review is to ensure that the Science Inventory fully represents the Agency’s peer review products 

and to determine if the peer review decisions are consistent with the guidance presented in this 

Handbook. ORD will consult, as necessary, with the appropriate persons in each organization 

(see Section 1.5). ORD will then consolidate the information and findings for the SPC and the 

Deputy Administrator. Any conflicts arising from the review will be resolved by the Deputy 

Administrator. 

 1.4.5 How Will EPA Submit an Annual Report to OMB?

EPA expects to submit an annual report to OMB by December 15 of each year that summarizes 

the peer reviews that were conducted during the previous fiscal year for highly influential 

scientific assessments and influential scientific information. Release of any reviewer 

information retrieved by a personal identifier will be performed in accordance with the Privacy 

Act, 5 USC § 552a as amended, and as interpreted in OMB implementing guidance, 40 FR 

28,948 (July 9, 1975). According to the OMB Bulletin, the annual report should include the 

following: 

a) The number of peer reviews conducted subject to the Bulletin 

b) The number of times alternative procedures were invoked 

c) The number of times waivers or deferrals were invoked (and in the case of deferrals, the 

length of time elapsed between the deferral and the peer review) 

d) Any decision to appoint a peer reviewer pursuant to any exception to the applicable 

independence or conflict of interest standards of the Bulletin 
 Peer Review Handbook Page 22 

e) The number of peer review panels that were conducted in public and the number that 

allowed public comment 

f) The number of public comments provided on the peer review plans, and 

g) The number of peer reviewers used who were recommended by professional societies. 

1.5 The Roles of People and Organizations in Peer Review

1.5.1 Who is Ultimately Accountable for Peer Review?

Under the Peer Review Policy, the Administrator has designated the Assistant Administrators 

and Regional Administrators (AAs and RAs) to be accountable for implementing the Policy in 

their respective organizations. The Deputy Administrator is ultimately responsible for peer 

review across the Agency and is the final arbitrator of conflicts and concerns about peer review. 

1.5.2 Who are the Agency Staff involved in Peer Review?

The principal Agency staff involved are Decision Makers (and their line managers), Peer Review 

Leaders and Peer Review Coordinators. In addition, there are many other critical staff in each 

Office and Region who have responsibility for peer review activities (e.g., Office and Division 

Peer Review Coordinators, technical information managers, and, of course, any Agency staff 

who serve as internal peer reviewers). Finally, ORD has oversight responsibility, as designated 

by the Deputy Administrator, for ensuring the Agency’s Peer Review Policy is implemented. 

1.5.3 Who are the Decision Makers and What are Their Responsibilities?

The AA/RA is the ultimate Decision Maker for his/her organization and is accountable for the 

decisions regarding the identification of influential scientific information and highly influential 

scientific assessments and the mechanism(s) of peer review utilized for each of the products. 

The AA/RA may designate Office Directors and/or Division Directors as the front-line Decision 

Makers. 

Furthermore, the Decision Makers commit the resources needed to ensure a proper peer review. 

Decision Makers are responsible for ensuring that the peer reviews are properly performed and 

documented. 

In order to ensure greater independence of peer reviews, it is important to strictly separate the 

management of work products from the actual peer review of those work products. Therefore, 

the Decision Maker and the Peer Review Leader for a work product should not be the same 

person. 
 Peer Review Handbook Page 23 

The Decision Maker needs appropriate training on how to manage the peer review process. The 

Peer Review Coordinator for the Office can advise the Decision Maker on how to obtain the 

appropriate training; the training approach varies among the different EPA offices. 

Specific responsibilities of the Decision Maker(s) are these: 

a) Determine which work products in their organization call for peer review and which ones 

are influential scientific information or highly influential scientific assessments; 

b) Designate (in conjunction with the Project Manager) a Peer Review Leader to organize 

each peer review; 

c) Provide advice, guidance, and support to the Peer Review Leader in the preparation, 

conduct, and completion of the peer review; 

d) Ensure that sufficient funds are designated in the office’s budget request to conduct the 

peer review; also ensure that adequate resources and/or extramural management support 

are available for the peer review; 

e) Establish a realistic peer review schedule; 

f) Designate the stage(s) of product development where peer review is appropriate; 

g) Consider public comments on the peer review plans and consider and decide if the public 

will be asked to nominate potential peer reviewers for influential scientific information 

and for highly influential scientific assessments; 

h) For highly influential scientific assessments, decide whether it is feasible and appropriate 

to make the draft scientific assessment available to the public for comment at the same 

time it is submitted for peer review (or during the peer review process) and whether it is 

feasible and appropriate to sponsor a public meeting where oral presentations on 

scientific issues can be made to the peer reviewers by interested members of the public. 

When public comment on the assessment is sought, provide peer reviewers with 

significant comments raised by the public; 

i) Ensure all relevant issues and comments raised by the peer reviewer(s) are adequately 

addressed and documented for the record, and where appropriate, incorporated into the 

work product that is used as basis for decision making. 

1.5.4 Who are the Peer Review Coordinators and What are Their Responsibilities?

The Peer Review Coordinator is designated by the AA/RA to coordinate and monitor peer 

review activities in his/her respective Office or Region. This person should be of sufficient 
 Peer Review Handbook Page 24 

stature and judgment to have the access to and confidence of all levels of Office or Regional 

management. The Peer Review Coordinator is the main contact for his/her organization and can 

also direct interested parties to other persons/contacts in the Office on specific work products 

(e.g., Peer Review Leader). 

Specific responsibilities of the Peer Review Coordinator are these: 

 a) General oversight responsibility for the Office’s or Region’s peer review process; 

 b) Report peer review activities to the AA/RA; 

 c) Help mediate difficult issues between the organization and others; if an issue cannot be 

resolved, then the Peer Review Coordinator can bring the issue to the attention of the 

appropriate level Decision Makers in each organization for resolution; 

 d) Function as the liaison with ORD and the Science Policy Council (SPC) to: 

1) Represent the Office/Region before the SPC on peer review issues; 

2) Advise ORD of any changes in the list of work products and peer review mechanisms 

during the annual reporting, and when necessary, at other times; 

3) Participate in Agency peer review training, workshops, etc., as requested and 

disseminate this information to the organization; coordinate and/or present training 

within the organization. 

 e) Submit information on the organization’s peer review plans and activities as needed to 

implement the Peer Review Policy (See Section 1.4). On a product specific basis, this 

data submission responsibility may be re-designated within the organization: 

 1) Assure that all Science Inventory records for peer review products are accurate and 

comply with OMB reporting requirements (see Section 1.4.2); 

 2) Assure the Science Inventory record addresses all issues and reporting requirements 

for the annual report to OMB and sign the record, making it available to the public 

via the Science Inventory website. 

 f) Establish procedures to ensure that the work product peer review documentation (i.e., 

peer review record) is filed and maintained in an appropriate manner (see Section 2.5); 

 g) Provide advice, guidance, and support to the various Peer Review Leaders for the 

performance of the peer reviews; 
 Peer Review Handbook Page 25 

 h) Distribute Agency-wide peer review guidance and materials to appropriate Office/Region 

personnel, as requested. 

1.5.5 Who are the Peer Review Leaders and What are Their Responsibilities?

The Peer Review Leader is assigned by the Decision Maker to organize, conduct and complete 

the peer review for a specific individual work product. The Peer Review Leader will obtain the 

assistance and support of the Peer Review Coordinator as well as any others within the Agency 

to help perform the peer review. The Peer Review Leader will be chosen on a case by case basis 

depending on the work product needing peer review. The Peer Review Leader cannot be the 

Decision Maker. The Peer Review Leader could be the Project Manager for the work product. 

The Peer Review Leader should have appropriate training on how to conduct a peer review 

before conducting the peer review. The Peer Review Coordinator for the Office can advise the 

Peer Review Leader on how to obtain the appropriate training; the training approach varies 

among the different EPA offices, however, uniform training modules are available. 

Specific responsibilities of the Peer Review Leader are: 

a) Keep the Decision Maker informed of the status of a given project; provide the Peer 

Review Coordinator with data for the annual report; 

b) Organize, conduct, and complete the peer review following Agency procedures: 

1) Establish and maintain the peer review record for the specific individual peer 

review currently being performed (see Section 2.5); this includes providing the 

peer review summary information in the Science Inventory for the Peer Review 

Coordinator to sign when the peer review is completed; 

2) Select the peer reviewers in consultation with others involved with the peer 

review (e.g., Decision Maker) and ensure that conflict of interest issues are 

addressed and documented in the peer review record (see Section 3.4.6); 

3) Advise peer reviewers of their responsibilities, including preparation of a peer 

review report if the product is influential scientific information or a highly 

influential scientific assessment. 

c) Provide information to the Decision Maker (including all appropriate managers in the 

Peer Review Leader’s chain of command) on the charge, profile of peer reviewers, the 

peer review comments, and a proposal on how to address the comments. Obtain Decision 

Maker approval on the approach to responding to peer reviewer comments. Clearly 

identify any peer review comments for the Decision Maker that will not be addressed in 
 Peer Review Handbook Page 26 

the agreed upon approach. (See Section 2.5.3 for specific contents of the peer review 

record): 

1) Make the peer review report for influential scientific information and highly 

influential scientific assessment publicly available; 

2) For highly influential scientific assessments, develop the Agency response to the 

peer review and make it publicly available. 

d) Notify the Peer Review Coordinator that the peer review is completed for the annual 

report; 

e) Archive the peer review record in a manner consistent with the organization’s archiving 

procedures. 

When a contractor is used to conduct a peer review, some of the above responsibilities are 

assumed by the contractor (see Section 3.6). In addition, when peer reviews are completed 

through a Federal Advisory Committee, some of the above responsibilities are assumed by the 

Designated Federal Officer (DFO). 

1.5.6 Who are the Peer Reviewers?

Peer reviewers are individuals who have technical expertise in the subject matter of the work 

product undergoing peer review. Peer reviewers can come from EPA, another Federal agency, 

or from outside of the Federal government. 

1.5.7 What are the Responsibilities of Peer Reviewers?

Peer reviewers need to be willing participants in the peer review process – they should agree to 

read all materials, participate fully, act ethically, and protect confidential information that arises. 

Peer reviewers should maintain the confidentiality of the product, perform the review in a timely 

manner, and be unbiased and objective. Peer Reviewers should not be given any products or 

data that may reveal confidential business information (CBI) unless the peer reviewer is internal 

to EPA and has the appropriate clearance (see Section 3.4.7). 

1.5.8 What is an Independent Peer Reviewer?

An independent peer reviewer is an expert who was not associated with the generation of the 

specific work product either directly by substantial contribution to its development or indirectly 

by significant consultation during the development of 

the specific product. The independent peer reviewer, 

thus, is expected to be objective (See Sections 1.2.6 

and 1.2.7 for further information). 

The quality of the peer review is 

dependent on the competence and 

independence of the reviewers.
 Peer Review Handbook Page 27 

Independence is freedom from institutional or ideological bias regarding the issues under review 

and is necessary for objective, fair, and responsible evaluation of the work product. If a selected 

reviewer has a particular scientific or technical perspective, it may be desirable to balance the 

review with peer reviewers with other perspectives. In addition to being independent, ideally, 

peer reviewers should be free of real or perceived conflicts-of-interest. Conflict of interest is 

different from independence and is discussed in Section 3.4.5 and 3.4.6. If there are potential 

conflicts of interest (real or perceived), they should be fully understood and addressed by the 

peer review leader (or other appropriate official) to ensure a credible peer review. 

1.5.9 When does an Agency Internal Peer Reviewer Qualify as Independent?

An Agency independent internal peer reviewer is one who comes from a different organizational 

unit than the one where the review question or document originates. A different organizational 

unit usually denotes, at minimum, a different Office (i.e., above Division level in programs; 

above Branch level in Regions) within the organization. In particular, a reviewer should not 

come from within the chain of command, either upward or downward. Agency staff may not 

serve as peer reviewers for the final peer review of a highly influential scientific assessment 

unless a specific exception is granted under Section III.3.c of the OMB Bulletin. 

1.5.10 What is a Peer Review Panel?

A peer review panel can range from a few individuals to ten or more, depending on the issue 

being investigated, the time available and any limitations on resources. Individuals who serve as 

peer reviewers should have appropriate scientific and technical expertise such that the review 

panel covers the broad spectrum of expertise necessary to address the issues/questions presented 

in the charge. 

1.5.11 What is a Subject Matter Expert?

A subject matter expert is one who has specific scientific and technical expertise in the matter 

under review. The importance of scientific and technical expertise in the subject matter is 

obvious, however, knowledge or just “knowing” about the subject area is not equivalent to 

expertise in the subject matter. For Agency decisions, a multi-disciplinary group of experts 

corresponding to the disciplines that contribute to complex Agency decisions is often necessary 

for a full and complete peer review. For example, a risk assessment that relies on both animal 

and human data usually would call for experts in both areas for a complete review. For 

economic analyses, experts from the economic disciplines are necessary. 

1.5.12 What is the Role of the Science Policy Council (SPC)?

According to the Peer Review Policy statement: “The Science Policy Council is responsible for 

overseeing Agency-wide implementation of this policy, including: promoting consistent 
 Peer Review Handbook Page 28 

interpretation; assessing Agency-wide progress; developing recommendations for revisions of 

the policy as necessary; and issuing the Peer Review Handbook, which provides additional 

information and procedures on implementing this policy.” The SPC meets its responsibilities 

through coordination with the Peer Review Coordinators, the Peer Review Advisory Group 

(PRAG) and the Office of Research and Development (ORD). 

The SPC, PRAG and ORD are not responsible for identifying specific products for peer review 

or determining the level of review or mechanism for that review; those functions are the 

responsibility of management within each Office or Region. 

 1.5.13 What is the Role of the Peer Review Advisory Group (PRAG)?

The Science Policy Council has created the PRAG to assist in the implementation of the 

Agency's Peer Review Policy. The primary role of the PRAG is to provide interpretation of the 

policy and to assist the SPC and Agency Offices and Regions in preparing updates to the Peer 

Review Handbook. 

 1.5.14 What is the Role of the Office of Research and Development (ORD)?

The Deputy Administrator has designated the Office of Research and Development (ORD) to 

provide oversight to the Program Offices and Regions in the collection and review of 

information that is contained in the Science Inventory (for detailed information, see Section 1.4). 
 Peer Review Handbook Page 29 

2. Planning A Peer Review

2.1 Overview Statement

Planning a peer review is a critical first step to ensuring a successful peer review of a work 

product. The initial step is to determine whether your work product should be peer reviewed. 

Once you have determined that a peer review will be conducted, the Decision Maker and Peer 

Review Leader need to plan an appropriate review. This includes the determination of resources 

(budget and personnel), schedule for the completion of the peer review, mechanism for peer 

review, development of the peer review record, and, finally, the selection of peer reviewers. 

2.2 Determining Which Work Products to Peer Review

2.2.1 What are Scientific and/or Technical Work Products?

The first step in determining which work products should be peer reviewed is to identify 

products that are scientific and/or technical in nature. The term scientific and/or technical work 

products is generally consistent with the term “scientific information” in the OMB Bulletin10. 

Scientific and/or technical work products are used to support a research agenda, regulatory 

program, policy position or other Agency position or action. Scientific and/or technical work 

products include economic and social science work products. Categories of work products 

include, for example: risk assessments, technical studies and guidance, analytical methods, 

scientific database designs, technical models, technical protocols, statistical survey/studies, 

technical background materials, technical guidance (except for guidance providing policy 

decisions), research plans, and research strategies (re journal articles, see section 2.3.4). 

Products that would not be considered scientific and/or technical work products can include 

those: that address procedural matters (e.g., planning, reporting, coordination, notification); that 

are primarily policy statements (e.g., relocation policy); that are conference proceedings (unless 

the proceedings are used as the scientific basis for an Agency action or decision); or that are 

decision documents, e.g., Environmental Impact Statement (EIS), Record of Decision (ROD), or 

Economic Analysis (EA). For any of these, the decision document itself is not subject to the 

Peer Review Policy, but its underlying scientific and/or technical models, data, and/or work 

products upon which these documents are based are candidates for peer review. In addition, the 

following Agency documents are not considered scientific and/or technical work products under 

the Peer Review Policy: strategic plans, analytic blueprints, and goals documents.

 

10OMB defines “scientific information” as “factual inputs, data, models, analyses, technical information, or 

scientific assessments based on the behavioral and social sciences, public health and medical sciences, life and earth 

sciences, engineering, or physical sciences. . .” (OMB Bulletin, Section I.5.)
 Peer Review Handbook Page 30 

Making final determinations concerning which work products are scientific and/or technical is a 

responsibility of the Decision Maker (See Section 1.5.3). All scientific and/or technical work 

products should be entered into the peer review component of the Science Inventory. 

2.2.2 What Scientific and/or Technical Work Products Need Peer Review?

The principle underlying the Peer Review Policy is 

that all influential scientific and technical work 

products used in decision making will be peer 

reviewed. The process for identifying which 

products are “influential” and then determining the 

mechanism of review will take into account various 

criteria and the circumstances surrounding the use of 

the work product. To maintain flexibility, the Decision Maker(s) for peer review should consider 

the full field of possible work products to identify those additional products that might still 

warrant peer review as well as the full spectrum of peer review mechanisms for each product. 

When in doubt about whether a 

work product merits peer review, 

decide to peer review it.

OMB defines influential to mean information that an agency reasonably can determine will have 

or does have a clear and substantial impact on important public policies or private sector 

decisions. EPA has provided further guidance on making determinations with respect to whether 

a document is “influential” in its Information Quality Guidelines.11 Once a product has been 

identified as being influential scientific information, the Decision Maker should determine 

whether it meets OMB’s definition of a “highly influential scientific assessment” (see Section 

2.2.4). The OMB Bulletin calls for additional peer review procedures for highly influential 

scientific assessments. 

2.2.3 How Does One Determine Whether a Scientific and/or Technical Work 

Product is Influential Scientific Information?

Generally, determinations whether a scientific and/or technical work product is “influential” will 

occur on a case-by-case basis. The continuum of work products covers the range from the 

obviously influential, which clearly need peer review, to those products which clearly are not 

influential and don’t need peer review. The majority of EPA’s work products fall in-between 

those two ends of the continuum and should be closely evaluated and assessed with respect to 

certain factors (see below). The Decision Maker should make the judgment as to whether a work 

product is influential scientific information. There is no easy, single “yes/no” test that applies to 

the whole continuum of work products for determining whether a work product is influential 

 

11Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information 

Disseminated by the Environmental Protection Agency, EPA/260R-02-008 (October 2002), 

(http://www.epa.gov/quality/informationguidelines) 
 Peer Review Handbook Page 31 

scientific information. A useful rule of thumb is that if there is any doubt about whether a work 

product needs peer review, go ahead and subject it to peer review. 

Generally, scientific and/or technical work products that are used to support a regulatory 

program or policy position and that meet one or more of the following factors would be 

considered to be influential scientific information: 

a) Establishes a significant precedent, model, or methodology; 

b) Likely to have an annual effect on the economy of $100 million or more, or adversely 

affect in a material way the economy, a sector of the economy, productivity, competition, 

jobs, the environment, public health or safety, or State, Tribal, or Local governments or 

communities; 

c) Addresses significant controversial issues; 

d) Focuses on significant emerging issues; 

e) Has significant cross-Agency/interagency implications; 

f) Involves a significant investment of Agency resources; 

g) Considers an innovative approach for a previously defined problem/process/methodology; 

h) Satisfies a statutory or other legal mandate for peer review. 

Historically, EPA has defined products subject to peer review in terms of having “major impact.” 

“Major impact” means that the work product may have applicability to a broad spectrum of 

regulated entities and other stakeholders, or that it will have narrower applicability, but with 

significant consequences on a smaller geographic or practical scale. The Agency developed the 

previously listed factors to consider when determining if a product had a “major impact.” The 

Agency has also linked its use of the term "influential" to the term “major” in its Information 

Quality Guidelines. The OMB Peer Review Bulletin provides similar factors to use in 

determining if a scientific assessment is highly influential, thereby creating some overlap 

between the two definitions. Scientific assessments that meet the criteria of both influential and 

highly influential should be considered as highly influential. 

Typically, an influential scientific and/or technical work product that has a reasonable likelihood 

of supporting a regulatory decision or policy/guidance of major impact may also have a clear and 

substantial impact on private sector decisions. The scientific and/or technical work products that 

underlie many of the Agency’s major rulemakings, policy and guidance documents of general 

applicability would be designated “influential” under this construct because of their far-reaching 

impacts. 
 Peer Review Handbook Page 32 

The novelty or controversy associated with the work product may determine whether it is 

influential scientific information. Influential scientific information may be novel or innovative, 

precedential, controversial, or emerging (“cutting edge”). An application of an existing, 

adequately peer-reviewed methodology or model to a situation that departs significantly from the 

situation it was originally designed to address may make peer review appropriate. Similarly, a 

modification of an existing, adequately peer-reviewed methodology or model that departs 

significantly from its original approach may also make peer review appropriate. Determining 

what constitutes a “significant departure” as used in this Section is the responsibility of the 

Decision Maker. 

 

In summary, an influential scientific or technical work product has a major impact, involves 

precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory 

obligation to conduct a peer review. 

2.2.4 How Does One Determine Whether Influential Scientific Information is a 

Highly Influential Scientific Assessment?

Once a scientific and/or technical work product has been categorized as “influential scientific 

information,” the Decision Maker should determine whether the product meets OMB’s definition 

of a highly influential scientific assessment (see also Figure 4). As with the categorization of a 

work product as influential scientific information, the decision whether or not to elevate a work 

product to the highly influential category occurs on a case-by-case basis and is largely based on 

evaluation of the same factors as identified above.

ote that “highly influential scientific 

f

as

sment

All work products EPA may 

generate or use

Influential Scientific Information

Highly Influential Scientific

Assessments

N

assessments” differ from “influential 

scientific information” both in terms o

their degree of influence and in 

substance. OMB defines highly

influential scientific assessments

influential scientific information that

“the agency or the Administrator 

determines to be a scientific asses

a) Could have a potential impact of 

 

b) Is novel, controversial, or 

terest.”

that: 

more than $500 million in any 

year, or Figure 4-EPA Work Products

This diagram depicts the relation between all EPA work 

products and those considered Influential Scientific 

precedent-setting or has Information or Highly Influential Scientific Assessments 

significant interagency in
 Peer Review Handbook Page 33 

OMB defines a scientific assessment as “an evaluation of a body of scientific or technical 

”

12. 

he OMB Bulletin includes the following examples of assessments that could be considered 

) State-of-science reports; 

ogy assessments; 

of-evidence analyses; 

alyses; 

safety, or ecological risk assessments; 

gical characterizations of substances; 

d assessment models; 

inations; 

 Exposure assessments. 

MB’s definition of a highly influential scientific assessment overlaps significantly with the 

nt 

he more far-reaching or significant the impacts of a scientific assessment are, the more 

e 

th a 

determinations that should be made by the Decision Maker. 

 

knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions,

and/or applies best professional judgment to bridge uncertainties in the available information

T

highly influential scientific assessments: 

a

 

b) Technol

 

c) Weight-

 

d) Meta-an

 

e) Health, 

 

f) Toxicolo

 

g) Integrate

 

h) Hazard determ

i)

O

factors you should consider when assessing whether a work product is influential scientific 

information. The decision to categorize a product as a highly influential scientific assessme

first depends on determining whether the influential scientific information is a “scientific 

assessment,” and then on assessing the likely degree of the product’s impact. 

T

appropriate it is to categorize the product as a highly influential scientific assessment. Th

determination as to whether the degree of novelty, precedent, and controversy associated wi

particular assessment warrants designation as a highly influential scientific assessment should be

made by the Decision Maker. If the influential scientific assessment involves significant issues 

that are truly “cutting edge,” it may be appropriate to designate it as a highly influential scientific

assessment. The greater the “significant departure,” as discussed above, the more appropriate it 

is to include the product as a highly influential scientific assessment. These are subjective 

12OMB Bulletin, Section I.7. 
 Peer Review Handbook Page 34 

OMB also directs agencies to consider the degree of potential economic impact of a scientific 

ssessment in deciding whether it should be categorized as a highly influential scientific 

e 

Ec nom l hnical work products, 

nd as such it may be appropriate to peer review them. If those work products are ISIs, then they 

 or protocol 

used to conduct economic analyses within a program; 

c) financial methodologies, 

particularly those that are recognized to be outside of mainstream economic practices; 

d)

Congressional mandates (e.g., the Clean Air Act reports to Congress on benefits and 

e) reference (e.g., contingent valuation) and revealed preference surveys (e.g., 

recreational travel cost surveys) developed to assist in the economic analysis of a 

f) s and expenditures for environmental protection (e.g., financial 

needs surveys, pollution abatement expenditures surveys); 

g) he state-of-science in 

economic theory, methodologies or modeling (in particular, the technical feasibility of 

h) alyses of existing published literature and supporting data on the 

measurement of economic benefits, costs and impacts); 

i) s, particularly those supporting 

economically significant rules, if the models and corresponding use of the data have not 

been previously subject to adequate peer review. 

a

assessment. As explained above, OMB defines highly influential scientific assessments as thos

scientific assessments that may have an impact of more than $500 million in any year. 

 2.2.5 What Economic Work Products Need Peer Review?

o ic ana yses (EA) and reports are considered scientific and/or tec

a

should be peer reviewed if they have not already been subject to adequate peer review (See 

section 2.2.3). The following economic work products. should generally be peer reviewed. 

a) Internal Agency guidance for conducting economic and financial analysis; 

b) Economic and financial methodologies that will serve as a principal method

Unique or novel applications of existing economic and

Broad-scale economic assessments of regulatory programs, such as those required by 

costs); 

Stated p

regulation or program; 

National surveys of cost

Economic research plans developed to assess and advance t

the plan’s components); 

Meta-analyses (i.e., re-an

Data and analytical models underlying economic analyse
 Peer Review Handbook Page 35 

If th e

should 

Adviso ommittee, other appropriate outside 

rganizations, or individuals that have expertise in the technical economic issues raised in the 

?

enerally, if an EA applies accepted, previously peer-reviewed methods in a straightforward 

manner u

“economically

pically do not utilize innovative or untried economic methods. It is unnecessary to conduct 

en 

ial 

some 

 as part of the regulatory development 

rocess, including input received from other EPA offices represented on the workgroup for the 

at 

 a 

petition, jobs, the 

nvironment, public health or safety, or State, Local, or Tribal governments or communities. 

es types of economic work products support a highly influential scientific assessment, you 

conduct an external peer review. External peer reviews can be provided by the Science 

ry Board’s Environmental Economics Advisory C

o

economic work product. 

2.2.6 Should Economic Work Products Prepared in Support of Regulations that

are Classified as "Major" or "Economically Significant" be Peer Reviewed

G

, it wo ld not undergo an additional peer review. EAs prepared to support “major” or 

 significant” regulations (these terms are defined at the end of this Section) 

ty

peer reviews of straightforward applications or transfers of accepted, previously peer-reviewed

economic methods or analyses. The procedures used to transfer or adapt an economic work 

product are generally established by separate economic guidance documents which have be

peer reviewed. Therefore, EAs that are developed using these procedures do not normally 

undergo an additional peer review, even those prepared in support of “major” and “economically

significant” rules. To the extent that the underlying data and models are considered “influent

scientific information,” they are subject to peer review, including questions regarding the 

applicability of the specific data set or model for the use. 

Even when peer review is not needed, additional peer input can be beneficial in the development 

of economic work products for “major” and “economically significant” rules. At present, 

peer input of these analyses is already likely to be included

p

rule, from the Agency’s Regulatory Steering Committee, and from the public as part of the 

public comment process for the rule. But there may be added benefit to employing additional 

peer input procedures, such as actively soliciting input from economists in other Federal 

Agencies on the quality and completeness of the economic analysis. 

Under Section 3(f)(1) of Executive Order 12866, “economically significant” rules are those th

may have an annual effect on the economy of $100 million or more or adversely affect in

material way the economy, a sector of the economy, productivity, com

e

The term “major,” as defined in the Congressional Review Act, means a rule that has resulted in 

or is likely to result in: an annual effect on the economy of $100 million or more; a major 

increase in costs or prices for consumers, individual industries, Federal, State, or Local 

government agencies, or geographic regions; or significant adverse effects on competition, 

employment, investment, productivity, innovation, or on the ability of United States-based 

enterprises to compete with foreign-based enterprises in domestic and export markets. 
 Peer Review Handbook Page 36 

 2.2.7 What Other Economic Work Products Might Benefit from Peer Review?

There may be other economic work products not covered in the preceding sections for which 

peer review might be useful. Examples of such work products are presented below; however, by 

listing these examples, there is no intent to establish a presumption of peer review for these work 

products. 

a) Analyses measuring the economic impacts and effectiveness of adopting market-based or 

economic incentives as regulatory management instruments; 

b) The technical analyses supporting economic policies established under other government 

organizations (e.g., economic models used to study transportation, economic 

development, and international trade policies). 

 

Most of these types of economic work products do not exhibit the degree of complexity, or 

establish an innovative or untried approach, that would warrant a peer review. However, other 

factors, such as the potential significance of the analysis for cross-Agency or interagency 

practices, or the significance of the issues addressed, may make peer review desirable. 

 2.2.8 What Other Social Science Work Products Need Peer Review?

Typically, a social science work product is one that includes empirical, logic-based approaches 

to answer technical questions about human motivation, human behavior, social interactions, and 

social processes, which are relevant to the environmental issues being addressed. The term 

“behavior” includes overt actions; underlying psychological processes such as cognition, 

emotion, temperament and motivation; and bio-behavioral interactions. The term “social” 

includes sociocultural, socioeconomic, and sociodemographic status; biosocial interactions; and 

the various levels of social context from small groups to complex cultural systems. Examples of 

social science work products include analyses and/or evaluations related to such topics as 

pollution prevention, risk communication, environmental information, environmental justice, 

quality of life, decision-making, and public participation. 

The following social science work products should normally undergo peer review: 

a) Internal Agency guidance for conducting social impact assessments and other 

community cultural assessments related to different environmental protection approaches 

such as community-based watershed protection (heretofore referred to as social 

assessments); 

b) New social science methodologies that will serve as a principal method or protocol used 

to conduct social assessments; 
 Peer Review Handbook Page 37 

c)

groups, interviews, network analyses, comparative analyses, and content analyses; 

e) Innovative research or analyses that address the human dimensions of environmental 

feasibility 

gency to 

se for a specific purpose. The receiving agency's guidance for peer review may likely be 

dif ren me minimal 

ommon guidance for the Federal government. Regardless, if EPA plans to use any work 

s 

Unique or novel applications of existing social science methods such as surveys, focus

d) New national surveys of values, perceptions and preferences related to environmental 

protection; 

protection or environmental change in terms of social trends, future predictions and/or 

behavioral generalizations; 

f) Social science research plans developed to assess and advance the state-of-science in 

social science theory, methodologies or modeling (in particular, the technical 

of the plan’s components). 

2.2.9 How Should Peer Review be Handled for Products Developed Under an 

Interagency Agreement (IAG)?

Under an interagency agreement (IAG), EPA provides funds to another agency for that a

u

fe t from EPA Peer Review Policy, although the OMB Bulletin establishes so

c

products from that agreement, EPA should decide whether those documents need review under 

EPA Peer Review Policy. 

2.2.10 Should Products from Contracts, Grants, and Cooperative Agreement

Receive Peer Review?

If there is a scientific and/or technical work product resulting from a grant, contract, or 

cooperative agreement and it is considered influential scientific information or a highly 

influential scientific assessment and it will likely be used in Agency decision-making, the work

product needs peer review. Since it would probably result in a perceived, if not real, conflict of 

terest, the group that is generating the work product usually cannot conduct or perform the peer 

rganizations 

at have adequate and well established recognized procedures for peer review, such as the 

Nat w the 

produc

vehicle cal work 

products from contracts, grants, or cooperative agreements to support decision making unless the 

ork products are peer reviewed for both scientific and technical rigor and applicability to the 

spe

e aware that contracts are very different from grants and cooperative agreements. Please note 

that there are important legal restrictions on the direct use of work products developed under 

in

review of its own work product. Exceptions may be made in certain instances for o

th

ional Academy of Sciences (NAS). In practice, the Agency may need to peer revie

t on its own, or arrange with an independent third group (e.g., via another extramural 

) to conduct the peer review. The Agency should not use scientific and/or techni

w

cific use to be made of the product. 

B
 Peer Review Handbook Page 38 

gra a

http://in

nts nd cooperative agreements in the Agency’s decision-making process (see 

tranet.epa.gov/ogd/) for additional information). 

 s 

operative Agreements?

It m

through

PA may legally provide financial assistance for research that is intended to stimulate or support 

dev p

resultin

journal opposed to a report tailored to EPA’s specific needs and 

quirements. EPA can consider these work products just as it can review other published 

sc ntif free, 

nonexclusive a products for Federal purposes, even if the 

cipient has copyrighted the material (see 40 CFR 30.36(a)). 

ld be 

he following are options for peer reviewing the product: 

a) EPA can have the product peer reviewed with the participation of the assistance 

2.2.11 How Does Peer Review Apply to Products Generated Through EPA Grant

or Co

ay be desirable to peer review scientific and/or technical work products that are generated

 EPA grants or cooperative agreements, but special considerations apply. 

E

elo ment of scientific knowledge that is not primarily for EPA’s direct use or benefit. The

g work products might be widely disseminated either through publication in scientific 

s or through other means, as

re

ie ic works when formulating its programs and policies. Further, EPA retains a royalty 

nd irrevocable right to use the work

re

EPA may determine that the recipient’s work product is influential scientific information 

because: 1) it will be used to support an EPA program or policy position, and 2) it meets one or 

more of the seven factors outlined in Section 2.2.3. In this situation, the work product shou

peer reviewed. 

T

agreement recipient/author. In this case, EPA could arrange for an independent peer 

review of the product itself, or may contract with a third party to conduct the review. 

EPA would also enter into a contract with the author (formerly the recipient), wh

would task the author to prepare a response to the peer reviewers’ comments, and to 

revise or prepare an addendum to the product in response to peer reviewer comments as 

determined appropriate by EPA. 

A caveat to this approach is that it may be difficult to get the recipient/author to agre

allow EPA to determine how to revise the product in response to the peer review 

comments. 

b) 

ich 

e to 

EPA can have the product peer reviewed without the participation of the recipient/author. 

EPA could arrange for the peer review itself, or could contract with a third party to 

conduct the review. In this case, however, the work product would not be revised to 

incorporate the peer review comments. Instead, EPA would receive the comments and 

prepare a statement that documents EPA’s own response to the comments. The EPA 

Decision Maker who is using the work product to support an EPA program or policy 
 Peer Review Handbook Page 39 

decision should be provided information on both the conclusions of the rec

product and EPA’s own conclusions from the pee

ipient’s work 

r review. 

c) Recipients can get their products peer reviewed on their own. Recipients may determine 

on their own that peer review would benefit the credibility of their product. Provided 

EPA agrees that a peer review would further the public purpose of the assistance 

r 

ts for, and 

fund, an independent peer review of their product. In either case, EPA would need to 

he 

t 

iversity under an assistance 

greement, even where the agreement does not specifically provide for this access. Nevertheless, 

t to 

e of the agreement, we also have a specific right to authorize peer reviewers to 

se the data for Federal purposes under 40 CFR 30.36(c)(2). 

PA cannot obtain access to the raw data if EPA specifically bargains away this right in the 

as n

Again, 

for you

ers of their Work Products?

As note

purpose

agreem ent to peer reviewers in exchange for their review of a scientific and/or 

chnical work product is allowable as a fee for professional services under assistance 

agreem

referred , 

Profess em 

39, Pro

Attach

Govern

agreement, EPA may include funds for the peer review in the agreement. (See 2.2.10 fo

additional information.) Alternatively, the recipient may make arrangemen

evaluate whether the peer review process undertaken by the recipient was acceptable for 

the purposes for which EPA was planning to use the work product. EPA may accept the

peer review if it determines that it is of appropriate quality and that EPA could defend t

peer review as if it were conducted by EPA itself. 

Under options a) or b), issues may arise over obtaining access for peer reviewers to the raw data 

used by the recipient to generate the work product. Under 40 CFR 30.36(c)(2), EPA has a righ

to obtain raw data produced by a non-profit organization or un

a

it may be prudent to include a specific term in the assistance agreement clarifying this poin

avoid misunderstandings. EPA may have to pay for obtaining access to the data if its transmittal

imposes additional costs on the recipient. Assuming our use of the data is incidental to the 

principal purpos

u

E

sista ce agreement. 

consult the Office of General Counsel (OGC) for help in drafting appropriate language

r assistance agreement. 

2.2.12 Can the Recipient of a Grant or Cooperative Agreement Use Agreement 

Funds to Pay Peer Review

d in Section 2.2.11, provided EPA agrees that a peer review would further the public 

 of the assistance agreement, EPA may include funds for the peer review in the 

ent. A paym

te

ents. (To accurately characterize this cost, however, it is important that the payment be 

 to as a fee, rather than an honorarium.) See OMB Circular A-21, Section J, item 32

ional Services Cost (Educational Institutions), OMB Circular A-122, Attachment B, It

fessional Services Costs (Non-Profit Organizations), and OMB Circular A-87, 

ment B, Item 33, Professional Services Costs (State, Local and Indian Tribal 

ments). See http://www.whitehouse.gov/OMB/circulars/ for further details. 
 Peer Review Handbook Page 40 

operative Agreements Need 

to be Reported in the Science Inventory 

As a m ne 

under g e aware 

of the o

decisio

should

Invento

A site-specific decision itself is not subject to peer review and does not need peer review as 

ing 

e review through the “scoping” and 

teragency review processes that are part of NEPA, this is not usually considered peer review. 

formation or a highly influential scientific assessment, then the relevant 

ortions should be peer reviewed. 

reparing the NEPA document may suggest the extent of review 

e document should get (See sections 2.2.3 and 2.2.4). If EPA is developing the document as 

part of 

influential scie

eceive independent peer review. If not, (the document is not influential, will have little impact, 

 policy 

chnical 

2.2.13 Do Products Generated Under EPA Grants or Co

atter of practice, EPA organizations are encouraged to include scientific work being do

rants and cooperative agreements in the Science Inventory so that Agency staff ar

ngoing work. If a grant or cooperative agreement product is likely to be used in Agency 

n making (assuming this use is incidental to the principal purpose of the agreement), it 

 generally be considered a candidate for peer review and noted as such in the Science 

ry. 

2.2.14 Should Site-Specific Decisions be Subject to Peer Review?

described in the Peer Review Policy. However, if a site-specific decision is supported by 

influential scientific information or a highly influential scientific assessment, that work product 

should be peer reviewed. Generally speaking, you should closely examine how the underly

scientific and/or technical work product is adapted to the site-specific circumstances. 

2.2.15 Should NEPA Products be Subject to Peer Review?

Although an Environmental Impact Statement (EIS) prepared under the requirements of the 

National Environmental Policy Act (NEPA) receives extensiv

in

If the underlying scientific and/or technical data, models, analyses, or work products are 

influential scientific in

p

In general, the Agency’s role in p

th

an EPA action/decision (EPA is the Lead Agency under NEPA), and the document is 

ntific information or a highly influential scientific assessment, then it should 

r

is non-controversial, etc.), then peer input might be appropriate. 

On the other hand, if EPA is reviewing an EIS from another agency (EPA is not the Lead 

Agency under NEPA), it is likely that we are reviewing the EIS for conflicts with EPA

and general environmental concerns. In such a case, EPA should ask if the underlying scientific

and/or technical work product that supports the EIS has been peer reviewed to avoid concerns

about the full credibility and soundness of the EIS based on the science and technical support. 

EPA should work with the other organization/agency to ensure that scientific and/or te

work products receive peer review adequate for EPA purposes. 
 Peer Review Handbook Page 41 

2.2.16 Should Environmental Regulatory Models be Peer Reviewed?

Generally, yes nvironmental regulatory models have 

een published by the Agency. These can be found on the EPA web site under the Science 

. Specific guidelines for the peer review of e

b

Policy Council home page (http://www.epa.gov/osa/spc/). In 2000, the Science Policy Council 

established the Agency’s Council for Regulatory Environmental Modeling (CREM) which 

promotes consistency and consensus among environmental modelers and users. The CREM is a 

good resource for you to contact for specific questions regarding peer review of environmental

regulatory models (http://cfpub.epa.gov/crem/). 

2.2.17 Should Another Organization’s Work Products That Have Been Submitted 

comes 

r 

to EPA for Use in Decision Making be Peer Reviewed?

Any scientific and/or technical work product that is used in Agency decision making and is 

considered influential scientific information or a highly influential scientific assessment be

a candidate for peer review regardless of whether the work product is produced by the Agency o

another organization. Therefore, all work products important to EPA environmental decision 

making that are independently generated by other organizations (e.g., other Federal agencies, 

teragency groups, State and Tribal bodies, environmental groups, industry, educational 

ins tut or peer review. 

 and 

gional 

 is hoped that if the other organization has the work product independently peer reviewed, the 

ure 

al 

ld be needed. See Section 3.4.9 

r considerations of when an outside party conducts and/or funds peer review of their own work 

arranging the peer review, the 

roduct should be entered in the Science Inventory. 

in

ti ions, international bodies) need to be considered as candidates f

If possible, when EPA knows that a work product is being generated by another organization

may be of interest to EPA for future use, the appropriate EPA office(s) should work with that 

organization, and others, as appropriate (e.g., the states), to promote the use of peer review. For 

example, the Office of International Affairs (OIA), as well as the impacted Program or Re

Office(s), should be included when international products are being considered for EPA use. 

It

peer review will meet the intent of the Agency’s Peer Review Policy and EPA’s proposed use of 

the product (i.e., the peer review is basically equivalent to what EPA would do). Agency staff 

from the appropriate office(s) should examine closely the particulars of the peer review to ens

independence and a conscious effort to incorporate the peer reviewers’ comments into the fin

work product. If there are perceived, or real, conflicts of interest, this may preclude the use of 

that peer review and, in those instances, another peer review wou

fo

product and submits it to the Agency. 

If the outside organization does not have the work product peer reviewed and EPA decides it 

should be peer reviewed, the appropriate EPA office(s) should ensure peer review of that work 

product occurs prior to the Agency’s use of the work product in decision making. Peer review 

can be accomplished by asking the outside organization to do so, or, if it declines, EPA may 

conduct or arrange for the peer review. If EPA is conducting or 

p
 Peer Review Handbook Page 42 

 Scientific 

Information or Highly Influential Scientific Assessments Still be Peer 

eer 

2.3

2.3.1 What Information Is Exempt from the OMB Bulletin Provisions?

ecrecy 

f) Accounting, budget, actuarial, and financial information, and; 

2.2.18 Can Work Products That are Not Determined to be Influential

Reviewed?

Yes. Scientific and/or technical work products that are not categorized as influential scientific 

information or a highly influential scientific assessment may nonetheless be candidates for p

review. For example, a project manager may decide to use peer review because of particular 

program needs and goals. Peer review may also be warranted because it adds substantial value 

to the work product. 

 Determining Which Work Products Do Not Receive Peer Review

The following information does not need to be peer reviewed, even if it might be considered a 

highly influential scientific assessment or influential scientific information: 

a) Information related to national security, foreign affairs, or negotiations involving 

international trade or treaties where peer review would interfere with the need for s

or promptness; 

b) Information disseminated in the course of an individual adjudication or permit 

proceeding (including a registration, approval, licensing, site-specific determination), 

unless peer review is practical and appropriate and the information is scientifically or 

technically novel or likely to have precedent-setting influence on future adjudications 

and/or permit proceeding; 

c) Information involving a health or safety dissemination that is time-sensitive; 

d) A regulatory impact analysis or regulatory flexibility analysis subject to interagency 

review under Executive Order 12866, except for underlying data and analytical models 

used; 

e) Routine statistical information (e.g., periodic demographic and economic statistics) and 

analyses of these data to compute standard indicators and trends; 

g) Information disseminated in connection with routine rules that materially alter 

entitlements, grants, user fees, or loan programs, or the rights and obligations of 

recipients thereof. 
 Peer Review Handbook Page 43 

There may be c s in which it is not necessary for a work product to undergo peer 

view. For example, peer review is not generally conducted: 

e 

ot generally undergo additional peer review, even if the product supported a 

significant Agency decision; 

b) If an application of an adequately peer-reviewed work product does not depart 

c

porting Control 

Techniques Guidelines (CTGs) and Effluent Limitation Guidelines (ELGs). Such use 

. 

rmation That Is Not Exempt, Are There Circumstances Where the 

Peer Review Provisions of the OMB Bulletin Can Be Waived or Deferred?

The Ad

influen

compel

unusua not otherwise covered by the exemptions, such as situations 

here unavoidable legal deadlines prevent full implementation of the Bulletin’s peer review 

pro dinarily 

arrant waiver of the provisions because those deadlines should be negotiated to permit time for 

con

approp

are deferred, peer review should be conducted as soon as practicable thereafter. Deferrals of 

eer review of influential scientific information and highly influential scientific assessments 

sho 

 peer review of a work product is not planned, an explanation should be included in the 

data s

ate Peer 

Scientific papers (articles) that are peer reviewed by a credible refereed scientific journal may 

not generally need to undergo further peer review. However, you should expect that there may 

2.3.2 Are There Other Circumstances Where Peer Review Is Not Necessary?

ircumstance

re

a) For work that has been previously reviewed in a manner consistent with OMB’s Bulletin 

and this Handbook. For example, a cancer risk assessment methodology or an exposur

modeling technique that was the subject of earlier peer review of appropriate technical 

merit would n

signifi antly from its scientific or technical approach; 

c) When the scientific and/or technical methodologies or information being used are 

commonly accepted in the field of expertise, e.g., many products sup

should provide the appropriate documentation to support the commonly held view

2.3.3 For Info

ministrator may waive or defer the peer review provisions of the OMB Bulletin for 

tial scientific information or highly influential scientific assessments if there is a 

ling rationale for the waiver or deferral. The use of waivers is expected to be limited to

l and compelling situations 

w

visions. According to the Bulletin, deadlines found in consent decrees will not or

w

ducting a peer review. Deferral of some or all of the peer review provisions may be an 

riate way to accommodate immovable deadlines. If any of the OMB Bulletin provisions 

p

uld be approved by the Administrator. 

If

ba e entry for that work product in the Science Inventory. 

2.3.4 Does Publication in a Refereed Scientific Journal Mean That Adequ

Review Has Been Performed?
 Peer Review Handbook Page 44 

be cases where EPA determines that a more rigorous or transparent review process is necessary,

.g., if a particular journal review process did not address questions that EPA determines should 

pplicability of the data presented 

r use in the Agency’s models). Also see Section 2.4.4 for additional information. 

Genera

directs EPA to use available voluntary consensus standards in its regulatory activities unless to 

o so would be inconsistent with applicable law or otherwise impractical. For purposes of the 

NT als

specifi s practices) that are developed or

dopted by voluntary consensus bodies (such as ISO or ASTM). The general purpose of the 

NT

“reinve ds

would

organiz

urposes of the Agency’s peer review policy. EPA reserves the right to conduct a peer review if 

it d ter r

purposes of th

es

test

tion and highly influential scientific assessments 

hen it involves well-qualified external reviewers, is intensive in its examination, and operates 

e and 

at can be reasonably achieved. 

rranging for the most appropriate and feasible peer review will involve good judgment 

re ardi e

as consideratio es. Developing a peer review plan that provides 

r appropriate depth, timing, and content is an important matter for early consideration by the 

(e

be addressed before using or disseminating the information, such as the extent of uncertainty

inherent in the findings, the sensitivity to model choice, or the a

fo

2.3.5 Do Voluntary Consensus Standards Undergo Peer Review?

lly, no. The National Technology Transfer and Advancement Act of 1995 (NTTAA) 

d

TAA, voluntary consensus standards are defined as technical standards (e.g., materi

cations, test methods, sampling procedures, and busines

a

TAA is to reduce private and governmental costs by avoiding having the government 

nt the wheel” in the development of technical standards. Voluntary consensus standar

normally not undergo peer review because the underlying process used by issuing 

ations to develop and approve these standards is generally considered adequate for 

p

e mines that the particular standard it wants to use is not a voluntary consensus standard fo

e NTTAA.

2.4 Choosing a Peer Review Mechanism

2.4.1 How Do You Determine the Appropriate 

Peer Review Mechanism?

During the planning of a peer review, the Decision Maker 

and the Peer Review Leader may consider several mechanisms for the peer review of scientific 

and/or technical work products. These options range from consultations with EPA colleagu

not involved in developing the product to a large and formal panel of outside subject matter 

experts. The peer review effort might be a focused one-time evaluation, or could encompass 

several examinations over the course of a project. In principle, peer review provides the grea

credibility for influential scientific informa

The mechanism of the peer

review should match the

importance and complexity of

the work product.

w

through a more or less formal and transparent process. As a practical matter, however, tim

resource considerations in many cases impose limitations on wh

A

g ng the xtent to which the peer review will improve the credibility of the product as well 

n of substance, time, and resourc

fo

Decision Maker and Peer Review Leader. Note that use of peer input or public or stakeholder 

involvement does not constitute peer review. 
 Peer Review Handbook Page 45 

While the following information is described in more detail elsewhere, these are important 

considerations in determing a peer review mechanism. The approach best suited to a specific 

work product will depend on the nature of the topic and the intended final product. Generally

the more novel or complex the science or technology, the greater the cost implications of the 

impending decision, and the more controversial the issue, then the stronger the indica

, 

tion is for a 

ore extensive and involved peer review and for external peer review in particular. Certain 

work p s enerally these 

ill be products with large impacts (e.g., those that support Tier 1 and Tier 2 rulemakings). 

d 

s 

match 

 work 

rk best in all situations. Some useful guidance includes: 

d to support the most important decisions, or 

work products that have special importance in their own right, external peer review is the 

ovel and/or controversial the 

produc the more the Decision Maker should consider 

implementing a peer review involving external experts; 

ntific perspective within a discipline). 

m

roduct may clearly lend themselves to extensive external peer review; g

w

Other work products may not need a large scale external peer review and may utilize a less 

involved, less resource intensive review. The peer review of some products may be better serve

with some form of internal peer review or a combination of internal and external peer review. 

It is important to make the choice of peer review mechanism at the time that the work is planned

(for products supporting rule makings, at the analytic blueprint stage) so that peer review cost

and time can be budgeted into the work plan. Essentially, the level of peer review should 

the impact and complexity of the work product. For example, a rule under development carries 

considerable weight and deserves careful handling and attention; therefore, the supporting

product deserves similar care and attention for its peer review. Factors that need to be 

considered include: use of internal vs. external peer reviewers; individual letter reviews vs. panel 

reviews; the number of reviewers; the timing of the peer review. No single peer review 

mechanism is likely to wo

a) Influential scientific information intende

approach of choice. Generally, the more complex, n

t, or the higher impact it has, 

b) Highly influential scientific assessments are expected to undergo external peer review. 

When time and resources allow, panels are preferable; 

c) Work products that are less complex, novel, or controversial, or have a lower impact 

might be subject to a less extensive, less resource-intensive review processes; 

d) Group discussion with peer reviewers can be very helpful at some point in the peer 

review process as it allows interaction among peer reviewers with different perspectives 

and expertise. Public panels are more transparent than closed discussions. On the other 

hand, simply soliciting individual comments is easier, faster, and less expensive. 

Individual review is probably more appropriate for peer review at the early stages of a 

product’s development or for products with less impact and complexity; 

e) More reviewers are necessary for more complex projects (more disciplines) and for more

controversial topics (differences in scie
 Peer Review Handbook Page 46 

f) Strict time constraints, such as a court-ordered deadline, can make a less involved or 

formal peer review mechanism imperative. Decision Makers and Peer Review Leaders 

should make maximum efforts to ensure that such a process is systematic and objectiv

2.4.2 What are Some Examples of Internal Peer Review Mechanism?

a) Individual independent experts from within the Agency (e.g., ORD experts on non-ca

effects of lead (Pb) review a draft article on benchmark dose completed by a Program

Office); 

e. 

ncer 

b) An ad hoc panel of independent experts from within the Agency (e.g., an independent 

e scientists); 

to 

c)

sory Panel (SAP), ORD’s Board 

of Scientific Counselors (BOSC), or the Clean Air Scientific Advisory Committee 

e) Agency-appointed special board or commission (e.g., a review of the risk assessment 

te: 

f)

are a review 

internal workgroup convened to examine the case for the classification of a chemical as a 

carcinogen); 

c) Technical merit review by scientists in an Agency laboratory (e.g., an initial review of the

risk assessment for a regional incinerator by Agency scientists). 

2.4.3 What are Some Examples of External Peer Review Mechanisms?

a) Independent experts from outside the Agency (e.g., a letter review by outsid

b) An ad hoc panel of independent experts outside the Agency (e.g., a group is convened 

develop a consensus on the carcinogenicity of a particular industrial chemical); 

Agency-sponsored peer review workshops (e.g., a review of potential indicators of 

ecosystem damage); 

d) Review by an established Federal Advisory Committee Act mechanism such as the 

Science Advisory Board (SAB), FIFRA Scientific Advi

(CASAC) - e.g., a review of a criteria document for a particular chemical risk; 

methodology prepared by the Clean Air Act Commission on Risk Assessment). No

The Office of General Counsel should be consulted regarding EPA's authority to 

establish and finance the activities of a commission or board. 

Review by the NAS (e.g., a review of the state of current knowledge about children’s 

health risks from pesticide exposures). 

There are other bodies that may provide external commentary on Agency work products but 

 not considered peer review mechanisms, such as interagency committees (e.g.,
 Peer Review Handbook Page 47 

of prospective research plans by the Committee on the Environment and Natural Resources 

gov

Hum

reviews initiated by non-governmental groups (e.g., a Society of Risk Analysis review of 

can d

Peer review of journal articles (written by EPA or non-EPA authors) performed by a credible, 

efereed scientific journal contributes to the scientific and technical credibility of the reviewed 

pro

scientif

satisfac eer review . However, in some cases, peer review of an Agency work 

roduct that uses these articles may be conducted (see Section 2.4.5). 

EPA-au or not, are included in 

e Science Inventory to highlight the extensive work EPA produces in the scientific literature. 

rior to submitting an article to a journal for peer review, EPA employees are encouraged to 

hav

ommon practice in certain parts of EPA. Articles may also need examination in accordance 

wit

herself ll also 

pply. 

ork Products Become Candidates for Peer Review when PeerReviewed Journal Articles are Used in Support of that Work Product?

In most

coordinated by the White House), committees convened by another Federal agency or 

ernment organization (e.g., a review of the Dioxin Reassessment by the Health and 

an Services Committee to Coordinate Environmentally Related Programs), and/or 

cer gui elines). 

2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?

r

duct. Generally, EPA considers peer review by such journals as adequate for reviewing the

ic credibility and validity of the findings (or data) in that article, and therefore, a 

tory form of p 13

p

thored journal articles, whether used in an Agency work product

th

P

e the article internally peer reviewed (see Section 2.4.2); such internal peer review is already 

c

h any organizational clearance procedures, especially when the author is presenting him or 

as an EPA employee. For EPA employees, conflict of interest law and policy wi

a

2.4.5 Do Agency W

 instances, Agency work products are candidates for peer review even when supported by

viewed journal article(s). Although the use of articles that have been peer reviewed 

e journal strengthens the scientific and technical credibility of any work product in

peer-re by a 

credibl which 

the article(s) appears or is referenced, it does not automatically eliminate the need to consider 

whe ay 

not cov

action. the 

article( . A journal article 

uthored by EPA employees should be used in the same manner as an article published by 

any

 

ther the work product itself should be peer reviewed. In most cases, journal peer review m

er issues and concerns that the Agency may want peer reviewed to support an Agency 

 Under these circumstances, the scientific and/or technical work product in which 

s) appears or is referenced becomes a candidate for peer review

a

one else in a credible, well-recognized journal. 

 13 G

Diss

(http

uidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information 

eminated by the Environmental Protection Agency, EPA/260R-02-008 (October 2002), 

://www.epa.gov/quality/informationguidelines) 
 Peer Review Handbook Page 48 

If an Agency work product is based solely on a single article that has received peer review by a 

ible journal (e.g., where a model is suggested for a singular use that fits a specific Agen

d), peer review of the Agency work product may or may not be necessary depending on

ely you apply the findings from the article. If an Agency work product is based on two 

re articles that have received peer review by a credible journal(s), the Agency work produc

cred cy 

nee how 

clos or 

mo t 

generally should be considered for peer review. Decisions to make (or not make) a work product 

a candidate for peer review should be documented in the peer review record. 

ne important factor to remember with regard to the use of articles that have received journal 

so such 

2.4.6 When and How Often Should Peer Review Occur?

ming and the frequency of peer review. Options abound, each with merits depending on the 

 development, particularly where it involves complex tasks, has decision branching 

oints, or could be expected to produce controversial findings. In addition, early review could be 

be fic h

extensive prim

etermine when the peer review(s) should occur, considering the type of work product under 

O

peer review concerns the availability of documentation from that peer review. Ideally, EPA 

should maintain a clear, easily accessible record of the peer review to ensure the credibility and 

validity of the peer review (see Section 2.5 for details on the peer review record). However, the

documentation from a journal peer review may not normally be available to the Agency, 

documentation is not expected in the peer review record. 

The Decision Maker and Peer Review Leader have significant discretion in deciding on the 

ti

context and specified peer review objectives. In many situations, a single peer review event, 

beginning when the final draft work product becomes available, is the approach taken. However,

it is increasingly apparent that peer review performed earlier in the work product development

stages provides a superior approach for some work products. There may be substantial 

incremental benefit to conducting more than one peer review during the whole process of work 

product

p

ne ial at t e stage of research design or data collection planning where the product involves 

ary data collection. The Decision Maker and Peer Review Leader should 

d

development and at what point in its development process a peer review would be most 

beneficial. 

Other types of work products that could benefit from early, up-front peer review in their 

development are scientific and technical planning products. Examples of such products are 

research proposals, plans, and strategies. Also, while not products per se, ongoing research 

programs can be peer reviewed. 

Remember though, that while more than one peer review can be beneficial, the distinction 

between peer input and peer review should be kept in mind. Experts providing input during the 

development or planning stages of the work product generally do not become peer reviewers of 

that product (see Sections 1.2.2 to 1.2.7 for full discussion on this distinction). 
 Peer Review Handbook Page 49 

2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review?

Several factors impact how quickly a peer review may be desired. These include deadlines for 

completion of a project, research program, or rulemaking, funding availability, availability of 

quality peer reviewers, the possible need to seek public comment on the peer review product, an

statutory and/or court-ordered deadlines. 

d 

eer review sometimes leads to new information and analyses. Reviewers may make 

ducts could be of sufficient magnitude 

 warrant a revision to the proposed action or rule, Decision Makers should exercise diligence in 

comple e ble. 

ons 

r 

t 

, 

s. 

review. 

ity for conducting a peer review can be negotiable when more than one Agency 

ffice or Region or other agencies are involved. Often, the degree of involvement by any of the 

e lead 

w Record and Peer Review Reports

 review 

omments were addressed. It includes sufficient documentation for an uninvolved person to 

nderstand what actually happened and why. The Peer Review Leader (with the program 

P

recommendations for new research that would alter the work product and thus modify the 

scientific/technical basis for the action or rule it supports. For this reason, a completed peer 

review is desirable before issuing any proposal for public comment. If that is not logistically 

possible because of court or statutory deadlines, or other appropriate reasons, the Decision 

Maker should make every effort to complete the peer review before the close of the comment 

period. Because peer review comments on such work pro

to

ting th peer review prior to the proposal stage whenever possi

Whenever feasible and appropriate, Offices should make a draft scientific assessment that is 

highly influential available to the public for comment at the same time it is submitted for peer 

review (or during the peer review process) and sponsor a public meeting where oral presentati

on scientific issues can be made to the peer reviewers by interested members of the public. 

When employing a public comment process as part of the peer review, Offices should, wheneve

practical, provide peer reviewers with access to public comments that address significan

scientific or technical issues. To ensure that public participation does not unduly delay activities

Offices should specify time limits for public participation throughout the peer review proces

2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the 

Peer Review?

The organization of the Decision Maker is normally responsible for conducting the peer 

Responsibil

O

organizations and agencies and their ability to fund peer review will determine who has th

for the peer review. 

2.5 Creating the Peer Revie

2.5.1 What is the Peer Review Record?

The peer review record is the formal record (file) of decision on the conduct of the peer review, 

including the type of peer review performed and an explanation of how the peer

c

u
 Peer Review Handbook Page 50 

ma e 

r development of the work. Once the peer review is completed, it is the responsibility of the 

entry in the Science Inventory Database. While 

me information from the peer review record appears in the database, the paper peer review 

, and 

ion 

b) Materials and information (including the charge

nager if there is one) creates a separate, clearly marked peer review file within the overall fil

fo

Peer Review Leader to ensure that the peer review record is filed and maintained in accordance 

with the organization’s document retention procedures. 

The Peer Review Record is separate from the 

so

record is the official record of the peer review. 

2.5.2 How Can the Peer Review Record Improve the Peer Review Process?

A good peer review record allows future reference to what happened during the peer review

helps Decision Makers make appropriate use of peer reviewer input. In addition, a good record

helps ensure that EPA’s Peer Review Policy is implemented. The Peer Review Leader is 

responsible for ensuring that the peer review record for individual work products is collected and 

maintained until completion of the peer review effort. 

2.5.3 What Should Be in the Peer Review Record?

The peer review record should include all materials considered by the individual peer reviewers, 

the peer review report, and other input. Such materials include, at a minimum (see also Sect

4.3.1): 

a) The draft work product submitted for peer review; 

) given to the peer reviewers; 

c)

inform eer reviewers (such as reviewers’ names, affiliations, and a 

statement concerning potential conflicts and their resolution, if applicable); 

of 

m, or other written record, approved by the Decision Maker, responding to 

the peer review comments specifying acceptance or, where thought appropriate, rebuttal 

se to the peer review 

report addressing each comment. 

f) The final work product. 

Leader 

ults 

d. 

The peer review report, which summarizes the peer review findings and contains 

ation about the p

d) Logistical information about conduct of the peer review (such as times and locations 

meetings); 

e) A memorandu

and non-acceptance. The Office should prepare a written respon

When deciding if particular materials should be included in the record, the Peer Review 

should consider whether the materials would help reconstruct the peer review process and res

at a later time. If the materials may be helpful, they should be part of the peer review recor
 Peer Review Handbook Page 51 

2.5.4 How and When Should You Develop and Post Peer Review Reports and 

Agency Responses for Influential Scientific Information and Highly 

Influential Scientific Assessments? 

Offices are expected to make peer review reports of the influential scientific information and 

dings and conclusions. The peer review report 

hould either (a) include a verbatim copy of each reviewer’s comments (either with or without 

sp cifi u arate 

nd dissenting views, although attribution of comments to names is not necessary. The names of 

rts on the Science Inventory website 

http://cfpub.epa.gov/si/

highly influential scientific assessments publicly available to implement the provisions of the 

OMB Bulletin. Offices should instruct peer reviewers to prepare a report that describes the 

nature of their review and the nature of their fin

s

e c attrib tions) or (b) represent the views of the group as a whole, including any disp

a

the reviewers and their organizational affiliations should be included in the report. For highly 

influential scientific assessments, the report should also include the charge to the reviewers and a

short paragraph on both the credentials and relevant experiences of each peer reviewer. 

EPA will post or provide a link to the peer review repo

( ) along with all materials related to the peer review (charge statement 

an Ag t is likely to be enhanced if the 

ublic understands how the Agency addressed the specific concerns raised by the peer reviewers. 

n response to the peer review report explaining (a) the agency’s agreement or 

isagreement with the views expressed in the report, (b) the actions that have or will be 

und Office believes those actions satisfy 

ny key concerns or recommendations in the report. These responses will also be posted on the 

Sci

The

proper docketing procedures are followed for a peer review of a work product supporting a rule. 

 EPA relies on influential scientific information or a highly influential scientific assessment to 

sup t

the pro

view Record Building Process Begin?

An early start at developing and maintaining a peer review record will help ensure the record is 

omplete and helpful. Ideally, the record begins when the decision to peer review a work 

d ency response). The credibility of the final work produc

p

Offices should consider preparing a written response for inclusion in the peer review report. For 

highly influential scientific assessments, the OMB Bulletin explicitly calls for Offices to prepare

a writte

d

ertaken to respond to the report, and (c) the reasons the

a

ence Inventory website. 

2.5.5 What Should You Do with a Peer Review Record That Pertains to a 

Rulemaking Action?

 Peer Review Leader should coordinate with his/her program’s docket office to see that 

If

por a regulatory action, the preamble should include a discussion of how EPA implemented 

visions of the OMB Bulletin. See Appendix C for a template to use for this purpose. 

2.5.6 When Should the Peer Re

c

product is made. The Peer Review Leader should construct the peer review record from this 

point on – doing so will avoid potentially time-consuming reconstruction at a later point. Note 

that the peer review record is not complete until it contains a copy of the final work product 
 Peer Review Handbook Page 52 

wh ch a d 

also be created

2.5.7 What are the Differences in Record Keeping for a Review by Individuals 

 by 

 review 

cord themselves until the peer review is complete. Minimally, the file should be maintained 

ord may 

tely 

 Peer Review Handbook. The documents contained in the peer 

eview record should be maintained in accordance with the Agency’s record-keeping retention 

sch du u m at 

http://www.ep

i ddresses the peer review comments. If a product is not peer reviewed, a record shoul

 which explains why the product was not peer reviewed. 

Compared to Review by a Panel?

Generally, the content of the peer review record would be the similar. In the case of a review

individuals, the peer review record would typically contain each individual's comments, while 

for a panel review, the record typically contains a summary or other synthesis documenting the 

panel's deliberations. 

2.5.8 Where Should the Peer Review Record be Kept and For How Long?

During the active conduct of the peer review, the Peer Review Leaders maintain the peer

re

until one year after the completed peer review is reported in the next annual reporting cycle. 

After that, the peer review record should be maintained for a reasonable period of time, as 

delineated by the organization's document retention policy. Establishment and maintenance of 

the archive where the peer review records ultimately reside are an organization’s responsibility 

(i.e., not that of an individual program manager or Peer Review Leader). Generally, to allow 

flexibility, individual Offices and Regions should decide the appropriate level of organizational 

responsibility and how to ensure the record is routinely available. The peer review rec

be kept with other records relating to the overall project, as long as it is easily and separa

identifiable. 

There are also specific procedures regarding the use of dockets for record-keeping; however, 

these are not covered in this

r

e le for s ch records. For details, see EPA’s National Records Management Progra

a.gov/records. One long-term archiving mechanism may be the formal archiving 

t the Federal Records Center. 

is 

andbook for planning, conducting, and completing a peer review. When you follow this formal 

p cess a cord of that 

ternal peer review should be included in the peer review record. This includes all the materials 

t 

l”), 

l 

a

2.5.9 Are Internal Peer Review Comments Included in the Peer Review Record? 

To be considered a legitimate peer review, internal EPA peer reviews should be formally 

conducted and documented. Such a process would be consistent with the guidance found in th

H

ro to obt in peer review from EPA peers (see Section 1.5.9), then the whole re

in

detailed in Section 2.5.3 (also see Section 4.3.1). Conducting a formal internal peer review is no

the same thing as informal input from your EPA colleagues (i.e., “colleagues down the hal

nor peer input from Agency personnel helping to develop the work product, nor organizationa

review and clearance processes. Such inputs from these informal processes should not be placed
 Peer Review Handbook Page 53 

in the peer review record. The peer review record should contain only the information obtained

when you conduct a formal internal peer review. 

In some w.

In such a case, ill stand as the peer review of record since it 

viewed as more independent in nature, has broader fields of available expertise which can be 

2.6.1 What Budgetary Factors Should You 

ied

l

a normal part of doing business. Peer review resource 

onsiderations should also be addressed in the analytic blueprint for Agency rule-making actions. 

.7 Legal Considerations

a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA 

 cases, an internal EPA peer review may be followed by a separate external peer revie

however, the external peer review w

is

brought to bear on the issues and often includes greater depth for specific disciplines. 

2.6 Budget Planning Peer review is part of the 

normal cost of doing business.

Consider in a Peer Review?

Resources necessary for the implementation of the Peer Review Policy should be requested 

through the usual Agency budgetary processes. The budget formulation process within the 

Executive Branch is followed, after appropriation bills are passed by Congress, by budget 

execution. These two processes provide opportunities to secure resources for activities carr

out by Headquarters and Regional offices, including peer review. The work products which wil

undergo peer review should have adequate funding requests for the coming fiscal year. 

Similarly, adequate funding should appear in the actual approved operating plan to ensure the

peer review can be conducted. For purposes of budget planning, the costs of peer review would

include the FTE cost of staff, the contract or other costs associated with the use of outside peer 

reviewers, and the administrative costs of conducting a review (copying, travel expenses, etc).

2.6.2 What Input is Needed for the Annual Budget Formulation and Budget 

Execution Process?

Senior Management in Program Office and Regions (including Decision Makers and budget 

officers) should ensure that budget requests include anticipated resources for peer review. Peer 

review should be considered as

c

2

2.7.1 What Are the Legal Ramifications of the Peer Review Policy?

The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it 

confirms the importance of peer review where appropriate, outlines relevant principles, and 

identifies factors Agency staff should consider in implementing the policy. Except where 

provided otherwise by law, peer review is not a formal part of or substitute for notice and 

comment rulemaking or adjudicative procedures. EPA’s decision to conduct peer review in any

particular case is wholly within the Agency’s discretion. Similarly, nothing in the Policy creates 
 Peer Review Handbook Page 54 

decisions rely on scientific and/or technical work products that have been subjected to peer 

review, the remarks of peer reviewers should be included in the record for that decision. 

 consult 

ith their usual OGC/RC advisors for legal advice or referral. Headquarters attorneys have 

 and can be consulted as needed (e

ontractual responsibilities; ethics and potential conflicts of i

2.8 Federal Advisory Committee Act (FACA) Considerations

with 

, 

s subject 

ontractor-run peer reviews. 

 most cases, Federal Advisory Committee Act (FACA) requirements apply to EPA-run peer 

s, 

a) The group is established, controlled, or managed by EPA; 

b) The group has a fixed membership, established purpose, and a set agenda; and 

c) The group strives to produce collegial, rather than individual, advice to EPA. 

bit 

ning 

rts in 

2.7.2 Is Legal Advice Needed?

AA/RA staff and management should work regularly and closely with individual OGC/Regional

Counsel (RC) staff assigned to Agency activities. Peer Review Leaders should initially

w

specialties in specific areas .g., FACA considerations, 

c nterest). 

The Federal Advisory Committee Act, 5 USC App. 2, imposes certain open meeting (public 

announcement in the Federal Register), balanced membership, and chartering requirements (

the approval of the General Services Administration (GSA)) before the Agency establishes

controls or manages an “advisory committee” for advice or recommendations. Peer review 

carried out by formal and established (chartered) Federal advisory committees, such as the 

Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel (SAP), is alway

to FACA requirements. However, FACA considerations do not apply to every EPA and 

contractor-run peer review. 

In the next Section you will find information on the applicability of FACA to EPA- or 

c

2.8.1 When Do FACA Requirements Apply to EPA-Run Peer Reviews?

In

reviews that are conducted by formal and established (chartered) Federal advisory committee

such as the Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel (SAP). 

These groups have the following characteristics: 

EPA-run peer reviews that are not originally intended to be subject to FACA, but which exhi

the above characteristics, may unintentionally become subject to FACA. Questions concer

the applicability of FACA to peer review meetings should be addressed to the FACA expe

the Cross-Cutting Issues Law Office of OGC, or the appropriate Office of Regional Counsel. 
 Peer Review Handbook Page 55 

2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? 

If EPA conducts a peer review with the purpose of obtaining advice from the individual peer 

reviewers and not for the purpose of obtaining a peer review product from the group (as a 

ollective or consensus body), the peer review, in most cases, would not be subject to FACA. 

ts are usually not subject to 

e requirements of FACA. 

 addition to ensuring that peer reviewers only provide comments as individuals, EPA officials 

n 

 the 

s well 

lso be advertised via other avenues (e.g., the Web, local 

ewspapers, and mailing lists). 

Become 

Subject to FACA? 

and 

actor report) are not 

ubject to FACA. Although FACA should not apply to contractor-run peer reviews, there are 

view (e.g., letter review, panel, workshop, etc.) for EPA: 

a) The outside party’s peer review may be subject to FACA if EPA establishes, manages or 

. 

gs; 

c

Peer review participants provide only their own views or recommendations and do not vote nor 

do they provide collective or consensus recommendations to EPA. When referring to the 

recommendations of the individual reviewers, EPA should not characterize these 

recommendations using terms such as “collective” or “consensus.” As a general matter, 

workshops and “letter reviews” that seek individual views or commen

th

In

may wish to lessen the potential for a challenge under FACA by seeking balanced participatio

at peer review meetings, and allowing interested members of the public to attend, and ensuring 

that they have access to appropriate materials. 

Non-FACA meetings may be announced in the Federal Register (providing that it is clear in

notice that such meetings are not subject to FACA) as it provides the public with useful 

information and a point of contact concerning the peer review. In addition, non-FACA (a

as FACA) meetings should a

n

2.8.3 How Do You Ensure that a Contractor-Run Peer Review Does Not 

Committees (or other peer review groups) established, controlled or managed by an outside 

organization (such as by an EPA contractor) to provide that outside organization with advice 

recommendations (that will be submitted eventually to EPA as a contr

s

things that you (i.e., EPA) can inadvertently do that may invoke FACA. 

The following are considerations that you should be aware of when a contractor runs a peer 

re

controls the peer review group (e.g., EPA selects panel members, runs the meeting, etc.); 

b) EPA should not provide contractors with a draft agenda or suggested format for meetings

EPA contractors should manage and control the process, including running any meetin
 Peer Review Handbook Page 56 

c) e peer reviewers 

(e.g., in a conference call with the contractor on the line) on the history or background of 

t only 

l the 

er the control of the meeting. The contractor may call on them to 

speak when appropriate, but EPA staff should limit their participation to providing 

tractor establishes, controls, or manages a peer 

review, the contractor need not avoid terms such as “collective” or “consensus” when 

leted the 

 the work assignment. EPA should not attempt to make changes 

in the contractor’s conclusions; this would compromise the independence of the peer 

review conducted by the contractor.

At the request of the EPA contractor, EPA can provide a briefing to th

the development of the document. EPA should only provide technical or background 

information and not use the call to take over the contractor’s peer review group. No

should the contractor be on the line, but it should be very clear to all participants that the

contractor is in charge of the call. The contractor, not EPA, should invite people to 

participate, make all administrative arrangements, conduct the meeting, and contro

agenda; 

d) EPA employees may attend the peer review panel meetings or workshops. However, 

they may not take ov

technical and/or background information, and not attempt to, or appear to, take over the 

contractor’s meeting; 

e) Since FACA does not apply when a con

reporting agreement among its peer reviewers (subcontractors); 

f) EPA may provide comments to the contractor on the contractor’s peer review only to the

extent that the Agency is verifying that the contractor has satisfactorily comp

report in accordance with
 Peer Review Handbook Page 57 

3. Conducting A Peer Review

Overview Statement

ccess and usefulness of any peer review depends on the quality of the draft work product

ted for peer review, the care given to the statement of the issues or "charge," the match 

n the peer

3.1

The su

submit

etwee review draft product and the form of peer review, the match between the peer 

ev

pee

each of

Figure

ompleted before selection of the peer reviewers to ensure that they have the appropriate 

xp

3.2 

As par

b

r iew draft product and the scientific/technical expertise of the reviewers, and Agency use of

r review comments in the final product. It is not simply enough to conduct a peer review; 

the foregoing elements deserves serious attention. 

2 shows the order of activities for conducting a peer review. The charge should be 

c

e ertise to address the questions raised.

Charge to the Peer Reviewers

3.2.1 What is a Charge?

t of each peer review, the Peer Review Leader formulates a clear, focused charge that 

es the technical and scientific issues on which the Agency would like feedback and 

ggestions for improving the document as a whole. This request signals the Agency's

ions. The charge to peer reviewers u

identifi

invites su

receptivity to expert recommendat sually makes two general 

requests. First, it focuses the review by presenting 

specific questions and concerns surrounding such 

issues as the comprehensiveness of the literature 

reviewed, the soundness of the method proposed, the 

scientific support for the assumptions employed, and 

the sensitivity of the results to alternative assumptions. 

Secondly, it invites general comments on the entire 

work product. The reviewers should be asked to ensure 

that the potential implications of the uncertainties for the technical conclusions drawn are clearly 

and transparently presented. The specific and general comments should focus on the scientific 

and technical merits of the work product and, where germane, whether the scientific/technical 

studies have been applied in a sound manner. Remember, the peer review is not for the decision 

or action itself, but for the underlying scientific and/or technical work product; reviewers should 

not be asked to provide advice on policy. Additionally, Focused questions greatly simplify the 

task of collating, analyzing and synthesizing peer review comments on a topical basis. The 

questions should be specific enough to encourage helpful comments, including constructive 

alternatives, but not so specific that they preclude creative responses (e.g., simply asking whether 

or not the reviewer agrees with the model choice or conclusion). Understanding the reviewer’s 

Time is well-spent preparing a good 

charge, as the charge is crucial for 

an effective peer review.
 Peer Review Handbook Page 58 

reasoning is pivotal to the A as well as crafting 

pecific improvements to the nses to these 

uestions help the Agency create a peer review record. As a general rule, time is well-spent 

 effective peer review. 

er 

’ comments; 

er responses (e.g., MS Word, Adobe Acrobat, e-mails, or 

ontains examples of successful charges that 

on Reque

nce A

 C ers? 

 charge to the peer reviewers, but EPA 

gency’s ability to address the reviewer’s concern 

s work product. The peer reviewers' written respo

q

preparing a good charge, as the charge is crucial for an

3.2.2 What are the Essential Elements of a Charge?

a) A brief overview or introduction (describe what the work product is, how it was 

developed, how it will be used) 

b) As needed, a brief description or listing of any background materials provided to the pe

reviewers 

c) The specific issues or questions to be addressed by the peer reviewer(s), including 

logistical details such as: 

1) The due date of reviewers

2) The format of review

text files); 

3) The point of contact, in case peer reviewers have questions. 

3.2.3 Where Can You Get an Example of a Charge?

Appendix D – Examples of Peer Review Charges, c

cover a variety of issues. Appendix E – Guidance 

Advisory Board, provides guidance for obtaining Scie

 3.2.4 Can a Stakeholder Provide Input to the

Yes. EPA may decide to obtain stakeholder input on the

makes the final determination on what elements to include in the charge to ensure that it meets 

EPA’s needs for the peer review. 

a) If you obtain stakeholder input, include interested parties to the extent feasible based 

upon statutory, regulatory, budgetary and/or time constraints. Do not limit input to one 

stakeholder or one side of a controversial issue (e.g., a responsible party or environmental

group). 

b) If EPA has hired a contractor to perform the peer review, it should still be EPA personnel 

who obtain stakeholder input and provide the list of charge questions to the contractor. 

sting a Review by the Science 

dvisory Board (SAB) services. 

harge to the Peer Review
 Peer Review Handbook Page 59 

c) If you form a committee of stakeholders to help develop the charge, be aware that your 

committee may become subject to the requirements of FACA (see Section 2.8). 

 3.2.5 Who Writes the Charge When You Hire a Contractor to Conduct the Peer 

Review?

 general, if EPA hires a contractor to perform the peer review, EPA should allow the contractor 

ind the 

contrac tions that EPA wants included in the charge letter to 

nsure that the peer review meets EPA’s needs. If the EPA project manager has them ready, the 

list 

contractor can then prepare and submit the actual charge letter to the peer reviewers. Prior to 

bmitting the charge letter to the reviewers, the contractor should be directed to give EPA an 

opp bmit 

the charge directly to the peer reviewers when the review is being conducted by a contractor. 

 3.2 elop the Charge to the Peer Reviewers?

No. EPA s

contractor, tho ovide assistance and advice in the development of the charge. 

3.3 Tim

ew?

ime 

cts 

f the peer review (e.g., contracting procedures). 

he schedule for peer review should take into account the overall rulemaking (or other decision 

ses, or recommendations for new research that would alter the 

ork product and thus modify the scientific/technical basis for the action. For this reason, it is 

usu

least be

approp

public for comment during the peer review process, and, if feasible and appropriate, sponsor a 

ublic meeting where oral presentations on scientific issues can be made to the peer reviewers by 

inte

review,

ientific or technical issues. 

In

ependence in conducting it. However, with regard to the charge, EPA should provide

tor with a substantive list of ques

e

of charge questions can be incorporated into the Statement of Work. Based on this list, the 

su

ortunity to review the charge letter to ensure that it meets EPA’s needs. EPA cannot su

.6 Is it Okay to Ask a Contractor to Dev

hould provide the charge questions to the contractor as discussed above. The 

ugh, may pr

e Line

3.3.1 What are the Factors in Scheduling a Peer Revi

The peer review schedule is a critical feature of the process. The schedule should take into 

account the availability of a quality draft work product, availability of appropriate experts, t

available for peer review comments, deadlines for the final work product, and logistical aspe

o

T

making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of the peer 

review should be included in the development of the analytic blueprint. Peer review sometimes

leads to new information and analy

w

ally advisable to complete the peer review before taking public comment on the rule, or at

fore the close of the public comment period. Note, however, that whenever feasible and 

riate, offices should make a draft highly influential scientific assessment available to the 

p

rested members of the public. When employing a public comment process as part of the peer 

 Offices should provide the reviewers access to the public’s comments that address 

sc
 Peer Review Handbook Page 60 

3.4 

3.4.1 What are Considerations for Selecting Peer Reviewers?

The choice of peer reviewers should be based primarily upon on the reviewers’ expertise, 

nowledge, skills, and experience and should include specialists from multiple disciplines, as 

nt 

t 

r an appearance of a lack of impartiality, and who are completely independent (see Section 

3 .5). v

esire to have appropriate technical balance and representation, means that the field of potential 

t, or 

ed a 

ariety of organizations rather than a single one for an extended period may provide expertise 

tive. The emphasis on independence and expertise applies equally to 

overnment experts and experts from the larger scientific community 

o ensure that the selected reviewers have the appropriate expertise, knowledge, skills, and 

he 

nflict of interest and do not appear to lack 

partiality. 

ntific 

gram or Regional Offices. 

Selection of Peer Reviewers

k

appropriate. The group of reviewers should be sufficiently broad and diverse to fairly represent 

the relevant scientific and technical perspectives and fields of knowledge; they should represe

a balanced range of technically legitimate points of view. Each peer reviewer should have 

recognized expertise that bears on the subject matter under discussion. It may sometimes be 

appropriate, as in the case of FACA committees, to consider "affiliation" (e.g., industrial, 

academic, or environmental community). 

The selection of independent peer reviewers is also critical to an effective peer review. EPA 

should always make every effort to use peer reviewers who do not have any conflict of interes

o

.4 Howe er, the very need to have experienced individuals on a peer review, along with the 

d

peer reviewers may include those that are less than independent, have a conflict of interes

might at least appear to lack impartiality. Sometimes selecting individuals who have serv

v

with diverse perspec

g

T

experience, the charge to the reviewers should be determined in advance of the selection of t

reviewers. Selection of peer reviewers should be made by identifying reviewers with the 

appropriate expertise and then narrowing the field of potential peer reviewers to those 

individuals that are independent, do not have a co

im

You should also consider requesting that the public, including scientific and professional 

societies, nominate peer reviewers. 

3.4.2 Where Do You Find Peer Reviewers?

Recommendations for potential peer reviewers can be identified by a number of organizations. 

These include external groups such as the affected party(ies), special interest groups, public 

interest groups, environmental groups, professional societies, trade or business associations, 

State organizations or agencies, Native American Tribes, colleges and universities, the National 

Research Council, and other Federal agencies with an involvement in or familiarity with the 

issue. Agency associated groups include the staff of the SAB or the SAP, and relevant scie

and technical experts from Pro
 Peer Review Handbook Page 61 

In certain circumstances, existing organizations such as the SAB, SAP or BOSC may be used to 

onduct a peer review. These groups establish their own criteria for accepting work. If you use 

su an i e.g., see Appendix E 

Guidance on Requesting a Review by the Science Advisory Board, for its procedures). SAB, 

r 

ses 

es, 

o posted so that the public may not only nominate, 

ut also comment on potential advisory committee members. See “Advisory Committee 

c

ch organ zation, you should coordinate directly with the organization (

- 

SAP and BOSC conduct formal, public, external peer reviews. 

Another method which might be used to find peer reviewers is public solicitation. The Science 

Inventory can assist the public solicitation process by announcing the opportunity for public 

nominations and by providing background on the review topic the public may find useful fo

identifying potential reviewers they may wish to nominate. EPA’s Science Advisory Board u

the Internet to solicit names for both ad hoc and standing advisory committees. These nam

along with short biographical sketches, are als

b

Meetings and Report Development: Process for Public Involvement” available from SAB’s 

website at http://www.epa.gov/sab/pdf/sabso_04_001.pdf

Occasionally, a member of the scientific community will offer his/her services for peer review 

during an ongoing peer review. These offers may be at no cost or based on an expectation that 

reimbursement will be made. Disposition of these unsolicited offers should be handled on a ca

by case basis by the Peer Review Leader, and as necessary, in consultation with the Peer Revie

Coordinator, the Office of General Counsel (OGC), and appropriate Decision Makers. 

If you use a contract mechanism to conduct a peer review, the contract

se 

w 

or may have its own pool 

f scientific and technical experts for peer review. EPA may provide contractors with 

ared 

pearance of a lack of impartiality concerns. Furthermore, when utilizing a 

ontract mechanism to conduct peer review, EPA is not permitted to direct the prime contractor 

wer 

h the peer reviewers are to be coordinated through the prime 

ontractor. See Section 3.6 for further information. 

eep in mind that for contracting purposes, contractors are required to obtain Contracting 

ould 

o

information on potential peer reviewers for conducting a peer review, if such a listing is prep

in alphabetical order. EPA should not require that the contractor select from a prepared list, nor

require that the contractor receive EPA approval before selecting any given peer reviewer 

(sometimes known as a “subcontractor”). However, EPA should review the list of peer

reviewers for conformance to work assignment specifications and adherence to conflict of 

interest and ap

c

to a specific subcontractor (or peer reviewer), nor is EPA permitted to direct the peer revie

(subcontractor). All interactions wit

c

K

Officer (CO) approval of subcontractors and the CO generally seeks the input of the Contracting

Officer Representative (the COR may be the Peer Review Leader in many cases) before 

approving the use of subcontractors. In this case, as noted above, the Peer Review Leader sh

review the list of potential reviewers for conformance to the work assignment specifications and 

adherence to conflict of interest and appearance of a lack of impartiality concerns. 
 Peer Review Handbook Page 62 

3.4.3 Are External or Internal Peer Reviewers Preferred?

External peer reviewers are generally preferred for influential scientific information, and are 

expected for highly influential scientific assessments. For some work products, like those 

reviewed at interim steps, either external or internal peer review may be appropriate. Selection 

f internal peer reviewers should be based upon technical expertise, available time and 

 of 

e work 

3.4.4 What is Important in the Mix of a Peer Review Panel?

er 

viewers. Naturally, experts whose understanding of the specific technical area(s) being 

oncerns 

have a 

portant 

duct. 

less 

t fall along the continuum. A review panel 

ould include experts that are considered “mainstream” (nearer the center of the continuum) as 

 help 

o

affiliation. That is, they should not come from the same Office or group producing the product 

or have any other connection with the product or document being peer reviewed. The choice

reviewers should be analyzed on a case-by-case basis. Reviewers employed by other Federal 

and State agencies may possess unique or indispensable expertise. Don’t forget that for highly 

influential scientific assessments, the use of internal peer reviewers is constrained by the OMB

Bulletin. External peer reviewers should be selected based upon technical expertise as well, 

however, you should not use individuals who have been involved in the development of th

product. (See Section 1.2.6; see also Sections 1.2.7 to 1.2.9). 

A peer review panel or group can number from just a few individuals to ten or more, depending

on the issue, the time and resources available, and the broad spectrum of expertise necessary to 

treat the range of issues/questions in the charge. Objective technical expertise and no conflict of 

interest or appearance of a lack of impartiality are critical characteristics for selecting pe

re

evaluated are necessary; nevertheless, it is also important to include a broad enough spectrum of 

other related experts to completely evaluate the relevant impacts on other less obvious c

(i.e., to comment not only if the job is being done right, but also whether the right job is being 

done). For example, for health related peer reviews, experts in such fields as ecology and 

economics may provide very useful insights. Although persons who are familiar with and 

substantial reputation in the field are often called upon repeatedly to be reviewers, it is im

to keep a balance with new people who bring fresh perspectives to the review of a work pro

The idea here is to avoid the repeated use of the same reviewer on multiple assessments un

his/her participation is essential and cannot be obtained elsewhere. 

There is usually a continuum of views on any issue. To the extent practicable, selected experts 

should have technically legitimate points of view tha

sh

well as those further along the continuum while generally avoiding the extremes. This will

maintain a balanced review panel, while allowing a broad range of views to be expressed and 

discussed. A balanced panel will allow consensus building if consensus is the object of a

particular peer review; if not, it provides a spectrum of views for the Agency to evaluate. As 

such, the group of reviewers should be sufficiently broad and diverse to fairly represent the 

relevant scientific and technical perspectives and fields of knowledge. 
 Peer Review Handbook Page 63 

A a ge u e had a predominant 

fluence on the position for a given organization on an issue, those who have clearly "taken 

t 

 a co ract mechanism or under 

e auspices of a Federal Advisory Committee. Finally, peer reviews may be conducted by 

ou ide iz

eview 

w leader or other appropriate 

fficial should be thoroughly familiar with the ethics policies and requirements of the 

tify 

s neral r le, experts who have made public pronouncements or hav

in

sides," may have an appearance of a lack of impartiality (see “Ethical Standards” below) and 

should be avoided. 

3.4.5 Ethical Standards

As discussed previously (see Section 2.4), there are various mechanisms through which peer 

reviews may be accomplished. Internal peer reviews can be conducted by independent experts

from within EPA, either individually or as ad hoc peer review panels. External peer reviews can

be conducted by individual experts (e.g., through letter reviews) who are either Regular 

Government Employees (RGEs) of another agency, experts hired as EPA Special Governmen

Employees (SGEs), or experts whose services are procured through a contract mechanism. In 

addition, external peer review panels can be convened through nt

th

ts organ ations such as the National Academy of Sciences (NAS). 

To ensure a credible peer review process, high ethical standards are applied to each of these 

mechanisms. These standards are embodied in the various laws, regulations, and other 

requirements that apply to peer reviewers who are either RGEs, SGEs, contractors, or who are 

employed by outside organizations (e.g., NAS). 

There are specific regulations that describe ethics considerations for contractor personnel. The 

Federal Acquisition Regulations (FAR), the Environmental Protection Agency Acquisition 

Regulations (EPAAR), and other internal Agency documents define ethical standards, such as 

Conflict of Interest (COI), and describe the Agency’s policies and procedures. This Peer R

Handbook provides some guidance regarding the application of this information to peer review 

(see Section 3.6 and Appendix F – Example Statements of Work for Contracts); however, 

specific questions should be addressed to the Office of Acquisition Management. 

For peer reviews conducted by outside organizations, the peer revie

o

organization conducting the review. For example, the NAS National Research Council has 

published their “Policy on Committee Composition and Balance and Conflicts of Interest.” 

The remainder of this section is focused on the laws and regulations governing the standards of 

ethical conduct for RGEs and SGEs. The standards for government employees are very 

thorough and, as such, may help the peer review leader or other appropriate official to iden

issues that may be more generically of concern with respect to the selection of peer reviewers.
 Peer Review Handbook Page 64 

Conflicts of Interest:

Public service is a public trust. As such, it is vitally important that government employees 

maintain the highest ethical standards. While there are many aspects of ethical behavior, for the 

urposes of peer review we are mostly concerned with compliance with applicable Federal ethics 

sta tes regulations are issued by the Office of Government Ethics 

GE) and are occasionally supplemented with additional regulations by EPA. 

lity.” 

o 

te as clear-cut as a 

conflict of interest” and calls for more of a judgment on the part of the determining official as to 

enerally, a conflict of interest arises when the person’s financial interests are affected by 

thics 

, see 

ttp://intranet.epa.gov/ogc/ethics/deos.htm

p

tu and regulations. The basic 

(O

Compliance with Federal ethics regulations can be a confusing and complex process. However,

the process is usually concerned with determining whether a RGE or a potential SGE has an 

ethics issue with respect to a “conflict of interest” or “an appearance of a lack of impartia

Each of these two concepts will be explained in more detail below. However, sometimes the tw

concepts are confused. A “conflict of interest” is concerned with matters of financial interest. 

There is either a conflict of interest or there is not. An “appearance of a lack of impartiality” 

(sometimes mischaracterized as a “potential conflict of interest”), can be financial or not 

financial in nature. The determination of an “appearance issue” is not qui

“

whether one exists. 

Let’s look at each of these two concepts in detail. 

 

G

his/her participation in a particular matter, when he/she, his/her associates or other individuals 

whose interests are imputed (as provided under Federal law) would derive benefit from 

incorporation of their point of view in an Agency product, and/or when their professional 

standing and status or the significance of their principal area of work might be affected by the 

outcome of the peer review. Whenever there are questions about conflicts of interest, you should

contact the appropriate Deputy Ethics Official, Ethics Assistant or the Designated Agency E

Official for clarification and assistance. For a list of EPA's Deputy Ethics Officials

h .

 that: 

capacity in any particular matter in which he, to his knowledge, or any person whose interests 

n 

element is missing, it is unlikely that there is a conflict of interest issue. However, the 

regulations concerning an appearance of a lack of impartiality may still apply and need to be 

considered. 

For Financial Conflict of Interest (COI) issues, the basic 18 USC § 208 provision states

An employee is prohibited from participating personally and substantially in an official 

are imputed to him under this statue has a financial interest, if the particular matter will have

a direct and predictable effect on that interest.

For a conflict of interest to be present, all elements in the above provision must be present. If a
 Peer Review Handbook Page 65 

O ry important aspect to c ne ve onsider is whether a charge to an advisory committee or other peer 

view panel constitutes a “particular matter.” A “particular matter” refers to matters that will 

 or a 

 particular 

atter involving specific parties means any judicial or other proceeding, application, request for 

s is 

n 

 a 

ct 

re

involve deliberation, decision, or action that is focused upon the interests of specific people,

discrete and identifiable class of people. It does not refer to consideration or adoption of broad 

policy options directed to the interests of a large and diverse group of people. A particular 

matter of general applicability means a particular matter that is focused on the interests of a 

discrete and identifiable class of persons, but does not involve specific parties. A

m

a ruling or other determination, contract, claim, controversy, investigation, charge, accusation, 

arrest or other particular matter involving a specific party or parties in which the United State

a party or has a direct and substantial interest. Participating personally means direct participatio

in the matter to be reviewed. Participating substantially refers to involvement that is of 

significance to the matter under consideration. For more information regarding the definition of

particular matter, see the Office of Government Ethics document “Standards of Ethical Condu

for Employees of the Executive Branch” available at http://www.usoge.gov. 

Another consideration is whether there will be a direct and predictable effect on the financial 

interests of the reviewers. A direct effect on a participant's financial interest exists if a close 

ausal link exists between any decision or action to be taken in the matter and any expected 

atter does not have a direct effect if 

sation is attenuated or is contingent upon the occurrence of events that are 

 has 

ot 

osed to a 

Es 

ee Act) committees are covered by certain 

xemptions from the conflict of interest statutes. The most significant of these is 5 CFR 

mployment. Specifically, this exemption permits SGEs serving on FACA committees to 

pa tions, 

po

em

urthermore, the question of whether a charge is a particular matter involving a specific party or 

ular 

.S. 

 contracts/grants as well as future representation back to the Federal government. 

or more information, see 18 USC § 207 or contact your Deputy Ethics Official/Ethics Assistant. 

c

effect of the matter on the financial interest. A particular m

the chain of cau

speculative or that are independent of, and unrelated to, the matter. A particular matter that

an effect on a financial interest only as a consequence of its effects on the general economy is n

considered to have a direct effect. A predictable effect exists if there is an actual, as opp

speculative, possibility that the matter will affect the financial interest. 

The question of whether a charge is a particular matter involving a specific party or parties or 

whether it is a particular matter of general applicability has important ramifications for reviewers 

who are Special Government Employees (SGEs) as defined under 18 USC § 202(a). SG

serving on FACA (Federal Advisory Committ

e

2640.203(g), which pertains to certain financial interests arising from an SGEs’s outside 

e

rticipate in particular matters of general applicability (e.g., development of general regula

licies or standards) where the disqualifying interest arises from the SGEs non-Federal 

ployment or prospective employment (but NOT due to financial holdings or consultancies). 

F

parties, or whether it is a particular matter of general applicability, has other ramifications for 

SGEs (and other Federal employees for that matter). For example, if a charge is a partic

matter involving a specific party or parties, there are certain restrictions regarding obtaining U

Government

F
 Peer Review Handbook Page 66 

Even if an SGE is determined to have a conflict of interest, that does not mean that it is 

impossible for her or him to participate in an advisory committee or other peer review. For 

example, in some cases it is possible to obtain a waiver for her or his participation. It is also 

possible to have a particular SGE be recused from certain portions of a review that would 

otherwise create a conflict of interest. However, you should always weigh the advantages a

disadvantages of this approach very carefully. In addition, public disclosure of these conflic

a manner compliant with the Privacy Act and other applicable regulations and practice, will a

to the integrity of the advisory committee and peer review process. Additional information for 

SGEs may be found in “Conflict of Interest and the Special Government Employee: A Summar

of Ethical Requirements Applicable to SGEs” available from the U.S. Office of Government 

Ethics (U.S. OGE) at 

nd 

ts in 

dd 

y 

http://www.usoge.gov.

Appearance of a Lack of Impartiality: 

The Code of Federal Regulations at 5 CFR § 2635.502(a) states that: 

Where an employee knows that a particular matter involving specific parties is likely to hav

a direct and predictable effect on the financial interest of a member of his household, or 

knows that a person with whom he has a covered relationship is or represents a party to s

matter, and where the person determines that the circumstances would cause a reasonable 

person with knowledge of the relevant facts to question his impartiality in the matter, the

employee should not participate in the matter unless he has informed the agency designee of 

the appearance problem and received authorization from the agency designee.” 

Further, § 2635.502(a)(2) states that, 

e 

uch 

nterest, 

nancial disclosure forms (described below) and public comment (where appropriate), it may be 

An employee who is concerned that circumstances other than those specifically described in

this section would raise a question regarding his impartiality should use the process 

described in this section to determine whether he should or should not participate in a 

particular matter.

Even though circumstances for specific candidates may not raise an issue of a conflict of i

each advisory committee member or peer reviewer should be evaluated to ensure that an 

appearance of a lack of impartiality does not preclude their participation. 

How do you evaluate candidates for appearance issues? In addition to collecting confidential 

fi

necessary on occasion to seek additional information from the potential advisory committee 

member. For example, you might want to consider asking additional questions which will help

you to determine whether there is an appearance issue. Some suggested questions would be: 
 Peer Review Handbook Page 67 

a) Do you know of any reason that you might be unable to provide impartial advice

matter to come before the Panel or any reason that your impartiality in the matter might 

be questioned? 

b) Have you had any previous involvement with the review document(s) under 

consideration? If so, please identify and describe that involvement. 

c) Have you served on previous advisory panels, committees or subcommittees that have 

addressed the topic under consideration? If so please identify those activities. 

d) Have you made any public statements

 on the 

 (written or oral) on the issue? If so, please identify 

those statements. 

e) Have you made any public statements that would indicate to an observer that you have 

ents. 

If 

de ar 

ad

De

 addition to the above considerations, you should check for a few more items to ensure 

Th n the 

outside activities of RGEs and SGEs, particularly involving activities involving the 

presentation of others before the Federal government. Section 203 prohibits an employee from 

d 

its an employee from 

ersonally representing anyone before any court or Federal agency or other specified Federal 

volving specific parties. Furthermore, the restrictions on SGEs are narrowly drawn to focus 

only on those matters in which the SGE actually participated for the Government, as well as, in 

some cases, those matters actually pending in the SGE’s own agency (in our case, EPA). 

taken a position on the issue under consideration? If so, please identify those statem

an SGE has an appearance of a lack of impartiality, it may still be possible to make a written 

termination that she or he may serve on an advisory committee or to ensure that a particul

visory committee member or peer reviewer is recused from certain areas of a review. See your

puty Ethics Official/Ethics Assistant for more information. 

A Few More Ethical Considerations:

In

compliance with Federal ethics laws: 

Restrictions on Representation:

ere are two statutes (18 USC § 203 and 18 USC § 205) that impose related restrictions o

re

receiving, agreeing to receive, or soliciting compensation for representation services before any 

court or Federal agency/entity in connection with any particular matter in which the Unite

States is a party or has a direct and substantial interest. Section 205 prohib

p

entity, in connection with any particular matter in which the United States is a party or has a 

direct and substantial interest. 

Fortunately, these restrictions are limited in their application to SGEs. One of the most 

significant limitations is that SGEs are restricted only in connection with particular matters 

in
 Peer Review Handbook Page 68 

Emoluments Clause of the U.S. Constitution:

The Department of Justice (DOJ) originally held that SGEs may be subject to the “Emoluments 

lause” of the United States Constitution (Article I, Section 9, clause 8) which prohibits persons 

wh on or 

receivi DOJ/Office of 

egal Counsel (March 9, 2005) states that SGEs who do not have access to classified 

info

in the F t is 

xtremely unlikely that any SGE performing a peer review for EPA would be subject to the 

Em

need m

he fine art of determining whether an advisory committee member or other peer reviewer has a 

ry 

ed to. 

The ma review, as well as maintaining the credibility of 

e Agency and the Agency’s scientific products, is of paramount importance. Peer review 

erest or 

. Finding a totally

C

o “hold office of profit or trust” in the Federal government from having any positi

ng any payment from a foreign government. However, an opinion from

L

rmation and who serve in purely advisory positions do not hold an “office of profit or trust”

ederal government and are not subject to the Emoluments Clause. Therefore, i

e

oluments Clause. As always, check with your Deputy Ethics Official/Ethics Assistant if you 

ore information. 

Putting It All Together:

T

conflict of interest or an appearance of a lack of impartiality can be difficult, confusing and 

complex. Needless to add, it is also important that any decision that is made concerning adviso

committee members or peer reviewers be appropriately documented and all applicable Federal 

record regulations and Privacy Act regulations be strictly adher

tter of obtaining a fair and credible peer 

th

leaders are strongly encouraged to obtain peer reviewers who do not have a conflict of int

an appearance of a lack of impartiality independent peer reviewer is a difficult 

nd often daunting task. See Section 3.4.6 for additional steps to take to ensure a credible peer 

review

rmation 

a

. 

For additional information on putting together an advisory panel (e.g., for highly influential 

scientific assessments), please refer to the SAB brochure “Overview of the Panel Fo

Process at the Environmental Protection Agency Science Advisory Board, September 2002" at 

http://www.epa.gov/sab/pdf/ec02010.pdf. 

3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of 

Conflicts of Interest?

Before finalizing the selection of reviewers, the Peer Review Leader should ascertain whether 

ach potential peer reviewer’s involvement in certain activities could pose a conflict of interest 

 is to 

flicts 

r 

 the 

e

(COI) or create an appearance of a lack of impartiality. One way of identifying conflicts

ask potential reviewers about work and clients (both current and prior) that might create con

or the appearance of a lack of impartiality in carrying out peer review activities. When the pee

review process is being conducted by a contractor, the requirement for addressing peer 

reviewers’ possible conflicts of interest should be highlighted in the Statement of Work or
 Peer Review Handbook Page 69 

work o ery Order, Task Order, etc.) and is a 

atter that is bound by contractual clauses with the Contracting Officer as the final Decision 

r 

that 

b) Exploring the issue directly with each participant before the review process takes place; 

c) Disclosing publicly at the beginning of meetings any previous involvement with the 

orm 3110-48 

(Confidential Financial Disclosure Form for Special Government Employees Serving on 

ny 

e) Collecting additional information through public comment, additional ethics questions 

f the peer review COI inquiry. This form is sent to 

each prospective peer reviewer by the Peer Review Leader (or contractor, in the case of 

actual review taking place. A follow-up contact with the Peer Review Leader (or 

contractor, in the case of contractor-run reviews) is then made to discuss any relevant 

is 

rdering instrument (i.e., Work Assignment, Deliv

m

Maker in contracting matters. 

You should take care to avoid ethical issues with respect to advisory committee members o

other peer reviewers and the work product under review. Remember, each situation is unique 

and should be treated on a case-by-case basis. The following are some considerations 

should be addressed in evaluating ethical issues: 

a) Paying attention to the employment, financial, and professional affiliations of the 

participants; 

issue; 

d) Obtaining confidential financial disclosure forms. In the cases of regular government 

employees (RGEs), use either the OGE Form 450 or the SF-278 Form as appropriate. In

the case of Special Government Employees (SGEs), use EPA F

Federal Advisory Committees at the U.S. Environmental Protection Agency). All these 

must be filed annually (with some exceptions). Remember, you can always discuss a

conflicts of interest or appearances of a lack of impartiality with your Deputy Ethics 

Official, Ethics Assistant or Deputy Agency Ethics Official as appropriate, but it is 

important to remember that there is no “attorney-client privilege” for these discussions; 

posed to potential advisory panel members and other appropriate means; 

f) In the case of non-Federal peer reviewers not on a FACA advisory panel (e.g., SAB, SAP

or BOSC), providing them a copy o

contractor-run reviews) to advise them of the need to address COI issues prior to the 

issues. The Peer Review Leader then documents this effort in the peer review record; th

includes a summary provided by the contractor documenting their inquiries and efforts. 

The Peer Review Leader should ensure that the peer review COI inquiry took place and that it 

appears in the peer review record. 
 Peer Review Handbook Page 70 

However, it should be strictly understood that Federal records maintained on individuals (defi

as any item, collection, or grouping of information about an individual that is maintained by a

agency, including, but not limited to, her or his e

ned 

n 

ducation, financial transactions, medical history, 

nd criminal or employment history and that contains his name, or the identifying number, 

nt 

a

symbol, or other identifying particular assigned to the individual, such as a finger or voice pri

or a photograph) are protected by the Privacy Act of 1974 (5 USC § 552a - Records maintained

on individuals). A copy can be found at http://www.usdoj.gov/04foia/privstat.htm. This Ac

states that no agency shall disclose any record wh

t 

ich is contained in a system of records by any 

eans of communication to any person, or to another agency, except pursuant to a written 

req . 

There are certain exceptions (e.g., pursuant to certain civil or criminal law enforcement activity). 

Thu

oes not contain information subject to the Privacy Act. Furthermore, Peer Review Leaders 

sho ee 

EPA’s National Records Management Program, http://www.epa.gov/records/

m

uest by, or with the prior written consent of, the individual to whom the record pertains

s, the peer review leader should ensure that any documentation in the peer review record 

d

uld ensure that any applicable EPA record-keeping procedures are followed (for details, s

). For information 

on Freedom of Information Act (FOIA) requests, please see http://www.epa.gov/foia. If you 

hav

contrac

Establi

Adviso

address

of a lac

Deputy of 

the Age

conflicts of interest or appearance issues in a contractual situation, obtain the involvement of the 

Con c

Of course, ethical issues do not necessarily arise merely because a peer reviewer knows 

som

e any questions, be sure to ask your appropriate Office of General Counsel and/or Office 

t official(s). 

shed peer review groups such as the Science Advisory Board (SAB), the Scientific 

ry Panel (SAP) and the Board of Scientific Counselors (BOSC) provide useful models for

ing balance and ethics issues. Assistance in identifying conflicts of interest/appearance 

k of impartiality and in providing an appropriate response can be obtained from the 

 Agency Ethics Official (located in the Office of the General Counsel) as well as any

ncy’s deputy ethics officials (DEOs) or Ethics Assistants. For assistance in evaluating 

tra ting Officer and the resources available within the Office of Acquisition Management. 

ething about the subject matter. In fact, experts with a stake in the outcome – and therefore a 

t or an appearance issue – may be some of the most knowledgeable and up-to-date exper

 they have concrete reasons to maintain their expertise. Such experts could be used 

d conflicts of interest are properly disclosed and appropriate waivers from the Office 

ment Ethics are obtained or, in the case of an appearance of a lack of impartiality, 

ure is made. In some cases, however, the conflict or appearance issue may be so direct

stantial as to rule out a particular expert. For instance, a potential peer reviewer may 

client or employer with a direct financial stake in the particular specific party matter 

conflic ts 

because

provide of 

Govern proper 

disclos

and sub

have a 

nder review, such as a Federal grant or contract to the potential peer reviewer or his/her 

review. However, review of a general 

ethodology that applies to numerous chemicals would not necessarily raise such a concern. 

u

employer that relates to the matter under review or the potential peer reviewer’s or their 

company’s work on a specific chemical under 

m
 Peer Review Handbook Page 71 

(Note: COI language should be made part of contracts/statements of work (SOW) or purchase 

orders (PO) – see Section 3.6.4.) 

A Peer Review Leader may also want to adopt measures that will prevent creation of conflic

the peer review is underway. Any measures contemplated that involve a contractual action must

be coordinated with the cognizant Contracting Officer. Some measures might include clauses in

a contract or purchase order that require reviewers to receive advance approval on future work

or place limits on such work, while they are performing the current peer review. Note that at 

some level these types of measures will discourage experts from serving as peer reviewers. 

Section 3.6.4 for further information dealing with contracts and suggestions for appropriate 

management controls.) 

ts as 

, 

 (See 

t a 

r of peer reviewers should be developed so that quick, effective peer review 

an be included for even the most rapidly moving products. 

e 

. 

e 

ffice of the General Counsel should be consulted on whether it is practical to obtain the consent 

hat 

. 

s of 

e (SGE) 

3.4.7 Are There Constraints to Selecting Peer Reviewers?

Sometimes the schedule for a peer review is accelerated due to a court-ordered deadline or other

time-sensitive requirements. In such cases, it is difficult, if not impossible to obtain external peer

reviewers in time to conduct a full external peer review. It may even be impossible to conduc

small scale internal peer review using just a few individuals. Mechanisms for identifying and

using a small numbe

c

Another possible constraint involves confidential business information (CBI). There are 

different definitions and types of CBI, depending on the statute that governs your action (e.g., 

Toxic Substances Control Act). To evaluate certain Agency-generated studies properly, som

peer reviewers may need access to CBI. However, unless the reviewers are Federal employees 

with CBI clearance, the Agency does not have the independent authority to disclose CBI to them

Therefore, whenever contemplating the use of outside peer reviewers, Agency staff should 

determine whether the reviewers will need access to CBI. If they do not have CBI clearance, th

O

of CBI submitters to disclose the information to peer reviewers. 

Offices need to be aware of the requirements of the Federal Advisory Committee Act (FACA) 

when establishing peer review mechanisms (see Section 2.8). Federal advisory committees t

are subject to chartering by the General Services Administration must hold meetings that are 

open to the public, and have balanced membership requirements. The Office of the General 

Counsel (OGC) should be consulted regarding the applicability of FACA to peer review panels

Occasionally, you may want to appoint a non-U.S. citizen to an advisory committee. Citizen

other countries may be appointed as representative members of an advisory committee. 

However, whether a non-U.S. citizen may be appointed as a special government employe

is a complicated question involving the Appropriations Act. 
 Peer Review Handbook Page 72 

Section 605 of the General Provisions of the Treasury and General Government Appropriations

Act provides, with some exceptions, that no appropriations can be used "to pay the compensation 

f any employee of the US Government" unless the person is a U.S. citizen or has filed a 

n 

sult 

tate. 

If e fo n tionship with the U.S. and 

o other exceptions apply to the individual, we may still hire the person as an SGE, but they 

 

3.4.8 If State Employees Are Used as Peer Reviewers, Can EPA Pay Them for This 

heir 

3.4.9 Does the Agency Recognize Peer Reviews That External Parties Conduct 

eer 

ex 

ppearance of a lack of impartiality for interested parties “paying” for a peer review of their own 

g peer review, the materials 

ould be treated by the Agency as anything else submitted for the Agency’s evaluation (i.e., 

o

declaration that they intend to become a U.S. citizen. There are a few exceptions to this 

restriction. One exception is for "nationals of those countries allied with the United States in a 

current defense effort." The appropriations restriction also provides that citizens of specific 

countries, refugees from certain countries, and students from China also are not subject to the 

restriction. If you want to hire a foreign national as an SGE, first contact OGC so that they ca

determine whether any exception applies to the person you are considering. Generally, to 

determine whether a country is allied with the U.S. in a current defense effort, OGC will con

with the Department of S

 

th reign ation in question does not have a qualifying defense rela

n

would have to agree in writing to forgo any compensation. For an SGE serving without 

compensation, we may reimburse the individual for his/her expenses (travel and per diem), but 

we may not make payments for "home work time" or time spent at advisory committee meetings.

The main point is to contact OGC if a foreign national is being considered for a FACA so that 

OGC can determine whether the person may be compensated. 

Service?

In some cases, this may be possible. First, the Peer Review candidate should determine if the 

State agency has a policy on whether its employees can perform this type of work, and whether 

the State policy will allow compensation. In most instances, EPA can also pay travel expenses 

(consult with your administrative staff for details). If the State person is not being paid for t

peer review services, they must sign an agreement stating that they do not expect payment (see 

Section 3.6.1 – Gratuitous Services). 

and/or Pay For?

There may be instances where parties external to EPA will want to conduct and/or pay for a p

review on a particular work product. This may look benign at first blush, but is a very compl

and sensitive situation that can raise significant concerns for conflicts of interest or an 

a

work product. While the Agency cannot prevent external parties from conducting and paying for 

a peer review, it is desirable that any such peer review is consistent with the intent of the 

Agency’s Peer Review Policy and implements the principles and guidance in this Handbook. If 

the external party submits their work product and accompanyin

sh
 Peer Review Handbook Page 73 

evaluation for scientific credibility and validity, as well as consistency with the provisions of the

Peer Review Handbook). 

3.5 Materials for Peer Reviewers

3.5.1 What Instructions Do You Give Peer Reviewers?

The Peer Review Leader is responsible for ensuring that peer reviewers understand their 

responsibilities (see Sections 3.2 and 3.6 if a contract is involved): 

a) Advise the Agency of actual or potential organizational or personal conflicts of interest or 

ther matters that would create the appearance of a lack of impartiality (see Section 

d) Inform on the planned extent of disclosure of names and attribution of comments. 

3.5 Materials Should be Sent to Peer Reviewers?

emember, documents should not be provided directly to a potential peer reviewer if that 

rev ew i the 

Agency provides the work product with associated background material to be peer reviewed to 

e prime contractor who in turn distributes these documents to the peer reviewers. In the case of 

 to 

o

3.4.6); 

b) Provide written comments in specified format by the specified deadline that are 

responsive to the charge; 

c) Comply with any requests for not disclosing draft work products to the public; 

.2 What 

For a peer review to be successful, peer reviewers should receive several documents at the 

beginning of the process. Typically, the most important among these documents are the charge 

letter and the current work product. The charge letter describes what the peer reviewers are 

being asked to do, and should serve to focus and structure the review. The work product is, of 

course, the material being subject to peer review. 

R

i er is go ng to be working under a contract or purchase order. In the case of a contract, 

th

a purchase order, the “charge or statement of work” must be part of the PO (purchase order) and 

the provision of any documents should be coordinated with the purchasing agent handling the 

order. 

Essential documentation for each peer reviewer includes: 

1) A current copy of the work product to be peer reviewed with associated background 

material. The work product should be of the best possible scientific/technical quality

ensure an adequate and useful peer review; 
 Peer Review Handbook Page 74 

2) A clear charge or statement of work, seeking informed comment on identified specific 

issues to properly focus the efforts of the peer reviewers and ensure that their individual 

efforts can be compared or contrasted; 

3) Information concerning the process that you use for the peer review, including the due 

d a point of contact in case 

the peer reviewer has questions. Responses should be written and submitted to the Peer 

y may 

ponse for the record should 

be made; 

4) aised 

tific issues; 

lic, even 

s. For example, all materials reviewed by the SAB and the 

FIFRA SAP are available. Agency managers may also decide that a broad accessibility 

products 

that are submitted for peer review (See Section 1.3.4). In other cases, confidentiality 

he 

need for confidentiality with regard to the release of Agency products that are stamped as 

ature release of draft 

Agency products, views, or positions may be inappropriate and can be damaging to the 

t, 

h 

tus (or a header or footer that states DRAFT 

status) on every page. In addition, in any solicitation for peer reviewers, the necessity for 

2) A bibliography and/or any particularly relevant scientific articles from the literature; 

3) A work product that has line numbering added in the margin for ease in providing and 

Peer Re o complete their task – they should not be 

verburdened with excess material. 

date of reviewer comments, the format of those responses, an

Review Leader by an agreed upon deadline. In certain rare cases, oral commentar

be sufficient; however, in such cases, a follow-up written res

For highly influential scientific assessments, copies of significant public comments r

on scien

5) In some cases, Agency materials being peer reviewed will be available to the pub

if they are marked as draft

has benefits for the Agency. Offices should include a disclaimer on draft work 

needs to be maintained. In these cases, each peer reviewer should be informed of t

"DRAFT" or "DRAFT - Do Not Cite, Quote, or Release." Prem

credibility of the Agency or the peer reviewer. Although it may not have legal effec

such language should be included in the charge to the peer reviewers when necessary. 

Other mechanisms to use in discouraging premature release include a disclaimer that 

appears in a separate section at the front of the document and creating the document wit

watermarks clearly delineating DRAFT sta

confidentiality and the non-release of materials should be emphasized. 

Useful, but not critical materials that may be sent to peer reviewers include: 

1) The name, address, and phone and fax numbers, and/or e-mail address of all peer 

reviewers working on the specific review; 

referencing comments. 

viewers should be given what is necessary t

o
 Peer Review Handbook Page 75 

a)

3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?

When EPA Conducts the Peer Review - The Peer Review Leader normally has 

administrative contacts with the reviewers during the development and conduct of the 

b) When the Contractor Conducts the Peer Review

peer review. In some cases (e.g., SAB peer review), peer reviewers may also receive a 

briefing on the product to be peer reviewed. Otherwise, the Peer Review Leader and 

other EPA staff should not contact the reviewers during the course of the review. Such 

contact can lead to perceived conflicts or inappropriate direction that could compromise 

the independence of the review. 

 - If peer review is conducted under a 

ould not have general contact and direction to the contractor’s staff 

or peer reviewers (sub-contractors). Note, when a peer review is conducted under a 

3.6 

A range

(gratuit

mechan

technic

Peer re

Regula

oversig tracts 

anagement Manual (CMM) for current approval levels. The CMM is available on the intranet 

contract mechanism, EPA should limit direct contact to the prime contractor’s designated 

representative and sh

contract, there are constraints where EPA staff are prohibited from contacting peer 

reviewers to avoid personal services arrangements. Personal services contracts exist 

when the nature of the relationship between the contractor and EPA can be characterized

as an employer-employee relationship. Any communications with peer reviewers should 

be coordinated through the prime contractor. 

Peer Review Services

 of peer review services are available to the Agency including internal, external 

ous services, contracts, purchase order), and Special Government Employee (SGE) 

isms. The mechanism selected is generally based on the nature of the scientific or 

al work product. 

view services are “advisory and assistance services”, as defined in Federal Acquisition 

tion (FAR) 37.201. These types of services require special approvals and management 

ht. See Figure 5, Item B Management Approvals, in Chapter 7 of the Con

M

at http://intranet.epa.gov/oamintra/policy/cmm.pdf. 

he provision of peer review products or services to EPA without compensation is provided as 

so-c

ithout compensation, EPA must ask them to sign an agreement whereby the person agrees to 

pro

receiving compensation for these services from EPA. An agreement, such as EPA Form 3100-14 

r a gratuitous services contract, must be executed because the Antideficiency Act (31 USC § 

as/fmsd/forms/f2.htm

3.6.1 What are Gratuitous Services for Peer Review?

T

alled “gratuitous” services. If a person wishes to perform peer review services for EPA 

w

vide the prescribed peer review services as gratuitous services, with no expectation of 

o

1342) prohibits the Agency’s acceptance of “voluntary” services. For a copy of EPA Form 

3100.14, see http://intranet.epa.gov/o . “Voluntary services” are services 
 Peer Review Handbook Page 76 

p vide P o 

PA. Note that persons cannot waive compensation (i.e., agree to provide gratuitous services) 

for ations 

concern

compen

volunte

attorne

t to Obtain Peer Review Services?

The

peer rev

scope c

solely f

A contr ase 

order is

Chapte

or assistance in preparing simplified acquisition packages for purchase orders, the Office of 

) has a guide called SAME: Simplified Acquisition Made Easy, 

hich is available on the intranet at http://intranet.epa.gov/oamintra/policy/index.htm

ro d to E A without an agreement in advance that such services are provided at no cost t

E

which there is a statutory right to payment, unless a law permits the waiver. For situ

ing State employees, see Section 3.4.9. Examples of laws that permit services without

sation are 5 USC § 3109 for experts or consultants and 5 USC § 3111 for student 

ers. If you are dealing with such gratuitous services, contact the appropriate OGC 

y for advice. 

3.6.2 Can You Use a Contrac

 Agency may obtain peer review services through a contract or purchase order. Typically,

iew services would be available under a “mission contract,” i.e., a contract with a broad 

overing a variety of services. It is also possible to have a contract or purchase order 

or peer reviews. 

act is awarded if the cost is over $100,000. If the cost is under $100,000, then a purch

 issued. For assistance in preparing the necessary pre-award contract documents, consult 

r 7 of the CMM. 

F

Acquisition Management (OAM

w . 

reparing the Statement of Work (SOW), consult Chapter 11 of the 

MM. The SOW should clearly specify that the contractor is responsible for preparing peer 

sibility 

lines. 

Th SO ontractor to include in the 

harge to the peer reviewers. While the contractor will be the one that will prepare and send the 

es of 

3.6.3 How Do You Write a Statement of Work for Contracts?

For general assistance in p

C

review evaluations and set forth guidelines for the peer review of scientific or technical 

documents. The contractor may perform the peer review with in-house staff, subcontractors or 

consultants. Any guidelines for performing peer reviews to ensure soundness and defen

should be developed by the Program Office and made part of the contract. The contractor would 

then ensure that the peer reviews adhere to the guide

e W should include the list of questions that EPA wants the c

c

formal charge to the peer reviewers, EPA should provide the list of questions to the contractor 

(see Section 3.2 for general discussion on charge to peer reviewers). 

The SOW cannot simply define the role of the prime contractor as arranging for the servic

others to perform peer reviews and logistics for meetings. Unless the prime contractor is clearly 

tasked with responsibility for performing peer reviews, individual peer reviewers' fees and 

associated travel expenses are not payable under the contract. 
 Peer Review Handbook Page 77 

EPA may pay for the reviewer's comments or evaluation, and also for attendance at a meeting 

with the Agency and other reviewers to discuss the results of the peer review. If the SOW calls 

for the preparation of comments or an evaluation, and specifies a meeting with the Agency and 

other peer reviewers to discuss the results of the peer review, payment is appropriate. The peer

reviewer's attendance at the meeting would then be part of contract performance. 

Example statements of work are found in Appendix F. 

2) the contractor’s work is free from conflicts of interest; 

uarded, and; 

 The 

ffice of Management and Budget Circular A-76, revised May 29, 2003, defines “inherently 

g ern lic interest as to 

andate performance by government personnel. These activities require the exercise of 

e 

gency policy is an IGA. Agency 

fficials make the official Agency decision regarding acceptability and/or quality of the 

s 

n 

 of their findings, a list of 

urces relied upon, and make clear and substantiate the methods and considerations upon which 

 is a 

3.6.4 What are Some Management Controls for Contracts?

Management controls ensure several things: 

1) the contractor does not perform inherently governmental activities (IGA); 

3) if provided to the contractor, confidential business information or sensitive information is 

appropriately safeg

4) improper relationships with contractor employees and subcontractors are avoided. 

Agency and Federal acquisition regulations prohibit contractors from performing IGA. 

O

ov mental activities” as activities that are so intimately related to the pub

m

substantial official discretion in the application of government authority and/or in making 

decisions for the government. 

The Agency cannot award contracts for IGA; however, certain functions, such as those involved

in peer review, may cross over into IGA if not properly managed. With peer review services, th

Agency is seeking only a contractor's recommendations, advice or analysis of a document, not a 

determination of what the policy should be. Determining A

o

document. To ensure that Agency officials are not improperly influenced by recommendation

in the peer review, the contract should include management controls. One possible control 

would be to direct the peer reviewers to submit with their evaluations or comments a descriptio

of the procedures used to arrive at their recommendations, a summary

so

their recommendations are based. To the extent possible, the contract should set forth any 

guidelines or criteria for performance of the peer review. Agency officials should document 

their evaluations of the quality and validity of the peer review. You should ensure that there

clear record of their review of the contractor's work and that Agency personnel made the final 
 Peer Review Handbook Page 78 

decisions. Such records of review could include notes from reviews of draft and final documen

by Agency personnel, minutes from progress meetings with contractors, and so forth. 

Conflict of Interest (COI) – Another important factor is that the objectivity of the peer review 

should not be improperly influenced or undermined by the contractor performing 

ts 

the review. To 

entify and avoid or mitigate actual or potential COI, the contract should include controls. 

tional COI clauses in the contract or 

urchase order is critical for peer review services. Usually, the EPA contracting officer (CO) 

will inc O f course without 

volvement by the EPA Project Officer. As a safeguard, the Project Officer should: 

a) Highlight the conflict of interest requirements in the Statement of Work (SOW) for the 

propriate conflict of interest 

clauses have been included, particularly EPAAR clause 1552.209-73, Notification of 

c) Work with the CO to develop contract-specific language regarding the peer review to 

impair the objectivity of peer reviewers. For example, a contractor may be advised to 

ny 

eir 

g any government service, for the preceding two years. 

ing 

 a 

n 

the preceding two years. 

id

Inclusion of Agency-developed individual and organiza

p

lude C I solicitation provisions and contract clauses as a matter o

in

procurement of the peer review services; 

b) Review the solicitation/contract to make sure that the ap

Conflicts of Interest Regarding Personnel; 

assist the contractor with identifying actual or potential conflicts of interest that might 

consider the following questions and issues when determining if a proposed peer 

reviewer may have an actual or potential conflict of interest: 

1) The sources and nature (obtained from a brief description of the work) of a

compensated and non-compensated employment of the panel member and th

spouse, includin

2) The sources of research support and project funding, including from any 

government source, for which the panel member served as the Principal 

Investigator, Significant Collaborator, Project Manager or Director during the 

preceding two years. For the panel member’s spouse, a general description of 

research and project activities in the preceding two years. 

3) The compensated consulting activities of the panel member during the preced

two years, including the names of clients if compensation provided 15% or more 

of annual compensation. For the panel member’s spouse, a general description of

consulting activities for the preceding two years. 

4) The sources of compensated expert witness activities of the panel member and

brief description of the issue and testimony during the preceding two years. For 

the panel member’s spouse, a general description of expert testimony provided i
 Peer Review Handbook Page 79 

5) The assets, including stocks, bonds, real estate, business, patents, trademarks, and 

royalties, for the panel member, their spouse and dependent children. 

Specifically, the financial holdings that collectively had a fair market value 

, 

mple, 

 consumer 

loans). 

tatements and/or positions of the panel member 

on, or closely related to, the matter under review. 

view materials) the individual has been asked 

to review (including previous peer reviews). 

on 

ose 

f the outcome of the 

review. 

Confidential B

Sensitive Info

Governm nt, it is unlikely that EPA will have authority to give reviewers access to confidential 

business rm 

Therefore, all d ation 

claimed as CB t is 

of a kind wher

submitter must itter has previously 

been informed that failure to assert a CBI claim may result in disclosure without notice. If the 

contracto ou

clauses can be 

required proce

requirements f use and disclosure of the data by 

contractor personnel. 

Personal serv

service contrac

deciding who to hire, supervising, and assigning tasks to contractor employees are personal 

greater than $15,000 at any time during the preceding two year period (excluding

for example, well-diversified mutual funds, money market funds, treasury bonds 

and personal residences). 

6) The liabilities over $10,000 owed by the panel member, their spouse, and 

dependent children at any time in the preceding one year (excluding, for exa

a mortgage on a personal residence, home equity loans, automobile and

7) A brief description of any public s

8) A brief description of any previous involvement of the panel member with the

development of the document (or re

9) A brief description of any other information that might reasonably raise a questi

about an actual or potential personal conflict of interest or bias including any 

financial benefit that might be gained by the panel member (or anyone wh

interests are imputed to the panel member) as a result o

usiness Information (CBI)/Privacy Act Protected Information and Other 

rmation – When peer reviewers are not employees of the United States 

e

info ation in the absence of consent for such disclosure by the CBI submitter.

ocuments provided to non-Federal reviewers must be screened for inform

I. Even where business information has not been explicitly claimed as CBI, if i

e the submitter might be expected to object to its release, prior to release the 

 be asked whether it wishes to assert a claim, unless the subm

r sh ld have access to CBI for your peer review, notify the CO so that the appropriate 

included in the contract or purchase order. These clauses will clearly identify any

dures or processes prior to release of any protected information, including any 

or confidentiality agreements, as well as limits on 

ices – Unless your organization has statutory authority to engage in personal 

ts, avoid treating contractor employees as Agency employees. For example, 
 Peer Review Handbook Page 80 

services. r a

of Contractor S tant that all 

Agency emplo ble 

on the intranet

 Fo dditional information, program officials should consult EPA Order 1901.1A, Use 

ervices to Avoid Improper Contracting Relationships. It is very impor

yees follow this order when interacting with contractors. The Order is availa

 at http://epawww.epa.gov/oamintra/policy/eo19011a.pdf. 

To avoid these improper relationships, program officials should write well-defined SOWs. The 

SOWs should set forth in detail a description of the work to be performed independently, 

including m is to be 

performed not 

ambiguous pro

considered supervision or assignment of tasks. 

3.

he prime contractor is responsible for selecting who will perform the peer review (EPA’s doing 

so may in

contractor or a

is a violation o In accordance with FAR clause 

2.244-2 Subcontracts, contractors may be required to obtain Contracting Officer (CO) consent 

or approv f

manager (WAM g 

the use of subc

EPA can establish criteria for the sort of individuals that might participate on a peer review 

anel. However, the Agency should not be involved in the selection of individual peer 

 may 

er 

 the 

PA acquires peer reviews through simplified acquisitions issued directly to peer reviewers or 

e 

ther 

 the anner in which it will be evaluated. The SOW should set forth what work

how the work is to be performed. Technical direction may be used to clarify 

visions to ensure efficient and effective contractor performance, and is not 

6.5 Can You Identify and/or Select Peer Reviewers When Using a Contract?

T

voke FACA; see Section 2.8), whether these individuals work directly for the 

re consultants or subcontractors to the prime contractor. Interfering in this process 

f Federal and Agency acquisition regulations.

5

al o subcontractors and the CO generally seeks the input of the work assignment 

 – the WAM may be the Peer Review Leader in many cases) before approvin

ontractors. 

p

reviewers, and should avoid commenting on the contractor’s selection of peer reviewers other 

than to determine whether the panel, once selected, meets the criteria established. EPA

identify a pool of qualified subcontractors and consultants to the prime contractor (listed in 

alphabetical order). If a list is provided, note on the list that this is a suggested list and oth

qualified candidates may exist who are not on the list. This is to prevent the impression that the

prime contractor can only choose someone on our list. The prime contractor is required to 

include a conflicts of interest clause substantially similar to the conflicts of interest clause in

primary contract in its subcontract to the peer reviewer. 

3.6.6 How is Travel Handled with Contracts or Purchase Orders?

Funds obligated on a contract or purchase order are available to pay for the costs of producing

the peer review including the travel costs and fee of the peer reviewer. 

E

through contracts with companies, which provide the peer review services. By issuing a 

purchase order or awarding a contract for peer review services, EPA may pay not only for th

peer review services/comments, but also for participation in a meeting with the Agency and o
 Peer Review Handbook Page 81 

reviewers to discuss comments. The scope of work of the contract must require the contractor or 

individual peer reviewer, as appropriate, to provide peer review services and indicate whether the 

contractor or peer reviewer will be required to discuss a specific peer review work product with 

the Agency and/or with other peer reviewers. Participation in a meeting to discuss a peer review 

ork product would then be part of the contract's performance. Thus, the contract may serve as 

 is defined in 18 USC § 202(a) as an officer or 

mployee of an agency who performs temporary duties, with or without compensation, for not 

m re th

er diem 

Es. 

w

the mechanism to pay for peer review services and associated travel expenses to provide 

comments to EPA. 

3.6.7 How is Travel Handled with Special Government Employees?

The term Special Government Employee (SGE)

e

o an 130 days in a period of 365 days, either on a full-time or intermittent basis. 

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid through 

the issuance of invitational travel orders (5 USC § 5703). These invitational travel and p

expenses should be charged to an appropriate EPA travel account. The Federal Travel 

Regulations govern the invited travelers reimbursement. 

Members of the SAB, SAP, and other FACA advisory committees are often classified as SG

It is not appropriate to reimburse travel or per diem expenses of advisory committee members 

(SGEs) through a contract. 
 Peer Review Handbook Page 82 

4. Completing A Peer Review

4.1 Overview

Performance of the formal peer review of the work product is not the final stage in the pro

evelopment. Rathe

duct’s 

r, it is an important stage in developing the work product, with the final 

e ion a . As a result, 

e ee w

ggestions in the final product assures a credible and defensible 

product for use in Agency decision-making. 

The peer review of a work product is not complete until the peer review comments are 

incorporated into the final version, or reasons are stated why such comments are not to be 

incorporated. However, for the purposes of documentation to the Agency’s Science Inventory 

only, the work product can be designated in the Science Inventory as having completed peer 

review once the peer review report is available on-line. 

The peer review record is complete only when it contains a copy of the final work product (when 

there is one) that addresses the peer review comments and a copy of the documentation outlining 

why comments were not incorporated, if any were not incorporated . (See Sec. 2.5.3 for a list of 

the minimum documentation for the peer review record.) 

4.2 Final Work Product

4.2.1 How Do You Incorporate Peer Review Comments into the Final Work 

Product?

The Peer Review Leader should carefully evaluate and analyze all peer review comments and 

recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers 

d

v rs (with ll comments addressed) representing the true end of the peer review

th p r revie process closes with four major activities: 

a) Evaluating peer review comments and recommendations 

b) Utilizing peer review comments for completing the final document, 

c) Organizing and maintaining a record of the peer review, and 

d) Completing the peer review record in the Science Inventory. 

Careful attention to all of these elements, singly and together, assures a credible and transparent 

peer review process. Conversely, inattention can nullify the peer review effort. A well-planned 

peer review applied to a reasonable quality starting work product, followed by a responsible 

utilization of peer review su
 Peer Review Handbook Page 83 

simplifies organizing and t are raised should be 

entified and evaluated.

jectivity of the comments should be evaluated. Analyses may include 

ther experts and staff within the Office and Agency. Adequate documentation 

n the Science Inventory.

Th Pe evie ppropriate managers in 

the ee ewers, the peer

view ments. It is the

spons val of the approach to

ddress learly identify for the

ecisio d and why, as well as

ny con

for the

e information EPA disseminates by 

entail significant impacts on the 

le, budget, or other resource needs. Management decisions to 

hese areas may be appropriate. 

b) The substantive issues or concerns expressed by peer reviewers may suggest that wider 

ct

es

analyzing comments. Also, any other issues tha

id

The validity and ob

consultation with o

is needed to show that comments are accepted or rejected. The documentation can be brief, but 

should address the legitimate, valid comments, whether accepted for incorporation in the final 

work product or not. The peer review record should contain a document describing the Agency’s

response to the peer review comments. The Agency’s response to the peer review report for 

highly influential scientific assessments should be posted o

e

P

er R w Leader should brief the Decision Maker (including all a

r Review Leader’s chain of command) on the charge, profile of peer revi

re comments, and provide a proposal on how to address the peer review com

re ibility of the Peer Review Leader to obtain Decision Maker appro

a ing the peer review comments. The Peer Review Leader should c

D n Maker any major peer review comments that will not be accepte

a troversial comments that should be resolved.

Comments that have significant impact on time, budgetary, or resource needs are particularly 

important and should be evaluated in consultation with 

management. These comments may lead to allocation of 

additional resources and a revised schedule for the 

ompletion of the work product if appropriate

The peer review is not complete until the 

peer review comments are incorporated 

into the final work product. c

final work products intended use. 

4.2.2 How Might Peer Review Comments 

Impact Your Work Product?

Peer review, when appropriate, enhances the quality of th

ensuring the information that governs EPA’s mission is reliable and accurate (objective) and is 

appropriate for the intended use (utility). A variety of changes to a work product may result 

from the comments provided during peer review: 

a) Peer review comments and recommendations may

planned project schedu

adjust one or more of t

scientific and technical consultation is needed to ensure the adequacy of the work produ

relative to its intended use. If the Agency agrees with the reviewers, additional resourc

and an extended delivery schedule may be necessary. 
 Peer Review Handbook Page 84 

c) The peer review comments and recommendations on a final product may provide a basis 

for bringing the associated project to closure. At times the Agency and its partners and 

stakeholders may not be able to come to consensus about a scientific or technical aspect 

charge 

along with 

escriptions of other public participation processes. It can be brief and does not need to describe 

sue and describe the reasons for the 

gency’s choices in the appropriate sections of the document. The level of detail that is 

f 

 peer-rev

n of its

w was not conducted. 

De vat (such as fact sheets, press releases, and 

brochures) do nderlying products were peer reviewed. 

rmation and highly influential scientific 

ssessments. Depending on the peer review process used, the reviewers names and affiliations 

may er 

informa cy 

Act, 5 U plementing guidance, 40 FR 

8,948 (July 9, 1975). 

Tho

attribut

In the o

comme litigation, the litigating parties 

of a work product. The scientific or technical issue in question can be noted in the 

to the reviewers as an area for which the Agency is specifically seeking peer review. 

4.2.3 What Should the Final Work Product Say About the Peer Review Process?

If the product has been peer reviewed, you should describe the peer review in the document. 

Frequently, this will be part of a description of the process of developing the product 

d

the process or discuss the peer-review comments in great detail. The description can be included 

in an introduction, preamble, or appendix. For influential scientific information and highly 

influential scientific assessments that support rulemaking, the peer review should be discussed in 

the preamble. 

When there are significant peer-review comments, and particularly if they are not being 

accepted, the document should generally discuss the is

A

provided is a matter of judgment and should reflect the importance and degree of controversy o

the issue. 

If a scientific or technical work product has not been

the document, perhaps in an introduction or descriptio

indicate the reasons that peer revie

iewed, this fact should be noted in 

 scope. This section should briefly 

ri ive products of scientific or technical products 

not need to discuss whether the u

4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous?

Generally, no. At a minimum, the names and affiliations of peer reviewers should be listed in

the peer review report for influential scientific info

a

 be made available to the public prior to the review commencing. Release of any review

tion retrieved by a personal identifier must be performed in accordance with the Priva

SC § 552a as amended, and as interpreted in OMB im

2

ugh the identity of reviewers becomes known at some point in the peer review process, the 

ion of specific comments to any given reviewer is kept confidential to the extent possible. 

rdinary course of events, you can often discuss comments received without attributing the 

nts to a specific reviewer. However, if a matter goes to
 Peer Review Handbook Page 85 

can

reviewe

review

Report

peer re ing 

eir names, credentials, and the extent of attribution of comments. 

 a peer reviewer requests anonymity at the outset of the peer review, the Peer Review Leader 

h the 

siness information). It is recognized that this may be a deterrent to possible peer 

viewers, but this is a reality that has to be understood. 

nce the Peer Review Leader has completed the peer review and the final work product (where 

 location of the peer review record 

ould be readily identifiable so interested parties can locate and obtain materials easily and 

if 

d work 

roduct may be sent to the peer reviewers for information. 

he Peer Review Leader should collect the materials identified in Section 2.5.3 and submit them 

ntory will become the primary public active record and archive for the 

eer review record, the Peer Review Leader is responsible for maintaining the on-site record. 

 discover the names of anyone who contributed to a Federal product, including peer 

rs. Also, in most cases, though specific comments are not attributed to specific peer 

ers, the names and affiliations of the peer reviewers are included in the Peer Review 

, which is included in the Science Inventory. Therefore, it is not possible to totally shield 

viewers. As noted previously, peer reviewers should be aware of EPA plans for releas

th

If

should inform the peer reviewer of the above possible eventualities. The Agency will in the 

ordinary course of events attempt to maintain the confidentiality of the peer reviewers and their 

attributable comments from public disclosure, but it is recognized in many instances, for 

example open public meetings, litigation, and Agency responses to Freedom of Information Act 

(FOIA) requests, this can’t be assured. Remember, the Agency is committed to working wit

fullest possible transparency to the public (except where statutorily constrained, such as with 

confidential bu

re

4.3 Completing the Peer Review Record

4.3.1 How Do You Complete the Peer Review Record? 

O

one is prepared), the peer review record is brought up to date and then archived according to that 

organization’s procedures (see Section 4.3.2). The peer review record should be indexed and 

maintained in an organization’s archive (repository). The

sh

quickly. The peer review record should be placed in any associated established public docket, 

so required, in addition to the organizational archive. As a courtesy, a copy of the revise

p

T

for archiving. Finally, the Agency Science Inventory entry for the project should be completed

and updated, including attachment of electronic copies of the documents. 

4.3.2 Where Should the Peer Review Records be Kept, and for How Long?

Though the Science Inve

p

During the active conduct of the peer review, the Peer Review Leaders maintain the peer review

record themselves until the peer review is totally completed. Minimally, the file should be 

maintained until one year after the completed peer review is reported in the next annual 

reporting. After that, the peer review record should be maintained for a “reasonable period of 
 Peer Review Handbook Page 86 

time.” Establishment and maintenance of the archive where the peer review records ulti

reside are an organization’s responsibility (i.e., not that of an individual program manager or 

Peer Review Leader). 

Generally, to allow flexibility, individual Offices and Regions should decide the appropriate 

level of organizational responsibility and how to ensure the record w

mately 

ill meet “routinely 

vailable.” The peer review record may be kept with other records relating to the overall project, 

n 

ption under the FOIA. (See 

ttp://www.epic.org/open_gov/foia/us_foia_act.html

a

as long as it is easily and separately identifiable. The peer review record should be maintained i

accordance with the Agency’s record-keeping schedule for such records. One long term 

archiving mechanism may be the formal archiving at the Federal Records Center; the Science 

Inventory will also maintain an electronic copy of the record. 

4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA? 

Yes, it is subject to release if EPA receives a Freedom of Information Act (FOIA) request, unless

the peer review information meets the criteria for an exem

h ).
 Peer Review Handbook Page A-1 

EPA’s Peer Review Policy 

January 31, 2006

Appendix A 
 Peer Review Handbook Page A-2 
 Peer Review Handbook Page A-3 

MEMORANDUM

SUBJECT: Peer Review Program

TO: Assistant Administrators 

General Counsel 

Inspector General 

Associate Administrators 

Regional Administrators 

Staff Office Directors 

We have made tremendous strides in improving our peer review program at EPA 

since the Agency’s Peer Review Policy was reaffirmed in 1994. Today I am updating the 

Peer Review Policy to emphasize the critical role that peer review plays in our efforts to 

ensure that EPA’s decisions rest on sound, credible science and data (see attached policy

statement). 

Peer review at EPA takes several different forms, ranging from informal 

consultations with Agency colleagues who were not involved in developing the product 

to the formal, public processes of the Science Advisory Board (SAB) and the FIFRA 

Scientific Advisory Panel (SAP). In any form, peer review assists EPA’s work by 

bringing independent expert experience and judgment to bear on issues before the 

Agency to the benefit of the final product. 

In 1994 the Science Policy Council (SPC) and its Steering Committee were asked 

to undertake an initiative to ensure that EPA has a comprehensive Agency-wide program

for implementing its Peer Review Policy. I commend the SPC for its diligence and 

success in meeting this objective. The SPC has made substantial improvements in the 

Peer Review Handbook, sponsored training of Agency managers and staff in peer review 

procedures, identified scientific and technical work products that merit peer review, and 

developed a publicly available data base of the peer review activities across the Agency. 

EPA has a strong and well-recognized peer review program as a direct result of these 

efforts. 
 Peer Review Handbook Page A-4 

In 2004 the Office of Management and Budget (OMB) issued a “Final 

Information Quality Bulletin for Peer Review” that contains provisions for peer review at 

all federal agencies. The OMB Bulletin applies to influential scientific information and 

highly influential scientific assessments. The SPC has updated the Agency's Peer 

Review Handbook, in part to incorporate the provisions of the OMB Bulletin, and to 

reflect the experience gained from implementing the program over the last decade. 

I ask that you continue to implement fully the provisions of our Peer Review 

Policy, and I expect the Science Policy Council to continue its role in overseeing and 

ram. We must ensure that our decisions are based 

n the highest quality, peer-reviewed scientific and technical information. 

strengthening EPA’s peer review prog

o

Attachment 

cc: Science Policy Council 

Science Policy Council Steering Committee
 Peer Review Handbook Page A-5 

PEER REVIEW AND PEER INVOLVEMENT 

NCY

ection 

ucts, including 

s. Peer 

ment their 

apabil

EPA strives to ensure that the scientific and technical bases of its decisions meet two 

important criteria: (1) they are based upon the best current knowledge from science, engineering, 

and other domains of technical expertise; and (2) they are credible. Peer review, a process based 

on the principles of obtaining the best technical and scientific expertise with appropriate 

independence, is central to sound science and helps the Agency meet these important criteria. 

Peer review occurs when scientifically and technically based work products are evaluated by 

levant experts who were not involved in creating the product. Properly applied, peer review 

ches the quality of work products but also adds a degree of credibility that cannot be 

chieved in any other way. Furthermore, peer review early in the development of work products 

me ture resources by steering the development along the most 

Peer review generally takes one of two approaches: 

• Internal

AT THE U.S. ENVIRONMENTAL PROTECTION AGE

 This document establishes the policy of the United States Environmental Prot

Agency (EPA) for peer review of scientifically and technically based work prod

economic and social science products, that are intended to inform Agency decision

review, a form of peer involvement, is one process through which EPA staff aug

c ities by inviting independent subject-matter experts to provide objective evaluation of the 

work product. 

PEER REVIEW 

re

not only enri

a

in so cases may conserve fu

efficacious course. 

, in which the reviewers are independent experts from inside EPA. 

• External, in which the reviewers are independent experts from outside EPA.

POLICY STATEMENT 

Peer review of all scientific and technical information that is intended to inform or 

support Agency decisions is encouraged and expected. Influential scientific information, 

including highly influential scientific assessments, should be peer reviewed in accordance with 

the Agency’s Peer Review Handbook. All Agency managers are accountable for ensuring that 

Agency policy and guidance are appropriately applied in determining if their work products are 

influential or highly influential, and for deciding the nature, scope, and timing of their peer 

review. For highly influential scientific assessments, external peer review is the expected 

procedure. For influential scientific information intended to support important decisions, or for 

work products that have special importance in their own right, external peer review is the 

approach of choice. Peer review is not restricted to the nearly final version of work products; in 

fact, peer review at the planning stage can often be extremely beneficial.
 Peer Review Handbook Page A-6 

LEGAL EFFECT

This po Rather, it

onfirms the importance of peer review where appropriate, outlines relevant principles, and 

licy

MPLEMENTATION

The Science Policy Council is responsible for overseeing Agency-wide implementation 

;

ain 

n

effective immediately.

licy statement does not establish or affect legal rights or obligations.

c

identifies factors Agency staff should consider in implementing the policy. On a continuing

basis, Agency management is expected to evaluate the policy as well as the results of its 

application throughout the Agency and undertake revisions as necessary. Therefore, the po

does not stand alone; nor does it establish a binding norm that is finally determinative of the

issues addressed. 

I

of this policy, including: promoting consistent interpretation; assessing Agency-wide progress

developing recommendations for revisions of the policy as necessary; and issuing the Peer 

Review Handbook, which provides additional information and procedures on implementing this 

policy. Assistant Administrators, Regional Administrators, and other senior managers rem

ultimately responsible for ensuring the appropriate application of Agency policy and guidance i

identifying work products subject to peer review, determining the type and timing of such 

review, documenting the process and outcome of each peer review, ensuring that the Science 

Inventory is kept current, and otherwise implementing the policy within their organizational 

units. 

 

The policy is
 Peer Review Handbook Page B-1 

Appendix B 

OMB Information Quality Bulletin for 

Peer Review and Preamble 
 Peer Review Handbook Page B-2 

OFFICE OF MANAGEMENT AND BUDGET 

inal Information Quality Bulletin for Peer Review 

DUCTION

his Bulletin establishes that important scientific information shall be peer reviewed by qualified 

ecialists before it is disseminated by the federal government. We published a proposed 

Bulletin on September 15, 2003. Based on public comments, we published a revised proposal 

for additional comment on April 28, 2004. We are now finalizing the April version, with minor 

revisions responsive to the public’s comments. 

The purpose of the Bulletin is to enhance the quality and credibility of the government’s 

scientific information. We recogn view are appropriate for 

different types of information. Un anted broad discretion to 

weigh the benefits and costs of usi anism for a specific 

info

info

the requirements inations, in 

order to ensure th gs. 

This Bulletin also applies stricter minimum requirements for the peer review of highly influential 

scientific assessments, which are a subset of influential scientific information. A scientific 

assessment is an evaluation of a body of scientific or technical knowledge that typically 

synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional 

judgment to bridge uncertainties in the available information. To ensure that the Bulletin is not 

too costly or rigid, these requirements for more intensive peer review apply only to the more 

important scientific assessments disseminated by the federal government. 

Even for these highly influential scientific assessments, the Bulletin leaves significant discretion 

to the agency formulating the peer review plan. In general, an agency conducting a peer review 

of a highly influential scientific assessment must ensure that the peer review process is 

transparent by making available to the public the written charge to the peer reviewers, the peer 

reviewers’ names, the peer reviewers’ report(s), and the agency’s response to the peer reviewers’ 

report(s). The agency selecting peer reviewers must ensure that the reviewers possess the 

necessary expertise. In addition, the agency must address reviewers’ potential conflicts of 

interest (including those stemming from ties to regulated businesses and other stakeholders) and 

independence from the agency. This Bulletin requires agencies to adopt or adapt the committee 

selection policies employed by the National Academy of Sciences (NAS)14 when selecting peer 

reviewers who are not government employees. Those that are government employees are subject 

to federal ethics requirements. The use of a transparent process, coupled with the selection of 

 

F

INTRO

T

sp

ize that different types of peer re

der this Bulletin, agencies are gr

ng a particular peer review mech

rmation product. The selection of an appropriate peer review mechanism for scientific 

rmation is left to the agency’s discretion. Various types of information are exempted from 

of this Bulletin, including time-sensitive health and safety determ

at peer review does not unduly delay the release of urgent findin

14 National Academy of Sciences, “Policy and Procedures on Committee Composition and Balance and Conflicts of 

Interest for Committees Used in the Development of Reports,” May 2003: Available at: 

http://www.nationalacademies.org/coi/index.html. 
 Peer Review Handbook Page B-3 

qualifie of go ernme scien e 

ent’s scientific products. 

aft to 

f the 

all 

rest to a 

cy 

f data 

 

d and independent peer reviewers, should improve the quality v nt c

while promoting public confidence in the integrity of the governm

PEER REVIEW

Peer review is one of the important procedures used to ensure that the quality of published 

information meets the standards of the scientific and technical community. It is a form of 

deliberation involving an exchange of judgments about the appropriateness of methods and the 

strength of the author’s inferences.15 Peer review involves the review of a draft product for 

quality by specialists in the field who were not involved in producing the draft. 

The peer reviewer’s report is an evaluation or critique that is used by the authors of the dr

improve the product. Peer review typically evaluates the clarity of hypotheses, the validity o

research design, the quality of data collection procedures, the robustness of the methods 

employed, the appropriateness of the methods for the hypotheses being tested, the extent to 

which the conclusions follow from the analysis, and the strengths and limitations of the over

product. 

Peer review has diverse purposes. Editors of scientific journals use reviewer comments to help 

determine whether a draft scientific article is of sufficient quality, importance, and inte

field of study to justify publication. Research funding organizations often use peer review to 

evaluate research proposals. In addition, some federal agencies make use of peer review to 

obtain evaluations of draft information that contains important scientific determinations. 

Peer review should not be confused with public comment and other stakeholder processes. The 

selection of participants in a peer review is based on expertise, with due consideration of 

independence and conflict of interest. Furthermore, notice-and-comment procedures for agen

rulemaking do not provide an adequate substitute for peer review, as some experts -- especially 

those most knowledgeable in a field -- may not file public comments with federal agencies. 

The critique provided by a peer review often suggests ways to clarify assumptions, findings, and

conclusions. For instance, peer reviews can filter out biases and identify oversights, omissions,

and inconsistencies.16 Peer review also may encourage authors to more fully acknowledge 

limitations and uncertainties. In some cases, reviewers might recommend major changes to the 

draft, such as refinement of hypotheses, reconsideration of research design, modifications o

collection or analysis methods, or alternative conclusions. However, peer review does not 

always lead to specific modifications in the draft product. In some cases, a draft is in excellent 

shape prior to being submitted for review. In others, the authors do not concur with changes 

suggested by one or more reviewers. 

15 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving 

Regulatory Decision Making, Carnegie Commission, New York, 1993: 75. 16 William W. Lowrance, Modern Science and Human Values, Oxford University Press, New York, NY 1985: 85. 
 Peer Review Handbook Page B-4 

Peer review may take a variety of forms, depending upon the nature and importance of the 

product. For example, the reviewers may represent one scientific discipline or a variety of 

disciplines; the number of reviewers may range from a few to more than a dozen; the names of 

ay be disclosed publicly or may remain anonymous (e.g., to encourage candor); 

e reviewers may be blinded to the authors of the report or the names of the authors may be 

ers 

ly or 

lic 

 of peer review are often only one of the criteria used to make decisions about journal 

ublication, grant funding, and information dissemination. For instance, the editors of scientific 

ut 

rly, 

w 

reports are an important factor in information dissemination decisions but rarely are the sole 

isions. 

here are a multiplicity of science advisory procedures used at federal agencies and across the 

he 

 

each reviewer m

th

disclosed to the reviewers; the reviewers may prepare individual reports or a panel of review

may be constituted to produce a collaborative report; panels may do their work electronical

they may meet together in person to discuss and prepare their evaluations; and reviewers may be

compensated for their work or they may donate their time as a contribution to science or pub

service. 

For large, complex reports, different reviewers may be assigned to different chapters or topics. 

Such reports may be reviewed in stages, sometimes with confidential reviews that precede a 

public process of panel review. As part of government-sponsored peer review, there may be 

opportunity for written and/or oral public comments on the draft product. 

The results

p

journals (rather than the peer reviewers) make final decisions about a manuscript’s 

appropriateness for publication based on a variety of considerations. In research-funding 

decisions, the reports of peer reviewers often play an important role, but the final decisions abo

funding are often made by accountable officials based on a variety of considerations. Simila

when a government agency sponsors peer review of its own draft documents, the peer revie

consideration. Agencies are not expected to cede their discretion with regard to dissemination or

use of information to peer reviewers; accountable agency officials must make the final dec

THE NEED FOR STRONGER PEER REVIEW POLICIES 

T

wide variety of scientific products prepared by agencies.17 In response to congressional inquiry, 

the U.S. General Accounting Office (now the Government Accountability Office) documented 

the variability in both the definition and implementation of peer review across agencies.18 T

Carnegie Commission on Science, Technology and Government19 has highlighted the 

importance of “internal” scientific advice (within the agency) and “external” advice (through 

scientific advisory boards and other mechanisms). 

 

17 Sheila Jasanoff, The Fifth Branch: Science Advisors as Policy Makers, Harvard University Press, Boston, 1990. 18 U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal Agencies Vary, 

d the Environment: Improving 

GAO/RCED-99-99, Washington, D.C., 1999. 

19 Carnegie Commission on Science, Technology, and Government, Risk an

Regulatory Decision Making, Carnegie Commission, New York, 1993: 90. 
 Peer Review Handbook Page B-5 

A wide variety of authorities have argued that peer review practices at federal agencies need

be strengthened.

 to 

cal, 

Prior to the development of this Bulletin, there were no government-wide standards concerning 

r 

ay to determine 

hen an important scientific information product is being developed by an agency, the type of 

y Act and OMB’s general authorities to 

versee the quality of agency information, analyses, and regulatory actions. In the Information 

20 Some arguments focus on specific types of scientific products (e.g., 

assessments of health, safety and environmental hazards). 21 The Congressional/Presidential 

Commission on Risk Assessment and Risk Management suggests that “peer review of economic 

and social science information should have as high a priority as peer review of health, ecologi

and engineering information.”22 

Some agencies have formal peer review policies, while others do not. Even agencies that have 

such policies do not always follow them prior to the release of important scientific products. 

when peer review is required and, if required, what type of peer review processes are 

appropriate. No formal interagency mechanism existed to foster cross-agency sharing of 

experiences with peer review practices and policies. Despite the importance of peer review fo

the credibility of agency scientific products, the public lacked a consistent w

w

peer review planned for that product, or whether there would be an opportunity to provide 

comments and data to the reviewers. 

This Bulletin establishes minimum standards for when peer review is required for scientific 

information and the types of peer review that should be considered by agencies in different 

circumstances. It also establishes a transparent process for public disclosure of peer review 

planning, including a web-accessible description of the peer review plan that the agency has 

developed for each of its forthcoming influential scientific disseminations. 

LEGAL AUTHORITY FOR THE BULLETIN

This Bulletin is issued under the Information Qualit

o

Quality Act, Congress directed OMB to issue guidelines to “provide policy and procedural 

 

20 National Academy of Sciences, Peer Review in the Department of Energy – Office of Science and Technology, 

Interim Report, National Academy Press, Washington, D.C., 1997; National Academy of Sciences, Peer Review in 

Environmental Technology Development: The Department of Energy – Office of Science and Technology, National 

Academy Press, Washington, D.C., 1998; National Academy of Sciences, Strengthening Science at the U.S. 

Environmental Protection Agency: Research-Management and Peer-Review Practices, National Academy Press, 

Washington, D.C. 2000; U.S. General Accounting Office, EPA’s Science Advisory Board Panels: Improved 

Policies and Procedures Needed to Ensure Independence and Balance, GAO-01-536, Washington, D.C., 2001; U. S. 

Environmental Protection Agency, Office of Inspector General, Pilot Study: Science in Support of Rulemaking 

2003-P-00003, Washington, D.C., 2002; Carnegie Commission on Science, Technology, and Government, In the 

National Interest: The Federal Government in the Reform of K-12 Math and Science Education, Carnegie 

Commission, New York, 1991; U.S. General Accounting Office, Endangered Species Program: Information on 

How Funds Are Allocated and What Activities are Emphasized, GAO-02-581, Washington, D.C. 2002. 21 National Research Council, Science and Judgment in Risk Assessment, National Academy Press, Wash

D.C., 1994. 

ington, 

eport, 

aking

22 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission R

Volume 2, Risk Assessment and Risk Management in Regulatory Decision-M , 1997:103. 
 Peer Review Handbook Page B-6 

guidance to Federal agencies for ensuring and maximizing the quality, objectivity, utility and 

integrity of information” disseminated by Federal agencies. Pub. L. No. 106-554, § 515

Information Quality Act was developed as a supplement to the Paperwork Reduction Act, 44 

U.S.C. § 3501 

(a). The 

et seq., which requires OMB, among other things, to “develop and oversee the 

implementation of policies, principles, standards, and guidelines to . . . apply to Federal agency 

dissemination of public information.” In addition, Executive Order 12866, 58 Fed. Reg. 51,735 

ct. 4, 1993), establishes that OIRA is “the repository of expertise concerning regulatory 

asonably obtainable scientific, technical, economic, or other information.” E.O. 12866, § 

Branch. 

dings 

l position of one or more agencies of the federal 

overnment. 

(O

issues,” and it directs OMB to provide guidance to the agencies on regulatory planning. E.O. 

12866, § 2(b). The Order also requires that “[e]ach agency shall base its decisions on the best 

re

1(b)(7). Finally, OMB has authority in certain circumstances to manage the agencies under the 

purview of the President’s Constitutional authority to supervise the unitary Executive 

All of these authorities support this Bulletin. 

THE REQUIREMENTS OF THIS BULLETIN

This Bulletin addresses peer review of scientific information disseminations that contain fin

or conclusions that represent the officia

g

Section I: Definitions

Section I provides definitions that are central to this Bulletin. Several terms are identical to o

based on those used in OMB’s government-wide information quality guideli

r 

nes, 67 Fed. Reg. 

8452 (Feb. 22, 2002), and the Paperwork Reduction Act, 44 U.S.C. § 3501 et seq. 

he term “Administrator” means the Administrator of the Office of Information and Regulatory 

3502(1). 

e 

t, 

In 

 of this Bulletin, the definition of “dissemination” modifies the definition in OMB’s 

T

Affairs in the Office of Management and Budget (OIRA). 

The term “agency” has the same meaning as in the Paperwork Reduction Act, 44 U.S.C. § 

The term “Information Quality Act” means Section 515 of Public Law 106-554 (Pub. L. No. 

106-554, § 515, 114 Stat. 2763, 2763A-153-154 (2000)). 

The term “dissemination” means agency initiated or sponsored distribution of information to th

public. Dissemination does not include distribution limited to government employees or agency 

contractors or grantees; intra- or inter-agency use or sharing of government information; or 

responses to requests for agency records under the Freedom of Information Act, the Privacy Ac

the Federal Advisory Committee Act, the Government Performance and Results Act, or similar

laws. This definition also excludes distribution limited to correspondence with individuals or 

persons, press releases, archival records, public filings, subpoenas and adjudicative processes. 

the context

government-wide information quality guidelines to address the need for peer review prior to 
 Peer Review Handbook Page B-7 

official dissemination of the information product. Accordingly, under this Bulletin, 

“dissemination” also excludes information distributed for peer review in compliance with this 

Bulletin or shared confidentially with scientific colleagues, provided that the distributing agen

includes an appropriate and clear disclaimer on the information, as explained more fully below

Finally, the Bulletin does not directly cover information supplied to the government by third 

parties (e.g., studies by private consultants, companies and private, non-profit organizations, or 

research institutions such as universities). However, if an agency plans to disseminate 

information supplied by a third party (e.g., using this information as the basis for an agency's 

factual determination that a particular behavior causes a disease), the requirements of the 

Bulletin apply, if the dissemination is "influential". 

In cases where a draft report or other information is released by an agency solely for purposes o

peer review, a question may arise as to wheth

cy 

. 

f 

er the draft report constitutes an official 

issemination" under information-quality guidelines. Section I instructs agencies to make this 

mer in the report: 

OT REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY 

AGENCY DETERMINATION OR POLICY.” 

In cases where the information is highly relevant to specific policy or regulatory 

ific 

formation presented at scientific meetings or shared confidentially with colleagues for 

BEEN FORMALLY 

ISSEMINATED BY [THE AGENCY] AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY 

n information product is not covered by the Bulletin unless it represents an official view of one 

by government-funded scientists (e.g., 

those supported extramurally or intramurally by federal agencies or those working in state or 

-

"d

clear by presenting the following disclai

“THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PREDISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY 

GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY]. IT

DOES N

deliberations, this disclaimer shall appear on each page of a draft report. Agencies also 

shall discourage state, local, international and private organizations from using 

information in draft reports that are undergoing peer review. Draft influential scient

in

scientific input prior to peer review shall include the disclaimer: “THE FINDINGS AND 

CONCLUSIONS IN THIS REPORT (PRESENTATION) HAVE NOT 

D

AGENCY DETERMINATION OR POLICY.” 

A

or more departments or agencies of the federal government. Accordingly, for the purposes of

this Bulletin, “dissemination” excludes research produced 

local governments with federal support) if that information is not represented as the views of a 

department or agency (i.e., they are not official government disseminations). For influential 

scientific information that does not have the imprimatur of the federal government, scientists

employed by the federal government are required to include in their information product a clear 

disclaimer that “the findings and conclusions in this report are those of the author(s) and do not 

necessarily represent the views of the funding agency.” A similar disclaimer is advised for non

government employees who publish government-funded research. 
 Peer Review Handbook Page B-8 

For the purposes of the peer review Bulletin, the term “scientific information” means

inputs, data, models, analyses, technical information, or scientific assessments related to such 

disciplines as the behavioral and social sciences, public health and medical sciences, life and 

earth sciences, engineering, or physical sciences. This includes any communication or 

representation of knowledge such as facts or data, in any medium or form, including textual, 

numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes 

information that an agency disseminates from a web page, but does not include the prov

hyperlinks on a web page to information that others disseminate. This definition excludes 

opinions, where the agency’s presentation makes clear that an individual’s opinion, rather 

statement of fact or of the agency’s findings and conc

 factual 

ision of 

than a 

lusions, is being offered. 

c policies 

-wide information quality guidelines 

nd the information quality guidelines of the agency. Information dissemination can have a 

signific he economic 

viabilit ributes. 

Alterna nce the 

respons 

sumptions, 

risk 

odels; hazard 

om 

d models used in scientific assessments have already 

een subject to some form of peer review (e.g., refereed journal peer review or peer review under 

The term “influential scientific information” means scientific information the agency reasonably 

can determine will have or does have a clear and substantial impact on important publi

or private sector decisions. In the term “influential scientific information,” the term "influential"

should be interpreted consistently with OMB's government

a

ant economic impact even if it is not part of a rulemaking. For instance, t

y of a technology can be influenced by the government’s characterization of its att

tively, the federal government's assessment of risk can directly or indirectly influe

e actions of state and local agencies or international bodies. 

One type of scientific information is a scientific assessment. For the purposes of this Bulletin, 

the term “scientific assessment” means an evaluation of a body of scientific or technical 

knowledge, which typically synthesizes multiple factual inputs, data, models, as

and/or applies best professional judgment to bridge uncertainties in the available information. 

These assessments include, but are not limited to, state-of-science reports; technology 

assessments; weight-of-evidence analyses; meta-analyses; health, safety, or ecological 

assessments; toxicological characterizations of substances; integrated assessment m

determinations; or exposure assessments. Such assessments often draw upon knowledge fr

multiple disciplines. Typically, the data an

b

Section II of this Bulletin). 

Section II: Peer Review of Influential Scientific Information

Section II requires each agency to subject "influential" scientific information to peer review prio

to dissemination. For dissemination of influential scientific information, Section II provides 

agencies broad discretion in determining what type of peer review is appropriate and what 

procedures should be employed to select appropriate reviewers. Agencies are directed to chose a

peer review mechanism that is adequate, giving due consideration to the novelty and complexity

of the science to be reviewed, the relevance of the information to decision making, the extent of 

prior peer reviews, and the expected benefits and costs of additiona

r 

l review. 
 Peer Review Handbook Page B-9 

The National Academy of Public Administration suggests that the intensity of peer review 

should be commensurate with the significance of the information being disseminated and the 

likely implications for policy decisions.23 Furthermore, agencies need to consider tradeoffs 

between depth of peer review and timeliness.24 More rigorous peer review is necessary 

information that is based on novel methods or presents complex challenges for interpretation.

Furthermore, the need for rigorous peer review is greater when the information contains 

precedent-setting methods or models, presents conclusions that are likely to change prevailing 

practices, or is likely to affect policy decisions that have a significant impact. 

This tradeoff can be considered in a benefit-cost framework. The costs of peer re

for 

 

view include 

both the direct costs of the peer review activity and those stemming from potential delay in 

ve to ensure that their peer 

view practices are characterized by both scientific integrity and process integrity. “Scientific 

anel; 

s 

eived 

ce to 

iewer comments; and adequacy of prior peer review. 

government and private actions that can result from peer review. The benefits of peer review are 

equally clear: the insights offered by peer reviewers may lead to policy with more benefits and/or

fewer costs. In addition to contributing to strong science, peer review, if performed fairly and 

rigorously, can build consensus among stakeholders and reduce the temptation for courts and 

legislators to second-guess or overturn agency actions.25 While it will not always be easy for

agencies to quantify the benefits and costs of peer review, agencies are encouraged to approach

peer review from a benefit-cost perspective. 

Regardless of the peer review mechanism chosen, agencies should stri

re

integrity,” in the context of peer review, refers to such issues as “expertise and balance of the 

panel members; the identification of the scientific issues and clarity of the charge to the p

the quality, focus and depth of the discussion of the issues by the panel; the rationale and 

supportability of the panel’s findings; and the accuracy and clarity of the panel report.” “Proces

integrity” includes such issues as “transparency and openness, avoidance of real or perc

conflicts of interest, a workable process for public comment and involvement,” and adheren

defined procedures.26 

When deciding what type of peer review mechanism is appropriate for a specific information 

product, agencies will need to consider at least the following issues: individual versus panel 

review; timing; scope of the review; selection of reviewers; disclosure and attribution; public 

participation; disposition of rev

 

23 National Academy of Public Administration, Setting Priorities, Getting Results: A New Direction for EPA, 

National Academy Press, Washington, D.C., 1995:23. 

24 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report, 

1997. 

25 Mark R. Powell, Science at EPA: Information in the Regulatory Process, Resources for the Future, Washington, 

D.C., 1999: 148, 176; Sheila Jasanoff, The Fifth Branch: Science Advisors as Policy Makers, Harvard University 

Press, Boston, 1990: 242. 

26 ILSI Risk Sciences Institute, “Policies and Procedures: Model Peer Review Center of 

Excellence,” 2002: 4. Available at http://rsi.ilsi.org/file/Policies&Procedures.pdf. 
 Peer Review Handbook Page B-10 

Individual versus Panel Review 

Letter reviews by several experts generally will be more expeditious than convening a panel o

experts. Individual letter reviews are more appropriate when a draft document covers only 

discipline or when premature disclosure of a sensitive report to a public panel could cause harm

to government or private interests. When time and resources warrant, panels are prefer

they tend to be more deliberative than individual letter reviews and the reviewers can learn from

each other. There are also multi-stage processes in which confidential letter r

f 

one 

able, as 

eviews are 

conducted prior to release of a draft document for public notice and comment, followed by a 

r 

reviewed by peers before the agency invests time 

and resources in implementing the model and interpreting the results. "Early" peer review occurs 

 and resource-draining litigation.” 

the peer 

eview and the specific advice sought. The importance of the information, which shapes the goal 

f the peer review, influences the charge. For instance, the goal of the review might be to 

determine the utility of a body of literature for drawing certain conclusions about the feasibility 

of a technology or the safety of a product. In this context, an agency might ask reviewers to 

t 

formal panel review. These more rigorous and expensive processes are particularly valuable fo

highly complex, multidisciplinary, and more important documents, especially those that are 

novel or precedent-setting. 

Timing of Peer Review 

As a general rule, it is most useful to consult with peers early in the process of producing 

information. For example, in the context of risk assessments, it is valuable to have the choice of

input data and the specification of the model 

in time to "focus attention on data inadequacies in time for corrections. 

When an information product is a critical component of rule-making, it is important to obtain 

peer review before the agency announces its regulatory options so that any technical corrections

can be made before the agency becomes invested in a specific approach or the positions of

interest groups have hardened. If review occurs too late, it is unlikely to contribute to the course 

of a rulemaking. Furthermore, investing in a more rigorous peer review early in the process 

“may provide net benefit by reducing the prospect of challenges to a regulation that later may 

trigger time consuming 27

Scope of the Review 

The “charge” contains the instructions to the peer reviewers regarding the objective of 

r

o

determine the relevance of conclusions drawn in one context for other contexts (e.g., differen

exposure conditions or patient populations). 

 

27 Fred Anderson, Mary Ann Chirba Martin, E Donald Elliott, Cynthia Farina, Ernest Gellhorn, John D. Gr

Boyden Gray, Jeffrey Holmstead, Ronald M. Levin, Lars Noah, Katherine Rhyne, Jonathan Baert Wiener, 

"Regulatory Improvement Legislation: Risk Assessment, Cost-Benefit Analysis, and Judicial Revie

aham, C. 

w," Duke 

Environmental Law and Policy Forum, Fall 2000, vol. XI (1): 132. 
 Peer Review Handbook Page B-11 

The charge to the reviewers should be determined in advance of the selection of the reviewers. 

 drafting the charge, it is important to remember the strengths and limitations of peer review. 

ting its 

te, reviewers should be asked to provide advice on the 

reasonableness of judgments made from the scientific evidence. However, the charge should 

wers are not to provide advice on the policy (e.g., the amount of 

ncertainty that is acceptable or the amount of precaution that should be embedded in an 

of the uncertainties for 

e technical conclusions drawn are clear. In addition, peer reviewers might be asked to consider 

In

Peer review is most powerful when the charge is specific and steers the reviewers to specific 

technical questions while also directing reviewers to offer a broad evaluation of the overall 

product. 

Uncertainty is inherent in science, and in many cases individual studies do not produce 

conclusive evidence. Thus, when an agency generates a scientific assessment, it is presen

scientific judgment about the accumulated evidence rather than scientific fact.28 Specialists 

attempt to reach a consensus by weighing the accumulated evidence. Peer reviewers can make 

an important contribution by distinguishing scientific facts from professional judgments. 

Furthermore, where appropria

make clear that the revie

u

analysis). Such considerations are the purview of the government.29 

The charge should ask that peer reviewers ensure that scientific uncertainties are clearly 

identified and characterized. Since not all uncertainties have an equal effect on the conclusions 

drawn, reviewers should be asked to ensure that the potential implications 

th

value-of-information analyses that identify whether more research is likely to decrease key 

uncertainties.30 Value-of-information analysis was suggested for this purpose in the report of the

Presidential/Congressional Commission on Risk Assessment and Risk Management.31 A 

description of additional research that would appreciably influence the conclusions of the 

assessment can help an agency assess and target subsequent efforts. 

Selection of Reviewers

Expertise. The most important factor in selecting reviewers is expertise: ensuring that the 

selected reviewer has the knowledge, experience, and skills necessary to perform the review. 

istics 

Agencies shall ensure that, in cases where the document being reviewed spans a variety of 

scientific disciplines or areas of technical expertise, reviewers who represent the necessary 

spectrum of knowledge are chosen. For instance, expertise in applied mathematics and stat

is essential in the review of models, thereby allowing an audit of calculations and claims of 

 

28 Mark R. Powell, Science at EPA: Information in the Regulatory Process, Resources for the Future, Washington, 

.C., 1999: 139. 

 Ibid. 

30 Granger Morgan and Max Henrion, “The Value of Knowing How Little You Know,” 

D

29

Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, 

Cambridge University Press, 1990: 307. 

31 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Repo

1997, Volume 1: 39, Volume 2: 91. 

rt, 
 Peer Review Handbook Page B-12 

significance and robustness based on the numeric data.32 For some reviews, evaluation of 

biological plausibility is as important as statistical modeling. Agencies shall consider requesting

that the public, including scientific and professional societies, nominate potential reviewers. 

Balance. While expertise is the primary consideration, reviewers should also be selected to 

present a diversity of scientific perspectives relevant to the subject. On most controversial 

the 

g., 

 and 

bjectivity. 

re

issues, there exists a range of respected scientific viewpoints regarding interpretation of 

available literature. Inviting reviewers with competing views on the science may lead to a 

sharper, more focused peer review. Indeed, as a final layer of review, some organizations (e.

the National Academy of Sciences) specifically recruit reviewers with strong opinions to test the

scientific strength and balance of their reports. The NAS policy on committee composition

balance33 highlights important considerations associated with perspective, bias, and o

Independence. In its narrowest sense, independence in a reviewer means that the reviewe

not involved in producing the draft document to be reviewed. However, for peer review o

documents, a broader view of independence is necessary to ensure cred

r was 

f some 

ibility of the process. 

eviewers are generally not employed by the agency or office producing the document. As the 

k, and 

l 

res a 

s have 

ork to be appropriate peer 

viewers for those agencies. This concern can be mitigated in situations where the scientist 

to be tested or the method to be developed, which effectively creates a 

uffer between the scientist and the agency. When an agency awards grants through a 

st’s 

n 

o that 

y 

 

R

National Academy of Sciences has stated, “external experts often can be more open, fran

challenging to the status quo than internal reviewers, who may feel constrained by organizationa

concerns.”34 The Carnegie Commission on Science, Technology, and Government notes that 

“external science advisory boards serve a critically important function in providing regulatory 

agencies with expert advice on a range of issues.”35 However, the choice of reviewers requi

case-by-case analysis. Reviewers employed by other federal and state agencies may possess 

unique or indispensable expertise. 

A related issue is whether government-funded scientists in universities and consulting firm

sufficient independence from the federal agencies that support their w

36 re

initiates the hypothesis 

b

competitive process that includes peer review, the agency’s potential to influence the scienti

research is limited. As such, when a scientist is awarded a government research grant through a

investigator-initiated, peer-reviewed competition, there generally should be no question as t

scientist's ability to offer independent scientific advice to the agency on other projects. This

contrasts, for example, to a situation in which a scientist has a consulting or contractual 

arrangement with the agency or office sponsoring a peer review. Likewise, when the agenc

32 William W. Lowrance, Modern Science and Human Values, Oxford University Press, New York, NY 1985: 86. 

alacademies.org/coi/index.html. epartment 

33 National Academy of Sciences, “Policy and Procedures on Committee Composition and Balance and Conflicts of 

Interest for Committees Used in the Development of Reports,” May 2003: Available at: 

http://www.nation 34 National Research Council, Peer Review in Environmental Technology Development Programs: The D

of Energy’s Office of Science and Technology, National Academy Press, Washington, D.C., 1998: 3. 35 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving 

Regulatory Decision Making, Carnegie Commission, New York, 1993: 90. 36 Lars Noah, “Scientific ‘Republicanism’: Expert Peer Review and the Quest for Regulatory Deliberation, Emory

Law Journal, Atlanta, Fall 2000:1066. 
 Peer Review Handbook Page B-13 

and a researcher work together (e.g., through a cooperative agreement) to design or implem

study, there is less independence from the agency. Furthermore, if a scientist has repeatedly 

served as a reviewer for the same agency, some may question whether that scientist is 

ent a 

ufficiently independent from the agency to be employed as a peer reviewer on agencyn 

ome agencies have built entire organizations to provide independent scientific advice while 

s

sponsored projects. 

As the foregoing suggests, independence poses a complex set of questions that must be 

considered by agencies when peer reviewers are selected. In general, agencies shall make a

effort to rotate peer review responsibilities across the available pool of qualified reviewers, 

recognizing that in some cases repeated service by the same reviewer is needed because of 

essential expertise. 

S

other agencies tend to employ ad hoc scientific panels on specific issues. Respect for the 

independence of reviewers may be enhanced if an agency collects names of potential reviewers 

(based on considerations of expertise and reputation for objectivity) from the public, including

scientific or professional societies. The Department of Energy’s use of the American Society of 

Mechanical Engineers to identify potential peer reviewers from a variety of different scientific 

societies provides an example of how professional societies can assist in the development of an 

independent peer review panel.37 

Conflict of Interest. The National Academy of Sciences defines “conflict of interest” as any 

financial or other interest that conflicts with the service of an individual on the review panel 

because it could impair the individual’s objectivity or could create an unfair competitive 

g 

o 

 and 

s 

ng 

practices of the National Academy of Sciences. Specifically, peer reviewers who are federal 

advantage for a person or organization.38 This standard provides a useful benchmark for 

agencies to consider in selecting peer reviewers. Agencies shall make a special effort to examine

prospective reviewers’ potential financial conflicts, including significant investments, consultin

arrangements, employer affiliations and grants/contracts. Financial ties of potential reviewers t

regulated entities (e.g., businesses), other stakeholders, and regulatory agencies shall be

scrutinized when the information being reviewed is likely to be relevant to regulatory policy. 

The inquiry into potential conflicts goes beyond financial investments and business relationships 

and includes work as an expert witness, consulting arrangements, honoraria and sources of grants 

and contracts. To evaluate any real or perceived conflicts of interest with potential reviewers

questions regarding the independence of reviewers, agencies are referred to federal ethic

requirements, applicable standards issued by the Office of Government Ethics, and the prevaili

employees (including special government employees) are subject to federal requirements 

governing conflicts of interest. See, e.g., 18 U.S.C. § 208; 5 C.F.R. Part 2635 (2004). With 

 

37 American Society for Mechanical Engineers, Assessment of Technologies Supported by the Office of Science and 

Technology, Department of Energy: Results of the Peer Review for Fiscal Year 2002, ASME Technical Publishing, 

Danvers, MA, 2003. 

38 National Academy of Sciences, “Policy and Procedures on Committee Composition and

nationalacademies.org/coi/index.html. 

Balance and Conflicts of Interest for Committees Used in the Development of Reports,” May 

2003: Available at: http://www.
 Peer Review Handbook Page B-14 

respect to reviewers who are not federal employees, agencies shall adopt or adapt the NAS 

policy for committee selection with respect to evaluating conflicts of interest.39 Both the NA

and the federal government recognize that under certain circumstances some conflict m

unavoidable in order to obtain the necessary expertise. 

S 

ay be 

See, e.g., 18 U.S.C. § 208(b)(3)

U.S.C. App. § 15 (go

; 5 

verning NAS committees). To improve the transparency of the process, 

when an agency determines that it is necessary to use a reviewer with a real or perceived conflict 

ions, the 

Peer re ewers must have a clear understanding of how their comments will be conveyed to the 

 issue 

eer review process. It is common at many journals and 

search funding agencies to disclose annually the slate of reviewers. Moreover, the National 

lly 

iations. 

 or 

, 

g. 

Public comments can be important in shaping expert deliberations. Agencies may decide that 

eer 

of interest, the agency should consider publicly disclosing those conflicts. In such situat

agency shall inform potential reviewers of such disclosure at the time they are recruited. 

Disclosure and Attribution: Anonymous versus Identified 

vi

authors of the document and to the public. When peer review of government reports is 

considered, the case for transparency is stronger, particularly when the report addresses an

with significant ramifications for the public and private sectors. The public may not have 

confidence in the peer review process when the names and affiliations of the peer reviewers are

unknown. Without access to the comments of reviewers, the public is incapable of determining 

whether the government has seriously considered the comments of reviewers and made 

appropriate revisions. Disclosure of the slate of reviewers and the substance of their comments 

can strengthen public confidence in the p

re

Academy of Sciences now discloses the names of its peer reviewers, without disclosing the 

substance of their comments. The science advisory committees to regulatory agencies typica

disclose at least a summary of the comments of reviewers as well as their names and affil

For agency-sponsored peer review conducted under Sections II and III, this Bulletin strikes a 

compromise by requiring disclosure of the identity of the reviewers, but not public attribution of 

specific comments to specific reviewers. The agency has considerable discretion in the 

implementation of this compromise (e.g., summarizing the views of reviewers as a group

disclosing individual reviewer comments without attribution). Whatever approach is employed

the agency must inform reviewers in advance of how it intends to address this issue. Information 

about a reviewer retrieved from a record filed by the reviewer's name or other identifier may be 

disclosed only as permitted by the conditions of disclosure enumerated in the Privacy Act, 5 

U.S.C. § 552a as amended, and as interpreted in OMB implementing guidance, 40 Fed. Re

28,948 (July 9, 1975). 

 

Public Participation 

peer review should precede an opportunity for public comment to ensure that the public receives 

the most scientifically strong product (rather than one that may change substantially as a result of 

peer reviewer suggestions). However, there are situations in which public participation in p

 

39 Ibid. 
 Peer Review Handbook Page B-15 

review is an important aspect of obtaining a high-quality product through a credible process

Agencies, however, should avoid open-ended comment periods, which may delay completion of

peer reviews and complicate the completion of the final work product. 

Public participation can take a variety of forms, including opportunities to provide oral 

comments before a peer review panel or requests to provide written comments to the peer 

reviewers. Another option is for agencies to publish a “request for comment” or other notice in 

which they solicit public comment before a panel of peer reviewers performs its work. 

. 

here 

lic 

r 

choose a 

ity 

ill 

ncy determines that a more rigorous or transparent review process is 

ermine a particular journal review process did not 

., the extent of uncertainty inherent in a finding) that the agency determines 

ould be addressed before disseminating that information. As such, prior peer review and 

y. 

 

Disposition of Reviewer Comments 

A peer review is considered completed once the agency considers and addresses the reviewers’ 

comments. All reviewer comments should be given consideration and be incorporated w

relevant and valid. For instance, in the context of risk assessments, the National Academy of 

Sciences recommends that peer review include a written evaluation made available for pub

inspection.40 In cases where there is a public panel, the agency should plan publication of the 

peer review report(s) and the agency’s response to peer reviewer comments. 

In addition, the credibility of the final scientific report is likely to be enhanced if the public 

understands how the agency addressed the specific concerns raised by the peer reviewers. 

Accordingly, agencies should consider preparing a written response to the peer review report 

explaining: the agency's agreement or disagreement, the actions the agency has undertaken o

will undertake in response to the report, and (if applicable) the reasons the agency believes those 

actions satisfy any key concerns or recommendations in the report. 

Adequacy of Prior Peer Review 

In light of the broad range of information covered by Section II, agencies are directed to 

peer review mechanism that is adequate, giving due consideration to the novelty and complex

of the science to be reviewed, the relevance of the information to decision making, the extent of 

55prior peer reviews, and the expected benefits and costs of additional review. 

Publication in a refereed scientific journal may mean that adequate peer review has been 

performed. However, the intensity of peer review is highly variable across journals. There w

be cases in which an age

necessary. For instance, an agency may det

address questions (e.g

sh

publication is not by itself sufficient grounds for determining that no further review is necessar

40 National Research Council, Risk Assessment in the Federal Government: Managing the Process, National 

Academy Press, Washington, D.C., 1983. 
 Peer Review Handbook Page B-16 

Section III: Peer Review of Highly Influential Scientific Assessments 

Whereas Section II leaves most of the considerations regarding the form of the peer review to the 

gency’s discretion, Section III requires a more rigorous form of peer review for highly 

o Section 

ing a 

; scope 

f the review; selection of reviewers; disclosure and attribution; public participation; and 

00 

most useful in 

ases where it is difficult for an agency to predict the potential economic effect of dissemination. 

e a 

ection III (2) clarifies that the principal findings, conclusions and recommendations in official 

 of Sciences that fall under this Section are generally presumed 

ot to require additional peer review. All other highly influential scientific assessments require a 

dge, experience 

nd skills required to perform the review whereas balance refers to the need for diversity in 

ot to 

 in 

lict 

As discussed under Section II, those peer reviewers who are federal employees, including 

pecial Government Employees, are subject to applicable statutory and regulatory standards for 

 

a

influential scientific assessments. The requirements of Section II of this Bulletin apply t

III, but Section III has some additional requirements, which are discussed below. In plann

peer review under Section III, agencies typically will have to devote greater resources and 

attention to the issues discussed in Section II, i.e., individual versus panel review; timing

o

disposition of reviewer comments. 

A scientific assessment is considered "highly influential" if the agency or the OIRA 

Administrator determines that the dissemination could have a potential impact of more than $5

million in any one year on either the public or private sector or that the dissemination is novel, 

controversial, or precedent-setting, or has significant interagency interest. One of the ways 

information can exert economic impact is through the costs or benefits of a regulation based on

the disseminated information. The qualitative aspect of this definition may be 

c

In the context of this Bulletin, it may be either the approach used in the assessment or the 

interpretation of the information itself that is novel or precedent-setting. Peer review can be

valuable in establishing the bounds of the scientific debate when methods or interpretations ar

source of controversy among interested parties. If information is covered by Section III, an 

agency is required to adhere to the peer review procedures specified in Section III. 

S

reports of the National Academy

n

review that meets the requirements of Section III of this Bulletin. 

With regard to the selection of reviewers, Section III(3)(a) emphasizes consideration of expertise

and balance. As discussed in Section II, expertise refers to the required knowle

a

scientific perspective and disciplines. We emphasize that the term "balance" here refers n

balancing of stakeholder or political interests but rather to a broad and diverse representation of 

respected perspectives and intellectual traditions within the scientific community, as discussed

the NAS policy on committee composition and balance.41 

Section III(3)(b) instructs agencies to consider barring participation by scientists with a conf

of interest. The conflict of interest standards for Sections II and III of the Bulletin are identical. 

S

41 National Academy of Sciences, “Policy and Procedures on Committee Composition and Balance and Conflicts of 

x.html. Interest for Committees Used in the Development of Reports,” May 2003: Available at: 

http://www.nationalacademies.org/coi/inde
 Peer Review Handbook Page B-17 

federal employees. For non-government employees, agencies shall adopt or adapt the NAS 

olicy for committee member selection with respect to evaluating conflicts of interest. 

y 

g as 

g the 

xperience and skills that are essential but cannot be obtained 

elsewhere. In evaluating the need for this exception, agencies shall use the NAS criteria for 

nt 

multiple 

assessments. Such repeated use should be avoided unless a particular reviewer’s expertise is 

 

l 

Section III(5) addresses opportunity for public participation in peer review, and provides that the 

ay 

assessment publicly available at the onset of a peer review process, the agency shall, whenever 

ived, the agency shall 

p

Section III(3)(c) instructs agencies to ensure that reviewers are independent of the agenc

sponsoring the review. Scientists employed by the sponsoring agency are not permitted to 

serve as reviewers for highly influential scientific assessments. This does not preclude Special 

Government Employees, such as academics appointed to advisory committees, from servin

peer reviewers. The only exception to this ban would be the rare situation in which a scientist 

from a different agency of a Cabinet-level department than the agency that is disseminatin

scientific assessment has expertise, e

assessing the appropriateness of using employees of sponsors (e.g., the government 

scientist must not have had any part in the development or prior review of the scientific 

information and must not hold a position of managerial or policy responsibility). 

We also considered whether a reviewer can be independent of the agency if that reviewer 

receives a substantial amount of research funding from the agency sponsoring the review. 

Research grants that were awarded to the scientist based on investigator-initiated, competitive, 

peer-reviewed proposals do not generally raise issues of independence. However, significa

consulting and contractual relationships with the agency may raise issues of independence or 

conflict, depending upon the situation. 

Section III(3)(d) addresses concerns regarding repeated use of the same reviewer in 

essential. Agencies should rotate membership across the available pool of qualified reviewers. 

Similarly, when using standing panels of scientific advisors, it is suggested that the agency rotate

membership among qualified scientists in order to obtain fresh perspectives and reinforce the 

reality and perception of independence from the agency. 

 

Section III(4) requires agencies to provide reviewers with sufficient background information, 

including access to key studies, data and models, to perform their role as peer reviewers. In this

respect, the peer review envisioned in Section III is more rigorous than some forms of journa

peer review, where the reviewer is often not provided access to underlying data or models. 

Reviewers shall be informed of applicable access, objectivity, reproducibility and other quality 

standards under federal information quality laws. 

agency shall, wherever possible, provide for public participation. In some cases, an assessment 

may be so sensitive that it is critical that the agency’s assessment achieve a high level of quality 

before it is publicized. In those situations, a rigorous yet confidential peer review process m

be appropriate, prior to public release of the assessment. If an agency decides to make a draft 

possible, provide a vehicle for the public to provide written comments, make an oral presentation 

before the peer reviewers, or both. When written public comments are rece
 Peer Review Handbook Page B-18 

ensure that peer reviewers receive copies of comments that address significant scientific issu

with ample time to consider them in their review. To avoid undue delay of agency activi

es 

ties, the 

agency shall specify time limits for public participation throughout the peer review process. 

e 

 has taken 

minate 

ite, including 

ll the materials related to the peer review such as the charge statement, peer review report, and 

then, 

ime 

. 

f 

 manage the peer review process in accordance with 

is Bulletin. The entity may be a scientific or professional society, a firm specializing in peer 

Section III(6) requires that agencies instruct reviewers to prepare a peer review report that 

describes the nature and scope of their review and their findings and conclusions. The report 

shall disclose the name of each peer reviewer and a brief description of his or her organizational

affiliation, credentials and relevant experiences. The peer review report should either summariz

the views of the group as a whole (including any dissenting views) or include a verbatim copy of

the comments of the individual reviewers (with or without attribution of specific views to 

specific names). The agency shall also prepare a written response to the peer review report, 

indicating whether the agency agrees with the reviewers and what actions the agency

or plans to take to address the points made by reviewers. The agency is required to disse

the peer review report and the agency's response to the report on the agency's webs

a

agency response to the review. If the scientific information is used to support a final rule 

where practicable, the peer review report shall be made available to the public with enough t

for the public to consider the implications of the peer review report for the rule being considered

 

Section III(7) authorizes but does not require an agency to commission an entity independent o

the agency to select peer reviewers and/or

th

review, or a non-profit organization with experience in peer review. 

Section IV: Alternative Procedures

Peer review as described in this Bulletin is only one of many procedures that agencies can 

employ to ensure an appropriate degree of pre-dissemination quality of influential scientific 

info r example, Congress has assigned the NAS a special role in advising the federal 

ational Academy of Sciences, (2) 

commission the National Academy of Sciences to peer review an agency draft scientific 

rmation. Fo

government on scientific and technical issues. The procedures of the NAS are generally quite 

rigorous, and thus agencies should presume that major findings, conclusions, and 

recommendations of NAS reports meet the performance standards of this Bulletin. 

As an alternative to complying with Sections II and III of this Bulletin, an agency may instead 

(1) rely on scientific information produced by the N

information product, or (3) employ an alternative procedure or set of procedures, specifically 

approved by the OIRA Administrator in consultation with the Office of Science and Technology

Policy (OSTP), that ensures that the scientific information product meets applicable 

information-quality standards. 

An example of an alternative procedure is to commission a respected third party other than the 

NAS (e.g., the Health Effects Institute or the National Commission on Radiation Protection and 

Measurement) to conduct an assessment or series of related assessments. Another example of 
 Peer Review Handbook Page B-19 

an alternative set of procedures is the three-part process used by the National Institutes of Healt

(NIH) to generate scientific guidance. Under that process, a scientific proposal or white paper is 

generated by a working group composed of external, independent scientific experts; that pap

h 

er is 

then forwarded to a separate external scientific council, which then makes recommendations to 

r 

process,” because it does not assure that qualified, impartial specialists in relevant fields have 

the agency. The agency, in turn, decides whether to adopt and/or modify the proposal. Fo

large science agencies that have diverse research portfolios and do not have significant 

regulatory responsibilities, such as NIH, an acceptable alternative would be 

to allow scientists from one part of the agency (for example, an NIH institute) to participate in 

the review of documents prepared by another part of the agency, as long as the head of the 

agency confirms in writing that each of the reviewers meets the NAS criteria relating to 

the appropriateness of using employees of sponsors (e.g., the government scientist must not 

have had any part in the development or prior review of the scientific information and must not

hold a position of managerial or policy responsibility). The purpose of Section IV is to 

encourage these types of innovation in the methods used to ensure pre-dissemination quality 

control of influential scientific information. 

The mere existence of a public comment process (e.g., notice-and-comment procedures under the

Administrative Procedure Act) does not constitute adequate peer review or an “alternative 

performed a critical evaluation of the agency's draft product.42 

Section V: Peer Review Planning

Section V requires agencies to begin a systematic process of peer review planning for influential 

hly influential scientific assessments) that the agency plans 

to disseminate in the foreseeable future. A key feature of this planning process is a webis 

s 

ach entry on the agenda shall include a preliminary title of the planned report, a short 

ill be 

he 

n’s requirements have been deferred or waived. If the agency, 

ministrator, has determined that it is appropriate to use a 

scientific information (including hig

accessible listing of forthcoming influential scientific disseminations (i.e., an agenda) that 

regularly updated by the agency. By making these plans publicly available, agencies will be

able to gauge the extent of public interest in the peer review process for influential scientific 

information, including highly influential scientific assessments. These web-accessible agenda

can also be used by the public to monitor agency compliance with this Bulletin. 

 E

paragraph describing the subject and purpose of the planned report, and an agency contact 

person. The agency shall provide its prediction regarding whether the dissemination w

“influential scientific information” or a “highly influential scientific assessment,” as the 

designation can influence the type of peer review to be undertaken. The agency shall discuss t

timing of the peer review, as well as the use of any deferrals. Agencies shall include entries in 

the agenda for influential scientific information, including highly influential scientific

assessments, for which the Bulleti

in consultation with the OIRA Ad

 

42 William W. Lowrance, Modern Science and Human Values, Oxford University Press, New York, NY 1985: 86. 
 Peer Review Handbook Page B-20 

Section IV “alternative procedure” for a specific dissemination, a description of that alternative 

procedure shall be included in the agenda. 

 

Furthermore, for each entry on the agenda, the agency shall describe the peer review plan. Each 

peer review plan shall include: (i) a paragraph including the title, subject and purpose of the 

planned report, as well as an agency contact to whom inquiries may be directed to learn t

specifics of the plan; (ii) whether the dissemination is likely to be influential s

information or a highly influential scientific assessment; (iii) the timing of the review (including

deferrals); (iv) whether the review will be conducted through a panel or individual letters (or

whether an alternative procedure will be exercised); (v) whether there will be opportunitie

the public to comment on the work product to be peer reviewed, and if so, how and when thes

opportunities will be provided; (vi) whether the agency will provide significant and relevant 

public comments to the peer reviewers before they conduct their review; (vii) the anticipa

number of reviewers (3 or fewer; 4-10; or more than 10); (viii) a succinct description of the 

primary disciplines or expertise needed in the r

he 

cientific 

s for 

e 

ted 

eview; (ix) whether reviewers will be selected by 

e agency or by a designated outside organization; and (x) whether the public, including 

x months. However, in some 

cases -- particularly for highly influential scientific assessments and other particularly important 

me to 

he peer review planning requirements of this Bulletin are designed to be implemented in 

uments 

fter 

th

scientific or professional societies, will be asked to nominate potential peer reviewers. The 

agency shall provide a link from the agenda to each document made public pursuant to this 

Bulletin. Agencies shall link their peer review agendas to the U.S. Government’s official web 

portal: firstgov at http://www.FirstGov.gov 

Agencies should update their peer review agendas at least every si

information -- more frequent updates of existing entries on the agenda, or the addition of new 

entries to the agenda, may be warranted. When new entries are added to the agenda of 

forthcoming reports and other information, the public should be provided with sufficient ti

comment on the agency's peer review plan for that report or product. Agencies shall consider 

public comments on the peer review plan. Agencies are encouraged to offer a listserve or similar 

mechanism for members of the public who would like to be notified by email each time an 

agency’s peer review agenda has been updated. 

T

phases. Specifically, the planning requirements of the Bulletin will go into effect for doc

subject to Section III of the Bulletin (highly influential scientific assessments) six months a

publication. However, the planning requirements for documents subject to Section II of the 

Bulletin do not go into effect until one year after publication. It is expected that agency 

experience with the planning requirements of the Bulletin for the smaller scope of documents 

encompassed in Section III will be used to inform implementation of these planning 

requirements for the larger scope of documents covered under Section II. 

Section VI: Annual Report

Each agency shall prepare an annual report that summarizes key decisions made pursuant to this 

Bulletin. In particular, each agency should provide to OIRA the following: 1) the number of 
 Peer Review Handbook Page B-21 

peer reviews conducted subject to the Bulletin (i.e., for influential scientific information and 

highly influential scientific assessments); 2) the number of times alternative procedures were 

voked; 3) the number of times waivers or deferrals were invoked (and in the case of deferrals, 

 in public 

re 

in

the length of time elapsed between the deferral and the peer review); 4) any decision to appoint a 

reviewer pursuant to any exception to the applicable independence or conflict of interest 

standards of the Bulletin, including determinations by the Secretary or Deputy Secretary 

pursuant to Section III (3) (c); 5) the number of peer review panels that were conducted

and the number that allowed public comment; 6) the number of public comments provided on the

agency’s peer review plans; and 7) the number of peer reviewers that the agency used that we

recommended by professional societies. 

Section VII: Certification in the Administrative Record 

If an agency relies on influential scientific information or a highly influential scientific 

assessment subject to the requirements of this Bulletin in support of a regulatory action, the 

agency shall include in the administrative record for that action a certification that explain

the agency has complied with the requirements of this Bulletin and the Information Quality A

Relevant materials are to be placed in the administrative record. 

s how 

ct. 

Section VIII: Safeguards, Deferrals, and Waivers

omply 

Section VIII also allows for a deferral or waiver of the requirements of the Bulletin where 

the 

ive 

d deferral 

not otherwise covered 

y the exemptions to the Bulletin, such as situations where unavoidable legal deadlines prevent 

lletin before information is disseminated. Deadlines found in 

onsent decrees agreed to by agencies after the Bulletin is issued will not ordinarily warrant 

Section VIII recognizes that individuals serving as peer reviewers have a privacy interest in 

information about themselves that the government maintains and retrieves by name or identifier 

from a system of records. To the extent information about a reviewer (name, credential, 

affiliation) will be disclosed along with his/her comments or analysis, the agency must c

with the requirements of the Privacy Act, 5 U.S.C. 552a, as amended, and OMB Circular A-130, 

Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a 

published System of Records Notice. Furthermore, the peer review must be conducted in a 

manner that respects confidential business information as well as intellectual property. 

necessary. Specifically, the agency head may waive or defer some or all of the peer review

requirements of Sections II or III of this Bulletin if there is a compelling rationale for waiver or 

deferral. Waivers will seldom be warranted under this provision because the Bulletin already 

provides significant safety valves, such as: the exemptions provided in Section IX, including 

exemption for time-sensitive health and safety information; the authorization for alternat

procedures in Section IV; and the overall flexibility provided for peer reviews of influential 

scientific information under Section II. Nonetheless, we have included this waiver an

provision to ensure needed flexibility in unusual and compelling situations 

b

full compliance with the Bu

c

waiver of the Bulletin’s requirements because those deadlines should be negotiated to permit 

time for all required procedures, including peer review. In addition, when an agency is 
 Peer Review Handbook Page B-22 

unavoidably up against a deadline, deferral of some or all requirements of the Bulletin (as 

opposed to outright waiver of all of them) is the most appropriate accommodation between the

need to satisfy immovable deadlines and the need to undertake proper peer review. If the agency

head defers any of the peer review requirements prior to dissemination, peer review should be 

conducted as soon as practicable thereafter. 

Section IX: Exemptions

There are a variety of situations where agencies need not conduct peer review under this 

Bulletin. These include, for example, disseminations of sensitive information related to certain 

ational security, foreign affairs, or negotiations involving international treaties and trade where 

r secrecy or promptness. 

it 

hat 

ings, licensing, approval and 

registration processes for specific product development activities as well as site-specific 

iew is practical and appropriate is left to the 

discretion of the agency. While this Bulletin is not broadly applicable to adjudications, agencies 

ential 

t or 

nimal infectious diseases. 

n

compliance with this Bulletin would interfere with the need fo

This Bulletin does not cover official disseminations that arise in adjudications and perm

proceedings, unless the agency determines that peer review is practical and appropriate and t

the influential dissemination is scientifically or technically novel (i.e., a major change in 

accepted practice) or likely to have precedent-setting influence on future adjudications or permit 

proceedings. This exclusion is intended to cover, among other th

activities. The determination as to whether peer rev

are encouraged to hold peer reviews of scientific assessments supporting adjudications to the

same technical standards as peer reviews covered by the Bulletin, including transparency and 

disclosure of the data and models underlying the assessments. Protections apply to confid

business information. 

The Bulletin does not cover time-sensitive health and safety disseminations, for example, a 

dissemination based primarily on data from a recent clinical trial that was adequately peer 

reviewed before the trial began. For this purpose, “health” includes public health, or plan

a

This Bulletin covers original data and formal analytic models used by agencies in Regulatory

Impact Analyses (RIAs). However, the RIA documents themselves are already reviewed 

through an interagency review process under E.O. 12866 that involves application of the 

principles and methods defined in OMB Circular A-4. In that respect, RIAs are excluded from 

coverage by this Bulletin, although agencies are encouraged to have RIAs reviewed by peers

within the government for adequacy and completeness. 

 

The Bulletin does not cover accounting, budget, actuarial, and financial information including 

that which is generated or used by agencies that focus on interest rates, banking, currency, 

securities, commodities, futures, or taxes. 
 Peer Review Handbook Page B-23 

Routine statistical information released by federal statistical agencies (e.g., periodic demog

and economic statistics) and analyses of these data to compute standard indicators and trends 

(e.g., unemployment and poverty rates) is excluded from this Bulletin. 

 

The Bulletin does not cover information dissem

raphic 

inated in connection with routine rules that 

materially alter entitlements, grants, user fees, or loan programs, or the rights and obligations of 

the 

quirements of this Bulletin even though the first dissemination was not. 

recipients thereof. 

If information is disseminated pursuant to an exemption to this Bulletin, subsequent 

disseminations are not automatically exempted. For example, if influential scientific information

is first disseminated in the course of an exempt agency adjudication, but is later disseminated in 

the context of a non-exempt rulemaking, the subsequent dissemination will be subject to 

re

Section X: OIRA and OSTP Responsibilities

OIRA, in consultation with OSTP, is responsible for overseeing agency implementation of this 

Bulletin. In order to foster learning about peer review practices across agencies, OIRA an

OSTP shall form an interagency workgroup on peer review that meets regularly, discusses 

progress and challenges, and recommends improvements to peer review practices. 

Section XI: Effective Date and Existing Law

d 

The requirements of this Bulletin, with the exception of Section V, apply to information 

disseminated on or after six months after publication of this Bulletin. However, the Bulletin 

does not apply to information that is already being addressed by an agency-initiated peer review 

process (e.g., a draft is already being reviewed by a formal scientific advisory committee 

stablished by the agency). An existing peer review mechanism mandated by law should be 

ntial 

tin. The requirements in Section V apply to documents subject to 

ection II of the Bulletin one year after publication of the final Bulletin. 

e

implemented by the agency in a manner as consistent as possible with the practices and 

procedures outlined in this Bulletin. The requirements of Section V apply to “highly influe

scientific assessments,” as designated in Section III of the Bulletin, within six months of 

publication of the final Bulle

S

Section XII: Judicial Review

This Bulletin is intended to improve the internal management of the Executive Branch and is no

intended to, and does not, create any right or benefit, substantive or procedural, enforceable a

law or in equity, against the United States, its agencies or

t 

t 

 other entities, its officers or 

employees, or any other person. 
 Peer Review Handbook Page B-24 
 Peer Review Handbook Page B-25 

Bulletin for Peer Review 

I. Definitions. 

For purposes of this Bulletin -- 

 1. the term “Administrator” means the Administrator of the Office of Information and 

Regulatory Affairs in the Office of Management and Budget (OIRA); 

 2. the term “agency” has the same meaning as in the Paperwork Reduction Act, 44 U.S.C. § 

3502(1); 

 3. the term “dissemination” means agency initiated or sponsored distribution of information to 

the public (see 5 C.F.R. 1320.3(d) (definition of “Conduct or Sponsor”)). Dissemination does 

not include distribution limited to government employees or agency contractors or grantees; 

intra- or inter-agency use or sharing of government information; or responses to requests for 

agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory 

Committee Act, the Government Performance and Results Act or similar law. This definition 

also excludes distribution limited to correspondence with individuals or persons, press releases, 

archival records, public filings, subpoenas and adjudicative processes. The term “dissemination” 

also excludes information distributed for peer review in compliance with this Bulletin, provided 

that the distributing agency includes a clear disclaimer on the information as follows: “THIS 

INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PREDISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY 

GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY]. 

IT DOES NOT REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY 

AGENCY DETERMINATION OR POLICY.” For the purposes of this Bulletin, 

“dissemination” excludes research produced by government-funded scientists (e.g., those 

supported extramurally or intramurally by federal agencies or those working in state or local 

governments with federal support) if that information does not represent the views of an agency. 

To qualify for this exemption, the information should display a clear disclaimer that “the 

findings and conclusions in this report are those of the author(s) and do not necessarily represent 

the views of the funding agency”; 

 4. the term “Information Quality Act” means Section 515 of Public Law 106-554 (Pub. L. No. 

106-554, § 515, 114 Stat. 2763, 2763A-153-154 (2000)); 

 5. the term “scientific information” means factual inputs, data, models, analyses, technical 

information, or scientific assessments based on the behavioral and social sciences, public health 

and medical sciences, life and earth sciences, engineering, or physical sciences. This includes 

any communication or representation of knowledge such as facts or data, in any medium or form, 

including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This 

definition includes information that an agency disseminates from a web page, but does not 

include the provision of hyperlinks to information that others disseminate. This definition does 

not include opinions, where the agency’s presentation makes clear that what is being offered is 

someone’s opinion rather than fact or the agency’s views; 

 6. the term “influential scientific information” means scientific information the agency 

reasonably can determine will have or does have a clear and substantial impact on important 

public policies or private sector decisions; and 
 Peer Review Handbook Page B-26 

 7. the term “scientific assessment” means an evaluation of a body of scientific or technical 

nowledge, which typically synthesizes multiple factual inputs, data, models, assumptions, 

fessional judgment to bridge uncertainties in the available information. 

hese assessments include, but are not limited to, state-of-science reports; technology 

analyses; meta-analyses; health, safety, or ecological risk 

 hazard 

r Review of Influential Scientific Information

k

and/or applies best pro

T

assessments; weight-of-evidence 

assessments; toxicological characterizations of substances; integrated assessment models;

determinations; or exposure assessments. 

II. Pee . 

 1. In General: To the extent permitted by law, each agency shall conduct a peer review on all

influential scientific information that the agency intends to disseminate. Peer reviewers shall

charged with reviewing scientific and technical matters, leaving policy determinations for the

agency. Reviewers shall be informed of applicable access, objectivity, reproducibility and oth

quality standards under the federal laws governing information access and quality. 

 2. Adequacy of Prior Peer Review

 be 

er 

: For information subject to this section of the Bulletin, 

agencies need not have further peer review conducted on information that has already been 

subjected to adequate peer review. In determining whether prior peer review is adequate, 

agencies shall give due consideration to the novelty and complexity of the science to be 

reviewed, the importance of the information to decision making, the extent of pr

and the expected benefits and costs of additional review. Principal findings, conclusions and

recommendations in official reports of the National Academy of Sciences are generally 

presumed to have been adequately peer reviewed. 

 3. Selection of Reviewers

ior peer reviews, 

: 

a. Expertise and Balance: Peer reviewers shall be selected based on expertise, expe

and skills, including specialists from multiple disciplines, as necessary. The group of reviewe

shall be sufficiently broad and diverse to fairly represent the relevant scientific and technical 

perspectives and fields of knowledge. Agencies shall consider requesting that the public, 

including scientific and professional societies, nominate potential reviewers. 

b. Conflicts

rience 

rs 

: The agency – or the entity selecting the peer reviewers – shall (i) ensure that 

g peer reviewers who are not 

ee 

those reviewers serving as federal employees (including special government employees) comply 

with applicable federal ethics requirements; (ii) in selectin

government employees, adopt or adapt the National Academy of Sciences policy for committ

selection with respect to evaluating the potential for conflicts (e.g., those arising from 

investments; agency, employer, and business affiliations; grants, contracts and consulting 

income). For scientific information relevant to specific regulations, the agency shall examine a 

reviewer’s financial ties to regulated entities (e.g., businesses), other stakeholders, and the

agency. 

c. Independence: Peer reviewers shall not have participated in development of the work 

product. Agencies are encouraged to rotate membership on standing panels across the pool of 

qualified reviewers. Research grants that were awarded to scientists based on investigatorinitiated, competitive, peer-reviewed proposals generally do not raise issues as to independence 

or conflicts. 
 Peer Review Handbook Page B-27 

 4. Choice of Peer Review Mechanism: The choice of a peer review mechanism (for exampl

letter reviews or ad hoc panels) for influential scientific information shall be based on the 

novelty and complexity of the information to be reviewed, the importance of the information to

decision making, the extent of prior peer review, and the expected benefits and costs of

as well as the factors regarding transparency described in II(5). 

 5. Transparency

e, 

 review, 

: The agency -- or entity managing the peer review -- shall instruct peer 

reviewers to prepare a report that describes the nature of their review and their findings and 

onclusions. The peer review report shall either (a) include a verbatim copy of each reviewer's 

represent the views of the group as 

a whole, including any disparate and dissenting views. The agency shall disclose the names of 

rulemaking 

c

comments (either with or without specific attributions) or (b) 

the reviewers and their organizational affiliations in the report. Reviewers shall be notified in 

advance regarding the extent of disclosure and attribution planned by the agency. The agency 

shall disseminate the final peer review report on the agency's website along with all materials 

related to the peer review (any charge statement, the peer review report, and any agency 

response). The peer review report shall be discussed in the preamble to any related 

and included in the administrative record for any related agency action. 

 6. Management of Peer Review Process and Reviewer Selection: The agency may commission 

independent entities to manage the peer review process, including the selection of peer 

reviewers, in accordance with this Bulletin. 

III. Additional Peer Review Requirements for Highly Influential Scientific Assessments. 

 1. Applicability: This section applies to influential scientific information that the agen

the Administrator determines to be a scientific asses

cy or 

sment that: 

(i) could have a potential impact of more than $500 million in any year, or 

(ii) is novel, controversial, or precedent-setting or has significant interagency interest. 

 2. In General: To the extent permitted by law, each agency shall conduct peer reviews on all 

information subject to this Section. The peer reviews shall satisfy the requirements of Section

of this Bulletin, as well as the additional requirements found in this Section. Principal find

conclusions and recommendations in official reports of the National Academy

 II 

ings, 

 of Sciences that 

fall under this Section are generally presumed not to require additional peer review. 

 3. Selection of Reviewers: 

a. Expertise and Balance: Peer reviewers shall be selected based on expertise, ex

and skills, including specialists from multiple disciplines, as necessary. The group of reviewers 

shall be sufficiently broad and diverse to fairly represent the relevant scientific and tech

perspectives and fields of knowledge. Agencies shall consider requesting that the public, 

including scientific and professional societies, nominate potential reviewers. 

b. Conflicts

perience 

nical 

: The agency – or the entity selecting the peer reviewers – shall (i) ensu

those revie

re that 

wers serving as federal employees (including special government employees) comply 

with ap

e 

 scientific assessments relevant to specific regulations, a reviewer’s financial ties to 

regulated entities (e.g., businesses), other stakeholders, and the agency shall be examined. 

plicable federal ethics requirements; (ii) in selecting peer reviewers who are not 

government employees, adopt or adapt the National Academy of Sciences’ policy for committe

selection with respect to evaluating the potential for conflicts (e.g., those arising from 

investments; agency, employer, and business affiliations; grants, contracts and consulting 

income). For
 Peer Review Handbook Page B-28 

c. Independence: In addition to the requirements of Section II (3)(c), which shall apply to

all reviews conducted under Section III, the agency -- or entity selecting the reviewers -- 

shall bar participation of scientists employed by the sponsoring agency unless the reviewer is 

employed only for the purpose of conducting the peer review (i.e., special government 

employees). The only exception to this bar would be the rare case where the agency determines, 

nd 

using the criteria developed by NAS for evaluating use of “employees of sponsors,” that a 

premier government scientist is (a) not in a position of management or policy responsibility a

(b) possesses essential expertise that cannot be obtained elsewhere. Furthermore, to be eligible 

for this exception, the scientist must be employed by a different agency of the Cabinet-level 

department than the agency that is disseminating the scientific information. The agency’s 

determination shall be documented in writing and approved, on a non-delegable basis, by the 

Secretary or Deputy Secretary of the department prior to the scientist’s appointment. 

d. Rotation: Agencies shall avoid repeated use of the same reviewer on multiple 

assessments unless his or her participation is essential and cannot be obtained elsewhere. 

 4. Information Access

 

: The agency -- or entity managing the peer review -- shall provide the 

reviewers with sufficient information -- including background information about key studies or 

models -- to enable them to understand the data, analytic procedures, and assumptions used to 

support the key findings or conclusions of the draft assessment. 

 5. Opportunity for Public Participation: Whenever feasible and appropriate, the agency shall 

ake the draft scientific assessment available to the public for comment at the same time it is 

it

ess as part of the peer review, 

the age omments that 

address

m

subm ted for peer review (or during the peer review process) and sponsor a public meeting 

where oral presentations on scientific issues can be made to the peer reviewers by interested 

members of the public. When employing a public comment proc

ncy shall, whenever practical, provide peer reviewers with access to public c

 significant scientific or technical issues. To ensure that public participation does not 

unduly delay agency activities, the agency shall clearly specify time limits for public 

participation throughout the peer review process. 

 6. Transparency: In addition to the requirements specified in II(5), which shall apply to all 

reviews conducted under Section III, the peer review report shall include the charge to the 

reviewers and a short paragraph on both the credentials and relevant experiences of each peer 

are a written response to the peer review report explaining (a) 

the age

th the 

 7. Ma

reviewer. The agency shall prep

ncy's agreement or disagreement with the views expressed in the report, (b) the actions 

the agency has undertaken or will undertake in response to the report, and (c) the reasons the 

agency believes those actions satisfy the key concerns stated in the report (if applicable). The

agency shall disseminate its response to the peer review report on the agency's website wi

related material specified in Section II(5). 

nagement of Peer Review Process and Reviewer Selection: The agency may commission 

independent entities to manage the peer review process, including the selection of peer 

reviewers, in accordance with this Bulletin. 
 Peer Review Handbook Page B-29 

IV. Alternative Procedures. 

As an alternative to complying with Sections II and III of this Bulletin, an agency may instead

(i) rely on the principal findings, conclusions and recommendations of a report produce

National Academy of Sciences; (ii) commission the National Academy of Sciences to peer 

review an agency’s draft scientific information; or (iii) employ an alternative scientific proc

or process, specifically approved by the Administrator in consultation with the Office of Science 

and Technology Policy (OSTP), that ensures the agency’s scientific information satisfies 

applicable information quality standards. The alternative procedure(s) may be applied to a 

designated report or group of reports. 

V. Peer Review Planning

: 

d by the 

edure 

. 

 1. Peer Review Agenda: Each agency shall post on its website, and update at least every six 

sted members of the public 

months, an agenda of peer review plans. The agenda shall describe all planned and ongoing 

influential scientific information subject to this Bulletin. The agency shall provide a link from 

the agenda to each document that has been made public pursuant to this Bulletin. Agencies are

encouraged to offer a listserve or similar mechanism to alert intere

when entries are added or updated. 

 2. Peer Review Plans: For each entry on the agenda the agency shall describe the peer revie

plan. Each peer review plan shall include: (i) a paragraph including the title, subject and pur

of the planned report, as well as an agency contact to whom inquiries may be directed to learn

the specifics of the plan; (ii) whether the dissemination is likely to be influential scientific 

information or a highly influential scientific assessment; (iii) the timing of the review (including 

deferrals); (iv) whether the review will be conducted through a panel or individual letters (or 

whether an alternative procedure will be employed); (v) whether there will be opportu

the public to comment on the work product to be peer review

w 

pose 

nities for 

ed, and if so, how and when these 

d 

opportunities will be provided; (vi) whether the agency will provide significant and relevant 

public comments to the peer reviewers before they conduct their review; (vii) the anticipate

number of reviewers (3 or fewer; 4-10; or more than 10); (viii) a succinct description of the 

primary disciplines or expertise needed in the review; (ix) whether reviewers will be selected by

the agency or by a designated outside organization; and (x) whether the public, including 

scientific or professional societies, will be asked to nominate potential peer reviewers. 

 3. Public Comment: Agencies shall establish a mechanism for allowing the public to comme

on the adequacy of the peer review plans. Agencies shall consider public comments on peer 

review plans. 

nt 

VI. Annual Reports. 

Each agency shall provide to OIRA, by December 15 of each year, a summary of the peer 

views conducted by the agency during the fiscal year. The report should include the following: 

) the number of peer reviews conducted subject to the Bulletin (i.e., for influential scientific 

formation and highly influential scientific assessments); 2) the number of times alternative 

rocedures were invoked; 3) the number of times waivers or deferrals were invoked (and in the 

re

1

in

p
 Peer Review Handbook Page B-30 

case of deferrals, the length of time elapsed between the deferral and the peer review); 4) any 

decision to appoint a reviewer pursuant to any exception to the applicable independence or 

conflict of interest standards of the Bulletin, including determinations by the Secretary pursuant 

to Section III(3)(c); 5) the number of peer review panels that were conducted in public and the 

number that allowed public comment; 6) the number of public comments provided on the 

agency’s peer review plans; and 7) the number of peer reviewers that the agency used that were 

recommended by professional societies. 

VII. Certification in the Administrative Record. 

If an agency relies on influential scientific information or a highly influential scientific 

 to support a regulatory action, it shall include in the 

inistrative record for that action a certification explaining how the agency has complied with 

assessment subject to this Bulletin

adm

the requirements of this Bulletin and the applicable information quality guidelines. Relevant 

materials shall be placed in the administrative record. 

VIII. Safeguards, Deferrals, and Waivers. 

 1. Privacy: To the extent information about a reviewer (name, credentials, affiliation) will be 

disclosed along with his/her comments or analysis, the agency shall comply with the 

requirements of the Privacy Act, 5 U.S.C. § 522a as amended, and OMB Circular A-130, 

Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a 

published System of Records Notice. 

 2. Confidentiality: Peer review shall be conducted in a manner that respects (i) confidential 

business information and (ii) intellectual property. 

 3. Deferral and Waiver: The agency head may waive or defer some or all of the peer review 

requirements of Sections II and III of this Bulletin where warranted by a compelling rationale. I

the agency head defers the peer review requirements prior to dissemination, peer review shall

conducted as soon as practicable. 

IX. Exemptions

f 

 be 

. 

Agencies need not have peer review conducted on information that is: 

 1. related to certain national security, foreign affairs, or negotiations involving international 

trade or treaties where compliance with this Bulletin would interfere with the need for secrecy

promptness; 

 or 

 2. disseminated in the course of an individual agency adjudication or permit proceeding 

proval, licensing, site-specific determination), unless the agency 

determines that peer review is practical and appropriate and that the influential dissemination is 

e-

(including a registration, ap

scientifically or technically novel or likely to have precedent-setting influence on future 

adjudications and/or permit proceedings; 

 3. a health or safety dissemination where the agency determines that the dissemination is tim

sensitive (e.g., findings based primarily on data from a recent clinical trial that was adequately

peer reviewed before the trial began); 
 Peer Review Handbook Page B-31 

 4. an agency regulatory impact analysis or regulatory flexibility analysis subject to interagency 

review under Executive Order 12866, except for underlying data and analytical models used

 5. routine statistical information released by federal statistical agencies (e.g., periodic 

demographic and economic statistics) and analyses of these data to compute standard indicators

and trends (e.g., unemployment and poverty rates); 

 6. accounting, budget, actuarial, and financial information, including that which is generated 

or used by agencies that focus on interest r

; 

ates, banking, currency, securities, commodities, 

tures, or taxes; or 

ne rules that materially alter entitlements, 

rants, user fees, or loan programs, or the rights and obligations of recipients thereof. 

fu

 7. information disseminated in connection with routi

g

X. Responsibilities of OIRA and OSTP. 

OIRA, in consultation with OSTP, shall be responsible for overseeing implementation of this

Bulletin. An interagency group, chaired by OSTP and OIRA, shall meet periodically to foster 

better understanding about peer review practices and to assess progress in implementing this 

Bulletin. 

XI. Effective Date and Existing Law.

The requirements of this Bulletin, with the exception of those in Section V (Peer Review 

Planning), apply to information disseminated on or after six months following publication of t

Bulletin, except that they do not apply

his 

 to information for which an agency has already provided a 

onsistent as possible with the practices and 

f this Bulletin. 

w

draft report and an associated charge to peer reviewers. Any existing peer review mechanisms 

mandated by law shall be employed in a manner as c

procedures laid out herein. The requirements in Section V apply to “highly influential scientific

assessments,” as designated in Section III of this Bulletin, within six months of publication of 

this Bulletin. The requirements in Section V apply to documents subject to Section II of this 

Bulletin one year after publication o

XII. Judicial Revie

This Bulletin is intended to improve the internal management of the executive branch, and is not 

intended to, and does not, create any right or benefit, substantive or procedural, enforceable at 

law or in equity, against the United States, its agencies or other entities, its officers or employees, 

or any other 
 Peer Review Handbook Page C-1 
 Peer Review Handbook Page C-2 

Appendix C 

Sound Science and Peer Review in 

Rulemaking Policy
 Peer Review Handbook Page C-3 

SOUND SCIENCE AND PEER REVIEW IN RULEMAKING

Several provisions of the December 2004, OMB Bulletin for peer review made it necessary for 

the Office of Policy, Economics, and Innovation (OPEI) to create conditional template language 

(Attachment A). This language should be used by rulewriters in the preamble of regulations that 

rely on influential scientific inform tific assessment, which are 

two categories of information defi

For proposed and final regulations that rely on influential scientific information or a highly 

influential scientific assessment, rulewriters sho use the template as a model to discuss peer

review in nicate

with rule s

included

OPEI also revised the Acti ssion of peer review

for influential scientific information or a highly influential scientific assessment (Attachment B). 

ation or a hig luential scien

ned in Section 2.2 of this handbook. 

hly inf

uld

the preamble where appropriate. In addition, peer review leaders should commu

writers and workgroup chairs to ensure that all appropriate peer review material i

in the docket, and that template language is included in the preamble. 

on Memorandum Framework to include a discu
 Peer Review Handbook Page C-4 

 ATTACHMENT A 

Λ

Peer Review (Conditional Template)

Read this first (but DO NOT insert it in your preamble): 

The OMB Final Information Quality Bulletin for Peer Review directs EPA to include a 

discussion of the peer review report and how the Agency complied with the provisions of the 

ulletin in the preamble of rulemakings that are supported by influential scientific information or 

atim

and

attribution

lanned by the Agency. You should ensure that the peer review report is placed in the docket to 

Use this template if your proposed or final rule is based on a work product containing influential 

scientific information or a highly influential scientific assessment. This language should appear 

in the Supplementary Information section of regulatory preambles under General 

Information. You may want to include the language under the heading: 

Did EPA conduct a peer review before issuing this notice? 

Π PROPOSED & FINAL ACTIONS: If you used a highly influential scientific 

assessment of influential scientific information to support this rulemaking, insert this into 

the preamble of your proposed or final rule, advanced notice of proposed rulemaking, or 

other substantive action: 

This regulatory action was supported by [influential scientific information or a highly 

influential scientific assessment]. Therefore, EPA conducted a peer review in accordance 

with OMB’s Final Information Quality Bulletin for Peer Review. [Insert a brief 

description of the peer review process along with any other relevant information.] The 

peer review report is located in the docket for today's action. According to the report, [insert 

a brief discussion of the peer review report. For more information about the peer 

review report, see the Peer Review Handbook.] 

B

highly influential scientific assessments. Peer review reports should either (a) include a verb

copy of each reviewer's comments (either with or without specific attributions) or (b) represent 

the views of the group as a whole, including any disparate and dissenting views. The Agency 

should disclose the names of the reviewers and their organizational affiliations in the report

should notify the reviewers in advance regarding the extent of the disclosure and

p

comply with the OMB Bulletin. 
 Peer Review Handbook Page C-5 

 ATTACHMENT B

FRAMEWORK FOR ACTION MEMORANDA 

PURPOSE OF T

This Framework provides guidance on the format and contents of Action Memoranda that 

accompany packages presented for signature. A draft of the memo also accompanies the Final 

gency Review (FAR) package. The Framework was developed to promote a consistent 

ecessary to accommodate the 

nique needs of each program while ensuring that critical information is provided to Agency 

xceed 5

HIS FRAMEWORK 

A

approach to these memoranda and to preserve the flexibility n

u

managers in a clear and concise manner. Generally, the Action Memoranda should not e

pages, and follows the format outlined in the Agency’s Correspondence Manual 

(http://epawww.epa.gov/rmpolicy/ads/manuals/1320.pdf). 

As indicated by the use of non-mandatory language such as “should,” “recommend,” and “may,”

this Framework provides recommendations and does not impose any legally binding 

requirements. Recommendations for each section of Action Memoranda are discussed below. 

Programs may include information on additional topics if they are relevant to a given action,

information quality issues, etc. 

e.g.,

ction Memoranda are considered internal, deliberative documents. At the bottom or top of 

Indicate if ther court imposed, 

statutory, or any one of a number of other deadlines, e.g., upcoming national meeting, 

Administrator’s priority, etc. 

Action Memorandum provides background information which 

briefly describes the rule, characterizes the environmental issue(s) or public health problem(s) 

being addressed, and summarizes the history of the action. It should explain why EPA is taking 

this

$

$

$

A

each page insert this reminder: “Internal Document Only; Do Not Cite, Quote or Release.” 

A description of the items to be included in an Action Memorandum follows. 

DEADLINE

e is a deadline of any sort and what that deadline is. It may be

OVERVIEW

The overview section of an

action, and where appropriate, cover the following points: 

Define any court or statutory deadline; 

Identify whether the action amends the Code of Federal Regulations, and if so, explain 

what kind of amendment (procedural, etc.); 

Identify other actions underway that will affect this particular program or sector; 
 Peer Review Handbook Page C-6 

$

requirements being addressed, and the goal intended by taking this action; 

$ Describe what th munity is affected

(performance standards, specific requirements); and 

entation of the action will be for States and regulated 

entities. 

mic 

iscuss the economic impacts on all affected entities including, where appropriate, possible 

icable, summarize the SBREFA Panel recommendations, and 

ow they were addressed in the rulemaking. 

t.

the role of both government entities and private sector stakeholders in the 

evelopment of the action. Summarize the concerns they have raised and what the Agency has 

r the action was developed under Tier 1, 2, or 3. If the action was Tier 1 or 2, 

ttach the summary memo from the Final Agency Review. Describe any outstanding issues from

Describe the specific environmental issue(s), public health problem(s) and/or statutory 

e action does, and specifically, how the regulated com

$ Describe how flexible the implem

IMPACTS

Summarize the costs and benefits, including a discussion of any non-monetized benefits, or 

contingent valuation and/or non-quantified benefits, of the action and the results of any econo

analyses. Explain the roles of cost-benefit and cost-effectiveness analyses in shaping the 

approach chosen. 

D

impacts on other Federal agencies, State, or Local governments, and Tribes. 

Summarize the results of the regulatory flexibility analysis and impacts on small entities (small 

businesses, small communities and small not-for-profit entities), including what was done to 

minimize those impacts. If appl

h

Describe the reporting and record-keeping burden and what the Agency has done to reduce i

STAKEHOLDER INVOLVEMENT

Briefly discuss

d

done to address them, or explain why we can’t. 

INTERNAL REVIEW

Identify whethe

a

Final Agency Review or other internal review. Identify Program Offices or Regions that 

participated in the development of the action, along with any outstanding issues and why they 

cannot be resolved or accommodated. Also, provide the basis for any decision made to not 

address an identified cross-media impact. 
 Peer Review Handbook Page C-7 

PEER REVIEW

If you did not use influential scientific information or a highly influential scientific assessment a

de ed by the Peer Review Handbook to support the action

s 

fin , include the following statement in 

There were no influential or highly influential products supporting this action as defined by 

ncy’s Peer Review Handbook. 

 of AAship] has followed the Agency's Peer Review Policy with respect to the 

underlying [influential scientific information or highly influential scientific assessment] 

ou may add any details you think are important, but you generally should not modify this 

nswer format, addressing the reader as “you,” using short sentences, etc. 

MB TRANSACTION

sues resulting from OMB's review under 

xecutive Order 12866, including any significant issues raised by other agencies participating in 

der Executive Order 12866. Explain any substantive changes made to 

e action as a result of recommendations from OMB or the other agencies. If OMB agreed that 

tion. 

escribe the type of response anticipated from the various audiences interested or impacted by 

the action. Identify both the involved stakeholders and the nature of their expected response. 

Characterize the likely reaction to the action by all interested parties including industry; 

environmental groups; Congress; State, Local, and Tribal governments; and OMB. Explain what 

the Agency has done to mitigate anticipated adverse reactions. 

the Action Memorandum: 

the Age

If you did use influential scientific information or a highly influential scientific assessment to

support the action, include the following statement: 

[Insert Name

supporting this action.

Y

compliance statement. If you used influential scientific information or a highly influential 

scientific assessment, but were not fully able to meet the Peer Review Policy, explain why. 

PLAIN LANGUAGE

Briefly describe which elements of plain language you used in the action, e.g., question and 

a

O

Identify the OMB determination and whether the action went to OMB for review under 

Executive Order 12866. Highlight significant is

E

the interagency review un

th

your action was not significant and, therefore, not subject to EO 12866 review, or if OMB 

waived review of your rule, please so indicate. If OMB was otherwise involved, e.g., was 

briefed or received a courtesy copy for informal review, describe the results of this interac

ANTICIPATED EXTERNAL REACTION 

D
 Peer Review Handbook Page C-8 

STAFF CONTACT

ECOMMENDATION

Ident ted to take (i.e., sign the rule or other document). 

to 

fficial), you should not include a 

oncurrence line at the bottom of the Action Memo or anything else that might cause the Action 

Mem

docum

any), not the Action Memo, which is only a pre-decisional briefing document. 

Give the name and phone number of the person to contact if there are questions about the 

package. 

R

ify the action the Administrator is expec

NOTE: Where an Action Memo accompanies another document (i.e., a rule or FR document) 

be signed by the Administrator (or other authorized o

c

o to be misinterpreted to be a Decision Memo (which it is not). The FR notice (or other 

ent such as an order), after being signed, will contain the Agency’s decision or action (if
 Peer Review Handbook Page C-9 
 Peer Review Handbook Page D-1 

Appendix D 

Examples Of Peer Review Charges
 Peer Review Handbook Page D-2 

Please Note

Appendix D - Examples Of Peer Review Charges

 – certain questions that are posed in charges can be responded to with a yes or 

o answer. Clearly, this is not the type of response the Agency generally wants, therefore, it is 

portant to phrase charge questio receive a fully satisfactory and

thoughtful response. Where a yes d, be sure to ask for a full

explanation supporting the yes or n

Ch shown

here cov looking

for the o ions

that need to be addressed and are therefore more complex in their nature. 

Other charges that have been used can be found on the SAB website at http://www.epa.gov/sab

n

im ns carefully to ensure that you

or no answer might be expecte

o answer.

arges can run the gamut from rather simplistic to highly complex. The examples

er a variety of types. Examples 1 through 4 have less complex questions and are

verall quality of the efforts. Examples 5 through 7 have numerous technical quest

 

and the SAP website at http://www.epa.gov/scipoly/sap/index.htm
 Peer Review Handbook Page D-3 

CHARGE EXAMPLE 1 - US EPA Science Advisory Board (SAB) Consultation on

Suspended and Bedded Sediments

Background: 

The Ecological Processes and Effects Committe

was asked by the Health and Ecological Criteria Division of the Office of Science and 

T

d

for consultant panel members to provide additional expertise to EPEC appeared in 

e Federal Register on July 30, 2003 (68 FR 44758-44760). 

ated

n streams in a wide range of concentrations – 

vels that might be perfectly normal in one water body would be indicative of impairment in 

s, and their

ffectiveness for dealing with all water quality impairments caused by SABS, especially as 

lity criteria for SABS or

ethodologies for deriving them on a site-specific basis was identified as one of the highest 

riorities for the water quality criteria program. As a result, the EPA Office of Water has 

oncluded that to better manage SABS in all types of water bodies and for all designated uses, 

tate and Tribal water quality managers need new and updated water quality criteria and 

formation for SABS. 

Charge to the panel: 

While many questions and much research remain, EPA seeks the opportunity for a consultation 

with the Science Advisory Board to gain advice and recommendations on the best potential 

approaches to developing water quality criteria for suspended and bedded sediments as will be

described in a draft Strategy for Developing Water Quality Criteria for Suspended and Bedded 

Sediments (SABS) to be prepared by the Office of Water. The Office of Water is also seeking 

recommendations on additional criteria development and approaches for uses of water bodies 

other than aquatic life, and it is also seeking advice on any potential criteria derivation 

methodology not included in the Strategy. 

e (EPEC) of the EPA Science Advisory Board 

echnology, Office of Water, to provide a consultation on potential approaches for a strategy for 

eveloping water-quality criteria for Suspended and Bedded Sediments (SABS). A request for 

nominations

th

In 1976, EPA issued a water quality criteria recommendation under the Clean Water Act for 

solids and turbidity. For a variety of reasons, the States seldom, if ever, use this criterion. It is 

questionable whether this criterion would achieve intended protection for all different design

uses for water bodies. SABS occurs naturally i

le

another. 

Although most States currently have water quality criteria that can be applied to manage SABS,

these are typically based on turbidity, suspended solids or settle-able solid

e

benchmarks for aquatic life protection based on natural levels, is questionable. In recent 

consultation with State representatives, the need for new water qua

m

p

c

S

in
 Peer Review Handbook Page D-4 

CHARGE EXAMPLE 2 - Economics - Benefits Transfer from Adults to Children

Background

PA established the Office of Children’s Health Protection (OCHP) in 1997 to support its efforts 

lt 

 the 

fforts, this paper discusses the 

enefits transfer technique as it applies to estimating values for children’s health. The first 

e 

to transfer adult-oriented values to estimate child-related values (Neumann and 

reenwood 1999). However, the results of this paper suggest that transfer of these value 

the policy 

t 

is support the proposition that the value of children’s health effects 

should be estimated differently than adult health effects? Why or why not? 

) Identify issues for further research that would improve our ability to estimate values for 

E

to increase the protection of children’s health throughout its programs. One of the many difficu

issues the Office is addressing is the appropriate treatment of children’s health effects in

economic analyses performed by the Agency. Policy analysis efforts at the Agency often rely on 

the benefits transfer technique, and very few of the Agency’s benefit transfers have explicitly 

addressed children’s health issues. In addition, no accepted systematic process for conducting 

benefits transfer currently exists. To assist the Agency in its e

b

section provides some general background on the technique, and its application to estimate 

health-related values. The second section raises important general issues to consider when 

conducting a benefits transfer for children’s health values. The last section discusses the 

implications of using the benefits transfer method to estimate values for children’s health. Th

scarcity and state of existing child-oriented health valuation literature suggests that it may be 

necessary 

G

estimates to children at best provides estimates for a scoping analysis. In cases where these 

scoping exercises indicate that children's health values may be a crucial component in 

analysis, primary research should be undertaken to estimate child-related values. 

a) Assess the appropriateness of transferring health benefit values estimated for adult 

populations to children. Describe the specific issues that arise in these transfers. Which 

variables or situations improve or decrease the appropriateness of transferring benefi

values from adults to children? 

b) Does the analys

c

children’s health effects. 
 Peer Review Handbook Page D-5 

CHARGE EXAMPLE 3 - Economics - Study of Municipal Government Costs and 

Financial Impacts from Environmental Regulations

ackground

n

ent 

es at a

al

t to recognize

at additional environmental requirements will have to compete with other infrastructure needs 

welfa

Give ic services, this study examines what additional 

vestments the new environmental legislation will require Local governments to undertake, and 

ts

and r re considered

uring the regulatory process in those situations permitted by environmental statutes. The 

a com ents, and to determine whether they will place a significant 

burden on the fiscal conditions of Local governments, and require them to significantly increase 

existing charges for improved environmental services. 

Please find attached a copy of the draft study and appendices for your review. This version of 

the report reflects Agency comments received on an earlier draft. We expect that this version of 

the report, with some additional minor modifications, will be the final version. Your comments 

will be useful in preparing the final version of the report and discussing the findings of the 

reports with the public. 

To assist in your review of the report, we ask that you pay particular attention to the following 

questions: 

B

Municipalities play a major role in supplying environmental services. Local governments have 

taken responsibility for providing drinking water, sewage treatment, and waste disposal in a 

majority of communities. Over the past fifteen to twenty years, most of the mandates found in 

the Federal environmental legislation enacted in the early 1970s have been met. The increase i

the number of people served and improvements in the quality of local environmental services 

have been considerable, as has the investment in public infrastructure to meet these laws. 

Recent revisions to the environmental legislation have established a broader and more string

set of standards to be met by suppliers of environmental services. As a result, many Local 

governments are now faced with having to maintain all or some part of their public servic

higher level of performance. To meet these new standards will require additional investments in

capital, and increases in rates charged to customers for environmental services. 

Improvements in environmental services are but one of several demands being made of loc

public infrastructure. Studies prepared on public infrastructure needs and the availability of

funds to meet these needs indicate that there will be an excess demand for money to rebuild and 

improve upon the existing stock of public infrastructure. Therefore, it is importan

th

(e.g., highways, bridges), as well as other public services (e.g., police, education, health and 

re programs) provided at the local level. 

n the increasing demand for publ

in

the likelihood that they will face difficulties raising the necessary funds through capital marke

evenues from customers. The economic impacts of individual EPA actions a

d

unique feature of this study is its attempt to estimate the cumulative costs and impacts of meeting 

bined set of EPA requirem
 Peer Review Handbook Page D-6 

a) Do State and Local governments and financial markets consider household costs 

(measured as a percent of household income) and selected municipal financial 

mation (debt service to general revenues or taxable property values) when 

evaluating the ability of enterprise systems and municipalities to issue bonds or obtain 

ed 

from the analysis? 

infor

loans? 

b) Having selected a series of financial indicators, are the criteria used in the analysis 

acceptable? 

c) Do the results support our conclusions? What additional conclusions can be reach

d) What modifications would you suggest be made to the recommendation section? 
 Peer Review Handbook Page D-7 

CHARGE EXAMPLE 4 - External Peer Review of the Design of the Targeted National 

Sewa

Backgro

The United States Environmental Protection Agency (EPA), Office of Water is charged with 

rotecting public health and the environment from adverse exposure to chemicals and microbial 

agent

sediments. In support of this mission the Office of Water/Office of Science and Technology 

ST) develops human health and environmental standards, health criteria, health advisories, and 

techn

eer review is an important component of the scientific process. It provides a focused, objective 

evalu

ther document submitted for review. The criticisms, suggestions and new ideas provided by 

e peer reviewers ensure objectivity, stimulate creative thought, strengthen the reviewed 

ocument, and confer scientific credibility on the product. Comprehensive, objective peer 

view leads to good science and product acceptance within the scientific community and is 

quired for the technical document that support Agency regulations and guidelines. 

Targeted National Survey of Pollutants in Sewage Sludge

As EPA described in the December 2003 Federal Register notice (68 FR 75531), EPA 

plans to conduct a targeted survey of pollutants (select pollutants) in sewage sludge. EPA has 

designed a targeted survey of selected chemical pollutants for which adequate data are available, 

or will be available, for conducting an exposure and hazard assessment concerning their presence 

in treated sewage sludge. Microbial pollutants (pathogens and indicator organisms) in sewage 

sludge are also included in the survey design. 

A survey will provide feedback for updating the science and technology of sewage sludge 

applied to land, disposed of in a surface disposal unit, or incinerated. The new concentration 

data would be used to assess human and ecological risks of identified, unregulated pollutants 

found in sewage sludge and identify pollutants for potential regulation. In addition, the data on 

microbial constituents in sewage sludge will help in determining the effectiveness of pathogen 

reduction and elimination processes that are specified in the 40 CFR Part 503 Standards. 

EPA is committed in FY 2005 to starting a limited analytical survey of pollutants found in 

sewage sludge. EPA expects this survey to address the pollutants identified by the exposure and 

hazard screening assessment as presenting a potential hazard, as identified in the current section 

405(d)(2)(C) biennial review (68 FR 75531). The Agency has evaluated the extent to which 

sampling and analytical methodologies will allow expansion of the survey scope within available 

resources to include additional pollutants. 

Questions for Peer Review Charge 

ge Sludge Survey for Pollutants in Sewage Sludge 

und Information 

p

s in water media, such as ambient and drinking waters, wastewater/sewage sludge, and 

(O

ical guidance documents for water and water-related media. 

P

ation of a research proposal, publication, risk assessment, health advisory, guidance or

o

th

d

re

re
 Peer Review Handbook Page D-8 

a) The sample design proposes excluding facilities with certain treatment types (i.e., 

less than secondary treatment, partial treatment, and ponds), remotely located facilities 

uous states), and non-publicly owned facilities (i.e., Tribal and 

private). Do these exclusions appear to be appropriate? If you disagree with EPA’s 

d 

s 

tics as other POTWs for 

purposes of biosolids sampling. Do you agree with the inclusion of the biosolids 

 they 

 of 

 are highly correlated. Do you 

agree that this is a reasonable assumption? If you disagree, your response should identify 

ting mass 

s to the rationale for selecting the binomial 

distribution. Your response also should recommend another approach, describe this 

e 

(i.e., outside the contig

proposed exclusions, your response should explain which groups should be included an

why (e.g., what information will be lost if they are excluded). 

b) The sample design proposes including facilities that the Clean Watershed Need

Survey (CWNS) identifies as biosolids handling facilities, because preliminary review 

indicates that they have the same operations and characteris

handling facilities? If you disagree, your response should explain how their 

characteristics (e.g., type of treatment) would differ from other POTWs. 

c) The sample design proposes including facilities with no discharge, because

also may produce biosolids as a result of treatment. Do you agree with the inclusion

no discharge facilities? If you disagree, your response should explain how their 

characteristics would differ from other POTWs. 

d) The sample design assumes that flow and mass

possible data sources for mass data, or describe alternative approaches to estima

values based upon available data. 

e) The sample design proposes facility selection based upon the binomial 

distribution. Do you agree that this is the appropriate approach? If you disagree, your 

response should describe your objection

approach (unless it was included as an alternative in the sample design), and provid

rationale for its selection. 
 Peer Review Handbook Page D-9 

CHARGE EXAMPLE 5 - External Peer Review of the Draft National Strategy for the 

BACK

EPA’s 

life, hu larity). EPA is initiating a 

ational Strategy for the Development of Water Quality Criteria for Suspended and Bedded 

ime

sedime

repeate nts 

resultin ic 

levels. 

PA’s Strategy for Water Quality Standards and Criteria, dated August 2003, is a document 

waters.

greates

sedime ominant polluting factors in 305(b) 

ater Quality Reports in rivers and streams, lakes and reservoirs, ponds, wetlands and ocean 

. 

were no

prioriti

suspend hese priorities were developed based on extensive 

stening sessions and discussions with States and other partners. 

To asse

the spe ut 

many S

Inform f 

some type were identified in 32 of 53 States, Territories and Washington, DC. But many States 

ave not adopted such sediment criteria into their water quality standards. As a result, SABS 

roblems are identified but perhaps underestimated, and few controls (water quality standards, 

ermit limits, TMDLs) are in place to reduce the overall impact of SABS. 

lso, as part of the current effort to develop national SABS criteria, the EPA Science Advisory 

Board met on October 2, 2003 to discuss the strategy, including various approaches for 

establishing criteria. The Science Advisory Board recommended that several approaches to 

SABS criteria development should be synthesized in an overall approach that can identify 

impaired SABS conditions as compared to appropriate reference conditions. These 

recommendations are presented in the draft strategy document. 

The methodologies presented in this draft strategy will be used by the States, Tribes, and 

Territories to better manage SABS in water bodies across the country. This strategy presents 

Development of Water Quality Criteria for Suspended and Bedded Sediments (SABS) 

GROUND INFORMATION:

Office of Science and Technology (OST) has water quality criteria programs for aquatic 

man health, and for nutrients (total N, P, DO, chlorophyll, and c

N

Sed nts (SABS). SABS occur naturally in all types of waterbodies. In appropriate amounts, 

nt is essential to aquatic ecosystems. However, imbalanced SABS conditions have 

dly ranked high as a major cause of waterbody impairment. An imbalance of sedime

g from human activities can impact ecological integrity at several scales and troph

E

which sets priorities to strengthen the foundation for protecting and restoring the Nation’s 

 Imbalance in loading of SABS to the aquatic systems is now considered one of the 

t causes of water quality impairment in the Nation. Turbidity, suspended solids, 

nt, and siltation have been consistently listed as d

W

shoreline waters. In 1998, for example, approximately 40% of assessed river miles in the U.S

t meeting their water quality goals due to excessive sediments. One of the top ten 

es listed in the document is producing and implementing a strategy for the development of 

ed and bedded sediment criteria. T

li

 

ss the effects of this growing environmental stressor, a number of States have identified

cific measurements and levels at which sediment impairment occurs in their waters, b

tates have not adopted such SABS criteria into their State water quality standards. 

ation from an EPA survey conducted in 2001 indicates that numeric sediment criteria o

h

p

p

A
 Peer Review Handbook Page D-10 

EPA’s blueprint for achieving the goals of developing sediment criteria methodologies and th

implementation of these criteria into water quality standards. 

e 

t of SABS Criteria 

re 

tates, under pressure to develop and issue total maximum daily loads (TMDLs) for SABS 

 and 

, a 

 actual 

r all waterbody types and all 

ons. The responsibility to develop and adopt SABS criteria, based on the 

) A list of resources from which States and Tribes can find assistance for deriving 

National Strategy for the Developmen

The Strategy for the development of SABS criteria describes the approach that EPA is planning 

to take for developing scientific information relating to SABS and working with States to 

encourage adoption of SABS criteria into State water quality standards pursuant to Section 

303(c) of the Clean Water Act. The draft strategy for developing water quality criteria for SABS

is modeled on the approach used by EPA to develop water quality criteria for nutrients, whe

appropriate. However, the strategy does not establish dates by which States must develop water 

quality standards for SABS. 

S

impaired water bodies, are moving forward on their own to develop new and improved SABS

criteria. The draft strategy highlights some of the State programs and looks for approaches

methods that may be useful, either directly, or with adaptation, to the entire nation. Because

current States’ efforts may serve as examples for development of numeric SABS criteria

summary of States’ programs are provided. In order to describe a possible SABS criteria 

development process, an Oregon case study was prepared for a hypothetical aquatic system, 

following a watercourse from the watershed to the receiving water (Section III.F of the draft 

strategy). The case study illustrates sediment criteria development through theoretical and

examples. 

EPA does not anticipate developing ecoregional SABS criteria fo

geographic regi

information and methodologies presented in the draft strategy, will rest with States. EPA’s draft 

national strategy includes the following major elements: 

a) Outline of actions to be taken by EPA, States, Territories, and Tribes to develop 

criteria, adopt criteria into water quality standards, and manage for SABS. 

b) Identification of appropriate indicators of water resource impairment due to 

imbalances of SABS. 

c) A process to stratify waterbodies by waterbody type, region, and designated uses 

for the development of SABS criteria. 

d) A series of approaches and technical tools for deriving SABS criteria specific to a 

region, waterbody-type, and designated use. 

e

SABS criteria. 
 Peer Review Handbook Page D-11 

f) A case study may be developed to illustrate SABS criteria development, 

addressing both the theoretical basis and an example using a synthesis of 

approaches. 

ollowing peer review and incorporation of peer review comments, the U.S. Environmental 

r 

ectivity, stimulate creative thought, strengthen the reviewed 

ocument, and confer scientific credibility on the product. Comprehensive, objective peer 

ng the 

sing 

Is a regional and watershed approach, as described in the strategy, likely to 

accomplish the desired goal of addressing the sediment imbalance problems in the 

ribes, given their resources? 

loping SABS 

criteria into water quality standards pursuant to Section 303(c) and 304(a) of the CWA? 

e) points and indicators described in the strategy reflective of 

the current or best available science? 

f) strategy adequate to help States, Tribes, 

and territories develop and adopt SABS criteria? 

g) a development through the 

theoretical and actual examples presented? 

 

F

Protection Agency (EPA) will publish a draft National Strategy for the Development of Water 

Quality Criteria for Suspended and Bedded Sediment (SABS) and will ask the scientific 

community and the public for its views on the scientific direction of its SABS strategy. Pee

review is an important component of the scientific process. It provides a focused, objective 

evaluation of a research proposal, publication, risk assessment, health advisory, guidance or 

other document submitted for review. The criticisms, suggestions and new ideas provided by 

the peer reviewers ensure obj

d

review leads to good science and product acceptance within the scientific community and is 

required for the technical document that support Agency regulations and guidelines. 

 

The Peer Reviewers charge is as follows: 

 

a) Does EPA’s draft national strategy adequately describe methods for assessi

expected SABS condition of a waterbody, for developing SABS criteria and for asses

attainment of criteria by States. 

b) 

nation’s surface waters? If not, what alternatives do you suggest? 

c) Does the draft strategy present a rational, practical and scientifically responsible 

approach that can be implemented by States and T

d) Is the state-of-the-science sufficient to write technical methods manuals that will 

allow States, Tribes and territories to determine the best method(s) for deve

Are the assessment end

Are the six major elements of the draft 

Does the case study illustrate sediment criteri
 Peer Review Handbook Page D-12 

CHARGE EXAMPLE 6 - CASAC Review of the Agency’s National Ambient Air 

Monitoring Strategy

e 

trategy 

nd 

across 

f a higher 

ribal (SLT) agencies 

ams and allowing SLT flexibility. The transition to the 

 creates a need for training that addresses new methods, information transfer 

areas w

will be e.g., satellite broadcasts and videos; hands-on sessions; 

uidance documents; vendor training of instrumentation; web-based training; and workshops. 

m 

assessm

he implementation plan incorporates action-oriented components of the Strategy, e.g., 

of the N r moving 

rward and will also alleviate some of the unnecessary burdens faced by monitoring agencies 

regulat

minimu

ecifications; periodic network assessments; and new quality assurance procedures. Written 

Subcom onitoring Technology Information Center 

MTIC) Web site. The Final Draft NAAMS document, which was updated following the July 

3 m eting o

http://w llstrat.pdf

Background. 

The draft National Ambient Air Monitoring Strategy (NAAMS or Strategy) was revised after th

Agency’s prior consultation with the former CASAC National Ambient Air Monitoring S

(NAAMS) Subcommittee in July 2003. This revision incorporates EPA’s responses to that 

Subcommittee’s recommendations. The primary recommendations from the NAAMS 

Subcommittee included a request for an implementation plan, and added emphasis on rural- a

ecosystem-oriented monitoring, support for the National Core Monitoring Network 

(NCore) Level 1 program, and training and quality assurance to enhance data consistency 

the Nation. 

The Strategy foresees moving resources from programs of decreasing value to those o

value which respects the partnership across EPA, State, Local, and T

retaining stability for the monitoring progr

NCore network

technologies, and an effective quality assurance program. There are programmatic and technical 

here some type of training or a transfer of information is required. These training needs 

 offered by various mechanisms, 

g

Broadening the Agency’s outreach to the health effects, atmospheric scientists and ecosyste

ent communities are included. 

T

regulation revisions, training, funding, and outreach approaches to facilitate the implementation 

Core program. The revised monitoring regulations will provide a legal basis fo

fo

and enhance the ability to introduce new technologies into our networks. Additionally, the 

ions introduce the NCore system of: multiple-tiered monitoring stations; adjustment of 

m requirements for specific pollutant measurements; new methods performance 

sp

Meeting Materials. OAQPS has posted written review and background materials for this 

mittee meeting on EPA’s Ambient M

(A

200 e f the former CASAC NAAMS Subcommittee, is at 

ww.epa.gov/ttn/amtic/files/ambient/monitorstrat/a . Additional background 

materials for this meeting are found on the “CASAC File Area” page of the AMTIC Web site at 

http://www.epa.gov/ttn/amtic/casacinf.html. Furthermore, it is our understanding that th

ffice will post a copy of the final agenda and

e SAB 

Staff O charge to the Subcommittee for this 

advisory meeting on the SAB Web site at http://www.epa.gov/sab (under “Meeting Agendas”) 

and the Subcommittee’s page at http://www.epa.gov/sab/panels/casac_aamm_subcom.html, 

respectively, in advance of the Subcommittee’s meeting. 
 Peer Review Handbook Page D-13 

Charge to the Panel: 

For this advisory meeting, OAQPS is requesting that the CASAC AAMM Subcommittee provide 

xpert advice and recommendations on the following charge questions, which focus on key 

 the 

ents (CO, 

a for 

Questions: Given limited budgetary resources, does this represent both an 

AMS 

all 

ation. 

sing 

s 

ut 

anorks 

etwork (STN) speciation 

) 

ean that some redundant monitors in 

e

implementation issues: 

a) The CASAC has expressed its support for the Agency’s proposal to redesign

routine PM monitoring network to support PM precursor gas measurem

SO2, NO/NOy, NH3, HNO3) at NCore Level II multiple-pollutant sites, and for 

air quality management decisions and to obtain relevant exposure dat

research programs. 

appropriate and adequate balance, as reflected by the relative resource allocations

provided in Section 11, “Draft Implementation Plan,” of the Final Draft NA

Document? In addition, are the relative adjustments in the training and guidance 

approaches proposed in the draft implementation plan consistent with the over

objectives of the Strategy? 

b) The implementation plan proposes a series of communication actions to advance 

the NCore Level 2 network, in order to more directly support long-term health 

effects research and provide better support to ecosystem assessments through an

increased level of coordin

Questions: Does the Subcommittee have additional suggestions for addres

this need for integration and communication to the broader community of “users,”

including scientific researchers (i.e., human health, atmospheric, ecological) and 

State, Local and Tribal (SLT) Agency representatives? More specifically, what i

the most effective manner for EPA both to reach-out to this broad user 

community and, where appropriate, to incorporate their feedback and design inp

on such issues as monitoring site locations and parameters? 

c) One of the remaining technical issues relates to harmonizing rural- and urb

based 2.5 chemical speciation networks such that both categories of netw

utilize consistent sampling and analysis protocols. For example, EPA is 

considering converting all of the Speciation Trends N

sites to Interagency Monitoring of Protected Visual Environments (IMPROVE

samplers and IMPROVE laboratory and sample handling protocols. 

Question: What are strengths and weaknesses of this approach? 

d) As EPA implements the National Ambient Air Monitoring Strategy to address 

multiple monitoring objectives, it will be looking to spatially optimize the 

ambient monitoring networks. This may m
 Peer Review Handbook Page D-14 

adjacent, but separate, geopolitical areas (e.g., neighboring counties) are 

ted” from a given network. Although technically sound, these divestments 

could result in data gaps which might, in turn, adversely impact regulatory 

ch 

es are scientifically justifiable; hence, the rationale for initiating 

discussion of these issues with the CASAC. 

cess? 

“dives

decision-making. The Agency is willing to adopt alternative approaches for 

assessing regulatory issues such as non-attainment designations, so long as su

approach

Question: Is it scientifically acceptable to generate air quality surfaces through 

modeled observations and/or integrated predictive/observational fields that would

be of appropriate uncertainty for use in the regulatory decision-making pro
 Peer Review Handbook Page D-15 

CHARGE EX

Assessment (3

Background:

The Office of S media, 

Multipathway, Model and this request was accepted

y the Board, becoming SAB Project #03-13. A “Widecast” Federal Register notice requesting 

nominations fo

In brief, the de

Identification Rule. EPA plans to use the modeling system to help inform managers on a variety 

of decisions in the waste program, such as setting concentration-based exit criteria for wastes in 

the hazardous waste management regulations, or deciding whether technology-based standards 

are protective of human health and the environment. 

In December 1995, EPA's Office of Solid Waste proposed to amend existing regulations for 

disposal of listed hazardous wastes under the Resource Conservation and Recovery Act(RCRA). 

This Hazardous Waste Identification Rule (HWIR) was designed to establish constituent-specific 

exit levels for low risk solid wastes that are currently captured in the RCRA subtitle C hazardous 

waste system. The EPA's Science Advisory Board (SAB) reviewed the proposed HWIR 

methodology for calculating exit concentrations, concluding that the methodology ``lacks the 

scientific defensibility for its intended regulatory use,'' the SAB also made recommendations 

that, when implemented, should provide an adequate scientific basis for establishing a risk-based 

methodology applicable at the national level for the waste program. 

In response, the Office of Solid Waste (OSW) collaborated with the Office of Research and 

Development (ORD) to develop and document a sound science foundation, supporting data for 

an assessment, and related software technology for an integrated, multimedia modeling system

(entitled 3MRA) following the recommendations of the SAB and other reviewers. The 

Multimedia, Multipathway, and Multireceptor Risk Assessment (3MRA) modeling system

represents a collection of science-based models and databases that have been integrated into a 

software infrastructure that is based on the FRAMES (Framework for Risk Analysis in 

Multimedia Environmental Systems) concept, which provides a computer-based environment for 

linking environmental models and databases and managing the large amounts of information 

within the system, including the visualization of outputs. This integrated multimedia modeling 

system provides national-level estimates of human and ecological risks resulting from longterm(chronic) chemical release from land-based waste management units. Over 45 experts 

participated in the peer review process of the underlying science within the 3MRA modeling 

system. 

Charge to the Panel: 

AMPLE 7 - The Multimedia, Multipathway, and Multireceptor Risk

MRA) Model

olid Waste (OSW) asked the Science Advisory Board to review the Multi

and Multireceptor Risk Assessment (3MRA)

b

r the Panel was published on April 11, 2003 (68 FR 17797).

velopment of the 3MRA methodology has a history in the Hazardous Waste 
 Peer Review Handbook Page D-16 

The SAB Staff Office Director and OSW staff concurred that the panel should focus upon

following four areas: assessment m

 the 

ethodology, 3MRA modeling system, modeling system 

valuation, and modeling system documentation. The panel will respond to the following charge 

) Charge Question 2a: Does the 3MRA modeling system provide a tool for performing 

vides 

he impacts on potential chemical 

exemption levels by allowing varying measures of protection based on the 

n? 

 the 

ity 

g 

ive 

e

questions: 

a) Charge Question 1: While EPA had the assessment methodology peer reviewed prior to 

the development of the 3MRA modeling system, does the SAB have any 

additional comments about the methodology as implemented? 

b

national risk assessments that facilitates consistent use of the science and pro

a mechanism for reproducing results? 

 

c) Charge Question 2b: Does the 3MRA modeling system provide Decision Makers 

sufficient flexibility for understanding t

number of receptors and/or number of sites protected, types of human and 

ecological receptors, and distance? 

d) Charge Question 2c: Does the 3MRA modeling system provide appropriate information 

for setting national risk-based regulations for the waste program? 

e) Charge Question 3a: Is the software development and verification testing approach 

implemented for the 3MRA modeling system sufficient to ensure confidence that 

the modeling results reflect the modeling system desig

f) Charge Question 3b: Given the thorough evaluations that EPA has implemented using

available data resources and technologies, while also recognizing the real world 

limitations that apply to validating the 3MRA modeling system, have we 

reasonably demonstrated through methodology design, peer review, qual

control, sensitivity analyses, and model comparison, that the 3MRA modelin

system will produce scientifically sound results of high utility and acceptance 

with respect to multimedia regulatory applications? 

g) Charge Question 4: Has EPA made substantive progress, relative to 1995, in designing

and preparing documentation for the 3MRA modeling system? Does the SAB 

have additional suggestions for improving the presentation of the comprehens

set of materials related to this modeling system? 

 
 Peer Review Handbook Page E-1 

Guidance on Requesting a Review by the 

S

Appendix E 

cience Advisory Board (SAB)
 Peer Review Handbook Page E-2 
 Peer Review Handbook Page E-3 

APPENDIX E – GUIDANCE ON REQUESTING A 

REVIEW BY THE SCIENCE ADVISORY BOARD (SAB)

Introduction

The Science Advisory Board Staff Office (SAB SO) solicits nominations for project proposals to 

be reviewed by three of the Agency’s independent scientific advisory committees the Science 

Advisory Board (SAB); the Clean Air Scientific Advisory Committee (CASAC); and the 

Advisory Council on Clean Air Compliance Analysis (Council). This Appendix provides 

guidance to EPA Program Offices and Regions regarding the project proposal nomination and 

selection process. More detailed information about the functions and advisory process for the 

SAB can be found in the Implementation Plan for the New Structural Organization of the EPA 

Science Advisory Board (EPA-SAB-04-002) available on the SAB website at 

http://www.epa.gov/science1/pdf/sab04002.pdf. 

Background

A key priority for the Administrator is to base Agency actions on sound scientific data, analyses, 

and interpretations, as well as independent peer review. The Administrator issued the Agency's

Peer Review Policy to increase the quality of the technical foundations upon which EPA's 

regulatory structures are built. The SAB, CASAC, and the Council provide mechanisms for 

EPA to receive independent scientific peer review and advice regarding the development and use 

of science at EPA. The following information is intended to help programs and Regions 

determine which scientific advisory projects to submit to the SAB, CASAC and Council. 

The SAB, CASAC, and Council provide several kinds of advisory functions including the 

following kinds of activities: 

• Consultation – Provides non-consensus, oral advice on a technical issue before EPA 

begins substantive work on that issue. 

• Advisory – Provides written advice on EPA's technical works-in-progress. 

• Peer Review – Conducts a review of EPA's final draft technical reports (e.g., 

guidelines, assessments, research strategies) or work products (e.g., analytical methods, 

models, databases). 

• Commentary – Provides forward-looking advice on an important technical or 

emerging issue in the form of a short communication (SAB function). 

 

• Original Study – Conducts original work on an emerging or overarching topic of 

importance to EPA (SAB function). 
 Peer Review Handbook Page E-4 

• Other Activities – Receives information briefings from EPA and conducts scientif

workshops on specific technical subject matters. 

ic

ecause resources are always limiting, the SAB Staff Office uses several criteria for selecting 

ls proposed by the Agency. Advisory project proposals best suited for 

onsideration by the SAB, CASAC, and Council are those that meet several of the following 

the

s)

s

ce-Driven Criteria

S Requires the commitment of substantial resources to scientific or technological 

undaries, within or outside EPA (includes 

international boundaries) 

In addi posals

for a sp

projects. Table 1 depicts the processes for identifying, nominating, and selecting Agency 

ominations, emerging issues for strategic advice, and de novo studies. The SAB, CASAC and 

Counci linked

to spec

Proces

B

project proposa

c

criteria: 

• General Criterion 

S Provides an opportunity to make a difference in the science that supports

Agency's mission 

 • Client-Related Criteria

 S Supports major regulatory or risk management initiatives 

 S Serves leadership interests (e.g. the Administrator or Congres

 S Supports EPA strategic prioritie

• Scien

S Involves scientific approaches that are new to the Agency 

 S Addresses areas of substantial uncertainties 

• Problem-Driven Criteria

 S Involves major environmental risks 

 S Relates to emerging environmental issues 

 S Exhibits a long-term outlook 

• Organizational Criteria

 S Serves as a model for future Agency methods 

development 

S Transcends organizational bo

S Strengthens the Agency's basic capabilities. 

tion, the SAB Staff Office considers the overall mix of the nominated project pro

ecific fiscal year, as well as the time and available resources needed to take on the 

n

l employ similar processes, but projects for the latter two advisory committees are

ific statutory mandates. 

s for Submitting Nominations

activity, or issue to the SAB, for a peer review, 

advisory, or consultation, is requested to complete the two-step process, described below. 

Any Office desiring to take a product,
 Peer Revi Han ew dbook Page E-5 

TABLE 1 – PRO

STEPS TO SAB 

PROJECT

SELECTIO

C S FOR S T ES ELEC ING SCIEN V CE AD ISORY BOARD (SAB) P

TYPE 

 OF N

PROJECT

I IFICATI R

O

IO N E 

C R

DENT ON DISC 

MINA

FO

A

RA FO

ND N

USS

TIO

ION 

N

PR RITIZATION A D SELECTION

S

DEFININ

OPE AND 

G TH

CHA GE

AGENCY RE ch

sc

ffi

oj ul

n na

vi

ic

on

t 

tra

na

inistra

nually

 neede

ie in

: 

ci ou

P gi

A

Select on: - Stat uirem- Age 

hi jo

cu

A fice 

- the requ g EPA Program nal o

QUESTS A

diOprbere

ieved

ussio

ces t

ects 

efit fr

ew 

 throug

ns withi

o determ

which wo

om exter

h 

n EPA 

ine 

d 

l 

StafnomAssiAdmor RAdm- An- As

f Off

inati

stan

inis

egio

e rec

s fro

tors 

l 

tors 

d ba

eive

m E

(AA

(RA

sis 

s 

PA 

s) 

s) 

Achwith- S - E- S

ved t

ence 

A Pro

B 

ion b

utory

ncy 

hrou

Poli

gra

ased

 Req

needs

gh jo

cy C

m/Re

t dis

ncil 

onal 

ent 

cuss

offic

ions

es 

Acdis- S

eved thro

ssions w

B Staff Of

estin

/Regio

ugh 

ith: 

int 

ffice

IDENTIISSUESADVICSTUDI

FYING E

FOR SA

E AND D

ES

ch th

te ns : 

PA ra gio

ffi

ientific C unit

ublic 

ience Ad B

AB Staff O

 Annual M

 Committe

tings 

d Meetings 

Meetings with Public Science Policy CouncilEPA Program/Regional Offices Scientific Workshops 

Achie gh joinwith: - Sci y Council - EPA /Regional- SAB To identif ects for fuconsiderat- Strategic Advice by SAB- De novo project Selection based on: - Impact on Agency’s mi- Innovative nature of project - Agency needs 

Achieved ugh jo- Design AB Committ- EPA Program/ Regional Office 

MERGING

B STRATEGIC

E NOVO

AInEOScPScS

ieved

ractio

 Prog

ces 

rough 

 with

m/Re

omm

visory

ffice 

nal 

y 

oard

SABSABMeeBoar

eetin

e 

g 

ved t

ence 

 Pro

hrou

Polic

gram

y proj

ion. 

 SAB 

t discuss

 offic

rther 

ssion 

ions 

es 

 thro

discussions with: 

ated S

ee 

int 
Step 1 – Project Identification and Nomination. inistrators and 

Regional Administrators are as ce Director a memorandum that 

lists all advisory project nom ons, with the highest priority nominations for the next fiscal 

year identified. SAB Pr 2 b r nominated projects should be 

attached to the memorandum. The S Staff Office and the Science Policy Council also request 

this mem e y f t other hig rio ience activities that the Assistant 

Administrator or Regional Adm

latter inf B m t d d nominated projects within 

the context of hig rity s ien ities at EPA. The SAB Staff Office requests this 

information to assist the Science Policy Council in ost 

appropr et of projec y to bring SAC , or the Council. 

Step 2 – Electronic Pro inators are asked to subm t an electron ject sheet 

for each individual project to be considered for SA

sheets are created after establishing or updating the related Peer Review Project or Science 

Activity in the Agency’s Science Inventory. The Science Inventory entr

their Peer R view Coordinator ience Activity Coordin Office 

or Region. Project sheets shoul

submitted projects f r w n m

Office o gion the r ce. The 

electronic Project sheet m e accessed and completed through EPA’s i e In ntory 

database. The inform e El onic data sheet ar

Process fo

 Each year, the Assistant Adm

elo

ked to send the SAB Staff Offi

inati

oject Sheets (see Item w) fo

AB

o six 

t the SA

genc

orandum

ormation is requ

iate s

e

r Re

r Keeping Informed

 to

h-p

 id

rio

ntif

ested, so tha

c

ts fo

our

inistrator has deci

ce activ

he

h-p

ded not to nom

ay be

 its discussions and decisions about the m

 to t

rity sc

ter un

 Boa

inate for SAB peer review. The 

erstan

r t A he rd, CA

ject Sheet.

hic

 S

 Nom

c

ro

aff

i

ed be

e D

ic

tion. The project 

r S

 pro

ci

B, CASAC, or Council atten

eeting has

ies m

pecific Prog

cluding previously 

ire

ust be approved by 

 fo

or S

d be filled out for all desired 

o p

 St

ator for the s

projects, in

ram

ogram

en

o

and

h 

AB

ject planning

Office Director or the As

 occurr

sociat

tween the Pr

cto

ay b

 about the Decisions Made

 Sc

e provided in Table 2. 

enc ve

ation fields required for th ectr

 rece p tions, the SAB Staff Office will discuss pr priorities 

with each E A Pr ram disc ith the EPA 

Science cy Council. T ffice will also discuss pro priorities with the SAB, which 

includes the CASAC and Council chairs. The SAB Staff Office will then develop a 

recomm EPA and SAB rankings taking into account 

available resources. The ice will consult with the EPA Admi nd Deputy 

Administrator to develop an annua

Additionally, projects may be adde r t

EPA se xecut management, Congress, and the SAB. 

duct Data

projects nominated for the Board and the status of decisions made on them. The database will 

also provide ongoing information about the status ports and meetings related to 

projects accepted for Board advice in Fiscal Year 2006. 

After

P

 Poli

ended list of advisory projects based on

nior e

EPA Staff may refer to the SAB Pro

iving

og

roject nomina

 Offices and R

oj

iorit

ect

egion, and uss overall Agency pr ies w

he Staff O

 SAB Staff Off

l operating plan that includes 

d or deleted at any tim

ject

the highest priority projects. 

ing

nistrator a

e du he fiscal year, as requested by 

ive

base in an ongoing way to identify all the 

 of SAB re
EPA Staff with questions about the SAB Product Database or the process for submitting 

ominations to the EPA SAB may contact Angela Nugent (nugent.angela@epa.gov, 202-343-

21.

n

9981) or Ms. Patricia Thomas (thomas.patricial.epa.gov, 202-343-9974). 

 

Table 2: Information Fields for SAB Project Sheet 

1. Project Title

2. Project Short Title

3. Fiscal Year SAB Activity Desired to Begin

4. Quarter SAB Activity Desired to Begin

5. Requesting Organization

6. Requesting Office

7. Requesting Official 

8. Requesting Official's Title

9. Program Contact 

10. Program Contact's Phone

11. Program Contact's Mail Code

12. Background for this Project:

13. Tentative Charge:

14. Applicable GPRA Goal and Objective 

15. Description of and citation for any legal obligation/directive for SAB Review: 

16. Principal interested and affected parties

17. Type of SAB advice requested

18. Why Should the SAB Advise on this Project?

19. Disciplinary Expertise 

20. Budget: 

 FY 

 Extramural Budget 

 FTE

Past Peer Reviews:

22. Quality Management/Quality Assurance: 

 
Appendix F 

ents Of Work for

racts

Users of this Peer Review Handbook need to be aware that the examples contained in this 

Appendix are generalized statements of work prepared (and in some cases modified) to 

emphasize certain important features (e.g., attention to conflicts of interest, responsibilities of 

c charge to peer reviewers). Please 

e

time of award (for example, see EPAAR 1552.212-71, alternate I). 

Example Statem

Cont

ontra

e sur

ctors or contracting officers, development and use of the 

b that proper and currently approved contract language is used in any EPA contract 

document at the
 Peer Review Handbook Page F-9 
 Peer Review Handbook Page F-10 

STATEMENT OF WORK - EXAMPLE 1 - Peer Review of Prioritization Tool Report

Work Assignment No.: 

Title: Peer Review of Prioritization Tool Report 

Work Assignment Manager (WAM): 

Name: John Q. Government Employee 

Address: Office of Solid Waste 

Phone No.: (202) 564-XXXX 

Background: 

The Waste Minimization Branch (WMB) in the Office of Solid Waste (OSW) is in the process of 

implementing the Waste Minimization National Plan, announced by the Agency on November 

18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction over waste 

management, in keeping with the policy stated in the 1984 amendments to the Resource, 

Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act (PPA). The 

Plan outlines major goals, objectives, and action items to achieve national reductions in the 

generation of hazardous wastes. 

One of the objectives of the Plan is to: "develop a framework for setting national priorities; 

develop and distribute a flexible screening tool for identifying priorities at individual facilities; 

[and] identify constituents of concern." This objective is a key building block in implementing 

subsequent objectives of the Plan. 

In September 1995, WMB formed the Waste Minimization Prioritization Team, which includes 

representatives from EPA Regions and states, to implement this objective. The Team has 

worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools that 

are currently available. The Team plans to summarize this work, along with its 

recommendations, in a report (referred to herein as the Prioritization Tool report) that would be 

available in draft form in July 1996. 

WMB and the Team wish to obtain independent peer review of the Prioritization Tool report 

prior to briefing EPA management. The report is being prepared with the support of ICF, Inc.; 

therefore, for the peer review to be considered independent, it must be performed by another 

contractor. 

Purpose and Scope of Work: 

The purpose of this work assignment is to provide support to WMB and the Team in finalizing 

the Prioritization Tool report by conducting an independent peer review of the report. 
 Peer Review Handbook Page F-11 

Work Statement: 

Task 1 -Management work plan and budget

ment, the contractor shall deliver a 

anagement work plan including a proposed level of effort, schedule, and budget for all tasks. 

ask 2 -Provide independent peer review of Prioritization Tool report

Within 15 days of CO approval of this work assign

m

T

and the Team in preparing the Prioritization Tool

g a review of the report. The contractor shall establish a 

anel of peer reviewers including three senior-level persons who collectively have extensive 

rticular areas to be identified by the WAM upon approval of the work assignment. 

cludes 

er

onduct the peer review, assemble the peer review comments 

nd recommendations in a peer review report organized by charge question, prepare an 

d

is not necessary that the peer reviewers jointly reach consensus on their findings and 

e of

g the report and

The contractor shall provide support to WMB

report by performin n independent peer

p

expertise in pa

Within three weeks of work assignment approval, receipt from the WAM of the necessary 

qualifications of peer reviewers (in a TD), and receipt from the WAM of the peer review 

“charge” (in a TD), whichever comes latest, the contractor shall identify the three peer reviewers 

and prepare a memo that lists the names of the peer reviewers and their affiliations and in

the peer reviewers’ bio’s. Within five weeks of WAM approval of the of the peer reviewers (via

a TD) and receipt of the draft Prioritization Tool report from the WAM (via a TD), whichev

comes later, the contractor shall c

a

introduction to the peer review report with a clear and concise overview of the comments, an

attach to the peer review report any marginal comments the peer reviewers had on the 

Prioritization Tool report. 

It

recommendations, since there may be limited overlap in the peer reviewers' areas of expertise 

and in the charge questions that they focus on. The contractor shall assume, for the purpos

estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200 

pages of appendices, and that each peer reviewer will spend 40 hours in reviewin

writing comments. EPA plans to provide the report to the contractor in mid-July. 

Deliverables and Schedule: 
 Peer Review Handbook Page F-12 

Task Deliverable Schedule

1 Work plan and budget Within 15 days of CO approval of work assignment 

2 

Memo identifying peer reviewers Within 3 weeks of work assignment approval, receipt of pee

qualifications from WAM, and receipt of charge from WAM, 

r reviewer

whichever comes latest 

3 Peer review report Within five weeks of WAM approval of

draft Prioritization Tool report from WA

peer reviewers and receipt of 

M, whichever comes later

Other Requirements: 

CONTRACTOR COMMUNICATIONS

Upon approval of the Work Plan, the contractor shall maintain at least weekly communicati

with the Work Assignment Manager regarding the status of work on the Work Assignment

CONFLICT OF INTEREST (COI)

ons

.

 [Revised Section] 

The contractor must adhere to the following requirements: 

a) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior 

to commencement of any work, notify both the CO and Project Officer of any 

actual or potential organizational or personal conflicts of interest. 

b) Provide a written certification, within 20 days of receipt of a Work Assignment, 

 work under this Work Assignment or relating 

to this Work Assignment have been informed of their obligation to report 

personal and organizational conflicts of interest to the CO. 

3) The Contractor recognizes its continuing obligation to identify and report 

any conflicts of interest arising during performance of this Work 

Assignment. 

c) If a conflict of interest is identified during performance under this Work 

Assignment, the Contractor shall immediately make a full disclosure in writing to 

the CO. The disclosure shall include a description of action which the Contractor 

QRT, or similar tasking document, that: 

1) Either all conflicts of interest have been reported to the CO or that no 

conflicts of interest exist. The contractor is directed to assure that none of 

the conflicts disclosed are so direct and substantial as to rule out a 

particular reviewer. 

2) All personnel who perform
 Peer Review Handbook Page F-13 

has taken or proposes to take, after consultation with the CO, to avoid, mitigate, 

or neutralize the conflict of interest. 

d) peer w, submit to

review candidates: 

1 flict o f

a lack of imparti

have had or presently have a financial relationship with EPA. 

mmarizes for EPA the Contractor’s efforts to

After selecting the reviewers but before starting the peer revie

EPA documentation that shows that the contractor has determined if the peer 

) have a con f interest or a situation that could create the appearance o

ality in relation to the work product, and 

2) 

This documentation su

identify and propose resolution of these concerns with peer review candidates. 

EXPENDITURE OF FUNDS/HOURS

In addition to the requirements of the contract, the contractor shall notify both the Project Officer 

and the Work Assignment Manager when 75% of funds or hours for this Work Assignment have 

been expended. 

INFORMATION COLLECTION

ny other provision of this Work Assignment notwithstanding, the contractor shall not proceed

ith an form

or mor blic

pproval will c f Management and Budget as required 

y the Paperwo Red

Only Federal agencies

respondent." State ag

oliciti g similar information applies to any coll

equally to "willing par

ny qu stion of

f the circumst es i

undertaken until the C e

ability of the PR is determined to be applicable, the Contractor shall not 

itiate col

The General S

Interagency Re

A

w y in ation collection where the same or similar information will be collected from ten 

e pu respondents until written approval is received from the Contracting Officer. This

a ite an approval number from the Office o

b rk uction Act (PRA).

and their employees are exempt from the PRA definition of "public 

encies and their employees are classified as "public respondents." 

S n ection method, i.e., written, oral, electronic, etc., 

and utilizing any approach, i.e., surveys, phone calls, focus groups, TQM, etc. The PRA applies

ticipants" and participation that is mandated by law. 

A e applicability of the PRA shall be resolved by submitting a complete description 

o anc n a written request to the Contracting Officer. No collection shall be 

ontracting Officer provides written notice to the Contractor as to th

applic A. If the PRA

in any lection until the requisite approval is received.

ervices Administration (GSA), under FIRMR Bulletin B-2 administers the 

ports Management Program as derived from 44 USC Chapters 29 and 31. All 
 Peer Review Handbook Page F-14 

work performe

done in full co

ONF NT

d under this Work Assignment involving Federal interagency reporting must be 

mpliance with these GSA procedures.

C IDE IAL BUSINESS INFORMATION

If this Work Assignment requires use of RCRA Confidential Business Information (CBI), the 

ontract must specifically authorize the contractor to have access to RCRA CBI and the 

Security Manual and i lan budget all 

stimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon as 

rk 

ssignment, whichever occurs first. 

PRINTING A

c

contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA CBI 

n the contract. The contractor shall identify in the Work P

e

it is no longer needed under this Work Assignment or before the expiration of the Wo

A

ND DUPLICATION

The contractor is prohibited from performing any printing under the Government Printing and 

inding Regulations. Duplication is allowed to the extent it does not exceed 5,000 impressions 

ll duplication jobs in excess of 5,000 impressions, the EPA WAM will 

etermine in advance if the work can be performed more cost effectively and under the job or 

p. If the total number of photocopies for this Work 

ssignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan the 

B

of a single-page document or 25,000 impressions of a multiple-page stand-alone document, is 

limited to one color (black) copies, and does not exceed the maximum image size of 10 3/4 by 14 

1/4 inches. For a

d

time constraints at the EPA Print Sho

A

photocopying costs by task and deliverable. 

WORK ASSIGNMENT/WORK PLAN BUDGETS

The contractor shall not exceed either the dollar or PL hour budget contained in the approved 

ork Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not exceed the PL 

ECHNICAL DIRECTION

W

hour budget of the QRT. 

T

1) (APR 1984) 

DEVIATION). Other than the Designated WAM, only the Project Officer and the Contracting 

tractor in accomplishing the 

Statement of Work; 

The Designated Work Assignment Manager (WAM) on this Work Assignment is authorized to 

provide technical direction to the extent allowed under EPAAR (1552.237-7

(

Officer are authorized to provide technical direction. 

Technical direction includes: 

(1) Direction to the contractor which assists the con

(2) Comments on and approval/acceptance of reports or other deliverables. 
 Peer Review Handbook Page F-15 

Technical direction must be within the contract and the Work Assignment statement of work. 

The Project Officer and the WAM do not have the authority to issue technical direction which 

(1) institutes additional work outside the scope of either the contract or this Work 

(4) alters the period of performance or deliverable due dates; or 

(5) changes any of the other express terms or conditions of the contract or Work 

r. If the 

CTIONS

Assignment; 

(2) constitutes a change as defined in the "Changes" clause; 

(3) causes an increase or decrease in the estimated cost of the contract or Work 

Assignment; 

Assignment. 

Technical direction will be issued in writing or confirmed in writing within five (5) calendar 

days after verbal issuance. The technical direction memorandum will be provided to the 

contractor and copies will be forwarded to the Contracting Officer and the Project Office

contractor has not received written confirmation within five (5) calendar days of an oral 

issuance, the contractor must notify the Project Officer. 

INHERENTLY GOVERNMENTAL FUN

ental functions (IGF) under this Work 

ssignment. If during the course of developing the plan of work, through receipt of technical 

e 

er. 

 AND SAFETY

The contractor shall not perform any inherently governm

A

direction, or in carrying out the assignment any portion of the effort is considered to possibly b

an inherently governmental function, the contractor must immediately notify the Project Officer 

and the Contracting Offic

OCCUPATIONAL HEALTH

Facility site visits conducted under a Work Assignment that include on-site inspections or 

sampling must be conducted in full compliance with the Department of Labor, Occupational

Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440 

(Occupational Health and Safety Manual). 
 Peer Review Handbook Page F-16 

TRAVEL COSTS

gulations in incurring allowable travel costs under this Work Assignment, and correspondingly 

must at all time

subsistence cei

QUICK SP

The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal 

re

s seek and obtain Government rates whenever available and observe current 

lings.

RE ONSE TASKS

Each Q Re t

Officer. The c respond by letter to the Project Officer with copies to the WAM 

nd the CO within two working days, giving a brief description of the plan of work, including 

best est task.

No task l e evel

of effort for ea e limited to a maximum of 250 labor hours.

a

uick sponse Task (QRT) shall be confirmed in writing and approved by the Projec

ontractor shall

a

imate of hours (by P-level) and a break-out of costs to accomplish the

shal xceed a duration of 30 calendar days from start date to completion date. The l

ch task shall b

Quick Response Task Requests do not change the dollar or professional labor hour budgets of

Work Assignment. 
 Peer Review Handbook Page F-17 

STATEMENT OF WORK - EXAMPLE 2 - External Peer Review of Protozoa Method 

evelopment Criteria Document

anager: Sally Q. Government Employee 

 Office of Water 

 U.S. Environmental 

Protection Agency 

ACKGROUND INFORMATION:

ter is charged with 

rotecting public health and the environment from adverse exposure to chemicals and microbials 

ssion OW’s Office of Science and Technology (OST) develops health 

standards, health criteria, health advisories, and technical guidance documents for water and 

water-related media. Under this work assignment, documents prepared by OST are to undergo 

peer review. 

Peer review is an important component of the scientific process. It provides a focused, objective 

evaluation of a research proposal, publication, risk assessment, health advisory, guidance or 

other document submitted for review. The criticism, suggestions and new ideas provided by the 

peer reviewers stimulate creative thought, strengthens the reviewed document and confer 

credibility on the product. Comprehensive, objective peer reviews leads to good science and 

product acceptance within the scientific community. 

Under this work assignment, the contractor will receive one document (Protozoa Method 

Development Criteria Document) for peer review which is related to human health and 

ecological effects. 

STATEMENT OF WORK:

Task 1. The contractor shall develop a work plan to address all tasks in this work 

assignment. The work plan shall describe the steps that will be taken by the 

contractor to provide for peer review, including selection of peer reviewer 

candidates with appropriate expertise, determining absence of conflict of interest, 

document and reference distribution, establishing schedules, preparing the peer 

review report, and submittal of the peer review package. Curriculum vitae for all 

persons assigned to complete this work assignment shall be provided. All P levels, 

D

Period of Performance: Work Plan Approval to August 1, 1997 

Work Assignment M

LOE: 196 hours 

SOW: 2.4 

B

The United States Environmental Protection Agency (EPA), Office of Wa

p

in water media, such as ambient and drinking waters, wastewater/sewage sludge and sediments. 

In support of this mi
 Peer Review Handbook Page F-18 

hours and total costs for each task will be provided and costs greater than $100.0

shall be itemized in deta

0 

il. 

determine their 

availability for the task and absence of conflict of interest, and establish a schedule 

hat none of the conflicts 

disclosed is so direct and peer 

reviewers shall participa

than 40 hours to this tas ble for peer reviewers to commit to less 

than 40 hours. The peer review will be conducted for the Protozoa Method 

Reviewers selected by and working for the 

icer in writing prior to their 

beginning work. Minimally, all peer reviewers shall be accomplished in protozoan 

ry and analysis from water. Approval submissions shall 

include the reviewers’ names and curriculum vitae. 

ers 

ent to be reviewed. 

in 

CHEDULE AND DELIVERABLES:

ask 2. 1 week after work plan approval 

ask 3. 1 week after selection of peer reviewers 

Task 4. 

TRAVEL: N

to this work 

Task 2. The contractor shall select a group of peer reviewers and 

for the peer review. The contractor is directed to ensure t

 substantial as to rule out a particular reviewer. Three

te in the review. No single peer reviewer may charge more 

k. It is fully accepta

Development Criteria Document. 

contractor shall be approved by the EPA Project Off

methods for sample recove

Task 3. The contractor shall arrange for the selected peer reviewers to review the EPA 

document. Prepare the charge to the peer reviewers based on technical direction 

received from the EPA WAM. Provide the peer reviewers with copies of the 

candidate report and all relevant references and instruct the selected peer review

to undertake the review. The WAM will provide the contractor with the final 

version of the docum

Task 4. The contractor shall monitor peer reviewers’ progress to ensure timely completion. 

The contractor shall collate peer review comments, and organize the comments 

the peer review “for comments” document. The contractor shall provide the peer 

review document and all materials submitted by the peer reviewers to the EPA

WAM. 

S

Task 1. (Work Plan) 15 days after receipt of work assignment 

T

T

1 week after receiving comments from the peer reviewers 

o travel is anticipated under this work assignment. Any travel directly chargeable

assignment must be submitted and approved by the Project Officer. 
 Peer Review Handbook Page F-19 

STATEMEN

TRODUCTION

Background

The mission 

protect publi

microorganis

sediments. I

develops effl the 

national prog on 

strategies to 

and sedimen

methods for onment and assesses, 

mediates or otherwise manages and communicates the risks associated with exposure to 

In January 1

Credible Sci

policy for pe June 

7, 1994 and program. The Agency policy guides the 

volvement of peers in the Agency’s development of products. This involvement may be 

implement a

work produc l 

decision mak

urpose

he purpose of this contract is to provide peer consultation and peer review support services to 

ces within the Office of Water 

an utilize this contract for efforts within this Statement of Work, if the OST Project Officer 

es c ST projections and requirements). The purpose of 

eer review is to uncover any technical problems or unresolved issues for use in revising a 

 reflect sound technical information and 

nalyses. These processes enhance the quality of and provide for objective review and 

 have no vested interest 

 related policy or implementation issues. 

 or more of the 

ollowing: 

o Support major regulatory decisions or policy/guidance of major impact 

T OF WORK - EXAMPLE 3 - Technical Support for Peer Review Services

IN

of the United States Environmental Protection Agency’s Office of Water (OW) is to 

c health and the environment from adverse effects of pollutants (e.g., chemicals and 

ms) in media such as ambient water, drinking water, wastewater, sewage sludge and 

n fulfillment of this mission, OW’s Office of Science and Technology (OST)

uent guidelines, human health and ecological criteria, health advisories, directs

ram for adoption of Water Quality Standards, develops prevention and remediati

reduce the risk to human health and aquatic life resulting from contaminated fish 

t, provides technical support to the Total Maximum Daily Load program, analytical 

detection and quantification of pollutants in the envir

re

contaminants and microbial pathogens. 

993, responding to recommendations in the report “Safeguarding the Future: 

ence, Credible Decisions,” Administrator William Reilly issued an Agency-wide 

er review. Administrator Carol Browner confirmed and reissued the policy on 

instituted an Agency-wide implementation

in

through peer consultation or peer review of products. The goal of the Peer Review Policy is to 

 comprehensive Agency-wide program to ensure that major scientific and technical 

ts receive critical scrutiny from scientific and technical experts as part of the overal

ing process. 

P

T

OST consistent with the Agency Peer Review Policy. (Other offi

c

determin apacity is sufficient, based upon O

p

preliminary product so that the final work product will

a

evaluation of scientific or technical work by knowledgeable experts who

in

Scientific and technical work products identified for peer review may address one

f
 Peer Review Handbook Page F-20 

o Establish a significant precedent, model or methodology 

o Address a controversial issue 

n a significant emerging issue 

o Have significant cross-Agency/interagency implications 

volve significant investment of Agency resources 

o Consider an innovative approach for a previously defined 

and

ok.” 

 

ng and Technical Studies; Reports to Congress; Technical Guidance 

nd Methodologies; Economic Evaluations; Statistical Analyses; Environmental Models; 

Publications, Products, and Approaches; Analytical Methods; and Policies and 

trategies. The contractor shall provide the necessary professional and technical personnel for 

ods

o Monte Carlo Analyses 

Engineering and Technology Analyses 

o Economic Analyses 

o Focus o

o In

 problem/process/methodology 

o Satisfy a statutory or other legal mandate for peer review. 

o Other products deemed appropriate by OST 

The contractor is responsible for preparing reports on peer review evaluations and shall provide 

support for all stages or portions of the peer review process. The term peer review shall mean 

both peer consultation and peer review of products. Peer review will include initial research 

protocol or work plan, preliminary review of an interim draft, review of a draft final work 

product and final evaluation of the work product. All peer review procedures, products,

responsibilities shall be in accordance with the latest revision of the “U.S. Environmental 

Protection Agency, Peer Review Handbo

Statement of Work

1.0 Administration and Management

1.1 Candidates for peer review include scientific, engineering and economic documents,

methodologies, strategies or positions that are used to support a research agenda, regulatory 

program, policy position or other OST decision. OST has several general categories of scientific

and technical work products which are subject to peer review: Criteria Methodologies/Criteria 

and Advisories; Engineeri

a

Technical

S

review of the following types of products: 

 

 o Toxicological Assessments 

 o Exposure Assessments 

 o Analytic Methods 

 o Statistical Analyses 

 o Dose-Response Assessments 

o Microbial Risk Assessments 

o Environmental Sampling Meth

o Risk Assessment Guidelines 

o 

o Technology Analyses 
 Peer Review Handbook Page F-21 

1 The contractor shall appoint a person to serv .2 e as the Program Manager for this contract. 

int of contact for the contract; shall be responsible for 

e per ma all be designated a key person in accordance 

he contractor shall 

for the direction and selection 

volved in the 

.3 , equipment and necessary 

rofessional, technical and support personnel for performance of work required under this 

s 

ns are 

.1 Selection of Peer Reviewers

he contractor shall provide Peer Review Services. Peer reviewers will be selected for 

 be 

ute to complex Agency decisions is often necessary 

mplete peer review. For example, a risk assessment that relies on both animal 

as for a complete review. The Project Officer 

ill provide the contractor with the document to be 

ailable, provide the contractor with names of known 

e required disciplines for the contractor to make final 

ith submis imeframe identified in the Work Assignment, 

f candidate peer reviewers, either from the pool provided 

y EPA rom erts, or from a list independently identified by the 

ilability for the task. The number of peer 

reviewe ll be identified in the work assignment. The contractor 

 along with their curriculum vitae to the EPA 

The Program Manager shall be the main po

th for nce of work under this contract, and sh

with the Key Personnel Clause in EPAAR 1552.237-72(APR 1984). T

manage all aspects of this contract and shall be solely responsible 

of all contractor, subcontractor and consultant personnel who may be in

performance of the contract. 

1 The contractor shall furnish the facilities, materials

p

contract. The contractor shall develop and institute internal management controls that will 

ensure that, in the performance of this contract, the contractor will not provide personal service

nor perform inherently governmental functions. For these purposes, personal services are 

defined in EPA Order 1900.1 dated October 31, 1985, and inherently governmental functio

defined in EPA Order 1900.2 dated October 22, 1990, and OFPP Policy Letter 92-1, dated 

December 1, 1992. 

2.0 Peer Review Support 

2

T

independence and scientific/technical expertise. The scientific/technical peer reviewers shall

recognized nationally or internationally in their field and have a general knowledge of 

environmental science issues as well as the specific knowledge, expertise or experience as 

required by the Work Assignment. An independent peer reviewer is an expert who was not 

associated with the generation of the specific work product either directly by substantial 

contribution to its development or indirectly by consultation during the development of the

specific product (see Note below). Knowledge about the subject area is not equivalent to 

expertise in the subject matter. For Agency decision, a multi-disciplinary group of experts 

corresponding to the disciplines that contrib

for a full and co

and human data often requires experts in both are

(PO)/Work Assignment Manager (WAM) w

reviewed. The PO/WAM will, when av

experts (listed in alphabetical order) in th

selection. 

W sion of the work plan or within the t

the contractor shall identify a group o

b , f contractor in-house staff exp

contractor (or a combination) and determine their ava

rs required for each review wi

shall submit its selection of peer reviewers,
 Peer Review Handbook Page F-22 

PO/W M. The Agency will review the proposed candidates for consistency with the 

requirements of the work assignment , based either on qualifications, conflicts of interest, or past 

direct involvement with the work under review, however, EPA shall refrain from suggesting 

particular individuals to replace such candidates. Once the reviewers are confirmed, the 

PO/WAM will provide the contractor with the final version of the charge and copies of relevant 

reference materials. 

A

he contractor shall not be involved in the review of documents prepared by the contractor, or its 

 peer 

rest. 

 the 

g as the disciplinary expertise requirements of the panel are also met. The 

ontractor shall attempt in all cases to fulfill this criteria. Cultural diversity is also to be sought, 

tise is the number one criterion for selection. 

t shall be independent experts which have not been 

ubstantially involved in the development of the product. In special cases where the number of 

time; the 

 prior 

h a 

ect 

w that 

ill 

he EPA Work Assignment). The charge to peer reviewers will usually make two 

eneral requests. First, it will focus the review by presenting specific questions and concerns 

e 

d 

to the 

T

subcontractor or consultants. It is the responsibility of the contractor to ensure that all

reviews are conducted in a manner to avoid all actual, potential or apparent conflicts of inte

The contractor shall obtain and provide written certification from all peer reviewers prior to

commencement of work that no personal or organizational conflicts of interest exist. 

It is the Agency’s policy to have panels of experts that represent diverse constituencies, e.g. 

industry, academia, public interest groups, Federal (non-EPA) Agencies, State and Local 

governments, so lon

c

however, disciplinary exper

Note: Peer reviewers of a produc

s

experts in a disciplinary area is very small or not available at the specifically designated 

contractor may select a peer reviewer who has been a peer consultant or prior reviewer of the 

product. If this selection is necessary, the contractor shall submit documentation of the

involvement in the development process of the product by the potential reviewer, along wit

control plan to avoid bias or lack of impartiality, for EPA CO (Contracting Officer) and Proj

Officer review and consent. If special circumstances exist where the expertise is so narro

another peer reviewer isn’t available, the EPA CO/PO, along with the Peer Review Leader, w

determine the appropriate involvement and controls for the potential reviewer, so that no 

perception of bias or conflict exists. 

2.2 Information and Charge to Peer Reviewers 

EPA will provide to the contractor written technical direction identifying the charge to be given

to the peer reviewers. The peer reviewers shall consider EPA-supplied and EPA-specified 

documents along with EPA-provided instructions on the exact nature of the assignment (as 

outlined in t

g

that the Agency expects the reviewers to address. Secondly, it invites general comments on the

entire work product. The contractor shall provide the charge to the peer reviewers along with th

product to be peer reviewed and all reference materials, and instruct the peer reviewers to 

undertake the review. Where appropriate, raw data and/or data in summary form will be provide

and reviewers will be expected to rely on their assessment of such data in performing their 

review. Each peer reviewer must be informed of the need for confidentiality with regard 
 Peer Review Handbook Page F-23 

release of Agency products that are stamped as “DRAFT” or “DRAFT - Do Not Cite, 

Release.” The peer reviewer and the contractor will agree that the draft shall not be given to 

others who are not under contractor direction without the written approval of the PO/WAM. 

Any requests for such drafts should be forwarded to the EPA Contracting Officer for actio

2.3 Peer Review Report

Quote or 

n.

nt. 

s 

ich their 

commendations are based. When a document assigned under a task order requires more than 

opy) , 

rmat and deadline will 

e provided in the work assignment issued for the work. The contractor shall submit to the EPA 

A shall 

ent 

ing of and transcription of any proceedings. The scientific 

xpertise required shall include the broad array of the scientific disciplines relevant to the 

 a document or issue paper is 

viewed, the contractor may be required to meet and document widely varying topics which will 

ostThe contractor shall monitor peer reviewers’ progress to ensure timely completion. All peer 

review comments shall be submitted to the EPA PO/WAM for the particular work assignme

Peer reviewers will be required to submit with their evaluations or comments a description of the 

procedures used to arrive at their recommendations; a summary of their findings; a list of source

relied upon; and make clear and substantiate the methods and considerations upon wh

re

one peer reviewer, the contractor shall summarize all review comments and present this 

summary in an integrated manner that facilitates an overall evaluation (e.g., an annotated c

and shall collate them into a peer review report. The contractor shall be tasked with the 

compilation, categorization, and summarization of the comments. The peer review report shall 

be submitted to EPA in a specified format by a specified deadline. The fo

b

PO/WAM the peer review report and all materials submitted by the peer reviewers. EP

evaluate and analyze all peer review comments and recommendations to ensure technical 

soundness and adherence to Agency policy. Agency officials should document their evaluations 

of the quality and validity of the peer review. 

2.4 Peer Review Meetings 

The contractor shall discuss peer review comments with the EPA PO, WAM, and/or other peer 

reviewers in telephone conferences, peer review workshops, and/or other meetings. Telephone 

conferences, workshops and meetings shall be conducted in accordance with work assignm

specifications, which may require tap

e

specific document under development or review. In instances when

re

be defined specifically by the work assignment. The contractor shall organize and provide 

support in arranging and conducting peer review meetings. The contractor shall obtain all p

meeting comments, collect and compile all comments and suggested document revision, and

return all required information/documentation to the PO/WAM. Occasionally, searches of 

recent literature and hard copy or microfiche retrieval of germane papers may be required. 
 Peer Review Handbook Page F-24 

2.5 Peer Review Meeting Support

Peer Review meeting logistical support includes bringing together appropriate experts to addr

various issues and concerns raised by EPA for selected products. The contractor may be 

ess 

quired to (if specified in the work assignment): 

1) prepare and distribute presentation materials, including pre-conference information , 

l 

10) A draft and final report may be prepared, in accordance with the requirements as 

signment, and in accordance with the specifications of 

Statement of Work Parts 2.3 and 2.4 herein. 

e, 

ontractor (including subcontractors and consultants with a contractual agreement with the 

prime), who, by reason of their role within the scope of the Work Assignment are required to 

attend the meeting. The contractor may be expected to reserve a block of lodging rooms for all 

reviewers and participants in a meeting. However, those participants who are Federal employees 

must then make their own room reservations, travel arrangements, and travel funding 

arrangements. 

re

agenda, pre-registration forms, fact sheets, overheads, slide presentations, etc.; 

2) arrange for meeting space when government space is not available, reserve meeting 

rooms, copying and audiovisual equipment, furniture/room setups, microphones, and 

related services to accommodate meeting needs; 

3) provide for, prepare, and convey all materials such as name badges, table tents, visua

aids, displays, etc., needed for pre-conference and onsite activities; 

4) inspect the conference site in accordance with expected facilities and services; 

5) handle registration, distribution of meeting materials, and attendance list; 

6) provide on-site recorder and/or facilitator; 

7) arrange reporting , transcription and/or note taking services, typing, reproduction, and 

related support services as required by meeting needs, including rapid photocopying for 

important materials to be copied at meetings; 

8) operate audiovisual equipment, microphones, other presentation related support 

equipment, and/or assist participants in operating such equipment; 

9) handle close-down of the site, reviewing all bills for accuracy, and completing 

payments for equipment rental, etc. 

specified in the work as

11) If required, distribute draft proceeding summaries to the participants for comment and 

make final revisions to the document. 

The length of the meeting, expected attendance, seating arrangements and audiovisual equipment

needs will be specifically identified in the work assignment issued by the CO. 

The contractor shall arrange travel only in accordance with the authority and limitations of the 

Section H Clause entitled “Approval of Contractor Travel”, i.e., use of contract funds to 

reimburse travel is strictly limited to logistical support for peer reviewers who directly contribute

to the requirements of specific contract performance related to the statement of work of the 

contract which supports the mission of OST. In providing logistical support within the scop

the contractor may claim transportation, lodging and subsistence for the personnel of the 

c