EPA/100/B-06/002



# U.S. Environmental Protection Agency

Peer Review Handbook

3rd Edition

Prepared for the U.S. Envi ronmental Protection Agency by Members of the Peer Review Advisory Group, for EPA’s Science Policy Council



## Science Policy Council U.S. Environmental Protection Agency Washington, DC 20460This report is dedicated to the memory of our friend and colleague, Nash Gerald. Nash was a wonderful colleague who thoughtfully strived to perfect the use of peer review at EPA.

Nash joined EPA in 1989. In addi tion to his duties coordinating science/policy and research for th e Office of Air Quality Planning and Standards, he consistently cont ributed to considerations of the Agency's peer review practices. Nash’s courteous and indefatigable approach to resolv ing details greatly enhanced the

revision of this Handbook. He will be missed.Peer Review Handbook Page iii



# Disclaimer

This Handbook was developed by EPA to provide gui dance to EPA staff and managers who are planning and conducting peer reviews. This Handbook is intended to improve the internal management of EPA by providing recommended pro cedures and approaches for EPA staff and managers. This Handbook is a guidance manual and not a rule or regulation. The Handbook does not replace existing laws or regulation, doe s not change or substitute for any legal requirement, and is not legally enforceable. Th is Handbook does not create or confer legal rights or impose any legally binding requirements on EPA or any party. The use of non-mandatory language such as "may," "can" or "should" in this Handbook does not connote a requirement but does indicate EPA’s strongly prefe rred approach to ensure the qua lity of peer reviews conducted or initiated by EPA. Mention of trade name s or commercial products does not constitute

endorsement or recommendation for use.Peer Review Handbook Page iv



## The Peer Review Advisory Group

Connie Bosma ORD

Jennifer Bowen OPEI Michele Burgess OSWER (now at USDA) Mary E Clark OAR Kerry Dearfield ORD (now at USDA)

Patrick Easter OSWER

Daniel Fort SAB

Nash Gerald OAR Roland Hemmett Region 2

Vincia Holloman OEI Joyce Jatko ORD (now at Navy)

Barbara Klieforth ORD

Steven Knott OPPTS Andrew McIntosh AO (now at GAO)

John E Miller ORD

Manisha Patel OGC Mary Reiley OW Nancy Wentworth OEIPeer Review Handbook Page v



# Table Of Contents

DISCLAIMER ............................................................................................................................... iii

FOREWORD ............................................................................................................................... xiii

LIST OF ACRONYMS ................................................................................................................ xv

QUICK START GUIDE TO TH E PEER REVIEW PROCESS

................................................... 1 Figure 1 – Flowchart for Planning a Peer Review

...................................................................... 2 Figure 2 - Flowchart for Conducting a Peer Review .................................................................. 3 Figure 3 - Flowchart for Co mpleting a Peer Review

.................................................................. 4 Manager’s Planning Checklist for Peer Review ......................................................................... 5 Regulatory Action Developmen t Checklist for Workgroups

...................................................... 6

PEER REVIEW GUIDANCE ........................................................................................................ 9

1. THE NEED FOR PEER REVIEW.......................................................................................... 11

1.1 Overview Statement...................................................................................................... 11

1.2 Understanding Peer Review.......................................................................................... 11

1.2.1 Why Use Peer Review? ........................................................................................ 11

1.2.2 What is Peer Involvement? ................................................................................... 12

1.2.3 What is Peer Review? ........................................................................................... 12

1.2.4 What is Peer Input?............................................................................................... 12

1.2.5 How is Peer Review Different from Peer Input?.................................................. 13 1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer Reviewer

for the Same Work Product Later in the Process? ................................................................ 13

1.2.7 Can the Same Peer Reviewer be Used More Than Once if a Product Will Be Peer Reviewed More Than Once, and Can the Same Peer Reviewer be Used Again and Again for

Different Products? ............................................................................................................... 13

1.2.8 How is Peer Review Different from Public Comment?........................................ 14

1.2.9 How is Peer Review Different from Stakeholder Involvement?

.......................... 14

1.2.10 What Role Does Peer Review Have in Regulatory Development?

...................... 14

1.2.11 How Does the Tiering Process Affect Peer Review? ........................................... 15

1.2.12 Should You Discuss Peer Review in the Analytic Blueprint for Your Regulation?

15

1.2.13 What Role does Peer Review Ha ve in Regulatory Negotiations? ........................ 161.2.14 Should You Address Peer Review in the Preamble of a Regulation?

.................. 16

1.2.15 How is Peer Review Documented in the Action Memo for Regulations?

........... 16

1.3. Peer Review and Information Quality .......................................................................... 16

1.3.1 Overview Statement.............................................................................................. 16 1.3.2 How Does Peer Review Ensure the Qu ality of Information EPA Disseminates? 17 1.3.3 How Do You Ensure the Quality of Data Supporting EPA’s Information

Products? 17 1.3.4 Are Work Products Undergoing Peer Re view Subject to the Guidelines?

........... 18

1.4 Annual Agency Reporting ............................................................................................ 19

1.4.1 What Are EPA’s Reporting Practices and What Information Is Publicly

Available? ............................................................................................................................. 19

1.4.3 How Will the Public Be Allowed to Co mment on the Peer Review Plans for

Influential Scientific Information and Hi ghly Influential Scien tific Assessments?

............. 21 1.4.4 What Is ORD’s Role in Reviewing th e Science Inventory and How Will This

Review Be Conducted?......................................................................................................... 21 1.4.5 How Will EPA Submit an Annual Report to OMB? ............................................ 21

1.5 The Roles of People and Organizations in Peer Review .............................................. 22

1.5.1 Who is Ultimately Accountable for Peer Review?............................................... 22 1.5.2 Who are the Agency Staff involved in Peer Review?........................................... 22 1.5.3 Who are the Decision Makers and What are Their Responsibilities? .................. 22 1.5.4 Who are the Peer Review Coordinators and What are Their Responsibilities? ... 23

1.5.5 Who are the Peer Review Leaders a nd What are Their Responsibilities?

............ 25

1.5.6 Who are the Peer Reviewers? ............................................................................... 26 1.5.7 What are the Responsibiliti es of Peer Reviewers?

............................................... 26 1.5.8 What is an Independent Peer Reviewer? .............................................................. 26 1.5.9 When does an Agency Internal Peer Reviewer Qualify as Independent? ............ 27

1.5.10 What is a Peer Review Panel? .............................................................................. 27 1.5.11 What is a Subject Matter Expert? ......................................................................... 27 1.5.12 What is the Role of the Science Policy Council (SPC)?....................................... 27 1.5.13 What is the Role of the Peer Review Advisory Group (PRAG)?......................... 28

1.5.14 What is the Role of the Office of Research and Development (ORD)?

............... 28

2. PLANNING A PEER REVIEW.............................................................................................. 29

2.1 Overview Statement...................................................................................................... 29 Peer Review Handbook Page vi2.2 Determining Which Work Products to Peer Review .................................................... 29

2.2.1 What are Scientific and/or Technical Work Products? ......................................... 29

2.2.2 What Scientific and/or Technical Work Products Need Peer Review?

................ 30

2.2.3 How Does One Determine Whether a Scient ific and/or Technical Work Product is

Influential Scientific Information?........................................................................................ 30

2.2.4 How Does One Determine Whether Influent ial Scientific Information is a Highly

Influential Scientific Assessment?........................................................................................ 32

2.2.5 What Economic Work Products Need Peer Review?........................................... 34 2.2.6 Should Economic Work Products Prepared in Support of Regulations that are Classified as Major or Economically Significant be Peer Reviewed?

.................................. 35

2.2.7 What Other Economic Work Products Might Benefit from Peer Review?

.......... 36

2.2.8 What Other Social Science Work Products Need Peer Review?

.......................... 36 2.2.9 How Should Peer Review be Handled for Products Developed Under an Interagency Agreement (IAG)? ............................................................................................ 37 2.2.10 Should Products from Contracts, Grants , and Cooperative Agreements Receive

Peer Review? ................................................................................................................... ..... 37 2.2.11 How Does Peer Review Apply to Produc ts Generated Through EPA Grants or Cooperative Agreements?..................................................................................................... 38 2.2.12 Can the Recipient of a Grant or Cooper ative Agreement Use Agreement Funds to Pay Peer Reviewers of their Work Products?....................................................................... 39

2.2.13 Do Products Generated Under EPA Grants or Cooperative Agreements Need to be Reported in the Science Inventory ................................................................................... 40 2.2.14 Should Site-Specific Decisions be Subject to Peer Review? ................................ 40

2.2.15 Should NEPA Products be S ubject to Peer Review? ............................................ 40 2.2.16 Should Environmental Regulatory Models be Peer Reviewed?

........................... 41 2.2.17 Should Another Organization’s Work Products That Have Been Submitted to EPA for Use in Decision Making be Peer Reviewed?.......................................................... 41

2.2.18 Can Work Products That are Not Determ ined to be Influential Scientific Information or Highly Influential Scientific Assessments Still be Peer Reviewed?

............ 42

2.3 Determining Which Work Products Do Not Receive Peer Review

............................. 42

2.3.1 What Information Is Exempt fr om the OMB Bulletin Provisions? ...................... 42

2.3.2 Are There Other Circumstances Wher e Peer Review Is Not Necessary?

............ 43 2.3.3 For Information That Is Not Exempt, Are There Circumstances Where the Peer

Review Provisions of the OMB Bulle tin Can Be Waived or Deferred?

............................... 43

2.3.4 Does Publication in a Refereed Scien tific Journal Mean That Adequate Peer

Review Has Been Performed? .............................................................................................. 43 2.3.5 Do Voluntary Consensus Sta ndards Undergo Peer Review?

............................... 44 Peer Review Handbook Page viiPeer Review Handbook Page vii i

2.4 Choosing a Peer Review Mechanism ........................................................................... 44

2.4.1 How Do You Determine the Appropr iate Peer Review Mechanism?

.................. 44 2.4.2 What are Some Examples of Internal Peer Review? ............................................ 46 2.4.3 What are Some Examples of External Peer Review? ........................................... 46

2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?.................... 47 2.4.5 Do Agency Work Products Become Candidates for Peer Review when Peer

Reviewed Journal Articles are Used in Support of that Work Product?

............................... 47

2.4.6 When and How Often Should Peer Review Occur? ............................................. 48

2.4.7 What Factors are Considered in Set ting the Time Frame for Peer Review?

........ 49 2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the Peer

Review? 49

2.5 Creating the Peer Review Reco rd and Peer Review Reports

....................................... 49

2.5.1 What is the Peer Review Record?......................................................................... 49

2.5.2 How Can the Peer Review Record Improve the Peer Review Process?

............... 50

2.5.3 What Should Be in the Peer Review Record? ...................................................... 50 2.5.4 How and When Should You Develop and Po st Peer Review Reports and Agency Responses for Influential Scientific Info rmation and Highly Influential Scientific

Assessments? ................................................................................................................... ..... 51

2.5.5 What Should You Do with a Peer Review Record That Pertains to a Rulemaking

Action? 51 2.5.6 When Should the Peer Review Record Building Process Begin? ........................ 51

2.5.7 What are the Differences in Record Keeping for a Review by Individuals Compared to Review by a Panel? ......................................................................................... 52

2.5.8 Where Should the Peer Review R ecord be Kept and For How Long? ................. 52

2.5.9 Are Internal Peer Review Comments Included in the Peer Review Record?

....... 52

2.6 Budget Planning............................................................................................................ 53

2.6.1 What Budgetary Factors Should Y ou Consider in a Peer Review? ...................... 53 2.6.2 What Input is Needed for the Annua l Budget Formulation and Budget Execution

Process? 53

2.7 Legal Considerations .................................................................................................... 53

2.7.1 What Are the Legal Ramifications of the Peer Review Policy?

........................... 53 2.7.2 Is Legal Advice Needed?...................................................................................... 54

2.8 Federal Advisory Committee Act (FACA) Considerations.......................................... 54Peer Review Handbook Page ix

2.8.1 When Do FACA Requirements A pply to EPA-Run Peer Reviews?

.................... 54 2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? ................................. 55

2.8.3 How Do You Ensure that a Contract or-Run Peer Review Does Not Become Subject to FACA?............................................................................................................... .. 55

3. CONDUCTING A PEER REVIEW........................................................................................ 57

3.1 Overview Statement...................................................................................................... 57

3.2 Charge to the Peer Reviewers....................................................................................... 57

3.2.1 What is a Charge? ................................................................................................. 57 3.2.2 What are the Essential Elements of a Charge? ..................................................... 58 3.2.3 Where Can You Get an Example of a Charge? .................................................... 58 3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?

.............. 58 3.2.5 Who Writes the Charge When You Hire a Contractor to Conduct the Peer

Review? 59 3.2.6 Is it Okay to Ask a Contractor to Deve lop the Charge to the Peer Reviewers?

... 59

3.3 Time Line.................................................................................................................. .... 59

3.3.1 What are the Factors in Scheduling a Peer Review?

............................................ 59

3.4 Selection of Peer Reviewers ......................................................................................... 60

3.4.1 What are Considerations fo r Selecting Peer Reviewers?

...................................... 60

3.4.2 Where Do You Find Peer Reviewers?.................................................................. 60

3.4.3 Are External or Internal Peer Reviewers Preferred?............................................. 62 3.4.4 What is Important in the Mi x of a Peer Review Panel?........................................ 62

3.4.5 Ethical Standards .................................................................................................. 63 3.4.6 What Techniques Help Ensure Disclo sure and Appropriate Resolution of

Conflicts of Interest?............................................................................................................. 68

3.4.7 Are There Constraints to Selecting Peer Reviewers? ........................................... 71 3.4.8 If State Employees Are Used as Peer Reviewers, Can EPA Pay Them for This

Service? 72 3.4.9 Does the Agency Recognize Peer Review s That External Parties Conduct and/or Pay For? 72

3.5 Materials for Peer Reviewers........................................................................................ 73

3.5.1 What Instructions Do You Give Peer Reviewers?................................................ 733.5.2 What Materials Should be Sent to Peer Reviewers?............................................. 73 3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?

.......... 75

3.6 Peer Review Services.................................................................................................... 75

3.6.1 What are Gratuitous Services for Peer Review?................................................... 75

3.6.2 Can You Use a Contract to Ob tain Peer Review Services?.................................. 76

3.6.3 How Do You Write a Statemen t of Work for Contracts?

..................................... 76 3.6.4 What are Some Management Controls for Contracts? ......................................... 77 3.6.5 Can You Identify and/or Select Peer Reviewers When Using a Contract?

.......... 80

3.6.6 How is Travel Handled with C ontracts or Purchase Orders? ............................... 80

3.6.7 How is Travel Handled with Special Government Employees?

........................... 81

4. COMPLETING A PEER REVIEW......................................................................................... 82

4.1 Overview................................................................................................................... .... 82

4.2 Final Work Product....................................................................................................... 82

4.2.1 How Do You Incorporate Peer Review Comments into the Final Work Product?

82

4.2.2 How Might Peer Review Commen ts Impact Your Work Product?

...................... 83

4.2.3 What Should the Final Work Product Sa y About the Peer Review Process?

....... 84

4.2.4 Can the Identity of Peer Re viewers be Kept Anonymous?

................................... 84

4.3 Completing the Peer Review Record............................................................................ 85

4.3.1 How Do You Complete th e Peer Review Record?............................................... 85 4.3.2 Where Should the Peer Review Records be Kept, and for How Long? ............... 85 4.3.3 Is Information Regarding a Peer Revi ew Subject to Rel ease under FOIA?

......... 86

APPENDIX A EPA’S P EER REVIEW POLICY

......................................................................... 1

APPENDIX B OMB INFORMATION QUAL ITY BULLETIN FOR P EER REVIEW AND

PREAMBLE ................................................................................................................................... 1

APPENDIX C SOUND SCIENCE AND PEER REVIEW IN RULEMAKING

.......................... 2

PEER REVIEW (CONDITIONAL TEMPLATE) ..................................................................... 4

FRAMEWORK FOR ACTION MEMORANDA...................................................................... 5 Peer Review Handbook Page xPeer Review Handbook Page xi

APPENDIX D EXAMPLES OF PEER REVIEW CHARGES

..................................................... 1

CHARGE EXAMPLE 1 - US EPA Science Advisory Board (SAB) Consultation on Suspended and Bedded Sediments

.............................................................................................. 3

CHARGE EXAMPLE 3 - Economics - Study of M unicipal Government Costs and Financial

CHARGE EXAMPLE 4 - External Peer Review of the Design of the Targeted National

CHARGE EXAMPLE 5 - External Peer Review of the Draft National Strategy for the

CHARGE EXAMPLE 6 - Clean Air Scientific A dvisory Committee Review of the Agency’s

CHARGE EXAMPLE 7 - The Multimedia, Multip athway, and Multireceptor Risk Assessment CHARGE EXAMPLE 2 - Economics - Benefits Transfer from Adults to Children

................ 4

Impacts from Environmental Regulations

.................................................................................. 5

Sewage Sludge Survey for Pollutants in Sewage Sludge

........................................................... 7

Development of Water Quality Criteria fo r Suspended and Bedded Sediments (SABS)

.......... 9

National Ambient Air Monitoring Strategy

.............................................................................. 12

(3MRA) Model

......................................................................................................................... 15

APPENDIX E GUIDANCE ON REQUESTING A REVIEW BY THE SCIENCE ADVISORY BOARD (SAB)

............................................................................................................................... 1

TABLE 1 – PROCESS FOR SELECTING SCIENCE ADVISORY BOARD (SAB)

PROJECTS

................................................................................................................................. 5

APPENDIX F EXAMPLE STATEMEN TS OF WORK FOR CONTRACTS

............................. 1

STATEMENT OF WORK - EXAM PLE 1 - Peer Review of Prioritization Tool Report

......... 2

STATEMENT OF WORK - EXAMPLE 2 External Peer Review of Protozoa Method Development Criteria Document

................................................................................................ 9

STATEMENT OF WORK - EXAMPLE 3 - Techni cal Support for Peer Review Services

.... 11Peer Review Handbook Page xiiPeer Review Handbook Page xii i



# Foreword

Science is the foundation that suppo rts all of our work here at EP A. Strong, independent science is of paramount importance to our environmental policies. The quality of science that underlies our regulations is vital to the credibility of EPA’s decisions and ultimately the Agency’s effectiveness in protecting human health and the environment. One important way to ensure

decisions are based on defensible science is to have an open and transparent pe er review process.

Consistent Agency-wide implementation of peer review has been an EPA priority for many years. Since issuing the Peer Review Policy in 1993, EPA has taken many steps to support and strengthen the policy. In January 1998, EPA issued the 1 st edition of the Peer Review Handbook to provide guidance about peer review and p eer review processes. The Handbook was updated in December 2000 to reflect feedback from EPA’s science community on improvements to the usability of the Handbook. The science communi ty found that the Handbook contributed greatly to the Agency goal of sound science and substant ially improved EPA’s peer review process.

In December 2004, the Office of Management and Budget (OMB) issued its “Final Information Quality Bulletin for Peer Review” (See Appendix B). This OMB Bulletin establishes government-wide guidance aimed at enhancing th e practice of peer review of government

science documents. The Bulletin includes guidance for all Federal agencies on what information should be subject to peer revi ew, for selecting peer reviewer s and for providing opportunities for public participation. It also defines a peer review planning process to allow the public to comment on peer review plans. While incorpor ating the provisions of the OMB Bulletin, EPA updated its Peer Review Policy to reiterate the Agency's commitment to peer review. This policy encourages and exp ects peer review of all scientific and technical information that is intended to inform or support Agency decisions a nd notes that influential scientific information, including highly influential scientific assessments , should be peer reviewed in accordance with

this Handbook.Peer Review Handbook Page xivPeer Review Handbook Page xv



# Acronyms

AA Assistant Administrator

AAS Advisory and Assistance Services

ADP Action Development Process ANPR Advance Notice of Proposed Rulemaking ASTM American Society of Testing and Materials

BOSC Board of Scientific Counselors

CBI Confidential Business Information CCMP Comprehensive Conserva tion and Management Plan

CENR Committee on the Environment and Natural Resources

CFR Code of Federal Regulations CMM Contracts Management Manual CO Contract(ing) Officer

COI Conflict of Interest COR Contracting Officer Representative CREM Council for Regulator y Environmental Modeling CTGs Control Techniques Guidelines

DEO Deputy Ethics Official DFO Designated Federal Officer

DRA Deputy Regional Administrator

EA Economic Analysis (previously kno wn as Regulatory Impact Analysis)

EC Executive Council of the SAB ECOC Ecological Chemicals of Concern EEAC Environmental Economics Advisory Committee EIMS Environmental Information Management System EIS Environmental Impact Statement

ELG Effluent Limitation Guidelines

EMAP Environment Monitori ng and Assessment Program

EO Executive Order

EPA Environmental Protection Agency EPAAR EPA Acquisition Regulations FACA Federal Advisory Committee Act

FAR Federal Acquisition Regulations FIFRA Federal Insecticide, Fungicide and Rodenticide Act

FIRMR Federal Information Re source Management Regulation

FOIA Freedom of Information Act FR Federal Register

FTE Full Time Equivalent

GSA General Services Administration

HHS Health and Human ServicesPeer Review Handbook Page xvi

HISA Highly Influential Scientific Assessment IDLH Immediately Dangerous to Life or Health Concentration IAG Interagency Agreement

IGA Inherently Governmental Activities

IGF Inherently Governmental Function IQGs Information Quality Guidelines

IRIS Integrated Risk Information System ISC-COMPDEP Industrial Source Comp lex - Complex Terrain Deposition

ISI Influential Scientific Information ISO International Organi zation for Standardization

LAN Local Area Network

LOC Level of Concern

LOE Level of Effort

NAS National Academy of Sciences NEPA National Environmental Policy Act

NOAA National Oceanic and Atmospheric Administration NODA Notice of Data Availability NPDES National Pollutant Discharge Elimination System

NPS National Park Service

NRC National Research Council NRMRL National Risk Manage ment Research Laboratory NTTAA National Technology Transfer and Advancement Act of 1995 OAM Office of Acquisition Management

OAP Office of Atmospheric Programs OAQPS Office of Air Qualit y Planning and Standards

OAR Office of Air and Radiation OARM Office of Administration and Resources Management

OCHP Office of Children’s Health Protection

OEI Office of Environmental Information

OGC Office of General Counsel

OGE Office of Government Ethics

OIA Office of International Affairs

OIG Office of the Inspector General

OIRA OMB's Office of Inform ation and Regulatory Affairs OMB White House Office of Management and Budget

ORC Office of Regional Counsel

ORD Office of Research and Development

ORIA Office of Radiation and Indoor Air ORMI Office of Regulatory Management and Information

OSA Office of the Science Advisor

OSP Office of Science PolicyPeer Review Handbook Page xvi i

OST Office of Science and Technology

OSW Office of Solid Waste OTAQ Office of Transportation and Air Quality PAHs Polynuclear aromatic hydrocarbons

PCB Polychlorinated biphenyl PCDD Polychlorinated dibenzo(p)dioxin PCDF Polychlorinated dibenzofuran

PEP Peconic Estuary Program

PL Professional Level PMCR Preliminary Model Calibration Report

PO Purchase Order PR Procurement Request

PPA Pollution Prevention Act PRA Paperwork Reduction Act PRAG Peer Review Advisory Group PRPT Peer Review Product Tracking Database Q&A Question and Answer QRT Quick Response Task RA Regional Administrator RC Regional Counsel RCRA Resource Conservation and Recovery Act

RFA Regulatory Flexibility Act

RfC Reference Concentration (Inhalation)

RFC Request for Correction

RfD Reference Dose (Oral)

RFR Request for Reconsideration

RGE Regular Government Employee

RIA Regulatory Impact Analysis (now known as Economic Analysis) RI/FS Remedial Investigation/Feasibility Study

ROD Record of Decision RSAC Research Strategies Advisory Committee

SAB Science Advisory Board SAME Simplified Acqui sition Made Easy

SAP Scientific Advisory Panel SBREFA Small Business Regulatory Fairness Act SERA Screening level Ec ological Risk Assessment

SGE Special Government Employee SI Science Inventory

SOPs Standard Operating Procedures

SOW Statement of Work

SPC Science Policy CouncilPeer Review Handbook Page xvi ii

SPC-SC Science Policy Council-Steering Committee

SRA Science Review Administrator

STAR Science To Achieve Results

TAMS Toxics Air Monitoring System

TD Technical Directive TQM Total Quality Management

UCL Upper Confidence Limits

URL Uniform Resource Locator

US United States

USC United States Code

USEPA United States Environmental Protection Agency

USGS United States Geological Survey

VSCF Value of a Statistical Cancer Fatality

VSL Value of Statistical Life

WAM Work Assignment Manager

WMB Waste Minimization Branch

WTI Waste Technologies IndustriesPeer Review Handbook Page 1



# Quick Start Guide To

The Peer Review Process

This Section provides flowcharts describing th e major steps in conductin g a peer review and checklists to help plan a successful peer review. Cross references to the appropriate sections in the Peer Review Handbook are shown in parenthese s and bolded. The flowcharts and checklists

included are:

$ Figure 1 - Flowchart for Planning a Peer Review
$ Figure 2 - Flowchart fo r Conducting a Peer Review
$ Figure 3 - Flowchart for Completing a Peer Review
$ Manager’s Planning Checklist for Peer Review
$ Regulatory Action Developm ent Checklist for WorkgroupsPeer Review Handbook Page 3

Figure 2 - Flowchart for Conducting a Peer Review

1. Develop the charge (§3.2)

– Determine which key issues to address

– Include in peer review record (§2.5.3)

2. Select a peer review mechanism (§2.4):

– Internal (§2.4.2)

– External (§2.4.3)

– Mail (e.g., Letter) review (§2.4.3)

– Face to face meeting (§2.4.3)

– One time or multiple meetings (§2.4.6)

– Include logistical information in the peer review record (§2.5.3)

3. Determine the specific timeline(§3.3):

– When will the review be started

– What are the intermediate check points

– What is the deadline for completion

4. Select peer reviewers (§3.4):

– Determine expertise needed (§§3.4.1, 3.4.4)

– Determine sources of peer reviewers (§3.4.2)

– If ISI/HISA, consider asking public to nominate peer reviewers (§3.4.2)

– Consider and address the balance of the panel (§3.4.4)

– Consider conflicts of interest (§§3.4.5, 3.4.6)

– Include documentation in peer review record (§2.5)

5. Materials for the peer review (§3.5):

– Obtain materials from Program for review

– Prepare instructions fo r peer reviews (§3.5.1)

– Forward materials to peer reviewers (§3.5.2)

– Include copy of materials in peer review record (§2.5)

6. Conduct the peer review:

– If ISI/HISA, ask reviewers to prepare peer review report (§2.5.4)

– If HISA, consider seeking public comment on work product and allowing the public to present to the peer reviewers (§§1.4.2, 1.4.3, 1.5.3, 2.4.7)

– If HISA, and if seeking public comment, provide peer reviewers with significant public comments

– Include in peer review record (§2.5)

Note: Some of these steps may occur concurrently. Work Product (Ready for Peer Review)

Develop the Charge

Select the Peer Review

Mechanism

Set the Timelines, Including Deadlines

Select Peer Reviewers

Send Materials to Peer

Reviewers & Conduct

Review

Complete Peer Review

and Obtain Comments

of Peer Reviewers

Continue With Completing a Peer

Review Work Product (Ready for Peer Review)

Develop the Charge

Select the Peer Review

Mechanism

Set the Timelines, Including Deadlines

Select Peer Reviewers

Send Materials to Peer

Reviewers & Conduct

Review

Complete Peer Review

and Obtain Comments

of Peer Reviewers

Continue With Completing a Peer

Review Set the Timelines, Including Deadlines

Select Peer Reviewers

Send Materials to Peer

Reviewers & Conduct

Review

Complete Peer Review

and Obtain Comments

of Peer Reviewers

Continue With Completing a Peer

Review Add to Peer Review Add to Peer Review

Record______________________________________________________________________________ Peer Review Handbook Page 5



### Manager’s Planning Ch ecklist for Peer Review

1) Title of Work Product : ___________________________________________________

2) What Decision/Rule/Regulation/A ction Does this Work Product Support : ________

3) Designation of Scientific and Technical Work Products

G Is the work product scient ific or technical __yes __no?

G Is the work product __influential scientific information (ISI), __highly influential scientific assessment (HISA), or __other? (See Section 2.2.3 and 2.2.4 of the Peer Review Handbook for an explanation of these terms).

4) Determining What Peer Review is Needed

G If ISI or HISA, peer review is needed.

G If not influential, is peer review still needed?

G What peer review mechanism is needed (internal and/or external)?

G When does the review need to be done?

G How much time will be needed to conduct/complete the review?

G Are there court ordered dead lines or other constraints?

G Has senior management (AA/RA/others) been informed of progress/problems?

G What would constitute success for this review?

5) Determining the Resources for Peer Review

G What is the priority of this project re lative to other projects in the same office?

G What resources are needed to conduct the review?

G What are the impacts of the review on personnel?

G Who will lead the peer review?

G Who will conduct the peer review?

G Who will maintain the peer review record?

G Where will the peer re view record be kept?

G What mechanism will be used for the peer review?

G Has the charge been developed?

G Has internal and external coordination been initiated/completed?

G Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes to the final work product) been made?

G How will results of the review be presented and addressed in the fina l work product (e.g., in a preamble, in an accompanying appendix

–as well as changes in the work product itself)?

G Has the work product been entered into the Science Inventory?

6) Comments:

_____________________________________________________________Peer Review Handbook Page 6



### Regulatory Action Development Checklist for Workgroups

This checklist will help workgroups plan for peer review in the larger context of regulatory development. Each numbered section corresponds to a time period in the regulatory development process.

1. Pe er Review Prior to Proposal

Tier 1 or Tier 2 Rule 1

G Is the peer review schedule incorpor ated into the Analytic Blueprint?

G Does this rule rely upon influential scientific information or a highly influential scientific

assessment?

G Will the work product be reviewed using external peer review?

Tier 3 Rule

G Is the peer review schedule incorporated in the plans (e.g., the analytic blueprint) for producing the action?

G Does this rule rely upon influential scientific information or a highly influential scientific

assessment?

G If an internal mechanism will be used for p eer review, is it acceptable according to Section

2.4 of the Peer Review Handbook?

2. Sending a Proposed Rule Forward for the Administrator’s Signature

G Has peer review been completed?

G Does the action memo indicate whether th e rule relies upon infl uential scientific information or a highly influential scientific assessment?

G If the proposed rule relies on influential scientific information or a highly influential scientific assessment, is there a discussion of the peer review in th e preamble of the rule?

3. Before the Proposed Rule Publishes

G Were the peer review report and any relevant materials in cluded in the docket for this rulemaking?

4. Peer Review Prior to Finalization

G Is a new peer review plan necessary as a result of new regulatory options?

5. Sending a Final Rule Forward for the Administrator’s Signature

G Has any new peer review been completed?

1 For further information on tiering and criteria used to determine the appropriate tier for an action, see http://intranet.epa.gov/adplibrary/adp/tiering/step2.htmPeer Review Handbook Page 7

G Does the action memo indicate whether the rule relies upon influential scientific information or a highly influential scientific assessment?

G If the final rule relies on influential scientific information or a highly influential scientific assessment, is there a discussion of the p eer review in the preamble of the rule?

6. Before the Final Rule Publishes

G Were the peer review report and any relevant materials in cluded in the docket for this rulemaking?Peer Review Handbook Page 8Peer Review Handbook Page 9



## U.S. Environmental Protection Agency

PEER REVIEW GUIDANCEPeer Review Handbook Page 10Peer Review Handbook Page 11



## 1. The Need For Peer Review

1.1 Overview Statement

Peer review at the U.S. Environmental Prot ection Agency (EPA) takes many different forms depending on the nature of the work product, rele vant statutory requirement s, and office-specific policies and practices. In January 1993, responding to recommendations in the report Safeguarding the Future: Credible Science, Credible Decisions 2 , EPA issued an Agency-wide policy for peer review. In 1994 EPA reaffirmed the central role of peer review and instituted an Agency-wide implementation program. In 1998, a Peer Review Handbook was created as a single, centralized form of implementation guidance for Agency staff and managers. The Peer

Review Handbook was revised and reissued in December 2000. The newly updated Peer Review Policy and this 3 rd Edition of the Handbook incorporate insights from the use of the 2 nd
Edition as well as the provisions of the OMB Fina l Information Quality Bulletin for Peer Review that was issued to help agencies enhance thei r peer review transparency and accountability. Among the changes introduced by the OMB Bulletin is the use of the terms “i nfluential scientific information” and “highly influential scien tific assessments” (see Sections 2.2.3 and 2.2.4 for explanations of these terms).

1.2 Understanding Peer Review

1.2.1 Why Use Peer Review? Peer review is not free; however, not doing peer

review can be costl y.

Peer review is intended to uncover any technical problems or unresolved issues in a preliminary (or draf t) work product through the use of independent experts. This information is then used to revise that draft product so that the final work product will reflect sound technical information and analys es. Peer review is a process for enhancing a scientific or technical work product so that th e decision or position taken by the Agency, based on that product, has a sound, credible basis. To be most effective, peer review of a scientific and/or technical work product should be incorpor ated into the up-front planning of any action based on the work product - this includes obtai ning the proper resource commitments (people and money) and establishing realistic schedules.

Peer review of scientific and/ or technical work products shoul d not be looked upon as another “hurdle” in the Agency decision making proce sses. Although conducting a peer review means

that time and resources have to be included in the decision making process, the benefits justify

the added cost. Peer review enhances the cred ibility and acceptance of the decision based on the work product. By ensuring a sound basis for deci sions, greater cost savings are realized since

2 EPA/600/9-91/050, March 1992Peer Review Handbook Page 12

decisions will not be challenged as often and extra effort will not be expended to go back and redo the work product. So while peer review is not free, the cost of not doing peer review is usually much more expensive. Furthermore, not conducting a peer revi ew may potentially place the Agency in the position of having to decide whether to try to de fend a scientifically questionable position – which can be very cos tly in terms of both resources, and, more importantly, credibility.

1.2.2 What is Peer Involvement?

Peer involvement is the process whereby Agen cy staff involve subject-matter experts from outside their program in one or more aspects of the development of work products. Peer involvement, therefore, constitutes active outreach to and participation by the broad scientific, engineering, public health, economics and other social science communities beyond the Agency (external), as well as within the Agency (int ernal). Typically, peer involvement takes two general forms: peer input (ongoing discussions duri ng the development of the work product) and

peer review (an evaluation of a work plan, preliminary draft or the like, or the final objective expert evaluation of the work product).

1.2.3 What is Peer Review?

Peer review is a documented critic al review of a specific Agency scientific and/or technical work product. Peer review is conducted by qualif ied individuals (or or ganizations) who are independent of those who performed the work, a nd who are collectively e quivalent in technical expertise (i.e., peers) to those who performed the original work. Peer review is conducted to

ensure that activities are technically supportable, competen tly performed, properly documented,

and consistent with esta blished quality criteria. Peer review is an in-depth assessment of the assumptions, calculations, extrapolations, al ternate interpretations , methodology, acceptance criteria, and conclusions pertaini ng to the specific majo r scientific and/or technical work product and of the documentation that supports them. Peer review may provide an evaluation of a subject where quantitative methods of analysis or measures of succe ss are unavailable or undefined such as research and development. P eer review is usually ch aracterized by a one-time

interaction or a limited number of interactions by independent peer review ers. Peer review is encouraged during the early stages of the project or methods sele ction, and/or as part of the culmination of the work product, as appropriate. Regardless of the timing of peer review, the goal is ensuring that the fina l product is technically sound.

1.2.4 What is Peer Input?

Many Agency work products are developed with the input of various sc ientific and technical experts inside and outside the Ag ency. Like the contribution made by peer reviewers, peer input

is valuable and enhances the scie ntific or technical ba sis of the products. Peer input, sometimes referred to as peer consultation, generally c onnotes an interaction during the development of an evolving Agency work product, providing an open ex change of data, insights, and ideas. PeerPeer Review Handbook Page 13

input may be characterized by a continued and itera tive interaction with sc ientific experts during work product development. A common example of peer input is the input received from workgroup members during development of a produc t. Many Agency products are developed through the efforts of a workgroup, which may include external experts, such as State and Tribal representatives. These workgroup members have an active, ongoing partic ipation in developing the work product. Another example of obtaining peer input is that of an Agency Office sending a draft work product to a list of stakeholder representatives fo r general comments (stakeholder representatives often include experts who could be considered “peers”).

1.2.5 How is Peer Review Different from Peer Input?

The key distinctions between peer input as de scribed above and formal peer review are the independence of the peer reviewers and their level of involvement. Th e goal of peer review is to obtain an independent, third-party review of the product from expe rts who have not substantially contributed to its development. When experts have a material st ake in the outcome of the peer review (such as a regulated party) or have particip ated substantially in the development of the product (such as a workgroup member), those expe rts’ reviews may not qualify as unbiased, independent peer review and may be better characterized as peer input.

It is clear that peer input provi des valuable cont ributions to the development of the work product. However, peer input does

not substitute for peer review. In other words, one cannot argue that a peer review is not necessary if a work product has received “enough” peer input. Peer Input is not a substitute for Peer Review

1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer

Reviewer for the Same Work Product Later in the Process?

Generally, the answer is no as that expert is no longer independent, but rath er a contributor to the work product. There may be special circumstance s where the expertise is so narrow that another peer reviewer is not available. The Peer Revi ew Leader (see Section 1.5.5) will normally be responsible for making this determination and documenting the decision in the peer review

record.

1.2.7 Can the Same Peer Reviewer be Used More Than Once if a Product Will Be Peer Reviewed More Than Once, and Can the Same Peer Reviewer be Used Again and Again for Different Products?

There is no prohibition on using the same peer re viewer more than once on the same product or for multiple products of the same office. However, it is preferable to use different people each time to provide a broader perspectiv e. When using a contractor to provide peer review services, you should recognize that contractors may have a “pool” of reviewers that they use regularly. If the same peer reviewers are used repeatedl y, they may lose thei r independence (or thePeer Review Handbook Page 14

appearance of independence) from the work product(s) . If a peer reviewer is asked to participate in multiple reviews of the same product it s hould be noted in the peer review record.

1.2.8 How is Peer Review Different from Public Comment?

Peer review and public comment are not the same . Public comment solicited from the general public through the Federal Register or by other means is often required by the Administrative Procedure Act, other relevant st atutes or both. Public comment may also be solicited for policy purposes. The Agency takes public comment on some strictly scientific products and almost all regulatory decisions. Public comm enters usually include a broad ar ray of people with an interest in the technical analysis or th e regulatory decision; some are sc ientific experts (who may provide some peer input), some are experts in other ar eas, and some are interested non-experts. The

critical distinction is that public comment does not necessarily draw the kind of independent, expert information and in-depth analyses expect ed from the peer review process. Public comment is open to all issues, whereas the peer review process is limite d to consideration of specified technical issues. While it may be an important component of EPA’s decision making process, public comment does not substitute for peer

review. Public comment does not substitute for peer review.

1.2.9 How is Peer Review Different from Stakeholder Involvement?

Stakeholder involvement occurs when the Agency works with external interest groups that have

some stake in or concerns about the outcome of the technical work product or regulatory position. This is an interactiv e process which usually involves other agencies, industry groups, regulated-community experts, envi ronmental groups, othe r interest groups that represent a broad spectrum of the regulated community, etc., and us ually strives for a consensus approach. The goal of peer review, on the ot her hand, is to obtain an independent, third-party revi ew f or ensuring scientific quality and technical credibility of the work product that supports a policy or decision. Stakeholder involvement is not a peer review mechanism.

1.2.10 What Role Does Peer Review Have in Regulatory Development?

Peer review of scientific and/or technical work products that s upport regulations is an important, fundamental step in the policy setting and re gulatory development processes and affirms the credibility of the Agency. Because the work pro ducts supporting regulations often are subject to intense scrutiny by the general public and the st akeholders involved, you should plan ahead and document the peer review process for such work products.Peer Review Handbook Page 15

In general, peer review should be completed prior to issuance of the proposed regulation. In some cases, work products that support final regul ations may need an additional peer review if

those scientific and/or techni cal work products change signif icantly after the public comment period.

A regulation itself is not subject to the Peer Review Policy. However, if a regulation is supported by influential scientific information or a highly influential scie ntific assessment, the underlying work product should be peer reviewed before EPA issues the proposed regulation. See Sections 2.2.3 and 2.2.4 for a more complete expl anation of what is c onsidered “influential scientific information” or a “highly influential scientific assessment” and for guidance in determining whether a work product contains this type of information. Remember, soliciting comments from stakeholders and the general public after a rule is proposed does not constitute peer review.

The original data and formal analytic models used in Economic Analyses (EAs) are subject to the OMB Bulletin if they constitute influentia l scientific information or highly influential scientific assessments. However, the straightforw ard application of data, models, and scientific and economic analyses used in previously p eer-reviewed EAs and economic assessments or other peer-reviewed products are not subject to additional formal peer review.

1.2.11 How Does the Rulemaking Tiering Process Affect Peer Review?

Tier 1 and Tier 2 rulemakings are, by definition, important Agency rulemakings 3 . Therefore, you should carefully scrutinize work products suppor ting Tier 1 and Tier 2 rules to determine whether they should undergo peer review. In almost all cases you should use external peer review for a work product that is intended to support a Tier 1 or Tier 2 rulemaking. Although acceptable in certain circumstances for influential scientific information, th e use of an internal peer review mechanism for such work products should be avoided.

Work products supporting Tier 3 rulemakings may also benefit from peer review. For work products supporting a Tier 3 rule, internal or ex ternal peer review may be appropriate depending on the nature of the product and other factors. You can find more information on the differences

between internal and external peer review in Section 2.4.

1.2.12 Should You Discuss Peer Review in the Analytic Blueprint for Your Regulation?

Workgroups should specifically addr ess peer review in each analytic blueprint. For peer review purposes, development of the analytic blueprint is the process whereby the workgroup identifies

3 For further information on tiering and criteria used to determine the appropriate tier for an action, see http://intranet.epa.gov/adplibrary/adp/tiering/step2.htmPeer Review Handbook Page 16

the supporting scientific and/or technical work products and r ecommends what kind of peer review is needed. The analytic blueprint should show the schedule of th e peer review in the context of the schedule for the overall rulemaking.

1.2.13 What Role does Peer Review Have in Regulatory Negotiations?

Regulatory negotiations are not ge nerally subject to peer review; however, to ensure final

decisions are based on sound and credible scien ce, where possible, work products that support the negotiation should be subjected to peer review before the negotiation takes place.

1.2.14 Should You Address Peer Review in the Preamble of a Regulation?

Generally, yes. For proposed and final regulations that rely on influential scientific information or a highly influential scientif ic assessment, you should discuss th e peer review report in the preamble, as described in the OMB Bulletin. If y ou are not the rulewriter, take steps to ensure the regulatory workgroup is aware of this pr ovision of the OMB Bulletin. See Appendix C – Sound Science and Peer Review in Rulemaking, for template information.

1.2.15 How is Peer Review Documented in the Action Memo for Regulations?

For all rules requiring the Administ rator's signature (proposed and final), indicate in the action memo what kind of peer review took place. See Appendix C for more information.

1.3. Peer Review and Information Quality

1.3.1 Overview Statement

EPA is committed to providing quality environmenta l information to its partners and the public. This commitment is integral to its mission to pr otect human health and the environment. One of its goals is that all parts of society have access to accurate information sufficient to effectively participate in managing human health and envi ronmental risks. To fulfill this and other important goals, EPA strives to rely upon inform ation of appropriate quality for each decision made. In early 2002, pursuant to the Information Quality Act 4 , the OMB developed government-wide guidance for “ensuring and maximizing” the quality of information Federal agencies disseminate to the public. 5 In late 2002, EPA developed and issued its Information

4 The Information Quality Act, also known as the Data Quality Act, was issued by Congress, under Section 515(a) of the Treasury and General Government Appropriations Act for Fiscal Year 2001 (Public Law 106-554;

H.R. 5658)

5 Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by Federal Agencies , OMB, 2002 (67 FR 8452).Peer Review Handbook Page 17

Quality Guidelines 6 for ensuring the information that supports EPA’s mission is reliable and accurate (objectivity); is appropriate for the in tended use (utility); and, is protected from compromise (integrity). Peer review, when appropriate, enhances the quality of the information

EPA disseminates.

1.3.2 How Does Peer Review Ensure th e Quality of Information EPA

Disseminates?

The Agency recognizes peer review as a co mponent of pre-dissemination review that complements and enhances the “objectivity” and “utility” of EPA’s information products. The Agency recommends that offices conduct pre-diss emination reviews of information to ensure that the information is of appropriate quality before it is disseminated to the public. Predissemination review is especia lly important for influential sc ientific information and highly

influential scientific assessments.

1.3.3 How Do You Ensure the Quality of Data Supporting EPA’s Information

Products?

EPA’s Quality System 7 provides guidance on systematic pla nning for work products that involve

the collection of new environmen tal data or the use of existi ng environmental data. The Agency’s policies for planning fo r the appropriate level of quali ty assurance and peer review ensure that the information the public receives is reliable and defensible. Documentation of this planning occurs at various stages in development of the final product:

a) A Quality Assurance Project Plan or similar documentation is developed and approved prior to the collection of new data, or use of existing data. Systematic planning helps

ensure that the data and information are of the right type, quality and quantity for the conclusions reached in the final work product.

b) EPA’s Action Development Process (ADP) 8 is another mechanism that assists the Agency in achieving the objectivity and transp arency of information used in developing

6 G uidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information Disseminated by the Environmental Protection Agency , EPA/260R-02-008 (October 2002), ( http://www.epa.gov/quality/informationguidelines )

7 EPA’s Quality System provides guidance on systematic planning for the management, collection and use

of all environmental data to ensure the data are of the right type, quantity and quality for the intended use. EPA Quality Manual for Environmental Programs 5360 A1. (May 2000), Section 1.3.1, http://www.epa.gov/QUALITY/qs-docs/5360.pdf

8 EPA's Action Development Pro cess: Guidance for EPA Staff on Developing Quality Actions, EPA June
2004, http://intranet.epa.gov/adplibrary/index.htmPeer Review Handbook Page 18

regulations, economic and policy decisions . The stakeholder involvement and consultation that are part of the ADP enha nce EPA’s efforts to disseminate quality information that is useful to the publi c, and are a valuable component of predissemination review.

c) The guidance entitled General Assessment Factors fo r Evaluating the Quality of Scientific and Tec hnical Information 9 is an additional resource for EPA’s staff as they evaluate the quality and util ity of information, regardless of the source. The document complements EPA’s Information Quality Guidelines by raising the awareness of the information-generating public about EPA’s ongoing interest in ensuring and enhancing the quality of information available for Agency use.

d) Peer review may be an important assessment t ool. For example, peer review may be used by researchers to confirm the robustness of a theory or design before proceeding to the next level, such as field sampling or fabricat ion of new equipment. This is particularly helpful in "cutting edge" situations where technical standards or other measures of validation may not yet exist.

1.3.4 Are Work Products Undergoing Peer Review Subject to the Guidelines?

Products that are undergoing peer review are not considered to be disseminated under EPA’s Information Quality Guidelines because they ar e dynamic documents that are subject to change and therefore, do not represent EPA’s final decision or position. These products should contain the following disclaimer:

This information is distributed solely for the purpose of pre-dissemination peer review under applicable inform ation quality guidelines. It has not been formally disseminated by EPA. It does not repr esent and should not be construed to represent any Agency de termination or policy.

In cases where the information is highly relevant to specific policy or re gulatory deliberations, this disclaimer should appear on each page of the work product. Peer review products that are disseminated are subject to EPA’s In formation Quality Guidelines.

9 General Assessment Factors for Evaluating the Quality of Scientific and Technical Information

, EPA (June 2003), http://www.epa.gov/osa/spc/pdfs/assess2.pdfPeer Review Handbook Page 19

1.4 Annual Agency Reporting

1.4.1 What Are EPA’s Reporting Practices and What Information Is Publicly

Available?

EPA maintains a database, called the Science Inventory ( http://cfpub.epa.gov/si ) , which is designed to track and report scie nce activities across the Agency, in cluding their status and peer review plans (See section 1.4.2). Offices are expect ed to keep this information current; agenda entries for the highly influential scientific assessments and influential scientific information products should be updated at least every six months. Real-time updates may occasionally be necessary, for instance if there is an imminent change in the timi ng for the peer review of a high visibility study or a change in the timing of the public availability of a draft of a highly

influential assessment.

EPA will make its peer review plans for highly in fluential scientific assessments and influential scientific information available for public co mment through the Science Inventory. Offices are to consider any comments that are submitted on the peer review plans for which they are responsible. The OMB Bulletin also calls for fina l peer review reports fo r the highly influential

scientific assessments and influential scientific information to be developed and made available to the public. EPA will make these reports available on the Science Inventory web site, which will directly link to the peer review agenda en try for that item. Additional information on the content of these reports is provided in Section 2.5.

The Deputy Administrator has designated the Office of Research and Development (ORD) to conduct an annual review of the peer review pl ans. As called for in OMB's Peer Review Bulletin, ORD expects to submit a report to OMB by December 15 of each year. This report will include information concerning the peer reviews conducted on the highly influential scientific assessments and influential scientific information during the previous fiscal year. ORD will generate this report from the informa tion in the Science Inventory database.

1.4.2 What Information Is Provided in the Science Inventory?

The Science Inventory database is designed to track and report peer review and other science activities across the Agency. The database is the single repository for product-specific peer review reporting and tracking and uses a common reporting form. Peer review work products are divided into three categories: highly influential scientific assessments, influential scientific information, and other products.

Each AA/RA will designate a Peer Review Coordina tor (See section 1.5.4) to ensure that their science activities and peer review products ar e fully represented, accurate, and complete. Specific information is included for each work product, including a summary or abstract of the product, what the product will be used for, a nd whether the product will unde rgo peer review. If the product will undergo peer review, additional info rmation regarding the peer review plans isPeer Review Handbook Page 20

provided. If the activity or produc t is considered to be influent ial scientific information or a highly influential scientific assessment, the follo wing information should be provided to address the provision of the OMB Bulletin that allows the public to view and comment on the Agency’s peer review plans for th ese activities or products:

a) A paragraph including the title, subjec t and purpose of the activity or product;

b) An Agency contact to whom inquiries may be directed to learn the specifics of the peer review plan;

c) Whether the dissemination is likely to be in fluential scientific information or a highly influential scientific assessment;

d) The timing of the review (including deferrals);

e) Whether the review will be conducted through a panel or individual le tters (or whether an alternative procedure will be employed);

f) Whether there will be opportunities for th e public to comment on the work product to be peer reviewed and, if so, how and when these opportunities will be provided;

g) Whether EPA will provide significant and relevant public comments to the peer reviewers before they conduct their review;

h) The anticipated number of reviewers (3 or fewer; 4-10; or more than 10);

i) A succinct description of the primary disc iplines or expertise needed in the review;

j) Whether reviewers will be selected by EPA or by a designated outside organization;

k) Whether the public, including scientific or professional societie s, will be asked to nominate peer reviewers.

If the product will not undergo peer review, Offices are asked to provide a statement on why peer review is not necessary (see Section 2.3 for further details).

Each record should be electronica lly signed by the Peer Review C oordinator. By signing it, the

Peer Review Coordinator indicate s that it is ready for ORD re view and, in the case of highly

influential scientific assessments and influentia l scientific information, is ready to be made available for public comment.Peer Review Handbook Page 21

1.4.3 How Will the Public Be Allowed to Co mment on the Peer Review Plans for

Influential Scientific Information and Highly Influential Scientific

Assessments?

The OMB Bulletin directs Federal agencies to make the peer review plans for influential scientific information and highly influential scientific assessments publicly available for comment. EPA will provide for this by making electr onic files of the peer review plans available on the Science Inventory. The public can then comment on these plans and comments will be directed to the peer review c ontact identified for the product. Offices are to consider the comments submitted by the public on the peer revi ew plans for highly influential scientific

assessments and influential scientific info rmation, in accordance with the OMB Bulletin.

1.4.4 What Is ORD’s Role in Reviewing th e Science Inventory and How Will This

Review Be Conducted?

ORD conducts an annual review of the peer review products in th e database. The purpose of this

review is to ensure that the Sc ience Inventory fully re presents the Agency’s peer review products and to determine if the peer review decisions ar e consistent with the gui dance presented in this Handbook. ORD will consult, as necessary, with th e appropriate persons in each organization (see Section 1.5). ORD will then consolidate the information and findings for the SPC and the Deputy Administrator. Any conflicts arising from the review will be resolved by the Deputy

Administrator.

1.4.5 How Will EPA Submit an Annual Report to OMB?

EPA expects to submit an annual report to OM B by December 15 of each year that summarizes the peer reviews that were conducted during the previous fiscal year for highly influential

scientific assessments and influential scien tific information. Release of any reviewer information retrieved by a personal identifier will be performed in accordance with the Privacy Act, 5 USC § 552a as amended, and as interp reted in OMB implementing guidance, 40 FR 28,948 (July 9, 1975). According to the OMB Bulle tin, the annual report should include the following:

a) The number of peer reviews conducted subject to the Bulletin

b) The number of times alternative procedures were invoked

c) The number of times waivers or deferrals were invoked (and in the case of deferrals, the length of time elapsed between th e deferral and the peer review)

d) Any decision to appoint a peer reviewer pursuant to a ny exception to the applicable independence or conflict of inte rest standards of the BulletinPeer Review Handbook Page 22

e) The number of peer review panels that were conducted in public and the number that allowed public comment

f) The number of public comments provi ded on the peer review plans, and

g) The number of peer reviewers used who were recommended by prof essional societies.

1.5 The Roles of People and Organizations in Peer Review

1.5.1 Who is Ultimately Accountable for Peer Review?

Under the Peer Review Policy, th e Administrator has designated the Assistant Administrators and Regional Administrators (AAs and RAs) to be accountable for implementing the Policy in their respective organizations. The Deputy Admi nistrator is ultimately responsible for peer review across the Agency and is the final arbitrator of conflicts and concerns about peer review.

1.5.2 Who are the Agency Staff involved in Peer Review?

The principal Agency staff involve d are Decision Makers (and thei r line managers), Peer Review

Leaders and Peer Review Coordinators. In addi tion, there are many other critical staff in each Office and Region who have responsibility for p eer review activities (e.g., Office and Division Peer Review Coordinators, tech nical information managers, and, of course, any Agency staff who serve as internal peer reviewers). Finall y, ORD has oversight resp onsibility, as designated by the Deputy Administrator, for ensuring the Ag ency’s Peer Review Policy is implemented.

1.5.3 Who are the Decision Makers and What are Their Responsibilities?

The AA/RA is the ultimate Decision Maker for his/her organization and is accountable for the decisions regarding the identification of influent ial scientific informati on and highly influential scientific assessments and the mechanism(s) of p eer review utilized for each of the products. The AA/RA may designate Office Di rectors and/or Division Direct ors as the front-line Decision

Makers.

Furthermore, the Decision Makers commit the res ources needed to ensure a proper peer review. Decision Makers are responsible for ensuring that the peer reviews are properly performed and

documented.

In order to ensure greater independence of peer re views, it is important to strictly separate the

management of work products from the actual peer review of those work products. Therefore,

the Decision Maker and the Peer Review Leader for a work product should not be the same

person.Peer Review Handbook Page 23

The Decision Maker needs appropriate training on how to manage the peer review process. The

Peer Review Coordinator for the Office can ad vise the Decision Maker on how to obtain the appropriate training; the training approach varies among the different EPA offices.

Specific responsibilities of the Decision Maker(s) are these:

a) Determine which work products in their orga nization call for peer review and which ones are influential scientific information or highly influential scientific assessments;

b) Designate (in conjunction with the Project Ma nager) a Peer Review Leader to organize each peer review;

c) Provide advice, guidance, and support to the Peer Review Leader in the preparation, conduct, and completion of the peer review;

d) Ensure that sufficient funds are designated in the office’s budget request to conduct the peer review; also ensure that adequate re sources and/or extramural management support are available for the peer review;

e) Establish a realistic peer review schedule;

f) Designate the stage(s) of product development where peer review is appropriate;

g) Consider public comments on the peer review plans and consider and decide if the public will be asked to nominate potential peer reviewers for influential scientific information and for highly influential scientific assessments;

h) For highly influential scientific assessments, decide whether it is feasible and appropriate

to make the draft scientific assessment avai lable to the public for comment at the same time it is submitted for peer review (or during th e peer review process) and whether it is feasible and appropriate to sponsor a pub lic meeting where oral presentations on scientific issues can be made to the peer reviewers by interested members of the public. When public comment on the assessment is sought, provide peer reviewers with significant comments raised by the public;

i) Ensure all relevant issues and comments rais ed by the peer reviewer(s) are adequately addressed and documented for the record, and where appropriate, incorporated into the work product that is used as basis for decision making.

1.5.4 Who are the Peer Review Coordinators and What are Their Responsibilities?

The Peer Review Coordinator is designated by the AA/RA to coordina te and monitor peer

review activities in his/her re spective Office or Region. This person should be of sufficientPeer Review Handbook Page 24

stature and judgment to have th e access to and confidence of all levels of Office or Regional management. The Peer Review Coordinator is th e main contact for his/her organization and can also direct interested parties to other persons /contacts in the Office on specific work products (e.g., Peer Review Leader).

Specific responsibilities of the Peer Review Coordinator are these:

a) General oversight responsibility for the Office’s or Region’s p eer review process;

b) Report peer review activities to the AA/RA;

c) Help mediate difficult issues between the organization and others; if an issue cannot be resolved, then the Peer Review Coordinator can bring the issue to the attention of the appropriate level Decision Makers in each organization for resolution;

d) Function as the liaison with ORD and the Science Policy Council (SPC) to:

1) Represent the Office/Region before the SPC on peer review issues;

2) Advise ORD of any changes in the list of work products and peer review mechanisms during the annual reporting, and when necessary, at other times;

3) Participate in Agency peer review tr aining, workshops, etc., as requested and disseminate this information to the organi zation; coordinate a nd/or present training within the organization.

e) Submit information on the organization’s peer review plans and activ ities as needed to implement the Peer Review Policy (See Section 1.4). On a product sp ecific basis, this data submission responsibility may be re-designated within the organization:

1) Assure that all Science Inventory reco rds for peer review pr oducts are accurate and comply with OMB reporting requirements (see Section 1.4.2);

2) Assure the Science Inventory record a ddresses all issues and reporting requirements for the annual report to OMB and sign th e record, making it available to the public via the Science Inventory website.

f) Establish procedures to ensure that the work product peer review documentation (i.e., peer review record) is filed and maintained in an appropriate manner (see Section 2.5);

g) Provide advice, guidance, and support to the various Peer Review Leaders for the performance of the peer reviews;Peer Review Handbook Page 25

h) Distribute Agency-wide peer review guida nce and materials to a ppropriate Office/Region personnel, as requested.

1.5.5 Who are the Peer Review Leaders a nd What are Their Responsibilities?

The Peer Review Leader is assigned by the D ecision Maker to organize, conduct and complete the peer review for a specific individual work product. The Peer Re view Leader will obtain the assistance and support of the Peer Review Coordina tor as well as any others within the Agency to help perform the peer review. The Peer Revi ew Leader will be chosen on a case by case basis depending on the work product needing peer review. The Peer Review Leader cannot be the

Decision Maker. The Peer Review Leader coul d be the Project Manager for the work product.

The Peer Review Leader should have appropria te training on how to conduct a peer review before conducting the peer review . The Peer Review Coordinato r for the Office can advise the

Peer Review Leader on how to obtain the appropriate training; the training approach varies among the different EPA offices, however, uni form training modules are available.

Specific responsibilities of th e Peer Review Leader are:

a) Keep the Decision Maker informed of the st atus of a given project; provide the Peer

Review Coordinator with da ta for the annual report;

b) Organize, conduct, and complete the peer review following Agency procedures:

1) Establish and maintain the peer review record for the specific individual peer review currently being perf ormed (see Section 2.5); this includes providing the peer review summary information in the Science Inventory for the Peer Review Coordinator to sign when the peer review is completed;

2) Select the peer reviewers in consultati on with others involved with the peer review (e.g., Decision Maker) and ensure that conflict of interest issues are addressed and documented in the peer review record (see Section 3.4.6);

3) Advise peer reviewers of their responsibilities, incl uding preparation of a peer review report if the produc t is influential scientific information or a highly

influential scientific assessment.

c) Provide information to the Decision Maker (i ncluding all appropriate managers in the Peer Review Leader’s chain of command) on the charge, profile of peer reviewers, the peer review comments, and a proposal on how to address the comments. Obtain Decision Maker approval on the approach to responding to peer reviewer comments. Clearly identify any peer review comments for the De cision Maker that will not be addressed inPeer Review Handbook Page 26

the agreed upon approach. (See Section 2.5.3 for specific contents of the peer review record):

1) Make the peer review report for influential scientific information and highly influential scientific assessment publicly available;

2) For highly influential scient ific assessments, develop the Agency response to the peer review and make it publicly available.

d) Notify the Peer Review Coordi nator that the peer review is completed for the annual

report;

e) Archive the peer review record in a manner consistent with the organization’s archiving procedures.

When a contractor is used to conduct a peer review, some of the above responsibilities are assumed by the contractor (see Section 3.6). In addition, when peer reviews are completed through a Federal Advisory Committee, some of the above responsibilities are assumed by the Designated Federal Officer (DFO).

1.5.6 Who are the Peer Reviewers?

Peer reviewers are individuals who have technica l expertise in the subject matter of the work product undergoing peer review. Peer reviewers can come from EPA, another Federal agency, or from outside of the Federal government.

1.5.7 What are the Responsibilities of Peer Reviewers?

Peer reviewers need to be willi ng participants in the peer review process – they should agree to read all materials, participate fully, act ethically, and protect confidential information that arises.

Peer reviewers should maintain th e confidentiality of the product, perform the review in a timely manner, and be unbiased and objective. Peer Reviewers should not be given any products or data that may reveal confidential business informa tion (CBI) unless the peer reviewer is internal to EPA and has the appropriate clearance (see Se ction 3.4.7).

1.5.8 What is an Independent Peer Reviewer?

An independent peer reviewer is an expert who was not associat ed with the generation of the specific work product either directly by substantial contribution to its development or indirectly by significant consultation during the development of the specific product. The independent peer reviewer, thus, is expected to be objective (See Sections 1.2.6 and 1.2.7 for further information). The quality of the peer review is dependent on the competence and independence of the reviewers.Peer Review Handbook Page 27

Independence is freedom from institutional or ideological bias regarding the issues under review and is necessary for objective, fair, and responsible evaluation of the work product. If a selected reviewer has a particular scientif ic or technical perspective, it may be desirable to balance the review with peer reviewers with other perspec tives. In addition to being independent, ideally, peer reviewers should be free of real or perceive d conflicts-of-interest. Conflict of interest is different from independence and is discussed in Section 3.4.5 a nd 3.4.6. If there are potential conflicts of interest (real or perceived), they should be fully unders tood and addressed by the peer review leader (or other appropriate o fficial) to ensure a credible peer review.

1.5.9 When does an Agency Internal Peer Reviewer Qualify as Independent?

An Agency independent internal peer re viewer is one who comes from a different organizational unit than the one where the review question or do cument originates. A different organizational unit usually denotes, at minimum, a different Office (i.e., abov e Division level in programs; above Branch level in Regions) within the organi zation. In particular, a reviewer should not

come from within the chain of command, either upward or down ward. Agency staff may not serve as peer reviewers for the final peer revi ew of a highly influentia l scientific assessment unless a specific exception is granted under Section II I.3.c of the OMB Bulletin.

1.5.10 What is a Peer Review Panel?

A peer review panel can range from a few indi viduals to ten or more, depending on the issue being investigated, the time available and any lim itations on resources. In dividuals who serve as peer reviewers should have appropriate scientific and technical expertise such that the review panel covers the broad spectrum of expertise nece ssary to address the issues/questions presented in the charge.

1.5.11 What is a Subject Matter Expert?

A subject matter expert is one who has specific sc ientific and technical expertise in the matter under review. The importance of sc ientific and technical expertise in the subject matter is obvious, however, knowledge or just “knowing” about the s ubject area is not equivalent to expertise in the subject matter. For Agency decisions, a multi-disciplinary group of experts corresponding to the disciplines th at contribute to complex Agency decisions is often necessary for a full and complete peer review. For exampl e, a risk assessment that relies on both animal and human data usually would call for experts in both areas for a complete review. For economic analyses, experts from the economic disciplines are necessary.

1.5.12 What is the Role of the Science Policy Council (SPC)?

According to the Peer Review Policy statement: “The Science Policy Counc il is responsible for overseeing Agency-wide implementation of this policy, including: promoting consistentPeer Review Handbook Page 28

interpretation; assessing Agency-wide progress; developing recommendati ons for revisions of the policy as necessa ry; and issuing the Peer Review Handbook , which provides additional information and procedures on implementing this policy.” The SPC m eets its responsibilities through coordination with the Peer Review C oordinators, the Peer Review Advisory Group (PRAG) and the Office of Research and Development (ORD).

The SPC, PRAG and ORD are not responsible for id entifying specific products for peer review or determining the level of re view or mechanism for that re view; those functions are the responsibility of management within each Office or Region.

1.5.13 What is the Role of the Peer Review Advisory Group (PRAG)?

The Science Policy Council has created the PRAG to assist in the implementation of the Agency's Peer Review Policy. The primary role of the PRAG is to provide interpretation of the policy and to assist the SPC a nd Agency Offices and Regions in preparing updates to the Peer

Review Handbook.

1.5.14 What is the Role of the Office of Research and Development (ORD)?

The Deputy Administrator has designated the Office of Research and Development (ORD) to provide oversight to the Program Offices a nd Regions in the collection and review of

information that is contained in the Science Inve ntory (for detailed inform ation, see Section 1.4).Peer Review Handbook Page 29



## 2. Planning A Peer Review

2.1 Overview Statement

Planning a peer review is a critical first step to ensuring a successful peer review of a work product. The initial step is to determine whether your work produc t should be peer reviewed. Once you have determined that a peer review will be conducted, the Decision Maker and Peer Review Leader need to plan an appropriate revi ew. This includes the determination of resources (budget and personnel), schedule for the completi on of the peer review, mechanism for peer review, development of the peer review record, and, finally, the selection of peer reviewers.

2.2 Determining Which Work Products to Peer Review

2.2.1 What are Scientific and/or Technical Work Products?

The first step in determining which work produc ts should be peer reviewed is to identify products that are scientific and/or technical in nature. The term scientific and/or technical work products is generally consistent with the term “scientific information” in the OMB Bulletin 10

. Scientific and/or technical work products are used to support a research agenda, regulatory program, policy position or other Agency position or action. Scie ntific and/or technical work products include economic and social science work products. Categories of work products include, for example: risk assessments, techni cal studies and guidance, analytical methods, scientific database designs, technical models, technical protocols, stat istical survey/studies, technical background materials, technical gu idance (except for guidance providing policy decisions), research plans, and research strategies (re journa l articles, see section 2.3.4).

Products that would not be consid ered scientific and/or techni cal work products can include those: that address procedural matters (e.g., planning, reporting, coordination, notification); that are primarily policy statements (e.g., relocation po licy); that are conference proceedings (unless the proceedings are used as the scientific basis for an Agency action or decision); or that are decision documents, e.g., Environmental Impact St atement (EIS), Record of Decision (ROD), or Economic Analysis (EA). For any of these, th e decision document itself is not subject to the Peer Review Policy, but its underlying scientific and/or technical models, data, and/or work products upon which these documents are based are candidates for peer review. In addition, the following Agency documents are not considered sc ientific and/or techni cal work products under the Peer Review Policy: strategic plans, analytic blueprints, and goals documents.

10 OMB defines “scientific information” as “factual inpu ts, data, models, analyses, technical information, or

scientific assessments based on the behavioral and social sc iences, public health and medi cal sciences, life and earth sciences, engineering, or physical scie nces. . .” (OMB Bulletin, Section I.5.)Peer Review Handbook Page 30

Making final determinations concerning which work products are scientific and/or technical is a responsibility of the Decision Ma ker (See Section 1.5.3). All scien tific and/or technical work products should be entered into the peer revi ew component of the Sc ience Inventory.

2.2.2 What Scientific and/or Technica l Work Products Need Peer Review?

The principle underlying th e Peer Review Policy is

that all influential scientific and technical work products used in decision making will be peer reviewed. The process for identifying which products are “influential” and then determining the

mechanism of review will ta ke into account various

criteria and the circumstan ces surrounding the use of the work product. To maintain flexibility, the Decision Maker(s) for peer review should consider the full field of possible work products to iden tify those additional products that might still warrant peer review as well as the full spectrum of peer review mechan isms for each product.

OMB defines influential to mean information that an agency reasonably can determine will have

or does have a clear and subs tantial impact on important public policies or private sector decisions. EPA has provided furt her guidance on making determinations with respect to whether a document is “influential” in its Information Quality Guidelines. 11 Once a product has been identified as being influential scientific information, the Decision Maker should determine whether it meets OMB’s definition of a “highly influential scientific assessment” (see Section 2.2.4). The OMB Bulletin calls for additional peer review procedures for highly influential

scientific assessments. When in doubt about whether a work product merits peer review, decide to peer review it.

2.2.3 How Does One Determine Whether a Scientific and/or Technical Work

Product is Influential Scientific Information?

Generally, determinations whether a scientific and/ or technical work produc t is “influential” will occur on a case-by-case basis. The continuum of work products covers the range from the obviously influential, which clear ly need peer review, to thos e products which clearly are not influential and don’t need peer review. The majo rity of EPA’s work pr oducts fall in-between

those two ends of the continuum and should be closely evaluated and as sessed with respect to certain factors (see below). The Decision Maker should make the judgment as to whether a work product is influential scientific in formation. There is no easy, single “yes/no” test th at applies to the whole continuum of work pr oducts for determining whether a work product is influential

11 G uidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information Disseminated by the Environmental Protection Agency , EPA/260R-02-008 (October 2002), ( http://www.epa.gov/quality/informationguidelines )Peer Review Handbook Page 31

scientific information. A useful rule of thumb is that if there is any doubt about whether a work product needs peer review, go ahead and subject it to peer review.

Generally, scientific and/or technical work pr oducts that are used to support a regulatory program or policy position and that meet one or more of the following factors would be

considered to be influential scientific information:

a) Establishes a significant preced ent, model, or methodology;

b) Likely to have an annual effect on the economy of $100 million or more, or adversely affect in a material way the economy, a s ector of the economy, productivity, competition, jobs, the environment, public health or safet y, or State, Tribal, or Local governments or communities;

c) Addresses significant controversial issues;

d) Focuses on significant emerging issues;

e) Has significant cross-Agency /interagency implications;

f) Involves a significant investme nt of Agency resources;

g) Considers an innovative approach for a pr eviously defined problem/process/methodology;

h) Satisfies a statutory or other legal mandate for peer review.

Historically, EPA has defined produc ts subject to peer review in terms of having “major impact.” “Major impact” means that the work product may have applicability to a broad spectrum of regulated entities and other stake holders, or that it will have narrower applicability, but with significant consequences on a smaller geographic or practical scale. The Agency developed the previously listed factors to consider when determining if a product had a “major impact.” The Agency has also linked its use of the term "influ ential" to the term “major” in its Information Quality Guidelines. The OMB Peer Review Bulletin provides similar factors to use in determining if a scientific assessment is highly influential, thereby creating some overlap

between the two definitions. Scientific assessments that meet the criteria of both influential and highly influential should be consider ed as highly influential.

Typically, an influential scientif ic and/or technical work produc t that has a reasonable likelihood of supporting a regulatory decision or policy/guidan ce of major impact may also have a clear and substantial impact on private sector decisions. The scientif ic and/or technical work products that underlie many of the Agency’s major rulemaking s, policy and guidance documents of general applicability would be designated “influential” under this constr uct because of their far-reaching impacts.Peer Review Handbook Page 32

The novelty or controversy associated with th e w ork product may determine whether it is

influential scientific information. Influential scientific information ma y be novel or innovative, precedential, controversial, or emerging (“cut ting edge”). An application of an existing, adequately peer-reviewed methodology or mode l to a situation that departs significantly from the situation it was originally design ed to address may make peer review appropriate. Similarly, a modification of an existing, adequately peer -reviewed methodology or model that departs significantly from its original approach may also make peer review appropriate. Determining what constitutes a “significant departure” as used in this Section is th e responsibility of the

Decision Maker.

In summary, an influential scientific or tech nical work product has a major impact, involves precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory obligation to conduct a peer review.

2.2.4 How Does One Determine Whether Infl uential Scientific Information is a Highly Influential Scientific Assessment?

Once a scientific and/or technical work product ha s been categorized as “influential scientific information,” the Decision Maker should determ ine whether the product meets OMB’s definition of a highly influential scientific assessment (see also Figure 4). As with the categorization of a work product as influential scientific information, the decision whether or not to elevate a work product to the highly influential category occurs on a case-by-case basis and is largely based on

evaluation of the same fact ors as identified above.

Note that “highly influential scientific

assessm ents” differ from “influential

scientific information” both in terms of their degree of influence and in substance. OMB defines highly

influential scientific assessments as

influential scientific information that “the agency or the Administrator

determines to be a scientific assessment

that:

a) Could have a pot ential impact of more than $500 million in any

year, or

b) Is novel, controversial, or precedent-setting or has significant interagency interest.” All work products EPA may
generate or use

Influential Scientific Information

Highly Influential Scientific

Assessments

Figure 4 -EPA Work Products This diagram depicts the relation between all EPA work products and those considered Influential Scientific Information or Highly Influential Scientific AssessmentsPeer Review Handbook Page 33

OMB defines a scientific assessment as “an eval uation of a body of scientific or technical knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information” 12

.

The OMB Bulletin includes the following examples of assessments that could be considered highly influential scientific assessments:

a) State-of-science reports;

b) Technology assessments;

c) Weight-of-evidence analyses;

d) Meta-analyses;

e) Health, safety, or ecological risk assessments;

f) Toxicological characteri zations of substances;

g) Integrated assessment models;

h) Hazard determinations;

i) Exposure assessments.

OMB’s definition of a highly influential scientif ic assessment overlaps significantly with the factors you should consider when assessing whethe r a work product is in fluential scientific information. The decision to categorize a product as a highly influentia l scientific assessment first depends on determining whether the influential scientific information is a “scientific assessment,” and then on assessing the likely degree of the product’s impact.

The more far-reaching or significant the impacts of a scientific assessment are, the more appropriate it is to categorize the product as a highly influential scientific assessment. The determination as to whether the degree of novelt y, precedent, and controversy associated with a particular assessment warrants designation as a hi ghly influential scientific assessment should be made by the Decision Maker. If the influential scientific assessment invol ves significant issues that are truly “cutting edge,” it may be appropriate to designate it as a highly influential scientific assessment. The greater the “significant depart ure,” as discussed above, the more appropriate it is to include the product as a hi ghly influential scientific a ssessment. These are subjective determinations that should be made by the Decision Maker.

12 OMB Bulletin, Section I.7.Peer Review Handbook Page 34

OMB also directs agencies to consider the degr ee of potential economic im pact of a scientific assessment in deciding whether it should be ca tegorized as a highly in fluential scientific assessment. As explained above, OMB defines highl y influential scientific assessments as those scientific assessments that may have an impact of more than $500 million in any year.

2.2.5 What Economic Work Products Need Peer Review?

Economic analyses (EA) and reports are consider ed scientific and/or technical work products, and as such it may be appropriate to peer review them. If those wo rk products are ISIs, then they should be peer reviewed if they have not already been subject to adequate peer review (See section 2.2.3). The following economic work pr oducts. should generally be peer reviewed.

a) Internal Agency guidance for conducti ng economic and financial analysis;

b) Economic and financial methodologies that will serve as a principal method or protocol

used to conduct economic an alyses within a program;

c) Unique or novel applications of exis ting economic and financial methodologies, particularly those that are r ecognized to be outside of mainstream economic practices;

d) Broad-scale economic assessments of regulat ory programs, such as those required by Congressional mandates (e.g., the Clean Air Act reports to Congress on benefits and costs);

e) Stated preference (e.g., contingent valuati on) and revealed preference surveys (e.g., recreational travel cost surveys) developed to assist in the ec onomic analysis of a regulation or program;

f) National surveys of costs and expenditures for environmental protection (e.g., financial needs surveys, pollution abatement expenditures surveys);

g) Economic research plans developed to a ssess and advance the state-of-science in economic theory, methodologies or modeling (in particular, the techni cal feasibility of the plan’s components);

h) Meta-analyses (i.e., re-analyses of existing published literature and supporting data on the measurement of economic benefits, costs and impacts);

i) Data and analytical models underlying econom ic analyses, particularly those supporting economically significant rules, if the models and corresponding use of the data have not been previously subject to adequate peer review.Peer Review Handbook Page 35

If these types of economic work products suppor t a highly influential sc ientific assessment, you should conduct an external peer review. Extern al peer reviews can be provided by the Science Advisory Board’s Environmental Economics A dvisory Committee, othe r appropriate outside organizations, or individuals that have expertise in the technica l economic issues raised in the economic work product.

2.2.6 Should Economic Work Products Prepa red in Support of Regulations that are Classified as "Major" or "Economi cally Significant" be Peer Reviewed?

Generally, if an EA applies accepted, previously peer-reviewed methods in a straightforward manner, it would not undergo an additional peer review. EAs prepared to support “major” or “economically significant” regulati ons (these terms are defined at the end of this Section) typically do not utilize innovative or untried economic methods. It is unnecessary to conduct peer reviews of straightforward applications or transfers of accepted, previously peer-reviewed economic methods or analyses. The procedures used to transfer or adapt an economic work product are generally established by separate economic guidance documents which have been peer reviewed. Therefore, EAs that are de veloped using these procedures do not normally undergo an additional peer review , even those prepared in support of “major” and “economically significant” rules. To the exte nt that the underlying data and m odels are consider ed “influential scientific information,” they are subject to peer review, including questions regarding the applicability of the specific data set or model for the use.

Even when peer review is not ne eded, additional peer input can be beneficial in the development of economic work products for “major” and “economi cally significant” rules. At present, some peer input of these analyses is already likely to be included as part of the regulatory development process, including input received from other EPA offices represented on the workgroup for the rule, from the Agency’s Regulatory Steering Co mmittee, and from the public as part of the public comment process for the rule. But ther e may be added benefit to employing additional peer input procedures, such as actively soliciting input from economists in other Federal Agencies on the quality and comp leteness of the economic analysis.

Under Section 3(f)(1) of Executiv e Order 12866, “economically signifi cant” rules are those that may have an annual effect on the economy of $10 0 million or more or adversely affect in a material way the economy, a sector of th e economy, productivity, competition, jobs, the environment, public health or safety, or State, Local, or Tribal governments or communities. The term “major,” as defined in the Congressional Review Act, mean s a rule that has resulted in or is likely to result in: an annual effect on the economy of $100 million or more; a major increase in costs or prices for consumers, i ndividual industries, Fede ral, State, or Local government agencies, or geographic regions; or significant adverse effects on competition, employment, investment, producti vity, innovation, or on the abil ity of United States-based enterprises to compete with foreign-based enterprises in domestic and export markets.Peer Review Handbook Page 36

2.2.7 What Other Economic Work Products Might Benefit from Peer Review?

There may be other economic work products not covered in the preceding sections for which peer review might be useful. Examples of such work products are presented below; however, by listing these examples, there is no intent to establ ish a presumption of peer review for these work products.

a) Analyses measuring the economic impacts and effectiveness of adop ting market-based or economic incentives as regulator y management instruments;

b) The technical analyses supporting economic policies established under other government organizations (e.g., economic models us ed to study transportation, economic development, and international trade policies).

Most of these types of economic work products do not exhibit the degr ee of complexity, or

establish an innovative or untried approach, that would warrant a peer review. However, other factors, such as the potential significance of the analysis for cross-Agency or interagency practices, or the significance of the issues a ddressed, may make peer review desirable.

2.2.8 What Other Social Science Work Products Need Peer Review?

Typically, a social science work product is one that includes empi rical, logic-based approaches to answer technical questions about human motivation, human beha vior, social interactions, and social processes, which are rele vant to the environmental issu es being addressed. The term “behavior” includes overt actions; underlying psychological processes such as cognition, emotion, temperament and motivation; and bio-be havioral interactions. The term “social” includes sociocultural, socioeconomic, and sociod emographic status; biosocial interactions; and the various levels of social context from small groups to complex cultural systems. Examples of social science work products in clude analyses and/or evaluation s related to such topics as pollution prevention, risk communication, environm ental information, environmental justice, quality of life, decision-ma king, and public participation.

The following social science work products should normally undergo peer review:

a) Internal Agency guidance for conducting social impact assessments and other community cultural assessments related to di fferent environmental protection approaches such as community-based watershed protec tion (heretofore refe rred to as social assessments);

b) New social science methodologies that will serve as a princi pal method or protocol used to conduct social assessments;Peer Review Handbook Page 37

c) Unique or novel applications of existing soci al science methods such as surveys, focus groups, interviews, network analyses, comp arative analyses, and content analyses;

d) New national surveys of values, perceptions and preferences related to environmental protection;

e) Innovative research or analys es that address the human di mensions of environmental protection or environmental change in terms of social trends, future predictions and/or behavioral generalizations;

f) Social science research plans developed to assess and advance the state-of-science in social science theory, methodologies or modeli ng (in particular, the technical feasibility of the plan’s components).

2.2.9 How Should Peer Review be Handled for Products Developed Under an Interagency Agreement (IAG)?

Under an interagency agreement (IAG), EPA provides funds to another agency for that agency to use for a specific purpose. The receiving agency's guidance for peer review may likely be

different from EPA Peer Revi ew Policy, although the OMB Bulletin establishes some minimal common guidance for the Federal government. Regardless, if EPA plans to use any work products from that agreement, EPA should deci de whether those documents need review under EPA Peer Review Policy.

2.2.10 Should Products from Contracts, Grants, and Cooperative Agreements

Receive Peer Review?

If there is a scientific and/or technical work product resulting from a grant, contract, or cooperative agreement and it is considered infl uential scientific information or a highly

influential scientific assessment and it will like ly be used in Agency decision-making, the work product needs peer review. Since it would probably result in a perceived, if not real, conflict of interest, the group that is genera ting the work product usually cannot conduct or perform the peer review of its own work product. Exceptions may be made in certain instances for organizations that have adequate and well esta blished recognized procedures for peer review, such as the National Academy of Sciences (NAS). In practi ce, the Agency may need to peer review the product on its own, or arrange with an independe nt third group (e.g., vi a another extramural vehicle) to conduct the peer revi ew. The Agency should not use sc ientific and/or technical work products from contracts, grants , or cooperative agreements to support decision making unless the work products are peer reviewed for both scientif ic and technical rigor and applicability to the specific use to be made of the product.

Be aware that contracts are very different from grants and coope rative agreements. Please note that there are important legal restrictions on the direct use of work products developed underPeer Review Handbook Page 38

grants and cooperative agreements in th e Agency’s decision-making process (see http://intranet.epa.gov/ogd/ ) for additional information).

2.2.11 How Does Peer Review Apply to Pro ducts Generated Through EPA Grants or Cooperative Agreements?

It may be desirable to peer review scientific and/or technical work pr oducts that are generated through EPA grants or cooperative agreemen ts, but special considerations apply.

EPA may legally provide financial assistance for re search that is intended to stimulate or support development of scientific knowledge that is not pr imarily for EPA’s direct use or benefit. The resulting work products might be widely dissem inated either through pub lication in scientific journals or through other means, as opposed to a report tailored to EPA’s specific needs and requirements. EPA can consider these work pr oducts just as it can review other published scientific works when formulating its programs and policies. Further, EPA retains a royalty free, nonexclusive and irrevocable right to use the work products for Federal purposes, even if the recipient has copyrighted the material (see 40 CFR 30.36(a)).

EPA may determine that the recipient’s work pr oduct is influential scientific information because: 1) it will be used to support an EPA pr ogram or policy position, and 2) it meets one or

more of the seven factors outlined in Section 2.2. 3. In this situation, the work product should be peer reviewed.

The following are options for peer reviewing the product:

a) EPA can have the product peer reviewed with the participation of the assistance agreement recipient/author . In this case, EPA could ar range for an independent peer review of the product itself, or may contract with a third party to conduct the review.

EPA would also ente r into a contract with the author (forme rly the recipient), which would task the author to pr epare a response to the peer reviewers’ comments, and to revise or prepare an addendum to the product in response to peer reviewer comments as determined appropriate by EPA.

A caveat to this approach is that it may be di fficult to get the recipi ent/author to agree to allow EPA to determine how to revise the product in response to the peer review

comments.

b) EPA can have the product peer reviewed w ithout the participation of the recipient/author

. EPA could arrange for the peer review itself, or could cont ract with a third party to

conduct the review. In this case, however, the work produc t would not be revised to incorporate the peer review comments. Instead, EPA woul d receive the comments and prepare a statement that documents EPA’s own response to the comments. The EPA Decision Maker who is using the work pr oduct to support an EPA program or policyPeer Review Handbook Page 39

decision should be provided information on bot h the conclusions of the recipient’s work product and EPA’s own conclusi ons from the peer review.

c) Recipients can get their products peer re viewed on their own . Recipients may determine on their own that peer review would benefit the credibility of their product. Provided EPA agrees that a peer review would further the public purpose of the assistance agreement, EPA may include funds for the peer review in the agreement (see 2.2.10 for additional information). Alte rnatively, the recipient may make arrangements for, and fund, an independent peer review of their pr oduct. In either case, EPA would need to evaluate whether the peer re view process undertaken by the recipient was acceptable for the purposes for which EPA was planning to use the work product. EPA may accept the peer review if it determines that it is of a ppropriate quality and that EPA could defend the peer review as if it were conducted by EPA itself.

Under options a) or b), issues may arise over obta ining access for peer reviewers to the raw data used by the recipient to generate the work pr oduct. Under 40 CFR 30.36(c)(2), EPA has a right to obtain raw data produced by a non-profit orga nization or university under an assistance agreement, even where the agreement does not specifically provide for this access. Nevertheless, it may be prudent to include a specific term in the assistance agreement clarifying this point to avoid misunderstandings. EPA may have to pay fo r obtaining access to the data if its transmittal imposes additional costs on the recipient. Assumi ng our use of the data is incidental to the principal purpose of the agreement, we also have a specific right to auth orize peer reviewers to use the data for Federal pur poses under 40 CFR 30.36(c)(2).

EPA cannot obtain access to the raw data if EPA specifically bargains away this right in the assistance agreement.

Again, consult the Office of General Counsel ( OGC) for help in drafti ng appropriate language for your assistance agreement.

2.2.12 Can the Recipient of a Grant or Cooperative Agreement Use Agreement Funds to Pay Peer Reviewers of their Work Products?

As noted in Section 2.2.11, provided EPA agrees that a peer review would further the public purpose of the assistance agreement, EPA may include funds for the peer review in the agreement. A payment to peer reviewers in exch ange for their review of a scientific and/or technical work product is allo wable as a fee for professiona l services under assistance agreements. (To accurately characterize this cost , however, it is important that the payment be referred to as a fee, rather than an honorariu m.) See OMB Circular A-21, Section J, item 32, Professional Services Cost (Educational Institutions), OMB Circular A-122, Attachment B, Item 39, Professional Services Costs (Non-Profit Organizations), and OMB Circular A-87, Attachment B, Item 33, Professional Services Costs (State, Local and Indian Tribal Governments). See http://www.whitehouse.gov/OMB/circulars/ for further details.Peer Review Handbook Page 40

2.2.13 Do Products Generated Under EPA Grants or Cooperative Agreements Need to be Reported in the Science Inventory

As a matter of practice, EPA orga nizations are encouraged to incl ude scientific work being done under grants and cooperative agreemen ts in the Science Inventory so that Agency staff are aware of the ongoing work. If a grant or cooperative ag reement product is likely to be used in Agency decision making (assuming this use is incidental to the principal purpose of the agreement), it should generally be considered a candidate for pe er review and noted as such in the Science Inventory.

2.2.14 Should Site-Specific Decision s be Subject to Peer Review?

A site-specific decision itself is not subject to peer review and does not need peer review as

described in the Peer Review Policy. However, if a site-spe cific decision is supported by influential scientific information or a highly infl uential scientific assessment, that work product should be peer reviewed. Generally speaki ng, you should closely examine how the underlying scientific and/or technical work product is adapted to the site-specific circumstances.

2.2.15 Should NEPA Products be Subject to Peer Review?

Although an Environmental Impact Statement (E IS) prepared under the requirements of the National Environmental Policy Act (NEPA) receive s extensive review through the “scoping” and interagency review processes that are part of NEPA, this is not usually considered peer review. If the underlying scientific and/or technical data, models, analyses, or work products are influential scientific information or a highly infl uential scientific assessment, then the relevant portions should be peer reviewed.

In general, the Agency’s role in preparing the NEPA document may suggest the extent of review the document should get (See sections 2.2.3 and 2.2.4). If EPA is developing the document as part of an EPA action/decision (EPA is the Lead Agency under NEPA), and the document is influential scientific information or a highly in fluential scientific as sessment, then it should receive independent peer review. If not, (the docum ent is not influential, will have little impact, is non-controversial, etc.), then peer input might be appropriate.

On the other hand, if EPA is reviewing an EIS from another agency (EPA is not the Lead Agency under NEPA), it is likely that we are reviewing the EIS for conflicts with EPA policy and general environmental concerns. In such a case, EPA should ask if th e underlying scientific and/or technical work product that supports the EIS has been peer reviewed to avoid concerns about the full credibility and soundness of the EI S based on the science and technical support. EPA should work with the other organization/agency to ensure that scient ific and/or technical work products receive peer review adequate for EPA purposes.Peer Review Handbook Page 41

2.2.16 Should Environmental Regulatory Models be Peer Reviewed?

Generally, yes. Specific guideline s for the peer review of envir onmental regulatory models have been published by the Agency. These can be found on the EPA web site under the Science Policy Council home page ( http://www.epa.gov/osa/spc/ ). In 2000, the Science Policy Council established the Agency’s Council for Regulat ory Environmental Modeling (CREM) which promotes consistency and consensus among enviro nmental modelers and users. The CREM is a good resource for you to contact for specific questi ons regarding peer revi ew of environmental regulatory models ( http://cfpub.epa.gov/crem/ ).

2.2.17 Should Another Organization’s Work Products That Have Been Submitted to EPA for Use in Decision Making be Peer Reviewed?

Any scientific and/or technical work product that is used in Agency decision making and is considered influential scientific information or a highly influential scientific assessment becomes a candidate for peer review regard less of whether the work produc t is produced by the Agency or another organization. Therefore, all work products important to EPA environmental decision making that are independently generated by other organizations (e.g., other Federal agencies, interagency groups, State and Tribal bodies, environmental groups, industry, educational institutions, international bodies) need to be considered as candidates for peer review.

If possible, when EPA knows that a work produc t is being generated by another organization and may be of interest to EPA for future use, the appropriate EPA office(s) should work with that organization, and others, as appropriate (e.g., the st ates), to promote the use of peer review. For example, the Office of Internati onal Affairs (OIA), as well as the impacted Program or Regional Office(s), should be included when internationa l products are being considered for EPA use.

It is hoped that if the other organization has the work product independently peer reviewed, the peer review will meet the intent of the Agency ’s Peer Review Policy and EPA’s proposed use of the product (i.e., the peer review is basically equivalent to what EPA would do). Agency staff from the appropriate office(s) should examine closel y the particulars of the peer review to ensure independence and a conscious effort to incorporat e the peer reviewers’ comments into the final work product. If there are perceived, or real, c onflicts of interest, this may preclude the use of that peer review and, in those instances, anothe r peer review would be needed. See Section 3.4.9 for considerations of when an outside party conduc ts and/or funds peer review of their own work product and submits it to the Agency.

If the outside organization does not have the work product peer reviewed and EPA decides it should be peer reviewed, the appropriate EPA offi ce(s) should ensure peer review of that work product occurs prior to the Agency’s use of th e work product in decision making. Peer review can be accomplished by asking the outside organization to do so, or, if it declines, EPA may conduct or arrange for the peer re view. If EPA is conducting or arranging the peer review, the product should be entered in the Science Inventory.Peer Review Handbook Page 42

2.2.18 Can Work Products That are Not Determ ined to be Influential Scientific Information or Highly Influential Scientific Assessments Still be Peer

Reviewed?

Yes. Scientific and/or technica l work products that are not categor ized as influential scientific information or a highly influential scientific assessm ent may nonetheless be candidates for peer review. For example, a project manager may deci de to use peer review because of particular program needs and goals. Peer review may also be warranted because it adds substantial value to the work product.

2.3 Determining Which Work Produc ts Do Not Receive Peer Review

2.3.1 What Information Is Exempt fr om the OMB Bulletin Provisions?

The following information does not need to be peer reviewed, even if it might be considered a highly influential scientific assessment or influential scientific information:

a) Information related to national security, foreign affairs, or negotiations involving international trade or treaties where peer revi ew would interfere with the need for secrecy

or promptness;

b) Information disseminated in the course of an individual adjudication or permit proceeding (including a registration, approval, licensing, site-speci fic determination), unless peer review is practical and appropriate and the information is scientifically or technically novel or likely to have preceden t-setting influence on future adjudications and/or permit proceeding;

c) Information involving a health or safety dissemination that is time-sensitive;

d) A regulatory impact analysis or regulatory fl exibility analysis subject to interagency review under Executive Order 12866, except fo r underlying data and analytical models used;

e) Routine statistical informati on (e.g., periodic demographic and economic statistics) and analyses of these data to comput e standard indicators and trends;

f) Accounting, budget, actuarial, and financial information, and;

g) Information disseminated in connection with routine rules that materially alter entitlements, grants, user fees, or loan programs, or the rights and obligations of recipients thereof.Peer Review Handbook Page 43

2.3.2 Are There Other Circumstances Wh ere Peer Review Is Not Necessary?

There may be circumstances in which it is no t necessary for a work product to undergo peer review. For example, peer revi ew is not generally conducted:

a) For work that has been previ ously reviewed in a manner consistent with OMB’s Bulletin and this Handbook. For example, a cancer risk assessment methodology or an exposure modeling technique that was the subject of ea rlier peer review of appropriate technical merit would not generally undergo additional pe er review, even if the product supported a significant Agency decision;

b) If an application of an adequately pe er-reviewed work product does not depart significantly from its scientif ic or technical approach;

c) When the scientific and/or technical methodologies or information being used are commonly accepted in the field of expert ise, e.g., many products supporting Control Techniques Guidelines (CTGs) and Effluent Limitation Guidelines (ELGs). Such use should provide the appropriate documentati on to support the comm only held view.

2.3.3 For Information That Is Not Exem pt, Are There Circumstances Where the

Peer Review Provisions of the OMB Bu lletin Can Be Waived or Deferred?

The Administrator may waive or defer the peer review provisions of the OMB Bulletin for influential scientific information or highly influential scientific assessments if there is a compelling rationale for the waiver or deferral. Th e use of waivers is expected to be limited to unusual and compelling situations not otherwise c overed by the exemptions, such as situations where unavoidable legal deadlines prevent full implementation of the Bulletin’s peer review provisions. According to the Bulletin, deadlines found in consent decrees will not ordinarily warrant waiver of the provisions because those d eadlines should be negotiated to permit time for conducting a peer review. Deferral of some or all of the peer review provisions may be an appropriate way to accommodate immovable deadlin es. If any of the OMB Bulletin provisions are deferred, peer review should be conducted as soon as practicable thereafter. Deferrals of peer review of influential scientific informa tion and highly influential scientific assessments should be approved by the Administrator.

If peer review of a work product is not pla nned, an explanation should be included in the database entry for that work pr oduct in the Science Inventory.

2.3.4 Does Publication in a Refereed Scient ific Journal Mean That Adequate Peer

Review Has Been Performed?

Scientific papers (articles) that are peer reviewed by a credible refereed scientific journal may not generally need to undergo further peer revi ew. However, you should expect that there mayPeer Review Handbook Page 44

be cases where EPA determines that a more rigor ous or transparent review process is necessary, (e.g., if a particular journal revi ew process did not a ddress questions that EPA determines should be addressed before using or di sseminating the information, such as the extent of uncertainty inherent in the findings, the sensitivity to model c hoice, or the applicabilit y of the data presented for use in the Agency’s models). Also see Section 2.4.4 for additional information.

2.3.5 Do Voluntary Consensus Standards Undergo Peer Review?

Generally, no. The National Technology Tran sfer and Advancement Act of 1995 (NTTAA) directs EPA to use available voluntary consensus standards in its regulatory activities unless to do so would be inconsistent with applicable la w or otherwise impractical. For purposes of the NTTAA, voluntary consensus standards are de fined as technical standards (e.g., materials specifications, test methods, sampling procedures , and business practices) that are developed or adopted by voluntary consensus bodies (such as IS O or ASTM). The general purpose of the NTTAA is to reduce private and governmen tal costs by avoiding having the government “reinvent the wheel” in the development of techni cal standards. Volunt ary consensus standards would normally not undergo peer review becau se the underlying process used by issuing organizations to develop and approve these sta ndards is generally considered adequate for purposes of the Agency’s peer review policy. EPA reserves the right to c onduct a peer review if it determines that the particular standard it want s to use is not a voluntary consensus standard for purposes of the NTTAA.

2.4 Choosing a Peer Review Mechanism

2.4.1 How Do You Determine the Appropriate

Peer Review Mechanism? The mechanism of the peer

review should match the importance and complexity of the work product.

During the planning of a peer review, the Decision Maker and the Peer Review Leader may consider several mechanisms for the peer review of scientific and/or technical work products. These options range from cons ultations with EPA colleagues not involved in developing the product to a larg e and formal panel of outside subject matter experts. The peer review effort might be a focused one-time evalua tion, or could encompass several examinations over the course of a project. In principle, peer review provides the greatest credibility for influential scientific informati on and highly influential scientific assessments when it involves well-qualified external reviewer s, is intensive in its examination, and operates through a more or less formal and transparent pro cess. As a practical matter, however, time and resource considerations in many cases impose lim itations on what can be reasonably achieved. Arranging for the most appropriate and feas ible peer review will involve good judgment regarding the extent to which th e peer review will improve the cr edibility of the product as well as consideration of substance, time, and resources. Developing a peer re view plan that provides for appropriate depth, timing, and content is an important matter for early consideration by the

Decision Maker and Peer Review Le ader. Note that use of peer input or public or stakeholder involvement does not constitute peer review.Peer Review Handbook Page 45

While the following information is described in more detail elsewhere, these are important considerations in determing a pe er review mechanism. The approach best suited to a specific work product will depend on the nature of the to pic and the intended final product. Generally, the more novel or complex the science or techno logy, the greater the cost implications of the impending decision, and the more controversial the i ssue, then the stronger the indication is for a more extensive and involved peer review and for external peer review in particular. Certain work products may clearly lend themselves to ex tensive external peer review; generally these will be products with large impacts (e.g., those that support Tier 1 and Tier 2 rulemakings). Other work products may not need a large scal e external peer review and may utilize a less involved, less resource intensive re view. The peer review of some products may be better served with some form of internal peer review or a combination of internal a nd external peer review.

It is important to make the choice of peer review mechanism at the time that the work is planned (for products supporting rule makings, at the analy tic blueprint stage) so that peer review costs and time can be budgeted into the work plan. Esse ntially, the level of peer review should match the impact and complexity of the work product. For example, a rule under development carries considerable weight and deserves careful handling and attention; therefore, the supporting work product deserves similar care and attention for it s peer review. Factors that need to be considered include: use of internal vs. external p eer reviewers; individual letter reviews vs. panel reviews; the number of review ers; the timing of the peer re view. No single peer review mechanism is likely to work best in all si tuations. Some usef ul guidance includes:

a) Influential scientific information intended to support the most important decisions, or work products that have special importance in th eir own right, external peer review is the approach of choice. Generally, the more complex, novel and/or controversial the product, or the higher impact it has, the more the D ecision Maker should consider implementing a peer review i nvolving external experts;

b) Highly influential scientific assessments ar e expected to undergo ex ternal peer review. When time and resources allow, panels are preferable;

c) Work products that are less complex, novel, or controversial, or have a lower impact might be subject to a less extensive, le ss resource-intensive review processes;

d) Group discussion with peer reviewers can be very helpful at some point in the peer review process as it allows interaction among peer reviewers with different perspectives and expertise. Public panels are more transp arent than closed discussions. On the other hand, simply soliciting indivi dual comments is easier, fast er, and less expensive. Individual review is probably more appropriate for peer review at the early stages of a product’s development or for products with less impact and complexity;

e) More reviewers are necessary for more comp lex projects (more disciplines) and for more controversial topics (differe nces in scientific perspective within a discipline).Peer Review Handbook Page 46

f) Strict time constraints, such as a cour t-ordered deadline, can make a less involved or formal peer review mechanism imperative. Decision Makers and Peer Review Leaders

should make maximum efforts to ensure that such a process is syst ematic and objective.

2.4.2 What are Some Examples of Internal Peer Review Mechanism?

a) Individual independent experts from within the Agency (e.g., ORD experts on non-cancer effects of lead (Pb) review a draft artic le on benchmark dose completed by a Program Office);

b) An ad hoc panel of independent experts from wi thin the Agency (e.g., an independent internal workgroup convened to examine the cas e for the classification of a chemical as a carcinogen);

c) Technical merit review by scientists in an Ag ency laboratory (e.g., an initial review of the risk assessment for a regional incinerator by Agency scientists).

2.4.3 What are Some Examples of External Peer Review Mechanisms?

a) Independent experts from outside the Agency (e .g., a letter review by outside scientists);

b) An ad hoc panel of independent experts outside th e Agency (e.g., a group is convened to develop a consensus on the carcinogenicity of a particular industrial chemical);

c) Agency-sponsored peer review workshops (e.g., a review of poten tial indicators of ecosystem damage);

d) Review by an established Federal Adviso ry Committee Act mechanism such as the Science Advisory Board (SAB), FIFRA Scientific Advisory Panel (SAP), ORD’s Board of Scientific Counselors (B OSC), or the Clean Air Scientific Advisory Committee (CASAC) - e.g., a review of a criteria documen t for a particular chemical risk;

e) Agency-appointed special board or commi ssion (e.g., a review of the risk assessment methodology prepared by the Clean Air Ac t Commission on Risk Assessment). Note:

The Office of General Counsel should be consulted regarding EPA's authority to

establish and finance the activit ies of a commission or board.

f) Review by the NAS (e.g., a review of the state of current knowledge about children’s health risks from pesticide exposures).

There are other bodies that may provide extern al commentary on Agency work products but are not considered peer review mechanisms, such as interagency committees (e.g., a reviewPeer Review Handbook Page 47

of prospective research plan s by the Committee on the Environment and Natural Resources coordinated by the White House), committees convened by another Federal agency or government organization (e.g., a review of the Dioxin Reassessment by the Health and

Human Services Committee to Coordinate Environmentally Related Programs), and/or reviews initiated by non-government al groups (e.g., a Society of Risk Analysis review of cancer guidelines).

2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?

Peer review of journal article s (written by EPA or non-EPA aut hors) performed by a credible, refereed scientific journal contributes to the scie ntific and technical credibility of the reviewed product. Generally, EPA considers peer review by such journals as adequate for reviewing the scientific credibility and validity of the findings (or data) in that article, and therefore, a satisfactory form of peer review 13 . However, in some cases, peer review of an Agency work product that uses these articles may be conducted (see Section 2.4.5).

EPA-authored journal articles, whether used in an Agency work product or not, are included in the Science Inventory to highlight the extensive work EPA produces in the scientific literature.

Prior to submitting an article to a journal for peer review, EPA employees are encouraged to have the article internally peer reviewed (see Sec tion 2.4.2); such internal peer review is already common practice in certain parts of EPA. Artic les may also need examination in accordance with any organizational clearance procedures, especially when th e author is presenting him or herself as an EPA employee. For EPA employees , conflict of interest law and policy will also apply.

2.4.5 Do Agency Work Products Become Ca ndidates for Peer Review when PeerReviewed Journal Articles are Used in Support of that Work Product?

In most instances, Agency wo rk products are candidates for peer review even when supported by peer-reviewed journal article(s). Although the use of articles that have been peer reviewed by a credible journal strengthens the scientific and technical credibility of any work product in which the article(s) appears or is referenced, it does not automatically eliminate the need to consider whether the work product itself shoul d be peer reviewed. In most cases, journal peer review may

not cover issues and concerns th at the Agency may want peer reviewed to support an Agency

action. Under these circumstances , the scientific and/or techni cal work product in which the article(s) appears or is referenced becomes a candidate for peer revi ew. A journal article authored by EPA employees should be used in the same manner as an article published by anyone else in a credible, well-recognized journal.

13 G uidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information Disseminated by the Environmental Protection Agency , EPA/260R-02-008 (October 2002), ( http://www.epa.gov/quality/informationguidelines )Peer Review Handbook Page 48

If an Agency work product is based solely on a single article that ha s received peer review by a credible journal (e.g., where a model is suggested for a singular use that fits a specific Agency need), peer review of the Agency work produc t may or may not be n ecessary depending on how closely you apply the findings from the article. If an Agency work product is based on two or more articles that have received peer review by a credible journal(s), the Agency work product generally should be considered fo r peer review. Deci sions to make (or not make) a work product a candidate for peer review should be documented in the peer review record.

One important factor to remember with regard to the use of articles that have received journal peer review concerns the availa bility of documentation from that peer review. Ideally, EPA should maintain a clear, easily access ible record of the peer review to ensure the credibility and validity of the peer review (see Section 2.5 for details on the peer review record). However, the documentation from a journal peer review may not normally be available to the Agency, so such documentation is not expected in the peer review record.

2.4.6 When and How Often Should Peer Review Occur?

The Decision Maker and Peer Review Leader have significant discreti on in deciding on the timing and the frequency of peer review. Op tions abound, each with merits depending on the context and specified peer review objectives. In many situations, a single peer review event, beginning when the final draft work product becomes available, is the approach taken. However, it is increasingly apparent that peer review performed earlier in the work product development stages provides a superior a pproach for some work products . There may be substantial incremental benefit to conducting more than one peer review dur ing the whole process of work product development, particularly where it i nvolves complex tasks, has decision branching points, or could be expected to produce controversial findings. In addition, early review could be beneficial at the stage of research design or data collection planning where the product involves extensive primary data collection. The Deci sion Maker and Peer Review Leader should determine when the peer review(s) should occu r, considering the type of work product under development and at what point in its development process a peer review would be most

beneficial.

Other types of work products that could benef it from early, up-front pe er review in their development are scientific and technical planning products. Examples of such products are research proposals, plans, and strategies. Also, while not products per se , ongoing research programs can be peer reviewed.

Remember though, that while more than one peer review can be beneficial, the distinction between peer input and peer revi ew should be kept in mind. E xperts providing input during the development or planning stages of the work pr oduct generally do not become peer reviewers of that product (see Sections 1.2.2 to 1.2.7 for full discussion on this distinction).Peer Review Handbook Page 49

2.4.7 What Factors are Considered in Setti ng the Time Frame for Peer Review?

Several factors impact how quickly a peer review may be desire d. These include deadlines for completion of a project, research program, or rulemaking, funding availab ility, availability of quality peer reviewers, the possible need to se ek public comment on the peer review product, and statutory and/or court-ordered deadlines.

Peer review sometimes leads to new inform ation and analyses. Reviewers may make

recommendations for new research that would alter the work product and thus modify the

scientific/technical basis for the action or rule it suppor ts. For this reason, a completed peer

review is desirable before issu ing any proposal for public comment. If that is not logistically possible because of court or statutory deadlin es, or other appropriate reasons, the Decision Maker should make every effort to complete th e peer review before the close of the comment period. Because peer review comments on such wo rk products could be of sufficient magnitude to warrant a revision to the proposed action or rule, Decision Makers should exercise diligence in completing the peer review prior to the proposal stage whenever possible.

Whenever feasible and appropriate , Offices should make a draft scientific assessment that is highly influential available to the public for comment at the same time it is submitted for peer review (or during the peer review process) and sponsor a public meeting where oral presentations on scientific issues can be made to the peer reviewers by interested members of the public. When employing a public comment process as part of the peer review, Offices should, whenever practical, provide peer review ers with access to public comments that address significant

scientific or technical issues. To ensure that public participat ion does not unduly delay activities, Offices should specify time limits for public pa rticipation throughout the peer review process.

2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the

Peer Review?

The organization of the Decision Ma ker is normally responsible fo r conducting the peer review. Responsibility for conducting a peer review can be negotiable when more than one Agency Office or Region or other agenci es are involved. Often, the degree of involvement by any of the organizations and agencies and their ability to fund peer review will determine who has the lead for the peer review.

2.5 Creating the Peer Review Record and Peer Review Reports

2.5.1 What is the Peer Review Record?

The peer review record is the formal record (f ile) of decision on the cond uct of the peer review, including the type of peer re view performed and an explanat ion of how the peer review

comments were addressed. It includes suffici ent documentation for an uninvolved person to understand what actually happened and why. Th e Peer Review Leader (with the programPeer Review Handbook Page 50

manager if there is one) creates a separate, clearly marked peer review file within the overall file for development of the work. Once the peer review is completed, it is the responsibility of the Peer Review Leader to ensure that the peer revi ew record is filed and maintained in accordance with the organization’s documen t retention procedures.

The Peer Review Record is separate from the entry in the Science Inve ntory Database. While some information from the peer review record appears in the database, the paper peer review record is the official record of the peer review.

2.5.2 How Can the Peer Review Record Improve the Peer Review Process?

A good peer review record allows future referenc e to what happened during the peer review, and helps Decision Makers make appropriate use of p eer reviewer input. In addition, a good record helps ensure that EPA’s Peer Review Policy is implemented. The Peer Review Leader is responsible for ensuring that the peer review record for individua l work products is collected and maintained until completion of the peer review effort.

2.5.3 What Should Be in the Peer Review Record?

The peer review record should include all materi als considered by the individual peer reviewers, the peer review report, and other input. Such materials include, at a minimum (see also Section 4.3.1):

a) The draft work product submitted for peer review;

b) Materials and information ( including the charge ) given to the peer reviewers; c) The peer review report, which summarizes the peer review fi ndings and contains information about the peer reviewers (such as reviewers’ names, affiliations, and a statement concerning poten tial conflicts and their re solution, if applicable);

d) Logistical information about c onduct of the peer review (suc h as times and locations of meetings);

e) A memorandum, or other written record, a pproved by the Decision Maker, responding to the peer review comments specifying accepta nce or, where thought appropriate, rebuttal and non-acceptance. The Office should prepar e a written response to the peer review report addressing each comment.

f) The final work product.

When deciding if particular materials should be included in the record, the Peer Review Leader

should consider whether the materi als would help reconstruct the pe er review process and results at a later time. If the materials may be helpful, they should be part of the peer review record.Peer Review Handbook Page 51

2.5.4 How and When Should You Develop a nd Post Peer Review Reports and Agency Responses for Influential Scientific Information and Highly

Influential Scientific Assessments?

Offices are expected to make peer review report s of the influential scientific information and highly influential scientific assessments publicly available to implement the provisions of the OMB Bulletin. Offices should instruct peer revi ewers to prepare a repor t that describes the

nature of their review and the nature of their findings and conc lusions. The peer review report should either (a) include a verb atim copy of each revi ewer’s comments (either with or without specific attributions) or (b) repr esent the views of the group as a whole, including any disparate and dissenting views, although attribution of commen ts to names is not necessary. The names of the reviewers and their organizational affiliations should be included in the report. For highly influential scientific assessments, the report should also include the charge to the reviewers and a short paragraph on both the credentials and re levant experiences of each peer reviewer.

EPA will post or provide a link to the peer re view reports on the Science Inventory website ( http://cfpub.epa.gov/si/ ) along with all materials related to the peer review (charge statement and Agency response). The credibility of the fina l work product is likely to be enhanced if the public understands how the Agency addressed the speci fic concerns raised by the peer reviewers. Offices should consider preparing a written response for inclusion in the peer review report. For highly influential scientific assessments, the OMB Bulletin explicitly calls for Offices to prepare a written response to the peer review report explaining (a) the agency’s agreement or disagreement with the views expressed in the re port, (b) the actions th at have or will be undertaken to respond to the report, and (c) the reasons the Office believes those actions satisfy any key concerns or recommendati ons in the report. These respons es will also be posted on the Science Inventory website.

2.5.5 What Should You Do with a Peer Review Record That Pertains to a Rulemaking Action?

The Peer Review Leader should coordinate with his/her program’s docket office to see that proper docketing procedures are followed for a peer review of a work product supporting a rule. If EPA relies on influential scientific information or a highly influential scientific assessment to support a regulatory action, the pr eamble should include a discu ssion of how EPA implemented the provisions of the OMB Bulletin . See Appendix C for a template to use for this purpose.

2.5.6 When Should the Peer Review Record Building Process Begin?

An early start at developing and maintaining a peer review record will help ensure the record is complete and helpful. Ideally, the record begi ns when the decision to peer review a work product is made. The Peer Review Leader should construct the peer review record from this point on – doing so will avoid pot entially time-consuming reconstr uction at a later point. Note that the peer review record is not complete until it contains a copy of the final work productPeer Review Handbook Page 52

which addresses the peer review comments. If a product is not peer reviewed, a record should also be created which explains w hy the product was not peer reviewed.

2.5.7 What are the Differences in Record Keeping for a Review by Individuals Compared to Review by a Panel?

Generally, the content of the peer review record w ould be the similar. In the case of a review by individuals, the peer review r ecord would typically contain ea ch individual's comments, while for a panel review, the record typically contai ns a summary or other synthesis documenting the panel's deliberations.

2.5.8 Where Should the Peer Review Record be Kept and For How Long?

During the active conduct of the peer review, the Peer Review Lead ers maintain the peer review record themselves until the peer review is complete. Minimall y, the file should be maintained until one year after the completed peer review is reported in the next annual reporting cycle. After that, the peer review r ecord should be maintained for a reasonable period of time, as delineated by the organization's document retent ion policy. Establishment and maintenance of the archive where the peer review records ultimately reside are an organization’s responsibility (i.e., not that of an individual program manager or Peer Review Leader). Generally, to allow flexibility, individual Offices a nd Regions should decide the appr opriate level of organizational responsibility and how to ensure th e record is routinely available. The peer review record may be kept with other reco rds relating to the overall project, as long as it is easily and separately

identifiable. There are also specific procedur es regarding the use of dockets for record-keeping; however,

these are not covered in this Peer Review Handbook. The documents contained in the peer

review record should be maintained in accordan ce with the Agency’s record-keeping retention schedule for such records. For details, see EPA’s National Records Management Program at http://www.epa.gov/records . One long-term archiving mechan ism may be the formal archiving

at the Federal Records Center.

2.5.9 Are Internal Peer Review Comments Included in the Peer Review Record?

To be considered a legitimate peer review, internal EPA peer reviews should be formally conducted and documented. Such a process would be consistent with the guidance found in this Handbook for planning, conducting, and completing a pe er review. When you follow this formal process to obtain peer review from EPA peers (s ee Section 1.5.9), then th e whole record of that internal peer review shou ld be included in the peer review re cord. This includes all the materials detailed in Section 2.5.3 (also see Section 4.3.1). Conducting a formal internal peer review is not the same thing as informal input from your EPA colleagues (i.e., “colleagues down the hall”), nor peer input from Agency personnel helping to develop the work product, nor organizational review and clearance processes. Such inputs from these informal processes should not be placedPeer Review Handbook Page 53

in the peer review record. Th e peer review record should cont ain only the information obtained when you conduct a formal internal peer review.

In some cases, an internal EPA peer review may be followed by a separate external peer review. In such a case, however, the external peer review will stand as the peer review of record since it is viewed as more independent in nature, has br oader fields of available expertise which can be brought to bear on the issues and often includ es greater depth for specific disciplines.

2.6 Budget Planning

2.6.1 What Budgetary Factors Should You

Consider in a Peer Review? Peer review is part of the normal cost of doing business.

Resources necessary for the implementation of the Peer Review Policy should be requested through the usual Agency budgetary processes. The budget formulation process within the Executive Branch is followed, after appropria tion bills are passed by Congress, by budget execution. These two processes pr ovide opportunities to secure resources for activities carried out by Headquarters and Regional offices, includi ng peer review. The work products which will undergo peer review should have adequate f unding requests for the coming fiscal year. Similarly, adequate funding should appear in th e actual approved operating plan to ensure the peer review can be conducted. For purposes of budget planning, the costs of peer review would include the FTE cost of staff, th e contract or other costs associated with the use of outside peer reviewers, and the administrative costs of c onducting a review (copying, tr avel expenses, etc).

2.6.2 What Input is Needed for the Annual Budget Formulation and Budget

Execution Process?

Senior Management in Program Office and Regions (including Decision Makers and budget officers) should ensure that budge t requests include anticipated reso urces for peer review. Peer

review should be considered as a normal part of doing busine ss. Peer review resource

considerations should also be a ddressed in the analytic blueprint for Agency rule-making actions.

2.7 Legal Considerations

2.7.1 What Are the Legal Ramificati ons of the Peer Review Policy?

The Peer Review Policy does not establish or a ffect legal rights or obligations. Rather, it confirms the importance of peer review where appropriate, outlines re levant principles, and identifies factors Agency staff should consid er in implementing the policy. Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicativ e procedures. EPA’s decision to conduct peer review in any particular case is wholly within the Agency’s discretion. Similarly, nothing in the Policy creates a legal requirement that EPA respond to peer re viewers. However, to the extent that EPAPeer Review Handbook Page 54

decisions rely on scientific and/ or technical work pr oducts that have been subjected to peer review, the remarks of peer reviewers should be included in the record for that decision.

2.7.2 Is Legal Advice Needed?

AA/RA staff and management should work regular ly and closely with individual OGC/Regional Counsel (RC) staff assigned to Agency activities. Peer Review Leaders should initially consult with their usual OGC/RC advisors for legal advi ce or referral. Headqua rters attorneys have specialties in specific areas and can be cons ulted as needed (e.g., FACA considerations, contractual responsibilitie s; ethics and potential conflicts of interest).

2.8 Federal Advisory Committee Act (FACA) Considerations

The Federal Advisory Committee Act, 5 USC A pp. 2, imposes certain open meeting (public announcement in the Federal Register), balanced membership, and chartering requirements (with the approval of the General Se rvices Administration (GSA)) befo re the Agency establishes, controls or manages an “advisory committee” fo r advice or recommendations. Peer review carried out by formal and established (charter ed) Federal advisory committees, such as the Science Advisory Board (SAB) or the FIFR A Scientific Advisory Panel (SAP), is always subject to FACA requirements. However, FACA c onsiderations do not a pply to every EPA and contractor-run peer review.

In the next Section you will find information on the applicability of FACA to EPA- or contractor-run peer reviews.

2.8.1 When Do FACA Requirements Ap ply to EPA-Run Peer Reviews?

In most cases, Federal Advisory Committee Act (FACA) requirements apply to EPA-run peer reviews that are conducted by formal and estab lished (chartered) Federal advisory committees, such as the Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel (SAP). These groups have the following characteristics:

a) The group is established, controlled, or managed by EPA;

b) The group has a fixed membership, esta blished purpose, and a set agenda; and

c) The group strives to produce collegial, rather than individual, advice to EPA.

EPA-run peer reviews that are not originally in tended to be subject to FACA, but which exhibit

the above characteristic s, may unintentionally become subject to FACA. Questions concerning the applicability of FACA to p eer review meetings should be a ddressed to the FACA experts in the Cross-Cutting Issues Law Office of OGC, or the appropriate Offi ce of Regional Counsel.Peer Review Handbook Page 55

2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA?

If EPA conducts a peer review with the purpose of obtaining advice from the individual peer reviewers and not for the purpose of obtaining a peer review product from the group (as a collective or consensus body), the peer review, in most cases, w ould not be subject to FACA. Peer review participants provi de only their own views or r ecommendations and do not vote nor do they provide collective or consensus recomm endations to EPA. When referring to the recommendations of the individual reviewers, EPA should not characterize these recommendations using terms such as “collective” or “consensus.” As a general matter, workshops and “letter reviews” th at seek individual views or co mments are usually not subject to the requirements of FACA.

In addition to ensuring that peer reviewers onl y provide comments as individuals, EPA officials may wish to lessen the potential for a challenge under FACA by seeking balanced participation at peer review meetings, and allowing interested members of the public to attend, and ensuring that they have access to appropriate materials.

Non-FACA meetings may be announced in the Federal Register (provi ding that it is clear in the notice that such meetings are not subject to FACA) as it prov ides the public with useful information and a point of cont act concerning the peer review. In addition, non-FACA (as well as FACA) meetings should al so be advertised via othe r avenues (e.g., the Web, local newspapers, and mailing lists).

2.8.3 How Do You Ensure that a Contractor-Run Peer Review Does Not Become Subject to FACA?

Committees (or other peer review groups) esta blished, controlled or managed by an outside organization (such as by an EPA c ontractor) to provide that outs ide organization with advice and recommendations (that will be submitted eventua lly to EPA as a contractor report) are not subject to FACA. Although FACA should not appl y to contractor-run peer reviews, there are things that you (i.e., EPA) can in advertently do that may invoke FACA.

The following are considerations that you should be aware of when a contractor runs a peer review (e.g., letter review, pa nel, workshop, etc.) for EPA:

a) The outside party’s peer review may be subject to FACA if EPA establishes, manages or controls the peer review gr oup (e.g., EPA selects panel member s, runs the meeting, etc.);

b) EPA should not provide contractor s with a draft agenda or sugge sted format for meetings. EPA contractors should manage and control th e process, including running any meetings;Peer Review Handbook Page 56

c) At the request of the EPA contractor, EPA can provide a briefing to the peer reviewers (e.g., in a conference call with the contractor on the line) on the history or background of the development of the document. EPA s hould only provide technical or background

information and not use the call to take over the contractor’s peer review group. Not only

should the contractor be on the li ne, but it should be very clear to all participants that the contractor is in charge of the call. The contractor, not EPA, should invite people to participate, make all administrative arrang ements, conduct the mee ting, and control the agenda;

d) EPA employees may attend the peer review panel meetings or workshops. However, they may not take over the control of the meeting. The contractor may call on them to speak when appropriate, but EPA staff s hould limit their participation to providing technical and/or background information, and no t attempt to, or appear to, take over the contractor’s meeting;

e) Since FACA does not apply when a contractor establishes, controls, or manages a peer review, the contractor need not avoid terms such as “collective” or “consensus” when reporting agreement among its peer reviewers (subcontractors);

f) EPA may provide comments to the contractor on the contractor’s peer review only to the extent that the Agency is verifying that th e contractor has satisf actorily completed the report in accordance with the work assignment. EPA should not attempt to make changes in the contractor’s conclusions; this woul d compromise the independence of the peer review conducted by the contractor.Peer Review Handbook Page 57



## 3. Conducting A Peer Review

3.1 Overview Statement

The success and usefulness of any peer review de pends on the quality of the draft work product submitted for peer review, the care given to the st atement of the issues or "charge," the match between the peer review draft pr oduct and the form of peer review, the match between the peer review draft product and the scient ific/technical expertis e of the reviewers, and Agency use of peer review comments in the final product. It is not simply enough to conduct a peer review; each of the foregoing elements deserves serious attention.

Figure 2 shows the order of activities for conduc ting a peer review. The charge should be completed before selection of the peer reviewer s to ensure that they have the appropriate expertise to address the questions raised.

3.2 Charge to the Peer Reviewers

3.2.1 What is a Charge?

As part of each peer review, the Peer Revi ew Leader formulates a clear, focused charge that

identifies the technical and scientific issues on which the Agency would like feedback and invites suggestions for improving the document as a whole. This request signals the Agency's receptivity to expert recommendati ons. The charge to peer reviewers usually makes two general requests. First, it focuses the review by presenting specific questions and c oncerns surrounding such issues as the comprehensiveness of the literature reviewed, the soundness of the method proposed, the scientific support for the assumptions employed, and the sensitivity of the results to alternative assumptions. Secondly, it invites general comments on the entire work product. The reviewers should be asked to ensure that the potential implications of the uncertainties for the technical conclusions drawn are clearly and transparently presented. The specific and general comments should focus on the scientific and technical merits of the work product and, where germane, whether the scientific/technical studies have been applied in a sound manner. Re member, the peer review is not for the decision or action itself, but for the underlying scientific and/or technical work product; reviewers should not be asked to provide advice on policy. Additi onally, Focused questions greatly simplify the task of collating, analyzing and synthesizing p eer review comments on a topical basis. The questions should be specific enough to encour age helpful comments, including constructive alternatives, but not so specific that they preclude creative responses (e.g., simply asking whether or not the reviewer agre es with the model choice or conclu sion). Understanding the reviewer’s Time is well-spent preparing a good charge, as the charge is crucial for an effective peer review.Peer Review Handbook Page 58

reasoning is pivotal to the Agency’s ability to address the reviewe r’s concern as well as crafting specific improvements to the work product. Th e peer reviewers' written responses to these questions help the Agency create a peer review record. As a general rule, time is well-spent preparing a good charge, as the charge is crucial for an effective peer review.

3.2.2 What are the Essential Elements of a Charge?

a) A brief overview or introduction (describe what the work product is, how it was developed, how it will be used)

b) As needed, a brief description or listing of any background materials provided to the peer

reviewers

c) The specific issues or questions to be a ddressed by the peer reviewer(s), including logistical details such as:

1) The due date of reviewers’ comments;

2) The format of reviewer responses (e.g., MS Word, Adobe Acrobat, e-mails, or text files);

3) The point of contact, in case peer reviewers have questions.

3.2.3 Where Can You Get an Example of a Charge?

Appendix D – Examples of Peer Review Charges, contains examples of successful charges that cover a variety of issues. Appendix E – Gu idance on Requesting a Review by the Science Advisory Board, provides guidanc e for obtaining Science Advisory Board (SAB) services.

3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?

Yes. EPA may decide to obtain stakeholder inpu t on the charge to the peer reviewers, but EPA

makes the final determination on what elements to include in the charge to ensure that it meets EPA’s needs for the peer review.

a) If you obtain stakeholder input, include interested parties to the extent feasible based upon statutory, regulatory, budgetary and/or time constraints. Do not limit input to one

stakeholder or one side of a controversial issue (e.g., a respon sible party or environmental group).

b) If EPA has hired a contractor to perform the peer review, it should still be EPA personnel who obtain stakeholder input and provide the lis t of charge questions to the contractor.Peer Review Handbook Page 59

c) If you form a committee of stakeholders to he lp develop the charge, be aware that your committee may become subject to the re quirements of FACA (see Section 2.8).

3.2.5 Who Writes the Charge When You Hire a Contractor to Conduct the Peer

Review?

In general, if EPA hires a contractor to perfor m the peer review, EPA should allow the contractor independence in conducting it. However, with regard to the charge, EPA should provide the

contractor with a substantive lis t of questions that EPA wants in cluded in the charge letter to ensure that the peer review m eets EPA’s needs. If the EPA pr oject manager has them ready, the list of charge questions can be in corporated into the Statement of Work. Based on this list, the contractor can then prepare and submit the actual ch arge letter to the peer reviewers. Prior to submitting the charge letter to the reviewers, the contractor should be directed to give EPA an opportunity to review the charge letter to ensure that it meets EPA’s needs. EPA cannot submit the charge directly to the peer reviewers when the review is being conducted by a contractor.

3.2.6 Is it Okay to Ask a Contractor to De velop the Charge to the Peer Reviewers?

No. EPA should provide the char ge questions to the contract or as discussed above. The contractor, though, may provide assistance and advice in the development of the charge.

3.3 Time Line

3.3.1 What are the Factors in Scheduling a Peer Review?

The peer review schedule is a critical feature of the process. The schedule should take into account the availability of a quality draft work product, availability of appropriate experts, time available for peer review comments, deadlines fo r the final work product, and logistical aspects of the peer review (e.g., contracting procedures).

The schedule for peer review should take into account the overall rulemaking (or other decision making) schedule. For rules, in particular those in Ti er 1 and Tier 2, the scheduling of the peer review should be included in the development of the analytic blue print. Peer review sometimes leads to new information and analyses, or recomme ndations for new research that would alter the work product and thus modify the scientific/techni cal basis for the action. For this reason, it is usually advisable to complete the peer review before taking public comment on the rule, or at least before the close of the public comment peri od. Note, however, that whenever feasible and appropriate, offices should make a draft highly influential scientific assessment available to the public for comment during the peer review proce ss, and, if feasible and appropriate, sponsor a public meeting where oral presenta tions on scientific issues can be made to the peer reviewers by interested members of the public. When employing a public comment process as part of the peer review, Offices should provide the reviewers access to the pu blic’s comments that address

scientific or technical issues.Peer Review Handbook Page 60

3.4 Selection of Peer Reviewers

3.4.1 What are Considerations for Selecting Peer Reviewers?

The choice of peer reviewers should be base d primarily upon on the reviewers’ expertise, knowledge, skills, and experience and should include specialists from multiple disciplines, as appropriate. The group of reviewer s should be sufficiently broad a nd diverse to fairly represent

the relevant scientific and technical perspectives and fields of knowledge; they should represent a balanced range of technically legitimate point s of view. Each peer reviewer should have recognized expertise that bear s on the subject matter under disc ussion. It may sometimes be appropriate, as in the case of FACA committees , to consider "affiliation" (e.g., industrial, academic, or environmental community).

The selection of independent peer reviewers is al so critical to an effective peer review. EPA should always make every effort to use peer re viewers who do not have an y conflict of interest or an appearance of a lack of impartiality, and who are completely independent (see Section 3.4.5). However, the very need to have experien ced individuals on a peer review, along with the desire to have appropriate techni cal balance and representation, m eans that the field of potential peer reviewers may include those that are less th an independent, have a conflict of interest, or might at least appear to lack impartiality. Sometimes selecting individu als who have served a variety of organizations rather than a single one for an extend ed period may provide expertise with diverse perspective. The emphasis on i ndependence and expertise applies equally to government experts and experts from the larger scient ific community

To ensure that the selected re viewers have the appropriate ex pertise, knowledge, skills, and experience, the charge to the reviewers should be determined in advance of the selection of the reviewers. Selection of peer reviewers s hould be made by identifying reviewers with the appropriate expertise and then narrowing the field of potential peer reviewers to those individuals that are independent, do not have a conflict of intere st and do not appear to lack impartiality.

You should also consider requesting that the public, including scientif ic and professional societies, nominate peer reviewers.

3.4.2 Where Do You Find Peer Reviewers?

Recommendations for potential pe er reviewers can be identified by a number of organizations. These include external groups such as the aff ected party(ies), special interest groups, public interest groups, environmental groups, profession al societies, trade or business associations, State organizations or agencies, Native American Tribes, colleges and universities, the National Research Council, and other Federal agencies with an involvement in or familiarity with the issue. Agency associated groups include the staff of the SAB or the SAP, and relevant scientific and technical experts from Program or Regional Offices.Peer Review Handbook Page 61

In certain circumstances, existing organizations such as the SAB, SAP or BOSC may be used to conduct a peer review. These groups establish th eir own criteria for accepting work. If you use such an organization, you should coordinate dire ctly with the organization (e.g., see Appendix E

- Guidance on Requesting a Review by the Science Advisory Board, for its procedures). SAB, SAP and BOSC conduct formal, pub lic, external peer reviews.

Another method which might be used to find peer reviewers is public so licitation. The Science Inventory can assist the public solicitation process by announc ing the opportunity for public nominations and by providing background on the re view topic the public may find useful for identifying potential reviewers they may wish to nominate. EPA’s Scien ce Advisory Board uses

the Internet to solicit names for both ad hoc and standing advisory committees. These names, along with short biographical sket ches, are also posted so that the public may not only nominate, but also comment on potential advisory co mmittee members. See “Advisory Committee Meetings and Report Development: Process for Public Involvement” available from SAB’s

website at http://www.epa.gov/sab/pdf/sabso_04_001.pdf

Occasionally, a member of the scientific community will offer his/her services for peer review during an ongoing peer review. These offers may be at no cost or based on an expectation that

reimbursement will be made. Di sposition of these unsolicited offers should be handled on a case by case basis by the Peer Review Leader, and as n ecessary, in consultation with the Peer Review Coordinator, the Office of Ge neral Counsel (OGC), and a ppropriate Decision Makers.

If you use a contract mechanism to conduct a peer review, the contractor may have its own pool of scientific and technical experts for peer review. EPA may provi de contractors with information on potential peer reviewers for conducti ng a peer review, if such a listing is prepared in alphabetical order. EPA should not require that the contractor select from a prepared list, nor require that the contractor receive EPA approva l before selecting any given peer reviewer (sometimes known as a “subcontractor”). Howe ver, EPA should review the list of peer reviewers for conformance to work assignment specifications and adhere nce to conflict of interest and appearance of a lack of impartia lity concerns. Furthermore, when utilizing a contract mechanism to conduct peer review, EPA is not permitted to direct the prime contractor to a specific subcontractor (or peer reviewer), nor is EPA permitted to direct the peer reviewer (subcontractor). All inte ractions with the peer reviewers are to be coordinated through the prime

contractor. See Section 3.6 for further information.

Keep in mind that for contracting purposes, contractors are required to obtain Contracting Officer (CO) approval of subcontra ctors and the CO generally seeks the input of the Contracting Officer Representative (the COR may be the Peer Review Leader in many cases) before approving the use of subcontractors. In this case, as noted above , the Peer Review Leader should review the list of potential revi ewers for conformance to the work assignment specifications and adherence to conflict of interest and appearance of a lack of impartiality concerns.Peer Review Handbook Page 62

3.4.3 Are External or Internal Peer Reviewers Preferred?

External peer reviewers are generally preferred for influential scientific information, and are expected for highly influential scientific asse ssments. For some work products, like those reviewed at interim steps, either external or internal peer revi ew may be appropriate. Selection of internal peer reviewers should be based upon technical expertise, available time and affiliation. That is, they should not come fr om the same Office or group producing the product or have any other connection with the product or document being pe er reviewed. The choice of reviewers should be analyzed on a case-by-case basis. Reviewers employed by other Federal and State agencies may possess unique or indispensa ble expertise. Don’t forget that for highly influential scientific assessments, the use of inte rnal peer reviewers is constrained by the OMB Bulletin. External peer reviewers should be se lected based upon techni cal expertise as well, however, you should not use individuals who have b een involved in the development of the work product. (See Section 1.2.6; see also Sections 1.2.7 to 1.2.9).

3.4.4 What is Important in the Mix of a Peer Review Panel?

A peer review panel or group can number from ju st a few individuals to ten or more, depending on the issue, the time and resources available, and the broad spectrum of expertise necessary to treat the range of issues/questions in the charge . Objective technical e xpertise and no conflict of interest or appearance of a lack of impartiality are critical characteristics for selecting peer reviewers. Naturally, experts whose understand ing of the specific technical area(s) being evaluated are necessary; nevertheless, it is also important to include a broad enough spectrum of other related experts to completely evaluate th e relevant impacts on ot her less obvious concerns (i.e., to comment not only if the job is being done right, but also whether the right job is being done). For example, for health related peer re views, experts in such fields as ecology and economics may provide very useful insights. Al though persons who are familiar with and have a substantial reputation in the field are often called upon repeatedly to be reviewers, it is important to keep a balance with new people who bring fresh perspectives to the review of a work product. The idea here is to avoid the repeated use of the same reviewer on multiple assessments unless his/her participation is essential and cannot be obtained elsewhere.

There is usually a continuum of views on any issue. To the extent practicable, selected experts should have technically legitimate points of view that fall along the continuum. A review panel should include experts that are considered “mains tream” (nearer the center of the continuum) as well as those further along the continuum while generally avoiding the extremes. This will help maintain a balanced review panel, while allowi ng a broad range of views to be expressed and discussed. A balanced panel wi ll allow consensus building if consensus is the object of a particular peer review; if not, it provides a spectrum of views for the Agency to evaluate. As such, the group of reviewers should be sufficientl y broad and diverse to fairly represent the relevant scientific and technical perspectives and fields of knowledge.Peer Review Handbook Page 63

As a general rule, experts who have made public pronouncements or have had a predominant influence on the position for a given organizatio n on an issue, those who have clearly "taken sides," may have an appearance of a lack of im partiality (see “Ethical Standards” below) and

should be avoided.

3.4.5 Ethical Standards

As discussed previously (see Section 2.4), ther e are various mechanisms through which peer reviews may be accomplished. Internal peer reviews can be conducted by independent experts from within EPA, either individually or as ad hoc peer review panels. Ex ternal peer reviews can be conducted by individual experts (e.g., through letter reviews) w ho are either Regular Government Employees (RGEs) of another agen cy, experts hired as EPA Special Government Employees (SGEs), or experts whose services are procured through a contract mechanism. In addition, external peer review panels can be convened throug h a contract mechanism or under the auspices of a Federal A dvisory Committee. Finally, p eer reviews may be conducted by outside organizations such as the Na tional Academy of Sciences (NAS).

To ensure a credible peer revi ew process, high ethical standard s are applied to each of these

mechanisms. These standards are embodied in the various laws, re gulations, and other requirements that apply to peer reviewers who are either RGEs, SGEs, contractors, or who are employed by outside organizations (e.g., NAS).

There are specific regulations that describe ethics considerations for contractor personnel. The Federal Acquisition Regulations (FAR), the Environmental Protection Agency Acquisition Regulations (EPAAR), and other internal Agency documents define ethical standards, such as Conflict of Interest (COI), and de scribe the Agency’s policies and procedures. This Peer Review Handbook provides some guidance regarding the applic ation of this information to peer review (see Section 3.6 and Appendix F – Example Stat ements of Work for Contracts); however, specific questions should be addressed to the Office of Acquisition Management.

For peer reviews conducted by outside organizations, the peer review leader or other appropriate official should be thoroughly familiar with the ethics policies and requirements of the organization conducting the review. For exampl e, the NAS National Research Council has published their “Policy on Comm ittee Composition and Balance a nd Conflicts of Interest.”

The remainder of this section is focused on the laws and regulations governing the standards of

ethical conduct for RGEs and SGEs. The st andards for government employees are very thorough and, as such, may help the peer review l eader or other appropriat e official to identify issues that may be more generica lly of concern with respect to the selection of peer reviewers.Peer Review Handbook Page 64

Conflicts of Interest:

Public service is a publi c trust. As such, it is vitally important that government employees maintain the highest ethical standards. While th ere are many aspects of ethical behavior, for the purposes of peer review we are mostly concerned with compliance with applicable Federal ethics statutes and regulations. The ba sic regulations are issued by th e Office of Government Ethics (OGE) and are occasionally supplemented with additional regulations by EPA.

Compliance with Federal ethics re gulations can be a confusing a nd complex process. However, the process is usually concerned with determin ing whether a RGE or a potential SGE has an ethics issue with respect to a “conflict of intere st” or “an appearance of a lack of impartiality.” Each of these two concepts will be explained in more detail below. However, sometimes the two concepts are confused. A “conflict of interest” is concerned with matters of financial interest.

There is either a conflict of in terest or there is not. An “app earance of a lack of impartiality” (sometimes mischaracterized as a “potential c onflict of interest”), can be financial or not

financial in nature. The determ ination of an “appearance issue” is not quite as clear-cut as a “conflict of interest” and calls for more of a judgment on the part of the determining official as to

whether one exists.

Let’s look at each of these two concepts in detail.

Generally, a conflict of interest arises when the person’s financial in terests are affected by his/her participation in a particular matter, when he/she, his/her associat es or other individuals whose interests are imputed (as provided unde r Federal law) would derive benefit from incorporation of their po int of view in an Agency product, and/or when their professional standing and status or th e significance of their principal area of work might be affected by the outcome of the peer review. Whenever there ar e questions about conflicts of interest, you should contact the appropriate Deputy Ethi cs Official, Ethics Assistant or the Designated Agency Ethics

Official for clarification and assistance. For a list of EPA's Deputy Ethics Officials, see http://intranet.epa.gov/ogc/ethics/deos.htm

.

For Financial Conflict of Intere st (COI) issues, the basic 18 USC § 208 provision states that:

An employee is prohibited from participati ng personally and substantially in an official capacity in any particular matter in which he, to his knowledge, or any person whose interests are imputed to him under this statute has a financ ial interest, if the particular matter will have a direct and predictable e ffect on that interest.

For a conflict of interest to be present, all elemen ts in the above provision must be present. If an element is missing, it is unlikely that there is a conflict of interest issue. However, the regulations concerning an appearance of a lack of impartiality may still apply and need to be

considered.Peer Review Handbook Page 65

One very important aspect to consider is whether a charge to an advisory committee or other peer review panel constitutes a “particular matter.” A “particular matter” refers to matters that will involve deliberation, decision, or ac tion that is focused upon the inte rests of specific people, or a

discrete and identifiable class of people. It does not refer to c onsideration or a doption of broad policy options directed to the in terests of a large a nd diverse group of people. A particular matter of general applicability means a particular matter that is focused on the interests of a

discrete and identifiable class of persons, but does not involve specific parties. A particular matter involving specific parties means any judici al or other proceeding, application, request for a ruling or other determination, contract, claim, controversy, investigation, charge, accusation, arrest or other particular matter involving a specific party or partie s in which the United States is a party or has a direct and substa ntial interest. Participating pers onally means direct participation in the matter to be reviewed. Participating substantially refers to involvement that is of significance to the matter under consideration. For mo re information regarding the definition of a particular matter, see the Office of Government Ethics document “Standards of Ethical Conduct for Employees of the Executive Branch” available at http://www.usoge.gov

.

Another consideration is whether there will be a direct and predictable effect on the financial

interests of the reviewers. A di rect effect on a participant's financial interest exists if a close causal link exists between any d ecision or action to be taken in the matter and any expected effect of the matter on the financial interest. A pa rticular matter does not ha ve a direct effect if

the chain of causation is attenuated or is co ntingent upon the occurren ce of events that are speculative or that are independent of, and unrelated to, the matter. A particular matter that has an effect on a financial interest only as a conseq uence of its effects on the general economy is not considered to have a direct effect. A predictable e ffect exists if there is an actual, as opposed to a speculative, possibility that the matter will affect the financial interest.

The question of whether a charge is a particular matter involving a specific party or parties or whether it is a particular matter of general applicability has impor tant ramifications for reviewers who are Special Government Employees (SGEs) as defined under 18 USC § 202(a). SGEs serving on FACA (Federal Advisory Committ ee Act) committees are covered by certain exemptions from the conflict of interest stat utes. The most signifi cant of these is 5 CFR 2640.203(g), which pertains to certain financial interests arising from an SGEs’s outside employment. Specifically, this exemption permits SGEs serving on FACA committees to participate in particular matters of general applicability (e.g., development of general regulations, policies or standards) where the disqualifying interest aris es from the SGEs non-Federal employment or prospective empl oyment (but NOT due to financ ial holdings or consultancies).

Furthermore, the question of whether a charge is a particular matter involvi ng a specific party or parties, or whether it is a part icular matter of genera l applicability, has other ramifications for SGEs (and other Federal employees for that matter). For example, if a charge is a particular matter involving a specific party or parties, there are certain restri ctions regarding obtaining U.S. Government contracts/grants as well as future representation back to the Federal government. For more information, see 18 USC § 207 or contact your Deputy Ethics Official/Ethics Assistant.Peer Review Handbook Page 66

Even if an SGE is determined to have a conf lict of interest, that does not mean that it is impossible for her or him to participate in an advisory committee or other peer review. For example, in some cases it is possible to obtain a wa iver for her or his part icipation. It is also possible to have a particular SGE be recused from certain portions of a review that would

otherwise create a confli ct of interest. However, you shoul d always weigh the advantages and disadvantages of this approach ve ry carefully. In addition, public disclosure of these conflicts in a manner compliant with the Privacy Act and othe r applicable regulations and practice, will add to the integrity of the advisory committee and p eer review process. Additional information for SGEs may be found in “Conflict of Interest and the Special G overnment Employee: A Summary of Ethical Requirements Applicable to SGEs” av ailable from the U.S. Office of Government Ethics (U.S. OGE) at http://www.usoge.gov

.

Appearance of a Lack of Impartiality:

The Code of Federal Regulations at 5 CFR § 2635.502(a) states that:

Where an employee knows that a particular matter involving specific parties is likely to have a direct and predictable effect on the financial interest of a member of his household, or knows that a person with whom he has a covered relationship is or represents a party to such matter, and where the person determines that the circumstances woul d cause a reasonable person with knowledge of the relevant facts to question his impartiality in the matter, the employee should not participate in the matter unless he has informed the agency designee of the appearance problem and received authorization from the agency designee.”

Further, § 2635.502(a)(2) states that,

An employee who is concerned that circumstanc es other than those sp ecifically described in this section would raise a question regardi ng his impartiality should use the process

described in this section to determine whether he should or should not participate in a particular matter.

Even though circumstances for specific candidates ma y not raise an issue of a conflict of interest, each advisory committee member or peer review er should be evaluated to ensure that an appearance of a lack of impartiality does not preclude their participation.

How do you evaluate candidates for appearance i ssues? In addition to collecting confidential financial disclosure forms (described below) a nd public comment (where appropriate), it may be necessary on occasion to seek additional info rmation from the potential advisory committee member. For example, you might want to consider asking additional questions which will help you to determine whether there is an appearance issue. Some suggested questions would be:Peer Review Handbook Page 67

a) Do you know of any reason that you might be unable to provide im partial advice on the matter to come before the Panel or any reas on that your impartiality in the matter might be questioned?

b) Have you had any previous involvement with the review document(s) under consideration? If so, please iden tify and describe that involvement.

c) Have you served on previous advisory panels , committees or subcommittees that have addressed the topic under c onsideration? If so please identify those activities.

d) Have you made any public statements (written or oral) on the issue? If so, please identify

those statements.

e) Have you made any public statements that w ould indicate to an observer that you have taken a position on the issue under consideration? If so, pleas e identify those statements.

If an SGE has an appearance of a lack of impart iality, it may still be possible to make a written determination that she or he may serve on an advi sory committee or to ensure that a particular advisory committee member or peer reviewer is r ecused from certain areas of a review. See your Deputy Ethics Official/Ethics Assistant for more information.

A Few More Ethical Considerations:

In addition to the above considerations, you s hould check for a few more items to ensure compliance with Federal ethics laws:

Restrictions on Representation:

There are two statutes (18 USC § 203 and 18 USC § 205) that impose related restrictions on the

outside activities of RGEs and SGEs, particularly invol ving activities involving the representation of others before the Federal government. Secti on 203 prohibits an employee from receiving, agreeing to receive, or soliciting comp ensation for representation services before any court or Federal agency/entity in connection wi th any particular matter in which the United States is a party or ha s a direct and substantial interest. Section 205 prohibits an employee from personally representing anyone befo re any court or Federal agency or other specified Federal entity, in connection with any particular matter in which the United States is a party or has a

direct and substantial interest.

Fortunately, these restrictions are limited in th eir application to SGEs . One of the most significant limitations is that SGEs are restrict ed only in connection wi th particular matters involving specific parties. Furthe rmore, the restrictions on SGEs are narrowly drawn to focus only on those matters in which the SGE actually pa rticipated for the Government, as well as, in some cases, those matters actually pending in the SGE’s own agency (in our case, EPA).Peer Review Handbook Page 68

Emoluments Clause of the U.S. Constitution:

The Department of Justice (DOJ) originally held that SGEs may be subject to the “Emoluments Clause” of the United States Constitution (Article I, Section 9, clause 8) which prohibits persons who “hold office of profit or trust” in the Federal government from having any position or receiving any payment from a foreign government . However, an opinion from DOJ/Office of Legal Counsel (March 9, 2005) states that SGEs who do not have access to classified information and who serve in purely advisory positi ons do not hold an “office of profit or trust” in the Federal government and are not subject to the Emoluments Clause. Therefore, it is extremely unlikely that any SGE performing a p eer review for EPA would be subject to the Emoluments Clause. As always, check with your Deputy Ethics Officia l/Ethics Assistant if you

need more information.

Putting It All Together:

The fine art of determining whether an advisory committee member or other peer reviewer has a conflict of interest or an appearance of a lack of impartiality can be difficult, confusing and complex. Needless to add, it is also important that any decision that is made concerning advisory committee members or peer reviewers be appropriately documented and all applicable Federal record regulations and Privacy Act re gulations be strictly adhered to.

The matter of obtaining a fair and credible peer re view, as well as maintaining the credibility of the Agency and the Agency’s scientific products , is of paramount importance. Peer review leaders are strongly encouraged to obtain peer reviewers who do not have a conflict of interest or an appearance of a lack of impartiality. Finding a totally independent peer re viewer is a difficult and often daunting task. See Section 3.4.6 for additional steps to take to ensure a credible peer

review.

For additional information on putting together an advisory panel (e.g., for highly influential scientific assessments), please refer to the SAB brochure “Overview of the Panel Formation Process at the Environmental Protection Agency Science Advisory Board, September 2002" at http://www.epa.gov/sab/pdf/ec02010.pdf

.

3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of

Conflicts of Interest?

Before finalizing the selection of reviewers, the Peer Review Leader should ascertain whether each potential peer reviewer’s involvement in cer tain activities could pose a conflict of interest (COI) or create an appearance of a lack of impa rtiality. One way of id entifying conflicts is to ask potential reviewers about work and clients (both current and prior) that might create conflicts or the appearance of a lack of im partiality in carrying out peer re view activities. When the peer review process is being conducted by a cont ractor, the requirement for addressing peer reviewers’ possible conflicts of interest should be highlighted in the Statement of Work or thePeer Review Handbook Page 69

work ordering instrument (i.e., Work Assignment, Delivery Order, Task Order, etc.) and is a matter that is bound by contractual clauses with the Contracting Officer as the final Decision Maker in contracting matters.

You should take care to avoid ethical issues with respect to advisory committee members or other peer reviewers and the work product under review. Rememb er, each situation is unique and should be treated on a case-by-case basis. The following are some considerations that

should be addressed in ev aluating ethical issues:

a) Paying attention to the employment, financia l, and professional affiliations of the participants;

b) Exploring the issue directly with each particip ant before the review process takes place;

c) Disclosing publicly at the be ginning of meetings any previous involvement with the issue;

d) Obtaining confidential financial disclosure fo rms. In the cases of regular government employees (RGEs), use either the OGE Form 450 or the SF-278 Form as appropriate. In the case of Special Government Employees (SGEs), use EPA Form 3110-48 (Confidential Financial Disclosure Form fo r Special Government Employees Serving on Federal Advisory Committees at the U.S. E nvironmental Protection Agency). All these must be filed annually (with some exceptions ). Remember, you can always discuss any conflicts of interest or appearances of a l ack of impartiality with your Deputy Ethics Official, Ethics Assistant or Deputy Agency Ethics Official as appropriate, but it is important to remember that there is no “atto rney-client privilege” for these discussions;

e) Collecting additional information through pub lic comment, additional ethics questions posed to potential advisory panel me mbers and other appropriate means;

f) In the case of non-Federal peer reviewers not on a FACA advisory panel (e.g., SAB, SAP or BOSC), providing them a copy of the peer re view COI inquiry. This form is sent to each prospective peer reviewer by the Peer Revi ew Leader (or contractor, in the case of contractor-run reviews) to advi se them of the need to addr ess COI issues prior to the actual review taking place. A follow-up c ontact with the Peer Review Leader (or contractor, in the case of cont ractor-run reviews) is then ma de to discuss any relevant

issues. The Peer Review Leader then documents this effort in the peer review record; this includes a summary provided by th e contractor documenting their inquiries and efforts.

The Peer Review Leader should ensure that th e peer review COI inquiry took place and that it appears in the peer review record.Peer Review Handbook Page 70

However, it should be strictly understood that Fede ral records maintained on individuals (defined as any item, collection, or grouping of information about an individual that is maintained by an agency, including, but not limited to, her or his education, financial transactions, medical history, and criminal or employment history and that contains his name, or the identifying number, symbol, or other identifying partic ular assigned to the individual , such as a finger or voice print or a photograph) are protected by the Privacy Ac t of 1974 (5 USC § 552a - Records maintained on individuals). A copy can be found at http://www.usdoj.gov/04foia/privstat.htm . This Act states that no agency shall disclose any record which is contained in a system of records by any means of communication to any person, or to another agency, except pu rsuant to a written request by, or with the prior writ ten consent of, the individual to whom the record pertains. There are certain exceptions (e.g., pu rsuant to certain civil or crim inal law enforcement activity).

Thus, the peer review leader s hould ensure that any documentati on in the peer review record does not contain information subject to the Priv acy Act. Furthermore, Peer Review Leaders should ensure that any applicable EPA record-kee ping procedures are followed (for details, see EPA’s National Records Management Program see h

, http://www.epa.gov/records/ ). For

information on Freedom of Information Act (FOIA) requests, please ttp://www.epa.gov/foia . If you have any questions, be sure to ask your appr opriate Office of Genera l Counsel and/or Office contract official(s).

Established peer review groups such as the Sc ience Advisory Board (S AB), the Scientific Advisory Panel (SAP) and the Boar d of Scientific Counselors (BOS C) provide useful models for addressing balance and ethics issu es. Assistance in identifying conflicts of interest/appearance of a lack of impartiality and in providing an appropriate response can be obtained from the Deputy Agency Ethics Official (l ocated in the Office of the Gene ral Counsel) as well as any of the Agency’s deputy ethics offici als (DEOs) or Ethics Assistants . For assistance in evaluating conflicts of interest or appearance issues in a contractual situation, obtain the involvement of the Contracting Officer and the resources availabl e within the Office of Acquisition Management.

Of course, ethical issues do not necessarily arise merely because a peer reviewer knows something about the subject matter. In f act, experts with a stake in the outcome – and therefore a conflict or an appearance issue – may be some of the most knowledgeable and up-to-date experts because they have concrete reasons to maintain their expertise. Such experts could be used provided conflicts of interest ar e properly disclosed and appropria te waivers from the Office of Government Ethics are obtained or, in the case of an appearance of a lack of impartiality, proper disclosure is made. In some cases, however, the conflict or appearance issue may be so direct and substantial as to rule out a particular expe rt. For instance, a potential peer reviewer may have a client or employer with a direct financia l stake in the particular specific party matter under review, such as a Federal grant or contract to the potential peer reviewer or his/her employer that relates to the matter under review or the potential peer reviewer’s or their company’s work on a specific chemical under review. However, review of a general methodology that applies to numerous chemicals w ould not necessarily ra ise such a concern.Peer Review Handbook Page 71

( Note : COI language should be made part of cont racts/statements of work (SOW) or purchase orders (PO) – see Section 3.6.4.)

A Peer Review Leader may also wa nt to adopt measures that will prevent creation of conflicts as the peer review is underway. Any measures cont emplated that involve a contractual action must be coordinated with the cognizant Contracting Offi cer. Some measures might include clauses in a contract or purchase order that require review ers to receive advance approval on future work, or place limits on such work, while they are performing the current peer review. Note that at some level these types of measures will discourage experts from serving as peer reviewers. (See Section 3.6.4 for further information dealing w ith contracts and suggestions for appropriate management controls.)

3.4.7 Are There Constraints to Selecting Peer Reviewers?

Sometimes the schedule for a peer review is accelerated due to a court-ordered deadline or other time-sensitive requirements. In such cases, it is difficult, if not impossible to obtain external peer reviewers in time to conduct a full external peer review. It may even be impossible to conduct a small scale internal peer review using just a fe w individuals. Mechanisms for identifying and using a small number of peer reviewers should be de veloped so that quick, effective peer review can be included for even the most rapidly moving products.

Another possible constraint i nvolves confidential business information (CBI). There are different definitions and type s of CBI, depending on the statut e that governs your action (e.g., Toxic Substances Control Act). To evaluate certain Agency-generated studies properly, some peer reviewers may need access to CBI. However, unless the re viewers are Federal employees with CBI clearance, the Agency does not have the independent authority to disclose CBI to them. Therefore, whenever contemplating the use of outside peer reviewers, Agency staff should

determine whether the reviewers will need access to CBI. If they do not have CBI clearance, the

Office of the General Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to disclose the information to peer reviewers.

Offices need to be aware of the requirements of the Federal Advisory Committee Act (FACA) when establishing peer review mechanisms (see S ection 2.8). Federal ad visory committees that are subject to chartering by the General Services Administration must hol d meetings that are open to the public, and have balanced membership requirements. The Office of the General Counsel (OGC) should be consulted regarding the a pplicability of FACA to peer review panels.

Occasionally, you may want to appoint a non-U.S. c itizen to an advisory committee. Citizens of other countries may be appointed as represen tative members of an advisory committee. However, whether a non-U.S. citi zen may be appointed as a sp ecial government employee (SGE) is a complicated question involvi ng the Appropriations Act.Peer Review Handbook Page 72

Section 605 of the General Provisions of the Tr easury and General Gove rnment Appropriations Act provides, with some exceptions, that no appropriations can be used "to pay the compensation of any employee of the US Government" unless the person is a U.S. citizen or has filed a declaration that they intend to become a U.S. citizen. There are a few exceptions to this restriction. One exception is for "n ationals of those countries alli ed with the United States in a

current defense effort." The a ppropriations restriction also pr ovides that citizens of specific countries, refugees from certain countries, and st udents from China also are not subject to the restriction. If you want to hire a foreign national as an SGE, first contact OGC so that they can determine whether any exception applies to th e person you are considering. Generally, to determine whether a country is allied with the U. S. in a current defense effort, OGC will consult with the Department of State.

If the foreign nation in question does not have a qualifying defens e relationship with the U.S. and no other exceptions apply to the individual, we may still hire the person as an SGE, but they would have to agree in writing to forgo a ny compensation. For an SGE serving without compensation, we may reimburse the individual for his/her expenses (trave l and per diem), but we may not make payments for "home work time" or time spent at advisory committee meetings. The main point is to contact OGC if a foreign na tional is being considered for a FACA so that OGC can determine whether the person may be compensated.

3.4.8 If State Employees Are Used as Peer Reviewers, Can EPA Pay Them for This

Service?

In some cases, this may be possible. First, th e Peer Review candidate should determine if the State agency has a policy on whether its employees can perform this type of work, and whether the State policy will allow compensation. In most instances, EPA can also pay travel expenses (consult with your administrative st aff for details). If the State person is not being paid for their peer review services, they must sign an agreemen t stating that they do not expect payment (see Section 3.6.1 – Gratuitous Services).

3.4.9 Does the Agency Recognize Peer Reviews That External Parties Conduct and/or Pay For?

There may be instances where par ties external to EPA will want to conduct and/or pay for a peer review on a particular work product. This may l ook benign at first blush, but is a very complex

and sensitive situation that can raise significant concerns for conflicts of interest or an appearance of a lack of impartiality for interested parties “paying” for a peer review of their own work product. While the Agency cannot prevent external parties from conducting and paying for a peer review, it is desirable that any such peer review is consistent with the intent of the Agency’s Peer Review Policy and implements th e principles and guidance in this Handbook. If the external party submits their work produc t and accompanying peer review, the materials should be treated by the Agency as anything el se submitted for the Agency’s evaluation (i.e.,Peer Review Handbook Page 73

evaluation for scientific credibility and validity, as well as consistency with the provisions of the Peer Review Handbook).

3.5 Materials for Peer Reviewers

3.5.1 What Instructions Do You Give Peer Reviewers?

The Peer Review Leader is responsible for ensuring that peer revi ewers understand their responsibilities (see Sections 3.2 a nd 3.6 if a contract is involved):

a) Advise the Agency of actual or potential organi zational or personal conflicts of interest or other matters that would create the appearan ce of a lack of impartiality (see Section 3.4.6);

b) Provide written comments in specified format by the specified deadline that are responsive to the charge;

c) Comply with any requests for not disclo sing draft work products to the public;

d) Inform on the planned extent of disclosure of names and attribution of comments.

3.5.2 What Materials Should be Sent to Peer Reviewers?

For a peer review to be successful, peer revi ewers should receive several documents at the beginning of the process. Typically, the most important among these documents are the charge letter and the current work produc t. The charge letter describe s what the peer reviewers are being asked to do, and should serve to focus and st ructure the review. Th e work product is, of course, the material being subject to peer review.

Remember, documents should not be provided direct ly to a potential peer reviewer if that reviewer is going to be working unde r a contract or purchase order. In the case of a contract, the Agency provides the work product with associated background material to be peer reviewed to the prime contractor who in turn di stributes these documents to the pe er reviewers. In the case of a purchase order, the “charge or statement of work ” must be part of the PO (purchase order) and the provision of any documents should be coor dinated with the purchas ing agent handling the

order.

Essential documentation for each peer reviewer includes:

1) A current copy of the work product to be peer reviewed with associated background material. The work product should be of the best possible scientific /technical quality to ensure an adequate and useful peer review;Peer Review Handbook Page 74

2) A clear charge or statement of work, seek ing informed comment on identified specific issues to properly focus the efforts of the pe er reviewers and ensure that their individual efforts can be compared or contrasted;

3) Information concerning the process that you use for the peer revi ew, including the due date of reviewer comments, the format of those responses, and a point of contact in case the peer reviewer has questions. Responses should be written and submitted to the Peer Review Leader by an agreed upon deadline. In certain rare cases, oral commentary may be sufficient; however, in such cases, a follo w-up written response for the record should be made;

4) For highly influential scientific assessments, copies of significant public comments raised on scientific issues;

5) In some cases, Agency materials being peer re viewed will be availabl e to the public, even if they are marked as drafts. For exampl e, all materials reviewed by the SAB and the FIFRA SAP are available. Agency managers may also decide that a broad accessibility has benefits for the Agency. Offices should include a disclaimer on draft work products that are submitted for peer review (See S ection 1.3.4). In other cases, confidentiality needs to be maintained. In these cases, each peer reviewer should be informed of the need for confidentiality with re gard to the release of Agency products that are stamped as "DRAFT" or "DRAFT - Do Not Cite, Quote, or Release." Premature release of draft Agency products, views, or positions may be inappropriate and can be damaging to the credibility of the Agency or the peer revi ewer. Although it may not have legal effect, such language should be included in the charge to the peer reviewers when necessary. Other mechanisms to use in discouraging pr emature release include a disclaimer that appears in a separate section at the front of the document and creating the document with watermarks clearly delineating DRAFT status (o r a header or footer that states DRAFT status) on every page. In addi tion, in any solicitation for peer reviewers, the necessity for confidentiality and the non-release of materials should be emphasized.

Useful, but not critical materials that ma y be sent to peer reviewers include:

1) The name, address, and phone and fax numbe rs, and/or e-mail address of all peer reviewers working on the specific review;

2) A bibliography and/or any partic ularly relevant scientific articles from the literature;

3) A work product that has line numbering adde d in the margin for ease in providing and referencing comments.

Peer Reviewers should be given what is necessary to complete their task – they should not be

overburdened with excess material.Peer Review Handbook Page 75

3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review?

a) When EPA Conducts the Peer Review - The Peer Review Leader normally has

administrative contacts with the reviewer s during the developmen t and conduct of the peer review. In some cases (e.g., SAB peer review), peer reviewers may also receive a briefing on the product to be p eer reviewed. Otherwise, th e Peer Review Leader and

other EPA staff should not contact the reviewer s during the course of the review. Such contact can lead to perceived conflicts or in appropriate direction that could compromise the independence of the review.

b) When the Contractor Conducts the Peer Review - If peer review is conducted under a contract mechanism, EPA should limit direct co ntact to the prime contractor’s designated representative and should not have general contact and direction to th e contractor’s staff or peer reviewers (sub-contra ctors). Note, when a peer review is conducted under a contract, there are constraints where EPA staff are prohibited from contacting peer reviewers to avoid personal services arrang ements. Personal services contracts exist when the nature of the relationship between the contractor and EPA can be characterized as an employer-employee relationship. Any co mmunications with peer reviewers should be coordinated through the prime contractor.

3.6 Peer Review Services

A range of peer review services are availabl e to the Agency including internal, external (gratuitous services, contract s, purchase order), and Special Government Employee (SGE) mechanisms. The mechanism selected is genera lly based on the nature of the scientific or technical work product.

Peer review services are “advisory and assistan ce services”, as defined in Federal Acquisition Regulation (FAR) 37.201. These ty pes of services require speci al approvals and management oversight. See Figure 5, Item B Management Approvals, in Chapter 7 of the Contracts Management Manual (CMM) for current approval le vels. The CMM is available on the intranet

at http://intranet.epa.gov/o amintra/policy/cmm.pdf

.

3.6.1 What are Gratuitous Services for Peer Review?

The provision of peer review products or servi ces to EPA without compensation is provided as so-called “gratuitous” services. If a person wishes to perform peer review services for EPA without compensation, EPA must ask them to si gn an agreement whereby the person agrees to provide the prescribed peer review services as gratuitous services, wi th no expectation of receiving compensation for these services from EPA. An agreement, such as EPA Form 3100-14 or a gratuitous services contract, must be executed because the Antideficiency Act (31 USC § 1342) prohibits the Agency’s acceptance of “vol untary” services. For a copy of EPA Form 3100.14, see http://intranet.epa.gov/oas/fmsd/forms/f2.htm . “Voluntary services” are servicesPeer Review Handbook Page 76

provided to EPA without an agreement in advance th at such services are provided at no cost to EPA. Note that persons cannot waive compensati on (i.e., agree to provid e gratuitous services) for which there is a statutory ri ght to payment, unless a law perm its the waiver. For situations concerning State employees, see Section 3.4.9. Exam ples of laws that permit services without compensation are 5 USC § 3109 for experts or consultants and 5 USC § 3111 for student volunteers. If you are dealing with such grat uitous services, contac t the appropriate OGC attorney for advice.

3.6.2 Can You Use a Contract to Obtain Peer Review Services?

The Agency may obtain peer review services th rough a contract or purch ase order. Typically, peer review services would be available under a “mission contract,” i.e., a contract with a broad scope covering a variety of servi ces. It is also possible to ha ve a contract or purchase order solely for peer reviews.

A contract is awarded if the cost is over $100,000. If the cost is under $100,000, then a purchase order is issued. For assistance in preparing th e necessary pre-award contract documents, consult Chapter 7 of the CMM.

For assistance in preparing simplified acquisitio n packages for purchase orders, the Office of Acquisition Management (OAM) ha s a guide called SAME: Simplified Acquisition Made Easy,

which is available on the intranet at http://intranet.epa.gov/oamintra/policy/index.htm

.

3.6.3 How Do You Write a Statement of Work for Contracts?

For general assistance in preparing the Statemen t of Work (SOW), cons ult Chapter 11 of the CMM. The SOW should clearly specify that the c ontractor is responsible for preparing peer review evaluations and set forth guidelines for the peer review of scientific or technical documents. The contractor may perform the peer review with in-house staff, subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness and defensibility should be developed by the Program Office and made part of the contract. The contractor would then ensure that the peer revi ews adhere to the guidelines.

The SOW should include the list of questions that EPA wants the contractor to include in the charge to the peer reviewers. While the contra ctor will be the one that will prepare and send the formal charge to the peer reviewers, EPA should provide the list of ques tions to the contractor (see Section 3.2 for general discussion on charge to peer reviewers).

The SOW cannot simply define the role of the prim e contractor as arranging for the services of others to perform peer reviews and logistics for meetings. Unless the prime contractor is clearly tasked with responsibility for performing peer reviews, individual peer reviewers' fees and associated travel expenses ar e not payable under the contract.Peer Review Handbook Page 77

EPA may pay for the reviewer's comments or ev aluation, and also for at tendance at a meeting with the Agency and other reviewers to discuss th e results of the peer review. If the SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the Agency and other peer reviewers to discuss th e results of the peer review, pa yment is appropriate. The peer reviewer's attendance at the meeting would then be part of contract performance.

Example statements of work are found in Appendix F.

3.6.4 What are Some Management Controls for Contracts?

Management controls en sure several things:

1) the contractor does not perform in herently governmental activities (IGA);

2) the contractor’s work is fr ee from conflicts of interest;

3) if provided to the contractor, confidential business information or sensitive information is appropriately safeguarded, and;

4) improper relationships with contractor employees and subcontractors are avoided.

Agency and Federal acquisition regulations prohibit contractors from performing IGA. The Office of Management and Budget Circular A-76, revised May 29, 2003, defines “inherently governmental activities” as activities that are so intimately related to the public interest as to mandate performance by government personnel. These activities require the exercise of

substantial official di scretion in the applica tion of government authority and/or in making decisions for the government.

The Agency cannot award contracts for IGA; howev er, certain functions, su ch as those involved in peer review, may cross over in to IGA if not properly managed. With peer review services, the Agency is seeking only a contractor's recommenda tions, advice or analysis of a document, not a determination of what the policy should be. De termining Agency policy is an IGA. Agency officials make the official Agency decision re garding acceptability and/or quality of the document. To ensure that Agency officials are not improperly influenced by recommendations in the peer review, the contract should incl ude management controls. One possible control would be to direct the peer revi ewers to submit with their evalua tions or comments a description of the procedures used to arrive at their recommendations, a summ ary of their findings, a list of sources relied upon, and make clear and substantia te the methods and considerations upon which

their recommendations are based. To the extent possible, the contract should set forth any guidelines or criteria for perf ormance of the peer review. Agency officials should document their evaluations of the quality and validity of the peer review. You should ensure that there is a

clear record of their review of the contractor's work and that Agency personnel made the finalPeer Review Handbook Page 78

decisions. Such records of revi ew could include notes from revi ews of draft and final documents by Agency personnel, minutes from progress m eetings with contractors, and so forth.

Conflict of Interest (COI) – Another important factor is that the objectivity of the peer review should not be improperly influenced or undermined by the contractor performing the review. To identify and avoid or mitigate actual or potentia l COI, the contract should include controls. Inclusion of Agency-developed individual and or ganizational COI clauses in the contract or purchase order is critical for peer review serv ices. Usually, the EPA contracting officer (CO) will include COI solicitation provisions and cont ract clauses as a matter of course without involvement by the EPA Project Officer. As a safeguard, the Project Officer should:

a) Highlight the conflict of interest requiremen ts in the Statement of Work (SOW) for the procurement of the peer review services;

b) Review the solicitation/contract to make sure that the appropriate c onflict of interest clauses have been included, particular ly EPAAR clause 1552.209-73, Notification of Conflicts of Interest Regarding Personnel;

c) Work with the CO to develop contract-speci fic language regarding the peer review to

assist the contractor with iden tifying actual or potential conf licts of interest that might impair the objectivity of peer reviewers. For example, a contractor may be advised to consider the following questions and issues when determining if a proposed peer reviewer may have an actual or potential conflict of interest:

1) The sources and nature (obtained from a brief description of the work) of any compensated and non-compensated employme nt of the panel member and their spouse, including any government service, for the pre ceding two years.

2) The sources of research support a nd project funding, in cluding from any government source, for which the pane l member served as the Principal Investigator, Significant Co llaborator, Project Manager or Director during the preceding two years. For the panel member’s spouse, a general description of research and project activities in the preceding two years.

3) The compensated consulti ng activities of the panel me mber during the preceding two years, including the names of client s if compensation provided 15% or more of annual compensation. For the panel memb er’s spouse, a general description of consulting activities for the preceding two years.

4) The sources of compensated expert witn ess activities of the panel member and a brief description of the issue and testim ony during the preceding two years. For the panel member’s spouse, a general desc ription of expert testimony provided in the preceding two years.Peer Review Handbook Page 79

5) The assets, including stocks, bonds, real es tate, business, patents, trademarks, and royalties, for the panel member, th eir spouse and dependent children. Specifically, the financial holdings that collectively had a fair market value greater than $15,000 at any time during the preceding two year period (excluding, for example, well-diversified mutual f unds, money market funds, treasury bonds and personal residences).

6) The liabilities over $10,000 owed by the panel member, their spouse, and dependent children at any time in the pr eceding one year (excluding, for example, a mortgage on a personal residence, home equity loans, autom obile and consumer loans).

7) A brief description of any public st atements and/or positions of the panel member on, or closely related to, the matter under review.

8) A brief description of any previo us involvement of the panel member with the development of the document (or review ma terials) the individual has been asked to review (including pr evious peer reviews).

9) A brief description of any other inform ation that might reasonably raise a question about an actual or potent ial personal conflict of inte rest or bias including any financial benefit that might be gained by the panel member (or anyone whose interests are imputed to the panel memb er) as a result of the outcome of the

review.

Confidential Business Information (CBI)/Pr ivacy Act Protected Information and Other

Sensitive Information – When peer reviewers are not employees of the United States Government, it is unlikely that EPA will have aut hority to give reviewers access to confidential

business information in the absence of consen t for such disclosure by the CBI submitter. Therefore, all documents provided to non-Federal reviewers must be screened for information

claimed as CBI. Even where business information ha s not been explicitly claimed as CBI, if it is of a kind where the submitter might be expected to object to its release, prior to release the

submitter must be asked whether it wishes to as sert a claim, unless the submitter has previously

been informed that failure to a ssert a CBI claim may result in disc losure without notice. If the

contractor should have access to CBI for your peer review, notify th e CO so that the appropriate clauses can be included in the contract or purch ase order. These clauses will clearly identify any required procedures or processes prior to rele ase of any protected information, including any requirements for confidentiality agreements, as well as limits on use and disclosure of the data by contractor personnel.

Personal services – Unless your organization has statutor y authority to engage in personal service contracts, avoid treating contractor employees as Agency employees. For example, deciding who to hire, supervising, and assigning tasks to contractor employees are personalPeer Review Handbook Page 80

services. For additional information, program officials should consult EPA Order 1901.1A, Use of Contractor Services to Avoi d Improper Contracting Relationships . It is very important that all Agency employees follow this order when interac ting with contractors. The Order is available

on the intranet at http://epawww.epa.gov/oami ntra/policy/eo19011a.pdf

.

To avoid these improper relationships, program of ficials should write well-defined SOWs. The SOWs should set forth in detail a description of the work to be performed independently, including the manner in which it wi ll be evaluated. The SOW should set forth what work is to be performed not how the work is to be performed. Technical direction ma y be used to clarify ambiguous provisions to ensure efficient and e ffective contractor performance, and is not considered supervision or assignment of tasks.

3.6.5 Can You Identify and/or Select Pe er Reviewers When Using a Contract?

The prime contractor is responsible for selecti ng who will perform the peer review (EPA’s doing so may invoke FACA; see Section 2.8), whethe r these individuals work directly for the

contractor or are consultants or subcontractors to the prime contract or. Interfering in this process is a violation of Federal and Agency acquisiti on regulations. In accordance with FAR clause 52.244-2 Subcontracts, contractors may be required to obtain Contracting Officer (CO) consent or approval of subcontractors a nd the CO generally seeks the i nput of the work assignment manager (WAM – the WAM may be the Peer Re view Leader in many cases) before approving

the use of subcontractors.

EPA can establish criteria for the sort of indi viduals that might partic ipate on a peer review panel. However, the Agency should not be i nvolved in the selecti on of individual peer reviewers, and should avoid comm enting on the contractor’s selec tion of peer reviewers other than to determine whether the panel, once sel ected, meets the criteria established. EPA may identify a pool of qualified subcontractors and co nsultants to the prime contractor (listed in alphabetical order). If a list is provided, note on the list that this is a sugges ted list and other qualified candidates may exist who ar e not on the list. This is to prevent the impression that the prime contractor can only choose someone on our lis t. The prime contract or is required to include a conflicts of interest clause substantially similar to the conflicts of interest clause in the primary contract in its subcont ract to the peer reviewer.

3.6.6 How is Travel Handled with Contracts or Purchase Orders?

Funds obligated on a contract or purchase order are available to pay for the costs of producing the peer review including the travel costs and fee of the peer reviewer.

EPA acquires peer reviews through simplified acquisi tions issued directly to peer reviewers or through contracts with companies, which provid e the peer review services. By issuing a purchase order or awarding a contract for peer review services, EPA ma y pay not only for the peer review services/comments, but also for participation in a m eeting with the Agency and otherPeer Review Handbook Page 81

reviewers to discuss comments. The scope of work of the contract must re quire the contractor or individual peer reviewer, as appr opriate, to provide peer review se rvices and indicate whether the contractor or peer reviewer will be required to discuss a specific peer review work product with the Agency and/or with other peer reviewers. Participation in a meeting to discuss a peer review work product would then be part of the contract's performance. Thus, the contract may serve as the mechanism to pay for peer review services and associated travel expenses to provide

comments to EPA.

3.6.7 How is Travel Handled with Special Government Employees?

The term Special Government Employee (SGE) is defined in 18 USC § 202( a) as an officer or employee of an agency who performs temporary duties, with or without compensation, for not more than 130 days in a period of 365 days, either on a full-time or intermittent basis.

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid through

the issuance of invitational trav el orders (5 USC § 5703). These invitational travel and per diem expenses should be charged to an appropria te EPA travel account. The Federal Travel Regulations govern the invited travelers reimbursement.

Members of the SAB, SAP, and other FACA adviso ry committees are often classified as SGEs. It is not appropriate to reimbur se travel or per diem expenses of advisory committee members (SGEs) through a contract.Peer Review Handbook Page 82



## 4. Completing A Peer Review

4.1 Overview

Performance of the formal peer review of the wo rk product is not the fina l stage in the product’s development. Rather, it is an important stage in developing the work pr oduct, with the final version (with all comments addresse d) representing the true end of the peer review. As a result, the peer review process closes with four major activities:

a) Evaluating peer review co mments and recommendations

b) Utilizing peer review comments for completing the final document,

c) Organizing and maintaining a re cord of the peer review, and

d) Completing the peer review r ecord in the Science Inventory.

Careful attention to all of these elements, singly and together, assures a credible and transparent peer review process. Conversely, inattention can nullify the peer review effort. A well-planned peer review applied to a reas onable quality starting work pr oduct, followed by a responsible utilization of peer review sugge stions in the final product assu res a credible and defensible product for use in Agency decision-making.

The peer review of a work product is not co mplete until the peer review comments are incorporated into the final vers ion, or reasons are stated why such comments are not to be incorporated. However, for the purposes of doc umentation to the Agency’s Science Inventory only, the work product can be designated in th e Science Inventory as having completed peer review once the peer review re port is available on-line.

The peer review record is complete only when it contains a copy of the final work product (when there is one) that addresses the peer review comments and a copy of the documentation outlining why comments were not incorporated, if any were not incorporated . (See Sec. 2.5.3 for a list of the minimum documentation for the peer review record.)

4.2 Final Work Product

4.2.1 How Do You Incorporate Peer Revi ew Comments into the Final Work

Product?

The Peer Review Leader should carefully evalua te and analyze all peer review comments and recommendations. As discussed earlier, a caref ully crafted charge to the peer reviewersPeer Review Handbook Page 83

simplifies organizing and an alyzing comments. Also, any other issues that are raised should be

identified and evaluated.

The validity and objectiv ity of the comments should be evaluated. Analyses may include consultation with other experts and staff within the Office and Agency. Adequate documentation is needed to show that comments are accepted or rejected. The documentation can be brief, but should address the legitimate, valid comments, wh ether accepted for incorporation in the final work product or not. The peer review record s hould contain a document describing the Agency’s response to the peer review comm ents. The Agency’s response to the peer review report for highly influential scientific assessments should be posted on the Science Inventory.

The Peer Review Leader should brief the Decision Maker (including all appropriate managers in the Peer Review Leader’s chain of command) on th e charge, profile of peer reviewers, the peer review comments, and provide a proposal on how to address the peer review comments. It is the responsibility of the Peer Review Leader to obtain Decision Maker approval of the approach to addressing the peer review commen ts. The Peer Review Leader should clearly identify for the Decision Maker any major peer review comments that will not be accepted and why, as well as any controversial comments that should be resolved.

Comments that have significan t impact on time, budgetary, or re source needs are particularly important and should be eval uated in consultation with management. These comments may lead to allocation of The peer review is not complete until the additional resources and a revised schedule for the peer review comments are incorporated completion of the work product if appropriate for the into the final work product. final work products intended use.

4.2.2 How Might Peer Review Comments Impact Your Work Product?

Peer review, when appropriate, enhances the quality of the information EPA disseminates by ensuring the information that governs EPA’s mission is reliable and accura te (objective) and is appropriate for the intended use (utility). A variety of changes to a work product may result from the comments provided during peer review:

a) Peer review comments and recommendations may entail significant impacts on the planned project schedule, budget, or other resource needs. Management decisions to adjust one or more of these areas may be appropriate.

b) The substantive issues or c oncerns expressed by peer revi ewers may suggest that wider

scientific and technical consultation is needed to ensure the adequacy of the work product relative to its intended use. If the Agency agrees with the reviewers, additional resources and an extended delivery schedule may be necessary.Peer Review Handbook Page 84

c) The peer review comments and recommendati ons on a final product may provide a basis for bringing the associated proj ect to closure. At times th e Agency and its partners and stakeholders may not be able to come to cons ensus about a scientific or technical aspect of a work product. The scientif ic or technical issue in questio n can be noted in the charge to the reviewers as an area for which the Ag ency is specifically seeking peer review.

4.2.3 What Should the Final Work Product Say About the Peer Review Process?

If the product has been peer reviewed, you should describe the peer review in the document. Frequently, this will be part of a description of the process of devel oping the product along with descriptions of other public participation proces ses. It can be brief and does not need to describe the process or discuss th e peer-review comments in great detail. The description can be included in an introduction, preamble, or appendix. For influential scientific information and highly influential scientific assessments that support ru lemaking, the peer review should be discussed in the preamble.

When there are significant peer-review commen ts, and particularly if they are not being accepted, the document should generally discuss the issue and describe the reasons for the Agency’s choices in the appropriate sections of the document. The le vel of detail that is provided is a matter of judgment and should reflect the importance and degree of controversy of

the issue.

If a scientific or technical work product has not been peer-reviewed, this fact should be noted in the document, perhaps in an introduction or descri ption of its scope. This section should briefly indicate the reasons that p eer review was not conducted.

Derivative products of scientific or technical products (such as fact sheets, press releases, and brochures) do not need to discuss whether th e underlying products were peer reviewed.

4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous?

Generally, no. At a minimum, the names and aff iliations of peer reviewers should be listed in the peer review report for influential scientific information and highly influential scientific assessments. Depending on the peer review proce ss used, the reviewers names and affiliations may be made available to the publ ic prior to the review commen cing. Release of any reviewer information retrieved by a personal identifier must be performed in accordance with the Privacy Act, 5 USC § 552a as amended, and as interp reted in OMB implementing guidance, 40 FR 28,948 (July 9, 1975).

Though the identity of reviewers becomes known at some point in the peer review process, the attribution of specific comments to any given reviewer is kept confidential to the extent possible. In the ordinary course of events, you can often di scuss comments received without attributing the comments to a specific reviewer. However, if a matter goes to litigation, the litigating partiesPeer Review Handbook Page 85

can discover the names of anyone who contributed to a Federal product, including peer reviewers. Also, in most cases, though specific comments are not attributed to specific peer reviewers, the names and affiliations of the p eer reviewers are included in the Peer Review Report, which is included in the Science Inventory. Therefore, it is not possible to totally shield peer reviewers. As noted previously, peer revi ewers should be aware of EPA plans for releasing their names, credentials, and the ex tent of attribution of comments.

If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review Leader should inform the peer reviewer of the above po ssible eventualities. The Agency will in the ordinary course of events attempt to maintain th e confidentiality of the peer reviewers and their attributable comments from public disclosure, but it is recognized in many instances, for example open public meetings, litigation, and Agen cy responses to Freedom of Information Act (FOIA) requests, this can’t be assured. Remember, the Agency is committed to working with the fullest possible transparency to the public (excep t where statutorily cons trained, such as with confidential business information). It is recognized that this may be a deterrent to possible peer reviewers, but this is a real ity that has to be understood.

4.3 Completing the Peer Review Record

4.3.1 How Do You Complete the Peer Review Record?

Once the Peer Review Leader has completed the peer review and the final work product (where one is prepared), the peer review record is brought up to date and then archived according to that organization’s procedures (see Section 4.3.2). Th e peer review record should be indexed and maintained in an organization’s archive (repository ). The location of the peer review record should be readily identifiable so interested pa rties can locate and obtai n materials easily and quickly. The peer review record should be placed in any associat ed established public docket, if so required, in addition to the organizational archiv e. As a courtesy, a copy of the revised work product may be sent to the p eer reviewers for information.

The Peer Review Leader should collect the mate rials identified in Sect ion 2.5.3 and submit them for archiving. Finally, the Agency Science Inve ntory entry for the project should be completed and updated, including attachment of electronic copies of the documents.

4.3.2 Where Should the Peer Review Reco rds be Kept, and for How Long?

Though the Science Inventory will become the prim ary public active record and archive for the peer review record, the Peer Re view Leader is responsible for maintaining the on-site record. During the active conduct of the peer review, the Peer Review Lead ers maintain the peer review record themselves until the peer review is totally completed. Minimally, the file should be maintained until one year after the completed pe er review is reported in the next annual reporting. After that, the peer re view record should be maintain ed for a “reasonable period ofPeer Review Handbook Page 86

time.” Establishment and maintenance of the archive where the peer review records ultimately reside are an organization’s res ponsibility (i.e., not that of an individual program manager or Peer Review Leader).

Generally, to allow flexibility, individual Offi ces and Regions should decide the appropriate level of organizational responsib ility and how to ensure the record will meet “routinely available.” The peer review record may be kept with other records relating to the overall project, as long as it is easily and separate ly identifiable. The peer review record should be maintained in accordance with the Agency’s record-keeping sc hedule for such records. One long term archiving mechanism may be the formal archivin g at the Federal Records Center; the Science Inventory will also maintain an electronic copy of the record.

4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA?

Yes, it is subject to release if EPA receives a Freedom of Inform ation Act (FOIA) request, unless the peer review information meets the criteria for an exemption under the FOIA. (See http://www.epic.org/open_gov/foia/us_foia_act.html ).Peer Review Handbook Page A-1



# Appendix A

EPA’s Peer Review Policy

January 31, 2006Peer Review Handbook Page A-2Peer Review Handbook Page A-3

MEMORANDUM

SUBJECT: Peer Review Program

TO: Assistant Administrators

General Counsel Inspector General

Associate Administrators Regional Administrators

Staff Office Directors

We have made tremendous strides in improving our peer review program at EPA since the Agency’s Peer Review Policy was reaffirmed in 1994. Today I am updating the Peer Review Policy to emphasize the critical role that peer review plays in our efforts to ensure that EPA’s decisions rest on sound, credible science and data (see attached policy statement).

Peer review at EPA takes several different forms, ranging from informal consultations with Agency colleagues who were not involved in developing the product to the formal, public processes of the Science Advisory Board (SAB) and the FIFRA Scientific Advisory Panel (SAP). In any form, peer review assists EPA’s work by bringing independent expert experience and judgment to bear on issues before the Agency to the benefit of the final product.

In 1994 the Science Policy Council (SPC) and its Steering Committee were asked to undertake an initiative to ensure that EPA has a comprehensive Agency-wide program for implementing its Peer Review Policy. I commend the SPC for its diligence and success in meeting this objective. The SPC has made substantial improvements in the Peer Review Handbook, sponsored training of Agency managers and staff in peer review procedures, identified scientific and technical work products that merit peer review, and developed a publicly available data base of the peer review activities across the Agency. EPA has a strong and well-recognized peer review program as a direct result of these

efforts.Peer Review Handbook Page A-4

In 2004 the Office of Management and Budget (OMB) issued a “Final Information Quality Bulletin for Peer Review” that contains provisions for peer review at all federal agencies. The OMB Bulletin applies to influential scientific information and highly influential scientific assessments. The SPC has updated the Agency's Peer Review Handbook, in part to incorporate the provisions of the OMB Bulletin, and to reflect the experience gained from implementing the program over the last decade.

I ask that you continue to implement fully the provisions of our Peer Review Policy, and I expect the Science Policy Council to continue its role in overseeing and strengthening EPA’s peer review program. We must ensure that our decisions are based on the highest quality, peer-reviewed scientific and technical information.

Attachment

cc: Science Policy Council Science Policy Council Steering CommitteePeer Review Handbook Page A-5

PEER REVIEW AND PEER INVOLVEMENT

AT THE U.S. ENVIRONMEN TAL PROTECTION AGENCY

This document establishes the policy of the United States Environmental Protection Agency (EPA) for peer review of scientifica lly and technically based work products, including economic and social science products, that are intended to inform Agency decisions. Peer review, a form of peer involvement, is one process through which EPA staff augment their capabilities by inviting independent subject-matter experts to provi de objective evaluation of the work product.

PEER REVIEW

EPA strives to ensure that the scientific a nd technical bases of its decisions meet two important criteria: (1) they are based upon the best current knowle dge from science, engineering, and other domains of technical expe rtise; and (2) they are credible . Peer review, a process based on the principles of obtaining th e best technical and scientific expertise with appropriate independence, is central to sound science and help s the Agency meet these important criteria.

Peer review occurs when scie ntifically and technically base d work products are evaluated by relevant experts who were not involved in creating the product. Properly applied, peer review not only enriches the quality of work products but also adds a degree of cr edibility that cannot be achieved in any other way. Furthermore, peer re view early in the development of work products in some cases may conserve future resour ces by steering the development along the most

efficacious course.

Peer review generally take s one of two approaches:

• Internal , in which the reviewers are indepe ndent experts from inside EPA.

• External , in which the reviewers are inde pendent experts from outside EPA.

POLICY STATEMENT

Peer review of all scientific and technical information that is intended to inform or support Agency decisions is encouraged and ex pected. Influential scientific information, including highly influential scientific assessments, should be peer reviewed in accordance with the Agency’s Peer Review Handbook. All Agency managers are accountable for ensuring that Agency policy and guidance are appr opriately applied in determining if their work products are influential or highly influential, and for deci ding the nature, scope, and timing of their peer review. For highly influential scientific assessm ents, external peer review is the expected procedure. For influential scientific informati on intended to support important decisions, or for work products that have special importance in th eir own right, external peer review is the approach of choice. Peer review is not restricted to the nearly final version of work products; in fact, peer review at th e planning stage can often be extremely beneficial.Peer Review Handbook Page A-6

LEGAL EFFECT

This policy statement does not establish or aff ect legal rights or ob ligations. Rather, it confirms the importance of peer review where appropriate, outlines re levant principles, and identifies factors Agency staff should consider in implementing the policy. On a continuing basis, Agency management is expected to eval uate the policy as well as the results of its application throughout the Agency and undertake re visions as necessary. Therefore, the policy does not stand alone; nor does it establish a binding norm that is finally determinative of the

issues addressed.

IMPLEMENTATION

The Science Policy Council is responsible for overseeing Agency-wide implementation of this policy, including: promoting consistent interpretation; assessi ng Agency-wide progress; developing recommendations for revisions of the policy as necessary; and issuing the Peer

Review Handbook , which provides additional information and procedures on implementing this policy. Assistant Administrators, Regional Administrators, and other senior managers remain ultimately responsible for ensuring the appropriate application of Agency policy and guidance in identifying work products subject to peer revi ew, determining the type and timing of such review, documenting the process and outcome of each peer review, ensuring that the Science Inventory is kept current, a nd otherwise implementing the polic y within their organizational

units.

The policy is effective immediately.Peer Review Handbook Page B-1



# Appendix B OMB Information Quality Bulletin for

Peer Review and PreamblePeer Review Handbook Page B-2



### OFFICE OF MANAGEMENT AND BUDGET Final Information Quality Bulletin for Peer Review

I NTRODUCTION

This Bulletin establishes that important scientific information shall be peer reviewed by qualified specialists before it is disseminated by the federal government. We published a proposed Bulletin on September 15, 2003. Based on public comments, we published a revised proposal for additional comment on April 28, 2004. We are now finalizing the April version, with minor revisions responsive to the public’s comments.

The purpose of the Bulletin is to enhance the quality and credibility of the government’s scientific information. We recognize that diffe rent types of peer review are appropriate for different types of information. Under this Bull etin, agencies are gran ted broad discretion to weigh the benefits and costs of using a particular peer re view mechanism for a specific information product. The selection of an appr opriate peer review m echanism for scientific information is left to the agency’s discretion. Various types of information are exempted from the requirements of this Bulletin, including time-s ensitive health and safety determinations, in order to ensure that peer review does not unduly delay the releas e of urgent findings.

This Bulletin also applies stricter minimum require ments for the peer review of highly influential scientific assessments, which are a subset of influential scientif ic information. A scientific assessment is an evaluation of a body of scien tific or technical know ledge that typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information. To en sure that the Bulletin is not too costly or rigid, these requirements for more intensive peer review apply only to the more important scientific assessments disse minated by the federal government.

Even for these highly influential scientific assess ments, the Bulletin leaves significant discretion to the agency formulating the peer review plan. In general, an agency conducting a peer review of a highly influential scientif ic assessment must ensure that the peer review process is transparent by making available to the public the wr itten charge to the peer reviewers, the peer reviewers’ names, the peer review ers’ report(s), and the agency’s re sponse to the peer reviewers’ report(s). The agency selecti ng peer reviewers must ensure that the reviewers possess the necessary expertise. In add ition, the agency must address re viewers’ potential conflicts of interest (including those stemmi ng from ties to regulated busine sses and other stakeholders) and independence from the agency. This Bulletin requires agencies to adopt or adapt the committee selection policies employed by the Na tional Academy of Sciences (NAS) 14 when selecting peer reviewers who are not government employees. T hose that are government employees are subject to federal ethics requirements. The use of a transparent process, coupled with the selection of

14 National Academy of Sciences, “Policy and Procedures on Committee Composition and Balance and Conflicts of Interest for Committees Used in the Development of Reports,” May 2003: Available at: http://www.nationalacademies.org/coi/index.html.Peer Review Handbook Page B-3

qualified and independent peer reviewers, should improve the quality of government science while promoting public confidence in the integrity of the government’s scientific products.

P EER R EVIEW

Peer review is one of the important procedures used to ensure that the quality of published

information meets the standards of the scientific and technical community . It is a form of deliberation involving an exchange of judgments about the appropriateness of methods and the strength of the author’s inferences. 15 Peer review involves the review of a draft product for quality by specialists in the field who we re not involved in producing the draft.

The peer reviewer’s report is an evaluation or cri tique that is used by the authors of the draft to improve the product. Peer review typically evalua tes the clarity of hypothese s, the validity of the research design, the quality of data collecti on procedures, the robus tness of the methods employed, the appropriateness of the methods fo r the hypotheses being tested, the extent to which the conclusions follow from the analysis, and the strengths and li mitations of the overall product.

Peer review has diverse purposes. Editors of scientific journals use reviewer comments to help

determine whether a draft scientific article is of sufficient quality, importan ce, and interest to a field of study to justify publica tion. Research funding organizatio ns often use peer review to evaluate research proposals. In addition, some federal agencies make use of peer review to

obtain evaluations of draft information that cont ains important scientific determinations.

Peer review should not be confus ed with public comment and othe r stakeholder processes. The selection of participants in a peer review is based on expertise, with due consideration of independence and conflict of inte rest. Furthermore, notice-and- comment procedures for agency rulemaking do not provide an adequate substitute fo r peer review, as some experts -- especially those most knowledgeable in a field -- may not file public comments w ith federal agencies.

The critique provided by a peer review often suggests ways to clarify assumptions, findings, and conclusions. For instance, peer reviews can filter out biases and identify oversights, omissions,

and inconsistencies. 16 Peer review also may encourage authors to more fully acknowledge limitations and uncertainties. In some cases, re viewers might recommend major changes to the draft, such as refinement of hypotheses, reconsider ation of research desig n, modifications of data collection or analysis methods, or alternative conclusions. However, peer review does not always lead to specific modifica tions in the draft product. In so me cases, a draft is in excellent shape prior to being submitted for review. In others, the authors do not concur with changes suggested by one or more reviewers.

15 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving Regulatory Decision Making , Carnegie Commission, New York, 1993: 75.

16 William W. Lowrance, Modern Science and Human Values , Oxford University Press, New York, NY 1985: 85.Peer Review Handbook Page B-4

Peer review may take a variety of forms, depending upon the nature and importance of the product. For example, the reviewers may repres ent one scientific discip line or a variety of disciplines; the number of reviewers may range fr om a few to more than a dozen; the names of each reviewer may be disclosed publicly or may remain anonymous (e.g., to encourage candor); the reviewers may be blinded to the authors of the report or the names of the authors may be disclosed to the reviewers; the reviewers may prepare individual reports or a panel of reviewers may be constituted to produce a collaborative repor t; panels may do their work electronically or they may meet together in pers on to discuss and prepare their ev aluations; and reviewers may be compensated for their work or they may donate th eir time as a contribution to science or public

service.

For large, complex reports, different reviewers may be assigned to different chapters or topics. Such reports may be reviewed in stages, sometimes with confid ential reviews that precede a public process of panel review. As part of government-sponsored peer review, there may be opportunity for written and/or oral pub lic comments on the draft product.

The results of peer review are of ten only one of the criteria used to make decisions about journal publication, grant funding, and inform ation dissemination. For instan ce, the editors of scientific journals (rather than the peer reviewers) make final decisions about a manuscript’s appropriateness for publication based on a variet y of considerations. In research-funding decisions, the reports of peer reviewers often play an important role, but the final decisions about funding are often made by accountable officials based on a variety of considerations. Similarly, when a government agency sponsors peer review of its own draft documen ts, the peer review reports are an important factor in information dissemination decisions but rarely are the sole consideration. Agencies are not expected to cede their discretion with regard to dissemination or use of information to peer reviewers; accountable agency officials must make the final decisions.

T HE N EED FOR S TRONGER P EER R EVIEW P OLICIES

There are a multiplicity of science advisory proc edures used at federal agencies and across the wide variety of scientific products prepared by agencies. 17 In response to congressional inquiry, the U.S. General Accounting Office (now the G overnment Accountability Office) documented the variability in both the definition and impl ementation of peer review across agencies. 18 The Carnegie Commission on Science, Technology and Government 19 has highlighted the importance of “internal” scien tific advice (within the agency ) and “external” advice (through scientific advisory boards and other mechanisms).

17 Sheila Jasanoff, The Fifth Branch: Science Advisors as Policy Makers , Harvard University Press, Boston, 1990.

18 U.S. General Accounting Office, Federal Research: Peer Review Practices at Federal Agencies Vary

, GAO/RCED-99-99, Washington, D.C., 1999.

19 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving Regulatory Decision Making , Carnegie Commission, New York, 1993: 90.Peer Review Handbook Page B-5

A wide variety of authorities have argued that peer review practices at federal agencies need to be strengthened. 20 Some arguments focus on specific types of scientif ic products (e.g., assessments of health, safety and environmental hazards). 21 The Congressional/Presidential Commission on Risk Assessment and Risk Manageme nt suggests that “peer review of economic

and social science information should have as high a priority as p eer review of health, ecological, and engineering information.” 22

Some agencies have formal peer review policies, while others do not. Even agencies that have such policies do not always follow them prior to the release of important scientific products.

Prior to the development of this Bulletin, th ere were no government-wid e standards concerning when peer review is required and, if require d, what type of peer review processes are appropriate. No formal interagency mechanism existed to foster cross-agency sharing of experiences with peer review pr actices and policies. Despite th e importance of peer review for the credibility of agency scientific products, th e public lacked a consis tent way to determine when an important scientific information product is being developed by an agency, the type of peer review planned for that product, or whether there would be an opportunity to provide

comments and data to the reviewers.

This Bulletin establishes minimum standards for when peer review is required for scientific information and the types of peer review that should be considered by agencies in different circumstances. It also establishes a transparen t process for public disclosure of peer review planning, including a web-accessible de scription of the peer review plan that the agency has developed for each of its forthcoming in fluential scientific disseminations.

L EGAL A UTHORITY FOR THE B ULLETIN

This Bulletin is issued under the Information Quality Act and OMB’s general authorities to oversee the quality of agency info rmation, analyses, and regulatory actions. In the Information Quality Act, Congress directed OMB to issue guidelines to “provide policy and procedural

20 National Academy of Sciences, Peer Review in the Department of Energy – Office of Science and Technology

, Interim Report, National Academy Press, Washington, D.C ., 1997; National Academy of Sciences, Peer Review in Environmental Technology Development: The Department of Energy – Office of Science and Technology , National Academy Press, Washington, D.C., 1998; National Acad emy of Sciences, Strengthen ing Science at the U.S. Environmental Protection Agency: Resear ch-Management and Peer-Review Practices , National Academy Press, Washington, D.C. 2000; U.S. General Accounting Office, EPA’s Science Advisory Board Panels: Improved

Policies and Procedures Needed to Ensure Independence and Balance , GAO-01-536, Washington, D.C., 2001; U. S. Environmental Protection Agency, Office of Inspector General, Pilot Study: Science in Support of Rulemaking

2003 -P-00003, Washington, D.C., 2002; Carnegie Commission on Science, Technology, and Government, In the

National Interest: The Federal Government in the Reform of K-12 Math and Science Education , Carnegie Commission, New York, 1991; U.S. General Accounting Office , Endangered Species Program: Information on How Funds Are Allocated and What Activities are Emphasized , GAO-02-581, Washington, D.C. 2002.

21 National Research Council, Science and Judgment in Risk Assessment , National Academy Press, Washington, D.C., 1994.

22 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report, Volume 2, Risk Assessment and Risk Mana gement in Regulatory Decision-Making , 1997:103.Peer Review Handbook Page B-6

guidance to Federal agencies for ensuring and maximizing the quality, ob jectivity, utility and integrity of information” disseminated by Federa l agencies. Pub. L. No. 106-554, § 515(a). The Information Quality Act was developed as a supplement to the Paperwork Reduction Act, 44

U.S.C. § 3501 et seq ., which requires OMB, among other things, to “develop and oversee the implementation of policies, principles, standards, and guidelines to . . . apply to Federal agency dissemination of public information.” In addition, Executive Order 12866, 58 Fed. Reg. 51,735 (Oct. 4, 1993), establishes that OIRA is “the repository of expertise concerning regulatory issues,” and it directs OMB to provide guidance to the agencies on regulatory planning. E.O. 12866, § 2(b). The Order also requires that “[e]ach agency shall base its decisions on the best reasonably obtainable scientif ic, technical, economic, or ot her information.” E.O. 12866, § 1(b)(7). Finally, OMB has authority in certain circumstances to manage the agencies under the purview of the President’s Constitutional authority to supervise the unita ry Executive Branch.

All of these authorities support this Bulletin.

T HE R EQUIREMENTS OF THIS B ULLETIN

This Bulletin addresses peer revi ew of scientific information di sseminations that contain findings or conclusions that represent th e official position of one or more agencies of the federal

government.

Section I: Definitions

Section I provides definitions that are central to this Bulletin. Se veral terms are identical to or based on those used in OMB’s government-wid e information quality guidelines, 67 Fed. Reg. 8452 (Feb. 22, 2002), and the Paperwork Reduc tion Act, 44 U.S.C. § 3501 et seq

.

The term “Administrator” means the Administrato r of the Office of Information and Regulatory

Affairs in the Office of Mana gement and Budget (OIRA).

The term “agency” has the same meaning as in the Paperwork Reduction Act, 44 U.S.C. § 3502(1).

The term “Information Quality Act” means S ection 515 of Public Law 106-554 (Pub. L. No. 106-554, § 515, 114 Stat. 2763, 2763A-153-154 (2000)).

The term “dissemination” means agency initiated or sponsored distribution of information to the public. Dissemination does not include distribu tion limited to government employees or agency

contractors or grantees; intra- or inter-agency use or shar ing of government information; or responses to requests for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act, the Govern ment Performance and Results Act, or similar

laws. This definition also ex cludes distribution limited to corr espondence with individuals or persons, press releases, archival records, public f ilings, subpoenas and adjudicative processes. In the context of this Bulletin, the definition of “d issemination” modifies the definition in OMB’s government-wide information quality guidelines to address the need for p eer review prior toPeer Review Handbook Page B-7

official dissemination of the informati on product. Accordingly, under this Bulletin,

“dissemination” also excludes information distribu ted for peer review in compliance with this Bulletin or shared confidentially with scientific colleagues, provided that the distributing agency includes an appropriate and clear disclaimer on the information, as explained more fully below. Finally, the Bulletin does not directly cover in formation supplied to the government by third parties (e.g., studies by private consultants, companies and private, non-profit organizations, or research institutions such as universities). However, if an agency plans to disseminate information supplied by a third party (e.g., using th is information as the basis for an agency's factual determination that a particular behavi or causes a disease), the requirements of the Bulletin apply, if the dissemination is "influential".

In cases where a draft report or other information is released by an agency solely for purposes of peer review, a question may arise as to whethe r the draft report cons titutes an official "dissemination" under information-quality guidelines. Section I instructs ag encies to make this clear by presenting the following disclaimer in the report:

“THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE

DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY]. IT

DOES NOT REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY

AGENCY DETERMINATION OR POLICY.”

In cases where the information is highly relevant to specific policy or regulatory deliberations, this disclaimer shall appear on e ach page of a draft report. Agencies also shall discourage state, local, internati onal and private organizations from using information in draft reports th at are undergoing peer review. Draft influential scientific information presented at scientific meetings or shared confidentially with colleagues for scientific input prior to peer revi ew shall include the disclaimer: “ THE FINDINGS AND CONCLUSIONS IN THIS REPORT (PRESENTATION) HAVE NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY] AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY

AGENCY DETERMINATION OR POLICY. ”

An information product is not covered by the Bulletin unless it represents an official view of one or more departments or agencies of the federa l government. Accordingly, for the purposes of this Bulletin, “dissemination” excludes research pr oduced by government-funded scientists (e.g., those supported extramurally or intramurally by fe deral agencies or thos e working in state or local governments with federal s upport) if that information is not represented as the views of a department or agency (i.e., they are not official government disse minations). For influential

scientific information that does not have the im primatur of the federal government, scientists employed by the federal government are required to include in their information product a clear disclaimer that “the findings and conclusions in this report are those of the author(s) and do not necessarily represent the views of the funding agency.” A similar disclaimer is advised for non

government employees who publish government-funded research.Peer Review Handbook Page B-8

For the purposes of the peer review Bulletin, th e term “scientific information” means factual inputs, data, models, analyses, technical informa tion, or scientific assessments related to such disciplines as the behavioral a nd social sciences, public health and medical sciences, life and earth sciences, engineering, or physical scie nces. This includes any communication or representation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographic, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks on a web page to information that others disseminate. This definition excludes opinions, where the agency’s presentation makes clea r that an individual’s opinion, rather than a statement of fact or of the agency’s fi ndings and conclusions, is being offered.

The term “influential scientific information” means scientific information the agency reasonably

can determine will have or does have a clear an d substantial impact on important public policies or private sector decisions. In the term “influential sc ientific information,” the term "influential" should be interpreted consisten tly with OMB's government-wide information quality guidelines and the information quality guidelines of the agency. Information dissemination can have a significant economic impact even if it is not pa rt of a rulemaking. Fo r instance, the economic viability of a technology can be influenced by the government’s characteriza tion of its attributes. Alternatively, the federal government's assessment of risk ca n directly or indirectly influence the response actions of state and local agencies or international bodies.

One type of scientific information is a scientific assessment. For the purposes of this Bulletin,

the term “scientific assessment ” means an evaluation of a body of scientific or technical knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information. These assessments include, but are not limited to, state-of-science re ports; technology assessments; weight-of-evidence analyses; meta-a nalyses; health, safety, or ecological risk assessments; toxicological characterizations of substances; integrated assessment models; hazard determinations; or exposure assessments. Such assessments often draw upon knowledge from multiple disciplines. Typically, the data and mode ls used in scientific assessments have already been subject to some form of peer review (e.g., re fereed journal peer review or peer review under Section II of this Bulletin).

Section II: Peer Review of In fluential Scientific Information

Section II requires each agency to subject "influential" scientific information to peer review prior

to dissemination. For dissemination of influen tial scientific information, Section II provides agencies broad discretion in determining what type of peer review is appropriate and what procedures should be employed to select appropriate reviewers. Agencies are directed to chose a peer review mechanism that is adequate, giving due consideration to the novelty and complexity of the science to be reviewed, the relevance of the information to decisi on making, the extent of prior peer reviews, and the expected be nefits and costs of additional review.Peer Review Handbook Page B-9

The National Academy of Public Administration suggests that the intens ity of peer review should be commensurate with the significance of the information being disseminated and the likely implications for policy decisions. 23 Furthermore, agencies need to consider tradeoffs between depth of peer review and timeliness. 24 More rigorous peer review is necessary for information that is based on novel methods or pres ents complex challenges for interpretation. Furthermore, the need for rigorous peer revi ew is greater when the information contains precedent-setting methods or models, presents conc lusions that are likely to change prevailing practices, or is likely to affect policy d ecisions that have a significant impact.

This tradeoff can be considered in a benefit-cost framework. The costs of peer review include both the direct costs of the peer review activ ity and those stemming from potential delay in government and private actions that can result from peer review. Th e benefits of peer review are equally clear: the insights offered by peer reviewer s may lead to policy with more benefits and/or fewer costs. In addition to contributing to str ong science, peer review, if performed fairly and rigorously, can build consensus among stakeholders and reduce the temptation for courts and legislators to second-guess or overturn agency actions. 25 While it will not always be easy for agencies to quantify the benefits and costs of peer review, agenci es are encouraged to approach peer review from a benefit-cost perspective.

Regardless of the peer review mechanism chosen, ag encies should strive to ensure that their peer review practices are characterized by both scientif ic integrity and process integrity. “Scientific integrity,” in the context of peer review, refers to such issues as “expertise and balance of the panel members; the identification of the scientific issues and clarity of th e charge to the panel; the quality, focus and depth of the discussion of the issues by the pa nel; the rationale and supportability of the panel’s findings; and the acc uracy and clarity of the panel report.” “Process integrity” includes such issues as “transparenc y and openness, avoidance of real or perceived conflicts of interest, a workable process for pub lic comment and involvement,” and adherence to defined procedures. 26
When deciding what type of peer review mechanism is appropriate for a specific information product, agencies will need to consider at least th e following issues: individual versus panel review; timing; scope of the revi ew; selection of reviewers; disc losure and attribution; public participation; disposition of reviewer comments; and adequacy of prior peer review.

23 National Academy of Public Administration, Setting Pr iorities, Getting Results: A New Direction for EPA, National Academy Press, Washington, D.C., 1995:23.

24 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report,

1997.

25 Mark R. Powell, Science at EPA: In formation in the Regulatory Process , Resources for the Future, Washington, D.C., 1999: 148, 176; Sheila Jasanoff, The Fifth Branch: Science Advisors as Policy Makers , Harvard University Press, Boston, 1990: 242.

26 ILSI Risk Sciences Institute, “Policies a nd Procedures: Model Peer Review Center of Excellence,” 2002: 4. Available at http://r si.ilsi.org/file/Pol icies&Procedures.pdf.Peer Review Handbook Page B-10

Individual versus Panel Review

Letter reviews by several experts ge nerally will be more expediti ous than convening a panel of experts. Individual letter review s are more appropriate when a draft document covers only one discipline or when premature disclosure of a sensitive report to a public panel could cause harm to government or private interests. When tim e and resources warrant, panels are preferable, as they tend to be more deliberative than individual letter reviews and the reviewers can learn from each other. There are also multi-stage proces ses in which confidential letter reviews are conducted prior to release of a draft document for public no tice and comment, followed by a formal panel review. These more rigorous and expensive processes are particularly valuable for highly complex, multidisciplinary, and more important documents, especially those that are novel or precedent-setting.

Timing of Peer Review

As a general rule, it is most useful to consu lt with peers early in the process of producing information. For example, in the context of risk assessments, it is valuable to have the choice of input data and the specification of the model reviewed by peers before the agency invests time and resources in implementing the model and interpreting the results. "Early" peer review occurs

in time to "focus attentio n on data inadequacies in time for corrections.

When an information product is a critical compone nt of rule-making, it is important to obtain peer review before the agency announces its regu latory options so that any technical corrections can be made before the agency becomes invested in a specific approach or the positions of interest groups have hardened. If review occurs too late, it is unlik ely to contribute to the course of a rulemaking. Furthermore, investing in a mo re rigorous peer review early in the process “may provide net benefit by reducing the prospect of challenges to a regulation that later may trigger time consuming and resource-draining litigation.” 27

Scope of the Review

The “charge” contains the instructions to the p eer reviewers regarding th e objective of the peer review and the specific advice s ought. The importance of the in formation, which shapes the goal of the peer review, influences the charge. For instance, the goal of th e review might be to determine the utility of a body of literature for dr awing certain conclusions about the feasibility of a technology or the safety of a product. In th is context, an agency might ask reviewers to

determine the relevance of conclusions drawn in one context for other contexts (e.g., different exposure conditions or patient populations).

27 Fred Anderson, Mary Ann Chirba Ma rtin, E Donald Elliott, Cynthia Farina, Ernest Gellhorn, John D. Graham, C. Boyden Gray, Jeffrey Holmstead, Ronald M. Levin, La rs Noah, Katherine Rhyne, Jonathan Baert Wiener, "Regulatory Improvement Legislation: Risk Assessment, Cost-Benefit Analysis, and Judicial Review," Duke Environmental Law and Policy Forum , Fall 2000, vol. XI (1): 132.Peer Review Handbook Page B-11

The charge to the reviewers should be determined in advance of the selection of the reviewers. In drafting the charge, it is impor tant to remember the strengths and limitations of peer review. Peer review is most powerful when the charge is specific and steers th e reviewers to specific technical questions while also directing review ers to offer a broad evaluation of the overall product.

Uncertainty is inherent in science, and in many cases in dividual studies do not produce conclusive evidence. Thus, when an agency genera tes a scientific assessment, it is presenting its scientific judgment about the accumulated evidence rath er than scientific fact. 28 Specialists attempt to reach a consensus by weighing the accu mulated evidence. Peer reviewers can make an important contribution by distinguishing scie ntific facts from professional judgments. Furthermore, where appropriate, reviewers should be asked to provide advice on the reasonableness of judgments made from the scie ntific evidence. However, the charge should make clear that the reviewers are not to pr ovide advice on the polic y (e.g., the amount of uncertainty that is acceptable or the amount of precaution that should be embedded in an analysis). Such considerations are the purview of the government. 29

The charge should ask that peer reviewers ensu re that scientific un certainties are clearly

identified and characterized. Since not all uncerta inties have an equal effect on the conclusions drawn, reviewers should be asked to ensure that the potential imp lications of the uncertainties for

the technical conclusions drawn ar e clear. In addition, peer review ers might be asked to consider value-of-information analyses that identify whet her more research is likely to decrease key

uncertainties. 30 Value-of-information analysis was suggested for this purpose in the report of the Presidential/Congressional Commission on Risk Assessment and Risk Management. 31 A description of additional resear ch that would appreciably infl uence the conclusions of the assessment can help an agency assess and target subsequent efforts.

Selection of Reviewers

Expertise . The most important factor in selecting reviewers is expertise: ensuring that the selected reviewer has the knowledge, experience, and skills necessary to perform the review. Agencies shall ensure that, in cases where th e document being reviewed spans a variety of scientific disciplines or areas of technical expertise, review ers who represent the necessary spectrum of knowledge are chosen. For instance, e xpertise in applied mathematics and statistics is essential in the review of models, thereby allowing an audit of calcu lations and claims of

28 Mark R. Powell, Science at EPA: Information in the Regulatory Process , Resources for the Future, Washington, D.C., 1999: 139.

29 Ibid.

30 Granger Morgan and Max Henrion, “The Value of Knowing Ho w Little You Know,” Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis

, Cambridge University Press, 1990: 307.

31 Presidential/Congressional Commission on Risk Assessment and Risk Management, Risk Commission Report, 1997, Volume 1: 39, Volume 2: 91.Peer Review Handbook Page B-12

significance and robustness ba sed on the numeric data. 32 For some reviews, evaluation of biological plausibility is as impor tant as statistic al modeling. Agencies sh all consider requesting that the public, including scientif ic and professional societies, nominate potential reviewers.

Balance . While expertise is the pr imary consideration, reviewers should also be selected to represent a diversity of scientific perspectives relevant to the s ubject. On most controversial issues, there exists a range of respected scien tific viewpoints regardi ng interpretation of the available literature. Inviting reviewers with competing views on the science may lead to a sharper, more focused peer review. Indeed, as a final layer of review, some organizations (e.g., the National Academy of Sciences) specifically recr uit reviewers with strong opinions to test the scientific strength and balance of their reports. The NAS policy on committee composition and

balance 33 highlights important considerat ions associated with perspe ctive, bias, and objectivity.

Independence

. In its narrowest sense, independence in a reviewer means that the reviewer was not involved in producing the draft document to be reviewed. However, for peer review of some documents, a broader view of independence is nece ssary to ensure credibility of the process. Reviewers are generally not employed by the agen cy or office producing the document. As the National Academy of Sciences has stated, “external experts often can be more open, frank, and challenging to the status quo than internal reviewers, who may f eel constrained by organizational

concerns.” 34 The Carnegie Commission on Science, Technology, and Government notes that “external science advisory boards serve a critica lly important function in providing regulatory agencies with expert advice on a range of issues.” 35 However, the choice of reviewers requires a case-by-case analysis. Review ers employed by other federal a nd state agencies may possess unique or indispensable expertise. A related issue is whether government-funded scien tists in universities and consulting firms have sufficient independence from the federal agencies th at support their work to be appropriate peer reviewers for those agencies. 36 This concern can be mitigated in situations where the scientist initiates the hypothesis to be tested or the met hod to be developed, which effectively creates a buffer between the scientist and the agency. When an agency awards grants through a competitive process that includes peer review, the agency’s potential to influence the scientist’s research is limited. As such, when a scientist is awarded a government research grant through an investigator-initiated, peer-reviewed competition, th ere generally should be no question as to that scientist's ability to offer independent scientific advice to the agency on other projects. This contrasts, for example, to a situation in wh ich a scientist has a consulting or contractual arrangement with the agency or office sponsoring a peer review. Likewise, when the agency

32 William W. Lowrance, Modern Science and Human Values , Oxford University Press, New York, NY 1985: 86.

33 National Academy of Sciences, “Policy and Procedures on Committee Composition and Balance and Conflicts of Interest for Committees Used in the Development of Reports,” May 2003: Available at: http://www.nationalacademies.org/coi/index.html.

34 National Research Council, Peer Review in Environmental Technology Development Programs: The Department of Energy’s Office of Science and Technology , National Academy Press, Washington, D.C., 1998: 3.

35 Carnegie Commission on Science, Technology, and Government, Risk and the Environment: Improving Regulatory Decision Making , Carnegie Commission, New York, 1993: 90.

36 Lars Noah, “Scientific ‘Republicanism’: Expert Peer Review and the Quest for Regulatory Deliberation , Emory

Law Journal , Atlanta, Fall 2000:1066.Peer Review Handbook Page B-13

and a researcher work together (e.g., through a cooperative agreem ent) to design or implement a study, there is less independence from the agency. Furthermore, if a scientist has repeatedly served as a reviewer for the same agency, some may question whether that scientist is sufficiently independent from the agency to be employed as a peer reviewer on agencysponsored projects.

As the foregoing suggests, independence poses a complex set of questions that must be considered by agencies when peer reviewers are selected. In general, agencies shall make an effort to rotate peer review responsibilities across the available pool of qualified reviewers, recognizing that in some cases repeated service by the same reviewer is needed because of essential expertise.

Some agencies have built entire organizations to provide independent sc ientific advice while other agencies tend to employ ad hoc scientific panels on spec ific issues. Respect for the independence of reviewers may be enhanced if an agency collects names of potential reviewers (based on considerations of e xpertise and reputation for object ivity) from the public, including scientific or professional societie s. The Department of Energy’s use of the American Society of Mechanical Engineers to identify potential peer re viewers from a variety of different scientific societies provides an example of how professional societies can a ssist in the development of an independent peer review panel. 37

Conflict of Interest . The National Academy of Sciences defines “conflict of interest” as any

financial or other interest that conflicts with th e service of an individual on the review panel because it could impair the individual’s objec tivity or could create an unfair competitive advantage for a person or organization. 38 This standard provides a useful benchmark for agencies to consider in selecti ng peer reviewers. Agencies shall make a special effort to examine prospective reviewers’ potential financial conflicts, including si gnificant investments, consulting arrangements, employer affiliations and grants/contracts. Financial ties of potential reviewers to regulated entities (e.g., businesses), other stak eholders, and regulatory agencies shall be scrutinized when the information being reviewed is likely to be relevant to regulatory policy. The inquiry into potential conflicts goes beyond financial investments a nd business relationships and includes work as an expert witness, consulting a rrangements, honoraria and sources of grants and contracts. To evaluate any real or perceived conflicts of interest with potential reviewers and questions regarding the independe nce of reviewers, agencies ar e referred to federal ethics requirements, applicable standards issued by the Office of Government Et hics, and the prevailing practices of the National Academy of Sciences. Specifically, peer reviewers who are federal employees (including special government empl oyees) are subject to federal requirements governing conflicts of interest. See

, e.g ., 18 U.S.C. § 208; 5 C.F.R. Part 2635 (2004). With

37 American Society for Mechanical Engineers, Assessment of Technologies Supported by the Office of Science and Technology, Department of Energy: Results of the Peer Review for Fiscal Year 2002 , ASME Technical Publishing, Danvers, MA, 2003.

38 National Academy of Sciences, “Policy and Procedures on Committee Composition and

Balance and Conflicts of Interest for Committees Used in the Development of Reports,” May

2003: Available at: http://www.nationalacademies.org/coi/index.html.Peer Review Handbook Page B-14

respect to reviewers who are not federal empl oyees, agencies shall adopt or adapt the NAS policy for committee selection with respect to evaluating conflicts of interest. 39 Both the NAS and the federal government recognize that under certain circumstances some conflict may be

unavoidable in order to obtain th e necessary expertise. See

, e.g ., 18 U.S.C. § 208(b)(3); 5

U.S.C. App. § 15 (governing NAS committees). To improve the transparency of the process, when an agency determines that it is necessary to use a reviewer with a real or perceived conflict of interest, the agency should consider publicly di sclosing those conflicts. In such situations, the agency shall inform potential reviewers of such disclosure at the time they are recruited.

Disclosure and Attribution: Anonymous versus Identified

Peer reviewers must have a clear understanding of how their comments will be conveyed to the authors of the document and to the public. When peer review of government reports is considered, the case for transparency is stronger, particularly when the re port addresses an issue with significant ramifications for the public a nd private sectors. The public may not have confidence in the peer review process when the na mes and affiliations of the peer reviewers are

unknown. Without access to the comments of reviewer s, the public is incapable of determining whether the government has seriously consider ed the comments of reviewers and made appropriate revisions. Disclosure of the slate of reviewers and the substance of their comments can strengthen public confidence in the peer review process. It is common at many journals and research funding agencies to disclose annually th e slate of reviewers. Moreover, the National Academy of Sciences now discloses the names of its peer reviewers, without disclosing the

substance of their comments. The science adviso ry committees to regulatory agencies typically disclose at least a summary of the comments of reviewers as well as their names and affiliations.

For agency-sponsored peer review conducted under Sections II and III, th is Bulletin strikes a compromise by requiring disclosure of the identity of the reviewers, but not public attribution of specific comments to specific reviewers. The agency has considerable discretion in the implementation of this compromise (e.g., summar izing the views of reviewers as a group or disclosing individual reviewer co mments without attribut ion). Whatever approach is employed, the agency must inform reviewers in advance of how it intends to address this issue. Information about a reviewer retrieved from a record filed by the reviewer's name or other identifier may be disclosed only as permitted by the conditions of disclosure enumerated in the Privacy Act, 5

U.S.C. § 552a as amended, and as interpre ted in OMB implementing guidance, 40 Fed. Reg. 28,948 (July 9, 1975).

Public Participation

Public comments can be important in shaping expe rt deliberations. Agencies may decide that peer review should precede an opportunity for pub lic comment to ensure that the public receives the most scientifically strong product (rather than one that may change substantially as a result of peer reviewer suggestions). However, there are situations in which public participation in peer

39 Ibid.Peer Review Handbook Page B-15

review is an important aspect of obtaining a high-quali ty product through a cr edible process. Agencies, however, should avoid open-ended co mment periods, which may delay completion of peer reviews and complicate the completion of the final work product.

Public participation can take a variety of forms, including opportuni ties to provide oral comments before a peer review panel or reques ts to provide written comments to the peer reviewers. Another option is for agencies to publish a “request for comment” or other notice in which they solicit public comment before a pane l of peer reviewers performs its work.

Disposition of Reviewer Comments

A peer review is considered completed once the agency considers and addresses the reviewers’ comments. All reviewer comments should be gi ven consideration and be incorporated where relevant and valid. For instance, in the cont ext of risk assessments, the National Academy of Sciences recommends that peer review include a written eval uation made available for public inspection. 40 In cases where there is a public panel, the agency should plan publication of the peer review report(s) and the agency’s re sponse to peer reviewer comments.

In addition, the credibility of the final scientific report is likely to be enhanced if the public understands how the agency addressed the specific concerns raised by the peer reviewers. Accordingly, agencies should consider prepari ng a written response to the peer review report explaining: the agency's agreement or disagreem ent, the actions the agency has undertaken or will undertake in response to the report, and (if a pplicable) the reasons the agency believes those actions satisfy any key concerns or recommendations in the report.

Adequacy of Prior Peer Review

In light of the broad range of information covere d by Section II, agencies are directed to choose a peer review mechanism that is adequate, giving due consideration to the novelty and complexity of the science to be reviewed, the relevance of the information to decisi on making, the extent of 55prior peer reviews, and the expected be nefits and costs of additional review.

Publication in a refereed scientific journal ma y mean that adequate peer review has been performed. However, the intensity of peer review is highly variable acro ss journals. There will be cases in which an agency determines that a more rigorous or transp arent review process is necessary. For instance, an agen cy may determine a particular journal review process did not address questions (e.g., the extent of uncertainty i nherent in a finding) that the agency determines

should be addressed before diss eminating that information. As such, prior peer review and publication is not by itself sufficient grounds for determining that no further review is necessary.

40 National Research Council, Risk Assessment in the Federal Government: Managing the Process , National Academy Press, Washington, D.C., 1983.Peer Review Handbook Page B-16

Section III: Peer Review of Highly Influential Scientific Assessments

Whereas Section II leaves most of the considerations regarding the fo rm of the peer review to the agency’s discretion, Section III requires a more rigorous form of peer review for highly influential scientific assessments. The requirements of Section II of this Bulletin apply to Section III, but Section III has some additional requirements, which are discussed below. In planning a peer review under Section III, agencies typical ly will have to devote greater resources and attention to the issues discussed in Section II, i. e., individual versus pa nel review; timing; scope of the review; selection of revi ewers; disclosure and attribut ion; public participation; and disposition of reviewer comments.

A scientific assessment is considered "highl y influential" if the agency or the OIRA

Administrator determines that the dissemination could have a potential impact of more than $500 million in any one year on either the public or private sector or that the dissemination is novel, controversial, or precedent-sett ing, or has significant interagenc y interest. One of the ways information can exert economic impact is through the costs or benefits of a regulation based on the disseminated information. The qualitative aspect of this definition may be most useful in cases where it is difficult for an agency to pred ict the potential economic effect of dissemination. In the context of this Bulletin, it may be either the approach used in the assessment or the interpretation of the information itself that is novel or precedent-setting. Peer review can be valuable in establishing the bounds of the scientific debate when methods or interpretations are a source of controversy among interested parties. If information is covered by Section III, an agency is required to adhere to the peer re view procedures specified in Section III.

Section III (2) clarifies that the principal finding s, conclusions and recommendations in official reports of the National Academy of Sciences that fall under this Section are generally presumed not to require additional peer revi ew. All other highly influential scientific assessments require a review that meets the requirements of Section III of this Bulletin.

With regard to the selection of reviewers, Sect ion III(3)(a) emphasizes co nsideration of expertise

and balance. As discussed in Section II, expertise refers to the required knowle dge, experience and skills required to perform the review whereas balance refers to the need for diversity in scientific perspective and disciplines. We emphasi ze that the term "balance" here refers not to balancing of stakeholder or politic al interests but rather to a br oad and diverse representation of respected perspectives and intellectual traditions within the scientific community, as discussed in the NAS policy on committee composition and balance. 41

Section III(3)(b) instructs agencies to consider barring particip ation by scientists with a conflict

of interest. The conflict of inte rest standards for Sections II a nd III of the Bulletin are identical. As discussed under Section II, those peer re viewers who are federal employees, including Special Government Employees, are subject to ap plicable statutory and regulatory standards for

41 National Academy of Sciences, “Policy and Procedures on Committee Composition and Balance and Conflicts of Interest for Committees Used in the Development of Reports,” May 2003: Available at: http://www.nationalacademies.org/coi/index.html.Peer Review Handbook Page B-17

federal employees. For non-government employees , agencies shall adopt or adapt the NAS policy for committee member selection with resp ect to evaluating conflicts of interest.

Section III(3)(c) instructs agenci es to ensure that reviewers are independent of the agency sponsoring the review. Scie ntists employed by the sponsoring agency are not permitted to serve as reviewers for highly infl uential scientific assessments. This does not preclude Special Government Employees, such as academics appointed to advisory committees, from serving as peer reviewers. The only exception to this ban w ould be the rare situati on in which a scientist from a different agency of a Cabinet-level depart ment than the agency that is disseminating the scientific assessment has expertise, experience a nd skills that are essentia l but cannot be obtained elsewhere. In evaluating the need for this exception, agencies shall use the NAS criteria for assessing the appropriateness of using em ployees of sponsors (e.g., the government scientist must not have had any part in the de velopment or prior revi ew of the scientific information and must not hold a position of ma nagerial or policy responsibility).

We also considered whether a reviewer can be independent of the agency if that reviewer

receives a substantial amount of research funding from th e agency sponsoring the review. Research grants that were awarded to the scient ist based on investigator-i nitiated, competitive, peer-reviewed proposals do not generally raise is sues of independence. However, significant consulting and contractual relati onships with the agency may ra ise issues of independence or conflict, depending upon the situation.

Section III(3)(d) addresses concerns regarding re peated use of the same reviewer in multiple assessments. Such repeated use should be avoi ded unless a particular re viewer’s expertise is essential. Agencies should rotate membership across the available pool of qualified reviewers. Similarly, when using standing panels of scientific advisors, it is suggested that the agency rotate membership among qualified scientists in order to obtain fresh perspectives and reinforce the reality and perception of independence from the agency.

Section III(4) requires ag encies to provide reviewers with sufficient background information, including access to key studies, data and models, to perform their role as peer reviewers. In this respect, the peer review envision ed in Section III is more rigor ous than some forms of journal peer review, where the reviewer is often not provided access to underlying data or models. Reviewers shall be informed of applicable acces s, objectivity, reproducib ility and other quality standards under federal information quality laws.

Section III(5) addresses oppor tunity for public participation in peer review, and provides that the agency shall, wherever possible, provide for publ ic participation. In some cases, an assessment may be so sensitive that it is cr itical that the agency’s assessment achieve a high level of quality before it is publicized. In thos e situations, a rigorous yet confidential peer review process may be appropriate, prior to p ublic release of the assessment. If an agency decides to make a draft assessment publicly available at th e onset of a peer review proce ss, the agency shall, whenever possible, provide a vehicle for the public to provi de written comments, make an oral presentation before the peer reviewers, or both. When written public comments are received, the agency shallPeer Review Handbook Page B-18

ensure that peer reviewers receive copies of co mments that address significant scientific issues with ample time to consider them in their review. To avoid undue delay of agency activities, the agency shall specify time limits for public part icipation throughout the p eer review process.

Section III(6) requires that agenci es instruct reviewers to prep are a peer review report that describes the nature and scope of their review and their findings and conclusions. The report shall disclose the name of each peer reviewer a nd a brief description of his or her organizational affiliation, credentials and relevant experiences. The peer review report should either summarize the views of the group as a whole (including any di ssenting views) or incl ude a verbatim copy of the comments of the individual reviewers (with or without attribution of specific views to specific names). The agency shall also prepare a written response to the peer review report, indicating whether the agency agrees with the re viewers and what actions the agency has taken or plans to take to address the points made by re viewers. The agency is required to disseminate the peer review report and the ag ency's response to the report on the agency's website, including all the materials related to the peer review such as the charge statement, peer review report, and agency response to the review. If the scientific information is used to support a final rule then, where practicable, the peer review report shall be made available to the pu blic with enough time for the public to consider the implications of the peer review report for the rule being considered.

Section III(7) authorizes but does not require an agency to commission an entity independent of the agency to select peer reviewers and/or mana ge the peer review process in accordance with this Bulletin. The entity may be a scientific or professional society, a firm specializing in peer review, or a non-profit or ganization with experience in peer review.

Section IV: Alternative Procedures

Peer review as described in this Bulletin is only one of many procedur es that agencies can employ to ensure an appropriate degree of pre- dissemination quality of influential scientific information. For example, Congress has assigned the NAS a special role in advising the federal government on scientific and technical issues. The procedures of the NAS are generally quite rigorous, and thus agencies should presum e that major findings, conclusions, and recommendations of NAS reports meet the performance standards of this Bulletin.

As an alternative to complying with Sections II and III of this Bulletin, an agency may instead (1) rely on scientific information produced by the National Academ y of Sciences, (2) commission the National Academy of Sciences to peer review an agency draft scientific information product, or (3) employ an alternativ e procedure or set of procedures, specifically approved by the OIRA Administrator in consul tation with the Office of Science and Technology Policy (OSTP), that ensures that the scient ific information product meets applicable information-quality standards.

An example of an alternative pr ocedure is to commission a respected third party other than the NAS (e.g., the Health Effects Institute or the National Commission on Radiation Protection and Measurement) to conduct an assessment or seri es of related assessments. Another example ofPeer Review Handbook Page B-19

an alternative set of procedures is the three-part process used by the National Institutes of Health (NIH) to generate scientific guida nce. Under that process, a scie ntific proposal or white paper is generated by a working group composed of external, independent scientific experts; that paper is then forwarded to a separate ex ternal scientific c ouncil, which then makes recommendations to the agency. The agency, in turn, decides whet her to adopt and/or modify the proposal. For large science agencies that have diverse re search portfolios and do not have significant regulatory responsibilities, such as NIH, an acceptable alternative would be to allow scientists from one part of the agency (f or example, an NIH institute) to participate in the review of documents prepared by another pa rt of the agency, as long as the head of the agency confirms in writing that each of the reviewers meets the NAS criteria relating to the appropriateness of using employees of sponsors (e.g., the government scientist must not have had any part in the development or prior re view of the scientific information and must not hold a position of managerial or policy responsib ility). The purpose of Section IV is to encourage these types of innovation in the met hods used to ensure pre-dissemination quality

control of influential sc ientific information.

The mere existence of a public comment proce ss (e.g., notice-and-commen t procedures under the Administrative Procedure Act) does not constitute adequate p eer review or an “alternative process,” because it does not assure that qualified, impartial specia lists in relevant fields have performed a critical evaluation of the agency's draft product. 42

Section V: Peer Review Planning

Section V requires agencies to begin a systematic process of peer review planning for influential scientific information (including highly influen tial scientific assessments) that the agency plans

to disseminate in the foreseeable future. A key feature of this planning process is a webaccessible listing of forthcoming in fluential scientific disseminati ons (i.e., an agenda) that is regularly updated by the agency. By making these plans publicly available, agencies will be able to gauge the extent of public interest in th e peer review process fo r influential scientific information, including highly infl uential scientific assessments. These web-accessible agendas can also be used by the public to monitor agency compliance with this Bulletin.

Each entry on the agenda shall include a pre liminary title of the planned report, a short paragraph describing the subjec t and purpose of the planned re port, and an agency contact person. The agency shall provide its predicti on regarding whether the dissemination will be “influential scientific information” or a “highly influential scientific assessment,” as the designation can influence the type of peer review to be undertaken. The agency shall discuss the timing of the peer review, as well as the use of any deferrals. Agencies sh all include entries in the agenda for influential scientific inform ation, including highly in fluential scientific assessments, for which the Bulletin’s requirements have been deferred or waived. If the agency,

in consultation with the OIRA Ad ministrator, has determined th at it is appropriate to use a

42 William W. Lowrance, Modern Science and Human Values , Oxford University Press, New York, NY 1985: 86.Peer Review Handbook Page B-20

Section IV “alternative pr ocedure” for a specific dissemination, a description of that alternative procedure shall be included in the agenda.

Furthermore, for each entry on the agenda, the agency shall describe the peer review plan. Each peer review plan shall include : (i) a paragraph including the t itle, subject and purpose of the planned report, as well as an agency contact to whom inquiries may be directed to learn the specifics of the plan; (ii) whether the dissemi nation is likely to be influential scientific information or a highly influential scientific a ssessment; (iii) the timing of the review (including deferrals); (iv) whether the revi ew will be conducted through a panel or individual letters (or whether an alternative procedure will be exercised); (v) whether there will be opportunities for the public to comment on the work product to be peer reviewed, and if so, how and when these opportunities will be provided; (vi) whether the agency will provi de significant and relevant public comments to the peer reviewers before th ey conduct their review; (vii) the anticipated number of reviewers (3 or fewer; 4-10; or more than 10); (viii) a succinct desc ription of the primary disciplines or expertise n eeded in the review; (ix) whethe r reviewers will be selected by the agency or by a designated outside organi zation; and (x) whether the public, including scientific or professional societies, will be aske d to nominate potential peer reviewers. The agency shall provide a link from the agenda to each document made public pursuant to this Bulletin. Agencies shall link their peer review agendas to the U.S. Government’s official web portal: firstgov at http://www.FirstGov.gov

Agencies should update their peer review agendas at least every si x months. However, in some cases -- particularly for highly influential scien tific assessments and other particularly important information -- more frequent updates of existing entries on the agenda, or the addition of new entries to the agenda, may be warranted. Wh en new entries are added to the agenda of forthcoming reports and other information, the pub lic should be provided with sufficient time to comment on the agency's peer review plan for that report or product. Ag encies shall consider public comments on the peer review plan. Agencies are encouraged to offer a listserve or similar mechanism for members of the public who woul d like to be notified by email each time an agency’s peer review agenda has been updated.

The peer review planning requirements of this Bulletin are designed to be implemented in phases. Specifically, the planning requirements of the Bulletin will go into effect for documents subject to Section III of the Bulletin (highly infl uential scientific assessments) six months after publication. However, the planning requirements for documents subject to Section II of the Bulletin do not go into effect un til one year after publication. It is expected that agency experience with the planning requ irements of the Bulletin for the smaller scope of documents encompassed in Section III will be used to inform implementation of these planning requirements for the larger scope of documents covered under Section II.

Section VI: Annual Report

Each agency shall prepare an annual report that summarizes key decisions made pursuant to this Bulletin. In particular, each agency should provi de to OIRA the following: 1) the number ofPeer Review Handbook Page B-21

peer reviews conducted subject to the Bulletin (i .e., for influential scientific information and highly influential scientific assessments); 2) th e number of times alternative procedures were invoked; 3) the number of times waivers or deferra ls were invoked (and in the case of deferrals, the length of time elapsed between the deferral and the peer review ); 4) any decision to appoint a reviewer pursuant to any excepti on to the applicable independenc e or conflict of interest standards of the Bulletin, including determin ations by the Secretary or Deputy Secretary pursuant to Section III (3) (c); 5) the number of peer review pane ls that were conducted in public and the number that allowed public comment; 6) the number of public comments provided on the agency’s peer review plans; and 7) the number of peer reviewers that the agency used that were recommended by professional societies.

Section VII: Certification in the Administrative Record

If an agency relies on influential scientific information or a highly influential scientific assessment subject to the requirements of this Bu lletin in support of a regulatory action, the agency shall include in the administrative record for that action a certificat ion that explains how the agency has complied with the requirements of this Bulletin and the Information Quality Act. Relevant materials are to be pl aced in the administrative record.

Section VIII: Safeguards, Deferrals, and Waivers

Section VIII recognizes that individuals serving as peer reviewer s have a privacy interest in information about themselves that the government maintains and retrieves by name or identifier from a system of records. To the extent in formation about a reviewer (name, credential, affiliation) will be disclosed along with his/her comments or analysis, the agency must comply with the requirements of the Privacy Act, 5 U.S.C. 552a, as amended, and OMB Circular A-130, Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a published System of Records Notice. Furthermor e, the peer review must be conducted in a manner that respects confidential business information as well as intellectual property.

Section VIII also allows for a deferral or waiv er of the requirements of the Bulletin where necessary. Specifically, the agency head may waive or defer some or all of the peer review requirements of Sections II or III of this Bulletin if there is a compelling rationale for waiver or

deferral. Waivers will seldom be warranted under th is provision because the Bulletin already provides significant safety valves, such as: the exemptions provided in Section IX, including the exemption for time-sensitive health and safety in formation; the authorization for alternative procedures in Section IV; and th e overall flexibility provided fo r peer reviews of influential

scientific information under Section II. Nonethel ess, we have included this waiver and deferral provision to ensure needed flexibility in unusual and compelling situations not otherwise covered by the exemptions to the Bulletin, such as situa tions where unavoidable legal deadlines prevent full compliance with the Bulletin before info rmation is disseminated. Deadlines found in consent decrees agreed to by agencies after the Bulletin is issued will not ordinarily warrant waiver of the Bulletin’s requirements because those deadlines should be negotiated to permit time for all required procedures, including peer review. In addition, when an agency isPeer Review Handbook Page B-22

unavoidably up against a deadline, deferral of so me or all requirements of the Bulletin (as opposed to outright waiver of all of them) is the most appropriate accommodation between the need to satisfy immovable deadlines and the need to undertake proper peer review. If the agency head defers any of the peer review requirement s prior to dissemination, peer review should be conducted as soon as practicable thereafter.

Section IX: Exemptions

There are a variety of situations where agencies need not conduct peer review under this Bulletin. These include, for example, disseminations of sensitive information related to certain national security, foreign affairs, or negotiations involving intern ational treaties and trade where compliance with this Bulletin would interfere wi th the need for secrecy or promptness.

This Bulletin does not cover official dissemina tions that arise in adjudications and permit proceedings, unless the agency determines that peer review is practical and appropriate and that the influential dissemination is scientifically or technically novel (i .e., a major change in accepted practice) or likely to have precedent-setting influence on future adjudications or permit proceedings. This exclusion is intended to cover, among other things, licensing, approval and registration processes for speci fic product development activities as well as site-specific activities. The determination as to whether peer re view is practical and appr opriate is left to the discretion of the agency

. While this Bulletin is not broadly applicable to adjudications, agencies are encouraged to hold peer reviews of scien tific assessments supporting adjudications to the same technical standards as peer reviews cove red by the Bulletin, including transparency and disclosure of the data and models underlying the assessments. Protections apply to confidential

business information.

The Bulletin does not cover time-sensitive health and safety disseminations, for example, a dissemination based primarily on data from a recen t clinical trial that was adequately peer reviewed before the trial began. For this purpo se, “health” includes public health, or plant or

animal infectious diseases.

This Bulletin covers original data and formal analytic models used by agencies in Regulatory Impact Analyses (RIAs). However, the RIA documents themselves are already reviewed through an interagency review process under E. O. 12866 that involves application of the principles and methods defined in OMB Circular A-4. In that respect, RIAs are excluded from coverage by this Bulletin, although agencies are encouraged to have RIAs reviewed by peers within the government for ad equacy and completeness.

The Bulletin does not cover accounting, budget, act uarial, and financial information including that which is generated or used by agencies that focus on interest rates, banking, currency, securities, commodities, futures, or taxes.Peer Review Handbook Page B-23

Routine statistical information released by federa l statistical agencies (e .g., periodic demographic and economic statistics) and analys es of these data to compute standard indicators and trends (e.g., unemployment and poverty rates) is excluded from this Bulletin.

The Bulletin does not cover information dissemina ted in connection with routine rules that materially alter entitlements, grants, user fees, or loan programs, or the ri ghts and obligations of recipients thereof.

If information is disseminated pursuant to an exemption to this Bulletin, subsequent disseminations are not automatically exempted. For example, if influential scientific information is first disseminated in the course of an exempt agency adjudication, but is later disseminated in the context of a non-exempt rulemaking, the subs equent dissemination will be subject to the requirements of this Bulletin even though the first dissemination was not.

Section X: OIRA and OSTP Responsibilities OIRA, in consultation with OSTP, is responsible for overseeing agency implementation of this

Bulletin. In order to foster le arning about peer revi ew practices across agencies, OIRA and OSTP shall form an interagency workgroup on p eer review that meets regularly, discusses progress and challenges, and recommends im provements to peer review practices.

Section XI: Effective Date and Existing Law

The requirements of this Bulletin, with the exception of Section V, apply to information disseminated on or after six months after publica tion of this Bulletin. However, the Bulletin does not apply to information that is already being addressed by an agency-initiated peer review process (e.g., a draft is alrea dy being reviewed by a formal scientific advisory committee established by the agency). An existing peer review mechanism mandated by law should be implemented by the agency in a manner as cons istent as possible wi th the practices and procedures outlined in this Bulletin. The requirements of Section V apply to “highly influential scientific assessments,” as designated in Sect ion III of the Bulletin, within six months of publication of the final Bulletin. The requirement s in Section V apply to documents subject to

Section II of the Bulletin one year after publication of the final Bulletin.

Section XII: Judicial Review

This Bulletin is intended to improve the internal management of the Executive Branch and is not intended to, and does not, create a ny right or benefit, substantive or procedural, enforceable at law or in equity, against the United States, its agencies or other entities, its officers or employees, or any other person.Peer Review Handbook Page B-24Peer Review Handbook Page B-25



### Bulletin for Peer Review

I. Definitions

.

For purposes of this Bulletin --

1. the term “Administrator” means the Ad ministrator of the Office of Information and Regulatory Affairs in the Office of Management and Budget (OIRA);

2. the term “agency” has the same meaning as in the Paperwork Reduction Act, 44 U.S.C. § 3502(1);

3. the term “dissemination” means agency init iated or sponsored distribution of information to the public (see 5 C.F.R. 1320.3(d) (definition of “Conduct or Sponsor”)). Dissemination does

not include distribution limited to government employees or agen cy contractors or grantees; intra- or inter-agency use or sh aring of government information; or responses to requests for agency records under the Freedom of Information Act, the Privacy Act, the Federal Advisory Committee Act, the Government Performance and Re sults Act or similar law. This definition

also excludes distribution limited to correspondence with individuals or persons, press releases, archival records, public filings, subpoenas and ad judicative processes. Th e term “dissemination” also excludes information distributed for peer re view in compliance with this Bulletin, provided that the distributing agency includes a clear disclaimer on the information as follows: “THIS

INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF PRE

DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY

GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY [THE AGENCY].

IT DOES NOT REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY

AGENCY DETERMINATION OR POLICY.” Fo r the purposes of this Bulletin, “dissemination” excludes research produced by government-funded scientists (e.g., those supported extramurally or intramurally by federal agencies or those working in state or local governments with federal support) if that informa tion does not represent the views of an agency. To qualify for this exemption, th e information should display a clear disclaimer that “the findings and conclusions in this report are those of the author(s ) and do not necess arily represent the views of the funding agency”;

4. the term “Information Quality Act” means Section 515 of Public Law 106-554 (Pub. L. No. 106-554, § 515, 114 Stat. 2763, 2763A-153-154 (2000));

5. the term “scientific info rmation” means factual inputs, da ta, models, analyses, technical information, or scientific assessments based on th e behavioral and social sciences, public health and medical sciences, life and earth sciences, engineering, or physical sciences. This includes any communication or repres entation of knowledge such as facts or data, in any medium or form, including textual, numerical, graphic, cartographi c, narrative, or audiovisual forms. This definition includes information that an agency disseminates from a web page, but does not include the provision of hyperlinks to information that others disseminate. This definition does not include opinions, where the agen cy’s presentation makes clear th at what is being offered is someone’s opinion rather than fact or the agency’s views;

6. the term “influential scientific inform ation” means scientific information the agency reasonably can determine will have or does have a clear and substantial impact on important public policies or privat e sector decisions; andPeer Review Handbook Page B-26

7. the term “scientific asse ssment” means an evaluation of a body of scientific or technical knowledge, which typically synthesizes multiple factual inputs, data, models, assumptions, and/or applies best professional judgment to bridge uncertainties in the available information. These assessments include, but are not limited to, state-of-science re ports; technology assessments; weight-of-evidence analyses; meta-a nalyses; health, safety, or ecological risk assessments; toxicological characterizations of substances; integrated assessment models; hazard determinations; or exposure assessments.

II. Peer Review of Influential Scientific Information

.

1. In General : To the extent permitted by law, each ag ency shall conduct a peer review on all influential scientific information that the agency intends to disseminate. Peer reviewers shall be charged with reviewing scientif ic and technical matters, leaving policy determinations for the agency. Reviewers shall be informed of applic able access, objectivity, reproducibility and other quality standards under the federal laws governing information access and quality.

2. Adequacy of Prior Peer Review : For information subject to this section of the Bulletin, agencies need not have furthe r peer review conducted on info rmation that has already been subjected to adequate peer re view. In determining whether prior peer review is adequate, agencies shall give due consideration to the novelty and complexity of the science to be reviewed, the importance of the in formation to decision making, the extent of prior peer reviews, and the expected benefits and costs of additiona l review. Principal fi ndings, conclusions and recommendations in official reports of the National Academy of Sciences are generally presumed to have been adequately peer reviewed.

3. Selection of Reviewers :

a. Expertise and Balance : Peer reviewers shall be select ed based on expertise, experience and skills, including specialists from multiple disc iplines, as necessary. The group of reviewers shall be sufficiently broad and diverse to fairly represent the relevant scientific and technical perspectives and fields of knowledge. Agenci es shall consider requesting that the public, including scientific and pr ofessional societies, nominate potential reviewers.

b. Conflicts : The agency – or the entity selecting the peer reviewers – shall (i) ensure that those reviewers serving as federal employees (including special government employees) comply with applicable federal ethics requirements; (ii) in selec ting peer reviewers who are not government employees

, adopt or adapt the National Academy of Sciences policy for committee selection with respect to evaluating the potential for conflicts (e.g., those arising from investments; agency, employer, and business affiliations; grants, contracts and consulting income). For scientific information relevant to specific regulations, the agency shall examine a

reviewer’s financial ties to regulated entities (e.g., businesse s), other stakeholders, and the

agency.

c. Independence : Peer reviewers shall not have part icipated in development of the work product. Agencies are encouraged to rotate me mbership on standing panels across the pool of qualified reviewers. Research grants that were awarded to scientists based on investigatorinitiated, competitive, peer-reviewed proposals gene rally do not raise issues as to independence

or conflicts.Peer Review Handbook Page B-27

4. Choice of Peer Review Mechanism : The choice of a peer review mechanism (for example, letter reviews or ad hoc panels) for influentia l scientific information shall be based on the novelty and complexity of the information to be reviewed, the importance of the information to decision making, the extent of prio r peer review, and the expected benefits and costs of review, as well as the factors regarding tran sparency described in II(5).

5. Transparency : The agency -- or entity managing the peer review -- shall instruct peer reviewers to prepare a report that describes the nature of their review and their findings and conclusions. The peer review re port shall either (a) include a ve rbatim copy of each reviewer's comments (either with or without specific attribu tions) or (b) represent th e views of the group as a whole, including any disparate a nd dissenting views. The agency shall disclose the names of the reviewers and their organizational affiliations in the report. Reviewers shall be notified in advance regarding the extent of disclosure and attribution planned by th e agency. The agency shall disseminate the final peer review report on the agency's we bsite along with all materials related to the peer review (a ny charge statement, the peer review report, and any agency response). The peer review report shall be disc ussed in the preamble to any related rulemaking

and included in the administrative re cord for any related agency action.

6. Management of Peer Review Process and Reviewer Selection : The agency may commission independent entities to manage the peer revi ew process, including th e selection of peer reviewers, in accordance with this Bulletin.

III. Additional Peer Review Requirements for Hi ghly Influential Scientific Assessments.

1. Applicability : This section applies to influential sc ientific information that the agency or

the Administrator determines to be a scientific assessment that: (i) could have a potential impact of mo re than $500 million in any year, or (ii) is novel, controversial, or precedent-setting or has signi ficant interagency interest.

2. In General : To the extent permitted by law, each ag ency shall conduct peer reviews on all information subject to this Section. The peer re views shall satisfy the re quirements of Section II of this Bulletin, as well as the additional requir ements found in this Section. Principal findings,

conclusions and recommendations in official reports of the Nati onal Academy of Sciences that fall under this Section are gene rally presumed not to require additional peer review.

3. Selection of Reviewers :

a. Expertise and Balance : Peer reviewers shall be select ed based on expertise, experience and skills, including specialists from multiple disc iplines, as necessary. The group of reviewers shall be sufficiently broad and diverse to fairly represent the relevant scientific and technical perspectives and fields of knowledge. Agencies shall consider requesting that the public, including scientific and pr ofessional societies, nominate potential reviewers.

b. Conflicts : The agency – or the entity selecting the peer reviewers – shall (i) ensure that those reviewers serving as federal employees (including special government employees) comply with applicable federal ethics requirements; (ii) in selec ting peer reviewers who are not government employees, adopt or adapt the National Academy of Sciences’ policy for committee selection with respect to evaluating the potential for conflicts (e.g., those arising from investments; agency, employer, and business affiliations; grants, contracts and consulting income). For scientific assessments relevant to specific regulations, a revi ewer’s financial ties to regulated entities (e.g., businesses) , other stakeholders, and the ag ency shall be examined.Peer Review Handbook Page B-28

c. Independence : In addition to the requirements of Section II (3)(c), which shall apply to

all reviews conducted unde r Section III, the agency -- or entity selecting the reviewers -- shall bar participation of scie ntists employed by the sponsoring agency unless the reviewer is employed only for the purpose of conducting the peer review (i.e., special government employees). The only exception to this bar would be the rare case where the agency determines, using the criteria developed by NAS for evalua ting use of “employees of sponsors,” that a premier government scientist is (a) not in a posi tion of management or po licy responsibility and (b) possesses essential expe rtise that cannot be obtained elsewh ere. Furthermore, to be eligible for this exception, the scientist must be employe d by a different agency of the Cabinet-level department than the agency that is disseminating the scientific information. The agency’s determination shall be documented in writing and approved, on a non-de legable basis, by the Secretary or Deputy Secretary of the department prior to the scientist’s appointment.

d. Rotation : Agencies shall avoid repeated use of the same reviewer on multiple assessments unless his or her pa rticipation is essential and can not be obtained elsewhere.

4. Information Access : The agency -- or entity managing the peer review -- shall provide the reviewers with sufficient information -- including background information about key studies or

models -- to enable them to unde rstand the data, analytic procedures, and assumptions used to support the key findings or conclusions of the draft assessment.

5. Opportunity for Public Participation : Whenever feasible and a ppropriate, the agency shall

make the draft scientific assessm ent available to the public for comment at the same time it is submitted for peer review (or during the peer review process) and sponsor a public meeting where oral presentations on scientific issues can be made to the peer reviewers by interested members of the public. When employing a public co mment process as part of the peer review, the agency shall, whenever practical, provide pe er reviewers with access to public comments that address significant scientific or technical issues. To ensure that public participation does not unduly delay agency activities, the agency sh all clearly specify time limits for public participation throughout the peer review process.

6. Transparency : In addition to the requirements specifi ed in II(5), which shall apply to all

reviews conducted under Section II I, the peer review report sha ll include the charge to the reviewers and a short paragraph on both the creden tials and relevant experiences of each peer reviewer. The agency shall prepare a written re sponse to the peer revi ew report explaining (a) the agency's agreement or disagreement with th e views expressed in the report, (b) the actions the agency has undertaken or will undertake in response to the report, and (c) the reasons the agency believes those actions satisfy the key concer ns stated in the report (if applicable). The agency shall disseminate its response to the peer review report on the agen cy's website with the related material specified in Section II(5).

7. Management of Peer Review Process and Reviewer Selection : The agency may commission independent entities to manage the peer revi ew process, including th e selection of peer reviewers, in accordance with this Bulletin.Peer Review Handbook Page B-29

IV. Alternative Procedures

.

As an alternative to complying with Sections II and III of this Bulletin, an agency may instead: (i) rely on the principal findings, conclusions and reco mmendations of a report produced by the National Academy of Sciences; (ii) commission the National Academy of Sciences to peer review an agency’s draft scientif ic information; or (iii) employ an alternative scientific procedure or process, specifically approved by the Administra tor in consultation with the Office of Science and Technology Policy (OSTP), that ensures the agency’s scientific information satisfies applicable information quality standards. The alternative procedure(s) may be applied to a designated report or group of reports.

V. Peer Review Planning

.

1. Peer Review Agenda : Each agency shall post on its we bsite, and update at least every six months, an agenda of peer review plans. The agenda shall descri be all planned and ongoing influential scientific information subject to this Bulletin. The agency shall provide a link from the agenda to each document that has been made public pursuant to this Bulletin. Agencies are encouraged to offer a listserve or similar mechanism to alert interested members of the public when entries are added or updated.

2. Peer Review Plans : For each entry on the agenda the agency shall describe the peer review plan. Each peer review plan shall include: (i) a paragraph incl uding the title, subject and purpose of the planned report, as well as an agency cont act to whom inquiries may be directed to learn the specifics of the plan; (ii) wh ether the dissemination is likely to be influential scientific information or a highly influential scientific a ssessment; (iii) the timing of the review (including deferrals); (iv) whether the revi ew will be conducted through a panel or individual letters (or whether an alternative procedure will be employed); (v) whether there will be opportunities for the public to comment on the work product to be peer reviewed, and if so, how and when these opportunities will be provided; (vi) whether the agency will provi de significant and relevant public comments to the peer reviewers before th ey conduct their review; (vii) the anticipated number of reviewers (3 or fewer; 4-10; or more than 10); (viii) a succinct desc ription of the primary disciplines or expertise n eeded in the review; (ix) whethe r reviewers will be selected by the agency or by a designated outside organi zation; and (x) whether the public, including scientific or professional societies, will be as ked to nominate potenti al peer reviewers.

3. Public Comment : Agencies shall establish a mechanism for allowing the public to comment on the adequacy of the peer review plans. Agencies shall consider public comments on peer review plans.

VI. Annual Reports

.

Each agency shall provide to OIRA, by Decem ber 15 of each year, a summary of the peer reviews conducted by the agency during the fiscal year. The report shoul d include the following: 1) the number of peer reviews conducted subject to the Bulletin (i.e., for influential scientific information and highly influentia l scientific assessments); 2) the number of times alternative procedures were invoked; 3) the number of times waivers or defe rrals were invoked (and in thePeer Review Handbook Page B-30

case of deferrals, the length of time elapsed be tween the deferral and the peer review); 4) any decision to appoint a reviewer pursuant to a ny exception to the appli cable independence or conflict of interest standards of the Bulletin, including determinations by the Secretary pursuant to Section III(3)(c); 5) the numbe r of peer review panels that were conducted in public and the number that allowed public comment; 6) th e number of public comments provided on the agency’s peer review plans; and 7) the number of peer reviewers that the agency used that were recommended by professional societies.

VII. Certification in the Administrative Record

.

If an agency relies on influential scientific information or a highly influential scientific assessment subject to this Bulletin to suppor t a regulatory action, it shall include in the

administrative record for that act ion a certification expl aining how the agency has complied with the requirements of this Bulletin and the appli cable information quality guidelines. Relevant materials shall be placed in the administrative record.

VIII. Safeguards, Deferrals, and Waivers

.

1. Privacy : To the extent information about a reviewer (name, credentials, affiliation) will be disclosed along with his/her comments or an alysis, the agency sh all comply with the requirements of the Privacy Act, 5 U.S.C. § 522a as amended, and OMB Circular A-130, Appendix I, 61 Fed. Reg. 6428 (February 20, 1996) to establish appropriate routine uses in a published System of Records Notice.

2. Confidentiality : Peer review shall be conducted in a manner that respects (i) confidential business information and (ii) intellectual property.

3. Deferral and Waiver : The agency head may waive or defe r some or all of the peer review requirements of Sections II and III of this Bulle tin where warranted by a compelling rationale. If the agency head defers the peer review requireme nts prior to dissemination, peer review shall be conducted as soon as practicable.

IX. Exemptions

.

Agencies need not have peer review conducted on information that is:

1. related to certain national security, forei gn affairs, or negotiations involving international trade or treaties where compliance with this Bulletin would interfere with th e need for secrecy or

promptness;
2. disseminated in the course of an indi vidual agency adjudication or permit proceeding (including a registration, approva l, licensing, site-specific dete rmination), unless the agency determines that peer review is practical and appr opriate and that the influential dissemination is scientifically or technically novel or likely to have precedent-setting influence on future adjudications and/or permit proceedings;

3. a health or safety dissemination where the agency determines that the dissemination is timesensitive (e.g., findings based primarily on data fr om a recent clinical trial that was adequately peer reviewed before the trial began);Peer Review Handbook Page B-31

4. an agency regulatory impact analysis or re gulatory flexibility analysis subject to interagency review under Executive Order 12866, except for unde rlying data and analytical models used;

5. routine statistical in formation released by federal statistical agenci es (e.g., periodic demographic and economic statistics) and analyses of these data to comput e standard indicators and trends (e.g., unemployment and poverty rates);

6. accounting, budget, actuarial, and financial information, including that which is generated or used by agencies that focus on interest rates, banking, currency, securities, commodities, futures, or taxes; or 7. information disseminated in connection with routine rules that materially alter entitlements, grants, user fees, or loan programs, or the rights and obligations of recipients thereof.

X. Responsibilities of OIRA and OSTP

.

OIRA, in consultation with OSTP, shall be resp onsible for overseeing implementation of this Bulletin. An interagency group, chaired by OSTP a nd OIRA, shall meet peri odically to foster better understanding about peer review practices and to assess progress in implementing this

Bulletin.

XI. Effective Date and Existing Law

.

The requirements of this Bulletin, with the ex ception of those in Section V (Peer Review Planning), apply to information disseminated on or after six months following publication of this Bulletin, except that they do not apply to inform ation for which an agency has already provided a draft report and an associated charge to peer reviewers. Any existing peer review mechanisms mandated by law shall be employed in a manner as consistent as possible with the practices and procedures laid out herein. The requirements in Section V apply to “highl y influential scientific assessments,” as designated in Section III of th is Bulletin, within six months of publication of this Bulletin. The requirements in Section V a pply to documents subject to Section II of this Bulletin one year after pub lication of this Bulletin.

XII. Judicial Review

This Bulletin is intended to improve the internal management of the executive branch, and is not intended to, and does not, create a ny right or benefit, substantive or procedural, enforceable at law or in equity, against the Unite d States, its agencies or other en tities, its officers or employees, or any otherPeer Review Handbook Page C-1Peer Review Handbook Page C-2



# Appendix C

Sound Science and Peer Review in Rulemaking PolicyPeer Review Handbook Page C-3



### SOUND SCIENCE AND PEER REVIEW IN RULEMAKING

Several provisions of the December 2004, OMB Bu lletin for peer review made it necessary for the Office of Policy, Economics, and Innovation (OPEI) to create conditional template language (Attachment A). This language should be used by rulewriters in the preamble of regulations that rely on influential scientific information or a highly influential scientific assessment, which are two categories of information defi ned in Section 2.2 of this handbook.

For proposed and final regulations that rely on influential scientific information or a highly influential scientific assessment, rulewriters should use the template as a model to discuss peer review in the preamble where appropriate. In addition, peer review l eaders should communicate with rulewriters and workgroup chairs to ensure that all appropriate peer review material is included in the docket, and that template language is included in the preamble.

OPEI also revised the Action Memorandum Framew ork to include a discussion of peer review

for influential scientific informa tion or a highly influential scient ific assessment (Attachment B).Λ Peer Review Handbook Page C-4

ATTACHMENT A



# Peer Review (Conditional Template)

Read this first (but DO NOT insert it in your preamble):

The OMB Final Information Quality Bulletin fo r Peer Review directs EPA to include a discussion of the peer review report and how th e Agency complied with the provisions of the Bulletin in the preamble of rulemakings that are s upported by influential scientific information or highly influential scientific assessments. Peer re view reports should either (a) include a verbatim copy of each reviewer's comments (either with or without specific attributions) or (b) represent the views of the group as a whole, including a ny disparate and dissenting views. The Agency

should disclose the names of the reviewers and th eir organizational affilia tions in the report and should notify the reviewers in advance regarding the extent of the disclosure and attribution planned by the Agency. You should ensure that th e peer review report is placed in the docket to comply with the OMB Bulletin.

Use this template if your proposed or final rule is based on a work produc t containing influential scientific information or a highly influential scientific assessment. This language should appear

in the Supplementary Information section of regulatory preambles under General

Information . You may want to include th e language under the heading:

Did EPA conduct a peer review be fore issuing this notice?

Π PROPOSED & FINAL ACTIONS: If you used a highly influential scientific

assessment of influential scientif ic information to support this rulemaking, ins ert this into the preamble of your proposed or final rule , advanced notice of proposed rulemaking, or

other substantive action:

This regulatory action was supported by [influential scientific information or a highly influential scientific assessment] . Therefore, EPA conducted a peer review in accordance with OMB’s Final Information Quality Bulletin for Peer Review. [Insert a brief description of the peer review process alon g with any other relevant information.] The peer review report is located in the docket for today's action. According to the report, [insert a brief discussion of the peer review repor t. For more information about the peer review report, see the Peer Review Handbook.]Peer Review Handbook Page C-5

ATTACHMENT B

FRAMEWORK FOR ACTION MEMORANDA

PURPOSE OF THIS FRAMEWORK

This Framework provides guidance on the format and contents of Action Memoranda that accompany packages presented for signature. A draft of the memo also accompanies the Final Agency Review (FAR) package. The Framew ork was developed to promote a consistent approach to these memoranda and to preserve the flexibility necessary to accommodate the unique needs of each program while ensuring that critical information is provided to Agency managers in a clear and concise manner. Ge nerally, the Action Memoranda should not exceed 5 pages, and follows the format outlined in the Agency’s Correspondence Manual ( /asd http://epawww/rmpolicy /manuals/1320.pdf.epa.gov ).

As indicated by the use of non-mandatory language such as “should,” “recommend,” and “may,” this Framework provides recommendations and does not impose any legally binding requirements. Recommendations for each section of Action Memoranda are discussed below. Programs may include information on additional topics if they are relevant to a given action, e.g., information quality issues, etc.

Action Memoranda are considered internal, deliberative documents. At the bottom or top of each page insert this reminder: “Internal Docu ment Only; Do Not Cite, Quote or Release.”

A description of the items to be incl uded in an Action Memorandum follows.

DEADLINE

Indicate if there is a deadline of any sort and what that deadline is. It may be court imposed, statutory, or any one of a nu mber of other deadlines, e.g., upcoming national meeting, Administrator’s priority, etc.

OVERVIEW

The overview section of an Action Memora ndum provides background information which briefly describes the rule, characterizes the envi ronmental issue(s) or publ ic health problem(s) being addressed, and summarizes th e history of the action. It s hould explain why EPA is taking this action, and where appropriate , cover the following points:

$ Define any court or statutory deadline;
$ Identify whether the action amends the Code of Federal Regulations, and if so, explain what kind of amendment (procedural, etc.);
$ Identify other actions underway that will affect this particular program or sector;Peer Review Handbook Page C-6

$ Describe the specific environm ental issue(s), public health problem(s) and/or statutory requirements being addressed, and the goal intended by taking this action;
$ Describe what the action does, and specifica lly, how the regulated community is affected (performance standards, specific requirements); and
$ Describe how flexible the implementation of the action will be for States and regulated

entities.

IMPACTS

Summarize the costs and benefits, including a discussion of any non-monetized benefits, or contingent valuation and/or non-quantified benefits , of the action and the results of any economic analyses. Explain the roles of cost-benefit a nd cost-effectiveness analyses in shaping the approach chosen.

Discuss the economic impacts on all affected entities including, wher e appropriate, possible impacts on other Federal agencies, Stat e, or Local governments, and Tribes.

Summarize the results of the regulatory flexibil ity analysis and impacts on small entities (small businesses, small communities and small not-for-p rofit entities), including what was done to minimize those impacts. If applicable, summarize the SBREFA Panel recommendations, and how they were addressed in the rulemaking.

Describe the reporting and record -keeping burden and what the Agency has done to reduce it.

STAKEHOLDER INVOLVEMENT

Briefly discuss the role of both government entit ies and private sector stakeholders in the development of the action. Summarize the concerns they have raised and what the Agency has done to address them, or explain why we can’t.

INTERNAL REVIEW

Identify whether the action was developed under Ti er 1, 2, or 3. If the action was Tier 1 or 2, attach the summary memo from the Final Agency Review. Describe any ou tstanding issues from Final Agency Review or other internal review . Identify Program Offices or Regions that participated in the development of the action, along with any outstanding issues and why they cannot be resolved or accommodated. Also, pr ovide the basis for any decision made to not address an identified cross-media impact.Peer Review Handbook Page C-7

PEER REVIEW

If you did not use influential scientific information or a highly influential scientific assessment as defined by the Peer Review Handbook to support th e action, include the following statement in

the Action Memorandum:

There were no influential or hi ghly influential products supporti ng this action as defined by the Agency’s Peer Review Handbook.

If you did use influential scientif ic information or a highly influe ntial scientific assessment to support the action, include the following statement:

[Insert Name of AAship] has followed the Agency 's Peer Review Policy with respect to the underlying [influential scientific information or hi ghly influential scient ific assessment] supporting this action.

You may add any details you think are important , but you generally should not modify this compliance statement. If you used influential sc ientific information or a highly influential scientific assessment, but were not fully able to meet the Peer Review Policy, explain why.

PLAIN LANGUAGE

Briefly describe which elements of plain la nguage you used in the action, e.g., question and answer format, addressing the reader as “you,” using short sentences, etc.

OMB TRANSACTION

Identify the OMB determination and whethe r the action went to OMB for review under Executive Order 12866. Highlight significant issu es resulting from OMB's review under Executive Order 12866, including any si gnificant issues raised by othe r agencies participating in the interagency review under Ex ecutive Order 12866. Explain any substantive changes made to

the action as a result of recomm endations from OMB or the other agencies. If OMB agreed that your action was not significant and, therefore, not subject to EO 12866 review, or if OMB waived review of your rule, please so indi cate. If OMB was otherw ise involved, e.g., was briefed or received a courtesy copy for informal re view, describe the result s of this interaction.

ANTICIPATED EXTERNAL REACTION

Describe the type of response an ticipated from the various audien ces interested or impacted by the action. Identify both the involv ed stakeholders and the nature of their expected response. Characterize the likely reaction to the action by all interested parties including industry; environmental groups; Congress; State, Local, an d Tribal governments; and OMB. Explain what the Agency has done to mitigate anticipated adverse reactions.Peer Review Handbook Page C-8

STAFF CONTACT

Give the name and phone number of the person to contact if there are questions about the package.

RECOMMENDATION

Identify the action the Administrator is expected to take (i.e., sign the ru le or other document).

NOTE : Where an Action Memo accompanies another document (i.e., a rule or FR document) to be signed by the Administrator (or other au thorized official), you should not include a

concurrence line at the bottom of the Action Memo or anything else that might cause the Action Memo to be misinterpreted to be a Decision Memo (which it is not). The FR notice (or other document such as an order), after being signed, will contain the Agency’s decision or action (if any), not the Action Memo, which is onl y a pre-decisional briefing document.Peer Review Handbook Page C-9Peer Review Handbook Page D-1



# Appendix D

Examples Of Peer Review ChargesPeer Review Handbook Page D-2



# Appendix D - Examples Of Peer Review Charges

Please Note – certain questions that ar e posed in charges can be responded to with a yes or no answer. Clearly, this is not th e type of response the Agency ge nerally wants, therefore, it is important to phrase charge questi ons carefully to ensure that y ou receive a fully satisfactory and thoughtful response. Where a yes or no answer mi ght be expected, be sure to ask for a full explanation supporting th e yes or no answer.

Charges can run the gamut from rather simp listic to highly complex. The examples shown here cover a variety of types. Examples 1 thro ugh 4 have less complex que stions and are looking for the overall quality of the effo rts. Examples 5 through 7 have numerous technical questions

that need to be addressed and are th erefore more complex in their nature.

Other charges that have been used can be found on the SAB website at http://www.epa.gov/sab

and the SAP website at http://www.epa.gov/scipoly/sap/index.htmPeer Review Handbook Page D-3

CHARGE EXAMPLE 1 - US EPA Science Advi sory Board (SAB) Consultation on Suspended and Bedded Sediments

Background:

The Ecological Processes and Effects Committee (EPEC) of the EPA Science Advisory Board was asked by the Health and Ecological Crit eria Division of the Office of Science and Technology, Office of Water, to provide a consul tation on potential approaches for a strategy for developing water-quality criter ia for Suspended and Bedded Sediments (SABS). A request for nominations for consultant panel members to prov ide additional expertise to EPEC appeared in the Federal Register on Ju ly 30, 2003 (68 FR 44758-44760).

In 1976, EPA issued a water quality criteria recommendation under the Clean Water Act for solids and turbidity. For a variety of reasons, the States seldom, if ever, use this criterion. It is questionable whether this criterion would achieve intended protection for all different designated uses for water bodies. SABS occurs naturally in streams in a wide range of concentrations – levels that might be perfectly normal in one water body would be indicative of impairment in

another.

Although most States currently have water quality criteria that can be applied to manage SABS, these are typically based on turbidity, suspe nded solids or settle-able solids, and their effectiveness for dealing with all water quality impairments caused by SABS, especially as benchmarks for aquatic life protection based on natural levels, is que stionable. In recent consultation with State representatives, the need for new water quality criteria for SABS or methodologies for deriving them on a site-specifi c basis was identified as one of the highest priorities for the water quality criteria progr am. As a result, the EPA Office of Water has concluded that to better manage SABS in all types of water bodies and for all designated uses, State and Tribal water quality managers n eed new and updated water quality criteria and

information for SABS.

Charge to the panel:

While many questions and much research remain, EPA seeks th e opportunity for a consultation with the Science Advisory Board to gain a dvice and recommendations on the best potential approaches to developing water quality criteria for suspended and bedded sediments as will be described in a draft Strategy fo r Developing Water Quality Criteria for Suspended and Bedded Sediments (SABS) to be prepared by the Office of Water. The Office of Water is also seeking recommendations on additional criteria development and approaches for uses of water bodies other than aquatic life, and it is also seeki ng advice on any potential criteria derivation methodology not included in the Strategy.Peer Review Handbook Page D-4

CHARGE EXAMPLE 2 - Economics - Benefits Transfer from Adults to Children

Background

EPA established the Office of Children’s Health Pr otection (OCHP) in 1997 to support its efforts to increase the protection of children’s health throughout its programs. One of the many difficult issues the Office is addressing is the appropriate treatment of children’s health effects in the economic analyses performed by the Agency. Policy analysis efforts at the Agency often rely on the benefits transfer technique, and very few of the Agency’s benefit transfers have explicitly

addressed children’s health issues. In addi tion, no accepted systematic process for conducting benefits transfer currently exists. To assist th e Agency in its efforts, this paper discusses the benefits transfer technique as it applies to estimating values for children’s health. The first section provides some general background on the technique, and its ap plication to estimate

health-related values. The sec ond section raises important gene ral issues to consider when conducting a benefits transfer for children’s he alth values. The last section discusses the implications of using the benefits transfer method to estimate values for children’s health. The scarcity and state of existing child-oriented hea lth valuation literature suggests that it may be necessary to transfer adult-oriented values to estimate child-related values (Neumann and Greenwood 1999). However, the results of this paper suggest that transfer of these value estimates to children at best provides estimates for a scoping analysis. In cases where these scoping exercises indicate that ch ildren's health values may be a crucial component in the policy analysis, primary research should be undert aken to estimate child-related values.

a) Assess the appropriateness of transferring health benef it values estimated for adult populations to children. Describe the specific issues that aris e in these transfers. Which variables or situations impr ove or decrease the appropria teness of transferring benefit

values from adults to children?

b) Does the analysis support the proposition th at the value of children’s health effects should be estimated differently than a dult health effects? Why or why not?

c) Identify issues for further research that would improve our ability to estimate values for

children’s health effects.Peer Review Handbook Page D-5

CHARGE EXAMPLE 3 - Economics - Study of Municipal Government Costs and Financial Impacts from Environmental Regulations

Background

Municipalities play a major ro le in supplying environmental se rvices. Local governments have taken responsibility for providing drinking water, sewage treatment, and waste disposal in a majority of communities. Over the past fifteen to twenty years, most of the mandates found in the Federal environmental legislation enacted in th e early 1970s have been met. The increase in the number of people served and improvements in the quality of local en vironmental services have been considerable, as has the investment in public infrastructure to meet these laws.

Recent revisions to the environmental legislation have established a broader and more stringent set of standards to be met by suppliers of environmental services. As a result, many Local governments are now faced with having to maintain al l or some part of their public services at a higher level of performance. To meet these ne w standards will require additional investments in capital, and increases in rates charged to customers for environmental services.

Improvements in environmental services are but one of several demands being made of local public infrastructure. Studies prepared on public infrastructure needs and the availability of

funds to meet these needs indicate that there wi ll be an excess demand for money to rebuild and improve upon the existing stock of public infrastructu re. Therefore, it is important to recognize that additional environmental requirements will ha ve to compete with other infrastructure needs (e.g., highways, bridges), as well as other public services (e.g., police, education, health and welfare programs) provide d at the local level.

Given the increasing demand for public serv ices, this study examines what additional investments the new environmental legislation w ill require Local governme nts to undertake, and the likelihood that they will f ace difficulties raising the nece ssary funds through capital markets and revenues from customers. The economic imp acts of individual EPA actions are considered during the regulatory process in those situati ons permitted by environmental statutes. The unique feature of this study is it s attempt to estimate the cumulati ve costs and impacts of meeting a combined set of EPA requirements, and to determine whether they will place a significant burden on the fiscal conditions of Local government s, and require them to significantly increase existing charges for improved environmental services.

Please find attached a copy of th e draft study and appendices for your review. This version of the report reflects Agency comments received on an earlier draft. We expect that this version of the report, with some additional minor modifica tions, will be the final version. Your comments will be useful in preparing the final version of the report and discussing the findings of the reports with the public.

To assist in your review of the report, we ask that you pay particular attention to the following questions:Peer Review Handbook Page D-6

a) Do State and Local governments and fi nancial markets consider household costs (measured as a percent of household income) and selected municipal financial information (debt service to general reve nues or taxable prop erty values) when evaluating the ability of enterprise systems and municipalities to issue bonds or obtain

loans?

b) Having selected a series of financial indica tors, are the criteria used in the analysis acceptable?

c) Do the results support our conclusions? What additional conclusions can be reached from the analysis?

d) What modifications would you suggest be made to the recommendation section?Peer Review Handbook Page D-7

CHARGE EXAMPLE 4 - External Peer Review of the Design of the Targeted National Sewage Sludge Survey for Pollutants in Sewage Sludge

Background Information

The United States Environmental Protection Agen cy (EPA), Office of Water is charged with protecting public health and th e environment from adverse exposure to chemicals and microbial agents in water media, such as ambient and drinking waters, wastewater/sewage sludge, and sediments. In support of this mission the O ffice of Water/Office of Science and Technology (OST) develops human health and environmental standards, health criteria, health advisories, and technical guidance documents for water and water-related media.

Peer review is an important component of the scie ntific process. It provides a focused, objective evaluation of a research proposal , publication, risk assessment, health advisory, guidance or other document submitted for review. The criticisms, suggestions and new ideas provided by the peer reviewers ensure objectivity, stimul ate creative thought, strengthen the reviewed document, and confer scientific credibility on the product. Comprehensive, objective peer review leads to good science and product accepta nce within the scientif ic community and is required for the technical document that s upport Agency regulations and guidelines.

Targeted National Survey of Pollutants in Sewage Sludge

As EPA described in the December 2003 Federal Register notice (68 FR 75531), EPA plans to conduct a targeted survey of pollutant s (select pollutants) in sewage sludge. EPA has designed a targeted survey of selected chemical pollutants for which adequate data are available, or will be available, for conducting an exposure and hazard assessment concerning their presence in treated sewage sludge. Micr obial pollutants (pathoge ns and indicator organisms) in sewage sludge are also included in the survey design.

A survey will provide feedback for updating the science and technology of sewage sludge applied to land, disposed of in a surface disposal unit, or incinerated. The new concentration data would be used to assess human and ecologi cal risks of identified, unregulated pollutants found in sewage sludge and identify pollutants fo r potential regulation. In addition, the data on microbial constituents in sewage sludge will help in determining the e ffectiveness of pathogen reduction and elimination processes that are sp ecified in the 40 CFR Part 503 Standards.

EPA is committed in FY 2005 to starting a limited analytical survey of pollutants found in sewage sludge. EPA expects this survey to ad dress the pollutants identi fied by the exposure and hazard screening assessment as presenting a potentia l hazard, as identified in the current section 405(d)(2)(C) biennial review (68 FR 75531). The Agency has evaluated the extent to which sampling and analytical methodologies will allow e xpansion of the survey scope within available resources to include additional pollutants.

Questions for Peer Review ChargePeer Review Handbook Page D-8

a) The sample design proposes excluding f acilities with certain treatment types (i.e., less than secondary treatment, partial treatment, and ponds), remotely located facilities (i.e., outside the contiguous states), and non-publicly owned facili ties (i.e., Tribal and private). Do these exclusions appear to be appropriate? If you disagree with EPA’s proposed exclusions, your response should expl ain which groups should be included and why (e.g., what information will be lost if they are excluded).

b) The sample design proposes including facilities that the Clean Watershed Needs Survey (CWNS) identifies as biosolids hand ling facilities, because preliminary review indicates that they have the same operati ons and characteristic s as other POTWs for purposes of biosolids sampling. Do you agr ee with the inclusion of the biosolids handling facilities? If you disagree, your response should explain how their characteristics (e.g., type of treatment) would differ from other POTWs.

c) The sample design proposes including f acilities with no discharge, because they also may produce biosolids as a result of trea tment. Do you agree with the inclusion of no discharge facilities? If you disagree, your response should explain how their

characteristics would differ from other POTWs.

d) The sample design assumes that flow and mass are highly correlated. Do you agree that this is a reasonable assumption? If you disagree, your response should identify possible data sources for mass da ta, or describe alternative approaches to estimating mass values based upon available data.

e) The sample design proposes faci lity selection based upon the binomial distribution. Do you agree that this is the appropriate approach? If you disagree, your response should describe your objections to the rationale for selecting the binomial distribution. Your response also should recommend another approach, describe this approach (unless it was incl uded as an alternative in th e sample design), and provide

rationale for its selection.Peer Review Handbook Page D-9

CHARGE EXAMPLE 5 - External Peer Review of the Draft National Strategy for the Development of Water Quality Criteria fo r Suspended and Bedde d Sediments (SABS)

BACKGROUND INFORMATION:

EPA’s Office of Science and Technology (OST) ha s water quality criteria programs for aquatic life, human health, and for nutrients (total N, P, DO, chlorophyll, a nd clarity). EPA is initiating a National Strategy for the Development of Water Quality Criteria for Suspended and Bedded Sediments (SABS). SABS occur naturally in all types of waterbodies. In appropriate amounts, sediment is essential to aquatic ecosystems. However, imbalanced SABS conditions have repeatedly ranked high as a major cause of waterbody impairment. An imbalance of sediments resulting from human activities can impact ecolo gical integrity at seve ral scales and trophic

levels.

EPA’s Strategy for Water Quality Standards and Criteria, dated A ugust 2003, is a document which sets priorities to st rengthen the foundation for protec ting and restoring the Nation’s waters. Imbalance in loading of SABS to the aquatic systems is now considered one of the greatest causes of water quality impairment in the Nation. Turbidity, suspended solids, sediment, and siltation have been consistently listed as dominant pollu ting factors in 305(b) Water Quality Reports in rivers and streams, lakes and reservoirs, ponds, wetlands and ocean shoreline waters. In 1998, for ex ample, approximately 40% of a ssessed river miles in the U.S. were not meeting their water qu ality goals due to excessive se diments. One of the top ten priorities listed in the document is producing and implementing a strategy for the development of suspended and bedded sediment criteria. These priorities were developed based on extensive listening sessions and discussions with States and other partners.

To assess the effects of this gr owing environmental stressor, a nu mber of States have identified the specific measurements and levels at which se diment impairment occurs in their waters, but many States have not adopted such SABS criteri a into their State water quality standards. Information from an EPA survey conducted in 2001 indicates that numeric sediment criteria of some type were identified in 32 of 53 States, Te rritories and Washington, DC. But many States have not adopted such sediment criteria into th eir water quality standards. As a result, SABS problems are identified but perhaps underestimate d, and few controls (wat er quality standards, permit limits, TMDLs) are in place to reduce the overall impact of SABS.

Also, as part of the current effort to develop national SABS criteria, the EPA Science Advisory Board met on October 2, 2003 to discuss th e strategy, including various approaches for establishing criteria. The Science Advisory Bo ard recommended that several approaches to SABS criteria development should be synthesized in an overall approach that can identify impaired SABS conditions as compared to appropriate reference conditions. These recommendations are presented in the draft strategy document.

The methodologies presented in this draft strategy will be used by the States, Tribes, and Territories to better manage SABS in water bodies across the country. This strategy presentsPeer Review Handbook Page D-10

EPA’s blueprint for achieving the goals of developing sediment criteria methodologies and the implementation of these criteria into water quality standards.

National Strategy for the Development of SABS Criteria

The Strategy for the development of SABS criteria desc ribes the approach that EPA is planning to take for developing scientific information relating to SABS and working with States to encourage adoption of SABS criteria into State water quality standards pursuant to Section 303(c) of the Clean Water Act. The draft strategy for developing water quality criteria for SABS is modeled on the approach used by EPA to deve lop water quality criteri a for nutrients, where appropriate. However, the strategy does not establish dates by whic h States must develop water quality standards for SABS.

States, under pressure to develop and issue total maximum daily loads (TMDLs) for SABS impaired water bodies, are moving forward on their own to develop new and improved SABS

criteria. The draft strategy highlights some of the State programs and looks for approaches and methods that may be useful, either directly, or with adaptation, to the entire nation. Because current States’ efforts may serve as examples for development of numeric SABS criteria, a summary of States’ programs are provided. In order to describe a possible SABS criteria development process, an Oregon case study was prepared for a hypothetical aquatic system, following a watercourse from the watershed to th e receiving water (Section III.F of the draft

strategy ). The case study illustrates sediment criteria development through theoretical and actual examples.

EPA does not anticipate developing ecoregional SABS criteria for all waterbody types and all geographic regions. The responsibility to de velop and adopt SABS criteria, based on the information and methodologies presented in the draft strategy , will rest with St ates. EPA’s draft national strategy includes the following major elements:

a) Outline of actions to be taken by EPA, States, Territories, and Tribes to develop criteria, adopt criteria into water quality standards, and manage for SABS.

b) Identification of appropriate indicato rs of water resource impairment due to

imbalances of SABS.

c) A process to stratify waterbodies by waterbody type, region, and designated uses for the development of SABS criteria.

d) A series of approaches and technical tools for deriving SABS criteria specific to a region, waterbody-type, and designated use.

e) A list of resources from which States and Tribes can find assistance for deriving

SABS criteria.Peer Review Handbook Page D-11

f) A case study may be developed to i llustrate SABS criteria development, addressing both the theoretical basis and an example using a synthesis of approaches.

Following peer review and incor poration of peer review comme nts, the U.S. Environmental Protection Agency (EPA) will publish a draft National Strategy for the Development of Water Quality Criteria for Suspended and Bedded Sediment (SABS) and will ask the scientific community and the public for its views on the scient ific direction of its SABS strategy. Peer review is an important component of the scientific process. It provides a focused, objective evaluation of a research proposal , publication, risk assessment, health advisory, guidance or other document submitted for review. The criticisms, suggestions and new ideas provided by the peer reviewers ensure objectivity, stimul ate creative thought, strengthen the reviewed document, and confer scientific credibility on the product. Comprehensive, objective peer review leads to good science and product accepta nce within the scientif ic community and is required for the technical document that s upport Agency regulations and guidelines.

The Peer Reviewers charge is as follows:

a) Does EPA’s draft national strategy adequately describe methods for assessing the expected SABS condition of a waterbody, for developing SABS criteria and for assessing attainment of criteria by States.

b) Is a regional and watershed approach, as described in the strategy , likely to accomplish the desired goal of addressing th e sediment imbalance problems in the nation’s surface waters? If not, what alternatives do you suggest?

c) Does the draft strategy present a rational, practical and scientifically responsible approach that can be implemented by St ates and Tribes, given their resources?

d) Is the state-of-the-science sufficient to write technical methods manuals that will allow States, Tribes and territo ries to determine the best method(s) for developing SABS criteria into water quality st andards pursuant to Section 303 (c) and 304(a) of the CWA?

e) Are the assessment endpoints and indicators described in the strategy reflective of

the current or best available science?

f) Are the six major elements of the draft strategy adequate to help States, Tribes, and territories develop a nd adopt SABS criteria?

g) Does the case study illustrate se diment criteria development through the theoretical and actual examples presented?Peer Review Handbook Page D-12

CHARGE EXAMPLE 6 - CASAC Review of th e Agency’s National Ambient Air Monitoring Strategy

Background.

The draft National Ambient Air Monitoring Strate gy (NAAMS or Strategy) was revised after the Agency’s prior consultation with the former CASAC National Am bient Air Monitoring Strategy (NAAMS) Subcommittee in July 2003. This revi sion incorporates EPA’s responses to that Subcommittee’s recommendations. The primary recommendations from the NAAMS Subcommittee included a request for an implemen tation plan, and added emphasis on rural- and ecosystem-oriented monitoring, support fo r the National Core Monitoring Network (NCore) Level 1 program, and training and quality assurance to enhance data consistency across

the Nation.

The Strategy foresees moving resources from prog rams of decreasing value to those of a higher value which respects the partne rship across EPA, State, Local, and Tribal (SLT) agencies retaining stability for the monitoring programs a nd allowing SLT flexibility. The transition to the NCore network creates a need for training that addresses new methods, information transfer technologies, and an effective quality assuranc e program. There are programmatic and technical areas where some type of training or a transfer of information is requir ed. These training needs will be offered by various mechanisms, e.g., satellite broadcasts and videos; hands-on sessions; guidance documents; vendor training of instru mentation; web-based tr aining; and workshops. Broadening the Agency’s outreach to the health effects, atmospheric scientists and ecosystem

assessment communities are included.

The implementation plan incorporates acti on-oriented components of the Strategy, e.g., regulation revisions, training, f unding, and outreach approaches to facilitate the implementation of the NCore program. The revised monitoring re gulations will provide a legal basis for moving forward and will also alleviate some of the unnecessary burdens faced by monitoring agencies and enhance the ability to introduce new tec hnologies into our networks. Additionally, the regulations introduce the NCore system of: multip le-tiered monitoring stations; adjustment of minimum requirements for specific pollutant measurements; new methods performance specifications; periodic network assessments; and new quality assurance procedures. Written Meeting Materials. OAQPS has posted written review and background materials for this Subcommittee meeting on EPA’s Ambient Mo nitoring Technology Information Center (AMTIC) Web site. The Final Draft NAAMS docu ment, which was updated following the July 2003 meeting of the former CASA C NAAMS Subcommittee, is at http://www.epa.gov/ttn/amtic/files/ambient/monitorstrat/allstrat.pdf . Additional background materials for this meeting are found on the “CASAC File Area” page of the AMTIC Web site at http://www.epa.gov/ttn/amtic/casacinf.html . Furthermore, it is our understanding that the SAB Staff Office will post a copy of the final agenda and charge to the Subcommittee for this advisory meeting on the SAB Web site at http://www.epa.gov/sab (under “Meeting Agendas”) and the Subcommittee’s page at http://www.epa.gov/sab/panels/casac_aamm_subcom.html

, respectively, in advance of the Subcommittee’s meeting.Peer Review Handbook Page D-13

Charge to the Panel:

For this advisory meeting, OAQPS is requesti ng that the CASAC AAMM Subcommittee provide expert advice and recommendations on the fo llowing charge questions, which focus on key implementation issues:

a) The CASAC has expressed its support fo r the Agency’s proposal to redesign the routine PM monitoring netw ork to support PM precursor gas measurements (CO, SO2, NO/NOy, NH3, HNO3) at NCore Level II multiple-pollutant sites, and for air quality management decisions and to obtain relevant exposure data for research programs.

Questions: Given limited budgetary resource s, does this represent both an appropriate and adequate balance, as reflect ed by the relative resource allocations provided in Section 11, “Draft Implemen tation Plan,” of the Final Draft NAAMS Document? In addition, are the relative adjustments in the training and guidance approaches proposed in the draft implemen tation plan consistent with the overall objectives of the Strategy?

b) The implementation plan proposes a seri es of communication actions to advance the NCore Level 2 network, in order to more directly support long-term health effects research and provide better su pport to ecosystem assessments through an

increased level of coordination.

Questions: Does the Subcommittee have additional suggestions for addressing this need for integration and communicati on to the broader community of “users,” including scientific research ers (i.e., human health, at mospheric, ecological) and State, Local and Tribal (SLT) Agency re presentatives? More specifically, what is

the most effective manner for EPA both to reach-out to this broad user community and, where appropriate, to inco rporate their feedback and design input on such issues as monitoring si te locations and parameters?

c) One of the remaining technical issues relates to harmonizing rural- and urbanbased 2.5 chemical speciation networks su ch that both categories of networks utilize consistent sampling and analys is protocols. For example, EPA is considering converting all of the Spec iation Trends Network (STN) speciation sites to Interagency Monitoring of Prot ected Visual Environments (IMPROVE) samplers and IMPROVE laboratory a nd sample handling protocols.

Question: What are strengths and weaknesses of this approach? d) As EPA implements the National Ambi ent Air Monitoring Strategy to address multiple monitoring objectives, it will be looking to spatially optimize the ambient monitoring networks. This may m ean that some redundant monitors inPeer Review Handbook Page D-14

adjacent, but separate, geopolitical areas (e.g., neighboring counties) are “divested” from a given network. Alt hough technically sound, these divestments could result in data gaps which might, in turn, adversely impact regulatory decision-making. The Agency is willing to adopt alternativ e approaches for assessing regulatory issues such as non-at tainment designations, so long as such approaches are scientifically justifiabl e; hence, the rationale for initiating

discussion of these issues with the CASAC.

Question: Is it scientifica lly acceptable to generate air quality surfaces through

modeled observations and/or integrated predictive/observa tional fields that would be of appropriate uncertainty for use in the regulatory decision-making process?Peer Review Handbook Page D-15

CHARGE EXAMPLE 7 - The Multimedia, Multipathway, and Multireceptor Risk Assessment (3MRA) Model

Background:

The Office of Solid Waste (OSW) asked the Science Advisory Board to review the Multimedia, Multipathway, and Multireceptor Risk Assessment (3MRA) Model and this request was accepted by the Board, becoming SAB Project #03-13. A “W idecast” Federal Register notice requesting nominations for the Panel was published on April 11, 2003 (68 FR 17797).

In brief, the development of the 3MRA me thodology has a history in the Hazardous Waste Identification Rule. EPA plans to use the modeling system to help inform managers on a variety of decisions in the waste program, such as settin g concentration-based exit criteria for wastes in the hazardous waste management regulations, or deciding whether technology-based standards are protective of human hea lth and the environment.

In December 1995, EPA's Office of Solid Waste proposed to amend existing regulations for disposal of listed hazardous wastes under th e Resource Conservation and Recovery Act(RCRA). This Hazardous Waste Identification Rule (HWIR) was designed to establish constituent-specific

exit levels for low risk solid wastes that are currently captured in the RCRA subtitle C hazardous waste system. The EPA's Science Advisory Board (SAB) reviewed the proposed HWIR methodology for calculating exit concentrations, concluding that the methodology ``lacks the scientific defensibility for its intended regulat ory use,'' the SAB also made recommendations that, when implemented, should provide an adequate scientific basis for es tablishing a risk-based methodology applicable at the national level for the waste program.

In response, the Office of Solid Waste (OSW) collaborated with the Office of Research and Development (ORD) to develop and document a sound science foundation, supporting data for an assessment, and related software technology for an integrated, multimedia modeling system (entitled 3MRA) following the recommendations of the SAB and other reviewers. The Multimedia, Multipathway, and Multireceptor Risk Assessment (3MRA) modeling system represents a collection of scienc e-based models and databases that have been integrated into a

software infrastructure that is based on th e FRAMES (Framework for Risk Analysis in Multimedia Environmental Systems) concept, whic h provides a computer-based environment for linking environmental models and databases a nd managing the large amounts of information within the system, including the visualization of outputs. This integrated multimedia modeling system provides national-level estimates of human and ecologi cal risks resulting from long

term(chronic) chemical release from land-ba sed waste management units. Over 45 experts participated in the peer revi ew process of the underlying scie nce within the 3MRA modeling

system.

Charge to the Panel:Peer Review Handbook Page D-16

The SAB Staff Office Director and OSW staff concurred that the panel should focus upon the following four areas: assessment methodology, 3MRA modeling system, modeling system evaluation, and modeling system documentation. Th e panel will respond to the following charge questions:

a) Charge Question 1: While EPA had the a ssessment methodology peer reviewed prior to the development of the 3MRA modeling system, does the SAB have any additional comments about the methodology as implemented?

b) Charge Question 2a: Does the 3MRA mode ling system provide a tool for performing

national risk assessments that facilitates consistent use of the science and provides a mechanism for reproducing results?

c) Charge Question 2b: Does the 3MRA m odeling system provide Decision Makers sufficient flexibility for understanding the impacts on potential chemical exemption levels by allowing varying measures of protection based on the number of receptors and/or number of sites protected, types of human and ecological receptors, and distance?

d) Charge Question 2c: Does the 3MRA mode ling system provide appropriate information for setting national risk-based re gulations for the waste program?

e) Charge Question 3a: Is the software de velopment and verifica tion testing approach implemented for the 3MRA modeling system sufficient to ensure confidence that the modeling results reflect the modeling system design?

f) Charge Question 3b: Given the thorough eval uations that EPA has implemented using the available data resources and technologies , while also recogni zing the real world limitations that apply to validating the 3MRA modeling system, have we reasonably demonstrated through met hodology design, peer review, quality control, sensitivity analyses, and model comparison, that the 3MRA modeling system will produce scientifically sound results of high utility and acceptance with respect to multimedia regulatory applications?

g) Charge Question 4: Has EPA made substant ive progress, relative to 1995, in designing and preparing documentation for the 3MRA modeling system? Does the SAB have additional suggestions for improving the presentation of the comprehensive set of materials related to this modeling system?Peer Review Handbook Page E-1



# Appendix E

Guidance on Requesting a Review by the Science Advisory Board (SAB)Peer Review Handbook Page E-2Peer Review Handbook Page E-3



## APPENDIX E – GUIDANCE ON REQUESTING A REVIEW BY THE SCIENCE ADVISORY BOARD (SAB)

Introduction

The Science Advisory Board Sta ff Office (SAB SO) solicits nomina tions for project proposals to be reviewed by three of the Agency’s independe nt scientific advisory committees the Science Advisory Board (SAB); the Clean Air Scien tific Advisory Committee (CASAC); and the Advisory Council on Clean Air Compliance Anal ysis (Council). This Appendix provides guidance to EPA Program Offices and Regions regarding the proj ect proposal nomination and selection process. More detailed information about the functions and advisory process for the

SAB can be found in the Implementation Plan for the New St ructural Organization of the EPA Science Advisory Board (EPA-SAB-04-002) availabl e on the SAB website at http://www.epa.gov/science1/pdf/sab04002.pdf

.

Background

A key priority for the Administrator is to base Agency actions on sound scientific data, analyses, and interpretations, as well as independent peer review. The Administrato r issued the Agency's Peer Review Policy to increase the quality of the technical founda tions upon which EPA's regulatory structures are built. The SAB, CA SAC, and the Council provide mechanisms for EPA to receive independent scient ific peer review and advice regarding the development and use of science at EPA. The following informa tion is intended to help programs and Regions

determine which scientific advi sory projects to submit to the SAB, CASAC and Council.

The SAB, CASAC, and Council provide several ki nds of advisory functions including the following kinds of activities:

• Consultation – Provides non-consensus, oral advi ce on a technical issue before EPA begins substantive work on that issue.

• Advisory – Provides written advice on EPA' s technical works-in-progress.

• Peer Review – Conducts a review of EPA's fi nal draft techni cal reports (e.g., guidelines, assessments, research strategies) or work products (e.g., analytical methods, models, databases).

• Commentary – Provides forward-looking advice on an important technical or emerging issue in the form of a short communication (SAB function).

• Original Study – Conducts original work on an emerging or overarching topic of importance to EPA (SAB function).Peer Review Handbook Page E-4

• Other Activities – Receives information briefings from EPA and conducts scientific workshops on specific technical subject matters.

Because resources are always limiting, the SAB Staff Office uses several criteria for selecting project proposals proposed by the Agency. A dvisory project proposals best suited for consideration by the SAB, CASAC, and Council are those that m eet several of the following

criteria:

• General Criterion

S Provides an opportunity to make a diffe rence in the scienc e that supports the Agency's mission

• Client-Related Criteria

S Supports major regulatory or risk management initiatives

S Serves leadership interests (e .g. the Administrator or Congress)

S Supports EPA strategic priorities

• Science-Driven Criteria

S Involves scientific approaches that are new to the Agency

S Addresses areas of su bstantial uncertainties

• Problem-Driven Criteria

S Involves major environmental risks

S Relates to emerging environmental issues

S Exhibits a long-term outlook

• Organizational Criteria

S Serves as a model for future Agency methods

S Requires the commitment of substantial resour ces to scientific or technological development

S Transcends organizational boundaries, within or outside EPA (includes international boundaries)

S Strengthens the Agency's basic capabilities.

In addition, the SAB Staff Office considers the overall mix of the nominated project proposals for a specific fiscal year, as well as the time a nd available resources needed to take on the projects. Table 1 depicts the processes for identifying, nominating, and selecting Agency nominations, emerging issues for strategic advice, and de novo studies. The SAB, CASAC and Council employ similar processes, but projects for the latter two advisory committees are linked to specific statutory mandates.

Process for Submitting Nominations Any Office desiring to take a product, activit y, or issue to the SAB, for a peer review, advisory, or consultation, is requested to comp lete the two-step process, described below.Peer Review HandbookPage E-5TABLE 1 – PROCESS FOR SELECTING SCIENCE ADVISORY BOARD (SAB) PROJECTSSTEPS TO SABTYPEPROJECTOFSELECTIONPROJECTIDENTIFICATIONFORA FORDISCUSSIONAND NOMINATIONPRIORITIZATION ANDSELECTIONDEFINING THESCOPE ANDCHARGEAPPROVALAGENCYREQUESTSAchieved throughdiscussions within EPAOffices to determineprojects which wouldbenefit from externalreviewStaff Office receivesnominations from EPAAssistantAdministrators (AAs)or RegionalAdministrators (RAs)- Annually- As needed basisAchieved through joint discussionswith:- Science Policy Council-EPA Program/Regional offices-SABSelection based on:- Statutory Requirement- Agency needsAchieved through jointdiscussions with:- SAB Staff Office- the requesting EPAProgram/Regional officeSAB Staff Office considers:- Availability of resourcesIDENTIFYINGEMERGINGISSUESFORSABSTRATEGICADVICEANDDE NOVOSTUDIESAchieved throughInteractions with:EPA Program/RegionalOfficesScientific CommunityPublicScience Advisory BoardSAB Staff OfficeSAB Annual MeetingSAB CommitteeMeetingsBoard MeetingsMeetings with PublicScience Policy CouncilEPA Program/RegionalOfficesScientific WorkshopsAchieved through joint discussionswith:- Science Policy Council- EPA Program/Regional offices-SABTo identify projects for furtherconsideration.- Strategic Advice by SAB-De novoSAB projectSelection based on:- Impact on Agency’s mission- Innovative nature of project- Agency needsAchieved through jointdiscussions with:- Designated SABCommittee- EPA Program/Regional OfficeSAB Staff Office considers:- Availability of resourcesPeer Review Handbook Page E-6

Step 1 – Project Identification and Nomination. Each year, the Assistant Administrators and Regional Administrators are aske d to send the SAB Staff Office Director a memorandum that lists all advisory project nominati ons, with the highest priority no minations for the next fiscal year identified. SAB Project Sheets (see Item 2 below) for nominated projects should be

attached to the memorandum. The SAB Staff Of fice and the Science Policy Council also request this memorandum to identify four to six other hi gh-priority science activities that the Assistant Administrator or Regional Administrator has deci ded not to nominate for SAB peer review. The latter information is requested, so that the SA B may better understand nominated projects within the context of high-priority science activities at EPA. The SAB Staff Office requests this information to assist the Science Policy Council in its discussions and decisions about the most appropriate set of projects for the Agency to bring to the Board, CASAC , or the Council.

Step 2 – Electronic Project Sheet. Nominators are asked to submit an electronic project sheet for each individual project to be considered for SAB, CASAC, or Council attention. The project sheets are created after establishing or updating the related Peer Review Project or Science Activity in the Agency’s Science Inventory. Th e Science Inventory entr ies must be approved by

their Peer Review Coordinator or Science Activity Coordinator for the specific Program Office or Region. Project sheets shoul d be filled out for all desired projects, including previously submitted projects for which no project planning meeting has occurred between the Program Office or Region and the SAB Staff Office Director or the Associat e Director for Science. The electronic Project sheet may be accessed and completed through EPA’s Science Inventory database. The information fields required for th e Electronic data sheet ar e provided in Table 2.

Process for Keeping Informed about the Decisions Made

After receiving project nomina tions, the SAB Staff Of fice will discuss pr oject priorities with each EPA Program Offices and Region, and disc uss overall Agency priorities with the EPA Science Policy Council. The Staff Office will also discuss project priorities with the SAB, which

includes the CASAC and Council chairs. Th e SAB Staff Office will then develop a recommended list of advisory projects based on EPA and SAB rankings taking into account

available resources. The SAB Staff Office will consult with the EPA Administrator and Deputy Administrator to develop an annua l operating plan that includes the highest priority projects. Additionally, projects may be adde d or deleted at any time during th e fiscal year, as requested by EPA senior executive management, Congress, and the SAB.

EPA Staff may refer to the SAB Product Data base in an ongoing way to identify all the projects nominated for the Board and the status of decisions made on them. The database will also provide ongoing information about the status of SAB reports and meetings related to projects accepted for Board advice in Fiscal Year 2006.Peer Review Handbook Page E-7

EPA Staff with questions about the SAB Pr oduct Database or the process for submitting nominations to the EPA SAB may contact Angela Nugent (nugent.angela@epa.gov, 202-343

- 9981) or Ms. Patricia Thomas (thomas.patricial.epa.gov, 202-343-9974).

Table 2: Information Fields for SAB Project Sheet

1. Project Title

2. Project Short Title

3. Fiscal Year SAB Activity Desired to Begin

4. Quarter SAB Activity Desired to Begin

5. Requesting Organization

6. Requesting Office

7. Requesting Official

8. Requesting Official's Title

9. Program Contact

10. Program Contact's Phone

11. Program Contact's Mail Code

12. Background for this Project:

13. Tentative Charge:

14. Applicable GPRA Goal and Objective

15. Description of and citation for any legal obligation/directive for SAB Review:

16. Principal interested and affected parties

17. Type of SAB advice requested

18. Why Should the SAB Advise on this Project?

19. Disciplinary Expertise

20. Budget:

FY
Extramural Budget

FTE

21. Past Peer Reviews:

22. Quality Management/Q uality Assurance:Peer Review Handbook Page F-1



# Appendix F

Example Statements Of Work for

Contracts

Users of this Peer Review Handbook need to be aware that the examples contained in this Appendix are generalized statements of work prepared (and in some cases modified) to emphasize certain important features (e.g., attention to conflicts of intere st, responsibilities of

contractors or contractin g officers, development and use of the charge to peer reviewers). Please be sure that proper and currently approved cont ract language is used in any EPA contract document at the time of award (for ex ample, see EPAAR 1552.212-71, alternate I).Peer Review Handbook Page F-2

STATEMENT OF WORK - EXAMPLE 1 - Peer Review of Prioritization Tool Report

Work Assignment No. :

Title : Peer Review of Prioritization Tool Report

Work Assignment Manager (WAM) :

Name:

Address:

Phone No.: John Q. Government Employee

Office of Solid Waste (202) 564-XXXX

Background :

The Waste Minimization Branch (WMB) in the Offi ce of Solid Waste (OSW) is in the process of implementing the Waste Minimization National Plan, announced by the Agency on November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction over waste management, in keeping with the policy stat ed in the 1984 amendments to the Resource, Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act (PPA). The Plan outlines major goals, objectives, and action items to achieve national reductions in the generation of hazardous wastes.

One of the objectives of the Pl an is to: "develop a framework for setting national priorities; develop and distribute a flexible sc reening tool for identifying priorities at individual facilities; [and] identify constituents of concern." This objective is a ke y building block in implementing subsequent objectives of the Plan.

In September 1995, WMB formed the Waste Mini mization Prioritization Team, which includes representatives from EPA Regions and states, to implement this objective. The Team has worked to assess stakeholder needs for prioritizati on tools and to evaluate prioritization tools that are currently available. The Team plans to summarize this work, along with its recommendations, in a report (r eferred to herein as the Prioritization Tool report) that would be available in draft form in July 1996.

WMB and the Team wish to obtain independent peer review of the Prioritization Tool report prior to briefing EPA management. The report is being prepared with th e support of ICF, Inc.; therefore, for the peer review to be considered independent, it must be performed by another

contractor.

Purpose and Scope of Work :

The purpose of this work assignment is to prov ide support to WMB and the Team in finalizing

the Prioritization Tool report by conducting an independent peer review of the report.Peer Review Handbook Page F-3

Work Statement :

Task 1 -Management work plan and budget

Within 15 days of CO approval of this work assignment, the contractor shall deliver a management work plan including a proposed level of effort, schedule, and budget for all tasks.

Task 2 -Provide independent peer review of Prioritization Tool report

The contractor shall provide support to WMB and the Team in preparing the Prioritization Tool report by performing an independent peer review of the report. The contractor shall establish a panel of peer reviewers includi ng three senior-level persons w ho collectively have extensive expertise in particular areas to be identified by the WAM upon approval of the work assignment.

Within three weeks of work assignment approval, receipt from the WAM of the necessary qualifications of peer reviewer s (in a TD), and receipt from the WAM of the peer review “charge” (in a TD), whichever comes latest, the cont ractor shall identify the three peer reviewers and prepare a memo that lists the names of the peer reviewers and their affiliations and includes the peer reviewers’ bio’s. Within five weeks of WAM approval of the of the peer reviewers (via a TD) and receipt of the draft Prioritization Tool report from the WAM (via a TD), whichever comes later, the contractor shall conduct the pe er review, assemble the peer review comments and recommendations in a peer review repor t organized by charge question, prepare an introduction to the peer review report with a clear and concise overview of the comments, and attach to the peer review report any marg inal comments the peer reviewers had on the
Prioritization Tool report.

It is not necessary that the peer reviewer s jointly reach consensus on their findings and recommendations, since there may be limited overla p in the peer reviewers' areas of expertise and in the charge questions that they focus on. The contractor shall a ssume, for the purpose of estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200 pages of appendices, and that each peer reviewer will spend 40 hours in reviewing the report and writing comments. EPA plans to provide the report to the contra ctor in mid-July.

Deliverables and Schedule :Peer Review Handbook Page F-4

Task Deliverable Schedule

1 Work plan and budget Within 15 days of CO approval of work assignment

2 Memo identifying peer reviewers Within 3 weeks of wo rk assignment approval, receipt of peer reviewer qualifications from WAM, and receipt of charge from WAM,

whichever comes latest

3 Peer review report Within five weeks of WA M approval of peer reviewers and receipt of

draft Prioritization Tool report from WAM, whichever comes later

Other Requirements :

CONTRACTOR COMMUNICATIONS

Upon approval of the Work Plan, th e contractor shall maintain at least weekly communications with the Work Assignment Mana ger regarding the st atus of work on the Work Assignment.

CONFLICT OF INTEREST (COI) [Revised Section]

The contractor must adhere to the following requirements:

a) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior to commencement of any work, notify bot h the CO and Project Officer of any actual or potential organizational or personal conflicts of interest.

b) Provide a written certifica tion, within 20 days of receipt of a Work Assignment, QRT, or similar task ing document, that:

1) Either all conflicts of interest ha ve been reported to the CO or that no

conflicts of interest exist. The contractor is direct ed to assure that none of

the conflicts disclosed are so direct and substantial as to rule out a particular reviewer.

2) All personnel who perform work under this Work Assi gnment or relating to this Work Assignment have been in formed of their obligation to report personal and organizational conflic ts of interest to the CO.

3) The Contractor recognizes its c ontinuing obligation to identify and report any conflicts of interest arising during performance of this Work Assignment.

c) If a conflict of inte rest is identified during performance under this Work Assignment, the Contractor shall immediat ely make a full disclosure in writing to

the CO. The disclosure shall include a de scription of action which the ContractorPeer Review Handbook Page F-5

has taken or proposes to take, after cons ultation with the CO, to avoid, mitigate,

or neutralize the conflict of interest.

d) After selecting the peer reviewers but before starting the peer review, submit to

EPA documentation that shows that the c ontractor has determined if the peer

review candidates:

1) have a conflict of interest or a s ituation that could crea te the appearance of a lack of impartiality in rela tion to the work product, and

2) have had or presently have a financial relationship with EPA.

This documentation summarizes for EPA the Contractor’s efforts to identify and propose resolution of these c oncerns with peer review candidates.

EXPENDITURE OF FUNDS/HOURS

In addition to the requirements of the contract, the contractor shall notify both the Project Officer and the Work Assignment Manager when 75% of f unds or hours for this Work Assignment have been expended.

INFORMATION COLLECTION

Any other provision of this Work Assignment notw ithstanding, the contract or shall not proceed with any information collection where the same or similar information will be collected from ten or more public respondents until wr itten approval is received from the Contracting Officer. This approval will cite an approval number from the Office of Management and Budget as required by the Paperwork Reduction Act (PRA).

Only Federal agencies and their employees ar e exempt from the PRA definition of "public respondent." State agencies and their employ ees are classified as "public respondents."

Soliciting similar information applies to any coll ection method, i.e., written, oral, electronic, etc., and utilizing any approach, i.e., surveys, phone calls, focus groups, TQM, etc. The PRA applies equally to "willing participants" and participation that is mandated by law.

Any question of applicability of the PRA shall be resolved by submitting a complete description of the circumstances in a written request to the Contracting Officer. No collection shall be undertaken until the Contracting Officer provides written notice to the Contractor as to the applicability of the PRA. If the PRA is determ ined to be applicable, the Contractor shall not initiate any collection until the requisite approval is received.

The General Services Administration (GSA), under FIRMR Bulletin B-2 administers the Interagency Reports Management Program as de rived from 44 USC Chapters 29 and 31. AllPeer Review Handbook Page F-6

work performed under this Work Assignment i nvolving Federal interage ncy reporting must be done in full compliance with these GSA procedures.

CONFIDENTIAL BUSINESS INFORMATION

If this Work Assignment requires use of RCRA Confidential Business Information (CBI), the contract must specifically authorize the c ontractor to have access to RCRA CBI and the contractor shall abide by all RCRA CBI require ments and stipulations found in the RCRA CBI Security Manual and in the contr act. The contractor shall iden tify in the Work Plan budget all estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon as it is no longer needed under th is Work Assignment or before the expiration of the Work Assignment, whichever occurs first.

PRINTING AND DUPLICATION

The contractor is prohibited from performing any printing under the Government Printing and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000 impressions of a single-page document or 25,000 impressions of a multiple-page sta nd-alone document, is limited to one color (black) copies, and does not exceed the maximum image size of 10 3/4 by 14 1/4 inches. For all duplication jobs in excess of 5,000 impressions, the EPA WAM will determine in advance if the work can be perfor med more cost effectively and under the job or time constraints at the EPA Print Shop. If the total number of photocopies for this Work Assignment exceeds 5,000 impressions, the contract or shall identify in their Work Plan the photocopying costs by task and deliverable.

WORK ASSIGNMENT/WORK PLAN BUDGETS

The contractor shall not exceed either the dollar or PL hour budget contained in the approved

Work Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not exceed the PL hour budget of the QRT.

TECHNICAL DIRECTION

The Designated Work Assignment Manager (WAM) on this Work Assignment is authorized to provide technical direction to the extent allowed under EPAAR (1552.237-71) (APR 1984) (DEVIATION). Other than the Designated WAM, only the Project Officer and the Contracting

Officer are authorized to provide technical direction.

Technical direction includes:

(1) Direction to the contractor which as sists the contractor in accomplishing the Statement of Work;

(2) Comments on and approval/acceptan ce of reports or other deliverables.Peer Review Handbook Page F-7

Technical direction must be with in the contract and the Work Assignment statement of work. The Project Officer and the WAM do not have the authority to issu e technical direction which

(1) institutes additional work outside the sc ope of either the contract or this Work Assignment;

(2) constitutes a change as de fined in the "Changes" clause;

(3) causes an increase or decrease in the estimated cost of the contract or Work Assignment;

(4) alters the period of performance or deliverable due dates; or

(5) changes any of the other express term s or conditions of the contract or Work Assignment.

Technical direction will be issu ed in writing or confirmed in writing within five (5) calendar days after verbal issuance. The technical direction memorandum will be provided to the contractor and copies will be forw arded to the Contracting Officer and the Project Officer. If the

contractor has not received written confirmation within five (5) calendar days of an oral issuance, the contractor must notify the Project Officer.

INHERENTLY GOVERNMENTAL FUNCTIONS

The contractor shall not perfor m any inherently governmental functions (IGF) under this Work Assignment. If during the course of developing the plan of wor k, through receip t of technical direction, or in carrying out the a ssignment any portion of the effort is considered to possibly be an inherently governmental function, the contract or must immediately notify the Project Officer and the Contracting Officer.

OCCUPATIONAL HEALTH AND SAFETY

Facility site visits conducted under a Work Assignment that include on-site inspections or sampling must be conducted in full complian ce with the Department of Labor, Occupational Safety, and Health Administration rule s under 29 CFR Part 1910 and EPA Order 1440 (Occupational Health and Safety Manual).Peer Review Handbook Page F-8

TRAVEL COSTS

The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal regulations in incurring allowa ble travel costs under this Wo rk Assignment, and correspondingly

must at all times seek and obtain Government rates whenever available and observe current subsistence ceilings.

QUICK RESPONSE TASKS

Each Quick Response Task (QRT) shall be c onfirmed in writing and approved by the Project

Officer. The contractor shall re spond by letter to the Project Of ficer with copies to the WAM and the CO within two working days, giving a br ief description of the plan of work, including best estimate of hours (by P-le vel) and a break-out of co sts to accomplish the task.

No task shall exceed a duration of 30 calendar days from start date to completion date. The level

of effort for each task shall be lim ited to a maximum of 250 labor hours.

Quick Response Task Requests do not change the dollar or professional la bor hour budgets of a Work Assignment.Peer Review Handbook Page F-9

STATEMENT OF WORK - EXAMPLE 2 - Ext ernal Peer Review of Protozoa Method Development Criteria Document

Period of Performance: Work Plan Approval to August 1, 1997

Work Assignment Manager: Sally Q. Government Employee

Office of Water

U.S. Environmental Protection Agency

LOE: 196 hours

SOW: 2.4

BACKGROUND INFORMATION:

The United States Environmental Protection Agen cy (EPA), Office of Water is charged with protecting public health and th e environment from adverse expos ure to chemicals and microbials in water media, such as ambient and drinking wa ters, wastewater/sewage sludge and sediments. In support of this mission OW’s Office of Science and Technology (OST) develops health standards, health criteria, h ealth advisories, and technical guidance documents for water and

water-related media. Under th is work assignment, documents prepared by OST are to undergo peer review.

Peer review is an important component of the scie ntific process. It provides a focused, objective evaluation of a research proposal , publication, risk assessment, health advisory, guidance or other document submitted for review. The criticism, suggestions and new ideas provided by the peer reviewers stimulate creative thought, st rengthens the reviewed document and confer credibility on the product. Comprehensive, ob jective peer reviews l eads to good science and product acceptance within th e scientific community.

Under this work assignment, the contractor will receive one document (Protozoa Method Development Criteria Document) for peer revi ew which is related to human health and ecological effects.

STATEMENT OF WORK:

Task 1. The contractor shall develop a work pl an to address all tasks in this work assignment. The work plan shall descri be the steps that will be taken by the contractor to provide for peer review, including sel ection of peer reviewer candidates with appropriate e xpertise, determining absenc e of conflict of interest, document and reference distribution, esta blishing schedules, preparing the peer review report, and submittal of the peer review package. Curriculum vitae for all persons assigned to complete this work a ssignment shall be provided. All P levels,Peer Review Handbook Page F-10

hours and total costs for each task will be provided and costs greater than $100.00

shall be itemized in detail.

Task 2. The contractor shall select a group of peer reviewers and determine their availability for the task a nd absence of conflict of inte rest, and establish a schedule for the peer review. The cont ractor is directed to ensure that none of the conflicts

disclosed is so direct and subs tantial as to rule out a par ticular reviewer. Three peer reviewers shall participate in the review. No single peer reviewer may charge more than 40 hours to this task. It is fully a cceptable for peer reviewers to commit to less than 40 hours. The peer review will be conducted for the Protozoa Method Development Criteria Document. Revi ewers selected by and working for the contractor shall be approve d by the EPA Project Officer in writing prior to their beginning work. Minimally, all peer revi ewers shall be accomplished in protozoan methods for sample recovery and analysis from water. Approval submissions shall

include the reviewers’ names and curriculum vitae.

Task 3. The contractor shall arrange for the sele cted peer reviewers to review the EPA document. Prepare the charge to the p eer reviewers based on technical direction

received from the EPA WAM. Provide th e peer reviewers with copies of the candidate report and all relevant references and instruct the sele cted peer reviewers to undertake the review. The WAM will pr ovide the contractor with the final

version of the document to be reviewed.

Task 4. The contractor shall monito r peer reviewers’ progress to ensure timely completion. The contractor shall collate peer review comments, and organize the comments in the peer review “for comments” document. The contractor shall provide the peer

review document and all materials subm itted by the peer reviewers to the EPA

WAM.

SCHEDULE AND DELIVERABLES:

Task 1. (Work Plan) 15 days afte r receipt of work assignment

Task 2. 1 week after work plan approval

Task 3. 1 week after selection of peer reviewers

Task 4. 1 week after receiving comments from the peer reviewers

TRAVEL: No travel is anticipated under this work assignment. A ny travel directly chargeable to this work assignment must be submitted and approved by the Project Officer.Peer Review Handbook Page F-11

STATEMENT OF WORK - EXAMPLE 3 - Tec hnical Support for Peer Review Services

INTRODUCTION

Background

The mission of the United States Environmental Pr otection Agency’s Office of Water (OW) is to protect public health and the environment from adverse effects of pollutants (e.g., chemicals and microorganisms) in media such as ambient water, drinking water, wastew ater, sewage sludge and sediments. In fulfillment of this mission, OW’s Office of Science and Technology (OST) develops effluent guidelines, hum an health and ecological criteria , health advisories, directs the national program for adoption of Water Quality St andards, develops prevention and remediation strategies to reduce the risk to human health and aquatic life re sulting from contaminated fish and sediment, provides technical support to the Total Maximum Daily Load program, analytical methods for detection and quan tification of pollutants in th e environment and assesses, remediates or otherwise manages and communicat es the risks associated with exposure to contaminants and microbial pathogens.

In January 1993, responding to recommendations in the report “Safeguarding the Future: Credible Science, Credible Decisions,” Administrator William Reilly issued an Agency-wide policy for peer review. Administ rator Carol Browner confirmed a nd reissued the policy on June 7, 1994 and instituted an Agency-wide implementa tion program. The Agency policy guides the involvement of peers in the Agency’s devel opment of products. This involvement may be through peer consultation or peer review of products. The goal of the Peer Review Policy is to implement a comprehensive Agency-wide program to ensure that major scientific and technical work products receive critical scru tiny from scientific and technical experts as part of the overall decision making process.

Purpose

The purpose of this contract is to provide peer consultation and peer revi ew support services to OST consistent with the Agency Peer Review Po licy. (Other offices with in the Office of Water

can utilize this contract for efforts within th is Statement of Work, if the OST Project Officer determines capacity is sufficient, based upon OST projections and requirements). The purpose of peer review is to uncover any technical problems or unresolve d issues for use in revising a preliminary product so that the final work pr oduct will reflect sound tech nical information and analyses. These processes enhance the quali ty of and provide fo r objective review and evaluation of scientific or technical work by know ledgeable experts who have no vested interest in related policy or implementation issues.

Scientific and technical work pr oducts identified for peer review may address one or more of the following:

o Support major regulatory decisions or policy/guidance of major impactPeer Review Handbook Page F-12

o Establish a significant precedent, model or methodology

o Address a controversial issue

o Focus on a significant emerging issue

o Have significant cross-Agency/interagency implications

o Involve significant invest ment of Agency resources

o Consider an innovative appr oach for a previously defined problem/process/methodology

o Satisfy a statutory or other legal mandate for peer review.

o Other products deemed appropriate by OST

The contractor is responsible for preparing repor ts on peer review evaluations and shall provide support for all stages or portions of the peer review process. Th e term peer review shall mean both peer consultation and peer re view of products. Peer review will include initial research protocol or work plan, prelimin ary review of an interim draft, review of a draft final work product and final evaluation of the work product. All peer review pr ocedures, products, and responsibilities shall be in acco rdance with the latest revisi on of the “U.S. Environmental Protection Agency, Peer Review Handbook .”

Statement of Work

1.0 Administration and Management

1.1 Candidates for peer review include scie ntific, engineering and economic documents, methodologies, strategies or positions that are used to support a research agenda, regulatory program, policy position or other OST decision. OST has several gene ral categories of scientific and technical work products which are subject to peer review: Criteria Methodologies/Criteria and Advisories; Engineering and Technical Studi es; Reports to Congress; Technical Guidance and Methodologies; Economic Evaluations; St atistical Analyses; Environmental Models; Technical Publications, Products , and Approaches; Analytical Methods; and Policies and Strategies. The contractor sh all provide the necessa ry professional and te chnical personnel for review of the following types of products:

o Toxicological Assessments

o Exposure Assessments

o Analytic Methods

o Statistical Analyses

o Dose-Response Assessments

o Microbial Risk Assessments

o Environmental Sampling Methods

o Monte Carlo Analyses

o Risk Assessment Guidelines

o Engineering and Technology Analyses

o Economic Analyses

o Technology AnalysesPeer Review Handbook Page F-13

1.2 The contractor shall appoint a person to serv e as the Program Manager for this contract. The Program Manager shall be the main point of c ontact for the contract; shall be responsible for the performance of work under this contract, an d shall be designated a key person in accordance with the Key Personnel Clause in EPAAR 1552.237-72(APR 1984). The contractor shall manage all aspects of this contract and shall be solely responsible for th e direction and selection of all contractor, subcontrac tor and consultant personnel who may be involved in the performance of the contract.

1.3 The contractor shall furnish the facilities, materials, equipment and necessary professional, technical and support personnel fo r performance of work required under this contract. The contractor shall develop and inst itute internal management controls that will ensure that, in the performance of this contract, the contractor will not provide personal services nor perform inherently governme ntal functions. For these pu rposes, personal services are defined in EPA Order 1900.1 dated October 31, 1985, and inherently governm ental functions are

defined in EPA Order 1900.2 dated October 22, 1990, and OFPP Policy Letter 92-1, dated December 1, 1992.

2.0 Peer Review Support

2.1 Selection of Peer Reviewers

The contractor shall provide Peer Review Serv ices. Peer reviewers will be selected for independence and scientific/technica l expertise. The scientific/tec hnical peer reviewers shall be recognized nationally or intern ationally in their field and have a general knowledge of environmental science issues as well as the sp ecific knowledge, expertise or experience as required by the Work Assignment. An independent peer reviewer is an expert who was not associated with the generation of the specifi c work product either directly by substantial contribution to its development or indirectly by consultation during th e development of the specific product (see Note below). Knowledge about the su bject area is not equivalent to expertise in the subject matter. For Agency decision, a multi-disciplinary group of experts corresponding to the disciplines th at contribute to complex Agency decisions is often necessary for a full and complete peer review. For exampl e, a risk assessment that relies on both animal and human data often requires experts in both areas for a complete review. The Project Officer (PO)/Work Assignment Manager (WAM) will provide the contractor with the document to be reviewed. The PO/WAM will, when available, provide the contractor with names of known experts (listed in alphab etical order) in the required disciplines for the contractor to make final

selection.

With submission of the work plan or within the timeframe iden tified in the Work Assignment, the contractor shall identify a gr oup of candidate peer reviewers, either from the pool provided by EPA, from contractor in-house staff experts, or from a list independently identified by the contractor (or a combination) and determine their availability for the task. The number of peer reviewers required for each review will be identif ied in the work assignment. The contractor shall submit its selection of p eer reviewers, along with thei r curriculum vitae to the EPAPeer Review Handbook Page F-14

PO/WAM. The Agency will review the propos ed candidates for consistency with the requirements of the work assignment , based either on qualifications, conflicts of interest, or past direct involvement with the work under review, however, EPA shall refrain from suggesting particular individuals to repl ace such candidates. Once th e reviewers are confirmed, the PO/WAM will provide the contractor with the final version of the charge and copies of relevant

reference materials.

The contractor shall not be involve d in the review of documents pr epared by the contractor, or its

subcontractor or consultants. It is the responsibility of the cont ractor to ensure that all peer

reviews are conducted in a manner to avoid all actual , potential or apparent conflicts of interest. The contractor shall obtain and provide written cert ification from all peer reviewers prior to the commencement of work that no personal or or ganizational conflicts of interest exist.

It is the Agency’s policy to have panels of experts that represent di verse constituencies, e.g. industry, academia, public interest groups, Federal (non-EPA) Agencies, State and Local governments, so long as the disciplinary expertise requirements of the pane l are also met. The contractor shall attempt in all cases to fulfill this criteria. Cultural diversity is also to be sought, however, disciplinary expertise is the number one criterion for selection.

Note : Peer reviewers of a product shall be independent experts which have not been substantially involved in the deve lopment of the product. In speci al cases where the number of experts in a disciplinary area is very small or not available at the speci fically designated time; the contractor may select a peer revi ewer who has been a peer consu ltant or prior reviewer of the product. If this selection is necessary, the c ontractor shall submit documentation of the prior involvement in the development process of the product by the potential re viewer, along with a control plan to avoid bias or lack of impartia lity, for EPA CO (Contrac ting Officer) and Project Officer review and consent. If special circumst ances exist where the expertise is so narrow that another peer reviewer isn’t available, the EPA CO/PO, along with the Peer Review Leader, will determine the appropriate involvement and cont rols for the potential reviewer, so that no perception of bias or conflict exists.

2.2 Information and Charge to Peer Reviewers

EPA will provide to the contractor written technical direction identifying the charge to be given to the peer reviewers. The peer reviewers shall consider EPA-supplied and EPA-specified documents along with EPA-provided instructions on the exact nature of the assignment (as outlined in the EPA Work Assignment). The ch arge to peer reviewers will usually make two general requests. First, it will focus the revi ew by presenting specific questions and concerns that the Agency expects the reviewers to addre ss. Secondly, it invites general comments on the entire work product. The contract or shall provide the charge to th e peer reviewers along with the product to be peer reviewed and all reference ma terials, and instruct the peer reviewers to undertake the review. Where appropriate, raw data and/or data in summary form will be provided and reviewers will be expected to rely on thei r assessment of such data in performing their review. Each peer reviewer must be informed of the need for confidentiality with regard to thePeer Review Handbook Page F-15

release of Agency products that are stamped as “DRAFT” or “DRAFT - Do Not Cite, Quote or Release.” The peer reviewer and the contractor will agree that the draft shall not be given to

others who are not under contract or direction without the writt en approval of the PO/WAM. Any requests for such drafts should be forwar ded to the EPA Contracting Officer for action.

2.3 Peer Review Report

The contractor shall monitor p eer reviewers’ progress to ensu re timely completion. All peer

review comments shall be submitted to the EPA PO/W AM for the particular work assignment. Peer reviewers will be required to submit with th eir evaluations or comments a description of the procedures used to arrive at their recommendations; a summary of their findings; a list of sources relied upon; and make clear and substantiate the methods and considerations upon which their

recommendations are based. When a document assi gned under a task order requires more than one peer reviewer, the contra ctor shall summarize all review comments and present this summary in an integrated manner that facilitate s an overall evaluation (e.g., an annotated copy) , and shall collate them into a p eer review report. The contract or shall be tasked with the compilation, categorization, and summarization of the comments. The peer review report shall be submitted to EPA in a specified format by a specified deadline. The format and deadline will be provided in the work assignment issued for th e work. The contractor shall submit to the EPA PO/WAM the peer review report and all materials submitted by the peer reviewers. EPA shall evaluate and analyze all peer review comment s and recommendations to ensure technical soundness and adherence to Agency policy. Agency officials should document their evaluations of the quality and validity of the peer review.

2.4 Peer Review Meetings

The contractor shall discuss peer review comments with the EPA PO, WAM, and/or other peer reviewers in telephone c onferences, peer review workshops, and/or other meetings. Telephone conferences, workshops and meetings shall be co nducted in accordance with work assignment specifications, which may require taping of and tran scription of any procee dings. The scientific expertise required shall include th e broad array of the scientific disciplines relevant to the specific document under development or review. In instances when a document or issue paper is reviewed, the contractor may be required to meet and document widely varying topics which will be defined specifically by the work assignment. The contractor shall organize and provide support in arranging and conducting p eer review meetings. The c ontractor shall obtain all postmeeting comments, collect and compile all co mments and suggested document revision, and return all required information/documentation to the PO/WAM. Occasionally, searches of recent literature and hard copy or microfiche retr ieval of germane papers may be required.Peer Review Handbook Page F-16

2.5 Peer Review Meeting Support

Peer Review meeting logistical support includes bringing together appropriate experts to address

various issues and concerns ra ised by EPA for selected products. The contractor may be required to (if specified in the work assignment):

1) prepare and distribute presentation material s, including pre-conference information , agenda, pre-registration forms, fact sheet s, overheads, slide presentations, etc.; 2) arrange for meeting space when government space is not available, reserve meeting rooms, copying and audiovisual equipmen t, furniture/room setups, microphones, and

related services to acco mmodate meeting needs; 3) provide for, prepare, and c onvey all materials such as name badges, table tents, visual aids, displays, etc., needed for pre-conference and ons ite activities; 4) inspect the conference site in accordance with expected facilities and services; 5) handle registration, distribution of mee ting materials, and attendance list; 6) provide on-site recorder and/or facilitator; 7) arrange reporting , tr anscription and/or note taking services, typing, reproduction, and related support servic es as required by meeting needs, including rapid photocopying for important materials to be copied at meetings; 8) operate audiovisual equipment, microphone s, other presenta tion related support equipment, and/or assist particip ants in operating such equipment; 9) handle close-down of the site, reviewi ng all bills for accuracy, and completing payments for equipment rental, etc. 10) A draft and final report may be prepare d, in accordance with the requirements as specified in the work assignment, and in accordance with the specifications of

Statement of Work Parts 2.3 and 2.4 herein. 11) If required, distribute draft proceeding summari es to the participants for comment and

make final revisions to the document.

The length of the meeting, expected attendance, seating arrangements and audiovisual equipment needs will be specifically identified in the work assignment issued by the CO.

The contractor shall arrange trav el only in accordance with the authority and limitations of the Section H Clause entitled “Approva l of Contractor Travel”, i.e., use of contract funds to reimburse travel is strictly limited to logistical support for peer reviewer s who directly contribute to the requirements of specific co ntract performance related to the statement of work of the contract which supports the mission of OST. In providing logistical s upport within the scope, the contractor may claim transportation, lodg ing and subsistence for the personnel of the contractor (including subcontract ors and consultants with a c ontractual agreement with the prime), who, by reason of their ro le within the scope of the Wo rk Assignment are required to attend the meeting. The contractor may be expect ed to reserve a block of lodging rooms for all reviewers and participants in a meeting. However, those participants who are Federal employees

must then make their own room reservati ons, travel arrangement s, and travel funding

arrangements.