United States Office of Science Policy Environmental Protection Office of Research and Development December 2000 Agency Washington, DC 20460 www.epa.gov EP A

100-B-00-001 

# Science Policy Council 

# HANDBOOK

2nd EditionEPA 100-B-00-001 December 2000 

# U.S. Environmental Protection Agency PEER REVIEW HANDBOOK 2nd Edition
Prepared for the U.S. Environmental P rotection Agency by members of the Peer Review Advisory Group, a group of EPA’s Science Policy Council Principal Authors Kerry L. Dearfield, Ph.D. A. Robert Flaak Office of Science Policy EPA Science Advisory Board Office of Resea rch and Developm ent Office of the A dministrator

Major Contributors
Jean C. Schum ann Office of Solid W aste and Emergenc y Response Roland B. Hemmett, Ph.D. Region 2 Brett Snyder Nancy W. Wentworth Office of Environmental Information Arnold M. Kuzmack, Ph.D. Office of Water

Office of Policy, Economics, and Innovation 

## Science Policy Council U.S. Environmental Protection Agency Washington, DC 20460Page ii Peer Review Handb ook



### DISCLAIMER
This document has been reviewed in accordance with U.S. Environmental Protection Agency policy a nd approved for publication and distribution to the Agency. Mention of trade names or commercial products does not constitute endorsement or recommendation for use.Peer Review Handb ook Page iii 

### TABLE OF CONTENTS

FOREWORD

.......................................................... Page xiii
ACKNOWLEDGMENTS

................................................ Page xvii
SUMMARY OF THE PEER REVIEW PROCESS

.............................. Page 1 Figure 1 - Flowchart for Planning a Peer Review

.......................... Page 2 Figure 2 - Flowchart for Conducting a Peer Review

........................ Page 3 Figure 3 - Flowchart for Completing a Peer Review

........................ Page 4 Manager’s Planning Checklist for Peer Review

........................... Page 5
PEER REVIEW GUI DANCE

............................................... Page 7
1. THE NEED FOR PEER REVIEW

......................................... Page 9 1.1 Overview Statement

........................................... Page 9 1.2 Understanding Peer Review

..................................... Page 9 1.2.1 Why use Peer Review?

.................................. Page 9 1.2.2 What is Peer Inv olvement?

.............................. Page 10 1.2.3 What is Peer Review?

.................................. Page 10 1.2.4 What is Peer Inp ut?

.................................... Page 10 1.2.5 How is Peer Review Different from Peer Input?

.............. Page 11 1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer Reviewer for the Same Work Product La ter in the Process?

........................................ Page 11 1.2.7 Can the Same Peer Reviewer be Used More than Once if a Product will Be Peer Reviewed More than Once, and can the Same Peer Review er be Used Aga in and Again for Different Products?

............................................ Page 12 1.2.8 How is Peer Review Different from Public Comment?

......... Page 12 1.2.9 How is Peer Review Different from Stakeholder I nvolvement?

.. Page 12 1.2.10 Wha t Ro le d oes Pee r Re vie w have i n th e Re gula tory Development Proc ess?

.................................. Page 13 1.2.11 What Role does Peer Review have in Reg ulatory Ne gotiations?

. Page 14 1.3 Annual Agency Reporting Requir ements

......................... Page 14 1.3.1 What are the Annua l Reporting Requir ements?

.............. Page 14 1.3.2 What Listin gs are Requir ed for the An nual Repor ting?

........ Page 14 1.3.3 When will the Handbook Itself be Revised?

................. Page 17Page iv Peer Review Handb ook

1.4 The Roles of People and Organizations in Peer Review

.............. Page 17 1.4.1 Who is Ultimately Accountable for Peer Review?

............ Page 17 1.4.2 Who are the Agency Staff involved in Peer Review?

.......... Page 17 1.4.3 Who are the Decision Makers & What are Their Responsibilities?

...................................... Page 17 1.4.4 Who are the Peer Review Leaders & What are Their Responsibilities?

...................................... Page 19 1.4.5 Who are the Peer Re view Coordinators & What are Their Responsibilities?

...................................... Page 20 1.4.6 Who are the Peer Re viewers?

............................ Page 21 1.4.7 What are the Respo nsibilities of Peer Reviewers?

............ Page 21 1.4.8 What is an Independent Peer Reviewer?

.................... Page 21 1.4.9 When does an Age ncy Int ernal Peer Revie wer Qualify as Independen t?

......................................... Page 22 1.4.10 What is a Peer Revie w Panel?

............................ Page 22 1.4.11 What is a Subject Matt er Expert?

......................... Page 22 1.4.12 What i s the R ole of the Sc ience Poli cy C ounc il (SPC)?

........ Page 23 1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)?

. Page 23 1.4.14 What is the Role of the Office of Research and Development (ORD)?

.................................. Page 23
2. PLANNING A PEER REVIEW

.......................................... Page 25 2.1 Overview Statement

.......................................... Page 25 2.2 Determining Which Work Products to Peer Review

................. Page 25 2.2.1 What are Scientifi c and Technical Work Products?

........... Page 25 2.2.2 What Scientific and Technical Work Products Need Peer Review?

............................................. Page 26 2.2.3 How Does One Determine Whether a Scientific and/or Technical Work Product is “Major”?

...................... Page 26 2.2.4 What Economic Work Products Need Peer Review?

.......... Page 27 2.2.5 Should Economic Work Products Prepared in Support of Regu lat ion s th at are C las sif iedas "M ajo r" or "Eco nom ica lly Significant" be Peer Reviewed?

........................... Page 28 2.2.6 What Other Economi c Work Products Mig ht Benefit from Peer Review?

......................................... Page 29 2.2.7 What Other Social Science Work Products Need Peer Review?

.. Page 30 2.2.8 How Should Peer Review be Handled for Products Developed under an Interagency Agreement (IAG)?

.................... Page 31 2.2.9 Should Products from Contracts, Grants, and Cooperative Agreements Receive Peer Review?

........................ Page 31Peer Review Handb ook Page v

2.2.10 How Does Peer Review Apply to Products Generated Through EPA Grants or Coope rative Agree ments?

................... Page 31 2.2.11 Can the Recipient of a Grant or Cooperative Agreement Use Agreement Fu nds to Pay Pee r Reviewers of th eir Work Products? Page 34 2.2.12 Do Products Generated under EPA Grants or Cooperative Agreements Ne ed to be Reported i n the Peer Review Product Tracking (PRPT) Database?

............................. Page 34 2.2.13 Should Site Specific Decisions be Subject to Peer Review?

..... Page 35 2.2.14 Should NEPA Products (e.g., EISs) be Subject to Peer Review?

. Page 35 2.2.15 Should Environmental Regulatory Models be Peer Reviewed?

.. Page 36 2.2.16 Is Peer Revie w Needed for Oth er Organization’s Work Products that Have be en Submit ted to EPA for Use in Dec ision Maki ng?

. Page 36 2.2.17 Can Work Products Th at are Not Deter mined to be Major Stil l be Peer Reviewed?

..................................... Page 37 2.3 Determining Which Work Products Do Not Receive Peer Review

...... Page 37 2.3.1 Are There Circu mstances When a Ma jor Work Product is No t Peer Reviewed?

....................................... Page 37 2.3.2 What Products Normally Do Not Need Peer Review?

......... Page 38 2.3.3 Do Voluntary Consensus Standards Require Peer Review?

..... Page 39 2.4 Choosing a Peer Re view Mechanism

............................. Page 39 2.4.1 How Do You Determine the Appropriate Peer Review Mechanism?

.......................................... Page 39 2.4.2 What are Examples of Internal Peer Review?

................ Page 41 2.4.3 What are Examples of External Peer Review?

............... Page 41 2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal?

............................................. Page 42 2.4.5 Do Agency Work Products Become Candidates for Peer Review when Peer Reviewed Journal Articles are Used in Support of that Work Pr oduct?

........................... Page 43 2.4.6 When and How Often Should Peer Review Occur?

........... Page 43 2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review?

......................................... Page 44 2.4.8 Which Office/Region or Other Agency is Responsible for Conducting the Peer Review?

............................ Page 45 2.5 Creating the Peer Review Record

............................... Page 45 2.5.1 What is the Peer Review Record?

......................... Page 45 2.5.2 How Can the Peer Review Record Improve the Peer Review Process?

...................................... Page 45 2.5.3 What Should Be in the Peer Review Record?

................ Page 45 2.5.4 What Should I Do with a Peer Review Record That Pertains to a Rulemaking Action?

................................... Page 46Page vi Peer Review Handb ook
2.5.5 When Should the Peer Review Record Building Process Begin?

. Page 46 2.5.6 What are the Differences in Record Keeping for a Review by an Individual Comp ared to a Panel?

.......................... Page 47 2.5.7 Where Should the Peer Review Record be Kept and For How Long?

........................................... Page 47 2.5.8 Are Internal Peer Review Comments Included in the Peer Review Record?

....................................... Page 47 2.6 Budget Planning

............................................. Page 48 2.6.1 What Budgetary Factors Should I Consider in a Peer Review?

... Page 48 2.6.2 What Input is Needed for the Annual Budget Formulation and Budget Executio n Process?

.............................. Page 48 2.7 Legal Considerations

......................................... Page 49 2.7.1 Are T here Legal Ram ifi cat ion s Fro m th e Pe er R evi ew P oli cy?

.. Page 49 2.7.2 Is Legal Advice Needed?

................................ Page 49 2.8 Federal Advisory Committee Act (FACA) Considerations

............ Page 49 2.8.1 When Do FACA Requirements Apply to EPA-Run Peer Reviews?

........................................ Page 50 2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA?

..... Page 50 2.8.3 How Do I Ensur e that a Contracto r-Run Peer Review Does Not Become Subject to FACA?

........................... Page 51
3. CONDUCTING A PEER REVIEW

....................................... Page 53 3.1 Overview Statement

.......................................... Page 53 3.2 Charge to the Peer Reviewers

.................................. Page 53 3.2.1 What is a Charge?

..................................... Page 53 3.2.2 What are the Essential Elements of a Charge?

............... Page 53 3.2.3 Where Can I Get an Example of a Charge?

.................. Page 54 3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers?

........................................... Page 54 3.2.5 Who Writes the Charg e When I Hire a Contractor to Condu ct the Peer Review?

...................................... Page 54 3.2.6 Is it Okay For Me to Ask a Contractor to Deve lop the Charge to the Peer Review ers?

.................................. Page 55 3.3 Time Line

.................................................. Page 55 3.3.1 What are the Factors in Scheduling a Peer Review?

........... Page 55 3.4 Selection of Peer Reviewers

................................... Page 55 3.4.1 What are Considera tions for Selecting Peer Reviewers?

........ Page 55 3.4.2 Where Do I Fi nd Peer Reviewer s?

......................... Page 56 3.4.3 Are External or Internal Peer Reviewers Preferred?

........... Page 57 3.4.4 What is Importa nt in the Mix of a Peer Review Panel?

......... Page 57 3.4.5 What is a Conflict of I nterest?

............................ Page 58Peer Review Handb ook Page vii
3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of Conflic ts of Interes t?

........................ Page 59 3.4.7 Can Parties Externa l to EPA Pay for Their Own Peer Re views?

. Page 61 3.4.8 Are There Constr aints to Selecting Peer Reviewers?

.......... Page 61 3.4.9 If State Employees are Used as Peer Reviewers, Can EPA Pay Them for this Service?

.............................. Page 62 3.5 Materials for Peer Reviewers

................................... Page 62 3.5.1 What Instructi ons Do You Give Pee r Reviewers?

............. Page 62 3.5.2 What Materials Shou ld be Sent to Peer Re viewers?

........... Page 63 3.5.3 How Closel y can EPA Interact w ith Peer Re viewers Dur ing the Review?

.......................................... Page 64 3.6 Peer Review Services

......................................... Page 65 3.6.1 What are Gratuitous Services for Peer Review?

.............. Page 65 3.6.2 Can I Use a Con tract to Obtain Pee r Review Service s?

........ Page 66 3.6.3 How Do I Write a Statement of Work fo r Contracts?

.......... Page 66 3.6.4 What are Advisory and Assistance Services (AAS) or Sensitive Activities?

........................................... Page 67 3.6.5 What are Some Mana gement Controls f or Contracts?

......... Page 67 3.6.6 Can I Identify and/o r Select Pee r Reviewers When Using a Contract?

........................................... Page 70 3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers?

........................................... Page 70 3.6.8 How is Travel Han dled with Contract s or Purchase Ord ers?

.... Page 72 3.6.9 How is Travel Han dled with Special G overnment Employ ees?

.. Page 72
4. COMPLETING A PEER REVIEW

....................................... Page 73 4.1 Overview

.................................................. Page 73 4.2 Final Work Product

.......................................... Page 73 4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product ?

................................... Page 73 4.2.2 What Actions are Potentially Forthcoming from Peer Review?

.. Page 74 4.2.3 What Should the Final Work Product Say About the Peer Review Process?

...................................... Page 74 4.2.4 Can the Identi ty of Peer Re viewers be Kept A nonymous?

...... Page 75 4.3 Completing the Peer Review Record

............................. Page 75 4.3.1 How Do I Complete the Peer Review Record?

............... Page 75 4.3.2 Where Should the Peer Review Records be Kept, and for How Long?

........................................... Page 76 4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA?

......................................... Page 77Page viii Peer Review Handb ook
SUBJECT INDEX

....................................................... Page 79 COMMONLY USED ACRONYMS

......................................... Page 81 APPENDIX A - U.S. ENVIRONME NTAL PROTECTI ON AGENCY PEER REVIEW POLI CY

................................................ Page A-1 APPENDIX B - SOUND SCIENCE AND PEER REVIEW I N RULEMAKING

..... Page B-1 APPENDIX C - EXAMPLES OF CHARGES

................................ Page C-1 CHARGE EXAMPLE 1 - Framework f or Developing a Living Resour ces Research and Monitoring Plan for the Peconic Estuary

.............. Page C-2 CHARGE EXAMPLE 2 -EPA Science Advisory Board (SAB) Review of the Environment Monitoring and Assessment Program (EMAP) Research Strategy and Research Plan

........................... Page C-5 CHARGE EXAMPLE 3 - EPA Science Advisory Board Revie w of the Agency's National Risk Mana gement Resear ch Laborato ry’s (NRMRL) Program

......................................... Page C-6 CHARGE EXAMPLE 4 - EPA Science Advisory Board (SAB) Review of the Technical Aspects of the Multi-Agency Radiation Survey and Site Investigat ion Manual (MARSSI M)

............................. Page C-7 CHARGE EXAMPLE 5 - Economics - Be nefits Transfer from Adults to Children

................................................ Page C-8 CHARGE EXAMPLE 6 - Economics - Stud y of Municipa l Government Costs and Financial Impacts from Environmental Regulations

........ Page C-9 CHARGE EXAMPLE 7 - Economics - Valuation of Fatal Cancer Risks

..... Page C-11 CHARGE EXAMPLE 8 - Hudson R iver PCBs Site Reasses sment RI/FS Modeling Approach Charge: Peer Review 1

..................... Page C-15 CHARGE EXAMPLE 9 - IRIS Pilot Program - I nstructions to Peer Reviewers for Reviewi ng IRIS Summari es and Supp orting Documentation

............................................ Page C-18 CHARGE EXAMPLE 10 - Charge to Re viewers for the WT I Draft Fina l Risk Assessment

.......................................... Page C-21 APPENDIX D - GUIDANCE ON REQUESTI NG A REVIEW BY THE US EPA SCIENCE ADVISORY BOARD (SAB)

.......................... Page D-1 APPENDIX E - EXAMPLE STATEMENTS OF WORK FOR CONTRACTS

...... Page E-1 STATEMENT OF WORK - EXAMPLE 1 - Stat ement of Work: Tec hnical Review Contractor for Panel Review of Assistance Agreement or Fellowship Applications

..................................... Page E-2Peer Review Handb ook Page ix
STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization Tool Report

............................................... Page E-5 STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of Protozoa Method Development Criteria Document

............... Page E-12 APPENDIX F - U SEFUL FORMS

......................................... Page F-1 PEER REVIEW CONF LICT OF I NTEREST INQUI RY

.................. Page F-2 PEER REVIEW CHECKLIST FOR DETERMIN ING WHETHER A WORK PRODUCT NEEDS PEER REVIEW

........ Page F-3 PEER REVIEW CHECKLIST FOR CONDUCTI NG A PEER REVIEW

..... Page F-7 VOLUNTEER SERVICE PROGRAM PARTICIPATI ON AGREEMENT

... Page F-14 APPENDIX G

......................................................... Page G-1 REFERENCES CONCERNING PEER REVIEW

....................... Page G-1Peer Review Handb ook Page xi 

### FOREWORD (2 nd

### Edition)

It has been three years since EPA's Science Policy Council (SPC) issued the 1st Edition of the Peer Review Handbook (January 1998). The origina l Handbook has been used exte nsively by personne l throughout the A gency fo r the peer revie w of Agency work products. At the time it was issued, we planned a two to three-year period during which Agency staff could use the Handbook and provide feedback on its utility, and at the same time, identify questions or issues not originally addressed. To date, we have received numerous comments and suggestions from reg ional and headquarters personnel concerning the Handbook

. These comments h ave been unifor mly positive and encouraging. The comments indicate that the Handbook contributed greatly to the Agency g oal of sound scien ce and substantia lly improving EPA's peer rev iew process.
We have coll ected the co mments and additiona l question s, discuss ed them wit h the SPC's Peer Review Advisory Group and Agency Peer Review Coordinators, and developed revised and new answers to the questions receiv ed. In additio n, we also incorpo rated many revisions in response to recommendations from the EPA Science Advisory Board (SAB; from "An SAB Report: Review of the Peer Review Program of the Environmental Protection Agency " a review by the Resea rch Strategies Advisory Comm ittee (RSAC) of the SAB; report #EPA-SAB-RSAC00-002 dat ed November 1999), the EPA Office o f the Inspecto r General (OIG; from "EPA's Selection of Peer Reviewers" ; report #1999-P-217 dated September 29, 1999), and the National Research Council (NRC; from "Strengthening Science at the U.S. Environmental Protection Agency: Research Manage ment and Peer Review Practices"; report issued 2000). After revising the Handbook , the revisions were circulated extensively throughout the Agenc y for final comment before the SPC approved the 2nd Edition of the Peer Review Handbook at their October 17, 2000 meeti ng. While we strived to make the Peer Review Handbook a definitive source of information about peer review and its processes, we need to sound one precautionary note. During the revision process, we heard many requests to put as much detail as possible into the Handbook to make it a one stop source for all things related to peer review. We have attempted to do this as much as possible; however, much of the detail in several instances delves into areas for which there is specialized expertise elsewhere in the Agency, par ticularly contracting and legal issues. If we provide all the detail requested in many of these areas, the Handbook would turn into a contract manual or legal book, something we did not want to do. Since many of these processes (like contracting) are updated frequently, w e did not want to outdate the Handbook unnecessarily. Also, we felt if the Handbook became much bigger than it is now, its utility andPage xii Peer Review Handb ook
ease of use would decrease. We have provided pointers and contacts where nec essary in the Handbook to help Agency people deal with these specific issues in relation to peer review. We are proud to bring the 2nd Edition to you for your use in peer review at EPA. The Handbook represents a la rge Agency collective effo rt to improve its scie nce to the highe st levels possible. Peer review done correctly significantly adds to our goal of sound and credible science that underlay s all Agency decisions and ac tions. Kerry Dearfield and Robert Flaak Co-Chairs, Peer Review Advisory Group (PRAG) Science Policy Council
Science Policy Council Norine Noonan, O RD, Vice-Chair Robert Brenner, OAR Steven Galson, OPPTS Albert McGartland, OPEI Michael Ryan, OCFO Elaine Stanley, OEI W. Michael McCabe, OA, Chair Donald Barnes, OA Tudor Davies, OW Sylvia Lo wrance, OECA William Muszynski, Region 2 Michael Shapiro, OSWER Ramona Trovato, OCHP Science Policy Council Steering Committee Donald Barnes, OA Reginald Cheatham, OEI William Farland, O RD Penelope Fenner-Crisp, OPPTS Jerri-Anne Garl, Region 5 Roland Hemmett, Region 2 Carl Mazza, OAR Jennifer Orme-Z avaleta, ORD Larry Reed, OSWER Rosemarie Russo, Region 4 Mary Ellen Weber, OPPTS William Wood, ORD Michael Brody, OCFO Patricia Cirone, Region 10 Michael Feldman, OCFO Michael Firestone, OCHP Peter Grevatt, OSWER Kate Mahaffey, OPPTS James Nelson, OGC Peter Preuss, ORD Joseph Reinert, OPEI Vanessa Vu, ORD Jeanette Wiltse, OW

Science P olicy Coun cil Staff Edward Bender Kerry Dea rfield James RowePeer Review Handbook Page xiii 

### FOREWORD (From 1 st

### Edition) EPA’s Science Policy Council (SPC) has organized this Peer Review Handbook as guidance to EPA s taff and manag ers on the orga nization and conduc t of peer review pursuant to the Administrator’s June 7, 1994 Peer Review Policy statement. The Handbook is based in part on the central themes set forth in the Policy statement (see Appendix A for the full policy ): Major scientifically and technically based work products related to Agency decisions normally should be p eer reviewed. Ag ency managers wit hin Headquarters, R egions, laboratories, and field components determine and are accountable for the decision whether to employ peer review in particular instances and, if so, its character, scope, and timing. These deci sions are made in co nformance with prog ram goals and prioritie s, resource constra ints, and statutory or court-ordered de adlines. For those wo rk products that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact, peer review at the planning stage can o ften be extreme ly beneficial. In ad dit ion , th e Han dbo ok augme nts the se them es b y sta tin g and exp lai nin g wid elyaccepted princ iples and practic es that have long guided peer re view in the univer sities, in private research org anizations, and at t he EPA and other g overnment age ncies. The goal of the Pe er Review Policy and this Handbook is to enhance the quality and credibility of Agency decisions by e nsuring that the scientific and technical work products underlying these decisions receive appropriate levels of peer review by independent scientific and technical experts. To serve this goal, the Handbook provides information and outlines procedures in several diffe rent areas: The Policy requires peer review of the basis of the decision (i.e., the underlying major scie ntific and/or technical work products), not the decision itself.

P basic principles and definitions, including distinctions between peer review and peer input, public comment, and stake holder involvemen t;

P preparing for peer review, including identify ing work products, identifying appropriate peer review mechanisms, and identifying qua lified experts; and,

P conducting and completing peer reviews, including materials required for pe er review, creat ing a peer rev iew record, and utilizing peer re view comments.Page xiv Peer Review Handb ook
This Handbook has t hree parts. The first contains flo w charts that outli ne the key s teps in conducting a peer review, along with a managers’ che cklist for planning peer reviews. The second part cont ains peer revie w guidance det ailing the proce dures outlined in th e flow charts in a question and answer format. The third part contains Appendices including the 1994 Peer Review Policy and examples to help perform quality peer reviews. So me procedures o utlined in the 1994 Policy have been completed while others are continuing to chang e in line with Agency experience. These changes have been incorporated into the current Ha ndbook.
Dorothy E. Patton, Ph.D. Executive Directo r, Science Policy CouncilPeer Review Handb ook Page xv 

### ACKNOWLEDGMENTS

Compiling a comprehensive handbook that incorporates the most relevant, as well as some of the obscure policies and guidance, regarding peer review and its application throughout EPA has been a herculean task involving not just the members of the Peer Review Advisory Group, but many others as well. We want to acknowledge the contributions of the headquarters and regional Peer Review Coordinators and their respective office and division coordinators, the Science Policy Council (SPC) and its St eering Committee , the Office of E nvironmental I nformation’s (OE I’s) Quality Assurance Staff, Office of Science Policy (OSP) Program Support Staff, and the many other individuals who provided comments on the revisions to the current edition. In particular, we want to acknowledge the following individuals who provided c ritical expertise and timely revisions (often several times) to various sections of the Peer Review Handbook: Hale Hawbecker (Office of General Counsel), Theresa Tr ainor (Office of Water), and Bruce Bakaysa (Office of Acquisition Management). Finally, we want to acknowle dge the tireless efforts and men torship of the rec ently retired Executive Director of the Science Policy Council, Dr. Dorothy Patton. She is one of the pillars of the Agency’s efforts to improve the quality of science and peer review and we are indebted to her.Peer Review Handb ook Page 1 

# U.S. Environmental Protection Agency
SUMMARY OF THE PEER REVIEW PROCESS

This Section of the Peer Review H andbook contains flowc harts and descriptions of the m ajor steps in conduc ting a peer review

. Cross refere nces to the appropriate sec tion on Peer Review Guidance are show n in parenthesis and bolded. The Managers Planning Checklist for Peer Review(on page 5) is designed to give Managers/Decision Makers a simple tool to help plan for a successful peer review. It asks questions that should be c onsidered by a m anager or Decisi on Maker during the pee r review process to insure that necessary actions ar e taking place. This checklist is also intended to be used by staff (especially Peer Review Leaders and Coord inators) to inform managers and Decision Makers on som e of the key steps and c onsiderations that ar e necessary in carrying out a s uccessful peer review. E xpanded Check Lists w hich can be adapted f or your use are also included in Appendix F - Useful Forms (courtesy of Region V).Page 2 Peer Review Handb ook 

### Figure 1 - Flowchart for Planning a Peer Review

1. Determination of “major scientifi c and technical wor k product”: -- It is a sci entific, engineeri ng, economic, or statistical document (§ 2.2.1) -- Determine if the work pro duct is major (§ 2.2.3, 2.2.4, 2.2.7) -- Major p roducts m eet certain criter ia (§ 2.2.3)
2. Work product is a candidate f or peer review : -- Major wor k products are subject to peer review (§ 2.2.2) -- Some non-major work products have to be evaluate d to determine if peer review is still warranted (§ 2.2.17)
3. Work p roduct is not a candidate for peer review : -- Most non-major work products are typicall y not candidates for peer review (§ 2.3.2) -- Major work p roduct consists onlyof science previously peer r eviewed an d adequate under the Agency’s Policy (§ 2.3.1) -- Place wor k produc t on List C (§ 1.3.2c))
4. If a work pro duct is subje ct to peer re view: -- Identi fy basis f or charge (§ 3.2.1) -- Identify key staff (§ 1.4) -- Creat e a peer rev iew recor d (§ 2.5) -- Ensure source of funding for the peer review (§ 2.6) -- Determine overall time frame for

peer review (§ 3.3.1) -- Place the work product on List B (§ 1.3.2b))Peer Review Handb ook Page 3 

### Figure 2 - Flowchart for Conducting a Peer Review

1. Develop the charge (§ 3.2) : -- Determine which key issues to add ress -- Include in peer revie w record (§ 2.5.3)
2. Select a peer review me chanism (§ 2.4) -- Inter nal(§ 2.4.2) -- External (§ 2.4.3) -- Mail (e.g., Letter ) review (§ 2.4.3) -- Face to face meeting (§ 2.4.3) -- One time or multiple meetings (§ 2.4.6) -- Include logisti cal information in peer review record (§ 2.5.3)
3. Determine the specific time li ne (§ 3.3) : -- When will t he review be st arted -- What are the intermed iate check p oints -- What is the deadl ine for completion
4. Select peer reviewers(§ 3.4) : -- Determine sources of peer reviewers(§ 3.4.2) -- Determine ex pertise requi red(§ 3.4.4) -- Consider balance/add ress (§ 3.4.4) -- Consider c onflicts of interest (§ 3.4.5 & 3.4.6) -- Include documentation in peer review record (§ 2.5.3)
5. Materials for t he peer review (§ 3.5) : -- Obtain mate rials f rom Program for review -- Prepare ins tructions for peer reviews (§ 3.5.1) -- Forward materials to peer reviewers (§ 3.5.2) -- Include copy of materials i n peer review record (§ 2.5.3)
6. Conduct the peer revi ew -- Obtain written comments from reviewers -- Include in peer revie w record (§ 2.5.3)
Note: Some of these steps may occur concurrently.Page 4 Peer Review Handb ook 

### Figure 3 - Flowchart for Completing a Peer Review

1. Evaluate comments from peer reviewers (§ 4.2.1) -- Respond and react to comme nts -- Obtain cl arificat ion, if n eeded
2. Brief your Decision Maker; obtain written management approval of response t o comments (§ 4.2.1)
3. Comments th at are con sidered, but not u sed (§ 4.3.1) -- Determine why not used and document -- Include comment s in peer review r ecord (§ 2.5.3)
4. Comments that are useful(§ 4.3.1) -- Revise the wo rk produ ct by incorpora ting comm ents -- Send r evised wor k product bac k to peer reviewers, if necessary -- Include comment s in peer review r ecord (§ 2.5.3)
5. Finalize wo rk produ ct (§ 4.3.1) -- Include in peer revie w record (§ 2.5.3) -- Move w ork prod uct from List B to List APeer Review Handb ook Page 5 

### Manager’s Planning Checklist for Peer Review 1) Title of Work Pro duct : _________________________ __________________________ 2) What D ecision/Rule/Regulation/A ction Does this Wo rk Product Su pport : ________

__________________________________ __________________________________ __________ 3) Determination of Major Scientific and Technical Work Products G Is the work product scientif ic or tec hnical __y es __no? G Is the work product __major or __non-major? 4) Determining W hat Peer Review is N eeded G If major , peer revi ew is needed? G If not major, is peer revie w still n eeded? G What peer review mechanism is needed (i nternal and/or exter nal)? G When does the review need to be don e? G How much time will be needed to conduc t/comple te t he re view? G Are there court ordered deadlines or other constraints? G Has senior management (AA /RA/others) been informed of progress/problems? G What wo uld const itu te s uccess f or th is r evie w? 5) Determining th e Resources for Peer Rev iew G What is the priority of this project relati ve to other projects in the same office? G What r esou rces are need ed to condu ct t he re view? G What are the im pacts of the review o n personnel? G Who wil l lead th e pee r rev iew? G Who wil l con duct the peer revi ew? G Who will maintai n the peer r eview record? G Where w ill the peer review reco rd be kept? G What mechan ism wi ll be used for the peer revi ew? G Has the char ge been developed? G Has intern al and exte rnal coordi nation bee n initiated/completed? G Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes to the final work product) been made? G How will resul ts of the review be presented and a ddressed i n the fin al work product (e.g., i n a preamble, in an a ccompanying appendix -- as well as changes in the work product itself)? G Has the work product been entered onto List B or C, as approp riate? 6) Comm ents:

__________________________________ ___________________________ See Appendix F (Use ful Forms) for expande d checklistsPeer Review Handb ook Page 7


# U.S. Environmental Protection Agency
PEER REVIEW GUI DANCEPeer Review Handbook Page 9 

### 1. THE NEED FOR PEER REVIEW

1.1 Overview Statem ent Peer review at the U.S. Environmental Protection Agency (EPA) take s many different forms depending on the nature of the work product, relevant statutoryrequirements, and officespecific policies and practices. In January 1993, r esponding to recommendations in the report Safeguarding the Future: Credible Science, Credible Decisions, Administrator Willia m Reilly issued an Agency-wide policy for peer review. Administrator Carol Browner reaffirmed the central role of peer review in the Agency on June 7, 1994 and instituted an Agency-wide imp lem ent ati on p rogr am ( see A ppe ndi x A - USE PA P eer R evi ew P oli cy). F oll owi ng Ag encywide implementation, office and region-specific standard operating procedur es (SOPs) were written and used f rom 1994 to 1998. I n 1998, this Peer Rev iew Handbook was created as a single, centralized form of implementation guidance for Agency staff and managers. The Peer Review Handbook was revised throughout calendar ye ar 2000 and was reissued as Edition 2 (the current version) in December 2000. Agency offices and regions, however, can still prepare brief, tailored guidan ce that meets thei r individual needs to supplement the in formation in this Handbook.
1.2 Understanding Peer Review
1.2.1 Why use Pe er Revie w? Peer review is intended to uncover any technical problems or unre solved issues in a preliminary (or draft) work product through the use of indepe ndent experts. This i nformation is then used to revise that draft product so that the final work product will reflect sound technical information and analyses. Peer review is a process for enhancing a scientific or technical Peer review is not free; work product so that the decision or position taken by the however, not doing peer Agency, based on that product, has a sound, credible basis. To review can be costly. be most effective, peer review of a major scientific and/ortechnical work product needs to be incorporated into the upfront planning of any action based on the work product - this includes obtaining the proper resource commitm ents (people an d money) and es tablishing realistic schedules. Peer review of major scientific and technical work products should not be looked upon as another “hurdle ” in the Agency decision making p rocesses. While pe er review requ ires that time and resources be planned into the decision making process, the benefits justify the added cost. Peer review enhances the credibility and acce ptance of the decision based on the work product.Page 10 Peer Review Handb ook By ensuring a sound basis for decisions, greate r cost savings are realized since decisions will not be challenged as often and extra effort will not be required to go back and re do the work product. So while peer review is not free, the cost of not doing peer review is usually much more expensive. Furthermore, not conducting a peer review can potentially place the Agency in the position of attempti ng to defend a sc ientifically invalid position -- w hich can be very costly in terms o f bo th r eso urce s, and m ore i mpo rtantl y, cre dib ili ty.
1.2.2 What is Peer Involvement? As defined in the Peer Review Policy, peer involvement is the process whereby Agency staff involve subject-matter experts from outside their program in one or more aspects of the development of work products. Peer involvement, therefore, constitutes active outreach to and participation by the broad scientific, engineering, a nd economics and social science communities beyond the Agency (external) as well as within the Agency (internal). Ty pically, peer involvement takes two general fo rms: peer input(ongoing discussions during the development of the work product) and peer review (an evaluation of a workplan, preliminary draft or the like, or most often, the critical, final objective expert evaluation of the work product).
1.2.3 What is Pe er Revie w? Peer review is a documented critical review of a specific Agency major scientific and/or technical work p roduct. The peer review is conduc ted by qualif ied individuals (or organizations) who are indepen dent of those who pe rformed the work , but who are colle ctively equ ivalent in technical expert ise (i.e., peers ) to those who perf ormed the origin al work. The pee r review is conducted to ensu re that activitie s are technical ly adequate , competently performed, prop erly documented, and s atisfy esta blished quality requirements. Th e peer review i s an in-depth assessment of the assumptions, calc ulations, extrapola tions, alternate i nterpretations, methodology , acceptance c riteria, and con clusions pertainin g to the specific major scientific and/or technical work product and of the documentation that supports them. Peer review may provide an evalu ation of a subject where quantitati ve methods of ana lysis or meas ures of success are unavailable or undefined; suc h as research a nd development. Pe er review is usua lly characterized by a one-time interaction or a limited number of interac tions by independent peer reviewers. Peer review can occur during the early stages of the project or methods selection, or as typically used, as part of th e culmination of the work product, en suring that the f inal product is technically sound.
1.2.4 What is Peer Input? Many Agency w ork products are developed with the input of various scientific and technical expert s inside and outside the Agency

. Like the con tribution made by peer reviewer s, peer input is valua ble and enhance s the scientific or technical basis o f the products. Pe er input,Peer Review Handbook Page 11 sometimes referred to as peer consultation, generally connotes an interaction during the development of an evolving A gency wo rk product, provi ding an open exch ange of data, i nsights, and ideas. Peer i nput may be c haracterized by a continued and ite rative interact ion with scientific experts during work product development. A common example of peer input is the input received from workgroup members during development of a product. Many Agency products are de veloped through the efforts of a workgroup, whic h may include external experts, such as State and Tribal representatives. These workgroup members have an ac tive, ongoing participation in developing the work product. Another example of obtaining peer input is of an Agency office sending a dra ft work product to a list of stakeholder representatives for general comments (stakeholder representatives often include experts who could be considered “peers”).
1.2.5 How is Peer Review Different from Peer Input? The key distinctions between peer input as described above and forma l peer review are the independence of the peer reviewers and their level of involvement. The goa l of peer review is to obtain an independent, third-party review of the product from experts who haven’t substantially contributed to its de velopment. When expe rts have a mater ial stake in the outc ome of the peer review (such as a regulated party ) or have participated substantially in the development of the product (such as a workgroup member), those experts’ reviews may not qualify as u nbiased, indepen dent peer revie w and may be better charac terized as peer in put. It is clear th at peer input prov ides valuable contributions to the development of the work product. Peer Input is not a However, peer input does not substitute for peer review. Once substitute for Peer Review a work product is considered major, it is a candidate for peer review and entered on List B ( Candidate Products f or Future Peer Review -- see Section 1.3.2b)) -- even though the work product may already have a substantial amount of peer input. In other words, one cannot argue that a peer review is not necessary if a major work product has receive d “enough” peer input. If the work product is not considered major and has had peer input, it is entered on List C ( Products for Which a Decision has been made not t o Peer Review -- see Section 1.3.2c)) with comments about the extent of the peer input.
1.2.6 Can Someone Who Provided Peer Input Become an Independent Peer Reviewer for the Sa me Work Produc t Later in the Pr ocess? Generally, the answer is no as that expert is no longer independent, but rather a contributor to the work product. The re may be sp ecial circumsta nces where the e xpertise is so narrow that another peer reviewer isn’t available. The Peer Review L eader (see Section 1.4.4) will normally be responsible for making this determination and documenting the decision in the peer review record.Page 12 Peer Review Handbook
1.2.7 Can the Same Pe er Reviewer be Us ed More than Once if a Product will Be Peer Reviewed More than Once, and can the Same Peer Reviewer be Used Again and Again for Differ ent Products? There is no prohib ition on using the sa me peer review er more than once on the same product or for mul tiple products of th e same office. H owever, it is a g ood idea to use some different p eople each t ime where po ssible to provide a br oader perspe ctive. Whe n using a contractor to provide peer review services, you should re cognize that contractors may have a “pool” of reviewers that they use regularly

. If the same peer reviewers are used repe atedly, they may lose their independence (or the appearanc e of independence) from the work product(s). If a peer reviewer is asked to participate in multiple reviews of the same product it should be noted in the peer review record.
1.2.8 How is Peer Review Different from Public Comment? Peer review and public comment are mutually exclusive

. Public comment solicited from the general public through the Federal Register or by other means is often required by the Administrative Procedures Act, relevant statutes or both. Public comment can also be solicited for policy purposes. The Agency takes public comment on some strictly scientific products and almost all regula tory decisi ons. Public commente rs usually in clude a broad ar ray of peo ple with an interest in the technical analysis or the regulatory decision; some are scientific experts (which may provide some peer input), some are experts in other Public comment does not subs titute areas, and some are interested non-experts. The for peer review. critical distinction is that public comment doesn’t necessarily draw the kind of independent, expert information and in -depth analy ses expected from the peer review process. Public c omment is open to all issues, whereas the peer review process is limited to consideration of technical issues. While it may be an important compo nent of the revie w process, public comment does not substitute for peer review.
1.2.9 How is Peer Review Different from Stakeholder Involvement? Stakeholder involvement occurs when the Agency works with external intere st groups that have some stake in or concerns over the outcome of the technical work product or reg ulatory position. This is an in teractive proc ess, working with other agencie s, industry g roups, regulat ed-community experts, environmental groups, other interest groups that Stakeholder involvement is not repr esen t a b road spe ctrum o f th e reg ula tedcom mun ity, a peer r eview mechani sm. etc., and usually strives for a consensus approach. The goal of peer review, on the other hand, is to obtain anPeer Review Handb ook Page 13 independent, third-party review for ensuring scientific integ rity and technical credibility of the work product that supports a policy or decision. Stakeholder involvement is not a peer review mechanism and it do es notsubstitute for peer review even though it adds value to the work product.
1.2.10 What Role does Pe er Review have in the Regulatory Deve lopment Proc ess? The peer review of scientific and technical work products that support rulemaking actions is an important, fun damental step in the policy settin g process and w hich affirms the c redibility of the Agency. Because new r ules, and the work products supporting them, must often withstand intense scrutiny by the gene ral public and the stakeholders involved in the action, the peer review process selected for such work products needs to be well planned and documented. The rule or regulation itse lf is not subject to th e Peer Review Polic y. However , if the rule or re gulation is supported by a major scientific and/or technical work product, that work product should be peer reviewed prior to its use in the rule (see Section 2.2 for determination of major work products). The decision to pe er review or not peer review an y scientific and/or technica l work product will be documented through the Agency’s annua l peer review reporting process (see Section 1.3). Remember, public comment and stakeholder involvement do not constitute peer review.
Tier 1 and Ti er 2 rulemak ings, are by defi nition im portant, m ajor Agency rulem akings within the Agency. Therefore, work products supporting Tier 1 and Tier 2 rules in particular (including rules that are determined to be “significant” by OMB under Executive Order 12866 because they have an economic impact of $100 millio n or more) should be closely scr utinized to determine whethe r they meet the criteria for major (see Sectio n 2.2.3). Work produ cts supporting Tier 3 rulemakings may also be considered major a nd thus candidates for peer review. External peer review is the procedure of first choice for a work produc t that is intended to support a Tier 1 or Tier 2 rulemaking. Although acceptable in certain circumstance s, any decision to use an internal peer review mechanism for such work products would be the exception rather than the rule. For work products supporting a Tier 3 rule, internal or external peer review may be appropriate depending on the nature of the product and other factors (see Section 2.4.1). F or Tier 1 and Tier 2 rulemakings, th e Final Agenc y Review/cl osure memo needs to indicate that the Peer Review Policy was followed. For Tier 3 rulemaking s, the action memo needs to indicate that the Peer Review Policy was followed. Analytic blueprints are required for Tier 1 and Tier 2 rule makings, and are encouraged for Tier 3 rulemakings; some individual EPA offices require it for Tier 3. For peer review purposes, development of the analytic blueprint is the process whereby the project manager identifies the supporting scientific and technical work products and identifies needed peer review. In th e directive fro m the Deputy Administrator (me mo from Acting De puty Administrator Robertson, June 15, 1999: Sound Science and Peer Review in Rulemaking; seePage 14 Peer Review Handb ook Appendix B - Sound Science and Peer Review in Rulemaking), planning for peer review must be specifically addressed in each new analy tic blueprint. You are required to integrate peer review into the rulemaking process and to include specific language addressing peer review in the action memorandum. The analytic blueprint shows the schedule of the peer review in the context of the schedule for the overall rulemaking. In genera l, peer review should be completed as early in the process as practicable. Where possible, peer review of work products should be completed prior to issuance of the proposed regulation. In some cases, support work products for f inal regulations may require an addit ional peer revie w if those scientif ic and technical work products change significantly after the public comment period.
1.2.11 What Role does Pe er Review have in Re gulatory Negotiat ions? Regulatory negotiations are not candidates for pee r review; however, to ensure final decisions are based on sound and credible science, the major scientific and technical work products that support the negotiation need peer review before the negotiation takes place

. 1.3 Annual Agency Reporting Requirements
1.3.1 What are the Annual R eporting Requirem ents? The Peer Review Coordinator (for each AA/RA; see Section 1.4.5) will organize an annual review of all peer review activities and submit this information to the Office of Research and Development ( ORD). ORD will staf f this function at th e direction of the Deputy Administrator. In the Deputy Administrator’s annual call for submissions, guidanc e on format and submission of this information will be provided. ORD will review the submissions for completeness, i.e., all information i s provided and pro ducts are acco unted for each y ear. ORD will then p rovide a rev iew of the co mpletenes s of the information i n the submi ssions th rough consultation with the appropriate persons in each organization (see Section 1.4). ORD will then consolidate the information and findings for the SPC and the Deputy Administrator. Any conflicts arising from the review will be resolved by the Deputy Administrator. The due date for the annual reporting will be announced each yea r in the annual call letter; however, for planning purposes, it is normally due in the early summer. The Of fice of Environmental Information (OEI) assist s ORD in these annua l reporting func tions.
1.3.2 What Listings are Required for the Annual Reporting?
The Peer Review Pr oduct Tracking (PRPT) database is a Lotus Notes s hared-databa se developed to track and report peer review activities across the Agency

. The database is the single repository for product-specific peer re view reporting and tracking and uses a common reporting form for all reporting. Work products for which peer review is completed ar e reported as List A (Peer Review Completed); candidate work products for future peer r eview are reportedPeer Review Handb ook Page 15 as List B (Peer Review Needed); and work products that do not meet peer r eview criteria are reported as List C (Peer Review Not Needed). List D (Scientific Articles/Reports Peer Reviewed Outside of EPA) was recently added largely to track the articles that EPA staff produce. Four listings of products are required for the annual reporting: a) List A - Products Peer Reviewed Since 1991 1) List A is a cumulative list of peer reviewed products from 1991 to the present. 2) Each work product entered onto List A must include information summarizing the peer review. The database will specify the ne eded information. The entry must be electronically signed (verified) by the Peer Review Leader, the Peer Review Coordinator, and the Decision Maker involved (see Section 1.4.3 to help determine who the Decision Makers are). b) List B - Candidate Products for Future Peer Review 1) List B is a list of products that are expected to be peer reviewed in the near future

. 2) This List conta ins major scientific and technical work products and any non-major scientific and technical work products for which peer review has been deemed necessary or appropriate

. All entries on List B must be electronically signed (verified) by the Peer Review Leader, the Peer Review Coordinator, and the Decision Maker involved. 3) Work products placed on List B remain on List B until they are either peer reviewed (after which they are moved to L ist A) or a decision is made not to peer review th at work product ( at which point it is mov ed to List C including the ra tionale for not co nducting peer r eview). If t he product is terminated, it is mov ed to List C. c) List C - Products for Which a Decision has Been Made Not to Peer Review 1) List C is a cumulative list of all scientific and technical work products that do not receive peer review (see Section 2.3).Page 16 Peer Review Handb ook 2) List C includes: (a) any major scientific and technical wor k products for which a decision w as made not to peer review; (b) those work products which were originally placed on L ist B, but for which it was decided that peer review was not necessary (e.g ., the product was not used in decision making; the project was canceled); and (c) all non-major scientific and technical work p roducts. 3) List C also includes several categories of work products that would not normally receive peer review. I t is not necessary to list these work products individually on List C, however, the total number prepared by the organization must be included on L ist C. The organiza tion may nee d to identify the individual products if requested (e.g., due to litigation, FOI A, etc.). Such categories can include, but may not be limited to: chemical action reports, RCRA permits, scientific analyses for Premanufacturing Notices (PMNs) that are conducted on a routine basis and that do not deviate from esta blished practice , and NPDES permits. 4) Each work produc t on List C needs a brief descriptio n of the reason(s ) it is not being peer r eviewed. All entr ies on List C must b e electronical ly signed (verified) by the Peer Review L eader, the Peer Review Coordinator, and the Decision Maker involved. d) List D - Scientific Articles and Reports That Are Peer Reviewed by Organizations Outside of EPA 1) List D includes EPA authored scientific papers (articles) that are peer reviewed by a credible referee d scientific journal (See Sections 2.4.4 and 2.4.5). 2) Listing such documents in this manner shows that these items have already been peer reviewed, but differ entiates them from Agency work products. In addition, it gives EPA an opportunity to highlight the extensive work it produces in the scientific literature. 3) It is not neces sary to list t hese papers i ndividual ly on List D, alt hough some organizations may choose to do so. However, at a minimum, the total number prep ared by the organization and peer reviewed by journals must be included on L ist D. The organ ization may ult imately nee d to identify the individual papers and where published, if requested (e.g., due to litigation, FOIA, etc.).Peer Review Handb ook Page 17 4) Once any o f these articles or reports are u sed in an Agenc y work produ ct, that work product becomes a candidate for peer review and must be listed on List B (or List C if a decision is made not to peer review the work product). 5) Entries on List D do not require an electronic signature.
1.3.3 When will the Handbook Itse lf be Revised? During each annual reporting cyc le, suggestions for revisions to the Handbook should be submitted. Periodically, a decision will be made by the Science Policy Council (SPC) on whether to revise the Handbook, in pa rt or total, or not ba sed on the sugg estions. The SPC will then direct the Peer Review Advisory Group (PRAG) to draft the revision(s) for Agency comment and SPC appr oval.
1.4 The Roles of People and Organizations in Peer Review
1.4.1 Who is Ulti mately A ccountabl e for Pee r Review? Under the June 7, 1994 Peer Review Policy (Appendix A - USEPA Peer Review Policy), the Administrator has designated the Assistant Administrators and Regional Administrators (AAs and RAs) to be accountable for implementing the Policy in their respec tive organizations. The Deputy Administrator is ultimately responsible for peer review across the Agency and is the final arbitrator of conflicts and concerns about peer review.
1.4.2 Who are the Agency St aff involved in Peer R eview? The principal Agency staff involved are De cision Makers (and their line managers), Peer Review Lead ers and Peer Revi ew Coordinators. I n addition, there a re many othe r critical staff in each office and region who have responsibility for peer review activities (e.g., office and division peer review coordinators, technical information managers, and, of course, a ny Agency staf f that serve as interna l peer reviewe rs). Finally , ORD has oversig ht, as designate d by the Depu ty Administrator, fo r ensuring the A gency’s Peer Review Policy requirements ar e met.
1.4.3 Who are the Decis ion Makers & What ar e Their Responsibilit ies? The AA/RA is the ultimate Decision Maker for their organization and is accountable for the decisions regarding the identification of major scientific and technical work products and the mechanism(s) of p eer review utili zed for each of th e products. The A A/RA may des ignate Office Directo rs and/or Division D irectors as the f ront-line Decisi on Makers.Page 18 Peer Review Handb ook Generally, the Decision Makers decide whether a work product is major and needs peer review or not, and what peer review mechanism to use. Furthermore, the Decision Maker s commit the resourc es needed to ensu re a proper pee r review. Decis ion Makers are r esponsible for ensuring that the peer reviews are properly performed and documented.
In order to e nsure greate r independence of peer review s, it is necessary to strictly se parate management of work products from the actual peer review of those work products. Therefore, the Decision Make r and the Peer Rev iew Leader (see Section 1.4.4) for a work pro duct should never be the same person. The Decision Maker needs appropriate training on how to manage the peer review process. The Pee r Review Coordina tor for the offic e can advise the Decision Maker o n how to obtain the approp riate training; the training app roach varies am ong the differe nt EPA offices. Specific re sponsibi lities of t he Decision Maker(s) are the follow ing: a) Determine which work products in their organization require peer review b) Designate (in c onjunction with the P roject Manage r) a Peer Review Leader to organize the peer review c) Provide advice, guidance, and support to the Peer Review Leader in the preparation, conduct, and completion of the peer review d) Ensure that suffi cient funds are d esignated in the office’s budg et request to conduct the peer review; also ensure that adequate resources and/or extramural management support are available for the peer review e) Establish a realis tic peer review schedule f) Designate the s tage(s) of pro duct development w here peer rev iew is appropriat e g) Ensure all relevant issues and comments raised by the peer review er(s) are adequately addressed and documented for the rec ord, and where appropriate, incorporated into the work product that is used as basis for decision making h) By signature, document the decisions made that are re ported in the annual reporting to the SPC and Deputy Administrator i) Certify any decision NOT to pe er review a pro duct by sig nature on a L ist C ( Products for Which a Decision has been made not to Peer Review ) submissionPeer Review Handb ook Page 19
1.4.4 Who are the Pee r Review Leaders & What are Their Res ponsibilities? The Peer Review Leader is assigned by the Decision Maker to organize, conduct and complete the peer review for a specific individualwork product. The Peer Review L eader will obtain the assistance and support of the Peer Review Coordinator (see below) as well as any others within the Agency to help perform the peer revie w. The Peer Review Leader will be chosen on a case by case basis depending on the wor k product needing peer review. The Peer Review Leader cannot be the Decision Maker. The Peer Review Leader could be the Project Manager for th e work product. The Peer Review Leader must have appropriate training on how to c onduct a peer review before conducting the peer review. The Peer Review Coordinator for the off ice can advise the Peer Review Leader on how t o obtain th e appropria te trainin g; the train ing approach varies among the different EPA offices however, uniform training modules are available. Specific responsibilities of the Peer Review Leader are these: a) Keep the Decision Maker informed of the status of a given project; provide Peer Review Coordinator with data for the annual report b) Organize, conduct, and complete the peer review following Agenc y procedures c) Establish and maintain the peer review record for the specific individual peer review currently being performed (se e Section 2.5); this includes providing the peer review summary information in the Peer Review Product Tracking (PRPT) Database for the Decision Maker to sign when the peer review is completed d) Select the peer reviewers in consultation with others involved with the peer review (e.g., Decision Maker) and ensure that conflict of interest issues are addressed and documented in the peer review record e) Advise peer reviewers of their responsibilities f) Provide informatio n to the Decision Ma ker (including a ll appropriate ma nagers in the Peer Review Leader’s chain of command) on the charg e, profile of peer reviewers, the peer review comments, and a proposal on how to address the comments. Obtain D ecision Maker ap proval on the appr oach to respondin g to peer reviewer comments. Clearly identify for the Decision Maker any peer review comments that will not be addressed in the agreed upon approach.Page 20 Peer Review Handb ook g) Notify the Peer Review Coordinator that the peer review is completed for the annual report h) Archive the peer review record in a manner consistent with their organization’s archiving procedures When a contractor is used to conduct a peer review, some of the above responsibilities are assumed by the contractor (see Section 3.6).
1.4.5 Who are the Pee r Review Coordinato rs & What are Their Responsibilities? The Peer Review Coordinator is designated by the AA/RA to coordinate and monitor peer review activities in their respective organization or organizational unit. This person must be of sufficient stature and judgment to have the access to and confidence of all levels of off ice or regional management when needed. The Peer Review Coordinator is the main contac t for their organization; they can also direct interested parties to other persons/contac ts in the office on specific work products (e.g., Peer Review Leader ). Specific responsibilities of the Peer Review Coordinator are these: a) General oversi ght responsibility for the Office’s or Region’s pee r review proce ss b) Report peer rev iew activities to th e AA/RA c) Help mediate difficult issues between their organization and others; if they can’t resolve issue, then bring the issue to the attention of the appropriate level Decision Makers in each organization for resolution. d) Function as the liaison with ORD and the Science Policy Council (SPC): 1) Represent offic e/region befo re the SPC 2) Advise ORD of any changes in the list of work products and peer r eview mechanisms during the annual reporting, and when necessary , at other times 3) Participate in Agency peer review training , workshops, etc., as requested and disseminate this information to the organization; coordinate and/or present training within their organizationPeer Review Handb ook Page 21 e) Submit information on organization’s peer review candidates for each y ear as requested (this is the annual reporting, see Section 1.3) 1) Generate and update Lists A, B, C, and D (See Section 1.3.2) 2) Assure the prope r approval sign ature on the compl eted submission with the accompanying explanation for any departures from the Policy f) Establish procedures to assure that the required work product peer review documentation (i.e., peer revie w record) is file d and maintained in an appropriate manner (see Section 2.5) g) Provide advice, guidance, and support to the various Peer Review Leade rs for the performance of the peer reviews h) Distribute Agen cy-wide pe er review guid ance and materi als to appropriat e office/region personnel, as requested
1.4.6 Who are the Pee r Reviewers? Peer reviewers are individuals who have technical expertise in the subject matter of the work product undergoing peer review. Peer reviewers c an come from EPA, another Federal agency, or from outside of the Federal g overnment.
1.4.7 What are the Respo nsibilities of Peer Reviewers?
Peer reviewer s need to be willing participants in the peer review pr ocess -- they should agree to read all materials, participate fully, a nd protect confidential information that arises. Peer reviewers shoul d maintain the conf identiality o f the product, pe rform the review in a timely manner, and be unbiased and objective.
1.4.8 What is an Independent Peer Reviewer? An independent peer reviewer is an expert who wasn’t associated with the generation of the spe cif ic w orkpro duc t ei the r di rect ly by substantialcontribution to its development or ind ire ctl y by significant consultation during the development of the specific product. The independent peer reviewer, thus, is expected to be objective (See Sections 1.2.6 & 1.2.7 for further information)Page 22 Peer Review Handbook Independen ce is freedom fr om institutional, ideological, or technical bias regarding the issues The quality of the pe er review is under review and is necessary for objective, fa ir, and dependent on the compe tence and responsible evaluation of the work product. If a independence of the reviewers. selected reviewer has a particular scientific or technical perspective, it may be desirable to balance the review with peer reviewers with other perspectives. Idea lly, peer reviewers should be free of real or perceived conflicts-of-interest or there should be a balancing of interests among peer reviewers. I f there are pote ntial conflicts of interest (real o r perceived), t hey should be fully identified to ensure a credible peer review. (See Sections 3.4.5 & 3.4.6 for further informa tion).
1.4.9 When does an Agency Internal Peer Reviewer Qualify as Independent? An Agency independent peer reviewe r is one who comes from a different organizational unit than the one where the review question or document originates. A different organizational unit usually denotes, at minimum, a different office (i.e., above division level in programs; above branch level in regions) within the organization. In particular, a revie wer shouldn’t come from within the chain of command, either upward or downward.
1.4.10 What is a Peer Re view Panel? A peer review panel can range from a few individuals to ten or more, depe nding on the issue being investigated, the time available and any limitations on resources. I ndividuals who serve as peer reviewers must have appropriate scientific and technical expertise such that the review panel covers the broad spectrum of expertise required to treat the issues/questions presented in the charge.
1.4.11 What is a Subject Matter Expert? A subject matter expert is one who has specific scientific and technical expertise in the matter under re view. The importa nce of scientific and technical expertise in the subject matte r is obvious, however, knowledge or just “knowing” about the subject a rea isn’t equiva lent to expertise in the sub ject matter. For Agency d ecisions, a multi-d isciplinary group of experts corresponding to the disciplines that contribute to complex Agency decisions is often necessa ry for a full and complete peer review. For example, a risk assessment that relies on both animal and human data usu ally requir es experts in both ar eas for a comple te review. For e conomic anal yses , ex pert s fr om t he co rres pon din g eco nom ic d isc ipl ine s ar e nec ess ary.Peer Review Handb ook Page 23
1.4.12 What is the Role of the Science Policy Council (SPC)? According to the 1994 Peer Review Po licy stateme nt: “The Science Policy Counci l is responsible for overseeing Agency- wide implementation. Its responsibilities include promoting consistent interpretation, assessing Agency-wide prog ress, and developing recommendations for revisions of the policy as necessary .” The SPC meets its responsibilities through coordination with the Peer Review Coordinators, the Peer Review Advisory Group (PRAG) and the Offic e of Research and Development (ORD). The SPC, PRAG and ORD are not responsible for identifying specific products for peer review or determining the level of review or mechanism for that review; those functions are the responsibility of management within each Office or Reg ion.
1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)?
The Science Polic y Council has created the Pee r Review Advisory Group (PRAG) to assist in the implementation of the Agency's Peer Review Policy

. The primary role of the PRAG is to provide interpretation of the policy and to assist the SPC and Agency Offices and Regions in the annual update of the Peer Review Handbook.
1.4.14 What is the Role of the Office of Research and Development (ORD)? The Deputy Administrator has designated the Office of Resea rch and Development (ORD) to provide oversight to the Program Offices and Regions in the collection and review of information that is contained in the annual submission of Lists A, B, C, and D (for detailed information, see Section 1.3). The O ffice of Environ mental Infor mation (OEI) assists ORD in its oversight role.Peer Review Handb ook Page 25 

### 2. PLANNING A PEER REVIEW

2.1 Overview Statem ent Planning a peer review is a critical first step to ensure a successful peer revie w of a work product. The initial step is to determine whether your work product requires peer review. Once you have determined that a peer review will be conducted, the Decision Makers and Peer Review Leaders need to plan an appropriate review. This includes identification of re sources (budget and personnel), the schedule for the completion of the peer review, the mechanism for peer review, the choice of peer reviewers, and the development of the peer r eview record.
2.2 Determining Which Work Products to Peer Review
2.2.1 What are Scientific and Technical Work P roducts? The first step in determining which work products require peer review, is to identify products that are scientific and/or technical in nature. Scientific and technical work products are used to support a research agenda, regulatory program, policy position or other Agency position or action. Scientific and technical work products include economic and social science work products. Categories of work products include, for example: risk assessments, technical studies and guidance, analytical methods, scientific database de signs, technical models, technical protocols, statistical survey/studies, technical background materials, technical g uidance (except for guidance providing policy judgments), rese arch plans, and research strategies. Products that wouldn’t be considered scientific and technical work products can include those: that address procedural matters (e.g., planning, reporting, coordination, notification); that are primarily policy state ments (e.g., re location policy ); that are conf erence proce edings (unless the proceedings are used as the scientific basis for an Agency action or decision); and that are decision documents (e.g., Record of Decision (ROD) -- the decision document itself is not subject to the Peer Review Policy, but the underlying scientific and/or technical support work product is a candidate for peer review). In addition, the following Agency documents are not considered scie ntific and/or tec hnical work produ cts under the Pee r Review Policy : strategic plans, analy tic blueprints, an d goals document s. Making final determinations concerning which work products are scientific and technica l is a responsibility of the Decision Maker (See Section 1.4.3)Page 26 Peer Review Handbook
2.2.2 What Scien tific and Te chnical Wo rk Produc ts Need P eer Revi ew? The principle und erlying th e Peer Review Polic y is that all major scientific and technical work products use d in decision making will be peer reviewed. The process for identifying When in doubt about whether to peer which of these products is “major” (and thus review a work product or not, always a candidate for peer review) and then determining the mechanism of rev iew will decide to make it a candidate for peer take into account various criteria and the review. circumstances surrounding the use of that work product. To maintain flexibility, the Decision Maker( s) for peer rev iew should conside r the full field of possible work prod ucts to identify those additional products that might still warrant peer review as well as the full spectr um of peer review mechanisms for each product. Once a decision is made to perform pe er review, the product is listed in the annual submission of Candidate Products for Future Peer Review (List B - see Section 1.3.2b)). If a decision is made not to peer review a scientific and technical work product, the product is listed in List C - Products for Which a Decision has Been Made Not to Peer Review (see Section 1.3.2c)).
2.2.3 How Does One Determine Whether a Scientific and/or Technical Work Product is “Major”? Determinations of a scientific and/or technical work product as “Major” will largely be case-by-case. The continuum of work products cover s the range from the obviously major, which clearly need peer review, to those products which a re not major and clearly don’t need peer review. The rest of the work products fall in-between those two distinctions. This “middleground” probably represents the majority of work products, each of which needs to be evaluated closely and be compared to certain criteria (see below). The Decision Maker needs to make a judgment as to whe ther a work produ ct meets the crite ria for major or n ot. There is no ea sy single yes/no test of major covering the wh ole continuum of wo rk products. A rul e of thumb to remember -- if there is any doubt about whether a work pr oduct needs peer review, then go ahead and consider it a candidate for peer review (and place it on L ist B - Candidate Products for Future Peer Review). Scientific and technical work products that are used to support a regulatory program or policy position and that meet one or more of the following criteria are candidatesfor peer review:a) Establish es a signifi cant precede nt, model, or methodo logy b) Addresses significant controversial issuesPeer Review Handb ook Page 27 c) Focuses on significant emerging issues d) Has significant cross-Agency/inter-ag ency implications e) Involves a significant investment of Agency resources f) Considers an innovative approach for a previously defined problem/p rocess/met hodology g) Satisfies a statutory or other legal mandate for peer re view Usually, a major scientific and/or technical work product supports a reg ulatory decision or policy/guidance of major impact. Major impact can mean that it will have applica bility to a broad spectrum of regulated entities and other stakeholders, or that it will have narrower applicability, but with significant consequences on a smaller geog raphic or practical scale. The scientific and/or technical work that underlies many of the Age ncy’s major rulemakings and policy and g uidance documen ts of general a pplicability would be designa ted “major” unde r this scope of impact c riterion becaus e of their far-r eaching or sig nificant impacts. The novelty or controversy associated with th e work product he lps determine whe ther it is major or not. A ma jor work product m ay be novel or innovative, pre cedential, contr oversial, or emerging (“cutting edge”). An application of an e xisting, adequately peer reviewed methodology or model to a situati on that departs signifi cantly from the situation i t was originally designed to address is a candidate for peer revi ew. Simil arly, a modifi cation of an existing, adequately peer reviewed methodology or model that departs signifi cantly from its original approach is a ca ndidate for peer review. Determ ination of “signi ficant departur e” as used in this Section is the responsibility of the Decision Maker. In summary, a major scientific or technical wor k product has a major impact, involves precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory obligation to conduct a peer review.
2.2.4 What Econo mic Work Product s Need Pe er Revie w? Economic analyses and reports are considered scientific a nd technical work products, and as such they are candidates for peer revie w. The following economic work pr oducts will normally be classified as major and will require peer review: a) internal Agenc y guidanc e for conducting economic and fina ncial analy sisPage 28 Peer Review Handb ook b) new economic and financial methodologies that will serve as a principal method or protocol used to conduct economic analyses within a program c) unique or novel ap plications of existin g economic and f inancial methodol ogies, particularly those that are re cognized to be out side of mainstrea m economic practices d) broad-scale economic assessments of regulatory pr ograms, such as those required by Congressional-mandates (e.g., the Clean Air Act r eports to Congress on benefits and cost s) e) newsta tedpreferen ce (e .g., con tin gent val uat ion ) and reve ale d pr efer ence surveys (e.g., recre ational travel cost surveys) developed to assis t in the economic a nalysis of a regulation or program f) new national surveys of costs and expenditures for environmental protection (e.g., financial needs surveys, po llution abatement e xpenditures survey s) g) economic resea rch plans develo ped to assess and a dvance the state -of-science in economic theory, methodologies or modeling (in particular, the tec hnical feasibility of the plan’s comp onents) h) new meta-analyses that re-analy ze existing published literature and supporting data on the measur ement of economic benefits, costs a nd impacts Generally , because of the nature of these t ypes of ec onomic work produc ts, you should conduct an external peer review. External peer reviews can be provided by the Science Advisory Board’s Environmental Economics Advisory Committee, or other appropriate outside organizations or individuals that have expertise in the technical economic issues raised in the economic work pr oduct.
2.2.5 Should Economic Work Products Prepared in Support of Regulations that are Classified as "Major" or "Economica lly Significant" be Pee r Reviewed? Normally they would not require peer review, if the economic work product applies accepted, previously peer reviewed methods in a straig htforward manner. Economic studies prepared to support “major” or “economically significa nt” regulations (“major” as defined below in this section) ty pically do no t utilize innovative o r untried economi c methods. It is unnecessary to conduct peer reviews of straig htforward applications or transfers of accepted, previously p eer reviewed e conomic methods or analyses. E conomic assessme nts prepared to support the regulatory development process routinely make use of previously published peerPeer Review Handb ook Page 29 reviewed literature and adopt tools that allow for the transfer or adaptation of these techniques and information. The procedures used to transfer or adapt this work will generally be established by separate economic guidance docume nts which have been peer reviewed. Therefore, ec onomic documents th at are develope d using these pro cedures will not no rmally require addition al peer review , even those prep ared in support of “major” and “ec onomically significant” r ules. Even where peer review is not needed, additional peer input can be beneficial in the development of economic work products for “major” and “economically significant” rules. At present, some peer input of these analyses is already likely to be included as part of the regulatory development process, including input rece ived from other EPA offices represented on the workgroup for the rule, the Agency’s Regulatory Steering Committee, and from the public as part of the public comment process for the rule. But there may be added benefit to employing additional peer input procedures, such as actively soliciting input from economists in other Agency o ffices on the qua lity and comp leteness of the e conomic analy sis. “Economically significant” rules under Executive Order 12866 ar e defined as rules that may have an annual effect on the economy of $100 million or more or adversely affect in a material way the economy, a sector of the economy, productivity, compe tition, jobs, the environment, public health or safety, or State, local, or tribal governments or c ommunities. The term “major,” as used in this Q&A, does not mean the same thing as the definition of “major” for purposes of deciding whether a work product requires peer revie w. Here, we are using the term as defined in the Congressional Review Act, which defines a “major rule” as one that has resulted in or is lik ely to resul t in: an annual eff ect on the econom y of $100 millio n or more; a major increase in costs or prices for consumers, individual industries, Federal, State, or local government agencies, or geographic reg ions; or significant adverse effects on competition, employment, investment, produc tivity, innova tion, or on the abili ty of U.S.-ba sed enterprises to compete with foreign-based enterprises in domestic and export markets.
2.2.6 What Other Economic Work Pro ducts Migh t Benefit fr om Peer Review? There may be other economic work products not covered in the pr eceding sections for which peer review might be useful. Examples of such work products are presented below; however, we do n ot intend to establis h a presumption of p eer review for these work produc ts. a) analyses me asuring the ec onomic impacts and effectiveness of adopting marketbased or economi c incentives as r egulatory management inst ruments b) analyses of economic policies established under other governme nt organizations (e.g., economic models used to study transportation, economic development, and international tra de policies)Page 30 Peer Review Handb ook

Mos t of the se types of ec ono mic work pr odu cts do n ot e xhi bitthe degr ee of comple xit y, or establish an innovative or untried approach, that would warrant a peer review. However , other factors, such as the potential significance of the analy sis for cross-Agency or inter-age ncy practices, or the significance of the issues addressed, may make peer review desirable.
2.2.7 What Other Social Sci ence Work Product s Need Pe er Revie w? Typically, a social science w ork product is one that includes empirical, logic-based approaches to answer technical questions about human motivation, human behavior, social interactions, and social processes, which are relevant to the environmental issues being addressed. The term “behavior” includes overt actions; underlying psychological processes such as cognition, emotion, temperament and motivation; and biobehavioral interactions. The term “social” includes sociocultural, socioeconomic, and sociodemographic status; biosocial interactions; and the various levels of social context from small groups to complex cultural systems. Examples of social science work products include analy ses and/or evaluations related to such topics as pollution prevention, risk communication, environmental information, environmental justice, quality of life, decision-making, and public participation. The following soc ial science work products will norma lly be class ified as major and will require peer review: a) internal Agency guidance for c onducting social impact assessments and other community cultural assessments related to different environmental protection approaches suc h as community -based watersh ed protection (he retofore refe rred to as social assessments). b) new social science methodologies that will serve as a principal method or protocol used to co nduct social asse ssments. c) unique or novel ap plications of existin g social scienc e methods such as su rveys, focus groups, interviews, network analyses, comparative a nalyses, and content analyses. d) new national surv eys of value s, perceptions an d preference s related to environmental protection. e) innovative research or analyses that address the human dimensions of environmental pr otection or envir onmental chang e in terms of socia l trends, future predictions and/or behavioral generalizations.Peer Review Handb ook Page 31 f) social science research plans developed to assess and advance the state-of-science in social science theory, methodologies or modeling (in particular, the technical feasibility of the plan’s comp onents)
2.2.8 How Should Peer Review be Handled for Products Developed under an Interagency Agreement (IAG)? Under an interagency agr eement (IAG), EPA provides funds to another agency for that agency t o use for a specif ic purpose. The r eceiving age ncy's gu idance for pee r review will most likely be different from the EPA peer review policy

. Therefore, if EPA plans to use any work products from that agreement, EPA must decide whether that document needs review under the EPA peer review policy (see Section 2.2.16).
2.2.9 Should Products from Contracts, Grants, and Cooperative Agreements Receive Peer Re view?
If there is a scientific and/or technical work p roduct resulting from a grant, contract, or cooperative agreement and it is considered major and will likely beused in Agency decisionmaking, the work product needs pe er review. Since it would probably result in a perce ived, if not real, conflict of interest, a group that is generating the work product usually cannot conduct or perform the pe er review of its o wn work product. E xceptions may be made in certa in instances for organizations that have adequate and well established recognized procedure s for peer review, such as the National Academy of Scienc es (NAS). In practice, the Agency may need to peer re view the product o n its own, or arra nge with an indep endent third gro up (e.g., via another extramural vehicle) to conduct the peer review. The Agency should not use the major scientific and te chnical work pro ducts from contra cts, grants, or c ooperative ag reements to support decision making unless the work products are peer reviewed for both scientific and technical rigo r and applicabili ty to the spec ific use to be made of the product. Be aware that contracts are very different from grants and cooperative agreements. Please note that there are important legal restrictions (discussed in 2.2.10 below) on the direct use of work products developed under grants and cooperative agre ements in the Agency’s decision-making process.
2.2.10 How Does Peer Review Apply to Products Generated Through EPA Grants or Cooperative A greements? Major scientific and technical work products that are generated through EPA g rants or coo pera tiv e agr eem ent s ar e can did ate s fo r pee r rev iew , bu t sp eci al c ons ide rat ion s ap ply.Page 32 Peer Review Handb ook First, you need to understand some background information on the proper use of assistance agreements (grants and cooperative agr eements) versus contracts. Under the Federal Grant and Cooperative Agreement Act, grants or cooperative ag reements may only be used where the principal purpose of the agreement is to accomplish a public purpose that is authorized by statute. EPA may derive some incidental use or be nefit from the award as long as the principle purpose of the project is public support. However, if the principal purpose of the agreement is to obtain a product o r service for th e direct benefi t or use of EPA, a c ontract must be used rather th an a grant or c ooperative ag reement. EPA Order 5700.1, “Policy for Distinguishing Between Assistance a nd Acquisition,” includes the follo wing examples of p rojects that cannot be performed using grants or cooperative agreements: a) Research and stu dies which gathe r specific infor mation desired by EPA for its own use. b) Research which provides technical or analytical advice for EPA’s direct benefit or use, such as infor mation used to set g uidelines. c) Projects that produce specific information that will be directly incorporated by EPA into technical , policy, or r egulatory decisions. Note that under the Order, EPA may legally provide financial assistance for research that is intended to stimulate or support development of scientific knowledge that is not primarily for EPA’s direct use or benefit. The resulting work products would be widely disseminated e ither through publication in scientific journals or through other means as opposed to a report tailored to EPA’s specific needs and requirements. EPA can consider these work products just as it can review other published scientific works when formulating its programs and policies. Further, EPA retains a royalty free , nonexclusive and irrevocable right to use the work products for Federal purposes, even if the recipient has copyr ighted the material. [40 CFR 30.36(a).] Consult the Order and OGC for more information about making the proper choice between an assis tance agree ment and a contra ct. Provided that EPA’s use of a grant o r cooperative a greement rec ipient’s work prod uct is incidental to the agreement’s principal purpose, EPA may still determine that the rec ipient’s work product is a “major scientific or technical work product” under EPA’s peer review policy because: 1) it wil l be used to support a n EPA program or policy positio n (assuming this us e is incidental to the principal purpose of the agreement), and 2) it meets one or more of the seven criteria outlined in Section 2.2.3. In this situation, the work product requires peer review.Peer Review Handb ook Page 33 The following ar e options for peer reviewing the p roduct: a) EPA can have the product peer reviewed with the participation of the assistance agreement recipient/author

. In this case, EPA could arrange for an independent peer review of the product itself, or may contract with a third par ty to conduct the review. EPA would also enter into a contract with the author (formerly the recipient), which would task the author to prepare a response to the peer reviewers’ co mments, and to revi se or prepare a n addendum to the pr oduct in response to peer reviewer comments as determined appropriate by EPA. A caveat to this ap proach is that it ma y be diffic ult to get the rec ipient/author to agree to allow EPA to determine how to revise the product in response to the peer review comments. b) EPA can have the product peer reviewed without the participation of the recipient/author

. EPA could arrange for the peer review itself, or could contract with a third party to conduct the review. In this case, how ever, the work product would not be revise d to incorporate t he peer review comments. Inst ead, EPA would receive the comments and prepare a statement that documents its own response to the comments. The EPA Decision Maker(s) who is using the work product to support an EPA program or policy decision must be provided information on both the conclusions of the recipient’s work product and EPA’s own conclusions from the peer review. c) Recipients can get their products peer reviewed on their own

. Recipients may determine on thei r own that peer r eview would bene fit the credibility of their product. Provided EPA agrees tha t a peer review would further the public purpose of the assistance agreement, EPA may include funds f or the peer review in t he ag reem ent

. (S ee S ect ion2.2 .11 foradd iti ona l in format ion .) A lte rnat ive ly, the recipient may make arrangements for, a nd fund, an independent peer review of their product. In either case, EPA would need to evaluate whether the pe er review process undertaken by the recipient was ac ceptable for the purposes for which EPA was planning to use the work product. EPA may accept the pee r review if it determines that it fulfilled the requirements of EPA’s Peer Review Handbook and that EPA could defend the peer review as if it were conducted by EPA itself. Under options a) or b), issues may arise over obtaining acce ss for peer reviewers to the raw data used by the recipient to g enerate the wor k product. Under 40 CFR 30.36(c)(2 ), EPA has a right to obtain raw data produced by a non-profit org anization or university under an assistance agreement, even where the agreeme nt doesn’t specifically provide for this access.Page 34 Peer Review Handb ook Nevertheless, it may be prudent to include a specific term in the assistance a greement clarifying this point to avoid misu nderstandings. EPA must pay f or obtaining ac cess to the data if its transmittal imposes additional costs on the recipient. Assuming our use of the data is incidental to the principal purpose of the agreement, we also have a specific right to authorize pee r reviewers to use the data for Federal purposes under 40 CFR 30.36(c)(2). The only time EPA cannot obtain a ccess to the raw data is where EPA specifically bargains away this right in the assistance agre ement. Again, consult OGC for help in drafting appropriate language for your assistance agreement.
2.2.11 Can the Recipient o f a Grant or Cooper ative Agreem ent Use Agreem ent Funds to Pay Pe er Reviewers of the ir Work Products? As noted in Section 2.2.10 above, provided EPA agrees that a peer review would further the public purpose of the assistance agreement, EPA may include funds for the pee r review in the agreement. A payment to peer reviewe rs in exchange for their review of a scientific/technical work product is allowable as a fee for professional services under assistanc e agreements. (To accurately cha racterize this cost, however, it is important that the payment be referred to as a fee, rather than an honorarium.) See OMB Circular A-21, Section J, item 32, Professional Services Cost (Educational Institutions), OMB Circular A-122, Attachment B, I tem 39, Professional Services Costs (Non-Profit Organizations), and OMB Circular A-87, Attachment B, Item 33, Professional Services Costs (State, Local and I ndian Tribal Governments). (See http://www.whiteho use.gov/OMB/c irculars/ for further details.)
2.2.12 Do Products Generated under EPA Grants or Cooperative Agreem ents Need to be Reported in the Peer Review Product Tracking (PRPT) Database? If a grant or cooperative agre ement product is determined to be a major scientific and/or technical work p roduct and is used in Agency decision making ( assuming this use i s incidental to the principal purpose of the agreement), it must generally be considered a candidate for peer review. Therefore: a) if it is determined to be a major work product and will be peer reviewed by EPA, the work product is ent ered onto List B ( Candidate Products for Future Peer Review) as a “future peer review” product, then moved to List A ( Products Peer Reviewed Since 1991 ) when the peer review is completed. b) if it is determined to be a major work product, but will not be peer reviewed (see the possible reas ons for not peer r eviewing a majo r product in Sectio n 2.3.1), it isPeer Review Handb ook Page 35
entered onto L ist C ( Products for Which a Decision has been made not to Peer Review), with a justifica tion for not peer r eviewing it. c) if it is determined t o not be a major wor k product but EPA de cides that it still warrants peer review (see Secti on 2.2.17), it is ent ered onto List B ( Candidate Products for Future Peer Review) as a “future peer review” product, then moved to List A ( Products Peer Reviewed Since 1991 ) when the peer r eview is completed. d) if EPA decides to us e the work produc t (due to its importa nce in EPA environmental de cision making), b ut considers it not ma jor and not a cand idate for peer revie w, it is entered ont o List C ( Products for Which a Decision has been made not to Peer Review) with the reasons for it being non-major. e) if the grant or cooperative agreement product will not be used in Agenc y decision making, it is not g enerally c onsidered a can didate for peer review. Such pro ducts do not have to be listed.
2.2.13 Should Site S pecific De cisions be Subject t o Peer R eview? The site specific decision itself is not subject to peer review and doesn’t need peer review bas ed s ole ly onthe Pee r Re vie w Po lic y. Ho wev er, i f a s ite spe cif ic d eci sio n is sup portedby a major scientific and/or technical work product, that work product needs peer review. While the same considerations for major apply here, several of the c riteria above (see Section 2.2.3; specifically criteria b, c, d, and g) are considered more useful for regional consideration than other criteria. So generally speaking, a close examination of ho w the underly ing major scient ific and/or technical work product is adapted to the site specific circumstances is required.
2.2.14 Should NEPA Product s (e.g., EI Ss) be Subje ct to Pe er Revie w? Not everything requires peer review, a nd in the case of an Environmental Impact Statement (EIS) prepared under the requirements of the National Environmental Policy Act (NEPA), the document already has received extensive review (although not necessarily peer review) through the “scoping” and interagency review processes that are part of NEPA.
The rule of thumb is that if the underly ing scientific a nd/or technical w ork product is major, then the wo rk product needs peer review. I n general, the Agency’s role in the NEPA document would suggest what sort of review the document gets. If EPA is developing the document as part o f an EPA action/de cision (EPA is the L ead agency under NEPA), and it meets the definition of "major," then it needs independent peer review. If it is not a major workPage 36 Peer Review Handb ook product (little impact, non-controversial, etc.), then peer input/continuing involvement might well be appropriate. On the other hand, if EPA is reviewing an EIS from another Agenc y (EPA is not the Lead agency unde r NEPA), it is likely that we are reviewing for conflicts with EPA policy and general envir onmental concer ns. However, EPA m ust ask if the under lying major scientific and/or technica l work product tha t supports the EI S has been peer r eviewed. If not, this would raise concern about the full credibility and soundness of the EI S based on the science and technical support. EPA should work with the other organization/agency to ensure tha t the major scientific and/or technical work p roduct receive s peer review a dequate for EPA p urposes. 2.2.15 Should Environmental R egulatory Models be Peer Reviewed? Generally, yes. Specific g uidelines for the peer review of environmental regulatory models have been published by the Agency

. These can be found on the EPA web site under the Science Policy Council home page (http://www.epa.gov/ORD/spc).
2.2.16 Is Peer Review Needed for Other Organization’s Work Products that Have been Submitted to EPA for Use in Decision Making? Any scientific and/or technical work product that is used in Agency decision making and is considered major becomes a candidate for peer review rega rdless of whether the work product is produced by the Agency or another org anization. Theref ore, all major wor k products important to EPA environmental decision making that are independently generated by other organizations (e.g., other Fed eral agencie s, interagenc y groups, St ate and Tribal bo dies, environmental groups, industry, educational institutions, international bodies) need to be considered as candidates for peer review just as major work products gener ated by EPA are considered (these would then be included on List B ( Candidate Products for Future Peer Review) if EPA is conducting or arranging for the peer review). If possible, when EPA knows that a work product is being generated by another organization and is of interest to EPA for future use, the appropriate EPA office(s) should work with that organization, and others, as appropriate (e.g., the states), to promote the use of peer review. For example, the Office of International Activities (OI A) as well as the impacted program or regional office(s) should be included when international products are being considered for EPA use. It is hoped that if the other organization has the work product independently peer reviewed, the peer review will meet the intent of the Agency’s Peer Review Policy and EPA’s proposed use of the product (i.e., the peer review is basically e quivalent to what EPA would do). Agency staff from the appropriate offic e(s) should examine closely the particulars of the peerPeer Review Handb ook Page 37
review to ensure independence a nd a conscious ef fort to incorpora te the peer revi ewers’ comments into the fi nal work product. If there ar e perceived, or real, conflicts o f interest, this may preclude the use of that peer review and, in those instance s, another peer review would be needed. See Section 3.4.7 for considerations for when an outside party conduc ts and/or funds peer revi ew o f th eirownwork pr odu ct and s ubm its it t o th e Age ncy. If the outside organization doe s not have the major work product pee r reviewed and E PA decides it needs peer review, the appropriate EPA office(s) will have to ensure pee r review of that work product occurs prior to the Agency’s use of the work product in decision making. Peer review can be a ccomplished by asking the outside organization to do so, or if they refuse, EPA may conduct or arrange for the pee r review. If EPA is conducting or arranging the peer review, the product should be en tered on to List B ( Candidate Products for Future Peer Review).
2.2.17 Can Work Products That are Not Determined to be Major Still be Peer Reviewed? Yes, they could be. Scientific and technical work products that do not come under the “major” distinction discussed above may nonetheless be candidates for peer review. For example, a project manager may decide to use peer review because of particular program needs and goals. Peer review may also be warra nted because it adds substantial value to the work product. In these cases, the pro duct should be entered o nto List B ( Candidate Products for Future Peer Review) as a candidate for future peer review.
2.3 Determining Which Work Products Do Not Receive Peer Review
2.3.1 Are There Circumstances When a Major Work Product is Not Peer Reviewed? There may be circumstances where a wor k product is considered major, but a decision for no peer review can then be justified. For example: a) Additional peer r eview is not requir ed with work that ha s been previously reviewed by recognized expe rts or an expert bod y. For examp le, a cancer ri sk assessment methodology or an exposure modeling technique that was the subject of earlier peer review would not require additional peer review, even if the product supported a significant Agency decision. b) Additional peer review is not required if an application of an adequately peer reviewed work product does not depart significantly from its scientific or technical approach.Page 38 Peer Review Handb ook c) Additional peer review is not required when the scientific and/or technical methodologies or information being used are commonly acce pted in the field of expertise (e.g., Control Techniques Guidelines (CTGs) or other such compilations). This would need the appropriate documentation to support the commonly held view. d) Most often, a major work product would not receive peer review when the regulatory activity or a ction which the work product supports is terminated or canc ele d -- no f urther a cti on, inc lud ing p eer r evi ew, is n eces sary. e) In a few insta nces, statutory or court ordere d deadlines or othe r time constraints may limit or preclude peer review of product that would otherwise be c onsidered major. However, it is up to the Decisio n Maker(s) to mak e every a ttempt possible to assure that pee r review of major products occurs taking into acco unt these deadlines. With proper up-front planning for peer reviews of major products and the ability to tailor the method of review to the pro duct and circumst ances, it should rarely be the case that major products don’t receive some type of pee r review due to time constraints. f) Very rarely, r esource limitations may also restrict peer review. Programs or Regions will eval uate these circ umstances on a ca se by case basis; decisions wi ll be based on consultations involving line management, the Project Manager, the Peer Review Leader, and the Peer Review Coordinator. If peer review of a major scientific and/or technical wor k product is not conducted, a written justification must be placed in the “Justification for Non-Peer Review Product” Section of the data base e ntry for tha t product on Lis t C ( Products for Which a Decision has been made not to Peer Review). The justification is signed-off by the appropriate Decision Maker (see Section 1.3.2c)).
2.3.2 What Products Nor mally Do N ot Need P eer Revi ew?
Products that are not major scientific and technical work products normally do not require peer review under the intent of the Peer Review Policy

. Most of these scientific and technical work p roducts are then placed on List C ( Products for Which a Decision has Been Made Not to Peer Review) with a written explanation of why it was determined to be not major (see Section 1.3.2c)). This justific ation is signed-o ff by the ap propriate Decis ion Maker to assure that allscientific and technical work products received consideration for peer review. Some scientific and technical work products are not considered major and generally do not need to be plac ed on List C ( Products for Which a Decision has Been Made Not to PeerPeer Review Handbook Page 39
Review). These types of work products typic ally include: derivative products (i.e., a product that only summarizes an already peer r eviewed product or products; fact sheets), compendiums of existing models, methods and/or technologies; minor intermediate products (e.g., a technical memorandum from a contractor) describing methods or results incorporated in a large r product which will be peer reviewed; or preliminary or incidental analy ses prepared separately from the work product ultimately used to support an Agency action or decision (e.g., during the course of developing a rule, managers may direc t staff to prepare various “what if” analyses; those that aren’t used in the work product do not need to be listed).
2.3.3 Do Volunta ry Consens us Standard s Require P eer Revi ew? Generally, no. The National Technology Transfer and Advancement Act of 1995 (NTTAA) directs EPA to use available voluntary consensus standards in its reg ulatory activities unless to do so would be inconsistent with applicable law or otherwise impractical. For purposes of the NTTAA, voluntary consensus standards are defined a s technical standards (e.g., materials specifications, test methods, sampling procedures, and business practices) that are developed or adopted by voluntary c onsensus bodies (such as ISO, ASTM). The general purpose of the NTTAA is to reduce private and governmental costs by avoiding having the government “reinvent the wheel.” Voluntary conse nsus standards would normally not require peer review because the underlying pr ocess used by issuing organizations to develop and approve these standards is generally considered a dequate for purposes of the Agency’s pee r review policy. EPA reserves the right to conduct a peer review if it determines that the standard is not a voluntary consensus standard under the NTTAA.
2.4 Choosing a Peer R eview Mechanism
2.4.1 How Do You Determine the Appropri ate Peer Review Mechanism? During the planning of a peer review, the Decision Maker and the Peer Review Leader may consider several mechanisms for the peer revie w of major scientific and technical work products. These o ptions range fr om consultations wit h EPA colleague s not involved in developing the product to a large and formal panel of outside subject matter experts. The peer review effort might be a focuse d one-time evalua tion, or could The mechanism of the peer encompass several examinations over the course of a review matches the project. In p rinciple, peer r eview provides th e greatest importance and complexity of credibility for major work pr oducts when it invol ves well- the major work product. qualified externa l reviewers, is in tensive in its examination, and operates through a more or less formal process. As a pra ctical matter, ho wever, time and r esource conside rations in many cases impose limitations on what c an be reasonabl y achieve d. Arranging for the most appro priate and feas ible peer review will involve good judgment and a willingness to consider substance, time, and resource tra deoffs. Dev eloping a pee r review pla n that prov ides for app ropriate de pth, timi ng,Page 40 Peer Review Handb ook and content is an important matter for early consideration by the Decision Maker and Peer Review Leader. Note that use of peer input, public or stakeholder involvement does not qualify as peer review. The approach be st suited to a specif ic work product w ill depend on the na ture of the topic andthe int end ed fi nal pro duc t. G ener all y, th e mo re no vel or co mpl ex t he s cie nce o r te chn olo gy, the greater the cost implications of the impending decision, and the more controversial the issue, then the stronger the indication for a more extensive and involved peer review and for external peer review in particular. Certain work products will clearly le nd themselves to extensive external peer review; generally these will be produc ts with large impacts (e.g., those that support Tier 1 and Tier 2 rulemakings). Other major work products may not need a large scale external peer review and may utilize a less involved, less resource intensive review. The peer review of some products may be better served with some form of internal peer re view or a combination of internal and external peer review. It is important t o make the choice of peer review mechanism at the ti me that the work is planned (for products supporting rule makings, at the analytic bluepr int stage) so that peer review costs and time can be budgeted into the work plan. Essentially, the level of peer review matches the impact and complexity of the major work product. For example, a rule under development carries considerable weight and deserves careful ha ndling and attention; therefore, the supporting wo rk product deser ves similar care and attention for its peer review. Both internal and external peer review mechanisms are available, have been used in the past, and ha ve served to addre ss the needs and c hallenges of a particular pee r review situatio n. Nevertheless, no single peer review mechanism is likely to work best in all situations. Some useful guidance includes: a) Major work products intended to support the most important decisions, or that have special importance in their own right, ordinarily should be the subject of external peer review. Generally, the more complex, novel and/or controver sial the product, or the higher impact it h as, the more the D ecision Maker sho uld consider implemen ting a large-scale peer rev iew involving exte rnal experts. b) Major work products that are less complex, novel, or controversial, or have a lower impact may not need such a la rge-scale an d external peer r eview. These products might be subject to one of the less extensive, less resource-intensive review process es. c) Group discussion with peer reviewers can be very helpful a t some point in the peer review pr ocess. On the othe r hand, simply soliciting individu al comments is easier, faster , and less expensive

. Individual r eview is probably more appropriat ePeer Review Handb ook Page 41 for peer revie w at the early stages of a pro duct’s developme nt or for products with les s impact andcomple xit y. d) Strict time constraints, such as a court-ordered deadline, can make a less involved or formal peer review mechanism imperative. But Decision Makers and Peer Review Lead ers must make maximum e fforts to assure t hat such a proce ss is perceived as systematic and objective.
2.4.2 What are E xamples o f Internal P eer Revi ew? a) Independent experts from within the Agency ( e.g., ORD experts on non-cancer effects of lead reviews a draft article on benchmark dose) b) An ad hoc panel of independent experts from within the Agency (e.g., an independent internal workgroup convened to examine the case for the classification of a chemical as a carcinogen) c) Technical merit review by scientists in an Agency laboratory (e.g., an initial review of the ris k assessment for a regional incine rator by Ag ency scien tists)
2.4.3 What are E xamples o f External Peer Re view? a) Independent experts from outside the Agency (e.g., a letter review by outside scientists) b) An ad hoc panel of independent exper ts outside the Age ncy (e.g., a group is convened to develop a consensus on the carcinogenicity of a particular industrial chemical) c) Agency-sponsored peer review wor kshops (e.g., a review of potential indicators of ecosystem damage) d) Review by an established Federal advisory committee such as the Science Advisory Board (SAB), FI FRA Scientific Advisory Panel (SAP), ORD’s Board of Scientific Counselors, or the Clean Air Scientific Advisory Committee (e.g., a review of a criteria document for a particular chemical risk) e) Agency-based federal advisory committee (other than those established and discussed in d above)Page 42 Peer Review Handb ook f) Agency a ppointed special b oard or commission (e.g., a revie w of the risk assessment method ology pre pared by th e Clean Air Act Com mission on Risk Assessment) Note: The Office of General Counsel should be consulted regarding EPA's authority to establish and finance the activities of a commission or board. g) Interagency c ommittee (e.g., a review of prospective research plans by the Committee on the Envi ronment and Natur al Resources coo rdinated by the White House) h) A committee convened by another federal ag ency or government organization (e.g., a review of the Dioxin Reassessment by the Health and Human Services (HHS) Committee to Co ordinate Environ mentally Rel ated Programs) i) Review by non-governmental groups (e.g., a Society of Risk Analysis review of cancer guide lines) j) Review by the National Academy of Sciences (e.g., a review of the state of current knowled ge about childr en’s health risks from pesticide exposures)
2.4.4 What is the Role of P eer Review by a Re fereed Scientific J ournal?
Pee r rev iewof j ournal art icl es ( wri tte n by E PA o r no n-E PA auth ors ) per formedby a credible, refereed scientific journal contributes to the scientific and technical credibility of the reviewed product. EPA considers peer review by such journa ls as adequate for reviewing the scientific credibility and validity of the findings (or data) in that article, and is therefore a satisfactory form of peer review. However , in some cases, peer review of an Agency work product that uses these articles may be required (see Section 2.4.5). EPA authored journal articles, whether used in an Agency work product or not, are tracked using List D (Scientific Articles and Reports That Are Peer Reviewed by Organizations Outside of EPA-- see Section 1.3.2d) for details). This shows that these items have been peer reviewed, and d ifferentiates t hem from other Ag ency work p roducts. In a ddition, it gives EPA an opportunity to highlight the extensive work it produces in the scientific literature. It is not necessary to list these papers individually on List D (Scientific Articles and Reports That Are Peer Reviewed by Organizations Outside of EPA ), although some organizations may choose to do so. However, at a minimum, the total number prepared by the organization and peer reviewed by journals must be included on L ist D. The organization may ultimately ne ed to identify the individual pape rs and where pub lished, if reques ted (e.g., due t o litigation, FOIA, etc.).Peer Review Handb ook Page 43 Prior to submitting an article to a journal for peer review, EPA employees ar e encouraged to have the artic le internally peer reviewed (see Section 1.4.9); such internal peer review is already c ommon procedure in certain parts o f EPA. Articles ma y also need examination in accordance w ith any org anizational clear ance procedur es, especially when the author is presenting him or herself as an EPA employee. F or EPA employ ees, conflict of interest regu lat ion s wi ll a lso app ly.
2.4.5 Do Agency Work Products Become Candidates for Peer Review when Peer Reviewed Journal Articles are Used in Support of that Work Product? In most instances, major Agency work pr oducts are candidatesfor peer review even when supported by peer reviewed journal article(s). A lthough the use of articles that have been peer reviewed by a credible journal streng thens the scientific and technical credibility of any work product in wh ich the article( s) appears or is r eferenced, it d oes not automatica lly eliminate the need to have the work product itself peer reviewed. In most cases, journa l peer review may not cover issues and concerns that the Agency would wa nt peer reviewed to support an Agency action. Under these circumstances, the major scientific and/or technical work product in which the article(s) a ppears or is ref erenced becom es a candidate f or peer review

. A journal articl e authored by EPA employees would be used in the sa me manner as an article published by anyone els e in a credible, w ell recognized journal. If an Agency wor k product is based solely on a single article that has received peer review by a credible journa l (e.g., where a model is sugge sted for a singul ar use that fits a specific Agency need), peer r eview of the Agency work product may or may not be necessary depending on how closely y ou apply the findings from the article. If an Agency work product is based on two or more articles that have received peer review by a credible journal(s), the Agency work product generally becomes a candida te for peer review. Decisions to make (or not make) a work product a candidate for peer review needs to be documented in the peer r eview record. One important factor to remember with regard to the use of articles that have rec eived jou rnal peer revi ew d eal s wi th t he av ail abi lit y of d ocumen tat ionfro m th at p eer r evi ew. Idea lly, EPA needs to maintain a clear, easily accessible record of the peer review to assure the credibility and validity of the peer re view (see Section 2.5 for details on the peer review record). However, the documentation from a journal peer review would not normally be available to the Agency, so such documentation is not expected in the peer review re cord.
2.4.6 When and How Often Should Peer Review Occur? The Decision Maker and Peer Review Leader have sig nificant discretion in deciding on the timing and the frequency of peer review. O ptions abound, each with merits depending on the context and specif ied peer review objectives. I n many situat ions, a single pe er review even t,Page 44 Peer Review Handb ook beginning when the final draft work product becomes available, is the approach ta ken. However, it is increasingly apparent that peer re view performed earlier in the work product development stages is a superior approach for some work products. There may be substantial incremental benefit to conducting more than one peer review during the whole proc ess of work product development, particularly where it involves complex tasks, has decision branching points, or could be expected to produc e controversial findings. In addition, early review would be beneficial at the stage of research design or data c ollection planning where the product involves extensive primary data collection. The Decision Maker and Peer Review L eader need to determine when the peer review(s) should occur, considering the ty pe of work product under development and a t what point in its dev elopment process a peer review w ould be most beneficial. Other type s of work products that would benefi t from early , up-front peer r eview in their development are scientific and te chnical planning products. Examples of such products are research proposals, plans, and strategies. Also, while not products per se, ongoing re search programs can be peer reviewed. Remember though, that while more than one peer review can be beneficial, the distinction between peer input and peer review needs to be kept in mind. Experts providing input during the d evelopment or pla nning stages of the work product g enerally d o not become peer reviewers of that product (see Sections 1.2.2 to 1.2.7 for full discussion on this distinction).
2.4.7 What Fac tors are Considere d in Setting the Time Frame f or Peer Review? Several factors impact how quickly a peer review may be needed. These include dead lin es f or co mpl eti on o f a pr oje ct, res earc h pr ogra m, o r ru lem aki ng, f und ing a vai lab ili ty, availability of quality pe er reviewers, and statutory and/or court-or dered deadline s. Peer review sometimes leads to new information and analyses. Reviewers may make recommendations for new research that would alter the work product and thus modify the scientific/technical basis for the action or rule it supports. For this reason, a completed peer review is desira ble before issuin g any prop osal for public co mment. If that is not logistically possible because of court or statutory deadlines, or other appropriate r easons, the Decision Maker should make every effort to complete the peer r eview before the close of the comment period. Because peer review comments on such work products could be of sufficient mag nitude to warrant a revision to the proposed action or rule, Decision Makers should exercise diligence in completing the peer review prior to the proposal stage whenever possible.Peer Review Handb ook Page 45
2.4.8 Which Office/Region or Other Agency is Re sponsible for Conducting the Peer Re view? The organization of the Decision Maker is normally responsible for conducting the peer review. Responsibility for conducting a peer review ca n be negotiable when more than one Agency office or region or othe r agencies are involved. Usually, the deg ree of involvement by any of the organizations and agencies and their a bility to fund peer review will often determine who has the lead for the peer review.
2.5 Creating the Peer Review Record
2.5.1 What is the Peer Review Record? It is the formal record (file) of decision on the conduct of the peer re view, the type of peer r eview perfo rmed, and an expl anati on of h ow the peer r eview comme nts we re add resse d. It includes sufficie nt documentation f or an uninvolved pe rson to understand what actually happened and why. The Peer Review L eader (with the program manager if there is one) cre ates a separate, clearly marked peer r eview file Section within the overall file for development of the work. Once the pe er review is comp leted, it is the res ponsibility of the Peer Review L eader to ensure that the peer review record is filed and maintained in accordance with the or ganization’s procedures. The Peer Review Record is separate from the entry in the Peer Review Product Tracking (PRPT) Database. While some information from the peer review record appears in the database, the peer review record is the official record of the peer revie w. 2.5.2 How Can the Peer R eview Record Im prove the Peer Review Process? A good peer review record allows future reference to what happened during the peer review, and helps Decision Makers make appropriate use of peer reviewer input. In addition, a good record helps ensure that EPA’s Peer Review Policy is followed. The Pee r Review Leader is responsible for ensuring that the peer review record for individual work products is collected and maintained unt il completion of the peer review ef fort.
2.5.3 What Should Be in the Pe er Review Recor d? The peer review record should include all materials considered by the individual peer reviewers of the peer review panel, as well as their written comments and other input. Such materials include, at a minimum (see also Section 4.3.1): a) The draft work product submitted for peer reviewPage 46 Peer Review Handb ook b) Materials and information (including t he charge ) given to the peer reviewers c) Written comments, information, and materials received from the peer reviewers d) Information about the peer reviewers (such as reviewer s’ names, affiliations, and a statement concerning potential conflicts and their resolution) e) Logistical information about conduct of the peer review (such as times and locations of meeti ngs) f) A memorandum, or other written record, approved by the Dec ision Maker, responding to the peer review comments specifying a cceptance or, where thought appropriate, rebuttal and non-acceptance g) The final work product When deciding if particular materials should be included in the record, the Peer Review Leader shou ld consider wheth er the materials would help recons truct the peer r eview process and results at a later time. If the materials may be helpful, they should be part of the peer review record.
In addition to h ard copies of mat erials, Peer Rev iew Leader s need to maintain e lectronic copies of the materials (e.g., charge) that are nece ssary for the annual reports to the Peer Review Tracking Data base. Peer Revie w Leaders c onsult with their Pee r Review Coordina tors to identify those materials.
2.5.4 What Should I Do with a Peer Review Record That Pertains to a Rulemaking Actio n? The Peer Review Leader should coordinate with their program’s doc ket office to see that proper docketing requirements are satisfied for a peer review of a work product supporting a new rule. The Peer Review Leader is also responsible for notify ing the workgroup chair as well as the Peer Review Coordinator (for the annual report) that a peer review is completed.
2.5.5 When Should the Peer Review Record Buildin g Process Begin? An early start at developing and maintaining a peer review record will help ensure the record is complete and helpful. Ideally, the r ecord begins when the decision to peer review a work product is made. The Peer Review Leader needs to construct the pee r review record from this point on -- this will avoid potentially time-consuming reconstruction at a later point. NotePeer Review Handb ook Page 47 that the peer review record is not complete until it contains a copy of the final work product which addresses the peer review com ments.
2.5.6 What are the Differences in Record Keeping for a Review by an Individual Compared to a P anel? Strictly speaking, a true peer review requires more than a single reviewer. A review conducted by one individual will rarely provide the depth of commentary required to improve the work product. In addition, y ou will not receive the range of vi ews and richness necessary to ensure improveme nt. In the case of a review panel, there will often be conflicting comments that must be resolved. This resolution should be in the record.
2.5.7 Where Should the Peer Review Record be Kept and For How Long? During the active conduct of the peer review, the Peer Review L eaders maintain the peer review record with themselves unt il the peer revie w is totally c ompleted. Minimally , the file should be m aintaine d until one year a fter the c ompleted p eer revi ew is repo rted in the next annual reporting

. After that, the p eer review re cord should be main tained for a “re asonable period of time.” Establishment and maintenance of the archive where the peer review re cords ultimately reside are an organization’s responsibility (i.e., not that of an individual program manager or Peer Review Leader). Gene rally, to allow flexibility, individual offices and regions will decide the appropriate level of organizational responsibility and how they will meet any “routinely available” requirements. The peer revie w record may be kept with other records relating to the overall project, as long as it is easily and separa tely identifiable.
There are als o specific r equirement s regarding th e use of dock ets for recor d-keeping; however, these are not covered in this Peer Review Handbook. The documents contained in the peer review record should be maintained in accordance with the Agenc y’s record keeping retention schedule for such records (for details, see EPA’s National Records Manage ment Program; http://www.epa.g ov/records/ ). One long-term archiving mechanism may be the f ormal archiving at the Federal Records Center in Suitland, MD.
2.5.8 Are Internal Pe er Review Comm ents Included in the P eer Review Reco rd? An internal EPA peer review must be a formal process to be considered a legitimate peer review. Thi s process ad heres to th e guidance fou nd in this Handbook fo r planning, conductin g, and completing a peer review. When you follow this formal process to obtain pee r review from EPA peers (see Se ction 1.4.9), then the whole recor d of that internal p eer review is inc luded in the peer review record. This includes all the materials detailed in Section 2.5.3 (also see SectionPage 48 Peer Review Handbook 4.3.1). Conducting a formal internal peer review is not the same thing as informal input from your EPA colleagues (i.e., “colleague s down the hall”), nor peer input from Agency personne l helping to develop the work product, nor organizational review and clearance proc esses. Such inputs from these informal processes should not be placed in the peer review record. The peer review record should contain only the information obtained when y ou conduct a formal internal peer review. In some cases, an internal EPA peer review may be followed by a separate external peer review. In this case, the formal record of the internal peer r eview should be included in the peer review record, however, the external peer review will genera lly stand as the peer review of greater sign ificance since it is viewed more in dependent in natu re, has broader fields of availab le expertise which ca n be brought to be ar on the issues an d often includes g reater depth fo r specific disciplines. Peer Review Leaders should ensure that any applicable EPA record keeping regulations ar e followed (for d etails, see EPA’s National Records Management Pro gram; http://www.epa.g ov/records/ ).
2.6 Budget Planning
2.6.1 What Budget ary Fact ors Should I Consider i n a Peer R eview? Resources needed to implement the Peer Review Policy need to be requested throug h the usual Agency budgetary processes. The budget formulation process within the Executive Branch is followed, after appropriation bills are passed by Congress, by budget execution. These two processes provide opportunities to secure resources for activities Peer review is par t of the carried out by Headquarters a nd Regional offi ces, normal cost of doing including peer review. The major work products for which business. decisions for pee r review have be en made (List B ( Candidate Products for Future Peer Review) candidates) need to have adequate funding for peer review in budget re quests for the coming fiscal year. Similarly, adequate funding needs to appear in the actual approve d operating budget to ensure their conduct. For purposes of budget planning, the costs of peer review would include the F TE cost of staff, the contract or other costs associated with the use of outside peer reviewers, and the administrative costs of conducting a review (copying, tra vel expenses, etc).
2.6.2 What Input is Needed for the Annual Budget Formulation and Budget Execution Proce ss? Senior Management in Office and Regions (including Decision Makers and budg et officers) need to be sure that budget requests include anticipated resources for peer review. Peer review needs to be considered as a normal part of doing business. Peer review resour cePeer Review Handb ook Page 49 considerations should also be addressed in the analytic blueprint for Agenc y rule-making actions.
2.7 Legal Consideratio ns 2.7.1 Are There Legal Ramifications From the Peer Review Policy?
The Peer Review Po licy does not establish or affe ct legal right s or obligations. R ather, it confirms the importance of peer review where appropriate, outlines relevant principles, a nd identifies factors Agency staff should consider in implementing the policy

. Except where provided otherwise by law, peer review is not a formal part of or substitute for notice and comment rulemaking or adjudicative procedures. EPA’s decision to conduct peer review in any particular case is wholly within the Agency’s discretion. Similarly, nothing in the Policy creates a legal requir ement that EPA res pond to peer revi ewers. Howeve r, to the extent that E PA decisions rely on scientific and technical work products that have been subjec ted to peer review, the remarks of peer reviewers should be included in the record for that decision.
2.7.2 Is Legal Advice Nee ded? AA/RA staff and management should work regularly with individual OGC/Regional Counsel (RC) staff assigned to Ag ency activ ities. Peer Review Leaders sho uld initially c onsult with their customary OGC/RC advisors for legal advice or referr al. Headquarters attorneys have specialties in specific areas and can be consulted as needed (e.g., F ACA considerations (see below); contractual responsibilities; ethics and potential conflicts of interest).
2.8 Federal Advisor y Committee Act (FACA) Considerations The Federal Advisory Committee Act, 5 U.S.C. App. 2, imposes certain open meeting (public announcement in the Federal Register), balanced membership, and charte ring requirements (with the approval of the General Services Administration (GSA)) before the Agency establishes, controls or manages an “ advisory committee” for advice or recommendations. Peer review carried out by formal and e stablished (chartered) Federal advisory co mmittees, such as th e Science Advisor y Board (SA B) or the FI FRA Scientific Advisory Pan el (SAP), is always subject to FACA requirements. However, FACA does not apply to eve ry EPA and c ontractor-run p eer reviews. In this section you will find information on the applicability of FACA to EPA- or contractor-run peer reviews.Page 50 Peer Review Handb ook
2.8.1 When Do FACA Requi rements Apply to EPA-Run Peer Reviews? In most cases, Federal Advisory Committee Act (F ACA) requirements apply to EPA-run peer reviews that are conducted by formal and e stablished (chartered) Federal advisory committees, such as the Science Advisory Board (SAB) or the F IFRA Scientific Advisory Panel (SAP). These gr oups have the follo wing characte ristics: a) the group is established, controlled, or managed by EPA; b) the group has a fixed membership, established purpose, and a set agenda; and c) the group strives to produce collegial, rather than, individual advice to EPA EPA run peer reviews that are not originally intended to be subject to F ACA, but which exhibit the above characteristics, may unintentionally bec ome subject to FACA. Questions concerning the ap plicability of FACA to peer review mee tings should be addre ssed to the FACA experts in the Cross-Cutting Issues Law Office of OGC (Mail Code 2322A at Headquarters), or the appropriate Offi ce of Regional Coun sel.
2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? If EPA conducts a peer review with the purpose of obtaining advice f rom the individual peer reviewer s and not for the pu rpose of obtaining a peer review p roduct from the g roup (as a collective or consensus body), the peer review would, in most cases, not be subject to F ACA. Peer review participants provide only their own views or recommendations and do not vote nor do they provide collective or consensus recommendations to EPA. When referring to the recommendations of the individual r eviewers, EPA sho uld not characte rize these recommendations using terms such as “collective” or “consensus.” As a genera l matter, workshops and “letter reviews” that seek individual views or comments are usually not subject to the requirements of FACA.
In addition to e nsuring that pee r reviewers only provide comments a s individuals, EPA officials may wish to lessen the potential for a challenge under the F ederal Advisory Committee Act (FACA) by seeking balanced participa tion at peer review meetings, and allowing interested members of the public to attend, and ensuring that they have acce ss to appropriate materials. Non-FACA meetings may be announced in the Fede ral Register (providing that it is clear in the notice that such meetings are not subject to FACA) as it provides the public with useful information and a point of contact c oncerning the p eer review. I n addition, non-FA CA (as well as FACA) meetings should also be advertised via other avenues (e.g., the Web, local newspapers, and mailing lists).Peer Review Handb ook Page 51
2.8.3 How Do I En sure that a Contractor -Run Peer Rev iew Does No t Become Subject to FACA? Committees (or other peer review groups) established, controlled or managed by an outside organiza tion (such as by an EPA contracto r) to provide that outside organiza tion with advice and rec ommendations (tha t will be submitted ev entually to E PA as a contracto r report) are not subject to FACA. Although F ACA should not apply to contractor-r un peer reviews, there are things that you (i.e., EPA) can inadvertently do that may invoke FACA. The following ar e considerations that you shou ld be aware of w hen a contracto r runs a peer review (e.g., letter review, panel, workshop, etc.) for EPA: a) the outside party ’s peer review could be subject to FACA if EPA establ ishes, manages or controls the peer review group (e.g., EPA selec ts the members of the panel, runs the meeting, etc.). b) EPA should not provide contractors with a draft agenda or suggested format for peer review me etings. EPA contr actors should mana ge and control t he process, including running any meetings. c) At the request of the EPA contractor, EPA can provide a briefing to the peer reviewers (e.g., in a conference call with the contractor on the line) on the history or background of the developmen t of the peer rev iew document. EPA o nly should provide technical or background information and not use the call to take over the contractor’s peer review group. Not only should the contractor be “on the line” but it shou ld be very c lear to all partic ipants that the con tractor is in charge of the call. The contractor, not EPA, should invite people to participate, make all administrative arrangements, conduct the meeting, and control the agenda. d) EPA employees may attend the peer review panel meetings or workshops. However, they may not take ove r the control of the meeting. The contractor should call on them t o speak when appr opriate, but they should limit their participation to providing technical and/or background information, and not attempt to , or appear t o, take over the contrac tor’s meeti ng. e) Since FACA does no t apply when a contractor es tablishes, contro ls, or manages a peer review, the contractor need not avoid terms such as “collective” or “consensus” when reporting agreement among its peer re viewers (subcontractors).Page 52 Peer Review Handb ook f) EPA may provide comments to the contractor on the contractor’s peer re view only to the extent that the Agency is verify ing that the contractor has completed satisfactorily the report in accordance with the work assig nment. EPA should not attempt to make cha nges in the contr actor’s conclus ions; this would compr omise the independence of the peer review conducted by the contractor.Peer Review Handb ook Page 53 

### 3. CONDUCTING A PEER REV IEW

3.1 Overview Statem ent The success and usefulness of any peer review de pends on the quality of the draft work product submitted for peer review, the care given to the statement of the issues or "char ge," the match between the peer review draft product and the form of peer re view, the match between the peer review dr aft product and th e scientific/tec hnical expertise o f the reviewers, and Agency use of peer review comments in the final product. It is not simply enoug h to conduct a peer review; each of the foregoing elements requires serious attention.
3.2 Charge to the Peer Reviewers
3.2.1 What is a Charge? As part of each peer review, the Peer Review Lea der must formulate a clear, focused chargethat identifies re cognized issues a nd invites comments o r assistance. Thi s request signal s the Agency's awareness of pote ntial issues and its receptivity to expert recommendations. The charge to peer reviewers usually make s two general requests. First, it focuses the review by presenting specific questions and concerns that the Agen cy ex pect s th e rev iew ers to a ddr ess
. Se con dly, it invites general comments on the entire work The time spent preparing a good product. The spec ific and gener al comments should charge is well spent, and is crucial focus mostly on the scientific and technical merits of for an effective peer review. the work product and, where germane, whether the scientific/technical studies have been applied in a sound manner. Remember, the peer review is not for the decision or action itself, but for the underlying scientific and/or technica l work product. Focused questions greatly simplify the task of collating, analyzing and sy nthesizing peer review comments on a topic al basis. The ques tions should be spec ific enough to g et helpful commen ts, but not so specific (unless very specific points are needed to be addre ssed) that they preclude creative respo nses. Moreover, the written respo nses to these quest ions by peer reviewers help the Agency create a peer review record. As a general rule, the time drafting a g ood charge letter is well-spent and is necessary for an effective pee r review.
3.2.2 What are the Essential Elements of a Charge? a) Brief overview or introduction (describe what the work product is, how it was developed, how it will be used)Page 54 Peer Review Handb ook b) As needed, a brie f description or li sting of any background mat erials provided t o the peer reviewers c) The issues or ques tions to be address ed by the pe er reviewer(s ) 1) The due date of r eviewers’ comme nts 2) The format of reviewer responses 3) The point of contact in case peer reviewers have questions
3.2.3 Where Can I Get an Example of a Charge? Appendix C (Examples of Charges) contains examples of successful charges that cover a variety of issues. Appendix D (Guidance on Requesting a Review by the Science Advisory Board) provide s guidance for obtaining Scienc e Advisory Board (SAB) ser vices.
3.2.4 Can a Stakeholder P rovide Input to the Charge to the Pe er Reviewers?
Yes. EPA may decide to obtain st akeholder input o n the charge to the peer review ers, but EPA must make the final determination on what elements to include in the charge to ensure that it meets the needs of the EPA peer review. a) If you o btain stakeholder input, you must include any and all intereste d parties to the extent feasible based upon statuto ry, regu latory, budg etary and/ or time constraints. Do no t limit input to one stak eholder only (e.g., a respo nsible party or environmental group). b) If EPA has hir ed a contractor to perform the pe er review, it shou ld still be EPA personnel who obta in stakeholder inp ut and provide the list of charge q uestions to the contractor. c) If you form a committee of stakeholders to help deve lop the charge, be aware that your committee may become subject to the re quirements of FACA (see Section 2.8).
3.2.5 Who Writes the Charge When I Hire a Contractor to Conduct the Peer Review? In general, if EPA hires a contractor to perform the pee r review, EPA must allow the contractor independence in conducting the review. However, with reg ard to the charge, EPA canPeer Review Handb ook Page 55 and should provide the contractor with the substantive list of questions that EPA wants included in the charge letter to en sure that t he peer revie w meets EPA ’s needs. Th e list of ch arge questions can be in corporated into t he Statement of Wor k if the EPA projec t manager has th e list ready at that time. Based on this list, the contractor would then prepare and submit the ac tual charge letter to the peer reviewers. Prior to submitting the charge le tter to the peer reviewers, the contractor shoul d give EPA an oppo rtunity to re view the charg e letter to ensure that it meets EPA’s needs. EPA c annot submit the cha rge directly to the peer revie wers. 3.2.6 Is it Okay For Me to Ask a Contractor to Develop the Charge to the Peer Reviewers?
No. EPA will provide the charge questions to the contractor as discussed above. The contractor, though, may provide assistance and advice in the de velopment of the charge.
3.3 Time Line
3.3.1 What are t he Facto rs in Schedu ling a Pee r Review? The peer revie w schedule is a cr itical feature o f the process. Th e schedule must ta ke into account the ava ilability of a quality dra ft work product, a vailability o f appropriate e xperts, time available for using peer review comments, deadlines for the final work product, and log istical aspects of the peer review (e.g., contracting procedur es). The schedule for peer review should take into account the overall rulemaking (or other decision making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of the peer review should be included in the development of the analytic blueprint. Peer review sometimes leads to new information and analyses, or recommendations for new rese arch that would alter the work product and thus modify the scientific/technical basis for the action. For this reason, it is usually advisable to complete the peer review before taking public comment, or at least before the close of the public comment period.
3.4 Selection of Peer Reviewers
3.4.1 What are Considera tions for Selecting P eer Reviewers? Selection of independent peer reviewers is not a trivial task, and it is crucial to an effective peer review. It is important that peer reviewer s be selected for independence and scientific/technical expertise. Therefore, EPA should always make ever y effort to use peer reviewers who do not have any real or perce ived bias or conflict of interest and who are completely independent. However, the very need to have experienced individuals on a peer review, along with the desire to have appropriate technical balance and re presentation, can meanPage 56 Peer Review Handb ook that the selection of potential peer reviewers often comes from those who are considered as having a potenti al bias. To reduc e the concern th at a potential pan el may have unnecessary bias, it may be useful to obtain an informal review of the expertise and balance of potential peer reviewers from others in your organization, from OGC or even f rom outside groups. Sometimes selecting individuals who have served in a variety of org anizations rather than a single one for an extended period, provides expertise with diverse perspective. The emphasis on independence and expertise appl ies equally to government e xperts and experts f rom the larger scientific com mun ity. Some peer review s can be conducte d with two or three reviewers; othe rs involve panels of peer review ers. In eithe r case, each pe er reviewer sh ould have recog nized technical e xpertise that bears on the s ubject matter und er discussion. The peer reviewer s of a work produc t should represent a balanced range of technically legitimate points of view. In addition, cultural diversity and "address" (e.g., industrial, aca demic, or environmental community) are other factors that ca n play a role in selecting pee r reviewers.
3.4.2 Where Do I Find P eer Reviewers? Recommendations for potential peer reviewers can be identified from a number of organizations. T hese include exter nal groups such a s the affected p arty(ies), special intere st groups, public in terest groups, e nvironmental gr oups, professiona l societies, trade or business associations, sta te organizations or agencies, N ative American T ribes, college s and universities, the National Rese arch Council, and other Federal agencies with a n involvement in or f amiliarity with the issue. Agency associated groups include the staff o f the Science Advisory Board (SAB) or the Scientific Advisory Panel (SAP), and relevant scientific and technical experts from Program or Reg ional offices. In certain circumstances, existing peer review orga nizations such as the SAB or SAP may be used to conduct a peer review. These groups establish their own criteria for a ccepting work and coordin ation must be made dire ctly with them (see Append ix D - Guid ance on Requ esting a Review by the Science Advisory Board) for SAB procedures. Both SAB and SAP conduct formal, public, e xternal peer rev iews. Occasionally, a member of the scientific community will offer his/her services for peer review during an ongoing peer review. These offe rs may be at no cost or based on an expectation that reimbursement will be made. Disposition of these unsolicited offers will be handled on a case by case basis by the Peer Review Leader, and as necessary , in consultation with the Peer Revie w Coordinator, the Office of Gene ral Counsel (OGC), and appropriat e Decision Makers.Peer Review Handb ook Page 57 If you u se a contract me chanism to conduc t a peer review , the contractor will have its own pool of scientific and technical experts for peer review. With contractors, EPA can provide information on potential sources of peer reviewers for conducting a peer re view if such a listing is prepared in alphabetical order. In no way should EPA insist that the contractor select from EPA’s list, or require that the contractor receive EPA approval before selecting any peer reviewer (sometimes known as a “subcontractor”). However, EPA should review the list of reviewers for conformance to work assignment specifications and adhere nce to conflict of interest concerns. Furthermore, when utilizing a contract mechanism to conduct peer review, EPA is not permitted to direct the prime contractor to a specific subcontractor (or peer reviewer) nor is EPA permitted to direct the peer reviewer (subcontractor). All interactions with the peer reviewers must be coordinated through the prime contractor. (See Section 3.6 for further information) Keep in mind that fo r contracting p urposes, contra ctors are requi red to obtain Contracting Officer (CO) approval of subcontractors and the CO gener ally seeks the input of the work assignment manager (WAM -- the WAM may b e the Peer Review Leader in ma ny cases) before approving the use of subcontractors. In this case, as noted a bove, the Peer Review Leader should review the list of potential reviewers for conformance to the work assignment specifications a nd adherence to conflict of inter est concerns.
3.4.3 Are External or I nternal Peer R eviewers Prefe rred? External peer reviewers are generally preferred, particularly for most final work products. For some work products, like those reviewed at interim steps, either external or internal peer review may be appropriate. Selection of internal peer r eviewers should be based upon technical expertise, available time and "address" -- that is, they should not come from the immediate of fice or group producing the product or have any othe r connection with the product or document being peer reviewed. External peer reviewers should be selected based upon technical e xpertise as well, however, care must be taken not to use individuals who have been involved in the development of the work product. (See Section 1.2.6; see also Sections 1.4.6 to 1.4.9).
3.4.4 What is Important in the Mix of a Peer Review Panel? A peer review panel or group can number from just a few individuals to ten or more, depending on the issue, the time and resources ava ilable, and the br oad spectrum of e xpertise required to treat the range of issues/questions in the charge. Objective technical e xpertise and lack of a conflic t of interest are critical in selec ting peer revie wers. Naturally , experts whose understanding o f the specific te chnical area( s) being evalua ted are necessa ry; nevert heless, it is also important to inc lude a broad enou gh spectrum of o ther related expe rts to completely evaluate the relevant impacts on other less obvious concerns (i.e., to comment not only if the job is being done right, but also whether the right job is being done). For example, for health relate dPage 58 Peer Review Handb ook peer reviews, experts in such fields as ecology and economics ma y provide very useful insights. Although persons who are familiar and have a substantial reputation in the field are often called upon repeatedly to be reviewers, it is important to keep a balance with new people who bring fresh perspect ives to the review of a work produc t. There is usually a continuum of views on any issue. To the e xtent possible or practicable, selected experts should have technically legitimate points of views that fa ll along the continuum. A review panel should include experts that are considered “mainstream” (nearer the center of the continuum) as well as those further along the continuum (while generally avoiding the extremes). This will help maintain a balanced review panel, while allowing all views to be expressed and discussed. A balanced panel will allow consensus building (if consensus is the object of a parti cular peer rev iew; if not, it allow s a spectrum of (r e)views for the A gency to evaluate). As a general rule, experts who have made public pronouncements on an issue (e.g ., those who have clearly "taken sides") may have difficulty in being objective and should be avoided.
3.4.5 What is a Conflict of Interest? The matter of obtaining a fair and credible peer review, as well as maintaining the credibility of the Agency and the Agency’s scientific products, is of paramount importance

. Peer review leaders are strongly encoura ged to obtain peer reviewers who do not have a legal or perceived conf lict of interest (i.e., creates the appearance th at the peer revi ewer lacks impar tiality or objectivity). In reality , we recognize that a totally independent peer reviewer is rare and very difficult to identify (see Section 3.4.6 for steps to take to ensure a credible peer review). Conflict of interest is a situation in which, because of other activities or relationships with other persons, an individual is unable or potentially unable to render impartial assistance or advice to the Ag ency, or th e person’s objec tivity in perf orming the work i s or might be other wise impaired, or a person has an unfair competitive advantage. Generally , a conflict of interest of interest arises when the person is affected by his/her private inter ests, when he/she or his/her associates would derive benefit f rom incorporatio n of their point of v iew in an Agenc y product, or when their professional standing and status or the significance of their principal area of work might be affected by the outcome of the peer re view. Clearly, peer reviewers should not be placed in the position of reviewing their own research and analy ses that form the basis of the work product under review as this might impair their objectivity. Whenever there a re questions about conflicts of interest, you should contact the appropriate official in OGC for clar ification. Be aware that COI can be used in a gene ric fashion in speaking of common, everyday experience that also involves elements of ethics. In most cases, that is how it is treated in the Peer Review Handbook. However, COI is a very specific issue when used in a governmentalPeer Review Handb ook Page 59 contracting sense. In the contracting sense, the Fe deral Acquisition Regulations (FAR), the Environmental Protection Agency Acquisition Regulations (EPAAR), and other internal Ag ency documents define COI and describe the Agency ’s COI policies and procedures. The Handbook provides guidan ce in some of these aspects relate d to peer review (see Section 3.6 a nd Appendix E - Example Statements of Work for Contracts), however, specific questions should be addressed to the O ffice of Acquisi tion Managemen t.
3.4.6 What Techniques Help Ensure Disclosure and Appropriate Resolution of Conflicts of Interest?
Before finalizi ng the selection of reviewers, th e Peer Review L eader should asc ertain whether each potential peer reviewer’s involvement in certain activities could pose a conflict of interest (COI) or create the appearance that the pe er reviewer lacks impartiality. One way of identifying conflicts is to ask p otential reviewe rs about current and prior work, a nd prior clients that might create conflicts or the appearance of a lack of impartiality in carrying out peer review activities. This information obtained by the Peer Review Leader becomes part of the peer review record. When the peer review process is being conducted by a contractor, the requirement for addressing peer reviewers’ possible conflicts of interest should be highlighted in the Statement of Work of the work ordering instrument (e.g., Work Assignment, Delivery Order, Task Order, etc.) and is a matter that is bound by contractual clauses with the Contracting Of ficer as the final Decision Maker in contracting ma tters. Care must be taken to reduce actual or potential organizational or personal conflicts of interest between the reviewers and the work product under review. Remember, ea ch potential conflict situation is unique and must be treated on a case-by-case basis. The following are consi derat ions t hat sh ould b e addr essed in eva luati ng COI: a) attention to the employment, financial, and professional affiliations of the participants; b) exploring directly the issue with each of the participants before the re view process takes place; c) disclosing public ly at the beg inning of meeting s any previo us involvement with the issue; d) in the cases of regular government employe es and Special Government Employees (SGEs), filing annual (or updated, as appropriate) Confide ntial Financial Disclosure Forms (OGE Form 450), discussing any po tential conflicts of interest with their designated ethics official (DEO), and advising the Peer Review Lead er of any r elevant concer ns;Page 60 Peer Review Handb ook e) in the case of non-Federal peer reviewers, provide them a copy of the peer review COI inquiry (see Appendix F - Useful Forms). This form is sent to ea ch prospective peer reviewer by the Peer Review L eader (or contractor, in the case of contractor-ru n reviews) to adv ise them of the nee d to address COI issues prior to the actual review taking place. A follow-up contact with the Peer Review L eader (or contractor, in the case of contractor-run reviews) is then made to discuss any relevant issues. The Peer Review Leader then documents this effort in the peer review record; this includes a summary provided by the contractor documenting their inquiries an d efforts.
The Peer Review L eader needs to e nsure that the pee r review COI inquiry (se e Appendix F - Useful Forms) took place and this appears in the peer review record. You should e nsure that any documentation in the peer review record does not contain informa tion subject to privacy laws (for example, see The Privacy Act of 1974, 5 U.S.C. § 552a - Records maintained on individuals; a copy can be found at: http://www.usdoj.g ov/04foia/privsta t.htm). Peer Review Leaders should ensure that any applica ble EPA record keeping regulations are followed (for details, see EPA’s National Record s Management Pr ogram; http://www.epa.g ov/records/ ). For information on Freedom of Information Act (FOI A) requests, please see EPA’s Ethics Advisory 88-12 (http://intranet.e pa.gov/ogcr mo01/ethics/88-12.htm). If y ou have any questions, be sure to ask your appropriate Office of Genera l Counsel and/or office contract official(s). Established peer review groups such as the Science Advisory Board provide useful models for addressing balance and COI issues. Assistance in identify ing conflicts of interest and in providing an appropriate response can be obtained from the Office of the Gene ral Counsel. Additional advice can be obtained from one of the Agency’s designated ethics officials (DEOs). Assistance in evaluating conflicts of interest in a contractual situation require the involvement of the Contracting Officer and the resources available within the Office of Ac quisition Management. Of course, conflicts do not necessarily arise merely because a peer reviewer knows something about the subject matter. In fact, experts with a stake in the outcom e-- and therefore a potential confl ict -- may be some of the most know ledgeable and up-to-date exper ts because they have concrete reasons to maintain their expertise. Such experts could be used provided the potential conflic ts of interest are disclosed and the p eer review pan el or group bein g used as a whole is balance d. In some cas es, however, the conflict may be so direct and su bstantial as to rule out a particular expert, for instance, a potential peer reviewer who may have a client or employer with a direct financial stake in the particular specific pa rty matter under review (e.g., i) a Federal grant or contract to the potential peer reviewer or his/her employer that relates to the matter under re view; ii) the poten tial peer review er’s or their co mpany’s wor k on a specific chemical under review). However, review of a gener al methodology that applies to numerousPeer Review Handb ook Page 61 chemicals would not necessarily raise such a concern. ( Note : COI language should be made part of contracts/statements of work (SOW) or purchase orders (PO) -- see Section 3.6.) A Peer Review Leader may a lso want to adopt measures that will prevent creation of conflicts as the peer review is underway. Any measures contemplated that involve a contractual action must be coordinated with the cognizant Contracting Officer. Some measures might include clauses in a contract or purchase order that require reviewers to re ceive advance approval on future work, or place limits on such work, while they are performing the current peer review. Note that at some level these types of measures will discourage experts from se rving as peer reviewers. (See Section 3.6.5 for further information dealing with contracts and sugg estions for appropriate management controls.)
3.4.7 Can Parties Exte rnal to EPA Pay fo r Their Own Peer R eviews? There may be instances where parties external to EPA will want to conduct and/or pa y for a peer review on a particular work product (presumably their own work product or one they are closely interested in, or they wouldn’t be interested in expending resources). This may look benign at first blush, but is a very complex and sensitive situation that can raise significant concerns for perceived and/or actual conflicts of interest for interested parties “ paying” for a peer review of their own work product. While the Agency cannot pr event external parties from conducting and paying for a peer r eview, it is desirable that any such peer review meets the intent of the Age ncy’s Peer Review Policy and adheres to th e principles and g uidance in this Handbook. If the external party submits their work product and ac companying peer review, the materials will be treated by the Agency as anything else submitted for the Agency ’s evaluation. We will evaluate the work product and the peer review for scientific credibility and validity bef ore making any decisions based o n the materials. [This issue is under active Agency consideration at this time; therefore, users of this Handbook should be aware that the language in Section 3.4.7 will likely change in the future.]
3.4.8 Are There Constr aints to Selecting P eer Reviewers? Sometimes the need for a peer review is accelerated due to a court-order ed deadline or other time-sensit ive requirement s. In such cas es, it is difficult, i f not impossible to obt ain external peer re viewers in time to c onduct a full exter nal peer review

. It may e ven be impossible to conduct a small scale internal peer review using just a few individuals. Mechanisms for identifying and using a small number of peer reviewe rs should be developed so that quick, effective pee r review can be included for eve n the most rapidly moving products.Page 62 Peer Review Handb ook Another possible constraint involves confidential business information (CBI). To evaluate certa in Agency -generated s tudies properly , some peer revie wers may ne ed access to CBI. However, unless the reviewers are Fede ral employees with CBI cleara nce, the Agency doesn’t have the independent authority to disclose CBI to them. Ther efore, whenever contemplating the use of outside peer reviewers, Agency staff should determine whether the reviewers will need access to CBI. If they don’t have CBI cleara nce, the Office of the General Counsel should be co nsulted on whether it is practical to o btain the consent o f CBI submitte rs to disclose the infor mation to peer re viewers. Offices need to be aware of the requirements of the Federal Advisory Committee Act (FACA) when establishing peer review mechanisms (see Section 2.8). Federa l advisory committees that are subject to chartering by the Genera l Services Administration must hold meetings that are open to the public, and have balanced membership requirements. The Office of the General Counsel should be consulted regarding the applicability of FACA to peer review panels.
3.4.9 If State Employee s are Used as Pee r Reviewers, Can EP A Pay Them for th is Service? Maybe. Fir st, the Peer Revie w Leader ne eds to determine: a ) if the State ag ency has a policy on whether their employees c an perform this type of work for EPA, and b) if the person will be acting as an individual or as an employee of the State. I f the State allows t he person to be a peer revie wer for EPA and th e person is acting as an individual, w e can pay t hem for their peer review services. If the State allows the person to be a pee r reviewer and the person will be acting as a State employee, we can only pay them if State policy allows us to. The most efficient vehicle for paying the State employ ee will likely be a contract or purchase order (se e Sections 3.6.6 & 3.6.7), bu t you should c onsult with OGC and th e procurement of fice for advice

. In most instances, EPA can also pay travel expenses (consult with y our administrative staff for details). If we don’t pay a State person for the ir peer review services, they must sign an ag reement stating that they don’t expect payment (See Sec tion 3.6.1 - gratuitous services).
3.5 Materials for Peer Reviewers
3.5.1 What Instructions Do You Give Peer Reviewers?
The Peer Review L eader is respons ible for ensuring that peer revie wers understand their responsibilities (see Section 3.6 if a contract is involved):Peer Review Handb ook Page 63 a) Advise the Agency of actual or potential orga nizational or personal conflicts of interest or other matters that would create the appe arance of a lac k of impartiality (e.g., see Section 3.4.6) b) Provide written comments in specified format by the specified deadline that are responsiv e to the char ge c) Comply with an y requests f or not disclosing d raft work produc ts to the public
3.5.2 What Materials Shoul d be Sent to Peer Reviewers? For a peer review to be successful, peer reviewers should rece ive several documents at the beginning of the process. Typically , the most important among these documents are the charge letter and the current work product. The charg e letter describes what the peer reviewers are being aske d to do, and should se rve to focus and s tructure the re view. The work pr oduct is, of course, the material being subject to peer review. Remember, no documents should be provided directly to a potential peer reviewer w hen that reviewer is going to be wor king under a con tract or purcha se order. In the case of a co ntract, the Agency provides the work product with associated bac kground material to be peer reviewed to the prime contractor who in turn distributes these documents to the peer reviewers. In the case of a purch ase order, the “ charge or sta tement of work” mu st be part of the PO (purchase order) and the provision of any documents needs to be coordinated with the purc hasing agent handling the order. a) Essential documen tation for each p eer reviewer includes: 1) A current copy of the work product to be peer revie wed with associated background mat erial. The work p roduct needs to be of the best possible scientific/technical quality to ensure an adequate and useful pee r review. 2) A clear charge or statement of work seeking informed comment on identified issues to properly focus the efforts of the peer reviewe rs and ensure that their individual efforts can be compared or contrasted. 3) Some information concerning the process that you use for the peer review, including the due date of reviewe r comments, the fo rmat of those responses, and a point of contact in case the peer reviewer has questions. Responses should be written and submitted to the Peer Review Leader by an agreed upon deadline. In certain rare cases, oral commentary may bePage 64 Peer Review Handb ook
sufficient. However, in such cases, a follow-up written response for the record is required. 4) In some case s, Agency materials being peer reviewed will be available to the public, even if they are ma rked as drafts. F or example, all mate rials reviewed by the SAB are available. Ag ency managers may also decide that a broad acc essibility ha s benefits for the Agency. In other cas es, confidentiality needs to be maintained. In these case s, each peer reviewer must be informed of the need for con fidentiality with regard to th e release of Agency products that are stamped as "D RAFT" or "DRAFT - Do Not Cite, Quote, or Rele ase." Prematur e release of dr aft Agency products, views, or positions i s inappropriate a nd can be damag ing to the credib ility of the Agency or the peer reviewe r. While not necessarily having legal consequences, such language will be included in the charge to the pe er reviewers. Othe r mechanisms to use in discouraging premature rele ase include a disclaimer that appears in a separate section at the front of the document and creating the document with watermarks clearly delineating DRAFT status (or a header or footer that states DRAFT status) on every page. In addition, in any solicitation for peer r eviewers, the necessity for confidentiality and the non-release of materials shall be emphasized. b) Useful, but not cr itical materials t hat may be se nt to peer review ers include: 1) The name, address, and phone and fax numbers, and/or Internet address of each peer reviewer working on the specific review 2) A bibliography and/or any pa rticularly relevant scientific articles from the literature 3) A work product tha t has line numberin g added in the ma rgin for ease i n providing and re ferencing co mments c) Peer Reviewers should be given what is needed to complete their task -- they should not be overb urdened with exce ss material.
3.5.3 How Closely can EPA In teract wi th Peer R eviewers During the Review? a) When EPA Conducts the Peer Review - The Peer Review Leader normally has administrative contacts with the reviewers during the development and conduct of the peer review

. In some case s (e.g., SAB pe er review), pe er reviewers ma y also receive a briefing on the product to be peer reviewed. Otherwise, the PeerPeer Review Handb ook Page 65 Review Leader and other EPA staff should not contact the reviewers during the course of the review. Such contact can lead to perceived conflicts or inappropriate direction that could compromise the independence of the review. b) When the Contractor Conducts the Peer Review - If peer review is conducted under a contrac t mechanism, EPA mus t limit direct conta ct to the prime contractor’s designated representative and not have genera l contact and direction to the contractor’s staff or peer reviewers (sub-contractors). Note, whe n a peer review is conducted under a contract, there are constraints where EPA staff a re prohibited from contacting peer reviewers to avoid personal services arrangements. Personal services contracts exist when the nature of the relationship between the contractor and the EPA can be characterized as an employer - employee re lationship. Any communications with peer reviewers must be coordinated through the prime contractor.
3.6 Peer Review Services A range of peer review services are available to the Ag ency including internal, external (gratuitous services, contracts, purchase order), and Special Government Employ ee (SGE) mechanisms. The mechanism selected is generally based on the na ture of the scientific or technical work p roduct.
3.6.1 What are G ratuitous Services for Peer Review? The provision of peer review products or services to EPA without compensation are provided as so-called “gratuitous” services. If a per son wishes to perform peer review services for EPA without compensation, the EPA must ask them to sign an agreement whereby the person agrees to provide the prescribed peer review service s as gratuitous services, with no expectation of re ceiving compen sation for these s ervices from EPA (see Appendix F fo r EPA Form 3100-14 whic h is used in situations where a gratu itous services co ntract is not used). An agreement (su ch as Form 3100-14) must be executed because the Anti deficiency Act (31 U.S.C. §1342) prohibits the Agency’s acceptance of “ voluntary” services (“voluntary services” are services provided to EPA without an agreement in advance that such services are provided a t no cost to EPA). Note that persons cannot waive compensation (i.e., agree to provide gratuitous services) for which there is a statutory right to pay ment, unless a law permits the waiver. (For situations concerning State employees, see Section 3.4.10. La ws that permit services without compensation are 5 U.S.C. §3109 for experts or consultants and 5 U.S.C. §3111 for student volunteers.)Page 66 Peer Review Handb ook
3.6.2 Can I Use a Contrac t to Obtain Peer Review Services? The Agency may contra ct for peer rev iew services. Th e contract may be written solely for peer reviews or be included as one of several specifically described interrelated tasks in a contract that re quires the contra ctor to provide mor e than just peer r eview services

. For assistance i n preparing the necessary pre-award doc uments, progra m officials should consult Chapter 2 of the Contracts Manag ement Manual (CMM)

. Chapter 2 is available on EPA’s Intran et (http://epawww.epa.gov/oamintra/policy/cmm.pdf ) and on the Agency LAN Services (Administration/ Management/OAM Procurement Policy I nformation).
3.6.3 How Do I Write a State ment of Work for Con tracts? The Statement of Work (SOW) must clearly specify that the contr actor is responsible for preparing peer review evaluations and set forth guidelines for the pee r review of scientific or technical documents. The contractor may perform the pee r review with in-house staff, subcontractors o r consultants. Any guidelines for p erforming pee r reviews to ensu re soundness and defensibility must be developed by the prog ram office and made part of the contract. The contractor woul d then ensure tha t the peer revie ws adhere to the g uidelines.
The SOW should includ e the list of questio ns that EPA wants the contractor to inc lude in the charge to the peer reviewers. While the contractor will be the one that will prepare and send the formal charge to the peer reviewers, EPA should provide the list of questions to the contractor (see Section 3.2 for general discussion on charge to peer reviewers). Within the SOW, the contractor should be directed to inquire whether prospective peer reviewers have any actual or potential orga nizational or personal conflicts of interest or other matters that would create the appearance of a lack of impartiality , including whether they have had or presently have a financial relationship with EPA. Further, the SOW needs to specif y that contractors ar e required to pro vide EPA with a summa ry of the pr ocedures and ef forts made to identify, disclose, and ensure that no independence or conflict of interest c oncerns arise during the performanc e of the contrac t. The SOW cannot simply define the role of the prime contractor as arrang ing for the services of othe rs to perform pee r reviews and log istics for meeting s. Unless the prime contractor is clearly tasked with responsibility for performing peer reviews, individual peer reviewers' f ees and associa ted travel expense s are not pay able under the c ontract. The EPA may p ay for the r eviewer's com ments or evaluatio n, and also for att endance at a meeting with the Agency and other reviewe rs to discuss the results of the peer review. If the SOW calls for the preparation of comments or an evaluation, and specifies a meeting with thePeer Review Handb ook Page 67 Agency a nd other peer re viewers to discuss the results of the p eer review, pa yment is appropriate. The peer reviewer's attendance at the meeting would then be part of contract performance. Example statements of work are found in Appendix E (Example Statements of Work for Contracts).
3.6.4 What are Advisory a nd Assistance Servic es (AAS) or Sensit ive Activities? Contracts that provide services that support or improve Agency decision-making or policy development are subject to special management controls. These services include services acquired from non-Governmental sources by contract to support or improve Agency policy development, decision-making, management, and administration, or research and development activities. See Federal Acquisition Regulation (FAR) 37.201 for a more specific definition of AAS. Such service s may take th e form of informa tion, advice, opin ions, alternative s, conclusions, recommendations, training, and direct assistance. For additional information on advisory and assistance services and sensitive activities, program of ficials should review Chapter 2 of the CMM. New contracts for these services require management approvals prior to issuance of the solicitation. For the thresholds tha t have been esta blished for appro val of these justif ications, see Figure 5, Item B Management App rovals set forth after Chapter 2 of the CMM (for current approval levels).
3.6.5 What are Some M anagement Contr ols for Contracts? Contracting for peer review services is permitted. However, because of the potential for improper use of these contracts, special management controls are required. a) Inherently Gover nmental Functio ns (IGFs) – OFPP Policy Letter 92-1, dated September 23, 1992, describes (1) functions that are inherently gove rnmental and must be performed only by Government e mployees and (2) functions that may be contracted, but so closely support Government employ ees in their performance of IGFs that the contract terms a nd performance require close sc rutiny by Federal officials. Fede ral Acquisition Reg ulation (FAR) cov erage of inhe rently governmental functions is at FAR Subpart 7.5. Peer reviews represent only a contractor's recommendations, advice or analysis of a document. Agency officials must make the official Ag ency decision regarding acceptability and/or quality of the docume nt. To ensure that Agency officials are not improperly influenced by rec ommendations in the peer review, managementPage 68 Peer Review Handbook
controls must be inc luded in the contra ct. One possible c ontrol would be to require the pee r reviewers to su bmit with their eval uations or comment s a description of the procedures used to arrive at their recommendations; a summary of their findings; a list of sources relied upon; and make clear and substantiate the methods and considerations upon which their recommendations are based. To the extent possible, the contract should set forth any guidelines or criteria for performance of the peer review

. Agency officials should d ocument their evaluations of the quality and validity of the peer review. b) Conflict of Interest (COI)-- Another important factor is that the objectivity of the peer review should not be improperly influenced or undermined by the contractor performing the review. To identify and a void or mitigate actual or potential COI, the contract should include controls. Such controls might require the contractor to report on prior and current work, and prior clients that might create COI. Other controls might include Agency review and placing limits or advance approval on future work. There should also be procedures implemented to assure that the contractor does not gain an unfair advantage in future requirements as a result of their p erformance of peer reviews. Pr ogram officia ls should consult the Co ntracting Offi cer (CO) for spe cial contract c lauses. FAR coverage of conflicts of int erest is at FAR Subp art 9.5. The EPA Acquisition Regulations (EPAAR) at 48 CFR Subpart 1509.5 generally mandates conflicts of interest solicitation provisions and contract clauses, but makes them optional for procurements accomplished through the use of simplified acquisition procedures as set forth in FAR Part 13. Also see 48 CFR 1509.507-1(b)(3) and 48 CFR 1552.209-70, -71 & -72 as addi tional resource s. Contract for peer review services: An EPA contracting officer will include conflicts of interest solicitation provisions and contract clauses as a matter of course without involvement by the EPA project officer, if the peer review services are no t obtained pursuan t to the simplified ac quisition procedur es in FAR Part 13. I f the peer revie w services are subcontracted p ursuant to a prime contract, then the prime contractor is ordinarily required to include a c onflicts of interest clause substantially similar to the conflicts of interest clause in the primary contract in its subcontract to the peer reviewer. Although the EPA contracting officer and/or prime contractor has the primary responsibility to include the required conflicts of interest provisions/clauses, the EPA project officer may nevertheless wish to:Peer Review Handb ook Page 69 1) Highlight the conflict of interest requirements in the Statement of Work (SOW) for the procurement of the peer review services 2) Develop a specific conflict of interest clause regarding the pee r review at issue as a substitute or in addition to the standard conflic ts of interest clause 3) Review the solicitation/contract to make sure that the required conflicts of interest clause has been included 4) Ensure that the SOW includes a requirement for the contractor to provide EPA a summary of the work performed to identify and resolve conflict of interest/independence concerns with peer review candidates Simplified acquisitions of peer review services : Although conflic t of interests requirements are optional for simplified acquisitions, they are nevertheless a g ood idea. According ly, an EPA pr oject officer o btaining peer r eview services with simplified acquisition procedures (see Section 3.6.7) should request the purchasing agent/contracting officer to include a conflict of intere st solicitation provision and contract clause in the purchase order. c) Confidential Business Information (CBI)/Privacy Act Protected Information and Other Sensitive Information-- When peer reviewers are not employees of the United States G overnment, it is unli kely that the EPA will have autho rity to give reviewers access to confidential business information in the absence of consent for such disclosure by the CBI submitt er. Therefore , all documents provided to non-Federal reviewers must be screened for information claimed as CBI. Even wh ere business info rmation has not bee n explicitly cl aimed as CBI , if it is of a kind where the submitter might be expected to object to its release, prior to release the su bmitter must be aske d whether it wishe s to assert a claim , unless the submitter has previously been informed that failure to assert a CBI claim may result in disclosur e without notice. L anguage is inc luded in the contra ct to clearly identify any required proc edures or processes prior to release of any prote cted information, including any requirements for confidentiality agreements, as well as limits on use and disc losure of the data by contrac tor personnel. d) Personal services-- Under contracts, the EPA may not engag e the peer reviewers in any improper personal services relationships, i.e., an arrang ement under which contractor personnel are subject to relatively continuous supervision and direc t control by an Agency official or employee. These relationships are chara cterizedPage 70 Peer Review Handb ook as one where the contractor employee interac ts with the Agency in a manner similar to that of a Federal employee. To avoid these impr oper relationshi ps, program off icials should writewelldefined SOWs. The SO Ws should set forth the requirements in d etail for work to be performed independently, including the manner in which it will be evaluate d. The SOW must set forth what work is to be performed not how the work is to be performed. Technical direction may be used to clarify ambiguous technical requirements to ensure efficient and effective contractor performanc e, and is not considered supervision or assignment of tasks. For additional information, program offic ials should consult EPA Order 1901.1A, Use of Contractor Services to Avoid Improper Contracting Relationships and FAR Subpart 37.1
3.6.6 Can I Identify and/or Select Peer Reviewers When Using a Contract?
No. Program off icials cannot inte rfere in a contr actor's author ity and resp onsibility to perform work by "selecting" who will perf orm the peer review (doing so may invoke FACA -- see Section 2.8). The Federal Acquisition Regulation (FAR) governs the Contracting Of ficer’s (CO's) and program officials' relationship with the contractor. Keep in mind that for contr acting purposes, contractors are required to obtain Contracting Officer (CO) approva l of subcontractors and the CO generally seeks the input of the work assignment mana ger (WAM -- the WAM may be the Peer Review Leader in many cases) before approving the use of subcontractors.
EPA can establish criter ia

for the sort of individuals that might participate on a peer review panel. However, the Agency must not be involved in the selection of individual peer reviewers, and should avoid commenting on the contractor’s selection of peer reviewer s other than to determine whether the panel, once selected, meets the criteria established. EPA may identify a p ool of qualified su bcontractors an d consultants to the prime contracto r (listed in alphabetical or der), but cannot direct the use of any partic ular subcontrac tor or consultant.
3.6.7 Can I Use Simplified A cquisition Proce dures to Obtain Pee r Reviewers? In some instances, peer reviewers can be obtained via simplified acquisition procedur es. The acquisition of supplies or non-personal services from the open market and on a sole source basis when the aggregate amount involved in any one transaction does not exceed $100,000 constitutes a simplified acquisition (FAR Part 13). The same considerations in the Section 3.6.5 discussion on IGFs, COI, access to CBI , and personal services apply to simplified acquisitions. Normally, the Government issues a purchase order directly to the individual peer reviewer, instead of to a prime contractor who may subcontract for perf ormance of the peer review.Peer Review Handb ook Page 71 a) Approvals -- All small purchases for peer reviews are considered Advisory and Assistance Services (AAS). See Chapter 2 of the Contract Management Manual (CMM) for the appr opriate approva l levels for AAS. b) Competition-- The Federal Acquisition Regulation (FAR) requires competition for purchases in excess of $2,500. Purchases for more than $2,500 and not more than $100,000 are to be made only from small businesses unless the Contracting Officer is not able to obtain two or more offers from small businesses that are competitive in terms of market price, quality, and delivery

. Only one source need be solicited if the Contracting Off icer determines that only one source is reasonably available. Contracting Officers a re encouraged to use best value. c) Procurement Requests -- Program Off icers should inclu de the following in all PRs for the purcha se of peer revie ws: 1) A fixed-price amo unt at or below the simplified acquisi tion threshold 2) A detailed description of the requested services, inclusive of: (a) Total quantity per line item (b) Estimated unit price per line item (c) Total cost per line item (d) Specific deliverables for each line item (e) Total cost of the pu rchase reques t 3) Reference FAR Subpart 3.6 and Environmental Protection Agency Acquisition Regulation (EPAAR) Subpart 1503.601 regarding sources from Government employees or organizations owned and controlled by them. 4) Provide sources from small businesses, if available. 5) If the reque st is a sole source purchase, justif ication must be pro vided in accordance with the EPAAR Subpart 1513.170-1.Page 72 Peer Review Handb ook
3.6.8 How is Travel Handled with Contracts or Pur chase Orders? Funds obligated on a contract or purchase order are available to pay for the costs of producing the peer review including the travel costs and fee of the pee r reviewer. The EPA acquires peer reviews through simplified acquisitions issued directly to peer reviewers or through contract s with companies, whi ch provide the peer review se rvices. By issuing a purchase order or awarding a contract for pee r review services, the EPA may pay not only for the peer review services/comments, but also for participation in a mee ting with the Agency a nd other reviewe rs to discuss commen ts. The scope of w ork of the contra ct must require the contractor or individual peer reviewer as appropriate to provide pee r review services and indicate whether the contractor or peer reviewer will be required to discuss a spe cific peer review work product with the Agency and/or with other pee r reviewers. Participation in a meeting to discuss a peer review work product would then be part of the contract's pe rformance. Thus, the contract may serve as the mechanism to pay for peer review services and associated travel expenses to provide comments to the EPA.
3.6.9 How is Travel Handled with Spe cial Governme nt Employees? The term Special Government Employee (SGE) is defined in 18 U.S.C. 202(a) as an officer or employee of an agenc y who performs temporary duties, with or w ithout compensation, for not more than 1 30 days in a p eriod of 365 day s, either on a full -time or intermitte nt basis. Travel and per d iem expenses of expe rts hired as SGEs f or peer review may only be paid through the issuance of invitational travel orders (5 U.S.C. §5703). These invitational travel and per diem expenses should be charged to an appropriate EPA travel account. The Fede ral Travel Regulations gov ern the invited tra velers reimburse ment. Members of the SAB, SAP, and other FACA advisory committees are often broug ht on board as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory committee member s (SGEs) through a contract.Peer Review Handb ook Page 73 

### 4. COMPLETING A PEER REVIEW

4.1 Overview Performance of the formal peer review is not the final stage in the development of the work product. Rather, it is an important stage in developing the work product, with the final work product rep resenting the tr ue end of the pee r review. As a re sult, the peer re view process closes with three major activities: evaluating comments and recommendations, utilizing peer review comm ents for com pleting th e final work product, an d organizin g and mainta ining a record of the peer review. Careful attention to all of these elements, singly and together, assure s a credible peer review process. Conversely, inattention can nullify the peer review attempt. A well-planned peer review applied to a reasonable quality starting w ork product, followed by responsible, visible utilization of peer review suggestions in the final product assures a credible product for use in Agency deci sion-maki ng. The peer review is not complete until The peer review is not complete until the the peer review c omments are peer review comments are incorporated into the incorporated into the f inal work final work product, or reasons are stated why product. such comments are not to be incorporated. However, for th e purposes of the annual report to the Agency’s Peer Review Product Tracking Database only, the work product can be moved from List B ( Candidate Products for Future Peer Review) (or List C - Products for Which a Decision has been made not to Peer Review ) in some instances) to List A (Products Peer Reviewed Since 1991) when the Decision Maker decides on how the peer review comments will be addressed and this decision is documented in the peer review record. The peer review record is completed only when it contains a c opy of the final work product (when the re is one) that ad dresses the peer review comments.
4.2 Final Work Product
4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product? The Peer Review L eader must care fully evalu ate and analy ze all peer revie w comments and recommendations. As discussed earlier, a carefully c rafted charge to the peer reviewers simplifies organizing and analyzing comments. Also, any other issues that are raised need to be identified and evaluated.Page 74 Peer Review Handb ook The validity and objectivity of the comments need to be eva luated. Analyses may include consultat ion with other exper ts/staff within the Office and/or A gency. A dequate documentation is needed to show that comments are accepted or rejected – the documentation can be brief, bu t must address the le gitimate, valid c omments. The peer review record must contain a document describing the Agency’s r esponse to the peer review comments.
The Peer Review L eader should brie f the Decision Mak er (including a ll appropriate managers in the Peer Review Leader’s chain of c ommand) on the charge, profile of peer reviewers, the peer review comments, and provide a proposal on how to address the peer r eview comments. It is the responsibility of the Peer Review L eader to obtain Decision Maker approval of the approach to addressing the peer review co mments. The Peer Re view Leader should clearly identify for the De cision Maker any major peer review comments that will not be accepted and why.
Comments that have s ignificant impac t on time, budgeta ry, and/or r esource requir ements are particular ly important a nd need to be eval uated in consultat ion with managem ent. These comments may lead to allocation of additional resources and a revised schedule for the completion of the w ork product.
4.2.2 What Acti ons are P otentiall y Forthc oming fro m Peer Review? Peer review comments and recommendations may entail significant impacts on the planned project schedule, budget, or other resource requirements. Manag ement decisions related to revisions in one or more of these areas may be appropriate. The substantive issues or concerns expressed by peer reviewers ma y suggest that wider scientific and te chnical consulta tion is needed to en sure the adequa cy of the wo rk product. The peer revie w comments and re commendations on a final product may provide a basis for bringing the associated project to closure.
4.2.3 What Should the Final Wo rk Product Say Abou t the Peer Re view Process? If the product has been peer reviewed, y ou should describe the peer review in the document. Frequently, this will be part of a description of the process of developing the product. It can be bri ef and does not ne ed to describe the process or discus s the peer-revi ew comments in great de tail. The d escripti on can be i ncluded i n an introd uction, pr eamble, o r appendi x. When there were significant peer-review comments, and particularly if they are not being accepted, the document will generally discuss the issue and descr ibe the reasons for the Agency’s choices in the appropriate sections of the doc ument. The level of detail that is neededPeer Review Handb ook Page 75 is a matter of judgment and will depend on the importance and degree of controversy of the issue. If a major scientific or technical product has not been peer-re viewed, this fact should be noted in the document, perhaps in an introduction or description of its scope. This section should briefly indicate the reasons that peer review was not conducte d. Derivative prod ucts of major scie ntific or technic al products (such as fact sheets, p ress releases, and brochures) do not need to discuss whether the underly ing products were peer reviewed.
4.2.4 Can the Identity of P eer Reviewers be Kept Anonymous? Yes, up to a point. I n the ordinary course of event s, you can of ten discuss commen ts received without attributing the comments to a specific reviewer. However, if the matter has gone to litigation, the litigating parties can discover the names of any one who contributed to a Federal product, including peer reviewers. Therefore, it is not possible to totally shield peer reviewers. In addition, it may be difficult to shield the names of the pe er reviewers when the Agency is responding to a Freedom of I nformation Act (FOIA) request. If a peer reviewer requests anony mity at the outset of the peer review, the Peer Review Leader nee ds to inform the pee r reviewer of th e above possible e ventualities. The Agency w ill in the ordinary course of events attempt to maintain the confidentiality of the peer reviewers and their comments from public disclosure, but it is recognized in many instances, for example open public meetings a nd the above circ umstances, this ca n’t be assured. Re member, the Age ncy is committed to workin g “as if in a fish bowl” and most of its activities are tr ansparent to the p ublic (except where confidential business information is concerned). It is recognized that this may be a deterrent to possible peer reviewers, but this is a reality that has to be understood.
4.3 Completing the Peer Review Record
4.3.1 How Do I Complete t he Peer Revie w Record? Once the Peer Review Leader has completed the peer r eview and the final work product (where one is prepared), the peer review record is broug ht up to date and then archived according to that organization’s procedure (see Section 4.3.2). The peer re view record must be indexed and maintained in an organization’s archive (repository). The location of the pe er review record needs to be read ily identifia ble so interested parties can loca te and obtain mate rials easily and q uickly. The peer review re cord should be pla ced in any a ssociated establ ished public docket, if required, in addition to the organizational archive. As a courtesy, a copy of the revised work product may be sent to the peer reviewer s for information.Page 76 Peer Review Handb ook
The Peer Review Leader will collect the following materials for the pe er review record and submit for archiving; including at least (see also Section 2.5.3): a) The draft work product submitted for peer review b) Materials and information (including t he charge ) given to the peer reviewers c) Written comments, information, and materials received from the peer reviewers d) Information about the peer reviewers (such as reviewer s’ names, affiliations, and a summary of efforts to identify potential conf licts and their resolution) e) Logistical information about conduct of the peer review (such as times and locations of meeti ngs) f) A memorandum, or other written record, approved by the Dec ision Maker, responding to the peer review comments specifying a cceptance or, where thought appropriate, rebuttal and non-acceptance g) The final work product
4.3.2 Where Should the Peer Review Records be Kept, and for How Long? During the active conduct of the peer review, the Peer Review L eaders maintain the peer review record with themselves unt il the peer revie w is totally c ompleted. Minimally , the file should be m aintaine d until one year a fter the c ompleted p eer revi ew is repo rted in the next annual reporting

. After that, the p eer review re cord should be main tained for a “re asonable period of time.” Establishment and maintenance of the archive where the peer review re cords ultimately reside are an organization’s responsibility (i.e., not that of an individual program manager or Peer Review Leader). Gene rally, to allow flexibility, individual offices and regions will decide the appropriate level of organizational responsibility and how they will meet the “routinely available” requirement. The peer re view record may be kept with other records relating to the overall project, as long as it is easily and separa tely identifiable. The peer review record should be maintained in accordance with the Agency’s record keeping schedule for such records. One long term archiving mechanism may be the formal archiving at the Federal Records Center in Suitland, MD. (Note: This is the same question as Section 2.5.7, but applies in this chapter as wel l).Peer Review Handb ook Page 77
4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA? Yes, it is subject to release if EPA receives a Freedom of Informa tion Act (FOIA) request, unless the peer review information meets the criteria for an exemption under the FOI A. (http://www.epic.org/open_gov/foia/us_foia_act.html).Peer Review Handb ook Page 79 

# SUBJECT INDEX This is an alphabetical listing of subjects from the Handbook and the pertinent page numbers where they are found.

- A - Administ rative Procedures Act (12) Administ rator (9, 17) Advisory and Assis tance Ser vices (66) Analyti c bluepri nts (13, 25, 40, 49) Annual repor ting r equirement s (14, 18) Anonymity of peer re viewers (75) Archivi ng (47, 75) Assist ant Administrat ors (17) - B - Balancing of peer re viewers (22) Board of Scien tifi c Counselor s (BOSC) (41) Budget (18, 40, 48) - C - Candidate f or peer revi ew (26) Categories of work pr oducts (16, 25) Charge (46, 53, 76) Chemical a ction re ports (16) Clean Air Scient ific Advisory Committee (41) Compensation (65) Competiti on (70) Completing a peer revi ew (4, 73) Conducting a peer revi ew (3, 53) Conference proceedings (25) Confidential business inf ormation (CBI) (61, 69) Confident ial Fin ancial Disclosu re (59) Confident ialit y (63) Conflic ts of i nterest (22, 31, 58, 67) Contracti ng offi cer (57, 63, 68, 69) Contracts (31, 64, 65, 68) Contracts Management Manual (CMM) (65, 67) Controvers y (27, 40) Cooperative agreement s (31) Court order ed deadlin es (44, 61) - D - Decision- Maker (17, 26, 39) Deputy Admini strat or (14, 17, 18) Designat ed Ethics Offici al (DEO) (59) Division Director s (17) Docket (46, 47, 75) - E - Economic work pr oducts (27) Electroni c records ( 46) Environment al Impact Statemen t (EIS) (35) Environment al regul atory model s (36) Executiv e Order 12866 (13, 29) External peer revi ew (40, 41, 57) - F - Federal Acqui sition Regulat ion (FAR) (67) Federal Advisor y Commit tee Act (FACA) (49, 62) Federal Regi ster (12, 50) Final wor k product (73) Freedom of In formati on Act (FOIA) ( 16, 42, 60, 77) Frequency of peer revi ew (43) - G - Grants (31) Gratuit ous servi ces (65) - I - Independent peer revi ewer (10, 11, 21, 55) Inherently governmental functions (IG Fs) (67) Interac ting wi th peer r eviewers (64) Interag ency agreement (31) Interna l peer rev iew (41) Interna l peer rev iewer (22) - J - Journal a rticl e (16, 42) - L - Legal considerati ons (49) Letter reviews (41, 50) List A (15) List B (15) List C (15, 16) List D (16, 42) Litigation (16, 75) - M - Major impact ( 27, 40) Major scientif ic and technical work product (5, 9, 15, 26, 37)Page 80 Peer Review Handb ook
Managers planning check list f or peer review (5) Materials for peer r eviewers (62) Mechanisms f or peer revi ew (39) - N - National Academy of Sci ences (31, 42) NEPA Products (35) Non-major scientific and technical work products ( 16, 37, 38) Novelty ( 27, 40) NPDES permits (16) - O - Office Di rectors (17) Offi ce of Research and De velopmen t (ORD) (14, 20, 23) Open meeting (49) Other organ ization’s work products ( 36) - P - Peer consul tation (11) Peer input (10) Peer invol vement (10) Peer revie w (9, 10) Peer Review Adv isory Grou p (PRAG) (17, 23) Peer revie w comments ( 46, 47, 73, 76) Peer Review Coor dinator ( 14, 17, 20, 23) Peer Review Leader (17-19, 39, 47, 61, 62, 73, 75) Peer revie w materials (62) Peer revie w mechanis m (5, 39) Peer revie w panel (22, 56, 57) Peer Revi ew Pr oduc t Track ing (PRPT) database ( 14, 19, 34, 45) Peer revie w record (5, 21, 45, 75) Peer revie w schedule (18) Peer revie w services (65) Peer revie wers (21, 46, 55, 56, 61, 70, 76) Personal services (69) Planning a peer revi ew (2, 25) Premanufact uring not ices (PMNs) (16) Privacy Act (60) Procurement request s (71) Project mana ger (18, 19, 37) Public comment (12) - R - RCRA permits ( 16) Record of Decis ion (ROD) (25) Records manage ment (60) Regional Admi nistr ators (17) Regulator y developmen t process (13) Regulator y negoti ations (14) Rulemakin g (13, 27, 46) - S - Science Advi sory Board (SAB) ( 28, 41, 50, 56) Science Pol icy Counci l (SPC) (14, 17, 20, 23) Scienti fic Advisory Panel (SAP) (41, 50, Scientific and technical work product (41, 50, 56) Selecti on of peer r eviewers (25) Sensit ive act iviti es (55) Simplified acquisiti ons of peer revie w services (66) Site specific decision s (69, 70) Social Sci ence work pr oduct (35) Special Gover nment Employ ee (SGE) (30) Stakeholder invol vement (59, 72) State empl oyees (12, 13, 40, 54) Statement of work f or contrac ts (62) Strateg ic plans (59, 66) Subcontrac tor (25) Subject mat ter exper t (51, 57) - T - Tier 1 and Tier 2 (22) Tier 3 (13, 55) Time frame f or peer revi ew (13) Training (44, 55) Travel (18, 19) - V - Voluntary consensus standar ds (71) Voluntary services (39) - W - Work Assignment Manager (WAM) (57)Peer Review Handb ook Page 81 

# COMMONLY USED ACRONYMS

AA Assistant Administrator AAS Advisory and Assistance Serv ices ASTM American So ciety of Testin g and Material s BOSC Board of Scien tific Counselo rs CBI Confidential Business Information CFR Code of Federal Regulations CMM Contracts Mana gement Man ual CO Contract(ing) Officer COI Conflict of Interest DEO Designated Ethics Official EIS Environmental Im pact Statement EPA Environmental Protection Agen cy EPAAR EPA Acquisition Regulations FACA Federal Advisory Comm ittee Act FAR Federal Acquisition Regulations FIFRA Federal Insecticide, Fungicide and Rodenticide Act FOIA Freedom of Information Act FTE Full Time Equivalent GSA General Services Administration HHS Health and Human Services IAG Interagency Agreem ent IGF Inherently Governm ental Function IRIS Integrated Risk I nformation System ISO International Organization for S tandardization LAN Local Area Ne twork NAS National Academy of Sciences NEPA National Environmental Policy Act NPDES National Pollutant Discharge Elimination System NTTAA National Technology Transfer and Adva ncement Act of 1995 OAM Office of Acquisition Management OEI Office of Environmental I nformation OGC Office of General Counsel OMB Office of Management and Budg et ORD Office of Research and Developm ent PMNs Premanufacture Notice PO Purchase Order PRAG Peer Review Advisory G roup PRPT Peer Review Product Tracking D atabase RA Regional Administrator RC Regional Counsel RCRA Resource Conservation and Recov ery Act RIAs Regulatory Impact AnalysesPage 82 Peer Review Handb ook
ROD Record of Decision SAB Science Adv isory Board SAP Scientific Advisory Panel SGE Special Government Emp loyee SOPs Standard Operating Procedures SOW Statement of Work SPC Science Policy Council USC United States Code WTI Waste Technologies Industries WAM Work Assignment ManagerPeer Review Handb ook Page A-1



# APPENDIX A - U.S. ENVIRONMENT AL PROTECTION AGENCY PEER REVIEW POLICY

June 7, 1994Page A-2 Peer Review Handb ook

June 7, 1994

MEMORANDUM
SUBJECT: Peer Review Program
TO: Assistant Administrators General Counsel Inspector General Associate Administrators Regional Administrators Staff Office Directors
Today, I am reaffirming the central role of peer review in our efforts to e nsure that EPA policy decisions rest on sound, credible science and data (see attac hed policy statement). Toward that end, as its first majo r task, EPA ’s Science P olicy Council (SPC) is i nstituti ng a program to expand and improve peer review in all EPA offices. This memorandum gives an overview of cur rent practices and outlines the ne w program. Peer Review Practices and Policy Peer review at EPA takes several different forms, ranging from infor mal consultations with Agency colleagues who were not involved in developing the product to the for mal, public processes of the Science Advisory Board (SAB) and the FIFRA Scientific Advisory Panel (SAP). In any form, peer review a ssists the Agency’s work by br inging independent expert experience and ju dgment to bear o n issues before th e Agency to the benefit of th e final product.
EPA’s Peer Review Policy, whic h responds in part t o recommendation s in the “Credible Science, Credible Decisions” report, outlines general principles for peer review a t EPA. Dif fere nt E PA o ffi ces h ave u nde rtakenvari ous imp lem ent ing a cti vit ies , in clu din g an A genc ywide information and planning workshop, internal guideline development, and numerous specific peer reviews. Even with these activities, however, I am conc erned that EPA does not yet have a comprehensive Agency -wide program for implementing its Peer Review Policy. I therefore welc ome the SPC initiative toward effecti ve, efficient imp lementation of the policy in all the program areas to which it a pplies.Peer Review Handb ook Page A-3 Expanding and Improving Peer Review The Science Policy Council and its Steering Committee have outlined a dual-track implementation program of planning and assistance for all Agency offices. The first track has three major miles tones. First, during the next few weeks, Steering Committee members will consult with senior management in each office to exchange information on current peer r eview activities, assistance needed, pos sible obst acles to im plementat ion, and im plementat ion planni ng. Second, using information and materials developed during the first stage, peer review task groups in each office will develop standard operating procedures (SOPs) for use in each office, based in part on generic guidance to be issued by the SPC and in part on peer review needs and capab ilities specific to each office. T he resulting SOPs w ill delineate as a ppropriate the scope of application of peer review with respect to various type s of scientific and technical work products such as reports of original research, risk assessments, and analy tical methods of economic analysis. OARM and OGC staff will assist each office as neede d on legal, budget and administrative matters. Each AA and RA will submit draft SOPs for Steering Committee review by July 15. Third, the SPC review group will work with each office to complete each plan by September 15. In parallel with the above, consistent with the Peer Review Policy, the Scienc e Policy Council will work with each AA and RA to identify “major scientific and technical work products” as peer review candidates for the coming y ear. This process will consider existing and new plans for internal review s and for Science Advisory Board (SAB), FI FRA Scientific Advisory Panel (SAP), and other external reviews. The two-fold objective is to plan reviews for technical products covered by the Peer Review Policy and to gain experience with options and obstacles. We w ill use this experien ce to review and revise the SOPs as n eeded. Also, to establish a baseline for comparison, each AA and RA will identify the “major tec hnical products” comple ted within his/her p rogram during the past 12 months. The Science Policy Council has sent additional information to each office offering guidance on the procedures that you are a sked to develop and the schedule for these activities. Please note, however, that because the policy is effective immediate ly, current peer review planning should c ontinue on presen t schedules in par allel with develop ing the formal SO Ps.Page A-4 Peer Review Handb ook To begin this process, I have asked each Assistant Administrator and Regional Administrator to designate a Peer Review Coordinator to work with the Steering Committee on implementation activities specific to each office. I am very pleased that the Science Policy Council is taking th is important step. A comprehensive p eer review pro gram is essentia l to maintaining and improving the quality of the analy ses that underlie Agency actions. I look forward to w ork ing w ith you and yo ur s taf f onthi s im portan t ac tiv ity.

/s/
Carol M. Browner

Attachment

cc: Science Policy Council Science Policy Council Steering CommitteePeer Review Handb ook Page A-5 

### PEER REVIEW AND PEER INVOLVEMENT AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY This document establishes the policy of the United States Environmental Protection Agency (EPA) for peer review of scientifically and technically base d work products that are intended to support Agency decisions. Peer review is presented in the c ontext of the broader concept, peer i nvolvement.
BACKGROUND The report “Saf eguarding the Future: Credible Science, Credibl e Decisions”1 focused on the state of science at EPA. The panel of experts who prepared the report emphasized the importance of peer review, especially external peer review, and the need for broader and more systematic u se of it at EPA to eva luate scientific and technical wo rk products. Thei r specific recommendation r egarding pee r review reads as follows: “Quality assurance and pee r review should be applied to the plan ning and results of all scientific and technical efforts to obtain data used for guidance and decisions at EPA, including such efforts in the program and regional offices. Such a requirement is essential if EPA is to be perceived as a credible, unbiased source of environmental and health information, both in the United States and throughout the world.” In response to this recommendation, then-Administrator Reilly directed staffto develop an EPAwide policy statement, which he issued in January, 1993. The par agraphs below preserve the core of that ear lier statement whi le updating it to sp ecify the r ole of the Science Policy Counci l in guiding furth er implementation of the policy

. Effective use o f peer review is indispensable for fulfilling the EPA mission and therefore deserves high-priority a ttention from program managers and s cientists within all pertinent Headq uarters and Reg ional Offices.

1 EPA/600/9-91/050, March 1992.Page A-6 Peer Review Handb ook
PEER INVOLVEMENT A ND PEER REVIEW EPA strives to ensure that the scientific and technical underpinnings of its decisions meet two important criteria: they should be based upon the best current knowledge f rom science, engineering, and other domains of technical expertise; and they should be judged credible by those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement -- that is, they augment their capabilities by inviting r elevant subject-matter experts from outside the program to be come involved in on e or more aspect s of the developme nt of the work prod ucts that support policie s and actions. One particularly important type of pe er involvement occurs when scientifically and technically based work products undergo peer review -- that is, when they are evaluated by relevant experts from outside the program who are peers of the program staff , consultants, and/or contractor personnel who prepared the product. Properly a pplied, peer review not only enriches the quality of work products but also adds a degree of cre dibility that cannot be achieved in any other way. Further, peer review ea rly in the development of work products in some cases may conserve future resources by steering the development along the most efficacious course. Peer review generally takes one of two f orms. The review team may consist primarily of relevant experts from within EPA, alb eit individuals who h ave no other invol vement with respect to the work product that is to be evaluated (internal peer review ). Or the review team may consist primarily of independent experts from outside EPA ( external peer review).
POLICY STATEMENT Major scientifically and technically ba sed work products related to Agency decisions normally sho uld be peer-rev iewed. Agency managers within Headquarters, Regions, laboratories, a nd field componen ts determine and a re accountable for the decision w hether to employ peer review in partic ular instances a nd, if so, its chara cter, scope, an d timing. These decisions are made in conformance with program goals and priorities, resource constraints, and statutory or court-ordered deadlines. For those work produc ts that are intended to support the most important decisions or that have special importance in their own right, external peer review is the procedure of choice. Peer review is not restricted to the penultimate version of work products; in fact , peer review a t the planning sta ge can often be extremely be neficial.
SCOPE Agency managers routinely make regulatory and other decisions that necessar ily involve many diffe rent considerat ions. This policy applies to major wo rk products that a re primarily scientific and technical in nature and may contribute to the basis for policy or regulatory decisions. By contrast, this policy does not apply to non-major or nontechnical matters that Agency ma nagers conside r as they ma ke decisions. Simila rly, this polic y does not ap ply to these ultimate decisions

.Peer Review Handb ook Page A-7 This policy applies where appropriate, as determined by the National and Regional Program Managers, to major scientifically and tec hnically based work products initiated subsequent to the d ate of issuance. Peer review shou ld be employe d to the extent reas onable to relevant work products that currently are under deve lopment. This policy does not apply to the bases for past decisions, unless and until the relevant scientific and technical issues are considered ane w in the Agency ’s decision-makin g processes. Except where it is required by law, formal peer review (a s distinguished from the Agency’s normal internal review procedur es) should be conducted in a manner that will not cause EPA to miss or need extension of a statutory or court-ordered deadline. Ag ency managers still may undertake peer review if it can be conducted conc urrently with necessary r ulemaking steps. LEGAL EFFECT This policy st atement does not e stablish or affec t legal rights o r obligations. Ra ther, it confirms the importance of peer review where appropriate, outlines relevant principles, a nd identifies factors Agency staff should consider in implementing the policy

. On a continuing basis, Agency management is e xpected to evaluat e the policy as well as the res ults of its application throughout the Agency and undertake r evisions as necessary. Therefore, the policy does not stand alone; nor does it establish a binding norm that is finally determinative of the issues addressed. Minor variations in its application from one instance to another are appropriate and expected; they thus are not a legitimate basis for dela ying or complicating action on otherwise satis factory sc ientific, technic al, and regula tory produc ts. Except where prov ided otherwise by law, peer revie w is not a formal pa rt of or substitute for notice and co mment rulemaking or adjudicative p rocedures. EPA’s decision whethe r to con duc t pe er re vie w inany p art icu larcase is w hol ly wi thi n th e Age ncy’s dis cret ion

. Si mil arl y, nothing in this policy creates a legal requirement that EPA respond to pee r reviewers. However, to the extent that EPA decisions rely on scientific and technical work products that have been subjected to peer review, the remarks of peer reviewers should be included in the rec ord for that decision.
IMPLEMENTATION The Science Policy Council is responsible for overseeing Age ncy-wide implementation. Its responsibi lities include prom oting consistent i nterpretation, a ssessing Agen cy-wide pr ogress, anddev elo pin g rec omm end ati ons forrevi sio ns o f th e po lic y as n eces sary. The Science Policy Council will oversee a peer-review work g roup, which will include representative s from program u nits throughout EPA to effect a cons istent, workable implementation of the policy. The work group will assist the programs in (1) formulating and, as necessary , revising standa rd operating pr ocedures (SOPs) f or peer review consistent with thisPage A-8 Peer Review Handb ook policy; (2) identifying work produc ts that are subject to review; and (3) for each major work product, selecting an appropriate level and timing of peer review. In assisting the programs, the work group will take into account statutory and court deadlines, resou rce implications, and availability of disinterested p eer reviewer s. The group will work closely with Headquarters offices and the Reg ional Offices toward ensuring effective, efficient uses of peer review in supporting their mission objectives. However, the Assistant Administrators an d Regional Adminis trators remain ul timately re sponsible for deve loping SOPs, identifying work products subject to peer review, determining the type and timing of such review, documenting the process and outcome of each peer review, and othe rwise implementing the policy w ithin their orga nizational units. Because peer review can be ti me-consuming an d expensive, Age ncy manag ers within Headquarters, Regions, labora tories, and field components are e xpected to plan car efully with respect to its use -- taking account of program priorities, resource considera tions, and any other relevant co nstraints as well as t he policy goal o f achieving h igh-qualit y, credible und erpinnings for decisions. External peer reviewers should be chosen carefully to ensure an independent and objective evaluation. The affiliations of peer reviewers should be identified on the public record, so as to avoid under cutting the cre dibility of the peer-review p rocess by c onflicts of intere st. The policy is effective immed iately. The peer-review w ork group mentio ned above will identify the focal point to whom c omments and questio ns should be addre ssed and, from time to time, will provide f urther informati on about implementa tion activities.

/s/ DATE: JUN 7 1994 CAROL M. BROWNE R, ADMINISTRATOR APPROVED:Peer Review Handb ook Page B-1



# APPENDIX B - SOUN D SCIENCE AND PEER REVIEW IN RULEMAK ING
Using EPA’s r egulatory agend a as the cornerstone of the Agency’s mission, the Deputy Adm inistrator’s m emo implem ents an additional re quirement as pa rt of the ongoing effort to u se peer review to enhance the quality and credibility of Agency decisio ns.Page B-2 Peer Review Handb ook June 15, 1999
MEMORANDUM
SUBJECT: Sound Science and Peer Review in Rulemaking
TO: The Regulatory Policy Counci l Over a year ago, the Age ncy enhanced its peer review policy and procedures with the finalization and release of the Peer Review Handbook. The Handbook was developed to serve as a guide for Agency scientists and manag ers on the organization and conduct of peer review. With our regulato ry agend a as the corners tone of the Agen cy’s mission, I would now like to implement an additional requirement as part of the ongoing effort to use peer review to enha nce the quality a nd credibility of Agency decisions. Beginning July 12, 1999, the action memorandum accompany ing each rule submitted for signature must explicitly address the use of peer review in that rulemaking. The Administrator and I want to ensure that the Agency takes complete advantage of the benefits that thorough peer reviews can provide in our deliberations on the many challenging decisions that must be made. A key step in producing cre dible regulator y decisions s upported by well-documented scientific data is ensuring that our work is based on broadly accepted sc ientific studies. Peer review is an essential tool for accomplishing th is goal. In addition, I ask that, beginn ing today, planning for pe er review be spe cifically addressed in each new Analytic Blueprint. The g eneral guidance on integrating peer reviews into the rulemaking process is included as Attachment A. Detailed guidance on action memoranda, updated to address peer review, is provided in Attachment B. Guidance reg arding which scientific or technical studies must be peer reviewed is contained in the fact sheet on “Peer Review in the Rulemaking Process” in Attachment C and in the Peer Review Handbook available at http://www.epa.g ov/ordntrnt/ORD/sp c/sopmenu.htm. I am also takin g this opportunity to bring to your attention a memorandum from the O ffice of General Counsel that discusses the implications of a recent decision in a challenge to a peer review brought a gainst the Agency under the Federal Advisory Committee Act (Attachment D) [note added: this attachment is not attached to the Peer Review Handbook ]. I apprecia te your supp ort in this effort. For additional inf ormation about pee r review in the rulemaking process, please contact your Steering Committee representative, your peer review coordinator (Attachment E) [note added: this attachment is not attached to the Peer Review Handbook ], or Phil Sc hwartz in t he Offic e of Policy at xxx-xxxx. [signed 6/15/99, AX 9807793] Peter D. Robertson Acting Deputy AdministratorPeer Review Handb ook Page B-3
ATTACHMENT A Regulatory Management Guidance PEER REVIEW REQUIR EMENTS Office of Policy FOR EPA RULEMAKING June 1999 Reference: RMD 5
BACKGROUND On February 25, 1998, the Agency reaffirmed its commitment to quality decision-making by issuing the compr ehensive Peer Re view Handbook. Th e Peer Review Ha ndbook is available at: http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm, which amplified and clarified key Agenc y peer review policy issues. The Handbook states that, in genera l, all major scientific and technical work products related to Agency decisions should normally be peer reviewed. Peer review is a documented, critical review of a work product, performed by experts who are independent of those who developed the product. Work products that are developed to support regulations, such as risk assessments, should be evaluated to determine whether they meet the criter ia for "major scientific and/or technical work products" and thus require peer review (See Attac hment C). Regulations themselves are not peer reviewed. The straight-forward a pplication of accepted, previously peer-reviewed economic methods or analy ses in Regulatory Impact Analy ses (RIA's), are not typically subject to formal pee r review. If, however, the particular facts and circumstances o f any piece of economic ana lysis in an RI A warrant peer review, the Ag ency will accommodate thos e needs on a case -by-case basis.
PLANNING FOR PEER REVIEW Remember that: 

### � Analytical Blueprints should discuss the plan fo r peer review o f major scientific and technical pr oducts. This should i nclude a schedul e for the review and should identify the resources that will be needed. 

### � When negotiating rule deadlines with courts or with litigants, be sure that your proposals allow a dequate time for any neede d peer reviews. PEER REVIEW REQUIR EMENTS Program offices have the primary responsibility for complying with the Agency peer review requirements. Senior managers must confirm in their Action Memoranda that their offices have complied with the peer review requirements; detailed explanations are not necessary . If no major scientific and technical product as defined by the Peer Review Ha ndbook was used to support the action, includ e the following s tatement in the ac tion memorandum:Page B-4 Peer Review Handb ook
Peer Review There were no major scientific or technical products supporting this action as defined by the Agency's Pee r Review Handbook . We did not, the refore, submit any support documents for pee r review. If a major scientific or technical product was used to support the action, include the following statement in the Ac tion Memorandum:
Peer Review [Insert Name of AAs hip] has complied with the Agenc y's Peer Review re quirements with respect to the underlying major scientific and technical products supporting this action. You may add any details y ou think important, but you may not modify this compliance statement. If you utilized peer-reviewed produc ts, but you could not fully comply with the peer revi ew re qui rem ent s, e xpl ainin t he A cti on M emo rand um w hy th e dev iat ionwas nece ssary. FINAL AGENCY REVIEW (FAR) For rules that go through Final Agency Review, OP will document compliance in the FAR memorandum. This documentation will be based on the peer review information included in the draft Action Memo randum. These requirements will be effective on July 12, 1999.After that date, OP's Office of Regulatory Management and I nformation (ORMI) will only forward for signa ture those actions that inclu de one of the following: 

### � a statement that no major scientific or technical doc uments were utilize d to support th e rulemakin g; 

### � a statement of co mpliance with the p eer review re quirements for ma jor scientific and technical do cuments; 

### � an explanation for deviation from the peer review re quirements. Questions?Contact your Regulatory Steering Committee representative or Phil Schwartz in the Regulatory Management Division at (202) 260-5493.Peer Review Handb ook Page B-5
ATTACHMENT B Regulatory Management Guidance Office of Policy FRAMEWORK FOR ACTION MEMORANDA Revised June 1999 Reference RMD-6
OVERVIEW The overview provides background information which briefly de scribes the rule, characterizes the environmental issue(s) or public health problem(s) being addressed, and summarizes the histo ry of the re gulatory a ction. It should explain why EPA is taking this action, and wher e appropriate, c over the following points: 

### � Define the court or statutory deadline; 

### � Identify whether the action amends the CFR, a nd if so, explain what kind of amendment (technical, procedural, etc.); 

### � Identify other regulatory actions underway that will affect this particular progra m or sector; 

### � Describe the specific environmental issue(s) or public health problem(s) being addressed, and the goal intended by taking this action; 

### � Describe what th e action does, an d specifically , how the regula ted community is affected (performance standards, specific requirements); and 

### � Describe how flexible the implementation of the action will be for states and regulated enti ties.
REGULATORY IMP ACTS Summarize the costs and benefits of the regulatory action (include costs to State and local governme nts, and tribal com munities) and the r esults of the econ omic and risk analyses. Discuss the economic impacts on all stakeholders including, where a ppropriate, the results of the regulatory flexibility analy sis and the specific effects of the action on small entities. Explain the roles of cost-benefit and cost-effectiveness analyses in shaping the regulatory approach.
Describe the re porting and Reco rd keeping burd en and what the Ag ency has do ne to reduce it. I ndicate possible i mpacts on other F ederal agenc ies.Page B-6 Peer Review Handb ook
ADMINISTRATOR'S P RIORITIES If appropri ate, discuss how th e regulatory action relates to the Administrator' s priorities. Explain how the reg ulatory ac tion relates to a sp ecific priority only where the relationship is significant, c lear and distinct.
STAKEHOLDER INVOLVEMENT Briefly discuss the role of both government entities and private sec tor stakeholders in the development of the action. Summarize the concerns they have raised and wha t the Agency h as done to address them.
INTERNAL REVIEW Identify whether the reg ulatory action was developed under Tier 1,2, or 3. Describe a ny outstanding issues from workgroup closure or other internal review. Ide ntify program offices or regions with outstanding issues and indicate why they cannot be resolved or accommodated. Also, provide the basis for any decision made to not address an identified cross- media impact. Wher e there is not a fo rmal workgroup, identify th ose offices that have reviewed the package.
PEER REVIEW If you did not utilize major scientific and technical products as defined by the Peer Review Handbook to support the acti on, include the fo llowing statemen t: There were no major scientific or technical products supporting this action as defined by the Agen cy's Peer Revie w Handbook. We did not, t herefore, submit any support documents fo r peer review. If a major scientific or technical product has been used to support the action, include the following statem ent: [Insert Name of AAship] has followed the Agenc y's Peer Review re quirements with respect to the unde rlying, major scientific and technical product s supporting this action. You may add any details y ou think important, but you may not modify this compliance statement. If you utilized pe er-reviewed p roducts, but y ou could not fully comply with the peer revi ew re qui rem ent s, e xpl ainwhy the d evi ati on w as n eces sary.Peer Review Handb ook Page B-7
PLAIN LANGUAGE Either in the action memorandum or in a separate note briefly describe which elements of plain language have been addressed.
OMB TRANSACTION Highlight significant issues resulting from OMB's review under Executive Order 12866. Explain any c hanges made to t he regulatory action as a resul t of that review. I f OMB review was waiv ed, please indica te this.
ANTICIPATED PU BLIC REACTION Describe the type of public response anticipated and identify both the involved stakeholders and the nature of their expected response. Characterize the likely reaction to the action by all interested pa rties including in dustry; envi ronmental groups; Congress; state, local, and tribal governments; and OMB. Explain why the Agency should take the action despite any controversial response anticipated from the public. If the regulatory ac tion will not be issued for public comment (i.e., direct final rule, administrative sta y, etc.), e xplain the basis for t he Agency 's decision not to s olicit comment. RECOMMENDATION Identify the action the administrator is expected to take.
NOTE: An Action Memo shou ld generally not exceed 5 page s.Page B-8 Peer Review Handb ook
ATTACHMENT C Fact Sheet 

### Peer Review in th e Rulemaking Pr ocess
What Role does Pe er Review have in the Regulatory Deve lopment Proc ess? Peer review of scientific and technical work products affirms the scientific credibility of the Agency's rulemakings. Work products nee d to be well planned and documented because new rules, and the work products supporting them, may have substantial cost impacts and must often withst andint ens e publi c scr uti ny. What Work P roducts N eed Peer Review? While rules themselves are not subject to peer review, any major scientific and/or technical work products that support rules should be peer reviewed. (See Section 2.2 of the Peer Review Handbook http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm to determine which work products are considered major.) Designations of a scientific and/or technical work pr oduct as "major" will largely be case-by -case. Those that are used to support a regulatory program or policy position and that meet one or more of the following criteria are c andidates for peer review:a) Establish es a signifi cant precede nt, model, or methodo logy b) Addresses significant controversial issues c) Focuses on significant emerging issues d) Has significant cross-Agency/inter-ag ency implications e) Involves a significant investment of Agency resources f) Considers an innovative approach for a previously defined problem/p rocess/met hodology The straight-f orward applica tion of accepted , previously peer-reviewe d economic methods or analyses in Regulatory I mpact Analyses (RIA's), ar e not typically subject to formal peer review. I f, however, the p articular fact s and circumstanc es of any pie ce of economic analysis in an RIA warrant peer r eview, the Agency will accommodate those needs on a case-by-case basis. The decision whether to peer review any work product will be documented through the Agency's annual pee r review reporting process (see Section 1.3 of the Handbook). When should determinations about the need for peer review be made? The need for, an d scheduling of, peer reviews sh ould be addresse d in the Analy tic Blueprint that is required for all Tier 1 and 2 regulatory a ctions. Work products supporting Tier 3 rule s may al so be c onsid ered m ajor a nd Ana lytic B luepr ints are en coura ged for Tier 3 rules . In any event, the need for, and scheduling of, pee r reviews should be addressed early in the development of Tie r 3 actions.Peer Review Handb ook Page C-1



# APPENDIX C - EX AMPLES OF CHARGES

Please note -- certain questions that are posed in charges can be responded to with a yes or no answer. Clea rly, this is not t he type of respo nse we generally want, therefore, it is important to phrase charge questions carefully to ensure that you receive a fully satisfactory and thoughtful response. Where a y es or no answer might be expecte d, be sure to ask for a full explanation sup porting the yes or no answer. Charges can run the gamut from rather simplis tic to highly com plex. The examples shown here cover a vari ety of types

. Examples 1 through 6 have less complex questions and are looking for the overall quality of the reports. Examples 7 through 11 have numerous technical questions that need to be addressed and are therefore more complex in their nature.Page C-2 Peer Review Handb ook
CHARGE EXAMPLE 1 - Framework fo r Developing a Living R esources Resea rch and Monitoring Plan for the Peconic Estuary Background The Peconic Estuary is located on the eastern end of L ong Island, New York. Under the Federal Clean Water Act, the Peconic Estuary was named a n "Estuary of National Significance" in 1992. Because of its high concentration of rare, threatened and endang ered species and habitats, The Nature Conservancy named the Peconic Ecosy stem as one of the "Last Great Places in the Western Hemisphere." The Peconic Estuary Program (PEP) released its draft Comprehe nsive Conservation and Management Plan (CCMP) in September 2, 1999. The draft Plan addresses the environmental management issues facing the estuary and its wate rshed, including brown tide (a nuisance algal bloom), nutrients, habitats and living resources, path ogens, and toxics, as well post-CCMP management structure, public education and outreach, and financing. A final Plan is now under preparation. Th e preparation of the Framework d ocument is called for in the draft CCM P. With population increasing in the watershed, the Peconic Estuary is being threate ned by over-developme nt and misuse of its r esources. To ful ly realize th e impacts of peopl e and their activities on this system, there must be a better understanding of how the Peconic Estuary functions ecologically. This knowledge can only be achieved through comprehensive research and monitoring of the entire ecosy stem. The long-term goal of this Framework for Developing a L iving Resources Research and Monitoring Plan is t o develop a strate gy of coor dinated resear ch and monitoring to fill significant information gaps and assist in the planning, conservation, and management of the Peconic Estuary. A key component of this strategy is to develop an applied, multi-scale, integrated approach to gain a better understanding of the estuar y. To achieve, this goal, the Framework for Developing a Living Resource s Research and Monitoring Plan sets broad priorities, provides the context for specific studies, guidelines for a detailed living resources research and monitoring plan, and stimulus for funding agencies and orga nizations and researchers. The short-term goal of this Framework is to provide a basis for sec uring funding for the priority initiatives descr ibed in the Frame work document.
In order of i mportance, the ob jectives of this F ramework docume nt are to: 1) understand threats so as to improve resource protection through management and conservation; 2) examine the biolog y and ecolo gy of par ticular organi sms identified in the CCMP as important eithe r due to their comme rcial or recre ational value or t heir role in the food web and ecosystem; and,Peer Review Handb ook Page C-3 3) undertake system-wide studies of the Peconic Estuary to understand the sign ifi canc e of k ey hab ita ts and e cos yste m pr odu cti vit y. Recommendations included in this framework are based on information gaps identified in the Characterization Report of the Living Resources of the Peconic Estuary (Bortman and Niedowski 1998), the Habitat Module of the draft CCMP, PEP Natural Resource Subcommittee meetings, and recommendations made at a PEP-sponsored Living Resources Resear ch and Monitoring workshop held in 1998. Copies of the draft CCMP and Characterization Report are included with this package.
Tasks The peer review charge is to assess the adequacy of this document to provide a framework for integrated, system-wide ec ological research and monitoring to understand the dynamic, multi-scale ecological patterns and processes that sustain biota and the ir supporting natural syst ems in t he P econ ic E stu ary. Specific Questions: Peer Review ers shall an swer/comm ent on the fo llowing: 1) Is the orga nization of the docu ment appropriate and does it presen t the material in a cl ear a nd c onc ise man ner? Ple ase e xpl ainful ly. 2) In y our op inion , wha t are the w eakest and the st rong est a spec ts of t he fr amew ork? Please make suggestions on how the weakest parts can be strengthened. 3) Are there any elements missing from the framework w hich you think need to be included or which would strengthen the document? Please explain fully. 4) Are you aware of any other significant data/studies that are relevant and should be inc lud ed o r referen ced i n th is d ocumen t? P lea se ex pla in f ull y. 5) Is the stated g oal realistic? Are t he stated objecti ves adequately met? Please explain fully. 6) Examine and critiqu e the interrelat ionship among the specified assessment, research, and monitoring priorities. 7) Does this framework present a holistic approach to preserve, protect and restore the estuary? Does the approach cover all the relevant and important are as that warr antinv est igat ion ? If n ot, ple ase s tat e wh at s hou ld b e do ne d iff eren tly.Page C-4 Peer Review Handb ook 8) Are the initial set of assessment, re search, and moni toring prioritie s adequate to provi de the infor matio n nee ded to successf ully prote ct an d mana ge th is sy stem? Please comment on the selection of priorities. Were any significant priorities mis sed ? Wh at o the r pr ior iti es s hou ld b e sp eci fie d an d wh y?Peer Review Handb ook Page C-5
CHARGE EXAMPLE 2 -EPA Science A dvisory Board (SAB) R eview of the Environm ent Monitoring and Assessment Program (EMAP) Research Strate gy and Research Plan
The Science Advisory Board (SAB) is asked to review the Environment Monitoring and Assessment Program (EMAP) Research Strategy and Rese arch Plan. The review is requested by the Office of Re search and Dev elopment (ORD) wi th the following sp ecific charg e issues: 1) Previous peer reviews recommended that EMAP develop a close working relation with EPA Program Offices and other federal monitoring efforts. Does the EMAP strategy support the [Office of Science and Technolog y Policy's Committee on Environment and Natural Resources] CENR National Monitoring Framework and EPA Program Off ices? 2) Previous peer reviews recommended that EMAP initiate a focused research program on indicator development. Does the intramural EMAP program on ecological indicator development, coordinated with the [ORD Science To Achi eve R esult s] STA R soli cita tions, res pond t o this rese arch need ? 3) Previous peer re views recommend ed that the EMAP des ign be modified to include a set of nonrandomly selected sentinel sites with intensive data collection. Does the development of Index Sites as outdoor laboratories in the national parks ([National Park Service] NPS and [US Geological Survey] USGS) and selected estuaries ([Natio nal Oceanic and Atmospheric Admin istration] NOAA) a dd this dimension to the EMA P? 4) Previous peer reviews recommended that EMAP combine effects-oriented and stressor-orient ed monitoring app roaches. Do the f ocused geogr aphic demonstration pilot studies (initially in the Mid-Atlantic region) combine these elements?Page C-6 Peer Review Handb ook CHARGE EXAMPLE 3 - EPA Science Advisory Board Review of the Agency's National Risk Management Research Laboratory’s (NRMRL) Program
The Office of Research and Development (ORD) requests that the Science Advisory Boardrevie w the A gency's N ation al Ris k Mana gemen t Rese arch Laborat ory’s (N RMRL) program. In the “Strate gic Plan for the O ffice of Resear ch and Developme nt” (EPA, 1996a), ORD described the relationship of risk assessment to the risk management process, and emphasized the need for scientific and engineering research to ena ble sound risk management decisions and actions. Within the framework of that strategic plan, NRMRL’s mission is to conduct research to reduce uncer tainties and costs a ssociated with mak ing and implement ing environmenta l risk management de cisions. NRMRL h as therefore de veloped a resea rch agenda to r educe risk uncertainty that also focuses on those important, relevant issues where it can make a difference. The charge to the SAB is to: 1) Examine and critique the research programmatic directions such as whether NRMRL is pursuing the most appropria te research pr oblem areas; 2) Comment on strateg ic directions, e.g., use of its cor e technical comp etencies, transition from primarily extramural to an intramural R&D organization, leveraging with other agencies and organization; 3) Review and comment on the effectiveness of NRMRL’s approach to science management, e.g., measures of success and science quality , soundness of peer review process; 4) Examine and critique the relationship of NRMRL’s risk management research and its intended role in the risk assessment/risk management paradigm; and 5) Review and comment on the strategic balance for the next decade among pollution prevention, technology development, remediation, and risk management assessment activi ties.Peer Review Handb ook Page C-7 CHARGE EXAMPLE 4 - EPA Science Advisory Board (SAB) Review of the Technical Aspects of the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM)
The EPA Science Advisory Board (SAB) is asked to review the technical aspects of the Multi-Agency Radiation Survey and Site I nvestigation Manual (MARSSIM). The review document was developed collaboratively by four Federal agencies, departments and commissions having authority for control of radioactive materials: Department of De fense, Department of Energy, Environmental Protection Agenc y, and Nuclear Regulatory Commission. MARSSIM addresses the need for a nationally consistent approa ch to conducting radiation surveys of p otentially r adioactively contaminated site s that are being considered for r elease to the public. A condition of release is a demonstration that residual radioactivity levels do not exceed a specified risk or dose level, also known as a release criterion. MARSSIM provides guidance to use rs performing a nd assessing the results of such a d emonstration for s urface soils and building surf aces. The SAB is asked b y the Agen cy’s Offic e of Radiation and Indoor Air (O RIA) to respond to the following charge in its review: 1) Is the overall approach to the planning, data acquisition, data assessment, and data interpreta tion as described in the MARSSIM te chnically a cceptable? Please exp lai n fu lly. 2) Are the methods and assumptions for demonstrating compliance with a dose- or ris k-b ased regu lat iontec hni cal ly ac cept abl e? P lea se ex pla in f ull y. 3) Are the hypotheses and statistical tests and their method of application app rop riate? Ple ase e xpl ainful ly.Page C-8 Peer Review Handb ook
CHARGE EXAMPLE 5 - Economics - Benefits Transfer from Adults to Children Background EPA established the Office of Children’s Health Protection (OCHP) in 1997 to support its efforts to increase the protection of children’s health throughout its programs. One of the many diffic ult issues the Offic e is addressing i s the appropriate treatment of chil dren’s health effects in the economic analyses performed by the Agency. Policy ana lysis efforts at the Agency often rely on the benefits transfer technique, and very few of the Agency’s benefit transfe rs have explicitly addressed children’s health issues. In addition, no accepted sy stematic process for conducting benefits transfer currently exists. To assist the Agency in its efforts, this paper discusses the benefits transfer technique as it applies to estimating values for children’s health. The first section provides some ge neral backgr ound on the techniq ue, and its applica tion to estimate health-related values. The second section raises important general issues to consider when conducting a benefits transfer for children’s health values. The last section discusses the implications of using the benefits transfer method to estimate values for children’s health. The scarcity and state of existing child-oriented health valuation literature sugg ests that it may be necessary to transfer adult-oriented values to estimate child-re lated values (Neumann and Greenwood 1999). However, the results of this paper suggest that transfer of the se value estimates to childr en at best provide s estimates for a s coping analy sis. In cases where these scoping exercises indicate that children's health values may be a crucial component in the policy analysis, pr imary rese arch should be und ertaken to estima te child-relate d values. 1) Assess the appropriateness of transferring health benefit values estimated for adult populations to children. Descr ibe the specific issues that arise i n these transfers. Which variables or situa tions improve or de crease the app ropriateness of transferring benefit values from adults to children? 2) Does the analy sis support the prop osition that the valu e of children’s h ealth effects should be estimated differently than adult health effec ts? Why or why not? 3) Identify issues for furthe r research tha t would improve our ability to est imate values for childr en’s health eff ects.Peer Review Handb ook Page C-9
CHARGE EXAMPLE 6 - Economics - St udy of Municipal Gover nment Costs and Financial Impact s from Environme ntal Regulations Background Municipalities play a major role in supplying environmenta l services. Local governments have taken responsibility for providing drinking wa ter, sewage treatment, and waste disposal in a majority of communities. Over the past fifteen to twenty years, most of the mandates found in the federal environmental legislation enacted in the early 1970s have been met. The increase in the number of people served and improvements in the quality of loc al environmental se rvices have bee n considerable, as has the investme nt in public infrast ructure to meet these laws. Recent revisions to the environmental legislation have established a broader and more stringent set of standards to be met by suppliers of environmental services. As a re sult, many local governments are now faced with having to maintain all or some part of their public services at a higher level of performance. To meet these new standards will require a dditional investments in capital, and increases in rates charged to customers for environmental services. Improvements in environmental services are but one of several demands being made of local public infra structure. Studie s prepared on pub lic infrastructu re needs and the availability of funds to meet the se needs indicate that there will be an excess demand f or money to r ebuild and improve upon th e existing stock of public infrastruc ture. Therefor e, it is important to recognize that additional environmental requirements will have to compete with other infrastructure needs (e.g., highway s, bridges), as well as other public services (e.g., police, education, heal th and welfare p rograms) provi ded at the local le vel. Given the increasing demand for public services, this study examines what additional investments the new environmental legislation will require local governments to undertake, and the likelihood that t hey will fac e difficulties ra ising the necess ary funds th rough capital ma rkets and revenues from customers. The economic impacts of individual EPA actions are considered during the regulatory process in those situations permitted by environmental statutes. The unique feature of this study is its attempt to estimate the cumulative costs and impacts of meeting a combined set of EPA requirements, and to determine whether they will place a significant bur den on the fiscal c onditions of local g overnments, and r equire them to significantly increase existing charges for improve d environmental services.
Please find attac hed a copy o f the draft study and appendices f or your rev iew. This version of the re port reflects Ag ency comme nts received on a n earlier draf t. We expect that this version of the report, with some additional minor modifications, will be the final version. Your comments w ill be usef ul in prepar ing the fina l version o f the report and discus sing the fin dings of the reports with the public.Page C-10 Peer Review Handb ook To assist in your review of the report, we ask that y ou pay particular attention to the following questio ns: 1) Do state and loca l governments an d financial mark ets consider house hold costs (measured as a percent of household income) and selected municipal financial information (debt service to general revenues or taxable property values) when evaluating the ability of enterprise sy stems and municipalities to issue bonds or obtai n loan s? 2) Having selected a series of financial indicators, are the criteria used in the anal ysis acc epta ble? 3) Do the results support our conclusions? What additional conclusions can be reac hed f rom th e ana lys is? 4) What modifications would you suggest be made to the recommendation section?Peer Review Handb ook Page C-11
CHARGE EXAMPLE 7 - Economics - V aluation of Fatal Can cer Risks Background The Science Advisory Board–Enviro nmental Economics Adv isory Committee (SABEEAC) review of t he draft Guidelines for Pre paring Economic An alyses (Guidel ines)helped the Agency to identify va lid and sound econo mic procedures t o use when conduc ting benefit-c ost analyses of environmental policies. Both the Guidelinesand the SAB-EEAC review (EPASAB-EEAC-99-020, dated September 1999) recognize that economic theory will evolve and new empirical lite rature will be fo rthcoming that ma y necessita te revisiting the analytic procedures contained in the Guidelines

. There may also be situations where the Guidelines do not provide sufficient detail to respond to analytic questions whose answers have broad implications for th e conduct of eco nomic analy sis at the EPA. Both circumstances h ave recently arisen, making it necessary for the Age ncy to submit for SAB-EEAC review a document that exam ines t he “be nefit trans fer” i ssues that a rise w hen us ing th e valu e of st atist ical l ife (V SL) literature that is based on accidental risks to estimate the economic benefits of environmental policies that redu ce fatal cance r risks.
The Guidelines provide informat ion and guidance on the valuation of reduced mortali ty ris ks ( Chapte r 7, pp. 37- 43)

. Th e Age ncy Guidelinesconclude - and we understan d the SABEEAC to have concurred in their review on this subject - that one practical and well-supported means to val ue cha nges i n mort ality r isks is to use th e Valu e of a S tatis tical Life (V SL) approach. Further, in response to the SAB-EEAC review, the Guidelines describe a number of important factors to consider in applying benefit transfer approa ches using VSL estimates from the empirical lite rature on wage -risk tradeoffs

. Recognizing tha t this is an important b enefit cat egor y, th e Age ncy Guidelinesstated that the EPA would “continue to conduct annual reviews of the risk valuat ion literature” a nd “reconsider and revise the re commendations in th ese guidelines accordingly.” Furthermore , the EPA would “seek advice from the Science Advisory Board as guidance recommendations are revised.” The Agency needs to return to the SAB-EEAC and obtain additional c ounsel on this subject. Some eco nomists within the g overnment have s uggested some particular appr oaches to dealing with the benefit-transfer issues. Since the Guidelines were drafted, a few r elevant articles have be en published that e xamine benefit tra nsfer issues surr ounding the use of VSL estimates when there is a passage of time (or latency pe riod) between the pollution exposure and harm, or when fa tal cancer risks are involve d. The importance of these issues wa s articulated in a recently proposed regulation to reduce human he alth risks from radon in drinking water. The proposed rule estimated the number of reduced fatal cancers resulting from diffe rent regulatory options. The Agency presented information on the economic values for the reductions in fatal cancer risks, a long with other qu antified benefi ts. A brief discuss ion of some of the be nefit transfer issues involved in this estimation was published in the preamble to the proposed rule for setting standar ds for exposure to r adon from drinking water sources ( Federal Register,Page C-12 Peer Review Handb ook November 2, 1999 volume 64, Number 211, pages 59245-59378). Quoting from the Federal Register notice requesting SAB review: Latency is one of a number of adjustments or fa ctors that are related to an evaluation of potential benefi ts associated with this rule, how those benefits are ca lculated, and whe n those economic benef its occur. Other f actors which may influence the es timate of economic benefits associated with avoided cancer fatalities include (1) a possible “cancer premium” (i.e., the additional value o r sum that people ma y be willing to pay to avo id the experience s of dread, pain and suffering , and diminished qua lity of life a ssociated with ca ncer-related illness and ultimate fatality); (2) the willingne ss of people to pay more over time to a void mortality risk as their income rises; (3) a possible premium for accepting involuntary risks as opposed to voluntary assumed risks; (4) the greater r isk aversion of the general popula tion compared to th e workers in the wage-risk valuation studies; (5) “altruism” or the willingness of people to pay more to reduce risk in other sectors of the population; and (6) a consideration of health status and life year s remaining at the time of premature mortality. U se of certain of these factors ma y signific antly increase the present value estimate. EPA therefore believes that adjustments should be considered simultaneously. The Agency also believes that there is currently neither a clear consensus among economists about ho w to simultaneously analyze ea ch of these adjust ments nor is there adeq uate empirical da ta to support defin itive quantitative estimates for all p otentially significant adjustment factors. As a result, the primary estimates of economic be nefits presented in the analy sis of this rule rel y on the unad justed $5.8 million es timate. However , EPA solicits comment on whethe r and how to conduc t these potential a djustments to econo mic benefits estimates together with any rationale or supporting data commenter s wish to offer. Because of the complexity of these issues, EP A will ask the Scien ce Advisory Board (SAB) to c onduct a review of these benefits transfe r issues associate d with economic va luation of adjustme nts in mortality risks. In its analysis of the f inal rule, EPA will attempt to develop and present an analysis and estimate of the latency struc ture and associated benefits transfer issues outlined previously consistent with the recommendations of the SAB and subject to resolution of any technical limitations of the data and models.” (page 59326) In the process of responding to reviews prepared during deliberations on the proposed radon rule, the Agency found that the Guidelineslack sufficient d etail on how to fully evaluate and characterize the different risk attributes that are central to a complete understanding of the benefit-cost implications of this rule. For example, time can pass between the point of initial exposure to a carcinogen, the biological manifestation or onset of cancer in the body , the medical diagnosis of cancer, and death caused by the c ancer. During development of policies affecting cancer risks, suggestions have been made to discount the VSL estimate (i.e., $5.8 million recommended in the Guidelines) to account for latencies, or the delay in time between reduc ed exposure and when the cancer death would have occurred absent the exposure reduction. Others argued that a suitable approach for valuing benefits from reduc ed cancer risks mus t co nsi der s imu lta neo usl y all of the benefit tra nsfer factors r elated to valuing cancer risks to ensure a careful and full treatment of benefits. There is evidence in the economics litera turePeer Review Handb ook Page C-13 regarding many such factors (e.g., potential premiums ascribed to cancer risk reductions due to a higher willingness to pay to avoid the dread, pain and suffer ing, morbidity effects, and other features of cancer endpoints) that may sugg est introducing upward adjustments factors which offset any potential downward adjustments caused by accounting for cancer latency. I n addition, proponents argue that adjustments for the age of population at risk, income, altruism and other risk characteristics (e.g., controllability, voluntariness) can all have some potential influence on the value of a statistical cancer fatality (VSCF) and there fore need to be reflected in the quantitative benefit assessment. While developing the primary benefit estimates for reduced fata l cancer risks in the proposed radon r ule, questions arose regarding the implementation of adjustments for some factors, but not others. For example, would it ever be appropriate to adjust only for latency periods, and not ot her factors, in t he valuation of re duced cancer d eaths? To help answe r this and related questions regarding the valuation of cancer risks, the Ag ency seeks the SAB-EEAC’s counsel. We furthe r ask that yo ur guidance re flect the ty pical uncertain ties facing EPA economists, including those surrounding the underlying risk assessments, the prediction (or la ck thereof) of late ncy period s for cancers, a nd the risk chara cteristics assoc iated with the VSL approach. Therefore, t he Agency propos es to seek re view of a “whi te paper” and list of cha rge questions by the SAB-EEAC on the valuation and benefit transfer practices a rising in the calculation of the econo mic benefits of reduced fat al cancer risks. The Agency seeks SABEEAC review of the treatment and presentation of quantitative and qualitative information for these types of benefits. Numeric case studies are included in the white pa per, to both identify and present prospective approaches to address these issues. The results of the SAB-EEAC review of this document, and responses to the specific charge questions, will be considered by the Agency during future revisions to the Guidelines,consistent with the Agency’s commitment to credible and c onsistent economic analysis in s upport of the polic y making pr ocess. Charge Questions: As the Committee considers the charge questions, it is asked to keep in mind the differing situations relating to differing degrees of data ava ilability or uncertainty in key parameters. 1) Does the white paper accurately describe the e mpirical economic literature relevant to the be nefit transfer i ssues that ensue w hen using the VSL literature to estim ate t he VSC F in a bene fit-c ost an aly sis? 2) Does the white paper present the important risk and demographic factors that can affe ct be nefi t tran sfer appr oach es tha t use V SL es timat es fo r VSCF?Page C-14 Peer Review Handb ook 3) Does the white pap er accurately describe attemp ts in the economic li terature to mea sur e VS CF d ire ctl y? 4) There are two numeric case studies of environmental cancer risks developed for the white paper. Each presents r isk assessment info rmation that forms the basis for quantify ing the number of statistical canc er fatalities tha t will be reduced a s a consequence of a hypothetical proposed environmental policy

. The case studies are then used to illustrate the outcome of using direct measures of the VSCF and benefit transfer adjustments to VSL estimates in order to calculate the VSCF. a) Which of the valuation approaches applied to the case study designated a s ALPHA are va lid to use? Does this ca se study omit a ny credible alternative protocols for valuing reductions in fatal cancer risks for benefit-cost an alyses of e nvironmental prog rams? b) Which of the valuation approaches applied to the case study designated a s OMEGA are valid t o use? Does this case s tudy omit any credible alternative protocols for valuing reductions in fatal cancer risks for benefit-cost an alyses of e nvironmental prog rams? 5) Which economic methods illustrated with the case studies, or additional methods identified by the Committee under charge questi ons 4.a and 4.b, se rve as credible protocols for the Agency to use in representing qua ntitative data, qualitative information, and sensitivity analyses for the economic va lue of reduced fatal cancer risks re ported in benefit -cost analy ses?Peer Review Handb ook Page C-15
CHARGE EXAM PLE 8 - Huds on River PCB s Site Reas sessment RI/ FS Modeling A pproach Charge: Pe er Revie w 1 Members of this peer review will be tasked to determine whether the models being used to support the decision-making process for the Reassessment, and the assumptions therein, are appropriate. The peer reviewers will base their assessment on the review the Preliminary Model Calibration Report (PMCR), an updated Technical Scope of Work for the Baseline Modeling Report (Appendix B of the PMCR) and the responses to selected comments received from stakeholders dur ing the public co mment period on the PMCR. In October 1996, EPA released the Preliminary Mode l Calibration Report (PMCR), which described the models, datasets and assumptions being used as part of the Hudson River PCB Reassessment RI/FS. The PMCR represents the status of the preliminary PCB modeling effort as of Fall 1995. Datasets, database corrections and other pertinent information which became available after October 1995 were not incorporated within the fate and tr ansport modeling presented in the PMCR. The PMCR was an interim document prepared to describe work in progress and was not intended to be a conclusive report. In particula r the HUDTOX model presented in the PMCR was not intended to be used as a predictive tool to assess remedial action scenarios. In addition, while time-varying mechanistic models of bioaccumulation will be used along with other models to predict fish body burdens, these models are not described in the PMCR. The PMCR was not formally peer reviewed at the time of publication, but was distributed to interested parties who were invited to submit comments and questions. Written responses were made to all of these comments and questions. In addition, the work plan contained in Appendix B of the PMCR has been revised to reflect the ongoing work being conducted as part of the Baseline Modeling effort. Results from this effort will be presented in a Baseline Modeling Report that will be formally peer reviewe d. The peer reviewers are requested to determine whether the models being used to support the decision-making process for the Reassessment RI/FS, and the assumptions therein, are appropriate. Th e peer reviewe rs are not being asked whether th ey would con duct the work in the same manner, only whethe r the work being conducted will y ield scientifica lly credible conclusions. It is sugge sted that the revie wer first read t he PMCR. The Response s to Comments provides information on the context of the PMCR within the overall modeling effort and additional details beyond the PMCR results. The current work plan as revised in June 1998 reflects the ongoing Baseline Modeling effort and revisions to some of the orig inal modeling tasks proposed in A ppendix B of the PMCR. I n addition, the USEPA /TAMS Phase 2 datab ase has been considerably revised. New datasets have be en added and some earlier datasets have been extensively revised.Page C-16 Peer Review Handb ook The peer revi ewers are ask ed to comme nt on the fo llowing: 1) Is EPA using appropriate models, datasets and assumptions on which to base a scientifically credible decision? 2) Will the models, with the associated datasets and assumptions, be able to answer the following pr incipal study questions as state d in the PMCR? a) When will PCB levels in the fish population recover to levels meeting human health and ecological risk criteria under No Action? b) Can remedies other than No Action significantly shorten the time required to achieve acc eptable risk leve ls? c) Are there contaminated sediments now buried and effectively se questered from the foo d chain whi ch are likel y to become “react ivated” fol lowing a major flood, resu lting in an incre ase in contaminat ion of the fish population? 3) Specific question s: a) Are the modeling approaches suitable for developing quantitative relationships betw een external for cing functions ( e.g., hyd raulic flows, solids and PCB loads, sediment initial c onditions, etc.) a nd PCB concentrations i n the water column , sediments and fis h? Are the models adequate for discriminating between water-related and sediment-relate d sources of PCBs? b) Are the spatial a nd temporal scale s of the modeling a pproaches adeq uate to answer the principal study questions? If not, what levels of spatial and temporal resolution are required to answer these questions? What supporting data are required for calibration/ validation of these spatial and temporal scales? c) It is contemplated that PCB concentrations in fish will be estimated using several modeling approaches: an empirical probabilistic model derived from Hudson River data, a steady state model that takes into account mechanisms of bioaccumulation body burdens, and a time-vary ing mechanistic model (not included in the PMCR). A bi-variate statistical model may al so be used to provid e insight into acc umulations. This multi

model approach is being contemplated because of the uncertainties associated with any individual model. Is this a reasonable approa ch or should predictions be made using a s ingle “best” mod el?Peer Review Handb ook Page C-17 d) Is the level of process resolution in the models adequate to answer the principal study questions? If not, what processes and what levels of resolution are r equired to answe r these questions? What supporting data (such as data to support specifications of a mixed depth layer, solids and scour dynamics, groundwater inflow, etc.) are required for these processes and le vels o f resoluti on? e) The results of the modeling effort will be used, in part, to support human and ecologica l risk assessments. In your judgment, will the models provide estimates adequate for this purpose? 4) Are there any changes to the work ef fort outlined in the revised work plan that would significantly improve the outcome? 5) In terms of evaluating the overall and specific effec ts and behavior of PCBs in the Hud sonRiv er, a re there any s eri ous flaws i n th e mo del ing a ppr oach (th eory, structure, physical parameters, etc.) that would limit or invalidate any conclusions or further work based upon the re sults of these mode ls?
Recommendat ions Based on your reading and analy sis of the information provided, please identify and submit an explanation of your overall recommendation for the modeling effort for the H udson River PCB Reassess ment RI/FS: 1) Acceptable as i s 2) Acceptable with minor revision (as indicated) 3) Acceptable with major revision (as outlined) 4) Not acceptable (under any circumstance)Page C-18 Peer Review Handb ook
CHARGE EXAMPLE 9 - IRIS Pilot Program - Instructions to Peer Reviewers for Reviewing IRIS Summaries and Supporting Documentation

The U.S. EPA is condu cting a peer re view of the scien tific basis support ing the health hazard and dose response assessments for the subject chemical that will appear on the Agency ’s online database, the Integrated Risk Informa tion System (IRIS). Materials to be revie wed include the summary information that will appear on IRI S (the inhalation reference concentration [RfC], oral reference dose [RfD], and cancer assessment) and the supporting document, the Toxicological Review, which will also be made available to the public. A listing of Agency Guidelines and Methodologies that were used in the development of these hazard and dose-response assessments included the following: The Risk Assessment Guidelines (1986), the (new) Proposed Guidelines for Carcinogen Risk Assessment ( 1996), Guidelines for De velopmental Toxicity Risk Assessment, (p roposed) Int erim Policy f or Particle Size and Limit Concentration Issues in Inhalation Toxicity, ( proposed) Guidelines for Neurotoxicity Risk Assessment, Methods for Derivation of Inhalation Reference Concentrations and Application of Inhalation Dosimetry, Recommendations for and Doc umentation of Biological Val ues for Use in Risk A ssessment and Use of the Benchmar k Dose Approach i n Health Risk Assessment. Copies of these documents (and/or their relevant Sections) will be made to the revie wer upon reques t. Peer review is me ant to ensure that science is used c redibly and appropriately in derivation of these dose-response assessments. You have been chosen as an expert on the chemical under c onsideration, on a scientific discip line related to at least one of the a ssessments, or in the field of r isk assessment. At l east three peer reviewers per chemical are be ing chosen to review the scientific basis of these draft dose-response assessments before they are forwarded on to the EPA's Consensus Process for final approval and adoption by the EPA. These hazard and dose-response assessments will then appear on IRIS and become available as Agency consensus health effect information. The primary function of the peer reviewer should be to judg e whether the choice, use, and interpretation of data employed in the derivation of the assessments is appropriate and scientifically sound. This review is not of the recommended Agency risk assessment guidelines or methodologies used to derive cancer or RfD/C assessments as these have been reviewed by external scientif ic peers, the pub lic, and EPA Scienc e Advisory Boards. The revie wer’s comments on the application of these guidelines/methodologies within the individual assessments is, how ever, welcomed and encourag ed. For example, t he reviewer ma y ascerta in whether or not there is data sufficient to support use of other than default assumptions for areas such as sensitive subpopulations or linear cancer extrapolation. The reviewer may a lso have opi nio ns o n other a reas of uncer tai nty s uchas s ubc hro nic to c hro nic dur ati on ( whe n on ly a subchronic study is available) or an incomplete data base but should focus on the specific a rea of uncertainty rather than on the magnitude of the overall estimate.Peer Review Handb ook Page C-19 Below are two groups of questions regarding this review. The first is a set of general questions that are meant to guide you through y our review. It is not imperative that you specifically answer each question of this group. The sec ond group of questions, however, are specific for the chemical assess ments and deal with areas of scient ific controvers y or uncer tainty in which the Agency may have to make a scientific judgment. Your input to this set of questions is considered vita l to the review pro cess. Questions for IRIS Peer Reviewers -General 1) Are you aware of any other data/studies that are relevant (i.e., useful for the hazard identification or dose-response assessment) for the assessment of the adverse health e ffects, both can cer and noncanc er, of this chemic al? Please exp lai n fu lly. 2) For the RfD and RfC, has the most appropriate critical effect been chosen (i.e., that adverse effect appearing first in a dose-response continuum)? For the cancer assessment, are the tumors observed biologically significant? relevant to human health? Points relevant to this determination include whether or not the choice follows from the dose-response assessment, whether the effect is considered adverse, and if the effect (inc luding tumors obse rved in the canc er assessment) and the species in which it is observe d is a valid model fo r humans. 3) Have the noncan cer and cance r assessments bee n based on the most a ppropriate studies? These studies should present the critical effect/cancer (tumors or appropriate precursor) in the clearest dose-response relationship. I f not, what other study (or s tudie s) sho uld be chose n and why ? 4) Studies included in the RfD and RfC under the heading "Supporting/Additional studies" are meant to lend scientific justification for the designation of critical effect by including any relevant pathog enesis in humans, a ny applicab le mechanistic infor mation, any e vidence corrob orative of the cr itical effect, o r to establish the comprehensiveness of the data base with respect to various endpoints (such as reproductive/developmental toxicity studies). Should other studies be include d under the "Suppo rting/Additional" category ? Should some studies be removed? 5) For the noncanc er assessments, a re there other d ata that should be c onsidered in developing the uncertainty factors or the modify ing factor? Do you consider that the data support use of different (default) values than those proposed? 6) Do the Confidence statements and weight-of-evidence statements present a clear rationale and accurately reflect the utility of the studies chosen, the relevancy of the effects (cancer and noncancer) to humans, and the comprehensive ness of thePage C-20 Peer Review Handb ook data base? Do these statements make sufficiently apparent all the underly ing assumptions and limitations of these assessments? If not, what needs to be added? Questions for IRIS Pe er Reviewers - Chemical Specific [example: cumene] 1) Based on the information noted in the Principal study currently designated (Cush man e t al., 1995) is the disco untin g of t he re nal effec ts in m ales justi fied ? Is sufficient rationale given to let stand the organ weight c hanges in female rats as a critical effe ct? 2) Is the information in the Toxicological Review sufficient to consider cumene as hav ing a lowpot ent ial forcaus ing r epro duc tiv e effect s? P lea se ex pla in f ull y. RECOMMENDATIONS Based on your reading and analy sis of the information provided, please identify your overall recomme ndation for the I RIS materials y ou have reviewe d as 1) Acceptable as i s 2) Acceptable with minor revision (as indicated) 3) Acceptable with major revision (as outlined) 4) Not acceptablePeer Review Handb ook Page C-21
CHARGE EXAMPLE 10 - Charge to Rev iewers for the WTI Dr aft Final Risk Assess ment
The draft final WTI risk assessment is divided into several volumes covering the scientifi c discipli nes of tox icology, envir onmental fate and tran sport, com bustion en gineering, atmospheric modeling, exposure assessment, ecological risk assessment, and accident analy sis. As a reviewer of the WTI draft final risk assessment, y ou should use your best technical knowledge and professional judgment to comment on the technical accuracy , completeness and scientific soundn ess of the assessm ent. Each revie wer is asked to fo cus on several sp ecific issues in his or her area of expertise with comments on other areas invited but optional. Your comments will be co nsidered in final izing the risk asse ssment. For the peer re view workshop re viewers will be or ganized into 5 wor k groups: Combustion Enginee ring, Air Disper sion and Deposition Modeling and Ac cident Analy sis, Toxicology, Exposure Assessment, and Ecological Risk Assessment. All reviewers should be famil iar wi th th e Exec utiv e Summ ary (Vol ume I) a nd the Facil ity Back groun d (Vol ume II) Sections of the dra ft risk assessment

. In addition, e ach work group should focus on spe cific Volumes as specified below: Workgroup Risk Assessment Volumes Combustion Engineering Volume II I - Facility Emissions Air Dispersion and Volume IV - At mospheric Volume VII - Accident Deposition Modeling and Dispersion and Deposition Analysis Accident Analy sis Modeling
Toxicolo gy Volume V - Human He alth Risk Assessment Volume VII - Accident Analysis Exposure Assessment Volume V - Human He alth Risk Assessment Volume VII - Accident Analysis Ecological Risk Assessment Volume VI - Screening Ecological Risk Assessment While reviewing these Sections of the document, please address the following general issues.Page C-22 Peer Review Handb ook 1) Comment on the organization of the risk assessment document. Does the layout follow a logical format? Is the presentation of information in the document clear, concise and easy to follow? 2) Does the executive summary accurately reflect the data and methodologies used and the conclusio ns derived in the r isk assessment? 3) Were the major re commendations of t he 1993 peer rev iew workshop for the risk assessment plan addressed? 4) As with any risk assessment, there are alway s additional data and method development efforts that could be undertaken to reduce the level of uncertainty

. However, are there any major data or methodolog ical gaps that would preclude the use of this risk assessment for decision making? If so, how should they be addressed? 5) What long-term re search would y ou recommend that could improve risk assessments of this type in the future? In addition, the following workgroup specific issues should be addressed.
Emissions Characterization Emissions characterization includes identification of substances of concern and the development of emission rates for these contaminants. Emission rates were developed through a combination of site specific stack test data and models. Please comment on the following issues with respect to thi s aspect of the dr aft risk assessme nt. 1) To characterize the nature of the emissions, waste stream profiles were developed and entered into a database. Several refinements and adjustments (e.g., the Subtraction Correction Factor for chlorinated compounds) were applied to the profiles before substances of concern were identified. Please comment on whether or not these adjustments are appropriate. What is the anticipated effect on the risk assessm ent? 2) Comment on the selected chemicals of concern. Have important chemicals been missed due to the selection technique? 3) Comment on the approaches used to estimate stack emission rates (e.g., use of the 95% UCL of the a rithmetic mean or the maximum detecte d value, whichev er is smaller, for high end em ission rates). Ar e the approache s appropriate? Are their effects on the risk assessment adequately characte rized? Comment on thePeer Review Handb ook Page C-23 adjustment made to PCDD/PCDF emission rates to account for brominated dioxin-like compounds. Also, comment on the approach to characterizing emission rates from fugitive sources (e.g., use of the TANKS 2 model for the Carbon Adsorption Bed). 4) Comment on the identified sources of fugitive emissions. Was the approach used to select these sources appropriate? Have important sources been missed? Have emissions from process upsets been given appropriate consideration? 5) There have bee n a number of cont rolled burns at the WTI facility

. Please comment on the ade quacy of th ese data in estima ting potential exposure. Please comment on the assumptions made from the tests in regard to composition of wastes received at WTI and emissions when the plant operates in the future. 6) Comment on the use of emission factors from coal burning to estimate the emission rate of fly ash from WTI. Are the fac tors used to adjust the coal emission rate appropriate? Are the uncertainties introduced from this approach adequately characterized? 7) Overall, is the identification of the key assumptions used in characterizing the nature and magnitude of emissions thorough? Are the magnitude and direction of effect of these assumptions on the ov erall risk assess ment accuratel y characterized ? Is the uncerta inty and var iability inhe rent in this analy sis adequately discussed? Does the sensitivity analy sis cover the major parameters expected to have a n effect on the r isk assessment? Dispersion and Deposit ion Modeling To develop this risk assessment, compu ter models have be en used with site spe cific data on emission rates and meteorological conditions to simulate the air concentrations and deposition rates for contaminants potentially emitted from the WTI facility

. The models used include the Industrial Source Complex - Complex Terrain Deposition (ISC-COMPDEP), the CALPUFF, an d the INPUFF models. In y our review, plea se address the fo llowing issues. 1) Since the 1993 peer review of the risk assessment plan, a number of efforts have been completed to reduce the uncertainty associated with the air dispe rsion and deposition modeling. These efforts include the collection of site-specific data for emission rates and meteorological conditions. Also, a wind tunnel study was conducted to evaluate the effects of the complex terrain surrounding the WTI facility. D oes the risk assessment document ade quately summ arize these activities? Is the link between these data collection efforts, the air dispersion models, and the risk assessment clearly established?Page C-24 Peer Review Handb ook 2) The results of 12 sets of sensitivity tests indicate that geophysica l variables (e.g., terrain) are more likely to affect dispersion and deposition than emission vari ables (e.g., s tack tempe ratu re). Were these sensi tivit y an aly ses a dequ ate? Comment on the conclusions reached. To further examine the effect of geophysic al variables, win d tunnel testing w as conducted to mo del the terrain induced flow effects expected near WTI. It wa s concluded that changes in peak concentrations a ttributed to these e ffects are re latively mino r and that the ISCCOMPDEP model is sufficiently conservative. Comment on this conclusion. Have these anal yses helped to characterize and/or reduce th e uncertainty in the air dispe rsion model ing as socia ted wi th th e comp lex t errai n surr oundi ng WTI. 3) The ISC-COMPDEP model does not allow for non-steady state conditions such as calm winds and strong temperature inversions. Therefore, CALPUFF was used to estimate air dispersion and deposition under these conditions. However, CALPUFF ga ve similar peak, 2 4 hour, and annua l average con centrations as ISCCOMPDEP. Comment on the adequacy of this analysis. Comment on the conclusions reached. Has this analysis helped to characterize and/or r educe the uncertainty in the air dispersi on modeling assoc iated with non-ste ady state meteorologica l conditions? 4) Atmospheric dispersion modeling was used to estimate air concentrations of hazardous chemicals for the accident analysis. The SL AB model was used for vapor releases from spills and the m ixing of incompatib le wastes. ISC

COMPDEP was used for releases associated with fires. Comment on the selection of the mo dels and inputs. Ar e they appr opriate selectio ns? 5) Overall, have adequate sensitivity tests been conducted to demonstrate the magnitude of variation in concentrations and deposition estimates with model inp uts ? Pl ease exp lai n fu lly.
Human Health Risks Human Health Risk Assessment includes hazard identification, dose-response evaluation, exposure assessme nt, and risk chara cterization. To de velop the risk asse ssment, potentially exposed populations have been identi fied and the mag nitude, frequen cy, and dur ation of their exposure quantified. This information was then integrated with the hazard identification and dose response ev aluation for the r isk characteri zation. For this risk assessment, both carcinogenic and non-carcin ogenic health e ffects have bee n evaluated. I n your revie w, please comment on the follo wing issues.Peer Review Handb ook Page C-25 Exposure 1) EPA's Exposure Assessment Guidelines identify certain exposure descriptors that should be u sed to chara cterize ex posure esti mates. The G uidelines define high end exposure estima tes as those repr esenting individu als above the 90th percentile on the exposure distribution but not higher than the individual in the population who has t he highest exposur e. Bounding expos ure estimates ar e those that are higher than the exposure incurred by the person in the population with the highest exposure

. Central tendenc y exposure e stimates are def ined as the best representation of the center of the exposure distribution (e.g., arithmetic mean for normal distributions). Comment on whether or not the WTI exposure assessment properly c haracterizes e ach of the exposur e estimates in ter ms of these descriptors. 2) The factors that go into estimating a central tendency or high end exposure, once the population has been defined, include the environmental media concentration, the intake rate, and the duration and/or frequency of e xposure. Comment on whether or not the WTI exposure assessment does an adequate job of describing the logical procedure of combining t hese factors to develop central ten dency, high end, and/or bounding estimates of exposure for each of the exposed subpopulations. 3) An important factor in an exposure assessment is identifying all of the important exposure sources
. Please comment on the adequacy of the WTI asse ssment in identifying the important sources and pathway s of exposure. 4) Have the key assumptions for estimation of chemical concentration and for estimation of exposure been identified? Are the magnitude and direction of effect corr ect f or the assum ption s that have been ident ified ? 5) Supposedly, c onservative assu mptions have been applied in this asse ssment to account for uncertainty. Are the conservative a ssumptions appropriately factored into the ultimate characterization of what descriptor best applies to each exposure estimate? Please comment on whether the uncertainties were confronted in an adeq uat e manne r. If the y were not , pl ease sta te w hatsho uldbe d one dif fere ntl y. Hazard Identification/Dose Response and Risk Characterization 1) To select surrogate compounds for quantitative risk assessment, a two step process was used in which chemicals were ranked on the basis of emission rate, toxicity (both c ancer and non-c ancer), and bio accumulation pote ntial. Please comment on this sele ction process. Ar e the ranking f actors appropri ate? CouldPage C-26 Peer Review Handb ook important compounds have been omitted from the analysis based on the ranking procedure? 2) For the majority of the chemicals of concern, tra ditional approac hes to dose response evalua tion were employ ed (e.g., use o f a slope factor for cancer an d use of a RfD/RfC for non-cancer). However for certain chemicals or g roups of compounds a diffe rent methodology was used. Specifi cally, dioxins, furans, PAHs, lead, mercury, nickel, chromium, acid gases, and par ticulate matter were given special c onsideration. Plea se comment on the me thodology u sed for these compounds. Was it appropriate? Have the uncertainties associated with the methodology been adequately characterized? Comment on the assumptions used due to a lack of chemical specific data. 3) Please comment on the selection of the overall population and the various subpopulations at r isk. Were site spec ific data, such a s the informal home gardening sur vey, prope rly utilized to identify the se subpopulations? 4) It is stated i n the risk a ssessment that average r isk estim ates are base d on average emission rates, average air dispersion/deposition within a subarea, and typica l exposure factors

. Further, maximum r isks are based on average emiss ion rates, typical exposure factors, and the maximum air concentration within a subarea. Please comment on t his use of the terms average and m aximum risks. Are the se descriptive term s appropriate g iven the paramet ers used to derive each? Please exp lai n fu lly. 5) Comment on whether or not the non-cancer risks of chemicals of concern have been adequatel y addresse d by the risk a ssessment? For example , has an adequate discussion of endo crine disrupter s been provided w hich either char acterizes their risks or clearly explains why their risks cannot be c haracterized? Further, have non-cancer ch ronic toxicities of d ioxins and furans be en adequately addressed in the risk assessmen t? 6) Please comment on w hether or not the u ncertainties ass ociated with the a dditivity and/or synergy of risks from pollutants emitted together from the WTI facility are adequately discussed in the ri sk assessment. 7) Have the key assumptions for estimation of dose and risk been identified? Are the magnitude and direction of effect correct for the assumptions that have been identified? Please comment on whether the uncertainties were confronted in an adeq uat e manne r. If the y were not , pl ease sta te w hatsho uldbe d one dif fere ntl y.Peer Review Handb ook Page C-27 8) Please comment on the overall adequacy of the risk charac terization. Does the risk characterization include a statement of confidence in the risk assessment including a discussion of the major uncertainties. Are the hazard identification, dose-response assessment, and exposure assessment clearly presented? Have sufficient risk descriptors which include important subgroups been presented and discussed? Screening Ecologic al Risk Assessment As with the human health risk assessment, the ecological risk assessment pulls together elements of exposure analysis and dose-response evaluations to develop a risk chara cterization. For the Screening level Ecological Risk Assessment (SERA), Ecological Chemicals of Concern (ECOC) and indicator species have been identified to provide conservative estimates of risk. Please address the following issues in your review. 1) Are there any components of the S ERA which yo u feel undermine the scientific validity of the assessment? If so, what are they and can you provide suggestions to strengthen the identified compon ents? 2) Is the organization of the document clear and does it present the material in a clear and concise manner consistent with the Framework for Ecol ogical Risk Assessment (EP A, 1 992 )? P lea se ex pla in f ull y. 3) Uncertainties a re discussed in num erous sections of the SERA and compose Section VIII of the SERA. I n each case, do these discussions cover all relevant and important aspe cts of the uncert ainties which y ou think should be ad dressed in the SERA? 4) In your opinion, what is the weakest and what is the stronge st aspect of the SERA? Can you make any suggestions on how the weake st parts can be str engt hen ed b y the Agen cy? 5) In Section II, are the stre ssors, ecological effects, and both the assessment and measurement endpoints adequately characterized? Are the five emission scenarios adequate to characterize the exposures for the WTI facility ? Are there other emission scenarios which you think should be included in the SERA? 6) In Section III , is the site characterization adequate to support the SERA? Why or why not? 7) In Section IV, is the tiered process used to identify the ecological chemicals of concern (ECOC) from the initial list of potential chemicals consideredPage C-28 Peer Review Handb ook scientifically defensible? Does application of this tiered approach support the statement made in the SERA "by focusing on the potential risk from the selected ECOCs, the SERA provides a thorough screening-level evaluation for the WTI faci lity ?" 8) In Sections V a nd VI, are th e exposure and ec ological effe cts adequately char acte rized ? Are t he mos t appr opria te es timat ion te chniq ues a vaila ble us ed? Are t he as sumpti ons cle arl y sta ted ? Pl ease exp lai n fu lly. 9) In Section VI II, are there any m ajor elements miss ing from the risk characterization which you think need to be included or which would strengthen the risk characterization? Does the risk characterization support the summary and conclusions presented in Section IX? 10) In Section IX, given the assumptions made and the processes use d to select and evaluate chemic als, receptors, and exposure pathw ays, do y ou think the SERA adeq uate ly m et its objec tive o f not i nadv erte ntly unde rest imati ng risk? Accident Analysis The Accident Analysis for the WTI incinerator involves e valuating the probability of an emergency incident occurring whic h results in the release of hazardous waste. The consequences of this release are also evaluated us ing exposure and h uman health effe cts information. Unlike the human health risk assessment which has a primary goal of quantify ing risks, the accident analysis typically provides information that can be used to reduce the likelihood, extent a nd impact of possibl e accidents. Plea se comment on the f ollowing issues in your revie w of this aspect of the risk assessmen t. 1) The WTI accident assessment selected five scenarios for quantitative evalua tion that were consid ered to be of prim ary conce rn. The scenari os are an on-site spill, an on-site fire, an on-site mixing of incompatible waste, an off-site spill, and an off-site spill and fire. Please comment on the selection of these scenarios. Were any significant scenarios missed? 2) Specific chemicals were selected to evaluate each scenario. Please comment on the selections. Would other chemicals have been more appropriate? 3) Chemical specific release rates are calculated for each sce nario. Please comment on the procedures used to estimate the release rates. Was an appropriate approach used?Peer Review Handb ook Page C-29 4) Atmospheric dispersion modeling was used to estimate air concentrations of hazardous chemicals. Specifically, the SLAB model wa s used for vapor releases from spills and the mixing of incompatible wastes. ISC-COMPDEP was used for releases associated with fires. Comment on the selection of the models and inputs. Are they appropriate selections? Should other models or inputs been used? 5) Please comment on the assessment's conclusions on the severity of consequence s and probability of occurrence. Has the report cor rectly categorized the severity of the consequences of the different accident scenarios? Has the assessment adequately justified the reported probability of occurrence of each of the accident events? 6) Key assumptions were made in the identification of accident scenarios and the description of the conservative and typical events. I ncluded were a description of the magnitude of the effect of th e assumptions and di rection of the ef fect. Please comment on the assumptions. Are they justified? Are the descriptions of the magnitude and directions of the effects correct? Has the accident assessment adequ ately co nfron ted th e unce rtain ties i nvolv ed in d oing t his t ype of an alysis ? If not, what else should be done? 7) Comment on the appropriateness of using IDLH values for c haracterizing the severity of consequences in the accident analy sis. Comment on the appropriateness of using 10 X LOC for chemicals for which I DLH values have not been established. 8) In the accident analysis, I DLH (or 10 X LOC) values were used to deter mine the downwind distances over which adverse human health effects might occur. To evaluate the uncertainty introduced by using the IDLH, a sensitivity a nalysis was conducted where these distances were recalculated using the L OC (a more stringent health criteria). Other sources of uncertainty that are identified in the accident analy sis include conce ntration averag ing times, chemic al concentratio ns, emission rates, an d meteorologica l conditions. For most of these param eters it is stated that conservative assumptions were used to avoid underestimating risks. Have the uncert ainties inherent i n the accident an alysis been adequately characterized? For those parameters where sensitivity analy ses were not conducted, is the conclusion that conservative assumptions have avoided underestimation valid?Peer Review Handb ook Page D-1


# APPENDIX D - GUIDANCE ON REQUESTING A REVIEW BY THE US EPA SCIENCE ADVISORY BOARD (SAB)Page D-2 Peer Review Handb ook
Guidance o n Requesti ng a Revie w By the US EPA Scienc e Advisory Board (SA B) Summary The US EPA Science Advisory Board (SAB or the “Board”) ann ually solicit s proposals for review projects every spring for the f ollowing fiscal year. This appendix provides guidance to Programs and Regions to help them submit requests for SAB reviews. Requests should be submitted to the SAB in both hard copy and electronic versions usually by mid-June for the following fiscal year. The requests may be part of the annual submissions that respond to the peer review a ctivities of the Agency, or t hey may be submitt ed directly to t he Board. Alt hough providing requests at one defined time in the spring helps with SAB planning, we recognize that projects also com e to light during other parts of the year. Plea se contact the SAB staff for detail s on making submissions during the remainder of the year (see e nd of this document for contacts). Background A key priority for the Administrator is to base Ag ency actions on sound scientific data, analyses, a nd interpretation s. The Administrato r issued the Age ncy's Pee r Review Policy to increase the quality of the technical foundations upon which EPA's reg ulatory structures are built. The SAB is a k ey scientif ic peer review mechanism availa ble to Programs a nd Regions in implementing the P eer Review Policy

. However, beca use the Board ha s finite resource s it cannot conduct all reviews. This document is designed to help Programs and Regions determine which projects to submit to the SAB. Note particularly that the SAB focuses on the technical underpinnings of Agency positions; i.e., risk assessment issues, in contrast to risk management issues. The topics that are best suited for the Board's agenda are those that satisfy several of the following criteria: 1) Integrate science into Ag ency actio ns in new way s. 2) Influence l ong-term techn ological develo pments. 3) Impact overall environmental protection. 4) Address novel sci entific problems o r principles. 5) Address problems that transcend federal-agency or other organizational boundaries. 6) Strengthen the A gency's basic capabiliti es. 7) Serve Congress ional or other lea dership interests

. 8) Deal with controv ersial issues. In suggesting issues for SAB involvement, Programs and Reg ions should note the breadth of SAB a ctivities:Peer Review Handb ook Page D-3 Historically, most of the outputs of the Board are in the form of "full" re ports. They prese nt the findings of peer reviews of Agency d ocument(s) and c ontain considerable de tail about the findi ngs and recomme ndations of the Bo ard. They a lso address the spec ific questions pose d by the Char ge to the Board

. "Letter" reports fulfill the same function as reports, but are simply shorter in length. Due to the need to be mor e responsive with a dvice, the Boar d has recently begun to produc e more short lette r reports than full reports, as they can be produced and finalized in less time.
The SAB has also in troduced the "Co nsultation” as a me ans of conferri ng--in public session--with the Agency on a technical matter befor e the Agency has begun substantive work on that issue. The goal is to leaven EPA's thinking on an issue by brainstorming a variety of approaches to th e problem very early in the development process. There is no attempt or intent to express an SAB consensus or to generate an SAB report. The B oard, via a brief letter simply notifies the Adminis trator that a Consultation has taken place. More recently , the Board intro duced a new vehi cle for communic ating with its clients -- the "Advisory” -- which provides, via a formal SAB consensus report, critical input on technical issues that arise during the Agency's issue development process. The Advisory g enerally i nvolves a review of a multi-ye ar Agency project. The inte nt is to provide some mid-c ourse assessment to see if the Age ncy is head ing in a scientif ically credible direction. In order to maintain an objective, arms-length relation with the Agency and its projects, the SAB review of the final pr oduct at some point in the future wil l in clu de ex pert s wh o di d no t parti cip ate in p roduci ng th e Advis ory. The Agenda Setting P rocess Each Assistant Adm inistrator and Reg ional Administrator is normally a sked to submit a list of candidate topics for SAB action/review. A "project sheet" (see attached e xample) is used to define each topic that is nominated for SAB review. The project sheet is prepared by the requesting office and contains the following information: 1) Project title/subject (Descriptive sho rt title of projec t) 2) Requesting Organization/Office (Primary of fice requesting review, AA/RA le vel) 3) Requesting Official(Name and position of senior official requesting review, usually office or division level - this is the person who may re ceive a summary briefing from the Chair following the review) 4) Program Contact (Name/phone number/mail code - this is the principal contact for SAB Staff to interact with during development of the SAB review)Page D-4 Peer Review Handb ook 5) Background (brief history of the project an d why it is impor tant) 6) Tentative C harge(what the SAB is being asked to comment on - usually a set of questions) 7) Tentative Schedule and Committee(when the review is expected to be conducted, e.g., Winter 2001; and which SAB committee is appropriate for the review -- final choice as to the review committee is at the discretion of the Board) 8) Budget Estimate (Rough estimates of Agency funding for the subject over the past 5 years (if applicable) and for the next 2 y ears (if applicable). The SAB Executive Committee has asked for this information to help it better apprec iate the level of Agency involvement in and commitment to the issue) 9) Preparer (name, phone, office of preparer of Project Sheet and the date prepare d) The proposed topic s will be examined an d discussed in a numb er of forums: 1) The individual SAB Committees- Throughout the late Spring and Summer, the SAB Committees will be examining options for the following fiscal year, inc lud ing a ll s ugge sti ons mad e by the A genc y. 2) The Science Polic y Council-Ste ering Committee (SPC-SC)- The SPC-SC usually meets in early summer to examine the proposals for ea ch fiscal y ear. The goal i s to provide cross-office critique/integration of the proposals. The SPC-SC will be used as a forum for continuing discussion throughout the process and throughout the year as new topics emerge. 3) The Deputy's perspective - In the summer, the Deputy Administrator will review the requests and p rovide insights on priorities. 4) The SAB Executive Committee- During its summe r meeting, the SA B’s EC will examine the nominate d topics, adding i ts own perspectiv e on an appropria te agenda, using its selection criteria. 5) The Administrator - In Septembe r, the list of proposed topics will be de livered to the Administrator f or information an d added insights. The completed project sheets should be submitted electronically to the SAB Deputy Staff Director (fowle.jack@epa.gov) and in signed hard copy (mail code 1400A).Peer Review Handb ook Page D-5 For general information, or to contact a specific SAB Staff member, please c all the SAB main phone line: (202) 564-4533. Please ask to be connected to the Staff person handling the major review area you wish to inquire about (e.g., human he alth and exposure, drinking water/water quality, ecology , engineering, modeling, air quality, economics, re search, and radiation).Page D-6 Peer Review Handb ook US EPA Science Advisory Board Proposed Project Project title/subject: Proposed Amendments to the Risk Assessment Guidelines for Carcinogens Requesting Organization/Office : Office of Research and Development (ORD) Requesting Official:Name, Title, Office/Organization Program Contact: Name, Ti tle, Offi ce/Orga nization, 2 02-260-xxxx Background: EPA's Health Risk Assessment Guidelines provide generic science and science policy guid ance on risk asse ssment issues for us e in all Agency offices. EPA has c urrently issued or proposed nine guidelines (or amendments) in this series, all of which have been submitted to the Science Advisory Board for review. The current guidelines for carcinogen risk assessment were re viewed by the SAB and issued as final guidance in 1986. In 1988, the Forum initiated a public process for considering amendments to these guidelines. A Risk Assessment Technical Panel considered submissions from the public as well as information developed by experts at two public workshops in revising these guideline s. Tentative Charge: Review the amended and expanded guidance, with special emphasis on (a) weight-of-evidence issues, (b) a new classification sy stem, (c) dose response modeling, and (d) the use of pharma cokinetic and met abolic data. A mor e detailed char ge will be nego tiated with SAB at a later date.
Tentative Schedule and Committee: Winter, 2001, Environmental Health Committee Budget: FY 1997 - $xxx and yy FTE FY 1998 - $xxx and yy FTE FY 1999 - $xxx and yy FTE FY 2000 - $xxx and yy FTE FY 2001 - estimated costs of $xxx and yy FTE each y ear Prepare r: Name, T itle, Off ice/Org anization , 202-260- xxxx Date: June 1, 2000Peer Review Handb ook Page E-1



# APPENDIX E - EXAMPLE STATEMENTS OF WORK FOR CONTRACTS

Users of this Peer Review Handbook need to be aware that the examples contained in this Appe ndix are generaliz ed statements of work prepared ( and in some cases m odified) to emp hasize certain im portant features ( e.g., attention to conflicts of interest, responsibilities of contractors or c ontracting officer s, development and use of the charge to peer reviewers). Please be sure that proper and currently a pproved contract la nguage is used in any EPA contract doc ument at the time of award (for example, see E PAAR 1552.212-71, alternate I).Page E-2 Peer Review Handb ook
STATEMENT OF WORK - EXAMPLE 1 - Statement of Work: Technical Review Contractor for P anel Review of Assista nce Agreeme nt or Fellowship Applic ations 1) Purpose The purpose of thi s contract is to pur chase peer re view services of a contractor wit h expertise in Exploratory Research; Environmental Chemistry

. The services are for peer reviewing ap plication s received i n response t o the Office of Research a nd Developm ent's (ORD's) 1997 Science to Achieve Results program. These reviews shall be completed and the evaluation sheets shall be prepared prior to the reviewer's participation in a 3-daypanel discussion to be held in Washington, D.C.on May 5 -7, 1997

. 2) Statement of Work ORD's National Center for Environmental Research and Quality Assura nce (NCERQA) is responsible for overseeing the recently expanded rese arch grants and fellowships programs. Each year NCERQA (alone or i n conjunction with o ther organizati ons) solicits applic ations in each of these pr ograms. The app lications to be rev iewed under this c ontract were su bmitted in response to the solicitation for the 1997 Science to Achieve Results program. As part of the selection process, NCERQA must conduct a peer review that is designed to evaluate the scientific quality of each application; this is accomplished through the ad hocuse of technical experts. The peer review services required by this contra ct necessitate the independent review of a maximum of 10applications and the preparation of a typedevaluation summar y and an ove rall rating for each of these applications. Each evaluation summary shall support and be consistent with the overall rating that is assigned; it also shall be completed prior to the contractor’s participation in the panel discussions. After the panel discussions for the applications assigned to the contractor, the contractor shall submit all completed evaluation summaries to the designated Science Review Administrator (SRA). The contractor also shall serve as the panel's rapporteur for approximately 6 - 8 of the applications assigned. As rapporteur, the contractor shall be responsible for preparing a typed evaluation summary (on-site typing support will be provide d by NCERQA) that reflects the panel's discuss ion of the respec tive application a s well as the pane l's overall rat ing (the criter ia for the panel's overall rating are the same as those for each pe er reviewer's overall rating). As rapporteur, the contractor shal l submit these pane l evaluation summar ies to the designa ted SRA prior to le aving the pan el meeting.Peer Review Handb ook Page E-3 Before the contractor shall be allowed to participate in the review process, the contra ctor shall have disclosed any actual or potential conflicts of interest and shall have signe d and submitted to NCERQA a Conflict of In terest/Confident iality Form. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer. Upon receipt of an approved Purchase Order (PO), NCERQA will send the following items to the contractor: a) A copy of the Purchase Order or the Purchase Orde r number b) The applications assigned to the contractor c) For grants, a set of abstracts for all the applications being reviewed by the panel d) For persons reviewing grant applications, a copy of the pertinent section(s) of the solicitation package to provide background information; for persons reviewing fellowship applications, information on how to access the solicitation package on the Internet e) A sample evaluation form to help the contractor prepare an acceptable evaluation form for each assigned application f) A blank evaluation form for each assigned application and the criteria for completing the form and determining the overall rating g) A blank and sample invoice as well as instructions for completing and submitting the invoice to EPA h) Information o n the point s of contact for additi onal infor mation (e. g., NCERQA's SRA) i) Logistics information on the location and time of the panel discussions NCERQA will transmit the above items under a cover letter. In this cover letter, NCERQA will provide additional details about each item, i ncluding (as ne eded) more spec ific instructions for the set of applications assigned to the reviewer. Each contractor shall be responsible for making his/her own travel reservations for hotel and transportation. 3) Reviewer Tasks a) Review the a ssigned app lications using the gui dance provi ded with NC ERQA's evaluation form. b) Submit completed evaluation forms to the SRA designated in the cover letter immediately following the panel discussions for the applications assigned to the contractor. THE COMPLETED FORMS MUST BE TYPED, AND THE EVALUATION SUMMARY FOR EACH APP LICATION MUST SUPP ORT AND BE CONSISTENT WIT H THE OVERALL RATING THAT IS ASSIGNED BY THE CONTRACTOR. IN SITUATIONSPage E-4 Peer Review Handb ook
WHERE THESE CONDITIONS AR E NOT MET, THE SRA WILL ASK THE CONTRACTOR TO REDO THE FORM. c) For those ap plication s for which t he contract or is servi ng as the pane l's rapporteur, submit a completed panel evaluation summary to the SRA designated in the cover letter prior to leaving the panel meeting. THE PANEL'S EVALUATION SUMMARY MUST BE TYPED (ON-SITE TYPING SUPPORT WILL BE PROVIDED BY NCERQA) AND BE CONSISTENT WITH THE PANEL'S OVERALL RATING. IN SITUATIONS WHERE THESE CONDITIONS AR E NOT MET, THE SRA WILL ASK THE CONTRACTOR TO REDO THE FORM. d) Make own airline and hotel accommodations for participation in the panel review meeting. Round-t rip air fare must be a commercial REFUNDABLE ticket.Peer Review Handb ook Page E-5
STATEMENT O F WORK - E XAMPLE 2 - Peer Re view of Prioritization Tool Report Work Assignment No. : Title : Peer Review of Prioritization Tool Report Work Assignment Ma nager (WAM): Name: John Q. Government Employee Address: Office of Solid Was te Phone No.: (202) 260-XXXX Background: The Waste Minimization Branch (WMB) in the Office of Solid Waste (OSW) is in the process of implementing the Waste Minimization National Plan, announced by the Agency on November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction over waste management, in keeping with the policy stated in the 1984 a mendments to the Resource, Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act (PPA). The Plan outlines major goals, objectives, and action items to achieve national reductions in the g eneration of haza rdous wastes. One of the objectives of the Plan is to: "develop a framework for setting national priorities; develop and distribute a flexible screening tool for identifying priorities at individual facilities; [and] ide ntify constit uents of concer n." This objective is a key build ing block in implementing subsequent objectives of the Plan. In September 1995, WMB formed the Waste Minimization Prioritization Team, which includes representatives from EPA regions and states, to implement this objective. The Team has worked to asse ss stakeholder ne eds for prioritiza tion tools and to eva luate prioritizatio n tools that are curre ntly availa ble. The Team pla ns to summarize this wo rk, along with its recommendations, in a report (referred to herein as the Prioritization Tool report) that would be available in draft form in July 1996. WMB and the Team wish to obtain independent peer review of the Prioritization Tool report prior to briefin g EPA management. The report i s being prepared with the su pport of ICF, Inc.; therefore, for the peer review to be consider ed independent, it must be performed by another contractor.Page E-6 Peer Review Handb ook Purpose and Scope of Work : The purpose of thi s work assignmen t is to provide suppor t to WMB and the Tea m in finalizing the Prioritization Toolreport by c onducting an inde pendent peer re view of the repor t. Work Statement : Task 1 -Management work plan and budget Within 15 days of CO approval of this work assignment, the contractor shall deliver a management wor k plan including a proposed level of effort, schedul e, and budget fo r all tasks. Task 2 -Provide independent peer review of Prioritization Toolreport The contractor shall provide support to WMB and the Team in preparing the Prioritization Toolreport by performing an independent peer re view of the report. The contractor shall establish a panel of peer reviewers including three senior-leve l persons who collectively have extensive expertise in particular areas to be identified by the WAM upon approval of the w ork assignment. Within three weeks of work assignment approval, receipt from the WAM of the necessary qualifications of peer reviewers ( in a TD), and receipt from the WAM of the peer review “charge” (in a TD), whichever comes latest, the contra ctor shall identify the three peer reviewers and prepare a memo that lists the names of the peer reviewers and their a ffiliations and includes the peer reviewers’ bio’s. Within five weeks of WAM approval of the of the peer reviewers (via a TD) and rece ipt of the draft Prioritization Toolreport from the WAM (via a TD), whichever comes later, the contractor shall conduct the peer review, asse mble the peer review comments and recommendations in a peer review report organized by charge question, prepare an introduction to the peer review report with a clear and concise over view of the comments, and attach to the peer review report any marg inal comments the peer reviewers had on the Prioritization Toolreport. It is not necessary that the peer revie wers jointly reach consensus on their findings and recommendations, since there may be limited overlap in the peer revie wers' areas o f expertise and in the charge questions that they focus on. The contractor shall assume, for the purpose of estimating costs, that the draft Prioritization Toolreport is roughly 100 pages in length with 200 pages of appendices, and that each peer reviewer will spend 40 hours in re viewing the report and wri tin g com men ts. EPA plans t o pr ovi de the re port to the con tra cto r inmid -Ju ly.Peer Review Handb ook Page E-7 Deliverables an d Schedule :

Task Deliverable Schedule 1 Work plan and budget Within 15 days of CO ap proval of work assignment 2 Memo identifying peer reviewers Within 3 weeks of wo rk assignment approval, receipt of pe er reviewe r qualifica tions from WAM, and r eceip t of cha rge fr om WAM, whichever comes latest 3 Peer review report Within five weeks of WAM approval of peer reviewers and receipt of draft Prioritizat ion Tool report from WAM, whichever comes later Other Requireme nts : CONTRACTOR COMMUNICATIONS Upon approval of the Work Plan, the co ntractor shall ma intain at least we ekly communications with the Work Assignment Manager regarding the status of work on the Work Assignment.
CONFLICT OF INTER EST (COI) [Revised Section]
The contractor must adhere to the following requi rements: a) Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior to commencement of any work, notify both the CO a nd PO of any actual or potential organ izational or person al conflicts of int erest. b) Provide a written c ertification, wit hin 20 days of receipt of a Work Assignment, QRT, or similar task ing document, tha t: 1) Either all conflicts of interest have been reported to the CO or that no conflicts of interest exist. The contractor is directed to assure that none of the conflicts disc losed are so dire ct and substantial as to rule out a particular reviewer. 2) All personnel who perform work under this Work Assignment or relating to this Work Assignment have been informed of their obligation to report personal and organizational conflicts of interest to the CO.Page E-8 Peer Review Handb ook 3) The Contractor recognizes its continuing obligation to identify and report any conflicts of interest arising during performanc e of this Work Assignment. c) If a conflict of interest is identified during performance under this Work Assignment, the Co ntractor shall imme diately mak e a full disclosure in writing to the CO. The disclosure shall include a description of action which the Contractor has taken or proposes to take, after consultation with the CO, to avoid, mitigate, or neutralize the c onflict of intere st. d) After selecting the peer review ers but before st arting the peer review, submit to EPA documentation that shows that the contractor has determined if the peer review candida tes: 1) have a conflict of interest or a situation that could create the appearance of a lack of impartiality in relation to the work product 2) have had or pres ently have a financial relat ionship with EPA
and summarizes for EPA its efforts to id entify and p ropose resolution of these concerns with pe er review cand idates.
EXPENDITURE OF FUNDS/HOURS In addition to the requirements of the contract, the contractor shall notify both the Project Officer and the Work Assignment Manager when 75% of funds or hours for this Work Assignment have been expended.
INFORMATION COLLECTION Any other provision of this Work Assignment notwithstanding, the contractor shall not proceed with any information collection where the same or similar information will be collec ted from ten or more public respondents until written approval is received from the Contracting Officer. This approval will cite an approval number from the Office of Manage ment and Budget as required by the Paperwork Reduction Act (PRA). Only Federal agencies and their employ ees are exempt from the PRA definition of "public respondent." State agencies and their employ ees are classified as "public respondents." Soliciting similar i nformation applie s to any colle ction method, i.e., written, oral, electronic, etc., and utilizing any approach, i.e., survey s, phone calls, focus groups, TQM, etc. The PRA applies equally to "willing participants" and participation that is mandated by law.Peer Review Handb ook Page E-9 Any question of applicability of the PRA shall be r esolved by s ubmitting a comple te description of the circumstances in a written request to the Contracting Officer. No collection shall be undertak en until the Contrac ting Officer pr ovides written noti ce to the Contrac tor as to the applicability of the PRA. If the PRA is determined to be applicable, the Contractor sha ll not initiate any collection until the requisite approval is received. The General Services Administration (GSA), under FIRMR Bulletin B-2 administers the Interage ncy Reports Management Pro gram as derive d from 44 U.S.C. Chap ters 29 and 31. All work performed under this Work Assignment involving federal interagency reporting must be done in full complia nce with these GSA procedures.
CONFIDENTIAL BUSIN ESS INFORMATION If this Work Assignment requires use of RCRA Confidential Business Information (CBI), the contract must specifically a uthorize the contractor to have access to RCRA CBI and the contractor s hall abide by all RCRA CBI re quirements and stip ulations found in the RCRA CBI Security Manual and in the contract. The contractor shall identify in the Work Plan budget all estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon as it is no longer needed under this Work Assignment or before the expiration of the Work Assignment, whic hever occurs f irst.
PRINTING AND D UPLICATION The contractor is prohibited from performing any printing unde r the Government Printing and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000 impressions of a single-page document or 25,000 impressions of a multiple-page stand-alone document , is limited to one col or (black ) copies, and does n ot exceed the maximum image siz e of 10 3/4 by 14 1/4 inches. For a ll duplication jobs in excess of 5,000 impr essions, the EPA WAM will determine in advance if the work can be performed more cost effectively and under the job or time cons traints at the EPA Pr int Shop. If the total number of pho tocopies for this Work Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan the photocopying costs by task and deliver able.
WORK ASSIGNMENT/ WORK PLAN BUDG ETS The contractor shall not exceed either the dollar or PL hour budget contained in the approved Work Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not exceed the PL hour budget of the QRT.Page E-10 Peer Review Handb ook
TECHNICAL DIRECTION The Designate d Work Assignment M anager (WAM) o n this Work Assignme nt is authorized to provide technical direction to the extent allowed under EPAAR (1552.237-71) (APR 1984) (DEVIATION). Other than the Desig nated WAM, only the Project Officer and the Contracting Officer are authorized to provide technical direction.
Technical direc tion includes: (1) Direction to the contractor which assists the contractor in accomplishing the Statement of Work (2) Comments on and approval/acceptance of reports or other deliverables Technical direction must be within the contract and the Work Assignment statement of work. The Projec t Officer and the WAM do nothave the authority to issue technical direction which (1) institutes additional work outside the scope of either the contract or this Work Assignment; (2) c onstitutes a chang e as defined in the "Changes" c lause; (3) cause s an increase or decrease in the estimated cost of the contract or Work Assignment; (4) alters the period of performance or deliverable due dates; or (5) changes any of the other express terms or conditions of the contract o r Work Assignment. Technical direction will be issued in writing or confirmed in writing within five (5) calendar day s after verbal issuance. The tec hnical direction memorandum will be provided to the co ntrac tor an d copi es wil l be fo rward ed to t he Con tract ing Officerand th e Proj ect Officer

. If the contractor has not received written confirmation within five (5) calendar day s of a oral issuance, the contractor must so notify the Project Officer.
INHERENTLY GOVERNMENTAL FUNCTIONS The contractor shall not perform a ny inheren tly govern mental functions ( IGF) under this Work Assignment. If during the course of developing the plan of work, thr ough receipt of technical direc tion, or in carry ing out the assig nment any po rtion of the effo rt is considered to possibly be an inherently gover nmental function, the contractor must immediately notify the Project Officer and the Contracting Officer.
OCCUPATIONAL HEA LTH AND SAFETY Facility site visits conducted under a Work Assignment that include on-site inspections or sampling must be conducted in full compliance with the Department of Labor, Occupational Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440 (Occupational Health and Safety Manual).Peer Review Handb ook Page E-11
TRAVEL COSTS The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal regulations in in curring allowa ble travel costs u nder this Work Assig nment, and corre spondingly must at all times seek and obtain Government rates whenever available and observe current subsistence ceil ings. QUICK RESPONSE TASKS Each Quick Response Task (QRT) shall be confirmed in writing and approved by the Project Officer. The contractor shall respond by letter to the PO with copies to the WAM and the CO within two wor king days, giving a brief description of the plan of work, incl uding best estimate of hours (by P-level) and a break-out of costs to accomplish the task. No task shall exceed a duration of 30 calendar days from start date to completion date

. The level of eff ort for each tas k shall be limited to a maximum of 250 labor h ours. Quick Response Task Requests do not change the dollar or professional labor hour budgets of a Work Assignment.Page E-12 Peer Review Handb ook
STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of Protozoa Method Development Cr iteria Docume nt
Period of Performance: Work Plan Approval to August 1, 1997 Work Assignment Manager: Sally Q. Government Employee Office of Water U.S. Environmental Protection Agency
LOE: 196 hours SOW: 2.4
BACKGROUND INFORMATION: The United States E nvironmental Prote ction Agency (EPA), Office o f Water is charg ed with protecting publi c health and the e nvironment from a dverse exposure to chemicals and mic robials in water media, such as ambient and drinking waters, wastewater/sewage sludg e and sediments. In support of t his mission OW’s Offic e of Science and Technology (OST) develops he alth standards, health criteria, health advisories, and technical guidance documents for water and water-relat ed media. Un der this wo rk assignme nt, docume nts prepare d by OST are to un dergo peer review. Peer review is an important component of the scientific process. It provides a focused, objec tive evaluation of a research proposal, publication, risk assessment, health advisory, g uidance or other document submitted for review. The criticism, suggestions and new ideas provided by the peer reviewers stimulate creative thought, strengthens the reviewed docume nt and confer credibility on the product. Comprehensive, objective peer reviews leads to g ood science and pro duc t ac cept ance withinthe sci ent ifi c com mun ity. Under this work assignment, the contractor will receive one document (Protozoa Method Development Criteria Document) for peer review which is related to human health and ecological ef fects.
STATEMENT OF WORK:
Task 1. The contractor shall develop a work plan to address all tasks in this work assignment. The work plan shall describe the steps that will be taken by the contractor to provide for peer review, including selection of peer reviewe r candidates with a ppropriate exper tise, determining absence of conf lict of interest, document and reference distribution, establishing schedules, preparing the peer review report, and submittal of the peer review pa ckage. Curric ulum vitae for all persons assigne d to complete this wo rk assignment sha ll be provided. All PPeer Review Handb ook Page E-13 levels, hours and total costs for each task will be provided and costs greater than $100.00 shall be ite mized in detail.
Task 2. The contractor shall select a g roup of peer re viewers and dete rmine their availability for the task and absence of conflict of interest, and establish a schedule for the peer review. The contractor is directed to assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular reviewer. Three peer reviewers shall participate in the review. No sing le peer reviewer may charge more than 40 hour s to this task. It is fully acceptable for peer reviewers to commit to less than 40 hours. The peer review will be conducted for the Protozoa Method Development Criteria Document. Reviewers selected by and working for the contractor shall be a pproved by the EPA Project Officer in writing prior to their beginning work. Minimally, all pee r reviewers shall be accompli shed in protozoan me thods for sample r ecovery a nd analysis from water. Approval submissions shall include the reviewers’ names and curriculum vitae.
Task 3. The contractor shall arrange for the selecte d peer reviewe rs to review the E PA document. Prepare the charge to the peer reviewers based on tec hnical direction received from the EPA WAM. Provide the peer reviewers with copies of the candidate report and all relevant references and instruct the selected pe er reviewers to und ertake the revi ew. The WAM will pro vide the contrac tor with the final version of the document to be reviewed.
Task 4. The contractor shall monitor peer reviewers’ pro gress to assure timely completion. The contractor shall collate peer review comments, and organize the comments in the peer review “for comments” document. Provide the peer review document and all ma terials submitted b y the peer r eviewers to the E PA WAM.
SCHEDULE AND DELIVERA BLES: Task 1. (Work Plan) 15 days after receipt of work assignment
Task 2. 1 week after work plan approval
Task 3. 1 week after selection of peer reviewers
Task 4. 1 week after receiving comments from the peer reviewers
TRAVEL: No travel is antic ipated under this w ork assignment. A ny travel di rectly cha rgeable to this work assignment must be submitted and approved by the project officer.Peer Review Handb ook Page F-1



# APPENDIX F - USEFUL FORMS
Peer Review Conflict of Interest Inquir y Peer Review Checklist for Determining Whether a Work Product Needs Peer Review (Template Provided by Region V) Peer Review Checklist for Conducting a Peer Review (Template Provided by Region V) Volunteer Service Program Participation Agreement (EPA Form 3100-14)Page F-2 Peer Review Handb ook

SUBJECT: PEER REVIEW CONFLICT OF INTEREST INQUIRY Dear (Peer Reviewer -- insert name): You have been requested by EPA to serve as a Peer Revie wer for______ __________(name of project )____________. Yo ur participation i n this review will be greatly appreciated. However, it is possible that your personal affiliations and involvement in particular activities could pose a conflict of in terest or crea te the appearan ce that you l ack impartiality in your involvement for this peer review. Although y our involvement in these activities is not necessarily grounds for exclusion from the peer review, y ou should consult the contact named below or other ap propriate offic ial to discuss these matters. Affilia tions or activities that could potentially lead to conflicts of interest might include: a) work or arrangements concerning future work in support of industries or other parties that could potentially be affected by regulatory developments or other actions based on material presented in the document (or review materials) that you have been asked to review; b) your personal benefit (or benefit of y our employer, spouse or dependent child) from the developments or other actions based on the document (or review materials) you have been asked to review; c) any previous involvement you have had with the de velopment of the document (or review materials) you have been asked to re view; d) any financial interest held by y ou (or your employer, spouse or de pendent child) that could be affected by y our participation in this matter; and e) any financ ial relationship y ou have or have h ad with EPA such as r esearch gra nts or cooperative a greements. Please contact ___________________ (name and contact info for Agency peer review official or prima ry contrac tor) to discuss any potential conflic t of interest issue s at your ea rliest convenience, but no later than _________.

[Be sur e to dat e and sign thi s in qui ry]Peer Review Handb ook Page F-3 

### PEER REVIEW CHECKLIST FOR DETE RMINING WHETHER A WORK PRODUCT NEEDS PEER REVIEW
Instructions : This checklist is based on the Agency’s Pe er Review Handbook and the October 2000 Region 5 Order “U.S. EPA Reg ion 5 Improved Policies and Procedures: Pe er Review, Records Management, and Work Product Au thorization of Scientific and Technical Work Products” which constitute Region 5’s standard operating procedur es for peer review. If you have any ques tions about peer review or need clarifica tion when com pleting this ch ecklist, please refer to the H andbook, ava ilable via the interne t at http://www.ep a.gov/ord/spc /2peerrev.htm

. Figure 1 on page 2 of the Hand book includes a useful flow chart and cross references to specific sections of the Handbook that are applicable to this checklist. You are also encourage d to consult with your Division or Office Pee r Review Coordinator. The Division/Office P eer Review Coordinators will periodically request information from this checklist in order to update the National Peer Review Database. 1. Title of Work Product:

2. Product Descr iption:

3. Project Manager: Name, Organization and Phone Number Please ci rcle the 4. Determination if Work Product is Scientific or Technical: appropriate answer a) Is the work pr oduct a scientific, engineering , economic, socia l yes no scie nce, or sta tisti cal d ocume nt? (Examples o f such docume nts include: risk assessments, technical studies and guid ance, analytical methods, scientific database designs, tech nical models, technical prot ocols, statisti cal surveys/stu dies, technica l background materials, and research plans a nd strategies.) b) Is the work pro duct a scien tific or te chnical doc ument resu lting yes no from a grant, c ontract or coope rative agree ment? c) Will the work product be used to support a research agenda, yes no regulatory program, policy position, or other Agency position or action? If you answered “no” to all of these questions, your work product is not subject to EPA’s peer review policy for scientific or technical work pr oducts and does not need to be placed on any of the peer review lists. Please proceed to #7 of this checklist. I f you answered “yes” to an y of these qu estions, your work product mig ht need peer re view; please continue on to #5 of this checklist.Page F-4 Peer Review Handb ook
5. Determination if Work Product is a Major Work Product: Determination of whether a work p roduct is “major” will largely be on a case-by -case basis. As the continuum of work prod ucts covers the rang e from the obviously major to those products that cle arly don’t need pee r review, there is no one single, easy ye s/no answer to the test of “major” (see Handbook, Section 2.2.3)

. There also is no s ingle definition of “significant.” Determination of “major” and “significant” are the responsibility of the Division or Office Director who is the official Decision Maker. Please c ircle t he appropriate answer a) Does the work product establish a significant pr ecedent, yes no mod el, or m eth odo logy? b) Does the work product address significant controversial yes no issues? c) Does the work product focus on significant emerging or yes no “cutting edge ” issues? d) Does the work product have significant cross-Agency or yes no inter-agency implications? e) Does the work product involve a significant investment of yes no agency r esources? f) Does the work product consider an innovative approach or yes no application for a previously defined problem, process or method olo gy? g) Is the wo rk product requir ed to be peer re viewed by st atute yes no or other legal mandate? h) Does the work product support a regulatory decision, yes no policy or g uidance of major impact? (Major impact can mean that it will have applicability to a broad spectrum of regulated entities and other stakeholders, or that it will have narrower applicability, but with significant consequences on a smaller geographic or practical scale.) i) Is the work product an application of or modification to yes no an existing, adequately peer reviewed methodology or model tha t de part s si gnificantl y fro m th e si tua tio n itwas ori gina lly designed to addr ess? If you answered “y es” to any of these questions, your work pr oduct needs peer review unless special circumstances exist; please continue on to #6. If you answe red “no” to all of these questions, your work product probably doe s not need peer review. However, peer review can alwa ys be done to improve the quality of the work produc t. Please procee d to #7 of this checklis t.Peer Review Handb ook Page F-5
6. Determination Whether Circumstances Exist Where a Major Work Product Would Not Be Peer Reviewed: Please ci rcle the appropriate answer a) Was the work product previously reviewed by recognized yes no exper ts or a n expe rt bod y? ( Note: Peer review of an EPA work product by a recognized refereed journal strengthe ns the scientific credibility of the work product but does no t eliminate the need to have the work product itself peer reviewed for issues and concerns to support an Agency action. See Sections 2.4.4 and 2.4.5 of the Handbook for more details. ) b) Are the scientific or technical methodologies or information yes no being used commonly accepted in the field of expertise? c) Has the reg ulatory ac tivity or act ion which the work product yes no supports been terminated or canceled? d) Is there a statutory or court ordere d deadline, or a time yes no constraint which may limit or preclude peer review of the work product? If you answered “y es” to any of these questions, your work pr oduct probably does not require peer review. This decision with the justification needs to be concurred with and signed off by the Division/Office Director. The decision with the justification must be retained in the peer review files and noted in Peer Review Work Product List C in the National Peer Rev iew Database. Co ntinue on to #7. I f you answe red “no” to all of these questions, proceed to #8.
7. Next Steps For W ork Products That Will Not Be Peer Revie wed: a) Division/Office Director concurs with the decision that the work product should not be peer reviewed.
Signature of Division/Office Director and Date Signed b) A copy of this completed checklist has been given to the Div/Off Peer Review Coord inator and put in the official peer r eview files in the Division/Office.
Signature of Div/Off Peer Review Coordinator and Date Signed

Location of Div/Off Peer Review FilesPage F-6 Peer Review Handb ook c) Work product has be en placed on Peer Review Work Produc t List C in the National Peer Review Database. (Note: This only applies to those work products subject to the peer review policy. )
Signature of Div/Off Peer Review Coordinator and Date Signed If all of the necessary informa tion is complete, you are done. You don’t need to proceed any furthe r with this checkli st.
8. Next Steps For W ork Products That Will Be Peer Re viewed: a) Division/Office Director has been consulted and concurs with the decision that the product should be peer reviewed.
Signature of Division/Office Director and Date Signed b) A copy of this completed checklist has been given to the Division/Office Peer Review Coordinator and put in the official peer review files in the Division/Office.
Signature of Div/Off Peer Review Coordinator and Date Signed

Location of Div/Off Peer Review Files c) Work product ha s been placed on Pe er Review Work Prod uct List B in the National Peer Review Database.
Signature of Div/Off Peer Review Coordinator and Date Signed Because y our work product w ill be peer revie wed, you ne ed to complete a se cond checklist entitled “Peer Review Check list for Conducting a Pe er Review. ”Peer Review Handbook Page F-7 

### PEER REVIEW CHECKLIST FOR CON DUCTING A PEER REVIEW
Instructions : This checklist is based on the Agency’s Pe er Review Handbook and the October 2000 Region 5 Order “U.S. EPA Reg ion 5 Improved Policies and Procedures: Pe er Review, Records Management, and Work Product Au thorization of Scientific and Technical Work Products” which constitute Region 5’s standard operating procedur es for peer review. If you have any ques tions about peer review or need clarifica tion when com pleting this ch ecklist, please refer to the H andbook, ava ilable via the interne t at http://www.ep a.gov/ord/spc /2peerrev.htm

. Pages 2-4 of the Handbook contain useful flowcharts and cross referenc es to specific s ections of the Handbook tha t are applicab le to this checklist. You are also encouraged to consult w ith your Division or Office Peer Review Coo rdinator. The Division/Office Peer Review Coordinators will periodically request information from this checklist in order to update the National Peer Review Database. 1. Title of Work Product:

2. Product Descr iption:

3. Project Manager: Name, Organization and Phone Number Check the box when 4. Up-front Considerat ions for Pl anning the P eer Revi ew: a) The Div/Office D irector has chos en a peer revie w leader for the project. (Note: The project manager and peer review leader can be the same person.) Name of Peer Review Leader: item is completed

Phone Number: Organization: b) The peer review leader has obtained appropriate peer review tr aining before conducting the peer review. c) Key questions and issues have been id entified t o include i n the charge to the peer revie wers. d) The Div/Office Records Coordinator has been consulted to insure that all the files, including electronic records, will be created, maintained, retained, and dis posed of appropr iately and i n accordance w ith Div/Office and A gency pr ocedures. e) A formal peer review record or file has been established, and provisions have been made t o store any e lectronic reco rds associated w ith the work product and peer review. Location of Re cord/File: Provisions for Elec tronic Records:Page F-8 Peer Review Handbook

Check the bo x when item is completed or circle the appropriate answer (NA = not applicable) f) There is a sourc e of adequate f unding to pay for external pee r review if external peer review is necessary and funding is nee ded. (Note: Contr acts NA can be used for peer review services. Ho wever, special management controls are required to ensure proper use of these co ntracts. See Section 3.6 of the Handbook for details.) Source of Fu nding: g) Resource limitations may restrict the peer review. (If “yes” was Yes No selected, a lim ited peer revie w might be co nsidered. Ho wever, only in very rare circumstances shouldresource limitations restr ict peer review. Peer review must be planned for as part of a project’s budget.) h) Amount of time needed for peer review(s) has been allotted given existing constrain ts of potential pee r reviewers, de adline for the fin al work product, logistics for the peer review, etc. Length of Time Needed:
5. Develop the Charg e to the Peer R eviewers: a) A clear, focused charge has been formulated that identifies recog nized issues, asks specific questions, and invites comments or assistance. b) The charge has been included in the peer review record.
6. Select the Pee r Review Mec hanism: a) The work product is novel, complex, co ntroversial, or h as great Yes No cost implications. (If the answer is “yes” to any of the above, serious thought should be given to conducting an external peer review. If the answer is “no” to all of the above, internal peer review is probably sufficient.) b) A determination ha s been made reg arding which co mponents or stages of the work product will be peer reviewed. (Note: Generally, peer review is recommended for each stage of a product’s development.) Components to be peer reviewed: c) A peer review mechanism (e.g., internal, external or a combination of both) has been chosen for the work product or stages of the work product. Mechanism: d) The work product either: 1) has been, or is being, generated as part Yes No of administrative or civil enforcement activities by U.S. EPA, or 2) likely wil l be used in the futu re to support admin istrative or civil enforcement activities by U.S. EPA. (If the answer is “yes” to either item above, then the Office of Regional Co uncil (ORC) must be consulted if the Peer Rev iew Leader be lieves an exter nal peer review is needed or i s preferable. ORC concurrence should be obtained.)Peer Review Handb ook Page F-9

Check the box when

item is completed, or circl e yes or no e) The work product is going to be peer reviewed via a refere ed, Yes No scientific journal. (If the answe r is “yes,” th e work produc t still should be considered for peer revi ew because journal peer review may not cover issues and concerns that the Agency would want peer reviewed in order to support an Agency ac tion.) f) Logistics fo r conducting the peer review (e.g., written com ments will be received by mail, or will be collected at a meeting) ha ve been included in the peer review record. g) The Div/Off Director has concurred with the recommended method of peer review. Date of Div/Off D irector Concurr ence: h) The concurrence of the Div/Off Director has been included in the peer review record.
7. Determ ine the Spe cific Tim e Line for t he Peer R eview: a) A start date for the peer review has been selected. Start Date: b) The amount of time t he peer review ers will be given to conduct the peer review has been determined. Number of Day s for Review: c) A due date for comments from the reviewers has been selected. Due Date: d) The amount of time necessary to incorporate comments from the peer reviewers into the work product has been determined. Number of Day s for Revision: e) A deadline for final completion of the work product has been determined. Due Date:
8. Select the Pee r Reviewers: a) Advice was sought in developing a list of potential peer reviewer candidates who are independent of the work product and have appropriate scientific and technical expertise. b) The expertise required for the peer review has been determined. c) In reviewing the candidates, a balance and a br oad enough spectrum of expertise were considered. d) In reviewing the candidates, a ny potential conflicts of inter est were considered.Page F-10 Peer Review Handb ook

Check the box when item is completed e) The peer reviewers have been selected and the process for selecting the reviewers, including inquiries and resolution of potential conflicts of interest, has been documented and included in the peer review record/file. (Note: Conflict of Interest Inquir y Forms are available from the Regional and Div/Off Peer Review Coordinators.)
9. Obtain and Tr ansmit Ma terials fo r Peer R eview: a) Instructions have been given to the peer reviewers which a sk for written comments in a specified format by the specified deadline that are responsive to the charge. b) The peer revie wers have been provided with the e ssential document s, data, and information to conduct their review. Date Peer Review ers Given Charg e/Materials: c) The peer reviewers have been instructed not to disclose draft work products to the public. d) The peer review record/file contains all the materials given to the peer reviewer s. 10. Conduct th e Peer R eview: a) Written comments ha ve been receiv ed from all peer reviewers. Date all comments were receive d: b) All clarification or additional information necessary from the peer reviewers is received. c) The validity and objectivity of the comments have bee n evaluated. d) Appropriate experts/staff/managers have been consulted on the potential impacts of the comments on the final work product, the project schedule , and budget. e) The peer review comments have been included in the peer review record/file.
11. Consider the Pee r Review Comm ents: a) Decisions have been made regarding which comments are accepted and will be incorporated into the final work product, and which comments will not be incorporated. b) A memo or other wr itten record has been prepared which responds to the peer review comments and specifies acceptance or, where thought appropriate, rebuttal and non-acceptance. c) The Div/Off Director has concurred with the decisions and written record on how to incorporate the peer reviewers comments in the work product and on which comments will not be incorporated. Date of Div/Off Director concurrence:Peer Review Handb ook Page F-11

Check the box when

item is completed, or circl e yes or no d) The concurrence of the Div/Off Director has been included in the peer review record/file. e) The memo or written record documenting how comments were handled and how the work product was revised has been included in the peer review record/file. f) The work product has been revise d to incorporate t he acceptable comments. g) The peer review performed during the process of developing the work product has b een summarized an d included in the wo rk product. h) It is necessary to send the revised work pr oduct back to the peer Yes No reviewers. (If the answer is “yes,” proceed to item #11i. If the answer is no, proceed to item #12.) i) Additional comments are received, evaluated, and incorporated into the work product, and placed in the peer review record.
12. Consider Other Com ments: a) Prior to finalization , the document nee ds additional inter nal Yes No and/or external programmatic review. (If the answer is“yes,” go to #12b. If the answer is “no,” proceed to #13.) b) Written comments by programmatic reviewers have been re ceived. c) Final decisions have been made regarding which comments are accepted and will be incorporated into the final work product, and which ones will not be incorporated. d) A memo or other written record has been prepared which responds to the programmatic review comments and specifies acceptance or, where thought appropriate, rebuttal and non-acceptance. e) Div/Off Director has concurred with the decisions and written record on how to in corporate the pr ogrammatic com ments. Date of Div/Off D irector concur rence: f) The memo or written record has been included in the peer review record/file. g) The work product has been revise d to incorporate t he acceptable programmatic c omments.
13. Finalize Work Product and Clo se Out Pe er Revie w: a) The work product has been completed. b) The Div/Off Dire ctor has approve d the work produc t. Date of Div/Off D irector Approva l: c) The Div/Off Director approval has been included in the peer review record/file.Page F-12 Peer Review Handb ook

Check the box when item is completed, or circle yes or no d) The Div/Off Dire ctor has judged the work product t o be sufficiently Yes No controversial, of significant enough interest to outside parties, or of wide enough distribution, such that it should also be authorized by the Regional Administrator (RA), or the Deputy RA (DRA). (If the answer is “yes,” proceed to #13e. If the answer is “no,” proceed to #13f.) e) The RA or DRA has a uthorized the work product. Date of RA or DRA A uthorization: f) The final work product has been included in the peer review record/file.
14. Publication and Rele ase of Reports: a) The Div/Off Director has approved publication or release of the work product. b) The written approval by the Div/Off Director has been included in the peer review record/file. c) The Div/Off Dire ctor has judged the work product t o be sufficiently Yes No controversial, of significant enough interest to outside parties or of wide enough distr ibution, such that its distribution or rel ease should also be authorized by the RA or DRA. If the answer is “yes,” proceed to #14d. If the answer is “no,”proceed to #15. (Note: The Div/Off Director’s decision to elev ate to the RA or DRA can be made concurrently with item #13d.) d) The RA or DRA has authorized distribution or release of the work product. Date of RA or DRA A uthorization:
15. Retention of Pee r Review Files and R ecords: a) The Div/Off official procedures for administrative records and the Agency’s record retention schedules have been examined to determine how lon g the peer rev iew record/file , including elec tronic records, should be retained. (Note: The required time of retention for final reports and supporting data varies dependin g upon the nature of the report, however, final reports which are mission related or have an EPA number and receive external distribu tion are generally permanent federal records.) b) The Div/Off Records Officer or the Regional Records Officer has bee n consulted to help determine how long the peer review record/file, including electronic records, should be retained.Peer Review Handb ook Page F-13

Check the box when item is completed c) A location for the completed peer review record/file has been identified, and pr ovisions have bee n made to retain e lectronic records associated with the work product and peer review. (Note: This can be the same location and provisions as identified in #4e.) Location of Re cord/File: Provisions for Elec tronic Records: d) Someone has been assigned the responsibility for maintaining the record/file and electronic records, and ensuring that they are either archived or destroyed appropriately

. (Note: This can be the same person as identified in #4a.) Contact Name and P hone No: Organization: 16. Closeout of Checklist: a) Items #1-15 of checklist have been completed.
Signature of Peer Review Leader and Date Signe d b) A copy of the completed checklist has been given to the Div/Off Peer Review Coordinator.
Signature of Div/Off Peer Review Coordinator and Date Signed c) The completed checklist has been included in official peer review record/file. d) The work product has been moved from Peer Review Work Product List B to List A in the National Peer Review Database. Date Product moved to List A:Page F-14 Peer Review Handb ook U.S. Environmental Protection Agency VOLUNTEER SERVICE PROGRAM PARTICIPATION AGREEMENT Name of Pr ogram Parti cipant Citizen of U.S. 

### � Yes 

### � No Project Supervisor

Proposed O rganizationa l Agree ment (Incl ude Geogr aphical Loc ation)

Proposed A rrival Date Actual Ar rival Date Proposed Termination Actual Termination

Sponsored by Educational Institution 

### � Yes 

### � No Name and Address of I nstitution (if “Yes” )

Describe Project (s) on Which Par ticipant Will Work, Including Scope and Anti cipated Hour s per Week

Facilities and Equipment to be Made Av ailable by EPA

Degree of Superv ision to be Exercised by EP A

Assistance and Degree of Cooper ation Require d by other A gency Pe rsonnel by Participant

Project Supervisor Cer tifies That Services to be Produced by the Progr am Participant are not Services P rovided for Through EPA Agency Operations

______ __________________________________ ______ Signature of Supervisor
THE VOLU NTEER AG REES THA T:
a. l claims fo r compensatio n from the G overnment of the Unite d States are waived fo r any serv ices perfo rmed; b. ernment has a non-exclus ive roya lty-fre e license to use or re produce and pate nt or copyr ight material which is develope d as part of and d uring participat ion in this prog ram; and c. to the administrative instructions and require ments of the agency w hile on EPA premises.

______ __________________________________ ______ Participant Signature

PERSONNEL OFFICE SIGNEE

CONCURRENCE

EPA Form 3100-14 (Rev. b 30) Repl aces the 7-72 edition and the 12-77 edition which was entitled “Unpaid Work-Study Program” Any and al The Gov
He/she will adhere

SIGNATURE OF PERSONNEL OFFICER OR DE DATEPeer Review Handb ook Page G-1 

# APPENDIX G - REFERENCES CONCERNING PEER REVIEW

American Chemical Society and the Conservation Foundation (1985) Issues in Peer Review of the Scientific Basis for Regulatory Decisions , Washington, DC, November 1985. Browner, C. (1994) Peer Review Program , Washington, DC, Memorandum issued June 7, 1994. (NOTE: Attached as Appendix A to this Handbook) Chubin, D. (1994) Grants peer review in theory and practice, Evaluation Review 18: 20-30. Chubin, D. and E. Hackett (1990) Peerless Science: Peer Review and Science Policy , Al bany, NY: State Universi ty of New Yo rk Press. Jasanoff, S. (1990) The Fifth Branch: Sc ience Advisors as Policymakers, Cambridge, MA: Harvard Univer sity Press. Kostoff, R. (1996) Peer Review in Selected Federal Agencies, presented at AAAS Annual Meeting, Baltimore, MD, February 9, 1996. National Academy of Sciences (1999) Evaluating Federal Research Programs: Research and the Government Performance and Results Act, Washington, DC: Na tional Academy Press, January 1999. National Environmental Policy Institute (1996) Enhancing the Integrity and Transparency of Science in the Re gulatory Process, Washington, DC: National Environmental Policy Institute, Fall 1996. National Research Council (1995) Interim Report of the Committee on Research and Peer Review in EPA , Washington, DC: National Academy Press, March 1995. National Research Council (2000) Strengthening Science at the U. S. Environmental Protection Agency: Research Management and Peer Review Practices, Washington, DC: National Academy Pr ess. Reilly, W. (1993) Peer-review Policy, Washington, DC, Memorandum issued January 19, 1993. Rennie, D. (1999) The Development and Rationale of Peer Review, In: Peer Review in He alth Sciences , Fiona Godlee and Tom Jefferson, eds., BMA Books, London.Page G-2 Peer Review Handbook Science Advisory Board (1999) An SAB Report: Review of the Peer Review Program of the Environmental Protection Agency, EPA-SAB-RSAC-00-002, U.S. Environmental Protection Agency, Science Advisory Board, Washington, DC, November 1999. Spitzer, H. (1995) Peer Review Practices in the Federal Government , Bethesda, MD: Environmental Ne twork, report pr epared for the A merican Ind ustrial Health Coun cil, April 26, 1995) USEPA (1992) Safeguarding the Future: Credible Science, Credible Decisions, U.S. Environmental Protection Agency, Washington, DC, March 1992. USEPA (1998) Science Policy Council Handbook: Peer Review, EPA 100-B-98-0 01, U.S. Environmental Protection Agency, Science Policy Council, Washington, DC, January 1998. USEPA (1999) EPA’s Selection o f Peer Reviewers , EPA OIG Repor t No. 1999-P-217, U.S. Environmental Prot ection Agency , Office of the I nspector Gener al (OIG), Was hington, DC, September 1999. U.S. General Accounting Office (1994) Peer Review: EPA Needs Implementation Procedures and Additional Control s , GAO/RCED-94-89, Washington, DC: U.S. Government Printing Office, February 1994. U.S. General Accounting Office (1996) Peer Review: EPA’s Implementation Remains Uneven, GAO/RCED-96-236, Washington, DC: U.S. Government Printing Office, September 1996. U.S. General Accounting Office (1999) Federal Research: Peer Review Practices at Federal Science Agenci es Vary, GAO/RCED-99-99, Washington, DC: U.S. Government Printing Office, March 1999.Peer Review Handb ook 

### Notes and CommentsPeer Review Handb ook 

### Notes and CommentsPeer Review Handb ook 

### Notes and CommentsPeer Review Handb ook 

### Notes and Comments