United States Office of Science Policy
 EPA 100-B-00-001 

Environmental Protection Office of Research and Development December 2000


Agency Washington, DC 20460 www.epa.gov


Science Policy Council


HANDBOOK


2nd Edition 
EPA 100-B-00-001 

December 2000 

U.S. Environmental Protection Agency


PEER REVIEW HANDBOOK


2nd Edition


Prepared for the U.S. Environmental Protection Agency 

by members of the Peer Review Advisory Group, 

a group of EPA’s Science Policy Council 

Principal Authors 

Kerry L. Dearfield, Ph.D. A. Robert Flaak


Office of Science Policy EPA Science Advisory Board


Office of Research and Development Office of the Administrator


Major Contributors 

Jean C. Schumann 

Office of Solid Waste 

and Emergency Response 

Roland B. Hemmett, Ph.D. 

Region 2 

Brett Snyder 

Nancy W. Wentworth 

Office of Environmental 

Information 

Arnold M. Kuzmack, Ph.D. 

Office of Water 

Office of Policy, Economics, 

and Innovation 

Science Policy Council


U.S. Environmental Protection Agency


Washington, DC 20460

Page ii Peer Review Handbook 

DISCLAIMER 

This document has been reviewed in accordance with U.S. Environmental Protection 

Agency policy and approved for publication and distribution to the Agency. Mention of trade 

names or commercial products does not constitute endorsement or recommendation for use. 
Peer Review Handbook Page iii


TABLE OF CONTENTS 

FOREWORD .......................................................... Page xiii


ACKNOWLEDGMENTS ................................................ Page xvii


SUMMARY OF THE PEER REVIEW PROCESS .............................. Page 1


Figure 1 - Flowchart for Planning a Peer Review .......................... Page 2


Figure 2 - Flowchart for Conducting a Peer Review ........................ Page 3


Figure 3 - Flowchart for Completing a Peer Review ........................ Page 4


Manager’s Planning Checklist for Peer Review ........................... Page 5


PEER REVIEW GUIDANCE ............................................... Page 7


1. 	THE NEED FOR PEER REVIEW ......................................... Page 9


1.1 Overview Statement ........................................... Page 9


1.2 Understanding Peer Review ..................................... Page 9


1.2.1 Why use Peer Review? .................................. Page 9


1.2.2 What is Peer Involvement? .............................. Page 10


1.2.3 What is Peer Review? .................................. Page 10


1.2.4 What is Peer Input? .................................... Page 10


1.2.5 How is Peer Review Different from Peer Input? .............. Page 11


1.2.6 Can Someone Who Provided Peer Input Become an 


Independent Peer Reviewer for the Same Work Product Later 


in the Process? ........................................ Page 11


1.2.7	 Can the Same Peer Reviewer be Used More than Once if a 


Product will Be Peer Reviewed More than Once, and can the 


Same Peer Reviewer be Used Again and Again for Different 


Products? ............................................ Page 12


1.2.8 How is Peer Review Different from Public Comment? ......... Page 12


1.2.9 How is Peer Review Different from Stakeholder Involvement? .. Page 12


1.2.10 What Role does Peer Review have in the Regulatory 


Development Process? .................................. Page 13


1.2.11 What Role does Peer Review have in Regulatory Negotiations? . Page 14


1.3	 Annual Agency Reporting Requirements ......................... Page 14


1.3.1 What are the Annual Reporting Requirements? .............. Page 14


1.3.2 What Listings are Required for the Annual Reporting? ........ Page 14


1.3.3 When will the Handbook Itself be Revised? ................. Page 17

Page iv Peer Review Handbook 

1.4	 The Roles of People and Organizations in Peer Review .............. Page 17


1.4.1 Who is Ultimately Accountable for Peer Review? ............ Page 17


1.4.2 Who are the Agency Staff involved in Peer Review? .......... Page 17


1.4.3 Who are the Decision Makers & What are Their 


Responsibilities? ...................................... Page 17


1.4.4 Who are the Peer Review Leaders & What are Their 


Responsibilities? ...................................... Page 19


1.4.5 Who are the Peer Review Coordinators & What are Their


Responsibilities? ...................................... Page 20


1.4.6 Who are the Peer Reviewers? ............................ Page 21


1.4.7 What are the Responsibilities of Peer Reviewers? ............ Page 21


1.4.8 What is an Independent Peer Reviewer? .................... Page 21


1.4.9 When does an Agency Internal Peer Reviewer Qualify as 


Independent? ......................................... Page 22


1.4.10 What is a Peer Review Panel? ............................ Page 22


1.4.11 What is a Subject Matter Expert? ......................... Page 22


1.4.12 What is the Role of the Science Policy Council (SPC)? ........ Page 23


1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)? . Page 23


1.4.14 What is the Role of the Office of Research and 


Development (ORD)? .................................. Page 23


2. 	PLANNING A PEER REVIEW .......................................... Page 25


2.1 Overview Statement .......................................... Page 25


2.2 Determining Which Work Products to Peer Review ................. Page 25


2.2.1 What are Scientific and Technical Work Products? ........... Page 25


2.2.2 What Scientific and Technical Work Products Need Peer 


Review? ............................................. Page 26


2.2.3 How Does One Determine Whether a Scientific and/or 


Technical Work Product is “Major”? ...................... Page 26


2.2.4 What Economic Work Products Need Peer Review? .......... Page 27


2.2.5 Should Economic Work Products Prepared in Support of 


Regulations that are Classified as "Major" or "Economically 


Significant" be Peer Reviewed? ........................... Page 28


2.2.6 What Other Economic Work Products Might Benefit from 


Peer Review? ......................................... Page 29


2.2.7 What Other Social Science Work Products Need Peer Review? .. Page 30


2.2.8 How Should Peer Review be Handled for Products Developed 


under an Interagency Agreement (IAG)? .................... Page 31


2.2.9 Should Products from Contracts, Grants, and Cooperative 


Agreements Receive Peer Review? ........................ Page 31

Peer Review Handbook Page v


2.2.10 How Does Peer Review Apply to Products Generated Through 


EPA Grants or Cooperative Agreements? ................... Page 31


2.2.11 Can the Recipient of a Grant or Cooperative Agreement Use 


Agreement Funds to Pay Peer Reviewers of their Work Products? Page 34


2.2.12	 Do Products Generated under EPA Grants or Cooperative 


Agreements Need to be Reported in the Peer Review Product 


Tracking (PRPT) Database? ............................. Page 34


2.2.13 Should Site Specific Decisions be Subject to Peer Review? ..... Page 35


2.2.14 Should NEPA Products (e.g., EISs) be Subject to Peer Review? . Page 35


2.2.15 Should Environmental Regulatory Models be Peer Reviewed? .. Page 36


2.2.16 Is Peer Review Needed for Other Organization’s Work Products 


that Have been Submitted to EPA for Use in Decision Making? . Page 36


2.2.17 Can Work Products That are Not Determined to be Major Still 


be Peer Reviewed? ..................................... Page 37


2.3 Determining Which Work Products Do Not Receive Peer Review ...... Page 37


2.3.1 Are There Circumstances When a Major Work Product is Not 


Peer Reviewed? ....................................... Page 37


2.3.2 What Products Normally Do Not Need Peer Review? ......... Page 38


2.3.3 Do Voluntary Consensus Standards Require Peer Review? ..... Page 39


2.4	 Choosing a Peer Review Mechanism ............................. Page 39


2.4.1 How Do You Determine the Appropriate Peer Review 


Mechanism? .......................................... Page 39


2.4.2 What are Examples of Internal Peer Review? ................ Page 41


2.4.3 What are Examples of External Peer Review? ............... Page 41


2.4.4 What is the Role of Peer Review by a Refereed Scientific 


Journal? ............................................. Page 42


2.4.5	 Do Agency Work Products Become Candidates for Peer 


Review when Peer Reviewed Journal Articles are Used in 


Support of that Work Product? ........................... Page 43


2.4.6 When and How Often Should Peer Review Occur? ........... Page 43


2.4.7 What Factors are Considered in Setting the Time Frame for 


Peer Review? ......................................... Page 44


2.4.8 Which Office/Region or Other Agency is Responsible for 


Conducting the Peer Review? ............................ Page 45


2.5	 Creating the Peer Review Record ............................... Page 45


2.5.1 What is the Peer Review Record? ......................... Page 45


2.5.2 How Can the Peer Review Record Improve the Peer 


Review Process? ...................................... Page 45


2.5.3 What Should Be in the Peer Review Record? ................ Page 45


2.5.4 What Should I Do with a Peer Review Record That Pertains to a


Rulemaking Action? ................................... Page 46

Page vi Peer Review Handbook 

2.5.5 When Should the Peer Review Record Building Process Begin? . Page 46


2.5.6 What are the Differences in Record Keeping for a Review by an 


Individual Compared to a Panel? .......................... Page 47


2.5.7 Where Should the Peer Review Record be Kept and For 


How Long? ........................................... Page 47


2.5.8 Are Internal Peer Review Comments Included in the Peer 


Review Record? ....................................... Page 47


2.6	 Budget Planning ............................................. Page 48


2.6.1 What Budgetary Factors Should I Consider in a Peer Review? ... Page 48


2.6.2 What Input is Needed for the Annual Budget Formulation and 


Budget Execution Process? .............................. Page 48


2.7	 Legal Considerations ......................................... Page 49


2.7.1 Are There Legal Ramifications From the Peer Review Policy? .. Page 49


2.7.2 Is Legal Advice Needed? ................................ Page 49


2.8	 Federal Advisory Committee Act (FACA) Considerations ............ Page 49


2.8.1 When Do FACA Requirements Apply to EPA-Run 


Peer Reviews? ........................................ Page 50


2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? ..... Page 50


2.8.3 How Do I Ensure that a Contractor-Run Peer Review Does 


Not Become Subject to FACA? ........................... Page 51


3. 	CONDUCTING A PEER REVIEW ....................................... Page 53


3.1 Overview Statement .......................................... Page 53


3.2 Charge to the Peer Reviewers .................................. Page 53


3.2.1 What is a Charge? ..................................... Page 53


3.2.2 What are the Essential Elements of a Charge? ............... Page 53


3.2.3 Where Can I Get an Example of a Charge? .................. Page 54


3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer 


Reviewers? ........................................... Page 54


3.2.5 Who Writes the Charge When I Hire a Contractor to Conduct 


the Peer Review? ...................................... Page 54


3.2.6 Is it Okay For Me to Ask a Contractor to Develop the Charge 


to the Peer Reviewers? .................................. Page 55


3.3 Time Line .................................................. Page 55


3.3.1 What are the Factors in Scheduling a Peer Review? ........... Page 55


3.4	 Selection of Peer Reviewers ................................... Page 55


3.4.1 What are Considerations for Selecting Peer Reviewers? ........ Page 55


3.4.2 Where Do I Find Peer Reviewers? ......................... Page 56


3.4.3 Are External or Internal Peer Reviewers Preferred? ........... Page 57


3.4.4 What is Important in the Mix of a Peer Review Panel? ......... Page 57


3.4.5 What is a Conflict of Interest? ............................ Page 58

Peer Review Handbook Page vii


3.4.6 What Techniques Help Ensure Disclosure and Appropriate 

Resolution of Conflicts of Interest? ........................ Page 59


3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews? . Page 61


3.4.8 Are There Constraints to Selecting Peer Reviewers? .......... Page 61


3.4.9 If State Employees are Used as Peer Reviewers, Can EPA 


Pay Them for this Service? .............................. Page 62


3.5	 Materials for Peer Reviewers ................................... Page 62


3.5.1 What Instructions Do You Give Peer Reviewers? ............. Page 62


3.5.2 What Materials Should be Sent to Peer Reviewers? ........... Page 63


3.5.3 How Closely can EPA Interact with Peer Reviewers During 


the Review? .......................................... Page 64


3.6	 Peer Review Services ......................................... Page 65


3.6.1 What are Gratuitous Services for Peer Review? .............. Page 65


3.6.2 Can I Use a Contract to Obtain Peer Review Services? ........ Page 66


3.6.3 How Do I Write a Statement of Work for Contracts? .......... Page 66


3.6.4 What are Advisory and Assistance Services (AAS) or Sensitive 


Activities? ........................................... Page 67


3.6.5 What are Some Management Controls for Contracts? ......... Page 67


3.6.6 Can I Identify and/or Select Peer Reviewers When Using 


a Contract? ........................................... Page 70


3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer 


Reviewers? ........................................... Page 70


3.6.8 How is Travel Handled with Contracts or Purchase Orders? .... Page 72


3.6.9 How is Travel Handled with Special Government Employees? .. Page 72


4. 	COMPLETING A PEER REVIEW ....................................... Page 73


4.1 Overview .................................................. Page 73


4.2 Final Work Product .......................................... Page 73


4.2.1 How Do I Incorporate Peer Review Comments into the 

Final Work Product? ................................... Page 73


4.2.2 What Actions are Potentially Forthcoming from Peer Review? .. Page 74


4.2.3 What Should the Final Work Product Say About the Peer 


Review Process? ...................................... Page 74


4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous? ...... Page 75


4.3	 Completing the Peer Review Record ............................. Page 75


4.3.1 How Do I Complete the Peer Review Record? ............... Page 75


4.3.2 Where Should the Peer Review Records be Kept, and for 


How Long? ........................................... Page 76


4.3.3 Is Information Regarding a Peer Review Subject to Release 


under FOIA? ......................................... Page 77

Page viii Peer Review Handbook 

SUBJECT INDEX ....................................................... Page 79


COMMONLY USED ACRONYMS ......................................... Page 81


APPENDIX A - U.S. ENVIRONMENTAL PROTECTION AGENCY PEER 


REVIEW POLICY ................................................ Page A-1


APPENDIX B - SOUND SCIENCE AND PEER REVIEW IN RULEMAKING ..... Page B-1


APPENDIX C - EXAMPLES OF CHARGES ................................ Page C-1


CHARGE EXAMPLE 1 - Framework for Developing a Living Resources 


Research and Monitoring Plan for the Peconic Estuary .............. Page C-2


CHARGE EXAMPLE 2 -EPA Science Advisory Board (SAB) Review of the 


Environment Monitoring and Assessment Program (EMAP) 


Research Strategy and Research Plan ........................... Page C-5


CHARGE EXAMPLE 3 - EPA Science Advisory Board Review of the 


Agency's National Risk Management Research Laboratory’s 


(NRMRL) Program ......................................... Page C-6


CHARGE EXAMPLE 4 - EPA Science Advisory Board (SAB) Review of the 


Technical Aspects of the Multi-Agency Radiation Survey and Site 


Investigation Manual (MARSSIM) ............................. Page C-7


CHARGE EXAMPLE 5 - Economics - Benefits Transfer from Adults 


to Children ................................................ Page C-8


CHARGE EXAMPLE 6 - Economics - Study of Municipal Government 


Costs and Financial Impacts from Environmental Regulations ........ Page C-9


CHARGE EXAMPLE 7 - Economics - Valuation of Fatal Cancer Risks ..... Page C-11


CHARGE EXAMPLE 8 - Hudson River PCBs Site Reassessment RI/FS


Modeling Approach Charge: Peer Review 1 ..................... Page C-15


CHARGE EXAMPLE 9 - IRIS Pilot Program - Instructions to Peer 


Reviewers for Reviewing IRIS Summaries and Supporting 


Documentation ............................................ Page C-18


CHARGE EXAMPLE 10 - Charge to Reviewers for the WTI Draft Final 


Risk Assessment .......................................... Page C-21


APPENDIX D - GUIDANCE ON REQUESTING A REVIEW BY THE US EPA 


SCIENCE ADVISORY BOARD (SAB) .......................... Page D-1


APPENDIX E - EXAMPLE STATEMENTS OF WORK FOR CONTRACTS ...... Page E-1


STATEMENT OF WORK - EXAMPLE 1 - Statement of Work: Technical 


Review Contractor for Panel Review of Assistance Agreement or 


Fellowship Applications ..................................... Page E-2

Peer Review Handbook Page ix


STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization 


Tool Report ............................................... Page E-5


STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of 


Protozoa Method Development Criteria Document ............... Page E-12


APPENDIX F - USEFUL FORMS ......................................... Page F-1


PEER REVIEW CONFLICT OF INTEREST INQUIRY .................. Page F-2


PEER REVIEW CHECKLIST FOR DETERMINING 


WHETHER A WORK PRODUCT NEEDS PEER REVIEW ........ Page F-3


PEER REVIEW CHECKLIST FOR CONDUCTING A PEER REVIEW ..... Page F-7


VOLUNTEER SERVICE PROGRAM PARTICIPATION AGREEMENT ... Page F-14


APPENDIX G ......................................................... Page G-1


REFERENCES CONCERNING PEER REVIEW ....................... Page G-1


Peer Review Handbook Page xi 

FOREWORD (2nd Edition) 

It has been three years since EPA's Science Policy Council (SPC) issued the 1st Edition 

of the Peer Review Handbook (January 1998). The original Handbook has been used extensively 

by personnel throughout the Agency for the peer review of Agency work products. At the time it 

was issued, we planned a two to three-year period during which Agency staff could use the 

Handbook and provide feedback on its utility, and at the same time, identify questions or issues 

not originally addressed. 

To date, we have received numerous comments and suggestions from regional and 

headquarters personnel concerning the Handbook. These comments have been uniformly 

positive and encouraging. The comments indicate that the Handbook contributed greatly to the 

Agency goal of sound science and substantially improving EPA's peer review process. 

We have collected the comments and additional questions, discussed them with the SPC's 

Peer Review Advisory Group and Agency Peer Review Coordinators, and developed revised and 

new answers to the questions received. In addition, we also incorporated many revisions in 

response to recommendations from the EPA Science Advisory Board (SAB; from "An SAB 

Report: Review of the Peer Review Program of the Environmental Protection Agency" a review 

by the Research Strategies Advisory Committee (RSAC) of the SAB; report #EPA-SAB-RSAC00-002 dated November 1999), the EPA Office of the Inspector General (OIG; from "EPA's 

Selection of Peer Reviewers" ; report #1999-P-217 dated September 29, 1999), and the National 

Research Council (NRC; from "Strengthening Science at the U.S. Environmental Protection 

Agency: Research Management and Peer Review Practices"; report issued 2000). After revising 

the Handbook , the revisions were circulated extensively throughout the Agency for final 

comment before the SPC approved the 2nd Edition of the Peer Review Handbook at their 

October 17, 2000 meeting. 

While we strived to make the Peer Review Handbook a definitive source of information 

about peer review and its processes, we need to sound one precautionary note. During the 

revision process, we heard many requests to put as much detail as possible into the Handbook to 

make it a one stop source for all things related to peer review. We have attempted to do this as 

much as possible; however, much of the detail in several instances delves into areas for which 

there is specialized expertise elsewhere in the Agency, particularly contracting and legal issues. 

If we provide all the detail requested in many of these areas, the Handbook would turn into a 

contract manual or legal book, something we did not want to do. Since many of these processes 

(like contracting) are updated frequently, we did not want to outdate the Handbook 

unnecessarily. Also, we felt if the Handbook became much bigger than it is now, its utility and 
Page xii Peer Review Handbook 

ease of use would decrease. We have provided pointers and contacts where necessary in the 

Handbook to help Agency people deal with these specific issues in relation to peer review. 

We are proud to bring the 2nd Edition to you for your use in peer review at EPA. The 

Handbook represents a large Agency collective effort to improve its science to the highest levels 

possible. Peer review done correctly significantly adds to our goal of sound and credible science 

that underlays all Agency decisions and actions. 

Kerry Dearfield and Robert Flaak


Co-Chairs, Peer Review Advisory Group (PRAG)


Science Policy Council


Science Policy Council 

W. Michael McCabe, OA, Chair


Donald Barnes, OA


Tudor Davies, OW


Sylvia Lowrance, OECA


William Muszynski, Region 2


Michael Shapiro, OSWER


Ramona Trovato, OCHP


Norine Noonan, ORD, Vice-Chair


Robert Brenner, OAR


Steven Galson, OPPTS


Albert McGartland, OPEI


Michael Ryan, OCFO


Elaine Stanley, OEI


Science Policy Council Steering Committee 

Donald Barnes, OA


Reginald Cheatham, OEI


William Farland, ORD


Penelope Fenner-Crisp, OPPTS


Jerri-Anne Garl, Region 5


Roland Hemmett, Region 2


Carl Mazza, OAR


Jennifer Orme-Zavaleta, ORD


Larry Reed, OSWER


Rosemarie Russo, Region 4


Mary Ellen Weber, OPPTS


William Wood, ORD


Michael Brody, OCFO 

Patricia Cirone, Region 10 

Michael Feldman, OCFO 

Michael Firestone, OCHP 

Peter Grevatt, OSWER 

Kate Mahaffey, OPPTS 

James Nelson, OGC 

Peter Preuss, ORD 

Joseph Reinert, OPEI 

Vanessa Vu, ORD 

Jeanette Wiltse, OW 

Science Policy Council Staff 

Edward Bender Kerry Dearfield James Rowe 
Peer Review Handbook Page xiii 

FOREWORD (From 1st Edition) 

EPA’s Science Policy Council (SPC) has organized this Peer Review Handbook as 

guidance to EPA staff and managers on the organization and conduct of peer review pursuant to 

the Administrator’s June 7, 1994 Peer Review Policy statement. The Handbook is based in part 

on the central themes set forth in the Policy statement (see Appendix A for the full policy): 

Major scientifically and technically based work products related to Agency decisions 

normally should be peer reviewed. Agency managers within Headquarters, Regions, 

laboratories, and field components determine and are accountable for the decision 

whether to employ peer review in particular instances and, if so, its character, scope, and 

timing. These decisions are made in conformance with program goals and priorities, 

resource constraints, and statutory or court-ordered deadlines. For those work products 

that are intended to support the most important decisions or that have special importance 

in their own right, external peer review is the procedure of choice. Peer review is not 

restricted to the penultimate version of work products; in fact, peer review at the 

planning stage can often be extremely beneficial. 

In addition, the Handbook augments these themes by stating and explaining widelyaccepted principles and practices that have long guided peer review in the universities, in private 

research organizations, and at the EPA and other government agencies. 

The goal of the Peer Review Policy and this 

Handbook is to enhance the quality and credibility of 

Agency decisions by ensuring that the scientific and 

technical work products underlying these decisions 

receive appropriate levels of peer review by independent 

scientific and technical experts. To serve this goal, the 

Handbook provides information and outlines procedures 

in several different areas: 

The Policy requires peer review 

of the basis of the decision (i.e., 

the underlying major scientific 

and/or technical work products), 

not the decision itself. 

P	 basic principles and definitions, including distinctions between peer review and 

peer input, public comment, and stakeholder involvement; 

P	 preparing for peer review, including identifying work products, identifying 

appropriate peer review mechanisms, and identifying qualified experts; and, 

P	 conducting and completing peer reviews, including materials required for peer 

review, creating a peer review record, and utilizing peer review comments. 
Page xiv Peer Review Handbook 

This Handbook has three parts. The first contains flow charts that outline the key steps in 

conducting a peer review, along with a managers’ checklist for planning peer reviews. The 

second part contains peer review guidance detailing the procedures outlined in the flow charts in 

a question and answer format. The third part contains Appendices including the 1994 Peer 

Review Policy and examples to help perform quality peer reviews. Some procedures outlined in 

the 1994 Policy have been completed while others are continuing to change in line with Agency 

experience. These changes have been incorporated into the current Handbook. 

Dorothy E. Patton, Ph.D.


Executive Director, Science Policy Council

Peer Review Handbook Page xv 

ACKNOWLEDGMENTS 

Compiling a comprehensive handbook that incorporates the most relevant, as well as 

some of the obscure policies and guidance, regarding peer review and its application throughout 

EPA has been a herculean task involving not just the members of the Peer Review Advisory 

Group, but many others as well. 

We want to acknowledge the contributions of the headquarters and regional Peer 

Review Coordinators and their respective office and division coordinators, the Science Policy 

Council (SPC) and its Steering Committee, the Office of Environmental Information’s (OEI’s) 

Quality Assurance Staff, Office of Science Policy (OSP) Program Support Staff, and the many 

other individuals who provided comments on the revisions to the current edition. 

In particular, we want to acknowledge the following individuals who provided critical 

expertise and timely revisions (often several times) to various sections of the Peer Review 

Handbook: Hale Hawbecker (Office of General Counsel), Theresa Trainor (Office of Water), 

and Bruce Bakaysa (Office of Acquisition Management). 

Finally, we want to acknowledge the tireless efforts and mentorship of the recently 

retired Executive Director of the Science Policy Council, Dr. Dorothy Patton. She is one of the 

pillars of the Agency’s efforts to improve the quality of science and peer review and we are 

indebted to her. 

Peer Review Handbook Page 1 

U.S. Environmental Protection Agency 

SUMMARY OF THE PEER REVIEW PROCESS


This Section of the Peer Review Handbook contains flowcharts and 

descriptions of the major steps in conducting a peer review. Cross references to 

the appropriate section on Peer Review Guidance are shown in parenthesis and 

bolded. 

The Managers Planning Checklist for Peer Review (on page 5) is designed 

to give Managers/Decision Makers a simple tool to help plan for a successful peer 

review. It asks questions that should be considered by a manager or Decision 

Maker during the peer review process to insure that necessary actions are taking 

place. This checklist is also intended to be used by staff (especially Peer Review 

Leaders and Coordinators) to inform managers and Decision Makers on some of 

the key steps and considerations that are necessary in carrying out a successful 

peer review. Expanded Check Lists which can be adapted for your use are also 

included in Appendix F - Useful Forms (courtesy of Region V). 
Page 2 Peer Review Handbook 

Figure 1 - Flowchart for Planning a Peer Review 

1.	 Determination of “major scientific and 

technical work product”: 

-- It is a scientific, engineering, 

economic, or statistical document 

(§ 2.2.1) 

-- Determine if the work pro duct is 

major (§ 2.2.3, 2.2.4, 2.2.7) 

-- Major p roducts m eet certain criter ia 

(§ 2.2.3) 

2.	 Work product is a candidate for peer 

review : 

-- Major work products are subject to 

peer review (§ 2.2.2) 

-- Some non-major work products have 

to be evaluated to determine if peer 

review is still warranted 

(§ 2.2.17) 

3. 	 Work p roduct is not a candidate for peer 

review : 

-- Most non-major work products are 

typically not candidates for peer 

review (§ 2.3.2) 

-- Major work p roduct consists only of 

science previously peer reviewed and 

adequate under the Agency’s Policy 

(§ 2.3.1) 

-- Place wor k produc t on List C 

(§ 1.3.2c)) 

4.	 If a work pro duct is subje ct to peer re view: 

-- Identify basis for charge (§ 3.2.1) 

-- Identify key staff (§ 1.4) 

-- Create a peer review record (§ 2.5) 

-- Ensure source of funding for the peer 

review (§ 2.6) 

-- Determine overall time frame for 

peer review (§ 3.3.1) 

-- Place the work product on 

List B (§ 1.3.2b)) 
Peer Review Handbook Page 3 

Figure 2 - Flowchart for Conducting a Peer Review 

1.	 Develop the charge (§ 3.2): 

-- Determine which key issues to add ress 

-- Include in peer review record (§ 2.5.3) 

2.	 Select a peer review me chanism (§ 2.4) 

-- Internal (§ 2.4.2) 

-- External (§ 2.4.3) 

-- Mail (e.g., Letter) review (§ 2.4.3) 

-- Face to face meeting (§ 2.4.3) 

-- One time or multiple meetings (§ 2.4.6) 

-- Include logistical information in peer 

review record (§ 2.5.3) 

3.	 Determine the specific time line (§ 3.3): 

-- When will the review be started 

-- What are the intermed iate check p oints 

-- What is the deadline for completion 

4.	 Select peer reviewers (§ 3.4): 

-- Determine sources of peer reviewers (§ 3.4.2) 

-- Determine expertise required (§ 3.4.4) 

-- Consider balance/add ress (§ 3.4.4) 

-- Consider c onflicts of interest (§ 3.4.5 & 3.4.6) 

-- Include documentation in peer 

review record (§ 2.5.3) 

5.	 Materials for the peer review (§ 3.5): 

-- Obtain materials from Program for review 

-- Prepare instructions for peer reviews (§ 3.5.1) 

-- Forward materials to peer reviewers (§ 3.5.2) 

-- Include copy of materials in peer 

review record (§ 2.5.3) 

6.	 Conduct the peer review 

-- Obtain written comments from reviewers 

-- Include in peer review record (§ 2.5.3) 

Note: Some of these steps may occur concurrently. 
Page 4 Peer Review Handbook 

Figure 3 - Flowchart for Completing a Peer Review 

1.	 Evaluate comments from peer reviewers (§ 4.2.1) 

-- Respond and react to comme nts 

-- Obtain clarification, if needed 

2.	 Brief your Decision Maker; obtain written 

management approval of response to 

comments (§ 4.2.1) 

3.	 Comments that are considered, but not used (§ 4.3.1) 

-- Determine why not used and document 

-- Include comments in peer review record (§ 2.5.3) 

4. Comments that are useful (§ 4.3.1) 

-- Revise the wo rk produ ct by 

incorpora ting comm ents 

-- Send revised work product back to 

peer reviewers, if necessary 

-- Include comments in peer review record (§ 2.5.3) 

5.	 Finalize wo rk produ ct (§ 4.3.1) 

-- Include in peer review record (§ 2.5.3) 

-- Move w ork prod uct from List B to List A 
Peer Review Handbook Page 5 

Manager’s Planning Checklist for Peer Review 

1) Title of Work Product: ___________________________________________________


2) What Decision/Rule/Regulation/Action Does this Work Product Support: ________


______________________________________________________________________________


3) Determination of Major Scientific and Technical Work Products


G Is the work product scientific or technical __yes __no?


G Is the work product __major or __non-major?


4) Determining What Peer Review is Needed


G If major, peer review is needed?


G If not major, is peer review still needed?


G What peer review mechanism is needed (internal and/or external)?


G When does the review need to be done?


G How much time will be needed to conduct/complete the review?


G Are there court ordered deadlines or other constraints?


G Has senior management (AA /RA/others) been informed of progress/problems?


G What would constitute success for this review?


5) Determining the Resources for Peer Review


G What is the priority of this project relative to other projects in the same office?


G What resources are needed to conduct the review?


G What are the impacts of the review on personnel?


G Who will lead the peer review?


G Who will conduct the peer review?


G Who will maintain the peer review record?


G Where will the peer review record be kept?


G What mechanism will be used for the peer review?


G Has the charge been developed?


G Has internal and external coordination been initiated/completed?


G Have arrangements for interim/final sign-offs (e.g., for the charge, the panel, on any changes


to the final work product) been made? 

G How will results of the review be presented and addressed in the final work product (e.g., in a 

preamble, in an accompanying appendix -- as well as changes in the work product itself)? 

G Has the work product been entered onto List B or C, as appropriate? 

6) Comments: _____________________________________________________________ 

See Appendix F (Useful Forms) for expanded checklists 

Peer Review Handbook Page 7 

U.S. Environmental Protection Agency 

PEER REVIEW GUIDANCE 

Peer Review Handbook Page 9 

1. THE NEED FOR PEER REVIEW 

1.1 Overview Statement 

Peer review at the U.S. Environmental Protection Agency (EPA) takes many different 

forms depending on the nature of the work product, relevant statutory requirements, and officespecific policies and practices. In January 1993, responding to recommendations in the report 

Safeguarding the Future: Credible Science, Credible Decisions, Administrator William Reilly 

issued an Agency-wide policy for peer review. Administrator Carol Browner reaffirmed the 

central role of peer review in the Agency on June 7, 1994 and instituted an Agency-wide 

implementation program (see Appendix A - USEPA Peer Review Policy). Following Agencywide implementation, office and region-specific standard operating procedures (SOPs) were 

written and used from 1994 to 1998. In 1998, this Peer Review Handbook was created as a 

single, centralized form of implementation guidance for Agency staff and managers. The Peer 

Review Handbook was revised throughout calendar year 2000 and was reissued as Edition 2 (the 

current version) in December 2000. Agency offices and regions, however, can still prepare brief, 

tailored guidance that meets their individual needs to supplement the information in this 

Handbook. 

1.2 Understanding Peer Review 

1.2.1 Why use Peer Review? 

Peer review is intended to uncover any technical problems or unresolved issues in a 

preliminary (or draft) work product through the use of independent experts. This information is 

then used to revise that draft product so that the final work 

product will reflect sound technical information and analyses. 

Peer review is a process for enhancing a scientific or technical Peer review is not free; 

work product so that the decision or position taken by the however, not doing peer 

Agency, based on that product, has a sound, credible basis. To review can be costly. 

be most effective, peer review of a major scientific and/or 

technical work product needs to be incorporated into the upfront planning of any action based on the work product - this includes obtaining the proper 

resource commitments (people and money) and establishing realistic schedules. 

Peer review of major scientific and technical work products should not be looked upon as 

another “hurdle” in the Agency decision making processes. While peer review requires that time 

and resources be planned into the decision making process, the benefits justify the added cost. 

Peer review enhances the credibility and acceptance of the decision based on the work product. 
Page 10 Peer Review Handbook 

By ensuring a sound basis for decisions, greater cost savings are realized since decisions will not 

be challenged as often and extra effort will not be required to go back and redo the work product. 

So while peer review is not free, the cost of not doing peer review is usually much more 

expensive. Furthermore, not conducting a peer review can potentially place the Agency in the 

position of attempting to defend a scientifically invalid position -- which can be very costly in 

terms of both resources, and more importantly, credibility. 

1.2.2 What is Peer Involvement? 

As defined in the Peer Review Policy, peer involvement is the process whereby Agency 

staff involve subject-matter experts from outside their program in one or more aspects of the 

development of work products. Peer involvement, therefore, constitutes active outreach to and 

participation by the broad scientific, engineering, and economics and social science communities 

beyond the Agency (external) as well as within the Agency (internal). Typically, peer 

involvement takes two general forms: peer input (ongoing discussions during the development of 

the work product) and peer review (an evaluation of a workplan, preliminary draft or the like, or 

most often, the critical, final objective expert evaluation of the work product). 

1.2.3 What is Peer Review? 

Peer review is a documented critical review of a specific Agency major scientific and/or 

technical work product. The peer review is conducted by qualified individuals (or organizations) 

who are independent of those who performed the work, but who are collectively equivalent in 

technical expertise (i.e., peers) to those who performed the original work. The peer review is 

conducted to ensure that activities are technically adequate, competently performed, properly 

documented, and satisfy established quality requirements. The peer review is an in-depth 

assessment of the assumptions, calculations, extrapolations, alternate interpretations, 

methodology, acceptance criteria, and conclusions pertaining to the specific major scientific 

and/or technical work product and of the documentation that supports them. Peer review may 

provide an evaluation of a subject where quantitative methods of analysis or measures of success 

are unavailable or undefined; such as research and development. Peer review is usually 

characterized by a one-time interaction or a limited number of interactions by independent peer 

reviewers. Peer review can occur during the early stages of the project or methods selection, or 

as typically used, as part of the culmination of the work product, ensuring that the final product is 

technically sound. 

1.2.4 What is Peer Input? 

Many Agency work products are developed with the input of various scientific and 

technical experts inside and outside the Agency. Like the contribution made by peer reviewers, 

peer input is valuable and enhances the scientific or technical basis of the products. Peer input, 
Peer Review Handbook Page 11 

sometimes referred to as peer consultation, generally connotes an interaction during the 

development of an evolving Agency work product, providing an open exchange of data, insights, 

and ideas. Peer input may be characterized by a continued and iterative interaction with 

scientific experts during work product development. A common example of peer input is the 

input received from workgroup members during development of a product. Many Agency 

products are developed through the efforts of a workgroup, which may include external experts, 

such as State and Tribal representatives. These workgroup members have an active, ongoing 

participation in developing the work product. Another example of obtaining peer input is of an 

Agency office sending a draft work product to a list of stakeholder representatives for general 

comments (stakeholder representatives often include experts who could be considered “peers”). 

1.2.5 How is Peer Review Different from Peer Input? 

The key distinctions between peer input as described above and formal peer review are 

the independence of the peer reviewers and their level of involvement. The goal of peer review 

is to obtain an independent, third-party review of the product from experts who haven’t 

substantially contributed to its development. When experts have a material stake in the outcome 

of the peer review (such as a regulated party) or have participated substantially in the 

development of the product (such as a workgroup member), those experts’ reviews may not 

qualify as unbiased, independent peer review and may be better characterized as peer input. 

It is clear that peer input provides valuable 

contributions to the development of the work product. Peer Input is not a

However, peer input does not substitute for peer review. Once substitute for Peer Review 

a work product is considered major, it is a candidate for peer 

review and entered on List B (Candidate Products for Future 

Peer Review -- see Section 1.3.2b)) -- even though the work product may already have a 

substantial amount of peer input. In other words, one cannot argue that a peer review is not 

necessary if a major work product has received “enough” peer input. If the work product is not 

considered major and has had peer input, it is entered on List C (Products for Which a Decision 

has been made not to Peer Review -- see Section 1.3.2c)) with comments about the extent of the 

peer input. 

1.2.6	 Can Someone Who Provided Peer Input Become an Independent Peer 

Reviewer for the Same Work Product Later in the Process? 

Generally, the answer is no as that expert is no longer independent, but rather a 

contributor to the work product. There may be special circumstances where the expertise is so 

narrow that another peer reviewer isn’t available. The Peer Review Leader (see Section 1.4.4) 

will normally be responsible for making this determination and documenting the decision in the 

peer review record. 
Page 12 Peer Review Handbook 

1.2.7	 Can the Same Peer Reviewer be Used More than Once if a Product will Be 

Peer Reviewed More than Once, and can the Same Peer Reviewer be Used 

Again and Again for Different Products? 

There is no prohibition on using the same peer reviewer more than once on the same 

product or for multiple products of the same office. However, it is a good idea to use some 

different people each time where possible to provide a broader perspective. When using a 

contractor to provide peer review services, you should recognize that contractors may have a 

“pool” of reviewers that they use regularly. If the same peer reviewers are used repeatedly, they 

may lose their independence (or the appearance of independence) from the work product(s). If a 

peer reviewer is asked to participate in multiple reviews of the same product it should be noted 

in the peer review record. 

1.2.8 How is Peer Review Different from Public Comment? 

Peer review and public comment are mutually exclusive. Public comment solicited from 

the general public through the Federal Register or by other means is often required by the 

Administrative Procedures Act, relevant statutes or both. Public comment can also be solicited 

for policy purposes. The Agency takes public comment on some strictly scientific products and 

almost all regulatory decisions. Public commenters usually include a broad array of people with 

an interest in the technical analysis or the regulatory 

decision; some are scientific experts (which may 

provide some peer input), some are experts in other Public comment does not substitute 

areas, and some are interested non-experts. The for peer review.

critical distinction is that public comment doesn’t 

necessarily draw the kind of independent, expert 

information and in-depth analyses expected from the peer review process. Public comment is 

open to all issues, whereas the peer review process is limited to consideration of technical issues. 

While it may be an important component of the review process, public comment does not 

substitute for peer review. 

1.2.9 How is Peer Review Different from Stakeholder Involvement? 

Stakeholder involvement occurs when the Agency works with external interest groups 

that have some stake in or concerns over the outcome of the technical work product or regulatory 

position. This is an interactive process, working with 

other agencies, industry groups, regulated-community 

experts, environmental groups, other interest groups that Stakeholder involvement is not 

represent a broad spectrum of the regulated community, a peer review mechanism. 

etc., and usually strives for a consensus approach. The 

goal of peer review, on the other hand, is to obtain an 
Peer Review Handbook Page 13 

independent, third-party review for ensuring scientific integrity and technical credibility of the 

work product that supports a policy or decision. Stakeholder involvement is not a peer review 

mechanism and it does not substitute for peer review even though it adds value to the work 

product. 

1.2.10 What Role does Peer Review have in the Regulatory Development Process? 

The peer review of scientific and technical work products that support rulemaking actions 

is an important, fundamental step in the policy setting process and which affirms the credibility 

of the Agency. Because new rules, and the work products supporting them, must often withstand 

intense scrutiny by the general public and the stakeholders involved in the action, the peer review 

process selected for such work products needs to be well planned and documented. The rule or 

regulation itself is not subject to the Peer Review Policy. However, if the rule or regulation is 

supported by a major scientific and/or technical work product, that work product should be peer 

reviewed prior to its use in the rule (see Section 2.2 for determination of major work products). 

The decision to peer review or not peer review any scientific and/or technical work product will 

be documented through the Agency’s annual peer review reporting process (see Section 1.3). 

Remember, public comment and stakeholder involvement do not constitute peer review. 

Tier 1 and Tier 2 rulemakings, are by definition important, major Agency rulemakings 

within the Agency. Therefore, work products supporting Tier 1 and Tier 2 rules in particular 

(including rules that are determined to be “significant” by OMB under Executive Order 12866 

because they have an economic impact of $100 million or more) should be closely scrutinized to 

determine whether they meet the criteria for major (see Section 2.2.3). Work products 

supporting Tier 3 rulemakings may also be considered major and thus candidates for peer 

review. External peer review is the procedure of first choice for a work product that is intended 

to support a Tier 1 or Tier 2 rulemaking. Although acceptable in certain circumstances, any 

decision to use an internal peer review mechanism for such work products would be the 

exception rather than the rule. For work products supporting a Tier 3 rule, internal or external 

peer review may be appropriate depending on the nature of the product and other factors (see 

Section 2.4.1). For Tier 1 and Tier 2 rulemakings, the Final Agency Review/closure memo 

needs to indicate that the Peer Review Policy was followed. For Tier 3 rulemakings, the action 

memo needs to indicate that the Peer Review Policy was followed. 

Analytic blueprints are required for Tier 1 and Tier 2 rulemakings, and are encouraged 

for Tier 3 rulemakings; some individual EPA offices require it for Tier 3. For peer review 

purposes, development of the analytic blueprint is the process whereby the project manager 

identifies the supporting scientific and technical work products and identifies needed peer 

review. In the directive from the Deputy Administrator (memo from Acting Deputy 

Administrator Robertson, June 15, 1999: Sound Science and Peer Review in Rulemaking; see 
Page 14 Peer Review Handbook 

Appendix B - Sound Science and Peer Review in Rulemaking), planning for peer review must be 

specifically addressed in each new analytic blueprint. You are required to integrate peer review 

into the rulemaking process and to include specific language addressing peer review in the action 

memorandum. The analytic blueprint shows the schedule of the peer review in the context of the 

schedule for the overall rulemaking. In general, peer review should be completed as early in the 

process as practicable. Where possible, peer review of work products should be completed prior 

to issuance of the proposed regulation. In some cases, support work products for final 

regulations may require an additional peer review if those scientific and technical work products 

change significantly after the public comment period. 

1.2.11 What Role does Peer Review have in Regulatory Negotiations? 

Regulatory negotiations are not candidates for peer review; however, to ensure final 

decisions are based on sound and credible science, the major scientific and technical work 

products that support the negotiation need peer review before the negotiation takes place. 

1.3 Annual Agency Reporting Requirements 

1.3.1 What are the Annual Reporting Requirements? 

The Peer Review Coordinator (for each AA/RA; see Section 1.4.5) will organize an 

annual review of all peer review activities and submit this information to the Office of Research 

and Development (ORD). ORD will staff this function at the direction of the Deputy 

Administrator. In the Deputy Administrator’s annual call for submissions, guidance on format 

and submission of this information will be provided. ORD will review the submissions for 

completeness, i.e., all information is provided and products are accounted for each year. ORD 

will then provide a review of the completeness of the information in the submissions through 

consultation with the appropriate persons in each organization (see Section 1.4). ORD will then 

consolidate the information and findings for the SPC and the Deputy Administrator. Any 

conflicts arising from the review will be resolved by the Deputy Administrator. The due date for 

the annual reporting will be announced each year in the annual call letter; however, for planning 

purposes, it is normally due in the early summer. The Office of Environmental Information 

(OEI) assists ORD in these annual reporting functions. 

1.3.2 What Listings are Required for the Annual Reporting? 

The Peer Review Product Tracking (PRPT) database is a Lotus Notes shared-database 

developed to track and report peer review activities across the Agency. The database is the 

single repository for product-specific peer review reporting and tracking and uses a common 

reporting form for all reporting. Work products for which peer review is completed are reported 

as List A (Peer Review Completed); candidate work products for future peer review are reported 
Peer Review Handbook Page 15 

as List B (Peer Review Needed); and work products that do not meet peer review criteria are 

reported as List C (Peer Review Not Needed). List D (Scientific Articles/Reports Peer Reviewed 

Outside of EPA) was recently added largely to track the articles that EPA staff produce. 

Four listings of products are required for the annual reporting: 

a) List A - Products Peer Reviewed Since 1991 

1)	 List A is a cumulative list of peer reviewed products from 1991 to the 

present. 

2)	 Each work product entered onto List A must include information 

summarizing the peer review. The database will specify the needed 

information. The entry must be electronically signed (verified) by the 

Peer Review Leader, the Peer Review Coordinator, and the Decision 

Maker involved (see Section 1.4.3 to help determine who the Decision 

Makers are). 

b) List B - Candidate Products for Future Peer Review 

1)	 List B is a list of products that are expected to be peer reviewed in the near 

future. 

2)	 This List contains major scientific and technical work products and any 

non-major scientific and technical work products for which peer review 

has been deemed necessary or appropriate. All entries on List B must be 

electronically signed (verified) by the Peer Review Leader, the Peer 

Review Coordinator, and the Decision Maker involved. 

3)	 Work products placed on List B remain on List B until they are either peer 

reviewed (after which they are moved to List A) or a decision is made not 

to peer review that work product (at which point it is moved to List C 

including the rationale for not conducting peer review). If the product is 

terminated, it is moved to List C. 

c) List C - Products for Which a Decision has Been Made Not to Peer Review 

1)	 List C is a cumulative list of all scientific and technical work products that 

do not receive peer review (see Section 2.3). 
Page 16 Peer Review Handbook 

2)	 List C includes: (a) any major scientific and technical work products for 

which a decision was made not to peer review; (b) those work products 

which were originally placed on List B, but for which it was decided that 

peer review was not necessary (e.g., the product was not used in decision 

making; the project was canceled); and (c) all non-major scientific and 

technical work products. 

3)	 List C also includes several categories of work products that would not 

normally receive peer review. It is not necessary to list these work 

products individually on List C, however, the total number prepared by the 

organization must be included on List C. The organization may need to 

identify the individual products if requested (e.g., due to litigation, FOIA, 

etc.). Such categories can include, but may not be limited to: chemical 

action reports, RCRA permits, scientific analyses for Premanufacturing 

Notices (PMNs) that are conducted on a routine basis and that do not 

deviate from established practice, and NPDES permits. 

4)	 Each work product on List C needs a brief description of the reason(s) it is 

not being peer reviewed. All entries on List C must be electronically 

signed (verified) by the Peer Review Leader, the Peer Review 

Coordinator, and the Decision Maker involved. 

d)	 List D - Scientific Articles and Reports That Are Peer Reviewed by Organizations 

Outside of EPA 

1)	 List D includes EPA authored scientific papers (articles) that are peer 

reviewed by a credible refereed scientific journal (See Sections 2.4.4 and 

2.4.5). 

2)	 Listing such documents in this manner shows that these items have 

already been peer reviewed, but differentiates them from Agency work 

products. In addition, it gives EPA an opportunity to highlight the 

extensive work it produces in the scientific literature. 

3)	 It is not necessary to list these papers individually on List D, although 

some organizations may choose to do so. However, at a minimum, the 

total number prepared by the organization and peer reviewed by journals 

must be included on List D. The organization may ultimately need to 

identify the individual papers and where published, if requested (e.g., due 

to litigation, FOIA, etc.). 
Peer Review Handbook Page 17 

4)	 Once any of these articles or reports are used in an Agency work product, 

that work product becomes a candidate for peer review and must be listed 

on List B (or List C if a decision is made not to peer review the work 

product). 

5) Entries on List D do not require an electronic signature. 

1.3.3 When will the Handbook Itself be Revised? 

During each annual reporting cycle, suggestions for revisions to the Handbook should be 

submitted. Periodically, a decision will be made by the Science Policy Council (SPC) on 

whether to revise the Handbook, in part or total, or not based on the suggestions. The SPC will 

then direct the Peer Review Advisory Group (PRAG) to draft the revision(s) for Agency 

comment and SPC approval. 

1.4 The Roles of People and Organizations in Peer Review 

1.4.1 Who is Ultimately Accountable for Peer Review? 

Under the June 7, 1994 Peer Review Policy (Appendix A - USEPA Peer Review Policy), 

the Administrator has designated the Assistant Administrators and Regional Administrators 

(AAs and RAs) to be accountable for implementing the Policy in their respective organizations. 

The Deputy Administrator is ultimately responsible for peer review across the Agency and is the 

final arbitrator of conflicts and concerns about peer review. 

1.4.2 Who are the Agency Staff involved in Peer Review? 

The principal Agency staff involved are Decision Makers (and their line managers), Peer 

Review Leaders and Peer Review Coordinators. In addition, there are many other critical staff in 

each office and region who have responsibility for peer review activities (e.g., office and division 

peer review coordinators, technical information managers, and, of course, any Agency staff that 

serve as internal peer reviewers). Finally, ORD has oversight, as designated by the Deputy 

Administrator, for ensuring the Agency’s Peer Review Policy requirements are met. 

1.4.3 Who are the Decision Makers & What are Their Responsibilities? 

The AA/RA is the ultimate Decision Maker for their organization and is accountable for 

the decisions regarding the identification of major scientific and technical work products and the 

mechanism(s) of peer review utilized for each of the products. The AA/RA may designate 

Office Directors and/or Division Directors as the front-line Decision Makers. 
Page 18 Peer Review Handbook 

Generally, the Decision Makers decide whether a work product is major and needs peer 

review or not, and what peer review mechanism to use. Furthermore, the Decision Makers 

commit the resources needed to ensure a proper peer review. Decision Makers are responsible 

for ensuring that the peer reviews are properly performed and documented. 

In order to ensure greater independence of peer reviews, it is necessary to strictly separate 

management of work products from the actual peer review of those work products. Therefore, 

the Decision Maker and the Peer Review Leader (see Section 1.4.4) for a work product should 

never be the same person. 

The Decision Maker needs appropriate training on how to manage the peer review 

process. The Peer Review Coordinator for the office can advise the Decision Maker on how to 

obtain the appropriate training; the training approach varies among the different EPA offices. 

Specific responsibilities of the Decision Maker(s) are the following: 

a) Determine which work products in their organization require peer review 

b)	 Designate (in conjunction with the Project Manager) a Peer Review Leader to 

organize the peer review 

c)	 Provide advice, guidance, and support to the Peer Review Leader in the 

preparation, conduct, and completion of the peer review 

d)	 Ensure that sufficient funds are designated in the office’s budget request to 

conduct the peer review; also ensure that adequate resources and/or extramural 

management support are available for the peer review 

e) Establish a realistic peer review schedule 

f) Designate the stage(s) of product development where peer review is appropriate 

g)	 Ensure all relevant issues and comments raised by the peer reviewer(s) are 

adequately addressed and documented for the record, and where appropriate, 

incorporated into the work product that is used as basis for decision making 

h)	 By signature, document the decisions made that are reported in the annual 

reporting to the SPC and Deputy Administrator 

i)	 Certify any decision NOT to peer review a product by signature on a List C 

(Products for Which a Decision has been made not to Peer Review) submission 
Peer Review Handbook Page 19 

1.4.4 Who are the Peer Review Leaders & What are Their Responsibilities? 

The Peer Review Leader is assigned by the Decision Maker to organize, conduct and 

complete the peer review for a specific individual work product. The Peer Review Leader will 

obtain the assistance and support of the Peer Review Coordinator (see below) as well as any 

others within the Agency to help perform the peer review. The Peer Review Leader will be 

chosen on a case by case basis depending on the work product needing peer review. The Peer 

Review Leader cannot be the Decision Maker. The Peer Review Leader could be the Project 

Manager for the work product. 

The Peer Review Leader must have appropriate training on how to conduct a peer review 

before conducting the peer review. The Peer Review Coordinator for the office can advise the 

Peer Review Leader on how to obtain the appropriate training; the training approach varies 

among the different EPA offices however, uniform training modules are available. 

Specific responsibilities of the Peer Review Leader are these: 

a)	 Keep the Decision Maker informed of the status of a given project; provide Peer 

Review Coordinator with data for the annual report 

b) Organize, conduct, and complete the peer review following Agency procedures 

c)	 Establish and maintain the peer review record for the specific individual peer 

review currently being performed (see Section 2.5); this includes providing the 

peer review summary information in the Peer Review Product Tracking (PRPT) 

Database for the Decision Maker to sign when the peer review is completed 

d)	 Select the peer reviewers in consultation with others involved with the peer 

review (e.g., Decision Maker) and ensure that conflict of interest issues are 

addressed and documented in the peer review record 

e) Advise peer reviewers of their responsibilities 

f)	 Provide information to the Decision Maker (including all appropriate managers in 

the Peer Review Leader’s chain of command) on the charge, profile of peer 

reviewers, the peer review comments, and a proposal on how to address the 

comments. Obtain Decision Maker approval on the approach to responding to 

peer reviewer comments. Clearly identify for the Decision Maker any peer 

review comments that will not be addressed in the agreed upon approach. 
Page 20 Peer Review Handbook 

g)	 Notify the Peer Review Coordinator that the peer review is completed for the 

annual report 

h)	 Archive the peer review record in a manner consistent with their organization’s 

archiving procedures 

When a contractor is used to conduct a peer review, some of the above responsibilities 

are assumed by the contractor (see Section 3.6). 

1.4.5 Who are the Peer Review Coordinators & What are Their Responsibilities? 

The Peer Review Coordinator is designated by the AA/RA to coordinate and monitor 

peer review activities in their respective organization or organizational unit. This person must be 

of sufficient stature and judgment to have the access to and confidence of all levels of office or 

regional management when needed. The Peer Review Coordinator is the main contact for their 

organization; they can also direct interested parties to other persons/contacts in the office on 

specific work products (e.g., Peer Review Leader). 

Specific responsibilities of the Peer Review Coordinator are these: 

a) General oversight responsibility for the Office’s or Region’s peer review process 

b) Report peer review activities to the AA/RA 

c)	 Help mediate difficult issues between their organization and others; if they can’t 

resolve issue, then bring the issue to the attention of the appropriate level 

Decision Makers in each organization for resolution. 

d) Function as the liaison with ORD and the Science Policy Council (SPC): 

1) Represent office/region before the SPC 

2)	 Advise ORD of any changes in the list of work products and peer review 

mechanisms during the annual reporting, and when necessary, at other 

times 

3)	 Participate in Agency peer review training, workshops, etc., as requested 

and disseminate this information to the organization; coordinate and/or 

present training within their organization 
Peer Review Handbook Page 21 

e)	 Submit information on organization’s peer review candidates for each year as 

requested (this is the annual reporting, see Section 1.3) 

1) Generate and update Lists A, B, C, and D (See Section 1.3.2) 

2)	 Assure the proper approval signature on the completed submission with 

the accompanying explanation for any departures from the Policy 

f)	 Establish procedures to assure that the required work product peer review 

documentation (i.e., peer review record) is filed and maintained in an appropriate 

manner (see Section 2.5) 

g)	 Provide advice, guidance, and support to the various Peer Review Leaders for the 

performance of the peer reviews 

h)	 Distribute Agency-wide peer review guidance and materials to appropriate 

office/region personnel, as requested 

1.4.6 Who are the Peer Reviewers? 

Peer reviewers are individuals who have technical expertise in the subject matter of the 

work product undergoing peer review. Peer reviewers can come from EPA, another Federal 

agency, or from outside of the Federal government. 

1.4.7 What are the Responsibilities of Peer Reviewers? 

Peer reviewers need to be willing participants in the peer review process -- they should 

agree to read all materials, participate fully, and protect confidential information that arises. Peer 

reviewers should maintain the confidentiality of the product, perform the review in a timely 

manner, and be unbiased and objective. 

1.4.8 What is an Independent Peer Reviewer? 

An independent peer reviewer is an expert who wasn’t associated with the generation of 

the specific work product either directly by substantial contribution to its development or 

indirectly by significant consultation during the development of the specific product. The 

independent peer reviewer, thus, is expected to be objective (See Sections 1.2.6 & 1.2.7 for 

further information) 
Page 22 Peer Review Handbook 

Independence is freedom from institutional, 

ideological, or technical bias regarding the issues The quality of the peer review is 

under review and is necessary for objective, fair, and dependent on the competence and 

responsible evaluation of the work product. If a independence of the reviewers. 

selected reviewer has a particular scientific or 

technical perspective, it may be desirable to balance 

the review with peer reviewers with other perspectives. Ideally, peer reviewers should be free of 

real or perceived conflicts-of-interest or there should be a balancing of interests among peer 

reviewers. If there are potential conflicts of interest (real or perceived), they should be fully 

identified to ensure a credible peer review. (See Sections 3.4.5 & 3.4.6 for further information). 

1.4.9 When does an Agency Internal Peer Reviewer Qualify as Independent? 

An Agency independent peer reviewer is one who comes from a different organizational 

unit than the one where the review question or document originates. A different organizational 

unit usually denotes, at minimum, a different office (i.e., above division level in programs; above 

branch level in regions) within the organization. In particular, a reviewer shouldn’t come from 

within the chain of command, either upward or downward. 

1.4.10 What is a Peer Review Panel? 

A peer review panel can range from a few individuals to ten or more, depending on the 

issue being investigated, the time available and any limitations on resources. Individuals who 

serve as peer reviewers must have appropriate scientific and technical expertise such that the 

review panel covers the broad spectrum of expertise required to treat the issues/questions 

presented in the charge. 

1.4.11 What is a Subject Matter Expert? 

A subject matter expert is one who has specific scientific and technical expertise in the 

matter under review. The importance of scientific and technical expertise in the subject matter is 

obvious, however, knowledge or just “knowing” about the subject area isn’t equivalent to 

expertise in the subject matter. For Agency decisions, a multi-disciplinary group of experts 

corresponding to the disciplines that contribute to complex Agency decisions is often necessary 

for a full and complete peer review. For example, a risk assessment that relies on both animal 

and human data usually requires experts in both areas for a complete review. For economic 

analyses, experts from the corresponding economic disciplines are necessary. 
Peer Review Handbook Page 23 

1.4.12 What is the Role of the Science Policy Council (SPC)? 

According to the 1994 Peer Review Policy statement: “The Science Policy Council is 

responsible for overseeing Agency-wide implementation. Its responsibilities include promoting 

consistent interpretation, assessing Agency-wide progress, and developing recommendations for 

revisions of the policy as necessary.” The SPC meets its responsibilities through coordination 

with the Peer Review Coordinators, the Peer Review Advisory Group (PRAG) and the Office of 

Research and Development (ORD). 

The SPC, PRAG and ORD are not responsible for identifying specific products for peer 

review or determining the level of review or mechanism for that review; those functions are the 

responsibility of management within each Office or Region. 

1.4.13 What is the Role of the Peer Review Advisory Group (PRAG)? 

The Science Policy Council has created the Peer Review Advisory Group (PRAG) to 

assist in the implementation of the Agency's Peer Review Policy. The primary role of the PRAG 

is to provide interpretation of the policy and to assist the SPC and Agency Offices and Regions 

in the annual update of the Peer Review Handbook. 

1.4.14 What is the Role of the Office of Research and Development (ORD)? 

The Deputy Administrator has designated the Office of Research and Development 

(ORD) to provide oversight to the Program Offices and Regions in the collection and review of 

information that is contained in the annual submission of Lists A, B, C, and D (for detailed 

information, see Section 1.3). The Office of Environmental Information (OEI) assists ORD in its 

oversight role. 

Peer Review Handbook Page 25 

2. PLANNING A PEER REVIEW 

2.1 Overview Statement 

Planning a peer review is a critical first step to ensure a successful peer review of a work 

product. The initial step is to determine whether your work product requires peer review. Once 

you have determined that a peer review will be conducted, the Decision Makers and Peer Review 

Leaders need to plan an appropriate review. This includes identification of resources (budget 

and personnel), the schedule for the completion of the peer review, the mechanism for peer 

review, the choice of peer reviewers, and the development of the peer review record. 

2.2 Determining Which Work Products to Peer Review 

2.2.1 What are Scientific and Technical Work Products? 

The first step in determining which work products require peer review, is to identify 

products that are scientific and/or technical in nature. Scientific and technical work products are 

used to support a research agenda, regulatory program, policy position or other Agency position 

or action. Scientific and technical work products include economic and social science work 

products. Categories of work products include, for example: risk assessments, technical studies 

and guidance, analytical methods, scientific database designs, technical models, technical 

protocols, statistical survey/studies, technical background materials, technical guidance (except 

for guidance providing policy judgments), research plans, and research strategies. 

Products that wouldn’t be considered scientific and technical work products can include 

those: that address procedural matters (e.g., planning, reporting, coordination, notification); that 

are primarily policy statements (e.g., relocation policy); that are conference proceedings (unless 

the proceedings are used as the scientific basis for an Agency action or decision); and that are 

decision documents (e.g., Record of Decision (ROD) -- the decision document itself is not 

subject to the Peer Review Policy, but the underlying scientific and/or technical support work 

product is a candidate for peer review). In addition, the following Agency documents are not 

considered scientific and/or technical work products under the Peer Review Policy: strategic 

plans, analytic blueprints, and goals documents. 

Making final determinations concerning which work products are scientific and technical 

is a responsibility of the Decision Maker (See Section 1.4.3) 
Page 26 Peer Review Handbook 

2.2.2 What Scientific and Technical Work Products Need Peer Review? 

The principle underlying the Peer Review Policy is that all major scientific and technical 

work products used in decision making will 

be peer reviewed. The process for identifying When in doubt about whether to peer which of these products is “major” (and thus review a work product or not, always a candidate for peer review) and then decide to make it a candidate for peer determining the mechanism of review will review. take into account various criteria and the 

circumstances surrounding the use of that 

work product. To maintain flexibility, the 

Decision Maker(s) for peer review should consider the full field of possible work products to 

identify those additional products that might still warrant peer review as well as the full spectrum 

of peer review mechanisms for each product. Once a decision is made to perform peer review, 

the product is listed in the annual submission of Candidate Products for Future Peer Review 

(List B - see Section 1.3.2b)). If a decision is made not to peer review a scientific and technical 

work product, the product is listed in List C - Products for Which a Decision has Been Made Not 

to Peer Review (see Section 1.3.2c)). 

2.2.3	 How Does One Determine Whether a Scientific and/or Technical Work 

Product is “Major”? 

Determinations of a scientific and/or technical work product as “Major” will largely be 

case-by-case. The continuum of work products covers the range from the obviously major, 

which clearly need peer review, to those products which are not major and clearly don’t need 

peer review. The rest of the work products fall in-between those two distinctions. This “middleground” probably represents the majority of work products, each of which needs to be evaluated 

closely and be compared to certain criteria (see below). The Decision Maker needs to make a 

judgment as to whether a work product meets the criteria for major or not. There is no easy 

single yes/no test of major covering the whole continuum of work products. A rule of thumb to 

remember -- if there is any doubt about whether a work product needs peer review, then go ahead 

and consider it a candidate for peer review (and place it on List B - Candidate Products for 

Future Peer Review). 

Scientific and technical work products that are used to support a regulatory program or 

policy position and that meet one or more of the following criteria are candidates for peer 

review: 

a) Establishes a significant precedent, model, or methodology 

b) Addresses significant controversial issues 
Peer Review Handbook Page 27 

c) Focuses on significant emerging issues 

d) Has significant cross-Agency/inter-agency implications 

e) Involves a significant investment of Agency resources 

f)	 Considers an innovative approach for a previously defined 

problem/process/methodology 

g) Satisfies a statutory or other legal mandate for peer review 

Usually, a major scientific and/or technical work product supports a regulatory decision 

or policy/guidance of major impact. Major impact can mean that it will have applicability to a 

broad spectrum of regulated entities and other stakeholders, or that it will have narrower 

applicability, but with significant consequences on a smaller geographic or practical scale. The 

scientific and/or technical work that underlies many of the Agency’s major rulemakings and 

policy and guidance documents of general applicability would be designated “major” under this 

scope of impact criterion because of their far-reaching or significant impacts. 

The novelty or controversy associated with the work product helps determine whether it 

is major or not. A major work product may be novel or innovative, precedential, controversial, 

or emerging (“cutting edge”). An application of an existing, adequately peer reviewed 

methodology or model to a situation that departs significantly from the situation it was originally 

designed to address is a candidate for peer review. Similarly, a modification of an existing, 

adequately peer reviewed methodology or model that departs significantly from its original 

approach is a candidate for peer review. Determination of “significant departure” as used in this 

Section is the responsibility of the Decision Maker. 

In summary, a major scientific or technical work product has a major impact, involves 

precedential, novel, and/or controversial issues, or the Agency has a legal and/or statutory 

obligation to conduct a peer review. 

2.2.4 What Economic Work Products Need Peer Review? 

Economic analyses and reports are considered scientific and technical work products, and 

as such they are candidates for peer review. The following economic work products will 

normally be classified as major and will require peer review: 

a) internal Agency guidance for conducting economic and financial analysis 
Page 28 Peer Review Handbook 

b)	 new economic and financial methodologies that will serve as a principal method 

or protocol used to conduct economic analyses within a program 

c)	 unique or novel applications of existing economic and financial methodologies, 

particularly those that are recognized to be outside of mainstream economic 

practices 

d)	 broad-scale economic assessments of regulatory programs, such as those required 

by Congressional-mandates (e.g., the Clean Air Act reports to Congress on 

benefits and costs) 

e)	 new stated preference (e.g., contingent valuation) and revealed preference surveys 

(e.g., recreational travel cost surveys) developed to assist in the economic analysis 

of a regulation or program 

f)	 new national surveys of costs and expenditures for environmental protection (e.g., 

financial needs surveys, pollution abatement expenditures surveys) 

g)	 economic research plans developed to assess and advance the state-of-science in 

economic theory, methodologies or modeling (in particular, the technical 

feasibility of the plan’s components) 

h)	 new meta-analyses that re-analyze existing published literature and supporting 

data on the measurement of economic benefits, costs and impacts 

Generally, because of the nature of these types of economic work products, you should 

conduct an external peer review. External peer reviews can be provided by the Science Advisory 

Board’s Environmental Economics Advisory Committee, or other appropriate outside 

organizations or individuals that have expertise in the technical economic issues raised in the 

economic work product. 

2.2.5	 Should Economic Work Products Prepared in Support of Regulations that 

are Classified as "Major" or "Economically Significant" be Peer Reviewed? 

Normally they would not require peer review, if the economic work product applies 

accepted, previously peer reviewed methods in a straightforward manner. Economic studies 

prepared to support “major” or “economically significant” regulations (“major” as defined below 

in this section) typically do not utilize innovative or untried economic methods. It is 

unnecessary to conduct peer reviews of straightforward applications or transfers of accepted, 

previously peer reviewed economic methods or analyses. Economic assessments prepared to 

support the regulatory development process routinely make use of previously published peer 
Peer Review Handbook Page 29 

reviewed literature and adopt tools that allow for the transfer or adaptation of these techniques 

and information. The procedures used to transfer or adapt this work will generally be 

established by separate economic guidance documents which have been peer reviewed. 

Therefore, economic documents that are developed using these procedures will not normally 

require additional peer review, even those prepared in support of “major” and “economically 

significant” rules. 

Even where peer review is not needed, additional peer input can be beneficial in the 

development of economic work products for “major” and “economically significant” rules. At 

present, some peer input of these analyses is already likely to be included as part of the 

regulatory development process, including input received from other EPA offices represented on 

the workgroup for the rule, the Agency’s Regulatory Steering Committee, and from the public as 

part of the public comment process for the rule. But there may be added benefit to employing 

additional peer input procedures, such as actively soliciting input from economists in other 

Agency offices on the quality and completeness of the economic analysis. 

“Economically significant” rules under Executive Order 12866 are defined as rules that 

may have an annual effect on the economy of $100 million or more or adversely affect in a 

material way the economy, a sector of the economy, productivity, competition, jobs, the 

environment, public health or safety, or State, local, or tribal governments or communities. The 

term “major,” as used in this Q&A, does not mean the same thing as the definition of “major” 

for purposes of deciding whether a work product requires peer review. Here, we are using the 

term as defined in the Congressional Review Act, which defines a “major rule” as one that has 

resulted in or is likely to result in: an annual effect on the economy of $100 million or more; a 

major increase in costs or prices for consumers, individual industries, Federal, State, or local 

government agencies, or geographic regions; or significant adverse effects on competition, 

employment, investment, productivity, innovation, or on the ability of U.S.-based enterprises to 

compete with foreign-based enterprises in domestic and export markets. 

2.2.6 What Other Economic Work Products Might Benefit from Peer Review? 

There may be other economic work products not covered in the preceding sections for 

which peer review might be useful. Examples of such work products are presented below; 

however, we do not intend to establish a presumption of peer review for these work products. 

a)	 analyses measuring the economic impacts and effectiveness of adopting marketbased or economic incentives as regulatory management instruments 

b)	 analyses of economic policies established under other government organizations 

(e.g., economic models used to study transportation, economic development, and 

international trade policies) 
Page 30 Peer Review Handbook 

Most of these types of economic work products do not exhibit the degree of complexity, 

or establish an innovative or untried approach, that would warrant a peer review. However, 

other factors, such as the potential significance of the analysis for cross-Agency or inter-agency 

practices, or the significance of the issues addressed, may make peer review desirable. 

2.2.7 What Other Social Science Work Products Need Peer Review? 

Typically, a social science work product is one that includes empirical, logic-based 

approaches to answer technical questions about human motivation, human behavior, social 

interactions, and social processes, which are relevant to the environmental issues being 

addressed. The term “behavior” includes overt actions; underlying psychological processes such 

as cognition, emotion, temperament and motivation; and biobehavioral interactions. The term 

“social” includes sociocultural, socioeconomic, and sociodemographic status; biosocial 

interactions; and the various levels of social context from small groups to complex cultural 

systems. Examples of social science work products include analyses and/or evaluations related 

to such topics as pollution prevention, risk communication, environmental information, 

environmental justice, quality of life, decision-making, and public participation. 

The following social science work products will normally be classified as major and will 

require peer review: 

a)	 internal Agency guidance for conducting social impact assessments and other 

community cultural assessments related to different environmental protection 

approaches such as community-based watershed protection (heretofore referred to 

as social assessments). 

b)	 new social science methodologies that will serve as a principal method or 

protocol used to conduct social assessments. 

c)	 unique or novel applications of existing social science methods such as surveys, 

focus groups, interviews, network analyses, comparative analyses, and content 

analyses. 

d)	 new national surveys of values, perceptions and preferences related to 

environmental protection. 

e)	 innovative research or analyses that address the human dimensions of 

environmental protection or environmental change in terms of social trends, 

future predictions and/or behavioral generalizations. 
Peer Review Handbook Page 31 

f)	 social science research plans developed to assess and advance the state-of-science 

in social science theory, methodologies or modeling (in particular, the technical 

feasibility of the plan’s components) 

2.2.8	 How Should Peer Review be Handled for Products Developed under an 

Interagency Agreement (IAG)? 

Under an interagency agreement (IAG), EPA provides funds to another agency for that 

agency to use for a specific purpose. The receiving agency's guidance for peer review will most 

likely be different from the EPA peer review policy. Therefore, if EPA plans to use any work 

products from that agreement, EPA must decide whether that document needs review under the 

EPA peer review policy (see Section 2.2.16). 

2.2.9	 Should Products from Contracts, Grants, and Cooperative Agreements 

Receive Peer Review? 

If there is a scientific and/or technical work product resulting from a grant, contract, or 

cooperative agreement and it is considered major and will likely be used in Agency decisionmaking, the work product needs peer review. Since it would probably result in a perceived, if 

not real, conflict of interest, a group that is generating the work product usually cannot conduct 

or perform the peer review of its own work product. Exceptions may be made in certain 

instances for organizations that have adequate and well established recognized procedures for 

peer review, such as the National Academy of Sciences (NAS). In practice, the Agency may 

need to peer review the product on its own, or arrange with an independent third group (e.g., via 

another extramural vehicle) to conduct the peer review. The Agency should not use the major 

scientific and technical work products from contracts, grants, or cooperative agreements to 

support decision making unless the work products are peer reviewed for both scientific and 

technical rigor and applicability to the specific use to be made of the product. 

Be aware that contracts are very different from grants and cooperative agreements. 

Please note that there are important legal restrictions (discussed in 2.2.10 below) on the direct 

use of work products developed under grants and cooperative agreements in the Agency’s 

decision-making process. 

2.2.10	 How Does Peer Review Apply to Products Generated Through EPA Grants 

or Cooperative Agreements? 

Major scientific and technical work products that are generated through EPA grants or 

cooperative agreements are candidates for peer review, but special considerations apply. 
Page 32 Peer Review Handbook 

First, you need to understand some background information on the proper use of 

assistance agreements (grants and cooperative agreements) versus contracts. Under the Federal 

Grant and Cooperative Agreement Act, grants or cooperative agreements may only be used 

where the principal purpose of the agreement is to accomplish a public purpose that is authorized 

by statute. EPA may derive some incidental use or benefit from the award as long as the 

principle purpose of the project is public support. However, if the principal purpose of the 

agreement is to obtain a product or service for the direct benefit or use of EPA, a contract must 

be used rather than a grant or cooperative agreement. 

EPA Order 5700.1, “Policy for Distinguishing Between Assistance and Acquisition,” 

includes the following examples of projects that cannot be performed using grants or cooperative 

agreements: 

a)	 Research and studies which gather specific information desired by EPA for its 

own use. 

b)	 Research which provides technical or analytical advice for EPA’s direct benefit or 

use, such as information used to set guidelines. 

c)	 Projects that produce specific information that will be directly incorporated by 

EPA into technical, policy, or regulatory decisions. 

Note that under the Order, EPA may legally provide financial assistance for research that 

is intended to stimulate or support development of scientific knowledge that is not primarily for 

EPA’s direct use or benefit. The resulting work products would be widely disseminated either 

through publication in scientific journals or through other means as opposed to a report tailored 

to EPA’s specific needs and requirements. EPA can consider these work products just as it can 

review other published scientific works when formulating its programs and policies. Further, 

EPA retains a royalty free, nonexclusive and irrevocable right to use the work products for 

Federal purposes, even if the recipient has copyrighted the material. [40 CFR 30.36(a).] 

Consult the Order and OGC for more information about making the proper choice 

between an assistance agreement and a contract. 

Provided that EPA’s use of a grant or cooperative agreement recipient’s work product is 

incidental to the agreement’s principal purpose, EPA may still determine that the recipient’s 

work product is a “major scientific or technical work product” under EPA’s peer review policy 

because: 1) it will be used to support an EPA program or policy position (assuming this use is 

incidental to the principal purpose of the agreement), and 2) it meets one or more of the seven 

criteria outlined in Section 2.2.3. In this situation, the work product requires peer review. 
Peer Review Handbook Page 33 

The following are options for peer reviewing the product: 

a)	 EPA can have the product peer reviewed with the participation of the assistance 

agreement recipient/author. In this case, EPA could arrange for an independent 

peer review of the product itself, or may contract with a third party to conduct the 

review. EPA would also enter into a contract with the author (formerly the 

recipient), which would task the author to prepare a response to the peer 

reviewers’ comments, and to revise or prepare an addendum to the product in 

response to peer reviewer comments as determined appropriate by EPA. 

A caveat to this approach is that it may be difficult to get the recipient/author to 

agree to allow EPA to determine how to revise the product in response to the peer 

review comments. 

b)	 EPA can have the product peer reviewed without the participation of the 

recipient/author. EPA could arrange for the peer review itself, or could contract 

with a third party to conduct the review. In this case, however, the work product 

would not be revised to incorporate the peer review comments. Instead, EPA 

would receive the comments and prepare a statement that documents its own 

response to the comments. The EPA Decision Maker(s) who is using the work 

product to support an EPA program or policy decision must be provided 

information on both the conclusions of the recipient’s work product and EPA’s 

own conclusions from the peer review. 

c)	 Recipients can get their products peer reviewed on their own. Recipients may 

determine on their own that peer review would benefit the credibility of their 

product. Provided EPA agrees that a peer review would further the public 

purpose of the assistance agreement, EPA may include funds for the peer review 

in the agreement. (See Section 2.2.11 for additional information.) Alternatively, 

the recipient may make arrangements for, and fund, an independent peer review 

of their product. In either case, EPA would need to evaluate whether the peer 

review process undertaken by the recipient was acceptable for the purposes for 

which EPA was planning to use the work product. EPA may accept the peer 

review if it determines that it fulfilled the requirements of EPA’s Peer Review 

Handbook and that EPA could defend the peer review as if it were conducted by 

EPA itself. 

Under options a) or b), issues may arise over obtaining access for peer reviewers to the 

w data used by the recipient to generate the work product. Under 40 CFR 30.36(c)(2), EPA 

s a right to obtain raw data produced by a non-profit organization or university under an 

sistance agreement, even where the agreement doesn’t specifically provide for this access. 

ra

ha

as
Page 34 Peer Review Handbook 

Nevertheless, it may be prudent to include a specific term in the assistance agreement clarifying 

this point to avoid misunderstandings. EPA must pay for obtaining access to the data if its 

transmittal imposes additional costs on the recipient. Assuming our use of the data is incidental 

to the principal purpose of the agreement, we also have a specific right to authorize peer 

reviewers to use the data for Federal purposes under 40 CFR 30.36(c)(2). 

The only time EPA cannot obtain access to the raw data is where EPA specifically 

bargains away this right in the assistance agreement. 

Again, consult OGC for help in drafting appropriate language for your assistance 

agreement. 

2.2.11	 Can the Recipient of a Grant or Cooperative Agreement Use Agreement 

Funds to Pay Peer Reviewers of their Work Products? 

As noted in Section 2.2.10 above, provided EPA agrees that a peer review would further 

the public purpose of the assistance agreement, EPA may include funds for the peer review in 

the agreement. A payment to peer reviewers in exchange for their review of a 

scientific/technical work product is allowable as a fee for professional services under assistance 

agreements. (To accurately characterize this cost, however, it is important that the payment be 

referred to as a fee, rather than an honorarium.) See OMB Circular A-21, Section J, item 32, 

Professional Services Cost (Educational Institutions), OMB Circular A-122, Attachment B, Item 

39, Professional Services Costs (Non-Profit Organizations), and OMB Circular A-87, 

Attachment B, Item 33, Professional Services Costs (State, Local and Indian Tribal 

Governments). (See http://www.whitehouse.gov/OMB/circulars/ for further details.) 

2.2.12	 Do Products Generated under EPA Grants or Cooperative Agreements Need 

to be Reported in the Peer Review Product Tracking (PRPT) Database? 

If a grant or cooperative agreement product is determined to be a major scientific and/or 

technical work product and is used in Agency decision making (assuming this use is incidental to 

the principal purpose of the agreement), it must generally be considered a candidate for peer 

review. Therefore: 

a)	 if it is determined to be a major work product and will be peer reviewed by EPA, 

the work product is entered onto List B (Candidate Products for Future Peer 

Review) as a “future peer review” product, then moved to List A (Products Peer 

Reviewed Since 1991) when the peer review is completed. 

b)	 if it is determined to be a major work product, but will not be peer reviewed (see 

the possible reasons for not peer reviewing a major product in Section 2.3.1), it is 
Peer Review Handbook Page 35 

entered onto List C (Products for Which a Decision has been made not to Peer 

Review), with a justification for not peer reviewing it. 

c)	 if it is determined to not be a major work product but EPA decides that it still 

warrants peer review (see Section 2.2.17), it is entered onto List B (Candidate 

Products for Future Peer Review) as a “future peer review” product, then moved 

to List A (Products Peer Reviewed Since 1991) when the peer review is 

completed. 

d)	 if EPA decides to use the work product (due to its importance in EPA 

environmental decision making), but considers it not major and not a candidate 

for peer review, it is entered onto List C (Products for Which a Decision has been 

made not to Peer Review) with the reasons for it being non-major. 

e)	 if the grant or cooperative agreement product will not be used in Agency decision 

making, it is not generally considered a candidate for peer review. Such products 

do not have to be listed. 

2.2.13 Should Site Specific Decisions be Subject to Peer Review? 

The site specific decision itself is not subject to peer review and doesn’t need peer review 

based solely on the Peer Review Policy. However, if a site specific decision is supported by a 

major scientific and/or technical work product, that work product needs peer review. While the 

same considerations for major apply here, several of the criteria above (see Section 2.2.3; 

specifically criteria b, c, d, and g) are considered more useful for regional consideration than 

other criteria. So generally speaking, a close examination of how the underlying major scientific 

and/or technical work product is adapted to the site specific circumstances is required. 

2.2.14 Should NEPA Products (e.g., EISs) be Subject to Peer Review? 

Not everything requires peer review, and in the case of an Environmental Impact 

Statement (EIS) prepared under the requirements of the National Environmental Policy Act 

(NEPA), the document already has received extensive review (although not necessarily peer 

review) through the “scoping” and interagency review processes that are part of NEPA. 

The rule of thumb is that if the underlying scientific and/or technical work product is 

major, then the work product needs peer review. In general, the Agency’s role in the NEPA 

document would suggest what sort of review the document gets. If EPA is developing the 

document as part of an EPA action/decision (EPA is the Lead agency under NEPA), and it meets 

the definition of "major," then it needs independent peer review. If it is not a major work 
Page 36 Peer Review Handbook 

product (little impact, non-controversial, etc.), then peer input/continuing involvement might 

well be appropriate. 

On the other hand, if EPA is reviewing an EIS from another Agency (EPA is not the 

Lead agency under NEPA), it is likely that we are reviewing for conflicts with EPA policy and 

general environmental concerns. However, EPA must ask if the underlying major scientific 

and/or technical work product that supports the EIS has been peer reviewed. If not, this would 

raise concern about the full credibility and soundness of the EIS based on the science and 

technical support. EPA should work with the other organization/agency to ensure that the major 

scientific and/or technical work product receives peer review adequate for EPA purposes. 

2.2.15 Should Environmental Regulatory Models be Peer Reviewed? 

Generally, yes. Specific guidelines for the peer review of environmental regulatory 

models have been published by the Agency. These can be found on the EPA web site under the 

Science Policy Council home page (http://www.epa.gov/ORD/spc). 

2.2.16	 Is Peer Review Needed for Other Organization’s Work Products that Have 

been Submitted to EPA for Use in Decision Making? 

Any scientific and/or technical work product that is used in Agency decision making and 

is considered major becomes a candidate for peer review regardless of whether the work product 

is produced by the Agency or another organization. Therefore, all major work products 

important to EPA environmental decision making that are independently generated by other 

organizations (e.g., other Federal agencies, interagency groups, State and Tribal bodies, 

environmental groups, industry, educational institutions, international bodies) need to be 

considered as candidates for peer review just as major work products generated by EPA are 

considered (these would then be included on List B (Candidate Products for Future Peer 

Review) if EPA is conducting or arranging for the peer review). 

If possible, when EPA knows that a work product is being generated by another 

organization and is of interest to EPA for future use, the appropriate EPA office(s) should work 

with that organization, and others, as appropriate (e.g., the states), to promote the use of peer 

review. For example, the Office of International Activities (OIA) as well as the impacted 

program or regional office(s) should be included when international products are being 

considered for EPA use. 

It is hoped that if the other organization has the work product independently peer 

reviewed, the peer review will meet the intent of the Agency’s Peer Review Policy and EPA’s 

proposed use of the product (i.e., the peer review is basically equivalent to what EPA would do). 

Agency staff from the appropriate office(s) should examine closely the particulars of the peer 
Peer Review Handbook Page 37 

review to ensure independence and a conscious effort to incorporate the peer reviewers’ 

comments into the final work product. If there are perceived, or real, conflicts of interest, this 

may preclude the use of that peer review and, in those instances, another peer review would be 

needed. See Section 3.4.7 for considerations for when an outside party conducts and/or funds 

peer review of their own work product and submits it to the Agency. 

If the outside organization does not have the major work product peer reviewed and EPA 

decides it needs peer review, the appropriate EPA office(s) will have to ensure peer review of 

that work product occurs prior to the Agency’s use of the work product in decision making. Peer 

review can be accomplished by asking the outside organization to do so, or if they refuse, EPA 

may conduct or arrange for the peer review. If EPA is conducting or arranging the peer review, 

the product should be entered on to List B (Candidate Products for Future Peer Review). 

2.2.17	 Can Work Products That are Not Determined to be Major Still be Peer 

Reviewed? 

Yes, they could be. Scientific and technical work products that do not come under the 

“major” distinction discussed above may nonetheless be candidates for peer review. For 

example, a project manager may decide to use peer review because of particular program needs 

and goals. Peer review may also be warranted because it adds substantial value to the work 

product. In these cases, the product should be entered onto List B (Candidate Products for 

Future Peer Review) as a candidate for future peer review. 

2.3 Determining Which Work Products Do Not Receive Peer Review 

2.3.1	 Are There Circumstances When a Major Work Product is Not Peer 

Reviewed? 

There may be circumstances where a work product is considered major, but a decision 

for no peer review can then be justified. For example: 

a)	 Additional peer review is not required with work that has been previously 

reviewed by recognized experts or an expert body. For example, a cancer risk 

assessment methodology or an exposure modeling technique that was the subject 

of earlier peer review would not require additional peer review, even if the 

product supported a significant Agency decision. 

b)	 Additional peer review is not required if an application of an adequately peer 

reviewed work product does not depart significantly from its scientific or 

technical approach. 
Page 38 Peer Review Handbook 

c)	 Additional peer review is not required when the scientific and/or technical 

methodologies or information being used are commonly accepted in the field of 

expertise (e.g., Control Techniques Guidelines (CTGs) or other such 

compilations). This would need the appropriate documentation to support the 

commonly held view. 

d)	 Most often, a major work product would not receive peer review when the 

regulatory activity or action which the work product supports is terminated or 

canceled -- no further action, including peer review, is necessary. 

e)	 In a few instances, statutory or court ordered deadlines or other time constraints 

may limit or preclude peer review of product that would otherwise be considered 

major. However, it is up to the Decision Maker(s) to make every attempt possible 

to assure that peer review of major products occurs taking into account these 

deadlines. With proper up-front planning for peer reviews of major products and 

the ability to tailor the method of review to the product and circumstances, it 

should rarely be the case that major products don’t receive some type of peer 

review due to time constraints. 

f)	 Very rarely, resource limitations may also restrict peer review. Programs or 

Regions will evaluate these circumstances on a case by case basis; decisions will 

be based on consultations involving line management, the Project Manager, the 

Peer Review Leader, and the Peer Review Coordinator. 

If peer review of a major scientific and/or technical work product is not conducted, a 

written justification must be placed in the “Justification for Non-Peer Review Product” Section 

of the data base entry for that product on List C (Products for Which a Decision has been made 

not to Peer Review). The justification is signed-off by the appropriate Decision Maker (see 

Section 1.3.2c)). 

2.3.2 What Products Normally Do Not Need Peer Review? 

Products that are not major scientific and technical work products normally do not 

require peer review under the intent of the Peer Review Policy. Most of these scientific and 

technical work products are then placed on List C (Products for Which a Decision has Been 

Made Not to Peer Review) with a written explanation of why it was determined to be not major 

(see Section 1.3.2c)). This justification is signed-off by the appropriate Decision Maker to 

assure that all scientific and technical work products received consideration for peer review. 

Some scientific and technical work products are not considered major and generally do 

not need to be placed on List C (Products for Which a Decision has Been Made Not to Peer 
Peer Review Handbook Page 39 

Review). These types of work products typically include: derivative products (i.e., a product that 

only summarizes an already peer reviewed product or products; fact sheets), compendiums of 

existing models, methods and/or technologies; minor intermediate products (e.g., a technical 

memorandum from a contractor) describing methods or results incorporated in a larger product 

which will be peer reviewed; or preliminary or incidental analyses prepared separately from the 

work product ultimately used to support an Agency action or decision (e.g., during the course of 

developing a rule, managers may direct staff to prepare various “what if” analyses; those that 

aren’t used in the work product do not need to be listed). 

2.3.3 Do Voluntary Consensus Standards Require Peer Review? 

Generally, no. The National Technology Transfer and Advancement Act of 1995 

(NTTAA) directs EPA to use available voluntary consensus standards in its regulatory activities 

unless to do so would be inconsistent with applicable law or otherwise impractical. For 

purposes of the NTTAA, voluntary consensus standards are defined as technical standards (e.g., 

materials specifications, test methods, sampling procedures, and business practices) that are 

developed or adopted by voluntary consensus bodies (such as ISO, ASTM). The general 

purpose of the NTTAA is to reduce private and governmental costs by avoiding having the 

government “reinvent the wheel.” Voluntary consensus standards would normally not require 

peer review because the underlying process used by issuing organizations to develop and 

approve these standards is generally considered adequate for purposes of the Agency’s peer 

review policy. EPA reserves the right to conduct a peer review if it determines that the standard 

is not a voluntary consensus standard under the NTTAA. 

2.4 Choosing a Peer Review Mechanism 

2.4.1 How Do You Determine the Appropriate Peer Review Mechanism? 

During the planning of a peer review, the Decision Maker and the Peer Review Leader 

may consider several mechanisms for the peer review of major scientific and technical work 

products. These options range from consultations with EPA colleagues not involved in 

developing the product to a large and formal panel of 

outside subject matter experts. The peer review effort 

might be a focused one-time evaluation, or could The mechanism of the peer 

encompass several examinations over the course of a review matches the 

project. In principle, peer review provides the greatest importance and complexity of 

credibility for major work products when it involves well- the major work product. 

qualified external reviewers, is intensive in its 

examination, and operates through a more or less formal 

process. As a practical matter, however, time and resource considerations in many cases impose 

limitations on what can be reasonably achieved. Arranging for the most appropriate and feasible 

peer review will involve good judgment and a willingness to consider substance, time, and 

resource tradeoffs. Developing a peer review plan that provides for appropriate depth, timing, 
Page 40 Peer Review Handbook 

and content is an important matter for early consideration by the Decision Maker and Peer 

Review Leader. Note that use of peer input, public or stakeholder involvement does not qualify 

as peer review. 

The approach best suited to a specific work product will depend on the nature of the topic 

and the intended final product. Generally, the more novel or complex the science or technology, 

the greater the cost implications of the impending decision, and the more controversial the issue, 

then the stronger the indication for a more extensive and involved peer review and for external 

peer review in particular. Certain work products will clearly lend themselves to extensive 

external peer review; generally these will be products with large impacts (e.g., those that support 

Tier 1 and Tier 2 rulemakings). Other major work products may not need a large scale external 

peer review and may utilize a less involved, less resource intensive review. The peer review of 

some products may be better served with some form of internal peer review or a combination of 

internal and external peer review. 

It is important to make the choice of peer review mechanism at the time that the work is 

planned (for products supporting rule makings, at the analytic blueprint stage) so that peer 

review costs and time can be budgeted into the work plan. Essentially, the level of peer review 

matches the impact and complexity of the major work product. For example, a rule under 

development carries considerable weight and deserves careful handling and attention; therefore, 

the supporting work product deserves similar care and attention for its peer review. Both 

internal and external peer review mechanisms are available, have been used in the past, and have 

served to address the needs and challenges of a particular peer review situation. Nevertheless, 

no single peer review mechanism is likely to work best in all situations. Some useful guidance 

includes: 

a)	 Major work products intended to support the most important decisions, or that 

have special importance in their own right, ordinarily should be the subject of 

external peer review. Generally, the more complex, novel and/or controversial 

the product, or the higher impact it has, the more the Decision Maker should 

consider implementing a large-scale peer review involving external experts. 

b)	 Major work products that are less complex, novel, or controversial, or have a 

lower impact may not need such a large-scale and external peer review. These 

products might be subject to one of the less extensive, less resource-intensive 

review processes. 

c)	 Group discussion with peer reviewers can be very helpful at some point in the 

peer review process. On the other hand, simply soliciting individual comments is 

easier, faster, and less expensive. Individual review is probably more appropriate 
Peer Review Handbook Page 41 

for peer review at the early stages of a product’s development or for products with 

less impact and complexity. 

d)	 Strict time constraints, such as a court-ordered deadline, can make a less involved 

or formal peer review mechanism imperative. But Decision Makers and Peer 

Review Leaders must make maximum efforts to assure that such a process is 

perceived as systematic and objective. 

2.4.2 What are Examples of Internal Peer Review? 

a)	 Independent experts from within the Agency (e.g., ORD experts on non-cancer 

effects of lead reviews a draft article on benchmark dose) 

b)	 An ad hoc panel of independent experts from within the Agency (e.g., an 

independent internal workgroup convened to examine the case for the 

classification of a chemical as a carcinogen) 

c)	 Technical merit review by scientists in an Agency laboratory (e.g., an initial 

review of the risk assessment for a regional incinerator by Agency scientists) 

2.4.3 What are Examples of External Peer Review? 

a)	 Independent experts from outside the Agency (e.g., a letter review by outside 

scientists) 

b)	 An ad hoc panel of independent experts outside the Agency (e.g., a group is 

convened to develop a consensus on the carcinogenicity of a particular industrial 

chemical) 

c)	 Agency-sponsored peer review workshops (e.g., a review of potential indicators 

of ecosystem damage) 

d)	 Review by an established Federal advisory committee such as the Science 

Advisory Board (SAB), FIFRA Scientific Advisory Panel (SAP), ORD’s Board 

of Scientific Counselors, or the Clean Air Scientific Advisory Committee (e.g., a 

review of a criteria document for a particular chemical risk) 

e)	 Agency-based federal advisory committee (other than those established and 

discussed in d above) 
Page 42 Peer Review Handbook 

f)	 Agency appointed special board or commission (e.g., a review of the risk 

assessment methodology prepared by the Clean Air Act Commission on Risk 

Assessment) Note: The Office of General Counsel should be consulted 

regarding EPA's authority to establish and finance the activities of a commission 

or board. 

g)	 Interagency committee (e.g., a review of prospective research plans by the 

Committee on the Environment and Natural Resources coordinated by the White 

House) 

h)	 A committee convened by another federal agency or government organization 

(e.g., a review of the Dioxin Reassessment by the Health and Human Services 

(HHS) Committee to Coordinate Environmentally Related Programs) 

i)	 Review by non-governmental groups (e.g., a Society of Risk Analysis review of 

cancer guidelines) 

j)	 Review by the National Academy of Sciences (e.g., a review of the state of 

current knowledge about children’s health risks from pesticide exposures) 

2.4.4 What is the Role of Peer Review by a Refereed Scientific Journal? 

Peer review of journal articles (written by EPA or non-EPA authors) performed by a 

credible, refereed scientific journal contributes to the scientific and technical credibility of the 

reviewed product. EPA considers peer review by such journals as adequate for reviewing the 

scientific credibility and validity of the findings (or data) in that article, and is therefore a 

satisfactory form of peer review. However, in some cases, peer review of an Agency work 

product that uses these articles may be required (see Section 2.4.5). 

EPA authored journal articles, whether used in an Agency work product or not, are 

tracked using List D (Scientific Articles and Reports That Are Peer Reviewed by Organizations 

Outside of EPA -- see Section 1.3.2d) for details). This shows that these items have been peer 

reviewed, and differentiates them from other Agency work products. In addition, it gives EPA 

an opportunity to highlight the extensive work it produces in the scientific literature. 

It is not necessary to list these papers individually on List D (Scientific Articles and 

Reports That Are Peer Reviewed by Organizations Outside of EPA), although some 

organizations may choose to do so. However, at a minimum, the total number prepared by the 

organization and peer reviewed by journals must be included on List D. The organization may 

ultimately need to identify the individual papers and where published, if requested (e.g., due to 

litigation, FOIA, etc.). 
Peer Review Handbook Page 43 

Prior to submitting an article to a journal for peer review, EPA employees are encouraged 

to have the article internally peer reviewed (see Section 1.4.9); such internal peer review is 

already common procedure in certain parts of EPA. Articles may also need examination in 

accordance with any organizational clearance procedures, especially when the author is 

presenting him or herself as an EPA employee. For EPA employees, conflict of interest 

regulations will also apply. 

2.4.5	 Do Agency Work Products Become Candidates for Peer Review when Peer 

Reviewed Journal Articles are Used in Support of that Work Product? 

In most instances, major Agency work products are candidates for peer review even 

when supported by peer reviewed journal article(s). Although the use of articles that have been 

peer reviewed by a credible journal strengthens the scientific and technical credibility of any 

work product in which the article(s) appears or is referenced, it does not automatically eliminate 

the need to have the work product itself peer reviewed. In most cases, journal peer review may 

not cover issues and concerns that the Agency would want peer reviewed to support an Agency 

action. Under these circumstances, the major scientific and/or technical work product in which 

the article(s) appears or is referenced becomes a candidate for peer review. A journal article 

authored by EPA employees would be used in the same manner as an article published by 

anyone else in a credible, well recognized journal. 

If an Agency work product is based solely on a single article that has received peer 

review by a credible journal (e.g., where a model is suggested for a singular use that fits a 

specific Agency need), peer review of the Agency work product may or may not be necessary 

depending on how closely you apply the findings from the article. If an Agency work product is 

based on two or more articles that have received peer review by a credible journal(s), the Agency 

work product generally becomes a candidate for peer review. Decisions to make (or not make) a 

work product a candidate for peer review needs to be documented in the peer review record. 

One important factor to remember with regard to the use of articles that have received 

journal peer review deals with the availability of documentation from that peer review. Ideally, 

EPA needs to maintain a clear, easily accessible record of the peer review to assure the 

credibility and validity of the peer review (see Section 2.5 for details on the peer review record). 

However, the documentation from a journal peer review would not normally be available to the 

Agency, so such documentation is not expected in the peer review record. 

2.4.6 When and How Often Should Peer Review Occur? 

The Decision Maker and Peer Review Leader have significant discretion in deciding on 

the timing and the frequency of peer review. Options abound, each with merits depending on the 

context and specified peer review objectives. In many situations, a single peer review event, 
Page 44 Peer Review Handbook 

beginning when the final draft work product becomes available, is the approach taken. 

However, it is increasingly apparent that peer review performed earlier in the work product 

development stages is a superior approach for some work products. There may be substantial 

incremental benefit to conducting more than one peer review during the whole process of work 

product development, particularly where it involves complex tasks, has decision branching 

points, or could be expected to produce controversial findings. In addition, early review would 

be beneficial at the stage of research design or data collection planning where the product 

involves extensive primary data collection. The Decision Maker and Peer Review Leader need 

to determine when the peer review(s) should occur, considering the type of work product under 

development and at what point in its development process a peer review would be most 

beneficial. 

Other types of work products that would benefit from early, up-front peer review in their 

development are scientific and technical planning products. Examples of such products are 

research proposals, plans, and strategies. Also, while not products per se, ongoing research 

programs can be peer reviewed. 

Remember though, that while more than one peer review can be beneficial, the 

distinction between peer input and peer review needs to be kept in mind. Experts providing 

input during the development or planning stages of the work product generally do not become 

peer reviewers of that product (see Sections 1.2.2 to 1.2.7 for full discussion on this distinction). 

2.4.7 What Factors are Considered in Setting the Time Frame for Peer Review? 

Several factors impact how quickly a peer review may be needed. These include 

deadlines for completion of a project, research program, or rulemaking, funding availability, 

availability of quality peer reviewers, and statutory and/or court-ordered deadlines. 

Peer review sometimes leads to new information and analyses. Reviewers may make 

recommendations for new research that would alter the work product and thus modify the 

scientific/technical basis for the action or rule it supports. For this reason, a completed peer 

review is desirable before issuing any proposal for public comment. If that is not logistically 

possible because of court or statutory deadlines, or other appropriate reasons, the Decision 

Maker should make every effort to complete the peer review before the close of the comment 

period. Because peer review comments on such work products could be of sufficient magnitude 

to warrant a revision to the proposed action or rule, Decision Makers should exercise diligence 

in completing the peer review prior to the proposal stage whenever possible. 
Peer Review Handbook Page 45 

2.4.8	 Which Office/Region or Other Agency is Responsible for Conducting the 

Peer Review? 

The organization of the Decision Maker is normally responsible for conducting the peer 

review. Responsibility for conducting a peer review can be negotiable when more than one 

Agency office or region or other agencies are involved. Usually, the degree of involvement by 

any of the organizations and agencies and their ability to fund peer review will often determine 

who has the lead for the peer review. 

2.5 Creating the Peer Review Record 

2.5.1 What is the Peer Review Record? 

It is the formal record (file) of decision on the conduct of the peer review, the type of 

peer review performed, and an explanation of how the peer review comments were addressed. It 

includes sufficient documentation for an uninvolved person to understand what actually 

happened and why. The Peer Review Leader (with the program manager if there is one) creates 

a separate, clearly marked peer review file Section within the overall file for development of the 

work. Once the peer review is completed, it is the responsibility of the Peer Review Leader to 

ensure that the peer review record is filed and maintained in accordance with the organization’s 

procedures. 

The Peer Review Record is separate from the entry in the Peer Review Product Tracking 

(PRPT) Database. While some information from the peer review record appears in the database, 

the peer review record is the official record of the peer review. 

2.5.2 How Can the Peer Review Record Improve the Peer Review Process? 

A good peer review record allows future reference to what happened during the peer 

review, and helps Decision Makers make appropriate use of peer reviewer input. In addition, a 

good record helps ensure that EPA’s Peer Review Policy is followed. The Peer Review Leader 

is responsible for ensuring that the peer review record for individual work products is collected 

and maintained until completion of the peer review effort. 

2.5.3 What Should Be in the Peer Review Record? 

The peer review record should include all materials considered by the individual peer 

reviewers of the peer review panel, as well as their written comments and other input. Such 

materials include, at a minimum (see also Section 4.3.1): 

a) The draft work product submitted for peer review 
Page 46 Peer Review Handbook 

b) Materials and information (including the charge) given to the peer reviewers 

c) Written comments, information, and materials received from the peer reviewers 

d)	 Information about the peer reviewers (such as reviewers’ names, affiliations, and 

a statement concerning potential conflicts and their resolution) 

e)	 Logistical information about conduct of the peer review (such as times and 

locations of meetings) 

f)	 A memorandum, or other written record, approved by the Decision Maker, 

responding to the peer review comments specifying acceptance or, where thought 

appropriate, rebuttal and non-acceptance 

g) The final work product 

When deciding if particular materials should be included in the record, the Peer Review 

Leader should consider whether the materials would help reconstruct the peer review process 

and results at a later time. If the materials may be helpful, they should be part of the peer review 

record. 

In addition to hard copies of materials, Peer Review Leaders need to maintain electronic 

copies of the materials (e.g., charge) that are necessary for the annual reports to the Peer Review 

Tracking Database. Peer Review Leaders consult with their Peer Review Coordinators to 

identify those materials. 

2.5.4	 What Should I Do with a Peer Review Record That Pertains to a 

Rulemaking Action? 

The Peer Review Leader should coordinate with their program’s docket office to see that 

proper docketing requirements are satisfied for a peer review of a work product supporting a new 

rule. The Peer Review Leader is also responsible for notifying the workgroup chair as well as 

the Peer Review Coordinator (for the annual report) that a peer review is completed. 

2.5.5 When Should the Peer Review Record Building Process Begin? 

An early start at developing and maintaining a peer review record will help ensure the 

record is complete and helpful. Ideally, the record begins when the decision to peer review a 

work product is made. The Peer Review Leader needs to construct the peer review record from 

this point on -- this will avoid potentially time-consuming reconstruction at a later point. Note 
Peer Review Handbook Page 47 

that the peer review record is not complete until it contains a copy of the final work product 

which addresses the peer review comments. 

2.5.6	 What are the Differences in Record Keeping for a Review by an Individual 

Compared to a Panel? 

Strictly speaking, a true peer review requires more than a single reviewer. A review 

conducted by one individual will rarely provide the depth of commentary required to improve 

the work product. In addition, you will not receive the range of views and richness necessary to 

ensure improvement. 

In the case of a review panel, there will often be conflicting comments that must be 

resolved. This resolution should be in the record. 

2.5.7 Where Should the Peer Review Record be Kept and For How Long? 

During the active conduct of the peer review, the Peer Review Leaders maintain the peer 

review record with themselves until the peer review is totally completed. Minimally, the file 

should be maintained until one year after the completed peer review is reported in the next 

annual reporting. After that, the peer review record should be maintained for a “reasonable 

period of time.” Establishment and maintenance of the archive where the peer review records 

ultimately reside are an organization’s responsibility (i.e., not that of an individual program 

manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions 

will decide the appropriate level of organizational responsibility and how they will meet any 

“routinely available” requirements. The peer review record may be kept with other records 

relating to the overall project, as long as it is easily and separately identifiable. 

There are also specific requirements regarding the use of dockets for record-keeping; 

however, these are not covered in this Peer Review Handbook. The documents contained in the 

peer review record should be maintained in accordance with the Agency’s record keeping 

retention schedule for such records (for details, see EPA’s National Records Management 

Program; http://www.epa.gov/records/). One long-term archiving mechanism may be the formal 

archiving at the Federal Records Center in Suitland, MD. 

2.5.8 Are Internal Peer Review Comments Included in the Peer Review Record? 

An internal EPA peer review must be a formal process to be considered a legitimate peer 

review. This process adheres to the guidance found in this Handbook for planning, conducting, 

and completing a peer review. When you follow this formal process to obtain peer review from 

EPA peers (see Section 1.4.9), then the whole record of that internal peer review is included in 

the peer review record. This includes all the materials detailed in Section 2.5.3 (also see Section 
Page 48 Peer Review Handbook 

4.3.1). Conducting a formal internal peer review is not the same thing as informal input from 

your EPA colleagues (i.e., “colleagues down the hall”), nor peer input from Agency personnel 

helping to develop the work product, nor organizational review and clearance processes. Such 

inputs from these informal processes should not be placed in the peer review record. The peer 

review record should contain only the information obtained when you conduct a formal internal 

peer review. 

In some cases, an internal EPA peer review may be followed by a separate external peer 

review. In this case, the formal record of the internal peer review should be included in the peer 

review record, however, the external peer review will generally stand as the peer review of 

greater significance since it is viewed more independent in nature, has broader fields of available 

expertise which can be brought to bear on the issues and often includes greater depth for specific 

disciplines. Peer Review Leaders should ensure that any applicable EPA record keeping 

regulations are followed (for details, see EPA’s National Records Management Program; 

http://www.epa.gov/records/). 

2.6 Budget Planning 

2.6.1 What Budgetary Factors Should I Consider in a Peer Review? 

Resources needed to implement the Peer Review Policy need to be requested through the 

usual Agency budgetary processes. The budget formulation process within the Executive Branch

is followed, after appropriation bills are passed by 

Congress, by budget execution. These two processes 

provide opportunities to secure resources for activities Peer review is part of the 

carried out by Headquarters and Regional offices, normal cost of doing 

including peer review. The major work products for which business. 

decisions for peer review have been made (List B 

(Candidate Products for Future Peer Review) candidates) 

need to have adequate funding for peer review in budget requests for the coming fiscal year. 

Similarly, adequate funding needs to appear in the actual approved operating budget to ensure 

their conduct. For purposes of budget planning, the costs of peer review would include the FTE 

cost of staff, the contract or other costs associated with the use of outside peer reviewers, and the 

administrative costs of conducting a review (copying, travel expenses, etc). 

2.6.2	 What Input is Needed for the Annual Budget Formulation and Budget 

Execution Process? 

Senior Management in Office and Regions (including Decision Makers and budget 

officers) need to be sure that budget requests include anticipated resources for peer review. Peer 

review needs to be considered as a normal part of doing business. Peer review resource 
Peer Review Handbook Page 49 

considerations should also be addressed in the analytic blueprint for Agency rule-making 

actions. 

2.7 Legal Considerations 

2.7.1 Are There Legal Ramifications From the Peer Review Policy? 

The Peer Review Policy does not establish or affect legal rights or obligations. Rather, it 

confirms the importance of peer review where appropriate, outlines relevant principles, and 

identifies factors Agency staff should consider in implementing the policy. Except where 

provided otherwise by law, peer review is not a formal part of or substitute for notice and 

comment rulemaking or adjudicative procedures. EPA’s decision to conduct peer review in any 

particular case is wholly within the Agency’s discretion. Similarly, nothing in the Policy creates 

a legal requirement that EPA respond to peer reviewers. However, to the extent that EPA 

decisions rely on scientific and technical work products that have been subjected to peer review, 

the remarks of peer reviewers should be included in the record for that decision. 

2.7.2 Is Legal Advice Needed? 

AA/RA staff and management should work regularly with individual OGC/Regional 

Counsel (RC) staff assigned to Agency activities. Peer Review Leaders should initially consult 

with their customary OGC/RC advisors for legal advice or referral. Headquarters attorneys have 

specialties in specific areas and can be consulted as needed (e.g., FACA considerations (see 

below); contractual responsibilities; ethics and potential conflicts of interest). 

2.8 Federal Advisory Committee Act (FACA) Considerations 

The Federal Advisory Committee Act, 5 U.S.C. App. 2, imposes certain open meeting 

(public announcement in the Federal Register), balanced membership, and chartering 

requirements (with the approval of the General Services Administration (GSA)) before the 

Agency establishes, controls or manages an “advisory committee” for advice or 

recommendations. Peer review carried out by formal and established (chartered) Federal 

advisory committees, such as the Science Advisory Board (SAB) or the FIFRA Scientific 

Advisory Panel (SAP), is always subject to FACA requirements. However, FACA does not 

apply to every EPA and contractor-run peer reviews. 

In this section you will find information on the applicability of FACA to EPA- or 

contractor-run peer reviews. 
Page 50 Peer Review Handbook 

2.8.1 When Do FACA Requirements Apply to EPA-Run Peer Reviews? 

In most cases, Federal Advisory Committee Act (FACA) requirements apply to EPA-run 

peer reviews that are conducted by formal and established (chartered) Federal advisory 

committees, such as the Science Advisory Board (SAB) or the FIFRA Scientific Advisory Panel 

(SAP). These groups have the following characteristics: 

a) the group is established, controlled, or managed by EPA; 

b) the group has a fixed membership, established purpose, and a set agenda; and 

c) the group strives to produce collegial, rather than, individual advice to EPA 

EPA run peer reviews that are not originally intended to be subject to FACA, but which 

exhibit the above characteristics, may unintentionally become subject to FACA. Questions 

concerning the applicability of FACA to peer review meetings should be addressed to the FACA 

experts in the Cross-Cutting Issues Law Office of OGC (Mail Code 2322A at Headquarters), or 

the appropriate Office of Regional Counsel. 

2.8.2 When Are EPA-Run Peer Reviews Not Subject to FACA? 

If EPA conducts a peer review with the purpose of obtaining advice from the individual 

peer reviewers and not for the purpose of obtaining a peer review product from the group (as a 

collective or consensus body), the peer review would, in most cases, not be subject to FACA. 

Peer review participants provide only their own views or recommendations and do not vote nor 

do they provide collective or consensus recommendations to EPA. When referring to the 

recommendations of the individual reviewers, EPA should not characterize these 

recommendations using terms such as “collective” or “consensus.” As a general matter, 

workshops and “letter reviews” that seek individual views or comments are usually not subject 

to the requirements of FACA. 

In addition to ensuring that peer reviewers only provide comments as individuals, EPA 

officials may wish to lessen the potential for a challenge under the Federal Advisory Committee 

Act (FACA) by seeking balanced participation at peer review meetings, and allowing interested 

members of the public to attend, and ensuring that they have access to appropriate materials. 

Non-FACA meetings may be announced in the Federal Register (providing that it is clear 

in the notice that such meetings are not subject to FACA) as it provides the public with useful 

information and a point of contact concerning the peer review. In addition, non-FACA (as well 

as FACA) meetings should also be advertised via other avenues (e.g., the Web, local 

newspapers, and mailing lists). 
Peer Review Handbook Page 51 

2.8.3	 How Do I Ensure that a Contractor-Run Peer Review Does Not Become 

Subject to FACA? 

Committees (or other peer review groups) established, controlled or managed by an 

outside organization (such as by an EPA contractor) to provide that outside organization with 

advice and recommendations (that will be submitted eventually to EPA as a contractor report) 

are not subject to FACA. Although FACA should not apply to contractor-run peer reviews, 

there are things that you (i.e., EPA) can inadvertently do that may invoke FACA. 

The following are considerations that you should be aware of when a contractor runs a 

peer review (e.g., letter review, panel, workshop, etc.) for EPA: 

a)	 the outside party’s peer review could be subject to FACA if EPA establishes, 

manages or controls the peer review group (e.g., EPA selects the members of the 

panel, runs the meeting, etc.). 

b)	 EPA should not provide contractors with a draft agenda or suggested format for 

peer review meetings. EPA contractors should manage and control the process, 

including running any meetings. 

c)	 At the request of the EPA contractor, EPA can provide a briefing to the peer 

reviewers (e.g., in a conference call with the contractor on the line) on the history 

or background of the development of the peer review document. EPA only 

should provide technical or background information and not use the call to take 

over the contractor’s peer review group. Not only should the contractor be “on 

the line” but it should be very clear to all participants that the contractor is in 

charge of the call. The contractor, not EPA, should invite people to participate, 

make all administrative arrangements, conduct the meeting, and control the 

agenda. 

d)	 EPA employees may attend the peer review panel meetings or workshops. 

However, they may not take over the control of the meeting. The contractor 

should call on them to speak when appropriate, but they should limit their 

participation to providing technical and/or background information, and not 

attempt to, or appear to, take over the contractor’s meeting. 

e)	 Since FACA does not apply when a contractor establishes, controls, or manages a 

peer review, the contractor need not avoid terms such as “collective” or 

“consensus” when reporting agreement among its peer reviewers 

(subcontractors). 
Page 52 Peer Review Handbook 

f)	 EPA may provide comments to the contractor on the contractor’s peer review 

only to the extent that the Agency is verifying that the contractor has completed 

satisfactorily the report in accordance with the work assignment. EPA should not 

attempt to make changes in the contractor’s conclusions; this would compromise 

the independence of the peer review conducted by the contractor. 
Peer Review Handbook Page 53 

3. CONDUCTING A PEER REVIEW 

3.1 Overview Statement 

The success and usefulness of any peer review depends on the quality of the draft work 

product submitted for peer review, the care given to the statement of the issues or "charge," the 

match between the peer review draft product and the form of peer review, the match between the 

peer review draft product and the scientific/technical expertise of the reviewers, and Agency use 

of peer review comments in the final product. It is not simply enough to conduct a peer review; 

each of the foregoing elements requires serious attention. 

3.2 Charge to the Peer Reviewers 

3.2.1 What is a Charge? 

As part of each peer review, the Peer Review Leader must formulate a clear, focused 

charge that identifies recognized issues and invites comments or assistance. This request signals 

the Agency's awareness of potential issues and its receptivity to expert recommendations. The 

charge to peer reviewers usually makes two general requests. First, it focuses the review by 

presenting specific questions and concerns that the 

Agency expects the reviewers to address. Secondly, 

it invites general comments on the entire work The time spent preparing a good product. The specific and general comments should charge is well spent, and is crucial focus mostly on the scientific and technical merits of for an effective peer review. the work product and, where germane, whether the 

scientific/technical studies have been applied in a 

sound manner. Remember, the peer review is not for 

the decision or action itself, but for the underlying scientific and/or technical work product. 

Focused questions greatly simplify the task of collating, analyzing and synthesizing peer review 

comments on a topical basis. The questions should be specific enough to get helpful comments, 

but not so specific (unless very specific points are needed to be addressed) that they preclude 

creative responses. Moreover, the written responses to these questions by peer reviewers help 

the Agency create a peer review record. As a general rule, the time drafting a good charge letter 

is well-spent and is necessary for an effective peer review. 

3.2.2 What are the Essential Elements of a Charge? 

a)	 Brief overview or introduction (describe what the work product is, how it was 

developed, how it will be used) 
Page 54 Peer Review Handbook 

b)	 As needed, a brief description or listing of any background materials provided to 

the peer reviewers 

c) The issues or questions to be addressed by the peer reviewer(s) 

1) The due date of reviewers’ comments 

2) The format of reviewer responses 

3) The point of contact in case peer reviewers have questions 

3.2.3 Where Can I Get an Example of a Charge? 

Appendix C (Examples of Charges) contains examples of successful charges that cover a 

variety of issues. Appendix D (Guidance on Requesting a Review by the Science Advisory 

Board) provides guidance for obtaining Science Advisory Board (SAB) services. 

3.2.4 Can a Stakeholder Provide Input to the Charge to the Peer Reviewers? 

Yes. EPA may decide to obtain stakeholder input on the charge to the peer reviewers, 

but EPA must make the final determination on what elements to include in the charge to ensure 

that it meets the needs of the EPA peer review. 

a)	 If you obtain stakeholder input, you must include any and all interested parties to 

the extent feasible based upon statutory, regulatory, budgetary and/or time 

constraints. Do not limit input to one stakeholder only (e.g., a responsible party 

or environmental group). 

b)	 If EPA has hired a contractor to perform the peer review, it should still be EPA 

personnel who obtain stakeholder input and provide the list of charge questions to 

the contractor. 

c)	 If you form a committee of stakeholders to help develop the charge, be aware that 

your committee may become subject to the requirements of FACA (see Section 

2.8). 

3.2.5	 Who Writes the Charge When I Hire a Contractor to Conduct the Peer 

Review? 

In general, if EPA hires a contractor to perform the peer review, EPA must allow the 

contractor independence in conducting the review. However, with regard to the charge, EPA can 
Peer Review Handbook Page 55 

and should provide the contractor with the substantive list of questions that EPA wants included 

in the charge letter to ensure that the peer review meets EPA’s needs. The list of charge 

questions can be incorporated into the Statement of Work if the EPA project manager has the list 

ready at that time. Based on this list, the contractor would then prepare and submit the actual 

charge letter to the peer reviewers. Prior to submitting the charge letter to the peer reviewers, the 

contractor should give EPA an opportunity to review the charge letter to ensure that it meets 

EPA’s needs. EPA cannot submit the charge directly to the peer reviewers. 

3.2.6	 Is it Okay For Me to Ask a Contractor to Develop the Charge to the Peer 

Reviewers? 

No. EPA will provide the charge questions to the contractor as discussed above. The 

contractor, though, may provide assistance and advice in the development of the charge. 

3.3 Time Line 

3.3.1 What are the Factors in Scheduling a Peer Review? 

The peer review schedule is a critical feature of the process. The schedule must take into 

account the availability of a quality draft work product, availability of appropriate experts, time 

available for using peer review comments, deadlines for the final work product, and logistical 

aspects of the peer review (e.g., contracting procedures). 

The schedule for peer review should take into account the overall rulemaking (or other 

decision making) schedule. For rules, in particular those in Tier 1 and Tier 2, the scheduling of 

the peer review should be included in the development of the analytic blueprint. Peer review 

sometimes leads to new information and analyses, or recommendations for new research that 

would alter the work product and thus modify the scientific/technical basis for the action. For 

this reason, it is usually advisable to complete the peer review before taking public comment, or 

at least before the close of the public comment period. 

3.4 Selection of Peer Reviewers 

3.4.1 What are Considerations for Selecting Peer Reviewers? 

Selection of independent peer reviewers is not a trivial task, and it is crucial to an 

effective peer review. It is important that peer reviewers be selected for independence and 

scientific/technical expertise. Therefore, EPA should always make every effort to use peer 

reviewers who do not have any real or perceived bias or conflict of interest and who are 

completely independent. However, the very need to have experienced individuals on a peer 

review, along with the desire to have appropriate technical balance and representation, can mean 
Page 56 Peer Review Handbook 

that the selection of potential peer reviewers often comes from those who are considered as 

having a potential bias. To reduce the concern that a potential panel may have unnecessary bias, 

it may be useful to obtain an informal review of the expertise and balance of potential peer 

reviewers from others in your organization, from OGC or even from outside groups. Sometimes 

selecting individuals who have served in a variety of organizations rather than a single one for an 

extended period, provides expertise with diverse perspective. The emphasis on independence 

and expertise applies equally to government experts and experts from the larger scientific 

community. 

Some peer reviews can be conducted with two or three reviewers; others involve panels 

of peer reviewers. In either case, each peer reviewer should have recognized technical expertise 

that bears on the subject matter under discussion. The peer reviewers of a work product should 

represent a balanced range of technically legitimate points of view. In addition, cultural 

diversity and "address" (e.g., industrial, academic, or environmental community) are other 

factors that can play a role in selecting peer reviewers. 

3.4.2 Where Do I Find Peer Reviewers? 

Recommendations for potential peer reviewers can be identified from a number of 

organizations. These include external groups such as the affected party(ies), special interest 

groups, public interest groups, environmental groups, professional societies, trade or business 

associations, state organizations or agencies, Native American Tribes, colleges and universities, 

the National Research Council, and other Federal agencies with an involvement in or familiarity 

with the issue. Agency associated groups include the staff of the Science Advisory Board (SAB) 

or the Scientific Advisory Panel (SAP), and relevant scientific and technical experts from 

Program or Regional offices. 

In certain circumstances, existing peer review organizations such as the SAB or SAP may 

be used to conduct a peer review. These groups establish their own criteria for accepting work 

and coordination must be made directly with them (see Appendix D - Guidance on Requesting a 

Review by the Science Advisory Board) for SAB procedures. Both SAB and SAP conduct 

formal, public, external peer reviews. 

Occasionally, a member of the scientific community will offer his/her services for peer 

review during an ongoing peer review. These offers may be at no cost or based on an 

expectation that reimbursement will be made. Disposition of these unsolicited offers will be 

handled on a case by case basis by the Peer Review Leader, and as necessary, in consultation 

with the Peer Review Coordinator, the Office of General Counsel (OGC), and appropriate 

Decision Makers. 
Peer Review Handbook Page 57 

If you use a contract mechanism to conduct a peer review, the contractor will have its 

own pool of scientific and technical experts for peer review. With contractors, EPA can provide 

information on potential sources of peer reviewers for conducting a peer review if such a listing 

is prepared in alphabetical order. In no way should EPA insist that the contractor select from 

EPA’s list, or require that the contractor receive EPA approval before selecting any peer 

reviewer (sometimes known as a “subcontractor”). However, EPA should review the list of 

reviewers for conformance to work assignment specifications and adherence to conflict of 

interest concerns. Furthermore, when utilizing a contract mechanism to conduct peer review, 

EPA is not permitted to direct the prime contractor to a specific subcontractor (or peer reviewer) 

nor is EPA permitted to direct the peer reviewer (subcontractor). All interactions with the peer 

reviewers must be coordinated through the prime contractor. (See Section 3.6 for further 

information) 

Keep in mind that for contracting purposes, contractors are required to obtain 

Contracting Officer (CO) approval of subcontractors and the CO generally seeks the input of the 

work assignment manager (WAM -- the WAM may be the Peer Review Leader in many cases) 

before approving the use of subcontractors. In this case, as noted above, the Peer Review Leader 

should review the list of potential reviewers for conformance to the work assignment 

specifications and adherence to conflict of interest concerns. 

3.4.3 Are External or Internal Peer Reviewers Preferred? 

External peer reviewers are generally preferred, particularly for most final work products. 

For some work products, like those reviewed at interim steps, either external or internal peer 

review may be appropriate. Selection of internal peer reviewers should be based upon technical 

expertise, available time and "address" -- that is, they should not come from the immediate office 

or group producing the product or have any other connection with the product or document being 

peer reviewed. External peer reviewers should be selected based upon technical expertise as 

well, however, care must be taken not to use individuals who have been involved in the 

development of the work product. (See Section 1.2.6; see also Sections 1.4.6 to 1.4.9). 

3.4.4 What is Important in the Mix of a Peer Review Panel? 

A peer review panel or group can number from just a few individuals to ten or more, 

depending on the issue, the time and resources available, and the broad spectrum of expertise 

required to treat the range of issues/questions in the charge. Objective technical expertise and 

lack of a conflict of interest are critical in selecting peer reviewers. Naturally, experts whose 

understanding of the specific technical area(s) being evaluated are necessary; nevertheless, it is 

also important to include a broad enough spectrum of other related experts to completely 

evaluate the relevant impacts on other less obvious concerns (i.e., to comment not only if the job 

is being done right, but also whether the right job is being done). For example, for health related 
Page 58 Peer Review Handbook 

peer reviews, experts in such fields as ecology and economics may provide very useful insights. 

Although persons who are familiar and have a substantial reputation in the field are often called 

upon repeatedly to be reviewers, it is important to keep a balance with new people who bring 

fresh perspectives to the review of a work product. 

There is usually a continuum of views on any issue. To the extent possible or 

practicable, selected experts should have technically legitimate points of views that fall along the 

continuum. A review panel should include experts that are considered “mainstream” (nearer the 

center of the continuum) as well as those further along the continuum (while generally avoiding 

the extremes). This will help maintain a balanced review panel, while allowing all views to be 

expressed and discussed. A balanced panel will allow consensus building (if consensus is the 

object of a particular peer review; if not, it allows a spectrum of (re)views for the Agency to 

evaluate). As a general rule, experts who have made public pronouncements on an issue (e.g., 

those who have clearly "taken sides") may have difficulty in being objective and should be 

avoided. 

3.4.5 What is a Conflict of Interest? 

The matter of obtaining a fair and credible peer review, as well as maintaining the 

credibility of the Agency and the Agency’s scientific products, is of paramount importance. Peer 

review leaders are strongly encouraged to obtain peer reviewers who do not have a legal or 

perceived conflict of interest (i.e., creates the appearance that the peer reviewer lacks impartiality 

or objectivity). In reality, we recognize that a totally independent peer reviewer is rare and very 

difficult to identify (see Section 3.4.6 for steps to take to ensure a credible peer review). 

Conflict of interest is a situation in which, because of other activities or relationships 

with other persons, an individual is unable or potentially unable to render impartial assistance or 

advice to the Agency, or the person’s objectivity in performing the work is or might be otherwise 

impaired, or a person has an unfair competitive advantage. Generally, a conflict of interest of 

interest arises when the person is affected by his/her private interests, when he/she or his/her 

associates would derive benefit from incorporation of their point of view in an Agency product, 

or when their professional standing and status or the significance of their principal area of work 

might be affected by the outcome of the peer review. Clearly, peer reviewers should not be 

placed in the position of reviewing their own research and analyses that form the basis of the 

work product under review as this might impair their objectivity. Whenever there are questions 

about conflicts of interest, you should contact the appropriate official in OGC for clarification. 

Be aware that COI can be used in a generic fashion in speaking of common, everyday 

experience that also involves elements of ethics. In most cases, that is how it is treated in the 

Peer Review Handbook. However, COI is a very specific issue when used in a governmental 
Peer Review Handbook Page 59 

contracting sense. In the contracting sense, the Federal Acquisition Regulations (FAR), the 

Environmental Protection Agency Acquisition Regulations (EPAAR), and other internal Agency 

documents define COI and describe the Agency’s COI policies and procedures. The Handbook 

provides guidance in some of these aspects related to peer review (see Section 3.6 and Appendix 

E - Example Statements of Work for Contracts), however, specific questions should be 

addressed to the Office of Acquisition Management. 

3.4.6	 What Techniques Help Ensure Disclosure and Appropriate Resolution of 

Conflicts of Interest? 

Before finalizing the selection of reviewers, the Peer Review Leader should ascertain 

whether each potential peer reviewer’s involvement in certain activities could pose a conflict of 

interest (COI) or create the appearance that the peer reviewer lacks impartiality. One way of 

identifying conflicts is to ask potential reviewers about current and prior work, and prior clients 

that might create conflicts or the appearance of a lack of impartiality in carrying out peer review 

activities. This information obtained by the Peer Review Leader becomes part of the peer review 

record. When the peer review process is being conducted by a contractor, the requirement for 

addressing peer reviewers’ possible conflicts of interest should be highlighted in the Statement 

of Work of the work ordering instrument (e.g., Work Assignment, Delivery Order, Task Order, 

etc.) and is a matter that is bound by contractual clauses with the Contracting Officer as the final 

Decision Maker in contracting matters. 

Care must be taken to reduce actual or potential organizational or personal conflicts of 

interest between the reviewers and the work product under review. Remember, each potential 

conflict situation is unique and must be treated on a case-by-case basis. The following are 

considerations that should be addressed in evaluating COI: 

a)	 attention to the employment, financial, and professional affiliations of the 

participants; 

b)	 exploring directly the issue with each of the participants before the review 

process takes place; 

c)	 disclosing publicly at the beginning of meetings any previous involvement with 

the issue; 

d)	 in the cases of regular government employees and Special Government 

Employees (SGEs), filing annual (or updated, as appropriate) Confidential 

Financial Disclosure Forms (OGE Form 450), discussing any potential conflicts 

of interest with their designated ethics official (DEO), and advising the Peer 

Review Leader of any relevant concerns; 
Page 60 Peer Review Handbook 

e)	 in the case of non-Federal peer reviewers, provide them a copy of the peer review 

COI inquiry (see Appendix F - Useful Forms). This form is sent to each 

prospective peer reviewer by the Peer Review Leader (or contractor, in the case of 

contractor-run reviews) to advise them of the need to address COI issues prior to 

the actual review taking place. A follow-up contact with the Peer Review Leader 

(or contractor, in the case of contractor-run reviews) is then made to discuss any 

relevant issues. The Peer Review Leader then documents this effort in the peer 

review record; this includes a summary provided by the contractor documenting 

their inquiries and efforts. 

The Peer Review Leader needs to ensure that the peer review COI inquiry (see Appendix 

F - Useful Forms) took place and this appears in the peer review record. You should ensure that 

any documentation in the peer review record does not contain information subject to privacy 

laws (for example, see The Privacy Act of 1974, 5 U.S.C. § 552a - Records maintained on 

individuals; a copy can be found at: http://www.usdoj.gov/04foia/privstat.htm). Peer Review 

Leaders should ensure that any applicable EPA record keeping regulations are followed (for 

details, see EPA’s National Records Management Program; http://www.epa.gov/records/). For 

information on Freedom of Information Act (FOIA) requests, please see EPA’s Ethics Advisory 

88-12 (http://intranet.epa.gov/ogcrmo01/ethics/88-12.htm). If you have any questions, be sure to 

ask your appropriate Office of General Counsel and/or office contract official(s). 

Established peer review groups such as the Science Advisory Board provide useful 

models for addressing balance and COI issues. Assistance in identifying conflicts of interest and 

in providing an appropriate response can be obtained from the Office of the General Counsel. 

Additional advice can be obtained from one of the Agency’s designated ethics officials (DEOs). 

Assistance in evaluating conflicts of interest in a contractual situation require the involvement of 

the Contracting Officer and the resources available within the Office of Acquisition 

Management. 

Of course, conflicts do not necessarily arise merely because a peer reviewer knows 

something about the subject matter. In fact, experts with a stake in the outcome -- and therefore 

a potential conflict -- may be some of the most knowledgeable and up-to-date experts because 

they have concrete reasons to maintain their expertise. Such experts could be used provided the 

potential conflicts of interest are disclosed and the peer review panel or group being used as a 

whole is balanced. In some cases, however, the conflict may be so direct and substantial as to 

rule out a particular expert, for instance, a potential peer reviewer who may have a client or 

employer with a direct financial stake in the particular specific party matter under review (e.g., 

i) a Federal grant or contract to the potential peer reviewer or his/her employer that relates to the 

matter under review; ii) the potential peer reviewer’s or their company’s work on a specific 

chemical under review). However, review of a general methodology that applies to numerous 
Peer Review Handbook Page 61 

chemicals would not necessarily raise such a concern. (Note: COI language should be made part 

of contracts/statements of work (SOW) or purchase orders (PO) -- see Section 3.6.) 

A Peer Review Leader may also want to adopt measures that will prevent creation of 

conflicts as the peer review is underway. Any measures contemplated that involve a contractual 

action must be coordinated with the cognizant Contracting Officer. Some measures might 

include clauses in a contract or purchase order that require reviewers to receive advance approval 

on future work, or place limits on such work, while they are performing the current peer review. 

Note that at some level these types of measures will discourage experts from serving as peer 

reviewers. (See Section 3.6.5 for further information dealing with contracts and suggestions for 

appropriate management controls.) 

3.4.7 Can Parties External to EPA Pay for Their Own Peer Reviews? 

There may be instances where parties external to EPA will want to conduct and/or pay 

for a peer review on a particular work product (presumably their own work product or one they 

are closely interested in, or they wouldn’t be interested in expending resources). This may look 

benign at first blush, but is a very complex and sensitive situation that can raise significant 

concerns for perceived and/or actual conflicts of interest for interested parties “paying” for a peer 

review of their own work product. While the Agency cannot prevent external parties from 

conducting and paying for a peer review, it is desirable that any such peer review meets the 

intent of the Agency’s Peer Review Policy and adheres to the principles and guidance in this 

Handbook. If the external party submits their work product and accompanying peer review, the 

materials will be treated by the Agency as anything else submitted for the Agency’s evaluation. 

We will evaluate the work product and the peer review for scientific credibility and 

validity before making any decisions based on the materials. 

[This issue is under active Agency consideration at this time; therefore, users of this Handbook 

should be aware that the language in Section 3.4.7 will likely change in the future.] 

3.4.8 Are There Constraints to Selecting Peer Reviewers? 

Sometimes the need for a peer review is accelerated due to a court-ordered deadline or 

other time-sensitive requirements. In such cases, it is difficult, if not impossible to obtain 

external peer reviewers in time to conduct a full external peer review. It may even be impossible 

to conduct a small scale internal peer review using just a few individuals. Mechanisms for 

identifying and using a small number of peer reviewers should be developed so that quick, 

effective peer review can be included for even the most rapidly moving products. 
Page 62 Peer Review Handbook 

Another possible constraint involves confidential business information (CBI). To 

evaluate certain Agency-generated studies properly, some peer reviewers may need access to 

CBI. However, unless the reviewers are Federal employees with CBI clearance, the Agency 

doesn’t have the independent authority to disclose CBI to them. Therefore, whenever 

contemplating the use of outside peer reviewers, Agency staff should determine whether the 

reviewers will need access to CBI. If they don’t have CBI clearance, the Office of the General 

Counsel should be consulted on whether it is practical to obtain the consent of CBI submitters to 

disclose the information to peer reviewers. 

Offices need to be aware of the requirements of the Federal Advisory Committee Act 

(FACA) when establishing peer review mechanisms (see Section 2.8). Federal advisory 

committees that are subject to chartering by the General Services Administration must hold 

meetings that are open to the public, and have balanced membership requirements. The Office 

of the General Counsel should be consulted regarding the applicability of FACA to peer review 

panels. 

3.4.9	 If State Employees are Used as Peer Reviewers, Can EPA Pay Them for this 

Service? 

Maybe. First, the Peer Review Leader needs to determine: a) if the State agency has a 

policy on whether their employees can perform this type of work for EPA, and b) if the person 

will be acting as an individual or as an employee of the State. If the State allows the person to 

be a peer reviewer for EPA and the person is acting as an individual, we can pay them for their 

peer review services. If the State allows the person to be a peer reviewer and the person will be 

acting as a State employee, we can only pay them if State policy allows us to. The most efficient 

vehicle for paying the State employee will likely be a contract or purchase order (see Sections 

3.6.6 & 3.6.7), but you should consult with OGC and the procurement office for advice. In most 

instances, EPA can also pay travel expenses (consult with your administrative staff for details). 

If we don’t pay a State person for their peer review services, they must sign an agreement 

stating that they don’t expect payment (See Section 3.6.1 - gratuitous services). 

3.5 Materials for Peer Reviewers 

3.5.1 What Instructions Do You Give Peer Reviewers? 

The Peer Review Leader is responsible for ensuring that peer reviewers understand their 

responsibilities (see Section 3.6 if a contract is involved): 
Peer Review Handbook Page 63 

a)	 Advise the Agency of actual or potential organizational or personal conflicts of 

interest or other matters that would create the appearance of a lack of impartiality 

(e.g., see Section 3.4.6) 

b)	 Provide written comments in specified format by the specified deadline that are 

responsive to the charge 

c) Comply with any requests for not disclosing draft work products to the public 

3.5.2 What Materials Should be Sent to Peer Reviewers? 

For a peer review to be successful, peer reviewers should receive several documents at 

the beginning of the process. Typically, the most important among these documents are the 

charge letter and the current work product. The charge letter describes what the peer reviewers 

are being asked to do, and should serve to focus and structure the review. The work product is, 

of course, the material being subject to peer review. 

Remember, no documents should be provided directly to a potential peer reviewer when 

that reviewer is going to be working under a contract or purchase order. In the case of a contract, 

the Agency provides the work product with associated background material to be peer reviewed 

to the prime contractor who in turn distributes these documents to the peer reviewers. In the 

case of a purchase order, the “charge or statement of work” must be part of the PO (purchase 

order) and the provision of any documents needs to be coordinated with the purchasing agent 

handling the order. 

a) Essential documentation for each peer reviewer includes: 

1)	 A current copy of the work product to be peer reviewed with associated 

background material. The work product needs to be of the best possible 

scientific/technical quality to ensure an adequate and useful peer review. 

2)	 A clear charge or statement of work seeking informed comment on 

identified issues to properly focus the efforts of the peer reviewers and 

ensure that their individual efforts can be compared or contrasted. 

3)	 Some information concerning the process that you use for the peer review, 

including the due date of reviewer comments, the format of those 

responses, and a point of contact in case the peer reviewer has questions. 

Responses should be written and submitted to the Peer Review Leader by 

an agreed upon deadline. In certain rare cases, oral commentary may be 
Page 64 Peer Review Handbook 

sufficient. However, in such cases, a follow-up written response for the 

record is required. 

4)	 In some cases, Agency materials being peer reviewed will be available to 

the public, even if they are marked as drafts. For example, all materials 

reviewed by the SAB are available. Agency managers may also decide 

that a broad accessibility has benefits for the Agency. In other cases, 

confidentiality needs to be maintained. In these cases, each peer reviewer 

must be informed of the need for confidentiality with regard to the release 

of Agency products that are stamped as "DRAFT" or "DRAFT - Do Not 

Cite, Quote, or Release." Premature release of draft Agency products, 

views, or positions is inappropriate and can be damaging to the credibility 

of the Agency or the peer reviewer. While not necessarily having legal 

consequences, such language will be included in the charge to the peer 

reviewers. Other mechanisms to use in discouraging premature release 

include a disclaimer that appears in a separate section at the front of the 

document and creating the document with watermarks clearly delineating 

DRAFT status (or a header or footer that states DRAFT status) on every 

page. In addition, in any solicitation for peer reviewers, the necessity for 

confidentiality and the non-release of materials shall be emphasized. 

b) Useful, but not critical materials that may be sent to peer reviewers include: 

1)	 The name, address, and phone and fax numbers, and/or Internet address of 

each peer reviewer working on the specific review 

2)	 A bibliography and/or any particularly relevant scientific articles from the 

literature 

3)	 A work product that has line numbering added in the margin for ease in 

providing and referencing comments 

c)	 Peer Reviewers should be given what is needed to complete their task -- they 

should not be overburdened with excess material. 

3.5.3 How Closely can EPA Interact with Peer Reviewers During the Review? 

a)	 When EPA Conducts the Peer Review - The Peer Review Leader normally has 

administrative contacts with the reviewers during the development and conduct of 

the peer review. In some cases (e.g., SAB peer review), peer reviewers may also 

receive a briefing on the product to be peer reviewed. Otherwise, the Peer 
Peer Review Handbook Page 65 

Review Leader and other EPA staff should not contact the reviewers during the 

course of the review. Such contact can lead to perceived conflicts or 

inappropriate direction that could compromise the independence of the review. 

b)	 When the Contractor Conducts the Peer Review - If peer review is conducted 

under a contract mechanism, EPA must limit direct contact to the prime 

contractor’s designated representative and not have general contact and direction 

to the contractor’s staff or peer reviewers (sub-contractors). Note, when a peer 

review is conducted under a contract, there are constraints where EPA staff are 

prohibited from contacting peer reviewers to avoid personal services 

arrangements. Personal services contracts exist when the nature of the 

relationship between the contractor and the EPA can be characterized as an 

employer - employee relationship. Any communications with peer reviewers 

must be coordinated through the prime contractor. 

3.6 Peer Review Services 

A range of peer review services are available to the Agency including internal, external 

(gratuitous services, contracts, purchase order), and Special Government Employee (SGE) 

mechanisms. The mechanism selected is generally based on the nature of the scientific or 

technical work product. 

3.6.1 What are Gratuitous Services for Peer Review? 

The provision of peer review products or services to EPA without compensation are 

provided as so-called “gratuitous” services. If a person wishes to perform peer review services 

for EPA without compensation, the EPA must ask them to sign an agreement whereby the 

person agrees to provide the prescribed peer review services as gratuitous services, with no 

expectation of receiving compensation for these services from EPA (see Appendix F for EPA 

Form 3100-14 which is used in situations where a gratuitous services contract is not used). An 

agreement (such as Form 3100-14) must be executed because the Antideficiency Act (31 U.S.C. 

§1342) prohibits the Agency’s acceptance of “voluntary” services (“voluntary services” are 

services provided to EPA without an agreement in advance that such services are provided at no 

cost to EPA). Note that persons cannot waive compensation (i.e., agree to provide gratuitous 

services) for which there is a statutory right to payment, unless a law permits the waiver. (For 

situations concerning State employees, see Section 3.4.10. Laws that permit services without 

compensation are 5 U.S.C. §3109 for experts or consultants and 5 U.S.C. §3111 for student 

volunteers.) 
Page 66 Peer Review Handbook 

3.6.2 Can I Use a Contract to Obtain Peer Review Services? 

The Agency may contract for peer review services. The contract may be written solely 

for peer reviews or be included as one of several specifically described interrelated tasks in a 

contract that requires the contractor to provide more than just peer review services. 

For assistance in preparing the necessary pre-award documents, program officials should 

consult Chapter 2 of the Contracts Management Manual (CMM). Chapter 2 is available on 

EPA’s Intranet (http://epawww.epa.gov/oamintra/policy/cmm.pdf) and on the Agency LAN 

Services (Administration/ Management/OAM Procurement Policy Information). 

3.6.3 How Do I Write a Statement of Work for Contracts? 

The Statement of Work (SOW) must clearly specify that the contractor is responsible for 

preparing peer review evaluations and set forth guidelines for the peer review of scientific or 

technical documents. The contractor may perform the peer review with in-house staff, 

subcontractors or consultants. Any guidelines for performing peer reviews to ensure soundness 

and defensibility must be developed by the program office and made part of the contract. The 

contractor would then ensure that the peer reviews adhere to the guidelines. 

The SOW should include the list of questions that EPA wants the contractor to include in 

the charge to the peer reviewers. While the contractor will be the one that will prepare and send 

the formal charge to the peer reviewers, EPA should provide the list of questions to the 

contractor (see Section 3.2 for general discussion on charge to peer reviewers). 

Within the SOW, the contractor should be directed to inquire whether prospective peer 

reviewers have any actual or potential organizational or personal conflicts of interest or other 

matters that would create the appearance of a lack of impartiality, including whether they have 

had or presently have a financial relationship with EPA. Further, the SOW needs to specify that 

contractors are required to provide EPA with a summary of the procedures and efforts made to 

identify, disclose, and ensure that no independence or conflict of interest concerns arise during 

the performance of the contract. 

The SOW cannot simply define the role of the prime contractor as arranging for the 

services of others to perform peer reviews and logistics for meetings. Unless the prime 

contractor is clearly tasked with responsibility for performing peer reviews, individual peer 

reviewers' fees and associated travel expenses are not payable under the contract. 

The EPA may pay for the reviewer's comments or evaluation, and also for attendance at a 

meeting with the Agency and other reviewers to discuss the results of the peer review. If the 

SOW calls for the preparation of comments or an evaluation, and specifies a meeting with the 
Peer Review Handbook Page 67 

Agency and other peer reviewers to discuss the results of the peer review, payment is 

appropriate. The peer reviewer's attendance at the meeting would then be part of contract 

performance. 

Example statements of work are found in Appendix E (Example Statements of Work for 

Contracts). 

3.6.4 What are Advisory and Assistance Services (AAS) or Sensitive Activities? 

Contracts that provide services that support or improve Agency decision-making or 

policy development are subject to special management controls. These services include services 

acquired from non-Governmental sources by contract to support or improve Agency policy 

development, decision-making, management, and administration, or research and development 

activities. See Federal Acquisition Regulation (FAR) 37.201 for a more specific definition of 

AAS. Such services may take the form of information, advice, opinions, alternatives, 

conclusions, recommendations, training, and direct assistance. For additional information on 

advisory and assistance services and sensitive activities, program officials should review Chapter 

2 of the CMM. 

New contracts for these services require management approvals prior to issuance of the 

solicitation. For the thresholds that have been established for approval of these justifications, 

see Figure 5, Item B Management Approvals set forth after Chapter 2 of the CMM (for current 

approval levels). 

3.6.5 What are Some Management Controls for Contracts? 

Contracting for peer review services is permitted. However, because of the potential for 

improper use of these contracts, special management controls are required. 

a)	 Inherently Governmental Functions (IGFs) – OFPP Policy Letter 92-1, dated 

September 23, 1992, describes (1) functions that are inherently governmental and 

must be performed only by Government employees and (2) functions that may be 

contracted, but so closely support Government employees in their performance of 

IGFs that the contract terms and performance require close scrutiny by Federal 

officials. Federal Acquisition Regulation (FAR) coverage of inherently 

governmental functions is at FAR Subpart 7.5. 

Peer reviews represent only a contractor's recommendations, advice or analysis of 

a document. Agency officials must make the official Agency decision regarding 

acceptability and/or quality of the document. To ensure that Agency officials are 

not improperly influenced by recommendations in the peer review, management 
Page 68 Peer Review Handbook 

controls must be included in the contract. One possible control would be to 

require the peer reviewers to submit with their evaluations or comments a 

description of the procedures used to arrive at their recommendations; a summary 

of their findings; a list of sources relied upon; and make clear and substantiate the 

methods and considerations upon which their recommendations are based. To the 

extent possible, the contract should set forth any guidelines or criteria for 

performance of the peer review. Agency officials should document their 

evaluations of the quality and validity of the peer review. 

b)	 Conflict of Interest (COI) -- Another important factor is that the objectivity of 

the peer review should not be improperly influenced or undermined by the 

contractor performing the review. To identify and avoid or mitigate actual or 

potential COI, the contract should include controls. Such controls might require 

the contractor to report on prior and current work, and prior clients that might 

create COI. Other controls might include Agency review and placing limits or 

advance approval on future work. There should also be procedures implemented 

to assure that the contractor does not gain an unfair advantage in future 

requirements as a result of their performance of peer reviews. Program officials 

should consult the Contracting Officer (CO) for special contract clauses. 

FAR coverage of conflicts of interest is at FAR Subpart 9.5. The EPA 

Acquisition Regulations (EPAAR) at 48 CFR Subpart 1509.5 generally mandates 

conflicts of interest solicitation provisions and contract clauses, but makes them 

optional for procurements accomplished through the use of simplified acquisition 

procedures as set forth in FAR Part 13. Also see 48 CFR 1509.507-1(b)(3) and 

48 CFR 1552.209-70, -71 & -72 as additional resources. 

Contract for peer review services: An EPA contracting officer will include 

conflicts of interest solicitation provisions and contract clauses as a matter of 

course without involvement by the EPA project officer, if the peer review 

services are not obtained pursuant to the simplified acquisition procedures in 

FAR Part 13. If the peer review services are subcontracted pursuant to a prime 

contract, then the prime contractor is ordinarily required to include a conflicts of 

interest clause substantially similar to the conflicts of interest clause in the 

primary contract in its subcontract to the peer reviewer. 

Although the EPA contracting officer and/or prime contractor has the primary 

responsibility to include the required conflicts of interest provisions/clauses, the 

EPA project officer may nevertheless wish to: 
Peer Review Handbook Page 69 

1)	 Highlight the conflict of interest requirements in the Statement of Work 

(SOW) for the procurement of the peer review services 

2)	 Develop a specific conflict of interest clause regarding the peer review at 

issue as a substitute or in addition to the standard conflicts of interest 

clause 

3)	 Review the solicitation/contract to make sure that the required conflicts of 

interest clause has been included 

4)	 Ensure that the SOW includes a requirement for the contractor to provide 

EPA a summary of the work performed to identify and resolve conflict of 

interest/independence concerns with peer review candidates 

Simplified acquisitions of peer review services: Although conflict of interests 

requirements are optional for simplified acquisitions, they are nevertheless a good 

idea. Accordingly, an EPA project officer obtaining peer review services with 

simplified acquisition procedures (see Section 3.6.7) should request the 

purchasing agent/contracting officer to include a conflict of interest solicitation 

provision and contract clause in the purchase order. 

c)	 Confidential Business Information (CBI)/Privacy Act Protected Information 

and Other Sensitive Information -- When peer reviewers are not employees of 

the United States Government, it is unlikely that the EPA will have authority to 

give reviewers access to confidential business information in the absence of 

consent for such disclosure by the CBI submitter. Therefore, all documents 

provided to non-Federal reviewers must be screened for information claimed as 

CBI. Even where business information has not been explicitly claimed as CBI, if 

it is of a kind where the submitter might be expected to object to its release, prior 

to release the submitter must be asked whether it wishes to assert a claim, unless 

the submitter has previously been informed that failure to assert a CBI claim may 

result in disclosure without notice. Language is included in the contract to clearly 

identify any required procedures or processes prior to release of any protected 

information, including any requirements for confidentiality agreements, as well as 

limits on use and disclosure of the data by contractor personnel. 

d)	 Personal services -- Under contracts, the EPA may not engage the peer reviewers 

in any improper personal services relationships, i.e., an arrangement under which 

contractor personnel are subject to relatively continuous supervision and direct 

control by an Agency official or employee. These relationships are characterized 
Page 70 Peer Review Handbook 

as one where the contractor employee interacts with the Agency in a manner 

similar to that of a Federal employee. 

To avoid these improper relationships, program officials should write welldefined SOWs. The SOWs should set forth the requirements in detail for work to 

be performed independently, including the manner in which it will be evaluated. 

The SOW must set forth what work is to be performed not how the work is to be 

performed. Technical direction may be used to clarify ambiguous technical 

requirements to ensure efficient and effective contractor performance, and is not 

considered supervision or assignment of tasks. For additional information, 

program officials should consult EPA Order 1901.1A, Use of Contractor Services 

to Avoid Improper Contracting Relationships and FAR Subpart 37.1 

3.6.6 Can I Identify and/or Select Peer Reviewers When Using a Contract? 

No. Program officials cannot interfere in a contractor's authority and responsibility to 

perform work by "selecting" who will perform the peer review (doing so may invoke FACA --

see Section 2.8). The Federal Acquisition Regulation (FAR) governs the Contracting Officer’s 

(CO's) and program officials' relationship with the contractor. Keep in mind that for contracting 

purposes, contractors are required to obtain Contracting Officer (CO) approval of subcontractors 

and the CO generally seeks the input of the work assignment manager (WAM -- the WAM may 

be the Peer Review Leader in many cases) before approving the use of subcontractors. 

EPA can establish criteria for the sort of individuals that might participate on a peer 

review panel. However, the Agency must not be involved in the selection of individual peer 

reviewers, and should avoid commenting on the contractor’s selection of peer reviewers other 

than to determine whether the panel, once selected, meets the criteria established. EPA may 

identify a pool of qualified subcontractors and consultants to the prime contractor (listed in 

alphabetical order), but cannot direct the use of any particular subcontractor or consultant. 

3.6.7 Can I Use Simplified Acquisition Procedures to Obtain Peer Reviewers? 

In some instances, peer reviewers can be obtained via simplified acquisition procedures. 

The acquisition of supplies or non-personal services from the open market and on a sole source 

basis when the aggregate amount involved in any one transaction does not exceed $100,000 

constitutes a simplified acquisition (FAR Part 13). The same considerations in the Section 3.6.5 

discussion on IGFs, COI, access to CBI, and personal services apply to simplified acquisitions. 

Normally, the Government issues a purchase order directly to the individual peer reviewer, 

instead of to a prime contractor who may subcontract for performance of the peer review. 
Peer Review Handbook Page 71 

a)	 Approvals -- All small purchases for peer reviews are considered Advisory and 

Assistance Services (AAS). See Chapter 2 of the Contract Management Manual 

(CMM) for the appropriate approval levels for AAS. 

b)	 Competition -- The Federal Acquisition Regulation (FAR) requires competition 

for purchases in excess of $2,500. Purchases for more than $2,500 and not more 

than $100,000 are to be made only from small businesses unless the Contracting 

Officer is not able to obtain two or more offers from small businesses that are 

competitive in terms of market price, quality, and delivery. Only one source need 

be solicited if the Contracting Officer determines that only one source is 

reasonably available. Contracting Officers are encouraged to use best value. 

c)	 Procurement Requests -- Program Officers should include the following in all 

PRs for the purchase of peer reviews: 

1) A fixed-price amount at or below the simplified acquisition threshold 

2)	 A detailed description of the requested services, inclusive of: 

(a) Total quantity per line item 

(b) Estimated unit price per line item 

(c) Total cost per line item 

(d) Specific deliverables for each line item 

(e) Total cost of the purchase request 

3)	 Reference FAR Subpart 3.6 and Environmental Protection Agency 

Acquisition Regulation (EPAAR) Subpart 1503.601 regarding sources 

from Government employees or organizations owned and controlled by 

them. 

4) Provide sources from small businesses, if available. 

5)	 If the request is a sole source purchase, justification must be provided in 

accordance with the EPAAR Subpart 1513.170-1. 
Page 72 Peer Review Handbook 

3.6.8 How is Travel Handled with Contracts or Purchase Orders? 

Funds obligated on a contract or purchase order are available to pay for the costs of 

producing the peer review including the travel costs and fee of the peer reviewer. 

The EPA acquires peer reviews through simplified acquisitions issued directly to peer 

reviewers or through contracts with companies, which provide the peer review services. By 

issuing a purchase order or awarding a contract for peer review services, the EPA may pay not 

only for the peer review services/comments, but also for participation in a meeting with the 

Agency and other reviewers to discuss comments. The scope of work of the contract must 

require the contractor or individual peer reviewer as appropriate to provide peer review services 

and indicate whether the contractor or peer reviewer will be required to discuss a specific peer 

review work product with the Agency and/or with other peer reviewers. Participation in a 

meeting to discuss a peer review work product would then be part of the contract's performance. 

Thus, the contract may serve as the mechanism to pay for peer review services and associated 

travel expenses to provide comments to the EPA. 

3.6.9 How is Travel Handled with Special Government Employees? 

The term Special Government Employee (SGE) is defined in 18 U.S.C. 202(a) as an 

officer or employee of an agency who performs temporary duties, with or without compensation, 

for not more than 130 days in a period of 365 days, either on a full-time or intermittent basis. 

Travel and per diem expenses of experts hired as SGEs for peer review may only be paid 

through the issuance of invitational travel orders (5 U.S.C. §5703). These invitational travel and 

per diem expenses should be charged to an appropriate EPA travel account. The Federal Travel 

Regulations govern the invited travelers reimbursement. 

Members of the SAB, SAP, and other FACA advisory committees are often brought on 

board as SGEs. It is not appropriate to reimburse travel or per diem expenses of advisory 

committee members (SGEs) through a contract. 
Peer Review Handbook Page 73 

4. COMPLETING A PEER REVIEW 

4.1 Overview 

Performance of the formal peer review is not the final stage in the development of the 

work product. Rather, it is an important stage in developing the work product, with the final 

work product representing the true end of the peer review. As a result, the peer review process 

closes with three major activities: evaluating comments and recommendations, utilizing peer 

review comments for completing the final work product, and organizing and maintaining a 

record of the peer review. 

Careful attention to all of these elements, singly and together, assures a credible peer 

review process. Conversely, inattention can nullify the peer review attempt. A well-planned 

peer review applied to a reasonable quality starting work product, followed by responsible, 

visible utilization of peer review suggestions in the final product assures a credible product for 

use in Agency decision-making. 

The peer review is not complete until The peer review is not complete until the the peer review comments are peer review comments are incorporated into the incorporated into the final work
 final work product, or reasons are stated why product.
 such comments are not to be incorporated. 


However, for the purposes of the annual report to


the Agency’s Peer Review Product Tracking


Database only, the work product can be moved from List B (Candidate Products for Future Peer


Review) (or List C - Products for Which a Decision has been made not to Peer Review) in some


instances) to List A (Products Peer Reviewed Since 1991) when the Decision Maker decides on


how the peer review comments will be addressed and this decision is documented in the peer


review record. 


The peer review record is completed only when it contains a copy of the final work 

product (when there is one) that addresses the peer review comments. 

4.2 Final Work Product 

4.2.1 How Do I Incorporate Peer Review Comments into the Final Work Product? 

The Peer Review Leader must carefully evaluate and analyze all peer review comments 

and recommendations. As discussed earlier, a carefully crafted charge to the peer reviewers 

simplifies organizing and analyzing comments. Also, any other issues that are raised need to be 

identified and evaluated. 
Page 74 Peer Review Handbook 

The validity and objectivity of the comments need to be evaluated. Analyses may 

include consultation with other experts/staff within the Office and/or Agency. Adequate 

documentation is needed to show that comments are accepted or rejected – the documentation 

can be brief, but must address the legitimate, valid comments. The peer review record must 

contain a document describing the Agency’s response to the peer review comments. 

The Peer Review Leader should brief the Decision Maker (including all appropriate 

managers in the Peer Review Leader’s chain of command) on the charge, profile of peer 

reviewers, the peer review comments, and provide a proposal on how to address the peer review 

comments. It is the responsibility of the Peer Review Leader to obtain Decision Maker approval 

of the approach to addressing the peer review comments. The Peer Review Leader should 

clearly identify for the Decision Maker any major peer review comments that will not be 

accepted and why. 

Comments that have significant impact on time, budgetary, and/or resource requirements 

are particularly important and need to be evaluated in consultation with management. These 

comments may lead to allocation of additional resources and a revised schedule for the 

completion of the work product. 

4.2.2 What Actions are Potentially Forthcoming from Peer Review? 

Peer review comments and recommendations may entail significant impacts on the 

planned project schedule, budget, or other resource requirements. Management decisions related 

to revisions in one or more of these areas may be appropriate. 

The substantive issues or concerns expressed by peer reviewers may suggest that wider 

scientific and technical consultation is needed to ensure the adequacy of the work product. 

The peer review comments and recommendations on a final product may provide a basis 

for bringing the associated project to closure. 

4.2.3 What Should the Final Work Product Say About the Peer Review Process? 

If the product has been peer reviewed, you should describe the peer review in the 

document. Frequently, this will be part of a description of the process of developing the product. 

It can be brief and does not need to describe the process or discuss the peer-review comments in 

great detail. The description can be included in an introduction, preamble, or appendix. 

When there were significant peer-review comments, and particularly if they are not being 

accepted, the document will generally discuss the issue and describe the reasons for the 

Agency’s choices in the appropriate sections of the document. The level of detail that is needed 
Peer Review Handbook Page 75 

is a matter of judgment and will depend on the importance and degree of controversy of the 

issue. 

If a major scientific or technical product has not been peer-reviewed, this fact should be 

noted in the document, perhaps in an introduction or description of its scope. This section 

should briefly indicate the reasons that peer review was not conducted. 

Derivative products of major scientific or technical products (such as fact sheets, press 

releases, and brochures) do not need to discuss whether the underlying products were peer 

reviewed. 

4.2.4 Can the Identity of Peer Reviewers be Kept Anonymous? 

Yes, up to a point. In the ordinary course of events, you can often discuss comments 

received without attributing the comments to a specific reviewer. However, if the matter has 

gone to litigation, the litigating parties can discover the names of anyone who contributed to a 

Federal product, including peer reviewers. Therefore, it is not possible to totally shield peer 

reviewers. In addition, it may be difficult to shield the names of the peer reviewers when the 

Agency is responding to a Freedom of Information Act (FOIA) request. 

If a peer reviewer requests anonymity at the outset of the peer review, the Peer Review 

Leader needs to inform the peer reviewer of the above possible eventualities. The Agency will 

in the ordinary course of events attempt to maintain the confidentiality of the peer reviewers and 

their comments from public disclosure, but it is recognized in many instances, for example open 

public meetings and the above circumstances, this can’t be assured. Remember, the Agency is 

committed to working “as if in a fishbowl” and most of its activities are transparent to the public 

(except where confidential business information is concerned). It is recognized that this may be 

a deterrent to possible peer reviewers, but this is a reality that has to be understood. 

4.3 Completing the Peer Review Record 

4.3.1 How Do I Complete the Peer Review Record? 

Once the Peer Review Leader has completed the peer review and the final work product 

(where one is prepared), the peer review record is brought up to date and then archived 

according to that organization’s procedure (see Section 4.3.2). The peer review record must be 

indexed and maintained in an organization’s archive (repository). The location of the peer 

review record needs to be readily identifiable so interested parties can locate and obtain materials 

easily and quickly. The peer review record should be placed in any associated established public 

docket, if required, in addition to the organizational archive. As a courtesy, a copy of the 

revised work product may be sent to the peer reviewers for information. 
Page 76 Peer Review Handbook 

The Peer Review Leader will collect the following materials for the peer review record 

and submit for archiving; including at least (see also Section 2.5.3): 

a) The draft work product submitted for peer review 

b) Materials and information (including the charge) given to the peer reviewers 

c) Written comments, information, and materials received from the peer reviewers 

d)	 Information about the peer reviewers (such as reviewers’ names, affiliations, and 

a summary of efforts to identify potential conflicts and their resolution) 

e)	 Logistical information about conduct of the peer review (such as times and 

locations of meetings) 

f)	 A memorandum, or other written record, approved by the Decision Maker, 

responding to the peer review comments specifying acceptance or, where thought 

appropriate, rebuttal and non-acceptance 

g) The final work product 

4.3.2 Where Should the Peer Review Records be Kept, and for How Long? 

During the active conduct of the peer review, the Peer Review Leaders maintain the peer 

review record with themselves until the peer review is totally completed. Minimally, the file 

should be maintained until one year after the completed peer review is reported in the next 

annual reporting. After that, the peer review record should be maintained for a “reasonable 

period of time.” Establishment and maintenance of the archive where the peer review records 

ultimately reside are an organization’s responsibility (i.e., not that of an individual program 

manager or Peer Review Leader). Generally, to allow flexibility, individual offices and regions 

will decide the appropriate level of organizational responsibility and how they will meet the 

“routinely available” requirement. The peer review record may be kept with other records 

relating to the overall project, as long as it is easily and separately identifiable. The peer review 

record should be maintained in accordance with the Agency’s record keeping schedule for such 

records. One long term archiving mechanism may be the formal archiving at the Federal 

Records Center in Suitland, MD. (Note: This is the same question as Section 2.5.7, but applies 

in this chapter as well). 
Peer Review Handbook Page 77 

4.3.3 Is Information Regarding a Peer Review Subject to Release under FOIA? 

Yes, it is subject to release if EPA receives a Freedom of Information Act (FOIA) 

request, unless the peer review information meets the criteria for an exemption under the FOIA. 

(http://www.epic.org/open_gov/foia/us_foia_act.html). 

Peer Review Handbook Page 79 

SUBJECT INDEX


This is an alphabetical listing of subjects from the Handbook and the pertinent page numbers 

where they are found. 

- A -

Administrative Procedures Act (12) 

Administrator (9, 17) 

Advisory and Assistance Services (66) 

Analytic blueprints (13, 25, 40, 49) 

Annual reporting requirements (14, 18) 

Anonymity of peer reviewers (75) 

Archiving (47, 75) 

Assistant Administrators (17) 

- B -

Balancing of peer reviewers (22) 

Board of Scientific Counselors (BOSC) (41) 

Budget (18, 40, 48) 

- C -

Candidate for peer review (26) 

Categories of work products (16, 25) 

Charge (46, 53, 76) 

Chemical action reports (16) 

Clean Air Scientific Advisory Committee 

(41) 

Compensation (65) 

Competition (70) 

Completing a peer review (4, 73) 

Conducting a peer review (3, 53) 

Conference proceedings (25) 

Confidential business information (CBI) 

(61, 69) 

Confidential Financial Disclosure (59) 

Confidentiality (63) 

Conflicts of interest (22, 31, 58, 67) 

Contracting officer (57, 63, 68, 69) 

Contracts (31, 64, 65, 68) 

Contracts Management Manual (CMM) (65, 

67) 

Controversy (27, 40) 

Cooperative agreements (31) 

Court ordered deadlines (44, 61) 

- D -

Decision-Maker (17, 26, 39) 

Deputy Administrator (14, 17, 18) 

Designated Ethics Official (DEO) (59) 

Division Directors (17) 

Docket (46, 47, 75) 

- E -

Economic work products (27) 

Electronic records (46) 

Environmental Impact Statement (EIS) (35) 

Environmental regulatory models (36) 

Executive Order 12866 (13, 29) 

External peer review (40, 41, 57) 

- F -

Federal Acquisition Regulation (FAR) (67) 

Federal Advisory Committee Act (FACA) 

(49, 62) 

Federal Register (12, 50) 

Final work product (73) 

Freedom of Information Act (FOIA) (16, 42, 

60, 77) 

Frequency of peer review (43) 

- G -

Grants (31) 

Gratuitous services (65) 

- I -

Independent peer reviewer (10, 11, 21, 55) 

Inherently governmental functions (IG Fs) 

(67) 

Interacting with peer reviewers (64) 

Interagency agreement (31) 

Internal peer review (41) 

Internal peer reviewer (22) 

- J -

Journal article (16, 42) 

- L -

Legal considerations (49) 

Letter reviews (41, 50) 

List A (15) 

List B (15) 

List C (15, 16) 

List D (16, 42) 

Litigation (16, 75) 

- M -

Major impact (27, 40) 

Major scientific and technical work product 

(5, 9, 15, 26, 37) 
Page 80 Peer Review Handbook 

Managers planning checklist for peer review 

(5) 

Materials for peer reviewers (62) 

Mechanisms for peer review (39) 

- N -

National Academy of Sciences (31, 42) 

NEPA Products (35) 

Non-major scientific and technical work 

products (16, 37, 38) 

Novelty (27, 40) 

NPDES permits (16) 

- O -

Office Directors (17) 

Office of Research and Development (ORD) 

(14, 20, 23) 

Open meeting (49) 

Other organization’s work products (36) 

- P -

Peer consultation (11) 

Peer input (10) 

Peer involvement (10) 

Peer review (9, 10) 

Peer Review Advisory Group (PRAG) (17, 

23) 

Peer review comments (46, 47, 73, 76) 

Peer Review Coordinator (14, 17, 20, 23) 

Peer Review Leader (17-19, 39, 47, 61, 62, 

73, 75) 

Peer review materials (62) 

Peer review mechanism (5, 39) 

Peer review panel (22, 56, 57) 

Peer Review Product Tracking (PRPT) 

database (14, 19, 34, 45) 

Peer review record (5, 21, 45, 75) 

Peer review schedule (18) 

Peer review services (65) 

Peer reviewers (21, 46, 55, 56, 61, 70, 76) 

Personal services (69) 

Planning a peer review (2, 25) 

Premanufacturing notices (PMNs) (16) 

Privacy Act (60) 

Procurement requests (71) 

Project manager (18, 19, 37) 

Public comment (12) 

- R -

RCRA permits (16) 

Record of Decision (ROD) (25) 

Records management (60) 

Regional Administrators (17) 

Regulatory development process (13) 

Regulatory negotiations (14) 

Rulemaking (13, 27, 46) 

- S -

Science Advisory Board (SAB) (28, 41, 50, 

56) 

Science Policy Council (SPC) (14, 17, 20, 

23) 

Scientific Advisory Panel (SAP) (41, 50, 

Scientific and technical work 

product (41, 50, 56) 

Selection of peer reviewers (25)


Sensitive activities (55)


Simplified acquisitions of peer review


services (66) 

Site specific decisions (69, 70) 

Social Science work product (35) 

Special Government Employee (SGE) (30) 

Stakeholder involvement (59, 72) 

State employees (12, 13, 40, 54) 

Statement of work for contracts (62) 

Strategic plans (59, 66) 

Subcontractor (25) 

Subject matter expert (51, 57) 

- T -

Tier 1 and Tier 2 (22) 

Tier 3 (13, 55) 

Time frame for peer review (13) 

Training (44, 55) 

Travel (18, 19) 

- V -

Voluntary consensus standards (71) 

Voluntary services (39) 

- W -

Work Assignment Manager (WAM) (57) 
Peer Review Handbook Page 81 

COMMONLY USED ACRONYMS


AA Assistant Administrator


AAS Advisory and Assistance Services


ASTM American Society of Testing and Materials


BOSC Board of Scientific Counselors


CBI Confidential Business Information


CFR Code of Federal Regulations


CMM Contracts Management Manual 


CO Contract(ing) Officer


COI Conflict of Interest


DEO Designated Ethics Official


EIS Environmental Impact Statement


EPA Environmental Protection Agency


EPAAR EPA Acquisition Regulations


FACA Federal Advisory Committee Act


FAR Federal Acquisition Regulations


FIFRA Federal Insecticide, Fungicide and Rodenticide Act


FOIA Freedom of Information Act


FTE Full Time Equivalent


GSA General Services Administration


HHS Health and Human Services


IAG Interagency Agreement


IGF Inherently Governmental Function


IRIS Integrated Risk Information System


ISO International Organization for Standardization


LAN Local Area Network


NAS National Academy of Sciences


NEPA National Environmental Policy Act


NPDES National Pollutant Discharge Elimination System


NTTAA National Technology Transfer and Advancement Act of 1995


OAM Office of Acquisition Management


OEI Office of Environmental Information


OGC Office of General Counsel


OMB Office of Management and Budget


ORD Office of Research and Development


PMNs Premanufacture Notice


PO Purchase Order


PRAG Peer Review Advisory Group


PRPT Peer Review Product Tracking Database


RA Regional Administrator


RC Regional Counsel


RCRA Resource Conservation and Recovery Act


RIAs Regulatory Impact Analyses

Page 82 Peer Review Handbook 

ROD Record of Decision


SAB Science Advisory Board


SAP Scientific Advisory Panel


SGE Special Government Employee


SOPs Standard Operating Procedures


SOW Statement of Work


SPC Science Policy Council


USC United States Code


WTI Waste Technologies Industries


WAM Work Assignment Manager

Peer Review Handbook Page A-1


APPENDIX A - U.S. ENVIRONMENTAL


PROTECTION AGENCY PEER REVIEW POLICY


June 7, 1994 
Page A-2 Peer Review Handbook 

June 7, 1994 

MEMORANDUM 

SUBJECT: Peer Review Program 

TO:	 Assistant Administrators 

General Counsel 

Inspector General 

Associate Administrators 

Regional Administrators 

Staff Office Directors 

Today, I am reaffirming the central role of peer review in our efforts to ensure that EPA 

policy decisions rest on sound, credible science and data (see attached policy statement). 

Toward that end, as its first major task, EPA’s Science Policy Council (SPC) is instituting a 

program to expand and improve peer review in all EPA offices. This memorandum gives an 

overview of current practices and outlines the new program. 

Peer Review Practices and Policy 

Peer review at EPA takes several different forms, ranging from informal consultations 

with Agency colleagues who were not involved in developing the product to the formal, public 

processes of the Science Advisory Board (SAB) and the FIFRA Scientific Advisory Panel 

(SAP). In any form, peer review assists the Agency’s work by bringing independent expert 

experience and judgment to bear on issues before the Agency to the benefit of the final product. 

EPA’s Peer Review Policy, which responds in part to recommendations in the “Credible 

Science, Credible Decisions” report, outlines general principles for peer review at EPA. 

Different EPA offices have undertaken various implementing activities, including an Agencywide information and planning workshop, internal guideline development, and numerous 

specific peer reviews. Even with these activities, however, I am concerned that EPA does not 

yet have a comprehensive Agency-wide program for implementing its Peer Review Policy. I 

therefore welcome the SPC initiative toward effective, efficient implementation of the policy in 

all the program areas to which it applies. 
Peer Review Handbook Page A-3 

Expanding and Improving Peer Review 

The Science Policy Council and its Steering Committee have outlined a dual-track 

implementation program of planning and assistance for all Agency offices. The first track has 

three major milestones. 

First, during the next few weeks, Steering Committee members will consult with senior 

management in each office to exchange information on current peer review activities, assistance 

needed, possible obstacles to implementation, and implementation planning. 

Second, using information and materials developed during the first stage, peer review 

task groups in each office will develop standard operating procedures (SOPs) for use in each 

office, based in part on generic guidance to be issued by the SPC and in part on peer review 

needs and capabilities specific to each office. The resulting SOPs will delineate as appropriate 

the scope of application of peer review with respect to various types of scientific and technical 

work products such as reports of original research, risk assessments, and analytical methods of 

economic analysis. OARM and OGC staff will assist each office as needed on legal, budget 

and administrative matters. Each AA and RA will submit draft SOPs for Steering Committee 

review by July 15. 

Third, the SPC review group will work with each office to complete each plan by 

September 15. 

In parallel with the above, consistent with the Peer Review Policy, the Science Policy 

Council will work with each AA and RA to identify “major scientific and technical work 

products” as peer review candidates for the coming year. This process will consider existing 

and new plans for internal reviews and for Science Advisory Board (SAB), FIFRA Scientific 

Advisory Panel (SAP), and other external reviews. The two-fold objective is to plan reviews 

for technical products covered by the Peer Review Policy and to gain experience with options 

and obstacles. We will use this experience to review and revise the SOPs as needed. Also, to 

establish a baseline for comparison, each AA and RA will identify the “major technical 

products” completed within his/her program during the past 12 months. 

The Science Policy Council has sent additional information to each office offering 

guidance on the procedures that you are asked to develop and the schedule for these activities. 

Please note, however, that because the policy is effective immediately, current peer review 

planning should continue on present schedules in parallel with developing the formal SOPs. 
Page A-4 Peer Review Handbook 

To begin this process, I have asked each Assistant Administrator and Regional 

Administrator to designate a Peer Review Coordinator to work with the Steering Committee on 

implementation activities specific to each office. I am very pleased that the Science Policy 

Council is taking this important step. A comprehensive peer review program is essential to 

maintaining and improving the quality of the analyses that underlie Agency actions. I look 

forward to working with you and your staff on this important activity. 

/s/ 

Carol M. Browner 

Attachment 

cc:	 Science Policy Council 

Science Policy Council Steering Committee 
Peer Review Handbook Page A-5 

PEER REVIEW AND PEER INVOLVEMENT


AT THE U.S. ENVIRONMENTAL PROTECTION AGENCY


This document establishes the policy of the United States Environmental Protection 

Agency (EPA) for peer review of scientifically and technically based work products that are 

intended to support Agency decisions. Peer review is presented in the context of the broader 

concept, peer involvement. 

BACKGROUND 

The report “Safeguarding the Future: Credible Science, Credible Decisions”1

 focused on 

the state of science at EPA. The panel of experts who prepared the report emphasized the 

importance of peer review, especially external peer review, and the need for broader and more 

systematic use of it at EPA to evaluate scientific and technical work products. Their specific 

recommendation regarding peer review reads as follows: 

“Quality assurance and peer review should be applied to the planning and results 

of all scientific and technical efforts to obtain data used for guidance and 

decisions at EPA, including such efforts in the program and regional offices. 

Such a requirement is essential if EPA is to be perceived as a credible, unbiased 

source of environmental and health information, both in the United States and 

throughout the world.” 

In response to this recommendation, then-Administrator Reilly directed staff to develop an EPAwide policy statement, which he issued in January, 1993. The paragraphs below preserve the 

core of that earlier statement while updating it to specify the role of the Science Policy Council 

in guiding further implementation of the policy. Effective use of peer review is indispensable 

for fulfilling the EPA mission and therefore deserves high-priority attention from program 

managers and scientists within all pertinent Headquarters and Regional Offices. 

1

 EPA/600/9-91/050, March 1992. 
Page A-6 Peer Review Handbook 

PEER INVOLVEMENT AND PEER REVIEW 

EPA strives to ensure that the scientific and technical underpinnings of its decisions meet 

two important criteria: they should be based upon the best current knowledge from science, 

engineering, and other domains of technical expertise; and they should be judged credible by 

those who deal with the Agency. EPA staff therefore frequently rely upon peer involvement --

that is, they augment their capabilities by inviting relevant subject-matter experts from outside 

the program to become involved in one or more aspects of the development of the work products 

that support policies and actions. 

One particularly important type of peer involvement occurs when scientifically and 

technically based work products undergo peer review -- that is, when they are evaluated by 

relevant experts from outside the program who are peers of the program staff, consultants, and/or 

contractor personnel who prepared the product. Properly applied, peer review not only enriches 

the quality of work products but also adds a degree of credibility that cannot be achieved in any 

other way. Further, peer review early in the development of work products in some cases may 

conserve future resources by steering the development along the most efficacious course. 

Peer review generally takes one of two forms. The review team may consist primarily of 

relevant experts from within EPA, albeit individuals who have no other involvement with 

respect to the work product that is to be evaluated (internal peer review). Or the review team 

may consist primarily of independent experts from outside EPA (external peer review). 

POLICY STATEMENT 

Major scientifically and technically based work products related to Agency decisions 

normally should be peer-reviewed. Agency managers within Headquarters, Regions, 

laboratories, and field components determine and are accountable for the decision whether to 

employ peer review in particular instances and, if so, its character, scope, and timing. These 

decisions are made in conformance with program goals and priorities, resource constraints, and 

statutory or court-ordered deadlines. For those work products that are intended to support the 

most important decisions or that have special importance in their own right, external peer review 

is the procedure of choice. Peer review is not restricted to the penultimate version of work 

products; in fact, peer review at the planning stage can often be extremely beneficial. 

SCOPE 

Agency managers routinely make regulatory and other decisions that necessarily involve 

many different considerations. This policy applies to major work products that are primarily 

scientific and technical in nature and may contribute to the basis for policy or regulatory 

decisions. By contrast, this policy does not apply to non-major or nontechnical matters that 

Agency managers consider as they make decisions. Similarly, this policy does not apply to these 

ultimate decisions. 
Peer Review Handbook Page A-7 

This policy applies where appropriate, as determined by the National and Regional 

Program Managers, to major scientifically and technically based work products initiated 

subsequent to the date of issuance. Peer review should be employed to the extent reasonable to 

relevant work products that currently are under development. This policy does not apply to the 

bases for past decisions, unless and until the relevant scientific and technical issues are 

considered anew in the Agency’s decision-making processes. 

Except where it is required by law, formal peer review (as distinguished from the 

Agency’s normal internal review procedures) should be conducted in a manner that will not 

cause EPA to miss or need extension of a statutory or court-ordered deadline. Agency managers 

still may undertake peer review if it can be conducted concurrently with necessary rulemaking 

steps. 

LEGAL EFFECT 

This policy statement does not establish or affect legal rights or obligations. Rather, it 

confirms the importance of peer review where appropriate, outlines relevant principles, and 

identifies factors Agency staff should consider in implementing the policy. On a continuing 

basis, Agency management is expected to evaluate the policy as well as the results of its 

application throughout the Agency and undertake revisions as necessary. Therefore, the policy 

does not stand alone; nor does it establish a binding norm that is finally determinative of the 

issues addressed. Minor variations in its application from one instance to another are 

appropriate and expected; they thus are not a legitimate basis for delaying or complicating action 

on otherwise satisfactory scientific, technical, and regulatory products. 

Except where provided otherwise by law, peer review is not a formal part of or substitute 

for notice and comment rulemaking or adjudicative procedures. EPA’s decision whether to 

conduct peer review in any particular case is wholly within the Agency’s discretion. Similarly, 

nothing in this policy creates a legal requirement that EPA respond to peer reviewers. However, 

to the extent that EPA decisions rely on scientific and technical work products that have been 

subjected to peer review, the remarks of peer reviewers should be included in the record for that 

decision. 

IMPLEMENTATION 

The Science Policy Council is responsible for overseeing Agency-wide implementation. 

Its responsibilities include promoting consistent interpretation, assessing Agency-wide progress, 

and developing recommendations for revisions of the policy as necessary. 

The Science Policy Council will oversee a peer-review work group, which will include 

representatives from program units throughout EPA to effect a consistent, workable 

implementation of the policy. The work group will assist the programs in (1) formulating and, 

as necessary, revising standard operating procedures (SOPs) for peer review consistent with this 
Page A-8 Peer Review Handbook 

policy; (2) identifying work products that are subject to review; and (3) for each major work 

product, selecting an appropriate level and timing of peer review. 

In assisting the programs, the work group will take into account statutory and court 

deadlines, resource implications, and availability of disinterested peer reviewers. The group will 

work closely with Headquarters offices and the Regional Offices toward ensuring effective, 

efficient uses of peer review in supporting their mission objectives. However, the Assistant 

Administrators and Regional Administrators remain ultimately responsible for developing SOPs, 

identifying work products subject to peer review, determining the type and timing of such 

review, documenting the process and outcome of each peer review, and otherwise implementing 

the policy within their organizational units. 

Because peer review can be time-consuming and expensive, Agency managers within 

Headquarters, Regions, laboratories, and field components are expected to plan carefully with 

respect to its use -- taking account of program priorities, resource considerations, and any other 

relevant constraints as well as the policy goal of achieving high-quality, credible underpinnings 

for decisions. External peer reviewers should be chosen carefully to ensure an independent and 

objective evaluation. The affiliations of peer reviewers should be identified on the public record, 

so as to avoid undercutting the credibility of the peer-review process by conflicts of interest. 

The policy is effective immediately. The peer-review work group mentioned above will 

identify the focal point to whom comments and questions should be addressed and, from time to 

time, will provide further information about implementation activities. 

/s/ 

DATE: JUN 7 1994 

CAROL M. BROWNER, ADMINISTRATOR 

APPROVED: 
Peer Review Handbook Page B-1 

APPENDIX B - SOUND SCIENCE AND PEER


REVIEW IN RULEMAKING


Using EPA’s regulatory agenda as the cornerstone of the Agency’s mission, 

the Deputy Administrator’s memo implements an additional requirement as part of 

the ongoing effort to use peer review to enhance the quality and credibility of 

Agency decisions. 
Page B-2 Peer Review Handbook 

June 15, 1999 

MEMORANDUM 

SUBJECT: Sound Science and Peer Review in Rulemaking 

TO: The Regulatory Policy Council 

Over a year ago, the Agency enhanced its peer review policy and procedures with the 

finalization and release of the Peer Review Handbook. The Handbook was developed to serve 

as a guide for Agency scientists and managers on the organization and conduct of peer review. 

With our regulatory agenda as the cornerstone of the Agency’s mission, I would now like to 

implement an additional requirement as part of the ongoing effort to use peer review to enhance 

the quality and credibility of Agency decisions. 

Beginning July 12, 1999, the action memorandum accompanying each rule submitted for 

signature must explicitly address the use of peer review in that rulemaking. The Administrator 

and I want to ensure that the Agency takes complete advantage of the benefits that thorough peer 

reviews can provide in our deliberations on the many challenging decisions that must be made. 

A key step in producing credible regulatory decisions supported by well-documented scientific 

data is ensuring that our work is based on broadly accepted scientific studies. Peer review is an 

essential tool for accomplishing this goal. 

In addition, I ask that, beginning today, planning for peer review be specifically 

addressed in each new Analytic Blueprint. The general guidance on integrating peer reviews 

into the rulemaking process is included as Attachment A. Detailed guidance on action 

memoranda, updated to address peer review, is provided in Attachment B. Guidance regarding 

which scientific or technical studies must be peer reviewed is contained in the fact sheet on 

“Peer Review in the Rulemaking Process” in Attachment C and in the Peer Review Handbook 

available at http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm. I am also taking this 

opportunity to bring to your attention a memorandum from the Office of General Counsel that 

discusses the implications of a recent decision in a challenge to a peer review brought against the 

Agency under the Federal Advisory Committee Act (Attachment D) [note added: this 

attachment is not attached to the Peer Review Handbook]. 

I appreciate your support in this effort. For additional information about peer review in 

the rulemaking process, please contact your Steering Committee representative, your peer review 

coordinator (Attachment E) [note added: this attachment is not attached to the Peer Review 

Handbook], or Phil Schwartz in the Office of Policy at xxx-xxxx. 

[signed 6/15/99, AX 9807793] 

Peter D. Robertson 

Acting Deputy Administrator 
Peer Review Handbook Page B-3 

ATTACHMENT A 

Regulatory Management Guidance PEER REVIEW REQUIREMENTS


Office of Policy FOR EPA RULEMAKING


June 1999


Reference: RMD 5


BACKGROUND 

On February 25, 1998, the Agency reaffirmed its commitment to quality decision-making by 

issuing the comprehensive Peer Review Handbook. The Peer Review Handbook is available at: 

http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm, which amplified and clarified key Agency 

peer review policy issues. The Handbook states that, in general, all major scientific and technical 

work products related to Agency decisions should normally be peer reviewed. Peer review is a 

documented, critical review of a work product, performed by experts who are independent of 

those who developed the product. Work products that are developed to support regulations, such 

as risk assessments, should be evaluated to determine whether they meet the criteria for "major 

scientific and/or technical work products" and thus require peer review (See Attachment C). 

Regulations themselves are not peer reviewed. The straight-forward application of accepted, 

previously peer-reviewed economic methods or analyses in Regulatory Impact Analyses (RIA's), 

are not typically subject to formal peer review. If, however, the particular facts and 

circumstances of any piece of economic analysis in an RIA warrant peer review, the Agency will 

accommodate those needs on a case-by-case basis. 

PLANNING FOR PEER REVIEW 

Remember that: 

� Analytical Blueprints should discuss the plan for peer review of major scientific 

and technical products. This should include a schedule for the review and should 

identify the resources that will be needed. 

� When negotiating rule deadlines with courts or with litigants, be sure that your 

proposals allow adequate time for any needed peer reviews. 

PEER REVIEW REQUIREMENTS 

Program offices have the primary responsibility for complying with the Agency peer review 

requirements. Senior managers must confirm in their Action Memoranda that their offices have 

complied with the peer review requirements; detailed explanations are not necessary. If no major 

scientific and technical product as defined by the Peer Review Handbook was used to support 

the action, include the following statement in the action memorandum: 
Page B-4 Peer Review Handbook 

Peer Review 

There were no major scientific or technical products supporting this action as defined by 

the Agency's Peer Review Handbook. We did not, therefore, submit any support 

documents for peer review. 

If a major scientific or technical product was used to support the action, include the following 

statement in the Action Memorandum: 

Peer Review 

[Insert Name of AAship] has complied with the Agency's Peer Review requirements with 

respect to the underlying major scientific and technical products supporting this action. 

You may add any details you think important, but you may not modify this compliance 

statement. If you utilized peer-reviewed products, but you could not fully comply with the peer 

review requirements, explain in the Action Memorandum why the deviation was necessary. 

FINAL AGENCY REVIEW (FAR) 

For rules that go through Final Agency Review, OP will document compliance in the FAR 

memorandum. This documentation will be based on the peer review information included in the 

draft Action Memorandum. 

These requirements will be effective on July 12, 1999. After that date, OP's Office of 

Regulatory Management and Information (ORMI) will only forward for signature those actions 

that include one of the following: 

� a statement that no major scientific or technical documents were utilized to 

support the rulemaking; 

� a statement of compliance with the peer review requirements for major scientific 

and technical documents; 

� an explanation for deviation from the peer review requirements. 

Questions? Contact your Regulatory Steering Committee representative or Phil Schwartz in the 

Regulatory Management Division at (202) 260-5493. 
Peer Review Handbook Page B-5 

ATTACHMENT B 

Regulatory Management Guidance 


Office of Policy FRAMEWORK FOR ACTION MEMORANDA


Revised June 1999


Reference RMD-6


OVERVIEW 

The overview provides background information which briefly describes the rule, 

characterizes the environmental issue(s) or public health problem(s) being addressed, and 

summarizes the history of the regulatory action. It should explain why EPA is taking this 

action, and where appropriate, cover the following points: 

� Define the court or statutory deadline; 

� Identify whether the action amends the CFR, and if so, explain what kind of 

amendment (technical, procedural, etc.); 

� Identify other regulatory actions underway that will affect this particular program 

or sector; 

� Describe the specific environmental issue(s) or public health problem(s) being 

addressed, and the goal intended by taking this action; 

� Describe what the action does, and specifically, how the regulated community is 

affected (performance standards, specific requirements); and 

� Describe how flexible the implementation of the action will be for states and 

regulated entities. 

REGULATORY IMPACTS 

Summarize the costs and benefits of the regulatory action (include costs to State and 

local governments, and tribal communities) and the results of the economic and risk 

analyses. Discuss the economic impacts on all stakeholders including, where appropriate, 

the results of the regulatory flexibility analysis and the specific effects of the action on 

small entities. Explain the roles of cost-benefit and cost-effectiveness analyses in shaping 

the regulatory approach. 

Describe the reporting and Record keeping burden and what the Agency has done to 

reduce it. Indicate possible impacts on other Federal agencies. 
Page B-6 Peer Review Handbook 

ADMINISTRATOR'S PRIORITIES 

If appropriate, discuss how the regulatory action relates to the Administrator's priorities. 

Explain how the regulatory action relates to a specific priority only where the relationship 

is significant, clear and distinct. 

STAKEHOLDER INVOLVEMENT 

Briefly discuss the role of both government entities and private sector stakeholders in the 

development of the action. Summarize the concerns they have raised and what the 

Agency has done to address them. 

INTERNAL REVIEW 

Identify whether the regulatory action was developed under Tier 1,2, or 3. Describe any 

outstanding issues from workgroup closure or other internal review. Identify program 

offices or regions with outstanding issues and indicate why they cannot be resolved or 

accommodated. Also, provide the basis for any decision made to not address an 

identified cross-media impact. Where there is not a formal workgroup, identify those 

offices that have reviewed the package. 

PEER REVIEW 

If you did not utilize major scientific and technical products as defined by the Peer 

Review Handbook to support the action, include the following statement: 

There were no major scientific or technical products supporting this action as 

defined by the Agency's Peer Review Handbook. We did not, therefore, submit 

any support documents for peer review. 

If a major scientific or technical product has been used to support the action, include the 

following statement: 

[Insert Name of AAship] has followed the Agency's Peer Review requirements 

with respect to the underlying, major scientific and technical products 

supporting this action. 

You may add any details you think important, but you may not modify this compliance 

statement. If you utilized peer-reviewed products, but you could not fully comply with 

the peer review requirements, explain why the deviation was necessary. 
Peer Review Handbook Page B-7 

PLAIN LANGUAGE 

Either in the action memorandum or in a separate note briefly describe which elements of 

plain language have been addressed. 

OMB TRANSACTION 

Highlight significant issues resulting from OMB's review under Executive Order 12866. 

Explain any changes made to the regulatory action as a result of that review. If OMB 

review was waived, please indicate this. 

ANTICIPATED PUBLIC REACTION 

Describe the type of public response anticipated and identify both the involved 

stakeholders and the nature of their expected response. Characterize the likely reaction 

to the action by all interested parties including industry; environmental groups; Congress; 

state, local, and tribal governments; and OMB. Explain why the Agency should take the 

action despite any controversial response anticipated from the public. 

If the regulatory action will not be issued for public comment (i.e., direct final rule, 

administrative stay, etc.), explain the basis for the Agency's decision not to solicit 

comment. 

RECOMMENDATION 

Identify the action the administrator is expected to take. 

NOTE: An Action Memo should generally not exceed 5 pages. 
Page B-8 Peer Review Handbook 

ATTACHMENT C 

Fact Sheet 

Peer Review in the Rulemaking Process 

What Role does Peer Review have in the Regulatory Development Process? 

Peer review of scientific and technical work products affirms the scientific credibility of 

the Agency's rulemakings. Work products need to be well planned and documented because new 

rules, and the work products supporting them, may have substantial cost impacts and must often 

withstand intense public scrutiny. 

What Work Products Need Peer Review? 

While rules themselves are not subject to peer review, any major scientific and/or 

technical work products that support rules should be peer reviewed. (See Section 2.2 of the Peer 

Review Handbook http://www.epa.gov/ordntrnt/ORD/spc/sopmenu.htm to determine which 

work products are considered major.) Designations of a scientific and/or technical work product 

as "major" will largely be case-by-case. Those that are used to support a regulatory program or 

policy position and that meet one or more of the following criteria are candidates for peer 

review: 

a) Establishes a significant precedent, model, or methodology 

b) Addresses significant controversial issues 

c) Focuses on significant emerging issues 

d) Has significant cross-Agency/inter-agency implications 

e) Involves a significant investment of Agency resources 

f)	 Considers an innovative approach for a previously defined 

problem/process/methodology 

The straight-forward application of accepted, previously peer-reviewed economic 

methods or analyses in Regulatory Impact Analyses (RIA's), are not typically subject to formal 

peer review. If, however, the particular facts and circumstances of any piece of economic 

analysis in an RIA warrant peer review, the Agency will accommodate those needs on a 

case-by-case basis. The decision whether to peer review any work product will be documented 

through the Agency's annual peer review reporting process (see Section 1.3 of the Handbook). 

When should determinations about the need for peer review be made? 

The need for, and scheduling of, peer reviews should be addressed in the Analytic 

Blueprint that is required for all Tier 1 and 2 regulatory actions. Work products supporting Tier 

3 rules may also be considered major and Analytic Blueprints are encouraged for Tier 3 rules. In 

any event, the need for, and scheduling of, peer reviews should be addressed early in the 

development of Tier 3 actions. 
Peer Review Handbook Page C-1 

APPENDIX C - EXAMPLES OF CHARGES


Please note -- certain questions that are posed in charges can be responded 

to with a yes or no answer. Clearly, this is not the type of response we generally 

want, therefore, it is important to phrase charge questions carefully to ensure that 

you receive a fully satisfactory and thoughtful response. Where a yes or no answer 

might be expected, be sure to ask for a full explanation supporting the yes or no 

answer. 

Charges can run the gamut from rather simplistic to highly complex. The 

examples shown here cover a variety of types. Examples 1 through 6 have less 

complex questions and are looking for the overall quality of the reports. Examples 

7 through 11 have numerous technical questions that need to be addressed and are 

therefore more complex in their nature. 
Page C-2 Peer Review Handbook 

CHARGE EXAMPLE 1 - Framework for Developing a Living Resources Research and 

Monitoring Plan for the Peconic Estuary 

Background 

The Peconic Estuary is located on the eastern end of Long Island, New York. Under the 

Federal Clean Water Act, the Peconic Estuary was named an "Estuary of National Significance" 

in 1992. Because of its high concentration of rare, threatened and endangered species and 

habitats, The Nature Conservancy named the Peconic Ecosystem as one of the "Last Great Places 

in the Western Hemisphere." 

The Peconic Estuary Program (PEP) released its draft Comprehensive Conservation and 

Management Plan (CCMP) in September 2, 1999. The draft Plan addresses the environmental 

management issues facing the estuary and its watershed, including brown tide (a nuisance algal 

bloom), nutrients, habitats and living resources, pathogens, and toxics, as well post-CCMP 

management structure, public education and outreach, and financing. A final Plan is now under 

preparation. The preparation of the Framework document is called for in the draft CCMP. 

With population increasing in the watershed, the Peconic Estuary is being threatened by 

over-development and misuse of its resources. To fully realize the impacts of people and their 

activities on this system, there must be a better understanding of how the Peconic Estuary 

functions ecologically. This knowledge can only be achieved through comprehensive research 

and monitoring of the entire ecosystem. 

The long-term goal of this Framework for Developing a Living Resources Research and 

Monitoring Plan is to develop a strategy of coordinated research and monitoring to fill 

significant information gaps and assist in the planning, conservation, and management of the 

Peconic Estuary. A key component of this strategy is to develop an applied, multi-scale, 

integrated approach to gain a better understanding of the estuary. To achieve, this goal, the 

Framework for Developing a Living Resources Research and Monitoring Plan sets broad 

priorities, provides the context for specific studies, guidelines for a detailed living resources 

research and monitoring plan, and stimulus for funding agencies and organizations and 

researchers. The short-term goal of this Framework is to provide a basis for securing funding for 

the priority initiatives described in the Framework document. 

In order of importance, the objectives of this Framework document are to: 

1)	 understand threats so as to improve resource protection through management and 

conservation; 

2)	 examine the biology and ecology of particular organisms identified in the CCMP 

as important either due to their commercial or recreational value or their role in 

the food web and ecosystem; and, 
Peer Review Handbook Page C-3 

3)	 undertake system-wide studies of the Peconic Estuary to understand the 

significance of key habitats and ecosystem productivity. 

Recommendations included in this framework are based on information gaps identified 

in the Characterization Report of the Living Resources of the Peconic Estuary (Bortman and 

Niedowski 1998), the Habitat Module of the draft CCMP, PEP Natural Resource Subcommittee 

meetings, and recommendations made at a PEP-sponsored Living Resources Research and 

Monitoring workshop held in 1998. Copies of the draft CCMP and Characterization Report are 

included with this package. 

Tasks 

The peer review charge is to assess the adequacy of this document to provide a 

framework for integrated, system-wide ecological research and monitoring to understand the 

dynamic, multi-scale ecological patterns and processes that sustain biota and their supporting 

natural systems in the Peconic Estuary. 

Specific Questions: 

Peer Reviewers shall answer/comment on the following: 

1)	 Is the organization of the document appropriate and does it present the material in 

a clear and concise manner? Please explain fully. 

2)	 In your opinion, what are the weakest and the strongest aspects of the framework? 

Please make suggestions on how the weakest parts can be strengthened. 

3)	 Are there any elements missing from the framework which you think need to be 

included or which would strengthen the document? Please explain fully. 

4)	 Are you aware of any other significant data/studies that are relevant and should be 

included or referenced in this document? Please explain fully. 

5)	 Is the stated goal realistic? Are the stated objectives adequately met? Please 

explain fully. 

6)	 Examine and critique the interrelationship among the specified assessment, 

research, and monitoring priorities. 

7)	 Does this framework present a holistic approach to preserve, protect and restore 

the estuary? Does the approach cover all the relevant and important areas that 

warrant investigation? If not, please state what should be done differently. 
Page C-4 Peer Review Handbook 

8)	 Are the initial set of assessment, research, and monitoring priorities adequate to 

provide the information needed to successfully protect and manage this system? 

Please comment on the selection of priorities. Were any significant priorities 

missed? What other priorities should be specified and why? 
Peer Review Handbook Page C-5 

CHARGE EXAMPLE 2 -EPA Science Advisory Board (SAB) Review of the Environment 

Monitoring and Assessment Program (EMAP) Research Strategy and Research Plan 

The Science Advisory Board (SAB) is asked to review the Environment Monitoring and 

Assessment Program (EMAP) Research Strategy and Research Plan. The review is requested by 

the Office of Research and Development (ORD) with the following specific charge issues: 

1)	 Previous peer reviews recommended that EMAP develop a close working relation 

with EPA Program Offices and other federal monitoring efforts. Does the EMAP 

strategy support the [Office of Science and Technology Policy's Committee on 

Environment and Natural Resources] CENR National Monitoring Framework and 

EPA Program Offices? 

2)	 Previous peer reviews recommended that EMAP initiate a focused research 

program on indicator development. Does the intramural EMAP program on 

ecological indicator development, coordinated with the [ORD Science To 

Achieve Results] STAR solicitations, respond to this research need? 

3)	 Previous peer reviews recommended that the EMAP design be modified to 

include a set of nonrandomly selected sentinel sites with intensive data collection. 

Does the development of Index Sites as outdoor laboratories in the national parks 

([National Park Service] NPS and [US Geological Survey] USGS) and selected 

estuaries ([National Oceanic and Atmospheric Administration] NOAA) add this 

dimension to the EMAP? 

4)	 Previous peer reviews recommended that EMAP combine effects-oriented and 

stressor-oriented monitoring approaches. Do the focused geographic 

demonstration pilot studies (initially in the Mid-Atlantic region) combine these 

elements? 
Page C-6 Peer Review Handbook 

CHARGE EXAMPLE 3 - EPA Science Advisory Board Review of the Agency's National 

Risk Management Research Laboratory’s (NRMRL) Program 

The Office of Research and Development (ORD) requests that the Science Advisory 

Board review the Agency's National Risk Management Research Laboratory’s (NRMRL) 

program. 

In the “Strategic Plan for the Office of Research and Development” (EPA, 1996a), ORD 

described the relationship of risk assessment to the risk management process, and emphasized 

the need for scientific and engineering research to enable sound risk management decisions and 

actions. Within the framework of that strategic plan, NRMRL’s mission is to conduct research 

to reduce uncertainties and costs associated with making and implementing environmental risk 

management decisions. NRMRL has therefore developed a research agenda to reduce risk 

uncertainty that also focuses on those important, relevant issues where it can make a difference. 

The charge to the SAB is to: 

1)	 Examine and critique the research programmatic directions such as whether 

NRMRL is pursuing the most appropriate research problem areas; 

2)	 Comment on strategic directions, e.g., use of its core technical competencies, 

transition from primarily extramural to an intramural R&D organization, 

leveraging with other agencies and organization; 

3)	 Review and comment on the effectiveness of NRMRL’s approach to science 

management, e.g., measures of success and science quality, soundness of peer 

review process; 

4)	 Examine and critique the relationship of NRMRL’s risk management research 

and its intended role in the risk assessment/risk management paradigm; and 

5)	 Review and comment on the strategic balance for the next decade among 

pollution prevention, technology development, remediation, and risk management 

assessment activities. 
Peer Review Handbook Page C-7 

CHARGE EXAMPLE 4 - EPA Science Advisory Board (SAB) Review of the Technical 

Aspects of the Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM) 

The EPA Science Advisory Board (SAB) is asked to review the technical aspects of the 

Multi-Agency Radiation Survey and Site Investigation Manual (MARSSIM). The review 

document was developed collaboratively by four Federal agencies, departments and 

commissions having authority for control of radioactive materials: Department of Defense, 

Department of Energy, Environmental Protection Agency, and Nuclear Regulatory Commission. 

MARSSIM addresses the need for a nationally consistent approach to conducting radiation 

surveys of potentially radioactively contaminated sites that are being considered for release to 

the public. A condition of release is a demonstration that residual radioactivity levels do not 

exceed a specified risk or dose level, also known as a release criterion. MARSSIM provides 

guidance to users performing and assessing the results of such a demonstration for surface soils 

and building surfaces. 

The SAB is asked by the Agency’s Office of Radiation and Indoor Air (ORIA) to 

respond to the following charge in its review: 

1)	 Is the overall approach to the planning, data acquisition, data assessment, and 

data interpretation as described in the MARSSIM technically acceptable? Please 

explain fully. 

2)	 Are the methods and assumptions for demonstrating compliance with a dose- or 

risk-based regulation technically acceptable? Please explain fully. 

3)	 Are the hypotheses and statistical tests and their method of application 

appropriate? Please explain fully. 
Page C-8 Peer Review Handbook 

CHARGE EXAMPLE 5 - Economics - Benefits Transfer from Adults to Children 

Background 

EPA established the Office of Children’s Health Protection (OCHP) in 1997 to support 

its efforts to increase the protection of children’s health throughout its programs. One of the 

many difficult issues the Office is addressing is the appropriate treatment of children’s health 

effects in the economic analyses performed by the Agency. Policy analysis efforts at the Agency 

often rely on the benefits transfer technique, and very few of the Agency’s benefit transfers have 

explicitly addressed children’s health issues. In addition, no accepted systematic process for 

conducting benefits transfer currently exists. To assist the Agency in its efforts, this paper 

discusses the benefits transfer technique as it applies to estimating values for children’s health. 

The first section provides some general background on the technique, and its application to 

estimate health-related values. The second section raises important general issues to consider 

when conducting a benefits transfer for children’s health values. The last section discusses the 

implications of using the benefits transfer method to estimate values for children’s health. The 

scarcity and state of existing child-oriented health valuation literature suggests that it may be 

necessary to transfer adult-oriented values to estimate child-related values (Neumann and 

Greenwood 1999). However, the results of this paper suggest that transfer of these value 

estimates to children at best provides estimates for a scoping analysis. In cases where these 

scoping exercises indicate that children's health values may be a crucial component in the policy 

analysis, primary research should be undertaken to estimate child-related values. 

1)	 Assess the appropriateness of transferring health benefit values estimated for 

adult populations to children. Describe the specific issues that arise in these 

transfers. Which variables or situations improve or decrease the appropriateness 

of transferring benefit values from adults to children? 

2)	 Does the analysis support the proposition that the value of children’s health 

effects should be estimated differently than adult health effects? Why or why 

not? 

3)	 Identify issues for further research that would improve our ability to estimate 

values for children’s health effects. 
Peer Review Handbook Page C-9 

CHARGE EXAMPLE 6 - Economics - Study of Municipal Government Costs and 

Financial Impacts from Environmental Regulations 

Background 

Municipalities play a major role in supplying environmental services. Local 

governments have taken responsibility for providing drinking water, sewage treatment, and 

waste disposal in a majority of communities. Over the past fifteen to twenty years, most of the 

mandates found in the federal environmental legislation enacted in the early 1970s have been 

met. The increase in the number of people served and improvements in the quality of local 

environmental services have been considerable, as has the investment in public infrastructure to 

meet these laws. 

Recent revisions to the environmental legislation have established a broader and more 

stringent set of standards to be met by suppliers of environmental services. As a result, many 

local governments are now faced with having to maintain all or some part of their public services 

at a higher level of performance. To meet these new standards will require additional 

investments in capital, and increases in rates charged to customers for environmental services. 

Improvements in environmental services are but one of several demands being made of 

local public infrastructure. Studies prepared on public infrastructure needs and the availability 

of funds to meet these needs indicate that there will be an excess demand for money to rebuild 

and improve upon the existing stock of public infrastructure. Therefore, it is important to 

recognize that additional environmental requirements will have to compete with other 

infrastructure needs (e.g., highways, bridges), as well as other public services (e.g., police, 

education, health and welfare programs) provided at the local level. 

Given the increasing demand for public services, this study examines what additional 

investments the new environmental legislation will require local governments to undertake, and 

the likelihood that they will face difficulties raising the necessary funds through capital markets 

and revenues from customers. The economic impacts of individual EPA actions are considered 

during the regulatory process in those situations permitted by environmental statutes. The 

unique feature of this study is its attempt to estimate the cumulative costs and impacts of 

meeting a combined set of EPA requirements, and to determine whether they will place a 

significant burden on the fiscal conditions of local governments, and require them to 

significantly increase existing charges for improved environmental services. 

Please find attached a copy of the draft study and appendices for your review. This 

version of the report reflects Agency comments received on an earlier draft. We expect that this 

version of the report, with some additional minor modifications, will be the final version. Your 

comments will be useful in preparing the final version of the report and discussing the findings 

of the reports with the public. 
Page C-10 Peer Review Handbook 

To assist in your review of the report, we ask that you pay particular attention to the 

following questions: 

1)	 Do state and local governments and financial markets consider household costs 

(measured as a percent of household income) and selected municipal financial 

information (debt service to general revenues or taxable property values) when 

evaluating the ability of enterprise systems and municipalities to issue bonds or 

obtain loans? 

2)	 Having selected a series of financial indicators, are the criteria used in the 

analysis acceptable? 

3)	 Do the results support our conclusions? What additional conclusions can be 

reached from the analysis? 

4) What modifications would you suggest be made to the recommendation section? 
Peer Review Handbook Page C-11 

CHARGE EXAMPLE 7 - Economics - Valuation of Fatal Cancer Risks 

Background 

The Science Advisory Board–Environmental Economics Advisory Committee (SABEEAC) review of the draft Guidelines for Preparing Economic Analyses (Guidelines) helped the 

Agency to identify valid and sound economic procedures to use when conducting benefit-cost 

analyses of environmental policies. Both the Guidelines and the SAB-EEAC review (EPASAB-EEAC-99-020, dated September 1999) recognize that economic theory will evolve and 

new empirical literature will be forthcoming that may necessitate revisiting the analytic 

procedures contained in the Guidelines. There may also be situations where the Guidelines do 

not provide sufficient detail to respond to analytic questions whose answers have broad 

implications for the conduct of economic analysis at the EPA. Both circumstances have recently 

arisen, making it necessary for the Agency to submit for SAB-EEAC review a document that 

examines the “benefit transfer” issues that arise when using the value of statistical life (VSL) 

literature that is based on accidental risks to estimate the economic benefits of environmental 

policies that reduce fatal cancer risks. 

The Guidelines provide information and guidance on the valuation of reduced mortality 

risks (Chapter 7, pp. 37-43). The Agency Guidelines conclude - and we understand the SABEEAC to have concurred in their review on this subject - that one practical and well-supported 

means to value changes in mortality risks is to use the Value of a Statistical Life (VSL) 

approach. Further, in response to the SAB-EEAC review, the Guidelines describe a number of 

important factors to consider in applying benefit transfer approaches using VSL estimates from 

the empirical literature on wage-risk tradeoffs. Recognizing that this is an important benefit 

category, the Agency Guidelines stated that the EPA would “continue to conduct annual reviews 

of the risk valuation literature” and “reconsider and revise the recommendations in these 

guidelines accordingly.” Furthermore, the EPA would “seek advice from the Science Advisory 

Board as guidance recommendations are revised.” 

The Agency needs to return to the SAB-EEAC and obtain additional counsel on this 

subject. Some economists within the government have suggested some particular approaches to 

dealing with the benefit-transfer issues. Since the Guidelines were drafted, a few relevant 

articles have been published that examine benefit transfer issues surrounding the use of VSL 

estimates when there is a passage of time (or latency period) between the pollution exposure and 

harm, or when fatal cancer risks are involved. The importance of these issues was articulated in 

a recently proposed regulation to reduce human health risks from radon in drinking water. The 

proposed rule estimated the number of reduced fatal cancers resulting from different regulatory 

options. The Agency presented information on the economic values for the reductions in fatal 

cancer risks, along with other quantified benefits. A brief discussion of some of the benefit 

transfer issues involved in this estimation was published in the preamble to the proposed rule for 

setting standards for exposure to radon from drinking water sources (Federal Register, 
Page C-12 Peer Review Handbook 

November 2, 1999 volume 64, Number 211, pages 59245-59378). Quoting from the Federal 

Register notice requesting SAB review: 

Latency is one of a number of adjustments or factors that are related to an evaluation of 

potential benefits associated with this rule, how those benefits are calculated, and when those 

economic benefits occur. Other factors which may influence the estimate of economic benefits 

associated with avoided cancer fatalities include (1) a possible “cancer premium” (i.e., the 

additional value or sum that people may be willing to pay to avoid the experiences of dread, pain 

and suffering, and diminished quality of life associated with cancer-related illness and ultimate 

fatality); (2) the willingness of people to pay more over time to avoid mortality risk as their 

income rises; (3) a possible premium for accepting involuntary risks as opposed to voluntary 

assumed risks; (4) the greater risk aversion of the general population compared to the workers in 

the wage-risk valuation studies; (5) “altruism” or the willingness of people to pay more to reduce 

risk in other sectors of the population; and (6) a consideration of health status and life years 

remaining at the time of premature mortality. Use of certain of these factors may significantly 

increase the present value estimate. EPA therefore believes that adjustments should be 

considered simultaneously. The Agency also believes that there is currently neither a clear 

consensus among economists about how to simultaneously analyze each of these adjustments 

nor is there adequate empirical data to support definitive quantitative estimates for all potentially 

significant adjustment factors. As a result, the primary estimates of economic benefits presented 

in the analysis of this rule rely on the unadjusted $5.8 million estimate. However, EPA solicits 

comment on whether and how to conduct these potential adjustments to economic benefits 

estimates together with any rationale or supporting data commenters wish to offer. Because of 

the complexity of these issues, EPA will ask the Science Advisory Board (SAB) to conduct a 

review of these benefits transfer issues associated with economic valuation of adjustments in 

mortality risks. In its analysis of the final rule, EPA will attempt to develop and present an 

analysis and estimate of the latency structure and associated benefits transfer issues outlined 

previously consistent with the recommendations of the SAB and subject to resolution of any 

technical limitations of the data and models.” (page 59326) 

In the process of responding to reviews prepared during deliberations on the proposed 

radon rule, the Agency found that the Guidelines lack sufficient detail on how to fully evaluate 

and characterize the different risk attributes that are central to a complete understanding of the 

benefit-cost implications of this rule. For example, time can pass between the point of initial 

exposure to a carcinogen, the biological manifestation or onset of cancer in the body, the medical 

diagnosis of cancer, and death caused by the cancer. During development of policies affecting 

cancer risks, suggestions have been made to discount the VSL estimate (i.e., $5.8 million 

recommended in the Guidelines) to account for latencies, or the delay in time between reduced 

exposure and when the cancer death would have occurred absent the exposure reduction. 

Others argued that a suitable approach for valuing benefits from reduced cancer risks 

must consider simultaneously all of the benefit transfer factors related to valuing cancer risks to 

ensure a careful and full treatment of benefits. There is evidence in the economics literature 
Peer Review Handbook Page C-13 

regarding many such factors (e.g., potential premiums ascribed to cancer risk reductions due to a 

higher willingness to pay to avoid the dread, pain and suffering, morbidity effects, and other 

features of cancer endpoints) that may suggest introducing upward adjustments factors which 

offset any potential downward adjustments caused by accounting for cancer latency. In addition, 

proponents argue that adjustments for the age of population at risk, income, altruism and other 

risk characteristics (e.g., controllability, voluntariness) can all have some potential influence on 

the value of a statistical cancer fatality (VSCF) and therefore need to be reflected in the 

quantitative benefit assessment. 

While developing the primary benefit estimates for reduced fatal cancer risks in the 

proposed radon rule, questions arose regarding the implementation of adjustments for some 

factors, but not others. For example, would it ever be appropriate to adjust only for latency 

periods, and not other factors, in the valuation of reduced cancer deaths? To help answer this 

and related questions regarding the valuation of cancer risks, the Agency seeks the SAB-EEAC’s 

counsel. We further ask that your guidance reflect the typical uncertainties facing EPA 

economists, including those surrounding the underlying risk assessments, the prediction (or lack 

thereof) of latency periods for cancers, and the risk characteristics associated with the VSL 

approach. 

Therefore, the Agency proposes to seek review of a “white paper” and list of charge 

questions by the SAB-EEAC on the valuation and benefit transfer practices arising in the 

calculation of the economic benefits of reduced fatal cancer risks. The Agency seeks SABEEAC review of the treatment and presentation of quantitative and qualitative information for 

these types of benefits. Numeric case studies are included in the white paper, to both identify 

and present prospective approaches to address these issues. The results of the SAB-EEAC 

review of this document, and responses to the specific charge questions, will be considered by 

the Agency during future revisions to the Guidelines, consistent with the Agency’s commitment 

to credible and consistent economic analysis in support of the policy making process. 

Charge Questions: 

As the Committee considers the charge questions, it is asked to keep in mind the 

differing situations relating to differing degrees of data availability or uncertainty in key 

parameters. 

1)	 Does the white paper accurately describe the empirical economic literature 

relevant to the benefit transfer issues that ensue when using the VSL literature to 

estimate the VSCF in a benefit-cost analysis? 

2)	 Does the white paper present the important risk and demographic factors that can 

affect benefit transfer approaches that use VSL estimates for VSCF? 
Page C-14 Peer Review Handbook 

3)	 Does the white paper accurately describe attempts in the economic literature to 

measure VSCF directly? 

4)	 There are two numeric case studies of environmental cancer risks developed for 

the white paper. Each presents risk assessment information that forms the basis 

for quantifying the number of statistical cancer fatalities that will be reduced as a 

consequence of a hypothetical proposed environmental policy. The case studies 

are then used to illustrate the outcome of using direct measures of the VSCF and 

benefit transfer adjustments to VSL estimates in order to calculate the VSCF. 

a)	 Which of the valuation approaches applied to the case study designated as 

ALPHA are valid to use? Does this case study omit any credible 

alternative protocols for valuing reductions in fatal cancer risks for 

benefit-cost analyses of environmental programs? 

b)	 Which of the valuation approaches applied to the case study designated as 

OMEGA are valid to use? Does this case study omit any credible 

alternative protocols for valuing reductions in fatal cancer risks for 

benefit-cost analyses of environmental programs? 

5)	 Which economic methods illustrated with the case studies, or additional methods 

identified by the Committee under charge questions 4.a and 4.b, serve as credible 

protocols for the Agency to use in representing quantitative data, qualitative 

information, and sensitivity analyses for the economic value of reduced fatal 

cancer risks reported in benefit-cost analyses? 
Peer Review Handbook Page C-15 

CHARGE EXAMPLE 8 - Hudson River PCBs Site Reassessment RI/FS 

Modeling Approach Charge: Peer Review 1 

Members of this peer review will be tasked to determine whether the models being used 

to support the decision-making process for the Reassessment, and the assumptions therein, are 

appropriate. The peer reviewers will base their assessment on the review the Preliminary Model 

Calibration Report (PMCR), an updated Technical Scope of Work for the Baseline Modeling 

Report (Appendix B of the PMCR) and the responses to selected comments received from 

stakeholders during the public comment period on the PMCR. 

In October 1996, EPA released the Preliminary Model Calibration Report (PMCR), 

which described the models, datasets and assumptions being used as part of the Hudson River 

PCB Reassessment RI/FS. The PMCR represents the status of the preliminary PCB modeling 

effort as of Fall 1995. Datasets, database corrections and other pertinent information which 

became available after October 1995 were not incorporated within the fate and transport 

modeling presented in the PMCR. The PMCR was an interim document prepared to describe 

work in progress and was not intended to be a conclusive report. In particular the HUDTOX 

model presented in the PMCR was not intended to be used as a predictive tool to assess remedial 

action scenarios. In addition, while time-varying mechanistic models of bioaccumulation will be 

used along with other models to predict fish body burdens, these models are not described in the 

PMCR. 

The PMCR was not formally peer reviewed at the time of publication, but was 

distributed to interested parties who were invited to submit comments and questions. Written 

responses were made to all of these comments and questions. In addition, the work plan 

contained in Appendix B of the PMCR has been revised to reflect the ongoing work being 

conducted as part of the Baseline Modeling effort. Results from this effort will be presented in a 

Baseline Modeling Report that will be formally peer reviewed. 

The peer reviewers are requested to determine whether the models being used to support 

the decision-making process for the Reassessment RI/FS, and the assumptions therein, are 

appropriate. The peer reviewers are not being asked whether they would conduct the work in 

the same manner, only whether the work being conducted will yield scientifically credible 

conclusions. 

It is suggested that the reviewer first read the PMCR. The Responses to Comments 

provides information on the context of the PMCR within the overall modeling effort and 

additional details beyond the PMCR results. The current work plan as revised in June 1998 

reflects the ongoing Baseline Modeling effort and revisions to some of the original modeling 

tasks proposed in Appendix B of the PMCR. In addition, the USEPA/TAMS Phase 2 database 

has been considerably revised. New datasets have been added and some earlier datasets have 

been extensively revised. 
Page C-16 Peer Review Handbook 

The peer reviewers are asked to comment on the following: 

1)	 Is EPA using appropriate models, datasets and assumptions on which to base a 

scientifically credible decision? 

2)	 Will the models, with the associated datasets and assumptions, be able to answer 

the following principal study questions as stated in the PMCR? 

a)	 When will PCB levels in the fish population recover to levels meeting 

human health and ecological risk criteria under No Action? 

b)	 Can remedies other than No Action significantly shorten the time required 

to achieve acceptable risk levels? 

c)	 Are there contaminated sediments now buried and effectively sequestered 

from the food chain which are likely to become “reactivated” following a 

major flood, resulting in an increase in contamination of the fish 

population? 

3) Specific questions: 

a)	 Are the modeling approaches suitable for developing quantitative 

relationships between external forcing functions (e.g., hydraulic flows, 

solids and PCB loads, sediment initial conditions, etc.) and PCB 

concentrations in the water column, sediments and fish? Are the models 

adequate for discriminating between water-related and sediment-related 

sources of PCBs? 

b)	 Are the spatial and temporal scales of the modeling approaches adequate 

to answer the principal study questions? If not, what levels of spatial and 

temporal resolution are required to answer these questions? What 

supporting data are required for calibration/ validation of these spatial and 

temporal scales? 

c)	 It is contemplated that PCB concentrations in fish will be estimated using 

several modeling approaches: an empirical probabilistic model derived 

from Hudson River data, a steady state model that takes into account 

mechanisms of bioaccumulation body burdens, and a time-varying 

mechanistic model (not included in the PMCR). A bi-variate statistical 

model may also be used to provide insight into accumulations. This multimodel approach is being contemplated because of the uncertainties 

associated with any individual model. Is this a reasonable approach or 

should predictions be made using a single “best” model? 
Peer Review Handbook Page C-17 

d)	 Is the level of process resolution in the models adequate to answer the 

principal study questions? If not, what processes and what levels of 

resolution are required to answer these questions? What supporting data 

(such as data to support specifications of a mixed depth layer, solids and 

scour dynamics, groundwater inflow, etc.) are required for these processes 

and levels of resolution? 

e)	 The results of the modeling effort will be used, in part, to support human 

and ecological risk assessments. In your judgment, will the models 

provide estimates adequate for this purpose? 

4)	 Are there any changes to the work effort outlined in the revised work plan that 

would significantly improve the outcome? 

5)	 In terms of evaluating the overall and specific effects and behavior of PCBs in the 

Hudson River, are there any serious flaws in the modeling approach (theory, 

structure, physical parameters, etc.) that would limit or invalidate any conclusions 

or further work based upon the results of these models? 

Recommendations 

Based on your reading and analysis of the information provided, please identify and 

submit an explanation of your overall recommendation for the modeling effort for the Hudson 

River PCB Reassessment RI/FS: 

1) Acceptable as is 

2) Acceptable with minor revision (as indicated) 

3) Acceptable with major revision (as outlined) 

4) Not acceptable (under any circumstance) 
Page C-18 Peer Review Handbook 

CHARGE EXAMPLE 9 - IRIS Pilot Program - Instructions to Peer Reviewers for 

Reviewing IRIS Summaries and Supporting Documentation 

The U.S. EPA is conducting a peer review of the scientific basis supporting the health 

hazard and dose response assessments for the subject chemical that will appear on the Agency’s 

online database, the Integrated Risk Information System (IRIS). Materials to be reviewed 

include the summary information that will appear on IRIS (the inhalation reference concentration 

[RfC], oral reference dose [RfD], and cancer assessment) and the supporting document, the 

Toxicological Review, which will also be made available to the public. 

A listing of Agency Guidelines and Methodologies that were used in the development of 

these hazard and dose-response assessments included the following: The Risk Assessment 

Guidelines (1986), the (new) Proposed Guidelines for Carcinogen Risk Assessment ( 1996), 

Guidelines for Developmental Toxicity Risk Assessment, (proposed) Interim Policy for Particle 

Size and Limit Concentration Issues in Inhalation Toxicity, (proposed) Guidelines for 

Neurotoxicity Risk Assessment, Methods for Derivation of Inhalation Reference Concentrations 

and Application of Inhalation Dosimetry, Recommendations for and Documentation of 

Biological Values for Use in Risk Assessment and Use of the Benchmark Dose Approach in 

Health Risk Assessment. Copies of these documents (and/or their relevant Sections) will be 

made to the reviewer upon request. 

Peer review is meant to ensure that science is used credibly and appropriately in 

derivation of these dose-response assessments. You have been chosen as an expert on the 

chemical under consideration, on a scientific discipline related to at least one of the assessments, 

or in the field of risk assessment. At least three peer reviewers per chemical are being chosen to 

review the scientific basis of these draft dose-response assessments before they are forwarded on 

to the EPA's Consensus Process for final approval and adoption by the EPA. These hazard and 

dose-response assessments will then appear on IRIS and become available as Agency consensus 

health effect information. 

The primary function of the peer reviewer should be to judge whether the choice, use, 

and interpretation of data employed in the derivation of the assessments is appropriate and 

scientifically sound. This review is not of the recommended Agency risk assessment guidelines 

or methodologies used to derive cancer or RfD/C assessments as these have been reviewed by 

external scientific peers, the public, and EPA Science Advisory Boards. The reviewer’s 

comments on the application of these guidelines/methodologies within the individual 

assessments is, however, welcomed and encouraged. For example, the reviewer may ascertain 

whether or not there is data sufficient to support use of other than default assumptions for areas 

such as sensitive subpopulations or linear cancer extrapolation. The reviewer may also have 

opinions on other areas of uncertainty such as subchronic to chronic duration (when only a 

subchronic study is available) or an incomplete data base but should focus on the specific area of 

uncertainty rather than on the magnitude of the overall estimate. 
Peer Review Handbook Page C-19 

Below are two groups of questions regarding this review. The first is a set of general 

questions that are meant to guide you through your review. It is not imperative that you 

specifically answer each question of this group. The second group of questions, however, are 

specific for the chemical assessments and deal with areas of scientific controversy or uncertainty 

in which the Agency may have to make a scientific judgment. Your input to this set of questions 

is considered vital to the review process. 

Questions for IRIS Peer Reviewers -General 

1)	 Are you aware of any other data/studies that are relevant (i.e., useful for the 

hazard identification or dose-response assessment) for the assessment of the 

adverse health effects, both cancer and noncancer, of this chemical? Please 

explain fully. 

2)	 For the RfD and RfC, has the most appropriate critical effect been chosen (i.e., 

that adverse effect appearing first in a dose-response continuum)? For the cancer 

assessment, are the tumors observed biologically significant? relevant to human 

health? Points relevant to this determination include whether or not the choice 

follows from the dose-response assessment, whether the effect is considered 

adverse, and if the effect (including tumors observed in the cancer assessment) 

and the species in which it is observed is a valid model for humans. 

3)	 Have the noncancer and cancer assessments been based on the most appropriate 

studies? These studies should present the critical effect/cancer (tumors or 

appropriate precursor) in the clearest dose-response relationship. If not, what 

other study (or studies) should be chosen and why? 

4)	 Studies included in the RfD and RfC under the heading "Supporting/Additional 

studies" are meant to lend scientific justification for the designation of critical 

effect by including any relevant pathogenesis in humans, any applicable 

mechanistic information, any evidence corroborative of the critical effect, or to 

establish the comprehensiveness of the data base with respect to various 

endpoints (such as reproductive/developmental toxicity studies). Should other 

studies be included under the "Supporting/Additional" category? Should some 

studies be removed? 

5)	 For the noncancer assessments, are there other data that should be considered in 

developing the uncertainty factors or the modifying factor? Do you consider that 

the data support use of different (default) values than those proposed? 

6)	 Do the Confidence statements and weight-of-evidence statements present a clear 

rationale and accurately reflect the utility of the studies chosen, the relevancy of 

the effects (cancer and noncancer) to humans, and the comprehensiveness of the 
Page C-20 Peer Review Handbook 

data base? Do these statements make sufficiently apparent all the underlying 

assumptions and limitations of these assessments? If not, what needs to be 

added? 

Questions for IRIS Peer Reviewers - Chemical Specific 

[example: cumene] 

1)	 Based on the information noted in the Principal study currently designated 

(Cushman et al., 1995) is the discounting of the renal effects in males justified? 

Is sufficient rationale given to let stand the organ weight changes in female rats as 

a critical effect? 

2)	 Is the information in the Toxicological Review sufficient to consider cumene as 

having a low potential for causing reproductive effects? Please explain fully. 

RECOMMENDATIONS 

Based on your reading and analysis of the information provided, please identify your 

overall recommendation for the IRIS materials you have reviewed as 

1) Acceptable as is 

2) Acceptable with minor revision (as indicated) 

3) Acceptable with major revision (as outlined) 

4) Not acceptable 
Peer Review Handbook Page C-21 

CHARGE EXAMPLE 10 - Charge to Reviewers for the WTI Draft Final Risk Assessment 

The draft final WTI risk assessment is divided into several volumes covering the 

scientific disciplines of toxicology, environmental fate and transport, combustion engineering, 

atmospheric modeling, exposure assessment, ecological risk assessment, and accident analysis. 

As a reviewer of the WTI draft final risk assessment, you should use your best technical 

knowledge and professional judgment to comment on the technical accuracy, completeness and 

scientific soundness of the assessment. Each reviewer is asked to focus on several specific 

issues in his or her area of expertise with comments on other areas invited but optional. Your 

comments will be considered in finalizing the risk assessment. 

For the peer review workshop reviewers will be organized into 5 work groups: 

Combustion Engineering, Air Dispersion and Deposition Modeling and Accident Analysis, 

Toxicology, Exposure Assessment, and Ecological Risk Assessment. All reviewers should be 

familiar with the Executive Summary (Volume I) and the Facility Background (Volume II) 

Sections of the draft risk assessment. In addition, each work group should focus on specific 

Volumes as specified below: 

Workgroup Risk Assessment Volumes 

Combustion Engineering Volume III - Facility 

Emissions 

Air Dispersion and Volume IV - Atmospheric Volume VII - Accident


Deposition Modeling and Dispersion and Deposition Analysis


Accident Analysis Modeling


Toxicology	 Volume V - Human Health 

Risk Assessment 

Volume VII - Accident 

Analysis 

Exposure Assessment	 Volume V - Human Health 

Risk Assessment 

Volume VII - Accident 

Analysis 

Ecological Risk Assessment	 Volume VI - Screening 

Ecological Risk Assessment 

While reviewing these Sections of the document, please address the following general 

issues. 
Page C-22 Peer Review Handbook 

1)	 Comment on the organization of the risk assessment document. Does the layout 

follow a logical format? Is the presentation of information in the document clear, 

concise and easy to follow? 

2)	 Does the executive summary accurately reflect the data and methodologies used 

and the conclusions derived in the risk assessment? 

3)	 Were the major recommendations of the 1993 peer review workshop for the risk 

assessment plan addressed? 

4)	 As with any risk assessment, there are always additional data and method 

development efforts that could be undertaken to reduce the level of uncertainty. 

However, are there any major data or methodological gaps that would preclude 

the use of this risk assessment for decision making? If so, how should they be 

addressed? 

5)	 What long-term research would you recommend that could improve risk 

assessments of this type in the future? 

In addition, the following workgroup specific issues should be addressed. 

Emissions Characterization 

Emissions characterization includes identification of substances of concern and the 

development of emission rates for these contaminants. Emission rates were developed through a 

combination of site specific stack test data and models. Please comment on the following issues 

with respect to this aspect of the draft risk assessment. 

1)	 To characterize the nature of the emissions, waste stream profiles were developed 

and entered into a database. Several refinements and adjustments (e.g., the 

Subtraction Correction Factor for chlorinated compounds) were applied to the 

profiles before substances of concern were identified. Please comment on 

whether or not these adjustments are appropriate. What is the anticipated effect 

on the risk assessment? 

2)	 Comment on the selected chemicals of concern. Have important chemicals been 

missed due to the selection technique? 

3)	 Comment on the approaches used to estimate stack emission rates (e.g., use of the 

95% UCL of the arithmetic mean or the maximum detected value, whichever is 

smaller, for high end emission rates). Are the approaches appropriate? Are their 

effects on the risk assessment adequately characterized? Comment on the 
Peer Review Handbook Page C-23 

adjustment made to PCDD/PCDF emission rates to account for brominated 

dioxin-like compounds. Also, comment on the approach to characterizing 

emission rates from fugitive sources (e.g., use of the TANKS 2 model for the 

Carbon Adsorption Bed). 

4)	 Comment on the identified sources of fugitive emissions. Was the approach used 

to select these sources appropriate? Have important sources been missed? Have 

emissions from process upsets been given appropriate consideration? 

5)	 There have been a number of controlled burns at the WTI facility. Please 

comment on the adequacy of these data in estimating potential exposure. Please 

comment on the assumptions made from the tests in regard to composition of 

wastes received at WTI and emissions when the plant operates in the future. 

6)	 Comment on the use of emission factors from coal burning to estimate the 

emission rate of fly ash from WTI. Are the factors used to adjust the coal 

emission rate appropriate? Are the uncertainties introduced from this approach 

adequately characterized? 

7)	 Overall, is the identification of the key assumptions used in characterizing the 

nature and magnitude of emissions thorough? Are the magnitude and direction of 

effect of these assumptions on the overall risk assessment accurately 

characterized? Is the uncertainty and variability inherent in this analysis 

adequately discussed? Does the sensitivity analysis cover the major parameters 

expected to have an effect on the risk assessment? 

Dispersion and Deposition Modeling 

To develop this risk assessment, computer models have been used with site specific data 

on emission rates and meteorological conditions to simulate the air concentrations and 

deposition rates for contaminants potentially emitted from the WTI facility. The models used 

include the Industrial Source Complex - Complex Terrain Deposition (ISC-COMPDEP), the 

CALPUFF, and the INPUFF models. In your review, please address the following issues. 

1)	 Since the 1993 peer review of the risk assessment plan, a number of efforts have 

been completed to reduce the uncertainty associated with the air dispersion and 

deposition modeling. These efforts include the collection of site-specific data for 

emission rates and meteorological conditions. Also, a wind tunnel study was 

conducted to evaluate the effects of the complex terrain surrounding the WTI 

facility. Does the risk assessment document adequately summarize these 

activities? Is the link between these data collection efforts, the air dispersion 

models, and the risk assessment clearly established? 
Page C-24 Peer Review Handbook 

2)	 The results of 12 sets of sensitivity tests indicate that geophysical variables (e.g., 

terrain) are more likely to affect dispersion and deposition than emission 

variables (e.g., stack temperature). Were these sensitivity analyses adequate? 

Comment on the conclusions reached. To further examine the effect of 

geophysical variables, wind tunnel testing was conducted to model the terrain 

induced flow effects expected near WTI. It was concluded that changes in peak 

concentrations attributed to these effects are relatively minor and that the ISCCOMPDEP model is sufficiently conservative. Comment on this conclusion. 

Have these analyses helped to characterize and/or reduce the uncertainty in the air 

dispersion modeling associated with the complex terrain surrounding WTI. 

3)	 The ISC-COMPDEP model does not allow for non-steady state conditions such 

as calm winds and strong temperature inversions. Therefore, CALPUFF was 

used to estimate air dispersion and deposition under these conditions. However, 

CALPUFF gave similar peak, 24 hour, and annual average concentrations as ISCCOMPDEP. Comment on the adequacy of this analysis. Comment on the 

conclusions reached. Has this analysis helped to characterize and/or reduce the 

uncertainty in the air dispersion modeling associated with non-steady state 

meteorological conditions? 

4)	 Atmospheric dispersion modeling was used to estimate air concentrations of 

hazardous chemicals for the accident analysis. The SLAB model was used for 

vapor releases from spills and the mixing of incompatible wastes. ISCCOMPDEP was used for releases associated with fires. Comment on the 

selection of the models and inputs. Are they appropriate selections? 

5)	 Overall, have adequate sensitivity tests been conducted to demonstrate the 

magnitude of variation in concentrations and deposition estimates with model 

inputs? Please explain fully. 

Human Health Risks 

Human Health Risk Assessment includes hazard identification, dose-response evaluation, 

exposure assessment, and risk characterization. To develop the risk assessment, potentially 

exposed populations have been identified and the magnitude, frequency, and duration of their 

exposure quantified. This information was then integrated with the hazard identification and 

dose response evaluation for the risk characterization. For this risk assessment, both 

carcinogenic and non-carcinogenic health effects have been evaluated. In your review, please 

comment on the following issues. 
Peer Review Handbook Page C-25 

Exposure 

1)	 EPA's Exposure Assessment Guidelines identify certain exposure descriptors that 

should be used to characterize exposure estimates. The Guidelines define high 

end exposure estimates as those representing individuals above the 90th 

percentile on the exposure distribution but not higher than the individual in the 

population who has the highest exposure. Bounding exposure estimates are those 

that are higher than the exposure incurred by the person in the population with the 

highest exposure. Central tendency exposure estimates are defined as the best 

representation of the center of the exposure distribution (e.g., arithmetic mean for 

normal distributions). Comment on whether or not the WTI exposure assessment 

properly characterizes each of the exposure estimates in terms of these 

descriptors. 

2)	 The factors that go into estimating a central tendency or high end exposure, once 

the population has been defined, include the environmental media concentration, 

the intake rate, and the duration and/or frequency of exposure. Comment on 

whether or not the WTI exposure assessment does an adequate job of describing 

the logical procedure of combining these factors to develop central tendency, high 

end, and/or bounding estimates of exposure for each of the exposed 

subpopulations. 

3)	 An important factor in an exposure assessment is identifying all of the important 

exposure sources. Please comment on the adequacy of the WTI assessment in 

identifying the important sources and pathways of exposure. 

4)	 Have the key assumptions for estimation of chemical concentration and for 

estimation of exposure been identified? Are the magnitude and direction of effect 

correct for the assumptions that have been identified? 

5)	 Supposedly, conservative assumptions have been applied in this assessment to 

account for uncertainty. Are the conservative assumptions appropriately factored 

into the ultimate characterization of what descriptor best applies to each exposure 

estimate? Please comment on whether the uncertainties were confronted in an 

adequate manner. If they were not, please state what should be done differently. 

Hazard Identification/Dose Response and Risk Characterization 

1)	 To select surrogate compounds for quantitative risk assessment, a two step 

process was used in which chemicals were ranked on the basis of emission rate, 

toxicity (both cancer and non-cancer), and bioaccumulation potential. Please 

comment on this selection process. Are the ranking factors appropriate? Could 
Page C-26 Peer Review Handbook 

important compounds have been omitted from the analysis based on the ranking 

procedure? 

2)	 For the majority of the chemicals of concern, traditional approaches to dose 

response evaluation were employed (e.g., use of a slope factor for cancer and use 

of a RfD/RfC for non-cancer). However for certain chemicals or groups of 

compounds a different methodology was used. Specifically, dioxins, furans, 

PAHs, lead, mercury, nickel, chromium, acid gases, and particulate matter were 

given special consideration. Please comment on the methodology used for these 

compounds. Was it appropriate? Have the uncertainties associated with the 

methodology been adequately characterized? Comment on the assumptions used 

due to a lack of chemical specific data. 

3)	 Please comment on the selection of the overall population and the various 

subpopulations at risk. Were site specific data, such as the informal home 

gardening survey, properly utilized to identify these subpopulations? 

4)	 It is stated in the risk assessment that average risk estimates are based on average 

emission rates, average air dispersion/deposition within a subarea, and typical 

exposure factors. Further, maximum risks are based on average emission rates, 

typical exposure factors, and the maximum air concentration within a subarea. 

Please comment on this use of the terms average and maximum risks. Are these 

descriptive terms appropriate given the parameters used to derive each? Please 

explain fully. 

5)	 Comment on whether or not the non-cancer risks of chemicals of concern have 

been adequately addressed by the risk assessment? For example, has an adequate 

discussion of endocrine disrupters been provided which either characterizes their 

risks or clearly explains why their risks cannot be characterized? Further, have 

non-cancer chronic toxicities of dioxins and furans been adequately addressed in 

the risk assessment? 

6)	 Please comment on whether or not the uncertainties associated with the additivity 

and/or synergy of risks from pollutants emitted together from the WTI facility are 

adequately discussed in the risk assessment. 

7)	 Have the key assumptions for estimation of dose and risk been identified? Are 

the magnitude and direction of effect correct for the assumptions that have been 

identified? Please comment on whether the uncertainties were confronted in an 

adequate manner. If they were not, please state what should be done differently. 
Peer Review Handbook Page C-27 

8)	 Please comment on the overall adequacy of the risk characterization. Does the 

risk characterization include a statement of confidence in the risk assessment 

including a discussion of the major uncertainties. Are the hazard identification, 

dose-response assessment, and exposure assessment clearly presented? Have 

sufficient risk descriptors which include important subgroups been presented and 

discussed? 

Screening Ecological Risk Assessment 

As with the human health risk assessment, the ecological risk assessment pulls together 

elements of exposure analysis and dose-response evaluations to develop a risk characterization. 

For the Screening level Ecological Risk Assessment (SERA), Ecological Chemicals of Concern 

(ECOC) and indicator species have been identified to provide conservative estimates of risk. 

Please address the following issues in your review. 

1)	 Are there any components of the SERA which you feel undermine the scientific 

validity of the assessment? If so, what are they and can you provide suggestions 

to strengthen the identified components? 

2)	 Is the organization of the document clear and does it present the material in a 

clear and concise manner consistent with the Framework for Ecological Risk 

Assessment (EPA, 1992)? Please explain fully. 

3)	 Uncertainties are discussed in numerous sections of the SERA and compose 

Section VIII of the SERA. In each case, do these discussions cover all relevant 

and important aspects of the uncertainties which you think should be addressed in 

the SERA? 

4)	 In your opinion, what is the weakest and what is the strongest aspect of the 

SERA? Can you make any suggestions on how the weakest parts can be 

strengthened by the Agency? 

5)	 In Section II, are the stressors, ecological effects, and both the assessment and 

measurement endpoints adequately characterized? Are the five emission 

scenarios adequate to characterize the exposures for the WTI facility? Are there 

other emission scenarios which you think should be included in the SERA? 

6)	 In Section III, is the site characterization adequate to support the SERA? Why or 

why not? 

7)	 In Section IV, is the tiered process used to identify the ecological chemicals of 

concern (ECOC) from the initial list of potential chemicals considered 
Page C-28 Peer Review Handbook 

scientifically defensible? Does application of this tiered approach support the 

statement made in the SERA "by focusing on the potential risk from the selected 

ECOCs, the SERA provides a thorough screening-level evaluation for the WTI 

facility?" 

8)	 In Sections V and VI, are the exposure and ecological effects adequately 

characterized? Are the most appropriate estimation techniques available used? 

Are the assumptions clearly stated? Please explain fully. 

9)	 In Section VIII, are there any major elements missing from the risk 

characterization which you think need to be included or which would strengthen 

the risk characterization? Does the risk characterization support the summary and 

conclusions presented in Section IX? 

10)	 In Section IX, given the assumptions made and the processes used to select and 

evaluate chemicals, receptors, and exposure pathways, do you think the SERA 

adequately met its objective of not inadvertently underestimating risk? 

Accident Analysis 

The Accident Analysis for the WTI incinerator involves evaluating the probability of an 

emergency incident occurring which results in the release of hazardous waste. The 

consequences of this release are also evaluated using exposure and human health effects 

information. Unlike the human health risk assessment which has a primary goal of quantifying 

risks, the accident analysis typically provides information that can be used to reduce the 

likelihood, extent and impact of possible accidents. Please comment on the following issues in 

your review of this aspect of the risk assessment. 

1)	 The WTI accident assessment selected five scenarios for quantitative evaluation 

that were considered to be of primary concern. The scenarios are an on-site spill, 

an on-site fire, an on-site mixing of incompatible waste, an off-site spill, and an 

off-site spill and fire. Please comment on the selection of these scenarios. Were 

any significant scenarios missed? 

2)	 Specific chemicals were selected to evaluate each scenario. Please comment on 

the selections. Would other chemicals have been more appropriate? 

3)	 Chemical specific release rates are calculated for each scenario. Please comment 

on the procedures used to estimate the release rates. Was an appropriate approach 

used? 
Peer Review Handbook Page C-29 

4)	 Atmospheric dispersion modeling was used to estimate air concentrations of 

hazardous chemicals. Specifically, the SLAB model was used for vapor releases 

from spills and the mixing of incompatible wastes. ISC-COMPDEP was used for 

releases associated with fires. Comment on the selection of the models and inputs. Are they appropriate selections? Should other models or inputs been used? 

5)	 Please comment on the assessment's conclusions on the severity of consequences 

and probability of occurrence. Has the report correctly categorized the severity of 

the consequences of the different accident scenarios? Has the assessment 

adequately justified the reported probability of occurrence of each of the accident 

events? 

6)	 Key assumptions were made in the identification of accident scenarios and the 

description of the conservative and typical events. Included were a description of 

the magnitude of the effect of the assumptions and direction of the effect. Please 

comment on the assumptions. Are they justified? Are the descriptions of the 

magnitude and directions of the effects correct? Has the accident assessment 

adequately confronted the uncertainties involved in doing this type of analysis? If 

not, what else should be done? 

7)	 Comment on the appropriateness of using IDLH values for characterizing the 

severity of consequences in the accident analysis. Comment on the 

appropriateness of using 10 X LOC for chemicals for which IDLH values have 

not been established. 

8)	 In the accident analysis, IDLH (or 10 X LOC) values were used to determine the 

downwind distances over which adverse human health effects might occur. To 

evaluate the uncertainty introduced by using the IDLH, a sensitivity analysis was 

conducted where these distances were recalculated using the LOC (a more 

stringent health criteria). Other sources of uncertainty that are identified in the 

accident analysis include concentration averaging times, chemical concentrations, 

emission rates, and meteorological conditions. For most of these parameters it is 

stated that conservative assumptions were used to avoid underestimating risks. 

Have the uncertainties inherent in the accident analysis been adequately 

characterized? For those parameters where sensitivity analyses were not 

conducted, is the conclusion that conservative assumptions have avoided 

underestimation valid? 

Peer Review Handbook Page D-1 

APPENDIX D - GUIDANCE ON REQUESTING A


REVIEW BY THE 


US EPA SCIENCE ADVISORY BOARD (SAB)

Page D-2 Peer Review Handbook 

Guidance on Requesting a Review 


By the US EPA Science Advisory Board (SAB)


Summary 

The US EPA Science Advisory Board (SAB or the “Board”) annually solicits proposals 

for review projects every spring for the following fiscal year. This appendix provides guidance 

to Programs and Regions to help them submit requests for SAB reviews. Requests should be 

submitted to the SAB in both hard copy and electronic versions usually by mid-June for the 

following fiscal year. The requests may be part of the annual submissions that respond to the 

peer review activities of the Agency, or they may be submitted directly to the Board. Although 

providing requests at one defined time in the spring helps with SAB planning, we recognize that 

projects also come to light during other parts of the year. Please contact the SAB staff for details 

on making submissions during the remainder of the year (see end of this document for contacts). 

Background 

A key priority for the Administrator is to base Agency actions on sound scientific data, 

analyses, and interpretations. The Administrator issued the Agency's Peer Review Policy to 

increase the quality of the technical foundations upon which EPA's regulatory structures are 

built. The SAB is a key scientific peer review mechanism available to Programs and Regions in 

implementing the Peer Review Policy. However, because the Board has finite resources it 

cannot conduct all reviews. This document is designed to help Programs and Regions determine 

which projects to submit to the SAB. Note particularly that the SAB focuses on the technical 

underpinnings of Agency positions; i.e., risk assessment issues, in contrast to risk management 

issues. 

The topics that are best suited for the Board's agenda are those that satisfy several of the 

following criteria: 

1) Integrate science into Agency actions in new ways. 

2) Influence long-term technological developments. 

3) Impact overall environmental protection. 

4) Address novel scientific problems or principles. 

5)	 Address problems that transcend federal-agency or other organizational 

boundaries. 

6) Strengthen the Agency's basic capabilities. 

7) Serve Congressional or other leadership interests. 

8) Deal with controversial issues. 

In suggesting issues for SAB involvement, Programs and Regions should note the 

breadth of SAB activities: 
Peer Review Handbook Page D-3 

Historically, most of the outputs of the Board are in the form of "full" reports. 

They present the findings of peer reviews of Agency document(s) and contain 

considerable detail about the findings and recommendations of the Board. They also 

address the specific questions posed by the Charge to the Board. "Letter" reports fulfill 

the same function as reports, but are simply shorter in length. Due to the need to be more 

responsive with advice, the Board has recently begun to produce more short letter reports 

than full reports, as they can be produced and finalized in less time. 

The SAB has also introduced the "Consultation” as a means of conferring--in 

public session--with the Agency on a technical matter before the Agency has begun 

substantive work on that issue. The goal is to leaven EPA's thinking on an issue by 

brainstorming a variety of approaches to the problem very early in the development 

process. There is no attempt or intent to express an SAB consensus or to generate an 

SAB report. The Board, via a brief letter simply notifies the Administrator that a 

Consultation has taken place. 

More recently, the Board introduced a new vehicle for communicating with its 

clients -- the "Advisory” -- which provides, via a formal SAB consensus report, critical 

input on technical issues that arise during the Agency's issue development process. The 

Advisory generally involves a review of a multi-year Agency project. The intent is to 

provide some mid-course assessment to see if the Agency is heading in a scientifically 

credible direction. In order to maintain an objective, arms-length relation with the 

Agency and its projects, the SAB review of the final product at some point in the future 

will include experts who did not participate in producing the Advisory. 

The Agenda Setting Process 

Each Assistant Administrator and Regional Administrator is normally asked to submit a 

list of candidate topics for SAB action/review. A "project sheet" (see attached example) is used 

to define each topic that is nominated for SAB review. The project sheet is prepared by the 

requesting office and contains the following information: 

1) Project title/subject (Descriptive short title of project) 

2) Requesting Organization/Office (Primary office requesting review, AA/RA level) 

3)	 Requesting Official (Name and position of senior official requesting review, 

usually office or division level - this is the person who may receive a summary 

briefing from the Chair following the review) 

4) 	 Program Contact (Name/phone number/mail code - this is the principal contact 

for SAB Staff to interact with during development of the SAB review) 
Page D-4 Peer Review Handbook 

5) Background (brief history of the project and why it is important) 

6)	 Tentative Charge (what the SAB is being asked to comment on - usually a set of 

questions) 

7)	 Tentative Schedule and Committee (when the review is expected to be conducted, 

e.g., Winter 2001; and which SAB committee is appropriate for the review -- final 

choice as to the review committee is at the discretion of the Board) 

8)	 Budget Estimate (Rough estimates of Agency funding for the subject over the 

past 5 years (if applicable) and for the next 2 years (if applicable). The SAB 

Executive Committee has asked for this information to help it better appreciate 

the level of Agency involvement in and commitment to the issue) 

9) Preparer (name, phone, office of preparer of Project Sheet and the date prepared) 

The proposed topics will be examined and discussed in a number of forums: 

1)	 The individual SAB Committees - Throughout the late Spring and Summer, the 

SAB Committees will be examining options for the following fiscal year, 

including all suggestions made by the Agency. 

2)	 The Science Policy Council-Steering Committee (SPC-SC) - The SPC-SC usually 

meets in early summer to examine the proposals for each fiscal year. The goal is 

to provide cross-office critique/integration of the proposals. The SPC-SC will be 

used as a forum for continuing discussion throughout the process and throughout 

the year as new topics emerge. 

3)	 The Deputy's perspective - In the summer, the Deputy Administrator will review 

the requests and provide insights on priorities. 

4)	 The SAB Executive Committee - During its summer meeting, the SAB’s EC will 

examine the nominated topics, adding its own perspective on an appropriate 

agenda, using its selection criteria. 

5)	 The Administrator - In September, the list of proposed topics will be delivered to 

the Administrator for information and added insights. 

The completed project sheets should be submitted electronically to the SAB Deputy Staff 

Director (fowle.jack@epa.gov) and in signed hard copy (mail code 1400A). 
Peer Review Handbook Page D-5 

For general information, or to contact a specific SAB Staff member, please call the SAB 

main phone line: (202) 564-4533. Please ask to be connected to the Staff person handling the 

major review area you wish to inquire about (e.g., human health and exposure, drinking 

water/water quality, ecology, engineering, modeling, air quality, economics, research, and 

radiation). 
Page D-6 Peer Review Handbook 

US EPA Science Advisory Board 

Proposed Project 

Project title/subject: Proposed Amendments to the Risk Assessment Guidelines for 

Carcinogens 

Requesting Organization/Office: Office of Research and Development (ORD) 

Requesting Official: Name, Title, Office/Organization 

Program Contact: Name, Title, Office/Organization, 202-260-xxxx 

Background: EPA's Health Risk Assessment Guidelines provide generic science and science 

policy guidance on risk assessment issues for use in all Agency offices. EPA has currently 

issued or proposed nine guidelines (or amendments) in this series, all of which have been 

submitted to the Science Advisory Board for review. 

The current guidelines for carcinogen risk assessment were reviewed by the SAB and issued as 

final guidance in 1986. In 1988, the Forum initiated a public process for considering 

amendments to these guidelines. A Risk Assessment Technical Panel considered submissions 

from the public as well as information developed by experts at two public workshops in revising 

these guidelines. 

Tentative Charge: Review the amended and expanded guidance, with special emphasis on (a) 

weight-of-evidence issues, (b) a new classification system, (c) dose response modeling, and (d) 

the use of pharmacokinetic and metabolic data. A more detailed charge will be negotiated with 

SAB at a later date. 

Tentative Schedule and Committee: Winter, 2001, Environmental Health Committee 

Budget:	 FY 1997 - $xxx and yy FTE 

FY 1998 - $xxx and yy FTE 

FY 1999 - $xxx and yy FTE 

FY 2000 - $xxx and yy FTE 

FY 2001 - estimated costs of $xxx and yy FTE each year 

Preparer: Name, Title, Office/Organization, 202-260-xxxx 

Date: June 1, 2000 
Peer Review Handbook Page E-1 

APPENDIX E - EXAMPLE STATEMENTS OF


WORK FOR CONTRACTS


Users of this Peer Review Handbook need to be aware that the examples 

contained in this Appendix are generalized statements of work prepared (and in 

some cases modified) to emphasize certain important features (e.g., attention to 

conflicts of interest, responsibilities of contractors or contracting officers, 

development and use of the charge to peer reviewers). Please be sure that proper 

and currently approved contract language is used in any EPA contract document 

at the time of award (for example, see EPAAR 1552.212-71, alternate I). 
Page E-2 Peer Review Handbook 

STATEMENT OF WORK - EXAMPLE 1 - Statement of Work: Technical Review 

Contractor for Panel Review of Assistance Agreement or Fellowship Applications 

1) Purpose 

The purpose of this contract is to purchase peer review services of a contractor with 

expertise in Exploratory Research; Environmental Chemistry. The services are for peer 

reviewing applications received in response to the Office of Research and Development's 

(ORD's) 1997 Science to Achieve Results program. These reviews shall be completed and the 

evaluation sheets shall be prepared prior to the reviewer's participation in a 3-day panel 

discussion to be held in Washington, D.C. on May 5 -7, 1997. 

2) Statement of Work 

ORD's National Center for Environmental Research and Quality Assurance (NCERQA) 

is responsible for overseeing the recently expanded research grants and fellowships programs. 

Each year NCERQA (alone or in conjunction with other organizations) solicits applications in 

each of these programs. The applications to be reviewed under this contract were submitted in 

response to the solicitation for the 1997 Science to Achieve Results program. As part of the 

selection process, NCERQA must conduct a peer review that is designed to evaluate the 

scientific quality of each application; this is accomplished through the ad hoc use of technical 

experts. 

The peer review services required by this contract necessitate the independent review of a 

maximum of 10 applications and the preparation of a typed evaluation summary and an overall 

rating for each of these applications. Each evaluation summary shall support and be consistent 

with the overall rating that is assigned; it also shall be completed prior to the contractor’s 

participation in the panel discussions. After the panel discussions for the applications assigned 

to the contractor, the contractor shall submit all completed evaluation summaries to the 

designated Science Review Administrator (SRA). 

The contractor also shall serve as the panel's rapporteur for approximately 6 - 8 of the 

applications assigned. As rapporteur, the contractor shall be responsible for preparing a typed 

evaluation summary (on-site typing support will be provided by NCERQA) that reflects the 

panel's discussion of the respective application as well as the panel's overall rating (the criteria 

for the panel's overall rating are the same as those for each peer reviewer's overall rating). As 

rapporteur, the contractor shall submit these panel evaluation summaries to the designated SRA 

prior to leaving the panel meeting. 
Peer Review Handbook Page E-3 

Before the contractor shall be allowed to participate in the review process, the contractor 

shall have disclosed any actual or potential conflicts of interest and shall have signed and 

submitted to NCERQA a Conflict of Interest/Confidentiality Form. The contractor is directed to 

assure that none of the conflicts disclosed are so direct and substantial as to rule out a particular 

reviewer. Upon receipt of an approved Purchase Order (PO), NCERQA will send the following 

items to the contractor: 

a) A copy of the Purchase Order or the Purchase Order number 

b) The applications assigned to the contractor 

c) For grants, a set of abstracts for all the applications being reviewed by the panel 

d)	 For persons reviewing grant applications, a copy of the pertinent section(s) of the 

solicitation package to provide background information; for persons reviewing 

fellowship applications, information on how to access the solicitation package on 

the Internet 

e)	 A sample evaluation form to help the contractor prepare an acceptable evaluation 

form for each assigned application 

f)	 A blank evaluation form for each assigned application and the criteria for 

completing the form and determining the overall rating 

g)	 A blank and sample invoice as well as instructions for completing and submitting 

the invoice to EPA 

h)	 Information on the points of contact for additional information (e.g., NCERQA's 

SRA) 

i) Logistics information on the location and time of the panel discussions 

NCERQA will transmit the above items under a cover letter. In this cover letter, 

NCERQA will provide additional details about each item, including (as needed) more specific 

instructions for the set of applications assigned to the reviewer. 

Each contractor shall be responsible for making his/her own travel reservations for hotel 

and transportation. 

3) Reviewer Tasks 

a) Review the assigned applications using the guidance provided with NCERQA's 

evaluation form. 

b) Submit completed evaluation forms to the SRA designated in the cover letter 

immediately following the panel discussions for the applications assigned to the contractor. 

THE COMPLETED FORMS MUST BE TYPED, AND THE EVALUATION SUMMARY 

FOR EACH APPLICATION MUST SUPPORT AND BE CONSISTENT WITH THE 

OVERALL RATING THAT IS ASSIGNED BY THE CONTRACTOR. IN SITUATIONS 
Page E-4 Peer Review Handbook 

WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE 

CONTRACTOR TO REDO THE FORM. 

c) For those applications for which the contractor is serving as the panel's 

rapporteur, submit a completed panel evaluation summary to the SRA designated in the cover 

letter prior to leaving the panel meeting. THE PANEL'S EVALUATION SUMMARY 

MUST BE TYPED (ON-SITE TYPING SUPPORT WILL BE PROVIDED BY NCERQA) 

AND BE CONSISTENT WITH THE PANEL'S OVERALL RATING. IN SITUATIONS 

WHERE THESE CONDITIONS ARE NOT MET, THE SRA WILL ASK THE 

CONTRACTOR TO REDO THE FORM. 

d) Make own airline and hotel accommodations for participation in the panel review 

meeting. Round-trip air fare must be a commercial REFUNDABLE ticket. 
Peer Review Handbook Page E-5 

STATEMENT OF WORK - EXAMPLE 2 - Peer Review of Prioritization Tool Report 

Work Assignment No.: 

Title: Peer Review of Prioritization Tool Report 

Work Assignment Manager (WAM): 

Name: John Q. Government Employee


Address: Office of Solid Waste


Phone No.: (202) 260-XXXX


Background: 

The Waste Minimization Branch (WMB) in the Office of Solid Waste (OSW) is in the 

process of implementing the Waste Minimization National Plan, announced by the Agency on 

November 18, 1994. The Plan reaffirms the Agency's commitment to promote source reduction 

over waste management, in keeping with the policy stated in the 1984 amendments to the 

Resource, Conservation, and Recovery Act (RCRA) and in the 1990 Pollution Prevention Act 

(PPA). The Plan outlines major goals, objectives, and action items to achieve national 

reductions in the generation of hazardous wastes. 

One of the objectives of the Plan is to: "develop a framework for setting national 

priorities; develop and distribute a flexible screening tool for identifying priorities at individual 

facilities; [and] identify constituents of concern." This objective is a key building block in 

implementing subsequent objectives of the Plan. 

In September 1995, WMB formed the Waste Minimization Prioritization Team, which 

includes representatives from EPA regions and states, to implement this objective. The Team 

has worked to assess stakeholder needs for prioritization tools and to evaluate prioritization tools 

that are currently available. The Team plans to summarize this work, along with its 

recommendations, in a report (referred to herein as the Prioritization Tool report) that would be 

available in draft form in July 1996. 

WMB and the Team wish to obtain independent peer review of the Prioritization Tool 

report prior to briefing EPA management. The report is being prepared with the support of ICF, 

Inc.; therefore, for the peer review to be considered independent, it must be performed by 

another contractor. 
Page E-6 Peer Review Handbook 

Purpose and Scope of Work: 

The purpose of this work assignment is to provide support to WMB and the Team in 

finalizing the Prioritization Tool report by conducting an independent peer review of the report. 

Work Statement: 

Task 1 -Management work plan and budget 

Within 15 days of CO approval of this work assignment, the contractor shall deliver a 

management work plan including a proposed level of effort, schedule, and budget for all tasks. 

Task 2 -Provide independent peer review of Prioritization Tool report 

The contractor shall provide support to WMB and the Team in preparing the 

Prioritization Tool report by performing an independent peer review of the report. The 

contractor shall establish a panel of peer reviewers including three senior-level persons who 

collectively have extensive expertise in particular areas to be identified by the WAM upon 

approval of the work assignment. 

Within three weeks of work assignment approval, receipt from the WAM of the 

necessary qualifications of peer reviewers (in a TD), and receipt from the WAM of the peer 

review “charge” (in a TD), whichever comes latest, the contractor shall identify the three peer 

reviewers and prepare a memo that lists the names of the peer reviewers and their affiliations and 

includes the peer reviewers’ bio’s. Within five weeks of WAM approval of the of the peer 

reviewers (via a TD) and receipt of the draft Prioritization Tool report from the WAM (via a 

TD), whichever comes later, the contractor shall conduct the peer review, assemble the peer 

review comments and recommendations in a peer review report organized by charge question, 

prepare an introduction to the peer review report with a clear and concise overview of the 

comments, and attach to the peer review report any marginal comments the peer reviewers had 

on the Prioritization Tool report. 

It is not necessary that the peer reviewers jointly reach consensus on their findings and 

recommendations, since there may be limited overlap in the peer reviewers' areas of expertise 

and in the charge questions that they focus on. The contractor shall assume, for the purpose of 

estimating costs, that the draft Prioritization Tool report is roughly 100 pages in length with 200 

pages of appendices, and that each peer reviewer will spend 40 hours in reviewing the report and 

writing comments. EPA plans to provide the report to the contractor in mid-July. 
Peer Review Handbook Page E-7 

Deliverables and Schedule: 

Task Deliverable Schedule 

1 Work plan and budget Within 15 days of CO ap proval of work 

assignment 

2 Memo identifying peer reviewers Within 3 weeks of wo rk assignment approval, 

receipt of peer reviewer qualifications from 

WAM, and receipt of charge from WAM, 

whichever comes latest 

3 Peer review report Within five weeks of WAM approval of peer 

reviewers and receipt of draft Prioritization 

Tool report from WAM, whichever comes 

later 

Other Requirements: 

CONTRACTOR COMMUNICATIONS 

Upon approval of the Work Plan, the contractor shall maintain at least weekly 

communications with the Work Assignment Manager regarding the status of work on the Work 

Assignment. 

CONFLICT OF INTEREST (COI) [Revised Section] 

The contractor must adhere to the following requirements: 

a)	 Upon receipt of a Work Assignment, QRT, or similar tasking document, and prior 

to commencement of any work, notify both the CO and PO of any actual or 

potential organizational or personal conflicts of interest. 

b)	 Provide a written certification, within 20 days of receipt of a Work Assignment, 

QRT, or similar tasking document, that: 

1)	 Either all conflicts of interest have been reported to the CO or that no 

conflicts of interest exist. The contractor is directed to assure that none of 

the conflicts disclosed are so direct and substantial as to rule out a 

particular reviewer. 

2)	 All personnel who perform work under this Work Assignment or relating 

to this Work Assignment have been informed of their obligation to report 

personal and organizational conflicts of interest to the CO. 
Page E-8 Peer Review Handbook 

3)	 The Contractor recognizes its continuing obligation to identify and report 

any conflicts of interest arising during performance of this Work 

Assignment. 

c)	 If a conflict of interest is identified during performance under this Work 

Assignment, the Contractor shall immediately make a full disclosure in writing to 

the CO. The disclosure shall include a description of action which the Contractor 

has taken or proposes to take, after consultation with the CO, to avoid, mitigate, 

or neutralize the conflict of interest. 

d)	 After selecting the peer reviewers but before starting the peer review, submit to 

EPA documentation that shows that the contractor has determined if the peer 

review candidates: 

1)	 have a conflict of interest or a situation that could create the appearance of 

a lack of impartiality in relation to the work product 

2) have had or presently have a financial relationship with EPA 

and summarizes for EPA its efforts to identify and propose resolution of these 

concerns with peer review candidates. 

EXPENDITURE OF FUNDS/HOURS 

In addition to the requirements of the contract, the contractor shall notify both the Project 

Officer and the Work Assignment Manager when 75% of funds or hours for this Work 

Assignment have been expended. 

INFORMATION COLLECTION 

Any other provision of this Work Assignment notwithstanding, the contractor shall not 

proceed with any information collection where the same or similar information will be collected 

from ten or more public respondents until written approval is received from the Contracting 

Officer. This approval will cite an approval number from the Office of Management and Budget 

as required by the Paperwork Reduction Act (PRA). 

Only Federal agencies and their employees are exempt from the PRA definition of 

"public respondent." State agencies and their employees are classified as "public respondents." 

Soliciting similar information applies to any collection method, i.e., written, oral, 

electronic, etc., and utilizing any approach, i.e., surveys, phone calls, focus groups, TQM, etc. 

The PRA applies equally to "willing participants" and participation that is mandated by law. 
Peer Review Handbook Page E-9 

Any question of applicability of the PRA shall be resolved by submitting a complete 

description of the circumstances in a written request to the Contracting Officer. No collection 

shall be undertaken until the Contracting Officer provides written notice to the Contractor as to 

the applicability of the PRA. If the PRA is determined to be applicable, the Contractor shall not 

initiate any collection until the requisite approval is received. 

The General Services Administration (GSA), under FIRMR Bulletin B-2 administers the 

Interagency Reports Management Program as derived from 44 U.S.C. Chapters 29 and 31. All 

work performed under this Work Assignment involving federal interagency reporting must be 

done in full compliance with these GSA procedures. 

CONFIDENTIAL BUSINESS INFORMATION 

If this Work Assignment requires use of RCRA Confidential Business Information 

(CBI), the contract must specifically authorize the contractor to have access to RCRA CBI and 

the contractor shall abide by all RCRA CBI requirements and stipulations found in the RCRA 

CBI Security Manual and in the contract. The contractor shall identify in the Work Plan budget 

all estimated costs for dealing with CBI requirements. All CBI must be returned to EPA as soon 

as it is no longer needed under this Work Assignment or before the expiration of the Work 

Assignment, whichever occurs first. 

PRINTING AND DUPLICATION 

The contractor is prohibited from performing any printing under the Government Printing 

and Binding Regulations. Duplication is allowed to the extent it does not exceed 5,000 

impressions of a single-page document or 25,000 impressions of a multiple-page stand-alone 

document, is limited to one color (black) copies, and does not exceed the maximum image size 

of 10 3/4 by 14 1/4 inches. For all duplication jobs in excess of 5,000 impressions, the EPA 

WAM will determine in advance if the work can be performed more cost effectively and under 

the job or time constraints at the EPA Print Shop. If the total number of photocopies for this 

Work Assignment exceeds 5,000 impressions, the contractor shall identify in their Work Plan 

the photocopying costs by task and deliverable. 

WORK ASSIGNMENT/WORK PLAN BUDGETS 

The contractor shall not exceed either the dollar or PL hour budget contained in the 

approved Work Plan. In addition, on Quick Response Tasks (QRTs) the contractor shall not 

exceed the PL hour budget of the QRT. 
Page E-10 Peer Review Handbook 

TECHNICAL DIRECTION 

The Designated Work Assignment Manager (WAM) on this Work Assignment is 

authorized to provide technical direction to the extent allowed under EPAAR (1552.237-71) 

(APR 1984) (DEVIATION). Other than the Designated WAM, only the Project Officer and the 

Contracting Officer are authorized to provide technical direction. 

Technical direction includes: 

(1) Direction to the contractor which assists the contractor in accomplishing the 

Statement of Work 

(2) Comments on and approval/acceptance of reports or other deliverables 

Technical direction must be within the contract and the Work Assignment statement of 

work. The Project Officer and the WAM do not have the authority to issue technical direction 

which (1) institutes additional work outside the scope of either the contract or this Work 

Assignment; (2) constitutes a change as defined in the "Changes" clause; (3) causes an increase 

or decrease in the estimated cost of the contract or Work Assignment; (4) alters the period of 

performance or deliverable due dates; or (5) changes any of the other express terms or conditions 

of the contract or Work Assignment. 

Technical direction will be issued in writing or confirmed in writing within five (5) 

calendar days after verbal issuance. The technical direction memorandum will be provided to 

the contractor and copies will be forwarded to the Contracting Officer and the Project Officer. If 

the contractor has not received written confirmation within five (5) calendar days of a oral 

issuance, the contractor must so notify the Project Officer. 

INHERENTLY GOVERNMENTAL FUNCTIONS 

The contractor shall not perform any inherently governmental functions (IGF) under this 

Work Assignment. If during the course of developing the plan of work, through receipt of 

technical direction, or in carrying out the assignment any portion of the effort is considered to 

possibly be an inherently governmental function, the contractor must immediately notify the 

Project Officer and the Contracting Officer. 

OCCUPATIONAL HEALTH AND SAFETY 

Facility site visits conducted under a Work Assignment that include on-site inspections 

or sampling must be conducted in full compliance with the Department of Labor, Occupational 

Safety, and Health Administration rules under 29 CFR Part 1910 and EPA Order 1440 

(Occupational Health and Safety Manual). 
Peer Review Handbook Page E-11 

TRAVEL COSTS 

The contractor shall follow the requirements of Subpart 31.2 of the FAR and the Federal 

regulations in incurring allowable travel costs under this Work Assignment, and correspondingly 

must at all times seek and obtain Government rates whenever available and observe current 

subsistence ceilings. 

QUICK RESPONSE TASKS 

Each Quick Response Task (QRT) shall be confirmed in writing and approved by the 

Project Officer. The contractor shall respond by letter to the PO with copies to the WAM and 

the CO within two working days, giving a brief description of the plan of work, including best 

estimate of hours (by P-level) and a break-out of costs to accomplish the task. 

No task shall exceed a duration of 30 calendar days from start date to completion date. 

The level of effort for each task shall be limited to a maximum of 250 labor hours. 

Quick Response Task Requests do not change the dollar or professional labor hour 

budgets of a Work Assignment. 
Page E-12 Peer Review Handbook 

STATEMENT OF WORK - EXAMPLE 3 - External Peer Review of Protozoa Method 

Development Criteria Document 

Period of Performance: Work Plan Approval to August 1, 1997 

Work Assignment Manager:	 Sally Q. Government Employee 

Office of Water 

U.S. Environmental Protection Agency 

LOE: 196 hours 

SOW: 2.4 

BACKGROUND INFORMATION: 

The United States Environmental Protection Agency (EPA), Office of Water is charged with 

protecting public health and the environment from adverse exposure to chemicals and microbials 

in water media, such as ambient and drinking waters, wastewater/sewage sludge and sediments. 

In support of this mission OW’s Office of Science and Technology (OST) develops health 

standards, health criteria, health advisories, and technical guidance documents for water and 

water-related media. Under this work assignment, documents prepared by OST are to undergo 

peer review. 

Peer review is an important component of the scientific process. It provides a focused, objective 

evaluation of a research proposal, publication, risk assessment, health advisory, guidance or 

other document submitted for review. The criticism, suggestions and new ideas provided by the 

peer reviewers stimulate creative thought, strengthens the reviewed document and confer 

credibility on the product. Comprehensive, objective peer reviews leads to good science and 

product acceptance within the scientific community. 

Under this work assignment, the contractor will receive one document (Protozoa Method 

Development Criteria Document) for peer review which is related to human health and 

ecological effects. 

STATEMENT OF WORK: 

Task 1.	 The contractor shall develop a work plan to address all tasks in this work 

assignment. The work plan shall describe the steps that will be taken by the 

contractor to provide for peer review, including selection of peer reviewer 

candidates with appropriate expertise, determining absence of conflict of interest, 

document and reference distribution, establishing schedules, preparing the peer 

review report, and submittal of the peer review package. Curriculum vitae for all 

persons assigned to complete this work assignment shall be provided. All P 
Peer Review Handbook Page E-13 

levels, hours and total costs for each task will be provided and costs greater than 

$100.00 shall be itemized in detail. 

Task 2.	 The contractor shall select a group of peer reviewers and determine their 

availability for the task and absence of conflict of interest, and establish a 

schedule for the peer review. The contractor is directed to assure that none of the 

conflicts disclosed are so direct and substantial as to rule out a particular 

reviewer. Three peer reviewers shall participate in the review. No single peer 

reviewer may charge more than 40 hours to this task. It is fully acceptable for 

peer reviewers to commit to less than 40 hours. The peer review will be 

conducted for the Protozoa Method Development Criteria Document. Reviewers 

selected by and working for the contractor shall be approved by the EPA Project 

Officer in writing prior to their beginning work. Minimally, all peer reviewers 

shall be accomplished in protozoan methods for sample recovery and analysis 

from water. Approval submissions shall include the reviewers’ names and 

curriculum vitae. 

Task 3.	 The contractor shall arrange for the selected peer reviewers to review the EPA 

document. Prepare the charge to the peer reviewers based on technical direction 

received from the EPA WAM. Provide the peer reviewers with copies of the 

candidate report and all relevant references and instruct the selected peer 

reviewers to undertake the review. The WAM will provide the contractor with 

the final version of the document to be reviewed. 

Task 4.	 The contractor shall monitor peer reviewers’ progress to assure timely 

completion. The contractor shall collate peer review comments, and organize the 

comments in the peer review “for comments” document. Provide the peer review 

document and all materials submitted by the peer reviewers to the EPA WAM. 

SCHEDULE AND DELIVERABLES: 

Task 1. (Work Plan) 15 days after receipt of work assignment 

Task 2. 1 week after work plan approval 

Task 3. 1 week after selection of peer reviewers 

Task 4. 1 week after receiving comments from the peer reviewers 

TRAVEL: No travel is anticipated under this work assignment. Any travel directly chargeable 

to this work assignment must be submitted and approved by the project officer. 

Peer Review Handbook Page F-1 

APPENDIX F - USEFUL FORMS


Peer Review Conflict of Interest Inquiry 

Peer Review Checklist for Determining 

Whether a Work Product Needs Peer Review 

(Template Provided by Region V) 

Peer Review Checklist for Conducting a Peer Review 

(Template Provided by Region V) 

Volunteer Service Program Participation Agreement 

(EPA Form 3100-14) 
Page F-2 Peer Review Handbook 

SUBJECT: PEER REVIEW CONFLICT OF INTEREST INQUIRY 

Dear (Peer Reviewer -- insert name): 

You have been requested by EPA to serve as a Peer Reviewer for______ 

__________(name of project)____________. Your participation in this review will be greatly 

appreciated. However, it is possible that your personal affiliations and involvement in particular 

activities could pose a conflict of interest or create the appearance that you lack impartiality in 

your involvement for this peer review. Although your involvement in these activities is not 

necessarily grounds for exclusion from the peer review, you should consult the contact named 

below or other appropriate official to discuss these matters. Affiliations or activities that could 

potentially lead to conflicts of interest might include: 

a)	 work or arrangements concerning future work in support of industries or other 

parties that could potentially be affected by regulatory developments or other 

actions based on material presented in the document (or review materials) that 

you have been asked to review; 

b)	 your personal benefit (or benefit of your employer, spouse or dependent child) 

from the developments or other actions based on the document (or review 

materials) you have been asked to review; 

c)	 any previous involvement you have had with the development of the document 

(or review materials) you have been asked to review; 

d)	 any financial interest held by you (or your employer, spouse or dependent child) 

that could be affected by your participation in this matter; and 

e)	 any financial relationship you have or have had with EPA such as research grants 

or cooperative agreements. 

Please contact ___________________ (name and contact info for Agency peer review 

official or primary contractor) to discuss any potential conflict of interest issues at your earliest 

convenience, but no later than _________. 

[Be sure to date and sign this inquiry] 
Peer Review Handbook Page F-3 

PEER REVIEW CHECKLIST FOR DETERMINING 

WHETHER A WORK PRODUCT NEEDS PEER REVIEW 

Instructions: This checklist is based on the Agency’s Peer Review Handbook and the October 2000 

Region 5 Order “U.S. EPA Region 5 Improved Policies and Procedures: Peer Review, Records 

Management, and Work Product Authorization of Scientific and Technical Work Products” which 

constitute Region 5’s standard operating procedures for peer review. If you have any questions about 

peer review or need clarification when completing this checklist, please refer to the Handbook, available 

via the internet at http://www.epa.gov/ord/spc/2peerrev.htm. Figure 1 on page 2 of the Handbook 

includes a useful flow chart and cross references to specific sections of the Handbook that are

applicable to this checklist. You are also encouraged to consult with your Division or Office Peer 

Review Coordinator. The Division/Office Peer Review Coordinators will periodically request

information from this checklist in order to update the National Peer Review Database. 

1. Title of Work Product: 

2. Product Description: 

3. Project Manager: 

Name, Organization and Phone Number 

Please circle the 

4. Determination if Work Product is Scientific or Technical: appropriate answer 

a) Is the work product a scientific, engineering, economic, social yes no 

science, or statistical document? (Examples of such documents 

include: risk assessments, technical studies and guidance, 

analytical methods, scientific database designs, technical models, 

technical protocols, statistical surveys/studies, technical 

background materials, and research plans and strategies.) 

b) Is the work product a scientific or technical document resulting yes no 

from a grant, contract or cooperative agreement? 

c) Will the work product be used to support a research agenda, yes no 

regulatory program, policy position, or other Agency 

position or action? 

If you answered “no” to all of these questions, your work product is not subject to EPA’s 

peer review policy for scientific or technical work products and does not need to be placed 

on any of the peer review lists. Please proceed to #7 of this checklist. If you answered 

“yes” to any of these questions, your work product might need peer review; please 

continue on to #5 of this checklist. 
Page F-4 Peer Review Handbook 

5.	 Determination if Work Product is a Major Work Product: 

Determination of whether a work product is “major” will largely be on a case-by-case 

basis. As the continuum of work products covers the range from the obviously major to 

those products that clearly don’t need peer review, there is no one single, easy yes/no 

answer to the test of “major” (see Handbook, Section 2.2.3). There also is no single 

definition of “significant.” Determination of “major” and “significant” are the 

responsibility of the Division or Office Director who is the official Decision Maker. 

Please circle the 

appropriate answer 

a) Does the work product establish a significant precedent, yes no 

model, or methodology? 

b) Does the work product address significant controversial	 yes no 

issues? 

c) Does the work product focus on significant emerging or	 yes no 

“cutting edge” issues? 

d) Does the work product have significant cross-Agency or yes no 

inter-agency implications? 

e) Does the work product involve a significant investment of	 yes no 

agency resources? 

f) Does the work product consider an innovative approach or	 yes no 

application for a previously defined problem, process or 

methodology? 

g) Is the work product required to be peer reviewed by statute yes no 

or other legal mandate? 

h) Does the work product support a regulatory decision,	 yes no 

policy or guidance of major impact? (Major impact can mean 

that it will have applicability to a broad spectrum of regulated 

entities and other stakeholders, or that it will have narrower 

applicability, but with significant consequences on a smaller 

geographic or practical scale.) 

i) Is the work product an application of or modification to 	 yes no 

an existing, adequately peer reviewed methodology or model 

that departs significantly from the situation it was originally 

designed to address? 

If you answered “yes” to any of these questions, your work product needs peer review 

unless special circumstances exist; please continue on to #6. If you answered “no” to all of 

these questions, your work product probably does not need peer review. However, peer 

review can always be done to improve the quality of the work product. Please proceed to 

#7 of this checklist. 
Peer Review Handbook Page F-5 

6.	 Determination Whether Circumstances Exist Where a Major Work Product 

Would Not Be Peer Reviewed: 

Please circle the 

appropriate answer 

a) Was the work product previously reviewed by recognized yes no 

experts or an expert body? (Note: Peer review of an EPA work 

product by a recognized refereed journal strengthens the 

scientific credibility of the work product but does not eliminate 

the need to have the work product itself peer reviewed for issues 

and concerns to support an Agency action. See Sections 2.4.4 and 

2.4.5 of the Handbook for more details.) 

b) Are the scientific or technical methodologies or information yes no 

being used commonly accepted in the field of expertise? 

c) Has the regulatory activity or action which the work product yes no 

supports been terminated or canceled? 

d) Is there a statutory or court ordered deadline, or a time yes no 

constraint which may limit or preclude peer review of the 

work product? 

If you answered “yes” to any of these questions, your work product probably does not 

require peer review. This decision with the justification needs to be concurred with and 

signed off by the Division/Office Director. The decision with the justification must be 

retained in the peer review files and noted in Peer Review Work Product List C in the 

National Peer Review Database. Continue on to #7. If you answered “no” to all of these 

questions, proceed to #8. 

7. Next Steps For Work Products That Will Not Be Peer Reviewed: 

a) Division/Office Director concurs with the decision that the work 

product should not be peer reviewed. 

Signature of Division/Office Director and Date Signed 

b) A copy of this completed checklist has been given to the Div/Off 

Peer Review Coordinator and put in the official peer review files in 

the Division/Office. 

Signature of Div/Off Peer Review Coordinator and Date Signed 

Location of Div/Off Peer Review Files 
Page F-6 Peer Review Handbook 

c) Work product has been placed on Peer Review Work Product List C 

in the National Peer Review Database. (Note: This only applies to those 

work products subject to the peer review policy.) 

Signature of Div/Off Peer Review Coordinator and Date Signed 

If all of the necessary information is complete, you are done. You don’t need to proceed 

any further with this checklist. 

8. Next Steps For Work Products That Will Be Peer Reviewed: 

a) Division/Office Director has been consulted and concurs with the decision that 

the product should be peer reviewed. 

Signature of Division/Office Director and Date Signed 

b) A copy of this completed checklist has been given to the Division/Office 

Peer Review Coordinator and put in the official peer review files in the 

Division/Office. 

Signature of Div/Off Peer Review Coordinator and Date Signed 

Location of Div/Off Peer Review Files 

c) Work product has been placed on Peer Review Work Product List B 

in the National Peer Review Database. 

Signature of Div/Off Peer Review Coordinator and Date Signed 

Because your work product will be peer reviewed, you need to complete a second checklist 

entitled “Peer Review Checklist for Conducting a Peer Review.” 
Peer Review Handbook Page F-7 

PEER REVIEW CHECKLIST FOR CONDUCTING A PEER REVIEW 

Instructions: This checklist is based on the Agency’s Peer Review Handbook and the October 2000 

Region 5 Order “U.S. EPA Region 5 Improved Policies and Procedures: Peer Review, Records 

Management, and Work Product Authorization of Scientific and Technical Work Products” which 

constitute Region 5’s standard operating procedures for peer review. If you have any questions about 

peer review or need clarification when completing this checklist, please refer to the Handbook, available 

via the internet at http://www.epa.gov/ord/spc/2peerrev.htm. Pages 2-4 of the Handbook contain useful 

flowcharts and cross references to specific sections of the Handbook that are applicable to this 

checklist. You are also encouraged to consult with your Division or Office Peer Review Coordinator. 

The Division/Office Peer Review Coordinators will periodically request information from this checklist 

in order to update the National Peer Review Database. 

1. Title of Work Product: 

2. Product Description: 

3.	 Project Manager: 

Name, Organization and Phone Number 

Check the box when 

4. Up-front Considerations for Planning the Peer Review: item is completed 

a) The Div/Office Director has chosen a peer review leader for the project. 

(Note: The project manager and peer review leader can be the same person.) 

Name of Peer Review Leader: 

Phone Number: 

Organization: 

b) The peer review leader has obtained appropriate peer review training 

before conducting the peer review. 

c) Key questions and issues have been identified to include in the charge 

to the peer reviewers. 

d) The Div/Office Records Coordinator has been consulted to insure 

that all the files, including electronic records, will be created, maintained, 

retained, and disposed of appropriately and in accordance with 

Div/Office and Agency procedures. 

e) A formal peer review record or file has been established, and provisions 

have been made to store any electronic records associated with 

the work product and peer review. 

Location of Record/File: 

Provisions for Electronic Records: 
Page F-8 Peer Review Handbook 

Check the bo x when item is 

completed or circle the 

appropriate answer 

(NA = not applicable) 

f)	 There is a source of adequate funding to pay for external peer review if 

external peer review is necessary and funding is needed. (Note: Contracts NA 

can be used for peer review services. However, special management controls 

are required to ensure proper use of these contracts. See Section 3.6 of the 

Handbook for details.) 

Source of Funding: 

g) Resource limitations may restrict the peer review. (If “yes” was Yes No 

selected, a limited peer review might be considered. However, only in 

very rare circumstances should resource limitations restrict peer review. 

Peer review must be planned for as part of a project’s budget.) 

h) Amount of time needed for peer review(s) has been allotted given 

existing constraints of potential peer reviewers, deadline for the final 

work product, logistics for the peer review, etc. 

Length of Time Needed: 

5. Develop the Charge to the Peer Reviewers: 

a) A clear, focused charge has been formulated that identifies recognized 

issues, asks specific questions, and invites comments or assistance. 

b) The charge has been included in the peer review record. 

6. Select the Peer Review Mechanism: 

a) The work product is novel, complex, controversial, or has great Yes No 

cost implications. (If the answer is “yes” to any of the above, serious 

thought should be given to conducting an external peer review. If the 

answer is “no” to all of the above, internal peer review is probably sufficient.) 

b) A determination has been made regarding which components 

or stages of the work product will be peer reviewed. (Note: Generally, 

peer review is recommended for each stage of a product’s development.) 

Components to be peer reviewed: 

c) A peer review mechanism (e.g., internal, external or a combination 

of both) has been chosen for the work product or stages of the work 

product. Mechanism: 

d) The work product either: 1) has been, or is being, generated as part Yes No 

of administrative or civil enforcement activities by U.S. EPA, or 

2) likely will be used in the future to support administrative or civil 

enforcement activities by U.S. EPA. (If the answer is “yes” to either 

item above, then the Office of Regional Council (ORC) must be consulted 

if the Peer Review Leader believes an external peer review is needed or is 

preferable. ORC concurrence should be obtained.) 
Peer Review Handbook Page F-9 

Check the box when 

item is com pleted, 

or circle yes or no 

e) The work product is going to be peer reviewed via a refereed, Yes No 

scientific journal. (If the answer is “yes,” the work product still 

should be considered for peer review because journal peer review 

may not cover issues and concerns that the Agency would want peer 

reviewed in order to support an Agency action.) 

f)	 Logistics for conducting the peer review (e.g., written comments will 

be received by mail, or will be collected at a meeting) have been 

included in the peer review record. 

g) The Div/Off Director has concurred with the recommended method 

of peer review. 

Date of Div/Off Director Concurrence: 

h) The concurrence of the Div/Off Director has been included in the 

peer review record. 

7. Determine the Specific Time Line for the Peer Review: 

a) A start date for the peer review has been selected. 

Start Date: 

b) The amount of time the peer reviewers will be given to conduct 

the peer review has been determined. 

Number of Days for Review: 

c) A due date for comments from the reviewers has been selected. 

Due Date: 

d) The amount of time necessary to incorporate comments from the 

peer reviewers into the work product has been determined. 

Number of Days for Revision: 

e) A deadline for final completion of the work product has been 

determined. 

Due Date: 

8. Select the Peer Reviewers: 

a) Advice was sought in developing a list of potential peer reviewer 

candidates who are independent of the work product and have 

appropriate scientific and technical expertise. 

b) The expertise required for the peer review has been determined. 

c) In reviewing the candidates, a balance and a broad enough 

spectrum of expertise were considered. 

d) In reviewing the candidates, any potential conflicts of interest 

were considered. 
Page F-10 Peer Review Handbook 

Check the box when 

item is completed 

e) The peer reviewers have been selected and the process for 

selecting the reviewers, including inquiries and resolution 

of potential conflicts of interest, has been documented and included 

in the peer review record/file. (Note: Conflict of Interest Inquiry Forms 

are available from the Regional and Div/Off Peer Review Coordinators.) 

9. Obtain and Transmit Materials for Peer Review: 

a) Instructions have been given to the peer reviewers which ask for 

written comments in a specified format by the specified deadline 

that are responsive to the charge. 

b) The peer reviewers have been provided with the essential documents, 

data, and information to conduct their review. 

Date Peer Reviewers Given Charge/Materials: 

c) The peer reviewers have been instructed not to disclose draft work 

products to the public. 

d) The peer review record/file contains all the materials given to the 

peer reviewers. 

10. Conduct the Peer Review: 

a) Written comments have been received from all peer reviewers. 

Date all comments were received: 

b) All clarification or additional information necessary from the peer 

reviewers is received. 

c) The validity and objectivity of the comments have been evaluated. 

d) Appropriate experts/staff/managers have been consulted on the 

potential impacts of the comments on the final work product, the 

project schedule, and budget. 

e) The peer review comments have been included in the peer review 

record/file. 

11. Consider the Peer Review Comments: 

a) Decisions have been made regarding which comments are

accepted and will be incorporated into the final work product, and

which comments will not be incorporated.

b) A memo or other written record has been prepared which responds

to the peer review comments and specifies acceptance or, where

thought appropriate, rebuttal and non-acceptance.

c) The Div/Off Director has concurred with the decisions and written

record on how to incorporate the peer reviewers comments in the

work product and on which comments will not be incorporated.

Date of Div/Off Director concurrence: 
Peer Review Handbook Page F-11 

Check the box when 

item is com pleted, 

or circle yes or no 

d) The concurrence of the Div/Off Director has been included 

in the peer review record/file. 

e) The memo or written record documenting how comments were 

handled and how the work product was revised has been 

included in the peer review record/file. 

f)	 The work product has been revised to incorporate the acceptable 

comments. 

g) The peer review performed during the process of developing the 

work product has been summarized and included in the work product. 

h) It is necessary to send the revised work product back to the peer Yes No 

reviewers. (If the answer is “yes,” proceed to item #11i. If the 

answer is no, proceed to item #12.) 

i)	 Additional comments are received, evaluated, and incorporated 

into the work product, and placed in the peer review record. 

12. Consider Other Comments: 

a) Prior to finalization, the document needs additional internal Yes No 

and/or external programmatic review. (If the answer is“yes,” go to 

#12b. If the answer is “no,” proceed to #13.) 

b) Written comments by programmatic reviewers have been received. 

c) Final decisions have been made regarding which comments are 

accepted and will be incorporated into the final work product, and 

which ones will not be incorporated. 

d) A memo or other written record has been prepared which responds 

to the programmatic review comments and specifies acceptance or, 

where thought appropriate, rebuttal and non-acceptance. 

e) Div/Off Director has concurred with the decisions and written 

record on how to incorporate the programmatic comments. 

Date of Div/Off Director concurrence: 

f)	 The memo or written record has been included in the peer review 

record/file. 

g) The work product has been revised to incorporate the acceptable 

programmatic comments. 

13. Finalize Work Product and Close Out Peer Review: 

a) The work product has been completed. 

b) The Div/Off Director has approved the work product. 

Date of Div/Off Director Approval: 

c) The Div/Off Director approval has been included in the 

peer review record/file. 
Page F-12 Peer Review Handbook 

Check the box when 

item is completed, or 

circle yes or no 

d) The Div/Off Director has judged the work product to be sufficiently Yes No 

controversial, of significant enough interest to outside parties, or of 

wide enough distribution, such that it should also be authorized by the 

Regional Administrator (RA), or the Deputy RA (DRA). (If the answer 

is “yes,” proceed to #13e. If the answer is “no,” proceed to #13f.) 

e) The RA or DRA has authorized the work product. 

Date of RA or DRA Authorization: 

f)	 The final work product has been included in the peer review 

record/file. 

14. Publication and Release of Reports: 

a) The Div/Off Director has approved publication or release of the 

work product. 

b) The written approval by the Div/Off Director has been included 

in the peer review record/file. 

c) The Div/Off Director has judged the work product to be sufficiently Yes No 

controversial, of significant enough interest to outside parties or of 

wide enough distribution, such that its distribution or release 

should also be authorized by the RA or DRA. If the answer is 

“yes,” proceed to #14d. If the answer is “no,” proceed to #15. 

(Note: The Div/Off Director’s decision to elevate to the RA or DRA can 

be made concurrently with item #13d.) 

d) The RA or DRA has authorized distribution or release of the work 

product. 

Date of RA or DRA Authorization: 

15. Retention of Peer Review Files and Records: 

a) The Div/Off official procedures for administrative records and the 

Agency’s record retention schedules have been examined to 

determine how long the peer review record/file, including electronic 

records, should be retained. (Note: The required time of retention for 

final reports and supporting data varies depending upon the nature 

of the report, however, final reports which are mission related or have 

an EPA number and receive external distribution are generally 

permanent federal records.) 

b) The Div/Off Records Officer or the Regional Records Officer has been 

consulted to help determine how long the peer review record/file, 

including electronic records, should be retained. 
Peer Review Handbook Page F-13 

Check the box when 

item is completed 

c) A location for the completed peer review record/file has been 

identified, and provisions have been made to retain electronic 

records associated with the work product and peer review. 

(Note: This can be the same location and provisions as identified in #4e.) 

Location of Record/File: 

Provisions for Electronic Records: 

d) Someone has been assigned the responsibility for maintaining the 

record/file and electronic records, and ensuring that they are either 

archived or destroyed appropriately. (Note: This can be the same person 

as identified in #4a.) 

Contact Name and Phone No: 

Organization: 

16. Closeout of Checklist: 

a) Items #1-15 of checklist have been completed. 

Signature of Peer Review Leader and Date Signed 

b) A copy of the completed checklist has been given to the Div/Off 

Peer Review Coordinator. 

Signature of Div/Off Peer Review Coordinator and Date Signed 

c) The completed checklist has been included in official peer review 

record/file. 

d) The work product has been moved from Peer Review Work Product 

List B to List A in the National Peer Review Database. 

Date Product moved to List A: 
Name of Program Participant Citizen of U.S. Project Supervisor 

� Yes � No 

Proposed Organizational Agreement (Include Geographical Location) 

Proposed Arrival Date Actual Arrival Date Proposed Termination Actual Termination 

Sponsored by Educational Name and Address of Institution (if “Yes” ) 

Institution 

� Yes � No 

Describe Project (s) on Which Par ticipant Will Work, Including Scope and Anti cipated Hour s per Week 

Facilities and Equipment to be Made Available by EPA 

Degree of Supervision to be Exercised by EPA 

Assistance and Degree of Cooperation Required by other Agency Personnel by Participant 

Project Supervisor Certifies That Services to be Produced by the Program Participant are not Services Provided for Through EPA Agency 

Operations 

______________________________________________ 

Signature of Supervisor 

THE VOLUNTEER AGREES THAT: 

a. Any and all claims for compensation from the Government of the United States are waived for any services performed; 

b. The Government has a non-exclusive royalty-free license to use or reproduce and patent or copyright material which is 

developed as part of and during participation in this program; and 

c. He/she will adhere to the administrative instructions and requirements of the agency while on EPA premises. 

______________________________________________ 

Participant Signature 

PERSONNEL OFFICE SIGNATURE OF PERSONNEL OFFICER OR DE SIGNEE DATE 

CONCURRENCE 

EPA Form 3100-14 (Rev. b 30) Replaces the 7-72 edition and the 12-77 edition which was entitled “Unpaid Work-Study Program” 

Page F-14 Peer Review Handbook 

U.S. Environmental Protection Agency


VOLUNTEER SERVICE PROGRAM PARTICIPATION AGREEMENT

Peer Review Handbook Page G-1 

APPENDIX G - REFERENCES CONCERNING


PEER REVIEW


American Chemical Society and the Conservation Foundation (1985) Issues in Peer Review of 

the Scientific Basis for Regulatory Decisions, Washington, DC, November 1985. 

Browner, C. (1994) Peer Review Program, Washington, DC, Memorandum issued June 7, 1994. 

(NOTE: Attached as Appendix A to this Handbook) 

Chubin, D. (1994) Grants peer review in theory and practice, Evaluation Review 18: 20-30. 

Chubin, D. and E. Hackett (1990) Peerless Science: Peer Review and Science Policy, Albany, 

NY: State University of New York Press. 

Jasanoff, S. (1990) The Fifth Branch: Science Advisors as Policymakers, Cambridge, MA: 

Harvard University Press. 

Kostoff, R. (1996) Peer Review in Selected Federal Agencies, presented at AAAS Annual 

Meeting, Baltimore, MD, February 9, 1996. 

National Academy of Sciences (1999) Evaluating Federal Research Programs: Research and 

the Government Performance and Results Act, Washington, DC: National Academy Press, 

January 1999. 

National Environmental Policy Institute (1996) Enhancing the Integrity and Transparency of 

Science in the Regulatory Process, Washington, DC: National Environmental Policy Institute, 

Fall 1996. 

National Research Council (1995) Interim Report of the Committee on Research and Peer 

Review in EPA, Washington, DC: National Academy Press, March 1995. 

National Research Council (2000) Strengthening Science at the U. S. Environmental Protection 

Agency: Research Management and Peer Review Practices, Washington, DC: National 

Academy Press. 

Reilly, W. (1993) Peer-review Policy, Washington, DC, Memorandum issued January 19, 1993. 

Rennie, D. (1999) The Development and Rationale of Peer Review, In: Peer Review in Health 

Sciences, Fiona Godlee and Tom Jefferson, eds., BMA Books, London. 
Page G-2 Peer Review Handbook 

Science Advisory Board (1999) An SAB Report: Review of the Peer Review Program of the 

Environmental Protection Agency, EPA-SAB-RSAC-00-002, U.S. Environmental Protection 

Agency, Science Advisory Board, Washington, DC, November 1999. 

Spitzer, H. (1995) Peer Review Practices in the Federal Government, Bethesda, MD: 

Environmental Network, report prepared for the American Industrial Health Council, April 

26, 1995) 

USEPA (1992) Safeguarding the Future: Credible Science, Credible Decisions, U.S. 

Environmental Protection Agency, Washington, DC, March 1992. 

USEPA (1998) Science Policy Council Handbook: Peer Review, EPA 100-B-98-001, U.S. 

Environmental Protection Agency, Science Policy Council, Washington, DC, January 1998. 

USEPA (1999) EPA’s Selection of Peer Reviewers, EPA OIG Report No. 1999-P-217, U.S. 

Environmental Protection Agency, Office of the Inspector General (OIG), Washington, DC, 

September 1999. 

U.S. General Accounting Office (1994) Peer Review: EPA Needs Implementation Procedures 

and Additional Controls, GAO/RCED-94-89, Washington, DC: U.S. Government Printing 

Office, February 1994. 

U.S. General Accounting Office (1996) Peer Review: EPA’s Implementation Remains Uneven, 

GAO/RCED-96-236, Washington, DC: U.S. Government Printing Office, September 1996. 

U.S. General Accounting Office (1999) Federal Research: Peer Review Practices at Federal 

Science Agencies Vary, GAO/RCED-99-99, Washington, DC: U.S. Government Printing 

Office, March 1999. 
Peer Review Handbook 

Notes and Comments 
Peer Review Handbook 

Notes and Comments 
Peer Review Handbook 

Notes and Comments 
Peer Review Handbook 

Notes and Comments