UNITED STATES ENVIRONMENTAL PROTECTION AGENCY

 WASHINGTON D.C. 20460 

 

OFFICE OF THE ADMINISTRATOR 

 SCIENCE ADVISORY BOARD 

June 18, 2010 

EPA-SAB-10-008 

The Honorable Lisa P. Jackson 

Administrator 

U.S. Environmental Protection Agency 

1200 Pennsylvania Avenue, N.W. 

Washington, D.C. 20460 

Subject: Review of EPA’s Microbial Risk Assessment Protocol 

Dear Administrator Jackson: 

In response to a request from EPA’s Office of Water (OW), the Science Advisory Board 

(SAB) convened the Drinking Water Committee to conduct a review of EPA's draft document, 

Protocol for Microbial Risk Assessment to Support Human Health Risk Assessment for WaterBased Media, henceforth referred to as “the MRA Protocol.” The Office of Water (OW) has 

performed quantitative microbial risk assessments (MRAs) in support of new regulations for 

microbial pathogens in drinking water under the Safe Drinking Water Act (SDWA). MRAs 

(although not formal quantitative MRAs) have also partially supported the development of 

health-based ambient water quality criteria and biosolids criteria under the Clean Water Act 

(CWA). These criteria have assisted in protecting against potential adverse human health 

outcomes and exposures to infectious disease microorganisms in recreational waters and from 

land application of wastewater biosolids. OW developed “the MRA Protocol” to provide 

Agency guidance for performing microbial risk assessments. Current Agency risk assessment 

guidance is geared towards chemical risk assessment, but MRAs do not fit completely within the 

chemical-risk framework because of microbial and host factors that are specific to microbial risk 

assessments. The MRA Protocol was developed to help risk assessors address these factors in a 

consistent manner. 

The SAB was asked to provide recommendations in several areas: how to improve the 

overall approach, the applicability of the protocol, the reasonableness of the protocol, the clarity 

of the protocol, the completeness and robustness of the protocol, and the ease of use of the 

protocol for conducting water-based microbial risk assessments. 
ii

The Committee commends the Agency for all the work undertaken and for taking a 

leadership role in the field of microbial risk assessment. This MRA document had been in 

development for many years and has undergone extensive internal and external review. It is 

important for EPA to complete this document as soon as possible as it will likely become an 

important document in this area. The key points and recommendations of the Committee are 

detailed in the report. Below is a brief summary. 

 Overall, the Committee finds the document to be comprehensive and inclusive of key 

information, but believes technical editing is needed to provide conciseness, clarity, and parallel 

structure between the chapters. The Committee also finds that the document does not fulfill its 

intended purpose as a “protocol.” A protocol generally implies a set of specific steps that would 

be undertaken to perform an MRA. This document does not provide those steps; rather, it serves 

as an excellent introduction to MRA by describing the conceptual framework, types of data and 

models, and the general process for undertaking an MRA. 

 

 The Committee strongly recommends the finalization and acceptance of this document, 

with appropriate modifications. The Committee recommends the following: (a) review and 

revision by a technical editor; (b) rename and restructure the document as an Introduction to 

MRA; (c) add more illustrative examples of actual EPA (or other) MRAs throughout the 

document with their strengths and weaknesses; (d) after publishing this document, develop a 

second, more advanced MRA document in the near future that would be a true protocol for 

conducting MRAs and/or a series of white papers that would address specific technical topics in 

greater detail. With such modifications, this document would represent a valuable foundation 

block in the field of MRA for the Agency. 

 

 Again, the SAB wants to express its overall admiration for the extensive work done by 

the Agency on this important topic of microbial risk. The SAB appreciates the opportunity to 

provide EPA with advice. We look forward to receiving the Agency’s response. 

Sincerely, 

 /Signed/ /Signed/ /Signed/ 

Dr. Deborah L. Swackhamer 

Chair 

EPA Science Advisory Board 

Dr. Joan B. Rose 

Former Chair 

Drinking Water Committee 

Dr. Jeffrey Griffiths 

Chair 

Drinking Water Committee

 
NOTICE 

This report has been written as part of the activities of the EPA Science Advisory Board, 

a public advisory committee providing extramural scientific information and advice to the 

Administrator and other officials of the Environmental Protection Agency. The Board is 

structured to provide balanced, expert assessment of scientific matters related to problems facing 

the Agency. This report has not been reviewed for approval by the Agency and, hence, the 

contents of this report do not necessarily represent the views and policies of the Environmental 

Protection Agency, nor of other agencies in the Executive Branch of the Federal government, nor 

does mention of trade names or commercial products constitute a recommendation for use. 

Reports of the EPA Science Advisory Board are posted on the EPA Web site at: 

http://www.epa.gov/sab. 

iii
U.S. Environmental Protection Agency 

Science Advisory Board 

Drinking Water Committee for the Review of 

EPA’s Microbial Risk Assessment Protocol

CHAIR 

Dr. Joan B. Rose, Professor and Homer Nowlin Chair for Water Research, Department of 

Fisheries and Wildlife, Michigan State University, East Lansing, MI 

MEMBERS 

Dr. Mark Borchardt, Director, Public Health Microbiology Laboratory, National Farm 

Medicine Center, Marshfield Clinic Research Foundation, Marshfield, WI 

Dr. Penelope Fenner-Crisp, Independent Consultant, North Garden, VA 

Dr. Jeffrey Griffiths, Associate Professor, Public Health and Community Medicine, School of 

Medicine, Tufts University, Boston, MA 

Dr. Gary King, Professor of Microbial Biology, Department of Biological Sciences, Louisiana 

State University, Baton Rouge, LA 

Dr. Joseph R. Landolph, Jr., Associate Professor, Molecular Microbiology and Immunology 

and Pathology, Keck School of Medicine and Associate Professor of Molecular Pharmacology 

and Pharmaceutical Science, School of Pharmacy, and Member, USC/Norris Comprehensive 

Cancer Center, University of Southern California, Los Angeles, CA 

Dr. Desmond F. Lawler, Bob R. Dorsey Professor of Engineering, Department of Civil, 

Architectural and Environmental Engineering, University of Texas, Austin, TX 

Dr. Christine Owen, Water Quality Assurance Officer, Tampa Bay Water, Clearwater, FL 

Dr. Richard Sakaji, Manager, Planning and Analysis for Water Quality, East Bay Municipal 

Utility District, Oakland, CA 

Dr. Gary Sayler, Beaman Distinguished Professor of Microbiology, and Ecology and 

Evolutionary Biology, Director of the Joint Institute for Biological Sciences and Director of 

the Center for Environmental Biotechnology, Oak Ridge National Laboratory, University of 

Tennessee 

Dr. David Sedlak, Professor, Department of Civil and Environmental Engineering, University 

of California-Berkeley, Berkeley, CA 

iv
Dr. Gina Solomon, Senior Scientist, Health and Environment Program, Natural Resources 

Defense Council, San Francisco, CA 

Dr. Laura J. Steinberg, Dean and Professor, College of Engineering and Computer Science, 

Syracuse University, Syracuse, NY 

Ms. Susan Teefy, Principal Engineer, Water Quality and Treatment Solutions, Inc., Canoga Park, 

CA 

SCIENCE ADVISORY BOARD STAFF 

Mr. Aaron Yeow, Designated Federal Officer, U.S. Environmental Protection Agency, 

Washington, DC 

v
U.S. Environmental Protection Agency 

Science Advisory Board 

BOARD 

CHAIR 

Dr. Deborah L. Swackhamer, Professor and Charles M. Denny, Jr., Chair in Science, 

Technology and Public Policy and Co-Director of the Water Resources Center, Hubert H. 

Humphrey Institute of Public Affairs, University of Minnesota, St. Paul, MN 

SAB MEMBERS 

Dr. David T. Allen, Professor, Department of Chemical Engineering, University of Texas, 

Austin, TX 

Dr. Claudia Benitez-Nelson, Associate Professor, Department of Earth and Ocean Sciences and 

Marine Science Program, University of South Carolina, Columbia, SC 

Dr. Timothy Buckley, Associate Professor and Chair, Division of Environmental Health 

Sciences, College of Public Health, The Ohio State University, Columbus, OH 

Dr. Thomas Burke, Professor, Department of Health Policy and Management, Johns Hopkins 

Bloomberg School of Public Health, Johns Hopkins University, Baltimore, MD 

Dr. Deborah Cory-Slechta, Professor, Department of Environmental Medicine, School of 

Medicine and Dentistry, University of Rochester, Rochester, NY 

Dr. Terry Daniel, Professor of Psychology and Natural Resources, Department of Psychology, 

School of Natural Resources, University of Arizona, Tucson, AZ 

Dr. George Daston, Victor Mills Society Research Fellow, Product Safety and Regulatory 

Affairs, Procter & Gamble, Cincinnati, OH 

Dr. Costel Denson, Managing Member, Costech Technologies, LLC, Newark, DE 

Dr. Otto C. Doering III, Professor, Department of Agricultural Economics, Purdue University, 

W. Lafayette, IN 

Dr. David A. Dzombak, Walter J. Blenko Sr. Professor , Department of Civil and 

Environmental Engineering, College of Engineering, Carnegie Mellon University, Pittsburgh, 

PA 

Dr. T. Taylor Eighmy, Vice President for Research, Office of the Vice President for Research, 

Texas Tech University, Lubbock, TX 

vi
Dr. Elaine Faustman, Professor, Department of Environmental and Occupational Health 

Sciences, School of Public Health and Community Medicine, University of Washington, Seattle, 

WA 

Dr. John P. Giesy, Professor and Canada Research Chair, Veterinary Biomedical Sciences and 

Toxicology Centre, University of Saskatchewan, Saskatoon, Saskatchewan, Canada 

Dr. Jeffrey Griffiths, Associate Professor, Department of Public Health and Community 

Medicine, School of Medicine, Tufts University, Boston, MA 

Dr. James K. Hammitt, Professor, Center for Risk Analysis, Harvard University, Boston, MA 

 

Dr. Rogene Henderson, Senior Scientist Emeritus, Lovelace Respiratory Research Institute, 

Albuquerque, NM 

Dr. Bernd Kahn, Professor Emeritus and Associate Director, Environmental Radiation Center, 

School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA 

Dr. Agnes Kane, Professor and Chair, Department of Pathology and Laboratory Medicine, 

Brown University, Providence, RI 

Dr. Nancy K. Kim, Senior Executive, New York State Department of Health, Troy, NY 

Dr. Catherine Kling, Professor, Department of Economics, Iowa State University, Ames, IA 

Dr. Kai Lee, Program Officer, Conservation and Science Program, David & Lucile Packard 

Foundation, Los Altos, CA 

Dr. Cecil Lue-Hing, President, Cecil Lue-Hing & Assoc. Inc., Burr Ridge, IL 

Dr. Floyd Malveaux, Executive Director, Merck Childhood Asthma Network, Inc., Washington, 

DC 

Dr. Lee D. McMullen, Water Resources Practice Leader, Snyder & Associates, Inc., Ankeny, 

IA 

Dr. Judith L. Meyer, Distinguished Research Professor Emeritus, Odum School of Ecology, 

University of Georgia, Lopez Island, WA 

Dr. Jana Milford, Professor, Department of Mechanical Engineering, University of Colorado, 

Boulder, CO 

Dr. Christine Moe, Eugene J. Gangarosa Professor, Hubert Department of Global Health, 

Rollins School of Public Health, Emory University, Atlanta, GA 

vii
Dr. Eileen Murphy, Manager, Division of Water Supply, New Jersey Department of 

Environmental Protection, Trenton, NJ 

Dr. Duncan Patten, Research Professor , Department of Land Resources and Environmental 

Sciences, Montana State University, Bozeman, MT 

Dr. Stephen Polasky, Fesler-Lampert Professor of Ecological/Environmental Economics, 

Department of Applied Economics, University of Minnesota, St. Paul, MN 

Dr. Stephen M. Roberts, Professor, Department of Physiological Sciences, Director, Center for 

Environmental and Human Toxicology, University of Florida, Gainesville, FL 

Dr. Amanda Rodewald, Associate Professor, School of Environment and Natural Resources, 

The Ohio State University, Columbus, OH 

Dr. Joan B. Rose, Professor and Homer Nowlin Chair for Water Research, Department of 

Fisheries and Wildlife, Michigan State University, East Lansing, MI 

Dr. Jonathan M. Samet, Professor and Flora L. Thornton Chair, Department of Preventive 

Medicine, University of Southern California, Los Angeles, CA 

 

Dr. James Sanders, Director and Professor, Skidaway Institute of Oceanography, Savannah, 

GA 

Dr. Jerald Schnoor, Allen S. Henry Chair Professor, Department of Civil and Environmental 

Engineering, Co-Director, Center for Global and Regional Environmental Research, University 

of Iowa, Iowa City, IA 

Dr. Kathleen Segerson, Professor, Department of Economics, University of Connecticut, Storrs, 

CT 

Dr. V. Kerry Smith, W.P. Carey Professor of Economics , Department of Economics , W.P 

Carey School of Business , Arizona State University, Tempe, AZ 

Dr. Herman Taylor, Professor, School of Medicine, University of Mississippi Medical Center, 

Jackson, MS 

Dr. Barton H. (Buzz) Thompson, Jr., Robert E. Paradise Professor of Natural Resources Law 

at the Stanford Law School and Perry L. McCarty Director, Woods Institute for the Environment, 

Stanford University, Stanford, CA 

Dr. Paige Tolbert, Associate Professor, Department of Environmental and Occupational Health, 

Rollins School of Public Health, Emory University, Atlanta, GA 

Dr. Thomas S. Wallsten, Professor and Chair, Department of Psychology, University of 

Maryland, College Park, MD 

viii
ix

Dr. Robert Watts, Professor of Mechanical Engineering Emeritus, Tulane University, 

Annapolis, MD 

SCIENCE ADVISORY BOARD STAFF 

Dr. Angela Nugent, Designated Federal Officer, U.S. Environmental Protection Agency, 

Washington, DC
ACRONYMS 

CWA Clean Water Act 

DWC Drinking Water Committee 

EPA Environmental Protection Agency 

HACCP Hazard Assessment and Critical Control Point 

ILSI International Life Sciences Institute 

MRA Microbial Risk Assessment 

OST EPA’s Office of Water, Office of Science and Technology 

OW EPA Office of Water 

SAB Science Advisory Board 

SDWA Safe Drinking Water Act

x
TABLE OF CONTENTS 

EXECUTIVE SUMMARY .......................................................................................................... 1 

INTRODUCTION......................................................................................................................... 6 

RESPONSE TO EPA CHARGE QUESTIONS......................................................................... 7 

1. Charge Question 1 – Overarching Considerations ............................................................ 7 

1.1 Utility of the Protocol for Meeting EPA’s Overall Needs, Particularly the 

Comprehensiveness and Robustness of the Protocol.................................................. 7 

1.2 Flow and Continuity Within and Between Chapters .................................................. 8 

1.3 Ease of Use and Utility for Outside Stakeholders ...................................................... 9 

1.4 Changes or Enhancement to the Protocol to Ensure That it Meets the Needs of 

 EPA and Outside Stakeholders ................................................................................... 9 

2. Charge Question 2 – Planning & Scoping and Problem Formulation (Chapter 2)...... 10 

2.1 Utility of Chapter to Ensure that Risk Assessments are Adequately 

 Conceptualized and Planned Appropriately.............................................................. 11 

2.2 Recommendations for Enhancing the Utility of the Chapter.................................... 11 

3. Charge Question 3 – Exposure (Chapter 3) ..................................................................... 11 

3.1 Additional Exposure Tools, Methods, and Approaches ........................................... 12 

3.2 Suggestions for Improvement................................................................................... 14 

4. Charge Question 4 – Human Health Effects (Chapter 4) ............................................... 17 

4.1 Scientifically Accepted Dose-Response Models ...................................................... 17 

4.2 Animal Dose-Response Models................................................................................ 18 

4.3 Human Health Outcomes.......................................................................................... 19 

4.4 Susceptible Populations ............................................................................................ 21 

4.5 Quality of Life........................................................................................................... 22 

5. Charge Question 5 – Risk Characterization (Chapter 5)................................................ 22 

5.1 Improvements to the Linkages between the Planning & Scoping and Problem 

Formulation, Exposure, and Human Health Chapters .............................................. 22 

5.2 Uncertainty, Variability, and Sensitivity Analysis ................................................... 24 

5.3 Other Recommendations........................................................................................... 24 

REFERENCES............................................................................................................................ 26 

APPENDIX – EPA CHARGE QUESTIONS........................................................................... 28 

xi 
EXECUTIVE SUMMARY 

EPA’s Office of Water (OW) requested that the Science Advisory Board (SAB) Drinking 

Water Committee (DWC) review its draft Protocol for Microbial Risk Assessment to Support 

Human Health Protection for Water-Based Media, henceforth referred to as the “the MRA 

Protocol.” There were five charge questions, which focused on an overview of the document 

and on the specific chapters of the document. These charge questions and responses are detailed 

in the report and the major recommendations from the Committee are highlighted below. The 

charge questions in their entirety are also presented in the Appendix of this report. 

Overall, the Committee finds the document to be comprehensive and inclusive of key 

information, but believes technical editing is needed to provide conciseness, clarity, and parallel 

structure between the chapters. However, the Committee finds that the document does not fulfill 

its intended purpose as a “protocol.” A protocol generally implies a set of specific steps that 

would be undertaken to perform, in this case, an MRA. This document does not provide those 

steps. However the Agency has done a tremendous amount of work on MRA and is commended 

for its leadership in this area, and this compilation serves as an excellent introduction to MRA by 

describing the conceptual framework, types of data and models, and the general process of 

performing an MRA. It is important that this document be finalized as soon as possible. The 

Committee recommends the following: 

• Add more examples of actual EPA (or other) MRAs throughout the document, with a 

concise summary of their strengths and weaknesses; 

• Rename and restructure the document as an Introduction to MRA rather than a 

Protocol; 

• Develop a second, more advanced MRA document that would provide a step-by-step 

process for conducting MRAs in the near future and/or a series of white papers that 

would address specific technical topics in greater detail; 

• Move the information from the appendices into the body of the text. 

• Clearly specify the target audience and, if there is more than one audience, clearly 

specify how they might differ in using the document; 

• Provide an index at the end of the document. 

Planning & Scoping and Problem Formulation Chapter 

The Committee finds that the Planning & Scoping and Problem Formulation chapter is 

generally useful. One recommendation the Committee has to improve this chapter is to: 

• Include some additional clarification to indicate when stakeholders should be 

consulted in the process, and whether the result of a planning/scoping and problem 

formulation exercise would be subject to external review. 

This chapter is also missing some information that would allow the reader to readily 

follow how the problem formulation is linked to the MRA process, and then to the desired end 

products. The linkage between the identified problem, and implementation of the MRA process 

1 
could be better outlined through the use of flow charts, figures, or logic trees. The Committee 

recommends: 

• Formatting all the diagrams in the chapter to the standard logic-diagram format. 

Exposure Chapter 

 

The Exposure chapter provides a good, concise discussion of the key issues related to 

exposure assessment and the role of exposure assessment in the overall risk assessment; however 

some weaknesses and omissions were identified. Sensitivity analyses of MRAs have shown that 

the greatest source of variability in risk assessment is from defining the exposure. It is therefore 

vital to consider a comprehensive range of possible exposures to adequately describe possible 

MRAs. The exposure profile, the sum result of exposure characterization, is not given adequate 

treatment in this chapter and is not comprehensive enough. The chapter focuses on endemic 

exposures rather than episodic exposures, the latter which are more likely to result in higher 

concentrations of pathogens in treated water. The chapter also addresses risks from recreational 

exposure more than risks from drinking water exposure. Drinking water examples, however, 

should be used throughout, including unusual exposure routes associated with biofilms and 

inhalation. The subject of indicator organisms instead of direct measurements of pathogens is 

not discussed and should be added. Indicator organisms are used extensively in environmental 

risk management as indicators of disinfection efficacy and provide many of the temporal and 

spatial data sets on sources, transport and fate. Both the uncertainty associated with using 

indicator organisms and situations in which indicator organisms are more or less likely to be 

present than the true pathogens of concern should also be addressed in this chapter. The 

Committee recommends: 

• Bolstering the discussion of the exposure profile; 

• Adding a discussion of drinking water exposures and the complex issues associated 

with doing so, including water treatment barriers; 

• Adding examples of MRAs used in drinking water exposure scenarios; 

• Adding a discussion of exposures from the use of reclaimed wastewater, recycled 

water, and gray water; 

• Adding a discussion of episodic exposures; 

• Adding a discussion of using indicator organisms instead of direct measurements of 

pathogens in water. 

Other suggestions for improving the chapter include: 

• Emphasizing not only the similarities but also the differences that exist between 

chemical exposure assessment and microbial exposure assessment; 

• Highlighting data needs for microbial exposure assessment, and how they differ from 

the data needs for chemical exposure assessment; 

• If possible, giving specific suggestions about possible data sources, or how data might 

be obtained, for performing an MRA; 

• Incorporating Appendix F, which is only one page in length, into this chapter; 

possibly as a “text box” if the authors think that it is disruptive to the overall flow. 

2 
• Adding examples of recently published MRAs used to guide beach closures and 

recreational exposures; 

• Expanding the discussion regarding analytical methods and the interpretation of 

results with a section heading. 

Human Health Effects Chapter 

 The Human Health Effects chapter is largely devoted to dose-response models. The 

Committee recommends separating this material into two chapters, one on dose-response and 

one on human health effects. The discussion on dose-response models was fairly comprehensive 

and thorough, when combined with Appendix G. The material from Appendix G should be 

brought into the text, but it should be rewritten by an expert statistician. The Committee strongly 

recommends: 

• Having an expert statistician rewrite the material in Appendix G for accuracy in its 

description of the extant literature and methods, and for appropriate continuity 

between the discussions of the various statistical techniques. 

Currently many animal feeding studies do not meet the criteria for developing statistically 

best-fit dose-response models. However, animal data sets are available and have been assessed. 

Thus this chapter should include a discussion of animal models that would be useful for the 

future, for example, the gnotobiotic piglet. The Committee recommends: 

• Adding a discussion of animal models that would be useful for the future, for 

example to address competency of the host to support infection and disease processes. 

Description of the data sets used for the dose-response curve fitting and the variability 

around the parameters should be included, and can be used further in MRA during uncertainty 

analysis. This characterization of the data sets may allow the reader to better understand how the 

quality of the data set and how the uncertainties in the data set affect the confidence one has in 

the dose-response curve. These details should include information about the microbial and host 

factors, such as pathogen strain and virulence, and host age. The Committee recommends: 

• Adding a discussion of uncertainty in dose-response modeling arising from data sets, 

low dose extrapolation, and microbial and host factors. 

Responding to the two charge questions on mathematical modeling of health outcomes 

and animal or in vitro models could have been more vigorous had the health effects of interest 

been defined. A better description of the human health outcomes associated with exposure to the 

wide range of environmental pathogens should be included in a very specific chapter devoted to 

this topic. A major observation was that this chapter contained little discussion of the health 

effects of interest and how mathematical approaches could be used to incorporate these effects 

into probability models or disease transmission models; the human health outcomes were not 

defined or delineated in sufficient detail. The majority of the chapter focused on dose-response 

analysis, obviously an important topic, but did not define the health effects of interest per se to 

which modeling should be applied. 

3 
The Committee recommends the following: 

• Making the dose-response section a separate chapter from the description of the 

health effects; 

• Expanding the discussion of health effects to include defining and delineating health 

outcomes; 

• Including a description of the major health syndromes and delineating the chronic and 

acute effects; 

• Providing examples of the health effects of interest, and how they have been used in 

prior risk assessments; 

• Adding specific health effects information in the compendium of mathematical 

models; 

• Adding a discussion of the spectrum of severity of health effects that may occur in a 

population; 

• Adding a discussion of the spectrum of syndromes caused by a single microbial 

agent; 

• Moving, or concisely reiterating, some or all of the susceptible population subsection 

in Chapter 2, to this chapter; 

• Adding a discussion of host response concepts such as timing or onset of infection 

and disease severity; 

• Removing the discussion about quality of life as it is not part of the risk assessment 

process. 

Risk Characterization Chapter 

The clarity and utility of the Risk Characterization chapter could be improved in several 

ways by connecting this chapter to the previous ones. It would be helpful to have summaries at 

the ends of the Exposure chapter and Human Health Effects chapter about what pieces of 

information need to be brought forward from those respective chapters and folded into a risk 

characterization. Then, at the beginning of the Risk Characterization chapter, it would be helpful 

to summarize the elements that need to be drawn from the earlier chapters and incorporated into 

a risk characterization. This section should explicitly refer back to the Planning & Scoping and 

Problem Formulation chapter. The discussion of dose-response models from the Human Health 

Effects chapter for use in the Risk Characterization chapter is confusing, and it is not clear which 

framework and models should be used to reflect different aspects of the risk assessment. The 

uncertainty, variability, and sensitivity analysis section of this chapter is good and does not omit 

any significant approaches or methods. To improve the chapter, the Committee recommends: 

• Adding summaries at the ends of Chapters 3 and 4 about what pieces of information 

need to be brought forward from those respective chapters and folded into a risk 

characterization; 

• Summarizing the elements that need to be drawn from the earlier chapters and 

incorporated into a risk characterization at the beginning of Chapter 5; 

• Trimming unnecessary detail and redundancy about the models in these chapters 

(perhaps, capturing the detail in a second more advanced MRA document and/or 

technical white papers); 

4 
• Clarifying explicitly the different roles that the (same or different) models play in 

each of these aspects of the risk assessment. These changes will ensure that the 

reader understands what needs to be accomplished by the modeling exercise in each 

step; 

• Adding text to explicitly refer back to the problem formulation, planning and scoping 

described in Chapter 2, especially Sections 2.1 and 2.2; 

• Placing some of the topics that are in Appendix D that are not already addressed in 

Chapter 5 (i.e., the topics covered in Sections D.4-D.10), in an overview/summary 

format within Chapter 5; 

• Acknowledging, and describing in an appropriate level of detail, the principles and 

practices regarding uncertainty, variability and sensitivity analysis presented in 

existing Agency documents that are, or could be, relevant to the conduct of MRAs; 

• Editing the chapter to better focus the chapter and to assure that the essential elements 

of the chapter are not lost in the tangential discussions and excessive detail. Some of 

the specific edits are detailed in the report. 

5 
INTRODUCTION 

This report was prepared by the Science Advisory Board (SAB) Drinking Water 

Committee (DWC) (the “Committee”) in response to a request by the Agency’s Office of Water 

(OW) to review their draft document, Protocol for Microbial Risk Assessment to Support Human 

Health Protection for Water-Based Media, henceforth referred to as “the MRA Protocol.”

The Office of Water (OW) has performed microbial risk assessments (MRA) in support 

of new regulations for microbial pathogens in drinking water under the Safe Drinking Water Act 

(SDWA). Specifically, quantitative MRAs have been performed in developing the Surface 

Water Treatment Rule and the Long-Term Enhanced Surface Water Treatment Rule. MRAs 

(although not formal quantitative MRAs) have also partially supported the development of 

health-based ambient water quality criteria and biosolids criteria under the Clean Water Act 

(CWA). These criteria have assisted in protecting against potential adverse human health 

outcomes and exposures to infectious disease microorganisms in drinking waters, recreational 

waters and land application of wastewater biosolids. OW developed “the MRA Protocol” to 

provide the Agency with more specific guidance on performing microbial risk assessments. 

Current Agency risk assessment guidance is geared towards chemical risk assessment, but MRAs 

do not fit completely within the chemical-risk framework because of microbial and human host 

factors that are specific to infectious disease and microbial risk assessments. The MRA Protocol 

was developed to improve risk assessments by Agency scientists and to address approaches to 

data and models in a consistent manner.

General features of the MRA Protocol document include: 1) a modular component 

concept; 2) flexibility to allow for unique Agency requirements which could be inserted or used 

to replace a set of default parameters; 3) discussion of various risk assessment applications 

including for regulatory purposes, outbreak investigation, identification and prioritization of 

research, investigation of risk-risk trade-offs, emergency response, and mitigation; 4) 

consistency with the companion EPA document, the Thesaurus of Terms Used in Microbial Risk 

Assessment, and 5) development of appendices showing details on dose-response modeling 

applications, flow diagrams for various types of assessments, and general considerations for 

conducting MRAs. 

The MRA Protocol document includes chapters on the following: 1) problem formulation, 

with planning, scoping, and tiered conceptual modeling; 2) exposure, which includes pathogen 

occurrence and exposure analysis; 3) human health effects, with dose-response and health effects, 

dose response modeling applications, and dynamic population susceptibility models; and 4) risk 

characterization, which applies EPA’s Risk Characterization Handbook, invoking uncertainty, 

variability, comparisons to similar risks, alternative approaches/solutions, and input to inform 

risk management decisions. 

The SAB was asked to provide recommendations in the following areas: how to improve 

the overall approach, the applicability of the protocol, the reasonableness of the protocol, the 

clarity of the protocol, the completeness and robustness of the protocol, and the ease of use of the 

protocol for conducting water-based microbial risk assessments. This task was posed to the 

6 
Committee in the form of specific charge questions detailed in the report below and also in the 

Appendix of the report. 

The Committee deliberated on the charge questions during a September 21-22, 2009 

face-to-face meeting and discussed its draft report in a subsequent conference call on November 

19, 2009. The Chartered SAB conducted a quality review of this document on March 24, 2010. 

Originally the charge question pertaining to Overarching Considerations was the last charge 

question, but the Committee believed that it was more appropriate as the first charge question, 

which is reflected below. The responses that follow represent the views of the Committee. 

RESPONSE TO EPA CHARGE QUESTIONS 

1. Charge Question 1 – Overarching Considerations 

EPA’s Office of Water, Office of Science and Technology (OST) would like this Protocol 

to provide a comprehensive and robust suite of approaches, tools, methods, and 

procedures to meet EPA’s overall needs in preparing for, and conducting typical MRAs. 

OST would also like the Protocol to be informative, easy to use and understand, and 

useful to outside stakeholders (states, communities, utilities, industry, and impacted 

parties). 

Please comment on the following: 

a) utility of the Protocol for meeting EPA’s overall needs, particularly on the 

comprehensiveness and robustness of the Protocol; 

b) flow and continuity within and between chapters; 

c) ease of use and utility for outside stakeholders; 

d) any changes or enhancements to the Protocol to ensure it meets the needs 

of EPA and outside stakeholders. 

1.1 Utility of the Protocol for Meeting EPA’s Overall Needs, Particularly the 

Comprehensiveness and Robustness of the Protocol 

 The Committee believes that the MRA Protocol is a valuable document that should be 

finalized quickly. However, this document is not really a protocol; that is, it does not lay out a 

step-by-step procedure for performing a microbial risk assessment. Rather, this document is an 

informative overview of the MRA process, describing the components of an MRA and 

delineating the data needs and possible models that could be used. 

 The document is quite comprehensive in terms of addressing the topic of microbial risk 

assessment at an introductory level, and the Committee did not identify major information gaps. 

The chapters are well organized but additional editing is needed to improve the flow of 

information, particularly in the use of topic sentences to introduce forth-coming ideas and 

provide transition between paragraphs and sections. The document reads much like a textbook 

and, in this respect, it is helpful and convenient to have all information on MRA readily available 

in one location. This document is very useful and could be one of the main resources on MRAs 

in the future. 

 

7 
• To make the document more comprehensive and robust, EPA should add examples of 

MRAs conducted on microbial pathogens. These examples should include a concise 

description of the strengths or weaknesses of the MRA; the specific data gaps which 

were highlighted in the MRA; and when new analytical tools and methodologies were 

either used or identified as needed. 

 The chapters introduce the readers to the various components of MRAs, whereas the 

Appendices go into greater detail with figures, tables and frameworks in each area. The 

discussion on the roles of the risk assessors and the risk managers is good and clearly delineates 

these roles. Attempts should be made to condense the writing of the document, to make it more 

concise, without sacrificing the clarity in the document and should undergo technical editing. 

 

 This document should be quite useful to professionals inside and outside of the 

government who conduct MRAs, and to scientists who are new to the field of MRA and who 

want to learn about this process. A determination should be made to establish the intent of this 

document: is it a Framework, a Guidance document, or an Introduction to Microbial Risk 

Assessment? In its present form, the document is more of an introduction to Microbial Risk 

Assessment than a Protocol. The Committee recommends the following: 

• Finalize this document as an Introduction to MRA; 

• In the near future, develop a more advanced document on MRA. 

 One could then progress from an Introduction to MRA (this document) to advanced 

topics in MRA to actual MRAs on specific microbes; the actual protocol would be the second 

document and would lay out step by step procedures on how to conduct MRAs. The examples of 

prior MRAs, and the concise descriptions of the strengths and weaknesses, data gaps, 

methodological needs, and usefulness to policy and regulatory decisions will inform this 

progression and help focus subsequent documents. 

1.2 Flow and Continuity Within and Between Chapters 

 The flow and continuity within all sections – the Executive Summary and 

Chapters 1-5 – are good but need to be improved. All of the chapters and the Executive 

Summary are well-written and informative. They are verbose in certain sections, however, and 

some condensation is warranted. The flow between the Problem Formulation, Exposure, Human 

Health Effects, and Risk Characterization chapters could be improved through the use of 

additional flow diagrams. Most of the Appendices are excellent, and add a wealth of detail to the 

document. 

 

Collectively, the DWC thinks that the information from the appendices should be moved 

into the body of the text. The Committee recommends that the Agency quickly produce (with 

good editing) this current MRA document as an introduction to MRA, while keeping some of the 

advanced and detailed discussions for inclusion in a second, more advanced MRA document 

and/or set of technical white papers. 

Suggestions from the DWC members that would improve the flow and while producing 

this as an overview of or introduction to MRA include: 

8 
• Placing Appendix A back into the text in the appropriate chapter; 

• Making Appendix B a separate chapter in the front or close to the front of the 

document; 

• Placing Appendix D toward the beginning of the document to illustrate general risk 

principles as a separate chapter; 

• Placing Appendix E at the end of one of the chapters as a concluding section in that 

chapter; 

• Placing Appendix F at the end of the appropriate chapter. 

• Appendix G should be removed from the Appendix and placed into the document in 

the dose-response section but only after a senior technical analyst with statistical 

expertise review Appendix G in particular, for accuracy in its description of the extant 

literature and methods, and for appropriate continuity between the discussions of the 

various statistical techniques (see response to charge question 4). 

1.3 Ease of Use and Utility for Outside Stakeholders 

 There was a good deal of discussion regarding the identities of the document’s intended 

audience and the primary stakeholders. The DWC concluded that this document was meant to 

first serve the scientists within the Agency and the groups they work with who may take on 

MRAs, and that the secondary audience and stakeholders would be the water industry. This 

document should be useful for these outside stakeholders as a primer for the scientist who is new 

to the field and wishes to understand the EPA’s MRA process. The stakeholders and scientists 

new to this area should be able to read the main chapters and understand them. 

1.4 Changes or Enhancement to the Protocol to Ensure That it Meets the Needs 

of EPA and Outside Stakeholders 

The intended primary purpose of this document was to inform and guide EPA staff and 

its contractors in conducting MRAs. An important secondary purpose was for the document to 

inform those outside the agency on how these EPA assessments are done. The document does a 

good job of describing the process thoroughly and helps a reader unfamiliar with the process to 

understand what an MRA might entail and the types of information needed. The Agency’s goal 

of transparency is furthered by this type of document. 

 The strength of the document, explaining the principles of MRA, may also be its 

weakness if the document was intended to be a protocol. The emphasis of the current 

document appears to be more on understanding MRA than in implementing an MRA. This focus 

is apparent in comparing Chapters 2 and 4. Chapter 2 has a sense of direction and describes a 

process of implementation with specific step-by-step instructions on how to formulate a problem 

to be addressed by MRA, and how to develop a suitable conceptual model. In contrast, Chapter 

4 is much more explanatory; the reader is not given directions, for example, on how to assess and 

choose dose-response models. 

 One idea for resolving this tradeoff of providing too much direction without enough 

explanation or vice versa, is to develop a step-by-step protocol subsequent to revising the current 

document (the current document being an Introduction to MRA). This first document will 

9 
provide the understanding, in which the steps in an eventual protocol can be linked to expanded 

explanations in this Introduction to MRA. Such a protocol could then follow the organization 

and format of other EPA protocols, such as Method 1623. It may help to create an overall visual 

schema, using a flow chart or decision tree, of the overall MRA process at the beginning of the 

document. This approach is used in an abbreviated fashion in Appendix D with Text boxes D.1, 

D.2, and D.3. It may further help to provide in the introduction a thorough description of actual 

well-developed MRAs as examples to guide the reader, such as the MRAs performed by EPA in 

support of the development of both the Surface Water Treatment Rule and the Long-Term 

Enhanced Surface Water Treatment Rule. 

Other enhancements to the document include the following: 

• Clearly specify the target audience and, if there is more than one audience, clearly 

specify how they might differ in using the document; 

• Provide an index at the end of the document; 

• Provide a better description of the Monte Carlo method and other appropriate 

probabilistic methods in the Risk Characterization chapter with appropriate reference 

(e.g., The Dutch MRA for drinking water); 

• Capture details on software or programming code for performing the risk 

characterization and the associated sensitivity and uncertainty analyses in a second 

advanced MRA document or separate white papers. 

 

 General principles from the Appendices should be captured into the appropriate chapters. 

Also, where appropriate, some of the detailed information from the chapters and appendices 

could be better placed into a protocol (as mentioned above in support of choosing appropriate 

dose-response models for example) and more advanced MRA documents. Separate white papers 

could be written to expand on detailed mathematical advances or uses for MRA in the future. 

These changes would improve the flow of the information. 

 With such modifications, this document could be entitled, Introduction to Microbial Risk 

Assessment. It could then be used in the same manner that the “Red Book” for carcinogenic risk 

assessment is used, as a foundation document for MRA. 

2. Charge Question 2 – Planning & Scoping and Problem Formulation (Chapter 2) 

Please comment on the utility of this chapter to ensure that risk assessments are 

adequately conceptualized and planned appropriately to address risk management’s 

issues. Please provide any recommendations for enhancing the utility of this chapter. 

Please comment on any enhancements or expanded guidance needed to allow users to 

prepare and conduct risk assessments to address a broad range of types of risk 

management questions. Examples of types of EPA uses of MRA may be: 

a) approaches to mitigation of environmentally-based microbial pathogen 

exposure risks; 

b) determination of acceptable health risks; 

c) identification of different exposure factors/routes; 

d) identification of microbial-based hazards in disease outbreaks; 

10 
e) development and prioritization of research needs; 

f) competing risks ranking. 

2.1 Utility of Chapter to Ensure that Risk Assessments are Adequately 

Conceptualized and Planned Appropriately 

Overall, this chapter provides a high-level discussion of how to plan and conduct an 

MRA. The structure described in this chapter, which involves formulating the problem and 

scoping out the entire process is excellent. It is particularly important in risk assessments to very 

specifically write down the questions that are being addressed and to develop a plan for 

addressing them. This approach applies to MRAs but also to nearly any technical investigation. 

The discussion of components in the conceptual model narrative were listed and concisely 

discussed, and in general, this chapter is well-written. The overall approach is sound and logical. 

It is particularly helpful to acknowledge up front that the conduct of an MRA is an iterative 

process. As the investigation/assessment proceeds, new information may point the investigator 

in a different direction, and the overall plan will be adjusted accordingly. 

2.2 Recommendations for Enhancing the Utility of the Chapter 

This chapter contains a good collection of common definitions that are unique to this 

field. An “outsider” to the MRA field might find this helpful in understanding the process. One 

recommendation to improve this chapter is to: 

• Include some additional clarification to indicate when stakeholders should be 

consulted in the process, and whether the result of a planning/scoping and problem 

formulation exercise would be subject to external review. 

The chapter could be improved if the diagrams were changed. For example, EPA has a 

general logic-diagram format used in drinking water regulations that is very helpful. Starting at 

the top, one proceeds in a downward direction, following a particular arrow. If there is a 

decision (yes/no) this is shown as a diamond, with arrows leading away from the corners of the 

diamond depending on the outcome of the decision. If there is an iteration, the arrow is shown 

looping back around to the starting point. Most of the figures in this chapter do not follow the 

standard logic-diagram format and if they were changed to this format, this would help tie all the 

pieces together to get to the end product of planning/scoping and problem formulation. The 

Committee recommends: 

• Formatting all the diagrams in the chapter to the standard logic-diagram format. 

3. Charge Question 3 – Exposure (Chapter 3) 

Please comment on any additional exposure tools, methods, or approaches that should be 

included to ensure a robust approach to adequately determining the microbial 

occurrence and human exposure factors relevant to health risks from water. This 

includes support for the estimation of the magnitude, frequency, duration, and also 

additional types of exposure to microbial pathogens by the water route, as well as the 

range of characteristics of the exposed population and their exposure profiles. 

11 
The Exposure chapter of the draft MRA protocol is a relatively short chapter in the 

overall document. The chapter has a good, concise discussion of the key issues related to 

exposure assessment and its role in the overall risk assessment. Points that are (properly) 

emphasized include the ideas that the life cycle and ecology of microorganisms are critical points 

for understanding the exposure pathways that ultimately lead to an exposure assessment. The 

current document also notes that the exposure duration and the population characteristics are 

important variables in assessing overall exposure. Exposure assessment is often very venue- and 

microbe-specific and previous EPA MRA examples could be included here, such as that 

performed for the Cryptosporidium risk assessment. This example would illustrate the exposure 

issues that need to be considered as well as delineate the type of specificity and quantitative data 

that are needed. 

In addition, recently published results on shower biofilms (Feazel, et al. 2009) raise 

questions about the extent to which an MRA can or should be extended to cover exposures that 

have not been considered previously or recognized as problems for water-borne pathogens. This 

also included pathogens such as Legionella. Incorporating “novel” routes and opportunistic 

pathogens may require new data for a number of variables, but ignoring these routes could result 

in unrealistic MRAs for some pathogens and some populations. 

3.1 Additional Exposure Tools, Methods, and Approaches 

The layout of the entire document is based on the breakdown shown in Figure 7 (p. 32 in 

the draft document), entitled “Analysis Phase Microbial Risk Assessment for Pathogens.” This 

chapter is concerned with the bottom three boxes on the left side of that document, called 

“Occurrence, Exposure Analysis, and Exposure Profile”. The profile is the net result of all the 

work that precedes the bottom line of the characterization of exposure. Despite its extreme 

importance to the MRA, the exposure profile is given inadequate treatment in this chapter, 

compared to the rest of the document. Uncertainty analysis reported in the literature on drinking 

water MRAs has shown that exposure assessment is the primary factor driving the distribution of 

risk outputs; thus it remains a very important aspect of the MRA. Two examples from the 

literature are cited and explained in some detail, but the reader is left to ascertain what 

constitutes an appropriate statement of the exposure profile and the significance of generating the 

exposure data. The examples would be much more valuable if the general principles were 

explained in more detail; this section is too general to be particularly useful. The Committee 

recommends: 

• Bolstering the discussion of the exposure profile. 

Exposure assessment for water applications will likely be far more complex in the future 

than is portrayed in this chapter. Throughout the chapter, the focus seems to be on natural water 

systems, where the human exposure is likely to occur through swimming or other recreational 

activities, i.e., where there is direct contact with raw untreated or insufficiently treated water. 

The protocol is supposed to be useful for that situation, but it must also be flexible enough to 

include more complex situations, such as assessing the risks associated with drinking water, 

where human exposure to pathogens has been attenuated by environmental factors or water 

treatment. MRA in drinking water is far more complex than MRA in recreational settings 

because the drinking water passes through a number of barriers before the water reaches the 

12 
consumer. The subject of drinking water and the presence of these barriers are not discussed in 

the MRA document. 

As the unit processes in a water treatment plant are designed to remove or inactivate 

pathogens, their role in preventing or minimizing human exposure to waterborne pathogens must 

be incorporated into the exposure assessment. This has been done previously for drinking water 

rules like the Surface Water Treatment Rule and more recently, the Long Term Enhanced 

Surface Water Treatment Rule (Haas, et al. 1993 and Regli, et al. 1991). How the treatment unit 

processes are configured also plays an important role in defining the strength and reliability 

(aided by redundancy) of the overall treatment barrier (ASCE and AWWA, 1990). There may 

be multiple filters feeding into a common header (in which case, each filter operates 

independently of the others in a “parallel” configuration). In the case of filtration followed by 

disinfection, the unit processes are in series, i.e., one step (filtration) followed by the next 

(disinfection); the unit processes may operate independently of each other in a series 

configuration, but not always. The concepts outlined in the ASCE and AWWA (1990) design 

manual have been used and demonstrated in particle counting studies and to establish regulatory 

log removal credits for Giardia and Cryptosporidium (Sakaji, et al. 1996). The statistical 

techniques used in this work are applicable to evaluating human exposure to pathogens in 

drinking water in large water treatment plants with multiple unit processes. The Committee 

recommends: 

• Adding a discussion of drinking water exposures and the complex issues associated 

with doing so, including water treatment barriers; 

Aside from drinking water, the document does not include exposure assessments 

associated with the use of reclaimed wastewater, recycled water, and gray water. All of these are 

important sources of water that will be used to amend current water supplies and/or improve 

water efficiency. Although the USEPA does not have current national programs in these subject 

areas, state agencies, which have recycling programs have used MRAs in their development of 

public policy for water recycling and will use the USEPA’s MRA document for this 

purpose. There is a body of published literature in the area of MRA that the Agency can cite, 

for use of recycled wastewater in agricultural irrigation, swimming, and landscape irrigation 

practices (Asano, et al. 1992; Soller, et. al. 2004; Rose et al., 1996). Because the terminology of 

reclaimed wastewater, recycled water, and gray water will be unfamiliar to many, these terms 

will need to be defined and explained in detail. Given the recent cross connection between a 

reclaimed wastewater irrigation line and potable water supply line in Coomara, Australia (this is 

not the first time; there have been cross connection incidents in California with recycled 

wastewater, none associated with a waterborne disease outbreak), this remains a route of 

exposure wherever water conservation occurs. The Agency may want to add a footnote about 

this. The Committee recommends: 

• Adding a discussion of exposures from the use of reclaimed wastewater, recycled 

water, and gray water. 

Another weakness of this chapter is that it focuses on endemic exposure risks and not 

episodic exposure risks. An exposure assessment in drinking water will need to consider all the 

13 
events leading to exposure and account for the likelihood of those events occurring. Pathogens 

in treated water supplies typically occur episodically rather than endemically. A series of 

seasonal events (such as a rare storm event on a watershed) can lead to a significant change in 

raw source water quality. If this change is severe enough it can lead to the overload and 

subsequent failure of a treatment system. However, the failure of water treatment systems is an 

extremely rare event because the events leading to it are not frequent and with adequate on-line 

real time monitoring are easily avoided. The low frequency of such failures needs to be captured 

and included in the drinking water exposure assessment. The Committee recommends: 

• Adding a discussion of episodic exposures. 

Connecting exposure assessment to management strategies is important for drinking 

water in particular. Thus another element of the assessment could incorporate a hazard 

assessment and critical control point analysis (HACCP) to determine where the “weak” or 

critical control points are, where the possible concentrations of pathogens and potential for 

exposure are greatest, and where barriers are needed and should be monitored (or strengthened) 

to prevent an outbreak. In order to prevent further delay in publishing this document, the 

HACCP concept could be identified in this document and developed further in the detailed 

protocol. 

This chapter is the shortest of all the chapters, but the brevity is largely due to the 

omission of several important points. As noted above, the chapter does not include anything 

about the drinking water pathway as a possible exposure route for microbial risk; it is essential 

that this be addressed in all parts of the chapter. Another serious omission is a lack of discussion 

on the use of indicator organisms instead of direct measurements of pathogens. Indicator 

organisms or surrogates are used extensively in environmental risk management as indicators of 

disinfection efficacy and provide many of the temporal and spatial data sets on sources, transport 

and fate, as stated in the analytical methods chapter of Haas, Rose, and Gerba (1999). This 

discussion should start with the four characteristics of a good microbial surrogate as referenced 

in Kay and Fricker (1997). The discussion should then evolve to explain how the choice of using 

indicator organisms contributes to the uncertainty. Situations in which indicator organisms are 

more or less likely to be present than the true pathogens of concern should be addressed in this 

chapter. The Committee recommends: 

• Adding a discussion of using indicator organisms instead of direct measurements of 

pathogens. 

3.2 Suggestions for Improvement 

The Committee’s suggestions reflect the comments above about omissions and 

weaknesses. The early part of the chapter should emphasize not only the similarities but also the 

differences that exist between chemical exposure assessment and microbial exposure assessment. 

Exposure assessments for microbes can be substantially more challenging than it is for chemicals. 

The delineation of these differences will highlight the data needs. The lack of available data is a 

major limitation in assessing exposure, or the likelihood of exposure, in the development of any 

MRA, so data needs should be highlighted since it adds considerably to the uncertainty. If 

14 
possible, giving specific suggestions about possible data sources, or how data might be obtained, 

for the performance of an MRA would be useful. The Committee recommends: 

• Emphasizing not only the similarities but also the differences that exist between 

chemical exposure assessment and microbial exposure assessment; 

• Highlighting data needs for microbial exposure assessment; 

• If possible, giving specific suggestions about possible data sources, or how data might 

be obtained, for the performance of an MRA; 

• Incorporating Appendix F, which is only one page in length, into this chapter; it could 

be done as a “text box” if the authors think that it is disruptive to the overall flow. 

The chapter can be improved by adding examples of MRAs throughout the chapter. 

These include: 

• MRAs used in drinking water exposures; 

• Recently published MRAs used to guide beach closures and recreational exposures. 

The Committee recommends that the discussion regarding analytical methods and the 

interpretation of results needs to be expanded and given its own section. Such a section would 

provide a better presentation regarding the interpretation of occurrence results, since different 

types of assays are used to enumerate pathogens. As noted earlier, the exposure assessment is 

the greatest source of uncertainty and the hazard assessment and critical control point analysis 

(HACCP) will probably point to the interpretation of the occurrence data as contributing the 

most to the overall uncertainty of the MRA. Some of these assays are designed only to detect 

and enumerate infectious pathogens, whereas others provide a response based on DNA or 

antigen (protein components) presence that does not differentiate between viable, infectious, and 

nonviable pathogens. 

The discussion on exposure associated with methods is confusing and should be 

reorganized and further related to the other chapters (e.g., the dose-response chapter). An 

example of this is found in the discussion on page 45 under the heading “What is the Level of 

Pathogens in the Water Body?”. The sentence that begins with “Assays used to quantify 

pathogens yield variable...” identifies three major points. These points relate to the analytical 

methods; the variable recovery rates associated with the assays; detection limits; and the fact that 

the assays may not provide information on the viability of pathogens, which affects their ability 

to infect humans. While these are important points, their importance is lost in the subsequent 

discussion on evaluating the efficacy of treatment. The example of the oocyst excystation assay 

not providing information on the infectivity of the trophozoites is accurate from the standpoint of 

evaluating the effectiveness of UV as a treatment technology, but misses a larger point. If the 

assay used to enumerate the oocysts in the Dupont human infectivity study (1995) provides the 

same level of enumeration accuracy (with the associated limitations) as the water sample assays 

for oocysts, then the major difference with excystation studies used to evaluate UV (early 

studies), would be the relative level between total count of oocysts and infectious oocysts. Thus 

what the discussion misses is that there may be a change in the relative ratio of infective, viable, 

and nonviable oocysts in the sample between the source of the oocysts and the raw water intake. 

The estimates of human infective dose used in developing the dose-response relationship have 

15 
uncertainty associated with the dose. Freshly-harvested oocysts were used as the inoculums in 

the human challenge studies and are often used in disinfection studies, yet the age of the oocysts 

in any given water sample can vary widely, as could the distribution of viable and infective 

oocysts. This difference leads to part of the uncertainty in the interpretation of the data, and thus 

the specific assumptions need to be clearly stated and used in exposure assessment. When new 

methods are introduced, such as cell culture techniques to enumerate the number of infectious 

oocysts in water samples, they may allow refinement of the dose-response relationship (Slifko, et 

al. 1997; Slifko, et al. 1999; Slifko, et al. 2002). This refinement can then be used to improve 

future exposure assessments. 

The subsequent paragraph jumps to a discussion of how environmental factors can affect 

the numbers of microbes and spectrum of microbes including bacteria and viruses. These 

analyses often have the opposite problem from oocyst measurements, as culture techniques may 

underestimate the dose. It is well known that culture techniques routinely used in the laboratory 

can not address the presence of all viable bacteria or viruses. It should be further pointed out to 

the reader that even the bacterial assays depend on the “state” of the bacteria as some may be in a 

stressed or quiescent stage and while non-culturable can be infectious once ingested. Thus the 

exposure is underestimated as is the risk. 

The analytical methods for detecting microorganisms in water are probably the biggest 

challenge and represent the largest source of difference between chemical and microbial risk 

assessments. The microbial methods include microscopic techniques that do not rely on the 

viability of the microbe (Cryptosporidium and Giardia). The method detection limits for 

microbial assays are written as less than one organism per volume of water processed (which 

might be considered analogous to one atom of a chemical contaminant in a volume of water), 

generally without consideration of recovery efficiencies or false positive/negative rates. The 

approach to establishing minimum limits of detection and practical quantitation limits for 

microbial methods is unlike the approach taken for analytical methods used to enumerate 

chemical concentrations. While the assumption of one organism per volume of water sampled as 

a method detection limit may work for some microbial assays, it is not valid for all assays. For 

example, the highly variable recovery rates for Giardia and Cryptosporidium may be affected by 

the amount of processing the sample goes through before the enumeration step. In turn, this 

affects the reliability and reproducibility with which one oocyst or cyst can be enumerated in a 

sample volume. Poor reproducibility contributes to increased uncertainty as the concentration 

approaches the minimum detection limit. 

Later in the chapter the statement “Correction of oocysts counts for viability...” indicates 

that correcting for viability has little impact on the concentration distribution. This statement 

needs to be clarified. Does this mean the shape of the distribution doesn’t change, but the 

position of the mean does? If so, assessing viability is still important, as the magnitude and 

strength of the treatment barrier may not need to be as great as regulations dictate. The statement 

seems to contradict the earlier arguments regarding the importance of viability for assessing 

treatment efficacy. 

16 
In summary then the Committee recommends: 

• Bolstering the discussion of the exposure profile; 

• Adding a discussion of drinking water exposures and the complex issues associated 

with doing so, including water treatment barriers; 

• Adding examples of MRAs used in drinking water exposure scenarios; 

• Adding a discussion of exposures from the use of reclaimed wastewater, recycled 

water, and gray water; 

• Adding a discussion of episodic exposures; 

• Adding a discussion of using indicator organisms instead of direct measurements of 

pathogens in water. 

Other suggestions for improving the chapter include: 

• Emphasizing not only the similarities but also the differences that exist between 

chemical exposure assessment and microbial exposure assessment; 

• Highlighting data needs for microbial exposure assessment; 

• If possible, giving specific suggestions about possible data sources, or how data might 

be obtained, for performing an MRA; 

• Incorporating Appendix F, which is only one page in length, into this chapter, 

possibly as a “text box” if the authors think that it is disruptive to the overall flow; 

• Adding examples of recently published MRAs used to guide beach closures and 

recreational exposures; 

• Expanding the discussion regarding analytical methods and the interpretation of 

results with a section heading. 

4. Charge Question 4 – Human Health Effects (Chapter 4) 

Please comment on any additional scientifically accepted dose-response models 

(including advanced and validated threshold, empirical, or mechanistic models) which 

should be included as tools for determining human dose-responses from waterborne 

exposures via oral, inhalation, and dermal routes, especially for low dose extrapolation. 

Please comment on whether any specific animal or in vitro dose-response protocols, 

models, and methods should be included in this chapter. If so, please describe their 

applications and limitations in establishing human dose-response curves. 

4.1 Scientifically Accepted Dose-Response Models 

The discussion of scientifically accepted dose-response models in the chapter, when 

combined with Appendix G, was comprehensive and thorough. General principles from 

Appendix G are critical to interpreting the chapter and therefore should be merged into the 

chapter. However, the Committee strongly recommends that the material in Appendix G be 

rewritten by an expert statistician for accuracy in its description of the extant literature and 

methods, and for appropriate continuity between the discussions of the various statistical 

techniques. 

17 
Description of the data sets used for the dose-response curve fitting and the variability 

around the parameters should be included, and can be used further in MRA during uncertainty 

analysis. This characterization of the data sets may allow the reader to better understand how the 

quality of the data set and how the uncertainties in the data set affect the confidence one has in 

the dose-response curve. These details should include information about the microbial and host 

factors, such as pathogen strain and virulence, and host age. Extrapolations to low dose always 

bring many uncertainties, because often, no data at the low doses are available. The doseresponse data sets for bacteria often reflect this lack of data at low doses, whereas those for 

viruses and parasites usually do include some low doses. The confidence levels surrounding the 

best-fit curve broaden with extrapolation to the low doses, and this concept should be included as 

one of the uncertainties surrounding the data. Furthermore, uncertainty is introduced when one 

extrapolates from information derived from a specific setting to the larger and more general 

circumstance. Take, for example, a specific study, which uses a specific experimental pathogen 

strain of microbe, and examines the consequences in a defined and specific human group or 

population. When generalizing from this specific study the assumption may be made that this 

microbe was representative of all of the pathogenic strains of the microbe, and that the human 

population that was studied was representative of the entire human population. This may not be 

true, and the uncertainty introduced by this assumption should be mentioned. The Committee 

recommends: 

• Adding a discussion of uncertainty in dose-response modeling arising from data sets, 

low dose extrapolation, and microbial and host factors. 

4.2 Animal Dose-Response Models 

A future research need is the exploration of the mathematical description of the doseresponse functions associated with variations in pathogen virulence, multiple doses, and mixtures. 

Animal models will provide the opportunity for advancing an understanding of the dose-response 

process. While it is acknowledged that many animal feeding studies do not meet the criteria for 

developing statistically best-fit dose-response models, this chapter should include some 

discussion of animal models that would be useful for the future. 

The gnotobiotic piglet model is one such model. It has been used for a number of human 

enteric (diarrheal) pathogens such as Campylobacter jejuni, Shigella dysenteriae, Salmonella, 

Cryptosporidium, Isospora, Helicobacter pylori, and Escherichia coli (Law et al., 2009; Jeong et 

al 2010). This model has been found to be useful for studying different manifestations of human 

disease. Gnotobiotic piglets have also been used to study a spectrum of rotavirus and even 

Norovirus isolates (Zhang et al., 2008). The application of this specific model is dependent upon 

the pathogen and the health outcome of interest. A broad range of other animal models ranging 

from the worm C. elegans to primate relatives of humans have also been studied to ascertain 

mechanisms of pathogenicity and microbial virulence. Further use of these models should be 

exploited to gain a better quantitative understanding of dose-response relationships relative to 

microbe-host-environment interactions with particular emphasis on susceptible populations. Of 

particular relevance would be those studies focused on immuno-compromised/incompetent and 

aged populations for which models such as nude mice have yielded important mechanistic 

insight and could be further extended to dose response/risk assessment relationships. 

18 
The data are scanty for many human pathogens, especially considering the spectrum of 

health effects upon which a modeling exercise must rest. Considerable information relevant to 

some pathogens (Cryptosporidum, E. coli) is available, but little exists for many other pathogens. 

These scientific gaps will need to be filled and in most cases, receptive and competent animal 

models will need to be used. 

4.3 Human Health Outcomes 

Responding to the two charge questions on mathematical modeling of health outcomes 

and animal or in vitro models could have been more vigorous had the health effects of interest 

been defined. A better description of the human health outcomes associated with exposure to the 

wide range of environmental pathogens should be included in a very specific chapter devoted to 

this topic. A major observation was that this chapter contained little discussion of the health 

effects of interest and how mathematical approaches could be used to incorporate these effects 

into probability models or disease transmission models; the human health outcomes were not 

defined or delineated in sufficient detail. The majority of the chapter focused on dose-response 

analysis, obviously an important topic, but did not define the health effects of interest per se to 

which modeling should be applied. For example, viral hepatitis may be a waterborne disease, 

but the word “hepatitis” is only mentioned in the entire document three times – once in a chart, 

once in an explanation that human hepatitis E is similar to the porcine variant; and once in a 

discussion of milder disease in children. The exposure models are elegant, but modeling must be 

grounded in factual data to have authenticity and to be useful to the reader of this document. The 

Committee recommends: 

• Making the dose-response section a separate chapter from the description of the 

health effects; 

• Expanding the discussion of health effects to include defining and delineating health 

outcomes. 

The first section of this chapter on health effects (4.1) mentioned a number of health 

effects “elements" that should be considered during risk assessment. These elements included 

duration and severity of illness, the morbidity and mortality and long-term health effects, 

transmission to others, and quality of life. These are described in a bit more detail over a 2-page 

section before the dose-response analysis overview (4.2) begins. 

The section on health effects does not include a description of some of the major health 

syndromes. These should include watery diarrhea, nausea and vomiting, illnesses similar to 

influenza, dysentery, hepatitis, meningitis, and others. Some inkling of these syndromes is given 

in the subsection that describes chronic sequelae (4.1.3) and lists some delayed effects of 

infection. However, there is no corresponding section on acute effects, and the list of chronic 

effects is illustrative, but not comprehensive. With such delineation of chronic and acute effects, 

it might be easier to identify where models, based in sound science, exist and where they do not. 

The Committee recommends: 

• Including a description of the major health syndromes and delineating the chronic and 

acute effects. 

19 
Earlier in the document, in section 2.2.4 (page 24) on the Scope of problem formation, 

the suggestions are made that the scope should include “Which infectious disease hazard is being 

addressed; ...which human populations will be included in the risk assessment; ...and what health 

outcome or endpoints are addressed by the risk assessment, including how the health outcome is 

measured.” The point is made in that section that the scope of the assessment (infection, disease 

symptom/s, mortality) must be defined. These different outcomes are all health effects to which 

the modeling can be applied. The document could be improved by providing examples of the 

health effects of interest, and how they have been used in prior risk assessments to give the 

reader a sense of the literature. Were this suggestion to be followed, a response to the charge 

questions with more detail could be provided. The Committee recommends: 

• Providing examples of the health effects of interest, and how they have been used in 

prior risk assessments. 

The section in the Health Effects chapter that begins on page 56 is an admirable 

compendium of the mathematical models which have been, or could be, used to model the 

effects of fairly generic exposures in a population. These models address the extent or likelihood 

of a general health effect in the population, not the modeling of the specific human health effects 

of interest. Human health effects information would allow a more robust analysis of the models 

that may, or may not, exist for specific health effects. Table IV of the chapter, on pages 69-70, 

provides information on pathogens and models used to describe their effects. An additional 

column (or a separate table) should be included that delineates the anticipated health effects of 

these pathogens. The Committee recommends: 

• Adding specific health effects information in the compendium of mathematical 

models. 

In the section on human health effects, there is no mention of the spectrum of severity of 

health effects that may occur in a population. While the median health effect of some infections 

may be minor, some individuals - only some of whom may belong to a susceptible population - 

may suffer uncommon, yet severe effects. By way of illustration, most children with diarrhea in 

the US have mild illness, so the severity of the average or median case of diarrhea is minor. 

However, some children with diarrhea become dehydrated and require admission to hospital, and 

a small number of these children will die. Thus the spectrum of the severity of the illness is 

broad, ranging from subclinical infection to death, while the severity of the median case is of 

minor consequence. The Committee recommends: 

• Adding a discussion of the spectrum of severity of health effects that may occur in a 

population. 

In addition, the health effects may include a spectrum of syndromes caused by a single 

microbial agent. An example of this variation can be seen with some enteroviruses, which most 

often cause diarrhea but also can cause meningoencephalitis (infection of the brain) and 

pericarditis (inflammation of the pericardium, which encloses the heart). While the diarrhea is 

typically not life-threatening, the meningoencephalitis may easily result in hospitalization, and 

the pericarditis may result in not only hospitalization but also life-long adverse effects. It would 

20 
be useful for the document to further highlight this point, perhaps in an example, to illustrate not 

only the severity but the spectrum of human health outcomes that are possible. The Committee 

recommends: 

• Adding a discussion of the spectrum of syndromes caused by a single microbial agent. 

A nexus of scientific advances is likely to substantially improve our understanding of the 

dose-response relationship between drinking water pathogens and human health outcomes in the 

near to mid-term future. These advances include: 

• The development and adoption of real-time pathogen detection in drinking water 

systems, spurred by homeland defense concerns; 

• The identification of specific genetic susceptibilities to microbial pathogens via rapid 

advances in genomic medicine; and 

• Improvements in surveillance systems which can better detect and follow human 

health syndromes, and specific disease, in the US population. 

The first will allow us to understand the magnitude of, and variance in, waterborne 

microbial pathogen occurrence using novel technologies that bypass the limitations of current 

methods. Real-time molecular or chromatographic monitoring will undoubtedly reveal 

exposures that we would not otherwise be aware of. Our current system of episodic or 

infrequent sampling, and detection via methods that depend upon pathogen growth, act as 

constraints in understanding the occurrence of pathogens. 

Genomic science is likely to identify specific genetic vulnerabilities to more microbial 

agents found in water, just as it is currently identifying the genetic basis for other infectious 

diseases such as Noroviruses, human immunodeficiency virus (HIV), severe acute respiratory 

syndrome (SARS), and Epstein-Bar virus (Lindesmith, et al. 2003). Finally, specific susceptible 

populations beyond our current understanding will be identified. We currently recognize infants 

and children, the elderly or pregnant, and those immuno-compromised by disease or drugs as 

susceptible to infectious agents (see more in the section below). These advances will help us to 

understand the full range of population-wide health effects as well as effects in specific sentinel 

populations. 

In aggregate, improved occurrence data and improved outcomes information should lead 

to far more robust risk assessment. 

4.4 Susceptible Populations 

The chapter on Problem Formation includes a subsection on susceptible populations 

[“Initial host characterization” pages 38-40, section 2.3.2] that is closely related to this chapter 

on Health Effects; the Committee recommends that some or all of that subsection should be 

moved to, or concisely reiterated in, this chapter. Certainly different populations may be 

affected by different routes of exposure. The example is given of behavioral elements, such as 

the ingestion of raw sea food, which is the critical route of exposure for some diseases. However, 

much of the discussion about susceptible populations in this section is relevant to how diseases 

21 
may be differentially expressed in susceptible populations. It is clear from the literature that 

certain populations including the elderly, immuno-compromised people, and young children are 

more susceptible to adverse outcomes (as seen with outbreaks of E. coli 0157: H7 and AIDs 

patients and Cryptosporidium). Currently there are no data that demonstrate that this 

susceptibility is a result of changes in dose-response functions. It appears likely that the 

increased susceptibility is related to an attenuated host response, once the infection has been 

initiated. The dose relationship between infectivity and age (young and old)/immunocompromised populations is ambiguous. There is a need to break down the host response into 

various components, including timing or onset of infection and disease severity. Such host 

response concepts have been acknowledged for food-borne bacterial infections (Buchanan, et al. 

2000) and need to be considered for the pathogen-host-environment interaction in MRA 

protocols. The Committee recommends: 

• Moving, or concisely reiterating, some or all of the susceptible population subsection 

in Chapter 2, to this chapter. 

• Adding a discussion of host response concepts such as timing or onset of infection 

and disease severity. 

4.5 Quality of Life 

The Committee does not believe that the quality of life discussion belongs in this 

document. Quality of life is not part of the risk assessment process, but rather could be part of 

the cost-benefit analysis. Although it could be used for comparison of different hazards and 

could be used as a metric for risk management, this is not the approach that EPA has used in the 

past. The Committee recommends removing this discussion from the document. 

5. Charge Question 5 – Risk Characterization (Chapter 5) 

Please comment on any improvements needed to achieve the necessary outputs or 

linkages between the components of the problem formulation, exposure, and health 

chapters to make risk characterization easier to conduct. Please comment on any 

additional approaches or methods to address uncertainty, variability, and sensitivity 

analysis of the various pathogen, health and exposure factors used in risk 

characterization. 

5.1 Improvements to the Linkages between the Planning & Scoping and Problem 

Formulation, Exposure, and Human Health Chapters 

This chapter is a good summary of risk characterization and of the models used in this 

area. It is written clearly and concisely. However, a number of improvements should be made to 

strengthen this important chapter. The Committee recommends: 

• Adding summaries at the ends of Chapters 3 and 4 about what pieces of information 

need to be brought forward from those respective chapters and folded into a risk 

characterization. 

• Summarizing the elements that need to be drawn from the earlier chapters and 

incorporated into a risk characterization at the beginning of Chapter 5. 

22 
Although these changes might seem simplistic or repetitive, they would help clarify the links 

between the components of the risk assessment and would improve the continuity of the 

document as a whole. 

The discussion of models occurs both in Chapter 4 (relative to dose-response) and in 

Chapter 5 (relative to risk characterization). This situation is somewhat confusing to the reader, 

since it is not always clear if the same or different models are applicable in each instance (i.e., 

does one use the same tool(s) to model dose-response and to characterize uncertainty and 

variability?). It also ends up being repetitive. The Committee recommends: 

• Trimming unnecessary detail and redundancy about the models in these chapters 

(perhaps, capturing the detail in a second more advanced MRA document and/or 

technical white papers); 

• Clarifying explicitly the different roles that the (same or different) models play in 

each of these aspects of the risk assessment. These changes will ensure that the 

reader understands what needs to be accomplished by the modeling exercise in each 

step. 

The Committee recommends that the Agency not use the terms “static” vs. “dynamic” 

modeling. These terms are not generally accepted in the MRA field. Both approaches described 

can be dynamic. We note that the Susceptible, Infected, Recovered (SIR) model takes into 

account the contagious nature of pathogens and of continuing, dynamic transmission. Many SIR 

models require a substantial number of assumptions to be made in order to derive a risk output. 

The Risk Characterization chapter should also explicitly refer back to the problem 

formulation, planning and scoping described in Chapter 2, especially Sections 2.1 and 2.2. It is 

important for the risk assessor to state at this stage whether, and how well, both the Statement of 

Concern and the Statement of Purpose and Objectives that were identified up-front in the risk 

assessment were, in fact, addressed. Although the topic of problem formulation is included as 

one of the items that should be addressed in the risk description summary at the end of the 

chapter (Section 5.5), it should be given greater emphasis elsewhere in the chapter as well. The 

Committee recommends: 

• Adding text to explicitly refer back to the problem formulation, planning and scoping 

described in Chapter 2, especially Sections 2.1 and 2.2. 

 

The title of Appendix D - MRA General Concepts, is misleading. One would expect this 

section to address principles and tools by which aspects of exposure, hazard, and dose-response 

assessment would be conducted. In fact, its entire focus is on Risk Characterization, but Chapter 

5 never makes reference to its existence or content. The Committee recommends: 

• Placing some of the topics that are in Appendix D that are not already addressed in 

Chapter 5 (i.e., the topics covered in Sections D.4-D.10), in an overview/summary 

format within Chapter 5. 

23 
5.2 Uncertainty, Variability, and Sensitivity Analysis 

Uncertainty, variability, and sensitivity analysis are important and deserve emphasis in 

this document. Section 5.4, which discusses these issues is good, and does not omit any 

significant approaches or methods. The Agency may choose to re-format this document such 

that this chapter presents the general principles of uncertainty, variability, and sensitivity analysis 

and the detailed discussions of the uncertainty, variability and sensitivity analyses required for a 

credible and complete risk characterization should be placed in the detailed follow-up document 

and/or, separately, in white papers on these topics. 

In any case, this document should acknowledge, and describe in an appropriate level of 

detail, the principles and practices regarding uncertainty, variability and sensitivity analysis 

presented in existing Agency documents that are, or could be, relevant to the conduct of MRAs. 

These include Guiding Principles for Monte Carlo Analysis (USEPA, 1997); Report of the 

Workshop on Selecting Input Distributions for Probabilistic Assessments (USEPA, 1999); 

Guidelines for Preparing Economic Analyses (USEPA, 2000a); Using Probabilistic Methods to 

Enhance the Role of Risk Analysis in Decision-making with Case Study Examples (External 

Review Draft) (USEPA, 2009). This latter document includes a case study on the Twodimensional Probabilistic Risk Analysis of Cryptosporidium in Public Water Supplies, with 

Bayesian Approaches to Uncertainty Analysis which was conducted in support of the 

development of the Long Term 2 Enhanced Surface Water Treatment Rule. This case study 

could serve as one of the case examples that the Committee is recommending to be added to the 

revised document. With the publication of so many guidelines, it would help to attain some 

degree of consistency in public policy if frameworks and approaches to risk assessment were 

consistent. This consistency could be achieved by referencing and using existing guidelines, 

noting exceptions or changes in the guidelines as dictated by legislative or executive direction. 

Guidelines are rarely recalled or revised based on new science, so changes to previous guidance 

need to be clearly and duly noted to improve the transparency of decision making within the 

agency. The Committee recommends: 

• Acknowledging, and describing in an appropriate level of detail, the principles and 

practices regarding uncertainty, variability and sensitivity analysis presented in 

existing Agency documents that are, or could be, relevant to the conduct of MRAs. 

The uncertainty analysis can be done using a Monte Carlo approach to examine each 

element in the MRA to determine contributions to overall risk outputs. Those factors (for 

example in exposure assessment) that may be contributing to most of the uncertainty of the final 

result may pinpoint where more data are necessary or where risk management decisions should 

be focused in the development of policy. Applying the HACCP system which has been used in 

food safety could be useful as part of this activity, whereby the risk characterization could be tied 

to best management approaches. 

5.3 Other Recommendations 

Overall, the Committee felt that Chapter 5 could benefit from significant editing to better 

focus the chapter and to assure that the essential elements of the chapter are not lost in the 

24 
tangential discussions and excessive detail. Specifically, the Committee recommends the 

following: 

• Edit Section 5.1.1 on Historical Context (minimize the verbiage) with the exception 

of the last paragraph on EPA policy; 

• Shorten Section 5.1.3 on Parsimony to one paragraph which defines the concept, and 

to state how to make the determination (drawing on the concepts outlined in 

Appendix G.1); 

• Change Section 5.1.2 to be consistent with EPA’s Risk Characterization Handbook

(USEPA, 2000b). (Because the document we were asked to review is primarily a 

document for EPA use, the terminology for risk assessment, and particularly for Risk 

Characterization, should be consistent with EPA’s risk assessment terminology. In 

some places in the document, the terminology appears to reflect the International Life 

Sciences Institute (ILSI) Framework for Microbial Risk Assessment rather than 

EPA’s own risk assessment terminology. For example, EPA’s Handbook does not 

define Risk Characterization as consisting of two major steps – risk estimation and 

risk description.) 

• Create a companion to Figure 9 that includes the same set of models but summarizes 

the pros and cons of each model choice (or the situations to which each model type is 

best suited); 

• Shorten the discussion on the various model types to focus on the pros and cons of 

each model type and when they should be used; 

• Remove the excessive detail on the models, such as Table 6 and Figure 12; 

• Shorten Section 5.2.3 or move it to a second more advanced MRA document and/or 

white papers. In particular, the lengthy literature review on Bayesian models on pp. 

84-86 should be removed and that section should be reduced to one paragraph. 

25 
REFERENCES 

Asano, T., Leong L.Y.C., Rigby M.G., Sakaji R.H. (1992). Evaluation of the California wastewater 

reclamation criteria using enteric virus monitoring data. Water Science and Technology, 26:1513-1524 

American Society of Civil Engineers (ASCE) and American Water Works Association (AWWA) (1990). 

Water Treatment Plant Design, McGraw-Hill Publishing Co., NY. 

Buchanan R.L., Smith J.L., Long W. (2000). Microbial risk assessment: Dose-response relations and risk 

characterization. International Journal of Food Microbiology, 58 (3), pp. 159-172. 

DuPont, H.L., Chappell, C.L., Sterling, C.R., Okhuysen, P.C., Rose, J.B., Jakubowski, W. (1995) 

The Infectivity of Cryptosporidium parvum in Healthy Volunteers. New England Journal of Medicine,

332 (13): 855-859 

Feazel, L. M., Baumgartner, L. K., Peterson, K. L., Frank, D. N., Harris, J. K., and Pace, N. R. (2009). 

Opportunistic pathogens enriched in showerhead biofilms. Proceedings of the National Academy of 

Sciences, 106(38):16393-16399. 

Haas C.N., Rose J.B., Gerba C., Regli S. (1993). Risk Assessment of Virus in Drinking Water. Risk 

Analysis, 13(5):545-552 

Jeong K.I., Zhang Q., Nunnari J., Tzipori S. (2010). A piglet model of acute gastroenteritis induced by 

Shigella dysenteriae Type 1. Journal of Infectious Diseases, 201(6):903-911. 

Kay, D., Fricker, C. (1997). Coliforms and E. Coli. Cambridge, United Kingdom: The Royal Society of 

Chemistry. 

Lindesmith L., Moe C., Marionneau S., Ruvoen N., Jiang X., Lindblad L., Stewart P., LePendu J., Baric 

R. (2003). Human susceptibility and resistance to Norwalk virus infection. Nature Medicine, 9:548–53. 

Law B.F., Adriance S.M., Joens L.A. (2009). Comparison of In Vitro Virulence Factors of 

Campylobacter jejuni to In Vivo Lesion Production. Foodborne Pathogens And Disease, 6(3): 377-385. 

Regli S., Rose J., Haas C.N., Gerba C. (1991). Modeling Risk from Giardia and Viruses in Drinking 

Water. Journal of the American Water Works Association, 83, 11, 76-84. 

Rose J.B., Dickson L.J., Farrah S.R., Carnahan R.P. (1996). Removal of Pathogenic and Indicator 

Microorganisms by A Full-Scale Water Reclamation Facility. Water Research, 30(11): 2785-2797. 

Sakaji, R.H., Godgrabe-Brewen, J. Lai, H.H. (1996). Interpreting Particle Counting Data from Full-Scale 

Water Treatment Plants. American Water Works Association Water Quality Technology Conference 

Proceedings 1995, November 12-15, 1995, New Orleans, LA, AWWA Denver CO. 

Slifko, T.R., Friedman, D., Rose, J.B., and Jankubowski, W. (1997). An In Vitro Method for Detecting 

Infectious Crytosporidium Oocysts with Cell Culture. Applied and Environmental Microbiology, 

63(9):3669-3675. 

26 
Slifko, T.R., Huffman, D.E. and Rose, J.B. (1999). A most probable assay for enumeration of infectious 

Cryptosporidium parvum oocycts. Applied and Environmental Microbiology, 65(9):3936-3941. 

Slifko, T.R., Huffman, D.E., Bertrand, D., Owens, J.H., Jakubowski, W., Haas, C.N. and Rose, J.B. 

(2002). Comparison of Animal Infectivity and Cell Culture Systems for Evaluation of Cryptosporidium 

parvum oocycts. Experimental Parasitology, 101:97-106. 

Soller, J., Olivieri, A.W., Eisenberg, J.N.S., Sakaji, R.H., Danielson, R. (2004). Evaluation of Microbial 

Risk Techniques and Applications. Water Environment Research Foundation, Alexandria, VA. 

U.S. EPA. (1997). Guiding Principles for Monte Carlo Analysis. U.S. Environmental Protection Agency, 

Risk Assessment Forum, Washington, DC, EPA/630/R-97/001. 

U.S. EPA. (1999). Report of the Workshop on Selecting Input Distributions for Probabilistic Assessments. 

Risk Assessment Forum, Washington, DC, EPA/630/R-98/004. 

U.S. EPA. (2000a). Guidelines for Preparing Economic Analyses. National Center for Environmental 

Economics, Washington, DC, EPA/240/R-00/003. 

U.S. EPA (2000b). Risk Characterization Handbook. Science Policy Council, Washington, DC, 

EPA/100/B-00/002.

U.S. EPA (2009). Using Probabilistic Methods to Enhance the Role of Risk Analysis in Decision-Making 

With Case Study Examples (External Review Draft). Risk Assessment Forum, Washington, DC,

EPA/100/R-09/001. 

Zhang W.P., Robertson D.C., Zhang C.X. , Bai W., Zhao M.J., Francis D.H. (2008). Escherichia coli

constructs expressing human or porcine enterotoxins induce identical diarrheal diseases in a piglet 

infection model. Applied And Environmental Microbiology, 74(18): 5832-5837. 

27 
APPENDIX – EPA CHARGE QUESTIONS 

28 
SAB REVIEW OF MRA PROTOCOL 

BACKGROUND 

Over the past decade, the Office of Science and Technology (OST) in the EPA’s Office of Water 

has been involved in the development of a Microbiological Risk Assessment (MRA) Protocol to 

better inform persons conducting EPA sponsored MRAs about available approaches, methods, 

and tools, thus enhancing the capability of the assessors to prepare successful products. Initially, 

OST enlisted the International Life Sciences Institute through a cooperative agreement to help 

develop a MRA framework based upon the specific or unique risk assessment factors that risk 

assessors need to consider in conducting MRAs in water media. Subsequently, the OST 

sponsored a number of workshops to identify existing or generally accepted developmental 

approaches, tools, methods, and procedures for application in populating the framework to 

establish the protocol for conducting MRAs, especially for water-based media (drinking water, 

recreational water, biosolids, shellfish growing water, etc.). 

At this time the OST has developed a draft MRA Protocol document that it believes captures the 

essential components for risk assessors to use to successfully conduct microbiological risk 

assessments in water media. The current Protocol focuses only on risk assessment components 

and does not broadly consider all aspects of risk management or risk communication although it 

is recognized that these features are essential components for conducting a successful risk 

analysis. After review by the EPA’s Science Advisory Board the OST will make essential 

modifications to the protocol and will then list this document on its website so that it will be 

available to all EPA staff and contractors involved in risk assessment as well as the general 

microbiology community. 

CHARGE QUESTIONS 

The following non-prioritized list of questions to the Science Advisory Board reviewers has been 

prepared to help EPA’s Office of Water, Office of Science and Technology, improve the MRA 

protocol’s effectiveness for users. It is envisioned that the SAB Reviewers will provide new 

insights and technical additions or modifications to improve the ease of use, technical robustness, 

clarity, and efficacy of the MRA protocol as a resource for guidance or support in conducting 

risk assessments. The focus of the MRA Protocol is to support professional microbiologists and 

risk assessors conducting water-based microbial risk assessments on conventional waterborne 

microbial pathogens and the water route of exposure. 

1. Overarching Considerations: 

OST would like this Protocol to provide a comprehensive and robust suite of approaches, 

tools, methods, and procedures to meet EPA’s overall needs in preparing for, and 

conducting typical MRAs. OST would also like the Protocol to be informative, easy to 

use and understand, and useful to outside stakeholders (states, communities, utilities, 

industry, and impacted parties). 

Please comment on the following: 

29 
a) utility of the Protocol for meeting EPA’s overall needs, particularly on the 

comprehensiveness and robustness of the Protocol; 

b) flow and continuity within and between chapters; 

c) ease of use and utility for outside stakeholders; 

d) any changes or enhancements to the Protocol to ensure it meets the needs of EPA 

and outside stakeholders. 

2. Planning/Scoping and Problem Formulation – Chapter 2: 

Please comment on the utility of this Chapter to ensure that risk assessments are 

adequately conceptualized and planned appropriately to address risk management’s issues. 

Please provide any recommendations for enhancing the utility of this Chapter. 

 Please comment on any enhancements or expanded guidance needed to allow users to 

prepare and conduct risk assessments to address a broad range of types of types of risk 

management questions. Examples of types of EPA uses of MRA may be: 

a) approaches to mitigation of environmentally-based microbial pathogen exposure 

risks; 

b) determination of acceptable health risks; 

c) identification of different exposure factors/routes; 

d) identification of microbial-based hazards in disease outbreaks; 

e) development and prioritization of research needs; 

f) competing risks ranking. 

 

3. Exposure – Chapter 3:

Please comment on any additional exposure tools, methods, or approaches that should be 

included to ensure a robust approach to adequately determining the microbial occurrence 

and human exposure factors relevant to health risks from water. This includes support for 

the estimation of the magnitude, frequency, duration, and also additional types of 

exposure to microbial pathogens by the water route, as well as the range of characteristics 

of the exposed population and their exposure profiles. 

4. Human Health Effects – Chapter 4:

Please comment on any additional scientifically accepted dose response models 

(including advanced and validated threshold, empirical, or mechanistic models) which 

should be included as tools for determining human dose responses from waterborne 

exposures via oral, inhalation, and dermal routes, especially for low dose extrapolation. 

Please comment on whether any specific animal or in vitro dose response protocols, 

models, and methods should be included in this Chapter. If so, please describe their 

applications and limitations in establishing human dose response curves. 

5. Risk Characterization – Chapter 5:

Please comment on any improvements needed to achieve the necessary outputs or 

linkages between the components of the problem formulation, exposure, and health 

chapters to make risk characterization easier to conduct. 

30 
31 

Please comment on any additional approaches or methods to address uncertainty, 

variability, and sensitivity analysis of the various pathogen, health and exposure factors 

used in risk characterization.