### Appendix A. 5 : Evaluation of the Suitability of Individual Combinations of

Indicators and Methods for Different Clean Water Act Programs

COMPLETION NOTICE

Evaluate the suitability of individual combinations of indicators and methods for different CWA pro grams (P18)

Summary of the Study This study was conducted to evaluate the strengths and weaknesses of both rapid and culture indicator/method combinations to method performance criteria and their appropriateness for different Clean Water Act programs. EP A identified and prioritized key concerns related to indicator/method performance and identified eight performance criteria: 1) limit of detection; (2) sensitivity; (3) specificity; (4) precision; (5) percent false positives and false negatives; (6)

abilit y to differentiate fecal sources; (7) performance in different waterbody types; and (8) indicator/illness health relationship established. The indicator/methods evaluated were methods

that showed a correlation to health in the freshwater and/or marine wate r epidemiological studies conducted by EPA under the National Epidemiological and Environmental Assessment of Recreational (NEEAR) Water Study or used to support the development of the 1986 ambient water quality criteria. Four methods (Enterococcus quantit ative polymerase chain reaction (qPCR), Bacteroidales qPCR, EPA Method 1600 and EPA Method 1603) were quantitatively evaluated with respect to the performance criteria. EPA evaluated the performance criteria based on published literature for the various in dicator/method combinations.

Additionally, qPCR and culture methods were evaluated qualitatively with respect to several method attributes/considerations for implementation purposes, and how well each method works for each Clean Water Act program. QPCR m ethods were evaluated relative to culture methods on the following evaluation criteria that were important to the CWA programs: (1) ease of use, (2) low cost, (3) time to results, (4) allowance for use of historical data for model development, (5) ability to demonstrate effectiveness of treatment from source to beach, (6) count/signal associated with human health risk pathogens, and (7) precision/accuracy.

Summary of Findings The quantitative evaluation of performance criteria revealed:

Enterococcus

2 3S qPCR (diluted crude extract) Bacteroidales

16S qPCR (diluted crude extract) Method 1600 Method 1603

50% Limit of

detection in CCE[CSE] 65 [904] 99

[1,380] Not reported Not reported

False positives 0%,19% 0% 6% 6% False negatives 1.1

- 4.5%, 2% 1.1

- 4.5% , 2% 6.5% 5% Specificity 100% 100% Not reported Not reportedUnited States Office of Water EPA 823

- R

- 10

- 004
Environmental Protection (4303T) December 2010 Agency



# Evaluation of the Suitability of Individual

Combinations of Indicators and Methods for Different Clean Water Act ProgramsOffice of Science and Technology

Office of Water U.S. Environmental Protection Agency Washington, DCEvaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

3 

## Table of Contents

List of Tables

................................ ................................ ................................ ................. 4 List of Acronyms and Abbreviations

................................ ................................ ............. 4



### Section

1 Introduction/Project Description

.. ................................ ................................ .................. 5

2 Background

... ................................ ................................ ................................ ................. 6

2.1 Indicator/Methods Evaluated in the 2003

- 2005 NEEAR Study

.................. 7

2.2 Indicator/Methods Evaluated in the 2007/2009 Marine Epi demiological Studies

................................ ................................ .............. 8

2.3 Other Indicator Methods

................................ ................................ .............. 9

3 Quantitative Evaluation of Indicator/Method Combinations Based

on Performance Criteria

................................ ................................ ............................... 11

3.1 Established Health Relationship

................................ ................................ 12

3.2 Limit of Detection

................................ ................................ ...................... 14

3.3 Precision

................................ ................................ ................................ ..... 14

3.4 Sensitivity

................................ ................................ ................................ .. 15

3.5 Specificity

................................ ................................ ................................ .. 15

3.6 Ability of Differ entiate Fecal Sources

................................ ....................... 16

3.7 Performance in Different Waterbody Types

................................ .............. 1 6

4 Evaluation of Appropriateness of Indicator/Methods for CWA Purposes

.................. 17

5 Study Limitations

................................ ................................ ................................ ......... 18

6 References

................................ ................................ ................................ .................... 1 8Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

4 

## List of Tables

Table 1 Indic a tors/Methods Tested by EPA in NEEAR Epidem iological Studies

............... 9

Table 2 Evaluation of Rapid and Culture Methods

................................ ............................ 13

Table 3 Evaluation of Rapid and Culture Methods for Clean Water Act Programs Implementation Purposes

................................ ................................ ....................... 17



## List of Acronyms & Abbreviations

BEACH Act Beaches Environmental Asses sment and Coastal Health

Act

CCE calibrator cell equivalents

CE cell equivalents

CFU colony forming unit

CLAT culture, latex agglutination, and typing

CSE calibrator sequence equivalents

CT Cycle Threshold

CWA Clean Water Act

DNA Deoxyribonucleic acid

EPA Environmental Protection Agency

GI gastrointestinal

Ml milliliter

MST Microbial Source Tracking

mTEC modified membrane

- Thermotolerant Escherichia coli agar NEEAR National Epidemiological and Environmental Assessment

of Recreationa l

NP DES National Pollutant Discharge Elimination System

PBS Phosphate B uffered S aline

PCR Polymerase Chain Reaction

POTW Publicly

- Owned Treatment Works qPCR Quantitative Polymerase Chain Reaction

RNA Ribonucleic acid

RSD Relative Standard Deviatio n SCCWRP Southern California Coastal Water Research Project

TMDL Total Maximum Daily Load WQS Water Quality StandardsEvaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

5 1.0 INTRODUCTION / PROJECT DESCRIPTIO N

Project P18 as described in Environmental Protection Agency’s (E PA’s ) Critical Path Science

Plan is to “ identify and evaluate indicator/method combinations for strengths and limitations with respect to fecal source identification (human, animal); and performance in different waterbody types. The important features of each indicator/method will b e described and the strengths and weaknesses of those features will be explained and evaluated. The ideal set of features will be proposed so that the indicator/methods can be compared to a hypothetical ideal

indicator/method. The indicators/methods under consideration will be ranked for each feature with respect to ability to differentiate fecal sources, performance in different water body types and appropriateness for different CWA purposes. EPA plans to use the results of this evaluation to inform the de cisions regarding which indicator/methods will be included in the new or revised

criteria and under what conditions those indicator/methods will be recommended .” The purpose of this white paper is to report on the result of the evaluation of indicator/meth ods assessed by EPA in support of the development of new or revised recreational water quality criteria for

bacteria.

Section 304(a)(1) of the CWA directs EPA to publish criteria for water quality accurately reflecting the latest scientific knowledge

. Th e criteria published by EPA under section 304(a) are intended to provide guidance to States in setting water quality standards ( WQS ) to protect public health as well as to maintain and restore water quality and ecosystem integrity. CWA 304(a) criteria are typically expressed as numbers (i.e., concentrations of pollutants) or narratives

that EPA recommends that states put into their WQS to protect waters for aquatic life, wildlife, consumption of aquatic organisms by humans, and primary contact recreation. C WA section 303 requires each state to adopt WQS for all waters of the state and to review, and revise them as necessary, every 3 years. Once adopted and approved, WQS are binding CWA regulatory

standards and effective for the following CWA purposes:

**•** Wate r Quality Assessments. Sections 303(d) and 305(b) provide that states are required to assess their waters on a regular basis to determine if they are meeting WQS. The

states’ water quality criteria are an essential baseline against which states determine whether particular waters are “ impaired. ”

**•** Total Maximum Daily Loads (TMDL)

. TMDL calculations are required for all waters that have been listed as "impaired" under section 303(d). A TMDL specifies the maximum amount of a pollutant that a waterbody can receive and still meet WQS, and "allocates" pollutant loadings among point and non

- point pollutant sources. TMDL calculations for impaired waters must be written to implement the applicable State WQS.

**•** National Pollutant Discharge Elimination System (N PDES). NPDES permits are required under section 402 for point source discharges of pollutants to waters of the United States. NPDES permits must include effluent limitations more stringent than required by technology regulations

, if necessary

, to meet wa ter quality standards, which

include state water quality criteria.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

6 **•** Non

- point Source Program. Water quality standards (including criteria) play a similarly important role under the CWA section 319 non

- point source program as part of the listing

and T MDL processes to determine whether best management practices or other risk management control strategies are needed to address non

- point source pollution.

**•** Recreational Water Monitoring and Notification. A State’s recreational water quality

criteria are used in beach monitoring and notification programs. States and beach managers typically make decisions about whether to issue advisories or closure notices by measuring the results of their monitoring against their WQS.

EPA’s current recommended criteria for bacteria are based on bacterial indicator organisms

– Escherichia coli ( E. coli ) and Enterococcus

– which generally do not cause illness, but have characteristics that make them good indicators of fecal contamination, and thus by inference, of pathoge ns capable of causing human illness such as acute gastrointestinal illness. EPA’s recommended bacteria criteria are intended to be adopted by states into state water quality standards to protect waters designated for recreational use activities such as swi mming.

2.0 BACKGROUND

Beginning with the freshwater National Epidemiological and Environmental Assessment of Recreational (NEEAR) Water Study in 2003, EPA has been evaluating recreational water quality using traditional culture methods and rapid genetic methods based on qPCR

. EPA’s research goal is to determine the indicator /method combination with the best correlation to illness in fresh

and marine recreational waters. EPA wa s also responding to a provision in the Beaches

Assessment and Coastal Health Act (BEACH Act) of 2000 which required the Agency to provide additional information for use in developing “appropriate, accurate, and expeditious and

cost

- effective methods (including predictive models) for detecting in a timely manner in coastal

recreatio n waters the presence of pathogens that are harmful to human health.”

A culture method for bacteria is one that allows direct propagation of the bacteria, and for the

visible (to the naked eye) increase in the number of cells, which can allow for enumerat ion via solid ( i.e ., membrane filtration technique) or liquid media (i.e., multiple tube fermentation technique). The currently approved EPA culture methods involve culturing and enumerating fecal indicator bacteria ( E. coli or Enterococcus spp. ). Current culture

- based m ethods require 24 to 48 hours to obtain results (Wade et al ., 2008).

Rapid methods allow for the direct measurement of genetic material such as DNA without the

need for incubation and can provide results between 2 to 6 hours from time of re ceipt by the laboratory to time results are available

. Rapid methods are being evaluated for the development of new or revised criteria because they provide more timely results than traditional culture

methods

, allow ing beach managers to make beach notific ation decisions on the same day water samples are collected.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

7 2. 1 Indicator / Methods Evaluated in the 2003

- 2005 NEEAR Stud y

T he rapid indicator/methods evaluated in the 2003

- 200 4 freshwater and 2005 (Biloxi, MS)

marine NEEAR studies were commercially

- available test methods or validated or experimental research protocols. The methods included in the NEEAR study met the following criteria: (1) results could be obtained within a few hours ; (2) Enterococcus or the new potential indicator, Bacteroidales, was detected by the

method ; (3) the detection limit of the method was sufficiently low to allow detection of indicators in the majority of samples from recreational water environments ; (4) the method was sufficiently resistant to water sa mple inhibitory effects to allow

detection of indicators in the majority of samples from recreational water

environments

.

The four tests included in the 2003

- 200 5 freshwater and marine NEEAR Water Stud ies are described below . The following description s are based on the methods available at that time and may not reflect recent changes to these methods.

1. Quantitative Polymerase Chain Reaction ( q PCR) Method , two new rapid gene probe methods developed by Dr. Richard Haugland of the EPA (Haugland et al

. , 2005) (hereafte r referred to as the EPA qPCR Assay) , were used to detect Enterococcus and

Bacteroid al es in water samples based on the collection of these organisms on membrane filters, extraction of their total DNA, and q PCR amplification ( i.e. , a process whereby

target DNA strands are doubled in each cycle of amplification) of a genus

- specific ( Enterococcus ) or order

– specific (Bacteroidales) rDNA sequence using the TaqMan TM

system.

2. RAPTOR Fiber optic Biosensor

, a portable, automated fiber optic biosensor develope d by Research International, Woodinville, Washington that can be used to detect microbiological and chemical analytes in water samples. The RAPTOR Biosensor, used in the summer of 2003, was removed from the study in 2004 because the capture and

detection antibodies were not sensitive enough.

3. Luminex 100 System , a compact flow cytometer

, developed by the Luminex Corporation, Austin, Texas and MiraiBio, Alameda, California, that analyzes immunoassays, complex genetic analyses, and/or enzymatic assays thro ugh the use of optics, fluidics, and advanced signal processing. The Luminex method was removed from further study in 2004 because it lacked adequate sensitivity

.

4. EPA Method 1600 is the EPA

- approved membrane filter method using mEI Agar for the

detec tion of Enterococcus in recreational water

. This method was included in the study

as a reference method for Enterococcus

.

Of the new methods described above, only the q PCR method produced results that were

sufficient to be included for further evaluation in the 2007 marine epidemiological studies in Fairhope, Alabama and Goddard, Rhode Island

. The antibodies used in the RAPTOR andEvaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

8 Luminex 100 methods lacked sensitivity for Enterococcus and Bacteroid al es but these methods may be useful if new antibodies c an be produced to increase the sensitivity and specificity of the

methods. Both the RAPTOR and Luminex methods were dropped from further evaluation.

In the 2003

- 2005 freshwater and marine studies E. coli methods were not tested . EPA did not

include an E. coli qPCR method because at that time, a method was not available. Since no qPCR method was available for comparison with the culture method, the culture method was not

included.

2. 2 Indicator/ Methods Evaluated in the 2007 /2009 Marine Epidemiological St udies

In 2007, EPA conducted two marine epidemiological studies in Fairhope, Alabama and Goddard,

Rhode Island at beaches predominantly impacted by treated Publicly

- Owned Treatment Works (POTW) effluent. In 2009, EPA initiated two additional epidemiologic al studies at a tropical beach impacted predominantly by human sources of fecal pollution, and a beach impacted by urban runoff using several rapid and cultural indicator/method combinations. The tropical epidemiological study was conducted at Boquerón Bea ch in Puerto Rico; and the urban runoff study at Surfside Beach in South Carolina. Additional information on the site selection process for the urban runoff epidemiological study can be found at http://www.e pa.gov/nheerl/neear/

.

Table 1 presents a comprehensive list of the methods tested in the EPA epidemiological studies conducted to support the development of new or revised recreational water quality criteria. Several methods were tested at multiple epide miological sites dating back to the freshwater Great Lakes studies that were initiated in 2003 as part of the NEEAR Water Study. E. coli

culture was not included in the methods tested in marine waters because E. coli is not stable and cells lyse easily in saltwater environments.

The Fecal Bacteroides assay was evaluate d because of its potential as a human associated marker. However, the e valuation of th i s indicator/method combination in the 2007 epidemiological studies resulted in its elimination due to i ts lack of human source specificity and the relatively high proportion of sample where target organisms were not detectable. A similar problem was encountered with the E. col i qPCR assay. However, evaluation of this assay was continued in 2009 in conjunct ion with an added DNA concentration and purification procedure

. The male

- specific F+ Coliphage methods were eliminated due to the low density of the indicator

in surface water.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

9 1 Haughland et al. , 2005

2 USEPA 2002

3 D ick and Field, 2004

4 S iefring et al ., 2008

5 Converse et al

., 2009

6 Shanks et al., 2009

7 Haugland et al., 2010.

8 C hern et al ., 2009

9 Chern et al., nd

10 Love and Sobsey, 2007

11 USEPA 2001

2. 3 Other Indicator Methods

2.3.1 Microbial Source Tracking Methods

There is uncertainty about the risk to human health associated with non

- human sources of fecal pollution. The ability to differentiate sources of fecal contamination site

- specifically may be important for an accurate assessment of the risks to human health from domestic

, agricultural

animals and wildlife. Source differentiation is also im portant with regard to CWA monitoring and assessment (§303[d] and §305[b]) and TMDL programs. S ome s tates have expressed a desire to be able to adjust the applicable criteria based on data that the indicator levels present in the waterbody are not due to human sources of fecal pollution. The stated concern is that o nce waters are listed as impaired, states must then expend resources to develop a TMDL to restore

waters that pose less of a risk to human health. In this context, mi crobial source tracking (MS T) methods are useful to supplement sanitary survey investigations (i.e., to identify sources of

contaminants or TMDL sources) and for risk analysis (human vs. non

- human vs. domestic animals). Most MST methods attempt to identify specific fecal sources to assist with prioritizing polluted areas for restoration. Table 1

– Indicators/ Methods Tested by EPA in NEEAR Epidemiological Studies

M ethods EPA Epidemiological Studies Where Methods

Were Used

Enterococcus spp. qPCR Freshwater: 2003, 2004 1

Marine: 2005, 2007, 2009 1

Enterococcus culture (EPA Method 1600) Freshwater: 2003, 2004 2

Marine: 2005, 2007, 2009 2

Total Bacteroidales spp. qPCR Freshwater: 2003, 2004 3

Marine: 2005, 2007, 2009 4

Fecal Bacteroides spp. qPCR Marine: 2007 5

Human

- specific Bacteroidales markers Marine: 2009 6,7

E. coli qPCR Marine: 2007 8 , 2009 8,9

Clostridium spp. qPCR Marine: 2007, 2009 8

F+ RNA Col iphage CLAT assay Marine: 2007 10

F+ DNA Coliphage CLAT assay Marine: 2007 10

F+ Coliphage 24

- hr SPOT assay Marine: 2007 11Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

10 EPA has been conducting research on rapid MST methods and has evaluated qPCR and PCR genetic markers of human and bovine pollution. This research includes a performance evaluation

of (1) seven PCR and qPCR assays targeting Bacteroidales genes reported to be associated with either ruminant (goat, sheep, deer and others) or bovine feces (Shanks et al ., 2010a) ; and (2) five PCR and ten qPCR assays targeting Bacteroidales genes reported to be associated with human feces (Shanks et al ., 2010b). The bovine assay study found large discrepancies in the performance of qPCR assays across different bovine populations; and recommended that the use

of bovine

- associated MST applications require an a priori charact erization of each watershed due to variability in genetic marker abundance and prevalence between populations. Study results also suggest that some assays are more suitable for the characterization of fecal contamination than others, and that the assay of choice can vary from one bovine population to another.

The human assay performance study included the evaluation of two qPCR assays for quantification of recent ly developed human

- associated genetic markers targeting putative

Bacteroidales

- like cell sur face

- associated genes (Shanks 2009), seven qPCR assays for quantification of 16s rRNA gene markers from human

- associated Bacteroidales species (Haugland et al ., 2010), and several other published assays. Some assays showed human source specificity levels exceeding 97% when tested against a panel of reference fecal samples originating from cattle, poultry, swine, and various wildlife animal sources. Based on assay performance and the prevalence of DNA targets in a collection of reference untreated sewage

sa mples collected from 54 different waste water treatment facilities in the United States , this research suggests a potential application for human

- associated quantitative methods for monitoring fecal pollution in ambient waters.

EPA is also conducting rese arch to identify genetic sequences that could form the basis of chicken and seagull

- associated MST methods as specific fecal source assays. EPA has selected

PCR

- based assays that could uniquely identify avian sources (primarily chicken and seagull) of

feca l pollution. Four assays are currently being evaluated and additional molecular data are being collected to further validate existing assays and determine if additional assays can be developed

. If EPA determines that chicken and seagull specific assays ca n be developed , EPA will evaluate chicken and seagull

- associated MST PCR

- based assays for sensitivity and specificity using reference fecal samples and environmental water samples with known sources of fecal

contamination

.

2.3.2 Chemical Methods

In addition t o the microbial indicators discussed in sections 2.2 and 2.3 , EPA also tested water samples for chemical constituents. The chemical constituents tested included coprostanol, a product of cholesterol metabolism in feces ; and urobilin, a bile pigment found in human feces and urine. Additionally, 48 different chemicals distinctly associated with humans that could be markers of human sewage or human fecal contamination (e.g., caffeine, cotininine, Acetaminophen, and codeine) were tested

. Preliminary data sugg est that chemical detections were not very frequent at freshwaters beaches, but were more frequent and occurred at higher

concentrations at marine beaches tested in 2005 and 2007. P reliminary analyses did not show a consistent relationship with health out comes
, and the c hemical measurements were discontinued in the 2009 epidemiological studies due to a lack of funding.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

11 2.3.3 Southern California Coastal Water Research Project Research

Southern California Coastal Water Research Project (SCCWRP) has been conductin g research

on indicator / methods with the goal of developing rapid methods that can augment or replace existing traditional culture methods for one or more types of indicator bacteria. SCCWRP is assessing water quality by measuring both traditional and non

- traditional indicators. Traditional

indicator methods include total coliform, fecal coliform, E nterococcus using membrane filtration

and Enterolert chromogenic substrate method

, E. coli using Colilert chromogenic substrate

method and membrane filtration

, and Coliphage

. Nontraditional measurements i nclude rapid methods for quantifying E nterococcus and E. coli

, Bacteroides

, Bacteroides thetaiotamicron

, adenovirus, norovirus, enterovirus and C oliphage (somatic and F+), among others. These

indicator methods a re being tested by SCCWRP as part of a 4

- year project on Rapid Bacterial Indicator Development and then validated in three large

- scale epidemiological studies conducted

at Avalon Bay B each, Malibu Surfrider Beach and Doheny State Beach in southern Califor nia

. As part of the epidemiological studies, SCCWRP has analyzed more than 4

, 000 water samples using 36 different analytical methods (SCCWRP, 2010 )

.

As part of its 2010/2011 Research Plan, SCCWRP is conducting a 3

- year study to assess which

source identi fication methods are optimal for differentiating fecal sources with the goal of bringing together a team of water quality experts experienced in source identification methods to

create a source identification manual , impl emen t selected protocols at several beaches of high

interest to California

, and then transition source identification capabilities to local laboratories.

3.0 QUANTITATIVE EVALUATION OF INDICATOR/ METHOD COMBINATIONS

BASED ON PERFORMANCE CRITE RIA

EPA’s approach to evaluate the suitability of individual co mbinations of indicators and me t h ods for different CWA programs (P18) was to conduct two separate analyses

– a quantitative analysis

of indicator/method combinations based on a set of performance criteria ; and a qualitative

evaluation of t he appropriateness of the qPCR and culture method s for each CWA program.

Enterococcus qPCR

, Bacteroidale s qPCR

, Method 1600 (membrane filtr ation method for

Enterococci

, ) and Method 1603 (membrane filtration method for E. coli using modified mTEC agar) were evaluated because they have an association with illness in swimmers ( Fleisher et al

., 2010; Wade et al ., 2008; Wade et al ., 2006; and USEPA 1986 ) (Table 2)

. The performance

criteria used to evaluate these methods are as follows: (1) e stablished health relationship, (2) limit of detection, (3) sensitivity, (4) specificity, (5) precision, (6) percent false positive and (7) percent false negative. Table 2 shows the results of the quantitative evaluation of the qPCR and

culture methods. The methods were al so e valuated on their ability to differentiate fecal sources and performance in different waterbody types.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

12 3.1 Established Health Relationship

3.1.1 Enterococcus Culture and qPCR

A d emonstrated relationship between indicator concentration and gastroint estinal (GI) illness is the most important criteri on for selection of the method for new or revised ambient water quality

criteria . The method must demonstrate a positive relationship between illness in humans and with indicator levels ( i.e ., as indicator levels increase, risk of illness increases) in recreational

waters

.

Enterococci are commonly found in the feces of humans and other warm

- blooded animals. Although some strains are ubiquitous and not related to fecal pollution, the presence of

enterococci in water is an indication of fecal pollution and the possible presence of enteric pathogens. Results from EPA’s recent epidemiological studies showed a correlation between qPCR measured E nterococc us levels to GI illness at freshwater beaches impacted by P OTW

sources (Wade et al. , 2008)

. A ddition ally , the recent studies point to significantly increased illness rates among swimmers 10 years and younger exposed above 35 CFU of Enterococcus using the culture method compared to non

- swimmers in the Great Lake s study, but the trend is not as strong (or evident) at low exposures. QPCR cell equivalent (CE) levels were a stronger predictor of GI illness than the CFU measure in the NEEAR study (Wade et al ., 2008) . Other

studies have shown enterococci levels measure d by culture methods correlated with GI illness

levels in marine and fresh waters (Au

- Yeung et al

., unpub ; Zmirou et al ., 2003)

. The meta

analysis of the Epibathe studies found an increased risk of gastroenteritis in both marine and

freshwater water sites when bathers were exposed to enterococci concentrations higher than 100

enterococci/100 ml in marine water and 200 enterococci/100 m l in fresh waters (Au

- Yeung et
al

., unpub ).

3.1.2 E. coli Culture and qPCR

E pidemiological studies conducted by EPA and SC CWRP separately found that there was no relationship between E. coli qPCR and GI illness in marine waters. There are no recent EPA data on the health relationship between E. coli culture and swimming

- related illness. E. coli

culture was not included in t he list of methods tested by EPA in the NEEAR epidemiological

studies. However, i n epidemiological studies conducted by EPA in the 1970s, there was an

association between E. coli culture and GI illness in fresh water but not marine waters

.

More recent da ta from randomized controlled trial epidemiological studies conducted in Europe

found a weak relationship between E. coli culture and GI illness in freshwater only

. A meta

analysis of the ( Epibathe ) studies found an increased risk of gastroenteritis in bot h marine and fresh water sites in bathers exposed to E. coli concentrations higher than the level that represents

the 2006/7/EC 1 “excellent quality” criteria (< 500 E.coli /100 ml) (Au

- Yeung et al

., unpub ).

1 Directive 2006/7/EC of the European Parliament and of the Council of 15 February 2006 concerning the management of bathing water quality and repealing Directive 76/160/EEC. The purpose of this Directive is to preserve, protect and improve the quality of t he environment and to protect human health by complementing Directive 2000/60/EC. This Directive lays down provisions for: (a) the monitoring and classification of bathing water quality; (b) the management of bathing water quality; and (c) the provision of information to the public on bathing water quality.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

13 Table 2

– Evaluation of Rapid and Culture Methods

Performance

Evaluation Criteria Rapid Methods Culture Methods

Enterococcus 23S qPCR

Diluted crude extract Bacteroidales 16S qPCR

Diluted crude extract Method 1600 ( Enterococcus by MF) Method 1603 ( E. coli by MF)

50% Limit of detection (CCE) 65 1 99 1 NR NR

50% Limit of detection (CSE) 904 1 1,380 1

− −

Sensitivity 100 % 2 NR NR NR Specificity 100% 3 100% 4 NR NR Precision (log 10 standard deviation) 0.24 4 0.20 4 2.2%

- 18.9% 5 25.9% 6

False positives 0% 1 , 19% 7 0% 1 6% 5 6 % 6

False negatives 1.1

- 4.5% 1 , 2.0% 7 1.1

- 4.5% 1 , 2.0% 7 6.5% 5 5 % 6

Health relationship

established Yes 8

, 9 Yes (Marine) 9 Yes Yes (Freshwater)

Note s :

Results presented in this table may be modified as additional information is collected and peer reviewed. For Method 1600 (Enterococcus by MF) and Method 1603(E. coli by MF) , performance criteria are based on the results of the inter

- laboratory validation in disinfected wastewater matrices. NR = Not reported.

1 Chern et al. , 2009

2 Ludwig and Schleifer, 2000

3 Frahm and Obst, 2003

4 Siefring et al., 2008

5 USEPA, 200 6a

6 USEPA 2006b

7 Haughland, et al., 2005

8 Wade et al. , 2008

9 Wade et al., 2010Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

14 3.1.3 Bacteroidales qPCR

Bacteroidales is a group of anaerobic bacteria commonly found in the gut of humans. Bacteroidales densities, as measured by qPCR, are orders of magnitude higher in raw waste

streams than enterococci or E. coli densities due in part to its greater abundance in feces. There are qPCR primers available that target the ge neral Bacteriodales population and other primers that are thought to target human

- specific strains. A positive relationship has been observed between general Bacteroidales and GI illness in marine waters in the NEEAR epidemiological studies (Wade et al ., 2 0 10 ).

3.2 Limit of detection

3.2.1 qPCR

The limit s of detection of the Enterococcus and Bacteroidales qPCR methods have been reported

as the estimated number of target organism calibrator cell equivalents (CCE) or target organism calibrator sequence e quivalents (CSE) that need to be present per total water sample filter extract in order to allow detection in 50% of analyses of that extract by the specified method (Chern et
al ., 2009). Chern et al. (2009) reported mean limits of detection of 65 CCE (904 CSE) for Enterococcus qPCR and 99 CCE (1,380 CSE) for Bacteroidales qPCR.

3.2.2 Cul t ure

The limits of detection of the Enterococcus and E. coli culture methods (Method 1600 and Method 1603, respectively) have not been reported.

3.3 Precision

3.3.1 q PCR

Precision of a qPCR method is the est imate of total variability of Cycle Threshold (CT) measurements obtained in analyses of common water sample filter extracts by single or mult iple

laboratories. It can be expressed in terms of standard deviation, i ncorporating between lab,

between run and random error estimates.

At this time

, EPA has completed a Single Laboratory Validation Study for Enterococcus qPCR and Bacteroidales qPCR. A Multi

- Laboratory Validation Study is on

- going ; however, the results

wi ll not be available until 2012 and are therefore, not presented or discussed in this report. Estimates of method precision (expressed as log 10 standard deviations of CCE) from analyses of
10,000 Enterococcus and Bacteroides cells spiked into 51 different f resh and marine water samples have been reported as 0.24 and 0.20, respectively (Si e fring et al ., 2008).

3.3.2 Culture

P recision for Method 1603 has been characterized by laboratory

- specific relative standard deviations (RSDs) from disinfected wastewater samples spiked with laboratory

- prepared. For Method 1603, the within

- laboratory pooled RSD was 25.9% (USEPA 2006b). P recision estimates in the range of 2.2%

- 18.9% has been reported for Method 1600 (USEPA 2006a)

.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

15 3.4 Sensitivity

Sensitivity can be expr essed as a percentage of targeted species that are detected by a method.

3.4.1 qPCR

It has been reported that the Enterococcus qPCR method detects all validly described species of Enterococcus ( Ludwig et al ., 2000). For the Bacteroidales qPCR method, th ere are no publish data available on the sensitivity of the method.

3.4.2 Culture

For the Enterococcus and E. coli culture methods there are no published reports available on method sensitivity. However, as part of the formal QA process for each laborato ry that uses Methods 1600 and 1603, the method s recommend that laboratories develop a statement of accuracy for method by calculating the average percent recovery and the standard deviation of the percent recovery (USEPA 2006a, b)

.

3.5 Specificity

Speci ficity is t he ability of a method to select and or distinguish the target bacteria under test from other bacteria in the same water sample. The specificity characteristic of a method is usually reported as the percent of false positive and false negative r esults. In the following sections, data are provided on the specificity, false positive and false negative rates for

Enterococcus qPCR, Bacteroidales qPCR, Method 1600, and Method 1603.

3.5.1 qPCR

The specificity of the Enterococcus qPCR method was estim ated from experimental analyses of five closely related non

- Enterococcus species showing at least 10,000 times higher limit of detection when compared to Enterococcus species (Frahm and Obst, 200 3 )

. For the

Bacteroidales qPCR method specificity was experim entally assessed using representative species of the related bacterial classes, Flavobacteria and Sphingobacteria (Siefri n g et al

. , 2008).

The percent false negatives for the Enterococcus and Bacteroidales qPCR methods have been reported as the percentage of water sample filter extracts that fail salmon DNA sample processing control assay quality control criterion of > 3 CT units higher than the mean of associated calibrator samples in the analysis of NEEAR study samples (Haugland et al ., 2005;

Chern et al ., 2009). Additionally, Griffith et al. (2007) reported false negative results of 58% for enteroccocci qPCR extracted using a multi

- step purification and concentration process after bead beating and 29% for enterococci qPCR bead beaten method. The a nalysi s for false negative rates was conducted relative to the State of California’s (AB411) standard of 104 cells/100 ml for enterococci (Griffith et al., 2007).

Percent false positives for the qPCR methods have been reported as the percentages of negative

con trol samples, consisting of clean filters that were subjected to the entire method including DNA extraction and qPCR analysis, that gave positive detection of target sequences (i.e.

, a true logarithmic amplification trace) during analyses of NEEAR study sa mples (Haugland et al

.,Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

16 2005; Chern et al ., 2009). It should be noted that the mean CT value from the false positive analyses shown for the study cited in footnote 7 in Table 2 was 43.65 which would not be detected as positive in a more recent version of t he method where only 40 thermal cycles are run.

Additionally, Griffith et al. (2007) reported false positive results of 3% for enteroccocci qPCR extracted using a multi

- step purification and concentration process after bead beating and 9% for

enterococc i qPCR bead beaten method. For E. coli qPCR, the false positive results were 0% before and 27% after adjustment for amplification efficiency. The analysis for false posi tive rates was conducted relative to the State of California’s (AB411) standard of 104 cells/100 ml for

enterococci and 400 cells/100 ml for E. coli (Griffith et al., 2007).

3.5.2 Culture

For the culture methods, the percent false negative is the percentage of the samples that had a negative result but were actually positive. For Method 16 00 and 1603, the percent false negatives are reported from analyses of various environmental water samples (USEPA 2000). Five percent

of the E. coli colonies observed gave a false negative reaction (USEPA

, 2006). In unspiked CSO samples, false negative ra tes of 37.5% and 1.7% have been reported for methods 1600 and 1603, respectively (USEPA

, 2008). Additionally, Francy et al. (2000) reported false negative results of

11% for the modified membrane

- Thermotolerant Escherichia coli agar (mTEC) (Method 1603)

wh en compared to the mTEC (EPA Method 1103.1). Griffith et al. (2007) reported false negative results of 10% for Method 1600 with respect to the AB411 standard of 104

enterococci/100 ml.

For Method 1603, the percent false positives reported from analyses o f various environmental water samples were <1% (USEPA, 2000) and averaged 6% for marine and fresh water samples (USEPA

, 2006). False positive confirmation rates were 0% and 6.7% for Method 1600 and 1603, respectively (USEPA

, 2008) were reported in unspiked CSO samples. Francy et al . (2000) reported false positive results of 0% for the modified mTEC (Method 1603) when compared to

the mTEC (Method 1103.1)

. Griffith et al. (2007) reported false positive results of 4% for Method 1600 with respect to the AB411 s tandard of 104 enterococci/100 ml.

3. 6 Ability to Differentiate Fecal S ources

This criterion evaluates the ability of the indicator/method combination to identify the sources contributing to fecal pollution in a waterbody. None of the indicator/methods evaluated are able to differentiate between fecal contamination sources. However, current research by EPA on microbial source tracking assays for use in detecting bovine and human fecal pollution suggests that these assays maybe useful in monitoring feca l contamination in ambient waters (see Section 2.4.1)

.

3. 7 Performance in Different Waterbody Types

T his criterion evaluates how well the method performs ( i.e., does it work or provide usable results) in different types of recreational waters ( e.g ., fr eshwater, marine water, temperate, tropical

, and subtropical waters ). Recent epidemiological studies conducted by EPA have shown

that Enterococcus qPCR and Bacteroidales qPCR performed well in both marine and

freshwaters.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

17 Results from earlier EPA epidemi ological studies found that Enterococcus culture works well in

both marine and freshwater , whereas E. coli only had good correlation (R 2 ≥ 0.5) with GI illness

in freshwater (Cabelli, 1983; Dufour

, 1984)

.

4.0 EVALUATION OF APPROP RIATENESS OF INDICAT OR/METHOD S FOR

CWA PURPOSES

The appropriateness of the method for the various CWA programs is important for state adoption and implementation of EPA’s new or revised criteria recommendations

. To assess the potential use and applicability of new fecal indicators an d methods to meet CWA program needs

, EPA’s Monitoring and Assessment, TMDLs, and NPDES programs evaluate d t he impact of new indicators/methods on their programs. Each program identified attributes necessary for indicator

methods to meet their program s’ CWA needs . The se attributes are shown in Table 4 and ranked on whether they are important (Yes) or not important (No) for each CWA program.

T able 3

– Evaluation of Rapid and Culture Methods for Clean Water Act Programs Implementation Purposes

Method Attribut e s Beach Program NPDES TMDL Assessment

Low Cost Yes Yes Yes Yes

Ease of Use Yes Yes Yes Yes Time to results (rapid/results within 4 hours of sample process/analysis) Yes No No No

Allows for use of historical data for model development, etc. Yes Yes Ye s Yes

Demonstrates effectiveness of treatment from

source to beach No Yes No No

Count/signal associated with human health risk pathogens Yes Yes Yes Yes

Precise/accurate Yes Yes Yes Yes

While beach advisory decisions and closures require same day resu lts to protect public health, for the Assessment, TMDLs and the NPDES programs same day results are not necessary. Therefore, rapid methods are suggested as a key component of the criteria for the Beach Program only. Other important attributes for indicat or/methods for the Beach Program include ease of use of the method and cost. Ease of use refers to how easily the method can be applied by those skilled in analysis. The cost associated with new indicator/method combination in criteria could include capi tal cost, training cost, per sample cost and additional sampling requirements. Capital costs include the upfront cost such as equipment purchase and space required to conduct the test. For example, when performing genetic testing, aside from the equipment needed (e.g., platform [i.e., specific machine]), laminar flow hoods, dedicated pipettes), space is needed, ideally in separate rooms, for reagent preparation (material not containing any genetic materials). Space is also needed for the two types of sampl e preparation, those containing high target sequence DNA concentrations such as DNA standards and calibrator samples, and those containing expected low target sequence DNA concentrations (e.g., filter blanks and water samples)

– the latter of

which should also be in separate laminar flow hoods ( USEPA, 2007).Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

18 Training costs are incurred prior to routine testing so that the user can perform the test within the performance criteria of the test ( USEPA, 2007). Training cost would be dependent on the types

of training available, i.e., whether workshop

- type training with hands

- on experience or completing a training module.

Sample cost can vary and becomes an important cost consideration with a new method depending on the volume of tests needed to be completed on a routine basis. Additional sampling is generally an effort that results from rapid testing. For example, if an early morning sample yields a positive result resulting in beach closure s
, it may then lead to additional sampling to determine if the beac h still needs to be closed in the mid

- afternoon ( USEPA, 2007). Therefore, depending on the number of designated bathing beaches that need to be monitored, the monitoring frequency, number of samples, and the need for additional sample analysis per beach, t he sample costs associated with the use of rapid method could become cost prohibitive or result in beaches being sampled less frequently or not re

- sampled to revise advisory or closure decision during the day.

Ease of use and cost are also important fact ors for Assessment, TMDL and NPDES programs. For the TMDL and Assessment programs the ability to use prior historical data for modeling loadings is also important. Current historical data for E. coli , enterococci, and fecal coliform are

derived from cultu re methods data. Criteria based on a rapid method without the necessary linkage between qPCR and the culture method would not allow for the use of historical data.

For the NPDES Program, the ability of the indicator/method to reflect the treatment efficac y ( e.g ., chlorination) is crucial. Traditional culture methods for fecal indicator bacteria detect only a subset of the total viable population within any given water sample. The qPCR method will theoretically detect all intact cells in a water sample whe ther they are viable or not.

Precision and accuracy of methods are also important to each program as well as the ability to

associate indicator counts/concentrations to health risk.

5 .0 STUDY LIMITATION S

There are several weaknesses in this evaluatio n with regard to how the project was defined in the Critical Path Science Plan. Due to limited data on various water body types, EPA is unable to evaluate method performance in different water body types based on the performance criteria in

Table 2. Addit ionally, method performance with respect to fecal source identification ( i.e

., human vs. animal) could not be evaluated because none of the current indicator/method combinations can distinguish between sources.

6 .0 REFERENCES

Au

- Yeung, H.C. , D. Rh. Tho mas, and R.L. Salmon. Nd. Appendix 2 Section 2 Risk of gastroenteritis from recreational bathing water: A meta

- analysis of randomized exposure trials in four European countries. From http://www.ies.aber.ac.uk/en/epibathe. Accessed on July 2009

.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

19 Cabelli, V.J. 1983. Health Effects Criteria for Marine Recreational Waters. EPA 600/1

- 80

- 031, U.S. Environmental Protection Agency, Cincinnati, OH.

Chern E.C., K. Brenner, L. Wymer

, and R .A. Haugland. 2009. Comparison of Fecal Indicator Bacteria Densities in Marine Recreational Waters by QPCR. Water Qual. Expo. Health 1(3),

203

- 214.

Chern E.C., S.C. Siefring, J. Paar, M. Doolittle, and R.A. Haugland

. Comparison of quantitative PCR assays for Escherichia coli targeting ribosomal RNA and single copy genes. Lett. Appl.

Microbiol. Manuscript Submitted.

Converse R.R., A.D. Blackwood, M. Kirs

, J.F. Griffith

, and R.T. Noble

. 2009. Rapid QPCR

based assay for fecal Bacteroides spp. as a tool fo r assessing fecal contamination in recreational

waters. Water Res 43:4828

– 4837. From http://dx.doi.org/10.1016/j.watres.2009.06.036

.

Dick L.K. and K

. G

. Field. 2004. Rapid estimation of numb ers of fecal Bacteroidetes by use of a quantitative PCR assay for 16SrRNA genes. Appl Environ Microbiol 70(9):5695

– 5697.

DuFour, A.P. 1984. Health Effects Criteria for Marine Recreational Waters. EPA 600/1

- 84

- 004. U.S. Environmental Protection Agency, Ci ncinnati, OH. Fleisher, J.M., L.E. Fleming, H.M. Solo

- Gabriele, J.K. Kish, C.D. Sinigalliano et al. 2010. The Beaches Study: health effects and exposures from non

- point source microbial contaminants in subtropical recreational marine waters. Int. J. Epide miol . 1

- 8. From http://ije.oxfordjournals.org

.

Frahm, E. and U. Obst. 2003 Application of the fluorogenic probe technique (TaqMan PCR) to the detection of Enterococcus spp. and Escherichia coli in water sampl es. Journal Microbiological Methods 52, 123

- 131.

Francy, D.S. and R.A. Darner. Comparison of methods for determining Esherichia coli

concentrations in recreational water. 2000

. Water Res. , 34:2770

- 2778.

Griffith, J

. F

. , Y

. Cao, C . D. McGee , S

. B. Weis berg . 2009. Evaluation of rapid methods and novel indicators for assessing microbiological beach water quality. Water Research , 43: 4900

- 4907

.

Griffith, J.F., D. Moore, C. McGee, and S.B. Weisberg. 2007. Beta Testing of Rapid Methods for Measuring Beach W ater Quality. Southern California Coast Water Research Project Technical Report 506. February.

Haugland, R.A., Siefring, S.C., Wymer, L.J., Brenner, K.P. and Dufour, A.P. 2005. Comparison of Enterococcus density measurements by quantitative polymerase chain reaction and membrane filter culture analysis at two freshwater recreational beaches. Water Research

, 39:559

- 568.

Haugland, R.A., M. Varma, M. Sivaganesan, C. Kelty, L. Peed, and O.C. Shanks. 2010. Evaluation of genetic markers from the 16S rRNA ge ne V2 region for use in quantitative detection of selected Bacteroidales species and human fecal waste by real time PCR. Syst. Appl.

Microbiol

. 33:348

- 357.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

20 Love D.C., and M.D. Sobse y M.D. Simple and rapid F+ C oliphage culture, latex agglutination, and ty ping assay to detect and source track fecal contamination. Appl. Environ. Microbiol

. 73:4110

– 8.

Ludwig, W. and K.

- H. Schleifer. (2000) How quantitative is quantitative PCR with respect to

cell counts? Systematic Applied Microbiology 23: 556

- 562.

Shan ks, O.C., C. Kelty, M. Sivaganesan, M. Varma, and R.A. Haugland. 2009. Quantitative PCR for genetic markers of human fecal pollution . Applied and Environmental Microbiology

, 75:5507

- 5513

Shanks, O.C., K. White, C.A. Kelty, S. Hayes, M. Sivaganesan, M. Jen kins, M. Varma, and

R.A. Haugland

. 2010a. Performance assessment PCR

- based assays targeting Bacteroidales genetic markers of bovine fecal pollution. Appl

. Environ Microbiol

., Mar;76(5):1359

- 66.

Shanks, O.C., White, K., Kelty, C.A., Sivaganesan, M., Blann on, J., Meckes, M., Varma, M., Haugland, R.A. 2010b. Performance of PCR

- based assays targeting Bacteroidales genetic markers of human fecal pollution in sewage and fecal samples. Environ Sci Technol . 44:6281

- 8. Siefring, S., M. Varma, E. Atikovic, L. Wymer and R.A. Haugland. 2008. Improved real

- time PCR assays for the detection of fecal indicator bacteria in surface waters with different instrument and reagent systems. J. Water and Health 6:225

- 237.

Southern California Coastal Water Research Project (SCCWR P) . FY 2010/2011 Research Plan

– Commissioned Approved. June 2010.

USEPA. 1986. Ambient Water Quality Criteria for Bacteria

– 1986. EPA 440/5

- 84

- 002. United States Environmental Protection Agency, Office of Water, Washington, DC.

USEPA. 2000. Improved En umeration Methods for the Recreational Water Quality Indicators:

Enterococci and Escherichia coli . EPA

- 821/R

- 97/004. U

. S

. Environmental Protection Agency, Office of Water (4303T), Washington

, DC

.

USEPA. 2001. Method 1601: Male

- specific (F+) and somat ic C oliphage in water by two

step enrichment procedure. Technical Report EPA 821

- R

- 01

- 030. U

. S

. Environmental Protection Agency, Office of Water, Washington

, DC

.

US EPA. 2002. Method 1600: E nterococci in water by membrane filtration using membrane

Ente rococcus indoxyl

- D

- glucoside agar (mEI), EPA 821/R

- 02/022

. U

. S

. Environmental Protection Agency, Office of Water, Washington

, DC.

USEPA. 2006a. Method 1600: E nterococci in water by membrane filtration using membrane

Enterococcus indoxyl

- D

- glucoside agar ( mEI), EPA 821

- R

- 0 6

- 009. U

. S

. Environmental Protection Agency, Office of Water, Washington

, DC.Evaluation of the Suitability of Individual Combinations of Indicators and Methods for Different Clean Water Act Programs

21 USEPA. 2006 b . Method 1603: Escherichia coli ( E. coli ) in Water by Membrane Filtration Using

Modified membrane

- Thermotolerant Escherichia coli Agar (Modified mT EC). EPA

- 821

- R

- 06

- 011. U.S. Environmental Protection Agency, Office of Water, Washington, DC.

US EPA. 2007. Guidelines establishing test procedures for the analysis of pollutants; Analytical methods for biological pollutants in wastewater and sewage sludge : F inal rule. Federal Register,

Vol

. 72, No. 57, March 26, 2007

.

US EPA. 2008. Results of the inter

- laboratory testing study for the comparison of methods for

detection and enumeration of Enterococci and E. coli in combined sewer overflows. EPA 821

- R

- 08

- 006. U.S. Environmental Protection Agency, Office of Water, Washington, DC.

Wade, T.J., N. Pai, J.N.S. Eisenberg, and J. M. Colford. 2003. Do U.S. Environmental Protection Agency water quality guidelines for recreational waters prevent gastrointestinal ill ness? A systematic review and meta

- analysis. Environmental Health Perspectives 111(8): 1102

- 1109.

Wade, T.J., R

. L. Calderon, E

. Sams, M

. Beach, K

. P. Brenner, A

. H. Williams, and A P. Dufour. 2006. Rapidly Measured Indicators of Recreational Water Quality Are Predictive of Swimming

- Associated Gastrointestinal Illness. Environmental Health Perspectives, 114(1):24

- 28.

Wade, T

. J., R.L. Calderon, KP Brenner, E. Sams, M. Beach, R. Haugland, L. Wymer, and A. P. Dufour. 2008. High Sensitivity of Children to Swi mming

- Associated Gastrointestinal Illness Results Using a Rapid Assay of Recreational Water Quality. Epidemiology , 19: 375

- 383.

Wade, T.J., E. Sams , KP Brenner , R. Haugland , E. Chern , et al. 2010. Rapidly measured indicators of recreational water quali ty and swimming

- associated illness at marine beaches: a prospective cohort study

. Environmental Health

, 9:66 . Available online at http://www.ehjournal.net/content/9/1/66

.

Zmirou, D., Pena, L., Ledran s, M., Letertre, A. 2003. Risks associated with the microbiological quality of bodies of fresh and marine water used for recreational purposes: summary estimates based on published epidemiological studies. Archives of Environmental Health 58(11): 703

- 711