# Presentation: 589959

## USEPA REGION 4 METHOD PERFORMANCE AND DATA COMPARABILITY PILOT PROJECT

- L. Huff1, 2, R. Frydenborg2, 4, M. Brossett2, 5, C. Jones2, 5, T. Pugh2, 5, M. Vogel2, 6, A. Dossett2, 7, B. Crouch2, 8, J. Glover2, 9, D. Arnwine2, 10, L. Dorn2, 11, C. Decker2, 11, J. Harrison2, 12, D. Melgaard2, 12, L. Gabanski13,**  **J. Diamond3, 14, S. Stribling3, 14, J. Gerritsen3, 14
**1****AL Dept of Environmental Management; 1400 Coliseum Boulevard, PO Box 301463, Montgomery, AL 36130-1463; e mail: **_[esh@adem.state.al.us](mailto:esh@adem.state.al.us)_**; ****2****R4 Workgroup Member; ****3****Technical Support; ****4****FL Dept. of Environmental Protection; ****5****GA Dept. of Natural Resources; ****6****KY Dept of Environmental Protection; ****7****MS Dept of Environmental Quality; ****8****NC Dept. of Environment and Natural Resources; ****9****SC Dept. of Health and Environmental Control; ****10****TN Dept. of Environment and Conservation; ****11****R4 EPA-Athens; ****12****R4 EPA-Atlanta; ****13****EPA Headquarters; ****14****Tetra Tech, Inc.**

- Introduction
- The water quality program of each EPA Region 4 state includes biological monitoring as an integral part of its assessment of water quality.
- Documenting method performance characteristics is essential as states struggle to develop biological indicators that link nutrient enrichment, sedimentation, and habitat degradation to biological community response.
- Documentation of data quality objectives and method performance characteristics will help R4 states to objectively determine how bioassessment data produced by their own and outside agencies should be used in management decisions.
- EPA conducted the National Wadeable Stream Assessment (WSA) using a single biological assessment protocol at over 500 sites.  The WSA provides an estimate of water quality on a national scale, but results may not agree with state results.  Documentation of method performance characteristics may provide EPA with a consistent measure to determine what data should be used to assess water quality, concentrate resources into improving state methods, and help understand why state assessments differ.
- There is currently no guidance for developing and documenting performance characteristics and comparability of bioassessment methods.
- We are conducting a region-wide pilot project to:
- Propose a framework for states to calculate and report data quality and performance characteristics of macroinvertebrate bioassessment methods .
- Review, test, and revise agreed-upon methods

- Development of a Regionalized General Stressor Gradient and Independent Assessments of Water Quality
- Bias, sensitivity, and responsiveness cannot be measured without independent assessment of the degree of impairment at each bioassessment site.  A critical element of this project is the development of a regional Generalized Stressor Gradient (RGSG).
- RGSG Factors: Quantifying Watershed Stress
**Landscape Level**:
- 2000 MRLC % Land Cover
- % Urban, % Agriculture, and Road Density
- Absolute Scale: Gradient thresholds set based on literature
**Habitat Alteration**:
- Habitat assessment data from each state
- Agencies use variations of the RR/GP forms in EPA’s RBP manual (Barbour et al. 1999)
- Relative Scale: Gradient thresholds based on “best available” in region.
**Chemical Pollution**:
- Water quality data collected by each agency
- Differences in parameters collected, when data collected, and number of site visits
- Most states have conductivity and nutrient data (NO3/NO2-N, NH3-N, TP)
- Relative Scale: Gradient thresholds based on “best available” in region.

- Majors Cr (Baldwin Co., AL)
- R4 1996 Joint Bioassessment Exercise

- Sougahatchee Cr (Lee Co., AL)
- R4 2004 Joint Bioassessment Exercise

- Data used to calculate and document Method Performance
- Region 4 includes 14 Level 3 Ecoregions and 99 Level 4 Ecoregions, none of which are present in all eight states. The project will be restricted to bioregions shared by 2 or more states, with existing data or where data can be collected during 2006.
- A survey conducted by ADEM was used to integrate the R4 MPDC Pilot Project into each state’s baseline monitoring activities and to use existing data where available.

- _Data used to measure method performance will include at least:_
- 5 pairs of duplicate samples along stressor gradient
- 10 reference reach samples within a site class
- 10 “test” reach samples along a stressor gradient
- Independent assessment of water quality at each site

- 1) B: Bioregion; L3: Level 3 Ecoregion; L4: Level 4 Ecoregion; 2) Bioregion or L4 in which project will be focused; AA: All available; 3) Months for which macroinvertebrate bioassessment criteria developed; 4) E: existing data; 2006: data collected during 2006; 5) JB: Joint Bioassessments – 3-6 conducted at a gradient of good, fair, and poor sites.

- Table 1. Summary of project areas, data sources, and joint bioassessments.

- Level 3 and 4 Ecoregions of the southeastern US

|  | Focus Region |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | 45 |  | 65 |  |  |  | 66 |  | 71 |  | 75 |  |
|  | NC | SC | AL | EPA | FL | MS | NC | TN | KY | TN | FL | GA |
| Site Classification1 | L3 | L3 | B | L3 | B | B | L3 | B | B | B | B | L4 |
| Focus Area2 | AA | AA | 65d/i | AA |  |  | AA | 66d/e/g | 71a/b/c/e/f/g/h | 71e & 71f/g/h | 75f/e | 75f |
| Sampling Period3 | 1-12 | 6-9 | 5-6 | 1-12 | 1-12 | 12-2 | 1-12 | 1-12 | 5-9 | 1-12 | 1-12 | 9-2 |
| Reference4 | E | E | E | 2006 | E | E | E | E | E | E | E | 2006 |
| Test4 | E | E | E | 2006 | E | E | E | E | E | E | E | 2006 |
| Duplicate4 | 2006 | 2006 | 2006 | 2006 | 2006 | E | 2006 | 2006 | 2006 | 2006 | E | 2006 |
| JBs5 | 3 in NC &              3 in SC  |  | 3 in AL  |  | 3 in MS |  | 3 in NC &            3 in TN |  | 3 in KY &             3 in TN |  | 5 in FL  |  |

- Method Performance Measures
- The project focuses on performance characteristics of each agency’s final assessments based on their established protocol. Four performance characteristics will be measured: precision, sensitivity, bias, and responsiveness.  Several methods of measuring these characteristics will be evaluated.

| Quality Control Measures | Definition | Calculation |
| --- | --- | --- |
| Sorting and Subsampling |  |  |
| Bias: Percent Sorting Efficiency (PSE) | A second sorter checks for missed specimens | PSE=(A/(A+B))*100
Where A=# of organisms found by 1st sorter; and, 
B= # orgs. missed by 1st sorter |
| Taxonomy |  |  |
| Precision: % difference in enumeration (PDE) | A 2nd taxonomist rechecks organism counts  | PDE=([n1-n2]/(n1+n2))*100
Where n1= # of orgs counted by 1st tax; and,
 n2= # of orgs counted by 2nd tax |
| Precision: % taxonomic disagreement (PTD) | A 2nd taxonomist re-identifies organisms | PTD=[1-(comppos/N)]*100
Where comppos= # of agreements; and, N=total # organisms in lger of two counts. |

**Table 2**. Quality control measures of specific steps in protocol to be calculated for each agency.

- R4 Joint Bioassessments
- We assume that methods with similar performance will provide similar assessments of water quality (Barbour et al. 1999).  With the RGSG, combined with results of side-by-side bioassessments, it may be possible to :
-  evaluate the use of performance characteristics as a measure of data comparability, and
-  determine what other information is needed to accurately evaluate data comparability.
- Six “mini” joint bioassessment workshops will be conducted among subsets of the nine participating agencies (Table 1). Each agency is conducting 3-6 joint bioassessments with another agency that shares the same bioregion or ecoregion.

- Quality Control Measures

- Method Performance Documentation Form and Database

- We are developing forms for submitting bioassessment data to the state.
- Entities will provide the following tables for reference, duplicate, and “test” sites:
- 1. Site description
- 2. Reference Reach Criteria
- 3. Bioassessment data
- 4. Ancillary data
- An ACCESS database will be created to house all R4 method performance data, evaluate the data in terms of site class, sampling period, and overall dataset, and to calculate and report performance characteristics.

- If performance characteristics do not meet data quality requirements, this information may help each agency determine how they could potentially address the problem by identifying those steps that are most variable.  If our assessments do differ, this information, in conjunction with the meta data associated with each of the reference and test samples, may help us determine why.