TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters



## Appendix B-14: Memo to FDEP Regarding Interval and Equivalence Tests for the SCI

(Appendix 3-C from USEPA. 2009. Proposed Methods and Approaches for Developing Numeric Nutrient Criteria for Florida’s Inland Waters . United States Environm ental Protection Agency, Office of Water, Office of Science and Technol ogy, Health and Ecological Criteria Division, Washington, DC.)

### Appendix 3-C



## Memo from Leska Fo re regarding interva l and equivalence tests for the Strea m Condition Index2 Statistical Design 136 NW 40th St. Seattle, WA 98107 (206) 708-5048

DATE: April 9, 2009

TO: Russ Frydenborg, Florida DEP

FROM: Leska Fore, Statistical Design

RE: Statistical Framework for Defining Impairment for SCI


## Objective
Identify a statistical test and approach fo r defining impairment status for the case in which a single test site is compared to a distribution of reference sites. Florida DEP’s stream condition index (SCI) for macroinvertebrate stream samples is used as a case study.


## Why use a statistical framework?

 A statistical model provides a hypothesis and a decision rule for determining whether a test site differs from reference condition.

 Uncertainty associated with an impa irment decision can be estimated.

 Type I (true reference site called impair ed) and Type II (true impaired site called reference) error can be compared and ba lanced to be equally protective of the resource and the regulated community.

 Provides a framework for future, unforeseen questions and needs, for example, how to deal with two replicat e visits to a test site.

 Methods are symmetric in that they can be equally applied to de-listing or listing a site as impaired.

 Makes listing decisions more robust to legal challenge.



## Statistical decision ≠ Stakeholder decision
Defining a threshold for biologi cal impairment involves scientif ic information and social values. Statistics can support this decision, but cannot ultimately be used to make the 

## MEMORANDUM3 decision. Once a threshold has been defined, a statistical framework can be used to structure the decision process ar ound the line of impairment.

This document addresses the statistical uncertainty around the threshold line.



## Criteria for selection of a statistical framework

 Applicable to our situation: comparison of one test site to a reference population.

 Legally and statistically defensible.

 Provides estimate of uncertainty asso ciated with impairment status calls.

 Can estimate false positive and negative rates.

 Can be used to model the effect of repeat visits to a site.



## Candidate statistical tests

Much of the literature related to thres hold testing for environmental compliance addresses the comparison of water chemistr y samples to a threshold (SWRCB, 2004). This testing scenario for chemical contam inants is somewhat the opposite from our situation in that contaminant thresholds are defined as a single value with no associated uncertainty, that is, no set of values from reference sites. Fo r biocriteria, thresholds are derived from a population of reference sites, not a single value. Furthermore, water chemistry test sites have multiple samples ra ther than a single value as is typical for biocriteria.
For water chemistry, the variability is associ ated with the test s ite rather than the reference sites. In contrast, for biocriteria we are comparing one test site value to multiple reference site values. Expectations for bioc riteria are based on local reference condition while thresholds for toxics and contaminants tend to be expressed as a single threshold that is nationally applied. Most of the papers related to statistical testing for thresholds address the chemical contaminant model.
An alternative approach for biocriteria thre shold testing was descri bed by Kilgour et al. (1998). Their method was based on interval and e quivalence tests. Both tests can be used to evaluate whether an observation at a test location falls within or outside of the normal range of variation for reference locations. Wh en results from the tests are interpreted together, errors due to false positives and negatives are equally likely.4 Table 1. Candidate statistical tests for comparing SC I values at a test site to a threshold derived from reference sites.
Test name Application to biocriteria threshold

Recommended : Combination of interval and equivalence tests (Kilgour et al., 1998) Both tests are designed to compare 1 test site against a population of reference sites; pair of tests controls for both Type I and Type II error
Two-sample t test Assumes multiple samples from both the test population and the reference population
One-sample t test Comparison to the mean, not to percentiles of the distribution; low power to detect differences
One-sample Wilcoxon signed rank test Nonparam etric analog to one-sample t test, compares median of a test population to a threshold; not designed to compare percentiles

Confidence interval around a proportion (Gibbons, 2001) Designed to test what proportion of test samples exceed a threshold; multiple test samples, one reference threshold

Exact binomial test Widely recommended for chemical testing of % exceedances of a threshold; compares multiple test values to a single reference value


## Interval and Equivalence Testing
The specific statistical methods described by K ilgour et al. (1998) are not particularly novel, i.e., the comparison of a statistic to th e noncentral F distribu tion, but the pairing of two tests (interval and equiva lence tests) and the applica tion to bioassessment endpoints represents a new approach. Their approach wa s developed specifically for the comparison of a single test site to a populat ion of reference sites for meas ures of biological condition.
Both the interval and the equivalence tests can be applied to the comparison of a single test site value with values from a population of reference sites. Both tests have a low Type I error rate (false positive), but they can be used to balance false positives and negatives because they test the opposite null hypotheses. The in terval null hypothesis tests for difference from reference and the equiva lence null hypothe sis tests for equality to reference (Table 2). In this way, the tests are symmetric. By pairing these two tests, we can balance for the error rates associated with falsely determining a reference site to be impaired (Type I) and missing a site that is truly impaired (Type II; Appendix A).
Although not relevant for this application to the SCI, the approach can also be generalized to a multivariate comparison of multiple indicators (e.g., metrics) against their respective th resholds (Bowman and Somers, 2006).5 Table 2. Comparison of the interval and equivale nce tests, hypotheses tested by each and their advantages and disadvantages relative to decisions about impairment.

Statistical test H O H A Advantages Disadvantages

Interval test Test site is within the range of reference sites Test site is not within the range of reference sites Protects the polluter (guards against unnecessary listing) Not as sensitive at detecting impairment

Equivalence test Test site is not within the range of reference sites Test site is within the range of reference sites Protects the environment (very sensitive to impairment) Very sensitive and may identify reference sites as impaired



## How do we specify the test?

Both tests use the same test statistic, the fa miliar F-statistic. Because we are testing for differences not from the mean, but for so me distance from the mean, e.g., the 10 th percentile (our defined threshold), we need to use a noncentral F test. The null and alternate hypotheses are mirror images of each for the interval a nd equivalence tests.

Interval test

The null hypothesis is that the test and reference sites are the same . The alternative hypothesis is that test an d reference sites are the different

.
H O (null): 

#      T ref , or “The difference between the mean SCI of the reference sites and the SCI at the test site is less than delta.” Delta can be any specified value, e.g., the 10 th percentile of SCI values for reference sites.

H A (alternate): 

#      T ref , or “The test site is more different than the specified difference (delta) from the mean SCI.”

Equivalence test

The null hypothesis is that the te st and reference sites are different . The alternative hypothesis is that test an d reference sites are the same

.
H O : 

#      T ref , or “The difference between the m ean SCI of the reference sites and the SCI at the test site is more than delta.”6 H A : 

#      T ref , or “The test site is not different from the specified difference from the mean SCI for reference sites.”

Test statistic 

#  

2 2

1         



ref ref

ref

n s TX F



## How does statistical test work?
Suppose we go to a stream site and sample fo r SCI and calculate a value of 46 out of a possible range of values from 0–100. We want to compare this observed value to the 10 th percentile of SCI values for re ference sites (Table 3). The 10 th percentile is derived from an assumption that reference sites follow an approximately normal distribution. The table value for the 10 th percentile of a normal distribut ion (–1.28) is used to define 10 th percentile of the reference dist ribution. This z-table value is also used to define the noncentrality parameter, lambda.

We calculate the F-statistic and use a statis tical program (e.g., R or Excel) to look up the p-value associated with the F-statistic with 1 and 63 degr ees of freedom and determine that we can reject the interval test and conc lude that the test s ite is different from reference condition.
Table 3. Worked example for comparing SCI at a te st site to SCI values at reference sites.
Description Value
Mean SCI (reference sites) 66.3
Standard deviation of SCI (reference sites) 12.6

Number of reference sites 64

10 th Percentile of SCI (reference sites) 66.3–(1.28*12.6) = 50

SCI for test site 46

Standardized z-value for 10 th percentile of reference (one-tailed test); used to define difference (delta) we are testing for –1.287 F-statistic 

##  

2 2

64 1 6.12 463.66
         = 166.1

Degrees of freedom (1, 63)
Lambda, noncentrality parameter; this value represents how far from the mean the test site should be, i.e., lower than the 10 th percentile 64 * (1.28) 2 = 104.9

p-value for F-statistic (interval test) p = 0.04, significantly lower than the 10 th %tile of reference

Conclusion “We are 95% confident the test site is impaired with impairment defined as below the 10 th percentile of reference.”



## Statistical Decision Rules for SCI

We can use the statistical framework of in terval and equivalence testing to estimate uncertainty for SCI values at a test site ba sed on different definitions of impairment.

Table 4. Statistical decision rules for SCI values at a test site for threshold values defined as the 1 st , 5 th , and 10 th percentiles of SCI values for reference sites. Shown are values for p < 0.05. N = 1 SCI value for each site.
Impairment threshold (description) Impairment threshold (numeric) Impaired Undetermined Reference

2.5 th percentile of reference 42 <37 37–45 >45

5 th percentile of reference 46 <41 41–49 >49

10 th percentile of reference 50 <47 47–53 >53


## Summary

 Caveat: Statistical tests provide a framework for comparison and testing around a threshold, but cannot be used alone to defi ne impairment thresholds. That decision also relies on scientific unde rstanding and social values.

The method described here combines two stat istical tests, the interval test and the equivalence test, to compare a si ngle test site value to a set of values from a population of reference sites. This approach satisfies most of our criteria for a statistical framework for8 a biocriteria threshold. Pairing the interval a nd equivalence tests to evaluate percentiles from the reference site distribution represents a simple statistical decision rule that is easy to describe and legally defensible. We can estimate error rates for false positives and false negatives and balance them. The method is symmetric and can be easily inverted for delisting without any change to Type I or Type II errors.
This approach does not deal exp licitly with the case in which more than one sample value is collected from a test site. This problem c ould be addressed in two ways. For the simple case of two repeat visits to a test site, a new distribution of reference sites could be developed based on the average SCI value for tw o repeat visits to each reference site. These could be the most recent visit or two visits that were randomly chosen.
A more general approach to incorporating test site variability could be accomplished with a simulation. Using our estimate of within -site variability, a di stribution around point estimates of SCI could be defined. Repeat sa mpling from this distribution could be used to generate a distribution of F-statistics. In this way, we could model the variability associated with a single SCI for a test site.
Combining the interval and equivalence tests provides a statistical framework to support impairment decisions, and is particularly suit ed to biocriteria becau se a single test value is compared to a population of reference values. 

## Appendix A

Example showing application of interval and equivalence testing for determining the status of a test site. The threshold for impairment is defined as the 10 th percentile of SCI values for reference sites. Shown are the SCI value, difference between mean of reference and SCI, numerator and denominator for the F statistic, F, p-value for F fo r the interval and equivalence tests, status call, and result for the interval and equivalence tests. See Table 3 for values used in calculations. SCI values greater than or equal to the mean for reference sites (SCI = 66–100) are not shown. SCI Mu-imp F_Num F_Den F P_Int P_Equ STATUS Interv. Equiv. 1 65.3 4264.1 2.48 1719.0 0.00 1.00 IMPAIRED >delta >=delta 2 64.3 4134.5 2.48 1666.7 0.00 1.00 IMPAIRED >delta >=delta 3 63.3 4006.9 2.48 1615.3 0.00 1.00 IMPAIRED >delta >=delta 4 62.3 3881.3 2.48 1564.6 0.00 1.00 IMPAIRED >delta >=delta 5 61.3 3757.7 2.48 1514.8 0.00 1.00 IMPAIRED >delta >=delta 6 60.3 3636.1 2.48 1465.8 0.00 1.00 IMPAIRED >delta >=delta 7 59.3 3516.5 2.48 1417.6 0.00 1.00 IMPAIRED >delta >=delta 8 58.3 3398.9 2.48 1370.2 0.00 1.00 IMPAIRED >delta >=delta 9 57.3 3283.3 2.48 1323.6 0.00 1.00 IMPAIRED >delta >=delta 10 56.3 3169.7 2.48 1277.8 0.00 1.00 IMPAIRED >delta >=delta 11 55.3 3058.1 2.48 1232.8 0.00 1.00 IMPAIRED >delta >=delta 12 54.3 2948.5 2.48 1188.6 0.00 1.00 IMPAIRED >delta >=delta 13 53.3 2840.9 2.48 1145.2 0.00 1.00 IMPAIRED >delta >=delta 14 52.3 2735.3 2.48 1102.7 0.00 1.00 IMPAIRED >delta >=delta 15 51.3 2631.7 2.48 1060.9 0.00 1.00 IMPAIRED >delta >=delta 16 50.3 2530.1 2.48 1019.9 0.00 1.00 IMPAIRED >delta >=delta 17 49.3 2430.5 2.48 979.8 0.00 1.00 IMPAIRED >delta >=delta9 SCI Mu-imp F_Num F_Den F P_Int P_Equ STATUS Interv. Equiv.

18 48.3 2332.9 2.48 940.4 0.00 1.00 IMPAIRED >delta >=delta 19 47.3 2237.3 2.48 901.9 0.00 1.00 IMPAIRED >delta >=delta 20 46.3 2143.7 2.48 864.2 0.00 1.00 IMPAIRED >delta >=delta 21 45.3 2052.1 2.48 827.2 0.00 1.00 IMPAIRED >delta >=delta 22 44.3 1962.5 2.48 791.1 0.00 1.00 IMPAIRED >delta >=delta 23 43.3 1874.9 2.48 755.8 0.00 1.00 IMPAIRED >delta >=delta 24 42.3 1789.3 2.48 721.3 0.00 1.00 IMPAIRED >delta >=delta 25 41.3 1705.7 2.48 687.6 0.00 1.00 IMPAIRED >delta >=delta 26 40.3 1624.1 2.48 654.7 0.00 1.00 IMPAIRED >delta >=delta 27 39.3 1544.5 2.48 622.6 0.00 1.00 IMPAIRED >delta >=delta 28 38.3 1466.9 2.48 591.3 0.00 1.00 IMPAIRED >delta >=delta 29 37.3 1391.3 2.48 560.9 0.00 1.00 IMPAIRED >delta >=delta 30 36.3 1317.7 2.48 531.2 0.00 1.00 IMPAIRED >delta >=delta 31 35.3 1246.1 2.48 502.3 0.00 1.00 IMPAIRED >delta >=delta 32 34.3 1176.5 2.48 474.3 0.00 1.00 IMPAIRED >delta >=delta 33 33.3 1108.9 2.48 447.0 0.00 1.00 IMPAIRED >delta >=delta 34 32.3 1043.3 2.48 420.6 0.00 1.00 IMPAIRED >delta >=delta 35 31.3 979.7 2.48 394.9 0.00 1.00 IMPAIRED >delta >=delta 36 30.3 918.1 2.48 370.1 0.00 1.00 IMPAIRED >delta >=delta 37 29.3 858.5 2.48 346.1 0.00 1.00 IMPAIRED >delta >=delta 38 28.3 800.9 2.48 322.9 0.00 1.00 IMPAIRED >delta >=delta 39 27.3 745.3 2.48 300.4 0.00 1.00 IMPAIRED >delta >=delta 40 26.3 691.7 2.48 278.8 0.00 1.00 IMPAIRED >delta >=delta 41 25.3 640.1 2.48 258.0 0.00 1.00 IMPAIRED >delta >=delta 42 24.3 590.5 2.48 238.0 0.00 1.00 IMPAIRED >delta >=delta 43 23.3 542.9 2.48 218.9 0.00 1.00 IMPAIRED >delta >=delta 44 22.3 497.3 2.48 200.5 0.01 0.99 IMPAIRED >delta >=delta 45 21.3 453.7 2.48 182.9 0.02 0.98 IMPAIRED >delta >=delta 46 20.3 412.1 2.48 166.1 0.04 0.96 IMPAIRED >delta >=delta 47 19.3 372.5 2.48 150.2 0.09 0.91 Undetermined <=delta >=delta 48 18.3 334.9 2.48 135.0 0.18 0.82 Undetermined <=delta >=delta 49 17.3 299.3 2.48 120.7 0.31 0.69 Undetermined <=delta >=delta 50 16.3 265.7 2.48 107.1 0.48 0.52 Undetermined <=delta >=delta 51 15.3 234.1 2.48 94.4 0.67 0.33 Undetermined <=delta >=delta 52 14.3 204.5 2.48 82.4 0.82 0.18 Undetermined <=delta >=delta 53 13.3 176.9 2.48 71.3 0.93 0.07 Undetermined <=delta >=delta 54 12.3 151.3 2.48 61.0 0.98 0.02 REFERENCE <=delta <delta 55 11.3 127.7 2.48 51.5 1.00 0.00 REFERENCE <=delta <delta 56 10.3 106.1 2.48 42.8 1.00 0.00 REFERENCE <=delta <delta 57 9.3 86.5 2.48 34.9 1.00 0.00 REFERENCE <=delta <delta 58 8.3 68.9 2.48 27.8 1.00 0.00 REFERENCE <=delta <delta 59 7.3 53.3 2.48 21.5 1.00 0.00 REFERENCE <=delta <delta 60 6.3 39.7 2.48 16.0 1.00 0.00 REFERENCE <=delta <delta 61 5.3 28.1 2.48 11.3 1.00 0.00 REFERENCE <=delta <delta 62 4.3 18.5 2.48 7.5 1.00 0.00 REFERENCE <=delta <delta 63 3.3 10.9 2.48 4.4 1.00 0.00 REFERENCE <=delta <delta 64 2.3 5.3 2.48 2.1 1.00 0.00 REFERENCE <=delta <delta 65 1.3 1.7 2.48 0.7 1.00 0.00 REFERENCE <=delta <delta 66 0.3 0.1 2.48 0.0 1.00 0.00 REFERENCE <=delta <delta10 

## References
Bowman, M. F. and K. M. Somers. 2006. Eval uating a novel Test Site Analysis (TSA) bioassessment approach. Journal of th e North American Benthological Society 25:712-727. Gibbons, R.D. 2003. A statistical approach for performing water quality impairment assessments. Journal of the American Water Resources Association 39:841–849. Kilgour, B. W., K. M. Somers, and D. E. Matthews. 1998. Using the normal range as a criterion for ecological significance in e nvironmental monitoring and assessment. Ecoscience 5:542–550. State Water Resources Control Board (SWRCB). 2004. Functional Equivalent Document: Water Quality Control Policy for Developing California s Clean Water Act Section 303(d) List. Califor nia Division of Water Quality. http://www.swrcb.ca.gov/water_issu es/programs/tmdl/303d_policydocs.shtml . See Document #100 on this web site.



## Additional References
Barnett, V. and Bown, M. 2002. Statistically meaningful standards fo r contaminated sites using composite sampling. Environmetrics 13: 1-13. Bowman, M. F., and K. M. Somers. 2005. Considerations when using the reference condition approach for bioassessment of freshwater ecosystems. Water Quality Research Journal of Canada 40:347–360. Goudey, R. 2007. Do statistical inferences allo wing three alternative decision give better feedback for environmentally precautionary decision-making? Journal of Environmental Management 65:338-344. Lin, Pi-Erh, D. Meeter, X.-F. Niu. 2000. A nonparametric procedure for listing and delisting impaired waters based on crit erion exceedances. Technical report submitted to Florida DEP. Parkhurst D.F. 2001. Statistical significance tests: equivalen ce and reverse tests should reduce misinterpretation. BioScience, 51, 1051–1057. Shabman, L. and E. Smith. 2003. Implications of Applying Statistically Based Procedures for Water Quality Assessment. Journa l of Water Resources Planning and Management. Smith, E. P., Ye, K., Hughes, C., and Sh abman, L. 2001. Statistical assessment of violations of water qualit y standards under Section 303d of the Clean Water Act. Environ. Sci. Technol., 35: 606–612. Smith, E. P., Zahran, A., Mahmoud, M., and Ye, K. 2003 . Evaluation of water quality using acceptance sampling by variables. Environmetrics, 14:373-386. Warn, A. E. 1989. Auditing the quality of effl uent discharges. Envi ronmental Monitoring and Assessment 12:165–181.