TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

Appendix B-14: Memo to FDEP Regarding Interval and 

Equivalence Tests for the SCI 

(Appendix 3-C from USEPA. 2009. Proposed Methods and Approaches for Developing Numeric 

Nutrient Criteria for Florida’s Inland Waters. United States Environmental Protection Agency, 

Office of Water, Office of Science and Technology, Health and Ecological Criteria Division, 

Washington, DC.) 
Appendix 3-C 

Memo from Leska Fore regarding interval and equivalence 

tests for the Stream Condition Index

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-1
2

Statistical Design 

 136 NW 40th St. 

Seattle, WA 98107 

(206) 708-5048 

DATE: April 9, 2009 

TO: Russ Frydenborg, Florida DEP 

FROM: Leska Fore, Statistical Design 

RE: Statistical Framework for Defining Impairment for SCI 

Objective 

Identify a statistical test and approach for defining impairment status for the case in 

which a single test site is compared to a distribution of reference sites. Florida DEP’s 

stream condition index (SCI) for macroinvertebrate stream samples is used as a case 

study. 

Why use a statistical framework? 

 A statistical model provides a hypothesis and a decision rule for determining whether 

a test site differs from reference condition. 

 Uncertainty associated with an impairment decision can be estimated. 

 Type I (true reference site called impaired) and Type II (true impaired site called 

reference) error can be compared and balanced to be equally protective of the 

resource and the regulated community. 

 Provides a framework for future, unforeseen questions and needs, for example, how 

to deal with two replicate visits to a test site. 

 Methods are symmetric in that they can be equally applied to de-listing or listing a 

site as impaired. 

 Makes listing decisions more robust to legal challenge. 

Statistical decision ≠ Stakeholder decision 

Defining a threshold for biological impairment involves scientific information and social 

values. Statistics can support this decision, but cannot ultimately be used to make the 

MEMORANDUM 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-2
3

decision. Once a threshold has been defined, a statistical framework can be used to 

structure the decision process around the line of impairment. 

This document addresses the statistical uncertainty around the threshold line. 

Criteria for selection of a statistical framework 

 Applicable to our situation: comparison of one test site to a reference population. 

 Legally and statistically defensible. 

 Provides estimate of uncertainty associated with impairment status calls. 

 Can estimate false positive and negative rates. 

 Can be used to model the effect of repeat visits to a site. 

Candidate statistical tests 

Much of the literature related to threshold testing for environmental compliance 

addresses the comparison of water chemistry samples to a threshold (SWRCB, 2004). 

This testing scenario for chemical contaminants is somewhat the opposite from our 

situation in that contaminant thresholds are defined as a single value with no associated 

uncertainty, that is, no set of values from reference sites. For biocriteria, thresholds are 

derived from a population of reference sites, not a single value. Furthermore, water 

chemistry test sites have multiple samples rather than a single value as is typical for 

biocriteria. 

For water chemistry, the variability is associated with the test site rather than the 

reference sites. In contrast, for biocriteria we are comparing one test site value to multiple 

reference site values. Expectations for biocriteria are based on local reference condition 

while thresholds for toxics and contaminants tend to be expressed as a single threshold 

that is nationally applied. Most of the papers related to statistical testing for thresholds 

address the chemical contaminant model. 

An alternative approach for biocriteria threshold testing was described by Kilgour et al. 

(1998). Their method was based on interval and equivalence tests. Both tests can be used 

to evaluate whether an observation at a test location falls within or outside of the normal 

range of variation for reference locations. When results from the tests are interpreted 

together, errors due to false positives and negatives are equally likely. 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-3
4

Table 1. Candidate statistical tests for comparing SCI values at a test site to a threshold derived 

from reference sites. 

Test name Application to biocriteria threshold 

Recommended: Combination of 

interval and equivalence tests (Kilgour 

et al., 1998) 

Both tests are designed to compare 1 test site against a 

population of reference sites; pair of tests controls for 

both Type I and Type II error 

Two-sample t test Assumes multiple samples from both the test 

population and the reference population 

One-sample t test Comparison to the mean, not to percentiles of the 

distribution; low power to detect differences 

One-sample Wilcoxon signed rank test Nonparametric analog to one-sample t test, compares 

median of a test population to a threshold; not designed 

to compare percentiles 

Confidence interval around a 

proportion (Gibbons, 2001) 

Designed to test what proportion of test samples 

exceed a threshold; multiple test samples, one 

reference threshold 

Exact binomial test Widely recommended for chemical testing of % 

exceedances of a threshold; compares multiple test 

values to a single reference value 

Interval and Equivalence Testing 

The specific statistical methods described by Kilgour et al. (1998) are not particularly 

novel, i.e., the comparison of a statistic to the noncentral F distribution, but the pairing of 

two tests (interval and equivalence tests) and the application to bioassessment endpoints 

represents a new approach. Their approach was developed specifically for the comparison 

of a single test site to a population of reference sites for measures of biological condition. 

Both the interval and the equivalence tests can be applied to the comparison of a single 

test site value with values from a population of reference sites. Both tests have a low 

Type I error rate (false positive), but they can be used to balance false positives and 

negatives because they test the opposite null hypotheses. The interval null hypothesis 

tests for difference from reference and the equivalence null hypothesis tests for equality

to reference (Table 2). In this way, the tests are symmetric. By pairing these two tests, we 

can balance for the error rates associated with falsely determining a reference site to be 

impaired (Type I) and missing a site that is truly impaired (Type II; Appendix A). 

Although not relevant for this application to the SCI, the approach can also be 

generalized to a multivariate comparison of multiple indicators (e.g., metrics) against 

their respective thresholds (Bowman and Somers, 2006). 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-4
5

Table 2. Comparison of the interval and equivalence tests, hypotheses tested by each and their 

advantages and disadvantages relative to decisions about impairment. 

Statistical 

test 

HO HA Advantages Disadvantages 

Interval test Test site is

within the 

range of 

reference sites 

Test site is not

within the 

range of 

reference sites 

Protects the 

polluter (guards 

against 

unnecessary 

listing) 

Not as sensitive at 

detecting impairment 

Equivalence 

test 

Test site is not

within the 

range of 

reference sites 

Test site is

within the 

range of 

reference sites 

Protects the 

environment (very 

sensitive to 

impairment) 

Very sensitive and 

may identify reference 

sites as impaired 

How do we specify the test? 

Both tests use the same test statistic, the familiar F-statistic. Because we are testing for 

differences not from the mean, but for some distance from the mean, e.g., the 10th

percentile (our defined threshold), we need to use a noncentral F test. The null and 

alternate hypotheses are mirror images of each for the interval and equivalence tests. 

Interval test

The null hypothesis is that the test and reference sites are the same. The alternative

hypothesis is that test and reference sites are the different. 

HO (null):  ref T    , or “The difference between the mean SCI of the reference sites 

and the SCI at the test site is less than delta.” Delta can be any specified value, e.g., the 

10th percentile of SCI values for reference sites. 

HA (alternate):  ref T    , or “The test site is more different than the specified 

difference (delta) from the mean SCI.” 

Equivalence test

The null hypothesis is that the test and reference sites are different. The alternative

hypothesis is that test and reference sites are the same. 

HO:  ref T    , or “The difference between the mean SCI of the reference sites and 

the SCI at the test site is more than delta.” 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-5
6

HA:  ref T    , or “The test site is not different from the specified difference from the 

mean SCI for reference sites.” 

Test statistic

 

2

2

1

















 

ref

ref

ref

n

s

TX

F

How does statistical test work? 

Suppose we go to a stream site and sample for SCI and calculate a value of 46 out of a 

possible range of values from 0–100. We want to compare this observed value to the 10th

percentile of SCI values for reference sites (Table 3). The 10th percentile is derived from 

an assumption that reference sites follow an approximately normal distribution. The table 

value for the 10th percentile of a normal distribution (–1.28) is used to define 10th

percentile of the reference distribution. This z-table value is also used to define the 

noncentrality parameter, lambda. 

We calculate the F-statistic and use a statistical program (e.g., R or Excel) to look up the 

p-value associated with the F-statistic with 1 and 63 degrees of freedom and determine 

that we can reject the interval test and conclude that the test site is different from 

reference condition. 

Table 3. Worked example for comparing SCI at a test site to SCI values at reference sites. 

Description Value 

Mean SCI (reference sites) 66.3 

Standard deviation of SCI (reference sites) 12.6 

Number of reference sites 64 

10th Percentile of SCI (reference sites) 66.3–(1.28*12.6) = 50 

SCI for test site 46 

Standardized z-value for 10th percentile of 

reference (one-tailed test); used to define 

difference (delta) we are testing for 

–1.28 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-6
7

F-statistic  

2

2

64

1 6.12

463.66

















 = 166.1 

Degrees of freedom (1, 63) 

Lambda, noncentrality parameter; this value 

represents how far from the mean the test site 

should be, i.e., lower than the 10th percentile 

64 * (1.28)2

 = 104.9 

p-value for F-statistic (interval test) p = 0.04, significantly lower than the 10th

%tile of reference 

Conclusion “We are 95% confident the test site is 

impaired with impairment defined as below 

the 10th percentile of reference.” 

Statistical Decision Rules for SCI 

We can use the statistical framework of interval and equivalence testing to estimate 

uncertainty for SCI values at a test site based on different definitions of impairment. 

Table 4. Statistical decision rules for SCI values at a test site for threshold values defined as the 

1st, 5th, and 10th percentiles of SCI values for reference sites. Shown are values for p < 0.05. N = 

1 SCI value for each site. 

Impairment threshold 

(description) 

Impairment 

threshold 

(numeric) 

Impaired Undetermined Reference

2.5th percentile of reference 42 <37 37–45 >45 

5th percentile of reference 46 <41 41–49 >49 

10th percentile of reference 50 <47 47–53 >53 

Summary 

 Caveat: Statistical tests provide a framework for comparison and testing around a 

threshold, but cannot be used alone to define impairment thresholds. That decision 

also relies on scientific understanding and social values. 

The method described here combines two statistical tests, the interval test and the 

equivalence test, to compare a single test site value to a set of values from a population of 

reference sites. This approach satisfies most of our criteria for a statistical framework for 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-7
8

a biocriteria threshold. Pairing the interval and equivalence tests to evaluate percentiles 

from the reference site distribution represents a simple statistical decision rule that is easy 

to describe and legally defensible. We can estimate error rates for false positives and 

false negatives and balance them. The method is symmetric and can be easily inverted for 

delisting without any change to Type I or Type II errors. 

This approach does not deal explicitly with the case in which more than one sample value 

is collected from a test site. This problem could be addressed in two ways. For the simple 

case of two repeat visits to a test site, a new distribution of reference sites could be 

developed based on the average SCI value for two repeat visits to each reference site. 

These could be the most recent visit or two visits that were randomly chosen. 

A more general approach to incorporating test site variability could be accomplished with 

a simulation. Using our estimate of within-site variability, a distribution around point 

estimates of SCI could be defined. Repeat sampling from this distribution could be used 

to generate a distribution of F-statistics. In this way, we could model the variability 

associated with a single SCI for a test site. 

Combining the interval and equivalence tests provides a statistical framework to support 

impairment decisions, and is particularly suited to biocriteria because a single test value 

is compared to a population of reference values. 

Appendix A 

Example showing application of interval and equivalence testing for determining the status of a 

test site. The threshold for impairment is defined as the 10th percentile of SCI values for reference 

sites. Shown are the SCI value, difference between mean of reference and SCI, numerator and 

denominator for the F statistic, F, p-value for F for the interval and equivalence tests, status call, 

and result for the interval and equivalence tests. See Table 3 for values used in calculations. SCI 

values greater than or equal to the mean for reference sites (SCI = 66–100) are not shown. 

SCI Mu-imp F_Num F_Den F P_Int P_Equ STATUS Interv. Equiv. 

1 65.3 4264.1 2.48 1719.0 0.00 1.00 IMPAIRED >delta >=delta

2 64.3 4134.5 2.48 1666.7 0.00 1.00 IMPAIRED >delta >=delta

3 63.3 4006.9 2.48 1615.3 0.00 1.00 IMPAIRED >delta >=delta

4 62.3 3881.3 2.48 1564.6 0.00 1.00 IMPAIRED >delta >=delta

5 61.3 3757.7 2.48 1514.8 0.00 1.00 IMPAIRED >delta >=delta

6 60.3 3636.1 2.48 1465.8 0.00 1.00 IMPAIRED >delta >=delta

7 59.3 3516.5 2.48 1417.6 0.00 1.00 IMPAIRED >delta >=delta

8 58.3 3398.9 2.48 1370.2 0.00 1.00 IMPAIRED >delta >=delta

9 57.3 3283.3 2.48 1323.6 0.00 1.00 IMPAIRED >delta >=delta

10 56.3 3169.7 2.48 1277.8 0.00 1.00 IMPAIRED >delta >=delta

11 55.3 3058.1 2.48 1232.8 0.00 1.00 IMPAIRED >delta >=delta

12 54.3 2948.5 2.48 1188.6 0.00 1.00 IMPAIRED >delta >=delta

13 53.3 2840.9 2.48 1145.2 0.00 1.00 IMPAIRED >delta >=delta

14 52.3 2735.3 2.48 1102.7 0.00 1.00 IMPAIRED >delta >=delta

15 51.3 2631.7 2.48 1060.9 0.00 1.00 IMPAIRED >delta >=delta

16 50.3 2530.1 2.48 1019.9 0.00 1.00 IMPAIRED >delta >=delta

17 49.3 2430.5 2.48 979.8 0.00 1.00 IMPAIRED >delta >=delta

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-8
9

SCI Mu-imp F_Num F_Den F P_Int P_Equ STATUS Interv. Equiv. 

18 48.3 2332.9 2.48 940.4 0.00 1.00 IMPAIRED >delta >=delta

19 47.3 2237.3 2.48 901.9 0.00 1.00 IMPAIRED >delta >=delta

20 46.3 2143.7 2.48 864.2 0.00 1.00 IMPAIRED >delta >=delta

21 45.3 2052.1 2.48 827.2 0.00 1.00 IMPAIRED >delta >=delta

22 44.3 1962.5 2.48 791.1 0.00 1.00 IMPAIRED >delta >=delta

23 43.3 1874.9 2.48 755.8 0.00 1.00 IMPAIRED >delta >=delta

24 42.3 1789.3 2.48 721.3 0.00 1.00 IMPAIRED >delta >=delta

25 41.3 1705.7 2.48 687.6 0.00 1.00 IMPAIRED >delta >=delta

26 40.3 1624.1 2.48 654.7 0.00 1.00 IMPAIRED >delta >=delta

27 39.3 1544.5 2.48 622.6 0.00 1.00 IMPAIRED >delta >=delta

28 38.3 1466.9 2.48 591.3 0.00 1.00 IMPAIRED >delta >=delta

29 37.3 1391.3 2.48 560.9 0.00 1.00 IMPAIRED >delta >=delta

30 36.3 1317.7 2.48 531.2 0.00 1.00 IMPAIRED >delta >=delta

31 35.3 1246.1 2.48 502.3 0.00 1.00 IMPAIRED >delta >=delta

32 34.3 1176.5 2.48 474.3 0.00 1.00 IMPAIRED >delta >=delta

33 33.3 1108.9 2.48 447.0 0.00 1.00 IMPAIRED >delta >=delta

34 32.3 1043.3 2.48 420.6 0.00 1.00 IMPAIRED >delta >=delta

35 31.3 979.7 2.48 394.9 0.00 1.00 IMPAIRED >delta >=delta

36 30.3 918.1 2.48 370.1 0.00 1.00 IMPAIRED >delta >=delta

37 29.3 858.5 2.48 346.1 0.00 1.00 IMPAIRED >delta >=delta

38 28.3 800.9 2.48 322.9 0.00 1.00 IMPAIRED >delta >=delta

39 27.3 745.3 2.48 300.4 0.00 1.00 IMPAIRED >delta >=delta

40 26.3 691.7 2.48 278.8 0.00 1.00 IMPAIRED >delta >=delta

41 25.3 640.1 2.48 258.0 0.00 1.00 IMPAIRED >delta >=delta

42 24.3 590.5 2.48 238.0 0.00 1.00 IMPAIRED >delta >=delta

43 23.3 542.9 2.48 218.9 0.00 1.00 IMPAIRED >delta >=delta

44 22.3 497.3 2.48 200.5 0.01 0.99 IMPAIRED >delta >=delta

45 21.3 453.7 2.48 182.9 0.02 0.98 IMPAIRED >delta >=delta

46 20.3 412.1 2.48 166.1 0.04 0.96 IMPAIRED >delta >=delta

47 19.3 372.5 2.48 150.2 0.09 0.91 Undetermined <=delta >=delta

48 18.3 334.9 2.48 135.0 0.18 0.82 Undetermined <=delta >=delta

49 17.3 299.3 2.48 120.7 0.31 0.69 Undetermined <=delta >=delta

50 16.3 265.7 2.48 107.1 0.48 0.52 Undetermined <=delta >=delta

51 15.3 234.1 2.48 94.4 0.67 0.33 Undetermined <=delta >=delta

52 14.3 204.5 2.48 82.4 0.82 0.18 Undetermined <=delta >=delta

53 13.3 176.9 2.48 71.3 0.93 0.07 Undetermined <=delta >=delta

54 12.3 151.3 2.48 61.0 0.98 0.02 REFERENCE <=delta <delta 

55 11.3 127.7 2.48 51.5 1.00 0.00 REFERENCE <=delta <delta 

56 10.3 106.1 2.48 42.8 1.00 0.00 REFERENCE <=delta <delta 

57 9.3 86.5 2.48 34.9 1.00 0.00 REFERENCE <=delta <delta 

58 8.3 68.9 2.48 27.8 1.00 0.00 REFERENCE <=delta <delta 

59 7.3 53.3 2.48 21.5 1.00 0.00 REFERENCE <=delta <delta 

60 6.3 39.7 2.48 16.0 1.00 0.00 REFERENCE <=delta <delta 

61 5.3 28.1 2.48 11.3 1.00 0.00 REFERENCE <=delta <delta 

62 4.3 18.5 2.48 7.5 1.00 0.00 REFERENCE <=delta <delta 

63 3.3 10.9 2.48 4.4 1.00 0.00 REFERENCE <=delta <delta 

64 2.3 5.3 2.48 2.1 1.00 0.00 REFERENCE <=delta <delta 

65 1.3 1.7 2.48 0.7 1.00 0.00 REFERENCE <=delta <delta 

66 0.3 0.1 2.48 0.0 1.00 0.00 REFERENCE <=delta <delta 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-9
10

References 

Bowman, M. F. and K. M. Somers. 2006. Evaluating a novel Test Site Analysis (TSA) 

bioassessment approach. Journal of the North American Benthological Society 

25:712-727. 

Gibbons, R.D. 2003. A statistical approach for performing water quality impairment 

assessments. Journal of the American Water Resources Association 39:841–849. 

Kilgour, B. W., K. M. Somers, and D. E. Matthews. 1998. Using the normal range as a 

criterion for ecological significance in environmental monitoring and assessment. 

Ecoscience 5:542–550. 

State Water Resources Control Board (SWRCB). 2004. Functional Equivalent 

Document: Water Quality Control Policy for Developing California s Clean Water 

Act Section 303(d) List. California Division of Water Quality. 

http://www.swrcb.ca.gov/water_issues/programs/tmdl/303d_policydocs.shtml. See 

Document #100 on this web site. 

Additional References 

Barnett, V. and Bown, M. 2002. Statistically meaningful standards for contaminated sites 

using composite sampling. Environmetrics 13: 1-13. 

Bowman, M. F., and K. M. Somers. 2005. Considerations when using the reference 

condition approach for bioassessment of freshwater ecosystems. Water Quality 

Research Journal of Canada 40:347–360. 

Goudey, R. 2007. Do statistical inferences allowing three alternative decision give better 

feedback for environmentally precautionary decision-making? Journal of 

Environmental Management 65:338-344. 

Lin, Pi-Erh, D. Meeter, X.-F. Niu. 2000. A nonparametric procedure for listing and 

delisting impaired waters based on criterion exceedances. Technical report 

submitted to Florida DEP. 

Parkhurst D.F. 2001. Statistical significance tests: equivalence and reverse tests should 

reduce misinterpretation. BioScience, 51, 1051–1057. 

Shabman, L. and E. Smith. 2003. Implications of Applying Statistically Based Procedures 

for Water Quality Assessment. Journal of Water Resources Planning and 

Management. 

Smith, E. P., Ye, K., Hughes, C., and Shabman, L. 2001. Statistical assessment of 

violations of water quality standards under Section 303d of the Clean Water Act. 

Environ. Sci. Technol., 35: 606–612. 

Smith, E. P., Zahran, A., Mahmoud, M., and Ye, K. 2003 . Evaluation of water quality 

using acceptance sampling by variables. Environmetrics, 14:373-386. 

Warn, A. E. 1989. Auditing the quality of effluent discharges. Environmental Monitoring 

and Assessment 12:165–181. 

TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters B14-10