TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-1 

4. 

Chapter 4: Methodology for Deriving U.S. EPA’s Proposed Criteria 

for Canals 

There are thousands of miles of canals in Florida, particularly in the southeastern part of the 

state. Canals are artificial waterways that are the result of modifications to existing rivers or 

streams, or waters that have been created for various purposes, including drainage and flood 

control (stormwater management), irrigation, navigation, and recreation. These canals also allow 

for the creation of many waterfront home sites in Florida. Ecosystems that existed in rivers and 

streams before their modification into canals are altered. These changes can affect fish and 

wildlife and plant growth. Newly created canals may have a tendency to fill with aquatic plants. 

Canals in south Florida vary greatly in size and depth. They can be anywhere from a few feet 

wide and a few feet deep to hundreds of feet wide and as deep as 30 to 35 feet. 

South Florida canals vary in their hydrology and behavior due to their size, function, and 

seasonality. Shallow canals with slow water flow have poor turnover of water and little flushing. 

Large canals also can have low flow and turnover during the dry season. In contrast, during the 

wet season these same large canals are flowing systems that quickly move large volumes of 

water, as they were designed to. Excess nutrients in canals, in combination with poor water 

circulation and decreased levels of dissolved oxygen, can lead to accelerated eutrophication and 

adverse impacts on other forms of aquatic life such as fish and other aquatic animals. In these 

canals, the accumulation of decaying organic matter on the canal bottom can also adversely 

affect healthy aquatic ecosystems. 

South Florida canals are highly managed waterways. Some canals are prone to an overabundance 

of aquatic plants. Without regular and frequent management, dense vegetation can clog the 

waterways, making navigation difficult and slowing the movement of water through the canal 

system. This can interfere with flood control, boating, and fishing. Aquatic plants (like plants in 

the terrestrial environment) respond and grow when fertilized with nutrients such as phosphorus 

and nitrogen, and thus nutrient runoff into canals is likely a significant contributor to both 

nuisance algal blooms and clogging of canal systems by aquatic plants. 

The U.S. Environmental Protection Agency (U.S. EPA) (USEPA 2000a) describes several 

general approaches, with certain options, that can be used to develop numeric nutrient criteria for 

streams. In all cases, consistent with the Clean Water Act (CWA), the objective is to determine a 

level of nutrients that will provide protection of aquatic life both in-stream and downstream of a 

point of application. 

One approach that EPA recommends is the reference condition approach. With this approach, a 

reference condition in terms of nutrient concentration is empirically determined from a statistical 

distribution of nutrient data that can be assumed to inherently protect aquatic life. One option 

used in the reference condition approach involves setting criteria based on an inclusive 

distribution of nutrient values obtained from minimally disturbed reference sites in a designated 

ecoregion (based on climate and geology). In this case, U.S. EPA recommends using an upper 

percentile value of this distribution to represent a level of nutrient concentration that will 

inherently protect aquatic life. This methodology will ensure that nutrient concentrations 

associated with biologically healthy, well-balanced communities would be considered acceptable 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-2 

as protective of those communities. Lacking reference or benchmark canal sites, U.S. EPA is 

proposing criteria based on the 75th percentile from all available freshwater canal data from sites 

located in waterbody IDs (WBIDs) that have not been identified for nutrient impairment. 

It is noted that this approach is an inference model and does not definitively demonstrate that 

exceeding the threshold established by the distribution of reference sites results in harm 

(impairment) to the aquatic life in a particular waterbody. Multiple factors, such as water 

velocity, residence time, availability of the other nutrient, presence of grazers, availability of 

light (due to tree cover and/or water transparency), and availability of suitable habitat, can 

strongly influence the expression of biological responses to nutrients across waterbodies. The 

“minimally affected reference water approach” identifies concentrations that are presumed to be 

inherently protective of the waterbody because those concentrations are associated with 

demonstrated minimally disturbed systems with healthy biological communities. 

Because soil or substrate type at the bottom of a canal can influence the nutrient cycling and 

relationships between the observed biological response and the total phosphorus (TP) and total 

nitrogen (TN) levels in canals, U.S. EPA used data on soil types in south Florida, along with 

knowledge of the Everglades Agricultural Area (EAA) and the Everglades Protection Area 

(EvPA), to subdivide the canal areas for criteria derivation (see Figures 4-1a and 4-1b). Thus the 

first step in this and other analyses was to group canals and canal data by soil type. The four 

groupings are the histosol and entisol soils of the EAA; histosol and entisol soils of the EvPA; 

spodosol and alfisol soils and areas west of the EvPA and EAA (hereafter, West Coast); and 

spodosol, entisol, and alfisol soils and areas east of the EvPA and EAA (hereafter East Coast). 

U.S. EPA performed the following steps to process the available data: 

 FDEP provided the “SFWMD Canal Stream” database for freshwater canals electronically. 

The database included 1,291,363 results for 2,174 sites. Similarly, FDEP provided “Broward 

County Canal Data,” which was an Excel sheet that contained 49,056 results associated with 

43 sites. Miami-Dade County provided its data, which included 293,159 results for 175 

stations. FDEP provided a STORET retrieval of Miami-Dade (Organization ID = 

“21FLDADE”) data as well. The STORET retrieval included 303,956 results for 1,041 sites. 

 U.S. EPA performed an extensive comparison of the FDEP-supplied Miami-Dade data with 

the data directly from Miami-Dade County. This comparison included a review of site 

locations, sampling dates, and parameters analyzed. Although the FDEP-supplied data 

included data for more sites, U.S. EPA found that 878 of these sites were not located in canals 

or did not include key parameters of interest (e.g., no nutrient data). The remaining 163 sites 

coincided with the Miami-Dade-supplied data set, but usually had a shorter period of record, 

with the Miami-Dade-supplied data set supplying data from 1998 to the present. For those 

sites with coincident sampling dates, selected results were compared and found to be identical. 

Based on this assessment, the FDEP-supplied Miami-Dade data were set aside and not 

considered further.3

 

3

 It is noted that FDEP does not have a record of auditing DERM’s laboratory or its contract laboratories. The DERM laboratory 

did participate in the Everglades round robin inter-laboratory phosphorus comparability exercises (Ken Weaver, personal 

communication, October 12, 2009). 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-3 

 The remaining 2,392 sites (South Florida Water Management District (SFWMD): 2,174, 

Broward: 43, Miami-Dade: 175) were attributed with ecoregions and WBID. Of these sites, 

994 were not located in the Everglades ecoregion and were removed from further analysis. In 

addition, 17 sites were dropped because of missing or poor location data; 101 sites were 

removed because they reported only “marine” matrix samples; 88 sites were dropped because 

more than 5% of the specific conductivity results were greater than 3,000 μS/cm;4 and 498 

sites were excluded because they were associated with stormwater treatment areas, marshes, 

wetlands, class IV canals, private pump stations, etc. and not applicable to the current 

analysis.5

 Finally, 163 sites were dropped because they were from Lake Okeechobee or 

nearby canals that might be expected to behave more like a lake than a freshwater canal. 

 Overall, 531 sites were maintained for further analysis, and they include samples collected 

from 1973 through 2009. Approximately two-thirds (64%) of the samples were collected from 

1999 through 2009. The samples are uniformly spaced throughout the year with each month 

having from 7% to 9% of the total samples. Considering the geographic diversity of the site 

locations and the temporal spread (both yearly and seasonally), these data take into 

consideration temporal, seasonal and spatial variability. 

 Using a geographic information system (GIS), site locations were attributed to one of the four 

canal regions. Because of the complexity of the existing canal network, all sites near or along 

the borders of the four canal regions were reviewed manually to ensure that sites were placed 

in the correct region. The result of these efforts is displayed in Figure 4-2. 

 Results with data qualifiers indicating estimated results—value not accurate, blank 

contamination, laboratory protocol issues, etc.––were deleted from the analysis. One-half the 

detection limit was used for results where the data qualifier indicated not-detected or the result 

was reported as “<.”6

 The sample type was also reviewed, and results with the following 

samples types were deleted: rainwater samples, well water samples, sediment samples, typha 

(cattails), cladium (sawgrass), and “NOB No Bottle Sample.” Results with questionable units 

were also deleted. For the Broward County Canal data, replicates were averaged. 

 In the analysis of benchmark sites in northern Florida, FDEP removed observations greater 

than the upper adjacent value.7

 That approach was considered for the “all-streams” analysis in 

northern Florida, but it was ultimately abandoned in lieu of a simple screening threshold of 

15 mg/L and 100 mg/L for TP and TN, respectively, to remove erroneous data. The same 

approach of deleting samples with a TP or TN concentration greater 15 mg/L or 100 mg/L 

was adopted herein: Two samples were deleted from the SFWMD Canal Stream database, and 

no samples were deleted from the Broward County or Miami-Dade County Canal Data. No 

chlorophyll observations were removed, with a maximum chlorophyll value of 482 μg/L 

observed. 

 The remaining data were organized by “Region-WBID-Year,” and observation counts were 

made in a manner identical to that used in FDEP’s analysis of benchmark sites. Whenever the 

chlorophyll observation count for a particular Region-WBID-Year was less than 4, all 

 

4

 Specific conductivity was examined to exclude sites with higher salinities, indicative of marine canals. 5

 FDEP and EPA staff performed this screening process manually by examining site descriptions and GIS displays. 6

 Remark code processing also applied to specific conductivity results analyzed in the above site-screening process. 7

 The upper adjacent value is defined as the largest observation that is less than or equal to the upper quartile plus 1.5 times the 

inter-quartile range. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-4 

chlorophyll observations for that Region-WBID-Year were dropped. A similar process for TP 

and TN was implemented. 

 The chlorophyll, TP, and TN data were transformed by taking the natural logarithm (LN(TP) 

and LN(TN)). Log transformation converts data that are positively skewed to more 

symmetrically (closer to normally) distributed data. The mean LN(chlorophyll), LN(TP), and 

LN(TN) were computed for each combination of Region-WBID-Year. 

Following the screening process to eliminate Region-WBID-Years with less than 4 samples, 

there were 68,544 samples,8 of which 5,524 samples had chlorophyll results. Summarizing 

(computing the mean of the transformed data) to Region-WBID-Year level resulted in 1,622 

records, of which 275 records had chlorophyll data. Of the 275 records with chlorophyll data, 

258 records had matched TN and/or TP data to enable regression analysis. Figure 4-3 shows the 

72 stations with chlorophyll data. Figure 4-4 shows the same 72 stations together with the 

WBIDs that FDEP has identified as nutrient-impaired. The 1,622 records summarized to the 

Region-WBID-Year level are the starting point for the analyses described in this section, as well 

as alternative analyses described later. 

The data were further screened by selecting only the sites located in WBIDs that were not listed 

as nutrient-impaired on FDEP’s 303(d) list. Allowing for potential inaccuracies in location data, 

sites that were within 1,000 meters of a WBID that was listed by FDEP as nutrient-impaired 

were excluded (see Figure 4-5). Using the means of the natural logarithm-transformed data at the 

Region-WBID-Year level, the mean and standard deviation were estimated by region as well as 

for all regions. The mean and standard deviation for TP were also computed for the West Coast, 

EAA, and East Coast regions combined. (This “three-region” result was developed in 

acknowledgement of the existing TP criteria for the EvPA.) Using the mean and standard 

deviation, the theoretical 75th (and other) percentiles were estimated assuming a lognormal 

distribution (see Table 4-1). 

The 75th percentile chlorophyll and TN concentrations across all four regions at sites located 

within WBIDs not listed was impaired for nutrients are 4.0 μg/L and 1.6 mg/L, respectively. The 

75th percentile TP concentration at sites located within WBIDs not listed as impaired for 

nutrients across the West Coast, EAA, and East Coast regions is 0.042 mg/L. 

4a. Other Analyses Conducted and Considered by U.S. EPA in the Derivation of 

Proposed Numeric Nutrient Criteria for Canals 

Another option that can be used in a reference condition approach is to extrapolate a reference 

condition of nutrient concentration that can be assumed to inherently protect aquatic life. In this 

application of the all canals approach, a distribution of nutrient data is assembled from all sites in 

an ecoregion (or, alternatively, a random selection of sites). Assuming the pool of sites in the 

distribution is large enough to represent all waters in the region and can be presumed to 

reasonably reflect the full range of ambient conditions with a disturbance gradient from least to 

most impacted, the selection of a lower percentile of this distribution (e.g., 25th percentile), 

 

8

 To be included, that sample must have at least one result associated with TN, TP, or chlorophyll. The process of excluding 

Region-WBID-Year with less than four observations was adopted from procedures used by FDEP in its reference condition 

approach analyses. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-5 

should provide a nutrient level that can be assumed to inherently protect aquatic life. Although an 

all sites data distribution approach does not directly link nutrient concentrations to valued 

ecological attributes (healthy, well-balanced populations of flora or fauna), it does presume that the 

25th percentile will closely approximate a reference condition for that geographic location. Several 

examples with data from actual reference sites have supported this assumption (USEPA 2000a, b). 

Using the means of the natural logarithm-transformed data at the Region-WBID-Year level (the 

1,622 records described earlier), the mean and standard deviation were estimated by region as 

well as for all regions. The mean and standard deviation for TP were also computed for the West 

Coast, EAA, and East Coast regions combined. (This three-region result was developed in 

acknowledgement of the existing TP criteria for the EvPA.) As one effort to consider canal 

benchmark conditions, summary statistics for sites along Rt. 41 were computed. The Rt. 41 sites 

(see Figure 4-1 and 4-2) are surrounded by the EvPA to the east and the Big Cypress National 

Preserve to the west. Site S356, the most eastward site, is one mile west of Krome Avenue on the 

L-29 canal, north side of Tamiami Trail. Site BC19, the most westward site, is on Rt. 41 at 

Bridge #69. Although these sites have not been vetted through the same process as the sites for 

the streams and rivers analyses, they are generally believed to represent canal sites with the 

lowest impact from anthropogenic activities in the study area. These summary statistics were 

used to compute theoretical distributions, assuming a lognormal distribution, and were backtransformed (see Table 4-2). The results follow: 

 Chlorophyll. The 25th percentile distribution ranged from 1.9 to 2.2 μg/L for the EvPA, West 

Coast, East Coast, and all regions. The 25th percentile distribution was 6.3 μg/L for the EAA. 

The 75th percentile distribution for Rt. 41 was 3.4 μg/L. 

 TN. The 25th percentile distribution ranged from 0.8 to 1.4 mg/L for the EvPA, West Coast, 

East Coast, and all regions. The 25th percentile distribution was 2.1 mg/L for the EAA. The 

75th percentile distribution for Rt. 41 was 1.3 mg/L. 

 TP. The 25th percentile distribution ranged from 0.013 to 0.023 mg/L for the EvPA, West 

Coast, East Coast, and all regions. Excluding the EvPA, the 25th percentile distribution was 

0.018 mg/L for the remaining three regions when combined. The 75th percentile distribution 

for Rt. 41 was 0.018 mg/L. 

4b. Duration and Frequency for Proposed Canals Criteria 

Aquatic life water quality criteria contain three components: magnitude, duration, and frequency. 

The criterion-duration for this magnitude (or averaging period) is specified as an annual 

geometric mean. U.S. EPA is proposing two expressions of allowable frequency, both of which 

are to be met. First, U.S. EPA proposes a no more than one in three years excursion frequency 

for the annual geometric mean criteria for canals. Second, U.S. EPA proposes that the long-term 

arithmetic average of annual geometric means not exceed the criterion-magnitude concentration. 

These proposed duration and frequency components of the criteria are consistent with the data 

set used to derive the criteria, which contained data from multiple years of record, all seasons, 

and a variety of hydrologic conditions. This frequency of excursions should not result in 

unacceptable effects on aquatic life as it will allow the canal aquatic system enough time to 

recover from the occasionally elevated year of nutrient loadings. The full rationale for both lakes 

(section 1d) and streams (section 2d) are also relevant for canals. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-6 

4c. Alternative Approach Considered by U.S. EPA 

i. Stressor-Response Approach 

Another approach described in U.S. EPA guidance for developing nutrient criteria (USEPA 

2000a) is to quantify a stressor-response relationship between excess nutrients and adverse 

effects on aquatic life, and then use that relationship to establish “effects-based” criteria. The 

necessary observed dose-response relationship could be described by a model (e.g., trophic state 

classification, regional predictive model, nutrient sensitive biocriteria), which in turn would link 

nutrient concentrations to the relative risk of environmental harm. Nutrient criteria based on a 

cause/effect relationship between nutrients and valued ecological attributes could thus be 

considered directly linked to maintaining designated uses. Such criteria might be scientifically 

defensible but have proved to be elusive. Although the underlying relationship certainly exists, in 

nature it has often been difficult to quantify because of the complexity of nutrient dynamics and 

the influence of interfering factors. 

In this analysis, matched chlorophyll and nutrient data were analyzed using linear regression. For 

those regression analyses that were statistically significant, an assumed geometric mean 

chlorophyll concentration of 20 μg/L was used to estimate the corresponding nutrient 

concentration that results from the 90% prediction interval of the regression equation. The 

selection of 20 μg/L does not imply a linkage to a particular designated use in canals. As 

described earlier and shown in Figures 4-3 and 4-4, 72 of the 531 sites had chlorophyll data for 

the stressor-response analysis performed. Those sites resulted in 258 records of chlorophyll 

matched with TP or TN. 

TP. Figure 4-6a is a bivariate plot of annual geometric mean chlorophyll as function of TP. 

Figure 4-6b presents the fitted regression equation along with the 90% confidence interval using 

data from across all regions. Figures 4-6c and 4-6d provide the regression analysis statistics and 

residual plots, respectively. This regression model explains 49% of the variance; the residual 

plots suggest that the residuals have a constant variance, are independent, and are normally 

distributed. Based on this regression analysis, an annual geometric mean TP concentration less 

than 0.124 mg/L would yield an annual geometric mean chlorophyll concentration of less than 

20 μg/L with a 90% confidence level. 

The TP regression analysis was rerun excluding data from the EvPA to evaluate model 

sensitivity (see Figure 4-7). There were negligible differences: An annual geometric mean TP 

concentration less than 0.124 mg/L would yield an annual geometric mean chlorophyll 

concentration of less than 20 μg/L with a 90% confidence level. 

The region-specific regression analyses for TP are shown in Figure 4-8. While the model for the 

EAA was not statistically significant, the regression models developed for the other three regions 

explained from 43% to 73% of the data variability. Residual plots for the EvPA, West Coast, and 

East Coast were visually examined and found to have constant variance, independence, and 

normal distribution. Region-specific regression diagnostics for statistically significant models are 

shown in Figures 4-9 through 4-11. The TP concentrations for the EvPA, West Coast, and East 

Coast regions that corresponded to a chlorophyll concentration of 20 μg/L were computed as 

0.052, 0.25, and 0.069 mg/L, respectively. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-7 

TN. Figure 4-12a is a bivariate plot of annual geometric mean chlorophyll as function of TN by 

canal region. Figure 4-12b presents the fitted regression equation along with the 90% confidence 

interval using data from across all regions. Figures 4-12c and 4-12d provide the regression 

analysis statistics and residual plots, respectively. This regression model explains 16.5% of the 

variance, and the residual plots suggest that the residuals are normally distributed. U.S. EPA 

acknowledges the visual appearance of a higher variance in the middle of the TN range. The 

lowest and highest chlorophyll concentrations were investigated further. The six records with 

annual geometric mean chlorophyll concentrations less than 1 μg/L are attributed to WBID 3283 

(1999 and 2003) and 3284 (1999, 2000, 2003 and 2006), both along Snake Creek Canal in 

northern Dade County. The five records with annual geometric mean chlorophyll concentrations 

greater than 15 μg/L are reported with data from 2002 through 2007 for WBIDs 3264 and 3226F 

(twice) (Hillsboro Canal in northern Broward County), 3271 (Pompano Canal in Broward 

County), 8063 (West Coast in Collier County), and 3261C (SR 29 Canal in Collier County). 

There was insufficient evidence in the data analyzed to suggest that these results were 

anomalous, and thus they were maintained. It is also noted that the higher spread in the data at 

mid-range TN levels may also be attributed, in part, to the higher frequency of observations in 

this range of the data. Based on this regression analysis, an annual geometric mean TN 

concentration less than 3.6 mg/L would yield an annual geometric mean chlorophyll 

concentration of less than 20 μg/L with a 90% confidence level. 

The region-specific regression analyses for TN are shown in Figure 4-13. The models for the 

EAA and EvPA regions were not statistically significant. The regression models developed for 

the West and East Coast regions explained from 9% to 21% of the data variability. Residual plots 

for the West and East Coast regions were visually examined and found to be normally 

distributed. Region-specific regression diagnostics for statistically significant models are shown 

in Figures 4-14 and 4-15. The variance associated with the West Coast residuals did show a 

tendency to increase as a function of TN. The TN concentrations for the West and East Coast 

regions that corresponded to a chlorophyll concentration of 20 μg/L were computed as 4.0 and 

2.35 mg/L, respectively. 

Alternative Regression Analysis. Quantile regression may also be used to analyze the same 

nutrient-chlorophyll data that were analyzed with linear regression. In this application, quantile 

regression computes an estimate on the 90th conditional quantile function of the response, given 

the stressor (TN or TP). The function presumes a linear specification for the quantile regression 

model, i.e., that the formula defines a model that is linear in parameters. The function minimizes 

a weighted sum of absolute residuals that can be formulated as a linear programming problem. 

To minimize the impact of skewed data, the log10 (logarithm, base 10) transformation was 

computed before applying the quantile regression algorithm. Table 4-3 presents the Spearman’s 

rho correlation coefficient, the quantile regression slope, and intercept terms. Candidate TN and 

TP endpoints were computed using the regression terms and an assumed chlorophyll 

concentration of 20 μg/L (see Table 4-3). Again, the selection of 20 μg/L does not imply a 

linkage to a particular designated use in canals. Figures 4-16 through 4-19 presents bivariate 

plots of chlorophyll as a function of TN and TP (in log10 space) overall and for the regions with 

significant correlations. Each plot displays the Spearman’s rho correlation coefficient, the 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-8 

LOESS9

 smoothed line (black line), and the quantile regression results (blue line and blue text 

for the regression terms). 

ii. Calculation of TP Criteria for the EAA Using a Downstream Protection Approach 

Florida has adopted and U.S. EPA has approved a long-term geometric mean TP criterion for the 

EvPA of 10 ppb. This 10 ppb long-term geometric mean water quality criterion applies 

throughout the EvPA. Walker (2005) determined that the ratio of the long-term flow-weighted 

mean to the long-term geometric mean across all stormwater treatment areas (STAs) for 24 water 

years is 1.18, with a standard error 0.04. The criterion has been calculated to translate into an 

equivalent maximum annual flow-weighted concentration of approximately 16 ppb at discharges 

into the Everglades (Payne et al. 2005; Walker 2005). Payne et al. (2005) concluded that a TP 

effluent limit of 15.5 ppb expressed as a maximum annual flow-weighted mean is recommended 

for incorporation into permits for all discharges into the EvPA to ensure that the discharges do 

not cause or contribute to exceedances of the phosphorus criterion in the downstream marsh 

receiving waters. This target can be reevaluated in the future as more data become available. 

Although the annual maximum limit has not been formally adopted by FDEP as a Water Quality 

Based Effluent Limit (WQBEL), at the present time it represents the direct translation of the 

long-term numeric criterion to the discharge such that all water delivered to the EvPA would 

meet the numeric criterion with a level of confidence. 

The 16 ppb annual maximum concentration limit might be used as a downstream protective 

value for the EvPA. This approach should ensure that all waters delivered to the EvPA at WBIDs 

3252B and 3265B would meet the existing TP criterion for the EvPA. 

Approach 1: Estimate STA inflow loads resulting in WQS in downstream waters 

To meet the 10 ppb criterion and protect downstream waters of the EvPA, STAs must discharge 

an annual maximum of 16 ppb. STA performance is a function of phosphorus loading rate 

(PLR), expressed as grams TP per acre of STA area per year (Kadlec 2006; Pietro et al. 2007; 

Goforth et al. 2007a–2007f). As PLR increases, the projected STA discharge TP concentration 

also increases. Relationships also have been explored for TP concentration at STA inflow versus 

TP outflow concentration (Pietro et al. 2007). The best predictor of STA outflow TP 

concentration appears to be PLR. 

Pietro et al. (2007) summarize annual STA performance across all STAs for the history of the 

program. From 1995 to 2006 the three years with the lowest STA outflow concentrations (1997, 

19 ppb; 1998, 20 ppb; and 1999, 19 ppb) had corresponding PLRs of 0.93, 0.82 and 0.75, and 

inflow TP concentrations of 98 ppb, 98 ppb, and 90 ppb, respectively. Outflow TP for the other 

nine years ranged from 20 ppb to 44 ppb; PLR was always greater than 1.0, ranging from 1.26 to 

3.51; and inflow TP concentrations were all greater than 100 ppb, ranging from 109 ppb to 147 

ppb. The years with the performance closest to the 16 ppb discharge goal all had PLRs less than 

1.0 g/m2/yr and inflow TP less than 100 ppb. This 100 ppb concentration might be the initial 

 

9

 We used a locally weighted scatterplot smoothing (LOESS) technique to visually examine possible trends along environmental 

gradients. The LOESS technique (Cleveland 1979) models nonlinear relationships where linear methods do not perform well, and 

fits simple models to localized subsets of the data to construct a function that describes, essentially, the central tendency of data 

along nutrient gradients. We used a smoothing factor of 0.7 to compute lowess curves. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-9 

STA inflow target concentration until such time that it is demonstrated that the STAs can attain 

the maximum annual flow-weighted concentration of 16 ppb discharge goal with a higher inflow 

concentration. 

iii. Upper Percentile Criterion with Low Exceedance Frequency 

The purpose of this analysis is to compute a “not to exceed” concentration for a single sample 

based on sites that meet the annual geometric mean criteria proposed for canals. The 

computational approach is the same as that used for other waterbodies of the inland criteria work. 

Using the canal water quality data prior to summarizing to the Region-WBID-Year level (i.e., the 

point with 68,544 samples, of which 5,524 samples had chlorophyll results), the geometric mean 

chlorophyll, TN, and TP were computed for each “site-year.” All data associated with site-year 

geometric means greater than the corresponding proposed nutrient criteria were removed from 

the analysis. 

The “site” standard deviation was computed using the remaining data, regardless of year. Using 

the proposed criteria and the standard deviation, the one-sided 95% prediction interval was used 

to estimate the not to exceed criteria for a single observation. Table 4-4 includes summary 

statistics and the estimated not to exceed concentration. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-10 

Table 4-1. Distributional Statistics Computed from the Natural Logarithm Transformed Data at 

Sites Located in WBIDs not Listed as Nutrient Impaired 

Observations Distribution Statistics 

Estimated from 

LN 

Transformed 

Data 

Percentiles Computed from Mean and 

Standard Deviation Back Transformed

 to Regular Units 

Parameter Region 

Number 

of 

WBIDYears 

Number 

of 

WBIDs

Mean 

LN 

Std 

Dev LN

10th

%tile 

LN 

25th

%tile 

LN 

50th

%tile 

LN 

75th

%tile 

LN 

90th

%tile 

LN 

EAA 

(Not Impaired) n/a n/a n/a n/a n/a n/a n/a n/a n/a 

EvPA 

(Not Impaired) 25 2 1.059 0.521 1.478 2.028 2.882 4.095 5.618 

West 

(Not Impaired) 78 9 1.005 0.393 1.652 2.097 2.732 3.561 4.519 

East 

(Not Impaired) 40 4 1.195 0.676 1.389 2.095 3.305 5.215 7.862 

Chlorophyll 

(percentiles 

are in μg/L) 

All 

(Not Impaired) 131 14 1.046 0.508 1.483 2.019 2.845 4.008 5.457 

 

EAA 

(Not Impaired) 3 1 0.320 0.327 0.906 1.105 1.377 1.717 2.095 

EvPA 

(Not Impaired) 134 7 0.398 0.391 0.902 1.144 1.489 1.939 2.458 

West 

(Not Impaired) 206 13 -0.055 0.408 0.561 0.719 0.946 1.246 1.596 

East 

(Not Impaired) 229 21 0.127 0.450 0.637 0.838 1.135 1.538 2.022 

TN 

(percentiles 

are in 

mg/L) 

All 

(Not Impaired) 547 40 0.137 0.459 0.636 0.841 1.146 1.563 2.065 

 

EAA 

(Not Impaired) 3 1 -3.299 0.244 0.0270 0.0313 0.0369 0.0436 0.0505

EvPA 

(Not Impaired) 158 7 -4.111 0.575 0.0078 0.0111 0.0164 0.0242 0.0343

West 

(Not Impaired) 220 14 -3.526 0.618 0.0133 0.0194 0.0294 0.0447 0.0650

East 

(Not Impaired) 293 23 -4.011 1.057 0.0047 0.0089 0.0181 0.0370 0.0703

3 Region 

(Not Impaired) 516 37 -3.800 0.925 0.0068 0.0120 0.0224 0.0418 0.0732

TP 

(percentiles 

are in 

mg/L) 

All 

(Not Impaired) 649 43 -3.868 0.875 0.0068 0.0116 0.0209 0.0377 0.0642
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-11 

Table 4-2. Distributional Statistics Computed from the Natural Logarithm Transformed Data 

Using All Data 

Observations Distribution Statistics 

Estimated from 

LN 

Transformed 

Data 

Percentiles Computed from Mean and 

Standard Deviation Back-Transformed to 

Regular Units 

Parameter Region 

Number 

of 

WBIDYears 

Number 

of 

WBIDs 

Mean 

LN 

Std Dev 

LN 

10th

%tile LN

25th

%tile LN

50th

%tile LN 

75th

%tile LN

90th

%tile LN

EAA 19 3 2.015 0.260 5.381 6.299 7.504 8.940 10.465 

EvPA 26 3 1.110 0.575 1.453 2.060 3.036 4.474 6.343 

West 94 11 1.158 0.530 1.614 2.227 3.183 4.550 6.275 

East 136 15 1.198 0.805 1.182 1.926 3.314 5.703 9.295 

ALL 262 30 1.226 0.711 1.370 2.109 3.407 5.504 8.475 

Chlorophyll 

(percentiles 

are in μg/L) 

Rt. 41 26 2 0.976 0.383 1.625 2.050 2.654 3.437 4.336 

 

EAA 250 14 0.976 0.334 1.729 2.118 2.654 3.325 4.073 

EvPA 344 22 0.555 0.340 1.127 1.385 1.742 2.190 2.692 

West 295 19 0.048 0.401 0.627 0.800 1.049 1.376 1.755 

East 558 49 0.243 0.381 0.782 0.986 1.274 1.648 2.077 

ALL 1355 89 0.406 0.487 0.803 1.080 1.500 2.084 2.801 

TN 

(percentiles 

are in 

mg/L) 

Rt. 41 86 5 0.088 0.296 0.748 0.895 1.093 1.334 1.596 

 

EAA 270 16 -2.660 0.572 0.0336 0.0476 0.0699 0.1028 0.1454

EvPA 365 23 -3.762 0.721 0.0092 0.0143 0.0232 0.0378 0.0586

West 313 20 -3.276 0.738 0.0147 0.0230 0.0378 0.0621 0.0972

East 649 51 -3.579 1.195 0.0060 0.0125 0.0279 0.0625 0.1290

ALL 1491 94 -3.415 1.008 0.0090 0.0167 0.0329 0.0648 0.1195

3 Region 1190 81 -3.328 1.050 0.0093 0.0177 0.0359 0.0728 0.1378

TP 

(percentiles 

are in 

mg/L) 

Rt. 41 110 5 -4.301 0.420 0.0079 0.0102 0.0136 0.0180 0.0232
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-12 

Table 4-3. Quantile Regression Results for Matched Annual Geometric Mean Chlorophyll and 

Nutrient Data 

Quantile Regression 

Parameter Region 

Spearman’s 

rho Intercept Slope 

Candidate 

Endpoint 

(mg/L) 

Overall 0.445 0.848 0.588 5.89 

EAA -0.058 

EvPA -0.130 

West 0.531 0.967 1.025 2.12 

TN 

East 0.394 0.851 0.643 5.01 

 

Overall 0.733 1.738 0.589 0.181 

EAA -0.430 

EvPA 0.816 2.595 1.039 0.057 

West 0.612 1.455 0.484 0.481 

TP 

East 0.739 1.807 0.621 0.153 

Candidate endpoint was computed assuming a chlorophyll concentration of 20 μg/L. 

Table 4-4. Results of “Not to Exceed” Criteria Computations 

Parameter 

Proposed 

Annual 

Geometric 

Mean 

Criteria 

Total 

Number 

of 

Observations 

Analyzed 

Total 

Number 

of SiteYears 

with 

Data 

Number of 

Site-Years 

with Data 

That Met 

Annual 

Geometric 

Mean Criteria

Number of 

Sites with 

Data That 

Met Annual 

Geometric 

Mean Criteria

Standard 

Deviation 

of 

Transfor

med Data 

Not to 

Exceed 

Value 

Chlorophyll 4.0 μg/L 5,524 596 412 58 0.8291 15.6 μg/L 

TN 1.6 mg/L 42,607 2,965 1,768 292 0.3040 2.6 mg/L 

TP 0.042 mg/L 67,000 3,781 2,516 344 0.6376 0.120 mg/L

Figure 4-1a. Location of EAA 

and EvPA. 

 Figure 4-1b. STATSGO soil orders for southern 

Florida. 

EA

EPA 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-13 

Figure 4-2. Site locations (n=513) with canal water quality data evaluated in this document. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-14 

Figure 4-3. Site locations (n=72) with chlorophyll data evaluated in this document. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-15 

Figure 4-4. Site locations (n=72) with chlorophyll data evaluated in this document together with 

Florida WBIDs that FDEP has identified as nutrient-impaired. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters

4-16 

Figure 4-5. Sites located within WBIDs not listed as nutrient-impaired. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-17 

a) 

TP

Chl_50

0.001 0.010 0.100

10.0

1.00.1

EAAEPAWestEastArea_Name

b)

TP

Chl_50

0.001 0.010 0.

10.0

1.00.1

logten(Chl_50) = 1.459 + 0.569

c) 

Regression Analysis: Chl_50 versus TP The regression equation is logten(Chl_50) = 1.459 + 0.5693 logten(TP) S = 0.218665 R-Sq = 49.2% R-Sq(adj) = 49.0% Source DF SS MS F P Regression 1 11.7784 11.7784 246.34 0.000 Error 254 12.1448 0.0478 Total 255 23.9232 

d)

Residual

Percent

-1.0 -0.5 0.0 0.5 1.0

99.9

99905010

1

0.1

Residual

0.00 

1.00.50.0

-0.5-1.0

Residual

Frequency

-0.6 -0.3 0.0 0.3 0.6 0.9

60453015

0

Residual

21 

1.00.50.0

-0.5-1.0

Normal Probability Plot of the Residuals R

Histogram of the Residuals Re

Residual Plots for logten(Ch

Figure 4-6. a) Bivariate plot of annual geometric mean chlorophyll as a function of TP by canal region, b) bivariate pregression and 90% prediction interval, c) regression analysis statistics, and d) residual plots from regression analysis
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-18 

a) 

TP

Chl_50

0.001 0.010 0.100

10.0

1.00.1

EAAEPAWestEastArea_Name

b)

TP

Chl_50

0.001 0.010 0

10.0

1.00.1

logten(Chl_50) = 1.452 + 0.570

c) 

Regression Analysis: Chl_50 versus TP The regression equation is logten(Chl_50) = 1.452 + 0.5707 logten(TP) S = 0.223073 R-Sq = 49.1% R-Sq(adj) = 48.9% Source DF SS MS F P Regression 1 10.9457 10.9457 219.96 0.000 Error 228 11.3456 0.0498 Total 229 22.2914 

d)

Residual

Percent

-1.0 -0.5 0.0 0.5 1.0

99.9

99905010

1

0.1

Residual

0.0

1.00.50.0

-0.5-1.0

Residual

Frequency

-0.6 -0.3 0.0 0.3 0.6 0.9

48362412

0

Residual

1 

1.00.50.0

-0.5-1.0

Normal Probability Plot of the Residuals 

Histogram of the Residuals Re

Residual Plots for logten

Figure 4-7. a) Bivariate plot of annual geometric mean chlorophyll as a function of TP by canal region excluding the 

fitted linear regression and 90% prediction interval, c) regression analysis statistics, and d) residual plots from regress
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-19 

a) 

TP

Chl_50

0.001 0.010 0.100

10.0

1.00.1

20

0.001

S 0.110211

R-Sq 9.7%R-Sq(adj) 4.4%

Regression

90% CI

90% PI

EAA TP

logten(Chl_50) = 0.6135 - 0.2755 logten(TP)

b)

TP

Chl_50

0.001 0.010 0.

10.0

1.00.1

0.052

EPA TP

logten(Chl_50) = 2.312 + 0.9844 

c) 

TP

Chl_50

0.001 0.010 0.100

10.0

1.00.1

20

0.25

S 0.173105R-Sq 44.1%R-Sq(adj) 43.4%

Regression

90% CI

90% PI

West Coast TP

logten(Chl_50) = 1.279 + 0.4892 logten(TP)

d)

TP

Chl_50

0.001 0.010 0.

10.0

1.00.1

0.069

East Coast TP

logten(Chl_50) = 1.692 + 0.6913 

Figure 4-8. Bivariate plot of annual geometric mean chlorophyll as a function of TP with fitted linear regression and 

interval for a) EAA (not statistically significant), b) EvPA, c) West Coast, and d) East Coast. (Residual plots for the and East Coast were visually examined and found to have constant variance, independence, and normal distribution
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-20 

Results for: EvPA Regression Analysis: Chl_50 versus TP The regression equation is logten(Chl_50) = 2.312 + 0.9844 logten(TP) S = 0.129383 R-Sq = 74.2% R-Sq(adj) = 73.2% Source DF SS MS F P Regression 1 1.15733 1.15733 69.14 0.000 Error 24 0.40176 0.01674 Total 25 1.55909 

Residual

Percent

-0.30 -0.15 0.00 0.15 0.30

99905010

1

Residual

0.00 0.

0.20.0

-0.2

Residual

Frequency

-0.2 -0.1 0.0 0.1 0.2 0.3

6.04.53.01.50.0

Residual

642 

0.20.0

-0.2

Normal Probability Plot of the Residuals Residua

Histogram of the Residuals Residuals 

EPA TP-Residual Plots for logten(C

Figure 4-9. EvPA regression analysis statistics for annual geometric mean chlorophyll as a function of TP and 

Results for: West Coast Regression Analysis: Chl_50 versus TP The regression equation is logten(Chl_50) = 1.279 + 0.4892 logten(TP) S = 0.173105 R-Sq = 44.1% R-Sq(adj) = 43.4% Source DF SS MS F P Regression 1 1.91713 1.91713 63.98 0.000 Error 81 2.42718 0.02997 Total 82 4.34431 

Residual

Percent

-0.50 -0.25 0.00 0.25 0.50

99.9

99905010

1

0.1

Residual

0.2 

0.500.250.00

-0.25-0.50

Residual

Frequency

-0.4 -0.2 0.0 0.2 0.4 0.6

201510

50

Residual

1 2

10 

0.500.250.00

-0.25-0.50

Normal Probability Plot of the Residuals Residua

Histogram of the Residuals Residuals 

West Coast TP-Residual Plots for logte

Figure 4-10. West Coast regression analysis statistics for annual geometric mean chlorophyll as a function of T

plots. 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-21 

Results for: East Coast Regression Analysis: Chl_50 versus TP The regression equation is logten(Chl_50) = 1.692 + 0.6913 logten(TP) S = 0.250179 R-Sq = 48.7% R-Sq(adj) = 48.3% Source DF SS MS F P Regression 1 7.4766 7.47656 119.45 0.000 Error 126 7.8863 0.06259 Total 127 15.3628 

Residual

Percent

-1.0 -0.5 0.0 0.5 1.0

99.9

99905010

1

0.1

Residual

0.00 0

1.00.50.0

-0.5-1.0

Residual

Frequency

-0.9 -0.6 -0.3 0.0 0.3 0.6 0.9

302010

0

Residual

30 20 11 0

1.00.50.0

-0.5-1.0

Normal Probability Plot of the Residuals Residua

Histogram of the Residuals Residuals 

East Coast TP-Residual Plots for logte

Figure 4-11. East Coast regression analysis statistics for annual geometric mean chlorophyll as a function of TP 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-22 

a) 

TN

Chl_50

0.1 1.0 10.0

10.0

1.00.1

EAAEPAWestEastArea_Name

b) 

TN

Chl_50

0.1 1.0 

10.0

1.00.1

3.6

logten(Chl_50) = 0.5238 + 0.5738 

c) 

Regression Analysis: Chl_50 versus TN The regression equation is logten(Chl_50) = 0.5238 + 0.5738 logten(TN) S = 0.279109 R-Sq = 16.8% R-Sq(adj) = 16.5% Source DF SS MS F P Regression 1 3.9005 3.90051 50.07 0.000 Error 248 19.3197 0.07790 Total 249 23.2202 

d) 

Residual

Percent

-1.0 -0.5 0.0 0.5 1.0

99.9

99905010

1

0.1

Residual

1.00.50.0

-0.5-1.0

Residual

Frequency

-0.9 -0.6 -0.3 0.30.0 0.6

40302010

0

Residual

21 0

1.00.50.0

-0.5-1.0

Normal Probability Plot of the Residuals R

Histogram of the Residuals Res

Residual Plots for logten(Ch

Figure 4-12. a) Bivariate plot of annual geometric mean chlorophyll as a function of TN by canal region, b) bivariaregression and 90% prediction interval, c) regression analysis statistics, and d) residual plots from regression analy
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-23 

a) 

TN

Chl_50

0.1 1.0 10.0

10.0

1.00.1

20

S 0.115747R-Sq 0.4%R-Sq(adj) 0.0%

Regression

90% CI

90% PI

EAA TN

logten(Chl_50) = 0.9085 - 0.0723 logten(TN)

b)

TN

Chl_50

0.1 1.0 

10.0

1.00.1

EPA TN

logten(Chl_50) = 0.4841 + 0.4881 

c) 

TN

Chl_50

0.1 1.0 10.0

10.0

1.00.1

20

4.0

S 0.206710R-Sq 22.3%R-Sq(adj) 21.3%

Regression

90% CI

90% PI

West Coast TN

logten(Chl_50) = 0.5965 + 0.5881 logten(TN)

d)

TN

Chl_50

0.1 1.0 

10.0

1.00.1

2.35

East Coast TN

logten(Chl_50) = 0.4645 + 0.7553 

Figure 4-13. Bivariate plot of annual geometric mean chlorophyll as a function of TN with fitted linear regression ainterval for a) EAA (not statistically significant), b) EvPA (not statistically significant), c) West Coast, and d) East C

plots for the West and East Coasts were visually examined and found to have independence and normal distributionassociated with the West Coast residuals did show a tendency to increase as a function of TN.) 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-24 

Results for: West Coast Regression Analysis: Chl_50 versus TN The regression equation is logten(Chl_50) = 0.5965 + 0.5881 logten(TN) S = 0.206710 R-Sq = 22.3% R-Sq(adj) = 21.3% Source DF SS MS F P Regression 1 0.94490 0.944901 22.11 0.000 Error 77 3.29015 0.042729 Total 78 4.23505 

Residual

Percent

-0.8 -0.4 0.0 0.4 0.8

99.9

99905010

1

0.1

Residual

0.2 

0.80.40.0

-0.4-0.8

Residual

Frequency

-0.6 -0.4 -0.2 0.60.40.20.0

241812

60

Residual

1 

2

10

0.80.40.0

-0.4-0.8

Normal Probability Plot of the Residuals Residua

Histogram of the Residuals Residuals 

West Coast TN-Residual Plots for logt

Figure 4-14. West Coast regression analysis statistics for annual geometric mean chlorophyll as a function of T

Results for: East Coast Regression Analysis: Chl_50 versus TN The regression equation is logten(Chl_50) = 0.4645 + 0.7553 logten(TN) S = 0.327084 R-Sq = 10.2% R-Sq(adj) = 9.4% Source DF SS MS F P Regression 1 1.5011 1.50111 14.03 0.000 Error 124 13.2660 0.10698 Total 125 14.7672 

Residual

Percent

-1.0 -0.5 0.0 0.5 1.0

99.9

99905010

1

0.1

Residual

0.2 

1.00.50.0

-0.5-1.0

Residual

Frequency

-0.9 -0.6 -0.3 0.90.60.30.0

241812

60

Residual

30 20 11 0

1.00.50.0

-0.5-1.0

Normal Probability Plot of the Residuals Residua

Histogram of the Residuals Residuals 

East Coast TN-Residual Plots for logte

Figure 4-15. East Coast regression analysis statistics for annual geometric mean chlorophyll as a function of TN 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-25 

Figure 4-16. Bivariate plot of annual geometric mean chlorophyll as a function of TN and TP across all 

regions together with Spearman’s rho correlation coefficient, LOESS smoother (black line, f=0.7), and 

quantile regression results (blue line and blue text). 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-26 

Figure 4-17. Bivariate plot of annual geometric mean chlorophyll as a function of TP for the 

EvPA together with Spearman’s rho correlation coefficient, LOESS smoother (black line, f=0.7), 

and quantile regression results (blue line and blue text). 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-27 

Figure 4-18. Bivariate plot of annual geometric mean chlorophyll as a function of TN and TP for the West 

Coast together with Spearman’s rho correlation coefficient, LOESS smoother (black line, f=0.7), and quantile 

regression results (blue line and blue text). 
TSD for U.S. EPA’s Proposed Numeric Nutrient Criteria for FL Inland Surface Fresh Waters 

4-28 

Figure 4-19. Bivariate plot of annual geometric mean chlorophyll as a function of TN and TP for the East 

Coast together with Spearman’s rho correlation coefficient, LOESS smoother (black line, f=0.7), and 

quantile regression results (blue line and blue text).