Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Appendix B: Statistical (StressorResponse) Analysis 

November 30, 2012
 
Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Contents 

Appendix B. Empirical Analyses.......................................................................................... B-1
 

B.1 Data ...............................................................................................................................B-1
 

B.2 Derivation of Chl-a Criteria to Meet Water Clarity Targets .........................................B-1
 

B.2.1 Data Screening..................................................................................................B-1
 

B.2.2 Statistical Model ...............................................................................................B-4
 

B.3	 Derivation of Numeric Nutrient Criteria to Meet Algal Bloom Frequency
 

Endpoint ......................................................................................................................B-7
 

B.4 Derivation of TN and TP Criteria to Achieve Chl-a Criteria........................................B-8
 

B.4.1 Data Screening..................................................................................................B-8
 

B.4.2 Statistical Model ...............................................................................................B-9
 

B.5 Computing Upper and Lower Bounds of Available Data ...........................................B-13
 

B.6 Nutrient Targets for Downstream Protective Value Calculation ................................B-13
 

B.7 References ...................................................................................................................B-15
 

B-i 
 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figures 

Figure B-1. Example of combining samples from neighboring stations in Indian River 

Lagoon. Filled gray circles show two sampling locations for which data were 

combined. Open circles show other sampling locations. .........................................B-2
 

Figure B-2. Example of estimating depth at a sampling location from sounding data. Left 

panel: open grey circles, sounding locations; contours indicate fitted 

bathymetric surface; filled red circle, sampling location. Right panel: depth 

predicted by nonparametric thin plate spline plotted against observed 

sounding depth. Filled red circle indicates the depth estimated for the 

sampled location. ......................................................................................................B-3
 

Figure B-3. Example of comparing measured Secchi Depth with estimated depth at the 

sampling location (shown as the horizontal line). Estimated depth at 

sampling location computed from nonparametric model prediction and mean 

tidal range, as described in the text. .........................................................................B-3
 

Figure B-4. Example of estimated relationship between chl-a and Kd. Red horizontal line 

shows Kd corresponding with seagrass depth target for the segment. Red 

vertical arrow shows annual geometric mean chl-a concentration predicted to 

be associated with Kd target. Green line segment shows the 5th to 95th 

percentile range of observed annual geometric mean values for chl-a. Open 

circles show measured annual average values of Kd (adjusted for the effect of 

turbidity and color) and chl-a. ..................................................................................B-5
 

Figure B-5. Estimates of the slopes between Kd and chl-a in different estuarine segments. 

Two panels from top to bottom show Northwest and Eastern estuaries. 

Vertical lines show estimates of the 90% confidence intervals on each slope, 

open circles show mean estimate of slope................................................................B-6
 

Figure B-6. Example of model of the relationship between annual geometric mean chl-a 

concentration and bloom frequency. Open circles: observed annual frequency 

of chl-a concentrations exceeding 20 μg/L versus annual geometric mean chla concentration for the same year; solid black line: modeled relationship 

between mean chl-a concentration and bloom frequency; horizontal red line: 

targeted bloom frequency of 10%; vertical red arrow: annual geometry mean 

chl-a concentration associated with targeted bloom frequency; green line 

segment: 5th to 95th percentile range of observed annual geometric mean chl-a 

concentrations. ..........................................................................................................B-8
 

Figure B-7. Example of estimated stressor-response relationship between TN and chl-a. 

Blue line: segment mean relationship. Grey lines: relationships estimated 

within different stations. Red horizontal line shows the chl-a criterion 

associated with the water clarity endpoint for each segment. Green segment 

shows the range of observed annual geometric mean TN values. Open circles 

show observed values of annual geometric mean chl-a and TN. .............................B-9
 

B-ii 
 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-8. Estimates of the slope of the linear relationship between annual geometric 

mean TP and chl-a among different segments. Two panels from top to bottom 

show Northwest and Eastern estuaries. Open circles: mean estimate of slope; 

vertical lines: estimated 90% confidence intervals. ...............................................B-11
 

Figure B-9. Estimates of the slope of the linear relationship between annual geometric 

mean TN and chl-a among different segments. Two panels from top to 

bottom show Northwest and Eastern estuaries. Open circles: mean estimate 

of slope; vertical lines: estimated 90% confidence intervals. ................................B-12
 

Figure B-10. Map of station locations in St. Lucie estuary. Stations labeled in order of 

maximum observed salinity (highest salinity is numbered 1). Station circled 

in red is located in the estuary pass. .......................................................................B-14
 

Figure B-11. Observed salinity and nutrient concentrations at estuary pass of St. Lucie. ........B-14
 

Figure B-12. Illustration of calculation of DPVs using a mixing/dilution relationship for 

Indian River Lagoon. Black diamond shows the estimated conditions in 

seawater. Green circles show proposed TP criteria for different segments. 

Green triangles show DPVs computed for each segment. Open circles show 

long-term average salinity and TP and different stations in the estuary. ...............B-15
 

B-iii 
 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Appendix B. Empirical Analyses 

Analyses of data collected from the field inform criteria derivation by identifying nutrient 

concentrations that are protective of biological endpoints. 

B.1 Data 

Water quality data were retrieved from the Impaired Waters Rule (IWR) Run 40 database (FDEP 

2010) and screened based on whether they were located within pre-defined estuary segments, 

yielding a total of 1,126,560 water quality samples collected at 26,495 sites. Different screens 

(described below) were applied to these data depending on the requirements of different 

analyses, which reduced the size of the database. 

Measurements of TN, TP, chlorophyll a (chl-a), nitrate+nitrite (NO3+NO2), and total Kjeldahl 

nitrogen were all log-transformed to reduce the skewness of the distributions. 

Measurements of Secchi depth were converted to an estimate of the light attenuation coefficient 

(Kd) using the following relationship (Holmes 1970): 

1.44 

kd = Secchi Depth

B.2 Derivation of Chl-a Criteria to Meet Water Clarity Targets 

B.2.1 Data Screening 

To derive chl-a criteria to meet water clarity targets, data retrieved from IWR Run 40 were 

further screened to include only samples with concurrent measurements of Secchi depth and chla. Chl-a measurements that were recorded as being below any detection limit that was 5 μg/L or 

greater were also excluded. 

In each estuary, all sampling locations were projected into the Universal Transverse Mercator 

coordinate system (UTM Zone 17). Then, data collected at different times from sites within 

350 m of one another were combined and analyzed as if collected from the same location (Figure 

B-1). The allowed separation distance of 350 m was selected by examining neighboring sites in 

which the data record at one site began after the data record at another site ended. In these pairs 

of sites, the slight change in sampling location may have resulted from a change in the datum on 

which the geographic location was based or a change in sampling protocol. Furthermore, EPA 

determined that natural characteristics of sites (e.g., salinity) within 350 m of one another in an 

estuary were assumed to be similar, and hence, relationships between Secchi depth and chl-a at 

these closely located sites were modeled as if the observations were collected at the same site. 

B-1 
 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-1. Example of combining samples from neighboring stations in Indian River Lagoon. Filled gray circles 

show two sampling locations for which data were combined. Open circles show other sampling locations. 

Depth at sampled locations was estimated using digital depth sounding data acquired from 

NOAA’s National Geophysical Data Center 

(http://www.ngdc.noaa.gov/mgg/bathymetry/relief.html). Depth sounding data in a 1 km × 1 km 

box surrounding each sampled location was selected, and a smooth bathymetric surface was 

estimated using a nonparametric thin plate spline with a maximum of 30 degrees of freedom 

(Wood and Augustin 2002). The maximum number of degrees of freedom for the nonparametric 

fit sets the upper limit on the degree to which the fitted surface “wiggles,” and exploratory 

analysis indicated that 30 degrees of freedom was sufficient to represent variations in depth, 

without overfitting the data (Wood and Augustin 2002). Once the nonparametric surface was fit, 

the depth at the sampled location was estimated from the mean depth predicted by the 

nonparametric model. To account for errors in the predicted depth, an estimate of the upper 90th 

prediction interval for the fitted value (i.e., mean plus 1.64 times the standard deviation of the 

residual deviation) was used as the estimated depth. 

Data collected from sample locations that were greater than 150 m from a recorded depth 

sounding were excluded from the dataset. Because depth can vary greatly over short distances 

and because depth soundings were relatively close to one another (Figure B-2), maintaining a 

relatively low cut-off distance of 150 m between sample location and the sounding location was 

reasonable. 

B-2 
 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-2. Example of estimating depth at a sampling location from sounding data. Left panel: open grey circles, 

sounding locations; contours indicate fitted bathymetric surface; filled red circle, sampling location. Right panel: 

depth predicted by nonparametric thin plate spline plotted against observed sounding depth. Filled red circle 

indicates the depth estimated for the sampled location. 

Each Secchi depth measurement was compared to the estimated depth at the sample location 

(Figure B-3). Since the depth sounding is referenced to mean lower-low water, the mean tidal 

range estimated for that location was added to the sounding depth before comparing with Secchi 

depth. Tidal range data for different estuaries were obtained from NOAA’s Center for 

Operational Oceanographic Products and Services (http://tidesandcurrents.noaa.gov) and the 

maximum tidal range observed in each segment was used. In some segments, tidal range 

information was not available, so tidal range data from nearby stations were used. 

Figure B-3. Example of comparing measured Secchi Depth with estimated depth at the sampling location (shown 

as the horizontal line). Estimated depth at sampling location computed from nonparametric model prediction and 

mean tidal range, as described in the text. 

B-3 
 

 

 

 

 

 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Secchi depth measurements that were greater than the estimated depth of the estuary at the 

sample location were treated as censored measurements, known only to be greater than the 

estimated depth. Light attenuation coefficients that were estimated from these censored Secchi 

depth measurements were also treated as censored data. Sampling locations at which the majority 

of Secchi depth measurements (i.e., greater than 50%) were censored were excluded from the 

calculation. 

Light attenuation coefficients, chl-a, turbidity, and color at each sampling location were 

summarized as annual geometric means. Sampled values recorded as being less than the 

detection limit were included in the computation of annual geometric means as half of the 

detection limit. At least four different samples of each parameter (Secchi depth, chl-a, turbidity, 

and color), including at least one sample collected in the summer (May–September) and one 

sample collected in the winter (October–April), were required to compute annual geometric 

means for that parameter. These requirements ensured that annual geometric mean values were 

estimated with sufficient precision for subsequent analysis. Relationships were estimated for an 

estuary segment only if at least two distinct stations with data satisfying all previous screens 

were available in the segment, to ensure the data sufficiently represent spatial variations within 

each segment. 

B.2.2 Statistical Model 

Relationships between annual geometric mean water clarity, as quantified by light attenuation 

coefficient, and chl-a were estimated using mixed effects models that represented the 

hierarchical structure of the data within sampling locations and within segments (Gelman and 

Hill 2007). In addition to chl-a, the model also quantified the effects of turbidity and color on 

observed water clarity. A model of the following form was fit: 

log(k

ഥ

d,ijk) = b0,jk + b1,jk log൫chlijk തതതതതതതതതത൯ + b3,jk log൫colorijk തതതതതതതത൯ + b2,jk log൫turbijk തതതതതതതതതതത൯ + εi

where k

ഥd,ijk, ത

chl തതതത

ijk, turbijk തതതതതതതതതത, and color തതതതതതതതijk are the annual geometric mean light attenuation 

coefficient, chl-a, turbidity, and color for year i at station k in segment j, b0..3,jk are regression 

coefficients for station k and segment j, and εi is a normally distributed error. The hierarchical 

structure of the data was represented by the regression coefficients, which were each computed 

as the sum of an overall mean value of the coefficient, a segment-specific value, and a stationspecific value: 

bm,jk = bm + δm,j + ∆m,k

where bm is the overall mean value for coefficient m, δm,j is the random effect for segment j, and 

∆m,k is the random effect for station k. Random effects associated with segments and stations 

were assumed to be normally distributed with a mean value of zero. 

In the northwest region of Florida (Panhandle and Big Bend), turbidity and color measurements 

were not collected in enough samples to include in the model, so in this region only the effect of 

chl-a on Kd was modeled: 

log(k

ഥ തതതതതതതത൯ d,ijk) = b0,jk + b1,jk log൫chlijk

B-4 
 

 

 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Models were fit using the package lme4 in R (R Development Core Team 2009). 

Chl-a values associated with depth targets (Dtarget) for each segment were computed by first 

converting the Dtarget for each segment to a Kd target using the following relationship, which 

computes the light attenuation coefficient that is necessary to achieve 20% light attenuation at 

the depth target: 

−ln(0.2) kd(target) = Dtarget

Then, the segment-specific relationship between Kd and chl-a was extracted from the fitted 

model, assuming that turbidity and color were constant at the mean value for the segment. This 

relationship was used to compute the chl-a criterion as the chl-a concentration that was 

associated with the targeted Kd value (Figure B-4). 

Figure B-4. Example of estimated relationship between chl-a and Kd. Red horizontal line shows Kd corresponding 

with seagrass depth target for the segment. Red vertical arrow shows annual geometric mean chl-a concentration 

predicted to be associated with Kd target. Green line segment shows the 5th to 95th percentile range of observed 

annual geometric mean values for chl-a. Open circles show measured annual average values of Kd (adjusted for the 

effect of turbidity and color) and chl-a. 

In some cases, the chl-a concentration associated with the Kd target was less than the minimum 

annual geometric mean chl-a concentration observed in the segment. In these cases, the chl-a 

criterion value was set at the lower bound of the observed range of data. Conversely, in some 

segments, the chl-a concentration associated with the Kd target was greater than the maximum 

geometric mean chl-a concentration observed in the segment. In these cases, the chl-a criterion 

was set at the upper bound of the observed range of data (see Section B.5). Relationships 

between Kd and chl-a varied among estuaries and estuary segments (Figure B-5). 

Relationships with negative and near-zero slopes generally occurred in estuarine segments near 

the inflows from large tributaries (e.g., Segment 1403, the Upper St. Lucie River); these 

relationships were likely affected by high concentrations of suspended solids and colored 

dissolved organic matter. Variations in the slope of the relationship between Kd and chl-a were 

B-5 
 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

also expected because the relationships can depend on existing conditions in the water body. For 

example, increasing chl-a concentration by 1 μg/L in waters that already have high 

concentrations of chl-a would have a smaller influence on water clarity than increasing chl-a by 

that same amount in waters that have low chl-a concentrations. The hierarchical structure of the 

model minimizes the potential effects of these issues by modeling changes within each sampling 

location, and thus, constraining the model to represent the relatively limited variations that occur 

specifically at each location. 

Figure B-5. Estimates of the slopes between Kd and chl-a in different estuarine segments. 

Two panels from top to bottom show Northwest and Eastern estuaries. Vertical lines show 

estimates of the 90% confidence intervals on each slope, open circles show mean estimate 

of slope. 

In segments in which the slope of the relationship between Kd and chl-a was very near zero (such 

that the confidence intervals included zero) or negative, water clarity was noted as being 

insensitive to changes in chl-a, and no chl-a criterion value associated with the water clarity 

endpoint was computed. Note, though, that additional data collection would improve the 

precision with which these relationships were estimated. 

B-6 
 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

B.3	 Derivation of Numeric Nutrient Criteria to Meet Algal Bloom Frequency 

Endpoint 

Relationships between annual geometric mean chl-a concentrations and the frequency of 

phytoplankton blooms were used to estimate annual geometric mean chl-a concentrations 

associated with meeting the phytoplankton bloom endpoint. First, annual geometric mean chl-a 

concentrations were estimated for each station. Station-year means were estimated from at least 

four distinct samples, with at least one sample collected in the winter and one sample collected in 

the summer. Stations were only retained if at least 3 years of data were available, and 

relationships were only estimated for estuary segments in which at least 2 stations satisfied all 

previous data screens. The additional screen of requiring 3 years of data for any single station 

(compared to only 1 year of data for Kd-chl-a models) was imposed for these models because 

estimated relationships were expected to vary more greatly among stations. More specifically, 

since these models only include a single explanatory variable, it was expected that variations in 

covariates at other stations would influence the results. In contrast, all of the primary explanatory 

factors were included in the Kd-chl-a models, lessening the potential variability of relationships 

among stations. Hence, a single pair of annual geometric mean Kd and chl-a values would inform 

model estimates. 

The final set of data was modeled using mixed models by fitting a model of the following form: 

pijk ln ቆ ቇ = d0,j + d1,jlog(chl തതതതijk) 1 − pijk

where pijk is the probability of chl-a concentration exceeding 20 μg/L in year i, segment j, and 

station k, and chl തതതതijk is the annual geometric mean chl-a concentration in year i, segment j, and 

station k. The regression coefficients include an effect that varies across segments as well as an 

overall mean effect as follows, 

dm,j = dm + δm,j + ∆m,k

Where m takes the value of 0 or 1, dm is the overall mean value for the regression coefficient and 

δm,j is a normally distributed random effect that differs for each segment, and ∆m,k is a normally 

distributed random effect that differs for each station. 

To derive criteria associated with the phytoplankton biomass endpoint, the annual geometric 

chl-a concentrations associated with pij = 0.1 (i.e., a bloom frequency of 10%) were computed 

(Figure B-6). 

As with other criteria derived using stressor-response relationships, in some cases the criterion 

associated with a bloom frequency of 10% was greater than the upper bound of the range of 

observed geometric mean chl-a concentrations. In these cases, to avoid extrapolating models 

beyond the range of data, criteria were set at the upper bound of the observed chl-a 

concentrations. 

B-7 
 

 

 

 

 

 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-6. Example of model of the relationship between annual geometric mean chl-a concentration and 

bloom frequency. Open circles: observed annual frequency of chl-a concentrations exceeding 20 μg/L versus 

annual geometric mean chl-a concentration for the same year; solid black line: modeled relationship between 

mean chl-a concentration and bloom frequency; horizontal red line: targeted bloom frequency of 10%; vertical red 

arrow: annual geometry mean chl-a concentration associated with targeted bloom frequency; green line segment: 

5th to 95th percentile range of observed annual geometric mean chl-a concentrations. 

B.4 Derivation of TN and TP Criteria to Achieve Chl-a Criteria 

Once chl-a criteria were derived to satisfy either the water clarity or the phytoplankton bloom 

endpoint, nutrient-chlorophyll relationships were required to calculate TN and TP criteria that 

were consistent with the chl-a criterion. 

B.4.1 Data Screening 

To estimate relationships between TN, TP, and chl-a, data retrieved from IWR Run 40 were 

further screened to include only samples with a measurement of chl-a, and at least one 

measurement of either TN, TP, or nitrate+nitrite. Station locations were then analyzed and 

stations that were within 350 m of one another were combined (see Section B.2.1). Samples with 

color measurement exceeding 50 PCU were excluded to minimize the effects of water color on 

estimated relationships. Annual geometric mean values for TN, TP, nitrate+nitrite, and chl-a 

were computed for each station-year combination in which at least 4 samples were available in 

that year and at least 1 sample was collected in the summer (May–September) and one sample 

was collected in the winter (October–April). Stations were only used in the analysis if at least 3 

annual means were available, and segments were only used in the analysis if at least 2 stations 

satisfying the screens described above were available. Here, as with the bloom frequency 

models, variations of the relationships among stations were expected to be greater than observed 

for the Kd-chl-a relationships, so more data were required for any single station to be included in 

the model. 

B-8 
 

 

 

 

 

 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

B.4.2 Statistical Model 

Mixed effects models were used to represent the hierarchical structure of the data within 

sampling locations and segments. More specifically, in each estuary segment, the model 

estimated the linear relationship between annual geometric mean chl-a and annual mean 

geometric TN or TP within stations and within segments. Annual geometric mean values of TN, 

TP, and chl-a were first calculated from sample data for each station. Then, a model of the 

following form was used to model the relationships between TP and chl-a: 

log൫chl a c0,jk + c1,jklog(TPതതതത

ijk) തതതതതതതijk൯ = 

where chl തതതതതതaതijk and TPതതതത

ijk are the geometric mean chl-a and TP concentrations for year i, in 

station k, in segment j. The regression coefficients were computed as the sum of an overall mean 

coefficient value, a segment-specific value, and a station-specific value as follows: 

cm,jk = cm + δm,j + ∆m,k

where m is equal to 0 or 1, δm,j is the random effect attributed to segment j, and ∆m,kis the 

random effect attributed to station k. The distributions of random effects at the station and 

segment level were assumed to be normal with a mean value of zero. An identical model was 

used to estimate the relationships between TN and chl-a, and in estuaries in which sufficient data 

were available, the same model form was used to estimate relationships between nitrate+nitrite 

and chl-a. An example stressor-response model for Segment 1504 between chl-a and TN is 

shown in Figure B-7. 

TN and TP criteria were determined as the point at which predicted annual geometric mean chl-a 

concentration was equivalent to the chl-a criterion. Nitrate+nitrite target values were also 

estimated where data were available, to help interpret relationships observed between TN and 

chl-a. 

Figure B-7. Example of estimated stressor-response relationship between TN and chl-a. Blue line: segment mean 

relationship. Grey lines: relationships estimated within different stations. Red horizontal line shows the chl-a 

criterion associated with the water clarity endpoint for each segment. Green segment shows the range of 

observed annual geometric mean TN values. Open circles show observed values of annual geometric mean chl-a 

and TN. 

B-9 
 

 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

As was observed when deriving chl-a criteria to meet water clarity endpoints, in some segments, 

the candidate TN or TP criterion estimated from the stressor-response relationship was outside 

the range of observed annual geometric mean TN or TP concentrations. When the candidate 

criterion was greater than the upper bound of the range of values observed for TN or TP, the 

criterion was set at the upper bound. Conversely, when the candidate criterion was less than the 

lower bound of the range of values observed for TN or TP, the criterion value was set to the 

lower bound. 

Estimates of the slope of the relationship between TP and chl-a were strongly positive for the 

vast majority of estuary segments (Figure B-8). The only segments in which near zero or 

negative slopes were observed were 1801 and 1802 (Lower St. Johns River and Trout River) and 

several segments in Charlotte Harbor. In contrast, a number of segments exhibited near zero 

(such that confidence intervals included zero) or negative slopes of the relationship between TN 

and chl-a (Figure B-9). Negative correlations between TN and chl-a likely reflect the 

complexities of the components of TN that are available for biological uptake. In certain 

locations, it seems likely that the majority of measured TN is composed of biological unavailable 

forms. Further analysis of relationships between nitrate+nitrite and chl-a support this 

interpretation (see estuary-specific description of analyses for Tampa Bay). 

The observed relationships between nutrients and chl-a conform with expectations, and provide a 

useful model for deriving nutrient criteria. 

B-10 
Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-8. Estimates of the slope of the linear relationship between annual geometric mean TP and chl-a 

among different segments. Two panels from top to bottom show Northwest and Eastern estuaries. Open 

circles: mean estimate of slope; vertical lines: estimated 90% confidence intervals. 

B-11 
Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-9. Estimates of the slope of the linear relationship between annual geometric mean TN and chl-a 

among different segments. Two panels from top to bottom show Northwest and Eastern estuaries. Open 

circles: mean estimate of slope; vertical lines: estimated 90% confidence intervals. 

B-12 
 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

B.5 Computing Upper and Lower Bounds of Available Data 

As described in the previous sections, when criteria were derived using stressor-response 

relationships outside the range of observed geometric mean concentrations, final criteria are 

based on either the upper or lower bound of the available data. Upper and lower bounds were 

quantified as the 10th and 90th percentiles of the available annual geometric means for a given 

analysis. In a few segments, too few distinct geometric mean concentrations for a particular 

parameter (i.e., < 10) were available to reliably estimate a 10th or a 90th percentile. In these 

segments, upper and/or lower bounds were not computed. 

B.6 Nutrient Targets for Downstream Protective Value Calculation 

A mixing/dilution model was used to calculate the freshwater downstream protective values 

(DPVs) necessary to achieve nutrient criterion values specified within different estuary 

segments. The mixing/dilution model is based on the premise that changes in nutrient 

concentrations starting from freshwater tributaries and through an estuary can be approximated 

as a conservative mixing process. Using this model, one can estimate the contributions of 

seawater and freshwater to any water quality sample, based on the observed salinity in the 

sample, freshwater salinity, and the salinity of seawater. With estimates of nutrient 

concentrations in seawater, one can also predict freshwater nutrient concentrations that are 

necessary to achieve targeted nutrient concentrations at estuarine salinities. 

One key component of this analysis is an estimate of TN and TP concentrations in seawater. To 

derive seawater concentrations of TN and TP, EPA examined available salinity data and station 

locations to identify stations that were most likely to reflect seawater conditions (Figure B-10). 

Based on this initial examination, EPA determined that data collected at the estuary pass of the 

St. Lucie estuary best characterized seawater salinity and nutrient concentrations. 

In the selected estuary pass station of St. Lucie, nutrient concentrations and salinity varied 

substantially over time, as expected, given different freshwater flow rates. A negative 

relationship between salinity and nutrient concentrations was observed, as one would expect, 

given that increased salinities indicate increased dilution by seawater (Figure B-11). At the 

highest salinities (> 35 PSU) TP concentrations converged to a single value (TP = 0.03 mg/L), 

which provided an estimate of seawater concentrations of TP. However, TN still exhibited 

substantial variability at these high salinities. This variability can likely be attributed to 

contributions of high salinity/high TN from side channels (see Figure B-10). Because of the 

potential effects of non-seawater contributions to TN, the minimum TN concentration observed 

between 34 and 36 PSU was used as an estimate of seawater TN (TN = 0.14 mg/L). 

B-13 
Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

Figure B-10. Map of station locations in St. Lucie estuary. Stations labeled in order of maximum observed salinity 

(highest salinity is numbered 1). Station circled in red is located in the estuary pass. 

Figure B-11. Observed salinity and nutrient concentrations at estuary pass of St. Lucie. 

B-14 
 

Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

For this analysis, the salinity of freshwater is assumed to be 2.7 PSU and that of seawater is 

assumed to be 36 PSU. 

Given an estimate of seawater salinity and nutrient concentrations and a targeted nutrient 

criterion value and long-term salinity value for a given segment, one can compute a freshwater 

DPV by simply fitting a straight line through the seawater value and the criterion value, and 

calculating a nutrient concentration at 2.7 PSU. This calculation can be written as follows for 

TN: 

TNcrit − TNsea DPV = (2.7 − Ssea) + TNsea Sseg − Ssea

where TNcrit is the proposed TN criterion for a particular segment, TNsea is the seawater 

concentration of TN, Sseg is the long-term salinity in the segment of interest, and Sseais the 

salinity of seawater (taken to be 36 PSU). The approach is illustrated graphically in Figure B-12. 

Figure B-12. Illustration of calculation of DPVs using a mixing/dilution relationship for Indian River Lagoon. Black 

diamond shows the estimated conditions in seawater. Green circles show proposed TP criteria for different 

segments. Green triangles show DPVs computed for each segment. Open circles show long-term average salinity 

and TP and different stations in the estuary. 

B.7 References 

FDEP. 2010. IWR Stations Run 40. http://www.dep.state.fl.us/water/basin411/download.htm. 

Last updated April 15, 2010. 

Gelman, A., and J. Hill. 2009. Data Analysis Using Regression and Multilevel/Hierarchical 

Models. Cambridge University Press, New York. 

Holmes, R.W. 1970. The Secchi disk in turbid coastal zones. Limnology and Oceanography. 

15:688-694. 

B-15 
Technical Support Document for U.S. EPA’s Proposed Rule for Numeric Nutrient Criteria, 

Volume 1 Estuaries 

R Development Core Team. 2009. R: A language and environment for statistical computing. 

R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0. 

http://www.R-project.org. 

Wood, S.N., and N.H. Augustin. 2002. GAMs with integrated model selection using penalized 

regression splines and applications to environmental modelling. Ecological Modelling 

157:157-177. 

B-16