1 of 1

Review of Comments on EPA’s Statistical Approach Used to Develop the Proposed MACT Floor Standards for Portland Cement Manufacturing

August 6, 20102 of 2
Revision of Data Analysis and Variability for Portland Cement NESHAP Prepared by Breda Munoz

Executive Summary to Appendix A of Portland Cement Association public Comments (comment 2845) 1. Page ES-1 Portland Cement Association (PCA) claims th at EPA’s intentions are to estimate the 99 percentile. However, I have reviewed docum ents that estimate the 99 confidence upper prediction limit (UPL), not the 99 th percentile.

Here are basic definitions: The 99th percentile is the value (or score) below wh ich 99 percent of the observations may be found

The confidence level describes the uncertainty associat ed with a sampling method used to collect the data. A 99% confidence level in an interval (e.g. prediction interval) means that we would expect 99% of the times the in terval estimates (lower and upper limits) to include the population parameter

The prediction interval is an estimate of an interval in which future observations will fall, with a certain probability (i.e . confidence level), given what has already been observed. The upper prediction limit (UPL) is the upper limit of the predicti on interval. The 99% confidence UPL is the upper limit of the pr ediction interval calculated with a 99% confidence (Neter, 1990)

PCA defines a percentile as a variable, a nd it is a population or sample value, not a random variable. The percentile of a populat ion is a parameter, has no variability associated with it, it is a fixed value. An estimate of the percentile has a variability associated with it, then a confidence in terval can be calculated for the population percentile using the sample percentile.

PCA’s statement, “the 99 th percentile should account fo r all but one percent of the possible values, with a specified level of confidence” is not correct. The 99 th percentile is the cut-off value below which 99% of the data are found, and this occurred with certainty.

In the second paragraph, PCA has several se ntences that are not correct. PCA seems to confuse the calculation of the percentiles in a sample or population, with confidence intervals for the popul ation percentiles.

PCA seems to conclude that the UPL was pr oposed to estimate the 99 percentile of the population. To this end, PCA proposes the uppe r tolerance limit (U TL). In the case of smaller datasets, the UPL is better than the UTL, (see discussion in Section 1).

2. Page ES-2.3 of 3
UTL is not for prediction purposes, so it won’t say what the data will be in the future. The prediction properties are a characteristic of the UPL. Both UPL and UTL are based on the present data, but UPL take into account the future observations variability. This is why EPA is using the UPL statistic rather than the UTL (along with the suitability of applying UPL to datasets like those in the NESHAP).

Reliability refers to consistency, agreement of values. Maybe PCA wa nted to assess the precision (variability) of both UPL and UTL? The UPL and UTL have different purposes (see Section 1 comments). An estimator is cons istent if it produces the same value under several repetitions. In this case, containing or not the 99 percentile can not be used to evaluate the reliability of the two estimato rs, UPL and UTL. By definition, the 99% UTL will probably contain the 99 percentile.

If EPA wants to estimate the 99 percentile, th en there are methods appropriate to do so. If the idea is to obtain an upper bound that is gr eater than the 99 per centile, then the UTL works. The UTL will generally be a larg er number than the 99 percentile.

3. Page ES-3

The assumption of a repres entative sample can be pr oved by comparing both the population and the sample in relevant characteristics (variables). For example if some type of material is used in the population, then the same relative quantity of this material should be present in the sample.

The sample will never have the same level in all the characteristics as the population. So it is important to define the most important characteristics or sour ces of variability, and demonstrate that both the sample and the population present the sa me distribution in these characteristics.

The statement that “UPL will provide a proba bility and not a propor tion” is not correct. The UPL will provide an upper bound for future observations with a given confidence limit. 4. Page ES-4 The terms in the variance used to generate the UPL were given erroneous names. Tthe names provided led PCA to try to relate these names with thos e produced in linear models, which has resulted in their attemp t to estimate components of variance.

‘Accuracy’ refers to agreemen t of the estimates with th e population value. PCA would like to test the accuracy of the UP L, not the reliability (precision).

5. Page ES-5 to ES-7 PCA claims of finding errors in the data wi ll generate different results than those obtained by EPA. A closed evaluation of th e datasets is important, otherwise EPA estimates and PCA estimates cannot be compared.4 of 4
Reference Executive Summary Neter, J., Wasserman, W., and Kutner, M. H. (1990) Applied Linear Statistical Models

, Third Edition, Irwin:IL

Section 1 of Appendix A to PCA Public Comments. 1. Page 1-1, paragraph 2 : PCA has noticed a mixing of two concepts, the 99 th percentile, the 99% confidence level and the 99% confidence level U pper Prediction Limit (UPL). Both terms are different and the document should clearly specify which term it is referring to. The NESHAP uses the 99% confidence level UPL.

The document titled “DEVELOPMENT OF THE MACT FLOORS FOR THE PROPOSED NESHAP FOR PORTLAND CEMENT” first shows the mixing of terms in the following paragraph:

“ We are proposing to use a 99th pe rcentile UPL. We have chosen the 99 th percentile UPL as a reasonab le upper limit because only one percent of future tests of the MACT pool of lowest emitting kilns will exceed the lim it if they are performing as well as the emission test data indicate (i.e., these kilns will achieve the limit 99 percent of the time in the future) .” [page 4, second paragraph before Section 2.]

The statement “... one percent of future tests of th e MACT pool of lowest emitting kilns will exceed the limit if they are performing as well as the emission test data indicate” is the interpretation of the 99 percentile. The correct interpretation for the 99 th confidence UPL is “... one percent of the time m- future tests of the MACT pool of lowest emitting kilns will exceed the limit if th ey are performing as well as th e emission test data indicate

The following modification corr ects the above paragraph:

“ We are proposing to use a 99th confidence UPL. We have chosen the 99 th confidence UPL as a reasonable uppe r limit for m-future tests of the MACT pool of lowest emitting kilns because we expect that m- future test results will be below the UPL 99 percent of the time .”

The PCA reviewer is correct in th at the UPL does not generate the 99 th percentile for the data. The UPL provides an upper bound for a mean value or for m-future observations, this upper bound will be achieved 99% of the times . Note that the term “99%” refers to a confidence level for the UPL not a percentile of the data. So the statement “In other words, it addresses probability, not a proportion” is correct.

2. Page 1-2: Section 1-1

. Assumptions 1 and 2 are typical in simila r studies. The selected data is assumed representative based on certain criteria established by th e subject-matter experts and engineers.5 of 5
If the statistical methods developed violates one or more of the assumptions listed in page 1-2 the resulting UPL will likely be wider th an that calculated under the assumption that all the assumptions hold (Hahn & Meeker, 1993).

t-test based confidence intervals are robust against moderate depa rtures from normality (Steenkamp & van Trijp, 1991).

When sample sizes are equal among groups, the t-test is robust to heterogeneous variances. However, when sample sizes are unequal and heterogene ous variances occur, the results of the t-test are compromised with respect to the likelihood of making a type I

error.

Relatively modest violations of independence can lead to substantial bias in estimated standard errors. In turn, the confidence levels and significance probabilities will be biased too (Gerald et. al, 1993)

Assumptions 4 and 6 will affect the width of the prediction limit. If the data are not independent then the variance term used in the calculation of the UPL is probably being underestimated and as a result a more c onservative UPL is produced (Oakes, 1990)

3. Page 1-3. Section 1.2. PCA is correct suggesting a section or appendi x with descriptive st atistics (boxplots and histograms, cumulative distributi on plots, outlier anal ysis, etc) will give an idea of the distribution. Add goodness of fit test for ev aluating any distribution assumption (e.g. normal).

4. Page 1-3. Section 1.3. PCA is correct regarding what seems a confusion in the use of terms (99 th percentile and 99% confidence level) as mentioned in “ Page 1-1, paragraph 2 ”.

5. Page 1-5. Section 1.3. The equations provided in the or iginal literature are incorrect and the result of an error when defining the “r” factor (i.e. multiplie r) in Gibbons paper (Gibbons, 1991, page 456).

An easy way to develop the correct equation is the following: The MACT floor limit is defined as an uppe r prediction limit for “m” new test runs. Suppose there are N sources, each providing n i runs. To determine the prediction upper limit for the mean of “m” new test runs we need to determine the distribution of:

m p X X

− , where p X is the pooled mean for the N sources, and

m X is the pooled mean for
the new test runs.

If both means, p X and

m X , have the same distribution, th en by the central limit theorem,

the random variable

m p X X

− has a normal distribution with mean 0 and variance
( ) m p Var X X

− .6 of 6

Then the (1- α )- upper prediction interval for

m p X X

− is given by:


# ( ) ( )

( ) 1 1

1 m p
m p m p

m p

m p m p X X z X X z Var X X Var X X

X X z Var X X α α

α − −

− − > ⇒−> −

−

⇒>+ −

The expression: 

# ( )
1 p m p X z Var X X

α

− +

− is the UPL

Further development on the variance term, 

# ( ) ( ) 

# (

mp m p Var X X Var X Var X −= + 

# ) , results in the expression found in the MACT report:

UPL: 2
2 1 1 1 N

p i i i xt s nN m

= ⎛ ⎞
+ + ⎜ ⎟ ⎝ ⎠ 

## ∑

Where the t -statistic is used instead of the z- statistic to account for small sample size and the unknown standard deviation.

In the same paragraph, PCA comments that th e tables in the webs ite are incorrect. We have revised Gibbons (1991) and we verified PCA comments that the column “two-sided K” for the “Tolerance factors (K) for 95% confidence and 95% coverage” table corresponds to the “95% coverage” column in table 2 for one-sided normal tolerance for 95% confidence and 95% coverage (Gibbons, 1991, p.568)

6. Page 1-6. Section 1.3. PCA recommends the use of Tolerance Limits (TL) instead of Prediction Limit (PL). As mentioned by Gibbons (1991) TLs are best for la rge datasets and PLs for small datasets, in other words, if we have to decide complia nce from few new test runs, then we need the upper prediction limit (UPL), but if we are more concerned about compliance based on a big proportion (say 95% or 99%) of the data, th en the upper tolerance limit (UTL) is best. Since the UTL is more conservative (larger cutoff value) then more facilities can be found in compliance than with the UTL.

When defining the TL PCA had a typo: UTL is calculated as: the mean plus the TL factor, not as the mean plus the TL, because the TL consists of both, the upper and the lower tolerance limits.

7. Page 1-7. Section 1.3. With respect to the robustness of the TL, Sharpe (1970) showed that it is generally robust to violations of normality.

8. Page 1-9. Section 1.47 of 7
RTI developed equation 7. See comments #5, Page 1-5. Section 1.3 or introduction to the development. The squared root term in equation 7 (p age 1-10, section 1.4) comes from the piece 

# ( )

m p Var X X

− as noted below: 

# ( )

m p Var X X

− 

# ( ) 2

p Var X
m σ = + (assuming independence of p X and

m X )
After assuming independence: 

# ( ) 2

2 2 1 1 1 1 ( ) N N i p i i i i s Var X Var x N N

= = = = 

## ∑ ∑

n

where 

# ( ) 2

1 2 ; 1,..., . ( 1) i n

ij i j
i
i x x s i
n = −

= =

− 

## ∑

N

Let’s estimate 2

1 1 N

i i s N σ

= = 

# ∑ 2 , resulting in 

# ( )

m p Var X X

− 22 2

2 2 11 1 NN N ii i

ii i i i s s ss Nm N n Nm N n

== = ⎛ ⎞
=+ = + ⎜ ⎟ ⎝ ⎠ 

## ∑∑ ∑ 2 i

An ANOVA approach would have resulted in similar values (except for rounding) since: 

# ( )

m p Var X X

− = M SE

m +

1 1 N

i i MSE

N n

= 

## ∑

. The linear model based appro aches suggested by PCA necess itates the use of special software (SAS). The proposed method was easily implemented in standard software such as EXCEL. Also, the terms between groups and among groups were incorrectly given to the two terms in the equation, which has lead PCA to suggest using ANOVA and other model approaches to estimate th e components of the equation.

.

References Section 1 :

Neter, J., Wasserman, W., and Kutner, M. H. (1990) Applied Linear Statistical Models

, Third Edition, Irwin:IL

Steenkamp, J.E.M., & van Trijp, H.C.M. (1991). The use of LISREL in validating marketing constructs. Intern ational Journal of Research in Marketing, 8: 283-299

Gerald J. Hahn and William Q. Meeker. 1993. Assumptions for Statistical Inference. The American Statistician, 47(1):1-11

Oakes M. W. (1990) Statistical Inference. Chestnut Hill, MA. Epidemiology Resources Inc.8 of 8
Gibbons, R.D. (1991). Statistical Tolerance limits for Ground-water monitoring. Ground water. 29(4);563-570

Gibbons, R.D. (1987) statistical prediction in tervals for the evaluation of ground water quality. Ground Wate r, 25(4):455-465

Sharpe, K. (1970). Robustness of Normal Tolerance Intervals. Biometrika, 57(1):71-78

Section 2 of Appendix A to PCA Public Comments. 1. Page 2-1. Section 2.1. According to last paragraph in page 2-1, emissions from kiln 1233 are lower in Hg concentrations than those expected from a repres entative kiln, if this is true, then results could be biased towards lower va lues, especially if statistics such as the mean are used.

2. Page 2-3. Section 2.2.1. As a result of the analysis, probably they used the inter quantile range (IQR=75% quartile – 25% quartile) . The rule for detecting ou tliers is the following: Q1-1.5×IQR and Q3+1.5×IQR, any data point outsi de these range is a possible outlier; and any data point outside Q1-3×IQR and Q3+3×IQR will be considered an outlier.

3. Page 2-3. Section 2.2.2. Some of the variance values in Table 2-2 pages 2-12 to 2-16 are smaller than 0.001 and are reported as 0.000 in tables. We recommend to either use more decimals or scientific notation to report variances and standard deviations. From table 2-2 the position of the mean with respect to the median suggests that most of the distributions are not normal but skewed. Also, if the skew statistic is not close to zero then the distribution is not symmetric. Table 2-3, page 2-17 shows the results of th e test for the normality assumption. The null hypothesis in this test is H0 : The data have a normal dist ribution, and the alternative hypothesis is Ha: The data do not have a norma l distribution. The small p-values of the different statistics (p<0.0001) provides ev idence against the null hypothesis. Then one can conclude, with a significance level greate r than 99%, that Hg mean concentrations are not normal distributed. Statistical tests for assessing log-normal, and weibull distributions for the Hg mean concentrations were not found. Fig 2-1 to Fig 2-6 (pages 2-35 to 2-40) show what appears to be quantile-quantile plots (qq-plots) for normal, lognormal and exponentia l distributions, with different parameters, for the limestone concentration. Are the limestone concentration the values obtained in each test run? From these plots nothing can be said regarding the Hg mean concentration. These plots are visual tools and can not be taken as a formal goodness of fit tests. There is no information in how the parameters for the different distributions used in the qq-plots were estimated. There is no way to know if other parameters would have provided a better fit. Figures 2-1 to 2-6 shows an outlier in the top ri ght portion of the plot.

Fig 2-10 to 2-15 (pages 2-44 to 2-49 show qq-plots for evaluating normal, lognormal and exponential distribution assump tion for the Hg mean concentration. Plot 2-11 ad 2-129 of 9
suggests a good fit to the lognormal and expone ntial distribution. Again a goodness of fit test would have provided a way to determine wh ich is the best fit. No information in the process to estimate the parameters used in the qq-plots were provided. There are twolognormal distributions (figures 2-11 and figure 2-14), how to decide which log-normal distribution best fitted the data?

Figures 2-10 to 2-15 show two outliers (the two points way above the straight line).

4. Page 2-3. Section 2.2.2. Statistical Conclusions The problem with outliers is that no one knows if the data point is wrong or not. The extreme value or outlier could be a correct pi ece of information if the distribution has indeed a long tail, so what are we observing is a valid number of the long tail. We should remove the outlier only if ther e is a technical reason for thei r aberrant behavior (Burke, 1997). Random variations generate occasional outliers by chance; they are part of the valid data and should generally be included in any calculations (Farrant, 1997, Hayden, 2005). So, the average concentrations above 96 ppb may be a valid observation from the same skewed distribution. Statistic tests shou ld not be the only source to decide if a datapoint needs to be removed from the data or not; that decision s hould be made after carefully reviewing the possible technical reasons that might ha ve resulted in the observance of such value..

Other tests for outliers are the Cochran’ test and the Grubb’s test (for normal data); and the Dixon tests for small da tasets (Farrant, 1997).

5. Page 2-4. Section 2.3. PCA should report the units of the measures reported in Table 2-5 (page 2-19 to 2-21). The skew statistics (skew column) reported sugg est that most of the distributions are not symmetric, suggesting a skewed distribu tion such as lognormal or weibull. Table 2-6 (page 2-22) reports the statistics for testing the normality assumption of Hg mean emissions. The p-values suggest that the normality assumption is not satisfied by the Hg mean concentrations. Figures 2-16 to 2-27 (pages 2-50 to 2-60) show the qq-plots of normal, log-normal and exponential distribution with different parameters for Hg emissions. We assumed Hg emissions correspond to Hg mean emissions. No detail on how the estimates of the parameters (mean and standard deviation fo r normal and log-normal, and scale and shape for exponential) used in the qq-plots were obtained and the reason to show two plots for normal, lognormal and exponential distributions . No testing for the weibull distribution was found. As mentioned before, the qq-plot s are visual aids and should not replace a formal goodness of fit test.

PCA should report the units in each table for the read er understanding.

The outlier testing results (Tab le 2-7, page 2-23) suggests ou tliers values are those above 209.5 rounded by PCA to 210 lb/1E6. Are these th e units for the values reported in table 2-7? What was the action taken as a result of the outlier detection?

6. Page 2-4. Section 2.5.10 of 10
Reason to replace kiln 1233 with kiln 1594 should be stated.

7. Page 2-5. Section 2.5. Table 2-9, page 2-27 shows an overall mean of 18.5 not 18.4 as reported by PCA in first line of page 2-5. Are the nu mbers reported in page 2-9 the result of the corrections described in page 2-2? The kiln numbers re ported in table 2-9 do not match any of the kiln numbers mentioned in page 2-2 as having data problems. Need to check if the same kilns selected by PCA as the 12% are the same used in EPA analysis.

In fig 2-31 (page 2-64), are these the Hg m ean emissions? The position of the median (horizontal lines) in the boxplots suggest sk ew distributions. What was the action taken by PCA regarding the outliers?

The larger variance suggest the presence of la rger numbers in the dataset used by PCA.

Fig 2-32, page 2-65, shows a bi-modal dist ribution, probably the combination of two skewed distributions -- look at the long tail at the ri ght. What lognormal, normal, exponential and weibull distribut ion were fitted to the data? How were the parameters of the distributions estimated?

There is a tendency to not specify the units and clarify the re sponse in the PCA report. It is not clear sometimes if they are referring to the daily emissions or the mean emissions.

Figures (2-57 to 2-60) show the qq-plots for the Hg daily emissions. There are some outliers, but in the ab sence of formal testing for the lognormal and weibull distributions, there is no support for th e graphical displays. Table 2-10 (page 2-28) provides different stat istic to evaluate the normality assumption. The Shapiro-Wilks, which is the most robus t of all of them, shows a small p-value, suggesting that the distribution of Hg daily emissions is not normal distributed.

8. Page 2-5. Section 2.6. The value reported in table 2-1 (page 2-11) as the pooled mean is 18.4. PCA can’t conclude without a test that the two values: 8.59 and 7.1 are “virtually identical”. A statistical test will provide evidence in favor or against such assumption. The two variance components in comment #5 of Section 1 were mistakenly referred as the between-source (VB) and w ithin-source (VW) variance co mponents. These terms are
the variance of the new runs and th e variance of the previous data: 

# ( ) 

# ( )

m p Var X Var X +

. The between group (between kilns) variability is expected to be larger than the within group (within kiln) variability since you expect to find more variation between kilns than internal variations in a singl e kiln. As was explained in comment #5, the terms involved in the construction of the UPL do not i nvolve an estimate of the between group variability, so the terms in the aforementi oned equation are estimat ed using reliable estimates. The pooled variance works under the assumption that the vari ances are the same across kilns. This assumption will not be appropriate if the different sources operate very differently, which would result in a dis tinctly different variance for each source.11 of 11

The difference in standard deviations seems to be due to some change in the data. Do we know if PCA is using the same top 12% kilns?

PCA is confused regarding the interpretation of the Upper Tolerance Limit. This limit does not reflect the 99 th percentile. As mentioned before , the tolerance limits constitute an interval that contains 99% of the data w ith some level of confidence. In general the tolerance limit will be greater than the 99 th percentile.

Table 2-12 was not found in the document to verify item a. in page 2-6. Results in item b mentioned in page 2-6 were found in table 2-45, page 2-33. As mentioned before it is not surprised that the UTL will be larger than the UPL.

Comment #6 in page 2-6 shows that PCA has confused the 99 th confidence UPL with the 99 th percentile.

The suggestion of using non-parameteric methods is a valid one given the non-normality of the data.

9. Page 2-7. Section 2.7 The 30 day rolling averages are common in tim e series. They are used to smooth shortterm fluctuations and emphasize long-term fluctuations . I think that what they have is a sample of 30 days period, assuming that the se lected 30 days is a representative sample of the whole year. The issue is to select a re presentative sample and unless someone can clearly identify most of the factors that will affect the emissions in a given year, it is reasonable to assume that in a given month we can observe all the possible values that one can observe in a given year. If months are different, then one can clus ter months in low concentration and high concentration months, then one can select some days from the low concentration and other days from the high concentration m onths. But if no differences are assumed between months, then selecting at random a month (30 days) would have provided a representative sample.

Since the analysis combines the days in a kiln (using the mean), then there is no need to worry about the independence of the days, because the kilns are assumed independent.

With respect to the homocedasticity assu mption the equation in comment #5 in this document is based on the assumption of different variances.

10. Page 2-8. Section 2-7.

The relative standard deviat ion (RSD), calculated as S

x , is a measure of precision of the
estimate (the mean in this case). Similar RSD will imply that the precision of two estimates is approximately the same; that doesn’t allow any conclusion regarding homocedasticity as PCA mistakenly concluded. It is not clear how the “mean RSD” was calculated when five quarries were used.12 of 12

To fully incorporate different sources of variability one would have to consider simulation methods such as bootstrapping or ja ckknife, and assume distributions for the quarry uncertainty, kilns uncertainty , runs uncertainty and so on.

In section 2.7.4 (page 2-8) PCA explores th e relationship between Hg concentration and Hg in limestone. The value of (coefficient of determination) is interpreted as the total variability in the response (Hg concentratio n) explained by Hg in limestone. The squared root of

, r , is the coefficient of correlation, and is interpreted as linear relationship between Hg in limestone and Hg concentration. In the simple linear models in Fig 2-61 (page 2-94) and Fig 2- 62 (page 2-95) the is 0.7134 and 0.9687, respectively. However, these two simple linear mode ls are affected by several high influence data points. Removal of these high influence data point s might change the resulting models. 2 r

2 r

2 r

On the other hand, the usefulness of the regr ession model depends of the width of the confidence prediction inte rval and the needs for precision (Neter, 1990). We observed that most of the points are outside the confidence bands surrounding the prediction models, suggesting poor prediction capabilities. However, a formal assessment looking at residuals, high leverage points, high influence points, etc. are needed to evaluate the goodness of fit of the proposed models

A possible explanation for the non relationshi p between Hg concentration and Hg in limestone when looking at the MACT floor data, is that limestone may be confounded with another variable (not included in the m odel) that could be mo re powerful explaining Hg concentration. In particular, Fig 2-62 can change dramatica lly if the two points w ith 0.7 or more Hg limestone are removed from the data. In th is case, probably the slope of the linear relationship would have change. After removi ng the high influence points, the remaining data (the cluster of points in page 2-65) ha ve Hg concentration mean values lower than 0.1. The MACT floor data should not be representative of the population of kilns, the whole sample where the MACT floor data were selected is the one that need to be representative. The MACT floor data should re present the best kiln s in terms of having lower Hg concentration values.

11. Page 2-9. Section 2.7. With respect to comment 2.7.5, EPA can clar ify how the data was obtained. To address long term variability, EPA should use data co llected over a long period of time, and that have similar range as those observed in similar or longe r time intervals.

Figures 2-78 and 2-79 (pages 2-110 and 2-111) are incomplete and not clear enough for interpretation.

12. Page 2-10. Section 2.8 The representativity of the data needs to be justified by comparing the characteristics of the sample with the characteristics of the population of kilns. Concentrate in13 of 13
demonstrating that the sample possesses the same distribution in the relevant characteristics of the population. To include a ll sources of va riability into the analysis will require use of simulation approaches. Some sources of variability can be controlled by stratifying the population of kilns according to levels of those variables creating the variability that ones desires to control.

Reference Section 2. Hubert, Mia and Van der Veeke, Stephan. 2008. Outlier detection for skewed data. Journal of Chemometrics. 22(3): 235-246

S. Burke. 1997. Scientific Data Management 1(1): 32–38

T.J. Farrant. 1997. Practical stat istics for the analytical sc ientist: A bench guide. Royal Society of Chemistry.

Hayden, Robert W. 2005. A dataset that is 44% outliers. Journal of Statistics Education Volume 13, Number 1. www.amstat.org/publications/jse/v13n1/datasets.hayden.html

Neter, J., Wasserman, W., and Kutner, M. H. (1990) Applied Linear Statistical Models

, Third Edition, Irwin:IL

Note for sections 3,4 and 5: Similar issues to discussed in Section 2 were found in sections 3, 4 and 5.Comments on Florida Department of Environmental Protection reply.

Page 3. The use of Monte Carlo simulation or bootstrap simulation has been used to account for different types of uncertainties. In particular, in the presence of non-detect values, simulations have been used to s ubstitute the non detect values for randomly selected plausible values derived from suita ble distributions such as Weibull or lognormal. In the presence of non-detects the use of standard statisti cs techniques produces biased estimates (Zhao and Frey, 2006 and Helsel, 2005). So, the suggestion by Florida Department of Environmental Protection follo ws this recent tendency to use simulation techniques to account for uncer tainties in the data. One of the uncertainties that researchers address is the pres ence of non-detects. The way to do that is to generate 1000 or more replicates of the data , calculate your analysis for each data set (e.g . calculate the UPL), take the mean of the 1000 or so UPLs , and report the mean the UPLs and the 2.5 and 97.5 percentiles of the UPLs as confidence limits.

Zhao, Yuchao and Frey, H. Christopher. 2006. Uncertainty for Data with Non-Detects: Air Toxic Emissions from Combustion. Huma n and Ecological Risk Assessment, 12: 1171–1191

Helsel DR. 2005. Nondetects and data analysis : Statistics for censored environmental data. Wiley, Hoboken, NJ, USA

Comments on HOLCIM (US) INC reply

Page 4, second paragraph of section B. HOL CIM had misunderstood the definition of the Upper Prediction Limit (UPL). The UPL proposed by EPA has a 99% confidence limit and it is not a confidence limit for the 99 pe rcentile. The use of some terms in EPA’s document has lead readers to think a bout a confidence interval for the 99 th percentile instead of a 99% confidence UPL.

The interpretation given by HOLCIM in the se cond paragraph mixes the concept of UPL and 99 percentile. Since the UPL are calculated for a set of “m” new tests, the interpretation is forced to link the “m” ne w tests together. So, one can conclude “we expect that 99% of the times, the m-new test s will be below the UPL, and only 1% of the times all the m-new tests will not fall below the UPL.”

The interpretation given by HOLCIM in th e second paragraph corresponds to a confidence interval for the 99 th percentile, not to the UP L with a confidence of 99%. HOLCIM mistakenly believes that EPA wants to calculate the 99 th percentile.

Page 5, first paragraph. In the first para graph, HOLCIM is commenting on the use of a confidence interval for the average of the 12 best performing MACT pool sources. The statement seems to be based in the “Centr al Limit Theorem” which states that the sampling distribution of the average is a norma l distribution; in other words, when the process of calculating the mean of the best 12 performing MA CT pool sources is repeated many times (40 or more) and we plot the calculated means (40 or more values), thedistribution or histogram of these means show s a symmetric shape, in this case the mean and the median coincide. Under this assumpti on, one could say that 50% of the data will be below or above the mean. In general, this is not the correct statement for a confidence interval based on the mean or average. The comment (“even the MACT floor kilns could exceed the limit half the time or more”) is appropriate for a confidence interval of the median, not for the mean. Page 5, third line from the bottom in second paragraph. The statement refers to a confidence interval for the 99 th percentile, not the UPL w ith a 99% confidence limit.

Page 5. Third paragraph. HOLCIM establishes that they understand from EPA documents that EPA wants to estimate the 99 th percentile or a confiden ce interval for the 99 percentiles. HOLCIM recognizes that EPA has calculated the UPL with 99% confidence and not the 99 th percentile, and insists that EPA wants to use the UPL to estimate the 99 th percentile. HOLCIM provides an appropr iate definition of the UPL with 99% confidence. HOLCIM comment (line 7) that the UPL is a pr obability is not correct, since the UPL is an upper bound for future observations. The comment that “the MACT floor will exceed the value (the UPL value) ...” (line 3 rd from bottom of third paragraph) is not comple tely correct, since the UPL is calculated as the mean of the 12 percent plus minus the variance of the MACT floor sources and the estimated variance of the new sources. It is very unlikely that it won’t contain the MACT floor sources “a signific ant number of times”.

Page 5. fourth paragraph. The comment that the UTL (upper tolerance limit) generates the 99 th percentile is not correct. It is very like ly that the UTL will be larger than the 99 th percentile and that the UPL will be smaller than the 99 th percentile.

Page 6. first paragraph. The comment that th e UTL is more reliable because it contains the 99 th percentiles is based on the mistaken assu mption that EPA intends to estimate the 99 th percentile, not to estimate an upper bound for new tests (i.e. a UPL).

Pages 9 and 10. HOLCIM calculated the UPL, UTL and 99 th percentiles for THC MACT floors. I think there is a missunderstanding of the use of the UPL. The UPL is an upper

bound for the mean of the new observations, 2
2 1 1 1 N

p i i i xt s nN m

= ⎛ ⎞
+ ⎜ ⎝ ⎠ 

## ∑ + ⎟ . I don’t see the

reason for HOLCIM calculating the UPL and U TL using the new data. They need to calculate the mean of the new data and compare it with the UPL.

In table 2, HOLCIM is comparing the UPL calcu lated with the existi ng 30 day data, with the actual 99 th percentile, which doesn’t make sens e since the UPL was not calculated as an estimator of the 99 th percentile. HOLCIM did not eval uate the validity of the UPL since they didn’t compare the mean of the ne w runs with the value of the calculated UPL using the existing data.Note that in Table 2, page 10, for the lo west five THC emitting sources, the kiln #5, Holcim Mason City Kiln 2, did not satisfy th e UPL since 10.91 > 10.4 but it is borderline.

In the remaining tables (Table 3, page 13; Table 4, page 15; table 6 page 19) show both the UPL and UTL; and in all these tables, th e average values for each kiln are below the UPL as expected.