1 of 1 

Review of Comments on EPA’s Statistical Approach Used to Develop the Proposed 

MACT Floor Standards for Portland Cement Manufacturing 

August 6, 2010
2 of 2 

Revision of Data Analysis and Variability for 

Portland Cement NESHAP 

Prepared by Breda Munoz 

Executive Summary to Appendix A of Portland Cement Association public 

Comments (comment 2845)

1. Page ES-1 

Portland Cement Association (PCA) claims that EPA’s intentions are to estimate the 99 

percentile. However, I have reviewed documents that estimate the 99 confidence upper 

prediction limit (UPL), not the 99th percentile. 

Here are basic definitions: 

The 99th percentile is the value (or score) below which 99 percent of the observations 

may be found 

The confidence level describes the uncertainty associated with a sampling method used to 

collect the data. A 99% confidence level in an interval (e.g. prediction interval) means 

that we would expect 99% of the times the interval estimates (lower and upper limits) to 

include the population parameter 

The prediction interval is an estimate of an interval in which future observations will fall, 

with a certain probability (i.e. confidence level), given what has already been observed. 

The upper prediction limit (UPL) is the upper limit of the prediction interval. The 99% 

confidence UPL is the upper limit of the prediction interval calculated with a 99% 

confidence (Neter, 1990) 

PCA defines a percentile as a variable, and it is a population or sample value, not a 

random variable. The percentile of a population is a parameter, has no variability 

associated with it, it is a fixed value. An estimate of the percentile has a variability 

associated with it, then a confidence interval can be calculated for the population 

percentile using the sample percentile. 

PCA’s statement, “the 99th percentile should account for all but one percent of the 

possible values, with a specified level of confidence” is not correct. The 99th percentile is 

the cut-off value below which 99% of the data are found, and this occurred with 

certainty. 

In the second paragraph, PCA has several sentences that are not correct. PCA seems to 

confuse the calculation of the percentiles in a sample or population, with confidence 

intervals for the population percentiles. 

PCA seems to conclude that the UPL was proposed to estimate the 99 percentile of the 

population. To this end, PCA proposes the upper tolerance limit (UTL). In the case of 

smaller datasets, the UPL is better than the UTL, (see discussion in Section 1). 

2. Page ES-2. 
3 of 3 

UTL is not for prediction purposes, so it won’t say what the data will be in the future. 

The prediction properties are a characteristic of the UPL. Both UPL and UTL are based 

on the present data, but UPL take into account the future observations variability. This is 

why EPA is using the UPL statistic rather than the UTL (along with the suitability of 

applying UPL to datasets like those in the NESHAP). 

Reliability refers to consistency, agreement of values. Maybe PCA wanted to assess the 

precision (variability) of both UPL and UTL? The UPL and UTL have different purposes 

(see Section 1 comments). An estimator is consistent if it produces the same value under 

several repetitions. In this case, containing or not the 99 percentile can not be used to 

evaluate the reliability of the two estimators, UPL and UTL. By definition, the 99% UTL 

will probably contain the 99 percentile. 

If EPA wants to estimate the 99 percentile, then there are methods appropriate to do so. If 

the idea is to obtain an upper bound that is greater than the 99 percentile, then the UTL 

works. The UTL will generally be a larger number than the 99 percentile. 

3. Page ES-3 

The assumption of a representative sample can be proved by comparing both the 

population and the sample in relevant characteristics (variables). For example if some 

type of material is used in the population, then the same relative quantity of this material 

should be present in the sample. 

The sample will never have the same level in all the characteristics as the population. So 

it is important to define the most important characteristics or sources of variability, and 

demonstrate that both the sample and the population present the same distribution in 

these characteristics. 

The statement that “UPL will provide a probability and not a proportion” is not correct. 

The UPL will provide an upper bound for future observations with a given confidence 

limit. 

4. Page ES-4 

The terms in the variance used to generate the UPL were given erroneous names. Tthe 

names provided led PCA to try to relate these names with those produced in linear 

models, which has resulted in their attempt to estimate components of variance. 

‘Accuracy’ refers to agreement of the estimates with the population value. PCA would 

like to test the accuracy of the UPL, not the reliability (precision). 

5. Page ES-5 to ES-7 

PCA claims of finding errors in the data will generate different results than those 

obtained by EPA. A closed evaluation of the datasets is important, otherwise EPA 

estimates and PCA estimates cannot be compared. 
4 of 4 

Reference Executive Summary

Neter, J., Wasserman, W., and Kutner, M. H. (1990) Applied Linear Statistical Models, 

Third Edition, Irwin:IL 

Section 1 of Appendix A to PCA Public Comments.

1. Page 1-1, paragraph 2: 

PCA has noticed a mixing of two concepts, the 99th percentile, the 99% confidence level 

and the 99% confidence level Upper Prediction Limit (UPL). Both terms are different and 

the document should clearly specify which term it is referring to. The NESHAP uses the 

99% confidence level UPL. 

The document titled “DEVELOPMENT OF THE MACT FLOORS FOR THE 

PROPOSED NESHAP FOR PORTLAND CEMENT” first shows the mixing of terms in 

the following paragraph: 

“We are proposing to use a 99th percentile UPL. We have chosen 

the 99th percentile UPL as a reasonable upper limit because only 

one percent of future tests of the MACT pool of lowest emitting 

kilns will exceed the limit if they are performing as well as the 

emission test data indicate (i.e., these kilns will achieve the limit 99 

percent of the time in the future).” [page 4, second paragraph 

before Section 2.] 

The statement “... one percent of future tests of the MACT pool of lowest emitting kilns 

will exceed the limit if they are performing as well as the emission test data indicate” is 

the interpretation of the 99 percentile. The correct interpretation for the 99th confidence 

UPL is “... one percent of the time m- future tests of the MACT pool of lowest emitting 

kilns will exceed the limit if they are performing as well as the emission test data indicate

The following modification corrects the above paragraph: 

“We are proposing to use a 99th confidence UPL. We have chosen 

the 99th confidence UPL as a reasonable upper limit for m-future 

tests of the MACT pool of lowest emitting kilns because we expect 

that m- future test results will be below the UPL 99 percent of the 

time.” 

The PCA reviewer is correct in that the UPL does not generate the 99th percentile for the 

data. The UPL provides an upper bound for a mean value or for m-future observations, 

this upper bound will be achieved 99% of the times. Note that the term “99%” refers to a 

confidence level for the UPL not a percentile of the data. So the statement “In other 

words, it addresses probability, not a proportion” is correct. 

2. Page 1-2: Section 1-1. 

Assumptions 1 and 2 are typical in similar studies. The selected data is assumed 

representative based on certain criteria established by the subject-matter experts and 

engineers. 
5 of 5 

If the statistical methods developed violates one or more of the assumptions listed in page 

1-2 the resulting UPL will likely be wider than that calculated under the assumption that 

all the assumptions hold (Hahn & Meeker, 1993). 

t-test based confidence intervals are robust against moderate departures from normality 

(Steenkamp & van Trijp, 1991). 

When sample sizes are equal among groups, the t-test is robust to heterogeneous 

variances. However, when sample sizes are unequal and heterogeneous variances occur, 

the results of the t-test are compromised with respect to the likelihood of making a type I 

error. 

Relatively modest violations of independence can lead to substantial bias in estimated 

standard errors. In turn, the confidence levels and significance probabilities will be biased 

too (Gerald et. al, 1993) 

Assumptions 4 and 6 will affect the width of the prediction limit. If the data are not 

independent then the variance term used in the calculation of the UPL is probably being 

underestimated and as a result a more conservative UPL is produced (Oakes, 1990) 

3. Page 1-3. Section 1.2. 

PCA is correct suggesting a section or appendix with descriptive statistics (boxplots and 

histograms, cumulative distribution plots, outlier analysis, etc) will give an idea of the 

distribution. Add goodness of fit test for evaluating any distribution assumption (e.g. 

normal). 

4. Page 1-3. Section 1.3. 

PCA is correct regarding what seems a confusion in the use of terms (99th percentile and 

99% confidence level) as mentioned in “Page 1-1, paragraph 2”. 

5. Page 1-5. Section 1.3. 

The equations provided in the original literature are incorrect and the result of an error 

when defining the “r” factor (i.e. multiplier) in Gibbons paper (Gibbons, 1991, page 456). 

An easy way to develop the correct equation is the following: 

The MACT floor limit is defined as an upper prediction limit for “m” new test runs. 

Suppose there are N sources, each providing ni runs. To determine the prediction upper 

limit for the mean of “m” new test runs we need to determine the distribution of: 

X m − X p , where X p is the pooled mean for the N sources, and X m is the pooled mean for 

the new test runs. 

If both means, X p and X m , have the same distribution, then by the central limit theorem, 

the random variable X m − X p has a normal distribution with mean 0 and variance 

( ) Var X X m p − . 
6 of 6 

Then the (1-α)- upper prediction interval for X m − X p is given by: 

( ) ( )

( )

1 1

1

m p

m p m p

m p

m p m p

X X

z X X z Var X X

Var X X

X X z Var X X

α α

α

− −

−

− > ⇒−> −

−

⇒>+ −

The expression: X p m + z Var X X 1−α ( − p ) is the UPL 

Further development on the variance term, Var X X Var X Var X ( mp m p −= + ) ( ) ( ) , results 

in the expression found in the MACT report: 

UPL: 2

2

1

1 1 N

p i

i i

xt s = nN m

⎛ ⎞

+ + ⎜ ⎟ ⎝ ⎠ ∑

Where the t-statistic is used instead of the z-statistic to account for small sample size and 

the unknown standard deviation. 

In the same paragraph, PCA comments that the tables in the website are incorrect. We 

have revised Gibbons (1991) and we verified PCA comments that the column “two-sided 

K” for the “Tolerance factors (K) for 95% confidence and 95% coverage” table 

corresponds to the “95% coverage” column in table 2 for one-sided normal tolerance for 

95% confidence and 95% coverage (Gibbons, 1991, p.568) 

6. Page 1-6. Section 1.3. 

PCA recommends the use of Tolerance Limits (TL) instead of Prediction Limit (PL). As 

mentioned by Gibbons (1991) TLs are best for large datasets and PLs for small datasets, 

in other words, if we have to decide compliance from few new test runs, then we need the 

upper prediction limit (UPL), but if we are more concerned about compliance based on a 

big proportion (say 95% or 99%) of the data, then the upper tolerance limit (UTL) is best. 

Since the UTL is more conservative (larger cutoff value) then more facilities can be 

found in compliance than with the UTL. 

When defining the TL PCA had a typo: UTL is calculated as: the mean plus the TL 

factor, not as the mean plus the TL, because the TL consists of both, the upper and the 

lower tolerance limits. 

7. Page 1-7. Section 1.3. 

With respect to the robustness of the TL, Sharpe (1970) showed that it is generally robust 

to violations of normality. 

8. Page 1-9. Section 1.4 
7 of 7 

RTI developed equation 7. See comments #5, Page 1-5. Section 1.3 or introduction to the 

development. 

The squared root term in equation 7 (page 1-10, section 1.4) comes from the piece 

Var X X ( m p − ) as noted below: 

Var X X ( ) m p − ( )

2

Var X p m

σ

= + (assuming independence of X p and X m ) 

After assuming independence: 

( )

2

2 2

1 1

1 1 ( )

N N

i

p i

i i i

s Var X Var x

N N = =

= = ∑ ∑ n

where 

( )2

2 1 ; 1,..., . ( 1)

i n

ij i

j

i

i

x x

s i

n

=

−

= = −

∑

N

Let’s estimate 2

1

1 N

i

i

s

N

σ == ∑ 2 , resulting in 

Var X X ( ) m p −

22 2

2 2

11 1

NN N

ii i

ii i i i

s s ss

== = Nm N n Nm N n

⎛ ⎞

=+ = + ⎜ ⎟ ⎝ ⎠ ∑∑ ∑

2

i

An ANOVA approach would have resulted in similar values (except for rounding) since: 

Var X X ( m p − ) = MSE

m

+

1

1 N

i i

MSE

N n =

∑ . 

The linear model based approaches suggested by PCA necessitates the use of special 

software (SAS). The proposed method was easily implemented in standard software such 

as EXCEL. Also, the terms between groups and among groups were incorrectly given to 

the two terms in the equation, which has lead PCA to suggest using ANOVA and other 

model approaches to estimate the components of the equation. 

. 

References Section 1: 

Neter, J., Wasserman, W., and Kutner, M. H. (1990) Applied Linear Statistical Models, 

Third Edition, Irwin:IL 

Steenkamp, J.E.M., & van Trijp, H.C.M. (1991). The use of LISREL in validating 

marketing constructs. International Journal of Research in Marketing, 8: 283-299 

Gerald J. Hahn and William Q. Meeker. 1993. Assumptions for Statistical Inference. The 

American Statistician, 47(1):1-11 

Oakes M. W. (1990) Statistical Inference. Chestnut Hill, MA. Epidemiology Resources 

Inc. 
8 of 8 

Gibbons, R.D. (1991). Statistical Tolerance limits for Ground-water monitoring. Ground 

water. 29(4);563-570 

Gibbons, R.D. (1987) statistical prediction intervals for the evaluation of ground water 

quality. Ground Water, 25(4):455-465 

Sharpe, K. (1970). Robustness of Normal Tolerance Intervals. Biometrika, 57(1):71-78 

Section 2 of Appendix A to PCA Public Comments.

1. Page 2-1. Section 2.1. 

According to last paragraph in page 2-1, emissions from kiln 1233 are lower in Hg 

concentrations than those expected from a representative kiln, if this is true, then results 

could be biased towards lower values, especially if statistics such as the mean are used. 

2. Page 2-3. Section 2.2.1. 

As a result of the analysis, probably they used the inter quantile range (IQR=75% quartile 

– 25% quartile) . The rule for detecting outliers is the following: Q1-1.5×IQR and 

Q3+1.5×IQR, any data point outside these range is a possible outlier; and any data point 

outside Q1-3×IQR and Q3+3×IQR will be considered an outlier. 

3. Page 2-3. Section 2.2.2. 

Some of the variance values in Table 2-2 pages 2-12 to 2-16 are smaller than 0.001 and 

are reported as 0.000 in tables. We recommend to either use more decimals or scientific 

notation to report variances and standard deviations. 

From table 2-2 the position of the mean with respect to the median suggests that most of 

the distributions are not normal but skewed. Also, if the skew statistic is not close to zero 

then the distribution is not symmetric. 

Table 2-3, page 2-17 shows the results of the test for the normality assumption. The null 

hypothesis in this test is H0: The data have a normal distribution, and the alternative 

hypothesis is Ha: The data do not have a normal distribution. The small p-values of the 

different statistics (p<0.0001) provides evidence against the null hypothesis. Then one 

can conclude, with a significance level greater than 99%, that Hg mean concentrations 

are not normal distributed. 

Statistical tests for assessing log-normal, and weibull distributions for the Hg mean 

concentrations were not found. 

Fig 2-1 to Fig 2-6 (pages 2-35 to 2-40) show what appears to be quantile-quantile plots 

(qq-plots) for normal, lognormal and exponential distributions, with different parameters, 

for the limestone concentration. Are the limestone concentration the values obtained in 

each test run? From these plots nothing can be said regarding the Hg mean concentration. 

These plots are visual tools and can not be taken as a formal goodness of fit tests. There 

is no information in how the parameters for the different distributions used in the qq-plots 

were estimated. There is no way to know if other parameters would have provided a 

better fit. 

Figures 2-1 to 2-6 shows an outlier in the top right portion of the plot. 

Fig 2-10 to 2-15 (pages 2-44 to 2-49 show qq-plots for evaluating normal, lognormal and 

exponential distribution assumption for the Hg mean concentration. Plot 2-11 ad 2-12 
9 of 9 

suggests a good fit to the lognormal and exponential distribution. Again a goodness of fit 

test would have provided a way to determine which is the best fit. No information in the 

process to estimate the parameters used in the qq-plots were provided. There are twolognormal distributions (figures 2-11 and figure 2-14), how to decide which log-normal 

distribution best fitted the data? 

Figures 2-10 to 2-15 show two outliers (the two points way above the straight line). 

4. Page 2-3. Section 2.2.2. Statistical Conclusions 

The problem with outliers is that no one knows if the data point is wrong or not. The 

extreme value or outlier could be a correct piece of information if the distribution has 

indeed a long tail, so what are we observing is a valid number of the long tail. We should 

remove the outlier only if there is a technical reason for their aberrant behavior (Burke, 

1997). Random variations generate occasional outliers by chance; they are part of the 

valid data and should generally be included in any calculations (Farrant, 1997, Hayden, 

2005). So, the average concentrations above 96 ppb may be a valid observation from the 

same skewed distribution. Statistic tests should not be the only source to decide if a 

datapoint needs to be removed from the data or not; that decision should be made after 

carefully reviewing the possible technical reasons that might have resulted in the 

observance of such value.. 

Other tests for outliers are the Cochran’ test and the Grubb’s test (for normal data); and 

the Dixon tests for small datasets (Farrant, 1997). 

5. Page 2-4. Section 2.3. 

PCA should report the units of the measures reported in Table 2-5 (page 2-19 to 2-21). 

The skew statistics (skew column) reported suggest that most of the distributions are not 

symmetric, suggesting a skewed distribution such as lognormal or weibull. 

Table 2-6 (page 2-22) reports the statistics for testing the normality assumption of Hg 

mean emissions. The p-values suggest that the normality assumption is not satisfied by 

the Hg mean concentrations. 

Figures 2-16 to 2-27 (pages 2-50 to 2-60) show the qq-plots of normal, log-normal and 

exponential distribution with different parameters for Hg emissions. We assumed Hg 

emissions correspond to Hg mean emissions. No detail on how the estimates of the 

parameters (mean and standard deviation for normal and log-normal, and scale and shape 

for exponential) used in the qq-plots were obtained and the reason to show two plots for 

normal, lognormal and exponential distributions. No testing for the weibull distribution 

was found. As mentioned before, the qq-plots are visual aids and should not replace a 

formal goodness of fit test. 

PCA should report the units in each table for the reader understanding. 

The outlier testing results (Table 2-7, page 2-23) suggests outliers values are those above 

209.5 rounded by PCA to 210 lb/1E6. Are these the units for the values reported in table 

2-7? What was the action taken as a result of the outlier detection? 

6. Page 2-4. Section 2.5. 
10 of 10 

Reason to replace kiln 1233 with kiln 1594 should be stated. 

7. Page 2-5. Section 2.5. 

Table 2-9, page 2-27 shows an overall mean of 18.5 not 18.4 as reported by PCA in first 

line of page 2-5. Are the numbers reported in page 2-9 the result of the corrections 

described in page 2-2? The kiln numbers reported in table 2-9 do not match any of the 

kiln numbers mentioned in page 2-2 as having data problems. Need to check if the same 

kilns selected by PCA as the 12% are the same used in EPA analysis. 

In fig 2-31 (page 2-64), are these the Hg mean emissions? The position of the median 

(horizontal lines) in the boxplots suggest skew distributions. What was the action taken 

by PCA regarding the outliers? 

The larger variance suggest the presence of larger numbers in the dataset used by PCA. 

Fig 2-32, page 2-65, shows a bi-modal distribution, probably the combination of two 

skewed distributions -- look at the long tail at the right. What lognormal, normal, 

exponential and weibull distribution were fitted to the data? How were the parameters of 

the distributions estimated? 

There is a tendency to not specify the units and clarify the response in the PCA report. It 

is not clear sometimes if they are referring to the daily emissions or the mean emissions. 

Figures (2-57 to 2-60) show the qq-plots for the Hg daily emissions. There are some 

outliers, but in the absence of formal testing for the lognormal and weibull distributions, 

there is no support for the graphical displays. 

Table 2-10 (page 2-28) provides different statistic to evaluate the normality assumption. 

The Shapiro-Wilks, which is the most robust of all of them, shows a small p-value, 

suggesting that the distribution of Hg daily emissions is not normal distributed. 

8. Page 2-5. Section 2.6. 

The value reported in table 2-1 (page 2-11) as the pooled mean is 18.4. PCA can’t 

conclude without a test that the two values: 8.59 and 7.1 are “virtually identical”. A 

statistical test will provide evidence in favor or against such assumption. 

The two variance components in comment #5 of Section 1 were mistakenly referred as 

the between-source (VB) and within-source (VW) variance components. These terms are 

the variance of the new runs and the variance of the previous data: Var X Var X ( ) m p + ( ). 

The between group (between kilns) variability is expected to be larger than the within 

group (within kiln) variability since you expect to find more variation between kilns than 

internal variations in a single kiln. As was explained in comment #5, the terms involved 

in the construction of the UPL do not involve an estimate of the between group 

variability, so the terms in the aforementioned equation are estimated using reliable 

estimates. 

The pooled variance works under the assumption that the variances are the same across 

kilns. This assumption will not be appropriate if the different sources operate very 

differently, which would result in a distinctly different variance for each source. 
11 of 11 

The difference in standard deviations seems to be due to some change in the data. Do we 

know if PCA is using the same top 12% kilns? 

PCA is confused regarding the interpretation of the Upper Tolerance Limit. This limit 

does not reflect the 99th percentile. As mentioned before, the tolerance limits constitute 

an interval that contains 99% of the data with some level of confidence. In general the 

tolerance limit will be greater than the 99th percentile. 

Table 2-12 was not found in the document to verify item a. in page 2-6. 

Results in item b mentioned in page 2-6 were found in table 2-45, page 2-33. As 

mentioned before it is not surprised that the UTL will be larger than the UPL. 

Comment #6 in page 2-6 shows that PCA has confused the 99th confidence UPL with the 

99th percentile. 

The suggestion of using non-parameteric methods is a valid one given the non-normality 

of the data. 

9. Page 2-7. Section 2.7 

The 30 day rolling averages are common in time series. They are used to smooth shortterm fluctuations and emphasize long-term fluctuations. I think that what they have is a 

sample of 30 days period, assuming that the selected 30 days is a representative sample of 

the whole year. The issue is to select a representative sample and unless someone can 

clearly identify most of the factors that will affect the emissions in a given year, it is 

reasonable to assume that in a given month we can observe all the possible values that 

one can observe in a given year. 

If months are different, then one can cluster months in low concentration and high 

concentration months, then one can select some days from the low concentration and 

other days from the high concentration months. But if no differences are assumed 

between months, then selecting at random a month (30 days) would have provided a 

representative sample. 

Since the analysis combines the days in a kiln (using the mean), then there is no need to 

worry about the independence of the days, because the kilns are assumed independent. 

With respect to the homocedasticity assumption the equation in comment #5 in this 

document is based on the assumption of different variances. 

10. Page 2-8. Section 2-7. 

The relative standard deviation (RSD), calculated as S

x

, is a measure of precision of the 

estimate (the mean in this case). Similar RSD will imply that the precision of two 

estimates is approximately the same; that doesn’t allow any conclusion regarding 

homocedasticity as PCA mistakenly concluded. It is not clear how the “mean RSD” was 

calculated when five quarries were used. 
12 of 12 

To fully incorporate different sources of variability one would have to consider 

simulation methods such as bootstrapping or jackknife, and assume distributions for the 

quarry uncertainty, kilns uncertainty, runs uncertainty and so on. 

In section 2.7.4 (page 2-8) PCA explores the relationship between Hg concentration and 

Hg in limestone. The value of (coefficient of determination) is interpreted as the total 

variability in the response (Hg concentration) explained by Hg in limestone. The squared 

root of , r , is the coefficient of correlation, and is interpreted as linear relationship 

between Hg in limestone and Hg concentration. In the simple linear models in Fig 2-61 

(page 2-94) and Fig 2-62 (page 2-95) the is 0.7134 and 0.9687, respectively. However, 

these two simple linear models are affected by several high influence data points. 

Removal of these high influence data points might change the resulting models. 

2

r

2

r

2

r

On the other hand, the usefulness of the regression model depends of the width of the 

confidence prediction interval and the needs for precision (Neter, 1990). 

We observed that most of the points are outside the confidence bands surrounding the 

prediction models, suggesting poor prediction capabilities. However, a formal assessment 

looking at residuals, high leverage points, high influence points, etc. are needed to 

evaluate the goodness of fit of the proposed models 

A possible explanation for the non relationship between Hg concentration and Hg in 

limestone when looking at the MACT floor data, is that limestone may be confounded 

with another variable (not included in the model) that could be more powerful explaining 

Hg concentration. 

In particular, Fig 2-62 can change dramatically if the two points with 0.7 or more Hg 

limestone are removed from the data. In this case, probably the slope of the linear 

relationship would have change. After removing the high influence points, the remaining 

data (the cluster of points in page 2-65) have Hg concentration mean values lower than 

0.1. 

The MACT floor data should not be representative of the population of kilns, the whole 

sample where the MACT floor data were selected is the one that need to be 

representative. The MACT floor data should represent the best kilns in terms of having 

lower Hg concentration values. 

11. Page 2-9. Section 2.7. 

With respect to comment 2.7.5, EPA can clarify how the data was obtained. To address 

long term variability, EPA should use data collected over a long period of time, and that 

have similar range as those observed in similar or longer time intervals. 

Figures 2-78 and 2-79 (pages 2-110 and 2-111) are incomplete and not clear enough for 

interpretation. 

12. Page 2-10. Section 2.8 

The representativity of the data needs to be justified by comparing the characteristics of 

the sample with the characteristics of the population of kilns. Concentrate in 
13 of 13 

demonstrating that the sample possesses the same distribution in the relevant 

characteristics of the population. To include all sources of variability into the analysis 

will require use of simulation approaches. Some sources of variability can be controlled 

by stratifying the population of kilns according to levels of those variables creating the 

variability that ones desires to control. 

Reference Section 2.

Hubert, Mia and Van der Veeke, Stephan. 2008. Outlier detection for skewed data. 

Journal of Chemometrics. 22(3): 235-246 

S. Burke. 1997. Scientific Data Management 1(1): 32–38 

T.J. Farrant. 1997. Practical statistics for the analytical scientist: A bench guide. Royal 

Society of Chemistry. 

Hayden, Robert W. 2005. A dataset that is 44% outliers. Journal of Statistics Education 

Volume 13, Number 1. www.amstat.org/publications/jse/v13n1/datasets.hayden.html 

Neter, J., Wasserman, W., and Kutner, M. H. (1990) Applied Linear Statistical Models, 

Third Edition, Irwin:IL 

Note for sections 3,4 and 5: 

Similar issues to discussed in Section 2 were found in sections 3, 4 and 5. 
Comments on Florida Department of Environmental Protection reply. 

Page 3. The use of Monte Carlo simulation or bootstrap simulation has been used to 

account for different types of uncertainties. In particular, in the presence of non-detect 

values, simulations have been used to substitute the non detect values for randomly 

selected plausible values derived from suitable distributions such as Weibull or lognormal. In the presence of non-detects the use of standard statistics techniques produces 

biased estimates (Zhao and Frey, 2006 and Helsel, 2005). So, the suggestion by Florida 

Department of Environmental Protection follows this recent tendency to use simulation 

techniques to account for uncertainties in the data. One of the uncertainties that 

researchers address is the presence of non-detects. The way to do that is to generate 1000 

or more replicates of the data, calculate your analysis for each data set (e.g. calculate the 

UPL), take the mean of the 1000 or so UPLs, and report the mean the UPLs and the 2.5 

and 97.5 percentiles of the UPLs as confidence limits. 

Zhao, Yuchao and Frey, H. Christopher. 2006. Uncertainty for Data with Non-Detects: 

Air Toxic Emissions from Combustion. Human and Ecological Risk Assessment, 12: 

1171–1191 

Helsel DR. 2005. Nondetects and data analysis: Statistics for censored environmental 

data. Wiley, Hoboken, NJ, USA 

Comments on HOLCIM (US) INC reply 

Page 4, second paragraph of section B. HOLCIM had misunderstood the definition of the 

Upper Prediction Limit (UPL). The UPL proposed by EPA has a 99% confidence limit 

and it is not a confidence limit for the 99 percentile. The use of some terms in EPA’s 

document has lead readers to think about a confidence interval for the 99th percentile 

instead of a 99% confidence UPL. 

The interpretation given by HOLCIM in the second paragraph mixes the concept of UPL 

and 99 percentile. Since the UPL are calculated for a set of “m” new tests, the 

interpretation is forced to link the “m” new tests together. So, one can conclude “we 

expect that 99% of the times, the m-new tests will be below the UPL, and only 1% of the 

times all the m-new tests will not fall below the UPL.” 

The interpretation given by HOLCIM in the second paragraph corresponds to a 

confidence interval for the 99th percentile, not to the UPL with a confidence of 99%. 

HOLCIM mistakenly believes that EPA wants to calculate the 99th percentile. 

Page 5, first paragraph. In the first paragraph, HOLCIM is commenting on the use of a 

confidence interval for the average of the 12 best performing MACT pool sources. The 

statement seems to be based in the “Central Limit Theorem” which states that the 

sampling distribution of the average is a normal distribution; in other words, when the 

process of calculating the mean of the best 12 performing MACT pool sources is repeated 

many times (40 or more) and we plot the calculated means (40 or more values), the 
distribution or histogram of these means shows a symmetric shape, in this case the mean 

and the median coincide. Under this assumption, one could say that 50% of the data will 

be below or above the mean. In general, this is not the correct statement for a confidence 

interval based on the mean or average. The comment (“even the MACT floor kilns could 

exceed the limit half the time or more”) is appropriate for a confidence interval of the 

median, not for the mean. 

Page 5, third line from the bottom in second paragraph. The statement refers to a 

confidence interval for the 99th percentile, not the UPL with a 99% confidence limit. 

 

Page 5. Third paragraph. HOLCIM establishes that they understand from EPA documents 

that EPA wants to estimate the 99th percentile or a confidence interval for the 99 

percentiles. HOLCIM recognizes that EPA has calculated the UPL with 99% confidence 

and not the 99th percentile, and insists that EPA wants to use the UPL to estimate the 99th

percentile. HOLCIM provides an appropriate definition of the UPL with 99% 

confidence. 

HOLCIM comment (line 7) that the UPL is a probability is not correct, since the UPL is 

an upper bound for future observations. 

The comment that “the MACT floor will exceed the value (the UPL value) ...” (line 3rd

from bottom of third paragraph) is not completely correct, since the UPL is calculated as 

the mean of the 12 percent plus minus the variance of the MACT floor sources and the 

estimated variance of the new sources. It is very unlikely that it won’t contain the MACT 

floor sources “a significant number of times”. 

Page 5. fourth paragraph. The comment that the UTL (upper tolerance limit) generates 

the 99th percentile is not correct. It is very likely that the UTL will be larger than the 99th

percentile and that the UPL will be smaller than the 99th percentile. 

Page 6. first paragraph. The comment that the UTL is more reliable because it contains 

the 99th percentiles is based on the mistaken assumption that EPA intends to estimate the 

99th percentile, not to estimate an upper bound for new tests (i.e. a UPL). 

Pages 9 and 10. HOLCIM calculated the UPL, UTL and 99th percentiles for THC MACT 

floors. I think there is a missunderstanding of the use of the UPL. The UPL is an upper 

bound for the mean of the new observations, 2

2

1

1 1 N

p i

i i

xt s = nN m

⎛ ⎞

+ ⎜

⎝ ⎠ ∑ + ⎟ . I don’t see the 

reason for HOLCIM calculating the UPL and UTL using the new data. They need to 

calculate the mean of the new data and compare it with the UPL. 

In table 2, HOLCIM is comparing the UPL calculated with the existing 30 day data, with 

the actual 99th percentile, which doesn’t make sense since the UPL was not calculated as 

an estimator of the 99th percentile. HOLCIM did not evaluate the validity of the UPL 

since they didn’t compare the mean of the new runs with the value of the calculated UPL 

using the existing data. 
Note that in Table 2, page 10, for the lowest five THC emitting sources, the kiln #5, 

Holcim Mason City Kiln 2, did not satisfy the UPL since 10.91 > 10.4 but it is borderline. 

In the remaining tables (Table 3, page 13; Table 4, page 15; table 6 page 19) show both 

the UPL and UTL; and in all these tables, the average values for each kiln are below the 

UPL as expected.