1 MEMORANDUM

TO: Docket ID No. EPA-HQ-OAR-2012-0522

FROM: Tina Ndoh Office of Air Quality Planning and Standards

DATE: June 9, 2015

SUBJECT: Approach for Applying the Upper Pr ediction Limit to Limited Datasets

I. What is Our Approach for Applying the Upper Prediction Limit to Limited Datasets?

The UPL approach addresses variability of emi ssions data from the best performing source or sources in setting MACT standards. The UPL al so accounts for uncertainty associated with emission values in a dataset, which can be in fluenced by components such as the number of samples available for developing MACT standard s and the number of samples that will be collected to assess compliance w ith the emission limit. The UPL approach has been used in many environmental science applications. 1,2,3,4,5,6 As explained in more detail in the memorandum entitled “Use of the Upper Predicti on Limit for Calculating MACT Floors”, which is located in the docket for this action, the EP A uses the UPL approach to reasonably estimate the emissions performance of the best performi ng source or sources to establish MACT floor
standards.

Sample size (the number of values in a particular dataset) is an important component of the UPL approach. First, the use of the UPL approach requ ires us to identify the distribution of the data, that is, whether a particular dataset is normally or non- normally distributed (see the memorandum entitled “Use of the Upper Predicti on Limit for Calculating MACT Floors”, which is located in the docket for this action). To de termine the distribution of the data, we use wellestablished tests (kurtosis and skewness tests), and th ese tests require at leas t three data points. In prior rulemakings, we used the kur tosis equation that is a built-in function in Excel software, but that equation requires at least 4 values in order to avoid dividi ng by zero, which precluded its use for datasets of 3. However, we recently further reviewed the a pplication of the UPL where data

1 Gibbons, R. D. (1987), Statistical Prediction Intervals for the Evaluation of Ground-Water Quality . Groundwater, 25: 455–465 and Hart, Barbara F. and Janet Chaseling, Optimizing Landfill Ground Water Analytes —New South Wales, Australia, Groundwater Monitoring & Remediation, 2003, 23, 2.

2 Wan, Can; Xu, Zhao; Pinson, Pierre; Dong, Zhao Yang; Wong, Kit Po. Optimal Prediction Intervals of Wind Power Generation. 2014. IEEE Transactions on Power Systems, ISSN 0885-8950, 29(3): pp. 1166 – 1174.

3 Khosravi, Abbas; Mazloumi, Ehsan; Nahavandi, Saeid; Creighton, Doug; van Lint, J. W. C. Prediction Intervals to
Account for Uncertainties in Travel Time Prediction. 2011. IEEE Transactions on Intelligent Transportation Systems, ISSN 1524-9050, 12(2):537 – 547.

4 Ashkan Zarnani; Petr Musilek; Jana Heckenbergerova. 2014. Clustering numerical weather forecasts to obtain statistical prediction intervals. Meteorological Applications, ISSN 1350-4827. 21(3): 605.

5 Rayer, Stefan; Smith, Stanley K; Tayman, Jeff. 2009. Empirical Prediction Intervals for County Population Forecasts. Population Research and Policy Review, 28 (6): 773 – 793.

6 Nicholas A Som; Nicolas P Zegre; Lisa M Ganio; Arne E Skaugset. 2012. Corrected prediction intervals for change detection in paired watershed studies. Hydrological Sciences Journal, ISSN 0262-6667, 57(1): 134 – 1432 are limited, and concluded that it is appr opriate to use another kurtosis estimator 7 that provides a meaningful result with just 3 values. The use of this estimator is important, as many, if not most,
of our new source emissions limits are based on 3 samples collected during an emissions test.

Once we determine the data distribution, the a ppropriate equation to be used in the UPL approach is selected based on the data distribution. We rec ognize that the use of the UPL approach for limited datasets introduces some am ount of uncertainty in the calculation of MACT standards, and, therefore, we are taking additional steps, discussed in this document, to ensure
that the level of the MACT standards is reasonabl e. We also note that after MACT standards are promulgated, we are required to review those st andards periodically, and for such reviews, we typically have significant additional HAP emi ssions data from the intervening years of compliance with which to further assess the actual performance of the various emission sources.

Regardless of the distribution of the data, UPL e quations have three well-defined components: an average, the t-score and a measure of variability that includes the actual variability of the data, the sample size, and the number of data point s that are averaged together to determine compliance with a particular emission limit. A t- score is a value that estimates the uncertainty and variability for a certain confidence level asso ciated with a specific number of data points. For a constant confidence level, t-scores decreas e as the number of samples increase. This means that, if the mean and variance remain constant , UPL values for a particular confidence level decrease as the number of samples increase. Figu re 1 and Table 1 show the t-scores for various sample sizes, and demonstrate that the t-score is highly variable at the smallest sample sizes and becomes relatively constant once the sample size is larger than a few data points. Consequently, we recognize that we need to take special care when smaller, limited sample sizes are used to establish emission limits, as the t-scores can ha ve a disproportionate effect, overwhelming the other components of the emission limit calculati on. In addition, we recognize that for a sample size of fewer than three data poi nts, which has a very large t-sc ore and precludes the appropriate
selection of a distributio n, we should not develop emission limits using the UPL. In other words, if fewer than 3 data points are available for use in determining an emission limit for a particular source, and no other data from sources in the subcategory are available, we would have to establish a different procedure for establishing the MACT floor that does not rely on the UPL.

7 Doric D, Nikolic-Doric E, Jevremovic V and Malisic J. 2009. On measuring skewness and kurtosis. Qual Quant.
43:481-493. DOI 10.1007/s11135-007-9128-9.3 Figure 1. Sample Sizes Versus One-sided, 99 Percent Confidence Interval T-Scores for
Normal Distributions

Table 1. Degrees of Freedom, T-Score for a On e-Tailed, 99 Percent Confidence Interval,
and Percent Difference in T-Sc ore for Each Sample Size

Sample Size (n) Degrees of Freedom (n-1) T-score (99 percent) Relative Percent
Difference a

2 1 31.82

3 2 6.96 78

4 3 4.54 35

5 4 3.75 17

6 5 3.36 10

7 6 3.14 7

8 7 3.00 5

9 8 2.90 3

a The relative percent difference be tween the t-score and the t-sc ore shown on the preceding row. 

### Number of SamplesT-score 

## T-score vs Number of Samples

2 3 4 5 6 7 8 9 0 5 10 15 20 25 30 35

78% reduction from 2 to 3 samples

35% reduction from 3 to 4 samples

17% reduction from 4 to 5 samples

10% reduction from 5 to 6 samples

7% reduction from 6 to 7 samples

5% reduction from 7 to 8 samples4 In the case of a dataset with a normal distri bution, the equation to cal culate the UPL is as
follows:

ܷܲܮ ൌ ݔ ̅ ൅ݐ ሾ ሺ ௡ିଵ ሻ

, ሺ ଵି஑ ሻ ሿ ඨ

ݏ ଶ ൬ 1

݊ ൅ 1

݉ ൰

Where ݔ = average or mean of test run data;

ݐ ሾ ሺ ௡ିଵ ሻ

, ሺ ଵି஑ ሻ ሿ = t score, the one-tailed t value of the Student’s t distribution for a specific
degree of freedom (n-1) and confidence level (1- α );

α = level of significance expr essed as a decimal (e.g., 1% significance = 0.01), note that

confidence level = 100 െ ሺ ߙ ൈ 100 ሻ ;

ݏ ଶ = variance of the dataset (test run);

݊ = number of values (test runs) in the dataset; and

݉ = number of values used to calcula te the test average (generally 3).

As shown in the equation, the sample size (n) dir ectly affects the UPL calculations in two ways. First, as mentioned earlier, the t-score is sele cted based on the sample size. Second, the sample size is a separate variable in th e equation, and as with the t-score, as the sample size gets larger (all other variables bein g held constant), the UPL gets smalle r (i.e., approaches the mean value). The sample size also influences both the mean ( ݔ ̅ (and variance (s 2 ) in that larger sample sizes provide better estimates of the population m ean and variance. The confidence level also influences the t-score, and is represente d in the UPL equation by the subscript “1- α .” As noted in the memorandum entitled “Use of the Upper Prediction Limit for Calculating MACT Floors”, which is located in the docket for this action, we typically choose the 99 percent UPL, which is another way of saying that α equals 0.01.

Note that the UPL equation for a lognormal dist ribution uses a z-score, instead of a t-score. 8 As with the normal UPL equation, the lognormal UPL e quation is affected by sample size and the variance of the data, and the basic concepts ou tlined in this discussi on apply to datasets regardless of th e distribution.

In summary, sample size is important for UPL calculations, as it is a key component of the tscore value and the calculation of the mean and variance of the sample. For the reasons presented above, for datasets below a certain size, further evaluation is warrant ed to ensure th at the results are a valid representation of the performance of the emission sources whose data are included in the dataset. The remainder of this document provides our assessment of what constitutes a limited dataset and what additional analyses are appropriate for ensuring that the UPL

8 Bhaumik, D.K. and Gibbons, R.D. “An Upper Prediction Limit for the Arithmetic Mean of a Lognormal Random Variable”, Technometrics, 46, 2004, 239-248.5 calculations are valid representations of the perf ormance of units as refl ected in the available
data.

II. What is a limited dataset?

While there are different options for choosing th e number of samples th at represents a limited dataset (or small sample size), we can use distinctions based on our observations in Figure 1 and Table 1 as an aid in defining what constitutes a limited dataset. As discussed earlier, the t-score is an influential parameter that can have a particularly large influence on an emission limit developed from a limited dataset. As such, the t- score is a key parameter for identifying what constitutes a limited dataset for purposes of MACT analyses. As shown in Figure 1 and Table 1, above, at the 99 percent confidence level there is a clea r distinction between t-score values for sample sizes of 3 and fewer when compared to t-sc ore values for sample sizes of more than 6. As Figure 1 and Table 1 show, the t-score changes dras tically from sample size equal to 2 to sample size equal to 3. The changes in the t-score are considerably less dramatic as the sample size approaches 9 data points and larger.

As noted earlier, we will not use the UPL approach for datasets with fewer than 3 values because of the high degree of uncertainty th at results from the size of the dataset and the related t-score. In fact, the t-score (see Figure 1 a nd table 1) for a dataset of 3 is 78 percent lower than a t-score for a dataset of 2, meaning that given an equal variance, the component of the UPL that includes
the t-score and variance would be more than 78 percent less when eval uating a 3-point dataset compared to a 2-point dataset. The table and figur e also show that the re lative change of the tscore for each additional data poi nt decreases as the sample size gets larger. As sample size increases, uncertainty decreases, but there is always some amount of un certainty. The discussion that follows presents our examination of the t-sc ore at various sample si zes. We rely on this examination to help identify a sample size below which we would further evaluate the data and the application of the UPL to ensu re that the amount of uncertainty is within a re asonable range. Based on Table 1, the lower bound of the sample size where such further eval uation is needed is 3 values, and a reasonable value (where the size of the t-statistic has much less influence on the eventual emission limit) lies between 4 and 9 values.

An additional factor to consider is that, in the context of MACT analyses, emission tests typically include 3 test runs (i ndependent data points), and, ther efore, our MACT floor dataset size typically is a multiple of 3. For this reas on, we first considered sample sizes that are multiples of 3 when identifying the size below which further scrutiny is needed. As can be calculated from the t-scores shown in Table 1, the relative difference between the t-score for 6 data points and the t-score for 9 data points is about 14 percent ((3.36 -2.90)/3.36). The relative
difference between the t-scores for 7 and 9 da ta points is about 8 percent ((3.14-2.90)/3.14).
Given that the test methods that are used to generate the data typically are accurate to within 10 to 20 percent 9 , this difference in t-score (between data sets with 7 and 9 runs) is less than the uncertainty in the test methods and would not have a large influence on the magnitude of an emission limit. Conversely, comparing the t-score for 3 runs to that for 7 runs, the relative difference in the t-score is 55 percent ((6.96-3.14 )/6.96), indicating that th ere is a possibility,

9 (ReMAP): PHASE 1, Preci sion of Manual Stack Emission Measuremen ts; American Society of Mechanical Engineers, Research Committ ee on Industrial and Municipal Waste, February 2001.6 especially if the variance is hi gh, that the t-score for 3 test ru ns may introduce a large amount of uncertainty in an emission limit, and, therefore, ad ditional analysis of a dataset of this size is
warranted. We also conducted the same type of evaluation on da tasets between 4 and 6 points, and while the change in t-score fo r datasets of these sizes could be considered to be sufficiently small, to be conservative, we conclude that a lim ited dataset, for purposes of MACT analyses, is
a dataset that includes at least 3 test runs and fewer than 7 test runs. Because of the uncertainty
associated with datasets at or below this size , further evaluation is warranted, and the types of additional analyses that we may choos e to conduct are explained below.

III. What approach will the EPA follow fo r MACT floors based on limited datasets?

When a MACT floor for either existing or new s ources is based on fewer than 7 data points, we
will further evaluate each individual dataset in order to ensure that the uncertainty associated
with a limited dataset does not cause the calculat ed emission limit to be so high that it does not reflect the average performance of the units upon which the limit is based after accounting for variability in the emissions of those units. Th e evaluation will include one or more of the following, depending on the specific dataset: confirming that the da ta distribution was selected correctly; after confirming the da ta distribution, ensuring that we use the most appropriate UPL equation 10 ; and, as necessary, comparing UPL equation components for the individual unit upon
which a new source floor is based with those of th e units in the existing source floor to determine
if our identification of the be st unit is reasonable. Each of the additional evaluations are
discussed below.

Confirming the data distribution is important because UPL equations and the emission limits derived from those UPL equations depend on the distribution. In prior rulemakings, we had not identified a way of ascertaining the data distribu tion for units with three samples, so we either assumed a normal or lognormal distribution. This particularly affected new source emissions limits, many of which were based on three samp les. However, we have since identified and adopted an established technique th at compares the ratio of skewne ss to the standard error of the skewness for both the raw and log transformed data . The lower of the two ratios identifies the data distribution that best repres ents the sample set. This ability to more precisely ascertain the
data distribution enables us to select the best option for the UP L equation given the small sample
size.

After confirming that we selected the best distribution based on the available data and ensuring that we used the correct equation to calculate a MACT floor value, we then examine key
variables that factor into the calculated floor values. The variables that are factored into the emission limit include the following: average, t-scor e, confidence level, variance, number of test runs (i.e., sample size), and number of runs in each test (i.e., the number of data points that are averaged together to determine compliance with a particular emi ssion limit). The considerations
related to each variable are discussed below.

10 For datasets with lognormal distribu tions, we recently determined that the most appropriate UPL equation, which is especially important for the smallest datasets that we use, is based on an approach described in Bhaumik, 2004, op. cit.7 The average, which typically is calculated as the mean value of a dataset, does not require additional consideration. Once it is calculated, it is used to rank the performance of units, and also is the key value upon which each emission li mit is based. As such, average values have already been compared within a particular data set, and additional assess ment is not necessary. However, in the overall assessment of variable s, when multiple best performing units have emission averages that are similar, we may look to other variables like th e variance to help to
inform our decision as to which un it is the single best performer.

As discussed earlier, the t-score is dependent on the sample size, and, therefore, we discuss these two variables together. As shown earlier in Figure 1, as the sa mple size decreases, the t-statistic increases. This is important because the t-statisti c is multiplied by the variance and a factor that involves the sample size and the number of test ru ns used for compliance (i.e., the “m” term in the UPL equation). For the smallest datasets (3 data points) that we consider under the UPL approach, a large variance coupled w ith a t-score that is significan tly greater than the t-score for
a dataset of 7 or more can create a situation where the emission limit in cludes a large amount of uncertainty. In such cases, we use our technical e xpertise to assess what level of emissions could realistically be expected from th e type of unit, controls, and ot her relevant factors, and would
ensure that the emission limit is reasonable.

The variance is another key variable that we would evaluate. Generally speaking, if our evaluation showed a very small amount of vari ability (e.g., a small vari ance), the emission limit would not contain an unacceptable amount of un certainty and variability because a small
variance would result in an emissi on limit that would be reasonably close to the mean of the data. With larger datasets, it is more likely that the de monstrated variance is trul y representative of the variation in the source’s emissions simply becau se there is more evidence of the source’s emissions over time. On the other hand, when a limited dataset includes a large amount of variability (e.g., a large va riance), we would need to carefully evaluate the data. For instance, consider the case where the pool of best performi ng units all have similar averages and the unit identified as the single best performer (based on average emissions only) has a limited dataset and a variance that greatly exceeds the variance of the other similarly-controlled best-performing units. In such a case, the MACT floor analysis ma y yield an emission limit for that unit (i.e., the new source MACT floor) that is higher than the existing source MACT floor, which is an indicator that further analysis is warranted. Care ful consideration of variance will allow us to
better ascertain which unit should be considered the best performer.

Another tool that could be used to adjust unrea sonable emission limits is the selection of the confidence level. In certain instances, we may determine that a limited dataset includes such a high degree of variability that th e 99 percent confidence level results in an emission limit that our experience suggests is higher than the average emissions limitation achieved by the best performing source or sources afte r accounting for variability. In such cases, we may choose to select a confidence level of 95 percent or 90 percen t. As an example, consider a unit identified as the best performing new source with a specifi c process or control device that has been demonstrated to operate far more efficiently (thus having the pot ential to lower emissions) in similar units, processes, or control devices. Under these c onditions, we may choose to acknowledge the better operation by lowering the confidence level (which lowers the emissions limit).8 Establishing and using this approach on a case- by-case basis for limited datasets will ensure consistent application of emissions limit development procedures, which will mitigate the additional uncertainty that coul d otherwise result from calculating MACT floor standards with
limited data. 11

IV. How did we apply the approach for limited data sets to limited datasets in the phosphoric acid manufacturing source category?

For the phosphoric acid manufacturing source cat egory, we have a limited dataset for the pollutant total F from phosphate rock calciners. Th erefore, we evaluated this specific dataset to determine whether it is appropriate to make any modifications to the approach used to calculate
the MACT floor.

For the phosphate rock calciner dataset, we performed the steps outlined above, including: ensuring that we selected the data distribution that best represents the dataset; ensuring that the correct equation for the distribut ion was then applied to the data; and comparing individual components of the limited dataset to determine if the total F standard based on the limited dataset reasonably represents the performa nce of the units included in the dataset. The results of this analysis are presented below.

The MACT floor dataset for total F from new an d existing phosphate rock calciners includes 3 test runs from 1 phosphate rock cal ciner. After determining that th e dataset is best represented by a normal distribution and ensuring that we used the correct equation fo r the distribution, we
considered the selection of a lower confidence level for determining the emission limit by evaluating whether the cal culated limit reasonabl y represents the performance of the unit upon which it is based. In this case, the calculated emission limit is about twice the shor t-term average emissions from the best performing source, indi cating that the emission limit is not unreasonable compared to the actual performance of the unit upon which the limit is based and is within the range that we see when we evaluate larger datasets using our MACT floor calculation procedures. Therefore, we determined that no changes to our sta ndard floor calculation procedure are warranted for this pollutant and su bcategory, and we are setting the MACT floor at
9.0E-04 lb/ton of rock feed for total F from new and existing phosphate rock calciners.

11 We note that in the recent D.C. Circuit Court of App eals decision in National Assn. of Clean Water Agencies v. EPA (NACWA), which involved challenges to EPA’s MACT standards for sewage sludge incinerators, the court identified two instances where we repl aced a new source floor value with the existing source value because the new source value was less stringent than the existing source va lue (the “anomalous result”). See 734 F.3d 1115. The procedures outlined here would eliminate the anomalous result in both instances in the rule at issue in NACWA. The
first instance in which the court in the NACWA decision id entified the anomalous result is where we established an emission limit using the UPL on a dataset that included only 2 data points. The other instance in which the court
identified the anomalous result is where we did not apply the most appropriate lognormal UPL equation.