1 MEMORANDUM

SUBJECT: Approach for Applying the Upper Prediction Limit to Limited Datasets

DATE: September 17 , 2014

FROM: Jeff Telander Office of Air Quality Planning and Standards

TO: Docket ID No. EPA

- HQ

- OAR

- 2013

- 0291

What is Our Approach for Appl ying the Upper Prediction Limit to Limited Dataset s ?

The UPL approach addresses variability of emissions data from the best performing source or sources in setting MACT standards. T he UPL also accounts for uncertainty associated with emission values in a dataset

, which can be influenced by components such as the number of samples available for developing MACT standards and the number of samples that will be collected to assess compliance with the emission limit. The UPL approach has been used in
many environmental science applications. 1

, 2

, 3

, 4

, 5

, 6 As explained in more detail in memorandum entitled “Use of the Upper Prediction Limit for Cal culating MACT Floors”, which is located in the docket for this action, the EPA uses the UPL approach to reasonably estimate the emissions performance of the best performing source or sources to establish MACT floor standards.

Sample size (the number of values in a particular dataset ) is an important component of the UPL approach

. First, the use of the UPL approach requires us to identify the distribution of the data

, that is, whether a particular dataset is normally or non

- normally distributed ( see the

m emorandum entitled “Use of the Upper Prediction Limit for Calculating MACT Floors”, which

1 Gibbons, R. D. (1987), Statistical Prediction Intervals for the Evaluation of Ground

- Water Quality . Groundwater, 25: 455

– 465 and Hart, Barbara F. and Janet Chaseling, Optimizing Landfill Ground Water Analytes

— New South Wales, Australia, Groundwater Monitoring & Remediation, 2003, 23, 2.
2 Wan, Can; Xu, Zhao; Pinson, Pierre; Dong, Zhao Yang; Wong, Kit Po. Optimal Prediction Intervals of Wind Power Generation. 2014. IEEE Transactions on Power Systems, ISSN 0885

- 8950, 29(3): pp. 1166

– 1174.
3 Khosravi, Abbas; Mazloumi, Ehsan; Nahavandi, Saeid; Creighton, Doug; van Lint, J. W. C. Prediction Intervals to Account for Uncertainties in Travel Time Prediction. 2011. IEEE Transactions on Intelligent Transportation Systems, ISSN 1524

- 9050, 12(2):537

– 547.
4 Ashkan Zarnani; Petr Musilek; Jana Heckenbergerova. 2014. Clustering numerical weather forecasts to obtain statistical prediction intervals. Meteorological Applications, ISSN 1350

- 4827. 21(3): 605.
5 Rayer, Stefan; Smith, Stanley K; Tayman, Jeff. 2009. Empirical Prediction Intervals for County Population Forecasts. Population Research and Policy Review, 28(6): 773

– 793.
6 Nicholas A Som; Nicolas P Zegre; Lisa M Ganio; Arne E Skaugset. 2012. Corrected prediction intervals for change detection in p aired watershed studies. Hydrological Sciences Journal, ISSN 0262

- 6667, 57(1): 134

– 1432 is located in the docket for this action ). To determine the distribution of the data , we use well

established tests ( kurtosis and skewness tests ) , and the se tests require a t least three data points

. In prior rulemakings , we used the kurtosis equation that is a built

- in function in Excel software , but that equation requires at least 4 values in order to avoid dividing by zero , which precluded its use for datas ets of 3

. However, we recently further reviewed the application of the UPL where data

are limited, and concluded that it is appropriate to use another kurtosis estimator 7 th at provides a meaningful result with just 3 values. The use of this estimator is important, as many, if not most, of our new source emissions limits are based on 3 samples collected during an emissions test

.

Once we determine t he data distribution

, the appropriate equation to be used in the UPL approach is selected based on the dat a distribution

. We recognize that the use of the UPL approach for limited dataset s introduces some amount of uncertainty in the calculation of MACT standards, and, therefore, we are taking additional steps , discussed in this document, to ensure that the le v el of the MACT standards is reasonable. We also note that after MACT standards are promulgated, we are required to review those standards periodically, and for such reviews, we typically have significant additional HAP emissions data from the intervening years of compliance with which to further assess the actual performance of the various emission sources.

Regardless of the distribution of the data, UPL equations ha ve three well

- defined components: an average, the t

- score and a measu re of variability that includes the actual variability of the data

, the sample size

, and the number of data points that are averaged together to determine compliance with a particular emission limit

. A t

- score is a value that estimates the uncertainty and variability for a certain confidence level associated with a specific number of data points

. For a constant confidence level, t

- scores decrease as the number of samples increase. This means that , if the mean and variance remain constan t, UPL values for a particular confidence level decrease as the number of samples increase. Figure 1 and Table 1 show the t

- scores for various sample sizes , and demonstrate that the t

- score is highly variable at the smallest sample sizes and becomes relati vely constant once the sample size is larger than a few data points

. Consequently,

w e recognize that we need to take special care when small er, limited sample sizes are used to establish emission limits, as the t

- scores can have a dispropor tionate effect, overwhelming the other components of the emission limit calculation. In addition, we recognize that for a sample size of fewer than three data points , which has a very large t

- score and precludes the appropriate selection of a distribution, we should not d evelop emission limits using the UPL. In other words, if fewer than 3 data points are available for use in determining an emission limit for a particular
source, and no other data from sources in the subcategory are available, we would have to establish a different procedure for establishing the MACT floor that does not rely on the UPL.

7 Doric D, Nikolic

- Doric E, Jevremovic V and Malisic J. 2009. On measuring skewness and kurtosis. Qual Quant. 43:481

- 493. DOI 10.1007/s11135

- 007

- 9128

- 9.3 Figure 1. Sample sizes versus one

- sided, 99 percent confidence interval t

- scores for normal distributions

Table 1. Degrees of Freedom, T

- Score for a One

- Tailed, 99 Percent Confidence Interval, and Percent Difference in T

- Score for Each Sample Size

Sample Size (n) Degrees of Freedom (n

1) t

- score (99 percent) Relative Percent Difference a 2 1 31.82 3 2 6.96 78 4 3 4.54 35 5 4 3.75 17 6 5 3.36 10 7 6 3.14 7 8 7 3.00 5 9 8 2.90 3

a The relative percent difference between the t

- score and the t

- score shown on the preceding row 

### Number of SamplesT-score 

## T-score vs Number of Samples

2 3 4 5 6 7 8 9 0 5 10 15 20 25 30 35
78% reduction from 2 to 3 samples

35% reduction from 3 to 4 samples

17% reduction from 4 to 5 samples

10% reduction from 5 to 6 samples

7% reduction from 6 to 7 samples

5% reduction from 7 to 8 samples4 In the case of a dataset with a normal distribution, the equation to calculate the UPL is as follows:

UUU = x ̅ + t [ ( n
− 1 )

, ( 1

− α ) ] �

s 2 � 1

n + 1

m �

where x = average or mean of test run data;

t [ ( n
− 1 )

, ( 1

− α ) ] = t score, the one

- tailed t value of the Student’s t distribution for a specific degree of freedom (n

- 1) and confidence level (1

- α);

α = level of significance expressed as a decimal (e.g., 1% significance = 0.01), note that confidence level = 100

− ( α × 100 ) ;

s 2 = variance of the dataset (test run);

n = number of values (test runs) in the dataset; and

m = number of values used to calcul ate the test average (generally 3).

As shown in the equation, the sample size (n) directly affects the UPL calculations in two ways. First, as mentioned earlier, the t

- score is selected based on the sample size. Second, the sample size is a separate vari able in the equation, and as with the t

- score, as the sample size gets larger (all other variables being held constant), the UPL gets smaller (i.e., approaches the mean value). The sample size also influences both the mean ( x ̅ ) and variance (s 2 ) in that la rger sample sizes provide better estimates of the population mean and variance. The confidence level also influences the t

- score, and is represented in the UPL equation by the subscript “1

- α .” As noted in the memorandum entitled “Use of the Upper Predicti on Limit for Calculating MACT Floors”, which is located in the docket for this action, we typically choose the 99 percent UPL, which is another way of saying that α equals 0.01.

Note that the UPL equation for a lognormal distribution uses a z

- score, inste ad of a t

- score. 8 As with the normal UPL equation, the lognormal UPL equation is affected by sample size and the variance of the data, and the basic concepts outlined in this discussion apply to datasets regardless of the distribution.

In summary, sample size is important for UPL calculations, as it is a key component of the t

score value and the calculation of the mean and variance of the sample. For the reasons presented above, for datasets below a certain size, further evaluation is w arranted to ensure that the results are a valid representation of the performance of the emission sources whose data are included in the dataset. The remainder of this document provides our assessment of what constitutes a

8 Bhaumik, D.K . and Gibbons, R.D.``An Upper Prediction Limit for the Arithmetic Mean of a Lognormal Random Variable", Technometrics, 46, 2004, 239

- 248.5 limited dataset and what addition al analyses are appropriate for ensuring that the UPL calculations are valid representations of the performance of units as reflected in the available data.

What is a limited dataset ?

While there are different options for choosing the number of samples that represents a limited dataset (or small sample size ) , we can use distinctions based on our observations in Figure 1 and Table 1 as an aid in defining what constitutes a limited dataset

. As discussed earlier, the t

- score is an influential parameter that can have a particularly large influence on an emission limit developed from a limited dataset . As such, the t

- score is a key parameter for identify ing what constitutes a limited dataset fo r purposes of MACT analyses. As shown in Figure 1 and Table 1, above

, at the 99 percent confidence level there is a clear distinction between t

- score values for sample sizes of 3 and fewer when compared to t

- score values for sample sizes of more than 6. As Figure 1 and Table 1 show, the t

- score changes drastically from sample size equal to 2 to sample size equal to 3 . The changes in the t

- score are considerably less dramatic as the sample size approaches 9 data points and larger.

As noted earlier, we will not use the UPL approach for dataset s with fewer than 3 values because of the high degree of uncertainty that results from the size of the dataset and the related t

- score. In fact, t he t

- score ( see F igure 1 and table 1) f or a dataset of 3 is 78 percent lower than a t

- score for a dataset of 2 , meaning that given an equal variance, the component of the UPL that includes the t

- score and variance would be more than 78 percent less when evaluating a 3

- point dataset compared to a 2

- point dataset

. The table and figure also show that the relative change of the t

score for each additional data point decreases as the sample size gets larger. As sample size increases, uncertainty decreases, but there is always some amount of uncertainty. The discussion that follows presents our examination of the t

- score at various sample sizes

. W e rely on t his examination to help identif y a sample size below which we would further evaluate the data and the application of the UPL to ensure that the amount of uncertainty is within a reasonable range

. Based on Table 1, the lower bound of the sample size where such further evaluation is needed is 3 values , and a reasonable value (where the size of the t

- statistic has much less influe nce on the eventual emission limit) lies between 4 and 9 values

.
An additional factor to consider is that, i n the context of MACT analyses, emission tests typically include 3 test runs (independent data points) , and, therefore, our MACT floor dataset siz e typically is a multiple of 3. For this reason, we first considered sample sizes that are multiples of 3 when identifying the size below which further scrutiny is needed. As can be calculated from the t

- scores shown in T able 1, the relative difference between the t

- score for 6 data point s and the t

- score for 9 data points is about 1 4 percent ((3. 36

- 2.90)/3. 36 )

. The relative difference between the t

- scores for 7 and 9 data points is about 8 percent ((3.14

- 2.90)/3.14). Give n that the test methods that are used to generate the data typically are accurate to within 10 to 20 percent 9 , this difference in t

- score (between dataset s with 7 and 9 runs) is less than the uncertainty in the test methods and would not have a large influence on the magnitude of an emission limit. Conversely, comparing the t

- score for 3 runs to that for 7 runs, the relative

9 (ReMAP): PHASE 1, Precision of Manual Stack Emission Measurements; American Society of Mechanical Engineers, Researc h Committee on Industrial and Municipal Waste, February 2001.6 difference in the t

- score is 5 5 percent ((6.96

- 3. 14 )/6.96) , indicating that there is a possibility, especially if the variance is high, that the t

- score for 3 test runs may introduce a large amount of uncertainty in an emission limit, and

, therefore

, additional analysis of a dataset of this size is warranted. We also conducted the same typ e of evaluation on dataset s between 4 and 6 points, and while the change in t

- score for dataset s of these sizes could be considered to be sufficiently small, to be conservative, we conclude that a limited dataset , for purposes of MACT analyses, is

a dataset that includes at least 3 test runs and fewer than 7 test runs

. Because of the uncertainty associated with dataset s at or below this size, further evaluation is warranted , and the types of additional analyses that we may choose to conduct are explained below.

What approach will the EPA follow for M ACT floors based on limited dataset s ?

When a MACT floor for either existing or new source s is based on fewer than 7 data points , we will further evaluate each individual dataset in order to ensure that the uncertainty associated with a limited dataset does not cause the calculated emission limit to be so high that it does not reflect the average performance of the units upon which the limit is based after accounting for variabili ty in the emissions of those units

. The evaluation w ill include one or more of the following, depending on the specific dataset : confirming that the data distribution was selected correctly ; after confirming the data distribution, ensur ing that we use the most appropriate UPL equation 10 ; and, as necessary, compar ing UPL equation components for the individual unit upon which a new source floor is based with those of the units in the existing source floor to determine if our identification of the best unit is reasonable

. Each of the additional evaluations are discussed below.

Confirming the data distribution is important because UPL equations and the emission limits derived from those UPL equations depend on the distribution. In prior rulemakings , we had no t i dentified a way of ascertaining the data distribution for units with three samples, so we either assumed a normal or lognormal distribution. This particularly affected new source emissions limits, many of which were based on three samples

. However, we have since identified and adopted a n established technique t hat compare s the ratio of skewness to the standard error of the skewness for both the raw and log transformed data. The lower of the two ratios identifies the data distribution that b est represents the sample set. This ability to more precisely ascertain the data distribution enables us to select the best option for the UPL equation given the small sample size

. After confirming that we selected the best distribution based on the avai lable data and ensuring that we used the correct equation to calculate a MACT floor value, we then examine key variables that factor into the calculated floor values. The variables that are factored into the emission limit include the following: average, t

- score, confidence level, variance, number of test runs (i.e., sample size), and number of runs in each test (i.e., the number of data points that are averaged together to determine compliance with a particular emission limit ). The considerations related t o each variable are discussed below.

10 For datasets with lognormal distributions, we recently determined that the most appropriate UPL equation, which is especially important for the smallest datasets that we use, is based on an ap proach described in Bhaumik, 2004, op. cit.7 The average, which typically is calculated as the mean value of a dataset , does not require additional consideration. Once it is calculated, it is used to rank the performance of units, and also is the key value upon w hich each emission limit is based. As such, average values have already been compared within a particular dataset , and additional assessment is not necessary. However, in the overall assessment of variables, when multiple best performing units have emissio n averages that are similar, we may look to other variables like the variance to help to inform our decision as to which unit is the single best performer.

As discussed earlier, the t

- score is dependent on the sample size, and, therefore, we discuss these two variable s together. As shown earlier in Figure 1, as the sample size decreases, the t

- statistic increases. This is important because the t

- statistic is multiplied by the variance and a factor that involves the sample size and the number of test runs used for compliance (i.e., the “m” term in the UPL equation) . For the smallest dataset s (3 data points) that we consider under the UPL approach, a large variance coupled with a t

- score that is significantly greater than the t

- score for

a dataset of 7 or mo re c an create a situation where the emission limit includes a large amount of uncertainty. In such cases, we use our technical expertise to assess what level of emissions could realistically be expected from the type of unit, controls, and other relevant f actors, and would ensure that the emission limit is r easonable.

The variance is another key variable that we would evaluate. Generally speaking, if our evaluation showed a very small amount of varia bility (e. g. , a small variance) , the emission limit woul d not contain an unacceptable amount of uncertainty and variability because a small variance would result in an emission limit that would be reasonably close to the mean of the data. With larger datasets, it is more likely that the demonstrated variance is truly representative of the variation in the source’s emissions simply because there is more evidence of the source’s emissions over time. On the other hand, when a limited dataset includes a large amount of variability ( e

. g. , a large variance ) , we would need to carefully evaluate the data. For instance, consider the case where the pool of best performing units all have similar averages and the unit identified as the single best performer (based on average emissions only) has a limited dataset and a variance that greatly exceeds the variance of the other similarly

- controlled best

- performing units. In such a case, the MACT floor analysis may yield an emission limit for that unit (i.e., the new source MACT floor) that is higher than the existing source MACT floor, which is an indicator that further analysis is warranted. Careful consideration of variance will allow us to better ascertain which unit should be considered the best performer.

Another tool that could be used to adjust unreasonable emi ssion limits is the selection of the confidence level . In certain insta n ces, we may determine that a limited dataset includes such a high degree of varia bility that the 99 percent confidence level results in an emission limit that our experience suggests i s high er than the average emissions limitation achieved by the best performing source or sources after accounting for variability . In such cases, we may choose to select a confidence level of 95 percent or 90 percent. As an example, consider a unit identif ied as the best performing new source with a specific process or control device that has been demonstrated to operate far more efficiently (thus having the potential to lower emissions) in similar units, processes, or control devices. Under these condition s, we may choose to acknowledge the better operation by lowering the confidence level (which lowers the emissions limit)

.8 Establishing and using this approach on a case

- by

- case basis for limited dataset s will ensure consistent application of emissions limit development procedures, which will mitigate the additional uncertainty that could otherwise result from calculating MACT floor standards with limited data. 11

How did we apply the approach for limited datasets to limited datasets in the brick and stru ctural clay products (BSCP) source category?

For the BSCP source category, we have limited datasets for the following pollutants and subcategories: mercury ( Hg ) for existing and new small tunnel kilns; particulate matter ( PM ) for

new tunnel kilns; and Hg for new large tunnel kilns

. For the BSCP alternate options we have limited datasets for the following pollutants and subcategories: PM for new large tunnel kilns and PM for new small tunnel kilns. W e evaluated these specific datasets to deter mine whether it is appropriate to make any modifications to the approach used to calculate MACT floors for each of these datasets.

For each dataset, we performed the steps outlined above, including: ensuring that we selected the data distribution that bes t represents each dataset; ensuring that the correct equation for the distribution was then applied to the data; and comparing individual components of each limited dataset to determine if the standards based on limited dataset s reasonably represent the
pe rformance of the units included in the dataset.

For each pollutant and subcategor y with a new source MACT floor limit based on the UPL, the best performing kiln was identified as the kiln with the lowest average emissions and the lowest variance. For Hg f rom small tunnel kilns , the existing source limits (lb/ton, μg/ dscm @ 7% O 2 , lb/hr) are based on one unit ; as such , they are the same as the new source limit s . For Hg from small tunnel kilns

, we found that the calculated existing emission limit s are abou t 1.1 times the
average emissions . All of these three existing limits are within the range that we see when we evaluate larger data sets using our MACT floor calculation procedures. Therefore , based on the results of the analyses for new and existing sources , w e determined that no changes to our standard floor calculation procedure were warranted and are proposing the MACT floor limits for the pollutants and subcategories shown in Table 2.

For the BSCP alternate opti ons , we determined that no changes to our standard floor calculation procedure were warranted for three of the four datasets (see Table 3). For these datasets, the new source MACT floor was either based on the 3xRDL or on the UPL for the best performing ki ln, which was identified as the kiln with the lowest average emissions and a variance similar to the kiln with the lowes t variance.

11 We note that in the recent D.C. Circuit Court of Appeals decision in National Assn. of Clean Water Agencies v. EPA (NACWA) , which involved challenges to EPA’s MACT standards for sewage sludge incinerators, the court identified two instances where we replaced a new source floor value with the existing source value because the new source value was less stringent than the existing source value (the “anomalous result”). See 734 F.3d 1115. The procedures outlined her e would eliminate the anomalous result in both instances in the rule at issue in NACWA. The first instance in which the court in the NACWA decision identified the anomalous result is where we established an emission limit using the UPL on a dataset that in cluded only 2 data points. The other instance in which the court identified the anomalous result is where we did not apply the most appropriate lognormal UPL equation.9 Table 2. Proposed BSCP MACT Floor Limits Based On Limited Datasets

Category Pollutant a Source ID Average Variance n Distribution UPL 3xRDL Floor Small Tunnel Kilns ( Existing

a nd New ) Hg lb/ton 531 1.76E

- 04 5.51E

- 11 3 Lognormal 2.0E

- 04 1.7E

- 05 2.0E

- 04 Hg μg/ dscm @ 7% O 2 531 61.87 2.18 3 Normal 70 6.1 70 Hg lb/hr 531 9.97E

- 04 1.76E

- 09 3 Lognormal 0.0011 2.0E

- 04 0.0011
Large Tunnel Kilns ( New ) Hg lb/ton 568 1.48E

- 05 8.83E

- 13 3 Normal 2.0E

- 05 8.0E

- 06 2.0E

- 05 Hg μg /dscm @ 7% O 2 568 10.45 1.13 3 Lognormal 13 6.4 13 Hg lb/hr 568 1.75E

- 04 1.25E

- 10 3 Normal 2.4E

- 04 8.0E

- 05 2.4E

- 04 All Tunnel Kilns ( New ) PM lb/ton 514 6.04E

- 03 1.08E

- 05 3 Lognormal 0.018 0.022 0.022 PM gr/dscf @ 7% O 2 514 2.00E

- 03 1.20E

- 06 3 Lognormal 0.0059 0.0061 0.0061

a lb/ton = pounds per ton; μg/dscm @ 7% O 2 = micrograms per dry standard cubic meter at 7 percent oxygen; lb/hr = pounds per hour; gr/dscf @ 7% O 2 = grains

per dry standard cubic meter at 7 percent oxygen.

Table 3. Selected Alternate BSCP MACT Floor Options Based On Limited Datasets

Category Pollutant a Source ID Average Variance n Distribution UPL 3xRDL Floor Alternate Small Tunnel Kilns ( New ) PM lb/ton 599 9.30E

- 03 2.99E

- 05 3 Lognormal 0.029 0.029 Alternate Large Tunnel Kilns ( New ) PM lb/ton 514 6.04E

- 03 1.08E

- 05 3 Lognormal 0.018 0.016 0.018 PM gr/dscf @ 7% O 2 514 2.00E

- 03 1.20E

- 06 3 Lognormal 0.0059 0.0077 0.0077

a lb/ton = pounds per ton; gr/dscf @ 7% O 2 = grains per dry standard cubic meter at 7 percent oxygen.10 The fourth of the BSCP alternate option datasets, t he new source MACT floor PM gr/dscf @ 7% O 2 dataset for small tunnel kilns

, includes 3 test runs from a single tunnel kiln (Kiln ID 599) that we identified as the top

- ranked unit based on average lb/ton emissions. After determining that the dataset is best re presented by a lognormal distribution and ensuring that we used the correct equation for the distribution, we evaluated the variance of this unit (Kiln ID 599) and concluded that further consideration of the variance was warranted. In particular, we noted that the variance of the dataset for this unit was almost twice as large as the variance of the dataset for the pool of best performing units that was used to calculate the existing source MACT floor. The high degree of variance in the dataset for the unit with the lowest average prompted us to question whether this unit was, in fact, the best performing unit and to evaluate which unit would better represent the best performing unit. The dataset for Kiln ID 665 includes 3 test runs and the average emissions are only about 18 percent higher than Kiln ID 599. While we find the average performance of these two units to be similar, Kiln ID 665 has a variance almost an order of magnitude lower than Kiln ID 599, thus indicating that Kiln ID 665 has a far more cons istent level of performance. For these reasons, we determined that Kiln ID 599 is not the best performing source for this pollutant

, and we are instead selecting Kiln ID 665 as the best performing source. After selecting the source upon which the new sourc e limit would be based,

we determined the distribution of the dataset to be lognormal

. Therefore, the new source MACT floor for this alternate option is 0.0040 gr/dscf @ 7% O 2 for PM from small tunnel kilns.

Table 4

. Alternate BSCP MACT Floor Option for Small Tunnel Kilns

Category Pollutant Source ID Average Variance n Distribution UPL 3xRDL Floor Alternate Small Tunnel Kilns (New) PM gr/dscf @ 7% O 2 599 2.31E

- 03 1.81E

- 06 3 Lognormal 0.0072 0.0072

665 0.002727 2.17E

- 07 3 Lognormal 0.0040 0.0040