1

MEMORANDUM

SUBJECT: Approach for Applying the Upper Prediction Limit to Limited Datasets

DATE: December 12, 2014

FROM: Toni Jones Office of Air Quality Planning and Standards

TO: Docket ID No. EPA-HQ-OAR-2003-0119

This memo is being provided for informational purposes for the context of reconsideration of the PM limit for the waste-burning kiln subcategory. It should be noted that, based on this memo, the changes proposed in the CISWI r econsideration proposal are not related to limited dataset issues. The CISWI reconsideration pr oposal does not propose new emission limits based on the potential approaches for limited dataset s presented within this memorandum.

I. What is Our Approach for Applying the Upper Prediction Li mit to Limited Datasets?

The UPL approach addresses variability of emissions data from the best performing source or sources in setting MACT standards. The UPL also accounts for uncertainty associated with emission values in a dataset, which can be influenced by components such as the number of samples available for developing MACT standards and the n umber of samples that will be collected to assess compliance with the emission limi t. The UPL approach has been used in many environmental science applications. 1,2,3,4,5,6 As explained in more detail in memorandum entitled “Use of the Upper Prediction Limit for Calcul ating MACT Floors”, which is located in

1 Gibbons, R. D. (1987), Statistical Prediction Intervals for the Evaluation of Ground- Water Quality . Groundwater, 25: 455–465 and Hart, Barbara F. and Janet Chaseling, Optimizing Landfill Ground Water Analytes —New South Wales, Australia, Groundwater Monitoring & Remediation, 2003, 23, 2.

2 Wan, Can; Xu, Zhao; Pinson, Pierre; Dong, Zhao Yang; W ong, Kit Po. Optimal Prediction Intervals of Wind Power Generation. 2014. IEEE Transactions on Power Syst ems, ISSN 0885-8950, 29(3): pp. 1166 – 1174.

3 Khosravi, Abbas; Mazloumi, Ehsan; Nahavandi, Saeid; Cr eighton, Doug; van Lint, J. W. C. Prediction Intervals to
Account for Uncertainties in Travel Time Prediction. 2011. IEEE Transactions on Intelligent Transportation Systems, ISSN 1524-9050, 12(2):537 – 547.

4 Ashkan Zarnani; Petr Musilek; Jana Heckenbergerova. 2014. C lustering numerical weather forecasts to obtain statistical prediction intervals. Meteorological Applica tions, ISSN 1350-4827. 21(3): 605.

5 Rayer, Stefan; Smith, Stanley K; Tayman, Jeff. 2009. Em pirical Prediction Intervals for County Population Forecasts. Population Research and Policy Review, 28(6): 773 – 793.

6 Nicholas A Som; Nicolas P Zegre; Lisa M Ganio; Arne E Skaugset. 2012. Corrected prediction intervals for change detection in paired watershed studies. Hydrological Sci ences Journal, ISSN 0262-6667, 57(1): 134 – 1432

the docket for this action, the EPA uses the UPL approach to reasonably estimate the emissions performance of the best performing source or sources to establish MACT floor standards.

Sample size (the number of values in a particular dat aset) is an important component of the UPL approach. First, the use of the UPL approach requires us t o identify the distribution of the data, that is, whether a particular dataset is normally or non-normally distributed (see the memorandum entitled “Use of the Upper Prediction Limit f or Calculating MACT Floors”, which is located in the docket for this action). To determine the distribution of the data, we use wellestablished tests (kurtosis and skewness tests), and these tests require at least three data points. In prior rulemakings, including this one, we used the kurtosis equa tion that is a built-in function in Excel software, but that equation requires at least 4 v alues in order to avoid dividing by zero, which precluded its use for datasets of 3. However, we rece ntly further reviewed the application of the UPL where data are limited, and concluded that it may be appropriate to use another
kurtosis estimator 7 that provides a meaningful result with just 3 values. Th e use of this estimator is important, as many, if not most, of our new source e missions limits are based on 3 samples collected during an emissions test.

Once we determine the data distribution, the appropriate equat ion to be used in the UPL approach is selected based on the data distribution. We re cognize that the use of the UPL approach for limited datasets introduces some amount of un certainty in the calculation of MACT standards, and, therefore, we may consider taking additio nal steps, discussed in this document, to
ensure that the level of the MACT standards is reasonabl e. We also note that after MACT standards are promulgated, we are required to review those s tandards periodically, and for such reviews, we typically have significant additional HAP e missions data from the intervening years of compliance with which to further assess the actual pe rformance of the various emission

sources.

Regardless of the distribution of the data, UPL equation s have three well-defined components: an average, the t-score and a measure of variability that i ncludes the actual variability of the data, the sample size, and the number of data points that ar e averaged together to determine compliance with a particular emission limit. A t-sco re is a value that estimates the uncertainty and variability for a certain confidence level associat ed with a specific number of data points. For a constant confidence level, t-scores decrease as the number of samples increase. This means that, if the mean and variance remain constant, UPL v alues for a particular confidence level decrease as the number of samples increase. Figure 1 and Table 1 show the t-scores for various sample sizes, and demonstrate that the t-score is highly variable at the smallest sample sizes and becomes relatively constant once the sample size is larger than a few data points. Consequently, we recognize that we need to take special care when small er, limited sample sizes are used to establish emission limits, as the t-scores can have a disproportionate effect, overwhelming the other components of the emission limit calculation. In addition, we recognize that for a sample size of fewer than three data points, which has a ver y large t-score and precludes the appropriate selection of a distribution, we should not develop emis sion limits using the UPL. In other words, if fewer than 3 data points are available for use in det ermining an emission limit for a particular

7 Doric D, Nikolic-Doric E, Jevremovic V and Malisic J. 2009. On measuring skewness and kurtosis. Qual Quant.
43:481-493. DOI 10.1007/s11135-007-9128-9.3

source, and no other data from sources in the subcategory are available, we would have to establish a different procedure for establishing the MACT floor that does not rely on the UPL.

Figure 1. Sample sizes versus one-sided, 99 percent confiden ce interval t-scores for normal

distributions

Table 1. Degrees of Freedom, T-Score for a One-Tailed, 99 P ercent Confidence Interval, and Percent Difference in T-Score for Each Sample Size

Sample Size (n) Degrees of Freedom (n1) t-score (99

percent) Relative Percent

Difference a

2 1 31.82

3 2 6.96 78

4 3 4.54 35

5 4 3.75 17

6 5 3.36 10

7 6 3.14 7

8 7 3.00 5



### Number of SamplesT-score 

## T-score vs Number of Samples

2 3 4 5 6 7 8 9 0 5 10 15 20 25 30 35

78% reduction from 2 to 3 samples

35% reduction from 3 to 4 samples

17% reduction from 4 to 5 samples

10% reduction from 5 to 6 samples

7% reduction from 6 to 7 samples

5% reduction from 7 to 8 samples4

9 8 2.90 3

a The relative percent difference between the t-score a nd the t-score shown on the preceding row

In the case of a dataset with a normal distribution, the equation to calculate the UPL is as
follows:

 = ̅ +  [ 

,   ] 

   1

 + 1

 

where  = average or mean of test run data;

 [ 

,   ] = t score, the one-tailed t value of the Student’s t distribution for a specific
degree of freedom (n-1) and confidence level (1- α );

α = level of significance expressed as a decimal (e.g., 1% s ignificance = 0.01), note that

confidence level = 100 −  × 100  ;

  = variance of the dataset (test run);

 = number of values (test runs) in the dataset; and

 = number of values used to calculate the test average ( generally 3).

As shown in the equation, the sample size (n) directl y affects the UPL calculations in two ways. First, as mentioned earlier, the t-score is selected b ased on the sample size. Second, the sample size is a separate variable in the equation, and as with the t-score, as the sample size gets larger (all other variables being held constant), the UPL gets smaller (i.e., approaches the mean value). The sample size also influences both the mean ( ̅ ) and variance (s 2 ) in that larger sample sizes provide better estimates of the population mean and variance . The confidence level also influences the t-score, and is represented in the UPL e quation by the subscript “1- α .” As noted in the memorandum entitled “Use of the Upper Prediction Limit for Calculating MACT Floors”, which is located in the docket for this action, we typic ally choose the 99 percent UPL, which is another way of saying that α equals 0.01.

Note that the UPL equation for a lognormal distribution us es a z-score, instead of a t-score. 8 As with the normal UPL equation, the lognormal UPL equation is affected by sample size and the variance of the data, and the basic concepts outlined i n this discussion apply to datasets regardless of the distribution.

8 Bhaumik, D.K. and Gibbons, R.D.``An Upper Prediction Lim it for the Arithmetic Mean of a Lognormal Random Variable", Technometrics, 46, 2004, 239-248.5

In summary, sample size is important for UPL calcula tions, as it is a key component of the tscore value and the calculation of the mean and varian ce of the sample. For the reasons presented above, for datasets below a certain size, further eval uation is warranted to ensure that the results are a valid representation of the performance of the e mission sources whose data are included in the dataset. The remainder of this document provides our a ssessment of what constitutes a limited dataset and what additional analyses are appropria te for ensuring that the UPL calculations are valid representations of the performan ce of units as reflected in the available
data.

II. What is a limited dataset?

While there are different options for choosing the num ber of samples that represents a limited dataset (or small sample size), we can use distinctio ns based on our observations in Figure 1 and Table 1 as an aid in defining what constitutes a limited dat aset. As discussed earlier, the t-score is an influential parameter that can have a particularl y large influence on an emission limit developed from a limited dataset. As such, the t-score is a key parameter for identifying what constitutes a limited dataset for purposes of MACT analy ses. As shown in Figure 1 and Table 1, above, at the 99 percent confidence level there is a clea r distinction between t-score values for sample sizes of 3 and fewer when compared to t-score valu es for sample sizes of more than 6. As Figure 1 and Table 1 show, the t-score changes drastically from sample size equal to 2 to sample size equal to 3. The changes in the t-score are considera bly less dramatic as the sample size approaches 9 data points and larger.

As noted earlier, we will not use the UPL approach for datasets with fewer than 3 values because of the high degree of uncertainty that results from the size of the dataset and the related t-score. In fact, the t-score (see Figure 1 and table 1) for a datas et of 3 is 78 percent lower than a t-score for a dataset of 2, meaning that given an equal variance, t he component of the UPL that includes the t-score and variance would be more than 78 percent le ss when evaluating a 3-point dataset compared to a 2-point dataset. The table and figure also s how that the relative change of the tscore for each additional data point decreases as the s ample size gets larger. As sample size increases, uncertainty decreases, but there is always so me amount of uncertainty. The discussion that follows presents our examination of the t-score a t various sample sizes. We rely on this examination to help identify a sample size below which we would further evaluate the data and the application of the UPL to ensure that the amount of uncertainty is within a reasonable range. Based on Table 1, the lower bound of the sample size whe re such further evaluation is needed is 3 values, and a reasonable value (where the size of the t-statistic has much less influence on the eventual emission limit) lies between 4 and 9 values.

An additional factor to consider is that, in the contex t of MACT analyses, emission tests typically include 3 test runs (independent data points), and, therefore, our MACT floor dataset size typically is a multiple of 3. For this reason, w e first considered sample sizes that are multiples of 3 when identifying the size below which furthe r scrutiny is needed. As can be calculated from the t-scores shown in Table 1, the rela tive difference between the t-score for 6 data points and the t-score for 9 data points is about 1 4 percent ((3.36-2.90)/3.36). The relative difference between the t-scores for 7 and 9 data points is about 8 percent ((3.14-2.90)/3.14). Given that the test methods that are used to generate th e data typically are accurate to within 106

to 20 percent 9 , this difference in t-score (between datasets with 7 a nd 9 runs) is less than the uncertainty in the test methods and would not have a la rge influence on the magnitude of an emission limit. Conversely, comparing the t-score fo r 3 runs to that for 7 runs, the relative difference in the t-score is 55 percent ((6.96-3.14)/6.96), i ndicating that there is a possibility, especially if the variance is high, that the t-score f or 3 test runs may introduce a large amount of uncertainty in an emission limit, and, therefore, add itional analysis of a dataset of this size is warranted. We also conducted the same type of evaluation on datasets between 4 and 6 points, and while the change in t-score for datasets of these s izes could be considered to be sufficiently small, to be conservative, we conclude that a limited dataset, for purposes of MACT analyses, is
a dataset that includes at least 3 test runs and fewer th an 7 test runs. Because of the uncertainty associated with datasets at or below this size, further evaluation is warranted, and the types of additional analyses that we may choose to conduct are explained below.

III. What approach could the EPA follow for MACT floor s based on limited datasets?

When a MACT floor for either existing or new sources i s based on fewer than 7 data points, we

could further evaluate each individual dataset in order t o ensure that the uncertainty associated
with a limited dataset does not cause the calculated emi ssion limit to be so high that it does not reflect the average performance of the units upon which th e limit is based after accounting for variability in the emissions of those units. The evaluat ion would include one or more of the following, depending on the specific dataset: confirming th at the data distribution was selected correctly; after confirming the data distribution, ensuring that we use the most appropriate UPL equation 10 ; and, as necessary, comparing UPL equation components for the individual unit upon
which a new source floor is based with those of the units in the existing source floor to determine

if our identification of the best unit is reasonable. Ea ch of the additional evaluations the EPA
could consider for CISWI are discussed below.

Confirming the data distribution is important because UPL e quations and the emission limits derived from those UPL equations depend on the distribution. In prior rulemakings, including this one, we had not identified a way of ascertaining the data distribution for units with three samples, so we either assumed a normal or lognormal d istribution. This particularly affected new source emissions limits, many of which were based on th ree samples. However, we have since identified an established technique that compares the ratio o f skewness to the standard error of the skewness for both the raw and log transformed data. T he lower of the two ratios identifies the data distribution that best represents the sample set. This ability to more precisely ascertain the data distribution enables us to select the best option f or the UPL equation given the small sample

size.

After confirming that we selected the best distribution ba sed on the available data and ensuring that we used the correct equation to calculate a MACT fl oor value, we could then examine key

variables that factor into the calculated floor values . The variables that are factored into the

9 (ReMAP): PHASE 1, Precision of Manual Stack Emission M easurements; American Society of Mechanical Engineers, Research Committee on Industrial and Municipal Waste, February 2001.

10 For datasets with lognormal distributions, we recently determined that the most appropriate UPL equation, which is especially important for the smallest datasets that we use, is based on an approach described in Bhaumik, 2004, op. cit.7

emission limit include the following: average, t-score, c onfidence level, variance, number of test runs (i.e., sample size), and number of runs in each test (i.e., the number of data points that are averaged together to determine compliance with a particul ar emission limit). The considerations
related to each variable are discussed below.

The average, which typically is calculated as the mean value of a dataset, does not require additional consideration. Once it is calculated, it is used to rank the performance of units, and also is the key value upon which each emission limit is based. As such, average values have already been compared within a particular dataset, and add itional assessment is not necessary. However, in the overall assessment of variables, whe n multiple best performing units have emission averages that are similar, we may look to other variables like the variance to help to inform our decision as to which unit is the single best pe rformer.

As discussed earlier, the t-score is dependent on the sam ple size, and, therefore, we discuss these two variables together. As shown earlier in Figure 1, as the sample size decreases, the t-statistic increases. This is important because the t-statistic is multiplied by the variance and a factor that involves the sample size and the number of test runs use d for compliance (i.e., the “m” term in the UPL equation). For the smallest datasets (3 data poi nts) that we consider under the UPL approach, a large variance coupled with a t-score that is significantly greater than the t-score for
a dataset of 7 or more can create a situation where the emission limit includes a large amount of uncertainty. In such cases, we use our technical expertis e to assess what level of emissions could realistically be expected from the type of unit, control s, and other relevant factors, and would

ensure that the emission limit is reasonable.

The variance is another key variable that we would eva luate. On one hand, if our evaluation showed a very small amount of variability (e.g., a sma ll variance), the emission limit would not contain an unacceptable amount of uncertainty and variabi lity because a small variance would result in an emission limit that would be reasonably c lose to the mean of the data. With larger datasets, it is more likely that the demonstrated vari ance is truly representative of the variation in the source’s emissions simply because there is more evi dence of the source’s emissions over time. On the other hand, when a limited dataset includes a large amount of variability (e.g., a large variance), we would need to carefully evaluate the da ta. For instance, consider the case where the pool of best performing units all have similar av erages and the unit identified as the single best performer (based on average emissions only) has a limited dataset and a variance that greatly exceeds the variance of the other similarly-cont rolled best-performing units. In such a case, the MACT floor analysis may yield an emission l imit for that unit (i.e., the new source MACT floor) that is higher than the existing source MACT floor, which is an indicator that further analysis is warranted. Careful consideration of variance will allow us to better ascertain which unit should be considered the best performer.

Another tool that could be used to adjust unreasonable emissi on limits is the selection of the confidence level. In certain instances, we may determine that a limited dataset includes such a high degree of variability that the 99 percent confidence le vel results in an emission limit that our experience suggests is higher than the average emissions limitation achieved by the best performing source or sources after accounting for variabili ty. In such cases, we may choose to select a confidence level of 95 percent or 90 percent. As a n example, consider a unit identified as8

the best performing new source with a specific process or control device that has been demonstrated to operate far more efficiently (thus having th e potential to lower emissions) in similar units, processes, or control devices. Under thes e conditions, we may choose to acknowledge the better operation by lowering the confidence level (which lowers the emissions limit).

Establishing and using this approach on a case-by-case basis fo r limited datasets would ensure consistent application of emissions limit development procedures, which would mitigate the additional uncertainty that could otherwise result from calculating MACT floor standards with
limited data. 11

11 We note that in the recent D.C. Circuit Court of Appeal s decision in National Assn. of Clean Water Agencies v. EPA (NACWA) , which involved challenges to EPA’s MACT standards for sewage sludge incinerators, the court identified two instances where we replaced a new source f loor value with the existing source value because the new source value was less stringent than the existing source va lue (the “anomalous result”). See 734 F.3d 1115. The procedures outlined here would eliminate the anomalous res ult in both instances in the rule at issue in NACWA. Th e first instance in which the court in the NACWA decision identified the anomalous result is where we established an emission limit using the UPL on a dataset that include d only 2 data points. The other instance in which the co urt identified the anomalous result is where we did not apply t he most appropriate lognormal UPL equation.