Date: October 30, 2007

To: Karen Rackley, EPA/SPPD

From: David Randall and Breda Munoz

Subject: Statistical Analysis of Valve Leak Frequencies Obtained by Industry and EPA for Petroleum Refining Process Units

I. Introduction

The U.S. Environmental Protection Agency (E PA) is currently revi ewing the new source performance standards for equipment leaks of VOC in the Synthetic Organic Chemicals Manufacturing Industry and in the Petroleum Re fining Industry (40 CFR part 60, subparts VV and GGG). In order to estimate the impacts of regulatory alternatives, the emission rates under both the existing requirements and the regulatory alternative must be determined. Calculating the emission rates requires an es timate of the percentage of equi pment that is leaking. Industry trade associations have provided leak frequency data for numerous process units that could be used directly in these impacts analyses. However, the EPA’s National Enforcement Investigations Center (NEIC) also conducts monito ring at some of the faci lities subject to these rules. In 1999, the NEIC published the resu lts of a comparative monitoring study at 17 petroleum refineries. 1 These results showed the percentage of valves identif ied as leaking by NEIC was always higher than th e results of monitoring conducted by the petroleum refiners, and the results differed by as much as a factor of 10. RTI has conducted an anal ysis of these results showing there is a statistically significant diffe rence between the leak frequencies identified by NEIC and the petroleum refiners, and the NEIC re sults are higher than the industry results by a

factor of at least 2.6 at the 99 percent confidence interval. Th e purpose of this memorandum is to describe the statistical analys is used to develop these results.

II. Background

Table 1 shows the results of the industry and NEIC monitoring programs at the 17

refineries. 1 The leak frequencies determined by the companies ranged from 0.2 percent to 3.6 percent, and the leak frequencies determined by NEIC ranged from 1.4 per cent to 11.5 percent. In the report of their study, NEIC theorizes th at the disparity between the NEIC and company results may be attributable to refineries not monitoring in the manner prescribed in Method 21 of 40 CFR part 60, Appendix A-7.

The data for each of the 17 facilities are for so me of the process units that are subject to equipment leak standards and that NEIC pre-select ed for monitoring as part of their inspection. The number of valves monitored by the companies should be the total number of valves that are2 subject to monitoring in the se lected process units during the most recent monitoring period. Based on information from the industry trade associat ion, it is likely that mo st of these refiners are conducting monitoring quarterly, but it is possibl e that some have impl emented an option that allows less frequent monitoring when low leak frequencies are maintained. All of the valves within a given process must be monitored within a calendar month under ei ther requirement (but not all processes would have to be monitored in the same month). Typically, NEIC tries to monitor about half of the valves in the selected pr ocess units, but less than half were monitored if the selected process units have a large number of valves. When monitoring, the NEIC inspectors rely on the company representatives to point out valves to monitor until they have reached the target number. With the exception of excluding most inaccessible valves, the valves that are monitored are selected as randomly as the compa ny representatives pick them out. NEIC makes no attempt to pick out certain types of valves or valves that are most likely to leak because of process conditions, age, or other factors. Th e companies and NEIC both follow Method 21 when monitoring.

III. Statistical Analysis

A. Hypotheses

The hypotheses tested by this study were:

1. Is the NEIC leak rate equal to the compa ny leak rate at each of the 17 refineries (which is equivalent to tes ting if the difference between th e two leak rates is equal to zero), and

2. Is the mean of the NEIC-to-company leak frequencies ratio equal to one? This hypothesis will allow us to assess the averag e difference between the two mean leak frequencies.

B. Methods and Results

Seventeen one-sample t-tests were conducted to test the first hypothesi s. If we assume that the valves monitored by NEIC were assumed to be random samples of the valves monitored by each company based on the discussion in Section II and that the reported re finery leak rate is the population value (fixed value), then to asse ss hypothesis 1 we can perform the standard onesample test for equality of proportions (or t- test). There is a cl ose relationship between confidence intervals and hypothesis testing. When a 95% confidence interval is constructed, all values in the interval are considered plausible values for the parameter being estimated. Values outside the interval are rejected as relatively impl ausible. If the value of the parameter specified by the null hypothesis is contai ned in the 95% interval, then the null hypothesis cannot be rejected at the 0.05 level. If the value specified by the null hypothesis is not in the interval then the null hypothesis can be reject ed at the 0.05 level. The fo llowing equation was used to calculate the upper and lower confidence limits of the differences (at the 95 percent significance level):3 

### n p p t p p df 

### ) ˆ 1 ( ˆ

, 2 / 0 

### −

±

− α 

### )

where:

α = the significance level (95% significance level) df = the degrees of freedom, which are calculated as n -1

n = the number of valves selected by NEIC for inspection

df t

, 2 / α = the tabular value from the t-distribution with df degrees of freedom

p ˆ = the NEIC leak ratio: Valves Number NEIC Leak Number NEIC

ˆ
= p and

P 0 = the company leak ratio (which is assumed fixed):

Valves Number Company Leak Number Company

0 = p

Results of the calculations are presented in Table 2. These results show the confidence intervals are above zero for 16 of the 17 companie s, which indicates the NEIC leak frequencies are different from the leak frequencies obtaine d by these companies. The only company for which the hypothesis of no difference in the leak frequencies cannot be rej ected is Company D. The confidence interval for this Company includ es zero, which suggests the leak frequencies obtained by NEIC and this company are not diffe rent at the 95 percent significance level.

For the second hypothesis, we consider the fa ct that we have two measurements for each

of the 17 refineries. This is known in the statistics literature as paired de signs. One approach to analyze these data is to calcul ate the ratio of the two pairs of measurements and transform the problem to a one-sample testing if the mean of th e ratios equals 1. To account for the deviation of the normality assumption, we log-transformed the data so the distribution of the sample

resembles normal distributed. We tested the se cond hypothesis using a one-sample t-test. The original ratios and the transformed ratios are sh own in Table 3. The following equation was used to calculate the upper and lo wer bounds of the mean of the transformed ratios:


# ( ) ( ) 

# [ ] ) log( ) log(

, 2 / 

### ratio Mean SE t ratio Mean df α 

### ±

where:

Mean (log( ratio )) = the mean of the log-transformed ratios, SE[ Mean (log( ratio )) = the standard error of the of the mean of the log-transformed ratios,

α = the significance level ( 99 percent significance level), df

= the degrees of freedom (16 degrees of freedom), and

t α /2, df = the tabular value from the t-distribution with df degrees of

freedom4 Table 4 shows the resulting statistics for the 99 percent significance level. The lower and upper bounds of the transfor med mean ratio are 0.95 and 2.1, respectively. Back transforming (exponentiating) gives the lower and upper bounds of the ratio as 2.6 and 8.4, respectively. Because this range does not include 1, the null hypothesis of equal leak frequencies is rejected. Furthermore, these results suggest that the NEIC leak frequencies are at least 2.6 times greater than the Company leak frequencies at the 99 percent confidence level.

IV. References

1. U.S. EPA, Office of Enforcement a nd Compliance Assurance. October 1999. Enforcement Alert: Proper Monitoring Esse ntial to Reducing 'Fugitive Emissions' Under Leak Detection and Repair Programs

.5 Table 1. Results of NEIC Comparative Monitoring Study

Refinery Company Monitoring: NEIC Monitoring Leak Rate (%)

Valves Leaks Valves Leaks Company NEIC A 7,694 170 3,363 354 2.3 10.5 B 7,879 223 3,407 216 2.8 6.3 C 3,913 22 2,008 108 0.6 5.4 D 2,229 26 1,784 24 1.2 1.4 E 5,555 96 2,109 112 0.7 5.3 F 42,505 124 3,053 53 0.3 1.7 G 14,307 226 3,852 236 1.6 6.1 H 20,719 736 3,351 179 3.6 5.3 I 5,339 9 2,754 84 0.2 3.1 J 8,374 78 2,981 55 0.9 1.8 K 6,997 101 1,658 114 1.4 6.9 L 12,686 26 3,228 125 0.2 3.8 M 4,160 40 1,926 222 1.0 11.5 N 5,944 29 2,487 106 0.5 4.3 O 7,181 112 2,897 130 1.6 4.5 P 8,532 203 4,060 181 2.4 4.5 Q 6,640 36 2,608 74 0.5 2.8 Total 170,717 2,266 47,526 2,372 1.3 (avg) 5.0 (avg)6 Table 2. Confidence Intervals for the Di fference Between NEIC Leak Rate and Company Leak Rate

Refinery Company

Leak Rate (%) (1) NEIC

Leak Rate (%) (2) NEIC Leak

Rate minus

Company

Leak Rate (2)-(1) 95% Lower

Confidence

Limit for

(2)-(1) 95% Upper

Confidence

Limit for

(2)-(1)

A 2.3 10.5 8.2 7.164 9.236

B 2.8 6.3 3.5 2.684 4.316

C 0.6 5.4 4.8 3.811 5.789

D 1.2 1.4 0.2 -0.345 0.745

E 0.7 5.3 4.6 3.644 5.556

F 0.3 1.7 1.4 0.941 1.859

G 1.6 6.1 4.5 3.744 5.256

H 3.6 5.3 1.7 0.941 2.459

I 0.2 3.1 2.9 2.253 3.547

J 0.9 1.8 0.9 0.423 1.377

K 1.4 6.9 5.5 4.280 6.720

L 0.2 3.8 3.6 2.940 4.260

M 1 11.5 10.5 9.075 11.925

N 0.5 4.3 3.8 3.003 4.597

O 1.6 4.5 2.9 2.145 3.655

P 2.4 4.5 2.1 1.462 2.738 Q 0.5 2.8 2.3 1.667 2.9337Table 3. Actual and Transformed Ratios of the NEIC Leak Rate to the Company Leak RateCompany Monitoring:NEIC MonitoringLeak Rate (%)ObservationRefineryValvesLeaksValvesLeaksCompanyNEICRatio = NEICleak Rate /Refinery LeakRatelogratio1A 7,694 170 3,363 354 2.3 10.5 4.5652 1.518472B 7,879 223 3,407 216 2.8 6.3 2.25 0.810933C 3,913 22 2,008 108 0.6 5.4 9 2.197224D 2,229 26 1,784 24 1.2 1.4 1.1667 0.154155E 5,555 96 2,109 112 0.7 5.3 7.5714 2.024386F 42,505 124 3,053 53 0.3 1.7 5.6667 1.73467G 14,307 226 3,852 236 1.6 6.1 3.8125 1.338298H 20,719 736 3,351 179 3.6 5.3 1.4722 0.386779I 5,339 9 2,754 84 0.2 3.1 15.5 2.7408410J 8,374 78 2,981 55 0.9 1.8 2 0.6931511K 6,997 101 1,658 114 1.4 6.9 4.9286 1.5950512L 12,686 26 3,228 125 0.2 3.8 19 2.9444413M 4,160 40 1,926 222 1.0 11.5 11.5 2.4423514N 5,944 29 2,487 106 0.5 4.3 8.6 2.1517615O 7,181 112 2,897 130 1.6 4.5 2.8125 1.0340716P 8,532 203 4,060 181 2.4 4.5 1.875 0.6286117Q 6,640 36 2,608 74 0.5 2.8 5.6 1.722778Table 4. Test at 99 Percent Confidence Level: The TTEST ProcedureStatisticsVariableNLowerCL MeanMeanUpper CLMeanLower CLStd DevStd DevUpper CLStd DevStd ErrMinimumMaximumlogratio 17 0.9498 1.5363 2.1229 0.5658 0.828 1.4606 0.2008 0.1542 2.9444T-TestsVariableDFT ValuePr > |t|logratio 16 2.67 0.0167