Date: October 30, 2007 

To: Karen Rackley, EPA/SPPD 

From: David Randall and Breda Munoz 

Subject: Statistical Analysis of Valve Leak Frequencies Obtained by Industry and EPA for 

Petroleum Refining Process Units 

I. Introduction 

The U.S. Environmental Protection Agency (EPA) is currently reviewing the new source 

performance standards for equipment leaks of VOC in the Synthetic Organic Chemicals 

Manufacturing Industry and in the Petroleum Refining Industry (40 CFR part 60, subparts VV 

and GGG). In order to estimate the impacts of regulatory alternatives, the emission rates under 

both the existing requirements and the regulatory alternative must be determined. Calculating 

the emission rates requires an estimate of the percentage of equipment that is leaking. Industry 

trade associations have provided leak frequency data for numerous process units that could be 

used directly in these impacts analyses. However, the EPA’s National Enforcement 

Investigations Center (NEIC) also conducts monitoring at some of the facilities subject to these 

rules. In 1999, the NEIC published the results of a comparative monitoring study at 17 

petroleum refineries.1

 These results showed the percentage of valves identified as leaking by 

NEIC was always higher than the results of monitoring conducted by the petroleum refiners, and 

the results differed by as much as a factor of 10. RTI has conducted an analysis of these results 

showing there is a statistically significant difference between the leak frequencies identified by 

NEIC and the petroleum refiners, and the NEIC results are higher than the industry results by a 

factor of at least 2.6 at the 99 percent confidence interval. The purpose of this memorandum is 

to describe the statistical analysis used to develop these results. 

II. Background 

Table 1 shows the results of the industry and NEIC monitoring programs at the 17 

refineries.1

 The leak frequencies determined by the companies ranged from 0.2 percent to 3.6 

percent, and the leak frequencies determined by NEIC ranged from 1.4 percent to 11.5 percent. 

In the report of their study, NEIC theorizes that the disparity between the NEIC and company 

results may be attributable to refineries not monitoring in the manner prescribed in Method 21 of 

40 CFR part 60, Appendix A-7. 

The data for each of the 17 facilities are for some of the process units that are subject to 

equipment leak standards and that NEIC pre-selected for monitoring as part of their inspection. 

The number of valves monitored by the companies should be the total number of valves that are 
 

2

subject to monitoring in the selected process units during the most recent monitoring period. 

Based on information from the industry trade association, it is likely that most of these refiners 

are conducting monitoring quarterly, but it is possible that some have implemented an option that 

allows less frequent monitoring when low leak frequencies are maintained. All of the valves 

within a given process must be monitored within a calendar month under either requirement (but 

not all processes would have to be monitored in the same month). Typically, NEIC tries to 

monitor about half of the valves in the selected process units, but less than half were monitored if 

the selected process units have a large number of valves. When monitoring, the NEIC inspectors 

rely on the company representatives to point out valves to monitor until they have reached the 

target number. With the exception of excluding most inaccessible valves, the valves that are 

monitored are selected as randomly as the company representatives pick them out. NEIC makes 

no attempt to pick out certain types of valves or valves that are most likely to leak because of 

process conditions, age, or other factors. The companies and NEIC both follow Method 21 when 

monitoring. 

III. Statistical Analysis 

A. Hypotheses

The hypotheses tested by this study were: 

1. Is the NEIC leak rate equal to the company leak rate at each of the 17 refineries 

(which is equivalent to testing if the difference between the two leak rates is equal to 

zero), and 

2. Is the mean of the NEIC-to-company leak frequencies ratio equal to one? This 

hypothesis will allow us to assess the average difference between the two mean leak 

frequencies. 

B. Methods and Results

 Seventeen one-sample t-tests were conducted to test the first hypothesis. If we assume 

that the valves monitored by NEIC were assumed to be random samples of the valves monitored 

by each company based on the discussion in Section II and that the reported refinery leak rate is 

the population value (fixed value), then to assess hypothesis 1 we can perform the standard onesample test for equality of proportions (or t-test). There is a close relationship between 

confidence intervals and hypothesis testing. When a 95% confidence interval is constructed, all 

values in the interval are considered plausible values for the parameter being estimated. Values 

outside the interval are rejected as relatively implausible. If the value of the parameter specified 

by the null hypothesis is contained in the 95% interval, then the null hypothesis cannot be 

rejected at the 0.05 level. If the value specified by the null hypothesis is not in the interval then 

the null hypothesis can be rejected at the 0.05 level. The following equation was used to 

calculate the upper and lower confidence limits of the differences (at the 95 percent significance 

level): 
 

3

n

p p p p t df

ˆ(1 ˆ)

0 / 2,

− − ± α

)

where: 

α = the significance level (95% significance level) 

 df = the degrees of freedom, which are calculated as n-1 

 n = the number of valves selected by NEIC for inspection 

df tα / 2, = the tabular value from the t-distribution with df degrees of freedom 

pˆ = the NEIC leak ratio: 

NEIC Number Valves

NEIC Number Leak 

pˆ = and 

 P0 = the company leak ratio (which is assumed fixed): 

 Company Number Valves

Company Number Leak

p0 =

 Results of the calculations are presented in Table 2. These results show the confidence 

intervals are above zero for 16 of the 17 companies, which indicates the NEIC leak frequencies 

are different from the leak frequencies obtained by these companies. The only company for 

which the hypothesis of no difference in the leak frequencies cannot be rejected is Company D. 

The confidence interval for this Company includes zero, which suggests the leak frequencies 

obtained by NEIC and this company are not different at the 95 percent significance level. 

 For the second hypothesis, we consider the fact that we have two measurements for each 

of the 17 refineries. This is known in the statistics literature as paired designs. One approach to 

analyze these data is to calculate the ratio of the two pairs of measurements and transform the 

problem to a one-sample testing if the mean of the ratios equals 1. To account for the deviation 

of the normality assumption, we log-transformed the data so the distribution of the sample 

resembles normal distributed. We tested the second hypothesis using a one-sample t-test. The 

original ratios and the transformed ratios are shown in Table 3. The following equation was used 

to calculate the upper and lower bounds of the mean of the transformed ratios: 

( ) log( ) [ (log( ))] / 2, Mean ratio t SE Mean ratio ± α df

where: 

Mean(log(ratio)) = the mean of the log-transformed ratios, 

 SE[Mean(log(ratio)) = the standard error of the of the mean of the log-transformed 

ratios, 

α = the significance level (99 percent significance level), 

df = the degrees of freedom (16 degrees of freedom), and 

 tα/2,df = the tabular value from the t-distribution with df degrees of 

freedom 
 

4

 Table 4 shows the resulting statistics for the 99 percent significance level. The lower and 

upper bounds of the transformed mean ratio are 0.95 and 2.1, respectively. Back transforming 

(exponentiating) gives the lower and upper bounds of the ratio as 2.6 and 8.4, respectively. 

Because this range does not include 1, the null hypothesis of equal leak frequencies is rejected. 

Furthermore, these results suggest that the NEIC leak frequencies are at least 2.6 times greater 

than the Company leak frequencies at the 99 percent confidence level. 

IV. References 

1. U.S. EPA, Office of Enforcement and Compliance Assurance. October 1999. 

Enforcement Alert: Proper Monitoring Essential to Reducing 'Fugitive Emissions' Under 

Leak Detection and Repair Programs. 
5 

Table 1. Results of NEIC Comparative Monitoring Study 

Refinery Company Monitoring: NEIC Monitoring Leak Rate (%) 

Valves Leaks Valves Leaks Company NEIC 

A 7,694 170 3,363 354 2.3 10.5 

B 7,879 223 3,407 216 2.8 6.3 

C 3,913 22 2,008 108 0.6 5.4 

D 2,229 26 1,784 24 1.2 1.4 

E 5,555 96 2,109 112 0.7 5.3 

F 42,505 124 3,053 53 0.3 1.7 

G 14,307 226 3,852 236 1.6 6.1 

H 20,719 736 3,351 179 3.6 5.3 

I 5,339 9 2,754 84 0.2 3.1 

J 8,374 78 2,981 55 0.9 1.8 

K 6,997 101 1,658 114 1.4 6.9 

L 12,686 26 3,228 125 0.2 3.8 

M 4,160 40 1,926 222 1.0 11.5 

N 5,944 29 2,487 106 0.5 4.3 

O 7,181 112 2,897 130 1.6 4.5 

P 8,532 203 4,060 181 2.4 4.5 

Q 6,640 36 2,608 74 0.5 2.8 

Total 170,717 2,266 47,526 2,372 1.3 (avg) 5.0 (avg) 
6 

Table 2. Confidence Intervals for the Difference Between NEIC Leak Rate and 

Company Leak Rate

Refinery 

Company 

Leak Rate 

(%) 

(1) 

NEIC 

Leak Rate 

(%) 

(2) 

NEIC Leak 

Rate minus 

Company 

Leak Rate 

(2)-(1) 

95% Lower 

Confidence 

Limit for 

(2)-(1) 

95% Upper 

Confidence 

Limit for 

(2)-(1) 

A 2.3 10.5 8.2 7.164 9.236 

B 2.8 6.3 3.5 2.684 4.316 

C 0.6 5.4 4.8 3.811 5.789 

D 1.2 1.4 0.2 -0.345 0.745 

E 0.7 5.3 4.6 3.644 5.556 

F 0.3 1.7 1.4 0.941 1.859 

G 1.6 6.1 4.5 3.744 5.256 

H 3.6 5.3 1.7 0.941 2.459 

I 0.2 3.1 2.9 2.253 3.547 

J 0.9 1.8 0.9 0.423 1.377 

K 1.4 6.9 5.5 4.280 6.720 

L 0.2 3.8 3.6 2.940 4.260 

M 1 11.5 10.5 9.075 11.925 

N 0.5 4.3 3.8 3.003 4.597 

O 1.6 4.5 2.9 2.145 3.655 

P 2.4 4.5 2.1 1.462 2.738 

Q 0.5 2.8 2.3 1.667 2.933 
7 

Table 3. Actual and Transformed Ratios of the NEIC Leak Rate to the Company Leak R

Company Monitoring: NEIC Monitoring Leak Rate (%) 

Observation Refinery Valves Leaks Valves Leaks Company NEIC

Ratio = N

leak Ra

Refinery 

Rate 

1 A 7,694 170 3,363 354 2.3 10.5 4.5652 B 7,879 223 3,407 216 2.8 6.3 2.25 3 C 3,913 22 2,008 108 0.6 5.4 9 4 D 2,229 26 1,784 24 1.2 1.4 1.1665 E 5,555 96 2,109 112 0.7 5.3 7.5716 F 42,505 124 3,053 53 0.3 1.7 5.6667 G 14,307 226 3,852 236 1.6 6.1 3.8128 H 20,719 736 3,351 179 3.6 5.3 1.4729 I 5,339 9 2,754 84 0.2 3.1 15.5 

10 J 8,374 78 2,981 55 0.9 1.8 2 11 K 6,997 101 1,658 114 1.4 6.9 4.92812 L 12,686 26 3,228 125 0.2 3.8 19 13 M 4,160 40 1,926 222 1.0 11.5 11.5 14 N 5,944 29 2,487 106 0.5 4.3 8.6 15 O 7,181 112 2,897 130 1.6 4.5 2.81216 P 8,532 203 4,060 181 2.4 4.5 1.875 17 Q 6,640 36 2,608 74 0.5 2.8 5.6 
8

Table 4. Test at 99 Percent Confidence Level: The TTEST Procedure 

Statistics 

Variable N 

Lower 

CL Mean Mean 

Upper CL 

Mean 

Lower CL 

Std Dev Std Dev 

Upper CL 

Std Dev Std Err M

logratio 17 0.9498 1.5363 2.1229 0.5658 0.828 1.4606 0.2008 0

T-Tests 

Variable DF T Value Pr > |t| 

logratio 16 2.67 0.0167