1 

## EPA Wood Heater Test Method Variability Study Analysis of Uncertainty, Repeatability and Reproducibility based on the EPA Accredited Laboratory Proficiency Test Database

Rick Curkeet, PE Chief Engineer – Hearth Products Intertek Testing Services Robert Ferguson Ferguson, Andors & Company2



# Introduction

• The precision of any test method is an extremely important tool in understanding the limitations of the data generated from the method. • No rigorous assessment of the precision of the test methods contained in the woodstove NSPS has ever been performed. • A large data base of proficiency test data from accredited laboratories has been assembled. • The objective of this paper is to perform that assessment, using the EPA proficiency test data base and broadly accepted tools for assessing repeatability and reproducibility.3



# EPA Test Methods

• **EPA Method 28**  – Certification and Auditing of Wood Heaters • **EPA Method 5H**  – Determination of Particulate Matter Emissions from Wood Heaters from a Stack Location • **EPA Method 5G**  **- 1**  - Determination of Particulate Matter Emissions from Wood Heaters (Dilution Tunnel Sampling Location), Single Filter Train with Dual 100mm Filters (filters at near ambient temperature) • **EPA Method 5G**  **- 2**  - Determination of Particulate Matter Emissions from Wood Heaters (Dilution Tunnel Sampling Location), Method 5H Filter Train (heated front filter with chilled impinger train) • **EPA Method 5G**  **- 3**  - Determination of Particulate Matter Emissions from Wood Heaters (Dilution Tunnel Sampling Location), Dual Filter Trains with Dual 47 mm Filters (filters at near ambient temperature)4 

## • All measurement processes have an inherent element of variability in the measurement result. • Each measurement that is made as part of the test procedure, and ultimately combined mathematically to produce a result, contributes to the overall uncertainty of that result. The more measurements made, the greater the overall uncertainty. • This variability can be evaluated empirically by performing multiple measurements of a single artifact and evaluating the mean and standard deviation of the test results. 

# Test Method Variability5 

## • The relative magnitude of the standard deviation to the mean is a measure of the dispersion (variability) of the measurement process. • This measure of dispersion includes variability that originates both from the measurement processes and from the inherent variability of the phenomenon being measured. • When the measurement involves performance of a process which might be affected by uncontrolled variables or by the ranges allowed for operational parameters within the process, the variability associated with the process is usually much greater than that that arises from the uncertainty of the actual measurements made.6 

# The NORMAL Distribution

NO RM AL DI STRI BUTI ON

-4 -3 -2 -1 0 1 2 3 4 STAN DARD DEVI ATI ON S 95% MEAN

68%7



# Terminology

• The closeness of agreement between independent test results obtained under stipulated conditions is the “ **precision** ” of the measurement. • Precision is usually broken down further into:

– **Repeatability** – The closeness of multiple measurements of the same artifact under the same conditions with the same equipment and operator. The symbol “ **r**” is often used to refer to repeatability. Repeatability can be evaluated by replicate tests of the same product at the same laboratory following the defined procedure.

– **Reproducibility** – The closeness of measurements made on the same or presumed to be identical artifacts by different laboratories, equipment and operators. The symbol “ **R** ” is often used for reproducibility. Reproducibility can be evaluated for any quantitative measurement system through an Interlaboratory Study or round robin test program.8 

# Quantifying **Test Method Precision**

• The primary data set available to evaluate test result variability is that generated through the EPA required Accredited Laboratory Proficiency Test Program.

– A total of 14 laboratories have participated over the years (to varying degrees).

– 5 individual test heaters have been used in the proficiency test program.910 

# • This database can be analyzed using standard statistical tools.

• This database can be further analyzed using advanced statistical procedures specifically intended to assess test method variability (ASTM E691).11 

## “Macro” Analysis Using Standard Statistical Tools12 

## • Since there is a wide range in the weighted average emissions performance for each stove in the database, both the standard deviation and the coefficient of variation (CV) were computed for each stove to avoid any potentially misleading assessments of variability that might result from the wide range in the calculated means for the individual test stoves.

– Note: The coefficient of variation represents the ratio of the standard deviation to the mean, and it is a useful statistic for comparing the degree of variation from one data series to another, even if the means are drastically different from each other.13 

# • It is apparent that the repeatability and reproducibility of the test procedures are not very good. • Even without applying any statistical tools, one can simply look at the results of testing the same stove in the same laboratory over two or more years to see the range of intra - laboratory results.14 

# • Using the results from all the laboratories for all the stoves over all the years of the program, it is possible to assess the interlaboratory precision.

– Using 2.8 times the standard deviation to estimate the potential range of results that could be expected with a 95% confidence level [Ref.: ASTM E177, 28.1], it is clear that the for any given test series the reproducibility is on the order of ± 4.9 – 9.8 grams per hour.15 

# • This means that for any emissions rate measured using the EPA test methods, the result could be 4.9 to 9.8 grams per hour higher or lower if the appliance were tested again at a different laboratory. • Even at one standard deviation (68% confidence level), the reproducibility interval is about 1.7 to 3.5 grams per hour, i.e., there is a 32% chance that a new test result would deviate by more than this amount.16 

## • The CV calculations further confirm that variability is high (>40 % CV) for the three test stoves with significant data sets but it is not consistent for all units. • Since CV is calculated from one standard deviation, the implied variability in the actual data is really about 2.8 times the CV at a 95% confidence level. For example, a CV of 40% implies that the results of any one test could vary by +/- 112% from a population mean with a probability of 5% or less of being farther away.17



# ASTM E177 3.1.12 repeatability limit (r) , n—the value below which the absolute difference between two individual test results obtained under repeatability conditions may be expected to occur with a probability of approximately 0.95 (95 %). 3.1.12.1 Discussion—The repeatability limit is 2.8 (1.96 x √2 ) times the repeatability standard deviation. This multiplier is independent of the size of the interlaboratory study. 3.1.12.2 Discussion—The approximation to 0.95 is reasonably good (say 0.90 to 0.98) when many laboratories (30 or more) are involved, but is likely to be poor when fewer than eight laboratories are studied.18 

# Precision Analysis Using ASTM E691

• Under the presumption of using consensus standards, we determined that ASTM E691 – Standard Practice for Conducting an Interlaboratory Study to Determine the Precision of a Test Method was appropriate for use here.19 

## • ASTM E691 provides procedures for planning and conducting studies of the precision of test methods, and also provides statistical techniques for performing precision analyses. • Since the EPA proficiency test program has been on - going for more than 20 years, the test program design components of ASTM E691 are not germane here. • However, the analytical procedures can be applied to the EPA proficiency test data for the purposes of determining precision (both r & R).20 

# • Separate tables were created for each of the different stoves that were tested over the years the proficiency test program was conducted where there is sufficient data to apply the procedures specified in ASTM E691.21222324 

# Conclusions from ASTM E691 Analyses

• The ASTM E691 analyses shows that the repeatability (within lab), **r** , of the weighted average emissions rate determined by the EPA test methods at the 95% confidence level is at best ± 2.9 grams per hour and typically about ± 3.5 to ± 5.4 grams per hour. The reproducibility (between labs), **R** , at the 95% confidence level is ± 4.5 to ± 6.4 grams per hour.25 

# Sources of Variability in EPA Wood

**Heater Testing**
• Given that 12 labs (five are still accredited) and all four particulate measurement methods are represented, it is important to try to understand if the repeatability and reproducibility issues are laboratory and method related or arise from a different source – that is, variable performance of the appliance itself.26 

# • To evaluate that question, we next investigated the potential sources of variability in the test methods.

– We started with the emissions measurement methods (5G - 1, 5G - 2, 5G - 3 and 5H), and then addressed Method 28, which specifies how the appliance is to be operated during emission testing.27 

# Emission Measurement Methods Method 5G - 1 • Potential sources of variability in Method 5G - 1 include the following:

– Accuracy of dilution tunnel gas flow and sample flow measurement and the resulting consistency of proportionality.

– Recovery of particulates from the probe and front half of filter holder using a solvent cleaner.

– Weighing errors for tare and final weights of filters and evaporation containers used to collect and weigh the catch. 

### • These sources combine to produce an estimated measurement uncertainty of about ± 2.7 to 3% of the emission value measured for a typical passing wood stove test.28 

### Method 5G - 2 • Potential sources of variability in Method 5G - 2 include the following:

– Accuracy of dilution tunnel gas flow and sample flow measurement and the resulting consistency of proportionality.

– Recovery of particulates from the probe and sampling line using a solvent cleaner.

– Recovery of particulate collected in the impingers and connecting glassware.

– Weighing errors for tare and final weights of filters and evaporation containers used to collect and weigh the catch. 

### • These sources combine to produce an estimated measurement uncertainty of about ± 3 – 3.5% of the emission value measured for a typical passing wood stove test.29 

## Method 5G - 3 • Potential sources of variability in Method 5G - 3 include the following:

– Accuracy of dilution tunnel gas flow and sample flow measurement and the resulting consistency of proportionality.

– Weighing errors for weights of filters, probes including front filter housings and filter seals. 

## • The measurement uncertainty of method 5G - 3 has been determined to be approximately ± 2.5% of the emission value measured for a typical passing wood stove test.30 

### Method 5H • Potential sources of variability in Method 5H include the following:

– Accuracy of flue gas flow and sample flow measurement and the resulting consistency of proportionality.

– Recovery of particulates from the probe and sampling line using a solvent cleaner.

– Recovery of particulate collected in the impingers and connecting glassware.

– Weighing errors for tare and final weights of filters and evaporation containers used to collect and weigh the catch. 

### • These sources combine to produce an estimated measurement uncertainty of about ± 20 to 30% of the total particulate mass determination. • The major contributor to MU is the difficulty in measurement of low and variable stack flow rates.31 

## Conclusions Regarding the Contribution of Emissions Measurement Method Uncertainty to

Overall Method Variability 

# • It is apparent that the emission measurement methods explain only a small part of the overall variability shown by the precision analyses.

– It should be noted that Method 5G - 3 showed lower variability in the analyses than the other emission measurement methods.32 

### Operating Protocol for the Appliance (EPA Method 28) • Potential Sources of Variability in Method 28 include the following:

– Fuel density variation from approximately 30 to 40 lb/ft3 (dry weight basis).

– Fuel moisture content variation from 19 to 25% dry basis (varies in uniformity as well as average).

– Fuel load configuration details.

– Coal bed size (20 - 25% of fuel load weight) and pre - burn temperature conditions.

– Loading time and start - up procedure.

– Ambient temperature, barometric pressure and humidity.

– Variations in control settings and resulting burn rates.

– Random uncontrollable variables such as when and how the fuel load settles, falls and collapses.33 

# • Using the EPA proficiency test data, it is possible to examine several of the various Method 28 parameters for their impact on precision.

– Impacts of burn rate variations. • Emissions performance is related to burn rate to some degree.

– Higher burn rates generally produce more complete combustion.34 

# • The proficiency test data does not lend itself well to determining the impact of burn rate variability over the full burn rate range of the heater in that each lab must adjust the air control(s) to a setting that will produce a burn rate in each of the required burn rate categories. • The exception is the high burn rate (Category 4) which is always run with the air control(s) fully open.3536



# Burn Rates

• The high burn rate variability is substantial, and it is certain that this variability is a contributing factor in the emissions variability. • There are many instances in the proficiency test data where two runs in one lab at very similar burn rates produced substantially different emissions rates. So burn rate variability does not explain much of the overall variability of emissions rates.37 

## – Impacts of fuel moisture variations

– Impacts of fuel density variations • The following charts show the relationship between emissions and fuel moisture content and emissions and load weight (assumed to relate to fuel density) for the 121 runs conducted on the catalytic proficiency test stove in 1987 and 1988. Using the “Corner Score” statistical test where a score of >11 indicates potential dependency between variables [Ref.: Mark’s Standard Handbook for Mechanical Engineers, 17 - 22], it is clear that there is no relationship.38 **1987 &1**  **988 Ca**  **t S to** **ve 1**  **- % M.C**  **. v E**  **mi ssions** 

y = -0 .1 496x + 7 .4 864 R 2= 0 .0 048 0 2 4 6 8 10 12 14 16 18 20

15 17 19 21 23 25 **%M .C**  **. D ry** **B asi**  **s E** **m s** **s i** **o n** **s (** **g /** **h )**  Em is si on s Ra te Medi anMC Medi anE mi ssi on s ra te Li ne ar (E mi ssi on s Ra te)

Corner Score +439 **1987 &1**  **988 Cat** 

**S to**  **ve 1**  **- L oad**  **Wt. v E**  **mis si**  **ons**

y = -0 .2 276x + 7 .3 68 R 2= 0 .0 048
0 2 4 6 8 10 12 14 16 18 20

9 10 11 12 13 14 15 16 **Load Wei**  **ght**

**(lb s)** **E m** **i s** **s i** **o n** **s (** **g /** **h )**  Emis sio ns Rat e Me d. E Me d. L aod Wt . Li near (E missio ns Rat e)

Corner Score +240 

### • By practical necessity, EPA Method 28 contains tolerances on operational and fueling parameters. These include fuel density, moisture content, laboratory ambient temperature, coal bed size, fuel load weight and several others. While it might be supposed that the method variability could be lessened by even tighter specifications within the test method relating to fuel and operating conditions, the data does not support this proposition. • Without these tolerances or specified ranges wood heater emission testing would become prohibitively expensive with significant additional costs incurred while attempting to control these variables within a significantly narrower range than specified. As demonstrated, these additional costs would not significantly reduce variability.41



# Final Conclusions

• The repeatability and reproducibility of wood heater emission testing as demonstrated by the EPA accredited laboratory proficiency test data is quite poor.

– At the 95% confidence level, repeatability for the EPA weighted average emission rate is at best ± 2.9 g/hour and ranged as high as ± 5.4 g/hour.

– The reproducibility was no better than ± 4.5 g/hour and ranged as high as ± 6.4 g/hour.42



# Final Conclusions **Em issions r** 

**& R** **- B**  **es t**  **Ca se** 

0 2 4 6 8 10 12 14 16 18 20

0. 5 1. 5 2. 5 3. 5 4. 5 5. 5 6. 5 7. 5 8. 5 9. 5 10 .5 11 .5 12 .5 13 .5 **Me an**  **E mis** 

**si ons**  **R at** **e (g**  **/h )** **9 5** **% C** **o n** **f i** **d e** **n c** **e L** **i m** **i t** **s** In tr al ab r Me an Emis si on g/ hr In te rl ab R43



# Ty pical Distribution44 

# • Further analyses of the potential sources of variability have shown that the emission measurement test methods (EPA Methods 5G - 1, 5G - 2, 5G - 3 and 5H) are not major contributors to the high overall variability being demonstrated by the data.

– However, Method 5G - 3 does demonstrate the lowest uncertainty of the four methods currently specified in the NSPS.45 

# • Analyses of the variability inherent in EPA Method 28 also indicate that the operational and fuel parameter tolerance ranges specified in that method are not major contributors to the high variability.

– Tightening these parameters to improve test precision would simply increase costs and the data show that such tightening would not significantly improve precision.46 

# • This leaves the only logical conclusion.

– Variability in wood heater emission testing results for any given appliance is most likely a function of the random nature of burning wood, no matter how tightly you try to control the process. • Many relatively small, uncontrollable variables that are inherent in the wood combustion process can combine to significantly affect the outcome of any given test.47 

## • These conclusions should not be taken to mean that the current test methodology is inadequate to characterize an appliance’s emissions performance.

– There is no question that the currently certified low emissions wood stoves produce far less particulate emissions than their pre - regulation predecessors which produced emissions rates of over 50 to 100 grams per hour.

– Our primary conclusion is that the current testing process simply cannot reliably distinguish emissions performance differences of less than 4 to 6 grams per hour. The process is certainly capable of reliably distinguishing between good and bad performance, but it cannot reliably distinguish between “good, better and best” performance.