1

EPA Wood Heater Test Method Variability Study

Analysis of Uncertainty, Repeatability and 

Reproducibility based on the EPA Accredited 

Laboratory Proficiency Test Database

Rick Curkeet, PE

Chief Engineer – Hearth Products

Intertek Testing Services

Robert Ferguson

Ferguson, Andors & Company
2

Introduction

• The precision of any test method is an extremely 

important tool in understanding the limitations of 

the data generated from the method.

• No rigorous assessment of the precision of the 

test methods contained in the woodstove NSPS 

has ever been performed.

• A large data base of proficiency test data from 

accredited laboratories has been assembled.

• The objective of this paper is to perform that 

assessment, using the EPA proficiency test data 

base and broadly accepted tools for assessing 

repeatability and reproducibility.
3

EPA Test Methods

•EPA Method 28 – Certification and Auditing of Wood Heaters

•EPA Method 5H – Determination of Particulate Matter Emissions from 

Wood Heaters from a Stack Location

•EPA Method 5G-1 - Determination of Particulate Matter Emissions 

from Wood Heaters (Dilution Tunnel Sampling Location), Single Filter 

Train with Dual 100mm Filters (filters at near ambient temperature)

•EPA Method 5G-2 - Determination of Particulate Matter Emissions 

from Wood Heaters (Dilution Tunnel Sampling Location), Method 5H 

Filter Train (heated front filter with chilled impinger train)

•EPA Method 5G-3 - Determination of Particulate Matter Emissions 

from Wood Heaters (Dilution Tunnel Sampling Location), Dual Filter 

Trains with Dual 47 mm Filters (filters at near ambient temperature)
4

• All measurement processes have an inherent 

element of variability in the measurement result.

• Each measurement that is made as part of the 

test procedure, and ultimately combined 

mathematically to produce a result, contributes 

to the overall uncertainty of that result. The 

more measurements made, the greater the 

overall uncertainty. 

• This variability can be evaluated empirically by 

performing multiple measurements of a single 

artifact and evaluating the mean and standard 

deviation of the test results. 

Test Method Variability
5

• The relative magnitude of the standard deviation 

to the mean is a measure of the dispersion 

(variability) of the measurement process. 

• This measure of dispersion includes variability 

that originates both from the measurement 

processes and from the inherent variability of the 

phenomenon being measured. 

• When the measurement involves performance of 

a process which might be affected by 

uncontrolled variables or by the ranges allowed 

for operational parameters within the process, 

the variability associated with the process is 

usually much greater than that that arises from 

the uncertainty of the actual measurements 

made. 
6

The NORMAL Distribution

NORMAL DISTRIBUTION

-4 -3 -2 -1 0 1 2 3 4

STANDARD DEVIATIONS

95%

MEAN

68%
7

Terminology

• The closeness of agreement between independent test 

results obtained under stipulated conditions is the 

“precision” of the measurement. 

• Precision is usually broken down further into:

– Repeatability – The closeness of multiple measurements of the 

same artifact under the same conditions with the same 

equipment and operator. The symbol “r” is often used to refer to 

repeatability. Repeatability can be evaluated by replicate tests of 

the same product at the same laboratory following the defined 

procedure.

– Reproducibility – The closeness of measurements made on the 

same or presumed to be identical artifacts by different 

laboratories, equipment and operators. The symbol “R” is often 

used for reproducibility. Reproducibility can be evaluated for any 

quantitative measurement system through an Interlaboratory 

Study or round robin test program.
8

Quantifying Test Method Precision

• The primary data set available to evaluate 

test result variability is that generated 

through the EPA required Accredited 

Laboratory Proficiency Test Program. 

– A total of 14 laboratories have participated 

over the years (to varying degrees).

– 5 individual test heaters have been used in 

the proficiency test program.
9
10

• This database can be analyzed using 

standard statistical tools.

• This database can be further analyzed 

using advanced statistical procedures 

specifically intended to assess test method 

variability (ASTM E691).
11

“Macro” Analysis Using Standard Statistical Tools
12

• Since there is a wide range in the weighted 

average emissions performance for each stove 

in the database, both the standard deviation and 

the coefficient of variation (CV) were computed 

for each stove to avoid any potentially 

misleading assessments of variability that might 

result from the wide range in the calculated 

means for the individual test stoves. 

– Note: The coefficient of variation represents the ratio 

of the standard deviation to the mean, and it is a 

useful statistic for comparing the degree of variation 

from one data series to another, even if the means 

are drastically different from each other. 
13

• It is apparent that the repeatability and 

reproducibility of the test procedures are 

not very good. 

• Even without applying any statistical tools, 

one can simply look at the results of 

testing the same stove in the same 

laboratory over two or more years to see 

the range of intra-laboratory results. 
14

• Using the results from all the laboratories 

for all the stoves over all the years of the 

program, it is possible to assess the interlaboratory precision.

– Using 2.8 times the standard deviation to 

estimate the potential range of results that 

could be expected with a 95% confidence 

level [Ref.: ASTM E177, 28.1], it is clear that 

the for any given test series the reproducibility 

is on the order of ± 4.9 – 9.8 grams per hour. 
15

• This means that for any emissions rate 

measured using the EPA test methods, 

the result could be 4.9 to 9.8 grams per 

hour higher or lower if the appliance were 

tested again at a different laboratory. 

• Even at one standard deviation (68% 

confidence level), the reproducibility 

interval is about 1.7 to 3.5 grams per hour, 

i.e., there is a 32% chance that a new test 

result would deviate by more than this 

amount. 
16

• The CV calculations further confirm that 

variability is high (>40 % CV) for the three test 

stoves with significant data sets but it is not 

consistent for all units. 

• Since CV is calculated from one standard 

deviation, the implied variability in the actual 

data is really about 2.8 times the CV at a 95% 

confidence level. For example, a CV of 40% 

implies that the results of any one test could vary 

by +/- 112% from a population mean with a 

probability of 5% or less of being farther away.
17

ASTM E177

3.1.12 repeatability limit (r), n—the value below which the 

absolute difference between two individual test results 

obtained under repeatability conditions may be expected to 

occur with a probability of approximately 0.95 (95 %).

3.1.12.1 Discussion—The repeatability limit is 2.8 (1.96 x√2 ) 

times the repeatability standard deviation. This multiplier is 

independent of the size of the interlaboratory study.

3.1.12.2 Discussion—The approximation to 0.95 is reasonably 

good (say 0.90 to 0.98) when many laboratories (30 or more) 

are involved, but is likely to be poor when fewer than eight 

laboratories are studied.
18

Precision Analysis Using ASTM E691

• Under the presumption of using 

consensus standards, we determined that 

ASTM E691 – Standard Practice for 

Conducting an Interlaboratory Study to 

Determine the Precision of a Test Method

was appropriate for use here. 
19

• ASTM E691 provides procedures for planning 

and conducting studies of the precision of test 

methods, and also provides statistical 

techniques for performing precision analyses. 

• Since the EPA proficiency test program has 

been on-going for more than 20 years, the test 

program design components of ASTM E691 are 

not germane here. 

• However, the analytical procedures can be 

applied to the EPA proficiency test data for the 

purposes of determining precision (both r & R). 
20

• Separate tables were created for each of 

the different stoves that were tested over 

the years the proficiency test program was 

conducted where there is sufficient data to 

apply the procedures specified in ASTM 

E691. 
21
22
23
24

Conclusions from ASTM E691 Analyses

• The ASTM E691 analyses shows that the 

repeatability (within lab), r, of the weighted 

average emissions rate determined by the 

EPA test methods at the 95% confidence 

level is at best ±2.9 grams per hour and 

typically about ±3.5 to ±5.4 grams per 

hour. The reproducibility (between labs), 

R, at the 95% confidence level is ±4.5 to 

±6.4 grams per hour. 
25

Sources of Variability in EPA Wood 

Heater Testing

• Given that 12 labs (five are still accredited) 

and all four particulate measurement 

methods are represented, it is important to 

try to understand if the repeatability and 

reproducibility issues are laboratory and 

method related or arise from a different 

source – that is, variable performance of 

the appliance itself. 
26

• To evaluate that question, we next 

investigated the potential sources of 

variability in the test methods. 

– We started with the emissions measurement 

methods (5G-1, 5G-2, 5G-3 and 5H), and 

then addressed Method 28, which specifies 

how the appliance is to be operated during 

emission testing.
27

Emission Measurement Methods

Method 5G-1

• Potential sources of variability in Method 5G-1 include 

the following:

– Accuracy of dilution tunnel gas flow and sample flow 

measurement and the resulting consistency of proportionality.

– Recovery of particulates from the probe and front half of filter 

holder using a solvent cleaner.

– Weighing errors for tare and final weights of filters and 

evaporation containers used to collect and weigh the catch.

• These sources combine to produce an estimated 

measurement uncertainty of about ±2.7 to 3% of the 

emission value measured for a typical passing wood 

stove test. 
28

Method 5G-2 

• Potential sources of variability in Method 5G-2 include 

the following:

– Accuracy of dilution tunnel gas flow and sample flow 

measurement and the resulting consistency of proportionality.

– Recovery of particulates from the probe and sampling line using 

a solvent cleaner.

– Recovery of particulate collected in the impingers and 

connecting glassware.

– Weighing errors for tare and final weights of filters and 

evaporation containers used to collect and weigh the catch.

• These sources combine to produce an estimated 

measurement uncertainty of about ±3 – 3.5% of the 

emission value measured for a typical passing wood 

stove test. 
29

Method 5G-3

• Potential sources of variability in Method 5G-3 

include the following:

– Accuracy of dilution tunnel gas flow and sample flow 

measurement and the resulting consistency of 

proportionality. 

– Weighing errors for weights of filters, probes including 

front filter housings and filter seals.

• The measurement uncertainty of method 5G-3 

has been determined to be approximately ±

2.5% of the emission value measured for a 

typical passing wood stove test. 
30

Method 5H

• Potential sources of variability in Method 5H include the 

following:

– Accuracy of flue gas flow and sample flow measurement and the 

resulting consistency of proportionality.

– Recovery of particulates from the probe and sampling line using 

a solvent cleaner.

– Recovery of particulate collected in the impingers and 

connecting glassware.

– Weighing errors for tare and final weights of filters and 

evaporation containers used to collect and weigh the catch.

• These sources combine to produce an estimated 

measurement uncertainty of about ± 20 to 30% of the 

total particulate mass determination. 

• The major contributor to MU is the difficulty in 

measurement of low and variable stack flow rates.
31

Conclusions Regarding the Contribution of 

Emissions Measurement Method Uncertainty to 

Overall Method Variability

• It is apparent that the emission 

measurement methods explain only a 

small part of the overall variability shown 

by the precision analyses. 

– It should be noted that Method 5G-3 showed 

lower variability in the analyses than the other 

emission measurement methods. 
32

Operating Protocol for the Appliance (EPA Method 28)

• Potential Sources of Variability in Method 28 include the 

following:

– Fuel density variation from approximately 30 to 40 lb/ft3 (dry 

weight basis).

– Fuel moisture content variation from 19 to 25% dry basis (varies 

in uniformity as well as average).

– Fuel load configuration details.

– Coal bed size (20-25% of fuel load weight) and pre-burn 

temperature conditions.

– Loading time and start-up procedure.

– Ambient temperature, barometric pressure and humidity.

– Variations in control settings and resulting burn rates.

– Random uncontrollable variables such as when and how the fuel 

load settles, falls and collapses.
33

• Using the EPA proficiency test data, it is 

possible to examine several of the various 

Method 28 parameters for their impact on 

precision.

– Impacts of burn rate variations. 

• Emissions performance is related to burn rate to 

some degree. 

– Higher burn rates generally produce more complete 

combustion. 
34

• The proficiency test data does not lend 

itself well to determining the impact of burn 

rate variability over the full burn rate range 

of the heater in that each lab must adjust 

the air control(s) to a setting that will 

produce a burn rate in each of the required 

burn rate categories. 

• The exception is the high burn rate 

(Category 4) which is always run with the 

air control(s) fully open. 
35
36

Burn Rates

• The high burn rate variability is substantial, and 

it is certain that this variability is a contributing 

factor in the emissions variability.

• There are many instances in the proficiency test 

data where two runs in one lab at very similar 

burn rates produced substantially different 

emissions rates. So burn rate variability does not 

explain much of the overall variability of 

emissions rates.
37

– Impacts of fuel moisture variations

– Impacts of fuel density variations

• The following charts show the relationship between 

emissions and fuel moisture content and emissions 

and load weight (assumed to relate to fuel density) 

for the 121 runs conducted on the catalytic 

proficiency test stove in 1987 and 1988. Using the 

“Corner Score” statistical test where a score of >11 

indicates potential dependency between variables 

[Ref.: Mark’s Standard Handbook for Mechanical 

Engineers, 17-22], it is clear that there is no 

relationship. 
38

1987 &1988 Cat Stove 1 - % M.C. v Emissions

y = -0.1496x + 7.4864

R

2

 = 0.0048

0

2

4

6

8

10

12

14

16

18

20

15 17 19 21 23 25

%M.C. Dry Basis

E

m

s

sio

n

s (g/h)

Emissions Rate

Median MC

Median Emissions rate

Linear (Emissions Rate)

Corner Score +4
39

1987 &1988 Cat Stove 1 - Load Wt. v Emissions

y = -0.2276x + 7.368

R

2

 = 0.0048

0

2

4

6

8

10

12

14

16

18

20

9 10 11 12 13 14 15 16

Load Weight (lbs)

E

mis

sio

n

s (g/h)

Emissions Rate

Med. E

Med. Laod Wt.

Linear (Emissions

Rate)

Corner Score +2
40

• By practical necessity, EPA Method 28 contains 

tolerances on operational and fueling parameters. 

These include fuel density, moisture content, laboratory 

ambient temperature, coal bed size, fuel load weight and 

several others. While it might be supposed that the 

method variability could be lessened by even tighter 

specifications within the test method relating to fuel and 

operating conditions, the data does not support this 

proposition. 

• Without these tolerances or specified ranges wood 

heater emission testing would become prohibitively 

expensive with significant additional costs incurred while 

attempting to control these variables within a significantly 

narrower range than specified. As demonstrated, these 

additional costs would not significantly reduce variability.
41

Final Conclusions

• The repeatability and reproducibility of wood 

heater emission testing as demonstrated by the 

EPA accredited laboratory proficiency test data 

is quite poor. 

– At the 95% confidence level, repeatability for the EPA 

weighted average emission rate is at best ±2.9 g/hour 

and ranged as high as ±5.4 g/hour. 

– The reproducibility was no better than ±4.5 g/hour 

and ranged as high as ±6.4 g/hour.
42

Final Conclusions

Emissions r & R - Best Case

0

2

4

6

8

10

12

14

16

18

20

0.51.52.53.54.55.56.57.58.59.5

10.511.512.513.5

Mean Emissions Rate (g/h)

9

5

%

C

o

n

fid

e

n

c

e

Lim its

Intralab

r

Mean Emission g/hr

Interlab

R
43

Typical Distribution
44

• Further analyses of the potential sources 

of variability have shown that the emission 

measurement test methods (EPA Methods 

5G-1, 5G-2, 5G-3 and 5H) are not major 

contributors to the high overall variability 

being demonstrated by the data. 

– However, Method 5G-3 does demonstrate 

the lowest uncertainty of the four methods 

currently specified in the NSPS.
45

• Analyses of the variability inherent in EPA 

Method 28 also indicate that the 

operational and fuel parameter tolerance 

ranges specified in that method are not 

major contributors to the high variability. 

– Tightening these parameters to improve test 

precision would simply increase costs and the 

data show that such tightening would not 

significantly improve precision.
46

• This leaves the only logical conclusion. 

– Variability in wood heater emission testing 

results for any given appliance is most likely a 

function of the random nature of burning 

wood, no matter how tightly you try to control 

the process. 

• Many relatively small, uncontrollable variables that 

are inherent in the wood combustion process can 

combine to significantly affect the outcome of any 

given test.
47

• These conclusions should not be taken to mean 

that the current test methodology is inadequate 

to characterize an appliance’s emissions 

performance. 

– There is no question that the currently certified low 

emissions wood stoves produce far less particulate 

emissions than their pre-regulation predecessors 

which produced emissions rates of over 50 to 100 

grams per hour. 

– Our primary conclusion is that the current testing 

process simply cannot reliably distinguish emissions 

performance differences of less than 4 to 6 grams per 

hour. The process is certainly capable of reliably 

distinguishing between good and bad performance, 

but it cannot reliably distinguish between “good, better 

and best” performance.