PSCAA Preliminary Reanalysis of 

- Wood Heater Test Method Variability (Curkeet, 2010)

- A Comparison Between In-Home Use and EPA Certified 

Values (Houck, 2012)

These analyses argue that 1) test method variability is too great, 

and 2)real-world emissions are insufficiently correlated to the test 

method, to justify lowering the standard.

Can these conclusions be objectively supported? 

How robust are the analyses and results? 

Are other conclusions possible? 
Overview

• Anyway you slice it, real-world emissions (5-10 g/hour) of certified stoves 

is too high for even modest growth in wood burning in urban areas such as 

the Tacoma NAA. Replacing all uncertified with certified would not allow 

for any new stoves to enter the airshed, or at the very least, would return 

the airshed to nonattainment in 5-10 years.

• We have concerns that existing analyses have problems with data 

representativeness, data treatment, completeness of analysis, and 

appropriateness of statistical methods

• Nonetheless, NSPS and certification values should be more representative 

of real-word burning. This requires:

– Significantly more comprehensive and statistically robust 1) analysis of existing 

data and 2) a priori design for test lab procedures and real-world evaluations

– Wider swath (estimated 10x) of real-world use data. Okay to sacrifice precision 

of measurements if the representativeness can be significantly increased.

– Significantly better experimental design for real-world testing. Burn rates, fuel 

moistures, etc, need to be more consistently employed. Campaigns with data 

not uniformly covering the independent variables must be omitted.

– More sophisticated model of independent variables, interactions, and 

emissions
Examples from Test Method Variability 

(Curkeet)
Note: outlier had 5-year gap with previous tests, only 2 labs evaluated, 
Lab deviation seems consistent across 

all stoves and years
Underlying data are log-normal. Normal 

quantile on left, log-normal on right 

(quantile vs emission rate)
A quick reanalysis of Test Method 

Variability data

• std dev of each labs average performance across three 

stoves, 1987-2000 is 1.4 g/hr, from 2-6 tests per lab

• std dev of deviation of individual runs from stove-average 

is about 1.8 g/hr, appears consistent across certification 

rate and test lab rate

• four stoves mean values show good linearity wrt certified 

values

• implies labs can reproduce certified values, lower values 

may need a few more repetitions to reduce uncertainty

• suggests that: with about 15 stoves (tests) in a lower 

category and a mean difference of 2 g/hour, there’s a > 

95% chance the true difference is at least 1 g/hour
Example of concerns from: A Comparison of In-Home 

Emissions and EPA (Houck, 2012)

• Includes data from a range of studies over many years, in different locations, using 

inconsistent burn rates, fuel moistures, fuel types, burn rate measurements, etc. No 

indication or analysis that these are representative.

• A number of stoves have bi-modal emissions vs burn rate relationships. No clear sign that 

these behaviors are representative and accurate. Also can’t apply normal statistics (based on 

a normal distribution).

• Analysis needs to account for non-normal distributions and correct for outliers and spurious 

data.

• Most underlying data (individual stove results and population results) are log-normal. 

• Stoves #4, #11 had anomalously high std deviations and are the highest values in the low cert 

value category, likely have data that are not representative.

• For stove #4, one of three campaigns (“laboratory simulated in-home use”) had wildly high 

values, didn’t fit distribution, very likely are outliers.

• Other catalytic stoves (e.g. #6) show two different regimes, one with much higher emissions 

and low burn rates, likely not consistent with design and not representative.

• several stoves had only one or two tests (certification requires 4)

• several stoves have only data with burn rate < 0.9

• Skewed distribution of burn rates compared to Method 28. “Real-world” is biased toward low 

burn rates, which unrepresentatively weights bi-modal regime.
All “Real-World” Emission Rate data, 

quantile plots, normal and log-normal 

of emission rate (g/hour)
Most of the highest points are from Stove 4.

Stove 4 quantile plots: normal and log-normal of 

emission rates in g/hour
Burn rates of “Real-World” data don’t 

agree with Method 28
Several stoves, mostly in the lowest emissions bins, have “L” shaped emissions vs

burn rate relationships. They appear to be bi-modal which means normal statistics 

are not applicable and the stoves may be outliers or unrepresentative. Stove 4 high 

data is exclusively from “Laboratary simulated in home use” while real in-home use 

was much lower
most stoves have more uni-modal relationship 
Which one of these is not like the other? Low certification, 

medium certification, high certification:

Emission rate (g/hour) vs burn rate divided in bins following Houck
data in lowest bins (1-2 g/hour [left], 2-3 

g/hour[right]) are not well behaved: quantile plots of 

normal (top) and ln-normal (bottom) data
“Real-World” Emission Factors vs Certified Emission Rates

binned by 1 ug/m3 increments, discarded spurious high emission rate 

values from two lowest bins, described above

• Perhaps discarding data for previous plots may prove to not be 

objectively justifiable? But, this exploratory exercise shows that a 

small change in the data can lead to a very different result. This 

illustrates the need for significantly more robust analysis, modeling, 

and planning, and calls into question the robustness of the paper’s 

conclusion.

• The papers also imply that high variability (low precision) prevents 

valid and statistically significant conclusions to made about the 

larger population of stoves.

• BUT, even a noisy relationship (lots of variability stove to stove, day 

to day, test to test) can produce an outcome with good confidence 

because of averaging (central limit theorem)

• For example, with a statistically significant relationship of real-world 

vs certified, even with high variability in the real-world vs certified 

relationship for individual stoves, real-world improvements will 

occur because of averaging (central limit theorem)

– when applied to a large number of devices in an air shed, say > 100, even 

an uncertainty of +/- 50% (per device) the uncertainty of the mean will be 

less than +/- 5%