# PSCAA Preliminary Reanalysis of

- Wood Heater Test Method Variability ( Curkeet , 2010)

- A Comparison Between In

- Home Use and EPA Certified Values (Houck, 2012)

These analyses argue that 1) test method variability is too great, and 2)real

- world emissions are insufficiently correlated to the test method, to justify lowering the standard. Can these conclusions be objectively supported? H ow robust are the analyses and results? Are other conclusions possible?

# Overview

• Anyway you slice it, real

- world emissions (5

- 10 g/hour) of certified stoves is too high for even modest growth in wood burning in urban areas such as the Tacoma NAA. Replacing all uncertified with certified would not allow for any new stoves to enter the airshed , or at the very least, would return the airshed to nonattainment in 5

- 10 years.

• We have concerns that existing analyses have problems with data representativeness, data treatment, completeness of analysis, and appropriateness of statistical methods

• Nonetheless, NSPS and certification values should be more representative of real

- word burning. This requires:

– Significantly more comprehensive and statistically robust 1) analysis of existing data and 2) a priori design for test lab procedures and real

- world evaluations

– Wider swath (estimated 10x) of real

- world use data. Okay to sacrifice precision of measurements if the representativeness can be significantly increased.

– Significantly better experimental design for real

- world testing. Burn rates, fuel moistures, etc , need to be more consistently employed. Campaigns with data not uniformly covering the independent variables must be omitted.

– More sophisticated model of independent variables, interactions, and emissions

# Examples from Test Method Variability ( Curkeet )Note: outlier had 5

- year gap with previous tests, only 2 labs evaluated,

# Lab deviation seems consistent across all stoves and years

# Underlying data are log

- normal. Normal quantile on left, log

- normal on right ( quantile vs emission rate)

# A quick reanalysis of Test Method Variability data

• std dev of each labs average performance across three stoves, 1987

- 2000 is 1.4 g/ hr , from 2

- 6 tests per lab

• std dev of deviation of individual runs from stove

- average is about 1.8 g/ hr , appears consistent across certification rate and test lab rate

• four stoves mean values show good linearity wrt certified values

• implies labs can reproduce certified values, lower values may need a few more repetitions to reduce uncertainty

• suggests that: with about 15 stoves (tests) in a lower category and a mean difference of 2 g/hour, there’s a > 95% chance the true difference is at least 1 g/hour

## Example of concerns from: A Comparison of In

- Home Emissions and EPA (Houck, 2012)

• Includes data from a range of studies over many years, in different locations, using inconsistent burn rates, fuel moistures, fuel types, burn rate measurements, etc. No indication or analysis that these are representative.

• A number of stoves have bi

- modal emissions vs burn rate relationships. No clear sign that these behaviors are representative and accurate. Also can’t apply normal statistics (based on a normal distribution).

• Analysis needs to account for non

- normal distributions and correct for outliers and spurious data.

• Most underlying data (individual stove results and population results) are log

- normal.

• Stoves #4 , # 11 had anomalously high std deviations and are the highest values in the low cert value category, likely have data that are not representative.

• For stove #4, one of three campaigns (“laboratory simulated in

- home use”) had wildly high values, didn’t fit distribution, very likely are outliers.

• Other catalytic stoves (e.g. #6) show two different regimes, one with much higher emissions and low burn rates, likely not consistent with design and not representative.

• several stoves had only one or two tests (certification requires 4 )

• several stoves have only data with burn rate < 0.9

• Skewed distribution of burn rates compared to Method 28. “Real

- world” is biased toward low burn rates, which unrepresentatively weights bi

- modal regime.

# All “Real

- World” Emission Rate data, quantile plots, normal and log

- normal of emission rate (g/hour)

## M ost of the highest points are from Stove 4. Stove 4 quantile plots: normal and log

- normal of emission rates in g/hour

# Burn rates of “Real

- World” data don’t agree with Method 28Several stoves, mostly in the lowest emissions bins, have “L” shaped emissions vs burn rate relationships. They appear to be bi

- modal which means normal statistics are not applicable and the stoves may be outliers or unrepresentative. Stove 4 high data is exclusively from “ Laboratary simulated in home use” while real in

- home use was much lower

## most stoves have more uni

- modal relationship

### Which one of these is not like the other? Low certification, medium certification, high certification: Emission rate (g/hour) vs burn rate divided in bins following Houck

## data in lowest bins (1

- 2 g/hour [left], 2

- 3 g/hour[right]) are not well behaved: quantile plots of normal (top) and ln

- normal (bottom) data“Real

- World” Emission Factors vs Certified Emission Rates binned by 1 ug /m3 increments, discarded spurious high emission rate values from two lowest bins, described above• Perhaps discarding data for previous plots may prove to not be objectively justifiable? But, this exploratory exercise shows that a small change in the data can lead to a very different result. This illustrates the need for significantly more robust analysis, modeling, and planning , and calls into question the robustness of the paper’s conclusion.

• The papers also imply that high variability (low precision) prevents valid and statistically significant conclusions to made about the larger population of stoves.

• BUT, even a noisy relationship (lots of variability stove to stove, day to day, test to test) can produce an outcome with good confidence because of averaging (central limit theorem)

• For example, with a statistically significant relationship of real

- world vs certified, even with high variability in the real

- world vs certified relationship for individual stoves, real

- world improvements will occur because of averaging (central limit theorem)

– when applied to a large number of devices in an air shed, say > 100, even an uncertainty of +/

- 50% (per device) the uncertainty of the mean will be less than +/

- 5%