Comments on Evaluation Procedure

Air Quality and Meteorological Mo

Bob Paine, ENSR

Panel Discussion at EPA’s 

9th Modeling Conference

October 10, 2008
Outline of Presentation

 Types of evaluation databases AERMOD evaluation review Evaluation tools Cox-Tikvart evaluation procedure BOOT/ASTM evaluation procedure (Joe Chang Evaluation databases (Joe Chang) Gridded met evaluation
Two Types of Evaluation Databases

▪ Tracer studies: short-term intensive studies, typ

multiple rows of samplers, each with many sites

• Can determine plume centerline and plume sigma-y• Can determine concentration trend with distance• Maximum concentrations on tracer arcs are used for e• Can evaluate predictions paired in time and distanc

e

• Limitation is short duration of study

▪ Long-term monitoring networks: year-long samp

few sites

• Statistics unpaired in time are necessary; paired in sp• Limitation is spatial resolution• Advantage is large number of hours in database
Review of AERMOD Evaluation

 How well does AERMOD predict peak ground-le

concentrations used for compliance with AQ sta

 Is AERMOD’s performance significantly better th

similar models?

 Evaluation databases were a mixture of tracer e

and long-term studies
Statistical Evaluation Tools Used for 

 Plots used extensively; they are often better than 

statistics

 Quantile-Quantile (Q-Q) plots: plot pairs of ranked 

and observations, unpaired in time

• Can be used for both types of evaluation datab

 Residual plots: plots of ratios of predicted/observe

downwind distance or wind speed, etc.

• Generally used only for tracer databases

 Estimates of Robust Highest Concentration, or RH

represents a smoothed estimate of the highest concentrations (from Cox-Tikvart evaluation techn

 Scatterplot (data paired in time and space) – only 

tracer databases
Scatter Plots –

Paired in Time and Space

(a)

(b) (c)

Source: Joe Chang lectureon model evaluation, 2006
Quantile-Quantile Plot

▪ Obse

and pare sranke

▪ To se

CDFs obserpredisimila

▪ Does 

ability to prein tim

(a)

(b) (c)

Source: Joe C
Residual Box Plots for a “Good” Model

▪ Plot model 

predictions/function of variable

▪ Group resid

ranges of avariable

▪ Use box plo

CDF of the group

▪ For exampl

points for ethe 2nd, 16t

98th percen

▪ A good mod

no trend in 

(a) (b)

(c) (d)

Source: Joe 
Residual Box Plots for a “Poor” Model

▪ A slide tre

residual is 

(a) (b)

(c) (d) Source: J
Important Evaluation Statistic is Fractional Bia

Co = observed concentration (or Maximum Arcwise Conc. for Cp = predicted concentrationFB of zero is perfect model; +/- 0.67 is within a factor of 2
Major Features of Cox-Tikvart Method

 RHC statistic used Resampling of data used to determine confiden

for differences in performances of models

 Composite performance measure (CPM) combi

absolute FBs for several averaging times

 Model Comparison Measure looks at differences 

between models to determine statistical significadifferences among models

 Best suited to long-term, sparse network evalua

databases
PM-10 Composite Performance Measure (CPM) - ISC

 With 90% Confidence Limits

0.0 0.5 1.0 1.5 

COMPOSITE PERFORMANCE MEASURE (CPM)

NLVNLA

NMVNMA

NHVNHA

BHV

BAS

MODEL

LEGENDB = BASE MODEL

N = NEW MODEL

L = EMISSION RESOM = EMISSION RESOH = EMISSION RESOA = ROADS AS AREV = ROADS AS VOL

(from 
PM-10 Model Comparison Measure (MCM) - ISCST

 With 90% Confidence Interval

MODEL PAIRS

-0.5 0.0 0

MODEL COMPARISON MEASURE (MCM)

NHV - NHANMV - NMANLV - NLA

(from B
BOOT Software Package (slides provided by Jo

 Developed by Hanna and Chang Best suited to tracer databases Widely distributed to (> 200) scientists in the fiel

through the European’s Harmonisation within AtDispersion Modelling for Regulatory Purposes – Validation Kit

 Is generic and can be used to evaluate different 

models, different kinds of outputs, and different data pairings
Primary References

▪ Chang, J.C., and S.R. Hanna, 2004: Air quality 

performance evaluation. Meteorol. and Atmos. 

167-196

▪ Chang, J. C., 2002: Methodologies for Evaluatin

Performance and Assessing Uncertainty of Atm

Dispersion Models. Ph.D. thesis, George MasoUniversity, Fairfax, VA 22030-4444, 277 pp

▪ These two references lead to numerous other c
Performance Measures in BOOT

(

)

( )

−

=

+

o p

o p

C C

F B

0 .5 C C

MG exp lnC lnC = − ( o p

)

( ) −

=

2

o p

o p

C C

N M S E

C C

( )   = −    

2 VG exp lnC lnC o p

= ≤ ≤

p

o

C

FAC2 % of data that satisfy 0.5 2.0

C

Fractional BiasNormalized Mean Sq

Geometric Mean BiaGeometric Variance

(

)

(

)

p o

o o p p

C C

C C C C

R − −

=

σ σ

Correlation Coefficie
Examples of BOOT Performance Plot

Model-BModel-A

Model-C

▪ A nice way to plot 

FB/NMSE) at the s

▪ A perfect model is 

center of the x-axis 

▪ MG for Model-B an

are significantly dif1.0
What Are “Observations”?

▪ Observations can be

• Directly measured by instruments• Products of other models or analysis procedures

▪ Direct observations are snapshots of an ensemb

model predictions often represent ensemble ave
ASTM (American Society for Testing and Mate

Procedure – Similar to BOOT

▪ Observations are snapshots (ensemble realizati▪ Model predictions are ensemble averages▪ The two cannot be directly compared▪ In order to compare model predictions to observ

some sort of averaging must first be performed

▪ ASTM suggests that this averaging be done ove

of similar conditions (e.g., for downwind distanc

atmospheric stability)
ASTM Procedure

▪ For short-range dispersion experiments where samplers are a

arcs, ASTM procedure also suggests near-centerline concentrepresentative of centerline values – creates Gaussian fit

Concentration

Center of mass,or centroid (yC)

0.67 σy

Near-centerline region

Cross-wind distribution wa center of mass (yC) and 

spread (σy)
Issues With ASTM Procedur

 Results sensitive to how the limited regim

defined

 Has so far only been demonstrated for sh

dispersion experiments with concentric saarcs

 Not clear how the procedure should be ap

the evaluation of 3-D Eulerian air quality mwhere predicted concentrations represenaverages over a grid volume, but observeconcentrations represent point measurem

See > 100 database references from Joe Chang at

http://www.ofcm.gov/homeland/gmu2005/Presentations/09Chang%202005%20GMU-OFCM%20Panel.ppt
Evaluation of Gridded Meteorological Data

▪ Gridded met data should not be used until thoro

evaluated with independent data

▪ There may be situations with poor met performa

complex terrain)

▪ Conditions of concern for dispersion modeling:

• Low wind frequency• Underestimation of wind speeds aloft (e.g., low-level j• Wind rose misrepresentation

▪ Sources of data for testing

• Need to find tall tower data, not just surface data• Private industrial met towers• Numerous wind energy assessment towers