Performance Evaluations Me

for Long Range Transpor

Models
Evaluation Paradigm for Lo

Range Transport Models

• LRT models play a unique role in air qual

modeling. This class of models plays sevroles.

– Emergency response modeling– Class I increments and visibility

• Requires additional level of skill to reflect 

and space considerations of LRT model u

– Statistical measures should examine spatiote

pairing ability of LRT models.
Evaluation Goal

• Develop meteorological and tracer datab

evaluation of long range transport models

• Develop a consistent and objective metho

evaluating long range transport (LRT) moused by the EPA.

• Promote the best scientific application of 

based upon lessons learned from evaluatand reflect this in EPA modeling guidance
Irwin Methodology

Plume Sigma-y Transit Time on Arc Observed 

Cros

Inte

Conc

Azimuth of Plume Plume Arrival Time

Centerline

Spatial Temporal Perfor

Method:• Compute 

n-hour average for each receptor on arc (ArcAverage p

• Use trapezoidal integration to ‘fit’ average plume on arc (PLMFIT 
Background from Origina

Performance Evaluations

• Measures employed by Irwin (1998) and EPA (1

provide useful diagnostic information about the performance of LRT modeling systems such as CALPUFF, but they do not always lend themselveasily to spatiotemporal analysis or direct modeintercomparison. 

• For tracer studies such as the Great Plains Trac

Experiment, Savannah River, and INEL74 where arcs of monitors were present, the Irwin evaluatiapproach was used. 

• In addition to the Irwin methodology, EPA augm

statistical measures focusing upon spatiotempocomparisons of model-observation pairings. 
Statistical Evaluation Methodo

• The model evaluation methodology employed fo

project was designed following the procedures oet al. (1998) and Draxler et al. (2001). 

• Statistical measures fall into four broad categori

– Scatter– Bias– Spatial– Cumulative distribution

• NOAA ARL DATEM performance evaluation pro

(STATMAIN) augmented by EPA with additiona

statistics for false alarm rates, probability of dete

and threat score.
NOAA’s Data Archive of Tracer Experimen

Meteorology (DATEM)
Statistical Measures - Sca

( ) 5.0  ×100

%





=

−

>

N

N

FOEX

Pi Ni

i

• Factor of Exce

(FOEX)

• Factor of 2/5 (• Normalized Me

Square Error (

• Pearson’s Cor

Coefficient (R)

( [ ] ) 100

0

0

×





−

=

−

=

N

N

y

y

x

x FA

α

α

( )

1

2

=

∑

Pi −

Mi

NPM

NMSE

(

)

(

)

( ) ( )





−





−

−

•

−

=

∑

∑

∑

2

2

M

M

P

P

M

M

P

P

R

i i

i

i i
Statistical Measures – Cumul

Distribution 

• Kolmogorov-Smirnov 

Parameter

(

KS

= MAX

C

M

k
Statistical Measures - Bia

• Mean Bias (B)

FB

=

2

B

(P

+

M

) • Fractional Bias 

=

∑( ) −

i

Pi Mi

N

B

1
Spatial Statistics

(

)

(

)

( ) ( ) 0

0

0

0 100

>

∪

>

>

∩

>

=

N

p

N

m

N

p

N

m FMS • Figure of merit in 

(FMS)

• Additional EPA M

– False Alarm Rate – Probability of De

(POD)

– Threat Score (TSa=forecast, not obsb=forecast and obsd=observed, not fo

×100

%





+

=

a

b

a FAR

×100

%





+

=

b

d

b POD

×100

%





+

+

=

a

b

d

b TS
Statistical Measure – Final M

Rank

• Model Rank (

measure of m“success” acrthe four broadcategories.

• Range from 0 

poorest and 4 performance.

• Unique meas

for direct inteamongst modsummarizes smodel into a snumber easily 

(

)

(

) 100

1

2 100

1 RANK

=

R

2

+

− FB

+ FMS

+

− KS
Example Performance Evalua

FLEXPART 6.2

•• EXP_DATA results file: gbl1_001.txt • Model variation: 001 Tracer number: 1 Station • -----------------------------------•• 3105 Unaveraged data points for processing• 0.00 Percentile input for zero measured• 0.00 Zero measured concentration value•• 0.00 Correlation coefficient• 0.06 T-value (|Slope|/Standard Error)• 269.56 Average measured concentration• 874.65 Average calculated concentration• 3.24 Ratio of calculated/measured• 118.44 Normalized mean square error• 1104 Number of pairs analyzed•• 605.09 Average bias [(C-M)/N]• 197.29 Lo 99 % confidence interval• 1012.90 Hi 99 % confidence interval• 1.06 Fractional bias [2B/(C+M)]•• 30.07 Fig of merit in space (%)•• 0.54 False Alarm Ratio• 0.27 Probability of Detection• 0.20 Threat score at 0.1 ngm-3 threshold•• -18.30 Factor exceeding [N(C>M)/N-0.5]• 5.53 Percent C/M ± 2• 12.77 Percent C/M ± 5• 30.07 Percent M>0 and C>0• 55.16 Percent M>0 and C=0• 14.76 Percent M=0 and C>0••• 51.00 Kolmogorov-Smirnov Parameter•• 1.26 Final rank (C,FB,FMS,KSP)