**Performance Evaluations Metrics for Long Range Transport** **Models Evaluation Paradigm for Long** **Range Transport Models • LRT models play a unique role in air quality** **modeling. This class of models plays several roles.** **– Emergency response modeling – Class I increments and visibility** **• Requires additional level of skill to reflect time and space considerations of LRT model use** **– Statistical measures should examine spatiotemporal pairing ability of LRT models.** **Evaluation Goal • Develop meteorological and tracer databases for** **evaluation of long range transport models. • Develop a consistent and objective method for** **evaluating long range transport (LRT) models used by the EPA.** **• Promote the best scientific application of models based upon lessons learned from evaluations** **and reflect this in EPA modeling guidance. Irwin Methodology** Observed MaximumTransit Time on ArcPlume Sigma-yCrosswindIntegratedConcentrationPlume Arrival TimeAzimuth of PlumeCenterlinePerformanceTemporalSpatial**Method: • Compute** n**-hour average for each receptor on arc (ArcAverage p rogram)** **• Use trapezoidal integration to ‘fit’ average plume on arc (PLMFIT program)** **Background from Original Performance Evaluations** **• Measures employed by Irwin (1998) and EPA (1998) provide useful diagnostic information about the** **performance of LRT modeling systems such as CALPUFF, but they do not always lend themselves** **easily to spatiotemporal analysis or direct model intercomparison.** **• For tracer studies such as the Great Plains Tracer Experiment, Savannah River, and INEL74 where distin** **ct arcs of monitors were present, the Irwin evaluation** **approach was used. • In addition to the Irwin methodology, EPA augmente** **d statistical measures focusing upon spatiotemporal** **comparisons of model-observation pairings. Statistical Evaluation Methodology** **• The model evaluation methodology employed for this project was designed following the procedures of Mo** **sca et al. (1998) and Draxler et al. (2001).** **• Statistical measures fall into four broad categori es** **– Scatter – Bias** **– Spatial – Cumulative distribution** **• NOAA ARL DATEM performance evaluation program (STATMAIN) augmented by EPA with additional spatial** **statistics for false alarm rates, probability of de tection,** **and threat score. NOAA’s Data Archive of Tracer Experiments and** **Meteorology (DATEM) Statistical Measures - Scatter** ( )%1005.0×−=>NNFOEXiiNiP**• Factor of Exceedance (FOEX)** **• Factor of 2/5 (FA2/5) • Normalized Mean** **Square Error (NMSE) • Pearson’s Correlation** **Coefficient (R)**[]()10000×−=−=NxxyyNFA 

# α

α( )21∑−=iiMPPMNNMSE()()( )( )−−−•−=∑∑∑22PPMMPPMMRiiiii**Statistical Measures – Cumulative Distribution** **• Kolmogorov-Smirnov Parameter** ()()kkPCMCMAXKS−=**Statistical Measures - Bias • Mean Bias (B)** **• Fractional Bias (FB)**()MPBFB+=2( )∑−=iiiMPNB1**Spatial Statistics**()()( ) ( )0000100>∪>>∩>=mNpNmNpNFMS**• Figure of merit in Space (FMS)** **• Additional EPA Metrics – False Alarm Rate (FAR)** **– Probability of Detection (POD)** **– Threat Score (TS)**a**=forecast, not observed**b**=forecast and observed**d**=observed, not forecast**%100×+=baaFAR%100×+=dbbPOD%100×++=dbabTS**Statistical Measure – Final Model Rank** **• Model Rank (RANK) – measure of model** **“success” across each of the four broader statistical** **categories. • Range from 0 to 4, with 0** **poorest and 4 best performance.** **• Unique measure allows for direct intercomparison** **amongst models and summarizes success of** **model into a single number easily relatable** ()()1001100212KSFMSFBRRANK−++−+=**Example Performance Evaluation – FLEXPART 6.2** **• • EXP_DATA results file: gbl1_001.txt** **• Model variation: 001 Tracer number: 1 Station s elect: All** **• ----------------------------------- •** **• 3105 Unaveraged data points for processing • 0.00 Percentile input for zero measured** **• 0.00 Zero measured concentration value •** **• 0.00 Correlation coefficient • 0.06 T-value (|Slope|/Standard Error)** **• 269.56 Average measured concentration • 874.65 Average calculated concentration** **• 3.24 Ratio of calculated/measured • 118.44 Normalized mean square error** **• 1104 Number of pairs analyzed •** **• 605.09 Average bias [(C-M)/N] • 197.29 Lo 99 % confidence interval** **• 1012.90 Hi 99 % confidence interval • 1.06 Fractional bias [2B/(C+M)]** **• • 30.07 Fig of merit in space (%)** **• • 0.54 False Alarm Ratio** **• 0.27 Probability of Detection • 0.20 Threat score at 0.1 ngm-3 threshold** **• • -18.30 Factor exceeding [N(C>M)/N-0.5]** **• 5.53 Percent C/M ± 2 • 12.77 Percent C/M ± 5** **• 30.07 Percent M>0 and C>0 • 55.16 Percent M>0 and C=0** **• 14.76 Percent M=0 and C>0 •** **• • 51.00 Kolmogorov-Smirnov Parameter** **• • 1.26 Final rank (C,FB,FMS,KSP)**