Comments on Evaluation Procedures forAir Quality and Meteorological ModelsBob Paine, ENSRPanel Discussion at EPA’s9thModeling ConferenceOctober 10, 2008Outline of PresentationTypes of evaluation databasesAERMOD evaluation reviewEvaluation toolsCox-Tikvart evaluation procedureBOOT/ASTM evaluation procedure (Joe Chang)Evaluation databases (Joe Chang)Gridded met evaluationTwo Types of Evaluation Databases▪Tracer studies: short-term intensive studies, typically withmultiple rows of samplers, each with many sites•Can determine plume centerline and plume sigma-y•Can determine concentration trend with distance•Maximum concentrations on tracer arcs are used forevaluation•Can evaluate predictions paired in time and distance•Limitation is short duration of study▪Long-term monitoring networks: year-long sampling at afew sites•Statistics unpaired in time are necessary; paired in space•Limitation is spatial resolution•Advantage is large number of hours in databaseReview of AERMOD EvaluationHow well does AERMOD predict peak ground-levelconcentrations used for compliance with AQ standards?Is AERMOD’s performance significantly better than that ofsimilar models?Evaluation databases were a mixture of tracer experimentsand long-term studiesStatistical Evaluation Tools Used for AERMODPlots used extensively; they are often better than“black box”statisticsQuantile-Quantile (Q-Q) plots: plot pairs of rankedpredictionsand observations, unpaired in time• Can be used for both types of evaluation databasesResidual plots: plots of ratios of predicted/observed conc vs.downwind distance or wind speed, etc.• Generally used only for tracer databasesEstimates of Robust Highest Concentration, or RHC,thatrepresents a smoothed estimate of the highestconcentrations (from Cox-Tikvart evaluation technique)Scatterplot (data paired in time and space) – only used fortracer databasesScatter Plots –Paired in Timeand Space(a)(b)(c)Source: Joe Chang lectureon model evaluation, 2006Quantile-Quantile Plot▪Observationsand predictionsare separatelyranked▪To see whetherCDFs given byobservations andpredictions aresimilar▪Does not testability of modelto predict pairedin time(a)(b) (c)Source: Joe Chang lectureResidual Box Plots for a “Good” Model▪Plot model residuals,predictions/observations as afunction of an independentvariable▪Group residuals according toranges of an independentvariable▪Use box plot to indicate theCDF of the n points in eachgroup▪For example, the significantpoints for each box indicatethe 2nd, 16th, 50th, 84th, and98thpercentiles▪A good model should haveno trend in model residuals(a)(b)(c) (d)Source: Joe Chang lectureResidual Box Plots for a “Poor” Model▪A slide trend in modelresidual is visible(a)(b)(c)(d)Source: Joe Chang lectureImportant Evaluation Statistic is Fractional BiasCo = observed concentration (or Maximum Arcwise Conc. for BOOT/ASTM)Cp = predicted concentrationFB of zero is perfect model; +/- 0.67 is within a factor of 2Major Features of Cox-Tikvart MethodRHC statistic usedResampling of data used to determine confidence intervalfor differences in performances of modelsComposite performance measure (CPM) combinesabsolute FBs for several averaging timesModel Comparison Measure looks at differences in CPMbetween models to determine statistical significance ofdifferences among modelsBest suited to long-term, sparse network evaluationdatabasesPM-10CompositePerformanceMeasure(CPM)-ISCST3With 90% Confidence Limits0.0 0.5 1.0 1.5 2.0COMPOSITE PERFORMANCE MEASURE (CPM)NLVNLANMVNMANHVNHABHVBAS

M O D E LLEGENDB = BASE MODELN = NEW MODELL = EMISSION RESOLUTION - LOWM = EMISSION RESOLUTION - MEDIUMH = EMISSION RESOLUTION - HIGHA = ROADS AS AREA SOURCESV = ROADS AS VOLUME SOURCES(from Brode, 2006)PM-10ModelComparisonMeasure(MCM)-ISCST3With 90% Confidence IntervalMODEL PAIRS-0.5 0.00.5MODEL COMPARISON MEASURE (MCM)NHV - NHANMV - NMANLV - NLA(from Brode, 2006)BOOT Software Package (slides provided by Joe Chang)Developed by Hanna and ChangBest suited to tracer databasesWidely distributed to (> 200) scientists in the field, mainlythrough the European’sHarmonisation within AtmosphericDispersion Modelling for Regulatory Purposes– ModelValidation KitIs generic and can be used to evaluate different kinds ofmodels, different kinds of outputs, and different kinds ofdata pairingsPrimary References▪ Chang, J.C., and S.R. Hanna, 2004: Air quality modelperformance evaluation.Meteorol. and Atmos. Phys.,87,167-196▪ Chang, J. C., 2002:Methodologies for EvaluatingPerformance and Assessing Uncertainty of AtmosphericDispersion Models. Ph.D. thesis, George MasonUniversity, Fairfax, VA 22030-4444, 277 pp▪ These two references lead to numerous other citationsPerformance Measures in BOOT()( )−=+o po pC CF B0 .5 C C()= −o pMG exp lnC lnC( )−=2o po pC CN M S EC C( ) = −  2o pVG exp lnC lnC= ≤ ≤poCFAC2 % of data that satisfy 0.5 2.0CFractional BiasNormalized Mean Square ErrorGeometric Mean BiasGeometric Variance()()p oo o p pC CC C C CR− −=σ σCorrelation CoefficientExamples of BOOT Performance PlotModel-BModel-AModel-C▪A nice way to plot MG/VG (orFB/NMSE) at the same time▪A perfect model is located at thecenter of the x-axis (green dot)▪MG for Model-B and Model-Care significantly different from1.0What Are “Observations”?▪ Observations can be• Directly measured by instruments• Products of other models or analysis procedures▪ Direct observations aresnapshotsof an ensemble, butmodel predictions often represent ensemble averagesASTM (American Society for Testing and Materials)Procedure – Similar to BOOT▪ Observations are snapshots (ensemble realizations)▪ Model predictions are ensemble averages▪ The two cannot be directly compared▪ In order to compare model predictions to observations,some sort of averaging must first be performed▪ ASTM suggests that this averaging be done overregimesof similar conditions (e.g., for downwind distance oratmospheric stability)ASTM Procedure▪For short-range dispersion experiments where samplers are arranged inarcs, ASTM procedure also suggests near-centerlineconcentrations asrepresentative of centerline values – creates Gaussian fitCross-winddirection (y)ConcentrationCenter of mass,or centroid (yC)0.67σyNear-centerline regionCross-wind distribution witha center of mass (yC) and aspread (σy)Issues With ASTM ProcedureResults sensitive to how the limited regimes aredefinedHas so far only been demonstrated for short-rangedispersion experiments with concentric samplingarcsNot clear how the procedure should be applied tothe evaluation of 3-D Eulerian air quality models,where predicted concentrations representaverages over a grid volume, but observedconcentrations represent point measurementsSee > 100 database references from Joe Chang athttp://www.ofcm.gov/homeland/gmu2005/Presentations/09-Chang%202005%20GMU-OFCM%20Panel.pptEvaluation of Gridded Meteorological Data▪Gridded met data should not be used until thoroughlyevaluated with independent data▪There may be situations with poor met performance (e.g.,complex terrain)▪Conditions of concern for dispersion modeling:•Low wind frequency•Underestimation of wind speeds aloft (e.g., low-level jet)•Wind rose misrepresentation▪Sources of data for testing•Need to find tall tower data, not just surface data•Private industrial met towers•Numerous wind energy assessment towers