### Kentucky Energy and Environment Cabinet Kentucky Division for Air Quality



# -------------------------------------------

Appendix F.2 Air Quality Model Performance Evaluation

1[This page intentionally left blank.]

2The Model Performance Evalua tion (MPE) was completed by the VISTAS contractor, ENVIRON, for the VISTAS region and is in cluded in Appendix M.1 – Technical Support Document for VISTAS Emissions and Air Qualit y Modeling to Support Regional Haze State Implementation Plans (Draft Final Report, dated November 14, 2007). However, some of the information specific to Mammoth Cave National Park is provided in this docum ent.

3Table of Contents
1. Overview

............................................................................................................................... .......5 1.1 IMPROVE Monitoring Sites

................................................................................................. 5 1.2 Particulate Matter and Component Species

.......................................................................... 5 1.3 20% Best and Worst Days

.................................................................................................... 6 2. Model Performance Statistics ......................................................................................................8 2.1 Statistical Tables

................................................................................................................. 10

4

### 1. Overview The discussion of model performance in this Appendix will focus on the comparison of observational data from the I nteragency M onitoring of PRO tected V isual E nvironments (IMPROVE) monitors and model output data from the 2002 VISTAS actual annual air quality modeling. The evaluation will primarily focus on the air quality model’s performance with respect to individual components of fine particulate matter (PM 2.5 ), as good model performance of the component species will dictate good model performance of total or reconstituted fine particulate matter. Model performance of the tota l fine particulate matter and the resulting total light extinction will also be provided as a means to discuss the overall model performance for this Implementation Plan.

1.1 IMPROVE Monitoring Sites
Mammoth Cave National Park is the one Cla ss I area in Kentucky. Ma mmoth Cave is represented by an IMPROVE mon itoring site located inside its park boundary. The Mammoth Cave IMPROVE monitoring site is identified as MACA1 (Mammoth Cave National Park). The Class I area and IMPROVE monitor associated with Kentucky are noted in Figure 1-1.

Mammoth Cave National Park

Figure 1-1: Class I area and IM PROVE monitor in Kentucky.

1.2 Particulate Matter and Component Species Regional haze is calculated by estimating li ght scattering and absorption by com ponents of PM 2.5 . Regional haze is measured by an extinction coefficient (b ext ), which represents the light attenuation resulting from scattering and absorpti on of light from ambient particulate matter plus scattering of light due to gas molecules in the air (i.e., Raylei gh scattering). Some scattering does occur from air molecules; however, the bulk of light extin ction is caused by the presence of suspended particles, partic ularly fine particles (aerodynamic mass <2.5 μ m).

5Fine particulate matter can be composed of varying amounts of several different species, including:

• Sulfates

• Nitra tes

• Ammonium

• Organic carbon (OC)

• Elem ental carbon (EC)

• Coarse particulate m atter (i.e., PM 10 - PM 2.5 ).

• Soil (also known as crustal m aterial, fine soil , major metal oxides, inorganic particulates, or other PM) The composition of particulate matter can vary f rom day today, allowing a day with a large concentration of sulfates to be closely followed by a day dominated by organic carbon. This is an important fact as some of the components of fi ne particulate matter are more effective at light scattering than others. Since there can be such a large variability in the concentrations of the component species, the model will actually be evaluated on a pe r species basis, followed by a cursory evaluation of total reconstructed fine particle mass. This is also a more stringent test of the model, as simply looking at particle pollution performance may pr ovide good results, even though the underlying constituent concentrations ar e incorrect. Only a brief overview of total fine particles after component evaluation is necessary, since the accur acy of the model in predicting the component species will dictate th e accuracy of the total fine particulate mass prediction.

1.3 20% Best and Worst Days Projections of light extinction fo r the evaluation of regional haze are based on a subset of days for which there is monitored data. IMPROVE monitors operate on an every third day polling frequency. Observations of component specie s on these days are used to calculate light extinction, which is the basis on which these days are ranked. Th ese rankings are then used to identify the 20% most impaired da ys, or “worst days”, and the 20% least impaired days, or “best days”, for use in visibility projections. Table 1-1 provides a list of the 20% best and worst days for Kentucky’s Class I area. Since the ultimate future year projections are based on these 20% best and worst days, the m odel performance evaluation will focus on the best and worst days for Mammoth Cave.

6Table 1-1: 20% Best and 20% Worst Days for Mammoth Cave National Park

Date Julian Date MACA1 01/14/02 14 20% Best Day 01/23/02 23 20% Best Day 02/01/02 32 20% Best Day 02/04/02 35 20% Best Day 02/13/02 44 20% Best Day 03/03/02 62 20% Best Day 03/09/02 68 20% Best Day 03/24/02 83 20% Best Day 03/27/02 86 20% Worst Day 03/30/02 89 20% Best Day 04/23/02 113 20% Best Day 04/26/02 116 20% Best Day 05/29/02 149 20% Worst Day 06/01/02 152 20% Worst Day 06/04/02 155 20% Worst Day 06/19/02 170 20% Worst Day 06/22/02 173 20% Worst Day 07/04/02 185 20% Worst Day 07/10/02 191 20% Worst Day 07/16/02 197 20% Worst Day 07/22/02 203 20% Worst Day 07/25/02 206 20% Worst Day 07/31/02 212 20% Worst Day 08/03/02 215 20% Worst Day 08/06/02 218 20% Worst Day 08/12/02 224 20% Worst Day 08/15/02 227 20% Best Day 08/21/02 233 20% Worst Day 08/24/02 236 20% Worst Day 08/27/02 239 20% Worst Day 09/02/02 245 20% Worst Day 09/08/02 251 20% Worst Day 09/11/02 254 20% Worst Day 09/17/02 260 20% Worst Day 09/20/02 263 20% Worst Day 09/23/02 266 20% Best Day 09/26/02 269 20% Best Day 10/05/02 278 20% Best Day 10/08/02 281 20% Best Day 10/14/02 287 20% Best Day 10/29/02 302 20% Best Day 11/04/02 308 20% Best Day 11/19/02 323 20% Best Day 12/04/02 338 20% Best Day 12/07/02 341 20% Worst Day 12/13/02 347 20% Best Day 12/22/02 356 20% Best Day 12/31/02 365 20% Best Day

7

### 2. Model Performance Statistics To quantify model performance, several statisti cal measures were calculated and evaluated for all the IMPROVE monitors within the VISTAS 12km domain and individually for the MPROVE monitor associated with Kentucky’s Class I area. The statistical measures selected were based on the recommendations outlined in section 18.4.1 of the USEPA’s Guidance On The Use Of Models And Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and Regional Haze (“ Attainment Guidance ”). In 2004 VISTAS established model performance goals and criteria f or components of fine particle mass (Table 2.1) based on previous mode l performance for ozone and fine particles. EPA modeling guidance for fine pa rticulate matter at the time noted that PM models might not be able to achieve the same level of perfor mance as ozone models. VISTAS’ evaluation considered several statistical perf ormance measures and displays. Table 2-1 Model performance goals and criteri a fo r components of fine particle mass. Fractional Bias Fractional Error Comment

< 15% < 35% Goal for PM model performance based on ozone model perform ance, considered excellent performance
< 30% < 50% Goal for PM model performance, considered good perform ance < 60% < 75% Criteria for PM model performance, considered average p erformance. Exceeding this level of performance indicates fundamental concerns with the modeling system and triggers diagnostic evaluation.

The statistical measures were calculated for al l of the com ponent specie s of particulate matter responsible for light extinction, an d for total light extinction. Fo r convenience, these statistical measures or metrics, along with a variety of a dditional statistical measures, are summarized in Table 2-2.

8Table 2-2 Statistical Metric Calculations Statistical Measure Shorthand Notation Mathematical Expression Notes Accuracy of Paired Peak A p

peak peak O OP

−

R 2 Coefficient of Determination P i = prediction at time and location i; 

# ( ) 

# ( ) ∑ ∑ ∑

= = =

− − ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

−−

N

i N

i i i N

i i i

OOPP OOPP

1 1 2 2 2

1 )()( O i = observation at time and location i; P = arithmetic average of P i , i = 1, 2, . . ., N; O = arithmetic average of O i , i = 1, 2, . . ., N; Normalized Mean Error NME


## ∑ ∑

=

= −

N

i i N

i ii

O OP

1 1 Reported as %

Root Mean Square Error RMSE 

## ( ) 2 1

1 2 1 ⎥ ⎦ ⎤ ⎢ ⎣ ⎡

− 

## ∑

= N

i ii OP N Reported as %

Mean Fractional Error MFE 

# ∑

= + − N

i ii ii OP OP N 1 2 Reported as %

Mean Absolute Gross Error MAGE 

## ∑

= − N

i ii OP N 1 1

Mean Normalized Gross Error MNGE 

## ∑

= − N

i i ii

O OP N 1 1 Reported as %

Mean Biased MB 

## ( ) 

## ∑

= − N

i ii OP N 1 1 Reported as concentration

Mean Normalized Bias MNB 

## ( ) 

# ∑

= − N

i i ii

O OP N 1 1 Reported as %

Mean Fractionalized Bias (Fractional Bias) MFB 

# ∑

= ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛

+ − N

i ii ii OP OP N 1 2 Reported as %

Normalized Mean Bias NMB 

## ( ) 

## ∑ ∑

=

= −

N

i i N

i ii

O OP

1 1 Reported as %

Bias Factor BF 

# ∑

= ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛ N

i i i O P N 1 1 Reported as BF:1 or 1:BF or in fractional notation (BF/1 or 1/BF)

92.1 Statistical Tables
The statistical data is calculated collectivel y f or the IMPROVE monitors within the VISTAS 12km domain, as well as individually for Mammoth Cave. These model performance statistics have been calculated separately for the 20% best days (Page 11) and 20% worst days (Page 12). Please see Appendix M.1 for detailed model pe rform ance evaluation information for Mammoth Cave National Park and other VISTAS Class I areas.

10