Kentucky Energy and Environment Cabinet 

Kentucky Division for Air Quality 

------------------------------------------- 

Appendix F.2 

Air Quality Model 

Performance Evaluation 

 1

Appendix F.2 - 1 

Kentucky Regional Haze SIP
[This page intentionally left blank.] 

 2

Appendix F.2 - 2 

Kentucky Regional Haze SIP
The Model Performance Evaluation (MPE) was completed by the VISTAS contractor, 

ENVIRON, for the VISTAS region and is included in Appendix M.1 – Technical Support 

Document for VISTAS Emissions and Air Quality Modeling to Support Regional Haze State 

Implementation Plans (Draft Final Report, dated November 14, 2007). 

However, some of the information specific to Mammoth Cave National Park is provided in this 

document. 

 3

Appendix F.2 - 3 

Kentucky Regional Haze SIP
Table of Contents 

1. Overview......................................................................................................................................5 

1.1 IMPROVE Monitoring Sites................................................................................................. 5 

1.2 Particulate Matter and Component Species.......................................................................... 5 

1.3 20% Best and Worst Days .................................................................................................... 6 

2. Model Performance Statistics......................................................................................................8 

2.1 Statistical Tables................................................................................................................. 10 

 4

Appendix F.2 - 4 

Kentucky Regional Haze SIP
 

1. Overview 

The discussion of model performance in this Appendix will focus on the comparison of 

observational data from the Interagency Monitoring of PROtected Visual Environments 

(IMPROVE) monitors and model output data from the 2002 VISTAS actual annual air quality 

modeling. The evaluation will primarily focus on the air quality model’s performance with 

respect to individual components of fine particulate matter (PM2.5), as good model performance 

of the component species will dictate good model performance of total or reconstituted fine 

particulate matter. Model performance of the total fine particulate matter and the resulting total 

light extinction will also be provided as a means to discuss the overall model performance for 

this Implementation Plan. 

1.1 IMPROVE Monitoring Sites 

Mammoth Cave National Park is the one Class I area in Kentucky. Mammoth Cave is 

represented by an IMPROVE monitoring site located inside its park boundary. The Mammoth 

Cave IMPROVE monitoring site is identified as MACA1 (Mammoth Cave National Park). The 

Class I area and IMPROVE monitor associated with Kentucky are noted in Figure 1-1. 

Mammoth Cave 

National Park 

Figure 1-1: Class I area and IMPROVE monitor in Kentucky. 

1.2 Particulate Matter and Component Species 

Regional haze is calculated by estimating light scattering and absorption by components of 

PM2.5. Regional haze is measured by an extinction coefficient (bext), which represents the light 

attenuation resulting from scattering and absorption of light from ambient particulate matter plus 

scattering of light due to gas molecules in the air (i.e., Rayleigh scattering). 

Some scattering does occur from air molecules; however, the bulk of light extinction is caused 

by the presence of suspended particles, particularly fine particles (aerodynamic mass <2.5 μm). 

 5

Appendix F.2 - 5 

Kentucky Regional Haze SIP
Fine particulate matter can be composed of varying amounts of several different species, 

including: 

• Sulfates 

• Nitrates 

• Ammonium 

• Organic carbon (OC) 

• Elemental carbon (EC) 

• Coarse particulate matter (i.e., PM10 - PM2.5). 

• Soil (also known as crustal material, fine soil, major metal oxides, inorganic particulates, 

or other PM) 

The composition of particulate matter can vary from day today, allowing a day with a large 

concentration of sulfates to be closely followed by a day dominated by organic carbon. This is 

an important fact as some of the components of fine particulate matter are more effective at light 

scattering than others. Since there can be such a large variability in the concentrations of the 

component species, the model will actually be evaluated on a per species basis, followed by a 

cursory evaluation of total reconstructed fine particle mass. This is also a more stringent test of 

the model, as simply looking at particle pollution performance may provide good results, even 

though the underlying constituent concentrations are incorrect. Only a brief overview of total 

fine particles after component evaluation is necessary, since the accuracy of the model in 

predicting the component species will dictate the accuracy of the total fine particulate mass 

prediction. 

1.3 20% Best and Worst Days 

Projections of light extinction for the evaluation of regional haze are based on a subset of days

for which there is monitored data. IMPROVE monitors operate on an every third day polling 

frequency. Observations of component species on these days are used to calculate light 

extinction, which is the basis on which these days are ranked. These rankings are then used to 

identify the 20% most impaired days, or “worst days”, and the 20% least impaired days, or “best 

days”, for use in visibility projections. Table 1-1 provides a list of the 20% best and worst days 

for Kentucky’s Class I area. 

Since the ultimate future year projections are based on these 20% best and worst days, the model 

performance evaluation will focus on the best and worst days for Mammoth Cave. 

 6

Appendix F.2 - 6 

Kentucky Regional Haze SIP
Table 1-1: 20% Best and 20% Worst Days for Mammoth Cave National Park 

Date Julian Date MACA1 

01/14/02 14 20% Best Day 

01/23/02 23 20% Best Day 

02/01/02 32 20% Best Day 

02/04/02 35 20% Best Day 

02/13/02 44 20% Best Day 

03/03/02 62 20% Best Day 

03/09/02 68 20% Best Day 

03/24/02 83 20% Best Day 

03/27/02 86 20% Worst Day 

03/30/02 89 20% Best Day 

04/23/02 113 20% Best Day 

04/26/02 116 20% Best Day 

05/29/02 149 20% Worst Day 

06/01/02 152 20% Worst Day 

06/04/02 155 20% Worst Day 

06/19/02 170 20% Worst Day 

06/22/02 173 20% Worst Day 

07/04/02 185 20% Worst Day 

07/10/02 191 20% Worst Day 

07/16/02 197 20% Worst Day 

07/22/02 203 20% Worst Day 

07/25/02 206 20% Worst Day 

07/31/02 212 20% Worst Day 

08/03/02 215 20% Worst Day 

08/06/02 218 20% Worst Day 

08/12/02 224 20% Worst Day 

08/15/02 227 20% Best Day 

08/21/02 233 20% Worst Day 

08/24/02 236 20% Worst Day 

08/27/02 239 20% Worst Day 

09/02/02 245 20% Worst Day 

09/08/02 251 20% Worst Day 

09/11/02 254 20% Worst Day 

09/17/02 260 20% Worst Day 

09/20/02 263 20% Worst Day 

09/23/02 266 20% Best Day 

09/26/02 269 20% Best Day 

10/05/02 278 20% Best Day 

10/08/02 281 20% Best Day 

10/14/02 287 20% Best Day 

10/29/02 302 20% Best Day 

11/04/02 308 20% Best Day 

11/19/02 323 20% Best Day 

12/04/02 338 20% Best Day 

12/07/02 341 20% Worst Day 

12/13/02 347 20% Best Day 

12/22/02 356 20% Best Day 

12/31/02 365 20% Best Day 

 7

Appendix F.2 - 7 

Kentucky Regional Haze SIP
2. Model Performance Statistics 

To quantify model performance, several statistical measures were calculated and evaluated for 

all the IMPROVE monitors within the VISTAS 12km domain and individually for the MPROVE 

monitor associated with Kentucky’s Class I area. The statistical measures selected were based 

on the recommendations outlined in section 18.4.1 of the USEPA’s Guidance On The Use Of 

Models And Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, 

PM2.5, and Regional Haze (“Attainment Guidance”). 

In 2004 VISTAS established model performance goals and criteria for components of fine 

particle mass (Table 2.1) based on previous model performance for ozone and fine particles. 

EPA modeling guidance for fine particulate matter at the time noted that PM models might not 

be able to achieve the same level of performance as ozone models. VISTAS’ evaluation 

considered several statistical performance measures and displays. 

Table 2-1 Model performance goals and criteria for components of fine particle mass. 

Fractional 

Bias 

Fractional 

Error 

Comment 

<15% <35% Goal for PM model performance based on ozone 

model performance, considered excellent 

performance 

<30% <50% Goal for PM model performance, considered good 

performance 

<60% <75% Criteria for PM model performance, considered 

average performance. Exceeding this level of 

performance indicates fundamental concerns with 

the modeling system and triggers diagnostic 

evaluation. 

The statistical measures were calculated for all of the component species of particulate matter 

responsible for light extinction, and for total light extinction. For convenience, these statistical 

measures or metrics, along with a variety of additional statistical measures, are summarized in 

Table 2-2. 

 8

Appendix F.2 - 8 

Kentucky Regional Haze SIP
Table 2-2 Statistical Metric Calculations 

Statistical Measure Shorthand 

Notation 

Mathematical Expression Notes 

Accuracy of Paired 

Peak 

Ap

peak

peak

O

− OP

 R2 Coefficient of 

Determination 

Pi= prediction at time and 

 location i; 

( )( )

∑ ∑

∑

= =

=

− −

⎥

⎦

⎤ ⎢

⎣

⎡ −−

N

i

N

i

i i

N

i

i i

OOPP

OOPP

1 1

2 2

2

1

)()(

Oi = observation at time and 

 location i; 

P = arithmetic average of Pi, 

 i = 1, 2, . . ., N; 

O = arithmetic average of Oi, 

 i = 1, 2, . . ., N; 

Normalized Mean 

Error 

NME 

∑

∑

=

=

−

N

i

i

N

i

ii

O

OP

1

1

Reported as % 

Root Mean Square 

Error 

RMSE 

( ) 2

1

1

1 2

⎥

⎦

⎤ ⎢

⎣

⎡ ∑ − =

N

i

OP ii N

Reported as % 

Mean Fractional 

Error 

MFE 

∑= +

− N

i ii

ii

OP

OP

N 1

2 Reported as % 

Mean Absolute 

Gross Error 

MAGE ∑=

−

N

i

OP ii N 1

1

Mean Normalized 

Gross Error 

MNGE 

∑=

N −

i i

ii

O

OP

N 1

1 Reported as % 

Mean Biased MB ∑( ) =

−

N

i

OP ii N 1

1 Reported as concentration 

Mean Normalized 

Bias 

MNB ( ) ∑=

− N

i i

ii

O

OP

N 1

1 Reported as % 

Mean 

Fractionalized Bias 

(Fractional Bias) 

MFB 

∑= ⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛

+

− N

i ii

ii

OP

OP

N 1

2 Reported as % 

Normalized Mean 

Bias 

NMB ( )

∑

∑

=

=

−

N

i

i

N

i

ii

O

OP

1

1

Reported as % 

Bias Factor BF 

∑= ⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛ N

i i

i

O

P

N 1

1 Reported as BF:1 or 1:BF or in 

fractional notation (BF/1 or 1/BF) 

 9

Appendix F.2 - 9 

Kentucky Regional Haze SIP
2.1 Statistical Tables 

The statistical data is calculated collectively for the IMPROVE monitors within the VISTAS 

12km domain, as well as individually for Mammoth Cave. These model performance statistics 

have been calculated separately for the 20% best days (Page 11) and 20% worst days (Page 12). 

Please see Appendix M.1 for detailed model performance evaluation information for Mammoth 

Cave National Park and other VISTAS Class I areas. 

 10

Appendix F.2 - 10 

Kentucky Regional Haze SIP
Appendix F.2 - 11 

Kentucky Regional Haze SIP
Appendix F.2 - 12 

Kentucky Regional Haze SIP