Appendix F.2 





Air Quality Model Performance Evaluation 

for the Great Smokys

Tennessee Department of Environment and Conservation 

Division of Air Pollution Control 

9th Floor L&C Annex 

401 Church Street 

Nashville, TN 37243

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 1

January, 18, 2008
The Model Performance Evaluation (MPE) was done by the VISTAS contractor, 

ENVIRON, for the VISTAS region and is included in Appendix M.1 – Technical 

Support Document for VISTAS Emissions and Air Quality Modeling to Support 

Regional Haze State Implementation Plans (Preliminary Draft Report, dated 

October 10, 2007). 

Some basic information specific to Great Smokys is provided here. 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 2

January, 18, 2008
20% Best and Worst Days for Great Smokys 

Projections of light extinction for the evaluation of regional haze are based on a subset of days for which 

there is monitored data. IMPROVE monitors operate on an every third day polling frequency. 

Observations of component species on these days are used to calculate light extinction, which is the basis 

on which these days are ranked. These rankings are then used to identify the 20% most impaired days, or 

“worst day”, and the 20% least impaired days, or the “best days”, for use in visibility projections. Table 1 

provides a list of the 20% best and worst days for Great Smokys. 

Since the ultimate future year projections are based on these 20% best and worst days, the model 

performance evaluation will focus on the best and worst days for Great Smokys. 

Table 1. 20% Best and 20% Worst Days for Great Smokys

Date Julian Date GRSM1 Date Julian Date GRSM1

01/08/02 8 20% Best Day 02/25/02 56 20% Worst Day 

01/11/02 11 20% Best Day 05/02/02 122 20% Worst Day 

01/14/02 14 20% Best Day 05/18/02 128 20% Worst Day 

01/23/02 23 20% Best Day 05/17/02 137 20% Worst Day 

02/01/02 32 20% Best Day 06/01/02 152 20% Worst Day 

02/07/02 38 20% Best Day 06/13/02 164 20% Worst Day 

02/19/02 50 20% Best Day 06/19/02 170 20% Worst Day 

02/28/02 59 20% Best Day 06/22/02 173 20% Worst Day 

03/03/02 62 20% Best Day 07/01/02 182 20% Worst Day 

03/09/02 68 20% Best Day 07/07/02 188 20% Worst Day 

03/30/02 89 20% Best Day 07/10/02 191 20% Worst Day 

04/26/02 116 20% Best Day 07/16/02 197 20% Worst Day 

07/13/02 194 20% Best Day 07/22/02 203 20% Worst Day 

09/14/02 257 20% Best Day 07/31/02 212 20% Worst Day 

09/26/02 269 20% Best Day 08/03/02 215 20% Worst Day 

10/26/02 299 20% Best Day 08/09/02 218 20% Worst Day 

10/29/02 302 20% Best Day 08/12/02 221 20% Worst Day 

11/04/02 308 20% Best Day 08/15/02 224 20% Worst Day 

11/16/02 320 20% Best Day 08/21/02 227 20% Worst Day 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 3

January, 18, 2008
Table 1. 20% Best and 20% Worst Days for Great Smokys

Date Julian Date GRSM1 Date Julian Date GRSM1

11/22/02 326 20% Best Day 09/05/02 233 20% Worst Day 

12/07/02 341 20% Best Day 09/08/02 248 20% Worst Day 

12/13/02 347 20% Best Day 9/11/02 251 20% Worst Day 

12/22/02 356 20% Best Day 09/11/02 254 20% Worst Day 

09/17/02 260 20% Worst Day 

Model Performance Statistics 

To quantify model performance, several statistical measures were calculated and evaluated for all the 

IMPROVE monitors within the VISTAS 12km domain and individually for the IMPROVE monitor 

associated with the Great Smokys. The statistical measures selected were based on the recommendations 

outlined in section 18.4.1 of the USEPA’s Guidance On The Use Of Models And Other Analyses for 

Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and Regional Haze (“Attainment 

Guidance”). 

In 2004 VISTAS established model performance goals and criteria for components of fine particle mass 

(Table 2) based on previous model performance for ozone and fine particles. EPA modeling guidance for 

fine particulate matter at the time noted that PM models might not be able to achieve the same level of 

performance as ozone models. VISTAS’ evaluation considered several statistical performance measures and 

displays. 

Table 2. Model Performance Goals and Criteria for Components of Fine Particle Mass 

Fractional Bias Fractional Error Comments 

#15% #35% Goal for PM model performance based on ozone 

model performance, considered excellent 

performance 

#30% #50% Goal for PM model performance, considered good 

performance 

#60% #75% Criteria for PM model performance, considered 

average performance. Exceeding this level of 

performance indicates fundamental concerns with the 

modeling system and triggers diagnostic evaluation. 

Statistical Tables 

The statistical data is calculated collectively for the IMPROVE monitors within the VISTAS 12 km domain, 

as well as individually for Great Smokys. These statistics have been calculated separately for the 20% worst 

(Tables 3 & 4) and 20% best days (Tables 5 & 6). 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 4

January, 18, 2008
Table 3. Model performance statistics for the Worst 20% Days across Class I areas in the 12 km VISTAS 

grid. 

12km bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT

MFB (%) -63.82 -121.18 -32.08 -63.17 22.51 188.69 -87.88 -49.58

MFE (%) 74.05 146.07 50.78 70.50 69.91 191.96 115.76 60.44

Ap -0.50 0.44 -0.83 -0.28 -0.43 8.80 -0.65 -0.43

NMB (%) -39.35 -18.22 -29.24 -42.51 36.38 18564.68 -69.20 -32.51

NME (%) 50.22 87.96 48.66 52.21 97.47 18666.07 74.92 44.58

MNB (%) -38.14 -39.83 -15.63 -41.16 156.91 7919.56 -39.56 -29.87

MNE (%) 51.57 102.75 43.37 51.71 189.98 7956.43 88.73 44.15

MB (μg/m3) -48.14 -1.38 -6.15 -2.24 0.38 12.46 -2.26 -55.75

MAGE (μg/m3) 61.44 6.66 10.23 2.75 1.01 12.52 2.44 76.45

RMSE (μg/m3) 82.20 13.69 28.78 3.59 1.79 20.70 3.53 101.79

Obs Max (μg/m3) 475.66 107.80 614.69 41.23 16.92 11.70 20.96 735.80

Pred Max (μg/m3) 235.90 154.76 102.23 29.67 9.61 114.64 7.23 419.01

Mean_Obs 122.36 7.57 21.03 5.26 1.04 0.07 3.26 171.51

Mean_Pred 74.22 6.19 14.88 3.03 1.42 12.52 1.01 115.76

Std_Dev_Obs 68.78 14.02 31.42 3.03 1.23 0.57 2.77 72.23

Std_Dev_Pre 59.70 18.80 12.73 3.01 1.23 16.52 1.03 74.73

Variance 6757.49 187.38 828.03 12.88 3.20 428.51 12.47 10360.90

R2 0.22 0.48 0.20 0.32 0.00 0.00 0.06 0.11

BF 0.62 0.60 0.84 0.59 2.57 4.23 0.58 0.70

N 512 512 510 510 512 512 512 512

Table 4. Model performance statistics for the Worst 20% Days at the Great Smoky Mountains Class I area. 

GRSM bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT

MFB (%) -63.76 -163.78 -1.34 -63.89 1.23 200.00 -100.86 -52.84

MFE (%) 65.31 197.12 37.84 64.35 43.61 200.00 124.35 54.90

Ap -0.31 -0.85 0.42 -0.25 0.74 -999.00 -0.83 -0.20

NMB (%) -43.46 -97.29 -0.61 -46.94 5.71 -999.00 -80.08 -38.18

NME (%) 44.65 97.64 39.61 47.41 43.59 -999.00 83.61 39.87

MNB (%) -43.44 -99.12 13.10 -46.37 48.53 -999.00 -27.41 -37.69

MNE (%) 45.09 99.12 44.16 46.84 80.70 -999.00 107.85 39.86

MB (μg/m3) -73.15 -2.67 -0.11 -2.25 0.06 7.32 -1.63 -79.67

MAGE (μg/m3) 75.14 2.68 7.46 2.28 0.42 7.32 1.70 83.20

RMSE (μg/m3) 92.01 3.23 10.13 2.54 0.57 8.15 2.15 100.92

Obs Max (μg/m3) 308.51 10.16 33.88 7.71 1.52 0.00 4.89 347.18

Pred Max (μg/m3) 212.34 1.54 47.97 5.81 2.66 17.08 0.82 277.94

Mean_Obs 168.29 2.75 18.83 4.80 0.97 0.00 2.03 208.67

Mean_Pred 95.15 0.07 18.71 2.55 1.02 7.32 0.40 129.00

Std_Dev_Obs 64.74 2.07 7.77 1.29 0.37 0.00 1.42 65.71

Std_Dev_Pre 58.12 0.31 8.98 1.16 0.53 3.68 0.15 66.87

Variance 8466.67 10.42 102.61 6.47 0.32 66.48 4.61 10185.05

R2 0.33 0.56 0.06 0.27 0.04 -999.00 0.00 0.30

BF 0.57 0.01 1.13 0.54 1.49 0.00 0.73 0.62

N 24 24 24 24 24 24 24 24

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 5

January, 18, 2008
Table 5. Model performance statistics for the Best 20% Days across Class I areas in the 12 km VISTAS 

grid. 

12km bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT

MFB (%) -115.20 -110.06 -25.88 -27.46 78.49 148.33 -81.50 -81.14

MFE (%) 122.49 132.75 57.01 55.29 114.38 184.31 104.25 88.50

Ap 2.48 1.09 2.16 1.54 1.73 4.23 -0.18 1.85

NMB (%) -61.28 -39.82 -3.89 -12.78 213.72 551.54 -62.01 -47.74

NME (%) 75.88 86.49 55.78 52.03 266.13 711.04 70.48 59.48

MNB (%) -61.79 -40.68 0.75 8.92 682.14 2051.72 -38.65 -48.42

MNE (%) 77.67 91.64 58.34 68.09 704.19 2147.79 78.76 60.38

MB (μg/m3) -9.52 -1.77 -0.18 -0.25 0.63 2.88 -1.34 -19.42

MAGE (μg/m3) 11.79 3.84 2.64 1.02 0.78 3.71 1.52 24.20

RMSE (μg/m3) 14.76 6.23 5.04 1.97 1.18 7.45 2.07 27.92

Obs Max (μg/m3) 41.57 30.49 21.39 9.96 2.95 15.38 9.86 65.36

Pred Max (μg/m3) 144.46 63.78 67.55 25.33 8.05 80.50 8.10 186.22

Mean_Obs 15.54 4.44 4.74 1.95 0.29 0.52 2.15 40.69

Mean_Pred 6.02 2.68 4.55 1.70 0.92 3.40 0.82 21.27

Std_Dev_Obs 7.06 3.93 2.50 1.09 0.32 1.78 1.68 10.12

Std_Dev_Pre 10.74 6.01 5.33 1.91 0.97 6.47 0.92 20.63

Variance 217.83 38.82 25.35 3.88 1.40 55.56 4.29 779.65

R2 0.06 0.11 0.12 0.06 0.00 0.01 0.14 0.09

BF 0.38 0.59 1.01 1.08 7.82 4.57 0.60 0.52

N 490 490 490 490 490 490 490 490

Table 6. Model performance statistics for the Best 20% Days and the Great Smoky Mountains Class I area.

GRSM bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT

MFB (%) -94.17 -85.46 4.96 -7.50 138.59 178.02 -92.16 -63.98

MFE (%) 106.13 121.52 35.87 46.24 146.55 192.47 100.92 73.86

Ap 1.26 0.42 0.48 0.80 5.01 1.79 -0.80 1.12

NMB (%) -46.57 -28.06 17.14 5.51 631.35 909.08 -69.24 -38.72

NME (%) 73.85 89.51 44.67 47.10 648.27 1026.45 72.85 55.67

MNB (%) -43.83 -18.32 18.89 8.54 1821.12 -1.90 -53.08 -37.82

MNE (%) 82.10 96.66 43.55 48.94 1826.58 88.88 64.36 58.76

MB (μg/m3) -8.72 -1.12 0.73 0.11 0.97 2.19 -0.79 -16.09

MAGE (μg/m3) 13.83 3.57 1.91 0.96 1.00 2.48 0.83 23.13

RMSE (μg/m3) 17.59 4.96 2.88 1.22 1.31 3.29 1.09 27.06

Obs Max (μg/m3) 31.91 10.95 8.95 3.38 0.69 3.59 3.75 52.59

Pred Max (μg/m3) 72.10 15.57 13.26 6.09 4.17 10.01 0.77 111.51

Mean_Obs 18.72 3.99 4.27 2.03 0.15 0.24 1.14 41.55

Mean_Pred 10.00 2.87 5.00 2.15 1.13 2.43 0.35 25.46

Std_Dev_Obs 8.30 3.09 1.77 0.72 0.17 0.84 0.78 9.77

Std_Dev_Pre 14.66 4.40 3.10 1.33 0.89 2.27 0.21 21.63

Variance 309.58 24.64 8.30 1.48 1.70 10.85 1.19 732.11

R2 0.03 0.03 0.18 0.16 0.01 0.02 0.03 0.03

BF 0.56 0.82 1.19 1.09 19.21 0.09 0.47 0.62

N 23 23 23 23 23 23 23 23

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 6

January, 18, 2008
Class I Area Model Performance Evaluation

Figures 1- 8 below summarize the CMAQ 2002 Base G2 12 km Actual base case model performance at

the Great Smokys for the best and worst 20% days and then for all days during 2002. 

Great Smoky Mountains National Park (GRSM), Tennessee-North Carolina 

2.3.12.1 GRSM Best 20 Percent Days 

Figure 1. CMAQ 2002 Base G2 12 km Actual base case model performance for the Best 

20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for SO4 (top 

left), NO3 (top right), OMC (bottom left) and EC (bottom right). 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 7

January, 18, 2008
 

Figure 2. CMAQ 2002 Base G2 12 km Actual base case model performance for the Best 

20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for Soil (top 

left), CM (top right), Sea Salt (bottom left) and Total Bext (bottom right). 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 8

January, 18, 2008
 

Figure 3. Bugle Plots of Fractional Bias and Fractional Gross Error for GRSM and 

the Best 20 Percent Days. 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 9

January, 18, 2008
 

GRSM Worst 20 Percent Days 

Figure 4. CMAQ 2002 Base G2 12 km Actual base case model performance for the Worst 

20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for SO4 (top 

left), NO3 (top right), OMC (bottom left) and EC (bottom right). 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 10

January, 18, 2008
 

Figure 5. CMAQ 2002 Base G2 12 km Actual base case model performance for the Worst 

20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for Soil (top 

left), CM (top right), Sea Salt (bottom left) and Total Bext (bottom right). 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 11

January, 18, 2008
 

Figure 6. Bugle Plots of Fractional Bias and Fractional Gross Error for GRSM and 

the Worst 20 Percent Days. 

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 12

January, 18, 2008
 

GRSM for All Days 

Figure 7. Time series of predicted and observed SO4, NO3, OMC, EC, Soil, CM, Sea 

Salt and PM2.5 mass across all monitoring days at Great Smoky Mountains (GRSM), 

Tennessee-North Carolina for the CMAQ 2002 Base G2 12 km base case simulation.

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 13

January, 18, 2008
 

Figure 8. (concluded) Time series of predicted and observed SO4, NO3, OMC, EC, 

Soil, CM, Sea Salt and PM2.5 mass across all monitoring days at Great Smoky Mountains 

(GRSM), Tennessee-North Carolina for the CMAQ 2002 Base G2 12 km base case simulation.

Pre-Hearing Draft

Regional Haze SIP for TN Class I Areas

Appendix F.2 - 14

January, 18, 2008