# Appendix F.2

Air Quality Model Performance Evaluation

for the Great Smok y s

Tennessee Department of Environment and Conservation Division of Air Pollution Control
9 th 

### Floor L&C Annex 401 Church Street Nashville, TN 37243

# The Model Performance Eval uation (MPE) was done by the VISTAS contractor, ENVIRON, for the VISTAS region an d is included in Appendix M.1 – Technical Support Document for VISTAS Emissions and Air Quality Mode ling to Support Regional Haze State Implementation Plans (Preliminary Draft Report, dated October 10, 2007).

Some basic information specific to Great Smokys is provided here.

## 20% Best and Worst Days for Great Smokys

Projections of light extinction for the evaluation of regional haze are ba sed on a subset of days for which there is monitored data. IMPROVE monitors op erate on an every third day polling frequency. Observations of component species on these days are us ed to calculate light extinction, which is the basis on which these days are ranked. These rankings are then used to identify the 20% most impaired days, or “worst day”, and the 20% least impaired days, or the “b est days”, for use in visi bility projections. Table 1 provides a list of the 20% best and worst days for Great Smokys.

Since the ultimate future year projections are based on these 20% best and worst days, the model performance evaluation will focus on the be st and worst days for Great Smokys.

Table 1. 20% Best and 20% Wor st Days for Great Smokys

Date Julian Date GRSM1 Date Julian Date GRSM1

01/08/02 8 20% Best Day 02/25/02 56 20% Worst Day

01/11/02 11 20% Best Day 05/02/02 122 20% Worst Day

01/14/02 14 20% Best Day 05/18/02 128 20% Worst Day

01/23/02 23 20% Best Day 05/17/02 137 20% Worst Day

02/01/02 32 20% Best Day 06/01/02 152 20% Worst Day

02/07/02 38 20% Best Day 06/13/02 164 20% Worst Day

02/19/02 50 20% Best Day 06/19/02 170 20% Worst Day

02/28/02 59 20% Best Day 06/22/02 173 20% Worst Day

03/03/02 62 20% Best Day 07/01/02 182 20% Worst Day

03/09/02 68 20% Best Day 07/07/02 188 20% Worst Day

03/30/02 89 20% Best Day 07/10/02 191 20% Worst Day

04/26/02 116 20% Best Day 07/16/02 197 20% Worst Day

07/13/02 194 20% Best Day 07/22/02 203 20% Worst Day

09/14/02 257 20% Best Day 07/31/02 212 20% Worst Day

09/26/02 269 20% Best Day 08/03/02 215 20% Worst Day

10/26/02 299 20% Best Day 08/09/02 218 20% Worst Day

10/29/02 302 20% Best Day 08/12/02 221 20% Worst Day

11/04/02 308 20% Best Day 08/15/02 224 20% Worst Day

11/16/02 320 20% Best Day 08/21/02 227 20% Worst Day

### Table 1. 20% Best and 20% Worst Days for Great Smokys

Date Julian Date GRSM1 Date Julian Date GRSM1

11/22/02 326 20% Best Day 09/05/02 233 20% Worst Day

12/07/02 341 20% Best Day 09/08/02 248 20% Worst Day

12/13/02 347 20% Best Day 9/11/02 251 20% Worst Day

12/22/02 356 20% Best Day 09/11/02 254 20% Worst Day

09/17/02 260 20% Worst Day



## Model Performance Statistics

To quantify model performance, seve ral statistical measures were cal culated and evaluated for all the IMPROVE monitors within the VI STAS 12km domain and individu ally for the IMPROVE monitor associated with the Great Smokys. The statistical measures selected were based on the recommendations outlined in section 18.4.1 of the USEPA’s Guidance On The Use Of Models And Other Analyses for Demonstrating Attainment of Air Qualit y Goals for Ozone, PM2.5, and Regional Haze (“ Attainment Guidance ”).

In 2004 VISTAS established model performance goals and criteria for components of fine particle mass (Table 2) based on previous model performance for oz one and fine particles. EPA modeling guidance for fine particulate matter at the time noted that PM mode ls might not be able to achieve the same level of performance as ozone models. VISTAS’ evaluation cons idered several statistical performance measures and displays.

Table 2. Model Performance Goals and Criteria for Components of Fine Particle Mass

Fractional Bias Fractional Error Comments
# 15% # 35% Goal for PM model performance based on ozone model performance, considered excellent performance # 30% # 50% Goal for PM model performance, considered good performance

# 60% # 75% Criteria for PM model performance, considered average performance. Exceeding this level of performance indicates fundamental concerns with the modeling system and triggers diagnostic evaluation.

Statistical Tables

The statistical data is calculated collectively for th e IMPROVE monitors within the VISTAS 12 km domain, as well as individually for Great Smokys . These statistics have been calcul ated separately for the 20% worst (Tables 3 & 4) and 20% be st days (Tables 5 & 6).

### Table 3. Model performance statistics for the Worst 20% Days across Class I areas in the 12 km VISTAS grid.

12km bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT MFB (%) -63.82 -121.18 -32.08 -63.17 22.51 188.69 -87.88 -49.58 MFE (%) 74.05 146.07 50.78 70.50 69.91 191.96 115.76 60.44
A p -0.50 0.44 -0.83 -0.28 -0.43 8.80 -0.65 -0.43 NMB (%) -39.35 -18.22 -29.24

- 42.51 36.38 18564.68 -69.20 -32.51 NME (%) 50.22 87.96 48.66 52. 21 97.47 18666.07 74.92 44.58 MNB (%) -38.14 -39.83 -15.63

- 41.16 156.91 7919.56 -39.56 -29.87 MNE (%) 51.57 102.75 43.37 51. 71 189.98 7956.43 88.73 44.15 MB ( μ g/m 3 ) -48.14 -1.38 -6.15 -2.24 0.38 12.46 -2.26 -55.75 MAGE ( μ g/m 3 ) 61.44 6.66 10.23 2.75 1.01 12.52 2.44 76.45 RMSE ( μ g/m 3 ) 82.20 13.69 28.78 3.59 1.79 20.70 3.53 101.79 Obs Max ( μ g/m 3 ) 475.66 107.80 614.69 41.23 16.92 11.70 20.96 735.80 Pred Max ( μ g/m 3 ) 235.90 154.76 102.23 29.67 9.61 114.64 7.23 419.01 Mean_Obs 122.36 7.57 21.03 5.26 1.04 0.07 3.26 171.51 Mean_Pred 74.22 6.19 14.88 3.03 1.42 12.52 1.01 115.76 Std_Dev_Obs 68.78 14.02 31. 42 3.03 1.23 0.57 2.77 72.23 Std_Dev_Pre 59.70 18.80 12. 73 3.01 1.23 16.52 1.03 74.73
Variance 6757.49 187.38 828.03 12.88 3.20 428.51 12.47 10360.90
R 2 0.22 0.48 0.20 0.32 0.00 0.00 0.06 0.11
BF 0.62 0.60 0.84 0.59 2.57 4.23 0.58 0.70
N 512 512 510 510 512 512 512 512



### Table 4. Model performance statistics fo r the Worst 20% Days at the Great Smoky Mountains Class I area.

GRSM bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT MFB (%) -63.76 -163.78 -1.34

- 63.89 1.23 200.00 -100.86 -52.84 MFE (%) 65.31 197.12 37.84 64.35 43.61 200.00 124.35 54.90
A p -0.31 -0.85 0.42 -0.25 0.74 -999.00 -0.83 -0.20 NMB (%) -43.46 -97.29 -0.61 -46. 94 5.71 -999.00 -80.08 -38.18 NME (%) 44.65 97.64 39.61 47. 41 43.59 -999.00 83.61 39.87 MNB (%) -43.44 -99.12 13.10 -46. 37 48.53 -999.00 -27.41 -37.69 MNE (%) 45.09 99.12 44.16 46. 84 80.70 -999.00 107.85 39.86 MB ( μ g/m 3 ) -73.15 -2.67 -0.11 -2.25 0.06 7.32 -1.63 -79.67 MAGE ( μ g/m 3 ) 75.14 2.68 7.46 2.28 0.42 7.32 1.70 83.20 RMSE ( μ g/m 3 ) 92.01 3.23 10.13 2.54 0.57 8.15 2.15 100.92 Obs Max ( μ g/m 3 ) 308.51 10.16 33.88 7.71 1.52 0.00 4.89 347.18 Pred Max ( μ g/m 3 ) 212.34 1.54 47.97 5.81 2.66 17.08 0.82 277.94 Mean_Obs 168.29 2.75 18.83 4.80 0.97 0.00 2.03 208.67 Mean_Pred 95.15 0.07 18.71 2.55 1.02 7.32 0.40 129.00 Std_Dev_Obs 64.74 2.07 7. 77 1.29 0.37 0.00 1.42 65.71 Std_Dev_Pre 58.12 0.31 8. 98 1.16 0.53 3.68 0.15 66.87
Variance 8466.67 10.42 102.61 6.47 0.32 66.48 4.61 10185.05
R 2 0.33 0.56 0.06 0.27 0.04 -999.00 0.00 0.30
BF 0.57 0.01 1.13 0.54 1.49 0.00 0.73 0.62
N 24 24 24 24 24 24 24 24

### Table 5. Model performance statistics fo r the Best 20% Days across Cla ss I areas in the 12 km VISTAS grid.

12km bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT MFB (%) -115.20 -110.06 -25.88 -27.46 78.49 148.33 -81.50 -81.14 MFE (%) 122.49 132.75 57.01 55.29 114.38 184.31 104.25 88.50
A p 2.48 1.09 2.16 1.54 1.73 4.23 -0.18 1.85 NMB (%) -61.28 -39.82 -3.89

- 12.78 213.72 551.54 -62.01 -47.74 NME (%) 75.88 86.49 55.78 52.03 266.13 711.04 70.48 59.48 MNB (%) -61.79 -40.68 0.75 8. 92 682.14 2051.72 -38.65 -48.42 MNE (%) 77.67 91.64 58.34 68. 09 704.19 2147.79 78.76 60.38 MB ( μ g/m 3 ) -9.52 -1.77 -0.18 -0.25 0.63 2.88 -1.34 -19.42 MAGE ( μ g/m 3 ) 11.79 3.84 2.64 1.02 0.78 3.71 1.52 24.20 RMSE ( μ g/m 3 ) 14.76 6.23 5.04 1.97 1.18 7.45 2.07 27.92 Obs Max ( μ g/m 3 ) 41.57 30.49 21.39 9.96 2.95 15.38 9.86 65.36 Pred Max ( μ g/m 3 ) 144.46 63.78 67.55 25.33 8.05 80.50 8.10 186.22 Mean_Obs 15.54 4.44 4.74 1.95 0.29 0.52 2.15 40.69 Mean_Pred 6.02 2.68 4.55 1.70 0.92 3.40 0.82 21.27 Std_Dev_Obs 7.06 3.93 2. 50 1.09 0.32 1.78 1.68 10.12 Std_Dev_Pre 10.74 6.01 5. 33 1.91 0.97 6.47 0.92 20.63
Variance 217.83 38.82 25.35 3.88 1.40 55.56 4.29 779.65
R 2 0.06 0.11 0.12 0.06 0.00 0.01 0.14 0.09
BF 0.38 0.59 1.01 1.08 7.82 4.57 0.60 0.52
N 490 490 490 490 490 490 490 490



### Table 6. Model performance statistics for the Best 20% Days and the Great Smoky Mountains Class I area.

GRSM bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXT MFB (%) -94.17 -85.46 4.96 -7 .50 138.59 178.02 -92.16 -63.98 MFE (%) 106.13 121.52 35.87 46.24 146.55 192.47 100.92 73.86
A p 1.26 0.42 0.48 0.80 5.01 1.79 -0.80 1.12 NMB (%) -46.57 -28.06 17.14 5.51 631.35 909.08 -69.24 -38.72 NME (%) 73.85 89.51 44.67 47. 10 648.27 1026.45 72.85 55.67 MNB (%) -43.83 -18.32 18.89 8. 54 1821.12 -1.90 -53.08 -37.82 MNE (%) 82.10 96.66 43.55 48.94 1826.58 88.88 64.36 58.76 MB ( μ g/m 3 ) -8.72 -1.12 0.73 0.11 0.97 2.19 -0.79 -16.09 MAGE ( μ g/m 3 ) 13.83 3.57 1.91 0.96 1.00 2.48 0.83 23.13 RMSE ( μ g/m 3 ) 17.59 4.96 2.88 1.22 1.31 3.29 1.09 27.06 Obs Max ( μ g/m 3 ) 31.91 10.95 8.95 3.38 0.69 3.59 3.75 52.59 Pred Max ( μ g/m 3 ) 72.10 15.57 13.26 6.09 4.17 10.01 0.77 111.51 Mean_Obs 18.72 3.99 4.27 2.03 0.15 0.24 1.14 41.55 Mean_Pred 10.00 2.87 5.00 2.15 1.13 2.43 0.35 25.46 Std_Dev_Obs 8.30 3.09 1. 77 0.72 0.17 0.84 0.78 9.77 Std_Dev_Pre 14.66 4.40 3. 10 1.33 0.89 2.27 0.21 21.63
Variance 309.58 24.64 8.30 1.48 1.70 10.85 1.19 732.11
R 2 0.03 0.03 0.18 0.16 0.01 0.02 0.03 0.03
BF 0.56 0.82 1.19 1.09 19.21 0.09 0.47 0.62
N 23 23 23 23 23 23 23 23

### Class I Area Model Performance Evaluation

Figures 1- 8 below summarize the CMAQ 2002 Base G2 12 km Actual base case model performance at the Great Smokys for the best and worst 20% days and then for all days during 2002.

C-123 Great Smoky Mountains National Pa rk (GRSM), Tennessee-North Carolina

2.3.12.1 GRSM Best 20 Percent Days

Figure 1

. CMAQ 2002 Base G2 12 km Actual base case model performance for the Best 20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for SO4 (top left), NO3 (top right), OMC (bottom left) and EC (bottom right).

### Figure 2

. CMAQ 2002 Base G2 12 km Actual base case model performance for the Best 20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for Soil (top left), CM (top right), Sea Salt (bottom left) and Total Bext (bottom right).

### Figure 3

. Bugle Plots of Fractional Bias and Fractional Gross Error for GRSM and the Best 20 Percent Days.

### GRSM Worst 20 Percent Days

Figure 4

. CMAQ 2002 Base G2 12 km Actual base case model performance for the Worst 20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for SO4 (top left), NO3 (top right), OMC (bottom left) and EC (bottom right).

### Figure 5

. CMAQ 2002 Base G2 12 km Actual base case model performance for the Worst 20 percent days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for Soil (top left), CM (top right), Sea Salt (bottom left) and Total Bext (bottom right).

### Figure 6

. Bugle Plots of Fractional Bias and Fractional Gross Error for GRSM and the Worst 20 Percent Days.

### GRSM for All Days

Figure 7

. 

### Time series of predicted and observed SO4, NO3, OMC, EC, Soil, CM, Sea Salt and PM 2.5 

### mass across all monitoring days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for the CMAQ 2002 Base G2 12 km base case simulation.

### Figure 8

. 

### (concluded) Time series of predicted and observed SO4, NO3, OMC, EC, Soil, CM, Sea Salt and PM 2.5 

### mass across all monitoring days at Great Smoky Mountains (GRSM), Tennessee-North Carolina for the CMAQ 2002 Base G2 12 km base case simulation.