west virginia department of environmental protection

Appendix F.2

Air Quality Model 

Performance Evaluation

Promoting a healthy environment.

WV Regional Haze SIP Appendix F.2 - 1
[This page intentionally left blank.]

WV Regional Haze SIP Appendix F.2 - 2
The Model Performance Evaluation (MPE) was done by the VISTAS

contractor, ENVIRON, for the VISTAS region and is included in

Appendix M.1 – Technical Support Document for VISTAS Emissions

and Air Quality Modeling to Support Regional Haze State

Implementation Plans (Preliminary Draft Report, dated October 10,

2007).

Some basic information specific to Dolly Sods is provided here.

WV Regional Haze SIP Appendix F.2 - 3
[This page intentionally left blank.]

WV Regional Haze SIP Appendix F.2 - 4
20% Best and Worst Days for Dolly Sods

Projections of light extinction for the evaluation of regional haze are based on a subset of days

for which there is monitored data. IMPROVE monitors operate on an every third day polling

frequency. Observations of component species on these days are used to calculate light

extinction, which is the basis on which these days are ranked. These rankings are then used to

identify the 20% most impaired days, or “worst day”, and the 20% least impaired days, or the

“best days”, for use in visibility projections. Table 1 provides a list of the 20% best and worst

days for Dolly Sods.

Since the ultimate future year projections are based on these 20% best and worst days, the model

performance evaluation will focus on the best and worst days for Dolly Sods. 

Table 1. 20% Best and 20% Worst Days for Dolly Sods

Date Julian Date Dolly Sods Date Julian Date Date

01/05/02 5 20% Best Day 05/02/02 122 20% Worst Day

01/14/02 14 20% Best Day 05/08/02 128 20% Worst Day

01/23/02 23 20% Best Day 05/17/02 137 20% Worst Day

01/26/02 26 20% Best Day 05/26/02 146 20% Worst Day

02/19/02 50 20% Best Day 05/29/03 149 20% Worst Day

03/09/02 68 20% Best Day 06/01/02 152 20% Worst Day

03/30/02 89 20% Best Day 06/04/02 155 20% Worst Day

04/14/02 104 20% Best Day 06/10/02 161 20% Worst Day

04/29/02 119 20% Best Day 06/13/02 164 20% Worst Day

06/07/02 158 20% Best Day 06/19/02 170 20% Worst Day

10/11/02 284 20% Best Day 06/22/02 173 20% Worst Day

10/20/02 293 20% Best Day 06/25/02 176 20% Worst Day

10/26/02 299 20% Best Day 07/01/02 182 20% Worst Day

10/29/02 302 20% Best Day 07/10/02 191 20% Worst Day

11/16/02 320 20% Best Day 07/25/02 206 20% Worst Day

11/22/02 326 20% Best Day 08/12/02 224 20% Worst Day

12/04/02 338 20% Best Day 08/15/02 227 20% Worst Day

12/07/02 341 20% Best Day 09/11/02 254 20% Worst Day

12/13/02 347 20% Best Day 09/14/02 257 20% Worst Day

WV Regional Haze SIP Appendix F.2 - 5
Table 1. 20% Best and 20% Worst Days for Dolly Sods

Date Julian Date Dolly Sods Date Julian Date Date

12/22/02 356 20% Best Day 09/17/02 260 20% Worst Day

12/25/02 359 20% Best Day 09/20/02 263 20% Worst Day

12/31/02 365 20% Best Day 9/29/02 272 20% Worst Day

10/02/02 275 20% Worst Day

 

Model Performance Statistics

To quantify model performance, several statistical measures were calculated and evaluated for

all the IMPROVE monitors within the VISTAS 12km domain and individually for the

IMPROVE monitor associated Dolly Sods. The statistical measures selected were based on the

recommendations outlined in section 18.4.1 of the USEPA’s Guidance On The Use Of Models

And Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and

Regional Haze (“Attainment Guidance”).

In 2004 VISTAS established model performance goals and criteria for components of fine

particle mass (Table 2) based on previous model performance for ozone and fine particles. EPA

modeling guidance for fine particulate matter at the time noted that PM models might not be able

to achieve the same level of performance as ozone models. VISTAS’ evaluation considered

several statistical performance measures and displays. 

Table 2. Model Performance Goals and Criteria for Components of Fine Particle Mass

Fractional Bias Fractional Error Comments

#15% #35% Goal for PM model performance based on ozone

model performance, considered excellent performance

#30% #50% Goal for PM model performance, considered good

performance

#60% #75% Criteria for PM model performance, considered

average performance. Exceeding this level of

performance indicates fundamental concerns with the

modeling system and triggers diagnostic evaluation.

WV Regional Haze SIP Appendix F.2 - 6
Statistical Tables

The statistical data is calculated collectively for the IMPROVE monitors within the VISTAS 12

km domain, as well as individually for Dolly Sods. These statistics have been calculated

separately for the 20% worst (Tables 3 & 4) and 20% best days (Tables 5 & 6). 

WV Regional Haze SIP Appendix F.2 - 7
[This page intentionally left blank.]

WV Regional Haze SIP Appendix F.2 - 8
Table 3. Model Performance Statistics for the VISTAS 12km Domain for the 20% Worst 

AllSite_AllDay bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS be

Fraction_Bias(%) -63.82298 -121.17516 -32.07985 -63.17410 22.51497 188.68991 -87Fraction_GErr(%) 74.04603 146.06810 50.77829 70.50087 69.91222 191.9608 11Ac_Paired_Peak -0.50405 0.43559 -0.83368 -0.28042 -0.43196 8.79674 -0Norm_Mean_Bias(%) -39.34555 -18.22497 -29.23614 -42.51103 36.38333 18564.67578 -69Norm_Mean_Err(%) 50.21605 87.96179 48.65946 52.20539 97.4745 18666.07031 74Mean_Norm_Bias(%) -38.13657 -39.83007 -15.63076 -41.16308 156.91199 7919.55859 -39Mean_Norm_GErr(%) 51.57135 102.75209 43.37334 51.70714 189.97557 7956.43359 88Mean_Bias -48.14245 -1.37973 -6.14909 -2.23798 0.37755 12.45686 -2Mean_Abs_GErr 61.44337 6.65920 10.23430 2.74833 1.01148 12.52489 2Geometric_Bias -0.74003 -2.60151 -0.35811 -0.70693 0.28415 1.71128 -1Geometric_Err 0.84703 2.92260 0.55342 0.77610 0.84541 2.6581 1Root_Mean_SErr 82.20394 13.68862 28.77551 3.58835 1.78845 20.70042 3Obs_Max 475.66281 107.80230 614.69092 41.23200 16.923 11.7016 2Pre_Max 235.90323 154.75974 102.23344 29.66984 9.613 114.63757 7Arith_Obs_Mean 122.35805 7.57055 21.03251 5.26446 1.03769 0.0671 3Arith_Pre_Mean 74.21560 6.19082 14.88340 3.02648 1.41524 12.52396 1Geome_Obs_Mean inf inf inf inf 0.77472 0.47292 2Geome_Pre_Mean inf 0.34576 inf 2.33338 1.02932 2.61811 0Stand_Devi_Obs 68.77800 14.01594 31.41555 3.03482 1.23243 0.57163 2Stand_Devi_Pre 59.70288 18.79670 12.73284 3.00580 1.22587 16.51884 1Corr_Variance 6757.48828 187.37831 828.02985 12.87626 3.19857 428.50751 12Coeff_Determi 0.21936 0.47704 0.19945 0.32257 0.00018 0.00126 0Bias_Factor 0.61863 0.59935 0.84369 0.58722 2.56912 4.22906 0Num_Points 512 512 510 510 512 512 

WV Regional Haze SIP 
Table 4. Model Performance Statistics for Dolly Sods12km Domain for the 20% W

AllDay_DOSO1 bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS be

Fraction_Bias(%) -72.68279 -116.04311 -46.03989 -106.63990 -18.5469 191.30435 -13Fraction_GErr(%) 79.51933 127.19271 65.51572 106.63990 42.44298 191.30435 13Ac_Paired_Peak -0.44867 0.19119 -0.36860 -0.72607 -0.37333 -999 -0Norm_Mean_Bias(%) -47.86574 -60.19393 -34.19370 -69.89289 -17.5115 -999 -8Norm_Mean_Err(%) 54.03939 77.85197 49.84565 69.89289 34.43992 -999 8Mean_Norm_Bias(%) -45.39338 -58.71173 -26.37871 -68.32784 3.87789 -999 -77Mean_Norm_GErr(%) 53.22950 75.27402 50.74098 68.32784 49.74221 -999 77Mean_Bias -73.36999 -1.72433 -4.04288 -2.89387 -0.14377 3.03404 -1Mean_Abs_GErr 82.83313 2.23016 5.89348 2.89387 0.28275 3.03404 1Geometric_Bias -0.76514 -1.91731 -0.51228 -1.23143 -0.18728 -999 -1Geometric_Err 0.83704 2.05064 0.70998 1.23143 0.45573 -999 1Root_Mean_SErr 113.34650 2.55479 6.90598 3.11407 0.38 3.37322 1Obs_Max 404.98950 5.68820 26.51380 7.50300 1.7766 0 4Pre_Max 223.28340 6.77574 16.74087 2.05532 1.11334 5.66848 0Arith_Obs_Mean 153.28288 2.86462 11.82346 4.14043 0.821 0 1Arith_Pre_Mean 79.91290 1.14029 7.78058 1.24657 0.67723 3.03404 0Geome_Obs_Mean 139.35542 2.67933 10.59959 3.98567 0.73189 -999 1Geome_Pre_Mean 61.76841 0.40035 6.35049 1.16331 0.60689 -999 0Stand_Devi_Obs 78.10185 1.10104 5.70874 1.21787 0.3246 0 1Stand_Devi_Pre 63.82932 1.72554 4.40868 0.43381 0.28867 1.50731 0Corr_Variance 12847.42969 6.52696 47.69260 9.69741 0.1444 11.37858 3Coeff_Determi 0.05653 0.01560 0.14631 0.07446 0.10027 -999 0Bias_Factor 0.54607 0.41288 0.73621 0.31672 1.03878 0 0Num_Points 23 23 23 23 23 23 

WV Regional Haze SIP 
Table 5. Model Performance Statistics for the VISTAS 12km Domain for the 20% 

AllSite_AllDay bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS be

Fraction_Bias(%) -115.19897 -110.05685 -25.87558 -27.45512 78.49191 148.33353 -8Fraction_GErr(%) 122.49083 132.74918 57.0146 55.29499 114.38155 184.30733 10Ac_Paired_Peak 2.47544 1.09211 2.15802 1.54198 1.73437 4.23499 -0Norm_Mean_Bias(%) -61.28178 -39.81734 -3.8786 -12.77569 213.71725 551.53925 -62Norm_Mean_Err(%) 75.88204 86.49456 55.78643 52.02515 266.12604 711.03851 70Mean_Norm_Bias(%) -61.79248 -40.68147 0.76093 8.91848 682.14447 2051.71558 -38Mean_Norm_GErr(%) 77.67429 91.64355 58.35357 68.09431 704.19354 2147.78906 78Mean_Bias -9.52442 -1.76987 -0.18369 -0.24939 0.62756 2.8772 -Mean_Abs_GErr 11.79359 3.84466 2.64201 1.01557 0.78146 3.70925 1Geometric_Bias -1.70498 -2.26525 -0.30405 -0.32102 1.06687 -0.67838 -1Geometric_Err 1.78588 2.55867 0.64003 0.60994 1.55508 2.23195 1Root_Mean_SErr 14.75895 6.23028 5.03647 1.96979 1.18422 7.4536 2Obs_Max 41.56650 30.4869 21.3894 9.964 2.945 15.3769 9Pre_Max 144.46182 63.78189 67.54807 25.32829 8.05273 80.49792 8Arith_Obs_Mean 15.54200 4.44497 4.73593 1.95207 0.29364 0.52167 2Arith_Pre_Mean 6.01758 2.6751 4.55224 1.70268 0.92121 3.39886 0Geome_Obs_Mean inf 3.33784 4.08561 1.63371 0 0.90877 1Geome_Pre_Mean 2.50002 0.36746 3.01446 1.18511 0.52763 0.46114 0Stand_Devi_Obs 7.06463 3.92602 2.50424 1.09105 0.3156 1.78235 1Stand_Devi_Pre 10.74304 6.01116 5.33102 1.91406 0.97091 6.47083 0Corr_Variance 217.82658 38.81642 25.36601 3.88008 1.40237 55.55618 Coeff_Determi 0.06251 0.11192 0.1215 0.06062 0.00267 0.01017 0Bias_Factor 0.38207 0.59197 1.00761 1.08252 7.82144 4.56691 0Num_Points 490 490 490 490 490 490 

WV Regional Haze SIP 
Table 6. Model Performance Statistics for Dolly Sods12km Domain for the 20% B

AllDay_DOSO1 bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS be

Fraction_Bias(%) -107.37399 -62.63736 -78.21556 -73.21972 104.57863 200 -84Fraction_GErr(%) 109.36848 98.95137 78.33873 76.44373 110.05344 200 12Ac_Paired_Peak -0.04082 0.45353 -0.51611 -0.46999 2.4739 -999 -0Norm_Mean_Bias(%) -62.79203 -24.5406 -57.40063 -49.56052 317.64697 -999 -79Norm_Mean_Err(%) 65.69522 64.67429 57.44202 51.87252 321.70764 -999 8Mean_Norm_Bias(%) -64.89848 -18.18808 -52.98637 -48.24676 926.53986 -999 20Mean_Norm_GErr(%) 67.01295 69.58811 53.11034 52.12767 931.31647 -999 14Mean_Bias -8.23004 -0.74423 -2.54523 -0.91678 0.45042 0.82124 -0Mean_Abs_GErr 8.61056 1.96133 2.54706 0.95954 0.45618 0.82124 0Geometric_Bias -1.64028 -0.95529 -0.89866 -0.86919 1.53116 -999 -1Geometric_Err 1.66025 1.36945 0.89989 0.90175 1.58632 -999 1Root_Mean_SErr 9.90401 2.683 3.15827 1.09748 0.58682 1.02365 0Obs_Max 23.23870 9.0214 9.0174 3.442 0.4287 0 Pre_Max 22.29012 13.11289 4.36343 1.8243 1.48926 2.75884 0Arith_Obs_Mean 13.10682 3.03263 4.43415 1.84981 0.1418 0 0Arith_Pre_Mean 4.87678 2.2884 1.88892 0.93303 0.59222 0.82124 0Geome_Obs_Mean 12.12716 2.3008 3.88151 1.69442 0.08938 -999 0Geome_Pre_Mean 2.35176 0.91345 1.58022 0.71045 0.41326 -999 0Stand_Devi_Obs 5.34473 2.39133 2.10428 0.76173 0.11039 0 0Stand_Devi_Pre 5.28463 3.0197 1.13213 0.52258 0.39587 0.62618 0Corr_Variance 98.08949 7.19851 9.97469 1.20445 0.34436 1.04787 0Coeff_Determi 0.18994 0.29621 0.18302 0.35023 0.0542 -999 0Bias_Factor 0.35102 0.81812 0.47014 0.51753 10.2654 0 1Num_Points 21 21 21 21 21 21 

WV Regional Haze SIP