# west virginia department of environmental protection



# Appendix F.2 Air Quality Model

Performance Evaluation

Promoting a healthy environment.

# [This page intentionally left blank.]

# The Model Performance Evaluation (MPE) was done by the VISTAS contractor, ENVIRON, for the VISTAS region and is included in Appendix M.1 – Technical Support Document for VISTAS Emissions and Air Quality Modeling to Support Regional Haze State Implementation Plans (Preliminary Draft Report, dated October 10, 2007).

Some basic information specific to Dolly Sods is provided here.

# [This page intentionally left blank.]

# 20% Best and Worst Days for Dolly Sods

Projections of light extinction for the evaluati on of regional haze are based on a subset of days for which there is monitored data. IMPROVE monitors operate on an every third day polling frequency. Observations of component species on these days are used to calculate light extinction, which is the basis on which these days are ranked. These rankings are then used to identify the 20% most impaired days, or “worst day”, and the 20% least impaired days, or the “best days”, for use in visibility projections. Table 1 provides a list of the 20% best and worst days for Dolly Sods.

Since the ultimate future year projections are based on these 20% best and worst days, the model performance evaluation will focus on the best and worst days for Dolly Sods.

Table 1. 20% Best and 20% Worst Days for Dolly Sods

Date Julian Date Dolly Sods Date Julian Date Date

01/05/02 5 20% Best Day 05/02/02 122 20% Worst Day

01/14/02 14 20% Best Day 05/08/02 128 20% Worst Day

01/23/02 23 20% Best Day 05/17/02 137 20% Worst Day

01/26/02 26 20% Best Day 05/26/02 146 20% Worst Day

02/19/02 50 20% Best Day 05/29/03 149 20% Worst Day

03/09/02 68 20% Best Day 06/01/02 152 20% Worst Day

03/30/02 89 20% Best Day 06/04/02 155 20% Worst Day

04/14/02 104 20% Best Day 06/10/02 161 20% Worst Day

04/29/02 119 20% Best Day 06/13/02 164 20% Worst Day

06/07/02 158 20% Best Day 06/19/02 170 20% Worst Day

10/11/02 284 20% Best Day 06/22/02 173 20% Worst Day

10/20/02 293 20% Best Day 06/25/02 176 20% Worst Day

10/26/02 299 20% Best Day 07/01/02 182 20% Worst Day

10/29/02 302 20% Best Day 07/10/02 191 20% Worst Day

11/16/02 320 20% Best Day 07/25/02 206 20% Worst Day

11/22/02 326 20% Best Day 08/12/02 224 20% Worst Day

12/04/02 338 20% Best Day 08/15/02 227 20% Worst Day

12/07/02 341 20% Best Day 09/11/02 254 20% Worst Day

12/13/02 347 20% Best Day 09/14/02 257 20% Worst Day

# Table 1. 20% Best and 20% Worst Days for Dolly Sods

Date Julian Date Dolly Sods Date Julian Date Date

12/22/02 356 20% Best Day 09/17/02 260 20% Worst Day

12/25/02 359 20% Best Day 09/20/02 263 20% Worst Day

12/31/02 365 20% Best Day 9/29/02 272 20% Worst Day

10/02/02 275 20% Worst Day



# Model Performance Statistics

To quantify model performance, several statistical measures were calculated and evaluated for all the IMPROVE monitors within the VISTAS 12km domain and individually for the IMPROVE monitor associated Dolly Sods. The statistical measures selected were based on the

recommendations outlined in section 18.4.1 of the USEPA’s Guidance On The Use Of Models And Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and Regional Haze (“ Attainment Guidance ”).

In 2004 VISTAS established model performance goals and criteria for components of fine particle mass (Table 2) based on previous model performance for ozone and fine particles. EPA modeling guidance for fine particulate matter at the time noted that PM models might not be able to achieve the same level of performance as ozone models. VISTAS’ evaluation considered several statistical performance measures and displays.

Table 2. Model Performance Goals and Criteria for Components of Fine Particle Mass

Fractional Bias Fractional Error Comments

# 15% # 35% Goal for PM model performance based on ozone model performance, considered excellent performance

# 30% # 50% Goal for PM model performance, considered good performance

# 60% # 75% Criteria for PM model performance, considered average performance. Exceeding this level of performance indicates fundamental concerns with the modeling system and triggers diagnostic evaluation.

# Statistical Tables

The statistical data is calculated collectively for the IMPROVE monitors within the VISTAS 12 km domain, as well as individually for Dolly Sods. These statistics have been calculated separately for the 20% worst (Tables 3 & 4) and 20% best days (Tables 5 & 6).

# [This page intentionally left blank.]Table 3. Model Performance Statistics for the VISTAS 12km Domain for the 20% Worst DaysAllSite_AllDay bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXTFraction_Bias(%) -63.82298 -121.17516 -32.07985 -63.17410 22.51497 188.68991 -87.88052 -49.57894Fraction_GErr(%) 74.04603 146.06810 50.77829 70.50087 69.91222 191.9608 115.76214 60.43559Ac_Paired_Peak -0.50405 0.43559 -0.83368 -0.28042 -0.43196 8.79674 -0.65483 -0.43055Norm_Mean_Bias(%) -39.34555 -18.22497 -29.23614 -42.51103 36.38333 18564.67578 -69.19688 -32.50518Norm_Mean_Err(%) 50.21605 87.96179 48.65946 52.20539 97.4745 18666.07031 74.91714 44.57608Mean_Norm_Bias(%) -38.13657 -39.83007 -15.63076 -41.16308 156.91199 7919.55859 -39.55885 -29.87465Mean_Norm_GErr(%) 51.57135 102.75209 43.37334 51.70714 189.97557 7956.43359 88.73123 44.15210Mean_Bias -48.14245 -1.37973 -6.14909 -2.23798 0.37755 12.45686 -2.2583 -55.75092Mean_Abs_GErr 61.44337 6.65920 10.23430 2.74833 1.01148 12.52489 2.44499 76.45420Geometric_Bias -0.74003 -2.60151 -0.35811 -0.70693 0.28415 1.71128 -1.30734 -0.52808Geometric_Err 0.84703 2.92260 0.55342 0.77610 0.84541 2.6581 1.43425 0.64225Root_Mean_SErr 82.20394 13.68862 28.77551 3.58835 1.78845 20.70042 3.53109 101.78848Obs_Max 475.66281 107.80230 614.69092 41.23200 16.923 11.7016 20.9585 735.80438Pre_Max 235.90323 154.75974 102.23344 29.66984 9.613 114.63757 7.23417 419.00604Arith_Obs_Mean 122.35805 7.57055 21.03251 5.26446 1.03769 0.0671 3.26359 171.51396Arith_Pre_Mean 74.21560 6.19082 14.88340 3.02648 1.41524 12.52396 1.00529 115.76298Geome_Obs_Mean inf inf inf inf 0.77472 0.47292 2.59834 infGeome_Pre_Mean inf 0.34576 inf 2.33338 1.02932 2.61811 0.70295 infStand_Devi_Obs 68.77800 14.01594 31.41555 3.03482 1.23243 0.57163 2.77285 72.22562Stand_Devi_Pre 59.70288 18.79670 12.73284 3.00580 1.22587 16.51884 1.02638 74.72571Corr_Variance 6757.48828 187.37831 828.02985 12.87626 3.19857 428.50751 12.46861 10360.89551Coeff_Determi 0.21936 0.47704 0.19945 0.32257 0.00018 0.00126 0.05701 0.10716Bias_Factor 0.61863 0.59935 0.84369 0.58722 2.56912 4.22906 0.57726 0.70125Num_Points 512 512 510 510 512 512 512 512.Table 4. Model Performance Statistics for Dolly Sods12km Domain for the 20% Worst DaysAllDay_DOSO1 bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXTFraction_Bias(%) -72.68279 -116.04311 -46.03989 -106.63990 -18.5469 191.30435 -133.52966 -70.31940Fraction_GErr(%) 79.51933 127.19271 65.51572 106.63990 42.44298 191.30435 133.52966 75.19762Ac_Paired_Peak -0.44867 0.19119 -0.36860 -0.72607 -0.37333 -999 -0.93009 -0.45986Norm_Mean_Bias(%) -47.86574 -60.19393 -34.19370 -69.89289 -17.5115 -999 -86.5834 -47.82595Norm_Mean_Err(%) 54.03939 77.85197 49.84565 69.89289 34.43992 -999 86.5834 52.20642Mean_Norm_Bias(%) -45.39338 -58.71173 -26.37871 -68.32784 3.87789 -999 -77.99408 -45.29449Mean_Norm_GErr(%) 53.22950 75.27402 50.74098 68.32784 49.74221 -999 77.99408 50.65216Mean_Bias -73.36999 -1.72433 -4.04288 -2.89387 -0.14377 3.03404 -1.36339 -88.24226Mean_Abs_GErr 82.83313 2.23016 5.89348 2.89387 0.28275 3.03404 1.36339 96.32455Geometric_Bias -0.76514 -1.91731-0.51228 -1.23143 -0.18728 -999 -1.76929 -0.71771Geometric_Err 0.83704 2.05064 0.70998 1.23143 0.45573 -999 1.76929 0.76886Root_Mean_SErr 113.34650 2.55479 6.90598 3.11407 0.38 3.37322 1.78193 129.42297Obs_Max 404.98950 5.68820 26.51380 7.50300 1.7766 0 4.8451 451.53500Pre_Max 223.28340 6.77574 16.74087 2.05532 1.11334 5.66848 0.33873 243.89001Arith_Obs_Mean 153.28288 2.86462 11.82346 4.14043 0.821 0 1.57466 184.50706Arith_Pre_Mean 79.91290 1.14029 7.78058 1.24657 0.67723 3.03404 0.21127 96.26483Geome_Obs_Mean 139.35542 2.67933 10.59959 3.98567 0.73189 -999 1.1725 172.30568Geome_Pre_Mean 61.76841 0.40035 6.35049 1.16331 0.60689 -999 0.19986 80.46085Stand_Devi_Obs 78.10185 1.10104 5.70874 1.21787 0.3246 0 1.21084 80.44410Stand_Devi_Pre 63.82932 1.72554 4.40868 0.43381 0.28867 1.50731 0.06943 68.62742Corr_Variance 12847.42969 6.52696 47.69260 9.69741 0.1444 11.37858 3.17527 16750.30469Coeff_Determi 0.05653 0.01560 0.14631 0.07446 0.10027 -999 0.31719 0.02687Bias_Factor 0.54607 0.41288 0.73621 0.31672 1.03878 0 0.22006 0.54706Num_Points 23 23 23 23 23 23 23 23Table 5. Model Performance Statistics for theVISTAS 12km Domain for the 20% Best DaysAllSite_AllDay bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXTFraction_Bias(%) -115.19897 -110.05685 -25.87558 -27.45512 78.49191 148.33353 -81.5044 -81.14001Fraction_GErr(%) 122.49083 132.74918 57.0146 55.29499 114.38155 184.30733 104.25344 88.50454Ac_Paired_Peak 2.47544 1.09211 2.15802 1.54198 1.73437 4.23499 -0.17893 1.84919Norm_Mean_Bias(%) -61.28178 -39.81734 -3.8786 -12.77569 213.71725 551.53925 -62.00702 -47.73489Norm_Mean_Err(%) 75.88204 86.49456 55.78643 52.02515 266.12604 711.03851 70.47713 59.48319Mean_Norm_Bias(%) -61.79248 -40.68147 0.76093 8.91848 682.14447 2051.71558 -38.65422 -48.42155Mean_Norm_GErr(%) 77.67429 91.64355 58.35357 68.09431 704.19354 2147.78906 78.75758 60.37934Mean_Bias -9.52442 -1.76987 -0.18369 -0.24939 0.62756 2.8772 -1.335 -19.42246Mean_Abs_GErr 11.79359 3.84466 2.64201 1.01557 0.78146 3.70925 1.51736 24.20263Geometric_Bias -1.70498 -2.26525 -0.30405 -0.32102 1.06687 -0.67838 -1.09384 -1.00189Geometric_Err 1.78588 2.55867 0.64003 0.60994 1.55508 2.23195 1.2741 1.07966Root_Mean_SErr 14.75895 6.23028 5.03647 1.96979 1.18422 7.4536 2.07051 27.92232Obs_Max 41.56650 30.4869 21.3894 9.964 2.945 15.3769 9.8644 65.35780Pre_Max 144.46182 63.78189 67.54807 25.32829 8.05273 80.49792 8.09939 186.21674Arith_Obs_Mean 15.54200 4.44497 4.73593 1.95207 0.29364 0.52167 2.15298 40.68818Arith_Pre_Mean 6.01758 2.6751 4.55224 1.70268 0.92121 3.39886 0.81798 21.26571Geome_Obs_Mean inf 3.33784 4.08561 1.63371 0 0.90877 1.54087 infGeome_Pre_Mean 2.50002 0.36746 3.01446 1.18511 0.52763 0.46114 0.51608 infStand_Devi_Obs 7.06463 3.92602 2.50424 1.09105 0.3156 1.78235 1.68186 10.11815Stand_Devi_Pre 10.74304 6.01116 5.33102 1.91406 0.97091 6.47083 0.91956 20.62970Corr_Variance 217.82658 38.81642 25.36601 3.88008 1.40237 55.55618 4.287 779.65582Coeff_Determi 0.06251 0.11192 0.1215 0.06062 0.00267 0.01017 0.1417 0.08924Bias_Factor 0.38207 0.59197 1.00761 1.08252 7.82144 4.56691 0.60219 0.51578Num_Points 490 490 490 490 490 490 490 490Table 6. Model Performance Statistics forDolly Sods12km Domain for the 20% Best DaysAllDay_DOSO1 bextSO4 bextNO3 bextOC bextEC bextSOIL bextSS bextCM totalBEXTFraction_Bias(%) -107.37399 -62.63736 -78.21556 -73.21972 104.57863 200 -84.35236 -97.79391Fraction_GErr(%) 109.36848 98.95137 78.33873 76.44373 110.05344 200 127.36462 97.79391Ac_Paired_Peak -0.04082 0.45353 -0.51611 -0.46999 2.4739 -999 -0.78878 -0.20720Norm_Mean_Bias(%) -62.79203 -24.5406 -57.40063 -49.56052 317.64697 -999 -79.16251 -61.85210Norm_Mean_Err(%) 65.69522 64.67429 57.44202 51.87252 321.70764 -999 84.3696 61.85210Mean_Norm_Bias(%) -64.89848 -18.18808 -52.98637 -48.24676 926.53986 -999 20.21571 -63.26039Mean_Norm_GErr(%) 67.01295 69.58811 53.11034 52.12767 931.31647 -999 143.06604 63.26039Mean_Bias -8.23004 -0.74423 -2.54523 -0.91678 0.45042 0.82124 -0.56105 -20.58063Mean_Abs_GErr 8.61056 1.96133 2.54706 0.95954 0.45618 0.82124 0.59796 20.58063Geometric_Bias -1.64028 -0.95529 -0.89866 -0.86919 1.53116 -999 -1.22902 -1.19088Geometric_Err 1.66025 1.36945 0.89989 0.90175 1.58632 -999 1.76542 1.19088Root_Mean_SErr 9.90401 2.683 3.15827 1.09748 0.58682 1.02365 0.79893 21.54301Obs_Max 23.23870 9.0214 9.0174 3.442 0.4287 0 1.927 43.68290Pre_Max 22.29012 13.11289 4.36343 1.8243 1.48926 2.75884 0.40702 34.63181Arith_Obs_Mean 13.10682 3.03263 4.43415 1.84981 0.1418 0 0.70874 33.27394Arith_Pre_Mean 4.87678 2.2884 1.88892 0.93303 0.59222 0.82124 0.14768 12.69331Geome_Obs_Mean 12.12716 2.3008 3.88151 1.69442 0.08938 -999 0.37565 32.53147Geome_Pre_Mean 2.35176 0.91345 1.58022 0.71045 0.41326 -999 0.10991 9.88810Stand_Devi_Obs 5.34473 2.39133 2.10428 0.76173 0.11039 0 0.60708 7.00297Stand_Devi_Pre 5.28463 3.0197 1.13213 0.52258 0.39587 0.62618 0.09535 7.56728Corr_Variance 98.08949 7.19851 9.97469 1.20445 0.34436 1.04787 0.63829 464.10141Coeff_Determi 0.18994 0.29621 0.18302 0.35023 0.0542 -999 0.10753 0.36167Bias_Factor 0.35102 0.81812 0.47014 0.51753 10.2654 0 1.20216 0.36740Num_Points 21 21 21 21 21 21 21 21