# Appendix 8.2

Model Performance Evaluation (TSD Chapter 3)August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-1 3.0 MODEL PERFORMANCE EVALUATION

In this Chapter we summarize the CMAQ model perform ance for the final 2002 36 km Base F base case simulation. This model performance focus es on the ability of the model to predict PM species within the CENRAP region. Details on the m odel performance are provided in Appendix C. Previously we have documented model pe rformance of interim versions of model base case simulations in reports (Morris et al., 20 05) and presentations to the CENRAP Work Groups and POG (e.g., Morris et al., 2006a,b).

3.1 Evaluation Methodology

EPA’s integrated ozone, PM 2.5 and regional haze modeling guidance calls for a co mprehensive, multi-layered approach to model performance testing , consisting of the four major components: operational, diagnostic, mechanistic (or scientific ) and probabilistic (EPA, 2007). The CMAQ model performance evaluation effort focused on the first two components, namely:

• Operational Evaluation : Tests the ability of the model to estimate PM conc entrations (both fine and coarse) and the components at PM 10 and PM 2.5 including the quantities used to characterize visibility (i.e., sulfate, nit rate, ammonium, organic carbon, elemental carbon, other PM 2.5 , and coarse matter (PM 2.5-10 ). This evaluation examines whether the measurements are properly represented by the model predictions but does not necessarily ensure that the model is getting “the right answer for the right reason”; and

• Diagnostic Evaluation: Tests the ability of the model to predict visibilit y and extinction, PM chemical composition including PM precursors (e. g., SOx, NOx, and NH 3 ) and associated oxidants (e.g., ozone and nitric acid); PM size distribution; temporal variation; spatial variation; mass fluxes; and components of l ight extinction (i.e., scattering and absorption).

In this final model performance evaluation for the 2002 Typical Base F CMAQ simulation, the operational evaluation has been given the greatest attention since this is the primarily thrust of EPA’s modeling guidance. However, we have also exa mined certain diagnostic features dealing with the model’s ability to simulate sub-regional a nd monthly/diurnal gas phase and aerosol concentration distributions. In the course of the CENRAP and other modeling process numerous diagnostic sensitivity tests were performe d to investigate and improve model performance. Key diagnostic tests performed are di scussed and the results for the rest are available on the CENRAP modeling website: http://pah.cert.ucr.edu/aqm/cenrap/index.shtml

.August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-2 3.2 Ambient Air Quality Data used in the Evaluatio n

The ground-level model evaluation database for 2002 was compiled by the modeling team using several routine and research-grade databases. The first is the routine gas-phase concentration measurements for ozone, SO2, NO 2 and CO archived in EPA’s Aerometric Information Re trieval System (AIRS) Air Quality System (AQS) database. O ther sources of observed information come from the various PM monitoring networks in the U.S. These include the Interagency Monitoring of Protected Visual Environments (IMPROV E); Clean Air Status and Trends Network (CASTNET); EPA Speciation Trends Network (S TN) of PM 2.5 species; and National Acid Deposition Network (NADP). During the course of the CENRAP modeling, the numerous base case simulations were evaluated across the con tinental U.S. (e.g., Morris et al., 2005). In this section and in Appendix C we focus our evaluat ion on model performance within the CENRAP region.

3.2 Operational Model Evaluation Approach

The CENRAP modeling databases will be used to devel op the visibility State Implementation Plan (SIP) as required by the Regional Haze Rule (R HR). Accordingly, the primary focus of the operational evaluation in this report is on the six components of fine particulate (PM 2.5 ) and coarse mass (PM 2.5-10 ) within the CENRAP region that are used to charact erize visibility at Class
I areas:

• Sulfate (SO4);

• Particulate Nitrate (NO3);

• Elemental Carbon (EC);

• Organic Mass Carbon (OMC);

• Other inorganic fine particulate (IP or Soil); and

• Coarse Mass (CM).

The model performance for ozone and precursor and p roduct species (e.g., SO 2 and HNO 3 ) is also evaluated to build confidence that the modelin g system is sufficiently reliable to project future-year visibility.

3.3 Model Performance Goals and Criteria

The issue of model performance goals for PM species is an area of ongoing research and debate. For ozone modeling, EPA has established performance goals for 1-hour ozone normalized mean bias and gross error of #± 15% and # 35%, respectively (EPA, 1991). EPA’s draft fine pa rticulate modeling guidance notes that performance goals for ozone should be viewed as upper bounds of model performance that PM models may not be able to always achieve and we should demand better model performance for PM components that mak e up a larger fraction of the PM mass than those that are minor contributors (EPA, 2001). EPA’s final modeling guidance does not list any specific model performance goals for PM and vis ibility modeling and instead provides a summary of PM model performance across several hist orical applications that can be used forAugust 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-3 comparisons if desired. Measuring PM species is no t as precise as ozone monitoring. In fact, the differences in measurement techniques for some spec ies likely exceed the more stringent performance goals, such as those for ozone. For ex ample, recent comparisons of the PM species measurements using the IMPROVE and STN measurement technologies found differences of approximately ∀ 20% (SO4) to ∀ 50% (EC) (Solomon et al., 2004).

For the CENRAP modeling we have adopted three level s of model performance goals and criteria for bias and gross error as listed in Tabl e 3-1. Note that we are not suggesting that these performance goals be adopted as guidance. Rather, we are just using them to frame and put the PM model performance into context and to facilitate model performance intercomparison across episodes, species, models and sensitivity tests.

Table 3-1. **Model performance goals and criteria used to assi st in interpreting modeling results** 

. Fractional Bias Fractional

Error Comment

#∀ **15%** # **35% Ozone model performance goal for which PM model**  **performance would be considered good – note that fo r**  **many PM species measurement uncertainties may excee d**  **this goal.**

#∀ **30%** # **50% Proposed PM model performance goal that we would ho**  **pe each PM species could meet** 

#∀ **60%** # **75% Proposed PM criteria above which indicates potentia**  **l fundamental problems with the modeling system.** 

As noted in EPA’s PM modeling guidance, less abunda nt PM species should have less stringent performance goals (EPA, 2001; 2007). Accordingly, we are also using performance goals that are a continuous function of average concentrations , as proposed by Dr. James Boylan at the Georgia Department of Natural Resources (GA DNR), t hat have the following features (Boylan, 2004):

• Asymptotically approaching proposed performance goa ls or criteria (i.e., the ∀ 30%/50%
and ∀ 60%/75% bias/error levels listed in Table 3-1) when the mean of the observed concentrations are greater than 2.5 ug/m 3

.

• Approaching 200% error and ∀ 200% bias when the mean of the observed concentrati ons are extremely small.

Bias and error are plotted as a function of average concentrations. As the mean concentration approach zero, the bias performance goal and criter ia flare out to ∀ 200% creating a horn shape, hence the name “Bugle Plots”. Dr. Boylan has defin ed three Zones of model performance: Zone
1 meets the ∀ 30%/50% bias/error performance goal and is consider ed “good” model performance; Zone 2 lies between the ∀ 30%/50% performance goal and ∀ 60%/75% performance criteria and is an area where concern f or model performance is raised; and Zone 3
lies above the ∀ 60%/75% performance criteria and is an area of ques tionable model performance.August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-4 3.4 Key Measures of Model Performance

Although we have generated numerous statistical per formance measures (see Table C-2 in Appendix C) that are available on the CENRAP model ing website, when comparing model performance across months, subdomains, networks, gr id resolution, models, studies, etc. it is useful to have a few key measurement statistics to be used to facilitate the comparisons. It is also useful to have a subset of the 2002 year that can represent the entire year so that a more focused evaluation can be conducted. We have found that the Mean Fractional Bias and Mean Fractional Gross Error appear to be the most consis tent descriptive measure of model performance (Morris et al., 2004b; 2005). The Frac tional Bias and Error normalize by the average of the observed and predicted value (see Ta ble C-2) because it provides descriptive power across different magnitudes of the model and observed concentrations and is bounded by -200% to +200%. This is in contrast to the normali zed bias and error (as recommended for ozone performance goals, EPA, 1991) that is normali zed by just the observed value so can “blow up” to infinity as the observed value approaches ze ro. In Appendix C we perform a focused evaluation of model performance for PM and gaseous species and four months of the 2002 year that are used to represent the seasonal variation i n performance:

• January

• April

• July

• October

Scatter plots of model predictions and observations for each PM species is presented for each of the four months along with performance statistics a nd predicted and observed time series plots at each CENRAP Class I area. Summary plots of monthly fractional bias and error are also presented.

3.5 Operational Model Performance Evaluation

A summary of the operational evaluation is presente d below. Just the monthly fractional bias performance metrics for each PM species using bar c harts and Bugle Plots are presented in this section. The reader is referred to Appendix C for the complete model performance evaluation.

3.5.1 Sulfate (SO4) Model Performance

Figure 3-1 compares the monthly SO4 fractional bias and error across the CENRAP region for the IMPROVE, STN and CASTNet monitoring networks. An under-prediction bias is clearly evident the first 8-10 months of the year. This un derestimation bias is greatest across the CASTNet network which persists throughout the year and is least for the STN network where it disappears by August-September. For the IMPROVE ne twork, the SO4 fractional bias is < ±20% for the first 2 and last 3 months of the yea r and ranges from -30% to -50% for the late Spring and Summer months.

Figure 3-1 also includes a Bugle Plot of monthly SO 4 fractional bias and error statistics and compares them against the proposed PM model perform ance goal and criteria (see Table 3-1).August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-5 For the STN network, it appears that SO4 performanc e for all months achieves the proposed PM model performance goal. For the IMPROVE network, a pproximately half of the months achieve the proposed PM performance goal with the other hal f exceeding the goal but within the performance criteria. Across the CASTNet network, most months exceed the proposed goal and are within the criteria. Although the CASTNet frac tional bias for some months is right at the performance criteria (≤±60%). With the exception o f two IMPROVE months, all of the monthly SO4 fractional error performance statistics achieve the proposed PM model performance goal.

3.5.2 Nitrate (NO3) Model Performance

Monthly NO3 model performance across he CENRAP regi on is characterized by a summer underestimation and winter overestimation bias (Fig ure 3-2). The summer underestimation bias is more severe exceeding -100%, whereas the winter overestimation bias is approximately 50%. So based on statistics alone, it appears the summer underestimation bias is a bigger concern than the winter overestimation bias. However, the Bugle Plots in the bottom part of Figure 3-2 shows that the summer underestimation bias occurs when NO 3 is very low and is not an important component of PM and visibility impairment. These s ummer values occur in the flared horn part of the Bugle Plot and in fact the summer NO3 perfor mance mostly achieves the model performance goal and always achieves the performanc e criteria. Whereas the winter overstated NO3 performance mostly doesn’t meet the performance goal and there are even some months/networks that don’t meet the performance cri teria.

3.5.3 Organic Matter Carbon (OMC) Model Performance

The OMC monthly fractional bias across IMPROVE and STN sites in the CENRAP region are shown in Figure 3-3. The bias performance for OMC at the IMPROVE sites is quite good throughout the year with values generally within ±2 0%, albeit with a slight winter overestimation and summer underestimation bias. At the urban STN sites, the model exhibits an underestimation bias throughout the year that range s from -20% to -50%. The urban underestimation of OMC is a fairly common occurrenc e and suggests there may be missing sources of organic aerosol emissions.

The good performance of the model for OMC at the IM PROVE sites is also reflected in the Bugle Plot (Figure 3-3, bottom) with the bias and e rror achieving the proposed PM model performance goal for all months of the year. At th e STN sites, however, the OMC bias falls between the proposed PM model performance goal and criteria, with error right at the goal for most months.August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-6 Figure 3-1. **Monthly fractional bias (%) for sulfate (SO4) acr oss the CENRAP region for the**  **CMAQ 2002 36 km Base F base case simulation.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-7 Figure 3-2. **Monthly fractional bias (%) for nitrate (NO3) acr oss the CENRAP region for the**  **CMAQ 2002 36 km Base F base case simulation.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-8 Figure 3-3. **Monthly fractional bias (%) for organic matter ca rbon (OMC) across the CENRAP**  **region for the CMAQ 2002 36 km Base F base case sim ulation.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-9 3.5.4 Elemental Carbon (EC) Model Performance

The monthly average bias and error for EC across th e IMPROVE and STN monitors in the CENRAP region are shown in Figure 3-4. The STN net work exhibits low bias year round, whereas the IMPROVE monitoring network exhibits a l arge under-prediction bias in the summer months (-40% to -60%) and much lower EC bias in the winter. The Bugle Plot puts the EC performance in context. The low EC concentrations at the IMPROVE sites results in bias values in the horn of the Bugle Plot. Thus, EC bias and e rror performance achieves the proposed PM performance goal for all months of the year.

3.5.5 Other PM 2.5 (Soil) Model Performance

Figure 3-5 displays the monthly variation in the So il fractional bias using IMPROVE measurements in the CENRAP region. During the wint er months, the model exhibits a very large (> 100%) overestimation bias. With the excep tion of July, the summer monthly bias is toward a slight over-prediction but generally less than 20% with errors of 60% to 80%. The July underestimation bias appears to be driven by impact s of high Soil values from wind blown dust events (e.g., see July 2002 discussion in Appendix C). The Bugle Plot indicates that the summer Soil performance achieves the PM performance goal, a few months in the Spring/Fall period fall between the performance goal and criteria and the w inter Soil performance exceeds the model performance criteria. Thus, the Soil performance i s a cause for concern.

3.5.6 Coarse Mass (CM) Model Performance

The monthly average fractional bias values for CM a re shown in Figure 3-6. In the winter the under-prediction bias is typically in the -60% to - 80% range. In the late Spring and Summer the under-prediction bias ranges from -120% to -160%. As this under-prediction bias is nearly systematic, then the errors are the same magnitude as the bias.

The Bugle Plots clearly show that the CM model perf ormance is a problem. The monthly bias exceeds both the performance goal and criteria for almost every month of the year. The error criteria are also exceeded for all months of the ye ar.August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-10 Figure 3-4. **Monthly fractional bias (%) for elemental carbon (EC) across the CENRAP region**  **for the CMAQ 2002 36 km Base F base case simulation**

**.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-11 Figure 3-5. **Monthly fractional bias (%) for other PM 2.5**  **(Soil) across the CENRAP region for the CMAQ 2002 36 km Base F base case simulation.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-12 Figure 3-6. **Monthly fractional bias (%) for coarse mass (CM) across the CENRAP region for**  **the CMAQ 2002 36 km Base F base case simulation.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-13 3.6 Diagnostic Model Performance Evaluation

The CASTNet and AQS networks also measure gas-phase species that are PM precursor or related species. The diagnostic evaluation of the 2002 36 km Base F CMAQ base case simulation for these compounds and the four seasona l months are presented in Appendix C. The displays for January are provided below as an examp le; the reader is referred to Appendix C for the rest of the monthly displays.

The CASTNet network measures weekly average samples of SO2, SO4, NO2, HNO3, NO3 and NH4. The AQS network collects hourly measurements of SO2, NO2, O3 and CO. A comparison of the SO2 and SO4 performance provides insight into whether the SO4 formation rate may be too slow or fast. For example, if SO4 is underestimated and SO2 is overestimated that may indicate too slow chemical conversion rate s. Analyzing the performance for SO4, HNO3, NO3, Total NO3 and NH4 provides insight into the equilibrium of these species. For example, if Total NO3 performs well but HNO3 and NO 3 do not, then there may be issues associated with the partitioning between the gaseou s and particle phases of nitrate. Causes for incorrect HNO3/NO3 partitioning could include inade quate ammonia emissions and/or poorly characterized meteorological conditions (e.g., temp erature).

3.6.1 Diagnostic Model Performance in January 2002

In January, SO2 is overstated across both the CASTN et and AQS sites with fractional bias values of 38% (Figure 3-7) and 31% (Figure 3-8), respectiv ely. SO4 is understated by -34% across the CASTNet monitors (Figure 3-7) and -12% and -13% for the IMPROVE and STN networks (Figure C-4a). Wet SO4 deposition is also overstat ed in January (+40%, Figure C-4a). Given that SO2 emissions are well characterized, these re sults suggest that the January SO4 underestimation may be partly due to understated tr ansformation rates of SO2 to SO4 and overstated wet SO4 deposition.

Total NO3 is overestimated by 35% on average across the CASTNet sites in the CENRAP region in January (Figure 3-7). HNO3 is underestim ated (-34%) and particle NO3 is overestimated (+61%) suggesting there are gas/parti cle equilibrium issues. An analysis of the time series of the four CASTNet stations reveals th at NO3, HNO3 and NH4 performance is actually very reasonable at the west Texas site and the HNO3 underestimation and NO3 overestimation bias is coming from the east Kansas, central Arkansas and northern Minnesota CASTNet sites (see Figure C-3 for site locations). One potential contributor for this performance problem could be overstated NH3 emissio ns. However, the Total NO3 overestimation bias suggests that the model estimat ed NOx oxidation rate may be too high in January.

The SO2, NO2, O3 and CO performance across the AQS sites in January is shown in Figure 3-8. The AQS monitoring network is primarily an urban-or iented network so it is not surprising that the model is underestimating concentrations of prim ary emissions like NO2 (-5%) and, particularly, CO (-67%) when a 36 km gri d is used. Ozone is also underestimated on average, especially the maximum values above 60 ppb

.August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-14 Figure 3-7. **January 2002 performance at CENRAP CASTNet sites for SO2 (top left), SO4 (top**  **right), HNO3 (middle left), NO3 (middle right), Tot al NO3 (bottom left) and NH4 (bottom right).** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-15 Figure 3-8. **January 2002 performance at CENRAP AQS sites for SO2 (top left), NO2 (top right),**  **O3 (bottom left) and CO (bottom right).**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-16 3.6.2 Diagnostic Model Performance In April

In April there is an average SO2 overestimation bia s across the CASTNet (+15%) and underestimation bias across the AQS (-10%) networks (Figures C-42 and C-43). SO4 is underestimated across all networks by -30% to -58% (Figure C-5a). The wet SO4 deposition bias is near zero. Both SO2 and SO4 are underest imated at the west Texas CASTNet monitor in April suggesting SO2 emissions in Mexico are lik ely understated.

The HNO3 performance in April is interesting with a lmost perfect agreement except for 5 modeled-observed comparisons that drives the averag e under-prediction bias of -29% (Figure C42). On Julian Day 102 there is high HNO3 at the M N, KS and OK CASTNet sites that is not captured by the model. Given that HNO3, NO3 and To tal NO3 are all underestimated by about the same amount (-30%), then part of the underestim ation bias is likely due to too slow oxidation of NOx.

There is a lot of scatter in the NO2 and O3 perform ance that is more or less centered on the 1:1 line of perfect agreement with bias values of -8% a nd -21%, respectively (Figure C-43). CO is underestimated by -72% with the model unable to pre dict CO concentrations above 1 ppm due to the use of the coarse 36 km grid spacing. Mobile s ources produce a vast majority of the CO emissions so AQS monitors for CO compliance are loc ated near roadways, which are not simulated well using a 36 km grid.

3.6.3 Diagnostic Model Performance In July

In July SO2 is slightly underestimated across the C ASTNet (-5%) and AQS (-12%) networks (Figures C-44 and C-45) and SO4 is more significant ly underestimated across all networks (-22% to -53%, Figure C-6a). Since wet SO4 is also underestimated it is unclear the reasons for why all sulfur species are underestimated.

The nitrate species are also all underestimated wit h the Total NO3 bias (-56%) being between the HNO3 bias (-35%) and NO3 bias (-115%). The modeled NO3 values are all near zero with little correlation with the observations, whereas the obse rved HNO3 and Total NO3 is tracked well with correlation coefficients of 0.74 and 0.76. Th ese results suggest that the July NO3 model performance problem is partly due to insufficient f ormation of Total NO3 but mainly due to too little incorrect partitioning of the Total NO3 into the particle NO3.

Again there is lots of scatter in the AQS NO2 scatt er plot for July (Figure C-45) resulting in a low bias (0%) but high error (65%). Ozone performa nce also exhibits a low bias (-15%) and error (20%), but the model is incapable of simulati ng ozone above 100 ppb. Although CO performance in July is better than the previous mon ths, it still has a large underestimation bias (- 82%).August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-17 3.6.4 Diagnostic Model Performance In October

SO2 is overstated in October across the CASTNet (+2 8%) and AQS (+33%) sites (Figures C-46 and C-47). Although SO4 is understated across the CASTNet sites (-24%), the bias across the IMPROVE (-6%) and STN (0%) sites are near zero (Fig ure C-7a).

Performance for HNO3 is fairly good with a low bias (+12%) and error (30%). But NO3 is overstated ( +34%) leading to an overstatement of T otal NO3 (+37%). The overstatement of NO3 leads to an overstatement of NH4 as well (Figur e C-46)

As seen in the other months, NO2 exhibits a lot of scatter resulting in a low correlation (0.22) and high error (61%) but low bias (12%). The model tends to under-predict the high and overpredict the low O3 observations resulting in a -29% bias and low correlation coefficient. CO is also under-predicted (-76%) for the reasons discuss ed previously.

3.7 Performance at CENRAP Class I Areas for the Worst a nd Best 20 Percent Days

In this section, and in section C.5 of Appendix C, we present the results of the model performance evaluation at each of the CENRAP Class I areas for the worst and best 20 percent days. Performance on these days is critical since they are the days used in the 2018 visibility projections discussed in Chapter 4. For each Clas s I area we compared the predicted and observed extinction of the worst and best 20 percen t days below. In Appendix C the PM speciesspecific extinction is also compared for the worst 20 percent days.

3.7.1 Caney Creek (CACR) Arkansas

The ability of the CMAQ model to estimate visibilit y extinction at the CACR Class I area on the 2002 worst and best 20 percent days is provide in F igures 3-9 and C-48. On most of the worst 20 percent days at CACR total extinction is dominat ed by SO4 extinction with some extinction due to OMC. On four of the worst 20 percent days e xtinction is dominated by NO3. The average extinction across the worst 20 percent days is underestimated by -33% (Figure 3-9), which is primarily due to a -51% underestimation of SO4 extinction combined with a 6% overestimation of NO3 extinction (Figure C-48). Pe rformance for OMC extinction at CACR on the worst 20 percent days is pretty good with a -20 % bias and 36% error, EC extinction is systematically underestimated, Soil extinction has low bias (-19%) buts lots of scatter and high error (74%), while CM extinction is greatly underes timated (bias of -153%).

On the best 20 percent days at CACR the observed ex tinction ranges from 20 to 40 Mm -1

, whereas then modeled extinction has a much larger r ange from 15 to 120 Mm -1 . Much of the modeled overestimation of total extinction on the b est 20% days (+44% bias) is due to NO3 overestimation (+94% bias).August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-18 Worst 20% Obs (left) vs Typ02g (right) at CACR1

**0 50**  **100 150**  **200 250**  **300 350**  **400**

**80 128 143 155 170 173 188 191 200 203 215 218 221 239 242 248 251 2 54 257 260 296 320 341 _ _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at CACR1

**0 20**  **40 60**  **80 100**  **120 140** 

**11 14 26 89 98 116 197 266 278 284 287 293 299 302 311 323 329 335 33 8 353 356 359 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-9. **Daily extinction model performance at Caney Creek ( CACR), Arkansas for the**  **worst (top) and best (bottom) 20 percent days durin g 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-19 3.7.2 Upper Buffalo (UOBU) Arkansas

Model performance at the UPBU Class I area for the worst and best 20 percent days is shown in Figures 3-10 and C-49. On most of the worst 20 per cent days at UPBU visibility impairment is dominated by SO4, although there are also two high NO3 days. The model underestimates the average of the total extinction on the worst 20 per cent days at UPBU by -40% (Figure 3-10), which is due to an underestimation of extinction du e to SO4, OMC and CM by, respectively, -46%, -33% and -179%.

On the best 20 percent days at UPBU, the model perf orms reasonably well with a low bias (2%) and error (42%). But again the model has a much wi der range in extinction values across the best 20 percent days (15 to 120 Mm -1 ) than observed (20 to 45 Mm -1 ). There are five days in which the modeled NO3 over-prediction is quite seve re and when those days are removed the range in the modeled and observed extinction on the best 20 percent days is quite similar, although the model gets much cleaner on the very cl eanest modeled days.

Worst 20% Obs (left) vs Typ02g (right) at UPBU1

**0 50**  **100 150**  **200 250**  **300 350** 

**128 143 158 170 173 179 191 197 203 206 215 218 221 224 239 242 248 251 254 257 269 296 320 332 _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**
**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at UPBU1

**0 20**  **40 60**  **80 100**  **120**

**14 26 47 68 89 98 116 119 164 167 176 227 266 278 281 287 290 302 311 323 335 353 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-10. **Daily extinction model performance at Upper Buffalo (UPBU), Arkansas for the**  **worst (top) and best (bottom) 20 percent days durin g 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-20 3.7.3 Breton Island (BRET), Louisiana

The observed total extinction on the worst 20 perce nt days at Breton Island is underestimated by -71% (Figure 3-11), which is due to an underestimat ion of each component of extinction (Figure C-50) by from -50% to -70% (SO4, OMC and Soil) to o ver -100% (EC and CM). The observed extinction on the worst 20 percent days ranges from 90 to 170 Mm -1 , whereas the modeled values drop down to as low as approximately 15 Mm -1 . On the best 20 percent days the range of the observed and modeled extinction is similarly (roughly 10 to 50 Mm -1 ) that results in a reasonably low bias (-22%), but there is little agr eement on which days are higher or lower resulting in a lot of scatter and high error (54%).

Worst 20% Obs (left) vs Typ02g (right) at BRET1

**0 20**  **40 60**  **80 100**  **120 140**  **160 180** 

**44 47 74 125 143 179 194 197 200 206 215 239 248 251 254 257 260 263 272 281 284 287 305 311 _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at BRET1

**0 5**  **10 15**  **20 25**  **30 35**  **40 45**  **50**

**8 14 32 38 68 101 119 122 158 170 176 185 188 191 209 212 233 236 245 299 302 335 356 _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-11. **Daily extinction model performance at Breton Island (BRET), Louisiana for the worst**  **(top) and best (bottom) 20 percent days during 2002**

**.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-21 3.7.4 Boundary Waters (BOWA), Minnesota

There are three types of days during the worst 20 p ercent days at BOWA, SO4 days, OMC days and NO3 days (Figure 3-12). The two high OMC days are likely fire impact events that the model captures to some extent on one day and not on the other. On the five high (> 20 Mm -1 ) NO3 extinction days the model predicts the observed extinction well on three days and overestimates by a factor of 3-4 on the other two h igh NO3 days. SO4 in underestimate by -43% on average across the worst 20 percent days at BOWA

.
With the exception of two days, the model reproduce s the total extinction for the best 20 percent days at BOWA quite well with a bias and error value of +14% and 22% (Figure 3-12). Without these two days, the modeled and observed extinction both range between 15 and 25 Mm -1

.

Worst 20% Obs (left) vs Typ02g (right) at BOWA1

**0 20**  **40 60**  **80 100**  **120 140**  **160**

**11 26 146 149 161 170 176 179 188 197 200 203 242 245 251 275 299 31 4 332 344 347 _ _ _ _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at BOWA1

**0 5**  **10 15**  **20 25**  **30 35**  **40**

**29 53 62 83 110 116 122 125 134 137 140 185 218 230 239 266 329 350 3 56 359 _ _ _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-12. **Daily extinction model performance at Boundary Wate rs (BOWA), Minnesota for the**  **worst (top) and best (bottom) 20 percent days durin g 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-22 3.7.5 Voyageurs (VOYA) Minnesota

VOYA is also characterized by SO4, NO3 and OMC days (Figure 3-13). Julian Days 179 and 200 are high OMC days that were also high OMC days at BOWA again indicating impacts from fires in the area that is not fully captured by the model. SO4 and NO3 extinction is fairly good and, without the fire days, OMC performance looks g ood as well (Figure C-52). On the best 20 percent days there is one day the modeled extinctio n is much higher than observed and a few others that are somewhat higher, but for most of th e best 20 percent days the modeled extinction is comparable to the observed values.

Worst 20% Obs (left) vs Typ02g (right) at VOYA2

**0 50**  **100 150**  **200 250** 

**5 11 71 86 104 131 146 152 170 179 188 197 200 221 242 245 254 260 27 2 284 299 332 344 347 _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at VOYA2

**0 5**  **10 15**  **20 25**  **30 35**  **40 45** 

**29 47 50 56 116 122 125 128 134 137 140 185 203 218 239 266 275 290 2 96 329 350 356 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-13. **Daily extinction model performance at Voyageurs (VO YA), Minnesota for the**  **worst (top) and best (bottom) 20 percent days durin g 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-23 3.7.6 Hercules Glade (HEGL) Missouri

On most of the worst 20 percent days at HEGL the ob served extinction ranges from 120 to 220
Mm -1 whereas model extinction ranging from 50 to 170 Mm -1 (Figure 3-14). However, there is one extreme day with extinction approaching 400 Mm -1 that the model does a very good job in replicating. Over all the days there is a modest u nderestimation bias in SO4 (-39%) and OMC (-39%) extinction, larger underestimation bias in E C (-62%) and CM (-118%) extinction and overestimation bias in Soil (+30%) extinction (Figu re C-53).

On the best 20 percent days there is one day where the model overstates the observed extinction by approximately a factor of four and a handful of other days that the model overstates the extinction by a factor of 2 or so, but most of the days both the model and observed extinction sites are around 40 Mm -1 ±10 Mm -1 . On the best 20 percent days when the observed ex tinction is overstated it is due to overstatement of the NO3

.

Worst 20% Obs (left) vs Typ02g (right) at HEGL1

**0 50**  **100 150**  **200 250**  **300 350**  **400**

**5 149 152 170 173 179 188 191 206 215 218 221 224 239 242 245 248 25 1 257 260 269 296 320 341 347 Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at HEGL1

**0 20**  **40 60**  **80 100**  **120 140**  **160**

**14 26 44 68 71 89 113 116 119 134 137 227 266 278 281 287 290 302 308 311 317 335 353 _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-14. **Daily extinction model performance at Hercules Glad e (HEGL), Missouri for the**  **worst (top) and best (bottom) 20 percent days durin g 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-24 3.7.7 Mingo (MING) Missouri

The worst 20 percent days at Ming are mainly high S O4 days with a few high NO3 days that the model reproduces reasonably well resulting in low b ias (+10%) and error (38%) for total extinction (Figure 3-15). The PM species specific performance is fairly good with low bias for SO4 (+4%), good agreement with NO3 on high NO3 days except for one day, low OMC (+23%) and EC (+3%) bias and larger bias in EC (+37%) and CM (-105%) extinction (Figure C-54).

For the best 20 percent days, there is one day the model is way to high due to overstated NO3 extinction and a few other days the model overstate s the observed extinction that is usually due to overrated NO3, but on most of the best 20 percen t days the modeled extinction is comparable to the observed values. This results in low bias ( +12%) and error (36%) for total extinction at MING for the best 20 percent days.

Worst 20% Obs (left) vs Typ02g (right) at MING1

**0 50**  **100 150**  **200 250**  **300 350**  **400**

**20 77 95 158 173 185 188 191 194 197 206 215 218 221 239 242 245 251 272 296 332 341 _ _ _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at MING1

**0 20**  **40 60**  **80 100**  **120 140**  **160 180**  **200**

**14 32 44 62 116 134 137 140 164 167 176 200 227 263 266 278 281 287 2 90 308 353 356 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-15. **Daily extinction model performance at Mingo (MING), Missouri for the worst (top)**  **and best (bottom) 20 percent days during 2002.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-25 3.7.8 Wichita Mountains (WIMO), Oklahoma

With the exception of an over-prediction on day 344 due to NO3, observed total extinction on the worst 20 percent days at WIMO is understated wi th a bias of -42% (Figure 3-16) that is primarily due to an underestimation of extinction d ue to SO4 (-48%) and OMC (-69%) (Figure C-55).

CMAQ total extinction performance for the average o f the best 20 percent days at WIMO is characterized by an overestimation bias (+21%) on m ost days that is primarily due to NO3 overprediction on several days. Again the modeled rang e of extinction on the best 20 percent days (12-60 Mm -1 ) is much greater than observed (20-35 Mm -1 ).

Worst 20% Obs (left) vs Typ02g (right) at WIMO1

**0 50**  **100 150**  **200 250** 

**29 83 95 101 110 113 125 128 131 143 170 173 176 179 218 221 224 242 245 254 260 269 272 344 _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at WIMO1

**0 10**  **20 30**  **40 50**  **60**

**8 11 14 26 32 41 44 47 53 98 116 263 278 281 287 302 311 314 317 323 35 3 356 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-16. **Daily extinction model performance at Wichita Mount ains (WIMO), Oklahoma for**  **the worst (top) and best (bottom) 20 percent days d uring 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-26 3.7.9 Big Bend (BIBE) Texas

The observed extinction on the worst 20 percent day s at BIBE is under-predicted on almost every day resulting in a fractional bias value of - 72% (Figure 3-17). Every component of extinction is underestimated on average for the wor st 20 percent days (Figure C-56) with the underestimation bias ranging from -24% (OMC) to -16 2% (CM). SO4 extinction, that typically represents the largest component of the total extin ction is understated by -94%.

The model does a better job in predicting the total extinction at BIBE for the best 20 percent days with average fractional bias and error values of +1 3% and 19% (Figure 3-17). With the exception of one day that the observed extinction i s overestimated by approximately a factor of 2, the modeled and observed extinction on the best 20 percent days at BIBE are both within 12 to 25 Mm -1 . However, there are some mismatches with the comp onents of extinction with the model estimating much lower contributions due to So il and CM.

Worst 20% Obs (left) vs Typ02g (right) at BIBE1

**0 20**  **40 60**  **80 100**  **120 140** 

**29 71 89 92 95 101 110 113 116 119 125 128 131 143 167 191 218 221 23 0 242 245 254 275 329 _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at BIBE1

**0 5**  **10 15**  **20 25** 

**11 14 38 56 77 194 212 263 281 287 299 302 308 320 326 338 344 347 35 0 353 359 362 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-17. **Daily extinction model performance at Big Bend (BIB E), Texas for the worst (top)**  **and best (bottom) 20 percent days during 2002.**August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-27 3.7.10 Guadalupe Mountains (GUMO) Texas

Most of the worst 30 percent days at GUMO are dust days with high Soil and CM that is not at all captured by the model (Figure 3-18). Extinctio n due to Soil and CM on the worst 20 percent days is underestimated by -105% and -191%, respecti vely (Figure C-57). Better performance is seen on the best 20 percent days with bias and erro r for total extinction of 8% and 21%, but the model still understates Soil and CM.

Worst 20% Obs (left) vs Typ02g (right) at GUMO1

**0 10**  **20 30**  **40 50**  **60 70**  **80 90**  **100**

**74 83 92 95 122 131 137 146 161 164 167 170 173 176 191 218 221 245 2 51 254 260 266 272 329 _ Avg**  Julian Day in Worst 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Best 20% Obs (left) vs Typ02g (right) at GUMO1

**0 5**  **10 15**  **20 25** 

**8 14 29 35 38 77 194 200 284 287 299 302 305 311 317 323 326 338 347 3 53 356 362 _ _ Av**  Julian Day in Best 20% groupbEXT (1/Mm) **bCM**

**bSOIL**

**bEC**

**bOC**

**bNO3**

**bSO4**

Figure 3-18. **Daily extinction model performance at Guadalupe Mou ntains (GUMO), Texas for**  **the worst (top) and best (bottom) 20 percent days d uring 2002.** August 2007

T:\BAR\Planning\Regional Haze SIP\Volume I\Kansas R egional Haze SIP and Appendices Oct 2009\Chapter 8 Appendices\Appendix 8.2 Model Performance Evaluatio n (TSD Chapter 3).doc 3-28 3.8 Model Performance Evaluation Conclusions

The model performance evaluation reveals that the m odel is performing best for SO4, OMC and EC. Soil performance is mixed with winter overesti mation bias but lower bias but high error in the summer. CM performance is poor year round. Th e operational evaluation reveals that SO4 performance usually achieves the PM model performan ce goal and always achieves the model performance criteria, although it does have an unde restimation bias that is greatest in the summer. NO3 performance is characterized by a wint er overestimation bias with an even greater summer underestimation bias. However, the summer u nderestimation bias occurs when NO3 is very low and it is not an important component of th e observed or predicted PM and visibility impairment. Performance for OMC meets the model pe rformance goal year round at the IMPROVE sites, but is characterized by an underesti mation bias at the more urban STN sites. EC exhibits very low bias at the STN sites and a su mmer underestimation bias at the IMPROVE sites, but meets the model performance goal through out the year. Soil has a winter overestimation bias that exceeds the model performa nce goal and criteria raising questions whether the model should be used for this species. Finally, CM performance is extremely poor with an under-prediction bias that exceeds the perf ormance goal and criteria. We suspect that much of the CM concentrations measured at the IMPRO VE sites is due to highly localized emissions that can not be simulated with 36 km regi onal modeling.

Performance for the worst 20 percent days at the CE NRAP Class I areas is generally characterized by an underestimation bias. Performa nce at the BRET, BIBE and GUMO Class I areas for the worst 20 percent days is particularly suspect and care should be taken in the interpretation of the visibility projections at the se three Class I areas.

The CMAQ 2002 36 km model appears to be working wel l enough to reliably make future-year projections for changes in SO4, NO3, EC and OMC at the rural Class I areas. Performance for Soil and especially CM is suspect enough that care should be taken in interpreting these modeling results. The model evaluation focused on the model’s ability to predict the components of light extinction mainly at the Class I areas. Additional analysis would have to be undertaken to examine the model’s ability to treat ozone and fine particulate to address 8-hour ozone and PM 2.5 attainment issues.