# Attachment F

Model Performance Evaluation

## Attachment F Contents

ASIP Model Performance Evaluation – Chap ter 3 of ASIP Technical Support Document

Appendix B of ASIP Technical Support Docu ment – State-by-State and Key Monitoring

Site PM 2.5 Model Evaluation

Appendix C of ASIP Technical Support Docume nt – Spatial Maps of Quarterly Average

PM 2.5 Components and Total Mass Concentrations with Superimposed Observations for the CMAQ 2002 12 km Base G2 Actual Base Case Simulation

Scatter Plots for K noxville-Area MonitorsJanuary 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-1 3.0 MODEL PERFORMANCE EVALUATION

3.1 INTRODUCTION

This Chapter summarizes the CMAQ 2002 36 km and 12 km Base G2 Actual base case simulation model performance evaluation. The focu s of the model performance evaluation is on

PM 2.5 and its component species in the urban and PM 2.5 nonattainment areas (NAAs). However, the evaluation of the CMAQ 2002 Base G2 base case simulation was also performed for PM species and visibility primarily at Class I areas as part of the VISTAS regional haze assessment.

The VISTAS model evaluation focused on the performance of PM component species and visibility at more rural (e .g., IMPROVE) monitors. ASIP also performed an urban ozone evaluation performance as part of the developmen t of the southeastern st ates 8-hour ozone SIPs

that were due in June 2007. The VISTAS re gional PM components and visibility and ASIP urban ozone model performance evaluations are directly relevant to the ASIP PM 2.5 evaluation as they share many of the same precursor and pr oduct species and atmospheric processes that produce elevated PM 2.5 levels in the urban and NAAs. Ap pendix B provides details on the ASIP

urban and NAAs PM 2.5 mass and PM 2.5 component species model performance evaluation that

are summarized for each ASIP state in Section 3.5 below. Appendix C provides a qualitative evaluation of the CMAQ base case for PM 2.5 components and total mass that is summarized in Section 3.6 However, before summarizing the urban and NAAs PM 2.5 model performance results, it is useful and informative to summarize the results of the VISTAS regional model performance evaluation (Morris et al., 2007) that is provided in Section 3.4.

3.2 CMAQ 2002 BASE G2 MODEL EVALUATION METHODOLOGY

VISTAS and ASIP share the same 2002 Base G2 Actual base case modeling results from the Community Multiscale Air Quality (CMAQ) modeling system Version 4.51 (Byun and Ching, 1999) with SOAmods enhancemen t (Morris et al., 2006). The VISTAS regional model performance evaluation focused on monitoring sites within the VISTAS states and VISTAS 12 km grid (Morris et al., 2007). The VISTAS a nd ASIP CMAQ evaluation of the 2002 Base G2 Actual base case is the product of numerous evaluations of predecessor CMAQ base case simulations using interim versions of the m odeling inputs and CMAQ model (e.g., Morris et al., 2004b,c; 2005; 2006). With the exception of Organi c Carbon Mass (OCM), the basic features of the model performance evaluation have essen tially remained unchanged from the initial

evaluations in Phase I and II of VISTAS (e.g., Morris et al., 2004a,b,c) to the current 2002 Base G2 actual base case model performance discusse d below. During the course of the VISTAS Phase II and ASIP modeling, VISTAS updated the secondary organic aerosol (SOA) module in CMAQ to include important SOA processes missing in the standard CMAQ model that significantly improved the OCM performance in the summer (Morris et al., 2006). The processes missing in the current standard versions of CMAQ (e.g., Versions 4.5 and 4.6) that were included as part of the SOAmods e nhancement are describe d in Section 1.3.3.3.1 and included SOA from sesquiterpenes and isoprene and the po lymerization of SOA so that it is no longer volatile. EPA intends to address this i ssue in the fall 2008 release of CMAQ, which is too late for VISTAS December 2007 regi onal haze or the ASIP April 2008 PM 2.5 SIPs. Thus, both VISTAS and ASIP have adopted CMAQ versi on 4.51 with SOAmods update s (Morris et al., 2006) as the core model for the modeling analysis.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-2 In the VISTAS modeling performance evalua tion, the CMAQ results were compared with

observational data from the I nteragency M onitoring of PRO tected V isual E nvironments (IMPROVE), S peciated T rends N etwork (STN), C lean A ir S tatus T rends Net work (CASTNet),

F ederal R eference M ethod (FRM) PM 2.5 mass, N ational A cid D eposition P rogram (NADP) and

S outh E ast A erosol R esearch and CH aracterization (SEARCH) study monitoring networks. The VISTAS evaluation focuses primarily on the oper ational model evaluati on of the air quality model’s performance with respect to individu al components of fine particulate matter (PM 2.5 ), as good model performance of the PM component speci es will dictate good model performance of total or reconstituted fine particulate matter from which visibility is based. The FRM network only collects total PM 2.5 mass and so is not as relevant for judging how well the model is predicting the components of light extinction so was not stressed in the VISTAS evaluation, however it is an important component of the ASIP PM 2.5 evaluation. VISTAS also performed a diagnostic evaluation analyzing th e ability of the mode l to reproduce gaseous PM precursor (e.g., SO2 and NOx) and product (e.g., HNO3) species.

The ASIP model performance ev aluation focused on ozone and PM 2.5 performance in urban and nonattainment areas (NAAs). The ASIP ozone model performance evaluation can be found at:

http://pah.cert.ucr.edu/vistas/vis tas2/evaluation_results/2002ga2a.mpe/

The ASIP PM 2.5 evaluation is summarized in Sections 3.5 and 3.6 below with more details provided in Appendices B and C. In addition, the full display of ASIP and VISTAS model evaluation products is availabl e on the project website at:

http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml

3.2.1 Evaluation Approach

EPA’s integrated ozone, PM 2.5 and regional haze modeling guida nce calls for a comprehensive, multi-layered approach to model performance testi ng, consisting of the four major components: operational, diagnostic, mechanis tic (or scientific) a nd probabilistic (EPA, 2007a). The CMAQ model performance evaluation effort for PM 2.5 discussed in this Chapter focused on the first two components of the EPA’s recommende d evaluation approach, namely:

• Operational Evaluation : Tests the ability of the model to estimate PM 2.5 mass concentrations and the components of PM 2.5 , that is sulfate, nitr ate, ammonium, organic carbon matter, elemental carbon, and other inorganic PM 2.5 . This evaluation examines whether the measurements are properly repres ented by the model predictions but does not necessarily ensure that the model is getting “the right answer for the right reason”; and

• Diagnostic Evaluation: Tests the ability of the model to predict visibility and extinction, PM chemical composition including PM precursors (e.g., SOx, NOx, and NH 3 ) and associated oxidants (e.g., ozone and nitric acid ); PM size distribution; temporal variation; spatial variation; mass fluxes; and components of light extinction (i .e., scattering and absorption).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-3 The diagnostic evaluation also includes the perf ormance of diagnostic tests to better understand model performance and identify potential flaws in the modeling system that can be corrected.

In this final model performa nce evaluation for the ASIP/VISTAS 2002 Actual Base G2 CMAQ 36/12 km base case simulation, the operational eval uation has been given the greatest attention since this is the primarily thrust of EPA’s mode ling guidance. However, we have also examined certain diagnostic features dealing with the model’s ability to simulate sub-regional and monthly/diurnal gas phase and ae rosol concentration distributions . In the course of the VISTAS/ASIP studies numerous diagnostic sensitiv ity tests were performed to investigate and improve model performance and test the model assumptions that are available on the VISTAS modeling website: http://pah.cert.ucr.edu/vistas/vistas2/ and reported by Morris and co-workers (2004a,b,c; 2005; 2006).

3.2.2 Particulate Matter and Component Species

PM 2.5 attainment is based on PM 2.5 mass measurements using Federal Reference Method (FRM) monitoring devices that cons ists of the following PM 2.5 components:

• Sulfate (SO4)

• Nitrate (NO3)

• Ammonium (NH4)

• Organic Carbon Matter Carbon (OCM)

• Elemental Carbon (EC) [also called Black Carbon (BC)] ]

• Other Inorganic PM 2.5 that is also referred to as Soil (also known as crustal material, fine soil, major metal oxides, or other PM 2.5 )

• Particle Bound Water (PBW)

• Sea Salt (that is mostly NaCl)

• Passive Mass (Blank Correction)

3.2.3 Ambient Air Quality Data for Model Performance Evaluation

A ground-level model evaluation database for 2002 was compiled by the modeling team using several routine and research-gra de databases. The focus of the VISTAS evaluation of the CMAQ model was on the PM components that can cause visibility impairment, whereas the ASIP model performance evaluation focus were on PM 2.5 mass and its compone nts. The primary monitoring networks available to evaluate this component of the CMAQ are: (a) Interagency Monitoring of Protected Visual Environments (IMPROVE); (b) Clean Air Status and Trends Network (CASTNET); (c) Southeastern Aerosol Research and Characterization (SEARCH); (d) EPA Federal Reference Method (FRM) PM 2.5 and PM 10 Mass Networks (EPA-FRM); (e) EPA Speciation Trends Network (STN) of PM 2.5 species; and (f) National Acid Deposition Network (NADP). These PM monitoring networks ma y also provide ozone and other gas phaseprecursors and product species, and visibility measurements at some sites. There are also gasphase criteria pollutant measures from the EPA’ s Air Quality System (AQS) that were used for

the urban ozone evaluation that is discussed elsewhere ( http://pah.cert.ucr.edu/ vistas/vistas2/evalua tion_results/2002ga2a.mpe/ ). Table 3-1 and Figure 3-1 summarizes the species collected and locati ons of the monitoring sites for the IMPROVE,January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-4 STN, CASTNet, NADP and SEARCH monitoring ne tworks use in the VISTAS and ASIP model

evaluation.

Table 3-1. Ambient monitoring data available in the VISTAS/ASIP region during 2002.

Monitoring

Network Chemical Species Measured Sampling Frequency;

Duration

IMPROVE Speciated PM 2.5 and PM 10 1 in 3 days; 24 hr

CASTNET Speciated PM 2.5 , Ozone Hourly, Weekly; 1 hr, Week

SEARCH 24-hr PM 25 (FRM Mass, OC, BC, SO 4 , NO 3 , NH 4 , Elem.); 24-hr PM coarse (SO 4 , NO 3 , NH 4 , elements); Hourly PM 2.5 (Mass, SO 4 , NO 3 , NH 4 , EC, TC); and Hourly gases (O 3 , NO, NO 2 , NO y , HNO 3 , SO 2 , CO) Daily, Hourly;

NADP WSO 4 , WNO 3 , WNH 4 Weekly

EPA-FRM Only total fine mass (PM 2.5 ) 1 in 3 days; 24 hr

EPA-STN Speciated PM 2.5 Varies; Varies
AIRS/AQS CO, NO, NO 2 , NO

x , O 3 Hourly; Hourly



### 250 500 750 1000 1250 1500 1750 2000 -1500 -1250 -1000

-750 -500 -250 0 250 500

IMPROVE

CASTNet

SEARCH

STN

NADP

AQS

Figure 3-1. IMPROVE, STN, CASTNet, NADP and SEARCH monitoring sites located in the VISTAS/ASIP region and the VISTAS/ASIP 12 km grid.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-5 3.3 Model Performance Statistics and Goals

To quantify model performance, several statistical measures were calculated and evaluated for all the IMPROVE, STN, CASTNet, SEARCH , FRM and NADP mon itors within the VISTAS/ASIP region or VISTAS/ASIP 12 km do main, individually fo r each monitor and individually for each VISTAS/ASIP state. The st atistical measures selected were based on the

recommendations outlined in section 18.4 of th e USEPA’s Guidance On The Use Of Models And Other Analyses for Demonstrating Attainme nt of Air Quality Goals for Ozone, PM2.5, and Regional Haze (EPA, 2007a).

In 2004, VISTAS established model performance goals and criteria for components of fine particle mass based on previous model performance for ozone and fi ne particles (e.g., Morris et al., 2004a,b,c). EPA modeling guidance for fine pa rticulate matter at the time noted that PM models might not be able to achieve the same level of performance as ozone models. The VISTAS and ASIP model performance evaluation considered several st atistical performance measures and displays. VISTAS/ASIP review ed numerous model performance evaluation metrics to evaluate their descriptive capabilit ies for summarizing the salient features of the model performance evaluation. Although numerous model performance statistics measures are routinely calculated, VISTAS/ASI P have found that the fractional bias and fractional gross error provide the best descriptive power over a wide range of concentr ations. The fractional bias and error are expressed as a percentage and are no rmalized by the average of the predicted and observed values. Consequently, they are bounded statistics, with the fr actional bias bounded by -

200% to +200%b and the fraction error bounded by 0 to 200%. Table 3-2a summarized the

formulas for the fractional bias and error statis tics. The VISTAS/ASIP model performance goals and criteria are gi ven in Table 3-3.

Table 3-2. Definitions of the fractional bias and fractional error statistical model performance

metrics.

Statistical Measure Shorthand

Notation Mathematical Expression Notes

Mean Fractional Gross Error (Fractional Error) MFE 

## ∑

= + − N

i i i i i O P O P

N 1 2 Reported as %

Mean Fractionalized Bias (Fractional Bias) MFB 

## ∑

= ⎟ ⎟ ⎠ ⎞ ⎜ ⎜ ⎝ ⎛

+ − N

i i i i i O P O P

N 1 2 Reported as %

Table 3-3. VISTAS/ASIP model performance goals and criteria for components of fine particle

mass. Fractional Bias Fractional Error Comment

<± 15% < 35% Goal for PM model performance based on ozone model performance, considered excellent performance

<± 30% < 50% Goal for PM model performance, considered good performance

<± 60% < 75% Criteria for PM model performance, considered average performance. Exceeding this level of performance indicates fundamental concerns with the modeling system and triggers diagnostic evaluation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-6 3.4 Regional Model Performance Evaluation

Details on the regional model performance eval uation for PM across the VISTAS/ASIP region km grid are provided in Chapter 3 and Appendi x B of the VISTAS Technical Support Document (TSD; Morris et al., 2007) that can be found at:

http://pah.cert.ucr.edu/vistas/vistas2/docs.shtml

Below we summarizes the key findings of th e of the VISTAS regional model performance

evaluations of PM 2.5 component species across the VISTAS region using monthly fractional bias and gross error performance statistics. More de tails, including scatter plots, quantile-quantile plots, Bugle Plots, etc. can be found in the VISTAS TSD referenced above.

3.4.1 Sulfate (SO4) Model Performance

Figure 3-2 compares the monthl y CMAQ SO4 concentration fracti onal bias and error across the VISTAS/ASIP 12 km grid region for the IMPROVE, STN, CASTNet, SEARCH Daily and SEARCH Hourly monitoring netw orks. The underprediction bias in SO4 concen trations is clearly evident across most of the year and mon itoring networks with the best (i.e., closest to zero) SO4 bias occurring in October. However, this underprediction bias is not severe usually almost always within ±20% and always within ±25% and almost always achieves the PM model performance goals for bias/error (i.e., within ± 30%/50%, see Table 3-2). The exception to this is the comparison with the SEARCH Hourly networ k that exhibits a positive overprediction SO4

bias for several months.

The SO4 monthly fractional error values are usually under 40% and, with one exception, always under 50% thereby achieving the PM performa nce goal (Figure 3-2). The exception is comparison with the SEARCH Hour ly network with error values ranging from 50% to 70%. The SEARCH Hourly SO4 measurements are made w ith research grade monitoring equipment and likely have higher uncertaintie s and artifacts than the ot her monitoring networks.

The summer SO4 underprediction bias is partly due to overstated convectiv e precipitation in the MM5 simulations (Olerud, 2003c,d). This is so mewhat confirmed by the SO4 wet deposition model performance evaluation that exhibits near zero bias during the winter when precipitation is dominated by synoptic weather ev ents, but has a positive overpred iction bias during the summer when convective precipitation is greatest (Morris et al., 2007).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-7 Figure 3-2. Monthly SO4 fractional bias (top) and fractional error (bottom) statistical measures for CASTNet, IMPROVE, STN and SEARCH (daily and hourly) monitoring sites in the VISTAS 12 km grid.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-8 3.4.2 Nitrate (NO3) Modeling Performance

The monthly fractional bias time series plot for NO3 clearly show s a seasonal dependence of this performance measure with a severe summer underp rediction bias of -100% to -150% and bias values generally within ±50% in the winter (F igure 3-3). The time seri es of fractional error

values also exhibits a seasonal dependence with an 80% error in the winter peaking to a 160%

error in the summer with a bell-like distribution. The large summer underprediction bias values occur when NO3 concentrations are extremely low (< 1.0 μ g/m 3 ) and, in fact, when the average

observed NO3 concentration exceeds 1 μ g/m 3 the model mostly achieves the model performance goal and almost always achieves the model perfor mance criteria for bias as indicated by the Bugle Plots in the VISTAS TSD (Morris et al., 2007). The mode l performance goal and criteria are achieved less often for NO3 fractional error, however the goal is always achieved across the CASTNet and usually achieved acro ss the IMPROVE monitoring networks.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-9 Figure 3-3. Monthly NO3 fractional bias (top) and fractional error (bottom) statistical measures for CASTNet, IMPROVE, STN and SEARCH daily and hourly monitoring sites in the VISTAS region.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-10 3.4.3 Organic Carbon Mass (OCM) Performance

The monthly fractional bias an d error plots for the 12 km CM AQ base case simulation OCM performance given in Figure 3- 4 that displays fa irly good performance across the IMPROVE monitors with bias usually wi thin ±15% and errors between 40% and 50%. However, at the urban STN sites the model exhibits a large OCM unde rprediction bias of -35% to -60%. Part of the underprediction bias at the STN sites may be due to measurement uncer tainties and artifacts and part may also be due to the model over dilu ting the urban OCM emissions through the coarse 12 km grid. The underprediction of urban OC M is a common problem in PM modeling and likely also points to uncertainties in the OCM and SOA precursor emission inventories.

One source of uncertainty in the OCM measuremen ts is the fact that OCM is actually derived

from OC measurements. The amount of additiona l elements (e.g., oxygen) attached to the OC to form OCM varies with the age and level of ph otochemical processing of the organic particles with OCM/OC ratios typically ranging from 1.2 to 2.2 with lower ratios for fresh and higher ratios for aged processed OCM. For the OC M model performance evaluation we used a 1.4

OCM/OC ratio that is consistent with the original IMPROVE equation and based on measurements from Los Angeles. The ne w IMPROVE equation uses a higher 1.8 OCM/OC ratio reflecting the fact that OCM that reaches the mostly rural Class I areas will be aged and subject to photochemical processi ng. There really is no one right OCM/OC factor to use in all cases. However, in interpreting the model perf ormance it is important to know which OCM/OC ratio was used and recognize that selection of anot her ratio could make a 30% or so difference in

the OCM measurements.

The STN OC measurements are also not blank corrected, which is believed to result in an approximate 0.5 μ g/m 3 positive artifact in the STN OC ob servations (which would be 0.5-0.9 μg/m 3 positive artifact in OCM depending on which OCM/ OC factor is used). The subtraction

of 0.5-0.9 μ g/m 3 from the measured OCM value woul d greatly improve the CMAQ OCM model performance across the STN network and br ing the CMAQ OCM bias closer to zero.

In the preliminary evaluation of the CMAQ model using early VISTAS Phas e I and II databases, the OCM performance was a great cause of concern due to a large summer OCM underprediction bias and the importance of OCM to the visibility extinction and PM budgets (Morris et al.., 2004b,c). This resulted in VI STAS conducting a focused research study on the

reasons for the summer OCM unde rprediction, the identification of processes missing in the CMAQ model and the development of the SOAm ods enhancement to CMAQ (see Section 1.3.3.3.1) that greatly improved the CMAQ OCM mode l performance (Morris et al., 2006). The CMAQ Version 4.51 with the SOAmods enhancements was used in the ASIP modeling as well.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-11 Figure 3-4. Monthly OCM fractional bias (top) and fractional gross error (bottom) statistical measures for IMPROVE, STN and SEARCH monitoring sites in the VISTAS region.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-12 3.4.4 Elemental Carbon (EC) Performance

The monthly fractional bias and error performa nce statistics for EC across the VISTAS 12 km domain are shown in Figure 3-5. Bias and error are lower in th e first and last quarters of the year. During the summer and adjacent months EC performance across the IMPROVE networks exhibits a large underprediction bias (-40% to - 80%) that peaks in June . The EC performance

across the SEARCH network is somewhat simila r to IMPROVE only the underprediction bias in the summer is not as great. Model performance compared to the STN network has fairly low EC fractional bias during quarters 1, 3 and 4 (achieving the most st ringent < ±15% ozone goal), but has larger underprediction bias during quarter 2 (-20% to -40%). The comparison to the STN network very low EC bias during most of the ye ar suggests that the anthropogenic EC emissions inventory may be adequately ch aracterized in urban areas.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-13 Figure 3-5. Monthly EC fractional bias (top) and fractional gross error (bottom) statistical measures for IMPROVE, STN and SEARCH daily and hourly monitoring sites in the VISTAS region.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-14 3.4.5 Other PM 2.5 or Soil Performance

The seasonal dependence of the Soil model performance across the IMPROVE and STN monitors in the VISTAS region is clearly evident in the monthly fractional bias time series plot in Figure 3-6 that show a large (> 100%) overprediction bias in the winter and near zero (IMPROVE) or much lower (SEARCH) bias in the summer. The modeled Soil values tend to always be between 0 and 2 μ g/m 3 year round, however the observed values are much lower in the winter (0 to 0.5 μ g/m 3 ) and comparable to the modeled values in the summer. These results suggest that the poorer winter Soil model pe rformance is likely due to incorrect emission temporal adjustment factors. For example, the effect of wetted surfaces that suppresses fugitive dust emissions may not be properly characterized in the seasonal adjustments to the emissions inventory.

The Soil model performance is confounded by the incommensurability between the modeled and measured Soil species. The IMPROVE observed Soil is built up of measured elements. The modeled “Soil”, on the other hand, is fine pa rticulate matter emissions that have not been explicitly identified and speciated as SO4, NO3, EC and OC in the PM speciation profiles used in the SMOKE emissions modeling. The em issions PM speciation profiles may have unidentified PM that is lumped in the other PM category that are not the same as the IMPROVE Soil. The measured Soil will typically be associ ated with fugitive dust emissions, so the separate tracking of fugitive dust may be one approach to separate the true Soil component from the other

PM 2.5 component. VISTAS investigated this issue with an inert (no chemistry) CMAQ sensitivity run that separate ly tracked just fugitive dust emissions and found improved Soil model performance with lower bias when just other PM 2.5 concentrations due to fugitive dust emissions were mapped to the Soil species.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-15 Figure 3-6. Monthly Soil fractional bias (top) and fractional gross error (bottom) statistical measures for IMPROVE and SEARCH Daily mo nitoring sites in the VISTAS region.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-16 3.4.6 Fine Particulate (PM 2.5 ) Performance

Figure 3-7 displays the mont hly time series of fracti onal bias and error for PM 2.5 mass concentrations for the IMPROVE, STN, SEARCH Daily, SEARCH Hourly and FRM networks in the VISTAS/ASIP region. The model genera lly is exhibiting good performance for total

PM 2.5 mass concentrations. For the first three a nd last three months of the year the model performs well for PM 2.5 with low fractional bias (<±20% ) and error (<40%) except for the SEARCH Hourly network. During April through July, the underpre diction bias becomes larger and is in the -20% to -45% range and the errors ar e also larger. This is driven in part by the SO4 underprediction bias discussed previously as SO4 is a major component of PM 2.5 in the summer

and the southeastern U.S.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-17 Figure 3-7. Monthly total PM 2.5 mass fractional bias (top) and fractional gross error (bottom) statistical measures in the VISTAS region.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-18 3.5 PM 2.5 MODEL PERFORMANCE BY STATE

In this section we evaluate the CMAQ base case for PM 2.5 at the mainly urban STN and FRM monitoring sites. Appendix B presents a detailed model performance evaluation by PM 2.5 species component and for total PM 2.5 mass by each state in the ASIP/VISTAS region. An example of the types of displays of model pe rformance from Appendix B are given below for Alabama and SO4 performance and Soccer Plots for PM 2.5 mass are also presented. For the remainder of the state-by-state scatter plot s, time series plots and Bugle Plots of PM 2.5 mass and components model performance the r eader is referred to Appendix B.

3.5.1 Alabama

Figure 3-8 is a repeat of Figure B-2a in Appe ndix B that contains a scatter plot and model performance statistics for SO4 at STN sites located in Alabama and the four seasonally representative months of Janua ry, April, July and October 2002. Similar scatter plots and performance statistics for the other PM 2.5 component species and total PM 2.5 mass are shown in Figure B-3b-g in Appendix B. Performance for SO4 is quite good with the points on the scatter plots clustered tightly around the 1:1 line of pe rfect agreement (Figures 3-8 and B-2a) and fractional bias and error performance metrics ac hieving the most stringe nt <±15% and < 35%, respectively, performance goals for ozone (Table 3-2). Performance for NO3 shown in Figure B-2b exhibits an underprediction tendency with fr actional bias values of approximately -20% in January, -110% in April, -175% in July and -10% in Oct ober. The large summer NO3 underprediction bias occurs when both the mode l and measured NO3 values are extremely low

with the modeled values near zero a nd the STN NO3 observed values are < 1 μ g/m 3 . Note that

under these summer conditions the NO3 will almost completely volatize off the FRM filter so the FRM would measure insignificant PM 2.5 mass due to NO3. Thus, these large summer NO3 underpredictions are not a concern as NO3 is not an important component of PM 2.5 mass. Performance for NH4 is also fairly good achievi ng the <±30%/50% fractional bias/error goal for PM species (Figure B-2c), which is not surpri sing given its strong link to SO4. With the exception of a few outliers, performance for orga nic carbon mass (OCM or OC) is fairly good in the winter, but underestimated by -20% to -30% in the summer and fall (Figure B-2d). The reasons for the OCM underpredic tion includes the fact that the STN OC measurements are not blank corrected and have high uncertainties (see SANDWICH discussion in Chapter 4) and there are also large uncertainties in the OC emissions and the form of OC emi ssions as particles or semi-volatile organic gases. EC model performa nce at STN sites in Alab ama is also fairly good mostly achieving the PM model performance goal, albeit with an under estimation tendency in the summer (Figure B-2f).

The CMAQ performance for PM 2.5 mass at 20 FRM monitoring site s in Alabama is shown in Figure B-2g. Performance for PM 2.5 mass is generally good achievi ng the most stringent ozone performance goal (<±bias/error) in January and October and th e PM goal all year round.

Time series of predicted (CMAQ 12 km results) and observed 24-hour PM 2.5 components and model performance statistics at the key North Birmingham STN and SEARCH site are shown in Figures B-3 and B-4, with the SO4 performance tim e series at these co-located sites reproduced as Figure 3-9. As discussed in Chapter 4, this is the site with the highest projected 2009 PM 2.5 Design Value in the ASIP region. SO4 performa nce is quite good with annual fractional bias

and errors of 6% and 29% achievi ng the most stringent ozone pe rformance goal at the STN siteJanuary 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-19 and fractional bias and error of 18% and 34% that almost achi eves the most stringent ozone performance goal at the SEARCH site. NO3 ha s a large underprediction bias at the North Birmingham site that is driven by near zero mo deled NO3 concentrations in the summer when the observed values are < 1 ug/m 3 (Figures B-3a and B-4a, bottom). However, the NO3 performance in the winter when observed values are above 1 ug/m 3 is much better. The EC and OCM performance at the North Birmingham STN site is characterized by an underprediction with fractional bias of approximately -30% an d -50%, respectively (Figures B-3b and B-4b). This underprediction is due in part to uncertain ties in the carbon measurements as well as the contributions of nearby industria l sources whose impacts are diluted across the 12 km CMAQ grid used. Performance for NH4 and PM 2.5 mass at the North Bi rmingham STN and SEARCH sites are also quite good with fractional bi as/error values of, re spectively, -5%/33% and

11%/29% at the STN site that achieves the more stringent ozone performa nce goal and values of, respectively, -24%/42% and 17% /31% that achieves the PM performance goal at the SEARCH site. The good performance of NH4 and PM 2.5 mass is not surprising give n that they are closely related to SO4 that exhibits good model performance.

Figures 3-10 reproduces Figure B-5 from Appendi x B and displays a Soc cer Plot of monthly fractional bias versus gross error for PM 2.5 mass performance across 20 FRM sites in Alabama and compares them with the three levels of bi as/error performance goals in Table 3-3. Total

PM 2.5 mass model performance across the FRM sites in Alabama is best in the winter months when the most stringent ozone < ±15%/35% bias/e rror goal is achieved. In the summer months,

the model tends to underestimate the FRM observed PM 2.5 mass by -15% to -25%, but still achieves the <±30%/50% PM model performance goal.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-20 Figure 3-8. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation (duplicate of Figure B-2a in Appendix B)January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-21 Figure 3-9. Time series of predicted and observed 24-hour sulfate (SO4) concentrations during 2002 at the STN Jefferson County, Alabama Site No. 01-073-0023 (top) and SEARCH BHM (bottom) co-located North Birmingham sites for the CMAQ 12 km 2002 Base G2 Actual base

case simulation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-22 Figure 3-10. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Alabama for the CMAQ 12 km 2002 Base G2 Actual base case simulation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-23 3.5.2 Florida

Figure B-7 summarizes the CMAQ PM 2.5 species model performance across the three STN sites in Florida and January, April, July and October 2002. SO4 is unde restimated at the Florida STN sites (Figure B-7a) with fractional bias values th at are closer to zero in the winter than summer (e.g., -7% January versus -34% July for the 12 km CMAQ results) and closer to zero using the 12 km than 36 km CMAQ modeling re sults (e.g., in January bias is -7% using 12 km and -22% using 36 km modeling results). Even with this underestimation bias, SO4 performance at the Florida STN CMAQ sites is fairly good. NO3 pe rformance, on the other hand, exhibits a large underprediction bias with fractiona l bias values that range from -70% in January to -170% in July (Figure B-7b). NH4 performance at the Fl orida STN sites is fairly good always achieving the <±30%/50% PM performance goal and some times even achieving the more stringent <±15%/35% ozone performance goal (Figure B-7c). It is interesting that in January and October when both SO4 and NO3 exhibit an underpredicti on bias, NH4 exhibits a slight overprediction bias. Given the linked relationship between thes e species this is somewhat surprising and may

be due to artifacts in the meas urements, or the modeled SO4 being more fully neutralized by

NH4 than in the measurements. The OCM and EC are characterized by underprediction tendency with the OCM fractional bias ranging from -55% to -95% and the CMAQ 36 km and 12 km producing nearly identical performance metrics (Figure B-7d). The EC performance is

better with fractional bias valu es ranging from -4% to -52% and the 12 km modeling results producing significantly better model performance metrics than the 36 km modeling results (Figure B-7e).

The PM 2.5 mass scatter plots and performance statisti cs in Florida are summarized across 3 STN sites and 28 FRM sites in Figur es B-7f and B-7g, respectively. The Soccer Plot of monthly

PM 2.5 bias and error performance for FRM sites in Florida is shown in Figure B-9, which is repeated as Figure 3-11 below. For the wi nter months of Nov, Dec, Jan and Feb, the PM 2.5 performance statistics are quite good with bi as/error values meeting the <±15%/35% ozone performance goal. The spring and fall months either achieve the <± 30%/50% or <±50%/125% performance goal and criteria. The May-Jul summe r months are at or exceed the -50% bias performance criteria levels (note that in the So ccer Plots if the symbol would be plotted outside of the range of the plot then it is plotted on the axis, such as the May and July -50% values in Figures 3-11 and B-9). Thus, the PM 2.5 performance in Florida is significantly worse than the

other ASIP states and is a cause for concern. Fortunately, Florida is one of the ASIP states without any violating PM 2.5 monitors (See Chapter 4).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-24 Figure 3-11. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Florida for the CMAQ 12 km 2002 Base G2 Actual base case simulation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-25 3.5.3 Georgia

There appears to be some valid data capture i ssues in the 2002 STN database for Georgia with only one valid site in January, 6 in April and 8 in July and October. This is due to some of the STN sites starting up in 2002. Th e CMAQ SO4 performance in Geor gia is generally pretty good with bias/error frequently achieving the <±15%/35% ozone perfor mance goal and always achieving the PM goal (Figure B-11a). The STN sites also have the summer nitrate underprediction performance issue as seen in th e other states (Figure B-11b). The NH4 model performance falls between the SO4 and NO3 perf ormance and, with the exception of January, which is based on one site so the model performan ce statistics are not very robust, it achieves the <±30%/50% PM performance goal (Figure B-11c). The carbon performance in Georgia appears to be slightly better than the other states, although s till with a general underestimation tendency. With the exception of July, when the CMAQ 36 km OCM performance exhi bits fractional bias and error values of -34% and 42% thereby ac hieving the <<±30%/50% PM performance goal,

the other three months achieve the more stringent <±15%/35% ozone performance goal (Figure B-11d). EC performance is variable with genera lly low bias but lots of scatter producing higher gross errors (Figure B-11e).

The performance of total PM 2.5 mass across the STN (Figure B-11f) and FRM (Figure B-11g) monitoring sites in Georgia is generally fairly good. July exhibits an u nderprediction tendency with fractional bias values of approximately -4 0% (STN) and -25% (FRM ) that is likely caused by the underestimation of SO4 and especially OCM. January is characterized by an overestimation tendency. But in general the PM 2.5 performance statistics mostly achieve the more stringent ozone performance goal.

An example time series of predicted and observe d PM concentrations and annual statistics for two sites in the Atlanta Georgia (a DeKalb County STN and the SE ARCH Jefferson Street sites) are shown in Figures B-12 and B-13. The model tr acks the observed SO4 c oncentrations at these sites extremely well produci ng low fractional bias (-5% and +9%) and gross error (27% and 30%). NO3 performance exhibits the underprediction tendency seen in the other states, but is

not as severe with a fractional bias of - 16% and ~80% error. The CMAQ 12 km results reproduces the OCM concentrations at the DeKalb County site surprisingly well with low bias (- 14%) and error (31%), with a slight degradation at the SEARCH JST site (-35% bias and 46% error). EC is overestimated (bias of 28%) at th e DeKalb site but has zer o error at the SEARCH site. Finally, the performance for PM 2.5 mass at the two Atlanta sites is extremely good with very low bias (2% and -3%) and error (25% and 24%).

Figure 3-12 below is reproduced from Figure B-14 in Appendix B and displays a Soccer Plot of monthly fractional bias and error across the 26 FRM sites in Geor gia. All months achieve the PM performance goal and 8 of the 12 months even achieve the more stringent ozone performance goal. The months that the PM 2.5 mass performance doesn’t achieve the ozone performance goal are due to an overestimation bi as in January and underestimation bias in the summer months of May-July that is drive primarily by an unde restimation of SO4 and OCM.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-26 Figure 3-12. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Georgia for the CMAQ 12 km 2002 Base G2 Actual base case simulation.

3.5.4 Kentucky

The monthly scatter plots and performance statis tics for PM species and sites in Kentucky are shown in Figure B-16 in Appendix B. The perf ormance for SO4 in January is mixed, with a group of points clustered around the 1:1 line and another further to the right that has been determined to be variations of SO4 performan ce in Kentucky on different STN sampling days in January. As seen in the other states, NO3 performance in Kentucky is characterized by underprediction bias that is larges t in the summer (Figure B-16b) w ith fractional bias is -30% to - 40% in January, -65% in April, -150% to -160% in July and near zero bi as in October due to NO3 underpredictions on some days being compen sated by overprediction on others. Like SO4, NH4 performance is fairly good albeit with an underp rediction bias that is largest in the summer (Figure B-16c). As seen for the other sites, OC M is underpredicted with the bias ranging from - 50% in January to -70% to -80% in the other three months (Figure B-16d). The OCM errors are the opposite sign but the sa me magnitude as the bias sugges ting that the OCM underprediction bias is almost systematic. The CMAQ 36 km a nd 12 km modeling results exhibits different EC performance in Kentucky. The CMAQ 12 km results have bias that is much closer to zero than the 36 km results (Figure B-16e). This is likel y due to the 36 km grid cells over dispersing theJanuary 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-27 urban EC emissions. Both the CMAQ 36 km and 12 km results exhibit a lot of variability producing a lot of scatter and high error metrics.

The performance of total PM 2.5 mass concentrations in Kent ucky is evaluated across 10 STN (Figure B-16f) and 23 FRM sites (Figure B-16g). Given the underprediction of most of the PM component species it is not surprising that total PM 2.5 mass is underpredicted across the STN sites by -16%, -42% to -55%, -33% and -6% to -13% for January, April, July and October, respectively. Across the FRM sites in Kentucky, however, the CMAQ model for PM 2.5 mass bias that is much closer to zero with values of -8%, -11% to -14%, -32% to -35% and +18% to -19%

for the same four months. The FRM PM 2.5 mass performance always achieves the PM performance goal and sometimes even achieve s the ozone model performance goal.

An example PM model performance for STN site s in Lexington (Fayette County) and Kenton County (Cincinnati NAA) are show n in Figures B-17 and B-18. Th e model performance at these

two sites is similar. At the Le xington site the model reproduces the temporal vari ations in SO4 quite well (Figure B-17a, top) at this site resu lting in low bias (-10%) a nd error (33%). Although the model shows some skill in reproducing th e observed higher NO3 values in the winter, especially in the first part of the year, the near zero modeled NO3 during the summer results in a

fractional bias value of -62% with 87% error, so does not ach ieve the PM performance goal (Figure B-17a, bottom). OCM is underpredicted with fractional bias values of -52% and an error of 58% just at the PM performance goal (Fi gure B-17b, top). EC exhibits much better performance at the Lexington, Kentucky site with a bias near zero (-4 %) and error of 36% so almost achieves the ozone performance goa l. The fractional bias for NH4 and PM 2.5 at Lexington are both approximately -10% w ith NH4 having an error of 36% and PM 2.5 an error of 28% thereby achieving the ozone performance goal.

Figure 3-13 below reproduces Figure B-19 fr om Appendix B that summarizes the Kentucky

PM 2.5 mass model performance using a Soccer Plot of monthly bias and error. With the exception of July, whose fractional bi as falls slightly below -50%, PM 2.5 performance in the other 11 months achieves the PM bias/error perf ormance goal of <-30%/75%. In fact 5 months, which occur mainly in the winter, even achieve the more stringent ozone model performance goal.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-28 Figure 3-13. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Kentucky for the CMAQ 12 km 2002 Base G2 Actual base case simulation

.

3.5.5 Mississippi

Performance of PM species across sites in Mi ssissippi is shown in Figure B-21. The CMAQ performance in Mississippi is generally worse th an the other ASIP states (with the exception of Florida). SO4 is underpredicted with fractional bi as values ranging from - 15% to -46% with the worse SO4 performance seen in the January/July winter/summer months than the April and October transitional months. NO3 performance metrics are extremely poor with fractional bias values of -80% to -90% in January and Apri l and -163% in July. In October some NO3 overpredictions begin to compensate for the fre quent near zero model predictions so that the underprediction bias is lower (-40%), but the error is still very high (~100%). OCM is underpredicted at sites in Mississippi with frac tional bias values of approximately -50% in January, April and July improving to -24% in October (Figure B- 21d). Performance for EC is better, albeit still with an underprediction bias, and again as seen for other states, the CMAQ 12 km modeling results produces much lower EC bi as than the 36 km mode ling results (Figure B21e). PM 2.5 performance in Mississippi across the STN and FRM networks are shown in Figures B-21f and B-21g, respectively. Across the STN network, PM 2.5 is underestimated by approximately -40% rising to -65% in July. Across the FRM network, the PM 2.5 underprediction bias is lower, at -14% in January and -25% in Ap ril and rising to approximate ly -50% in July. In

October the model is better able to replicate the observed PM 2.5 mass with CMAQ 12 km biasJanuary 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-29 and error values across the FRM network of 2% and 29%, re spectively, achieving the ozone model performance goals.

The Soccer Plot (Figures 3-14 and B-23) for PM 2.5 mass performance confirm that PM 2.5 performance in Mississippi is worse than seen for the other ASIP states, ex cept Florida. Only two months achieve the ozone model performa nce goal and only 8 months achieve the PM model performance goal. The model underpredic tion tendency during the four month period of March through July 2002 produ ces fractional bias for PM 2.5 that ranges from -50% to -100% thereby not achieving the PM model performance goa l. However, unlike Florida, the Mississippi monthly bias/error for PM 2.5 always achieves the <-50%/75% PM performance criteria.

Figure 3-14. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Mississippi for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-30 3.5.6 North Carolina

SO4 performance across 9 STN monitoring sites in North Carolina and four months from 2002 is shown in Figure B-25a. Although SO4 performance achieves the PM performance goal, it varies by month with January and July having fractional bias of -25% to -30%, April a fractional bias of approximately -20% and October having near zero fractional bias. Although NO3 is still greatly underestimated in North Carolina during the summ er (July bias of -105% to -142%), during January and October NO3 exhibits an overest imation bias with values of 84% and 35%, respectively (Figure B-25b). This is likely du e to overstated ammonia emissions during the winter and adjacent periods which is somewhat verified by the overstated ammonium during January and October that have fractional bias valu es in the +40% to +55% range (Figure B-25c). OCM is characterized by an underprediction bias ranging from -30% to -50%, although performance in the April and October transitional months achieves the PM performance goal (Figure B-25d). EC, on the other hand, achie ves the PM performance goal in January and

October but has lots of scatter in the other two mont hs exceeding the PM performance goal for bias in April (-34% to -41%) and for error in July (62%-65%). PM 2.5 mass performance across

the 9 STN and 36 FRM sites in North Carolina exhi bits monthly variations (Figures B-25f and B-25g). FRM PM 2.5 is overpredicted in January (15% to 18%) and October (25% to 30%), but underpredicted in April (-9% to -16%) and July (-37% to -39%). The STN PM 2.5 performance is similar except because the STN measurements tend to be higher so the underprediction bias is greater and the overpredi ction bias is less.

The Soccer Plot summary of FRM total PM 2.5 mass monthly model performance in North Carolina are shown in Figure 3- 15 (reproduced from Figure B- 28 in Appendix B). With the exception of the large summer underprediction bias in June and July, the other 10 months PM 2.5 performance achieves the <±30%/50% PM perfor mance goal with four of the months even achieving the more stringent ozone performance w ith four additional months right at the ozone goal.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-31 Figure 3-15. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in North Carolina for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.

3.5.7 South Carolina

The CMAQ performance for SO4 across the 4 ST N sites in South Carolina generally has an underprediction bias but is reasonably good al ways achieving the PM performance goal and sometimes even achieving the ozone performan ce goal (Figure B-30a). NO3 performance is poor with an underprediction bias generally from -40% to -180% that is largest in the summer when NO3 is lowest and not an important part of the PM 2.5 (Figure B-30b). NH4 performance is variable frequently achieving the PM performance goal with the worst performance in the summer with a large underpredictio n tendency. OCM is underestimated with bias of -15% to - 62%. EC performance also has an underestima tion tendency, although not is bad as seen for OCM. The CMAQ 12 km results also exhibits mu ch better EC performa nce than the CMAQ 36 km modeling results with th e 12 km EC model performan ce always achieving the PM performance goal and sometimes even achievi ng the ozone performance goal, whereas the CMAQ 36 km EC results never achieves the PM performance goal for the four months studied (Figure B-30e). PM 2.5 model performance across STN and FR M networks in South Carolina are shown in Figures B-30f and B-30g, respectively. PM 2.5 mass is usually underpredicted acrossJanuary 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-32 both networks, with the fractional bias across the STN network is worse and approximately 20 percentage points larger underprediction than the FRM network. In fact, except for July with a - 40% bias, the bias/error for PM 2.5 across the FRM network ach ieves the <±15%/35% ozone model performance goal.

The summary fractional bias and gross error Soccer Plot for total PM 2.5 mass across the FRM network in South Carolina is shown in Figures 3-16 and B-32. During the fall and winter months of August, September, November, December, Janua ry and February, the monthly bias and error

statistics for PM 2.5 achieve the very stringent ozone mo del performance goal. The three additional months of March, April and October ach ieve the PM performance goal. Whereas the summer months of May, June and July fail to achieve the PM performance goal (but does achieve the PM performance crit eria) due to large unde rprediction bias th at is caused by the underprediction of each of the PM component species.

Figure 3-16. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in South Carolina for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-33 3.5.8 Tennessee

The CMAQ does a good job predicting the observed SO4 concentrations in Tennessee with bias

and error values that either achieve the most st ringent ozone performance goal or are right at the edge of the ozone goal (Figure B- 34a). The CMAQ 12 km results are slightly better than the 36 km predictions and, unlike the other states wher e July had the largest underprediction bias, in Tennessee the SO4 performance in July is quite good with the 12 km results producing a 4% bias

and 32% error. The model exhibits little skill in its NO3 predictions (Figure B-34b) with the usual large underprediction bias in the summer (e .g., -170% in July). Although the bias is low in January and October (<±10%), the error is large (60% to 80%). NH4 performance is generally reasonably good with low bias and error in January and April, but an underprediction bias in July (-20% to -30%) and an overprediction bias in October (33% to 36%).

As seen in the other states, OCM exhibits a larg e underprediction bias that generally ranges from -40% in the winter fall and spring to -60% in summer (Figure B-34d). EC performance is quite good in January and October, but has an underpre diction tendency in April and July (Figure B34f). Tennessee PM 2.5 mass performance across the STN and FRM networks are shown in Figures B-34f and B-34g. Across the STN networks performance for PM 2.5 achieves the ozone goal in January and October, but ex hibits an underprediction bias in April and July such that in July it exceeds the PM performance goal. Ac ross the FRM network the model performs better

for PM 2.5 achieving or nearly achieving the ozone performance goal in January, April and October, but with an underprediction bias in July (-26% to -35%) that is at the PM performance goal.

The monthly fractional bias a nd error Soccer Plot for FRM PM 2.5 performance in Tennessee shown in Figure 3-17 and indicates that all months of the year achieve the PM model performance goal and that 5 of the months ev en achieve or are right at the ozone model performance goal. The months that do not ac hieve the ozone performa nce goal are due to a summer underprediction bias for May-July and a fall/winter overprediction bias for October and November and a too high error for December.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-34 Figure 3-17. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Tennessee for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.

3.5.9 Virginia

SO4 performance is fairly good in Virginia with low bias and error in July and October that achieves the <±15%/35% ozone bias/error perf ormance goal (Figure B-39a). Although January and April exhibit a -20% to -30% underpredict ion bias for SO4, it still achieves the PM performance goal. As seen for the other states , NO3 performance is po or and characterized by an underprediction bias that is greatest in the summer (-170%) when CM AQ estimates near zero NO3 predictions. NH4 performance achieves th e ozone performance goal in April and PM performance goal in January and October, but the -48% to -55% bias in July exceeds the PM performance goal (Figure B-39c).

OCM performance is characterized by an underpredic tion bias of from -30% to -60% with error of opposite sign and similar magnitude (Figure B-39d) . The fractional bias for EC is about 20- 30 percentage points higher using CMAQ with a 12 km grid than 36 km grid resulting in the 12 km results always achieving the PM performance goal but the 36 km results underprediction bias in April and October is in the -40% to -50% range so does not achieve the PM performance goal.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-35 The PM 2.5 mass performance across the FRM sites is fairly good achieving the ozone performance goal in January, April and Octobe r and the PM performance goal in all three months (Figure B-39g). Across the STN sites, the PM perfor mance goal is achieved in January, April and October but not in July due to a -36% to -44% underprediction bias.

Figure 3-18 below summarize the PM 2.5 model performance across 19 FRM sites in Virginia using a Soccer Plot. Monthly frac tional bias and gross error for PM 2.5 mass in Virginia achieves the PM performance goal for every month in 2002, with six of the months achieving the ozone performance goal and another two ju st at the ozone goal. As seen in the other states, the three summer months of May, June and July have an u nderprediction bias that does not meet the ozone performance goal, but still m eets the PM performance goal.

Figure 3-18. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Virginia for the CMAQ 12 km 2002 Base G2 Actual base case simulation.

3.5.10 West Virginia

There were no speciated PM 2.5 STN monitoring sites within West Virginia during the 2002 modeling year, so the PM 2.5 model evaluation is limited to the 16 FRM PM 2.5 mass monitoring sites. In January, April a nd July, CMAQ underestimates PM 2.5 mass across the FRM sites in West Virginia by approximately -30%, -25% an d -40% and achieves the PM performance goal in January and April, but not July (Figure B-44). In October the model has near zero bias and 32%January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-36 error so achieves the ozone model performance goal. The Soccer Plot in Figures 3-19 below

summarizes the FRM PM 2.5 mass performance in West Virginia in each month. Nine of the months achieve the PM performance goal, with the winter months even achieving or nearly achieving the ozone model performance goal. As seen for the other states, the underprediction bias during the May-July summer months resu lts in the bias falling outside of the PM performance goal, but within the PM performance criteria.

Figure 3-19. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in West Virginia for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-37 3.6 SPATIAL AND SEASONAL ANA LYSIS OFMODEL PERFORMANCE

This section summarizes the results in Appendix C that contains a general qualitative overview of the CMAQ 2002 Base G2 Actual base case model performance across the ASIP 12 km domain, with emphasis on temporal (seasonal) and spatial patterns within the domain. The evaluation is carried out for each of the major PM 2.5 components separately (sulfate, nitrate, ammonium, organic carbon mate rial, elemental carbon, soil PM 2.5 , sea-salt PM 2.5 ), as well as for

total PM 2.5 mass. The performance evaluation summar ized in this section and presented in Appendix C present spatial maps of modeled quarterly average PM 2.5 concentrations with overlaid quarterly average observations. For each PM 2.5 species, two sets of figures are provided, one comparing model predictions to obs ervations at the STN sites, the other to observations at the IMPROVE network sites (so il and sea-salt are available at the IMPROVE network only). Modeled spatial maps w ith overlaid observations for total PM 2.5 mass are compared for the FRM network, in addition to the STN and IMPROVE networks. Four figures are provided for each set corresponding to the averag e concentrations during the four quarters of the year (Q1: Jan-Mar; Q2: Apr-Jun; Q3: Jul-Se p; Q4: Oct-Dec). In Appendix C all of the spatial maps comparisons are presen ted, with the figures for the STN PM 2.5 species components

and the FRM PM 2.5 mass also provided below.

This evaluation allows for a general understandi ng of model performance and key issues. A more detailed and quantitative performance evaluation fo r monitoring sites within each ASIP states is provided in Appendix B and summ arized in Section 3.5 above.

The observed quarterly average PM 2.5 mass and PM 2.5 species concentrations were obtained by averaging all 24-hour measurements at a site that occurred during each three month quarter. The modeled quarterly average spatial maps were obtained by averaging the daily average concentrations for each day in the three month quart er and each grid cell. No attempt was made in the averaging to match the modeled daily av erage concentrations with the 1:3 day sampling frequency used by the monitoring networks. In fact, obtaining modeled quarterly average concentrations trying to match th e measurement days is problema tic since not all sites collect valid samples on every 1:3 day samp ling day. For some STN sites this resulted in a bias in the Q1 quarterly average predicted and observed co mparisons since many of the STN sites started up in 2002 and are missing data for the first part of 2002 so only have samples near the end of Q1. In the case of species li ke SO4 that has a strong seasonal trend, basing ob served Q1 averages on samples in March and modeled values averag ed across January-Febru ary-March introduces a seemingly underprediction bias that is artificial and an artifact of the network sampling periods. We know this is occurring at some STN sites, for example see discussion for Georgia in Section

3.5.3.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-38 3.6.1 Sulfate (SO4)

Modeled sulfate concentrations show a very strong seasonal pa ttern (Figure 3-20 below and Figures C-1a and C-1b in Appendix B), with p eak concentrations occurring during summer months (Quarters 2 and 3) when photochemistry is highest. A spatial pattern is also evident, with higher concentrations in the northeast, Ohio River valley, and southeast, compared to the upper Midwest and Florida, caused ma inly by the higher and denser SO 2 emissions in those regions compared to the latter two. In Florida, the impacts of individual major sources of SO 2 are evident (such as in the Tampa and Jacksonvill e areas), as their emissions remain relatively unmixed with emissions from other regions, bei ng surrounded by ocean. However, in most of the domain, a “regional” sulfate field is observed as a result of mixing of emissions from various regions, especially for the long averagi ng time presented here (three months).

Overall, the model seems to accurately simulate sulfate levels over the do main and captures both the temporal and spatial patterns exhibited in th e observations, both from the STN (Figures C-1a and 3-20) and IMPROVE (Figure C-1b) networks . The exception is Quarter 1 for the STN network, in which the observed concentrations s eem higher than the modeled ones, and there is not much agreement in the spatial pattern between the two. This is partly due to an artifact in the way the observations are presente d here. Since some of the ST N sites were not in operation during the first few months of 2002, observed Quarte r 1 averages might in fact be driven by observations during the latter (and warmer) pa rt of Quarter 1 (e.g., March samples), and therefore are biased high relative to the modeled Q1 SO4 concentrations.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-39 Figure 3-20. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) sulfate (SO4) concentrations over the VISTAS 12 km domain (for IMPROVE comparisons see Appendix C).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-40 3.6.2 Nitrate (NO3)

Nitrate is in many ways the “mirror” image of su lfate, with peak concentrations occurring during wintertime, when the cooler temperatures are fa vorable for particulate nitrate formation, and due

to lower SO4 as more ammonia is available to bond with nitrate, rather than sulfate (Figure 3-21 and Figures C-2a and C-2b in Ap pendix C). Also for this r eason, peak particulate nitrate

concentrations occur in areas of the domain wh ere ammonia emissions are highest, such as the

Midwest.

While overall the model does capture the seasonal and spatial variability in particulate nitrate concentrations (Figures C-2a,b), it does seem to overestimate concentrations during wintertime, especially over urban ce nters, such as in part of the Northeast and ove r Atlanta and Birmingham.

Winter overestimations are also evident over much of North Carolina. While it is impossible to

infer from these data alone on the cause for these overpredictions, they are related to either one or a combination of the model nitrate partitioning between gaseous and particulate phase, and the availability of gaseous ammonia, as reflected by the emissions inventory. The latter can be the common denominator between the overpredicti ons over the urban centers and over North Carolina as ammonia emissions in these areas, though originating from different sources (mobile sources in the urban centers; agricultural a mmonia emissions in North-Carolina), might be overestimated making more ammonia available for particulate formation compared to the

observations.

In the summer the model estimates near zero (< 0.25 μ g/m 3 ) particulate nitrate concentrations,

whereas the observed values are typically in the 0.25-0.75 μ g/m 3 range. So both the modeled and observed summer NO3 values suggest that it is not an important component of the total

PM 2.5 mass concentrations in the sout heastern U.S. during the summer.

Fairly similar trends are observed when compari ng the model to either the STN (Figures 3-21 and C-2a) or the IMPROVE (Fi gure C-2b) networks, however a more detailed analysis is provided based on the STN given its larger size and density.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-41 Figure 3-21. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) nitrate (NO3) concentrations over the VI STAS 12 km domain (IMPROVE comparisons are provided in Appendix C).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-42 3.6.3 Ammonium (NH4)

Ammonium concentrations depend, to a large degree, on the availability of sulfate and nitrate. Therefore, ammonium levels do not exhibit as strong seasonal pattern, due to the seasonal

tradeoff between sulfate and nitrate to which it bonds (Figures 3-22 and C-3).

Model performance when compared to the STN data (Figures 3-22 and C-3a ), seems to be better during summertime (Quarters 2 and 3) when the am monium is mainly associated with sulfate. During wintertime (Quarters 1 and 4), most of th e issues observed with nitrate overpredictions are also evident in the ammonium plots, especi ally over North-Carolina a nd the urban centers of

the Southeast.

Ammonium is not directly measured by IMPROVE, rather it is derived by assuming it completely neutralizes the meas ured sulfate and nitrate. A lthough assuming that nitrate is completely neutralized by ammoni um is, most of the time, a va lid assumption, the same may not be true for sulfate, especially in the summer months. Hence the observed ammonium comparison in the IMPROVE case (Figure C-3b) is likely overstated du ring the summer months and the seemingly underpredicted observed amm onium concentrations across the Appalachian Mountain IMPROVE monitoring sites in Quarter 3 in Figure C-3b are in part due to using

derived observed ammonium.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-43 Figure 3-22. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) ammonium (NH4) concentrations over the VI STAS 12 km domain (IMPROVE comparisons provided in Appendix C).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-44 3.6.4 Organic Carbon Material (OCM)

Evaluating model performance for OCM is complicat ed due to its dual primary/secondary nature and anthropogenic/biogenic sources. As a resu lt, the evaluation is dependant on a combination of very different emissions and formation proce sses. In addition, ther e is a fair amount of uncertainty associated with OC measurements with the major limitation being the inability to directly measure OCM. Instead, only OC is me asured and OCM is estimated (and compared to model results) by multiplying OC by an OCM/OC ra tio factor. OCM/OC ratio factors typically range from 1.2 to 2.2 depending on whether the OCM is fresh or aged. For this model performance evaluation analysis, an OCM/OC f actor of 1.4 has been used to convert the observed OC to OCM. Another comp licating factor is that different analysis methods are used in the STN (NIOSH) and IMPROVE networks that introduce additional measurement artifacts and

uncertainties.

The seasonal pattern of OCM concentrations wo uld depend on the relative contributions from secondary OCM (peak concentrat ions during summertime) and pr imary OCM. The seasonal pattern of primary OCM would depend on the seasona lity of activity and emissions from sources such as biomass burning and transportation. Gi ven a flat activity/emissions profile, primary OCM is would typically be higher in wintertime, due to reduced atmospheric mixing.

Comparing the modeled quarterly OCM predicti ons to STN observations (Figures 3-23 and C4a), both exhibit a fairly flat seasonal patter n. Overall, the modeled OCM concentrations are lower compared to the observations. The model seems to exhibit a much stronger spatial pattern, with peak concentrations occurring in the South eastern U.S. This is likely associated with biomass burning in the winter m onths, and biogenic secondary OCM formation in the summer

months. The increased modeled concentrations of secondary OCM are evident in Quarter 3, especially over the Northeast. When comp aring simulated concentrations to IMPROVE observations (Figure C-4b), bette r agreement is observed. Given that most IMPROVE sites are located in rural areas, this ma y be an indication that the OC M underprediction at urban sites (STN, Figures 3-23 and C-4a) may possibly be due to local spatial gradients and the comparison of 12 km grid-cell volume average co ncentrations to point measurements at urban sites. It is also partly due to measurement artif acts where the STN OC observati ons are not “blank corrected” so are biased high.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-45 Figure 3-23. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) organic carbon material (OCM) concentrations over the VISTAS 12 km domain (IMPROVE comparisons provided in Appendix C).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-46 3.6.5 Elemental Carbon (EC)

Evaluating model performance for EC, the model tends to underpredict concentrations when compared to the STN network (Figures 3-24 and C- 5a), while better agreement is observed with the IMPROVE network (Figure C-5b). This follows the same conclusion as for OCM, and may be reflecting the differences between grid-ce ll volume average concentrations and point measurements. These differences are expected to be more pronounced at urban sites (STN) compared to rural ones (IMPROVE), as at urban locations there are likely stronger local spatial gradients in EC concentrations. The STN and IMPROVE networks also use different methods for measuring EC that may also be contributin g to the differences in EC model performance using the two networks. In Section 3.5 and Appendix B we noted that the CMAQ 12 km modeling results exhibited superior model pe rformance to the CMAQ 36 km modeling results

across the urban STN network with bias closer to zero with the 12 km grid versus a large underprediction bias with the 36 km grid. Thus , the dilution of the urban EC emissions across coarse grid cell is sure ly contributing to the EC underpredict ion tendency at the STN monitoring

network.

Both the modeled values and obs ervations (at both networks) exhi bit higher concentrations in wintertime (Quarters 1 and 4), that is likely due to reduced atmospheric mixing compared to summertime. However, the seasonal pattern is much stronger in the model results compared to

the observations.

Another interesting finding has to do with relati vely high modeled EC concentrations in the western part of the Georgia-Flor ida border, especially in Quarter 1. This is due to modeled emissions from biomass burning ac tivities. However, nearby meas urements at both the STN and

IMPROVE networks do not exhibit the same trend.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-47 Figure 3-24. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) elemental carbon (EC) concentrations over the VISTAS 12 km domain (IMPROVE comparisons provided in Appendix C).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-48 3.6.6 Soil PM 2.5 (SOIL)

Soil PM 2.5 is measured/calculated/derived at the IM PROVE sites only (Figures 3-25 and C-6). In general, soil PM 2.5 is overpredicted compared to the observations. This is a common issue in grid-based models, where resuspended fugitive dust is assumed to be mixed uniformly in the first layer of the model, whereas in practice most of it is removed locally by impaction onto surfaces such as cars, buildings and vegetation. Higher c oncentrations are especi ally evident in urban centers, due to the way the model calculates th e emission and dispersion of resuspended road dust. However, this is not very evident in this comparison, since most IMPROVE sites are

located at rural areas. The ASIP/VISTAS m odeling did apply fugitive dust transport factors (FDTF) to fugitive dust emission source categories to account for dust that is deposited locally and not transported.

Another issue associated with th e Soil evaluation is the mismatch between how Soil is defined in

the measurement versus modeled values. The m easured Soil values are derived by building it up from the elements, whereas the modeled values are obtained from the emission speciation profiles that are assigned all PM 2.5 to the other PM 2.5 species that is not e xplicitly speciated as SO4, NO3, EC or OCM.

Also evident in the figures are the higher modeled concentrations in the western portion of the ASIP/VISTAS 12 km domain. This could be origin ating from various reasons, such as possibly higher emissions in that region, transport into the domain from the west, or differences in methodologies used to calculate soil emissi ons between the various Regional Planning Organizations (RPOs). The latter is likely the cas e, since a sharp spatia l gradient is observed between the Midwest RPO states (and west of Mississippi, in the portions of the CENRAP states) and the MANE-VU (north east) and VISTAS (southeast) states (see Figure 1-1 for definition of the RPO states).

Regardless of actual model perfor mance, in the context of St ate Implementation Plan (SIP) development and future attainment tests, mode led levels of soil components are of relatively little importance, since they are normalized according to observat ions, and little to no controls are being applied (so the Relative Reduction Fa ctor would be equal or close to unity).January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-49 Figure 3-25. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) soil PM 2.5 (SOIL) concentrations over the VISTAS 12 km domain.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-50 3.6.7 Sea-salt PM 2.5 (SeaSalt)

Sea-salt PM 2.5 is measured/calculated at some of th e IMPROVE sites only (see Figure C-7 in Appendix C). As expected, higher concentrations are measured at sites along the coastline. This spatial pattern is captured by the model as wel l; however, concentrations along the coastline are underpredicted compared to the measurements.

Since sea-salt PM 2.5 is not used for future attainment tests as part of the SIP development process, model performance in this case is of relatively little importance.

3.6.8 Total PM 2.5 Mass (PM 2.5 )

A comparison of the quarterly modeled spatial distribution of total PM 2.5 mass concentrations with observations from the FRM, STN and IMPROVE network are provided in Figure C-8, with the results for the FRM network repeated in Fi gure 3-26 below. The conclusions on the model performance for total PM 2.5 mass vary by which network is examined. For Q1 there are elevated

PM 2.5 concentrations in the major urban areas (e.g., Chicago-Gary, St . Louis, Atlanta, Northeast Corridor, etc.) due to concentrated urban emissions. There are also elevated PM 2.5 mass concentrations for Q1 in the western part of th e Georgia-Florida border that is due to biomass burning. The mainly rural IMPROVE monitors fail to capture many of these elevated areas, but

does confirm the relative clean co nditions in the Appalachian Mountai ns (Figure C-8c, top left).

The IMPROVE St. Marks measured PM 2.5 levels appears to refute the elevated PM 2.5 in Q1 along the western Florida-Georgia border. However, the STN (F igure C-8b) and FRM (Figures 3-26 and C-8a) spatial plots confirm that elevated PM 2.5 levels existed in this area; the lack of

elevated PM 2.5 at St. Marks is likely due to its coastal location and the sea breeze.

During Q2 the model predicts elevated total PM 2.5 mass levels from St. Louis across the Midwest into Ohio with highest values centered on Indian apolis, in southeastern Pennsylvania and up the Northeast Corridor and in the Birmingham and A tlanta urban areas in the southeast. The IMPROVE monitors suggest that the model is capturing the rural aspect of the spatial

distribution of the total PM 2.5 mass patterns, albeit with an unde rprediction bias (Figure C-8c, top right). However, the FRM (Figures 3-26 and C- 8a) and STN (Figure C-8b) plots indicates that the model is underestimating the spatia l extent of the elevated total PM 2.5 mass levels.

The highest seasonal PM 2.5 levels occur in the Q3 summer period when SO4 is the highest. The

observations indicate that the en tire interior portion of the AS IP/VISTAS 12 km grid domain is covered by elevated PM 2.5 levels, which is reproduced reasonab ly well by the model. However, the model is estimating slightly lower values and relatively cleaner areas over the Appalachian Mountains that are not supported by the ob servations. In addition, the observed PM 2.5 concentration gradient from high to low occurs further south than pred icted by the model. However, in general the model is doing a good j ob in reproducing the spatial distribution of

PM 2.5 in Q3.

In Q3 the model estimates elevated PM 2.5 levels across the upper Midwest (MO-IL-OH) and southwest (Northern AL and GA, SC and NC) th at is split by the Appa lachian Mountains and elevated levels in the Northeast Corridor. The IMPROVE network plots confirm the relative cleaner area in the Appalachian Mountains (Figur e C-8c) and the STN network plot confirms theJanuary 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-51 three areas of high PM 2.5 (Figure C-8b). The FRM network spatial map (Figure 3-26 and C-8a)

confirms the three areas of elevated PM 2.5 concentrations, but suggest s the southeast area is not as high as the others and that the PM 2.5 distribution should be spottier.

In conclusion, the model appears to do a good job in reproducing the spatial and temporal

variations in PM 2.5 concentrations across the ASIP/VIS TAS 12 km grid albeit with an underprediction bias. The spatia l distribution of the modeled PM 2.5 concentrations is smoother and less spotty than the observed distributions, wh ich is due in part to the coarse 12 km grid spacing used in the modeling.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-52 Figure 3-26. 2002 Quarterly averages of CMAQ simulated and FRM observed (diamonds) total

PM 2.5 mass concentrations over the VISTAS 12 km domain.January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-53 3.7 CONCLUSIONS OF MODEL PERFORMANCE EVALUATION

The quarterly average CMAQ PM 2.5 component species results near the FRM monitors are used in the Speciated Model Attainment Test (SMAT) to project the current year PM 2.5 Design Values to the 2009 future year for comparison with the PM 2.5 NAAQS. SMAT uses the relative changes CMAQ 2002 and 2009 modeling results to scale each of the PM 2.5 components of the current year PM 2.5 Design Values. These model derived scal ing factors are referred to as Relative Response Factors (RRFs). In Chapter 4 of this TSD, we present the results of the SMAT PM 2.5 Design Value projections for FRM monitoring sites in and near th e ASIP states. These results

indicate that most of the reductions in the PM 2.5 Design Values between 2002 and 2009 are due

to reductions in sulfate concentr ations. Thus, performance of th e model for sulfate is of most importance in the model performance evaluation.

Below we discuss the implications of the ma jor findings in the AS IP model performance

evaluation in the context of the modeling results are used to project future year PM 2.5 attainment through SMAT.

• Nitrate Underprediction Tendency : NO3 is routinely under predicted during the summer and adjacent months throughout the ASIP regi on. This underprediction is due to modeled NOx concentrations near zero, when observed va lues are low, but above zero (typically <

1 μ g/m 3 ). However, NO3 is almost always a ve ry minor to insignificant contributor to

total PM 2.5 mass at FRM monitors in the ASIP regi on (see stacked bar charts in Chapter 4). In fact, the maximum NO3 contribution to a 2009 projected PM 2.5 Design Value is

1.0 μ g/m 3 with a median value of 0.2 μ g/m 3 . Thus, the NO3 performance issues are not a big concern in the PM 2.5 projections.

• OCM Underpredictions : The OCM underprediction bias is a cause for concern since it is a major component of the PM 2.5 mass at ASIP FRM monitoring sites with maximum

contributions to the 2009 PM 2.5 Design Values of ~8 μ g/m 3 , minimum values of ~3 μ g/m 3

and a median value of ~4 μ g/m 3 . The reasons for the u nderestimation of OCM are unclear, but the fact that th e underpredictions are higher in the urban than rural areas suggest that there may be missing anthropoge nic emission sources, or possibly the urban OCM emissions are over diluted across the 12 km grid resolution used in the ASIP modeling. The changes in projected OCM concentrations between the current and projected PM 2.5 Design Values are mostly less than 5% (i.e., 0.95 < RRF OCM < 1.05). Thus, the changes in OCM between the current and future year are having a minor influence on the projected PM 2.5 Design Values.

• EC Performance Issues : The EC underprediction bias at the urban sites is partly due to

over dilution of the urban EC emissions due to the coarse (12 km) grid used. For the most part, the CMAQ model performed well for EC and the underprediction would not affect the relative changes in the model respons e to anthropogenic EC emissions changes. Therefore, any EC performance issues ar e not a cause for concern, although the model performance for EC was generally good.

• Sulfate Underprediction Bias : Although SO4 is performi ng well, it does have an underprediction bias that is la rgest in the summer months. But this underpr ediction is not severe and the model appears to be capturing the temporal and spatial variations in the

observed sulfate well and is responding to the SO2 emi ssion reductions between 2002January 2008

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-54 and 2009 in a manner as expected. Thus the model performance indicates that the modeled relative changes in SO4 concentr ations are likely a valid response.

• Soil Performance Issues : The CMAQ performance for the Soil species is quite poor. This Soil component of the 2009 projected PM 2.5 Design Value ranges from 0.4 to 1.8

μ g/m 3 . The RRFs for Soil indicate that it is mostly increasing, with summer (Q3) Soil RRFs typically ranging from 1.0 to 1.3. We address the Soil performance issues in Chapter 5 weight of evidence analysis where we assumed that the modeled Soil remains unchanged between 2002 and 2009 which did not significantly affect the projected 2009

PM 2.5 Design Values.

SO4 reductions dominate the changes in PM 2.5 Design Values between 2002 and 2009. SO4 performance is quite good in the CMAQ 2002 Base G2 Actual base case simulation almost always achieving the PM performance goal and fr equently also achieving the more stringent ozone performance goal. These factors provid e confidence in the De sign Value projections using the CMAQ Base G modeling results.APPENDIX B

Air Quality Model Performance Evaluation: State-by-State and Key Monitoring Site PM 2.5 Model EvaluationB-1 B.1 OVERVIEW

The discussion of model performance evaluation in this Appendix focuses on the evaluation of

the 2002 Base G2 Actual base cas e modeling results from the Comm unity Multiscale Air Quality (CMAQ) modeling system Version 4.5 with S OAmods enhancement across each VISTAS state and at key PM 2.5 monitors within the VISTAS states. The CMAQ results are compared with

observational data from the S peciated T rends N etwork (STN), F ederal R eference M ethod (FRM) PM 2.5 mass and S outh E ast A erosol R esearch and CH aracterization (SEARCH) study monitoring networks. The eval uation focuses primarily on the air quality model’s performance with respect to individual compone nts of fine particulate matter (PM 2.5 ) and total fine particulate matter (PM 2.5 ) mass within PM 2.5 nonattainment areas (NAAs) in the VISTAS region. Consequently, the focus is on the STN and FRM monitoring sites that are located within the

PM 2.5 NAAs although SEARCH also includes two si zes in NAAs (Atlanta and Birmingham). Previously we have presented an evaluati on of the CMAQ 2002 Base G2 Actual base case simulation focusing primarily on PM and visibility as part of the Visibility Improvements for

States and Tribal Associations in the South east (VISTAS) that focu sed on the regional model performance and performance at Cl ass I areas (Morris et al., 2007).

B.2 CMAQ EVALUATION METHODOLOGY

EPA’s integrated ozone, PM 2.5 and regional haze modeling guida nce calls for a comprehensive, multi-layered approach to model performance testi ng, consisting of the four major components: operational, diagnostic, mechanistic (or sc ientific) and probabilis tic (EPA, 2007a).

B.2.1 Evaluation Approach

The CMAQ model performance evaluation effort fo cused on the first two components, namely:

• Operational Evaluation : Tests the ability of the model to estimate PM 2.5 concentrations and the components of PM 2.5 (i.e., sulfate, nitrate, amm onium, organic carbon, elemental

carbon and other PM 2.5 ) within the PM 2.5 nonattainment areas. This evaluation examines whether the measurements are properly repres ented by the model predictions but does not necessarily ensure that the model is getting “the right answer for the right reason”; and

• Diagnostic Evaluation: Tests the ability of the model to predict PM chemical composition including PM precursors (e.g., SOx, NOx, and NH 3 ) and associated oxidants (e.g., ozone and nitric acid); PM size distributi on; temporal variation; spatial variation; mass fluxes; and components of light exti nction (i.e., scattering and absorption).

The diagnostic evaluation can al so include the performance of diagnostic tests to better understand model performance and identify potential flaws in the modeling system that can be corrected. The diagnostic evalua tion may also includes the use of “probing tools” to understandB-2 why the model obtains a given prediction; pr obing tools include Process Analysis (PA), decoupled direct method (DDM) and source apportionment (SA).

In this final model performance evaluation for the ASIP 2002 Actual Base G2 CMAQ 36/12 km base case simulation, the operational evaluation for PM 2.5 has been given the greatest attention since this is the primarily thrust of EPA’s mode ling guidance. However, we have also examined certain diagnostic features dealing with the model’s ability to simulate sub-regional and monthly/diurnal gas phase and aero sol concentration distributions. In the course of the ASIP and VISTAS studies, numerous diagnostic sensitiv ity tests were performed to investigate and improve model performance. Key diagnostic test s performed are discussed and the results for the rest are available on the proj ects modeling website: http://p ah.cert.ucr.edu/vistas/vistas2/.

B.2.2 Particulate Matter and Component Species

Fine particulate matter can be composed of vary ing amounts of several different species. The main components of PM 2.5 are as follows:

• Sulfate (SO4)

• Nitrate (NO3)

• Ammonium (NH4)

• Organic Carbon Matter (OCM)

• Elemental Carbon (EC)

• Soil (also known as crustal material, fine soil , major metal oxides, inorganic particulates, or other PM)

• Sea Salt (NaCl)

• Particle Bound Water (PBW)

B.2.3 Ambient Air Quality Data for ASIP Model Evaluation

A ground-level model evaluation database for 2002 was compiled by the modeling team using several routine and research-grade databases. The focus of the evaluation of the CMAQ model for use in simulating PM 2.5 mass concentrations in urban areas and NAAs. The primary monitoring networks available to evaluate this component of the CMAQ are: (a) EPA Federal

Reference Method PM 2.5 and PM 10 Mass Networks (EPA-FRM); (b) EPA Speciation Trends Network (STN) of PM 2.5 species; and (c) two sites from the Southeastern Aerosol Research and Characterization (SEARCH). These PM monitori ng networks may also provide ozone and other gas phase precursors and product species, and visibi lity measurements at some sites. There are also gas-phase criteria pollutant measurements in NAAs from the EPA’s Air Quality System (AQS) that are available for use. In addition, there are several more rural monitoring networks that collect PM or PM related species, such as IMPROVE, CASTNet and NADP. VISTAS used

these more rural networks to perform a regional evaluation of the CMAQ 2002 Base G2 base case that is presented elsewher e (Morris et al., 2007). The st udy team has performed a detailed evaluation of the CMAQ 2002 base case simulati on using the AQS that is available on the modeling website: http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml . Table B-1 and Figure B-1B-3 summarizes the species collected and locations of the monitoring sites for the FRM, STN and SEARCH monitoring networks use in the ASIP PM 2.5 model evaluation.

Table B-1a. Ambient PM 2.5 monitoring data available in PM 2.5 nonattainment areas in the ASIP

region during 2002.

Monitoring

Network Chemical Species Measured Sampling Frequency;

Duration

SEARCH 24-hr PM 25 (FRM Mass, OC, BC, SO 4 , NO 3 , NH 4 , Elem.); 24-hr PM coarse (SO 4 , NO 3 , NH 4 , elements); Hourly PM 2.5 (Mass, SO 4 , NO 3 , NH 4 , EC, TC); and Hourly gases (O 3 , NO, NO 2 , NO y , HNO 3 , SO 2 , CO) Daily, Hourly;

EPA-FRM Only total fine mass (PM 2.5 ) 1 in 3 days; 24 hr

EPA-STN Speciated PM 2.5 Varies; Varies

B.3 MODEL PERFORMANCE STATISTICS

To quantify PM 2.5 model performance, severa l statistical measures were calculated and evaluated

for all the monitors within each ASIP state wi th a NAA and at individua l monitors within the

NAAs. The statistical measures selected were based on the recommendations outlined in section

18.4.1 of the USEPA’s Guidance On The Use Of M odels And Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and Regional Haze (EPA, 2007a). VISTAS/ASIP has established model pe rformance goals and criteria for PM 2.5 and components of fine particle mass based on previous model pe rformance for ozone and fine particles that are shown in Table B-1b. EPA modeling guidance for fi ne particulate matter at the time noted that PM models might not be able to achieve the sa me level of performance as ozone models.

Table B-1b. Model performance goals and criteria for components of fine particle mass.

Fractional Bias Fractional Error Comment

< 15% < 35% Goal for PM model performance based on ozone model performance, considered excellent performance

< 30% < 50% Goal for PM model performance, considered good performance

< 60% < 75% Criteria for PM model performance, considered average performance. Exceeding this level of performance indicates fundamental concerns with the modeling system and triggers diagnostic evaluation.

B.4 PM 2.5 MODELING PERFORMANCE ACROSS THE ASIP STATES

In the following discussions we use selected monthly scatter plots, time seri es plots, soccer plots, bugle plots and model performance statistical measures from the UCR Analysis Tool application to the 2002 CMAQ Base G2 Actual base case simu lation in an operational evaluation of theB-4 model for PM 2.5 mass and components of PM 2.5 within each ASIP state. These results represent just a small subset of the model performa nce evaluation products of the CMAQ 2002 36/12 km Base G2 base case simulation. Complete model evaluation products are available on the project

website:

http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml

The focus of the model performance evaluation in this Appendix is on PM 2.5 and PM 2.5 components within PM 2.5 nonattainment areas (NAAs) and urba n areas in the ASIP region. The regional performance evaluation of the model for PM and visibility has been documented in the VISTAS Regional Haze SIP Technical Support Docu ment (TSD; Morris et al., 2007), which in addition to the products availa ble on the website includes:

• Regional operational and diagnostic model pe rformance evaluation for PM and visibility across the VISTAS/ASIP region in the sout heastern U.S. (Appendix B of VISTAS TSD);

• Model performance for PM and visibility at Class I areas in the VISTAS region. (Appendix C of VISTAS TSD); and

• Spatial model performance for PM compone nts across the VISTAS/ASIP 12 km grid. (Appendix D of VISTAS TSD).

The performance of the ASIP/V ISTAS CMAQ 2002 36/12 km Base G2 base case simulation for

ozone has also been evaluation a nd the results are available at:

http://pah.cert.ucr.edu/vistas/vis tas2/evaluation_results/2002ga2a.mpe/

B.4.1 Alabama

Figures B-2a-f contain s catter plots and model performance statistics for PM 2.5 component species at STN sites located in Alabama, the f our seasonally representative months of January, April, July and October and the CMAQ 2002 36/ 12 km Base G2 base case simulation. Performance for SO4 is quite good with the scatter plots clustere d tightly around the 1:1 line of perfect agreement (Figure B-2a) and fractional bi as and error performance metrics achieving the most stringent <±15% and < 35% performance goal (Table B-1b). Performance for NO3 exhibits an underprediction tendency with fractio nal bias values of approximately -20% in January, -110% in April, -175% in July and -1 0% in October (Figure B-2b). The large summer NO3 underprediction bias occurs when both the model and measured NO3 values are extremely

low with the modeled values near zero and with the STN NO3 observed values < 1 μ g/m 3 . Note that under these summer conditions, the NO3 will almost completely vola tilize off the FRM filter so the FRM would measure insignificant PM 2.5 mass due to NO3. Thus, these large summer NO3 underpredictions are not a concern as NO3 is not an important component of PM 2.5 mass. Performance for NH4 is also fairly good achievi ng the <±30%/50% fractional bias/error goal for PM species (Figure B-2c), which is not surpri sing given its strong link to SO4. With the exception of a few outliers, performance for orga nic carbon mass (OCM or OC) is fairly good inB-5 the winter, but underestimated by -20% to -30% in the summer and fall (Figure B-2d). The reasons for the OCM underpredic tion includes the fact that the STN OC measurements are not blank corrected and have high uncertainties (see SANDWICH discussion in Chapter 4) and there are also large uncertainties in the OC emissions and the form of OC emi ssions as particles or semi-volatile organic gases. EC model performance at STN sites in Alabama is also fairly good, mostly achieving the PM model performance goal, albeit with an under estimation tendency in the summer (Figure B-2e).

The CMAQ performance for PM 2.5 mass at 3 STN and 20 FRM monitoring sites in Alabama are shown in Figures B-2f and B-2g, respectively. Performance for PM 2.5 mass is generally good achieving the most stringent ozone performance goal (<±bias/error) in January and October and the PM goal all year round.

Time series of predicted (CMAQ 12 km results) and observed 24-hour PM 2.5 components and model performance statistics at the key North Bi rmingham STN site are shown in Figure B-3. As discussed in Chapter 4, this is the site with the highest projected 2009 PM 2.5 Design Value in the ASIP region. SO4 performance is quite good with annual frac tional bias and errors of 6% and 29% achieving the most stringe nt ozone performance goal (Figure B-3a, top). The fractional

bias for NO3 of -110% indicates a large underprediction bias that is driven by near zero modeled

NO3 concentrations in the summer wh en the observed values are < 1 ug/m 3 (Figure B-3a, bottom). However, the NO3 performance in the winter, when observed values are above 1 ug/m 3
is much better. The EC and OCM performa nce at the North Birmingham STN site is characterized by an underprediction with fractiona l bias of -30% and -49%, respectively (Figure B-3b). This underprediction is due in part to uncertainties in the STN carbon measurements as

well as the contributions of nearby industrial sources whose imp acts are diluted across the 12 km CMAQ grid used. Performance for NH4 and PM 2.5 mass at the North Birmingham STN site are also quite good with fractional bias/error values of -5%/33% and 11%/29% achieving the more stringent ozone performance goal. The good performance of NH4 and PM 2.5 mass is not surprising given that they are closely tied to SO4 that exhibits good model performance.

Figure B-4 contains time series of predicted and observed PM components and performance

statistics for the SEARCH BH M monitoring site also locat ed at North Birmingham.

Performance at the SEARCH BHM site is simila r to the STN site at the same location with generally good performance for SO4, NH4 and PM 2.5 mass and an underprediction bias for EC

and OCM. The SEARCH BHM site also includes performance for the “So il” species that is greatly overestimated with annual fractional bias values of +164% (Figure B-4d, top)

Figure B-5 displays a soccer plot of monthl y fractional bias versus gross error for PM 2.5 mass performance across 20 FRM sites in Alabama and compares them with the three levels of bias/error performance goals in Table B-1b. PM 2.5 model performance is best in the winter months when the most stringent ozone < ±15%/35% bias/error goal is achi eved. In the summer

months the model tends to underestimate the FRM observed PM 2.5 mass by -15% to -25%, but

still achieves the <±30%/50% PM model performance goal. Th e Bugle Plots of monthly PM 2.5 bias and error in Figure B-6 also confirm that the bias and error performance goals for PM are

achieved across FRM sites in Alabama.B-6 The PM 2.5 model performance products on the proj ect website also include a day-by-day

evaluation of the PM 2.5 model performance at STN sites. Th is is done with si de-by-side stacked

bar charts of PM 2.5 components for each sampling day in 2002 and each STN site in Alabama. These model performance displays can be found at:

http://pah.cert.ucr.edu/vistas/vistas2/e valuation_results/2002/ 2002ga2a/MPE/stackbar/ST N/AL_STN_stackbar_monthly.pdfB-7 Figure B-2a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-8 Figure B-2b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-9 Figure B-2c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-10 Figure B-2d. Scatter plots of predicted and observed or ganic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-11 Figure B-2e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-12 Figure B-2f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-13 Figure B-2g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in Alabama and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-14 Figure B-3a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Jefferson County, Alabama Site No. 01- 073-0023 (North Birmingham) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-15 Figure B-3b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Jefferson County, Alabama Site No. 01-073-0023 (North Birmingham) for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-16 Figure B-3c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Jefferson County, Alabama Site No. 01-073-0023 (North Birmingham) for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-17 Figure B-4a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the SEARCH North Birmingham (NBHM) Alabama
site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-18 Figure B-4b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the SEARCH North Birmingham (NBHM) Alabama site for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-19 Figure B-4c. Time series of predicted and observed 24-hour ammonium (NH4, top) and total carbon mass (TCM, bottom) concentrations during 2002 at the SEARCH North Birmingham (BHM) Alabama site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-20 Figure B-4d. Time series of predicted and observed 24-hour other PM 2.5 (Soil, top) and total fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the SEARCH North Birmingham (BHM) Alabama site for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-21 Figure B-5. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Alabama for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-22 Figure B-6. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites
in Alabama and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-23 B.4.2 Florida

Figure B-7 summarizes the CMAQ PM 2.5 species model performance across the three STN sites in Florida and January, April, July and October 2002. SO4 is unde restimated at the Florida STN sites (Figure B-7a) with fractional bias values th at are closer to zero in the winter than summer (e.g., -7% January versus -34% July 12 km results) and closer to zero using the 12 km than 36 km CMAQ modeling results (e.g., in January bias , is -7% using 12 km and -22% using 36 km modeling results). Even with this underestimation bias SO4 pe rformance at the Florida STN sites is fairly good. NO3 performance, on the ot her hand, exhibits a la rge underprediction bias with fractional bias values that range from -70% in January to -170% in July (Figure B-7b). NH4 performance at the Florida STN sites is fairly good always achieving the <±30%/50% PM performance goal and sometimes even achie ving the more stringent <±15%/35% ozone performance goal (Figure B-7c). It is interesti ng that in January and October when both SO4 and NO3 exhibit an underprediction bias, NH4 exhibits a slight overprediction bias. Given the linked relationship between these species this is somewhat surprising and may be due to artifacts in the measurements, or the modeled SO4 being mo re fully neutralized by NH4 than in the

measurements. The OCM and EC are character ized by underprediction tendency with the OCM fractional bias ranging from -55% to -95% with the CMAQ 36 km and 12 km producing nearly identical performance metrics (Figure B-7d). The EC performance is bette r with fractional bias values ranging from -4% to -5 2% and the 12 km modeling resu lts produce significantly better EC performance metrics than the 36 km modeling results (Figure B-7e).

The PM 2.5 mass performance in Florida is summarizes across 3 STN sites and 28 FRM sites in Figures B-7f and B-7g, respectivel y. Across the STN sites, the PM 2.5 fractional bias ranges from -27% to -75% for the four months examined, wh ereas across the FRM network the bias values are lower ranging from+2% to -69%. Given th e underprediction tendency of SO4, NO3 and carbon the general underprediction of total PM 2.5 mass is not unexpected. Why the underprediction bias is greater across the STN network than the FRM network is unclear but is likely related to the locations of the 3 STN and 28 FRM monitoring networks and the

measurement artifacts of the two sampling techniques (e.g., FR M will loose more of the volatile material).

An example predicted and observed time series and annual model performance metrics for 24- hour PM components at an STN s ite in Miami-Dade County Florid a is shown in Figure B-8. The summer sulfate underprediction tenden cy results in an a nnual fractional bias of -30% with error of 44% just barely achieving the PM model performance goals (Figure B-8a, top). NO3 performance at this Miami site is very poor w ith a -180% fractional bias with modeled NO3 on most days near zero, compared to around 0.5 μ g/m 3 measured values during the summer. OCM and EC are also underpredicted at the Miami site with fr actional bias values of, resp ectively, -123% and -84% (Figure B-8b). Ammonium performance is fairly good meeting the most stringent ozone performance goal, whereas total PM 2.5 is underpredicted due to the underprediction of most of the PM compone nts as discussed above (Figure B-8c).

The soccer plot of monthly PM 2.5 bias and error performance for FRM sites in Florida is shown in Figure B-9, with the Bugle Pl ots shown in Figure B- 10. For the winter months of Nov, Dec,B-24 Jan and Feb the PM 2.5 performance statistics are quite good with bias/error values meeting the <±15%/35% ozone performance goal. The sp ring and fall months either achieve the <±30%/50% or <±50%/125% performance goal and cr iteria. The May-Jul summer months are at or exceed the -50% performance crit eria levels (note that in the Soccer Plots if the symbol would be plotted outside of the range of the plot then it is plotted on the axis, su ch as the May and July -50% values in Figure B-9).B-25 Figure B-7a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base G2 Actual base

case simulation.B-26 Figure B-7b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base G2 Actual base

case simulation.B-27 Figure B-7c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-28 Figure B-7d. Scatter plots of predicted and observed or ganic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-29 Figure B-7e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-30 Figure B-7f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-31 Figure B-7g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in Florida and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-32 Figure B-8a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Miami-Dade County, Florida Site No. 12-
086-1016 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-33 Figure B-8b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Miami-Dade County,
Florida Site No. 12-086-1016 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-34 Figure B-8c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Miami-Dade County,
Florida Site No. 12-086-1016 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-35 Figure B-9. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Florida for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-36 Figure B-10. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites
in Florida and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-37 B.4.3 Georgia

There appears to be some valid data capture i ssues in the 2002 STN MP E database for Georgia with only one valid site in January, 6 in Apri l and 8 in July and October. The CMAQSO4 performance in Georgia is generally pretty good with bias/error frequently achieving the <±15%/35% ozone performance goal and always achieving the PM goal (Figure B-11a). The STN sites also have the summer nitrate underpredi ction performance issue as seen in the other states (Figure B-11b). The NH4 model performance falls between the SO4 and NO3 performance and, with the exception of January which is based on one site so the statistics are not very robust, it achieves the <±30%/50% PM performance goal (Figure B-11c). The carbon performance in Georgia appears to be slightly better than the other states, although still with a general underestimation tendency. With the exception of July, when the CMAQ 36 km OCM performance exhibits fractional bias and error values of -34% and 42% thereby achieving the <<±30%/50% PM performance goal, the other th ree months achieve the most stringent <±15%/35% ozone performance goal (Figure B-11d). EC performance is vari able with generally low bias but lots of scatter producin g higher gross errors (Figure B-11e).

The performance of total PM 2.5 mass across the STN (Figure B-11f) and FRM (Figure B-11g) monitoring sites in Georgia is generally fairly good. July exhibits an u nderprediction tendency with fractional bias values of approximately -4 0% (STN) and -25% (FRM ) that is likely caused by the underestimation of SO4 and especially OCM. January is characterized by an overestimation tendency. But in general the PM 2.5 performance statistics mostly achieve the most stringent ozone performance goal.

An example time series of predicted and observe d PM concentrations a nd annual statistics for a site in the Atlanta Georgia (D eKalb County) is shown in Figur e B-12. The model tracks the

observed SO4 concentrations at this site extr emely well producing low fr actional bias (-5%) and gross error (27%). NO3 performance exhibits the underpredi ction tendency seen in the other states, but is not as severe with a -16% fractiona l bias and 77% error. The CMAQ 12 km results reproduces the OCM concentrations at this site surpri singly well with low bi as (-14%) and error (31%). EC is overestimated (bias of 28%) at th is Atlanta site with fairly high error (44%) but still achieves the PM performance goal. NH4 is overs tated at this site on average (29% error) but also achieves the PM performance goa l. Finally, the performance for PM 2.5 mass at the DeKalb County STN site is extremely good with ve ry low bias (2%) and error (25%).

Figure B-13 displays another example of predic ted and observed PM species, time series and

annual statistics for a site in Atla nta, only this time for the SEARCH Jefferson Street (JST) site. Unlike the STN 1:3 sampling frequency, the SE ARCH JST site collects 24-hour samples everyday so has a richer model evaluation data base. The performance at the SEARCH JST Atlanta is similar to the STN DeKalb County si te with good performance for SO4 that achieves the ozone performance goal, and NO3 performan ce that has an overall annual underestimation tendency (bias of -16%) and high error (80%). It appears the summer NO3 underestimation bias is being compensated by a winter NO3 overestimati on bias that can be quite large (Figure B-13a, bottom). The SEARCH JST site had some data capture problems for OC M so the performance is not very meaningful, but does suggest and underp rediction tendency as seen at other sites. Unlike the DeKalb County STN site that suggests an EC overestimation bias (28%), at theB-38 SEARCH JST site the CMAQ 12 km modeling result s exhibits a zero bias and fractional gross error of 37% (Figure B-13b, bottom). The ST N and SEARCH sites use different sampling techniques for carbon which may help explain these differences. The observed NH4 at the JST site is reproduced well (Figure B-13c, top) with a fr actional bias of -5% a nd error of 37%. The

Soil term measures at th e JST site is greatly overestimated with fractional bias of 136% (Figure B-13d). The Soil performance issue is discussed extensively in the VISTAS regional haze TSD (Morris et al., 2007) and two poten tial reasons for its poor perform ance are: (1) in compatibilities between the measured Soil that is built up from measured elements and the “Soil” in the modeling which is the other PM 2.5 that consists of PM 2.5 that is not explicitly speciated as SO4, NO3, OCM or EC; and (2) impacts of local fugitive dust sources atthe monitor that are subgridscale to the CMAQ 36/12 km modeling so is not captured by the modeling. Performance for

total PM 2.5 mass at the JST site is excellent (Figure B-13d, bottom) with a low bias (-3%) and error (24%).

Figure B-14 and B-15 displays Soccer and Bugle Plots, respectively, of monthly fractional bias and error across the 26 FRM sites in Georgia. All months achieve the PM performance goal and 8 of the 12 months even achieve the more stringe nt ozone performance goal. The months that

the PM 2.5 mass performance doesn’t achieve the ozone performance goal are due to an overestimation bias in January and underestimation bias in May-July.B-39 Figure B-11a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-40 Figure B-11b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base G2 Actual base

case simulation.B-41 Figure B-11c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-42 Figure B-11d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-43 Figure B-11e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-44 Figure B-11f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-45 Figure B-11g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in Georgia and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-46 Figure B-12a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN DeKalb County, Georgia Site No. 13-089-
0002 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-47 Figure B-12b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN DeKalb County, Georgia Site No. 13-089-0002 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-48 Figure B-12c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN DeKalb County, Georgia Site No. 13-089-0002for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-49 Figure B-13a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the SEARCH Jefferson Street (JST) Georgia site
for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-50 Figure B-13b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the SEARCH Jefferson Street (JST) Georgia site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-51 Figure B-13c. Time series of predicted and observed 24-hour ammonium (NH4, top) and total carbon mass (TCM, bottom) concentrations during 2002 at the SEARCH Jefferson Street (JST) Georgia site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-52 Figure B-13d. Time series of predicted and observed 24-hour other PM 2.5 (Soil, top) and total fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the SEARCH Jefferson Street (JST) Georgia site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-53 Figure B-14. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Georgia for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-54 Figure B-15. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites in Georgia and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-55 B.4.4 Kentucky

The monthly scatter plots and performance statis tics for PM species and sites in Kentucky are shown in Figure B-16. The performance for SO4 in January is mixed, with a group of points

clustered around the 1:1 line and a nother further to the right resu lting in a net underprediction bias of approximately -30%. An examination of the more detailed perform plots on the website reveals that this dichotomy in January SO4 mode l performance in Kentucky is due to temporal rather than spatial differences in performance; of the three STN samplin g days in January 2002 CMAQ does a very good job in simulating the SO4 model perf ormance on January 14 and 26 with fractional bias of approximately -5%, but on January 20 the bias is closer to -80%. The SO4 performance in other months is character ized by an underprediction bias in April and

October of -30% to -50% and near zero bias in July, but with some scatter for an error of ~40%.

As seen in the other states, NO3 performance is characterized by underprediction bias that is largest in the summer (Figure B-16b). In Kentuc ky, the NO3 fractional bias is -30% to -40% in January, -65% in April and -150% to -160% in July. In October the NO3 underprediction is compensated by some overprediction so that there is near zero bias but a 50-55% error.

Like SO4, NH4 performance is fairly good albeit with an underprediction bias that is largest in the summer (Figure B-16c). As seen for the ot her sites, OCM is underpre dicted with the bias ranging from -50% in January to 70% to -80% in the other three months (Figure B-16d). The OCM errors are the opposite sign but the same magnitude as the bias suggesting that the OCM underprediction bias is almost systematic. The CMAQ 36 km and 12 km modeling results exhibits different EC performance in Kentucky. The CMAQ 12 km results have bias that is much closer to zero than the 36 km results (Figur e B-16e). This is likely due to the 36 km grid cells over dispersing the urban EC emissions. Both the CMAQ 36 km and 12 km results exhibit a lot of variability producing a lot of scatter and high error metrics.

The performance of total PM 2.5 mass concentrations in Kent ucky is evaluated across 10 STN (Figure B-16f) and 23 FRM sites (Figure B-16g). Given the underprediction of most of the PM component species it is not surprising that total PM 2.5 mass is underpredicted across the STN sites by -16%, -42% to -55%, -33% and -6% to -13% for January, April, July and October. Across the FRM sites in Kentucky, however, the CMAQ model for PM 2.5 mass bias that is much closer to zero with values of -8%, -11% to -14% , -32% to -35% and +18% to -19% for the same

four months. The FRM PM 2.5 mass performance always achieve s the PM performance goal and sometimes even achieves the ozone model performance goal.

An example PM model performance for a singl e STN site in Lexington (Fayette County) Kentucky is shown in Figure B-17. The model re produces the temporal va riations in SO4 quite well (Figure B-17a, top) at this site resulting in low bias (-10%) and error (33%). Although the model shows some skill in reproduc ing the observed higher NO3 values in the winter, especially in the first part of the year, the near zero m odeled NO3 during the summer results in a fractional

bias value of -62% with 87% er ror, so does not achieve the PM performance goal (Figure B-17a, bottom).B-56 OCM is underpredicted with fractional bias values of -52% and an error of 58% just at the PM performance goal (Figure B-17b, top). EC exhib its much better performa nce at the Lexington, Kentucky site with a bias near zero (-4%) and error of 36% so almost achieves the ozone performance goal.

The performance for NH4 and PM 2.5 mass is quite good, which is not surprising given the good SO4 performance and the strong link between SO4 and these tw o species in Kentucky. The

fractional bias for NH4 and PM 2.5 are both approximately -10% with NH4 having an error of

36% and PM 2.5 an error of 28% thereby achievi ng the ozone performance goal.

Figure B-18 displays another example time seri es performance for PM components at an STN site in Kenton County, Kentucky that lies in the Cincinnati NAA. The performance at the

Cincinnati NAA STN site is si milar to that at Lexington STN site with very good SO4 performance that produces bias/error values of -12%/30% that achieve s the ozone performance goal and NO3 with a net underestimation with bias /error of -47%/75% that is right at the PM performance criteria (Figure B-18a). OCM e xhibits a large underprediction bias (-57%), whereas performance for EC is better with bias ( 14%) and error (36%) that is right at the ozone performance goal (Figure B-18b). NH4 (-4%/37%) and PM 2.5 (-7%/31%) also exhibit low bias/error and good performance, which again is not surprising given their relationship with SO4 (Figure B-18c).

Figure B-19 and B-20 summ arized the Kentucky PM 2.5 mass model performance using Soccer and Bugle Plots of monthly bias and error. With the exception of July, whose fractional bias falls slightly below -50%, PM 2.5 performance in the other 11 m onths achieves the PM bias/error performance goal of <-30%/75%. In fact 5 months that occur mainly in the winter even achieve the more stringent ozone model performance goal.B-57 Figure B-16a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-58 Figure B-16b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-59 Figure B-16c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-60 Figure B-16d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-61 Figure B-16e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-62 Figure B-16f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-63 Figure B-16g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM si tes in Kentucky and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-64 Figure B-17a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Fayette County, Kentucky Site No. 21- 067-0012 (Lexington) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-65 Figure B-17b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Fayette County, Kentucky Site No. 21-067-0012 (Lexington) for the CMAQ 12 km 2002 Base G2 Actual base

case simulation.B-66 Figure B-17c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Fayette County, Kentucky Site No. 21-067-0012 (Lexington) for the CMAQ 12 km 2002 Base G2 Actual base

case simulation.B-67 Figure B-18a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Kenton County, Kentucky Site No. 21- 117-0007 (Cincinnati) site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-68 Figure B-18b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Kenton County, Kentucky Site No. 21-117-0007 (Cincinnati) site for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-69 Figure B-18c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Kenton County, Kentucky Site No. 21-117-0007 (Cincinnati) site for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-70 Figure B-19. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Kentucky for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-71 Figure B-20. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites in Kentucky and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-72 B.4.5 Mississippi

Performance of PM species across sites in Mississ ippi is shown in Figure B-21. SO4 is generally underpredicted with fractional bias values rangin g from -15% to -46%. Worst SO4 performance is seen in the January and July winter/summer months than the April and October transitional months. NO3 performance metrics are extremely poor wi th fractional bias values of -80% to -90% in January and April and - 163% in July. In October some NO3 overpredictions begin to compensate for the frequent near zero model predic tions so that the underpre diction bias is lower (-40%), but the error is still very high (~100%).

NH4 performance is characterized by lots of scatter a nd little correlation, although some skill is seen in October (Figure B-21c). OCM is underpr edicted at sites in Missi ssippi with fractional bias values of approximately -50% in January, April and July improving to -24% in October (Figure B-21d). Performance for EC is better, al beit still with an underprediction bias, and again seen as for other states, the CMAQ 12 km mode ling results produces much lower bias than the 36 km modeling results (Figure B-21e).

PM 2.5 performance in Mississippi across the STN a nd FRM networks are shown in Figures B-21f and B-21g, respectively. Across the STN network, PM 2.5 is underestimated by approximately - 40% rising to -65% in July. Across the FRM network, the PM 2.5 underprediction bias is lower, at -14% in January and -25% in Ap ril and rising to approximately - 50% in July. In October the model is better able to replicate the observed PM 2.5 mass with CMAQ 12 km bias and error

values across the FRM network of 2% and 29% achieving the ozone model performance goals.

Figure 22 displays an example mode l performance time series analys is at an STN site in Jones County in southeastern Mississipp i (Hattiesburg). SO4 performance is worse than seen in the

other ASIP states with a -36% fr actional bias and 44% error. NO 3 performance exhibits similar

characteristics as the other ASIP states with a summer underestimation bias that produces an

annual fractional bias value of -40%. Like the other states, OCM is underestimated by -43% on average, but unlike the other states the underpre diction fractional bias for EC exhibits an underprediction (-60%) that is even greater than seen for OCM.

Performance statistics for NH4 are reasonable with a near zero bias and 42% error, the near zero bias is due to a few days of large overprediction compensated for a general underprediction.

Given the fact that almost all of the PM com ponents are underestimated, the underprediction of

total PM 2.5 mass is not a surprise with a fractional bias value of -35%.

The Soccer Plot (Figure B-23) and Bugle Plot (Figure B-24) for PM 2.5 mass performance confirm

that PM 2.5 performance in Mississippi is worse than seen for the other ASIP states. Only two months achieve the ozone model performance goal and only 8 months achieve the PM model performance goal. The underprediction bias in March through July 2002 produces fractional bias that do not achieve the <±50% PM performance goal.B-73 Figure B-21a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Miss issippi and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-74 Figure B-21b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-75 Figure B-21c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Miss issippi and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-76 Figure B-21d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN site s in Mississippi and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-77 Figure B-21e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-78 Figure B-21f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-79 Figure B-21g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in Mississippi and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-80 Figure B-22a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Jones County, Mississippi Site No. 28-
067-0002 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-81 Figure B-22b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Jones County, Mississippi Site No. 28-067-0002 for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-82 Figure B-22c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Jones County, Mississippi Site No. 28-067-0002 for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-83 Figure B-23. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Mississippi for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-84 Figure B-24. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites in Mississippi and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-85 B.4.6 North Carolina

SO4 performance across 9 STN monitoring sites in North Carolina and four months from 2002 is shown in Figure B-25a. Although SO4 performan ce achieves the PM performance goal it varies by month with January and July having fractional bi as of -245% top -30%, April a fractional bias of approximately -20% and Oc tober having near zero fractiona l bias. Although NO3 is still greatly underestimated in Nort h Carolina during the summer (Jul y bias of -105% to -142%), during January and October NO3 exhibits an over estimation bias with values of 84% and 35%, respectively (Figure B-25b). This is likely du e to overstated ammonia emissions during the winter and adjacent periods which is somewhat verified by the overstated ammonium during January and October that have fr actional bias values in the +40% to +55% range (Figure B-25c)

OCM is characterized by an underprediction bias ranging from -30% to -50%, although performance in the April and October transition months achieves the PM performance goal (Figure B-25d). EC, on the other hand, achie ves the PM performance goal in January and

October but has lots of scatter in the other two mont hs exceeding the PM performance goal for bias in April (-34% to -41%) and for error in July (62%-65%).

PM 2.5 mass performance across the 9 STN and 36 FR M sites in North Carolina exhibits monthly variations (Figures B- 25f and B-25g). FRM PM 2.5 is overpredicted in Ja nuary (15% to 18%) and October (25% to 30%), but underpredicted in April (-9% to -16%) and July (-37% to -39%). The

STN PM 2.5 performance is similar except because th e STN measurements tend to be higher the underprediction bias is greater and the overprediction bias is less.

Figures B-26 and B-27 displays time series pe rformance analysis at two STN sites in North Carolina, one on Charlotte (Meckle nburg County) and the other in Raleigh (Wake County). So4 performance is good at both sites, albeit with an underprediction bias of -14% for Charlotte and - 19% for Raleigh with both sites having low er ror (31% and 32%). Although both sites have a net annual NO3 underprediction bias (-19% and -38%) due to the summer underprediction tendency discussed previously, during the winter there are so me days when the CMAQ model overpredicts the observed NO3 by a f actor of 2-3. OCM is underpre dicted by a far margin at the two sites (-34% and -50%). At the Charlotte site EC is overpredicted by 45% which occurs year round, whereas at the Raleigh site EC is predicted well with a zero bias and 34% error. NH4 is predicted reasonably well with bias/error valu es that achieve or nearly achieve the ozone performance goal. PM 2.5 is predicted well at the Charlotte site with low bias (-8%) and error (29%), but underestimated at the Ra leigh site with a bias of -28%.

The Soccer and Bugle Plots summary of FRM PM 2.5 mass model performan ce in North Carolina are shown in Figures B-28 and B-29. With the exception of the large summer underprediction bias in June and July, the other 10 months PM 2.5 performance achieves the <±30%/50% PM performance goal and four of the months even achieve the more stringent ozone performance with four additional months right at the ozone goal.B-86 Figure B-25a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-87 Figure B-25b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-88 Figure B-25c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-89 Figure B-25d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12
km Base G2 Actual base case simulation.B-90 Figure B-25e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-91 Figure B-25f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-92 Figure B-25g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in North Carolina and the CMAQ 2002 36 and 12
km Base G2 Actual base case simulation.B-93 Figure B-26a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Mecklenburg County, North Carolina Site No. 37-119-0041 (Charlotte) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-94 Figure B-26b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Mecklenburg County, North Carolina Site No. 37-119-0041 (Charlotte) for the CMAQ 12 km 2002 Base G2

Actual base case simulation.B-95 Figure B-26c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Mecklenburg County, North Carolina Site No. 37-119-0041 (Charlotte) for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-96 Figure B-27a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Wake County, North Carolina Site No. 37-183-0014 (Raleigh) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-97 Figure B-27b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Wake County, North Carolina Site No. 37-183-0014 (Raleigh) for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-98 Figure B-27c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Wake County, North Carolina Site No. 37-183-0014 (Raleigh) for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-99 Figure B-28. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in North Carolina for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-100 Figure B-29. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites
in North Carolina and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-101 B.4.7 South Carolina

The CMAQ performance for SO4 across the 4 ST N sites in South Carolina generally has an underprediction bias but is reasonably good al ways achieving the PM performance goal and sometimes even achieving the ozone performan ce goal (Figure B-30a). NO3 performance is poor with an underprediction bias generally from -40% to -180% that is largest in the summer when NO3 is lowest and not an important part of the PM 2.5 (Figure B-30b). NH4 performance is variable frequently achieving the PM performance goal with the worst performance in the summer with a large underpredictio n tendency. OCM is underestimated with bias of -15% to - 62%. EC performance also has an underestima tion tendency, although not is bad as seen for OCM. The CMAQ 12 km results also exhibits mu ch better EC performa nce than the CMAQ 36 km modeling results with th e 12 km EC model performan ce always achieving the PM performance goal and sometimes even achievi ng the ozone performance goal, whereas the CMAQ 36 km EC results never achieves the PM performance goal for the four months studied (Figure B-30e).

PM 2.5 model performance across STN and FRM ne tworks in South Carolina are shown in Figures B-30f and B-30g, respectively. PM 2.5 mass is usually underp redicted across both networks, with the fractional bias across the STN network is worse and approximately 20 percentage points larger underprediction than the FRM network. In fact, except for July with a - 40% bias, the bias/error for PM 2.5 across the FRM network ach ieves the <±15%/35% ozone model performance goal.

An example time series of predicted and obser ved PM concentrations and model performance metrics at the Greenville, South Carolina STN si te is given in Figure B-31. All PM components are underestimated on average with fractional bias values for SO4, NO3, NH4, OCM, EC and

PM 2.5 of, respectively, -23%, -97%, -19%, -46% , -11% and -30%. Although the model does show some skill in reproducing the temporal and seasonal variability in the observed PM

concentrations.

The summary fractional bias and gross e rror Soccer and Bugle Plots for total PM 2.5 mass across the FRM network in South Carolina are shown in Figures B-32 and B-33. During the fall and winter months of Aug, Sep, Nov, Dec, Jan and Fe b the monthly bias and error statis tics achieve the ozone model performance goal. The three a dditional months of Mar, Apr and Oct achieve the PM performance goal. Whereas the summer m onths of May, Jun and Jul fail to achieve the PM performance goal due to large underprediction bi as that is due to the underprediction of each of the PM component species.B-102 Figure B-30a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-103 Figure B-30b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-104 Figure B-30c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-105 Figure B-30d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12
km Base G2 Actual base case simulation.B-106 Figure B-30e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-107 Figure B-30f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-108 Figure B-30g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in South Carolina and the CMAQ 2002 36 and 12
km Base G2 Actual base case simulation.B-109 Figure B-31a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Greenville County, South Carolina Site
No. 45-045-0009 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-110 Figure B-31b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Greenville County,
South Carolina Site No. 45-045-0009 for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-111 Figure B-31c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Greenville County,
South Carolina Site No. 45-045-0009 for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-112 Figure B-32. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in South Carolina for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-113 Figure B-33. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites
in South Carolina and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-114 B.4.8 Tennessee

The CMAQ does a good job predicting the observed SO4 concentrations in Tennessee with bias

and error values that either achieve the most st ringent ozone performance goal or are right at the edge of the ozone goal (Figure B- 34a). The CMAQ 12 km results are slightly better than the 36 km predictions and, unlike the other states, wher e July had the largest underprediction bias, in Tennessee the SO4 performance in July is quite good with the 12 km results producing a 4% bias

and 32% error.

The model exhibits little skill in its NO3 pr edictions (Figure B-34b) with the usual large underprediction bias in the summer (e.g., -170% in July). Although the bias is low in January and October (<±10%), the error is large (60% to 80%). NH4 performance is generally reasonably good with low bias and error in January and April, but an underprediction bias in July (-20% to -30%) and an overprediction bias in October (33% to 36%).

As seen in the other states, OCM exhibits a larg e underprediction bias that generally ranges from -40% in the winter fall and spring to -60% in summer (Figure B-34d). EC performance is quite good in January and October, but has an underpre diction tendency in April and July (Figure B34f).

Tennessee PM 2.5 mass performance across the STN and FR M networks are shown in Figures B34f and B-34g. Across the STN networks performance for PM 2.5 achieves the ozone goal in January and October, but exhibits an underpredicti on bias in April and July such that in July it exceeds the PM performance goal. Across the FRM network the model performs better for

PM 2.5 achieving or nearly achieving the ozone performance goal in January, April and October, but with an underprediction bias in July (-26% to -35%) that is at the PM performance goal.

Example time series comparisons and annual model statistics are presented for two sites in Tennessee, one in Chattanooga, Hamilton County (Figure B-35) and one in Knoxville, Knox County (Figure B-36). The model reproduces the da ily variations in the SO4 at these two sites fairly well with an annual underp rediction bias of -17% and -9%. NO3 is underpredicted in the summer resulting in an annual underprediction bias in Chattanooga and Knoxville of -86% and -74%, respectively. OCM is underpredicted at both sites with bias values of -56% and -44%. EC is underestimated on average in Chattanooga (-23%) and overestimated in Knoxville (21%). The underprediction of the two main PM components at the two sites (SO4 and OCM) results in a net underprediction bias of PM 2.5 mass of -51% and -22%.

The monthly fractional bias and erro r Soccer and Bugle Plots for FRM PM 2.5 performance in Tennessee show that all months of the year ach ieve the PM model performance goal and that 5 of the months even achieve or are right at the ozone model performance goal. The months that do not achieve the ozone performance goal are due to a summer underprediction bias for MayJuly and a fall/winter overprediction bias for October and November and a too high error for

December.B-115 Figure B-34a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-116 Figure B-34b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-117 Figure B-34c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km Base G2

Actual base case simulation.B-118 Figure B-34d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-119 Figure B-34e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-120 Figure B-34f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-121 Figure B-34g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in Tennessee and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-122 Figure B-35a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Hamilton County, Tennessee Site No. 47- 065-4002 (Chattanooga) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-123 Figure B-35b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Hamilton County, Tennessee Site No. 47-065-4002 (Chattanooga) for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-124 Figure B-35c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Hamilton County, Tennessee Site No. 47-065-4002 (Chattanooga) for the CMAQ 12 km 2002 Base G2 Actual

base case simulation.B-125 Figure B-36a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Knox County, Tennessee Site No. 47- 093-1020 (Knoxville) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-126 Figure B-36b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Knox County, Tennessee Site No. 47-093-1020 (Knoxville) for the CMAQ 12 km 2002 Base G2 Actual base

case simulation.B-127 Figure B-36c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Knox County, Tennessee Site No. 47-093-1020 (Knoxville) for the CMAQ 12 km 2002 Base G2 Actual base

case simulation.B-128 Figure B-37. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Tennessee for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-129 Figure B-38. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites
in Tennessee and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-130 B.4.9 Virginia

SO4 performance is fairly good in Virginia with low bias and error in July and October that achieves the <±15%/35% ozone bias/error perf ormance goal (Figure B-39a). Although January and April exhibit a -20% to -30% underpredict ion bias for SO4, it still achieves the PM performance goal. As seen for the other states , NO3 performance is po or and characterized by an underprediction bias that is greatest in the summer (-170%) when CM AQ estimates near zero NO3 predictions. NH4 performance achieves th e ozone performance goal in April and PM performance goal in January and October, but the -48% to -55% bias in July exceeds the PM performance goal (Figure B-39c).

OCM performance is characterized by an underpredic tion bias of from -30% to -60% with error of opposite sign and similar magnitude (Figure B-39d) . The fractional bias for EC is about 20- 30 percentage points higher using CMAQ with a 12 km grid than 36 km grid resulting in the 12 km results always achieving the PM performance goal but the 36 km results underprediction bias in April and October is in the -40% to -50% range so does not achieve the PM performance goal.

The PM 2.5 mass performance across the FRM sites is fairly good achieving the ozone performance goal in January, April and Octobe r and the PM performance goal in all three months (Figure B-39g). Across the STN sites, the PM perfor mance goal is achieved in January, April and October but not in July due to a -36% to -44% underprediction bias.

Figure B-40 displays an example annual time seri es model performance ev aluation for a site in Richmond County, Virginia. SO4 performance at this Richmond County STN site is extremely good as the model follows the day-to-day variati ons quite well resulting in low annual fractional bias (-11%) and gross error (30%). NO3 performa nce, on the other hand, is characterized by a winter overprediction and summer unde rprediction bias with an annual bias and gross error of -46% and 91%, respectively. OCM at the Richmond site has an underprediction bias that occurs throughout the year resulting in an annual fractional bias of -46%. Better performance is seen for EC with bias (24%) and error (42%) th at achieves the PM performance goal. PM 2.5 performance is also fairly good w ith bias (-10%) and er ror (28%) that achieves the most stringent ozone performance goal.

Another example time series model performance anal ysis is given in Figu re B-41 for a site in Roanoke County, Virginia. Similarly good SO4 and poor NO3 performance is seen at the Roanoke County site. OCM is almost systematically underpredicted through the year with a

-50% bias and 51% error. Wher eas better performance is seen for EC that achieves the ozone performance goal. PM 2.5 performance is generally character ized by an underestimation tendency resulting in a -33% fractional bias.

Figures B-42 and B-43 summarize the PM 2.5 model performance across 19 FRM sites in Virginia using, respectively, Soccer and Bu gle Plots. Monthly fractiona l bias and gross error for PM 2.5 mass achieves the PM performance goal for every month in 2002, with six of the months achieving the ozone performance goal and another two just at the ozone goal. As seen in the other states, the three summer months of May, June and July have an underprediction bias that exceeds the ozone performance goal, but still meets the PM performance goal.B-131 Figure B-39a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-132 Figure B-39b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base G2 Actual base

case simulation.B-133 Figure B-39c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base G2 Actual

base case simulation.B-134 Figure B-39d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-135 Figure B-39e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-136 Figure B-39f. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base
G2 Actual base case simulation.B-137 Figure B-39g. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in Virginia and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-138 Figure B-40a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Richmond County, Virginia Site No. 51-
760-0020 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-139 Figure B-40b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Richmond County, Virginia Site No. 51-760-0020 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-140 Figure B-40c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Richmond County, Virginia Site No. 51-760-0020 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-141 Figure B-41a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate (NO3, bottom) concentrations during 2002 at the STN Roanoke County, Virginia Site No. 51-
770-0014 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-142 Figure B-41b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) and elemental carbon (EC, bottom) concentrations during 2002 at the STN Roanoke County, Virginia Site No. 51-770-0014 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-143 Figure B-41c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine particulate mass (PM 2.5 , bottom) concentrations during 2002 at the STN Roanoke County, Virginia Site No. 51-770-0014 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-144 Figure B-42. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in Virginia for the CMAQ 12 km 2002 Base G2 Actual base case simulation.B-145 Figure B-43. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites in Virginia and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-146 B.4.10 West Virginia

There were no speciated PM 2.5 STN monitoring sites within West Virginia during the 2002 modeling year, so the PM 2.5 model evaluation is limited to the 16 FRM PM 2.5 mass monitoring sites. In January, April a nd July, CMAQ underestimates PM 2.5 mass across the FRM sites in West Virginia by approximately -30%, -25% an d -40% and achieves the PM performance goal in January and April, but not July (Figure B-44). In October the model has near zero bias and 32% error so achieves the ozone model performance goa l. The Soccer and Bugle Plots in Figures B45 and B-46 summarizes the FRM PM 2.5 mass performance in West Virginia in each month. Nine of the months achieve the PM performance goal with the winter months even achieving or nearly achieving the ozone model performance goal. As seen for the other states, the underprediction bias during the May-July summer m onths results in the bias falling outside of the PM performance goal, but within the PM performance criteria.B-147 Figure B-44. Scatter plots of predicted and observed fine particulate mass (PM 2.5 ) concentrations for January, April, July and October 2002, FRM sites in West Virginia and the CMAQ 2002 36 and 12 km
Base G2 Actual base case simulation.B-148 Figure B-45. Soccer plot of monthly fractional bias versus fractional gross error for PM 2.5 mass at FRM monitors in West Virginia for the CMAQ 12 km 2002 Base G2 Actual base case

simulation.B-149 Figure B-46. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites in West Virginia and thee CMAQ 12 km 2002 Base G2 Actual base case simulation.B-150 B.5 CONCLUSIONS STATE-BY -STATE MODEL EVALUATION

The state-by-state model performance evaluation for P.M 2.5 mass and component species reveals several model performance tendencies of th e CMAQ 2002 Base G2 actual base case. SO 4 performance is by far constantly the best perf orming PM species frequently achieving the most stringent ozone performance goal. Although the SO 4 performance in some of the most southern states degraded some (e.g., Fl orida and Mississippi), NO 3 performance, on the other hand, was consistently poor with large summer underpredi ction bias. OCM was usually underpredicted, with the underprediction bias mo st severe in the summer (May – July). EC performance is usually fairly good frequently achieving the most stringent ozone model performance goal with the CMAQ 12 km modeling results exhibiting much superior EC performance to the 36 km modeling results. Performance for NH4 and total PM 2.5 mass is also good most of the time. This is not surprising since the SO 4 performance is usually quite good and SO 4 dominates the PM 2.5 mass and drives the NH4 performance in the Southeast U.S.APPENDIX C

Spatial Maps of Quarterly Average PM 2.5 Components and Total Mass Concentrations with Superimposed Observations for the CMAQ 2002 12 km Base G2 Ac tual Base Case SimulationF:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-3 C.1 OVERVIEW OF SPATIAL MODEL PERFORMANCE

The discussion in this appendix provides a gene ral qualitative overview of the 2002 Base G2 Actual base case model performance of the Comm unity Multiscale Air Quality (CMAQ) modeling system Version 4.5 with SOAmods enhancement acr oss the ASIP 12 km dom ain, with emphasis on temporal (seasonal) and spatial patterns within th e domain. The evaluation is carried out for each of the major PM 2.5 components separately (s ulfate, nitrate, ammonium, organic carbon material, elemental carbon, soil PM 2.5 , sea-salt PM 2.5 ), as well as for total PM 2.5 mass. For each species, two sets of figures are provided, one comparing model pr edictions to observations at the STN sites, the other to observations at the IMPROVE network sites (soil and sea-salt are available at the IMPROVE network only). Total PM 2.5 mass estimates are also compared to observations at the FRM network, in addition to the STN and IMPROVE networks. Four figures are provided for each set corresponding to the average concentrations duri ng the four quarters of th e year (Q1: Jan-Mar; Q2: Apr-Jun; Q3: Jul-Sep; Q4: Oct-Dec).

This evaluation allows for a general understandi ng of model performance and key issues. A more detailed and quantitative performance evaluation at key VISTAS states sites is provided in Appendix B and summarized in Chapter 3.

The observed quarterly average PM 2.5 mass and PM 2.5 species concentrati ons were obtained by averaging all 24-hour measurements at a site that occurred during each three month quarter. The modeled quarterly average spatial maps we re obtained by averagi ng the daily average concentrations for each day in the three month quarter and each grid cell. No attempt was made to match the modeled daily average concentrations with the 1:3 da y sampling frequency used by the monitoring networks. In fact, obtaining modeled quarterly average concentrations trying to match the measurement days is problematic since not all sites collect valid samples on every 1:3 day sampling day. In the case of the STN network, this results in a bias in the Q1 quarterly average predicted and observed comparisons since many of the STN sites started up in 2002 and have samples near the end of Q1 but not during the be ginning. For species like SO4 that has a strong seasonal trend, basing observed Q1 averages on samples in March and modeled values averaged across January-February-March introduces a seemingl y underprediction bias that is artificial and an artifact of the network sampling periods. We know this is occurring at some STN sites, for example see discussion for Georgia in Section 3.5.3.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-4 C.2 SPATIAL MODEL PERFORMANCE EVALUATION

C.2.1 Sulfate (SO4)

Modeled sulfate concentrations show a very str ong seasonal pattern (Figures C-1a,b), with peak concentrations occurring duri ng summer months (Quarters 2 and 3), when photochemistry is highest. A spatial pattern is also evident, with higher concentrations in the northeast, Ohio River valley, and southeast, compared to the upper Midw est and Florida, caused mainly by the higher and

denser SO 2 emissions in those regions compared to the latter two. In Florida, the impacts of individual major sources of SO 2 are evident (such as in the Tampa and Jacksonville areas), as their emissions remain relatively unmixed with emi ssions from other regi ons, being surrounded by ocean. However, in most of the dom ain, a “regional” sulfate field is observed as a result of mixing of emissions from various regions, especially for the long averaging time presented here (three months). Overall, the model seems to accura tely simulate sulfate levels over the domain, and captures both the temporal and spatial patterns exhibited in the observations, both from the STN (Figure C-1a) and IMPROVE (Figure C-1b) networks. The excepti on is Quarter 1 for the STN network, in which

the observed concentrations seem higher than the modeled ones, a nd there is not much agreement in the spatial pattern between the two. This is mainly due to an artifact in the way the observations are presented here discussed above. Since some of the STN sites were not in operation during the first few months of 2002, observed Quarter 1 averag es might in fact be driven by observations during the latter (and warmer) part of Quarter 1 (e.g., March samples), and therefore are biased high.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-5 Figure C-1a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) sulfate (SO4) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-6 Figure C-1b. 2002 Quarterly averages of CMAQ si mulated and IMPROVE observed (diamonds) sulfate (SO4) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-7 C.2.2 Nitrate (NO3)

Nitrate is in many ways the “mirror” image of su lfate, with peak concen trations occurring during wintertime, when the cooler temperatures are favor able for particulate nitrate formation and sulfate concentrations are lower making more ammonia available to bond wi th nitrate. The need for nitrate to be neutralized by a basic compound in order to be in the particulate state results in peak particulate nitrate concentrations occurring in areas of the doma in where ammonia emissions are highest, such as the Midwest.

While overall the model does capture the seasonal a nd spatial variability in particulate nitrate concentrations (Figures C-2a,b), it does seem to overestimate concentrati ons during wintertime, especially over urban ce nters, such as in part of the Nort heast and over Atlant a and Birmingham.

Winter overestimations are also evident over much of North Carolina. While it is impossible to

infer from these data alone on the cause for these ove rpredictions, they are related to either one or a combination of the model nitrate partitioning between gaseous and particulate phase, and the availability of gaseous ammonia, as reflected by the emissions inventory. The latter can be the common denominator between the overpredictions over the urban centers a nd over North Carolina, as ammonia emissions in these areas, though origin ating from different sour ces (mobile sources in the urban centers; agricultural ammonia emissions in North-Carolina), might be overestimated, making more ammonia available for particulat e formation compared to the observations.

In the summer, the model estimates near zero (< 0.25 μ g/m 3 ) particulate nitrate concentrations, whereas the observed values are typically in the 0.25-0.75 μ g/m 3 range. In any event, both the modeled and observed summer NO3 values suggest th at it is not an impor tant component of the

total PM 2.5 mass concentrations in the sout heastern U.S. during the summer.

Fairly similar trends are observed when compari ng the model to either th e STN (Figure C-2a) or the IMPROVE (Figure C-2b) networks, however a mo re detailed analysis is provided based on the STN, given its size and density.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-8 Figure C-2a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) nitrate (NO3) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-9 Figure C-2b. 2002 Quarterly averages of CMAQ si mulated and IMPROVE observed (diamonds) nitrate (NO3) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-10 C.2.3 Ammonium (NH4)

Ammonium concentrations depend, to a large degree, on the availa bility of sulfate and nitrate. Therefore, ammonium levels do not exhibit as st rong seasonal pattern, due to the seasonal tradeoff between sulfate and nitrate, to which it bonds.

Model performance, when compared to the STN da ta (Figure C-3a), seems to be better during summertime (Quarters 2 and 3), when the ammonium is mainly associated with sulfate. During wintertime (Quarters 1 and 4), most of the issues observed with ni trate overpredictions are also evident in the ammonium plots, especially ov er North-Carolina and the urban centers of the

Southeast.

Ammonium is not directly measured by IMPROVE, rather it is derived by assuming it completely

neutralizes the measured sulfate and nitrate. A ssuming that nitrate is completely neutralized by ammonium is most of the time a valid assumption, however the same may not be true for sulfate, especially in the summer months. Hence the observed ammonium comparison in the IMPROVE case (Figure C-3b) is likely oversta ted during the summer months when sulfate is less likely to be fully neutralized and the seemingly underpredicted observed ammonium concentrations across the Appalachian Mountain IMPROVE mon itoring sites in Quarter 3 in Fi gure C-3b are in part due to using derived observed ammonium.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-11 Figure C-3a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) ammonium (NH4) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-12 Figure C-3b. 2002 Quarterly averages of CMAQ si mulated and IMPROVE derived (diamonds) ammonium (NH4) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-13 C.2.4 Organic Carbon Material (OCM)

Evaluating model performance for OCM is complicat ed due to its dual pr imary/secondary nature and anthropogenic/biogenic precursor sources. As a result, the eval uation is dependant on a combination of very different emissions and formati on processes. In addition, there is a fair amount of uncertainty associated with OC measurements , with a major limitation being the inability to directly measure OCM. Instead, only OC is meas ured, and OCM is estimated (and compared to model results) by multiplying OC by an OCM/OC ratio factor. Such OCM/OC ratio factors typically range from 1.2 to 2.2 depending on whether the OCM is fr esh or aged, where we have

used an OCM/OC factor of 1.4. In addition, different analysis methods are used in the STN (NIOSH) and IMPROVE networks that in troduce additional meas urement artifacts.

The seasonal pattern of OCM concentrations w ould depend on the relative contributions from secondary OCM (peak concentrations during su mmertime) and primary OCM. The seasonal pattern of primary OCM would depend on the seasona lity of activity and emissions from sources such as biomass burning and transportation. Given a flat activity/emissions profile, primary OCM is would typically be higher in winter time, due to reduced atmospheric mixing.

Comparing model OCM predictions to STN observati ons (Figure C-4a), both exhibit a fairly flat seasonal pattern. Overall, the modeled concentra tions are lower compared to observations. The

model seems to exhibit a much st ronger spatial pattern, with peak concentrations occurring in the Southeast. This is likely associated with bi omass burning in the winter months, and biogenic secondary OCM formation in th e summer months. The increase d modeled-concentrations of secondary OCM are evident in Quarter 3, especi ally over the Northeast. When comparing

simulated concentrations to IMPROVE observations (Figure C-4b), better agreement is observed.

Given that most IMPROVE sites are located in rura l areas, this may be an indication that OCM is underpredicted mainly at urban sites (STN, Figure C- 4a), possibly due to local spatial gradients and the comparison of diffused 12 km grid-cell volume average concentrations to point measurements at urban sites. It is also pa rtly due to measurement artifacts where the STN OC observations are not “blank corrected” so are biased high.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-14 Figure C-4a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) organic carbon material (OCM) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-15 Figure C-4b. 2002 Quarterly averages of CMAQ si mulated and IMPROVE observed (diamonds) organic carbon material (OCM) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-16 C.2.5 Elemental Carbon (EC)

Evaluating model performance fo r EC, the model tends to underp redict concentrations when compared to the STN network (Figure C-5a), while better agreement is observed with the IMPROVE network (Figure C-5b). This follows th e same conclusion as for OCM and may be reflecting the differences between grid-cel l volume average concentrations and point measurements. These differences are expected to be more pronounced at urban sites (STN) compared to rural ones (IMPROVE), as at urban lo cations there are likely stronger local spatial gradients in EC concentrations. The STN and IMPROVE networks also use different methods for measuring EC that may also be contributing to th e differences in EC model performance using the two networks. In Chapter 3 and Appendix B we noted that the CMAQ 12 km modeling results exhibited superior EC model pe rformance to the CMAQ 36 km m odeling results across the urban

STN network so the dilution of the urban EC emissi ons across the coarse 12 km grid cell is surely contributing to the EC underprediction tendency.

Both the modeled values and observations (at bot h networks) exhibit highe r concentrations in wintertime (Quarters 1 and 4) that is likely due to reduced atmospheric mixing compared to summertime. However, the seasonal pattern is much stronger in the model results compared to the

observations.

Another interesting finding has to do with relati vely high modeled EC concentrations in the western part of the Georgia-Florida border, esp ecially in Quarter 1. This is due to modeled emissions from biomass burning ac tivities. However, nearby meas urements at both the STN and

IMPROVE networks do not exhibit the same trend.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-17 Figure C-5a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) elemental carbon (EC) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-18 Figure C-5b. 2002 Quarterly averages of CMAQ si mulated and IMPROVE observed (diamonds) elemental carbon (EC) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-19 C.2.6 Soil PM 2.5 (SOIL)

Soil PM 2.5 is measured/calculated/derived at the IMPR OVE sites only (Figure C-6). In general,

soil PM 2.5 is overpredicted compared to the observati ons. This is a common issue in grid-based models, where resuspended fugitive dust is assumed to be mixed uniformly in the first layer of the model, whereas in practice most of it is removed locally by impaction onto surfaces such as cars, buildings and vegetation. Higher co ncentrations are especially eviden t in urban centers, due to the way the model calculates the emission and dispersion of resuspended road dust. However, this is not very evident in this comparison, since most IMPROVE sites are located in rural areas. The ASIP/VISTAS modeling did apply fugitive dust trans port factors (FDTF) to fugitive dust emission source categories to account for dust that is deposited locally and not transported.

Another issues associated with the Soil evaluation is the mismatch between how Soil is defined in

the measured versus modeled values. The measur ed Soil values are derived by building it up from the elements, whereas the modeled values are obta ined from the emission speciation profiles that assign all PM 2.5 that is not explicitly speciated as SO4, NO3, EC or OCM to the other PM 2.5 category that is interpreted as Soil in the model performance evaluation.

Also evident in the figures are the higher modele d concentrations in th e western side of the domain. This could be originating for various r easons, such as possibly higher emissions in that region, transport into the domain from the west, or differences in methodologies used to calculate

crustal emissions between the va rious Regional Planning Organizati ons (RPOs). The latter is likely the case, since a sharp spatial gradient is obser ved between the Midwest RPO states (and west of Mississippi, in the porti ons of the CENRAP states) and th e MANE-VU (northeast) and VISTAS (southeast) states (see Figure 1-1 for definition of the RPO states).

Regardless of actual model perf ormance, in the context of PM 2.5 State Implementation Plan (SIP) development and future attainment tests, modeled le vels of soil components are of relatively little importance, since they are normalized according to observations, and little to no controls are being applied to fugitive dust sources (so the Relative Reduction Factor would be equal or close to unity).F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-20 Figure C-6. 2002 Quarterly averages of CMAQ simu lated and IMPROVE observed (diamonds)

soil PM 2.5 (SOIL) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-21 C.2.7 Sea-Salt PM 2.5 (SeaSalt)

Sea-salt PM 2.5 is measured/calculated at some of the IMPROVE sites only (Figure C-7). As expected, higher concentrations are measured at s ites along the coastline. This spatial pattern is captured by the model as well; however, concen trations along the coastline are underpredicted compared to the measurements.

Since sea-salt PM 2.5 is not used for future attainment tests as part of the SIP development process, model performance in this case is of relatively little importance.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-22 Figure C-7. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds)

sea-salt PM 2.5 (SeaSalt) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-23 C.2.8 Total PM 2.5 Mass (PM 2.5 )

A comparison of the modeled spatial dist ribution of quarterly average total PM 2.5 mass concentrations with observations from the FRM, STN and IMPROVE network are provided in Figure C-8. The conclusions on th e model performance for total PM 2.5 mass vary by which network is examined. For Q1, there are elevated PM 2.5 concentrations in the major urban areas (e.g., Chicago-Gary, St. Louis, Atlanta, Northeas t Corridor, etc.) and th e western part of the Georgia-Florida border, the latter is due to biomass burning. The mainly rural IMPROVE monitors fail to capture any of these elevated areas, but does confirm the relative clean conditions in the Appalachian Mountains (Figure C-8c, top le ft). The IMPROVE St. Marks measured PM 2.5 levels appears to refute the elevated PM 2.5 in Q1 along the western Florida- Georgia border. However, the STN (Figure C-8b) and FRM (F igure C-8a) spatial plots confirm that elevated PM 2.5 levels occur in this area during Q1; the lack of elevated PM 2.5 at St. Marks is likely due to its coastal location.

During Q2 the model predicts elevated total PM 2.5 mass levels from St. Louis across the Midwest into Ohio with highest values centered on Indian apolis, in southeastern Pennsylvania and up the Northeast Corridor, and in the Birmingham and Atlanta urban areas in the southeast. The IMPROVE monitors suggest that the model is capturi ng the rural aspect of th e spatial distribution

of the total PM 2.5 mass patterns, albeit with an underpredi ction bias (Figure C-8c, top right). However, the FRM (Figures 3-26 and C-8a) and ST N (Figure C-8b) plots indicates that the model is underestimating the spatial cove rage of the elevated total PM 2.5 mass levels.

The highest seasonal PM 2.5 levels occur in the Q3 summer pe riod when SO4 is the highest. The

observations indicate that the entire interior portion of the ASIP/VISTA S 12 km grid domain is covered by elevated PM 2.5 levels, which is reproduced reasonab ly well by the model. However, the model is estimating slight lower values and relatively cleaner area s over the Appalachian Mountains that are not suppor ted by the observations. In addition, the observed PM 2.5 concentration gradient from high to low occurs further south than pred icted by the model. However, in general the model is doing a good job in reproducing the spatial distribution of PM 2.5 in Q3.

In Q3 the model estimates elevated PM le vels across the upper Midwest (MO-IL-OH) and southwest (Northern AL and GA, SC and NC) th at is split by the Appa lachian Mountains and elevated levels in the Northeast Corridor. The IMPROVE network plots confirm the relative cleaner area in the Appalachian Mountains (Figur e C-8c) and the STN network plot confirms the three areas of high PM 2.5 (Figure C-8b).

Model performance evaluation for total PM 2.5 mass depends heavily on the performance for the individual components. As such, it is less mean ingful than the performance evaluation for the individual components. In addition, modeled total PM 2.5 is not used for future attainment tests as part of the SIP development process that uses th e Speciated Modeled Attain ment Test (SMAT) that

is based on the modeled PM 2.5 component species. The FRM network spatial map (Figure 3-26 and C-8a) confirms the three areas of elevated PM 2.5 concentrations, but sugge sts the southeast area is not as high as the others and that the PM 2.5 distribution should be spottier.

In conclusion, the model appears to do a good job in reproducing the spatial and temporal

variations in PM 2.5 concentrations across the ASIP/VIS TAS 12 km grid, albeit with an underprediction bias. The spatia l distribution of the modeled PM 2.5 concentrations is smoother and less spotty than the observed distri butions, which is due in part to the coarse 12 km grid spacing used in the modeling.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-24 Figure C-8a. 2002 Quarterly averages of CMAQ simulated and FRM observed (diamonds) total

PM 2.5 mass concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-25 Figure C-8b. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) total

PM 2.5 (PM25) concentrations over the VISTAS 12 km domain.F:\PM25 SIP Docs\Model Performance Ev aluation\Appendix_C_MPE-Spatial1.doc C-26 Figure C-8c. 2002 Quarterly averages of CMAQ si mulated and IMPROVE observed (diamonds)

total PM 2.5 (PM25) concentrations over the VISTAS 12 km domain.

## Scatter Plots for Knoxville-Area Monitors