Attachment F 

Model Performance Evaluation 
Attachment F Contents 

ASIP Model Performance Evaluation – Chapter 3 of ASIP Technical Support Document 

Appendix B of ASIP Technical Support Document – State-by-State and Key Monitoring 

Site PM2.5 Model Evaluation 

Appendix C of ASIP Technical Support Document – Spatial Maps of Quarterly Average 

PM2.5 Components and Total Mass Concentrations with Superimposed Observations for 

the CMAQ 2002 12 km Base G2 Actual Base Case Simulation 

Scatter Plots for Knoxville-Area Monitors 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-1 

3.0 MODEL PERFORMANCE EVALUATION 

3.1 INTRODUCTION 

This Chapter summarizes the CMAQ 2002 36 km and 12 km Base G2 Actual base case 

simulation model performance evaluation. The focus of the model performance evaluation is on 

PM2.5 and its component species in the urban and PM2.5 nonattainment areas (NAAs). However, 

the evaluation of the CMAQ 2002 Base G2 base case simulation was also performed for PM 

species and visibility primarily at Class I areas as part of the VISTAS regional haze assessment. 

The VISTAS model evaluation focused on the performance of PM component species and 

visibility at more rural (e.g., IMPROVE) monitors. ASIP also performed an urban ozone 

evaluation performance as part of the development of the southeastern states 8-hour ozone SIPs 

that were due in June 2007. The VISTAS regional PM components and visibility and ASIP 

urban ozone model performance evaluations are directly relevant to the ASIP PM2.5 evaluation as 

they share many of the same precursor and product species and atmospheric processes that 

produce elevated PM2.5 levels in the urban and NAAs. Appendix B provides details on the ASIP 

urban and NAAs PM2.5 mass and PM2.5 component species model performance evaluation that 

are summarized for each ASIP state in Section 3.5 below. Appendix C provides a qualitative 

evaluation of the CMAQ base case for PM2.5 components and total mass that is summarized in 

Section 3.6 However, before summarizing the urban and NAAs PM2.5 model performance 

results, it is useful and informative to summarize the results of the VISTAS regional model 

performance evaluation (Morris et al., 2007) that is provided in Section 3.4. 

3.2 CMAQ 2002 BASE G2 MODEL EVALUATION METHODOLOGY 

VISTAS and ASIP share the same 2002 Base G2 Actual base case modeling results from the 

Community Multiscale Air Quality (CMAQ) modeling system Version 4.51 (Byun and Ching, 

1999) with SOAmods enhancement (Morris et al., 2006). The VISTAS regional model 

performance evaluation focused on monitoring sites within the VISTAS states and VISTAS 12 

km grid (Morris et al., 2007). The VISTAS and ASIP CMAQ evaluation of the 2002 Base G2 

Actual base case is the product of numerous evaluations of predecessor CMAQ base case 

simulations using interim versions of the modeling inputs and CMAQ model (e.g., Morris et al., 

2004b,c; 2005; 2006). With the exception of Organic Carbon Mass (OCM), the basic features of 

the model performance evaluation have essentially remained unchanged from the initial 

evaluations in Phase I and II of VISTAS (e.g., Morris et al., 2004a,b,c) to the current 2002 Base 

G2 actual base case model performance discussed below. During the course of the VISTAS 

Phase II and ASIP modeling, VISTAS updated the secondary organic aerosol (SOA) module in 

CMAQ to include important SOA processes missing in the standard CMAQ model that 

significantly improved the OCM performance in the summer (Morris et al., 2006). The 

processes missing in the current standard versions of CMAQ (e.g., Versions 4.5 and 4.6) that 

were included as part of the SOAmods enhancement are described in Section 1.3.3.3.1 and 

included SOA from sesquiterpenes and isoprene and the polymerization of SOA so that it is no 

longer volatile. EPA intends to address this issue in the fall 2008 release of CMAQ, which is too 

late for VISTAS December 2007 regional haze or the ASIP April 2008 PM2.5 SIPs. Thus, both 

VISTAS and ASIP have adopted CMAQ version 4.51 with SOAmods updates (Morris et al., 

2006) as the core model for the modeling analysis. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-2 

In the VISTAS modeling performance evaluation, the CMAQ results were compared with 

observational data from the Interagency Monitoring of PROtected Visual Environments 

(IMPROVE), Speciated Trends Network (STN), Clean Air Status Trends Network (CASTNet), 

Federal Reference Method (FRM) PM2.5 mass, National Acid Deposition Program (NADP) and 

South East Aerosol Research and CHaracterization (SEARCH) study monitoring networks. The 

VISTAS evaluation focuses primarily on the operational model evaluation of the air quality 

model’s performance with respect to individual components of fine particulate matter (PM2.5), as 

good model performance of the PM component species will dictate good model performance of 

total or reconstituted fine particulate matter from which visibility is based. The FRM network 

only collects total PM2.5 mass and so is not as relevant for judging how well the model is 

predicting the components of light extinction so was not stressed in the VISTAS evaluation, 

however it is an important component of the ASIP PM2.5 evaluation. VISTAS also performed a 

diagnostic evaluation analyzing the ability of the model to reproduce gaseous PM precursor (e.g., 

SO2 and NOx) and product (e.g., HNO3) species. 

The ASIP model performance evaluation focused on ozone and PM2.5 performance in urban and 

nonattainment areas (NAAs). The ASIP ozone model performance evaluation can be found at: 

http://pah.cert.ucr.edu/vistas/vistas2/evaluation_results/2002ga2a.mpe/

The ASIP PM2.5 evaluation is summarized in Sections 3.5 and 3.6 below with more details 

provided in Appendices B and C. In addition, the full display of ASIP and VISTAS model 

evaluation products is available on the project website at: 

http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml

3.2.1 Evaluation Approach 

EPA’s integrated ozone, PM2.5 and regional haze modeling guidance calls for a comprehensive, 

multi-layered approach to model performance testing, consisting of the four major components: 

operational, diagnostic, mechanistic (or scientific) and probabilistic (EPA, 2007a). The CMAQ 

model performance evaluation effort for PM2.5 discussed in this Chapter focused on the first two 

components of the EPA’s recommended evaluation approach, namely: 

• Operational Evaluation: Tests the ability of the model to estimate PM2.5 mass 

concentrations and the components of PM2.5, that is sulfate, nitrate, ammonium, organic 

carbon matter, elemental carbon, and other inorganic PM2.5. This evaluation examines 

whether the measurements are properly represented by the model predictions but does not 

necessarily ensure that the model is getting “the right answer for the right reason”; and 

• Diagnostic Evaluation: Tests the ability of the model to predict visibility and extinction, 

PM chemical composition including PM precursors (e.g., SOx, NOx, and NH3) and 

associated oxidants (e.g., ozone and nitric acid); PM size distribution; temporal variation; 

spatial variation; mass fluxes; and components of light extinction (i.e., scattering and 

absorption). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-3 

The diagnostic evaluation also includes the performance of diagnostic tests to better understand 

model performance and identify potential flaws in the modeling system that can be corrected. 

In this final model performance evaluation for the ASIP/VISTAS 2002 Actual Base G2 CMAQ 

36/12 km base case simulation, the operational evaluation has been given the greatest attention 

since this is the primarily thrust of EPA’s modeling guidance. However, we have also examined 

certain diagnostic features dealing with the model’s ability to simulate sub-regional and 

monthly/diurnal gas phase and aerosol concentration distributions. In the course of the 

VISTAS/ASIP studies numerous diagnostic sensitivity tests were performed to investigate and 

improve model performance and test the model assumptions that are available on the VISTAS 

modeling website: http://pah.cert.ucr.edu/vistas/vistas2/ and reported by Morris and co-workers 

(2004a,b,c; 2005; 2006). 

3.2.2 Particulate Matter and Component Species 

PM2.5 attainment is based on PM2.5 mass measurements using Federal Reference Method (FRM) 

monitoring devices that consists of the following PM2.5 components: 

• Sulfate (SO4) 

• Nitrate (NO3) 

• Ammonium (NH4) 

• Organic Carbon Matter Carbon (OCM) 

• Elemental Carbon (EC) [also called Black Carbon (BC)] ]

• Other Inorganic PM2.5 that is also referred to as Soil (also known as crustal material, fine 

soil, major metal oxides, or other PM2.5) 

• Particle Bound Water (PBW) 

• Sea Salt (that is mostly NaCl) 

• Passive Mass (Blank Correction) 

3.2.3 Ambient Air Quality Data for Model Performance Evaluation 

A ground-level model evaluation database for 2002 was compiled by the modeling team using 

several routine and research-grade databases. The focus of the VISTAS evaluation of the 

CMAQ model was on the PM components that can cause visibility impairment, whereas the 

ASIP model performance evaluation focus were on PM2.5 mass and its components. The primary 

monitoring networks available to evaluate this component of the CMAQ are: (a) Interagency 

Monitoring of Protected Visual Environments (IMPROVE); (b) Clean Air Status and Trends 

Network (CASTNET); (c) Southeastern Aerosol Research and Characterization (SEARCH); (d) 

EPA Federal Reference Method (FRM) PM2.5 and PM10 Mass Networks (EPA-FRM); (e) EPA 

Speciation Trends Network (STN) of PM2.5 species; and (f) National Acid Deposition Network 

(NADP). These PM monitoring networks may also provide ozone and other gas phase 

precursors and product species, and visibility measurements at some sites. There are also gasphase criteria pollutant measures from the EPA’s Air Quality System (AQS) that were used for 

the urban ozone evaluation that is discussed elsewhere 

(http://pah.cert.ucr.edu/vistas/vistas2/evaluation_results/2002ga2a.mpe/). Table 3-1 and Figure 

3-1 summarizes the species collected and locations of the monitoring sites for the IMPROVE, 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-4 

STN, CASTNet, NADP and SEARCH monitoring networks use in the VISTAS and ASIP model 

evaluation. 

Table 3-1. Ambient monitoring data available in the VISTAS/ASIP region during 2002. 

Monitoring 

Network Chemical Species Measured 

Sampling Frequency; 

Duration 

IMPROVE Speciated PM2.5 and PM10 1 in 3 days; 24 hr 

CASTNET Speciated PM2.5, Ozone Hourly, Weekly; 1 hr, Week 

SEARCH 

24-hr PM25 (FRM Mass, OC, BC, SO4, NO3, 

NH4, Elem.); 24-hr PM coarse (SO4, NO3, 

NH4, elements); Hourly PM2.5 (Mass, SO4, 

NO3, NH4, EC, TC); and Hourly gases (O3, 

NO, NO2, NOy, HNO3, SO2, CO) Daily, Hourly; 

NADP WSO4, WNO3, WNH4 Weekly 

EPA-FRM Only total fine mass (PM2.5) 1 in 3 days; 24 hr 

EPA-STN Speciated PM2.5 Varies; Varies 

AIRS/AQS CO, NO, NO2, NOx, O3 Hourly; Hourly 

250 500 750 1000 1250 1500 1750 2000

-1500

-1250

-1000

-750

-500

-250

0

250

500

IMPROVE

CASTNet

SEARCH

STN

NADP

AQS

Figure 3-1. IMPROVE, STN, CASTNet, NADP and SEARCH monitoring sites located in the 

VISTAS/ASIP region and the VISTAS/ASIP 12 km grid. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-5 

3.3 Model Performance Statistics and Goals 

To quantify model performance, several statistical measures were calculated and evaluated for 

all the IMPROVE, STN, CASTNet, SEARCH, FRM and NADP monitors within the 

VISTAS/ASIP region or VISTAS/ASIP 12 km domain, individually for each monitor and 

individually for each VISTAS/ASIP state. The statistical measures selected were based on the 

recommendations outlined in section 18.4 of the USEPA’s Guidance On The Use Of Models 

And Other Analyses for Demonstrating Attainment of Air Quality Goals for Ozone, PM2.5, and 

Regional Haze (EPA, 2007a). 

In 2004, VISTAS established model performance goals and criteria for components of fine 

particle mass based on previous model performance for ozone and fine particles (e.g., Morris et 

al., 2004a,b,c). EPA modeling guidance for fine particulate matter at the time noted that PM 

models might not be able to achieve the same level of performance as ozone models. The 

VISTAS and ASIP model performance evaluation considered several statistical performance 

measures and displays. VISTAS/ASIP reviewed numerous model performance evaluation 

metrics to evaluate their descriptive capabilities for summarizing the salient features of the 

model performance evaluation. Although numerous model performance statistics measures are 

routinely calculated, VISTAS/ASIP have found that the fractional bias and fractional gross error 

provide the best descriptive power over a wide range of concentrations. The fractional bias and 

error are expressed as a percentage and are normalized by the average of the predicted and 

observed values. Consequently, they are bounded statistics, with the fractional bias bounded by -

200% to +200%b and the fraction error bounded by 0 to 200%. Table 3-2a summarized the 

formulas for the fractional bias and error statistics. The VISTAS/ASIP model performance goals 

and criteria are given in Table 3-3. 

Table 3-2. Definitions of the fractional bias and fractional error statistical model performance 

metrics. 

Statistical Measure Shorthand 

Notation 

Mathematical Expression Notes 

Mean Fractional Gross 

Error (Fractional Error) 

MFE 

∑= +

− N

i i i

i i

P O

P O

N 1

2 Reported as % 

Mean Fractionalized Bias 

 (Fractional Bias) 

MFB 

∑= ⎟

⎟

⎠

⎞

⎜

⎜

⎝

⎛

+

− N

i i i

i i

P O

P O

N 1

2 Reported as % 

Table 3-3. VISTAS/ASIP model performance goals and criteria for components of fine particle 

mass. 

Fractional Bias Fractional Error Comment 

<±15% <35% Goal for PM model performance based on ozone model 

performance, considered excellent performance 

<±30% <50% Goal for PM model performance, considered good 

performance 

<±60% <75% Criteria for PM model performance, considered average 

performance. Exceeding this level of performance 

indicates fundamental concerns with the modeling 

system and triggers diagnostic evaluation. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-6 

3.4 Regional Model Performance Evaluation 

Details on the regional model performance evaluation for PM across the VISTAS/ASIP region 

km grid are provided in Chapter 3 and Appendix B of the VISTAS Technical Support Document 

(TSD; Morris et al., 2007) that can be found at: 

http://pah.cert.ucr.edu/vistas/vistas2/docs.shtml 

Below we summarizes the key findings of the of the VISTAS regional model performance 

evaluations of PM2.5 component species across the VISTAS region using monthly fractional bias 

and gross error performance statistics. More details, including scatter plots, quantile-quantile 

plots, Bugle Plots, etc. can be found in the VISTAS TSD referenced above. 

3.4.1 Sulfate (SO4) Model Performance 

Figure 3-2 compares the monthly CMAQ SO4 concentration fractional bias and error across the 

VISTAS/ASIP 12 km grid region for the IMPROVE, STN, CASTNet, SEARCH Daily and 

SEARCH Hourly monitoring networks. The underprediction bias in SO4 concentrations is 

clearly evident across most of the year and monitoring networks with the best (i.e., closest to 

zero) SO4 bias occurring in October. However, this underprediction bias is not severe usually 

almost always within ±20% and always within ±25% and almost always achieves the PM model 

performance goals for bias/error (i.e., within ±30%/50%, see Table 3-2). The exception to this is 

the comparison with the SEARCH Hourly network that exhibits a positive overprediction SO4 

bias for several months. 

The SO4 monthly fractional error values are usually under 40% and, with one exception, always 

under 50% thereby achieving the PM performance goal (Figure 3-2). The exception is 

comparison with the SEARCH Hourly network with error values ranging from 50% to 70%. The 

SEARCH Hourly SO4 measurements are made with research grade monitoring equipment and 

likely have higher uncertainties and artifacts than the other monitoring networks. 

The summer SO4 underprediction bias is partly due to overstated convective precipitation in the 

MM5 simulations (Olerud, 2003c,d). This is somewhat confirmed by the SO4 wet deposition 

model performance evaluation that exhibits near zero bias during the winter when precipitation is 

dominated by synoptic weather events, but has a positive overprediction bias during the summer 

when convective precipitation is greatest (Morris et al., 2007). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-7 

Figure 3-2. Monthly SO4 fractional bias (top) and fractional error (bottom) statistical measures 

for CASTNet, IMPROVE, STN and SEARCH (daily and hourly) monitoring sites in the VISTAS 

12 km grid. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-8 

3.4.2 Nitrate (NO3) Modeling Performance 

The monthly fractional bias time series plot for NO3 clearly shows a seasonal dependence of this 

performance measure with a severe summer underprediction bias of -100% to -150% and bias 

values generally within ±50% in the winter (Figure 3-3). The time series of fractional error 

values also exhibits a seasonal dependence with an 80% error in the winter peaking to a 160% 

error in the summer with a bell-like distribution. The large summer underprediction bias values 

occur when NO3 concentrations are extremely low (< 1.0 μg/m3) and, in fact, when the average 

observed NO3 concentration exceeds 1 μg/m3 the model mostly achieves the model performance 

goal and almost always achieves the model performance criteria for bias as indicated by the 

Bugle Plots in the VISTAS TSD (Morris et al., 2007). The model performance goal and criteria 

are achieved less often for NO3 fractional error, however the goal is always achieved across the 

CASTNet and usually achieved across the IMPROVE monitoring networks. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-9 

Figure 3-3. Monthly NO3 fractional bias (top) and fractional error (bottom) statistical measures 

for CASTNet, IMPROVE, STN and SEARCH daily and hourly monitoring sites in the VISTAS 

region. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-10 

3.4.3 Organic Carbon Mass (OCM) Performance 

The monthly fractional bias and error plots for the 12 km CMAQ base case simulation OCM 

performance given in Figure 3-4 that displays fairly good performance across the IMPROVE 

monitors with bias usually within ±15% and errors between 40% and 50%. However, at the 

urban STN sites the model exhibits a large OCM underprediction bias of -35% to -60%. Part of 

the underprediction bias at the STN sites may be due to measurement uncertainties and artifacts 

and part may also be due to the model over diluting the urban OCM emissions through the coarse 

12 km grid. The underprediction of urban OCM is a common problem in PM modeling and 

likely also points to uncertainties in the OCM and SOA precursor emission inventories. 

One source of uncertainty in the OCM measurements is the fact that OCM is actually derived 

from OC measurements. The amount of additional elements (e.g., oxygen) attached to the OC to 

form OCM varies with the age and level of photochemical processing of the organic particles 

with OCM/OC ratios typically ranging from 1.2 to 2.2 with lower ratios for fresh and higher 

ratios for aged processed OCM. For the OCM model performance evaluation we used a 1.4 

OCM/OC ratio that is consistent with the original IMPROVE equation and based on 

measurements from Los Angeles. The new IMPROVE equation uses a higher 1.8 OCM/OC 

ratio reflecting the fact that OCM that reaches the mostly rural Class I areas will be aged and 

subject to photochemical processing. There really is no one right OCM/OC factor to use in all 

cases. However, in interpreting the model performance it is important to know which OCM/OC 

ratio was used and recognize that selection of another ratio could make a 30% or so difference in 

the OCM measurements. 

The STN OC measurements are also not blank corrected, which is believed to result in an 

approximate 0.5 μg/m3

 positive artifact in the STN OC observations (which would be 0.5-0.9 

μg/m3 positive artifact in OCM depending on which OCM/OC factor is used). The subtraction 

of 0.5-0.9 μg/m3 from the measured OCM value would greatly improve the CMAQ OCM model 

performance across the STN network and bring the CMAQ OCM bias closer to zero. 

In the preliminary evaluation of the CMAQ model using early VISTAS Phase I and II databases, 

the OCM performance was a great cause of concern due to a large summer OCM 

underprediction bias and the importance of OCM to the visibility extinction and PM budgets 

(Morris et al.., 2004b,c). This resulted in VISTAS conducting a focused research study on the 

reasons for the summer OCM underprediction, the identification of processes missing in the 

CMAQ model and the development of the SOAmods enhancement to CMAQ (see Section 

1.3.3.3.1) that greatly improved the CMAQ OCM model performance (Morris et al., 2006). The 

CMAQ Version 4.51 with the SOAmods enhancements was used in the ASIP modeling as well. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-11 

Figure 3-4. Monthly OCM fractional bias (top) and fractional gross error (bottom) statistical 

measures for IMPROVE, STN and SEARCH monitoring sites in the VISTAS region. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-12 

3.4.4 Elemental Carbon (EC) Performance 

The monthly fractional bias and error performance statistics for EC across the VISTAS 12 km 

domain are shown in Figure 3-5. Bias and error are lower in the first and last quarters of the 

year. During the summer and adjacent months EC performance across the IMPROVE networks 

exhibits a large underprediction bias (-40% to -80%) that peaks in June. The EC performance 

across the SEARCH network is somewhat similar to IMPROVE only the underprediction bias in 

the summer is not as great. Model performance compared to the STN network has fairly low EC 

fractional bias during quarters 1, 3 and 4 (achieving the most stringent < ±15% ozone goal), but 

has larger underprediction bias during quarter 2 (-20% to -40%). The comparison to the STN 

network very low EC bias during most of the year suggests that the anthropogenic EC emissions 

inventory may be adequately characterized in urban areas. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-13 

Figure 3-5. Monthly EC fractional bias (top) and fractional gross error (bottom) statistical 

measures for IMPROVE, STN and SEARCH daily and hourly monitoring sites in the VISTAS 

region. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-14 

3.4.5 Other PM2.5 or Soil Performance 

The seasonal dependence of the Soil model performance across the IMPROVE and STN 

monitors in the VISTAS region is clearly evident in the monthly fractional bias time series plot 

in Figure 3-6 that show a large (> 100%) overprediction bias in the winter and near zero 

(IMPROVE) or much lower (SEARCH) bias in the summer. The modeled Soil values tend to 

always be between 0 and 2 μg/m3 year round, however the observed values are much lower in 

the winter (0 to 0.5 μg/m3) and comparable to the modeled values in the summer. These results 

suggest that the poorer winter Soil model performance is likely due to incorrect emission 

temporal adjustment factors. For example, the effect of wetted surfaces that suppresses fugitive 

dust emissions may not be properly characterized in the seasonal adjustments to the emissions 

inventory. 

The Soil model performance is confounded by the incommensurability between the modeled and 

measured Soil species. The IMPROVE observed Soil is built up of measured elements. The 

modeled “Soil”, on the other hand, is fine particulate matter emissions that have not been 

explicitly identified and speciated as SO4, NO3, EC and OC in the PM speciation profiles used 

in the SMOKE emissions modeling. The emissions PM speciation profiles may have 

unidentified PM that is lumped in the other PM category that are not the same as the IMPROVE 

Soil. The measured Soil will typically be associated with fugitive dust emissions, so the separate 

tracking of fugitive dust may be one approach to separate the true Soil component from the other 

PM2.5 component. VISTAS investigated this issue with an inert (no chemistry) CMAQ 

sensitivity run that separately tracked just fugitive dust emissions and found improved Soil 

model performance with lower bias when just other PM2.5 concentrations due to fugitive dust 

emissions were mapped to the Soil species. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-15 

Figure 3-6. Monthly Soil fractional bias (top) and fractional gross error (bottom) statistical 

measures for IMPROVE and SEARCH Daily monitoring sites in the VISTAS region. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-16 

3.4.6 Fine Particulate (PM2.5) Performance 

Figure 3-7 displays the monthly time series of fractional bias and error for PM2.5 mass 

concentrations for the IMPROVE, STN, SEARCH Daily, SEARCH Hourly and FRM networks 

in the VISTAS/ASIP region. The model generally is exhibiting good performance for total 

PM2.5 mass concentrations. For the first three and last three months of the year the model 

performs well for PM2.5 with low fractional bias (<±20%) and error (<40%) except for the 

SEARCH Hourly network. During April through July, the underprediction bias becomes larger 

and is in the -20% to -45% range and the errors are also larger. This is driven in part by the SO4 

underprediction bias discussed previously as SO4 is a major component of PM2.5 in the summer 

and the southeastern U.S. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-17 

Figure 3-7. Monthly total PM2.5 mass fractional bias (top) and fractional gross error (bottom) 

statistical measures in the VISTAS region. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-18 

3.5 PM2.5 MODEL PERFORMANCE BY STATE 

In this section we evaluate the CMAQ base case for PM2.5 at the mainly urban STN and FRM 

monitoring sites. Appendix B presents a detailed model performance evaluation by PM2.5

species component and for total PM2.5 mass by each state in the ASIP/VISTAS region. An 

example of the types of displays of model performance from Appendix B are given below for 

Alabama and SO4 performance and Soccer Plots for PM2.5 mass are also presented. For the 

remainder of the state-by-state scatter plots, time series plots and Bugle Plots of PM2.5 mass and 

components model performance the reader is referred to Appendix B. 

3.5.1 Alabama 

Figure 3-8 is a repeat of Figure B-2a in Appendix B that contains a scatter plot and model 

performance statistics for SO4 at STN sites located in Alabama and the four seasonally 

representative months of January, April, July and October 2002. Similar scatter plots and 

performance statistics for the other PM2.5 component species and total PM2.5 mass are shown in 

Figure B-3b-g in Appendix B. Performance for SO4 is quite good with the points on the scatter 

plots clustered tightly around the 1:1 line of perfect agreement (Figures 3-8 and B-2a) and 

fractional bias and error performance metrics achieving the most stringent <±15% and < 35%, 

respectively, performance goals for ozone (Table 3-2). Performance for NO3 shown in Figure 

B-2b exhibits an underprediction tendency with fractional bias values of approximately -20% in 

January, -110% in April, -175% in July and -10% in October. The large summer NO3 

underprediction bias occurs when both the model and measured NO3 values are extremely low 

with the modeled values near zero and the STN NO3 observed values are < 1 μg/m3. Note that 

under these summer conditions the NO3 will almost completely volatize off the FRM filter so 

the FRM would measure insignificant PM2.5 mass due to NO3. Thus, these large summer NO3 

underpredictions are not a concern as NO3 is not an important component of PM2.5 mass. 

Performance for NH4 is also fairly good achieving the <±30%/50% fractional bias/error goal for 

PM species (Figure B-2c), which is not surprising given its strong link to SO4. With the 

exception of a few outliers, performance for organic carbon mass (OCM or OC) is fairly good in 

the winter, but underestimated by -20% to -30% in the summer and fall (Figure B-2d). The 

reasons for the OCM underprediction includes the fact that the STN OC measurements are not 

blank corrected and have high uncertainties (see SANDWICH discussion in Chapter 4) and there 

are also large uncertainties in the OC emissions and the form of OC emissions as particles or 

semi-volatile organic gases. EC model performance at STN sites in Alabama is also fairly good 

mostly achieving the PM model performance goal, albeit with an underestimation tendency in 

the summer (Figure B-2f). 

The CMAQ performance for PM2.5 mass at 20 FRM monitoring sites in Alabama is shown in 

Figure B-2g. Performance for PM2.5 mass is generally good achieving the most stringent ozone 

performance goal (<±bias/error) in January and October and the PM goal all year round. 

Time series of predicted (CMAQ 12 km results) and observed 24-hour PM2.5 components and 

model performance statistics at the key North Birmingham STN and SEARCH site are shown in 

Figures B-3 and B-4, with the SO4 performance time series at these co-located sites reproduced 

as Figure 3-9. As discussed in Chapter 4, this is the site with the highest projected 2009 PM2.5

Design Value in the ASIP region. SO4 performance is quite good with annual fractional bias 

and errors of 6% and 29% achieving the most stringent ozone performance goal at the STN site 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-19 

and fractional bias and error of 18% and 34% that almost achieves the most stringent ozone 

performance goal at the SEARCH site. NO3 has a large underprediction bias at the North 

Birmingham site that is driven by near zero modeled NO3 concentrations in the summer when 

the observed values are < 1 ug/m3

 (Figures B-3a and B-4a, bottom). However, the NO3 

performance in the winter when observed values are above 1 ug/m3

 is much better. The EC and 

OCM performance at the North Birmingham STN site is characterized by an underprediction 

with fractional bias of approximately -30% and -50%, respectively (Figures B-3b and B-4b). 

This underprediction is due in part to uncertainties in the carbon measurements as well as the 

contributions of nearby industrial sources whose impacts are diluted across the 12 km CMAQ 

grid used. Performance for NH4 and PM2.5 mass at the North Birmingham STN and SEARCH 

sites are also quite good with fractional bias/error values of, respectively, -5%/33% and 

11%/29% at the STN site that achieves the more stringent ozone performance goal and values of, 

respectively, -24%/42% and 17%/31% that achieves the PM performance goal at the SEARCH 

site. The good performance of NH4 and PM2.5 mass is not surprising given that they are closely 

related to SO4 that exhibits good model performance. 

Figures 3-10 reproduces Figure B-5 from Appendix B and displays a Soccer Plot of monthly 

fractional bias versus gross error for PM2.5 mass performance across 20 FRM sites in Alabama 

and compares them with the three levels of bias/error performance goals in Table 3-3. Total 

PM2.5 mass model performance across the FRM sites in Alabama is best in the winter months 

when the most stringent ozone < ±15%/35% bias/error goal is achieved. In the summer months, 

the model tends to underestimate the FRM observed PM2.5 mass by -15% to -25%, but still 

achieves the <±30%/50% PM model performance goal. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-20 

Figure 3-8. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, 

July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation (duplicate of Figure B-2a in Appendix B) 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-21 

Figure 3-9. Time series of predicted and observed 24-hour sulfate (SO4) concentrations during 

2002 at the STN Jefferson County, Alabama Site No. 01-073-0023 (top) and SEARCH BHM 

(bottom) co-located North Birmingham sites for the CMAQ 12 km 2002 Base G2 Actual base 

case simulation.
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-22 

Figure 3-10. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Alabama for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-23 

3.5.2 Florida 

Figure B-7 summarizes the CMAQ PM2.5 species model performance across the three STN sites 

in Florida and January, April, July and October 2002. SO4 is underestimated at the Florida STN 

sites (Figure B-7a) with fractional bias values that are closer to zero in the winter than summer 

(e.g., -7% January versus -34% July for the 12 km CMAQ results) and closer to zero using the 

12 km than 36 km CMAQ modeling results (e.g., in January bias is -7% using 12 km and -22% 

using 36 km modeling results). Even with this underestimation bias, SO4 performance at the 

Florida STN CMAQ sites is fairly good. NO3 performance, on the other hand, exhibits a large 

underprediction bias with fractional bias values that range from -70% in January to -170% in 

July (Figure B-7b). NH4 performance at the Florida STN sites is fairly good always achieving 

the <±30%/50% PM performance goal and sometimes even achieving the more stringent 

<±15%/35% ozone performance goal (Figure B-7c). It is interesting that in January and October 

when both SO4 and NO3 exhibit an underprediction bias, NH4 exhibits a slight overprediction 

bias. Given the linked relationship between these species this is somewhat surprising and may 

be due to artifacts in the measurements, or the modeled SO4 being more fully neutralized by 

NH4 than in the measurements. The OCM and EC are characterized by underprediction 

tendency with the OCM fractional bias ranging from -55% to -95% and the CMAQ 36 km and 

12 km producing nearly identical performance metrics (Figure B-7d). The EC performance is 

better with fractional bias values ranging from -4% to -52% and the 12 km modeling results 

producing significantly better model performance metrics than the 36 km modeling results 

(Figure B-7e). 

The PM2.5 mass scatter plots and performance statistics in Florida are summarized across 3 STN 

sites and 28 FRM sites in Figures B-7f and B-7g, respectively. The Soccer Plot of monthly 

PM2.5 bias and error performance for FRM sites in Florida is shown in Figure B-9, which is 

repeated as Figure 3-11 below. For the winter months of Nov, Dec, Jan and Feb, the PM2.5

performance statistics are quite good with bias/error values meeting the <±15%/35% ozone 

performance goal. The spring and fall months either achieve the <±30%/50% or <±50%/125% 

performance goal and criteria. The May-Jul summer months are at or exceed the -50% bias 

performance criteria levels (note that in the Soccer Plots if the symbol would be plotted outside 

of the range of the plot then it is plotted on the axis, such as the May and July -50% values in 

Figures 3-11 and B-9). Thus, the PM2.5 performance in Florida is significantly worse than the 

other ASIP states and is a cause for concern. Fortunately, Florida is one of the ASIP states 

without any violating PM2.5 monitors (See Chapter 4). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-24 

Figure 3-11. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Florida for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-25 

3.5.3 Georgia 

There appears to be some valid data capture issues in the 2002 STN database for Georgia with 

only one valid site in January, 6 in April and 8 in July and October. This is due to some of the 

STN sites starting up in 2002. The CMAQ SO4 performance in Georgia is generally pretty good 

with bias/error frequently achieving the <±15%/35% ozone performance goal and always 

achieving the PM goal (Figure B-11a). The STN sites also have the summer nitrate 

underprediction performance issue as seen in the other states (Figure B-11b). The NH4 model 

performance falls between the SO4 and NO3 performance and, with the exception of January, 

which is based on one site so the model performance statistics are not very robust, it achieves the 

<±30%/50% PM performance goal (Figure B-11c). The carbon performance in Georgia appears 

to be slightly better than the other states, although still with a general underestimation tendency. 

With the exception of July, when the CMAQ 36 km OCM performance exhibits fractional bias 

and error values of -34% and 42% thereby achieving the <<±30%/50% PM performance goal, 

the other three months achieve the more stringent <±15%/35% ozone performance goal (Figure 

B-11d). EC performance is variable with generally low bias but lots of scatter producing higher 

gross errors (Figure B-11e). 

The performance of total PM2.5 mass across the STN (Figure B-11f) and FRM (Figure B-11g) 

monitoring sites in Georgia is generally fairly good. July exhibits an underprediction tendency 

with fractional bias values of approximately -40% (STN) and -25% (FRM) that is likely caused 

by the underestimation of SO4 and especially OCM. January is characterized by an 

overestimation tendency. But in general the PM2.5 performance statistics mostly achieve the 

more stringent ozone performance goal. 

An example time series of predicted and observed PM concentrations and annual statistics for 

two sites in the Atlanta Georgia (a DeKalb County STN and the SEARCH Jefferson Street sites) 

are shown in Figures B-12 and B-13. The model tracks the observed SO4 concentrations at these 

sites extremely well producing low fractional bias (-5% and +9%) and gross error (27% and 

30%). NO3 performance exhibits the underprediction tendency seen in the other states, but is 

not as severe with a fractional bias of -16% and ~80% error. The CMAQ 12 km results 

reproduces the OCM concentrations at the DeKalb County site surprisingly well with low bias (-

14%) and error (31%), with a slight degradation at the SEARCH JST site (-35% bias and 46% 

error). EC is overestimated (bias of 28%) at the DeKalb site but has zero error at the SEARCH 

site. Finally, the performance for PM2.5 mass at the two Atlanta sites is extremely good with 

very low bias (2% and -3%) and error (25% and 24%). 

Figure 3-12 below is reproduced from Figure B-14 in Appendix B and displays a Soccer Plot of 

monthly fractional bias and error across the 26 FRM sites in Georgia. All months achieve the 

PM performance goal and 8 of the 12 months even achieve the more stringent ozone 

performance goal. The months that the PM2.5 mass performance doesn’t achieve the ozone 

performance goal are due to an overestimation bias in January and underestimation bias in the 

summer months of May-July that is drive primarily by an underestimation of SO4 and OCM. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-26 

Figure 3-12. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Georgia for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 

3.5.4 Kentucky 

The monthly scatter plots and performance statistics for PM species and sites in Kentucky are 

shown in Figure B-16 in Appendix B. The performance for SO4 in January is mixed, with a 

group of points clustered around the 1:1 line and another further to the right that has been 

determined to be variations of SO4 performance in Kentucky on different STN sampling days in 

January. As seen in the other states, NO3 performance in Kentucky is characterized by 

underprediction bias that is largest in the summer (Figure B-16b) with fractional bias is -30% to -

40% in January, -65% in April, -150% to -160% in July and near zero bias in October due to 

NO3 underpredictions on some days being compensated by overprediction on others. Like SO4, 

NH4 performance is fairly good albeit with an underprediction bias that is largest in the summer 

(Figure B-16c). As seen for the other sites, OCM is underpredicted with the bias ranging from -

50% in January to -70% to -80% in the other three months (Figure B-16d). The OCM errors are 

the opposite sign but the same magnitude as the bias suggesting that the OCM underprediction 

bias is almost systematic. The CMAQ 36 km and 12 km modeling results exhibits different EC 

performance in Kentucky. The CMAQ 12 km results have bias that is much closer to zero than 

the 36 km results (Figure B-16e). This is likely due to the 36 km grid cells over dispersing the 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-27 

urban EC emissions. Both the CMAQ 36 km and 12 km results exhibit a lot of variability 

producing a lot of scatter and high error metrics. 

The performance of total PM2.5 mass concentrations in Kentucky is evaluated across 10 STN 

(Figure B-16f) and 23 FRM sites (Figure B-16g). Given the underprediction of most of the PM 

component species it is not surprising that total PM2.5 mass is underpredicted across the STN 

sites by -16%, -42% to -55%, -33% and -6% to -13% for January, April, July and October, 

respectively. Across the FRM sites in Kentucky, however, the CMAQ model for PM2.5 mass bias 

that is much closer to zero with values of -8%, -11% to -14%, -32% to -35% and +18% to -19% 

for the same four months. The FRM PM2.5 mass performance always achieves the PM 

performance goal and sometimes even achieves the ozone model performance goal. 

An example PM model performance for STN sites in Lexington (Fayette County) and Kenton 

County (Cincinnati NAA) are shown in Figures B-17 and B-18. The model performance at these 

two sites is similar. At the Lexington site the model reproduces the temporal variations in SO4 

quite well (Figure B-17a, top) at this site resulting in low bias (-10%) and error (33%). Although 

the model shows some skill in reproducing the observed higher NO3 values in the winter, 

especially in the first part of the year, the near zero modeled NO3 during the summer results in a 

fractional bias value of -62% with 87% error, so does not achieve the PM performance goal 

(Figure B-17a, bottom). OCM is underpredicted with fractional bias values of -52% and an error 

of 58% just at the PM performance goal (Figure B-17b, top). EC exhibits much better 

performance at the Lexington, Kentucky site with a bias near zero (-4%) and error of 36% so 

almost achieves the ozone performance goal. The fractional bias for NH4 and PM2.5 at 

Lexington are both approximately -10% with NH4 having an error of 36% and PM2.5 an error of 

28% thereby achieving the ozone performance goal. 

Figure 3-13 below reproduces Figure B-19 from Appendix B that summarizes the Kentucky 

PM2.5 mass model performance using a Soccer Plot of monthly bias and error. With the 

exception of July, whose fractional bias falls slightly below -50%, PM2.5 performance in the 

other 11 months achieves the PM bias/error performance goal of <-30%/75%. In fact 5 months, 

which occur mainly in the winter, even achieve the more stringent ozone model performance 

goal. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-28 

Figure 3-13. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Kentucky for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 

3.5.5 Mississippi 

Performance of PM species across sites in Mississippi is shown in Figure B-21. The CMAQ 

performance in Mississippi is generally worse than the other ASIP states (with the exception of 

Florida). SO4 is underpredicted with fractional bias values ranging from -15% to -46% with the 

worse SO4 performance seen in the January/July winter/summer months than the April and 

October transitional months. NO3 performance metrics are extremely poor with fractional bias 

values of -80% to -90% in January and April and -163% in July. In October some NO3 

overpredictions begin to compensate for the frequent near zero model predictions so that the 

underprediction bias is lower (-40%), but the error is still very high (~100%). OCM is 

underpredicted at sites in Mississippi with fractional bias values of approximately -50% in 

January, April and July improving to -24% in October (Figure B-21d). Performance for EC is 

better, albeit still with an underprediction bias, and again as seen for other states, the CMAQ 12 

km modeling results produces much lower EC bias than the 36 km modeling results (Figure B21e). PM2.5 performance in Mississippi across the STN and FRM networks are shown in Figures 

B-21f and B-21g, respectively. Across the STN network, PM2.5 is underestimated by 

approximately -40% rising to -65% in July. Across the FRM network, the PM2.5 underprediction 

bias is lower, at -14% in January and -25% in April and rising to approximately -50% in July. In 

October the model is better able to replicate the observed PM2.5 mass with CMAQ 12 km bias 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-29 

and error values across the FRM network of 2% and 29%, respectively, achieving the ozone 

model performance goals. 

The Soccer Plot (Figures 3-14 and B-23) for PM2.5 mass performance confirm that PM2.5

performance in Mississippi is worse than seen for the other ASIP states, except Florida. Only 

two months achieve the ozone model performance goal and only 8 months achieve the PM 

model performance goal. The model underprediction tendency during the four month period of 

March through July 2002 produces fractional bias for PM2.5 that ranges from -50% to -100% 

thereby not achieving the PM model performance goal. However, unlike Florida, the Mississippi 

monthly bias/error for PM2.5 always achieves the <-50%/75% PM performance criteria. 

Figure 3-14. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Mississippi for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-30 

3.5.6 North Carolina 

SO4 performance across 9 STN monitoring sites in North Carolina and four months from 2002 is 

shown in Figure B-25a. Although SO4 performance achieves the PM performance goal, it varies 

by month with January and July having fractional bias of -25% to -30%, April a fractional bias of 

approximately -20% and October having near zero fractional bias. Although NO3 is still greatly 

underestimated in North Carolina during the summer (July bias of -105% to -142%), during 

January and October NO3 exhibits an overestimation bias with values of 84% and 35%, 

respectively (Figure B-25b). This is likely due to overstated ammonia emissions during the 

winter and adjacent periods which is somewhat verified by the overstated ammonium during 

January and October that have fractional bias values in the +40% to +55% range (Figure B-25c). 

OCM is characterized by an underprediction bias ranging from -30% to -50%, although 

performance in the April and October transitional months achieves the PM performance goal 

(Figure B-25d). EC, on the other hand, achieves the PM performance goal in January and 

October but has lots of scatter in the other two months exceeding the PM performance goal for 

bias in April (-34% to -41%) and for error in July (62%-65%). PM2.5 mass performance across 

the 9 STN and 36 FRM sites in North Carolina exhibits monthly variations (Figures B-25f and 

B-25g). FRM PM2.5 is overpredicted in January (15% to 18%) and October (25% to 30%), but 

underpredicted in April (-9% to -16%) and July (-37% to -39%). The STN PM2.5 performance is 

similar except because the STN measurements tend to be higher so the underprediction bias is 

greater and the overprediction bias is less. 

The Soccer Plot summary of FRM total PM2.5 mass monthly model performance in North 

Carolina are shown in Figure 3-15 (reproduced from Figure B-28 in Appendix B). With the 

exception of the large summer underprediction bias in June and July, the other 10 months PM2.5

performance achieves the <±30%/50% PM performance goal with four of the months even 

achieving the more stringent ozone performance with four additional months right at the ozone 

goal. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-31 

Figure 3-15. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in North Carolina for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 

3.5.7 South Carolina 

The CMAQ performance for SO4 across the 4 STN sites in South Carolina generally has an 

underprediction bias but is reasonably good always achieving the PM performance goal and 

sometimes even achieving the ozone performance goal (Figure B-30a). NO3 performance is 

poor with an underprediction bias generally from -40% to -180% that is largest in the summer 

when NO3 is lowest and not an important part of the PM2.5 (Figure B-30b). NH4 performance is 

variable frequently achieving the PM performance goal with the worst performance in the 

summer with a large underprediction tendency. OCM is underestimated with bias of -15% to -

62%. EC performance also has an underestimation tendency, although not is bad as seen for 

OCM. The CMAQ 12 km results also exhibits much better EC performance than the CMAQ 36 

km modeling results with the 12 km EC model performance always achieving the PM 

performance goal and sometimes even achieving the ozone performance goal, whereas the 

CMAQ 36 km EC results never achieves the PM performance goal for the four months studied 

(Figure B-30e). PM2.5 model performance across STN and FRM networks in South Carolina are 

shown in Figures B-30f and B-30g, respectively. PM2.5 mass is usually underpredicted across 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-32 

both networks, with the fractional bias across the STN network is worse and approximately 20 

percentage points larger underprediction than the FRM network. In fact, except for July with a -

40% bias, the bias/error for PM2.5 across the FRM network achieves the <±15%/35% ozone 

model performance goal. 

The summary fractional bias and gross error Soccer Plot for total PM2.5 mass across the FRM 

network in South Carolina is shown in Figures 3-16 and B-32. During the fall and winter months 

of August, September, November, December, January and February, the monthly bias and error 

statistics for PM2.5 achieve the very stringent ozone model performance goal. The three 

additional months of March, April and October achieve the PM performance goal. Whereas the 

summer months of May, June and July fail to achieve the PM performance goal (but does 

achieve the PM performance criteria) due to large underprediction bias that is caused by the 

underprediction of each of the PM component species. 

Figure 3-16. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in South Carolina for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-33 

3.5.8 Tennessee 

The CMAQ does a good job predicting the observed SO4 concentrations in Tennessee with bias 

and error values that either achieve the most stringent ozone performance goal or are right at the 

edge of the ozone goal (Figure B-34a). The CMAQ 12 km results are slightly better than the 36 

km predictions and, unlike the other states where July had the largest underprediction bias, in 

Tennessee the SO4 performance in July is quite good with the 12 km results producing a 4% bias 

and 32% error. The model exhibits little skill in its NO3 predictions (Figure B-34b) with the 

usual large underprediction bias in the summer (e.g., -170% in July). Although the bias is low in 

January and October (<±10%), the error is large (60% to 80%). NH4 performance is generally 

reasonably good with low bias and error in January and April, but an underprediction bias in July 

(-20% to -30%) and an overprediction bias in October (33% to 36%). 

As seen in the other states, OCM exhibits a large underprediction bias that generally ranges from 

-40% in the winter fall and spring to -60% in summer (Figure B-34d). EC performance is quite 

good in January and October, but has an underprediction tendency in April and July (Figure B34f). Tennessee PM2.5 mass performance across the STN and FRM networks are shown in 

Figures B-34f and B-34g. Across the STN networks performance for PM2.5 achieves the ozone 

goal in January and October, but exhibits an underprediction bias in April and July such that in 

July it exceeds the PM performance goal. Across the FRM network the model performs better 

for PM2.5 achieving or nearly achieving the ozone performance goal in January, April and 

October, but with an underprediction bias in July (-26% to -35%) that is at the PM performance 

goal. 

The monthly fractional bias and error Soccer Plot for FRM PM2.5 performance in Tennessee 

shown in Figure 3-17 and indicates that all months of the year achieve the PM model 

performance goal and that 5 of the months even achieve or are right at the ozone model 

performance goal. The months that do not achieve the ozone performance goal are due to a 

summer underprediction bias for May-July and a fall/winter overprediction bias for October and 

November and a too high error for December. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-34 

Figure 3-17. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Tennessee for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 

3.5.9 Virginia 

SO4 performance is fairly good in Virginia with low bias and error in July and October that 

achieves the <±15%/35% ozone bias/error performance goal (Figure B-39a). Although January 

and April exhibit a -20% to -30% underprediction bias for SO4, it still achieves the PM 

performance goal. As seen for the other states, NO3 performance is poor and characterized by 

an underprediction bias that is greatest in the summer (-170%) when CMAQ estimates near zero 

NO3 predictions. NH4 performance achieves the ozone performance goal in April and PM 

performance goal in January and October, but the -48% to -55% bias in July exceeds the PM 

performance goal (Figure B-39c). 

OCM performance is characterized by an underprediction bias of from -30% to -60% with error 

of opposite sign and similar magnitude (Figure B-39d). The fractional bias for EC is about 20-

30 percentage points higher using CMAQ with a 12 km grid than 36 km grid resulting in the 12 

km results always achieving the PM performance goal but the 36 km results underprediction bias 

in April and October is in the -40% to -50% range so does not achieve the PM performance goal. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-35 

The PM2.5 mass performance across the FRM sites is fairly good achieving the ozone 

performance goal in January, April and October and the PM performance goal in all three 

months (Figure B-39g). Across the STN sites, the PM performance goal is achieved in January, 

April and October but not in July due to a -36% to -44% underprediction bias. 

Figure 3-18 below summarize the PM2.5 model performance across 19 FRM sites in Virginia 

using a Soccer Plot. Monthly fractional bias and gross error for PM2.5 mass in Virginia achieves 

the PM performance goal for every month in 2002, with six of the months achieving the ozone 

performance goal and another two just at the ozone goal. As seen in the other states, the three 

summer months of May, June and July have an underprediction bias that does not meet the ozone 

performance goal, but still meets the PM performance goal. 

Figure 3-18. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Virginia for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 

3.5.10 West Virginia 

There were no speciated PM2.5 STN monitoring sites within West Virginia during the 2002 

modeling year, so the PM2.5 model evaluation is limited to the 16 FRM PM2.5 mass monitoring 

sites. In January, April and July, CMAQ underestimates PM2.5 mass across the FRM sites in 

West Virginia by approximately -30%, -25% and -40% and achieves the PM performance goal in 

January and April, but not July (Figure B-44). In October the model has near zero bias and 32% 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-36 

error so achieves the ozone model performance goal. The Soccer Plot in Figures 3-19 below 

summarizes the FRM PM2.5 mass performance in West Virginia in each month. Nine of the 

months achieve the PM performance goal, with the winter months even achieving or nearly 

achieving the ozone model performance goal. As seen for the other states, the underprediction 

bias during the May-July summer months results in the bias falling outside of the PM 

performance goal, but within the PM performance criteria. 

Figure 3-19. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in West Virginia for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-37 

3.6 SPATIAL AND SEASONAL ANALYSIS OFMODEL PERFORMANCE 

This section summarizes the results in Appendix C that contains a general qualitative overview 

of the CMAQ 2002 Base G2 Actual base case model performance across the ASIP 12 km 

domain, with emphasis on temporal (seasonal) and spatial patterns within the domain. The 

evaluation is carried out for each of the major PM2.5 components separately (sulfate, nitrate, 

ammonium, organic carbon material, elemental carbon, soil PM2.5, sea-salt PM2.5), as well as for 

total PM2.5 mass. The performance evaluation summarized in this section and presented in 

Appendix C present spatial maps of modeled quarterly average PM2.5 concentrations with 

overlaid quarterly average observations. For each PM2.5 species, two sets of figures are 

provided, one comparing model predictions to observations at the STN sites, the other to 

observations at the IMPROVE network sites (soil and sea-salt are available at the IMPROVE 

network only). Modeled spatial maps with overlaid observations for total PM2.5 mass are 

compared for the FRM network, in addition to the STN and IMPROVE networks. Four figures 

are provided for each set corresponding to the average concentrations during the four quarters of 

the year (Q1: Jan-Mar; Q2: Apr-Jun; Q3: Jul-Sep; Q4: Oct-Dec). In Appendix C all of the 

spatial maps comparisons are presented, with the figures for the STN PM2.5 species components 

and the FRM PM2.5 mass also provided below. 

This evaluation allows for a general understanding of model performance and key issues. A more 

detailed and quantitative performance evaluation for monitoring sites within each ASIP states is 

provided in Appendix B and summarized in Section 3.5 above. 

The observed quarterly average PM2.5 mass and PM2.5 species concentrations were obtained by 

averaging all 24-hour measurements at a site that occurred during each three month quarter. The 

modeled quarterly average spatial maps were obtained by averaging the daily average 

concentrations for each day in the three month quarter and each grid cell. No attempt was made 

in the averaging to match the modeled daily average concentrations with the 1:3 day sampling 

frequency used by the monitoring networks. In fact, obtaining modeled quarterly average 

concentrations trying to match the measurement days is problematic since not all sites collect 

valid samples on every 1:3 day sampling day. For some STN sites this resulted in a bias in the 

Q1 quarterly average predicted and observed comparisons since many of the STN sites started up 

in 2002 and are missing data for the first part of 2002 so only have samples near the end of Q1. 

In the case of species like SO4 that has a strong seasonal trend, basing observed Q1 averages on 

samples in March and modeled values averaged across January-February-March introduces a 

seemingly underprediction bias that is artificial and an artifact of the network sampling periods. 

We know this is occurring at some STN sites, for example see discussion for Georgia in Section 

3.5.3. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-38 

3.6.1 Sulfate (SO4) 

Modeled sulfate concentrations show a very strong seasonal pattern (Figure 3-20 below and 

Figures C-1a and C-1b in Appendix B), with peak concentrations occurring during summer 

months (Quarters 2 and 3) when photochemistry is highest. A spatial pattern is also evident, 

with higher concentrations in the northeast, Ohio River valley, and southeast, compared to the 

upper Midwest and Florida, caused mainly by the higher and denser SO2 emissions in those 

regions compared to the latter two. In Florida, the impacts of individual major sources of SO2

are evident (such as in the Tampa and Jacksonville areas), as their emissions remain relatively 

unmixed with emissions from other regions, being surrounded by ocean. However, in most of 

the domain, a “regional” sulfate field is observed as a result of mixing of emissions from various 

regions, especially for the long averaging time presented here (three months). 

Overall, the model seems to accurately simulate sulfate levels over the domain and captures both 

the temporal and spatial patterns exhibited in the observations, both from the STN (Figures C-1a 

and 3-20) and IMPROVE (Figure C-1b) networks. The exception is Quarter 1 for the STN 

network, in which the observed concentrations seem higher than the modeled ones, and there is 

not much agreement in the spatial pattern between the two. This is partly due to an artifact in the 

way the observations are presented here. Since some of the STN sites were not in operation 

during the first few months of 2002, observed Quarter 1 averages might in fact be driven by 

observations during the latter (and warmer) part of Quarter 1 (e.g., March samples), and 

therefore are biased high relative to the modeled Q1 SO4 concentrations. 

 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-39 

Figure 3-20. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

sulfate (SO4) concentrations over the VISTAS 12 km domain (for IMPROVE comparisons see 

Appendix C). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-40 

3.6.2 Nitrate (NO3) 

Nitrate is in many ways the “mirror” image of sulfate, with peak concentrations occurring during 

wintertime, when the cooler temperatures are favorable for particulate nitrate formation, and due 

to lower SO4 as more ammonia is available to bond with nitrate, rather than sulfate (Figure 3-21 

and Figures C-2a and C-2b in Appendix C). Also for this reason, peak particulate nitrate 

concentrations occur in areas of the domain where ammonia emissions are highest, such as the 

Midwest. 

While overall the model does capture the seasonal and spatial variability in particulate nitrate 

concentrations (Figures C-2a,b), it does seem to overestimate concentrations during wintertime, 

especially over urban centers, such as in part of the Northeast and over Atlanta and Birmingham. 

Winter overestimations are also evident over much of North Carolina. While it is impossible to 

infer from these data alone on the cause for these overpredictions, they are related to either one 

or a combination of the model nitrate partitioning between gaseous and particulate phase, and the 

availability of gaseous ammonia, as reflected by the emissions inventory. The latter can be the 

common denominator between the overpredictions over the urban centers and over North 

Carolina as ammonia emissions in these areas, though originating from different sources (mobile 

sources in the urban centers; agricultural ammonia emissions in North-Carolina), might be 

overestimated making more ammonia available for particulate formation compared to the 

observations. 

In the summer the model estimates near zero (< 0.25 μg/m3

) particulate nitrate concentrations, 

whereas the observed values are typically in the 0.25-0.75 μg/m3

 range. So both the modeled 

and observed summer NO3 values suggest that it is not an important component of the total 

PM2.5 mass concentrations in the southeastern U.S. during the summer. 

Fairly similar trends are observed when comparing the model to either the STN (Figures 3-21 

and C-2a) or the IMPROVE (Figure C-2b) networks, however a more detailed analysis is 

provided based on the STN given its larger size and density. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-41 

Figure 3-21. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

nitrate (NO3) concentrations over the VISTAS 12 km domain (IMPROVE comparisons are 

provided in Appendix C). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-42 

3.6.3 Ammonium (NH4) 

Ammonium concentrations depend, to a large degree, on the availability of sulfate and nitrate. 

Therefore, ammonium levels do not exhibit as strong seasonal pattern, due to the seasonal 

tradeoff between sulfate and nitrate to which it bonds (Figures 3-22 and C-3). 

Model performance when compared to the STN data (Figures 3-22 and C-3a), seems to be better 

during summertime (Quarters 2 and 3) when the ammonium is mainly associated with sulfate. 

During wintertime (Quarters 1 and 4), most of the issues observed with nitrate overpredictions 

are also evident in the ammonium plots, especially over North-Carolina and the urban centers of 

the Southeast. 

Ammonium is not directly measured by IMPROVE, rather it is derived by assuming it 

completely neutralizes the measured sulfate and nitrate. Although assuming that nitrate is 

completely neutralized by ammonium is, most of the time, a valid assumption, the same may not 

be true for sulfate, especially in the summer months. Hence the observed ammonium 

comparison in the IMPROVE case (Figure C-3b) is likely overstated during the summer months 

and the seemingly underpredicted observed ammonium concentrations across the Appalachian 

Mountain IMPROVE monitoring sites in Quarter 3 in Figure C-3b are in part due to using 

derived observed ammonium. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-43 

Figure 3-22. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

ammonium (NH4) concentrations over the VISTAS 12 km domain (IMPROVE comparisons 

provided in Appendix C). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-44 

3.6.4 Organic Carbon Material (OCM) 

Evaluating model performance for OCM is complicated due to its dual primary/secondary nature 

and anthropogenic/biogenic sources. As a result, the evaluation is dependant on a combination 

of very different emissions and formation processes. In addition, there is a fair amount of 

uncertainty associated with OC measurements with the major limitation being the inability to 

directly measure OCM. Instead, only OC is measured and OCM is estimated (and compared to 

model results) by multiplying OC by an OCM/OC ratio factor. OCM/OC ratio factors typically 

range from 1.2 to 2.2 depending on whether the OCM is fresh or aged. For this model 

performance evaluation analysis, an OCM/OC factor of 1.4 has been used to convert the 

observed OC to OCM. Another complicating factor is that different analysis methods are used in 

the STN (NIOSH) and IMPROVE networks that introduce additional measurement artifacts and 

uncertainties. 

The seasonal pattern of OCM concentrations would depend on the relative contributions from 

secondary OCM (peak concentrations during summertime) and primary OCM. The seasonal 

pattern of primary OCM would depend on the seasonality of activity and emissions from sources 

such as biomass burning and transportation. Given a flat activity/emissions profile, primary 

OCM is would typically be higher in wintertime, due to reduced atmospheric mixing. 

Comparing the modeled quarterly OCM predictions to STN observations (Figures 3-23 and C4a), both exhibit a fairly flat seasonal pattern. Overall, the modeled OCM concentrations are 

lower compared to the observations. The model seems to exhibit a much stronger spatial pattern, 

with peak concentrations occurring in the Southeastern U.S. This is likely associated with 

biomass burning in the winter months, and biogenic secondary OCM formation in the summer 

months. The increased modeled concentrations of secondary OCM are evident in Quarter 3, 

especially over the Northeast. When comparing simulated concentrations to IMPROVE 

observations (Figure C-4b), better agreement is observed. Given that most IMPROVE sites are 

located in rural areas, this may be an indication that the OCM underprediction at urban sites 

(STN, Figures 3-23 and C-4a) may possibly be due to local spatial gradients and the comparison 

of 12 km grid-cell volume average concentrations to point measurements at urban sites. It is also 

partly due to measurement artifacts where the STN OC observations are not “blank corrected” so 

are biased high. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-45 

Figure 3-23. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

organic carbon material (OCM) concentrations over the VISTAS 12 km domain (IMPROVE 

comparisons provided in Appendix C). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-46 

3.6.5 Elemental Carbon (EC) 

Evaluating model performance for EC, the model tends to underpredict concentrations when 

compared to the STN network (Figures 3-24 and C-5a), while better agreement is observed with 

the IMPROVE network (Figure C-5b). This follows the same conclusion as for OCM, and may 

be reflecting the differences between grid-cell volume average concentrations and point 

measurements. These differences are expected to be more pronounced at urban sites (STN) 

compared to rural ones (IMPROVE), as at urban locations there are likely stronger local spatial 

gradients in EC concentrations. The STN and IMPROVE networks also use different methods 

for measuring EC that may also be contributing to the differences in EC model performance 

using the two networks. In Section 3.5 and Appendix B we noted that the CMAQ 12 km 

modeling results exhibited superior model performance to the CMAQ 36 km modeling results 

across the urban STN network with bias closer to zero with the 12 km grid versus a large 

underprediction bias with the 36 km grid. Thus, the dilution of the urban EC emissions across 

coarse grid cell is surely contributing to the EC underprediction tendency at the STN monitoring 

network. 

Both the modeled values and observations (at both networks) exhibit higher concentrations in 

wintertime (Quarters 1 and 4), that is likely due to reduced atmospheric mixing compared to 

summertime. However, the seasonal pattern is much stronger in the model results compared to 

the observations. 

Another interesting finding has to do with relatively high modeled EC concentrations in the 

western part of the Georgia-Florida border, especially in Quarter 1. This is due to modeled 

emissions from biomass burning activities. However, nearby measurements at both the STN and 

IMPROVE networks do not exhibit the same trend. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-47 

Figure 3-24. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

elemental carbon (EC) concentrations over the VISTAS 12 km domain (IMPROVE comparisons 

provided in Appendix C). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-48 

3.6.6 Soil PM2.5 (SOIL) 

Soil PM2.5 is measured/calculated/derived at the IMPROVE sites only (Figures 3-25 and C-6). 

In general, soil PM2.5 is overpredicted compared to the observations. This is a common issue in 

grid-based models, where resuspended fugitive dust is assumed to be mixed uniformly in the first 

layer of the model, whereas in practice most of it is removed locally by impaction onto surfaces 

such as cars, buildings and vegetation. Higher concentrations are especially evident in urban 

centers, due to the way the model calculates the emission and dispersion of resuspended road 

dust. However, this is not very evident in this comparison, since most IMPROVE sites are 

located at rural areas. The ASIP/VISTAS modeling did apply fugitive dust transport factors 

(FDTF) to fugitive dust emission source categories to account for dust that is deposited locally 

and not transported. 

Another issue associated with the Soil evaluation is the mismatch between how Soil is defined in 

the measurement versus modeled values. The measured Soil values are derived by building it up 

from the elements, whereas the modeled values are obtained from the emission speciation 

profiles that are assigned all PM2.5 to the other PM2.5 species that is not explicitly speciated as 

SO4, NO3, EC or OCM. 

Also evident in the figures are the higher modeled concentrations in the western portion of the 

ASIP/VISTAS 12 km domain. This could be originating from various reasons, such as possibly 

higher emissions in that region, transport into the domain from the west, or differences in 

methodologies used to calculate soil emissions between the various Regional Planning 

Organizations (RPOs). The latter is likely the case, since a sharp spatial gradient is observed 

between the Midwest RPO states (and west of Mississippi, in the portions of the CENRAP 

states) and the MANE-VU (northeast) and VISTAS (southeast) states (see Figure 1-1 for 

definition of the RPO states). 

Regardless of actual model performance, in the context of State Implementation Plan (SIP) 

development and future attainment tests, modeled levels of soil components are of relatively 

little importance, since they are normalized according to observations, and little to no controls 

are being applied (so the Relative Reduction Factor would be equal or close to unity). 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-49 

Figure 3-25. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed 

(diamonds) soil PM2.5 (SOIL) concentrations over the VISTAS 12 km domain. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-50 

3.6.7 Sea-salt PM2.5 (SeaSalt) 

Sea-salt PM2.5 is measured/calculated at some of the IMPROVE sites only (see Figure C-7 in 

Appendix C). As expected, higher concentrations are measured at sites along the coastline. This 

spatial pattern is captured by the model as well; however, concentrations along the coastline are 

underpredicted compared to the measurements. 

Since sea-salt PM2.5 is not used for future attainment tests as part of the SIP development 

process, model performance in this case is of relatively little importance. 

3.6.8 Total PM2.5 Mass (PM2.5)

A comparison of the quarterly modeled spatial distribution of total PM2.5 mass concentrations 

with observations from the FRM, STN and IMPROVE network are provided in Figure C-8, with 

the results for the FRM network repeated in Figure 3-26 below. The conclusions on the model 

performance for total PM2.5 mass vary by which network is examined. For Q1 there are elevated 

PM2.5 concentrations in the major urban areas (e.g., Chicago-Gary, St. Louis, Atlanta, Northeast 

Corridor, etc.) due to concentrated urban emissions. There are also elevated PM2.5 mass 

concentrations for Q1 in the western part of the Georgia-Florida border that is due to biomass 

burning. The mainly rural IMPROVE monitors fail to capture many of these elevated areas, but 

does confirm the relative clean conditions in the Appalachian Mountains (Figure C-8c, top left). 

The IMPROVE St. Marks measured PM2.5 levels appears to refute the elevated PM2.5 in Q1 

along the western Florida-Georgia border. However, the STN (Figure C-8b) and FRM (Figures 

3-26 and C-8a) spatial plots confirm that elevated PM2.5 levels existed in this area; the lack of 

elevated PM2.5 at St. Marks is likely due to its coastal location and the sea breeze. 

During Q2 the model predicts elevated total PM2.5 mass levels from St. Louis across the Midwest 

into Ohio with highest values centered on Indianapolis, in southeastern Pennsylvania and up the 

Northeast Corridor and in the Birmingham and Atlanta urban areas in the southeast. The 

IMPROVE monitors suggest that the model is capturing the rural aspect of the spatial 

distribution of the total PM2.5 mass patterns, albeit with an underprediction bias (Figure C-8c, top 

right). However, the FRM (Figures 3-26 and C-8a) and STN (Figure C-8b) plots indicates that 

the model is underestimating the spatial extent of the elevated total PM2.5 mass levels. 

The highest seasonal PM2.5 levels occur in the Q3 summer period when SO4 is the highest. The 

observations indicate that the entire interior portion of the ASIP/VISTAS 12 km grid domain is 

covered by elevated PM2.5 levels, which is reproduced reasonably well by the model. However, 

the model is estimating slightly lower values and relatively cleaner areas over the Appalachian 

Mountains that are not supported by the observations. In addition, the observed PM2.5

concentration gradient from high to low occurs further south than predicted by the model. 

However, in general the model is doing a good job in reproducing the spatial distribution of 

PM2.5 in Q3. 

In Q3 the model estimates elevated PM2.5 levels across the upper Midwest (MO-IL-OH) and 

southwest (Northern AL and GA, SC and NC) that is split by the Appalachian Mountains and 

elevated levels in the Northeast Corridor. The IMPROVE network plots confirm the relative 

cleaner area in the Appalachian Mountains (Figure C-8c) and the STN network plot confirms the 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-51 

three areas of high PM2.5 (Figure C-8b). The FRM network spatial map (Figure 3-26 and C-8a) 

confirms the three areas of elevated PM2.5 concentrations, but suggests the southeast area is not 

as high as the others and that the PM2.5 distribution should be spottier. 

In conclusion, the model appears to do a good job in reproducing the spatial and temporal 

variations in PM2.5 concentrations across the ASIP/VISTAS 12 km grid albeit with an 

underprediction bias. The spatial distribution of the modeled PM2.5 concentrations is smoother 

and less spotty than the observed distributions, which is due in part to the coarse 12 km grid 

spacing used in the modeling. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-52 

Figure 3-26. 2002 Quarterly averages of CMAQ simulated and FRM observed (diamonds) total 

PM2.5 mass concentrations over the VISTAS 12 km domain. 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-53 

3.7 CONCLUSIONS OF MODEL PERFORMANCE EVALUATION 

The quarterly average CMAQ PM2.5 component species results near the FRM monitors are used 

in the Speciated Model Attainment Test (SMAT) to project the current year PM2.5 Design Values 

to the 2009 future year for comparison with the PM2.5 NAAQS. SMAT uses the relative changes 

CMAQ 2002 and 2009 modeling results to scale each of the PM2.5 components of the current 

year PM2.5 Design Values. These model derived scaling factors are referred to as Relative 

Response Factors (RRFs). In Chapter 4 of this TSD, we present the results of the SMAT PM2.5

Design Value projections for FRM monitoring sites in and near the ASIP states. These results 

indicate that most of the reductions in the PM2.5 Design Values between 2002 and 2009 are due 

to reductions in sulfate concentrations. Thus, performance of the model for sulfate is of most 

importance in the model performance evaluation. 

 

Below we discuss the implications of the major findings in the ASIP model performance 

evaluation in the context of the modeling results are used to project future year PM2.5 attainment 

through SMAT. 

• Nitrate Underprediction Tendency: NO3 is routinely underpredicted during the summer 

and adjacent months throughout the ASIP region. This underprediction is due to modeled 

NOx concentrations near zero, when observed values are low, but above zero (typically < 

1 μg/m3). However, NO3 is almost always a very minor to insignificant contributor to 

total PM2.5 mass at FRM monitors in the ASIP region (see stacked bar charts in Chapter 

4). In fact, the maximum NO3 contribution to a 2009 projected PM2.5 Design Value is 

1.0 μg/m3

with a median value of 0.2 μg/m3. Thus, the NO3 performance issues are not a 

big concern in the PM2.5 projections. 

• OCM Underpredictions: The OCM underprediction bias is a cause for concern since it is 

a major component of the PM2.5 mass at ASIP FRM monitoring sites with maximum 

contributions to the 2009 PM2.5 Design Values of ~8 μg/m3, minimum values of ~3 μg/m3 

and a median value of ~4 μg/m3. The reasons for the underestimation of OCM are 

unclear, but the fact that the underpredictions are higher in the urban than rural areas 

suggest that there may be missing anthropogenic emission sources, or possibly the urban 

OCM emissions are over diluted across the 12 km grid resolution used in the ASIP 

modeling. The changes in projected OCM concentrations between the current and 

projected PM2.5 Design Values are mostly less than 5% (i.e., 0.95 < RRFOCM < 1.05). 

Thus, the changes in OCM between the current and future year are having a minor 

influence on the projected PM2.5 Design Values. 

 

• EC Performance Issues: The EC underprediction bias at the urban sites is partly due to 

over dilution of the urban EC emissions due to the coarse (12 km) grid used. For the 

most part, the CMAQ model performed well for EC and the underprediction would not 

affect the relative changes in the model response to anthropogenic EC emissions changes. 

Therefore, any EC performance issues are not a cause for concern, although the model 

performance for EC was generally good. 

 

• Sulfate Underprediction Bias: Although SO4 is performing well, it does have an 

underprediction bias that is largest in the summer months. But this underprediction is not 

severe and the model appears to be capturing the temporal and spatial variations in the 

observed sulfate well and is responding to the SO2 emission reductions between 2002 
 

January 2008 

F:\PM25 SIP Docs\Model Performance Evaluation\Chap_3_MPE1.doc 3-54 

and 2009 in a manner as expected. Thus the model performance indicates that the 

modeled relative changes in SO4 concentrations are likely a valid response. 

• Soil Performance Issues: The CMAQ performance for the Soil species is quite poor. 

This Soil component of the 2009 projected PM2.5 Design Value ranges from 0.4 to 1.8 

μg/m3. The RRFs for Soil indicate that it is mostly increasing, with summer (Q3) Soil 

RRFs typically ranging from 1.0 to 1.3. We address the Soil performance issues in 

Chapter 5 weight of evidence analysis where we assumed that the modeled Soil remains 

unchanged between 2002 and 2009 which did not significantly affect the projected 2009 

PM2.5 Design Values. 

SO4 reductions dominate the changes in PM2.5 Design Values between 2002 and 2009. SO4 

performance is quite good in the CMAQ 2002 Base G2 Actual base case simulation almost 

always achieving the PM performance goal and frequently also achieving the more stringent 

ozone performance goal. These factors provide confidence in the Design Value projections 

using the CMAQ Base G modeling results. 
 

APPENDIX B 

Air Quality Model Performance Evaluation: 

State-by-State and Key Monitoring Site PM2.5 Model Evaluation 
 
 B-1 

B.1 OVERVIEW 

The discussion of model performance evaluation in this Appendix focuses on the evaluation of 

the 2002 Base G2 Actual base case modeling results from the Community Multiscale Air Quality 

(CMAQ) modeling system Version 4.5 with SOAmods enhancement across each VISTAS state 

and at key PM2.5 monitors within the VISTAS states. The CMAQ results are compared with 

observational data from the Speciated Trends Network (STN), Federal Reference Method 

(FRM) PM2.5 mass and South East Aerosol Research and CHaracterization (SEARCH) study 

monitoring networks. The evaluation focuses primarily on the air quality model’s performance 

with respect to individual components of fine particulate matter (PM2.5) and total fine particulate 

matter (PM2.5) mass within PM2.5 nonattainment areas (NAAs) in the VISTAS region. 

Consequently, the focus is on the STN and FRM monitoring sites that are located within the 

PM2.5 NAAs although SEARCH also includes two sizes in NAAs (Atlanta and Birmingham). 

Previously we have presented an evaluation of the CMAQ 2002 Base G2 Actual base case 

simulation focusing primarily on PM and visibility as part of the Visibility Improvements for 

States and Tribal Associations in the Southeast (VISTAS) that focused on the regional model 

performance and performance at Class I areas (Morris et al., 2007). 

B.2 CMAQ EVALUATION METHODOLOGY 

EPA’s integrated ozone, PM2.5 and regional haze modeling guidance calls for a comprehensive, 

multi-layered approach to model performance testing, consisting of the four major components: 

operational, diagnostic, mechanistic (or scientific) and probabilistic (EPA, 2007a). 

B.2.1 Evaluation Approach 

The CMAQ model performance evaluation effort focused on the first two components, namely: 

• Operational Evaluation: Tests the ability of the model to estimate PM2.5 concentrations 

and the components of PM2.5 (i.e., sulfate, nitrate, ammonium, organic carbon, elemental 

carbon and other PM2.5) within the PM2.5 nonattainment areas. This evaluation examines 

whether the measurements are properly represented by the model predictions but does not 

necessarily ensure that the model is getting “the right answer for the right reason”; and 

• Diagnostic Evaluation: Tests the ability of the model to predict PM chemical 

composition including PM precursors (e.g., SOx, NOx, and NH3) and associated oxidants 

(e.g., ozone and nitric acid); PM size distribution; temporal variation; spatial variation; 

mass fluxes; and components of light extinction (i.e., scattering and absorption). 

The diagnostic evaluation can also include the performance of diagnostic tests to better 

understand model performance and identify potential flaws in the modeling system that can be 

corrected. The diagnostic evaluation may also includes the use of “probing tools” to understand 
 B-2 

why the model obtains a given prediction; probing tools include Process Analysis (PA), 

decoupled direct method (DDM) and source apportionment (SA). 

In this final model performance evaluation for the ASIP 2002 Actual Base G2 CMAQ 36/12 km 

base case simulation, the operational evaluation for PM2.5 has been given the greatest attention 

since this is the primarily thrust of EPA’s modeling guidance. However, we have also examined 

certain diagnostic features dealing with the model’s ability to simulate sub-regional and 

monthly/diurnal gas phase and aerosol concentration distributions. In the course of the ASIP 

and VISTAS studies, numerous diagnostic sensitivity tests were performed to investigate and 

improve model performance. Key diagnostic tests performed are discussed and the results for 

the rest are available on the projects modeling website: http://pah.cert.ucr.edu/vistas/vistas2/. 

B.2.2 Particulate Matter and Component Species 

Fine particulate matter can be composed of varying amounts of several different species. The 

main components of PM2.5 are as follows: 

• Sulfate (SO4) 

• Nitrate (NO3) 

• Ammonium (NH4) 

• Organic Carbon Matter (OCM) 

• Elemental Carbon (EC) 

• Soil (also known as crustal material, fine soil, major metal oxides, inorganic particulates, 

or other PM) 

• Sea Salt (NaCl) 

• Particle Bound Water (PBW) 

B.2.3 Ambient Air Quality Data for ASIP Model Evaluation 

A ground-level model evaluation database for 2002 was compiled by the modeling team using 

several routine and research-grade databases. The focus of the evaluation of the CMAQ model 

for use in simulating PM2.5 mass concentrations in urban areas and NAAs. The primary 

monitoring networks available to evaluate this component of the CMAQ are: (a) EPA Federal 

Reference Method PM2.5 and PM10 Mass Networks (EPA-FRM); (b) EPA Speciation Trends 

Network (STN) of PM2.5 species; and (c) two sites from the Southeastern Aerosol Research and 

Characterization (SEARCH). These PM monitoring networks may also provide ozone and other 

gas phase precursors and product species, and visibility measurements at some sites. There are 

also gas-phase criteria pollutant measurements in NAAs from the EPA’s Air Quality System 

(AQS) that are available for use. In addition, there are several more rural monitoring networks 

that collect PM or PM related species, such as IMPROVE, CASTNet and NADP. VISTAS used 

these more rural networks to perform a regional evaluation of the CMAQ 2002 Base G2 base 

case that is presented elsewhere (Morris et al., 2007). The study team has performed a detailed 

evaluation of the CMAQ 2002 base case simulation using the AQS that is available on the 

modeling website: http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml. Table B-1 and Figure B-1 
 B-3 

summarizes the species collected and locations of the monitoring sites for the FRM, STN and 

SEARCH monitoring networks use in the ASIP PM2.5 model evaluation. 

Table B-1a. Ambient PM2.5 monitoring data available in PM2.5 nonattainment areas in the ASIP 

region during 2002. 

Monitoring 

Network Chemical Species Measured 

Sampling Frequency; 

Duration 

SEARCH 

24-hr PM25 (FRM Mass, OC, BC, SO4, NO3, 

NH4, Elem.); 24-hr PM coarse (SO4, NO3, 

NH4, elements); Hourly PM2.5 (Mass, SO4, 

NO3, NH4, EC, TC); and Hourly gases (O3, 

NO, NO2, NOy, HNO3, SO2, CO) Daily, Hourly; 

EPA-FRM Only total fine mass (PM2.5) 1 in 3 days; 24 hr 

EPA-STN Speciated PM2.5 Varies; Varies 

B.3 MODEL PERFORMANCE STATISTICS 

To quantify PM2.5 model performance, several statistical measures were calculated and evaluated 

for all the monitors within each ASIP state with a NAA and at individual monitors within the 

NAAs. The statistical measures selected were based on the recommendations outlined in section 

18.4.1 of the USEPA’s Guidance On The Use Of Models And Other Analyses for Demonstrating 

Attainment of Air Quality Goals for Ozone, PM2.5, and Regional Haze (EPA, 2007a). 

VISTAS/ASIP has established model performance goals and criteria for PM2.5 and components 

of fine particle mass based on previous model performance for ozone and fine particles that are 

shown in Table B-1b. EPA modeling guidance for fine particulate matter at the time noted that 

PM models might not be able to achieve the same level of performance as ozone models. 

Table B-1b. Model performance goals and criteria for components of fine particle mass. 

Fractional Bias Fractional Error Comment 

<15% <35% Goal for PM model performance based on ozone model 

performance, considered excellent performance 

<30% <50% Goal for PM model performance, considered good 

performance 

<60% <75% Criteria for PM model performance, considered average 

performance. Exceeding this level of performance 

indicates fundamental concerns with the modeling 

system and triggers diagnostic evaluation. 

B.4 PM2.5 MODELING PERFORMANCE ACROSS THE ASIP STATES 

In the following discussions we use selected monthly scatter plots, time series plots, soccer plots, 

bugle plots and model performance statistical measures from the UCR Analysis Tool application 

to the 2002 CMAQ Base G2 Actual base case simulation in an operational evaluation of the 
 B-4 

model for PM2.5 mass and components of PM2.5 within each ASIP state. These results represent 

just a small subset of the model performance evaluation products of the CMAQ 2002 36/12 km 

Base G2 base case simulation. Complete model evaluation products are available on the project 

website: 

http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml

The focus of the model performance evaluation in this Appendix is on PM2.5 and PM2.5 

components within PM2.5 nonattainment areas (NAAs) and urban areas in the ASIP region. The 

regional performance evaluation of the model for PM and visibility has been documented in the 

VISTAS Regional Haze SIP Technical Support Document (TSD; Morris et al., 2007), which in 

addition to the products available on the website includes: 

• Regional operational and diagnostic model performance evaluation for PM and visibility 

across the VISTAS/ASIP region in the southeastern U.S. (Appendix B of VISTAS 

TSD); 

• Model performance for PM and visibility at Class I areas in the VISTAS region. 

(Appendix C of VISTAS TSD); and 

• Spatial model performance for PM components across the VISTAS/ASIP 12 km grid. 

(Appendix D of VISTAS TSD). 

The performance of the ASIP/VISTAS CMAQ 2002 36/12 km Base G2 base case simulation for 

ozone has also been evaluation and the results are available at: 

http://pah.cert.ucr.edu/vistas/vistas2/evaluation_results/2002ga2a.mpe/

B.4.1 Alabama 

Figures B-2a-f contain scatter plots and model performance statistics for PM2.5 component 

species at STN sites located in Alabama, the four seasonally representative months of January, 

April, July and October and the CMAQ 2002 36/12 km Base G2 base case simulation. 

Performance for SO4 is quite good with the scatter plots clustered tightly around the 1:1 line of 

perfect agreement (Figure B-2a) and fractional bias and error performance metrics achieving the 

most stringent <±15% and < 35% performance goal (Table B-1b). Performance for NO3 

exhibits an underprediction tendency with fractional bias values of approximately -20% in 

January, -110% in April, -175% in July and -10% in October (Figure B-2b). The large summer 

NO3 underprediction bias occurs when both the model and measured NO3 values are extremely 

low with the modeled values near zero and with the STN NO3 observed values < 1 μg/m3

. Note 

that under these summer conditions, the NO3 will almost completely volatilize off the FRM filter 

so the FRM would measure insignificant PM2.5 mass due to NO3. Thus, these large summer 

NO3 underpredictions are not a concern as NO3 is not an important component of PM2.5 mass. 

Performance for NH4 is also fairly good achieving the <±30%/50% fractional bias/error goal for 

PM species (Figure B-2c), which is not surprising given its strong link to SO4. With the 

exception of a few outliers, performance for organic carbon mass (OCM or OC) is fairly good in 
 B-5 

the winter, but underestimated by -20% to -30% in the summer and fall (Figure B-2d). The 

reasons for the OCM underprediction includes the fact that the STN OC measurements are not 

blank corrected and have high uncertainties (see SANDWICH discussion in Chapter 4) and there 

are also large uncertainties in the OC emissions and the form of OC emissions as particles or 

semi-volatile organic gases. EC model performance at STN sites in Alabama is also fairly good, 

mostly achieving the PM model performance goal, albeit with an underestimation tendency in 

the summer (Figure B-2e). 

The CMAQ performance for PM2.5 mass at 3 STN and 20 FRM monitoring sites in Alabama are 

shown in Figures B-2f and B-2g, respectively. Performance for PM2.5 mass is generally good 

achieving the most stringent ozone performance goal (<±bias/error) in January and October and 

the PM goal all year round. 

Time series of predicted (CMAQ 12 km results) and observed 24-hour PM2.5 components and 

model performance statistics at the key North Birmingham STN site are shown in Figure B-3. 

As discussed in Chapter 4, this is the site with the highest projected 2009 PM2.5 Design Value in 

the ASIP region. SO4 performance is quite good with annual fractional bias and errors of 6% 

and 29% achieving the most stringent ozone performance goal (Figure B-3a, top). The fractional 

bias for NO3 of -110% indicates a large underprediction bias that is driven by near zero modeled 

NO3 concentrations in the summer when the observed values are < 1 ug/m3 (Figure B-3a, 

bottom). However, the NO3 performance in the winter, when observed values are above 1 ug/m3

is much better. The EC and OCM performance at the North Birmingham STN site is 

characterized by an underprediction with fractional bias of -30% and -49%, respectively (Figure 

B-3b). This underprediction is due in part to uncertainties in the STN carbon measurements as 

well as the contributions of nearby industrial sources whose impacts are diluted across the 12 km 

CMAQ grid used. Performance for NH4 and PM2.5 mass at the North Birmingham STN site are 

also quite good with fractional bias/error values of -5%/33% and 11%/29% achieving the more 

stringent ozone performance goal. The good performance of NH4 and PM2.5 mass is not 

surprising given that they are closely tied to SO4 that exhibits good model performance. 

Figure B-4 contains time series of predicted and observed PM components and performance 

statistics for the SEARCH BHM monitoring site also located at North Birmingham. 

Performance at the SEARCH BHM site is similar to the STN site at the same location with 

generally good performance for SO4, NH4 and PM2.5 mass and an underprediction bias for EC 

and OCM. The SEARCH BHM site also includes performance for the “Soil” species that is 

greatly overestimated with annual fractional bias values of +164% (Figure B-4d, top) 

Figure B-5 displays a soccer plot of monthly fractional bias versus gross error for PM2.5 mass 

performance across 20 FRM sites in Alabama and compares them with the three levels of 

bias/error performance goals in Table B-1b. PM2.5 model performance is best in the winter 

months when the most stringent ozone < ±15%/35% bias/error goal is achieved. In the summer 

months the model tends to underestimate the FRM observed PM2.5 mass by -15% to -25%, but 

still achieves the <±30%/50% PM model performance goal. The Bugle Plots of monthly PM2.5 

bias and error in Figure B-6 also confirm that the bias and error performance goals for PM are 

achieved across FRM sites in Alabama. 
 B-6 

The PM2.5 model performance products on the project website also include a day-by-day 

evaluation of the PM2.5 model performance at STN sites. This is done with side-by-side stacked 

bar charts of PM2.5 components for each sampling day in 2002 and each STN site in Alabama. 

These model performance displays can be found at: 

http://pah.cert.ucr.edu/vistas/vistas2/evaluation_results/2002/2002ga2a/MPE/stackbar/ST

N/AL_STN_stackbar_monthly.pdf
 B-7 

Figure B-2a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, 

July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-8 

Figure B-2b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-9 

Figure B-2c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-10 

Figure B-2d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for 

January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-11 

Figure B-2e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-12 

Figure B-2f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Alabama and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-13 

Figure B-2g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, FRM sites in Alabama and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-14 

Figure B-3a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Jefferson County, Alabama Site No. 01-

073-0023 (North Birmingham) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-15 

Figure B-3b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Jefferson County, 

Alabama Site No. 01-073-0023 (North Birmingham) for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-16 

Figure B-3c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Jefferson County, 

Alabama Site No. 01-073-0023 (North Birmingham) for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-17 

Figure B-4a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the SEARCH North Birmingham (NBHM) Alabama 

site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-18 

Figure B-4b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the SEARCH North 

Birmingham (NBHM) Alabama site for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-19 

Figure B-4c. Time series of predicted and observed 24-hour ammonium (NH4, top) and total 

carbon mass (TCM, bottom) concentrations during 2002 at the SEARCH North Birmingham 

(BHM) Alabama site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-20 

Figure B-4d. Time series of predicted and observed 24-hour other PM2.5 (Soil, top) and total 

fine particulate mass (PM2.5, bottom) concentrations during 2002 at the SEARCH North 

Birmingham (BHM) Alabama site for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-21 

Figure B-5. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Alabama for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-22 

Figure B-6. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Alabama and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-23 

B.4.2 Florida 

Figure B-7 summarizes the CMAQ PM2.5 species model performance across the three STN sites 

in Florida and January, April, July and October 2002. SO4 is underestimated at the Florida STN 

sites (Figure B-7a) with fractional bias values that are closer to zero in the winter than summer 

(e.g., -7% January versus -34% July 12 km results) and closer to zero using the 12 km than 36 

km CMAQ modeling results (e.g., in January bias, is -7% using 12 km and -22% using 36 km 

modeling results). Even with this underestimation bias SO4 performance at the Florida STN 

sites is fairly good. NO3 performance, on the other hand, exhibits a large underprediction bias 

with fractional bias values that range from -70% in January to -170% in July (Figure B-7b). 

NH4 performance at the Florida STN sites is fairly good always achieving the <±30%/50% PM 

performance goal and sometimes even achieving the more stringent <±15%/35% ozone 

performance goal (Figure B-7c). It is interesting that in January and October when both SO4 and 

NO3 exhibit an underprediction bias, NH4 exhibits a slight overprediction bias. Given the linked 

relationship between these species this is somewhat surprising and may be due to artifacts in the 

measurements, or the modeled SO4 being more fully neutralized by NH4 than in the 

measurements. The OCM and EC are characterized by underprediction tendency with the OCM 

fractional bias ranging from -55% to -95% with the CMAQ 36 km and 12 km producing nearly 

identical performance metrics (Figure B-7d). The EC performance is better with fractional bias 

values ranging from -4% to -52% and the 12 km modeling results produce significantly better 

EC performance metrics than the 36 km modeling results (Figure B-7e). 

The PM2.5 mass performance in Florida is summarizes across 3 STN sites and 28 FRM sites in 

Figures B-7f and B-7g, respectively. Across the STN sites, the PM2.5 fractional bias ranges from 

-27% to -75% for the four months examined, whereas across the FRM network the bias values 

are lower ranging from+2% to -69%. Given the underprediction tendency of SO4, NO3 and 

carbon the general underprediction of total PM2.5 mass is not unexpected. Why the 

underprediction bias is greater across the STN network than the FRM network is unclear but is 

likely related to the locations of the 3 STN and 28 FRM monitoring networks and the 

measurement artifacts of the two sampling techniques (e.g., FRM will loose more of the volatile 

material). 

An example predicted and observed time series and annual model performance metrics for 24-

hour PM components at an STN site in Miami-Dade County Florida is shown in Figure B-8. The 

summer sulfate underprediction tendency results in an annual fractional bias of -30% with error 

of 44% just barely achieving the PM model performance goals (Figure B-8a, top). NO3 

performance at this Miami site is very poor with a -180% fractional bias with modeled NO3 on 

most days near zero, compared to around 0.5 μg/m3

 measured values during the summer. OCM 

and EC are also underpredicted at the Miami site with fractional bias values of, respectively, 

-123% and -84% (Figure B-8b). Ammonium performance is fairly good meeting the most 

stringent ozone performance goal, whereas total PM2.5 is underpredicted due to the 

underprediction of most of the PM components as discussed above (Figure B-8c). 

The soccer plot of monthly PM2.5 bias and error performance for FRM sites in Florida is shown 

in Figure B-9, with the Bugle Plots shown in Figure B-10. For the winter months of Nov, Dec, 
 B-24 

Jan and Feb the PM2.5 performance statistics are quite good with bias/error values meeting the 

<±15%/35% ozone performance goal. The spring and fall months either achieve the 

<±30%/50% or <±50%/125% performance goal and criteria. The May-Jul summer months are at 

or exceed the -50% performance criteria levels (note that in the Soccer Plots if the symbol would 

be plotted outside of the range of the plot then it is plotted on the axis, such as the May and July 

-50% values in Figure B-9). 
 B-25 

Figure B-7a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, April, 

July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base G2 Actual base 

case simulation. 
 B-26 

Figure B-7b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base G2 Actual base 

case simulation. 
 B-27 

Figure B-7c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-28 

Figure B-7d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations for 

January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-29 

Figure B-7e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-30 

Figure B-7f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Florida and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-31 

Figure B-7g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, FRM sites in Florida and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-32 

Figure B-8a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Miami-Dade County, Florida Site No. 12-

086-1016 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-33 

Figure B-8b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Miami-Dade County, 

Florida Site No. 12-086-1016 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-34 

Figure B-8c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Miami-Dade County, 

Florida Site No. 12-086-1016 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-35 

Figure B-9. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Florida for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-36 

Figure B-10. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Florida and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-37 

B.4.3 Georgia 

There appears to be some valid data capture issues in the 2002 STN MPE database for Georgia 

with only one valid site in January, 6 in April and 8 in July and October. The CMAQSO4 

performance in Georgia is generally pretty good with bias/error frequently achieving the 

<±15%/35% ozone performance goal and always achieving the PM goal (Figure B-11a). The 

STN sites also have the summer nitrate underprediction performance issue as seen in the other 

states (Figure B-11b). The NH4 model performance falls between the SO4 and NO3 

performance and, with the exception of January which is based on one site so the statistics are 

not very robust, it achieves the <±30%/50% PM performance goal (Figure B-11c). The carbon 

performance in Georgia appears to be slightly better than the other states, although still with a 

general underestimation tendency. With the exception of July, when the CMAQ 36 km OCM 

performance exhibits fractional bias and error values of -34% and 42% thereby achieving the 

<<±30%/50% PM performance goal, the other three months achieve the most stringent 

<±15%/35% ozone performance goal (Figure B-11d). EC performance is variable with generally 

low bias but lots of scatter producing higher gross errors (Figure B-11e). 

The performance of total PM2.5 mass across the STN (Figure B-11f) and FRM (Figure B-11g) 

monitoring sites in Georgia is generally fairly good. July exhibits an underprediction tendency 

with fractional bias values of approximately -40% (STN) and -25% (FRM) that is likely caused 

by the underestimation of SO4 and especially OCM. January is characterized by an 

overestimation tendency. But in general the PM2.5 performance statistics mostly achieve the 

most stringent ozone performance goal. 

An example time series of predicted and observed PM concentrations and annual statistics for a 

site in the Atlanta Georgia (DeKalb County) is shown in Figure B-12. The model tracks the 

observed SO4 concentrations at this site extremely well producing low fractional bias (-5%) and 

gross error (27%). NO3 performance exhibits the underprediction tendency seen in the other 

states, but is not as severe with a -16% fractional bias and 77% error. The CMAQ 12 km results 

reproduces the OCM concentrations at this site surprisingly well with low bias (-14%) and error 

(31%). EC is overestimated (bias of 28%) at this Atlanta site with fairly high error (44%) but 

still achieves the PM performance goal. NH4 is overstated at this site on average (29% error) but 

also achieves the PM performance goal. Finally, the performance for PM2.5 mass at the DeKalb 

County STN site is extremely good with very low bias (2%) and error (25%). 

Figure B-13 displays another example of predicted and observed PM species, time series and 

annual statistics for a site in Atlanta, only this time for the SEARCH Jefferson Street (JST) site. 

Unlike the STN 1:3 sampling frequency, the SEARCH JST site collects 24-hour samples 

everyday so has a richer model evaluation database. The performance at the SEARCH JST 

Atlanta is similar to the STN DeKalb County site with good performance for SO4 that achieves 

the ozone performance goal, and NO3 performance that has an overall annual underestimation 

tendency (bias of -16%) and high error (80%). It appears the summer NO3 underestimation bias 

is being compensated by a winter NO3 overestimation bias that can be quite large (Figure B-13a, 

bottom). The SEARCH JST site had some data capture problems for OCM so the performance 

is not very meaningful, but does suggest and underprediction tendency as seen at other sites. 

Unlike the DeKalb County STN site that suggests an EC overestimation bias (28%), at the 
 B-38 

SEARCH JST site the CMAQ 12 km modeling results exhibits a zero bias and fractional gross 

error of 37% (Figure B-13b, bottom). The STN and SEARCH sites use different sampling 

techniques for carbon which may help explain these differences. The observed NH4 at the JST 

site is reproduced well (Figure B-13c, top) with a fractional bias of -5% and error of 37%. The 

Soil term measures at the JST site is greatly overestimated with fractional bias of 136% (Figure 

B-13d). The Soil performance issue is discussed extensively in the VISTAS regional haze TSD 

(Morris et al., 2007) and two potential reasons for its poor performance are: (1) incompatibilities 

between the measured Soil that is built up from measured elements and the “Soil” in the 

modeling which is the other PM2.5 that consists of PM2.5 that is not explicitly speciated as SO4, 

NO3, OCM or EC; and (2) impacts of local fugitive dust sources at the monitor that are subgridscale to the CMAQ 36/12 km modeling so is not captured by the modeling. Performance for 

total PM2.5 mass at the JST site is excellent (Figure B-13d, bottom) with a low bias (-3%) and 

error (24%). 

Figure B-14 and B-15 displays Soccer and Bugle Plots, respectively, of monthly fractional bias 

and error across the 26 FRM sites in Georgia. All months achieve the PM performance goal and 

8 of the 12 months even achieve the more stringent ozone performance goal. The months that 

the PM2.5 mass performance doesn’t achieve the ozone performance goal are due to an 

overestimation bias in January and underestimation bias in May-July. 
 B-39 

Figure B-11a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-40 

Figure B-11b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base G2 Actual base 

case simulation. 
 B-41 

Figure B-11c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-42 

Figure B-11d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-43 

Figure B-11e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-44 

Figure B-11f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Georgia and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-45 

Figure B-11g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in Georgia and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-46 

Figure B-12a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN DeKalb County, Georgia Site No. 13-089-

0002 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-47 

Figure B-12b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN DeKalb County, 

Georgia Site No. 13-089-0002 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-48 

Figure B-12c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN DeKalb County, 

Georgia Site No. 13-089-0002for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-49 

Figure B-13a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the SEARCH Jefferson Street (JST) Georgia site 

for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-50 

Figure B-13b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the SEARCH Jefferson Street 

(JST) Georgia site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-51 

Figure B-13c. Time series of predicted and observed 24-hour ammonium (NH4, top) and total 

carbon mass (TCM, bottom) concentrations during 2002 at the SEARCH Jefferson Street (JST) 

Georgia site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-52 

Figure B-13d. Time series of predicted and observed 24-hour other PM2.5 (Soil, top) and total 

fine particulate mass (PM2.5, bottom) concentrations during 2002 at the SEARCH Jefferson 

Street (JST) Georgia site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-53 

Figure B-14. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Georgia for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-54 

Figure B-15. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Georgia and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-55 

B.4.4 Kentucky 

The monthly scatter plots and performance statistics for PM species and sites in Kentucky are 

shown in Figure B-16. The performance for SO4 in January is mixed, with a group of points 

clustered around the 1:1 line and another further to the right resulting in a net underprediction 

bias of approximately -30%. An examination of the more detailed perform plots on the website 

reveals that this dichotomy in January SO4 model performance in Kentucky is due to temporal 

rather than spatial differences in performance; of the three STN sampling days in January 2002 

CMAQ does a very good job in simulating the SO4 model performance on January 14 and 26 

with fractional bias of approximately -5%, but on January 20 the bias is closer to -80%. The 

SO4 performance in other months is characterized by an underprediction bias in April and 

October of -30% to -50% and near zero bias in July, but with some scatter for an error of ~40%. 

As seen in the other states, NO3 performance is characterized by underprediction bias that is 

largest in the summer (Figure B-16b). In Kentucky, the NO3 fractional bias is -30% to -40% in 

January, -65% in April and -150% to -160% in July. In October the NO3 underprediction is 

compensated by some overprediction so that there is near zero bias but a 50-55% error. 

Like SO4, NH4 performance is fairly good albeit with an underprediction bias that is largest in 

the summer (Figure B-16c). As seen for the other sites, OCM is underpredicted with the bias 

ranging from -50% in January to 70% to -80% in the other three months (Figure B-16d). The 

OCM errors are the opposite sign but the same magnitude as the bias suggesting that the OCM 

underprediction bias is almost systematic. The CMAQ 36 km and 12 km modeling results 

exhibits different EC performance in Kentucky. The CMAQ 12 km results have bias that is 

much closer to zero than the 36 km results (Figure B-16e). This is likely due to the 36 km grid 

cells over dispersing the urban EC emissions. Both the CMAQ 36 km and 12 km results exhibit 

a lot of variability producing a lot of scatter and high error metrics. 

The performance of total PM2.5 mass concentrations in Kentucky is evaluated across 10 STN 

(Figure B-16f) and 23 FRM sites (Figure B-16g). Given the underprediction of most of the PM 

component species it is not surprising that total PM2.5 mass is underpredicted across the STN 

sites by -16%, -42% to -55%, -33% and -6% to -13% for January, April, July and October. 

Across the FRM sites in Kentucky, however, the CMAQ model for PM2.5 mass bias that is much 

closer to zero with values of -8%, -11% to -14%, -32% to -35% and +18% to -19% for the same 

four months. The FRM PM2.5 mass performance always achieves the PM performance goal and 

sometimes even achieves the ozone model performance goal. 

An example PM model performance for a single STN site in Lexington (Fayette County) 

Kentucky is shown in Figure B-17. The model reproduces the temporal variations in SO4 quite 

well (Figure B-17a, top) at this site resulting in low bias (-10%) and error (33%). Although the 

model shows some skill in reproducing the observed higher NO3 values in the winter, especially 

in the first part of the year, the near zero modeled NO3 during the summer results in a fractional 

bias value of -62% with 87% error, so does not achieve the PM performance goal (Figure B-17a, 

bottom). 
 B-56 

OCM is underpredicted with fractional bias values of -52% and an error of 58% just at the PM 

performance goal (Figure B-17b, top). EC exhibits much better performance at the Lexington, 

Kentucky site with a bias near zero (-4%) and error of 36% so almost achieves the ozone 

performance goal. 

The performance for NH4 and PM2.5 mass is quite good, which is not surprising given the good 

SO4 performance and the strong link between SO4 and these two species in Kentucky. The 

fractional bias for NH4 and PM2.5 are both approximately -10% with NH4 having an error of 

36% and PM2.5 an error of 28% thereby achieving the ozone performance goal. 

Figure B-18 displays another example time series performance for PM components at an STN 

site in Kenton County, Kentucky that lies in the Cincinnati NAA. The performance at the 

Cincinnati NAA STN site is similar to that at Lexington STN site with very good SO4 

performance that produces bias/error values of -12%/30% that achieves the ozone performance 

goal and NO3 with a net underestimation with bias/error of -47%/75% that is right at the PM 

performance criteria (Figure B-18a). OCM exhibits a large underprediction bias (-57%), 

whereas performance for EC is better with bias (14%) and error (36%) that is right at the ozone 

performance goal (Figure B-18b). NH4 (-4%/37%) and PM2.5 (-7%/31%) also exhibit low 

bias/error and good performance, which again is not surprising given their relationship with SO4 

(Figure B-18c). 

Figure B-19 and B-20 summarized the Kentucky PM2.5 mass model performance using Soccer 

and Bugle Plots of monthly bias and error. With the exception of July, whose fractional bias 

falls slightly below -50%, PM 2.5 performance in the other 11 months achieves the PM bias/error 

performance goal of <-30%/75%. In fact 5 months that occur mainly in the winter even achieve 

the more stringent ozone model performance goal. 
 B-57 

Figure B-16a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-58 

Figure B-16b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-59 

Figure B-16c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-60 

Figure B-16d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-61 

Figure B-16e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-62 

Figure B-16f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Kentucky and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-63 

Figure B-16g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in Kentucky and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-64 

Figure B-17a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Fayette County, Kentucky Site No. 21-

067-0012 (Lexington) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-65 

Figure B-17b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Fayette County, 

Kentucky Site No. 21-067-0012 (Lexington) for the CMAQ 12 km 2002 Base G2 Actual base 

case simulation.
 B-66 

Figure B-17c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Fayette County, 

Kentucky Site No. 21-067-0012 (Lexington) for the CMAQ 12 km 2002 Base G2 Actual base 

case simulation.
 B-67 

Figure B-18a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Kenton County, Kentucky Site No. 21-

117-0007 (Cincinnati) site for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-68 

Figure B-18b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Kenton County, 

Kentucky Site No. 21-117-0007 (Cincinnati) site for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-69 

Figure B-18c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Kenton County, 

Kentucky Site No. 21-117-0007 (Cincinnati) site for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-70 

Figure B-19. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Kentucky for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-71 

Figure B-20. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Kentucky and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-72 

B.4.5 Mississippi 

Performance of PM species across sites in Mississippi is shown in Figure B-21. SO4 is generally 

underpredicted with fractional bias values ranging from -15% to -46%. Worst SO4 performance 

is seen in the January and July winter/summer months than the April and October transitional 

months. NO3 performance metrics are extremely poor with fractional bias values of -80% to 

-90% in January and April and -163% in July. In October some NO3 overpredictions begin to 

compensate for the frequent near zero model predictions so that the underprediction bias is lower 

(-40%), but the error is still very high (~100%). 

NH4 performance is characterized by lots of scatter and little correlation, although some skill is 

seen in October (Figure B-21c). OCM is underpredicted at sites in Mississippi with fractional 

bias values of approximately -50% in January, April and July improving to -24% in October 

(Figure B-21d). Performance for EC is better, albeit still with an underprediction bias, and again 

seen as for other states, the CMAQ 12 km modeling results produces much lower bias than the 

36 km modeling results (Figure B-21e). 

PM2.5 performance in Mississippi across the STN and FRM networks are shown in Figures B-21f 

and B-21g, respectively. Across the STN network, PM2.5 is underestimated by approximately -

40% rising to -65% in July. Across the FRM network, the PM2.5 underprediction bias is lower, at 

-14% in January and -25% in April and rising to approximately -50% in July. In October the 

model is better able to replicate the observed PM2.5 mass with CMAQ 12 km bias and error 

values across the FRM network of 2% and 29% achieving the ozone model performance goals. 

Figure 22 displays an example model performance time series analysis at an STN site in Jones 

County in southeastern Mississippi (Hattiesburg). SO4 performance is worse than seen in the 

other ASIP states with a -36% fractional bias and 44% error. NO3 performance exhibits similar 

characteristics as the other ASIP states with a summer underestimation bias that produces an 

annual fractional bias value of -40%. Like the other states, OCM is underestimated by -43% on 

average, but unlike the other states the underprediction fractional bias for EC exhibits an 

underprediction (-60%) that is even greater than seen for OCM. 

Performance statistics for NH4 are reasonable with a near zero bias and 42% error, the near zero 

bias is due to a few days of large overprediction compensated for a general underprediction. 

Given the fact that almost all of the PM components are underestimated, the underprediction of 

total PM2.5 mass is not a surprise with a fractional bias value of -35%. 

The Soccer Plot (Figure B-23) and Bugle Plot (Figure B-24) for PM2.5 mass performance confirm 

that PM2.5 performance in Mississippi is worse than seen for the other ASIP states. Only two 

months achieve the ozone model performance goal and only 8 months achieve the PM model 

performance goal. The underprediction bias in March through July 2002 produces fractional bias 

that do not achieve the <±50% PM performance goal. 
 B-73 

Figure B-21a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-74 

Figure B-21b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-75 

Figure B-21c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-76 

Figure B-21d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-77 

Figure B-21e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-78 

Figure B-21f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Mississippi and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-79 

Figure B-21g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in Mississippi and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-80 

Figure B-22a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Jones County, Mississippi Site No. 28-

067-0002 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-81 

Figure B-22b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Jones County, 

Mississippi Site No. 28-067-0002 for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-82 

Figure B-22c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Jones County, 

Mississippi Site No. 28-067-0002 for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-83 

Figure B-23. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Mississippi for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 B-84 

Figure B-24. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Mississippi and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-85 

B.4.6 North Carolina 

SO4 performance across 9 STN monitoring sites in North Carolina and four months from 2002 is 

shown in Figure B-25a. Although SO4 performance achieves the PM performance goal it varies 

by month with January and July having fractional bias of -245% top -30%, April a fractional bias 

of approximately -20% and October having near zero fractional bias. Although NO3 is still 

greatly underestimated in North Carolina during the summer (July bias of -105% to -142%), 

during January and October NO3 exhibits an overestimation bias with values of 84% and 35%, 

respectively (Figure B-25b). This is likely due to overstated ammonia emissions during the 

winter and adjacent periods which is somewhat verified by the overstated ammonium during 

January and October that have fractional bias values in the +40% to +55% range (Figure B-25c) 

OCM is characterized by an underprediction bias ranging from -30% to -50%, although 

performance in the April and October transition months achieves the PM performance goal 

(Figure B-25d). EC, on the other hand, achieves the PM performance goal in January and 

October but has lots of scatter in the other two months exceeding the PM performance goal for 

bias in April (-34% to -41%) and for error in July (62%-65%). 

PM2.5 mass performance across the 9 STN and 36 FRM sites in North Carolina exhibits monthly 

variations (Figures B-25f and B-25g). FRM PM2.5 is overpredicted in January (15% to 18%) and 

October (25% to 30%), but underpredicted in April (-9% to -16%) and July (-37% to -39%). The 

STN PM2.5 performance is similar except because the STN measurements tend to be higher the 

underprediction bias is greater and the overprediction bias is less. 

Figures B-26 and B-27 displays time series performance analysis at two STN sites in North 

Carolina, one on Charlotte (Mecklenburg County) and the other in Raleigh (Wake County). So4 

performance is good at both sites, albeit with an underprediction bias of -14% for Charlotte and -

19% for Raleigh with both sites having low error (31% and 32%). Although both sites have a 

net annual NO3 underprediction bias (-19% and -38%) due to the summer underprediction 

tendency discussed previously, during the winter there are some days when the CMAQ model 

overpredicts the observed NO3 by a factor of 2-3. OCM is underpredicted by a far margin at the 

two sites (-34% and -50%). At the Charlotte site EC is overpredicted by 45% which occurs year 

round, whereas at the Raleigh site EC is predicted well with a zero bias and 34% error. NH4 is 

predicted reasonably well with bias/error values that achieve or nearly achieve the ozone 

performance goal. PM2.5 is predicted well at the Charlotte site with low bias (-8%) and error 

(29%), but underestimated at the Raleigh site with a bias of -28%. 

The Soccer and Bugle Plots summary of FRM PM2.5 mass model performance in North Carolina 

are shown in Figures B-28 and B-29. With the exception of the large summer underprediction 

bias in June and July, the other 10 months PM2.5 performance achieves the <±30%/50% PM 

performance goal and four of the months even achieve the more stringent ozone performance 

with four additional months right at the ozone goal. 
 B-86 

Figure B-25a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-87 

Figure B-25b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-88 

Figure B-25c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-89 

Figure B-25d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 

km Base G2 Actual base case simulation. 
 B-90 

Figure B-25e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-91 

Figure B-25f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in North Carolina and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-92 

Figure B-25g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in North Carolina and the CMAQ 2002 36 and 12 

km Base G2 Actual base case simulation. 
 B-93 

Figure B-26a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Mecklenburg County, North Carolina Site 

No. 37-119-0041 (Charlotte) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-94 

Figure B-26b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Mecklenburg 

County, North Carolina Site No. 37-119-0041 (Charlotte) for the CMAQ 12 km 2002 Base G2 

Actual base case simulation.
 B-95 

Figure B-26c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Mecklenburg County, 

North Carolina Site No. 37-119-0041 (Charlotte) for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-96 

Figure B-27a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Wake County, North Carolina Site No. 

37-183-0014 (Raleigh) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-97 

Figure B-27b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Wake County, North 

Carolina Site No. 37-183-0014 (Raleigh) for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-98 

Figure B-27c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Wake County, North 

Carolina Site No. 37-183-0014 (Raleigh) for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-99 

Figure B-28. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in North Carolina for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 B-100 

Figure B-29. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in North Carolina and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-101 

B.4.7 South Carolina 

The CMAQ performance for SO4 across the 4 STN sites in South Carolina generally has an 

underprediction bias but is reasonably good always achieving the PM performance goal and 

sometimes even achieving the ozone performance goal (Figure B-30a). NO3 performance is 

poor with an underprediction bias generally from -40% to -180% that is largest in the summer 

when NO3 is lowest and not an important part of the PM2.5 (Figure B-30b). NH4 performance is 

variable frequently achieving the PM performance goal with the worst performance in the 

summer with a large underprediction tendency. OCM is underestimated with bias of -15% to -

62%. EC performance also has an underestimation tendency, although not is bad as seen for 

OCM. The CMAQ 12 km results also exhibits much better EC performance than the CMAQ 36 

km modeling results with the 12 km EC model performance always achieving the PM 

performance goal and sometimes even achieving the ozone performance goal, whereas the 

CMAQ 36 km EC results never achieves the PM performance goal for the four months studied 

(Figure B-30e). 

PM2.5 model performance across STN and FRM networks in South Carolina are shown in 

Figures B-30f and B-30g, respectively. PM2.5 mass is usually underpredicted across both 

networks, with the fractional bias across the STN network is worse and approximately 20 

percentage points larger underprediction than the FRM network. In fact, except for July with a -

40% bias, the bias/error for PM2.5 across the FRM network achieves the <±15%/35% ozone 

model performance goal. 

An example time series of predicted and observed PM concentrations and model performance 

metrics at the Greenville, South Carolina STN site is given in Figure B-31. All PM components 

are underestimated on average with fractional bias values for SO4, NO3, NH4, OCM, EC and 

PM2.5 of, respectively, -23%, -97%, -19%, -46%, -11% and -30%. Although the model does 

show some skill in reproducing the temporal and seasonal variability in the observed PM 

concentrations. 

The summary fractional bias and gross error Soccer and Bugle Plots for total PM2.5 mass across 

the FRM network in South Carolina are shown in Figures B-32 and B-33. During the fall and 

winter months of Aug, Sep, Nov, Dec, Jan and Feb the monthly bias and error statistics achieve 

the ozone model performance goal. The three additional months of Mar, Apr and Oct achieve 

the PM performance goal. Whereas the summer months of May, Jun and Jul fail to achieve the 

PM performance goal due to large underprediction bias that is due to the underprediction of each 

of the PM component species. 
 B-102 

Figure B-30a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-103 

Figure B-30b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-104 

Figure B-30c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-105 

Figure B-30d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 

km Base G2 Actual base case simulation. 
 B-106 

Figure B-30e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-107 

Figure B-30f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in South Carolina and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-108 

Figure B-30g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in South Carolina and the CMAQ 2002 36 and 12 

km Base G2 Actual base case simulation. 
 B-109 

Figure B-31a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Greenville County, South Carolina Site 

No. 45-045-0009 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-110 

Figure B-31b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Greenville County, 

South Carolina Site No. 45-045-0009 for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-111 

Figure B-31c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Greenville County, 

South Carolina Site No. 45-045-0009 for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation.
 B-112 

Figure B-32. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in South Carolina for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 B-113 

Figure B-33. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in South Carolina and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-114 

B.4.8 Tennessee 

The CMAQ does a good job predicting the observed SO4 concentrations in Tennessee with bias 

and error values that either achieve the most stringent ozone performance goal or are right at the 

edge of the ozone goal (Figure B-34a). The CMAQ 12 km results are slightly better than the 36 

km predictions and, unlike the other states, where July had the largest underprediction bias, in 

Tennessee the SO4 performance in July is quite good with the 12 km results producing a 4% bias 

and 32% error. 

The model exhibits little skill in its NO3 predictions (Figure B-34b) with the usual large 

underprediction bias in the summer (e.g., -170% in July). Although the bias is low in January 

and October (<±10%), the error is large (60% to 80%). NH4 performance is generally 

reasonably good with low bias and error in January and April, but an underprediction bias in July 

(-20% to -30%) and an overprediction bias in October (33% to 36%). 

As seen in the other states, OCM exhibits a large underprediction bias that generally ranges from 

-40% in the winter fall and spring to -60% in summer (Figure B-34d). EC performance is quite 

good in January and October, but has an underprediction tendency in April and July (Figure B34f). 

Tennessee PM2.5 mass performance across the STN and FRM networks are shown in Figures B34f and B-34g. Across the STN networks performance for PM2.5 achieves the ozone goal in 

January and October, but exhibits an underprediction bias in April and July such that in July it 

exceeds the PM performance goal. Across the FRM network the model performs better for 

PM2.5 achieving or nearly achieving the ozone performance goal in January, April and October, 

but with an underprediction bias in July (-26% to -35%) that is at the PM performance goal. 

Example time series comparisons and annual model statistics are presented for two sites in 

Tennessee, one in Chattanooga, Hamilton County (Figure B-35) and one in Knoxville, Knox 

County (Figure B-36). The model reproduces the daily variations in the SO4 at these two sites 

fairly well with an annual underprediction bias of -17% and -9%. NO3 is underpredicted in the 

summer resulting in an annual underprediction bias in Chattanooga and Knoxville of -86% and 

-74%, respectively. OCM is underpredicted at both sites with bias values of -56% and -44%. 

EC is underestimated on average in Chattanooga (-23%) and overestimated in Knoxville (21%). 

The underprediction of the two main PM components at the two sites (SO4 and OCM) results in 

a net underprediction bias of PM2.5 mass of -51% and -22%. 

The monthly fractional bias and error Soccer and Bugle Plots for FRM PM2.5 performance in 

Tennessee show that all months of the year achieve the PM model performance goal and that 5 

of the months even achieve or are right at the ozone model performance goal. The months that 

do not achieve the ozone performance goal are due to a summer underprediction bias for MayJuly and a fall/winter overprediction bias for October and November and a too high error for 

December. 
 B-115 

Figure B-34a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-116 

Figure B-34b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-117 

Figure B-34c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km Base G2 

Actual base case simulation. 
 B-118 

Figure B-34d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-119 

Figure B-34e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-120 

Figure B-34f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Tennessee and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-121 

Figure B-34g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in Tennessee and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-122 

Figure B-35a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Hamilton County, Tennessee Site No. 47-

065-4002 (Chattanooga) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-123 

Figure B-35b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Hamilton County, 

Tennessee Site No. 47-065-4002 (Chattanooga) for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-124 

Figure B-35c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Hamilton County, 

Tennessee Site No. 47-065-4002 (Chattanooga) for the CMAQ 12 km 2002 Base G2 Actual 

base case simulation.
 B-125 

Figure B-36a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Knox County, Tennessee Site No. 47-

093-1020 (Knoxville) for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-126 

Figure B-36b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Knox County, 

Tennessee Site No. 47-093-1020 (Knoxville) for the CMAQ 12 km 2002 Base G2 Actual base 

case simulation.
 B-127 

Figure B-36c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Knox County, 

Tennessee Site No. 47-093-1020 (Knoxville) for the CMAQ 12 km 2002 Base G2 Actual base 

case simulation.
 B-128 

Figure B-37. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Tennessee for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 B-129 

Figure B-38. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Tennessee and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-130 

B.4.9 Virginia 

SO4 performance is fairly good in Virginia with low bias and error in July and October that 

achieves the <±15%/35% ozone bias/error performance goal (Figure B-39a). Although January 

and April exhibit a -20% to -30% underprediction bias for SO4, it still achieves the PM 

performance goal. As seen for the other states, NO3 performance is poor and characterized by 

an underprediction bias that is greatest in the summer (-170%) when CMAQ estimates near zero 

NO3 predictions. NH4 performance achieves the ozone performance goal in April and PM 

performance goal in January and October, but the -48% to -55% bias in July exceeds the PM 

performance goal (Figure B-39c). 

OCM performance is characterized by an underprediction bias of from -30% to -60% with error 

of opposite sign and similar magnitude (Figure B-39d). The fractional bias for EC is about 20-

30 percentage points higher using CMAQ with a 12 km grid than 36 km grid resulting in the 12 

km results always achieving the PM performance goal but the 36 km results underprediction bias 

in April and October is in the -40% to -50% range so does not achieve the PM performance goal. 

The PM2.5 mass performance across the FRM sites is fairly good achieving the ozone 

performance goal in January, April and October and the PM performance goal in all three 

months (Figure B-39g). Across the STN sites, the PM performance goal is achieved in January, 

April and October but not in July due to a -36% to -44% underprediction bias. 

Figure B-40 displays an example annual time series model performance evaluation for a site in 

Richmond County, Virginia. SO4 performance at this Richmond County STN site is extremely 

good as the model follows the day-to-day variations quite well resulting in low annual fractional 

bias (-11%) and gross error (30%). NO3 performance, on the other hand, is characterized by a 

winter overprediction and summer underprediction bias with an annual bias and gross error of 

-46% and 91%, respectively. OCM at the Richmond site has an underprediction bias that occurs 

throughout the year resulting in an annual fractional bias of -46%. Better performance is seen 

for EC with bias (24%) and error (42%) that achieves the PM performance goal. PM2.5

performance is also fairly good with bias (-10%) and error (28%) that achieves the most stringent 

ozone performance goal. 

Another example time series model performance analysis is given in Figure B-41 for a site in 

Roanoke County, Virginia. Similarly good SO4 and poor NO3 performance is seen at the 

Roanoke County site. OCM is almost systematically underpredicted through the year with a 

-50% bias and 51% error. Whereas better performance is seen for EC that achieves the ozone 

performance goal. PM2.5 performance is generally characterized by an underestimation tendency 

resulting in a -33% fractional bias. 

Figures B-42 and B-43 summarize the PM2.5 model performance across 19 FRM sites in Virginia 

using, respectively, Soccer and Bugle Plots. Monthly fractional bias and gross error for PM2.5

mass achieves the PM performance goal for every month in 2002, with six of the months 

achieving the ozone performance goal and another two just at the ozone goal. As seen in the 

other states, the three summer months of May, June and July have an underprediction bias 

that exceeds the ozone performance goal, but still meets the PM performance goal.
 B-131 

Figure B-39a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January, 

April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-132 

Figure B-39b. Scatter plots of predicted and observed nitrate (NO3) concentrations for January, April, 

July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base G2 Actual base 

case simulation. 
 B-133 

Figure B-39c. Scatter plots of predicted and observed ammonium (NH4) concentrations for January, 

April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-134 

Figure B-39d. Scatter plots of predicted and observed organic matter carbon (OCM) concentrations 

for January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-135 

Figure B-39e. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-136 

Figure B-39f. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, STN sites in Virginia and the CMAQ 2002 36 and 12 km Base 

G2 Actual base case simulation. 
 B-137 

Figure B-39g. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations 

for January, April, July and October 2002, FRM sites in Virginia and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-138 

Figure B-40a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Richmond County, Virginia Site No. 51-

760-0020 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-139 

Figure B-40b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Richmond County, 

Virginia Site No. 51-760-0020 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-140 

Figure B-40c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Richmond County, 

Virginia Site No. 51-760-0020 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-141 

Figure B-41a. Time series of predicted and observed 24-hour sulfate (SO4, top) and nitrate 

(NO3, bottom) concentrations during 2002 at the STN Roanoke County, Virginia Site No. 51-

770-0014 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-142 

Figure B-41b. Time series of predicted and observed 24-hour organic mass carbon (OCM, top) 

and elemental carbon (EC, bottom) concentrations during 2002 at the STN Roanoke County, 

Virginia Site No. 51-770-0014 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-143 

Figure B-41c. Time series of predicted and observed 24-hour ammonium (NH4, top) and fine 

particulate mass (PM2.5, bottom) concentrations during 2002 at the STN Roanoke County, 

Virginia Site No. 51-770-0014 for the CMAQ 12 km 2002 Base G2 Actual base case simulation.
 B-144 

Figure B-42. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in Virginia for the CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-145 

Figure B-43. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in Virginia and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-146 

B.4.10 West Virginia 

There were no speciated PM2.5 STN monitoring sites within West Virginia during the 2002 

modeling year, so the PM2.5 model evaluation is limited to the 16 FRM PM2.5 mass monitoring 

sites. In January, April and July, CMAQ underestimates PM2.5 mass across the FRM sites in 

West Virginia by approximately -30%, -25% and -40% and achieves the PM performance goal in 

January and April, but not July (Figure B-44). In October the model has near zero bias and 32% 

error so achieves the ozone model performance goal. The Soccer and Bugle Plots in Figures B45 and B-46 summarizes the FRM PM2.5 mass performance in West Virginia in each month. 

Nine of the months achieve the PM performance goal with the winter months even achieving or 

nearly achieving the ozone model performance goal. As seen for the other states, the 

underprediction bias during the May-July summer months results in the bias falling outside of 

the PM performance goal, but within the PM performance criteria. 
 B-147 

Figure B-44. Scatter plots of predicted and observed fine particulate mass (PM2.5) concentrations for 

January, April, July and October 2002, FRM sites in West Virginia and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 
 B-148 

Figure B-45. Soccer plot of monthly fractional bias versus fractional gross error for PM2.5 mass 

at FRM monitors in West Virginia for the CMAQ 12 km 2002 Base G2 Actual base case 

simulation. 
 B-149 

Figure B-46. Bugle plots of monthly fractional bias (top) and gross error (bottom) for FRM sites 

in West Virginia and thee CMAQ 12 km 2002 Base G2 Actual base case simulation. 
 B-150 

B.5 CONCLUSIONS STATE-BY-STATE MODEL EVALUATION 

The state-by-state model performance evaluation for P.M2.5 mass and component species reveals 

several model performance tendencies of the CMAQ 2002 Base G2 actual base case. SO4

performance is by far constantly the best performing PM species frequently achieving the most 

stringent ozone performance goal. Although the SO4 performance in some of the most southern 

states degraded some (e.g., Florida and Mississippi), NO3 performance, on the other hand, was 

consistently poor with large summer underprediction bias. OCM was usually underpredicted, 

with the underprediction bias most severe in the summer (May – July). EC performance is 

usually fairly good frequently achieving the most stringent ozone model performance goal with 

the CMAQ 12 km modeling results exhibiting much superior EC performance to the 36 km 

modeling results. Performance for NH4 and total PM2.5 mass is also good most of the time. This 

is not surprising since the SO4 performance is usually quite good and SO4 dominates the PM2.5

mass and drives the NH4 performance in the Southeast U.S. 
APPENDIX C 

Spatial Maps of Quarterly Average PM2.5 Components and 

Total Mass Concentrations with Superimposed Observations 

for the CMAQ 2002 12 km Base G2 Actual Base Case Simulation 

F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-3

C.1 OVERVIEW OF SPATIAL MODEL PERFORMANCE 

The discussion in this appendix provides a general qualitative overview of the 2002 Base G2 

Actual base case model performance of the Community Multiscale Air Quality (CMAQ) modeling 

system Version 4.5 with SOAmods enhancement across the ASIP 12 km domain, with emphasis on 

temporal (seasonal) and spatial patterns within the domain. The evaluation is carried out for each of 

the major PM2.5 components separately (sulfate, nitrate, ammonium, organic carbon material, 

elemental carbon, soil PM2.5, sea-salt PM2.5), as well as for total PM2.5 mass. For each species, two 

sets of figures are provided, one comparing model predictions to observations at the STN sites, the 

other to observations at the IMPROVE network sites (soil and sea-salt are available at the 

IMPROVE network only). Total PM2.5 mass estimates are also compared to observations at the 

FRM network, in addition to the STN and IMPROVE networks. Four figures are provided for each 

set corresponding to the average concentrations during the four quarters of the year (Q1: Jan-Mar; 

Q2: Apr-Jun; Q3: Jul-Sep; Q4: Oct-Dec). 

This evaluation allows for a general understanding of model performance and key issues. A more 

detailed and quantitative performance evaluation at key VISTAS states sites is provided in 

Appendix B and summarized in Chapter 3. 

The observed quarterly average PM2.5 mass and PM2.5 species concentrations were obtained by 

averaging all 24-hour measurements at a site that occurred during each three month quarter. The 

modeled quarterly average spatial maps were obtained by averaging the daily average 

concentrations for each day in the three month quarter and each grid cell. No attempt was made to 

match the modeled daily average concentrations with the 1:3 day sampling frequency used by the 

monitoring networks. In fact, obtaining modeled quarterly average concentrations trying to match 

the measurement days is problematic since not all sites collect valid samples on every 1:3 day 

sampling day. In the case of the STN network, this results in a bias in the Q1 quarterly average 

predicted and observed comparisons since many of the STN sites started up in 2002 and have 

samples near the end of Q1 but not during the beginning. For species like SO4 that has a strong 

seasonal trend, basing observed Q1 averages on samples in March and modeled values averaged 

across January-February-March introduces a seemingly underprediction bias that is artificial and an 

artifact of the network sampling periods. We know this is occurring at some STN sites, for 

example see discussion for Georgia in Section 3.5.3. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-4 

C.2 SPATIAL MODEL PERFORMANCE EVALUATION 

C.2.1 Sulfate (SO4) 

Modeled sulfate concentrations show a very strong seasonal pattern (Figures C-1a,b), with peak 

concentrations occurring during summer months (Quarters 2 and 3), when photochemistry is 

highest. A spatial pattern is also evident, with higher concentrations in the northeast, Ohio River 

valley, and southeast, compared to the upper Midwest and Florida, caused mainly by the higher and 

denser SO2 emissions in those regions compared to the latter two. In Florida, the impacts of 

individual major sources of SO2 are evident (such as in the Tampa and Jacksonville areas), as their 

emissions remain relatively unmixed with emissions from other regions, being surrounded by 

ocean. However, in most of the domain, a “regional” sulfate field is observed as a result of mixing 

of emissions from various regions, especially for the long averaging time presented here (three 

months). 

Overall, the model seems to accurately simulate sulfate levels over the domain, and captures both 

the temporal and spatial patterns exhibited in the observations, both from the STN (Figure C-1a) 

and IMPROVE (Figure C-1b) networks. The exception is Quarter 1 for the STN network, in which 

the observed concentrations seem higher than the modeled ones, and there is not much agreement 

in the spatial pattern between the two. This is mainly due to an artifact in the way the observations 

are presented here discussed above. Since some of the STN sites were not in operation during the 

first few months of 2002, observed Quarter 1 averages might in fact be driven by observations 

during the latter (and warmer) part of Quarter 1 (e.g., March samples), and therefore are biased 

high. 

 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-5 

Figure C-1a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) sulfate 

(SO4) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-6 

Figure C-1b. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

sulfate (SO4) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-7 

C.2.2 Nitrate (NO3) 

Nitrate is in many ways the “mirror” image of sulfate, with peak concentrations occurring during 

wintertime, when the cooler temperatures are favorable for particulate nitrate formation and sulfate 

concentrations are lower making more ammonia available to bond with nitrate. The need for 

nitrate to be neutralized by a basic compound in order to be in the particulate state results in peak 

particulate nitrate concentrations occurring in areas of the domain where ammonia emissions are 

highest, such as the Midwest. 

While overall the model does capture the seasonal and spatial variability in particulate nitrate 

concentrations (Figures C-2a,b), it does seem to overestimate concentrations during wintertime, 

especially over urban centers, such as in part of the Northeast and over Atlanta and Birmingham. 

Winter overestimations are also evident over much of North Carolina. While it is impossible to 

infer from these data alone on the cause for these overpredictions, they are related to either one or a 

combination of the model nitrate partitioning between gaseous and particulate phase, and the 

availability of gaseous ammonia, as reflected by the emissions inventory. The latter can be the 

common denominator between the overpredictions over the urban centers and over North Carolina, 

as ammonia emissions in these areas, though originating from different sources (mobile sources in 

the urban centers; agricultural ammonia emissions in North-Carolina), might be overestimated, 

making more ammonia available for particulate formation compared to the observations. 

In the summer, the model estimates near zero (< 0.25 μg/m3

) particulate nitrate concentrations, 

whereas the observed values are typically in the 0.25-0.75 μg/m3

 range. In any event, both the 

modeled and observed summer NO3 values suggest that it is not an important component of the 

total PM2.5 mass concentrations in the southeastern U.S. during the summer. 

Fairly similar trends are observed when comparing the model to either the STN (Figure C-2a) or 

the IMPROVE (Figure C-2b) networks, however a more detailed analysis is provided based on the 

STN, given its size and density. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-8 

Figure C-2a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) nitrate 

(NO3) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-9 

Figure C-2b. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

nitrate (NO3) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-10 

C.2.3 Ammonium (NH4) 

Ammonium concentrations depend, to a large degree, on the availability of sulfate and nitrate. 

Therefore, ammonium levels do not exhibit as strong seasonal pattern, due to the seasonal tradeoff 

between sulfate and nitrate, to which it bonds. 

Model performance, when compared to the STN data (Figure C-3a), seems to be better during 

summertime (Quarters 2 and 3), when the ammonium is mainly associated with sulfate. During 

wintertime (Quarters 1 and 4), most of the issues observed with nitrate overpredictions are also 

evident in the ammonium plots, especially over North-Carolina and the urban centers of the 

Southeast. 

Ammonium is not directly measured by IMPROVE, rather it is derived by assuming it completely 

neutralizes the measured sulfate and nitrate. Assuming that nitrate is completely neutralized by 

ammonium is most of the time a valid assumption, however the same may not be true for sulfate, 

especially in the summer months. Hence the observed ammonium comparison in the IMPROVE 

case (Figure C-3b) is likely overstated during the summer months when sulfate is less likely to be 

fully neutralized and the seemingly underpredicted observed ammonium concentrations across the 

Appalachian Mountain IMPROVE monitoring sites in Quarter 3 in Figure C-3b are in part due to 

using derived observed ammonium. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-11 

Figure C-3a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

ammonium (NH4) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-12 

Figure C-3b. 2002 Quarterly averages of CMAQ simulated and IMPROVE derived (diamonds) 

ammonium (NH4) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-13 

C.2.4 Organic Carbon Material (OCM) 

Evaluating model performance for OCM is complicated due to its dual primary/secondary nature 

and anthropogenic/biogenic precursor sources. As a result, the evaluation is dependant on a 

combination of very different emissions and formation processes. In addition, there is a fair amount 

of uncertainty associated with OC measurements, with a major limitation being the inability to 

directly measure OCM. Instead, only OC is measured, and OCM is estimated (and compared to 

model results) by multiplying OC by an OCM/OC ratio factor. Such OCM/OC ratio factors 

typically range from 1.2 to 2.2 depending on whether the OCM is fresh or aged, where we have 

used an OCM/OC factor of 1.4. In addition, different analysis methods are used in the STN 

(NIOSH) and IMPROVE networks that introduce additional measurement artifacts. 

The seasonal pattern of OCM concentrations would depend on the relative contributions from 

secondary OCM (peak concentrations during summertime) and primary OCM. The seasonal 

pattern of primary OCM would depend on the seasonality of activity and emissions from sources 

such as biomass burning and transportation. Given a flat activity/emissions profile, primary OCM 

is would typically be higher in wintertime, due to reduced atmospheric mixing. 

Comparing model OCM predictions to STN observations (Figure C-4a), both exhibit a fairly flat 

seasonal pattern. Overall, the modeled concentrations are lower compared to observations. The 

model seems to exhibit a much stronger spatial pattern, with peak concentrations occurring in the 

Southeast. This is likely associated with biomass burning in the winter months, and biogenic 

secondary OCM formation in the summer months. The increased modeled-concentrations of 

secondary OCM are evident in Quarter 3, especially over the Northeast. When comparing 

simulated concentrations to IMPROVE observations (Figure C-4b), better agreement is observed. 

Given that most IMPROVE sites are located in rural areas, this may be an indication that OCM is 

underpredicted mainly at urban sites (STN, Figure C-4a), possibly due to local spatial gradients and 

the comparison of diffused 12 km grid-cell volume average concentrations to point measurements 

at urban sites. It is also partly due to measurement artifacts where the STN OC observations are 

not “blank corrected” so are biased high. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-14 

Figure C-4a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

organic carbon material (OCM) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-15 

Figure C-4b. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

organic carbon material (OCM) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-16 

C.2.5 Elemental Carbon (EC) 

Evaluating model performance for EC, the model tends to underpredict concentrations when 

compared to the STN network (Figure C-5a), while better agreement is observed with the 

IMPROVE network (Figure C-5b). This follows the same conclusion as for OCM and may be 

reflecting the differences between grid-cell volume average concentrations and point 

measurements. These differences are expected to be more pronounced at urban sites (STN) 

compared to rural ones (IMPROVE), as at urban locations there are likely stronger local spatial 

gradients in EC concentrations. The STN and IMPROVE networks also use different methods for 

measuring EC that may also be contributing to the differences in EC model performance using the 

two networks. In Chapter 3 and Appendix B we noted that the CMAQ 12 km modeling results 

exhibited superior EC model performance to the CMAQ 36 km modeling results across the urban 

STN network so the dilution of the urban EC emissions across the coarse 12 km grid cell is surely 

contributing to the EC underprediction tendency. 

Both the modeled values and observations (at both networks) exhibit higher concentrations in 

wintertime (Quarters 1 and 4) that is likely due to reduced atmospheric mixing compared to 

summertime. However, the seasonal pattern is much stronger in the model results compared to the 

observations. 

Another interesting finding has to do with relatively high modeled EC concentrations in the 

western part of the Georgia-Florida border, especially in Quarter 1. This is due to modeled 

emissions from biomass burning activities. However, nearby measurements at both the STN and 

IMPROVE networks do not exhibit the same trend. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-17 

Figure C-5a. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) 

elemental carbon (EC) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-18 

Figure C-5b. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

elemental carbon (EC) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-19 

C.2.6 Soil PM2.5 (SOIL) 

Soil PM2.5 is measured/calculated/derived at the IMPROVE sites only (Figure C-6). In general, 

soil PM2.5 is overpredicted compared to the observations. This is a common issue in grid-based 

models, where resuspended fugitive dust is assumed to be mixed uniformly in the first layer of the 

model, whereas in practice most of it is removed locally by impaction onto surfaces such as cars, 

buildings and vegetation. Higher concentrations are especially evident in urban centers, due to the 

way the model calculates the emission and dispersion of resuspended road dust. However, this is 

not very evident in this comparison, since most IMPROVE sites are located in rural areas. The 

ASIP/VISTAS modeling did apply fugitive dust transport factors (FDTF) to fugitive dust emission 

source categories to account for dust that is deposited locally and not transported. 

Another issues associated with the Soil evaluation is the mismatch between how Soil is defined in 

the measured versus modeled values. The measured Soil values are derived by building it up from 

the elements, whereas the modeled values are obtained from the emission speciation profiles that 

assign all PM2.5 that is not explicitly speciated as SO4, NO3, EC or OCM to the other PM2.5 

category that is interpreted as Soil in the model performance evaluation. 

Also evident in the figures are the higher modeled concentrations in the western side of the 

domain. This could be originating for various reasons, such as possibly higher emissions in that 

region, transport into the domain from the west, or differences in methodologies used to calculate 

crustal emissions between the various Regional Planning Organizations (RPOs). The latter is likely 

the case, since a sharp spatial gradient is observed between the Midwest RPO states (and west of 

Mississippi, in the portions of the CENRAP states) and the MANE-VU (northeast) and VISTAS 

(southeast) states (see Figure 1-1 for definition of the RPO states). 

Regardless of actual model performance, in the context of PM2.5 State Implementation Plan (SIP) 

development and future attainment tests, modeled levels of soil components are of relatively little 

importance, since they are normalized according to observations, and little to no controls are being 

applied to fugitive dust sources (so the Relative Reduction Factor would be equal or close to unity). 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-20 

Figure C-6. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

soil PM2.5 (SOIL) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-21 

C.2.7 Sea-Salt PM2.5 (SeaSalt)

Sea-salt PM2.5 is measured/calculated at some of the IMPROVE sites only (Figure C-7). As 

expected, higher concentrations are measured at sites along the coastline. This spatial pattern is 

captured by the model as well; however, concentrations along the coastline are underpredicted 

compared to the measurements. 

Since sea-salt PM2.5 is not used for future attainment tests as part of the SIP development process, 

model performance in this case is of relatively little importance. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-22 

Figure C-7. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

sea-salt PM2.5 (SeaSalt) concentrations over the VISTAS 12 km domain.
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-23 

C.2.8 Total PM2.5 Mass (PM2.5)

A comparison of the modeled spatial distribution of quarterly average total PM2.5 mass 

concentrations with observations from the FRM, STN and IMPROVE network are provided in 

Figure C-8. The conclusions on the model performance for total PM2.5 mass vary by which 

network is examined. For Q1, there are elevated PM 2.5 concentrations in the major urban areas 

(e.g., Chicago-Gary, St. Louis, Atlanta, Northeast Corridor, etc.) and the western part of the 

Georgia-Florida border, the latter is due to biomass burning. The mainly rural IMPROVE monitors 

fail to capture any of these elevated areas, but does confirm the relative clean conditions in the 

Appalachian Mountains (Figure C-8c, top left). The IMPROVE St. Marks measured PM2.5 levels 

appears to refute the elevated PM2.5 in Q1 along the western Florida-Georgia border. However, the 

STN (Figure C-8b) and FRM (Figure C-8a) spatial plots confirm that elevated PM2.5 levels occur in 

this area during Q1; the lack of elevated PM2.5 at St. Marks is likely due to its coastal location. 

During Q2 the model predicts elevated total PM2.5 mass levels from St. Louis across the Midwest 

into Ohio with highest values centered on Indianapolis, in southeastern Pennsylvania and up the 

Northeast Corridor, and in the Birmingham and Atlanta urban areas in the southeast. The 

IMPROVE monitors suggest that the model is capturing the rural aspect of the spatial distribution 

of the total PM2.5 mass patterns, albeit with an underprediction bias (Figure C-8c, top right). 

However, the FRM (Figures 3-26 and C-8a) and STN (Figure C-8b) plots indicates that the model 

is underestimating the spatial coverage of the elevated total PM2.5 mass levels. 

The highest seasonal PM2.5 levels occur in the Q3 summer period when SO4 is the highest. The 

observations indicate that the entire interior portion of the ASIP/VISTAS 12 km grid domain is 

covered by elevated PM2.5 levels, which is reproduced reasonably well by the model. However, the 

model is estimating slight lower values and relatively cleaner areas over the Appalachian 

Mountains that are not supported by the observations. In addition, the observed PM2.5

concentration gradient from high to low occurs further south than predicted by the model. 

However, in general the model is doing a good job in reproducing the spatial distribution of PM2.5

in Q3. 

In Q3 the model estimates elevated PM levels across the upper Midwest (MO-IL-OH) and 

southwest (Northern AL and GA, SC and NC) that is split by the Appalachian Mountains and 

elevated levels in the Northeast Corridor. The IMPROVE network plots confirm the relative 

cleaner area in the Appalachian Mountains (Figure C-8c) and the STN network plot confirms the 

three areas of high PM2.5 (Figure C-8b). 

Model performance evaluation for total PM2.5 mass depends heavily on the performance for the 

individual components. As such, it is less meaningful than the performance evaluation for the 

individual components. In addition, modeled total PM2.5 is not used for future attainment tests as 

part of the SIP development process that uses the Speciated Modeled Attainment Test (SMAT) that 

is based on the modeled PM2.5 component species. The FRM network spatial map (Figure 3-26 

and C-8a) confirms the three areas of elevated PM2.5 concentrations, but suggests the southeast area 

is not as high as the others and that the PM2.5 distribution should be spottier. 

In conclusion, the model appears to do a good job in reproducing the spatial and temporal 

variations in PM2.5 concentrations across the ASIP/VISTAS 12 km grid, albeit with an 

underprediction bias. The spatial distribution of the modeled PM2.5 concentrations is smoother and 

less spotty than the observed distributions, which is due in part to the coarse 12 km grid spacing 

used in the modeling. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-24 

Figure C-8a. 2002 Quarterly averages of CMAQ simulated and FRM observed (diamonds) total 

PM2.5 mass concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-25 

Figure C-8b. 2002 Quarterly averages of CMAQ simulated and STN observed (diamonds) total 

PM2.5 (PM25) concentrations over the VISTAS 12 km domain. 
F:\PM25 SIP Docs\Model Performance Evaluation\Appendix_C_MPE-Spatial1.doc C-26 

Figure C-8c. 2002 Quarterly averages of CMAQ simulated and IMPROVE observed (diamonds) 

total PM2.5 (PM25) concentrations over the VISTAS 12 km domain. 
Scatter Plots for Knoxville-Area Monitors