APPENDIX B 

Air Quality Model Performance Evaluation: 

Regional Model Evaluation across the VISTAS Region
 B-2 

B.1 OVERVIEW 

The discussion of model performance evaluation in this Appendix focuses on the evaluation of 

the 2002 Base G2 Actual base case modeling results from the Community Multiscale Air Quality 

(CMAQ) modeling system Version 4.5 with SOAmods enhancement across the VISTAS states 

and VISTAS 12 km grid. The CMAQ results are compared with observational data from the 

Interagency Monitoring of PROtected Visual Environments (IMPROVE), Speciated Trends 

Network (STN), Clean Air Status Trends Network (CASTNet), Federal Reference Method 

(FRM) PM2.5 mass, National Acid Deposition Program (NADP) and South East Aerosol 

Research and CHaracterization (SEARCH) study monitoring networks. The evaluation will 

primarily focus on the air quality model’s performance with respect to individual components of 

fine particulate matter (PM2.5), as good model performance of the PM component species will 

dictate good model performance of total or reconstituted fine particulate matter (i.e., visibility). 

The FRM networks only collects total PM2.5 mass and these monitors are almost exclusively 

located in urban areas so are not as relevant for judging how well the model is predicting the 

components of light extinction at the more rural Class I areas. However, model performance of 

the total fine particulate matter and the resulting total light extinction will also be provided as a 

means to discuss the overall model performance. 

B.2 CMAQ Evaluation Methodology 

EPA’s integrated ozone, PM2.5 and regional haze modeling guidance calls for a comprehensive, 

multi-layered approach to model performance testing, consisting of the four major components: 

operational, diagnostic, mechanistic (or scientific) and probabilistic (EPA, 2007a). 

B.2.1 Evaluation Approach 

The CMAQ model performance evaluation effort focused on the first two components, namely: 

• Operational Evaluation: Tests the ability of the model to estimate PM concentrations 

(both fine and coarse) and the components at PM10 and PM2.5 including the quantities 

used to characterize visibility (i.e., sulfate, nitrate, ammonium, organic carbon, elemental 

carbon, other PM2.5, and coarse matter (PM2.5-10). This evaluation examines whether the 

measurements are properly represented by the model predictions but does not necessarily 

ensure that the model is getting “the right answer for the right reason”; and 

• Diagnostic Evaluation: Tests the ability of the model to predict visibility and extinction, 

PM chemical composition including PM precursors (e.g., SOx, NOx, and NH3) and 

associated oxidants (e.g., ozone and nitric acid); PM size distribution; temporal variation; 

spatial variation; mass fluxes; and components of light extinction (i.e., scattering and 

absorption). 
 B-3 

The diagnostic evaluation also includes the performance of diagnostic tests to better understand 

model performance and identify potential flaws in the modeling system that can be corrected. 

The diagnostic evaluation may also includes the use of “probing tools” to understand why the 

model obtains a given prediction; probing tools include Process Analysis (PA), decoupled direct 

method (DDM) and source apportionment (SA). 

In this final model performance evaluation for the VISTAS 2002 Actual Base G2 CMAQ 36/12 

km base case simulation, the operational evaluation has been given the greatest attention since 

this is the primarily thrust of EPA’s modeling guidance. However, we have also examined 

certain diagnostic features dealing with the model’s ability to simulate sub-regional and 

monthly/diurnal gas phase and aerosol concentration distributions. In the course of the VISTAS 

study numerous diagnostic sensitivity tests were performed to investigate and improve model 

performance. Key diagnostic tests performed are discussed and the results for the rest are 

available on the VISTAS modeling website: http://pah.cert.ucr.edu/vistas/vistas2/. 

B.2.2 Particulate Matter and Component Species 

Regional haze is calculated by estimating light scattering and absorption by components of 

PM2.5. Regional haze is measured by an extinction coefficient (bext), which represents the light 

attenuation resulting from scattering and absorption of light from ambient particulate matter plus 

scattering of light due to gas molecules in the air (i.e., Rayleigh scattering). 

Some scattering does occur from air molecules; however, the bulk of light extinction is caused 

by the presence of suspended particles, particularly fine particles (aerodynamic mass <2.5 μm). 

Fine particulate matter can be composed of varying amounts of several different species, 

including: 

• Sulfate (SO4) 

• Nitrate (NO3) 

• Ammonium (NH4) 

• Organic Matter Carbon (OMC) 

• Elemental Carbon (EC) 

• Coarse particulate matter or Coarse Mass (i.e., PM10 - PM2.5 or CM). 

• Soil (also known as crustal material, fine soil, major metal oxides, inorganic particulates, 

or other PM) 

• Sea Salt (NaCl) 

B.2.3 Ambient Air Quality Data for VISTAS Model Evaluation 

A ground-level model evaluation database for 2002 was compiled by the modeling team using 

several routine and research-grade databases. The focus of the regional evaluation of the CMAQ 

model for use in simulating visibility in the southeastern is on the PM components that can cause 

visibility impairment. The primary monitoring networks available to evaluate this component of 

the CMAQ are: (a) Interagency Monitoring of Protected Visual Environments (IMPROVE); (b) 
 B-4 

Clean Air Status and Trends Network (CASTNET); (c) Southeastern Aerosol Research and 

Characterization (SEARCH); (d) EPA Federal Reference Method PM2.5 and PM10 Mass 

Networks (EPA-FRM); (e) EPA Speciation Trends Network (STN) of PM2.5 species; and (f) 

National Acid Deposition Network (NADP). These PM monitoring networks may also provide 

ozone and other gas phase precursors and product species, and visibility measurements at some 

sites. There are also gas-phase criteria pollutant measures from the EPA’s Air Quality System 

(AQS) that are available for use. However, these sites are urban-oriented and not as useful for 

the regional evaluation of the CMAQ modeling system. The study team has performed a 

detailed evaluation of the CMAQ 2002 base case simulation using the AQS that is available on 

the modeling website: http://pah.cert.ucr.edu/vistas/vistas2/mpe2.shtml. Table B-1 and Figure 

B-1 summarizes the species collected and locations of the monitoring sites for the IMPROVE, 

STN, CASTNet, NADP and SEARCH monitoring networks use in the VISTAS model 

evaluation. 

Table B-1. Ambient monitoring data available in the VISTAS region during 2002. 

Monitoring 

Network Chemical Species Measured 

Sampling Frequency; 

Duration 

IMPROVE Speciated PM2.5 and PM10 1 in 3 days; 24 hr 

CASTNET Speciated PM2.5, Ozone Hourly, Weekly; 1 hr, Week 

SEARCH 

24-hr PM25 (FRM Mass, OC, BC, SO4, NO3, 

NH4, Elem.); 24-hr PM coarse (SO4, NO3, 

NH4, elements); Hourly PM2.5 (Mass, SO4, 

NO3, NH4, EC, TC); and Hourly gases (O3, 

NO, NO2, NOy, HNO3, SO2, CO) Daily, Hourly; 

NADP WSO4, WNO3, WNH4 Weekly 

EPA-FRM Only total fine mass (PM2.5) 1 in 3 days; 24 hr 

EPA-STN Speciated PM2.5 Varies; Varies 

AIRS/AQS CO, NO, NO2, NOx, O3 Hourly; Hourly 
 B-5 

250 500 750 1000 1250 1500 1750 2000

-1500

-1250

-1000

-750

-500

-250

0

250

500

IMPROVE

CASTNet

SEARCH

STN

NADP

AQS

Figure B-1. IMPROVE, STN, CASTNet, NADP and SEARCH monitoring sites located in the 

VISTAS region and the VISTAS 12 km grid. 

B.3 Model Performance Statistics 

To quantify model performance, several statistical measures were calculated and evaluated for 

all the monitors within the VISTAS region and within each VISTAS state. The statistical 

measures selected were based on the recommendations outlined in section 18.4.1 of the 

USEPA’s Guidance On The Use Of Models And Other Analyses for Demonstrating Attainment 

of Air Quality Goals for Ozone, PM2.5, and Regional Haze (EPA, 2007a). In 2004, VISTAS 

established model performance goals and criteria for components of fine particle mass based on 

previous model performance for ozone and fine particles. EPA modeling guidance for fine 

particulate matter at the time noted that PM models might not be able to achieve the same level 

of performance as ozone models. VISTAS’ evaluation considered several statistical 

performance measures and displays. 
 B-6 

B.4 Modeling Performance Across the VISTAS Region 

In the following discussions we use selected monthly scatter plots, time series plots and model 

performance statistical measures from the UCR Analysis Tool application to the 2002 CMAQ 

Base G2 Actual base case simulation in an operational evaluation of the model for PM species. 

We focus on the six main components of PM that are used to project visibility. 

B.4.1 Sulfate (SO4) Monthly Model Performance in the VISTAS Region 

B.4.1.1 SO4 in January 2002 

Figure B-2a displays scatter plots of predicted and observed SO4 concentrations or wet 

depositions for January 2002, sites in the VISTAS region using observations from the 

IMPROVE, STN, CASTNet and NADP monitoring networks and the CMAQ Base G2 Actual 

base case 36 km (red) and 12 km (blue) modeling results. The IMPROVE and STN SO4 

concentrations are 24-hour averages whereas the CASTNet SO4 concentrations and NADP SO4 

wet deposition are weekly averages. The January SO4 performance at the IMPROVE and STN 

networks in the VISTAS region is quite good with low fractional bias (-14%) and gross error 

(34%-38%) that almost achieves the most stringent model performance evaluation goals (Table 

C-1XXX). There is a larger SO4 concentration underestimation bias across the CASTNet 

network (-21% to -23%) with less scatter so there is lower gross error (30% to 32%). The lower 

scatter and error is likely due to the longer sampling period for CASTNet (weekly) versus 

IMPROVE/STN networks (24-hour) that reduces the variability in both the predicted and 

observed SO4 concentrations. Surprisingly the model exhibits low SO4 wet deposition bias 

across the VISTAS region (-7% to -8%) but lots of scatter so produces a high bias than for the 

concentrations (67% to 70%). This is surprising because obtaining good SO4 model 

performance requires the simultaneous occurrence of correct depiction of SO4 concentrations 

and precipitation model predictions. As noted by Olerud (2003c,d), the MM5 simulation does a 

respectable job in reproducing precipitation during synoptic rain events as occur in the winter, 

however it exhibited less skill in predicting precipitation during the summer convective events. 

Figure B-2b present quantile-quantile (Q-Q) plots for January 2002 SO4 concentration and wet 

deposition model performance. Q-Q plots compare the frequency distribution between the 

predicted and observed SO4 concentrations/depositions. Across the IMPROVE network the 

model slightly underestimates the frequency distribution of the lower SO4 below approximately 

5 ug/m3 and overpredictions the values above this value. Across the more urban STN network 

the model exhibits a slight underprediction of the observed frequency distribution across the 

VISTAS region during January 2002. Across the mixed urban/non-urban SEARCH network the 

model reproduces the observed frequency distribution of SO4 concentrations quite well, except 

for overestimating the highest values above approximately 4 ug/m3. Finally, the observed wet 

SO4 deposition in the VISTAS region and January 2002 is matched exactly by the model except 

for the very highest events that are underpredicted. It is unclear whether the model 

overestimation of the highest observed concentrations and underestimation of the highest 

observed wet deposition events are connected.
 B-7 

Figure B-2a. Scatter plots of predicted and observed sulfate (SO4) concentrations for January 2002 

and sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-8 

Figure B-2b. Q-Q plots of predicted and observed sulfate (SO4) concentrations or depositions for 

January 2002 and sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), 

SEARCH (bottom left) and NADP monitoring networks and the CMAQ 2002 12 km Base G2 Actual 

base case simulation
 B-9 

B.4.1.2 SO4 in April 2002 

In April CMAQ still exhibits reasonable good SO4 model performance although with an 

underestimation bias of -20% to -22% across the IMPROVE and STN networks. The scatter in 

the SO4 performance across these two networks is very tight with low fractions error values of 

30% to 35% (Figure B-3a). Across the CASTNet network in April 2002, both the 36 km and 12 

km versions of the model exhibits a higher underestimations bias (-25% to -26%), but low error 

(27% to 28%), which is due to the less variability in the weekly samples. Wet SO4 deposition 

performance in April across the VISTAS region is much worse than seen for January with an 

overprediction bias of 48% (12 km) and 50% (36 km) and fractions errors of approximately 

100%. The Q-Q plots of SO4 concentrations and deposition model performance in April shown 

in Figure C-3b indicates that the model does a reasonable job in reproducing the observed 

frequency distribution, although with a slight underprediction bias except for the highest values 

that are either reproduced fairly well (IMPROVE and STN) or underpredicted (SEARCH and 

NADP).. 
 B-10 

Figure B-3a. Scatter plots of predicted and observed sulfate (SO4) concentrations for April 2002 

and sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-11 

Figure B-3b. Q-Q plots of predicted and observed sulfate (SO4) concentrations for April 2002 and 

sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) 

and NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-12 

B.4.1.3 SO4 in July 2002 

The SO4 model performance in July 2002 across the VISTAS region is also fairly good. The 

CMAQ Base G2 12 km model simulation produces slight lower fractions bias than the 36 km 

model for the IMPROVE (-18% vs. -20%), STN (-13% vs. -16%) and CASTNet (-17% vs. -

23%) networks (Figure B-4a). The fractional error for July SO4 concentrations across these 

same networks ranges from 27% to 48% with the 12 km model exhibiting slight lower error than 

the 36 km model. The July SO4 wet deposition exhibits an overestimation bias of 39% to 45% 

and fractions errors of ~90%. The overestimation of the July 2002 SO4 wet deposition in the 

VISTAS region is consistent with the MM5 meteorological model evaluation that found that 

MM5 overstated the amount of summer precipitation due to convective activity. 

The model reproduces the observed frequency distribution of SO4 concentrations across the 

IMPROVE monitors in the VISTAS region during July 2002 very well (Figure B-4b). The Q-Q 

plots of July SO4 performance for the STN and CASTNet monitoring network across the 

VISTAS region also exhibits fairly good model performance, although with an underestimation 

bias for STN and overestimation bias for SEARCH at the high end. The observed frequency 

distribution of SO4 wet deposition across the VISTAS region is greatly overestimated by the 

model. This summer wet SO4 overestimation bias undoubtedly contributes to the 

underprediction of the SO4 concentrations and extinction for the Worst 20 percent days as 

discussed in Appendix C. 
 B-13 

Figure B-4a. Scatter plots of predicted and observed sulfate (SO4) concentrations for July 2002 and 

sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-14 

Figure B-4b. Q-Q plots of predicted and observed sulfate (SO4) concentrations for July 2002 and 

sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-15 

B.4.1.4 SO4 in October 2002 

In October 2002, CMAQ is doing a better job of reproducing the observed SO4 concentrations 

with much lower fractional bias values across the IMPROVE (0% to 4%) and STN (-6% to -

15%) and fractional errors of 33% to 48% (Figure C-5a). Good SO4 concentrations model 

performance for October 2002 and the VISTAS region is also seen across the CASTNet monitors 

with zero bias and fractions error of approximately 30% (i.e., meeting the most stringent ozone 

model performance goals). Although the SO4 wet deposition model performance bias is fairly 

low (~20%) there is lots of scatter producing large errors (~85%). 

The Q-Q plot comparisons (Figure C-5b) indicate that the model overstates both the SO4 

concentrations and deposition distribution in October 2002 with the overstatement increasing at 

the higher values. 
 B-16 

Figure B-5a. Scatter plots of predicted and observed sulfate (SO4) concentrations for October 2002 

and sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-17 

Figure B-5b. Q-Q plots of predicted and observed sulfate (SO4) concentrations for October 2002 

and sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) 

and NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-18 

B.4.1.5 SO4 Monthly Bias and Error 

Figure B-6a compares the monthly SO4 concentration fractional bias and error across the 

VISTAS region for the IMPROVE, STN, CASTNet, SEARCH Daily and SEARCH Hourly 

monitoring networks. Note that these monthly model performance statistics were calculated 

across the VISTAS 12 km grid versus just the VISTAS states for used in the monthly 

performance evaluation in Figures B-2 through B-5. The underprediction bias in SO4 

concentrations is clearly evident across most of the year and monitoring networks with the 

lowest SO4 bias in October. However, this underprediction bias is not severe usually not going 

lower than -20% and never going lower than -25%. The exception to this is the SEARCH 

Hourly network that exhibits a positive overprediction SO4 bas for several months. 

The SO4 monthly fractional error values are usually under 40% and, with one exception, always 

under 50% (Figure B-6a). The exception is the SEARCH Hourly network with error values 

ranging from 50% to 70%. 

Figure B-6b displays Bugle Plots of monthly SO4 fractional bias and error. Bugle Plots compare 

the monthly bias and error values with a performance goal and performance criteria and accounts 

for the concept that better performance is needed (i.e., more stringent model performance goals 

and criteria) for species with higher concentrations/extinctions and that performance goals and 

criteria can be relaxed as the species concentrations become low and are not longer an important 

component of the PM or extinction budget (see Chapter 3 and Appendix C for more discussion). 

The Bugle Plots in Figure B-6b confirm the previous analysis that SO4 model performance tends 

to be good across all months and, with one exception, always achieving the model performance 

goal for both bias and error. The one exception is for fractional error across the SEARCH 

Hourly network that is above the performance goal, but still achieves the performance criteria 

(Figure B-6b, bottom). 
 B-19 

Figure B-6a. Monthly SO4 fractional bias (top) and fractional gross error (bottom) statistical 

measures for CASTNet, IMPROVE, STN and SEARCH (daily and hourly) monitoring sites in the 

VISTAS 12 km grid. 
 B-20 

Figure B-6b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for SO4 in 

the VISTAS 12 km grid and comparison to model performance goals (blue) and criteria (yellow). 
 B-21 

B.4.2 Nitrate (NO3) Monthly Model Performance in the VISTAS Region 

The regional model performance evaluation for nitrate (NO3) follows the same format as given 

above for SO4. 

B.4.2.1 NO3 in January 2002 

The January NO3 performance (Figure B-7a) is not as good as seen for SO4. The January NO3 

fractional bias is in the -20% to -25% and very similar for the 36 km and 12 km CMAQ Base G2 

simulations for the STN and CASTNet monitoring networks across the VISTAS region, with 

fractional errors that range from 30% to 37%. However, across the IMPROVE network in the 

VISTAS region the CMAQ 12 km run exhibits a higher underprediction bias (-39%) than the 36 

km run (-23%), although it has lower error (90%) than the 36 km run (96%). The January NO3 

wet deposition performance for the 36 km and 12 km simulations are very similar with an 

approximate -7% bias and 36% gross error. 

The Q-Q plots comparing the predicted and observed January NO3 frequency distributions 

indicate that although there is a net underestimation bias in NO3 when averaged across the 

monitoring networks as shown IN Figure B-7a, the model overestimates the highest observed 

NO3 concentrations. This overestimation of the highest observed NO3 concentrations in January 

occurs for the IMPROVE, CASTNet and NADP networks, but not the more urban-oriented STN 

network.
 B-22 

Figure B-7a. Scatter plots of predicted and observed nitrate (NO3) concentrations for January 2002 

and sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-23 

Figure B-7b. Q-Q plots of predicted and observed nitrate (NO3) concentrations for January 2002 

and sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) 

and NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-24 

B.4.2.2 NO3 in April 2002 

The monthly NO3 performance degrades in April 2002 with the model exhibiting an 

underprediction bias of approximately -60% and -160% across the IMPROVE and STN 

monitoring in the VISTAS region and errors of over 100% and 160%, respectively. Better April 

NO3 performance is seen across the CASTNet monitoring network with bias values of -9% and -

18% for the CMAQ 36 km and 12 km base case simulations and errors of approximately 80%. 

April NO3 wet deposition is overestimated by 36% to 31% with errors of approximately 95%. 

Despite the -60% underestimation bias for April monthly NO3 across the IMPEOVE network, 

the Q-Q plot indicates an overestimation bias for the highest NO3 values (Figure B-8b). This is 

because most of the modeled NO3 values are near zero so show up as an underprediction bias for 

the very lowest observed NO3 values in the Q-Q bias, but there are some modeled values higher 

than observed that are accentuated in the Q-Q plot. 
 B-25 

Figure B-8a. Scatter plots of predicted and observed nitrate (NO3) concentrations for April 2002 and 

sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-26 

Figure B-8b. Q-Q plots of predicted and observed nitrate (NO3) concentrations for April 2002 and 

sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-27 

B.4.2.3 NO3 in July 2002 

NO3 monthly model performance in July is characterized by a large underprediction bias with 

fractional bias values of -150% to -160% across the IMPROVE and STN networks and of 

approximately -100% across the CASTNet monitoring network (Figure B-8a), This is due to the 

model most of the time predicting near zero NO3 concentrations in July as the hot temperatures 

in the southeastern U.S. will volatilize the particulate NO3 in the model to gaseous nitric acid. 

Even so, there are occurrences of overestimated model NO3 concentrations in July and for the 

IMPROVE network the highest modeled NO3 values exceed the measured values as shown in 

the Q-Q plot in Figure B-8b. 

Figure B-9a. Scatter plots of predicted and observed nitrate (NO3) concentrations for July 2002 and 

sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-28 

Figure B-9b. Q-Q plots of predicted and observed nitrate (NO3) concentrations for July 2002 and 

sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) and 

NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-29 

B.4.2.4 NO3 in October 2002 

The NO3 scatter plots for October 2002 suggest that the model is overestimating the observed 

NO3 but there is a net negative fraction bias of approximately -30% to 0-40% for the IMPROVE 

and STN networks and the VISTAS 12 km grid (Figure B-10a). This negative bias is due to the 

model predicting very low near zero NO3 concentrations when the observed NO3 is less than 

approximately a 0.5 ug/m3. However, when the observed value is greater than approximately 

0.5 ug/m3 at the IMPROVE and STN networks it appears that the model mostly overstates the 

observed value. Across the CASTNet network a positive bias is seen (32% to 42%) which may 

be due in part to volatilization of the particulate NO3 in the weekly sample. The magnitude of 

the NO3 wet deposition observed values is reproduced reasonable well with low bias (6% to 8%) 

but with lots of scatter around the 1:1 line producing errors of approximately 85%. 

The overprediction of the highest observed NO3 concentrations and deposition in October is 

clearly evident in the Q-Q plots (Figure B-10b). 
 B-30 

Figure B-10a. Scatter plots of predicted and observed nitrate (NO3) concentrations for October 

2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), CASTNet (bottom 

left) and NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-31 

Figure B-10b. Q-Q plots of predicted and observed nitrate (NO3) concentrations for October 2002 

and sites in the VISTAS region using IMPROVE (top left), CASTNet (top right), SEARCH (bottom left) 

and NADP monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-32 

B.4.2.5 NO3 Monthly Bias and Error 

The monthly fractional bias time series plot clearly shows the seasonal dependence of this 

performance measure with severe summer underprediction bias of -100% to -150% in the 

summer and bias values generally within ±50% in the winter (Figure B-11a). The time series of 

fraction error values also exhibit a seasonal dependence with an 80% error in the winter peaking 

to a 160% error in the summer with a bell-like distribution. At first glance it appears that the 

summer underprediction bias would adversely affect the regional haze modeling as that is when a 

vast majority of the worst 20 percent days occur at Class I areas in the VISTAS region. 

However, as shown in the Bugle Plots in Figure B-11b, these large summer underprediction bias 

values occur when NO3 concentrations are low (< 1.0 ugm3) and in fact when the average 

observed NO3 concentration exceeds 1 ug/m3 the model mostly achieves the model performance 

goal and almost always achieves the model performance criteria for bias (Figure B-11a top). The 

model performance goal and criteria is achieved less often for fractional error and NO3 (Figure 

B-11b, bottom), but the goal is always achieved across the CASTNet and usually achieved across 

the IMPROVE monitoring networks. 
 B-33 

Figure B-11a. Monthly NO3 fractional bias (top) and fractional gross error (bottom) statistical 

measures for CASTNet, IMPROVE, STN and SEARCH daily and hourly monitoring sites in the 

VISTAS region. 
 B-34 

Figure B-11b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for NO3 in 

the VISTAS region and comparison to model performance goals (blue) and criteria (yellow). 
 B-35 

B.4.3 Organic Matter Carbon (OMC) Monthly Model Performance in the VISTAS Region 

The CMAQ model simulates organic matter carbon (PMC) that includes both primary and 

secondary organic aerosol (OA) that includes carbon, oxygen and other compounds. However, 

the monitoring networks measure just the organic carbon (OC) portion of the OMC. Thus, the 

OC measurements need to be converted to OMC for the model evaluation. OMC/OC ratios tend 

to vary between 1.2 to 2.2 depending on the organic compounds and the amount of 

photochemistry the compounds have undergone, with newer OMC tending to have lower 

OMC/OC ratios than aged OMC. The original IMPROVE equation uses a 1.4 OMC/OC ratio, 

whereas the new IMPROVE equation uses a 1.8 OMC/OC ratio. In this model performance 

evaluation we used a 1.4 OMC/OC ratio to convert the measured OC to OMC. There is also 

concern with the STN OC measurements because they are not black corrected. In fact the 

differences in OC sampling methodologies results in differences in measurements that exceeds 

our performance goals (Solomon et al., 1994). Thus, these uncertainties in the OMC 

measurements need to be accounted for in the model performance evaluation. 

B.4.3.1 OMC in January 2002 

In January the CMAQ 12 km and 36 km Base G2 2002 Actual base case simulation exhibit a 

11% and 18% overprediction bias across the rural IMPROVE monitors and a -40% and -46% 

underprediction bias across the urban STN monitors in the VISTAS region with errors of 

approximately 50% (Figure B-12a). For the mixed urban/rural SEARCH monitoring network an 

OMC underprediction bias of -24% and -16% occurs using the 12 km and 36 km modeling 

results with errors of 40% to 50%. The SEARCH monitors also collected hourly measurements 

of total carbon mass (TCM=EC+OC) which was overestimated by the 12 km and 36 km 

modeling results by 20% and 12% with errors of approximately 70%. The Q-Q plots indicate the 

model is reproducing the low end of the observed OMC frequency distributions fairly well across 

the IMPROVE and STN networks but overestimates the high end, whereas the high end is 

underestimated across the SEARCH network. 
 B-36 

Figure B-12a. Scatter plots of predicted and observed organic matter carbon (OMC) concentrations 

for January 2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and 

SEARCH Daily (bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH 

Hourly network (bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-37 

Figure B-12b. Q-Q plots for January 2002 and the VISTAS 12 km grid of predicted and observed 

organic matter carbon (OMC) and elemental carbon (EC) across the IMPROVE network (top left and 

right) and SEARCH Daily OCM (bottom left) and SEARCH Hourly total carbon mass (TCM) (bottom 

right) using the CMAQ 2002 12 km Base G2 Actual base case simulation. 
 B-38 

B.4.3.2 OMC in April 2002 

In April 2002, the CMAQ Base B2 modeling results has a near zero bias across the IMPROVE 

monitors with a 39% error, and a -59% (12 km) and -54% (36 km) underprediction bias across 

the STNB monitors. There is also very low OMC bias across the SEARCH network in April, 

although the errors are higher (~60%). The SEACH hourly TCM also shows low OMC bias in 

April (11%-16%) and errors of 60%. The April OMC Q-Q plots are similar to January with the 

model overstating the observed OMC frequency distribution at the high end for IMPRVE and 

STN, but understating it for SEARCH. 

Figure B-13a. Scatter plots of predicted and observed organic matter carbon (OMC) 

concentrations for April 2002 and sites in the VISTAS region using IMPROVE (top left), STN (top 

right), and SEARCH Daily (bottom left) monitoring networks and for total carbon mass (TCM) for 

the SEARCH Hourly network (bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-39 

Figure B-13b. Q-Q plots for January April and the VISTAS 12 km grid of predicted and observed 

organic matter carbon (OMC) and elemental carbon (EC) across the IMPROVE network (top left 

and right) and SEARCH Daily OCM (bottom left) and SEARCH Hourly total carbon mass (TCM) 

(bottom right) using the CMAQ 2002 12 km Base G2 Actual base case simulation. 
 B-40 

B.4.3.3 OMC in July 2002 

In July the model underestimates OMC across the IMPOREVE (-13%), STN (-54% to -59%) 

and SEARCH networks (-34% to -40%) monitoring networks in the VISTAS states (Figure B14a). The scatter plots show that the model is exhibiting some skill in predicting the observed 

OMC, albeit with an underprediction bias. Unlike January and April, the observed high end of 

the OMC measurements is underestimated by the model across all three networks. 

Figure B-14a. Scatter plots of predicted and observed organic matter carbon (OMC) concentrations 

for July 2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and 

SEARCH Daily (bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH 

Hourly network (bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-41 

Figure B-14b. Q-Q plots for July 2002 and the VISTAS 12 km grid of predicted and observed 

organic matter carbon (OMC) and elemental carbon (EC) across the IMPROVE network (top left and 

right) and SEARCH Daily OCM (bottom left) and SEARCH Hourly total carbon mass (TCM) (bottom 

right) using the CMAQ 2002 12 km Base G2 Actual base case simulation. 
 B-42 

B.4.3.4 OMC in October 2002 

In October the model exhibits near zero bias across the IMPROVE, large underprediction bias 

across STN and smaller underprediction bias across the SEARCH networks (Figure B-15a). The 

highest observed OMC frequency distributions are overstated at the IMPROVE and STN but 

matched across the SEARCH monitors (Figure B-15b) 

Figure B-15a. Scatter plots of predicted and observed organic matter carbon (OMC) concentrations 

for October 2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and 

SEARCH Daily (bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH 

Hourly network (bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-43 

Figure B-15b. Q-Q plots for October 2002 and the VISTAS 12 km grid of predicted and observed 

organic matter carbon (OMC) and elemental carbon (EC) across the IMPROVE network (top left and 

right) and SEARCH Daily OCM (bottom left) and SEARCH Hourly total carbon mass (TCM) (bottom 

right) using the CMAQ 2002 12 km Base G2 Actual base case simulation. 
 B-44 

B.4.3.5 OMC Monthly Bias and Error 

The monthly fractional bias and error plot for the 12 km CMAQ base case simulation in Figure 

B-16a displays fairly good performance across the IMPROVE monitors with bias usually within 

±15% and errors between 40% and 50%. However, at the urban STN sites the model exhibits a 

large underprediction bias of -35% to -60%. Part of the underprediction bias may be due to 

measurement uncertainties (e.g., blank corrections) and part may be due to the model over 

diluting the urban OMC emissions through the coarse 36/12 km grid. The underprediction of 

urban OMC is a common problem in PM modeling and likely also points to uncertainties in the 

OMC and SOA precursor emission inventories. 

The Bugle Plots of monthly fractional bias and gross error (Figure B-16b) concurs that the OMC 

performance across the IMPROVE monitoring sites is fairly good achieving the model 

performance goals. But the OMC underprediction bias for the STN monitors is significant 

enough so that always falls out of the model performance goals and is right at the model 

performance criteria. The OMC performance across the SEARCH network is mixed usually 

achieving the model performance goal and always achieving the model performance criteria. 
 B-45 

Figure B-16a. Monthly OMC fractional bias (top) and fractional gross error (bottom) statistical 

measures for IMPROVE, STN and SEARCH monitoring sites in the VISTAS region. 
 B-46 

Figure B-16b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for OMC in 

the VISTAS region and comparison to model performance goals (blue) and criteria (yellow). 
 B-47 

B.4.4 Elemental Carbon (EC) Monthly Model Performance in the VISTAS Region 

Elemental carbon is measured at the IMPROVE, STN and SEARCH monitoring networks. 

SEARCH included 24-hur and hourly TCM measurements. In the OMC performance plots in 

Figures B-12 through B-15 the SEARCH hourly TCM results were plotted. In the EC 

performance plots in Figures B-17 through B-20 we display the SEARCH daily average TCM 

performance. 

B.4.4.1 EC in January 2002 

Elemental Caron (EC) in January 2002 is fairly good with fractions bias < ±15% across the 

IMPROVE and STN networks and errors of ~45% (Figure B-17). A larger underprediction bias 

and higher errors are seen across the SEARCH network for OMC and TCM. 
 B-48 

Figure B-17. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

January 2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and 

SEARCH Daily (bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH 

Daily network (bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-49 

B.4.4.2 EC in April 2002 

Reasonable performance is seen for OMC in April, but with an underprediction bias that ranges 

from -30% to -50% across the IMPORVE and STN monitoring networks in the VISTAS region 

with errors of 50% to 60% (Figure B-18). The OMC and TCM performance across the 

SEARCH network in April exhibits near zero bias but with a lot of scatter resulting in errors on 

~60%. 

Figure B-18. Scatter plots of predicted and observed elemental carbon (EC) concentrations for April 

2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and SEARCH Daily 

(bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH Daily network 

(bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-50 

B.4.4.3 EC in July 2002 

EC is underestimated in July 2002 in the VISTAS region across all monitoring networks. With 

the underestimation bias greater at the more rural IMPROVE (-41% and -53%) than urban STN 

(2% and -23%) networks (Figure B-19). It is interesting to note that EC model performance at 

the urban STN and partially urban SEARCH networks is much better for the 12 km modeling 

results (bias of 2% and -60%) than the 36 km results (-23% and -86%). 

Figure B-19. Scatter plots of predicted and observed elemental carbon (EC) concentrations for July 

2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and SEARCH Daily 

(bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH Daily network 

(bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-51 

B.4.4.4 EC in October 2002 

The EC underprediction bias also occurs in October (Figure B-20). Again lower bias is seen at 

the STN networks with the 12 km results (-10%) producing lower bias than the 36 km results (-

31%). 

Figure B-20. Scatter plots of predicted and observed elemental carbon (EC) concentrations for 

October 2002 and sites in the VISTAS region using IMPROVE (top left), STN (top right), and 

SEARCH Daily (bottom left) monitoring networks and for total carbon mass (TCM) for the SEARCH 

Daily network (bottom right) using the CMAQ 2002 36 and 12 km Base G2 Actual base case 

simulation. 
 B-52 

B.4.4.5 EC Monthly Bias and Error 

The monthly bias and error performance statistics for EC across the VISTAS region are shown in 

Figure B-21a. Bias and error are lower in the first quarter of the year, and then the IMPROVE 

and SEARCH networks begin to exhibit an underprediction bias that peaks in the summer and 

gets lower again with the onset of winter. Surprisingly, the STN network exhibits very low EC 

bias during the summer to fall. 

The Bugle Plot in Figure B-21b shows that although the EC underprediction produces large 

fractional bias values, because the EC concentrations are low the EC performance achieves the 

model performance goal for all months and networks since it is in the flared horn section of the 

Bugle Plot. 
 B-53 

Figure B-21a. Monthly EC fractional bias (top) and fractional gross error (bottom) statistical 

measures for IMPROVE, STN and SEARCH daily and hourly monitoring sites in the VISTAS 

region. 
 B-54 

Figure B-21b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for EC in 

the VISTAS region and comparison to model performance goals (blue) and criteria (yellow). 
 B-55 

B.4.5 Other PM2.5 (Soil) Monthly Model Performance in the VISTAS Region 

There are potential model-measurement incommensurability issues with the Soil model 

performance. The IMPROVE and SEARCH networks builds up Soil from the elements, whereas 

in the model Soil is mapped to the other PM2.5 category that is emissions not identified as SO4, 

NO3, OMC or EC. 

B.4.5.1 Soil in January, April, July and October 2002 

Soil model performance is poor in January with a systematic overprediction bias of 134% to 

157% with similar magnitude of errors (Figure B-22). The overprediction bias is not as large in 

April and no longer systematic (i.e., doesn’t occur across all predicted-observed pairs) with 

fraction bias values of ~20% (IMPROVE) and ~80% (SEARCH) (Figure B-23). Higher 

observed values in the measured Soil values in July lowers the model overprediction bias to 

approximately 40% across the SEARCH network with a -30% underprediction bias across the 

IMPORVE monitors (Figure B-24). The overprediction bias returns in October with fractional 

bias of ~100% (IMPROVE) to ~140% (SEARCH). 
 B-56 

Figure B-22. Comparison of predicted and observed other PM2.5 (Soil) concentrations for January 

2002 and sites in the VISTAS region using scatter plots (top) and Q-Q plots (bottom) and 

observations from the IMPROVE (left) and SEARCH Daily (left) monitoring networks for the CMAQ 

2002 12 km Base G2 Actual base case simulation. 
 B-57 

Figure B-23. Comparison of predicted and observed other PM2.5 (Soil) concentrations for April 2002 

and sites in the VISTAS region using scatter plots (top) and Q-Q plots (bottom) and observations 

from the IMPROVE (left) and SEARCH Daily (left) monitoring networks for the CMAQ 2002 12 km 

Base G2 Actual base case simulation. 
 B-58 

 

Figure B-24. Comparison of predicted and observed other PM2.5 (Soil) concentrations for July 2002 

and sites in the VISTAS region using scatter plots (top) and Q-Q plots (bottom) and observations 

from the IMPROVE (left) and SEARCH Daily (left) monitoring networks for the CMAQ 2002 12 km 

Base G2 Actual base case simulation. 
 B-59 

 

Figure B-25. Comparison of predicted and observed other PM2.5 (Soil) concentrations for October 

2002 and sites in the VISTAS region using scatter plots (top) and Q-Q plots (bottom) and 

observations from the IMPROVE (left) and SEARCH Daily (left) monitoring networks for the CMAQ 

2002 12 km Base G2 Actual base case simulation. 
 B-60 

B.4.5.2 Soil Monthly Bias and Error 

The seasonal dependence of the Soil model performance across the IMPROVE and STN 

monitors in the VISTAS region is clearly evident in the monthly bias time series plot in Figure 

B-26a. As noted in the scatter plots in Figure B-22 through B-25, the modeled Soil values tend 

to be between 0 and 2 μg/m3

 year round, however the observed values are much lower in the 

winter (0 to 0.5 μg/m3) and comparable to the modeled values in the summer. These results 

suggest that the poor Soil model performance in the winter, spring and fall is likely due to 

incorrect emission temporal adjustment factors. For example, the effects of wetted surfaces on 

fugitive dust emissions may not be properly characterized in the emissions inventory. 

The Bugle plots of bias and error also display the seasonal dependence of the Soil model 

performance with the model achieving the model performance goals for Soil at the IMPROVE 

monitoring network for approximately half the year and not achieving both the model 

performance goals and criteria at the IMPROVE network for the other half (Figure B-26b). 

Performance for Soil across the SEARCH network is worse with the model not achieving either 

the performance goals or criteria for most months. 
 B-61 

Figure B-26a. Monthly Soil fractional bias (top) and fractional gross error (bottom) statistical 

measures for IMPROVE and SEARCH Daily monitoring sites in the VISTAS region. 
 B-62 

Figure B-26b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for Soil in 

the VISTAS region and comparison to model performance goals (blue) and criteria (yellow). 
 B-63 

B.4.6 Coarse Mass (CM) and PM10 Mass Monthly Model Performance in the VISTAS 

Region 

Coarse Mass (CM) measurements are defined as the difference between the PM10 mass and 

PM2.5 mass measurements. Since IMPROVE network only speciates the PM2.5 fraction of PM, 

then any coarse SO4 or NO3 would be in the CM measurement. In the model, on the other hand, 

CM is due solely to primary PM emissions that have been speciated to the coarse fractions 

(PM2.5-10). Thus, there are potential incompatibility issues with the measured and modeled CM 

concentrations. 

In addition, CM has a much shorter transport distances than fine PM. In the modeling this is 

accounted for though fugitive dust transport factors (FDTF) that adjust the fugitive dust 

emissions downward (by ~75% on average in the VISTAS region) to account for the fact that 

these emissions are deposited locally and are not transported out of the grid cell where they are 

emitted. However, such local dust emissions can impact an IMPROVE monitor. Thus, much of 

the measured CM at the IMPROVE monitors is likely local in nature and subgrid-scale to the 

model. 

B.4.6.1 CM and PM10 in January, April, July and October 2002 

Figures B-27 through B30 display scatter plots of predicted and observed 24-hour CM and PM10

concentrations across the IMPROVE monitors in the VISTAS region for, respectively, January, 

April, July and October. CM exhibits an underprediction bias of approximately -50% in January 

and approximately -100% for the other three months with errors from 100% to 130%. The PM 

10 performance in January across the IMPORVE monitors is quite good with near zero bias and 

~30% error so achieves the most stringent ozone model performance goals (Figure B-27). 

However, an underprediction bias becomes larger in April (-35%) and even larger in July (-60%) 

only to come back to fairly good model performance again in October. 

Figure B-27. Scatter plots of predicted and observed coarse mass (CM) (left) and PM10 (right) 

concentrations for January 2002 and observations from the IMPROVE monitoring network for the 

CMAQ 2002 12 km Base G2 Actual base case simulation. 
 B-64 

Figure B-28. Scatter plots of predicted and observed coarse mass (CM) (left) and PM10 (right) 

concentrations for April 2002 and observations from the IMPROVE monitoring network for the CMAQ 

2002 12 km Base G2 Actual base case simulation. 

Figure B-29. Scatter plots of predicted and observed coarse mass (CM) (left) and PM10 (right) 

concentrations for July 2002 and observations from the IMPROVE monitoring network for the CMAQ 

2002 12 km Base G2 Actual base case simulation. 
 B-65 

Figure B-30. Scatter plots of predicted and observed coarse mass (CM) (left) and PM10 (right) 

concentrations for October 2002 and observations from the IMPROVE monitoring network for the 

CMAQ 2002 12 km Base G2 Actual base case simulation. 

B.4.6.5 CM Monthly Bias and Error 

Figure B-31a displays the monthly fraction bias and gross error values for CM across the 

IMPROVE monitors in the VISTAS region. The underprediction bias ranges from -60% to -

120% with errors of 80% to 140%. The Bugle Plot (Figure B-31b) shows that these performance 

statistics are either right at or exceeds the performance criteria indicating poor CM model 

performance. 
 B-66 

Figure B-31a. Monthly coarse mass (CM) fractional bias (top) and fractional gross error 

(bottom) statistical measures for IMPROVE monitoring sites in the VISTAS region. 
 B-67 

Figure B-31b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for coarse 

mass in the VISTAS region and comparison to model performance goals (blue) and criteria 

(yellow). 
 B-68 

B.4.7 PM2.5 Mass Monthly Model Performance in the VISTAS Region 

Below we present the CMAQ 2002 Base G2 12km and 36 km model performance for PM2.5

mass. In the model, PM2.5 mass is obtained by summing all of the PM species except CM. In 

the measurement, PM2.5 is obtained by the total PM on a filter collected by a sampler with a 2.5 

micron inlet cut point. Thus, the measured PM2.5 could exclude some SO4 and NO3 if they are 

in the coarse mode that would be included in the modeled PM .5 that assumes all SO4 and NO3 

are fine. 

B.4.7.1 PM2.5 in January 2002 

PM2.5 performance in January is quite good achieving the most stringent 1-hour ozone 

<±15%/<35% 1-hour ozone bias/error performance goal (Figure B-32a). PM2.5 is slightly 

overestimated across the IMPROVE, underestimated across the STN and near zero bias across 

the SEARXCH and FRM monitoring sites. 

Figure B-32a. Scatter plots of predicted and observed PM2.5 mass concentrations for January 2002 

and sites in the VISTAS region using IMPROVE (top left), SEARCH Daily (top right), STN (bottom 

left) and FRM (bottom right) monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-69 

Figure B-32a. (concluded) Scatter plots of predicted and observed PM2.5 mass concentrations for 

January 2002 and sites in the VISTAS region using IMPROVE (top left), SEARCH Daily (top right), 

STN (bottom left) and FRM (bottom right) monitoring networks and the CMAQ 2002 36 and 12 km 

Base G2 Actual base case simulation. 

Figure B-32b. Q-Q plots of predicted and observed PM2.5 mass concentrations for January 2002 and 

sites in the VISTAS region using IMPROVE (left) and SEARCH Daily (right) monitoring networks and 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-70 

B.4.7.2 PM2.5 in April 2002 

In April the PM2.5 model performance is still fairly good with errors in the 30% to 40% range, 

but with an underprediction bias of from -15% (IMPROPVE) to -35% (STN). 

Figure B-33a. Scatter plots of predicted and observed PM2.5 mass concentrations for April 2002 and 

sites in the VISTAS region using IMPROVE (top left), SEARCH Daily (top right), STN (bottom left) 

and FRM (bottom right) monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-71 

Figure B-33b. Q-Q plots of predicted and observed PM2.5 mass concentrations for April 2002 and 

sites in the VISTAS region using IMPROVE (left) and SEARCH Daily (right) monitoring networks and 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-72 

B.4.7.3 PM2.5 in July 2002 

The underprediction bias is higher in July (-35% to -50%) with errors higher as well (45% to 

55%). 

Figure B-34a. Scatter plots of predicted and observed PM2.5 mass concentrations for July 2002 and 

sites in the VISTAS region using IMPROVE (top left), SEARCH Daily (top right), STN (bottom left) 

and FRM (bottom right) monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-73 

Figure B-34b. Q-Q plots of predicted and observed PM2.5 mass concentrations for July 2002 and 

sites in the VISTAS region using IMPROVE (left) and SEARCH Daily (right) monitoring networks and 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-74 

B.4.7.4 PM2.5 in October 2002 

By October the performance has improved with lower bias and errors across all the networks. 

Figure B-35a. Scatter plots of predicted and observed PM2.5 mass concentrations for October 2002 

and sites in the VISTAS region using IMPROVE (top left), SEARCH Daily (top right), STN (bottom 

left) and FRM (bottom right) monitoring networks and the CMAQ 2002 36 and 12 km Base G2 Actual 

base case simulation. 
 B-75 

Figure B-35b. Q-Q plots of predicted and observed PM2.5 mass concentrations for October 2002 and 

sites in the VISTAS region using IMPROVE (left) and SEARCH Daily (right) monitoring networks and 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 

B.4.7.5 PM2.5 Monthly Bias and Error 

Figure B-26a displays the monthly time series of fractional bias and error for PM2.5 mass 

concentrations for the IMPROVE, STN, SEARCH Daily, SEARCH Hourly and FRM networks 

in the VISTAS region. For the first three and last three months of the year the model performs 

well for PM2.5 with low bias (<±20%) and error (<40% except for SEARCH Hourly). During 

April through July the underprediction bias becomes larger and is in the -20% to -45%) and the 

errors are also larger. 

The Bugle Plots confirm the good performance for PM2.5 mass concentrations that achieves the 

model performance goals most of the time and always achieves the model performance criteria 

(Figure B-36b). 
 B-76 

Figure B-36a. Monthly PM2.5 fractional bias (top) and fractional gross error (bottom) statistical 

measures in the VISTAS region. 
 B-77 

Figure B-36b. Bugle Plots of monthly fractional bias (top) and gross error (bottom) for PM2.5 in 

the VISTAS region and comparison to model performance goals (blue) and criteria (yellow). 
 B-78 

B.4.8 Sea Salt in the VISTAS Region 

Figure B-37 compares scatter plots of predicted and observed “Sea Salt” for IMPROVE monitors 

in the VISTAS region and January, April, July and October 2002. The IMPROVE “Sea Salt” is 

inferred from the chlorine ion measurement [Cl-], whereas CMAQ internally generates its Sea 

Salt emissions. The model is predicting near zero Sea Salt resulting in poor performance. As 

Sea Salt is not directly used in the 2018 visibility projections this is not a big cause for concern, 

but can affect aerosol concentrations and is an area that should be investigated further, 

Figure B-37. Scatter plots of predicted and observed Sea Salt concentrations in the VISTAS region 

using IMPROVE monitoring data and for January (top left), April (top right), July (bottom left) and 

October (bottom right) and the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-79 

B.5 Diagnostic Model Evaluation for Gas-Phase and Precursor Species 

The CASTNet and SEARCH networks also measure gas-phase species that are PM precursor or 

related species. The diagnostic evaluation of the 2002 36/12 km Base G2 CMAQ base case 

simulation for these compounds and the four seasonal months presented previously is provided 

below. Note that the EPA Air Quality Stations (AQS) compliance network also provides routine 

measurements of many gas-phase precursor (e.g., NOx and SO2) and product (e.g., O3) species 

as well. However, these sites tend to be urban-oriented and designed to measure maximum 

potential concentrations of criteria pollutants so would not provide a good measure of a model’s 

ability to estimate regional concentrations at the more rural Class I areas. Consequently the gasphase precursor and product species diagnostic evaluation used measured data from the 

CASTNet and SEARCH networks that included some more rural monitoring sites. 

The CASTNet network measures weekly average samples of SO2, SO4, NO2, HNO3, NO3 and 

NH4. Because the CASTNet samples are collected in a cartridge and sampled at a later date 

some of the NO3 may volatize into gaseous HNO3. Thus, we also evaluate against total nitrate 

(TNO3) that is defined as the sum of NO3 and HNO3. The SEARCH Hourly network collects 

hourly measurements of gaseous SO2, NO, NO2, NOy, O3 and CO and particulate SO4, NO3, 

and NH4. A comparison of the SO2 and SO4 performance provides insight into whether the 

SO4 formation rate may be too slow or fast. For example, if SO4 is underestimated and SO2 is 

overestimated, that may indicate too slow chemical conversion rate. Analyzing the performance 

for SO4, HNO3, NO3, Total NO3 and NH4 provides insight into the equilibrium of these 

species. For example, if TNO3 performs well but HNO3 and NO3 do not, then there may be 

issues associated with the partitioning between the gaseous and particle phases of nitrate (e.g., 

ammonia may be off). 
 B-80 

B.5.1 Diagnostic Model Performance in January 2002 

Figure B-38a. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the SEARCH Hourly monitoring network and January 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-81 

Figure B-38b. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the CASTNet monitoring network and January 2002 for the CMAQ 

2002 36 and 12 km Base G2 Actual base case simulation. 
 B-82 

Figure B-39. Scatter plots of predicted and observed gaseous SO2 (left) and particulate SO4 (right) 

from the SEARCH Hourly (top) and CASTNet (bottom) monitoring networks and January 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-83 

Figure B-40. Scatter plots of predicted and observed gaseous NO (top left), NO2 (top right), NOy 

(bottom left) and O3 (bottom right) from the SEARCH Hourly monitoring network and January 2002 

for the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-84 

B.5.2 Diagnostic Model Performance in April 2002 

Figure B-41a. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the SEARCH Hourly monitoring network and April 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-85 

Figure B-41b. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the CASTNet monitoring network and April 2002 for the CMAQ 

2002 36 and 12 km Base G2 Actual base case simulation. 
 B-86 

Figure B-42. Scatter plots of predicted and observed gaseous SO2 (left) and particulate SO4 (right) 

from the SEARCH Hourly (top) and CASTNet (bottom) monitoring networks and April 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-87 

Figure B-43. Scatter plots of predicted and observed gaseous NO (top left), NO2 (top right), NOy 

(bottom left) and O3 (bottom right) from the SEARCH Hourly monitoring network and April 2002 for 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-88 

B.5.3 Diagnostic Model Performance in July 2002 

Figure B-44a. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the SEARCH Hourly monitoring network and July 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-89 

Figure B-44b. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the CASTNet monitoring network and July 2002 for the CMAQ 

2002 36 and 12 km Base G2 Actual base case simulation. 
 B-90 

Figure B-45. Scatter plots of predicted and observed gaseous SO2 (left) and particulate SO4 (right) 

from the SEARCH Hourly (top) and CASTNet (bottom) monitoring networks and July 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-91 

Figure B-46. Scatter plots of predicted and observed gaseous NO (top left), NO2 (top right), NOy 

(bottom left) and O3 (bottom right) from the SEARCH Hourly monitoring network and July 2002 for 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-92 

B.5.4 Diagnostic Model Performance in October 2002 

Figure B-47a. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the SEARCH Hourly monitoring network and October 2002 for the 

CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-93 

Figure B-47b. Scatter plots of predicted and observed NO3 (top left), HNO3 (top right), NH4 (bottom 

left) and TNO3 (bottom right) from the CASTNet monitoring network and October 2002 for the CMAQ 

2002 36 and 12 km Base G2 Actual base case simulation. 
 B-94 

Figure B-48. Scatter plots of predicted and observed gaseous SO2 (left) and particulate SO4 (right) 

from the SEARCH Hourly (top) and CASTNet (bottom) monitoring networks and October 2002 for 

the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation. 
 B-95 

Figure B-49. Scatter plots of predicted and observed gaseous NO (top left), NO2 (top right), NOy 

(bottom left) and O3 (bottom right) from the SEARCH Hourly monitoring network and October 2002 

for the CMAQ 2002 36 and 12 km Base G2 Actual base case simulation.