CMAQ Model Performance Evaluation for 2000 in 

Central California 

Planning and Technical Support Division 

Air Resources Board 

California Environmental Protection Agency 

1001 I Street, Sacramento, CA 94814 

March 12, 2008
ii

DISCLAIMER

The conclusions and recommendations contained in this document are those of 

the authors and do not necessarily reflect the views and policies of the California 

Air Resources Board. The mention of trade names or commercial products does 

not constitute endorsement or recommendation for their use. 
iii

TABLE OF CONTENTS 

DISCLAIMER....................................................................................................................................ii

TABLE OF CONTENTS .................................................................................................................. iii

TABLE OF FIGURES ......................................................................................................................iv

1. Introduction: ............................................................................................................................ 1

2. Regional Air Quality Methodology .......................................................................................... 1

3. Chemical Mechanism.............................................................................................................. 2

4. Model Inputs and Setup.......................................................................................................... 2

4.1. Domain Structure ........................................................................................................... 2

4.2. Initial and Boundary Conditions ..................................................................................... 3

4.3. Emissions ....................................................................................................................... 3

4.4. Meteorological Inputs ..................................................................................................... 4

4.5. MM5 Simulation.............................................................................................................. 4

5. Model Performance Evaluation............................................................................................... 6

5.1. Spatial Patterns for PM mass and components............................................................. 6

5.2. Statistics for Mean Fractional Error (MFE) and Bias (MFB) ........................................ 12

5.3. Time-History Plots:....................................................................................................... 14

6. Conclusion: ........................................................................................................................... 14
iv

TABLE OF FIGURES 

Figure 1: The CCAQS modeling domain with the SJV modeling domain inset. ......................... 3

Figure 2. The location of the three nested grids adopted for numerical modeling of the SJV 

using MM5.................................................................................................................... 5

Figure 3: The annual average concentrations for PM2.5 and its components ............................. 7

Figure 4: The first quarter average concentrations for PM2.5 and its components...................... 8

Figure 5: The second quarter average concentrations for PM2.5 and its components................ 9

Figure 6: The third quarter average concentrations for PM2.5 and its components ................. 10

Figure 7: The fourth quarter average concentrations for PM2.5 and its components ................ 11

Figure 8: The annual average mean fractional error (MFE) and bias (MFB) for PM2.5 and its 

major components. The red line is the criteria and the blue line the goal. ............... 13

Figure 9: The time-history plots of PM2.5 and its major components at the Bakersfield 

monitoring station....................................................................................................... 15

Figure 10: The time-history plots of PM2.5 and its major components at the Fresno monitoring 

station......................................................................................................................... 16
1

1. Introduction: 

The San Joaquin Valley (SJV) of California is currently designated as 

nonattainment for PM2.5. Addressing this issue properly requires rigorous longterm chemical transport modeling to support the development of a plan to attain 

the National Ambient Air Quality Standard (NAAQS) for PM2.5. These chemical 

transport modeling exercises serve to determine future year attainment status for 

the SJV given projected emissions scenarios and also the most effective 

emissions reduction pathways to control PM concentrations for different seasons 

and regions of the domain. 

 

This document describes the performance evaluation of the United States 

Environmental Protection Agency (USEPA) Community Multiscale Air Quality 

(CMAQ) model (USEPA, 1999; CMAS, 2007) annual simulation conducted to 

assist the preparation of the State Implementation Plan for PM2.5 in the SJV. The 

base year for this simulation is 2000, for which a rich database of observations 

was collected as part of the California Regional PM10/PM2.5 Air Quality Study. 

This evaluation includes PM2.5 mass and its major components such as 

nitrate, sulfate, ammonium, organic carbon, elemental carbon, and dust. Since 

there are no universally accepted criteria for the evaluation of PM mass or 

components, we selected those proposed by Boylan and Russell (2006). Here 

we use the traditional statistical metrics, correlation diagrams, spatial plots, and 

time-history plots. 

2. Regional Air Quality Methodology: 

As stipulated in USEPA modeling guidance, a grid-based photochemical 

model is necessary to perform the modeled attainment test for PM2.5 (USEPA, 

2007). Such models offer the best available representation of important 

atmospheric processes and are an essential tool in analyzing the impacts of 

proposed emissions controls on pollutant concentrations. The USEPA 

recommends guidelines for choosing a model for use in the attainment test. For 

example, the model source code should be free or low cost, modeling elements 

should have undergone rigorous scientific peer-review, and the model should 

have been shown to perform well in the past for similar applications. 

The CMAQ model has been selected for use in the PM2.5 modeled 

attainment demonstration for the San Joaquin Valley Air Pollution Control District. 

CMAQ is a state-of-the-science “one-atmosphere” system that treats major 

atmospheric and land processes (e.g., advection, diffusion, gas phase chemistry, 

gas-particle mass transfer, nucleation, coagulation, wet and dry deposition, 

aqueous phase chemistry, etc.) and a range of species (e.g., anthropogenic and 
2

biogenic, primary and secondary, gaseous and particulate) in a comprehensive 

framework (USEPA, 1999; CMAS, 2007). 

CMAQ has been extensively peer-reviewed, is well-documented, and is 

regularly updated to reflect the latest changes in scientific understanding. CMAQ 

has been applied successfully in a range of environments and on many spatial 

and temporal scales. Given that CMAQ has also been applied successfully to 

episodic modeling in Central California, the CMAQ modeling system version 4.6 

with California-specific updates, as described in Liang and Kaduwela (2005), was 

selected for use in support of the PM2.5 modeled attainment demonstration. 

3. Chemical Mechanism: 

There are a number of gas-phase chemical mechanisms readily available 

for application in CMAQ (e.g., CB-IV, CB-V, SAPRC-99). The user has the 

additional option of whether to couple the chosen gas phase mechanism with 

aerosol and/or aqueous phase chemical processes. In order to simulate the 

complex mixture of PM2.5 species in the San Joaquin Valley, SAPRC99 coupled

with CMAQ aerosol code version 4 and aqueous phase chemistry has been 

chosen for this application. SAPRC-99, a complete update of SAPRC-90, is a 

detailed mechanism describing the gas-phase reactions of volatile organic 

compounds (VOCs) and oxides of nitrogen (NOx) (Carter, 2000). AE4-AQ, the 

fourth-generation CMAQ aerosol code with aqueous phase chemistry, when 

coupled with a gas phase mechanism, represents such phenomena as gas–

aerosol/aqueous phase mass transfer, chemical transformation of particulate 

species and their gas phase precursors, and the evolution of the aerosol size 

distribution. 

4. Model Inputs and Setup: 

4.1. Domain Structure: 

Two modeling domains were used for this work. The first modeling 

domain, named CCAQS, covered the Central Valley and its surroundings with 

63x63 lateral 12x12 km2 grid cells for each vertical layer. The CCAQS domain 

extends from the Pacific Ocean in the west to the Mojave Desert and western 

Nevada in the east and runs from the northern Sacramento Valley to the 

Tehachapi Mountains in the south. The second domain, named SJV, is fully 

nested within the CCAQS domain and covers the San Joaquin Valley with 80x89 

lateral 4x4 km2 grid cells for each vertical layer (Figure 1). The vertical structure 

for both domains is composed of 15 layers of varying thickness up to the top of 

the meteorological domain (100 mb). The finest resolution belongs to those 

layers closest to the surface and is determined largely by the vertical structure of 

the meteorological inputs. The surface layer is approximately 30 meters thick. 
3

Figure 1: The CCAQS modeling domain 

with the SJV modeling domain inset.

4.2. Initial and Boundary Conditions: 

Boundary conditions for the CCAQS domain were taken from the global 

chemical transport Model for Ozone And Related chemical Tracers (MOZART). 

Model boundary conditions for major species were extracted for the Central 

California modeling domain from MOZART results representative of the year 

2000. In addition to VOCs and inorganic gases, boundary conditions were 

extracted for ammonium, nitrate, sulfate, and organic and elemental carbon. 

Initial conditions were estimated as an average of the extracted boundary 

conditions for each species. The CCAQS domain provided the initial and 

boundary conditions for the nested SJV domain. All species reported in the 

three-dimensional output fields for the simulation over the CCAQS domain were 

included in the initial and boundary conditions for the SJV domain. While 

boundary conditions for the CCAQS domain were held constant for each month, 

boundary conditions for the SJV domain varied for each hour. The impact of 

initial conditions was minimized for each domain by simulating 8-day spin up 

periods prior to the simulation of each month of the year. 

4.3. Emissions: 

A spatially, temporally, and chemically resolved emissions inventory of 

combined area, mobile, and point sources was generated using the California 

Emissions Forecasting System (CEFS) version 1.06 with offline adjustments. 

The inventory includes emissions estimates for gaseous and particulate species 
4

of anthropogenic and biogenic origin. Gridded hourly emissions were developed 

for the CMAQ modeling domain for the years 2000, 2005, and 2014. Quality 

assurance checks of domain emissions totals and spatial distribution were 

performed at various steps in the emissions processing in order to ensure that 

the CMAQ emissions input files were sound. Additional information on emissions 

is provided separately. 

4.4. Meteorological Inputs: 

The meteorological input fields to CMAQ were generated with the 

Meteorology-Chemistry Interface Processor (MCIP) version 3.0. MCIP serves as 

a link between meteorological models like MM5 or WRF with CMAQ, and 

generates model-ready meteorological inputs like the wind and temperature 

fields necessary to drive the transport and chemistry calculations in CMAQ. 

Inputs to MCIP were generated using the PSU/NCAR Mesoscale Model (MM5) 

(version 3.6) (Grell et al, 1995). Planetary boundary layer and radiation 

characteristics were calculated in MCIP, and the Models-3 dry deposition routine 

(Pleim - with chlorine and mercury species) was chosen to represent dry 

deposition. 

4.5. MM5 Simulation: 

MM5 is a limited area, terrain-following sigma coordinate model on 

Lambert Conformal projection that was developed by Penn State and NCAR as a 

community mesoscale model. The model is based on non-hydrostatic, fully 

compressible motions that allow users to study the atmospheric motions at small 

scales by explicitly treating the effects of convective motions on atmospheric 

circulations. The MM5 model has been improved over more than two decades 

by contributions from a broad scientific community.

MM5 was set up for a 14 month simulation (December 1999 – January 

2001) with three nested grids using (70 x 70), (133 x 133), and (94 x 85) grid 

points in the (x, y) or (south-north, west-east) directions with 36, 12, and 4 km 

horizontal resolution in each grid, respectively (Figure 2). The vertical structure 

of the domain was defined with 30 layers extending to 100 mb at the top of the 

domain. The first two coarse grids defined the atmospheric initial and boundary 

conditions for the area at large scale, while the innermost grid with 4 km 

horizontal resolution resolved the fine details of atmospheric motions within the 

SJV modeling domain. MM5 has several options to calculate the components of 

internal and external forces acting on a volume of air, such as those for radiation, 

convection, cloud microphysics, soil fluxes, and boundary layer physics. While 

many sensitivity studies were conducted using various model options to find the 

best agreement with observations, an effort was also made to use the same 

model options from one simulation to the next. The Grell (1995) cumulus 

parameterization scheme for coarse grids was used along with the Blackadar 

boundary layer scheme for calculation of fluxes (Blackadar, 1979, Grell, 1995). 
5

The Dudhia simple ice scheme was used for the treatment of excess moisture 

(Dudhia, 1989) and the Dudhia cloud radiation scheme (Dudhia, 1993) was used 

for radiational heating and cooling of the atmosphere. The Blackadar multi-layer, 

force-restore method soil model (Blackadar, 1976) was used for soil physics in all 

grids. 

Initial and boundary conditions were prepared using the analyses of 

observations prepared by the National Center for Environmental Prediction 

(NCEP) and archived at NCAR. The 14-month period (December 1999 through 

January 2001) was first simulated using initial and boundary conditions (IC/BCs) 

with the analysis nudging option on the two coarse grids. Then, IC/BCs were 

prepared from the 12 km grid output for the initialization of the 4 km grid. The 

three-dimensional (3-D) wind and temperature values simulated by the model 

were compared against surface observations obtained from four SJV stations 

(Fresno, Bakersfield, Arvin, and Parlier) to study the temporal and 3-D spatial 

structure of atmospheric motions as well as to evaluate the model performance 

within the SJV. Figure 2 shows the MM5 domains employed to generate 

meteorological fields for the CMAQ simulation. A detailed performance 

evaluation of the MM5 simulation is provided separately. 

Figure 2. The location of the three nested grids 

adopted for numerical modeling of the SJV using 

MM5. 
6

5. Model Performance Evaluation: 

Methods for evaluating the performance of episodic ozone simulations are 

reasonably well developed and universally accepted. However, model evaluation 

for particulate matter (PM) simulations is still in an infant stage. Methods for 

performance evaluation of long-term simulations for both ozone and PM are 

neither reasonably developed nor universally accepted. 

Thus, we employ the following approach to evaluate the annual PM 

simulation for 2000. First, we will examine the spatial patterns of PM2.5 mass and 

its components on an annual and quarterly basis. Second we compute fractional 

statistics (mean fractional error and mean fractional bias) for each measurement 

location. We have also computed other traditional statistics such as mean 

normalized error and mean normalized bias, but we present only fractional 

statistics here. Third, we examine time-history plots for PM2.5 mass and its 

components. 

5.1. Spatial Patterns for PM mass and components: 

Figure 3 shows the annual average spatial patterns for PM mass, nitrate, 

sulfate, ammonium, organic carbon, elemental carbon and dust. As expected, 

PM mass and components are restricted to the valley floor. The maximum mass 

loading for this domain is near Visalia. The major contributors to the mass are 

dust and ammonium nitrate, and they also peak near Visalia. The second 

highest mass loading is in the Bakersfield area. The sulfate concentrations are 

very low in the majority of the domain (<5 μg/m3

) except in the vicinity of Taft, 

Bakersfield, and Stockton. The concentrations of both organic and elemental 

carbon are relatively small on an annual basis compared to those of dust and 

ammonium nitrate. 

Figures 4-7 show the quarterly average for PM2.5 mass and its major 

components. It is evident from these figures that nitrate formation follows the 

seasonal pattern expected based on seasonal temperatures and relative 

humidity. However, the nitrate concentrations for the third quarter appear to be 

excessive. The (12x12) km2

 simulation that provided the initial and boundary 

conditions for this (4x4) km2

 simulation does not show this behavior. Thus, the

cause of this excessive nitrate is not clear at this time. It is important to note that 

the Federal Reference Monitor (FRM) does not retain significant amounts of 

nitrates for both the second and third quarters. Thus, the excess nitrate 

concentrations simulated here do not have a significant impact on the future year 

design value determination. As expected, sulfate concentrations remain 

relatively low for all quarters. The organic and elemental carbon concentrations 

are higher in the first and fourth quarters, perhaps due to enhanced wood 

burning which was not controlled in the 2000 base year for this simulation. Dust 

becomes more wide-spread in the third and fourth quarters. 
Figuraverafor PMcomp
Figurquartconceand it
Figurquartconceand it
Figurquartconceand it
Figurquartconceand it
12

5.2. Statistics for Mean Fractional Error (MFE) and Bias (MFB): 

There are several statistical measures that have been used in the 

evaluation of episodic ozone simulations. However, their utility in the evaluation 

of PM models is not universally agreed upon. The general consensus among 

practitioners of photochemical air quality simulations is that novel concepts are 

needed to evaluate annual PM simulations. To that end, Boylan and Russell 

(2006) proposed a two-tier approach for model performance evaluation of PM. 

These two tiers are based on fractional statistics known as mean fractional error 

(MFE) and the mean fractional bias (MFB). The first tier is the model 

performance criteria. This is defined as “the level of accuracy that is considered 

to be acceptable for modeling applications.” The model performance criteria is 

met when both MFE ≤ +75% and MFB ≤ ±60%. The second, and more 

restrictive, tier is the model performance goal. This is defined as the “the level of 

accuracy that is considered to be close to the best a model can be expected to 

achieve.” The model performance goal is met when both MFE ≤ +50% and MFB 

≤ ±30%. This method also allows for less stringent criteria/goals at lower 

concentrations. 

The mean fractional error and mean fractional bias are defined as 

∑= +

−

=

N

i

o

i

m

i

o

i

m

i

C C

C C

N

MFE

1

2

, 

and 

∑= +

−

=

N

i

o

i

m

i

o

i

m

i

C C

C C

N

MFE

1

2

, 

where m Ci and o Ci are the modeled and observed concentration at a given 

monitor. The index i indicates an individual member of a set of N matching pairs 

of simulated and observed values at that monitor. These are centered statistics 

that give equal weights to simulated and measured values. Note here that we 

calculate these statistics, as shown in figure 8, for each monitoring site and each 

specie. 

Figure 8 shows both MFE and MFB for PM2.5 and its major components. 

Each point in these graphs represents an individual measurement location. We 

have used approximately 30 locations within the (4x4) km2 domain that reported 

24-hour PM2.5 filter measurements. Chemical speciation information was used 

directly when available for that location. For locations where speciation 

information was not available, information from a nearby speciation site was 

used. The general procedure used to assign un-speciated monitors to speciation 

monitors is described elsewhere.
Figuraveraerror (MFBmajored linthe b
14

In comparing the observations with simulations, we selected the best 

matched simulated value within nine grid cells of the monitor. The results, not 

shown here, are very comparable when a nine-cell average concentration is used 

instead. 

It is clear from figure 8 that the 2000 base simulation meets the 

performance criteria for both MFE and MFB for all species except for dust. The 

cause of overestimation of dust concentrations is not known at this time and 

further investigations of emissions inventories may be needed to better 

understand this discrepancy. It is interesting to note that the apparent 

overestimation of ammonium nitrate during the third quarter does not affect the 

model performance appreciably (despite the fact that the third quarter MFE and 

MFB, which are not shown here, do not meet model performance criteria). 

5.3. Time-History Plots: 

Figures 9-10 show time-history plots for PM2.5 and its major components 

at the Bakersfield and Fresno monitoring stations. These two stations measured 

high concentrations of particulate matter. It is clear from these two figures that 

the air quality model is able to reproduce the seasonality of particulate matter 

production observed in the Central Valley of California. The overestimation of 

ammonium nitrate in the third quarter is obvious from these figures. The cause 

of this overestimation is not known at this time. But, we suspect that the 

simulated meteorology may be responsible for this overestimation. It is also 

shown that dust is overestimated throughout the year. This may be due to 

overestimation of dust in the emission inventory. Organic carbon, elemental 

carbon, and sulfate show satisfactory model performance throughout the year. 

6. Conclusion: 

We have conducted photochemical particulate matter modeling for the 

San Joaquin Valley of central California for the base year of 2000. The model 

performance is satisfactory based on criteria proposed by Boylan and Russell 

(2006). These criteria were based on an analysis of various particulate-matter 

modeling conducted in several areas of the United States. Our simulation 

overestimates ammonium nitrate concentrations for the third quarter. However, 

this has no significant impact on the future-year design value determination since 

the Federal Reference Monitor does not report significant amounts of ammonium 

nitrate for the third quarter. The concentration of dust is also consistently 

overestimated for the entire year. This may not pose a significant problem since 

the model results are used in a relative sense and the dust component in the 

measurements is not very significant. Organic carbon, elemental carbon, and 

sulfate show satisfactory model performance. 
Figure of PM

compmonit
Figure of PM

compmonit
17

References: 

Blackadar, A. K. (1976) Modeling the Nocturnal Boundary Layer. Preprints of 

Third Symposium on Atmospheric Turbulence and Air Quality, Raleigh, 

NC. 19-22, October 1976, Amer. Meteor. Soc., Boston, 46-49. 

Blackadar, A. K. (1979) Advances in Environmental Science and Engineering, 1. 

No. 1. Pfaflin and Ziegler, Eds. Gordon and Breach Publishers, 50-85. 

Boylan, J.W. and A.G. Russell (2006) PM and light extinction model 

performance metrics, goals, and criteria for three-dimensional air quality 

models. Atmospheric Environment, 40:4946-4959. 

doi:10.1016/j.atmosenv.2005.09.087 

Carter, W. (2000) Documentation of the SAPRC-99 chemical mechanism for 

VOC reactivity assessment. Final Report to California Air Resources 

Board. Contract 92-329 and Contract 95-308, 

http://pah.cert.ucr.edu/ftp/pub/carter/pubs/s99txt.pdf 

Chow, J.C., Chen, L.W.A., Watson, J.G., Lowenthal, D.G., Magliano, K.A., 

Turkiewicz, K., and D.E. Lehrman (2006) PM2.5 chemical composition and 

spatiotemporal variability during the California Regional PM10/PM2.5 Air 

Quality Study (CRPAQS). J. Geophys. Res., 111, D10S04, 

doi:10.1029/2005JD006457. 

Chow, J.C., Watson, J.G., Lowenthal, D.H., Magliano, K. (2005) Loss of PM2.5

Nitrate from Filter Samples in Central California. J. Air & Waste Manage. 

Assoc. 55:1158-1168. 

Community Modeling and Analysis System (2007) CMAQ v4.6 Operational 

Guidance Document. 

http://www.cmaqmodel.org/op_guidance_4.6/manual.pdf

Dudhia, J. (1989) Numerical Study of Convection Observed During the Winter 

Monsoon Experiment Using a Two-Dimensional Model. J. Atmos. Sci., 46, 

3077-3107. 

Dudhia, J. (1993) Radiation Studies with a High-Resolution Mesoscale Model. 

Proceedings of the Third Atmospheric Radiation Measurement (ARM) 

Science Team Meeting. March 1-4, 1993, Norman, OK. pp. 363-366. 

Grell, G. A., J. Dudhia and D. R. Stauffer. (1995) A Description of the FifthGeneration Penn State/NCAR Mesoscale Model (MM5). NCAR Technical 

Note, NCAR/TN-398+STR. 
18

Hering, S.V. and Cass, G.R. (1999) The Magnitude of Bias in the Measurement 

of PM2.5 Arising from Volatilization of Particulate Nitrate from Teflon Filters. 

J. Air & Waste Manage. Assoc. 49:725-733. 

Liang, J. and A. Kaduwela, 2005: Microdevelopment of CMAQ for California 

Regional Particulate-Matter Air Quality Study. Proceedings of the 4th 

Annual CMAS Models-3 Users’ Conference; September 26-28, 2005, 

Chapel-Hill, NC. 

U.S. Environmental Protection Agency (1999) Science Algorithms of the EPA 

Models-3 Community Multiscale Air Quality (CMAQ) Modeling System.

EPA/600/R-99/030. 

http://www.epa.gov/asmdnerl/CMAQ/CMAQscienceDoc.html

U.S. Environmental Protection Agency. (2007) Guidance on the Use of Models 

and Other Analyses for Demonstrating Attainment of Air Quality Goals for 

Ozone, PM2.5, and Regional Haze. EPA-454/B-07-002