PHOTOCHEMICAL MODELING IN SUPPORT OF 

ATTAINING THE FEDERAL 8-HOUR OZONE 

AIR QUALITY STANDARD IN CENTRAL CALIFORNIA 

Volume 1: 

Model Performance 

California Air Resources Board 

Planning and Technical Support Division 

Sacramento, California 95814 

3/1/07 
MODEL PERFORMANCE EVALUATION 

2 

TABLE OF CONTENTS 

1 METEOROLOGICAL MODEL PERFORMANCE.................................................... 3

1.1 Meteorological Model Performance Metrics.................................................................. 3

1.2 Meteorological Model Performance Results .................................................................. 6

2 AIR QUALITY MODEL PERFORMANCE ............................................................. 37

2.1 Air Quality Model Performance Metrics...................................................................... 37

2.2 Air Quality Model Performance Results ...................................................................... 43

3 CONCLUSION – CUMULATIVE 1-HOUR AND 8-HOUR DAYS........................... 69

4 RESULTS FOR MODEL SENSITIVITY TESTS..................................................... 71

5 REFERENCES....................................................................................................... 72

6 APPENDIX – INFORMATION AVAILABLE FOR DOWNLOADING..................... 79
MODEL PERFORMANCE EVALUATION 

3 

For District review purposes, this document summarizes model performance 

procedures and results for meteorological modeling (Section 1) as well as air quality 

modeling (Section 2) for the July 1999 and July-August, 2000, episodes. The model 

performance evaluations are based on USEPA guidance (1991 and 2005) as well as 

recommendations from Emery (2001), Tesche (1994) and Tesche et al. (2001). 

The third section (Section 3) provides a summary of the performance analysis and 

Section 4 provides a tabular listing of complete graphical and statistical results that can 

downloaded via ftp from eos.arb.ca.gov. 

Of note is that the two episodes have both been extended by two days at the beginning 

of the original episode periods in an effort to increase the number of useable days for 

future year design value calculations. 

1 Meteorological Model Performance 

1.1 Meteorological Model Performance Metrics 

Meteorological model performance is assessed both quantitatively using statistical 

metrics as well as qualitatively against known conceptual meteorological flows and 

observed episodic meteorological features. 

1.1.1 Quantitative Performance Evaluation 

There are a number of statistical and graphical approaches for evaluating 

meteorological model outputs. However, none of them are independently conclusive. 

Most of these approaches involve comparisons between observed and simulated 

meteorological parameter values. These analyses pose a difficult challenge, since most 

of the available meteorological monitoring stations are located in urbanized areas. 

Thus, the majority of observations tend to represent those areas versus the full 

complexity of meteorology throughout the CCOS domain. Furthermore, since the use of 

objective analysis and observational nudging forces the meteorological modeling results 

towards the observations, model performance problems can increase in areas away 

from observation locations. 

It also needs to be recognized that output from the various meteorological models must 

be preprocessed for input into the air quality model. This preprocessing may 

inadvertently perturb the meteorological fields. Therefore, meteorological model 

performance should be based on the air quality model input files, rather than the 

meteorological model outputs. 

The SIP modeling domain is geographically very complex and the observational data on 

which meteorological model outputs were evaluated are not distributed uniformly. 

Therefore, it is unreasonable to evaluate model performance for the domain as a whole. 

For purposes of meteorological model performance analysis, the CCOS domain is 
MODEL PERFORMANCE EVALUATION 

4 

divided into sub-regions, representing areas of similar meteorological features. The 

graphical and statistical model evaluations will be done for each of these sub-regions. 

A number of standard statistical and graphical techniques are used for meteorological 

model performance analysis. The most widely used application is the METSTAT 

program (Tesche, 1994, Tesche et al, 2001). Two graphical representations of the 

METSTAT statistics were used in meteorological model performance analysis 

conducted here: a) “Root Mean Square Error (RMSE) of Wind Speed” vs. “Gross Error 

(E) of Wind Direction”, and b) “Bias Error (B)” vs. “Gross Error (E)” for temperature. 

Equations used for these comparisons were taken from the user documentation of the 

METSTAT program and are given below: 

Bias Error (B): calculated as the mean difference in prediction-observation 

pairings with valid data within a given analysis region and for a given time period 

(hourly or daily): 

Here, P and O indicate model predictions and observations, respectively. 

Similarly, I and J are the indices of grid points in x and y directions, respectively. 

Gross Error (E): calculated as the mean absolute difference in predictionobservation pairings with valid data within a given analysis region and for a given 

time period (hourly or daily): 

Note that the bias and gross error for winds are calculated from the predictedobserved residuals in speed and direction (not from vector components u and v). 

The direction error for a given prediction-observation pairing is limited to range 

from 0 to ±180°. 

Root Mean Square Error (RMSE): calculated as the square root of the mean 

squared difference in prediction-observation pairings with valid data within a 

given analysis region and for a given time period (hourly or daily): 

The RMSE, as is the gross error, is a good overall measure of model performance. 

However, since large errors are weighted heavily (due to squaring), large errors in small 

subregions may produce a large RMSE even though the errors may be small and quite 

acceptable elsewhere. 

∑∑( )

= =

= −

J

j

I

i

i

j

i Pj O

IJ

B

1 1

1

∑∑= =

= −

J

j

I

i

i

j

i Pj O

IJ

E

1 1

1

( )

2/1

1 1

1 2













= ∑∑ −

= =

J

j

I

i

i

j

i Pj O

IJ

RMSE
MODEL PERFORMANCE EVALUATION 

5 

Table 1-1 shows the criteria used to decide if the results of a given model fall within 

acceptable performance limits. 

Table 1-1 Statistical comparisons between observed and 

simulated meteorological parameter values. Statistical 

comparisons are made by model performance subregions.

________________________________________________ 

 Parameter Abbreviation Benchmark 

________________________________________________ 

 Wind Speed RMSE: < 2 m/s 

 Bias: < ±0.5 m/s 

 IOA: ≥ 0.6 

 Wind Direction Gross Error: < 30 deg 

 Bias: < ±10 deg 

 Temperature Gross Error: < 2 oK 

 Bias: < ±0.5 oK 

 IOA ≥ 0.8 

In an ideal situation, meteorological field evaluation would be done independent of the 

air quality model results. However, in practice, meteorological field evaluation is limited 

by the relative paucity of observational data, especially aloft. Therefore, base year air 

quality model performance was also considered in the selection of meteorological fields 

used for air quality simulations. 

Table 1-2 Graphical analysis of meteorological model fields. Time 

series plots are made for each station and spatial plots are made over 

the whole modeling domain. 

 Time-series plots of hourly mean air temperature 

 Time-series plots of hourly mean wind speeds. 

 Spatial plots of hourly wind vectors 

 Spatial plots of hourly air temperatures 

 
MODEL PERFORMANCE EVALUATION 

6 

1.1.2 Qualitative Performance Analyses 

Given episode-specific information on the meteorological features that were observed 

with field measurements, additional subjective analyses of observed versus predicted 

mesoscale features can be conducted. Examples of such qualitative analyses that will 

be considered are described below. 

1. Determine and compare modeled and observed horizontal flow patterns over the 

modeling domain. Features to consider include flow splitting, the structure of the 

sea breeze, urban circulations, local flows such as Fresno and Schultz eddy 

circulations, slope and drainage flows, up/down valley flows, and the existence of 

cloud formations. 

2. Study the 3-D spatial characteristics of the flow field by using time-height cross 

sections of wind profiler observations and the simulated wind field at the wind 

profiler location. 

3. Determine the spatial and temporal characteristics of the mixing layer height 

using available upper air observations, and compare it with the simulated 

behavior of mixing layer height. 

4. Perform some sensitivity tests to see the effects of certain model parameters on 

the model results, such as observational nudging vs. analysis nudging, the 

choice of soil physics, and boundary layer parameterizations. 

1.2 Meteorological Model Performance Results 

The following two sections present the results of meteorological model performance for 

the two modeling episodes, based on the criteria discussed above. For illustration 

purposes, a small number of the graphics that were produced are used in the 

subsequent discussions. However, all of the graphics that have been generated are 

available via ftp per the table in the Appendix. 

1.2.1 July 1999 Episode (Routine Episode) 

The July 1999 simulation covers the period from July 5th 12Z, 1999 to July 14th 12Z, 

1999. Meteorological model performance is assessed for the 7-day period spanning 

July 7th through July 13th

. 
MODEL PERFORMANCE EVALUATION 

7 

The July 1999 episode model performance is assessed for three MM5 runs as follows: 

Table 1-3. July-1999 Episode. Three ARB simulations are considered.

Simulation 

Number Abbreviation Description 

1 C108 7 day simulation; without FDDA 

2 C109 7 day simulation; with FDDA; includes all the 

available observational data. 

3 C110 7 day simulation; with FDDA; but excludes from

the FDDA file all known 2-meter station height 

data (i.e. CIMIS and NWS stations). 

The FDDA file for run C109 includes all of the available observational data that are 

available for this routine field measurement episode. The Bay Area Air Quality 

Management District and their contractors, AtMet and ENVIRON, produced FDDA data 

for the original ‘core’ episode days, July 9th through July 12th. For the extended 

episode days that fall outside of the original core days, ARB FDDA data are used. The 

third run (c110) has the same model setup as “c109”, except that data from sources 

utilizing 2-meter station heights are excluded from the C109 FDDA file. Simulation 

C110 is a sensitivity run to evaluate the effect of 2-meter observational station heights 

on MM5 performance for this episode. 

To calculate model performance statistics, the results from all three MM5 simulations 

are processed through the METSTAT program. Performance statistics and siteaveraged time series are calculated for 5 regions: Bay Area region, Sacramento region, 

Central San Joaquin Valley, Southern San Joaquin Valley and Northern San Joaquin 

Valley. The resulting statistics are presented in soccer plots, where, ideally, model 

performance statistics fall within the central box of the goal. 

Figure 1-1 shows sites-averaged time series for winds and temperatures in the Bay 

Area region. There is little difference between the three simulations. Wind speeds were 

generally under-predicted over the entire simulation period. On the other hand, 

temperature performance is good, as noted by the simulated diurnal pattern. The 

exception to this is that temperatures were over-predicted on July 13th. In terms of 

model performance statistics for the Bay Area region, Figure 1-2 shows the soccer goal 

plots of daily performance for winds and temperature. There was little difference 

between c109 and c110. 

Figure 1-3 shows sites-averaged time series for winds and temperatures in the 

Sacramento region. In general, there is little difference between the three simulations. 

Wind speeds are over-predicted on July 7th and July 12th. Otherwise, the wind speed 

performance is good over the simulation period. On the other hand, temperatures were 

under-predicted during the day and over-predicted in the morning. Figure 1-4 shows 

the soccer goal plots of daily performance for winds and temperature. 
MODEL PERFORMANCE EVALUATION 

8 

Figure 1-5 shows sites-averaged time series for winds and temperatures in the Central 

San Joaquin Valley. Wind speeds were generally over-predicted. Temperatures were 

under-predicted during the day and over-predicted in the morning. There was little 

difference between the three simulations. Figure 1-6 shows the soccer goal plots of 

daily performance for winds and temperature. 

Figure 1-7 shows sites-averaged time series for winds and temperatures in the 

Southern San Joaquin Valley. Wind speeds were generally over-predicted. 

Temperatures were under-predicted during the day and over-predicted in the morning. 

There was little difference between the three simulations. Figure 1-8 shows the soccer 

goal plots of daily performance for winds and temperature. 

Figure 1-9 shows sites-averaged time series for winds and temperatures in the Northern 

San Joaquin Valley. Wind speeds were generally over-predicted. Temperatures were 

under-predicted during the day and over-predicted in the morning. There was little 

difference between the three simulations. Figure 1-10 shows the soccer goal plots of 

daily performance for winds and temperature. 

It should be noted that both the ARB and the Bay Area Air Quality Management District 

have done much work to improve meteorological model performance for this episode. 

However, little additional progress has been made over the past two years and 

performance statistics still remain outside of the ‘ideal’ range. Alone, however, this is 

not grounds to dismiss the met simulations as poor. We are hopeful that statistical 

performance can be improved and will continue to work closely with the districts and 

other stakeholders, including CCOS contractors, with this goal in mind. 

Among the three July 1999 simulations, MM5 with observation nudging (c109 and c110) 

improves the wind speed and wind direction a little over c108, which has no observation 

nudging. Since there were no significant differences between c109 and c110, it is 

assumed that the 2-meter station data included in the FDDA file play no significant role 

in degrading model performance. As a result, c109 is used as input for the air quality 

model. 
MODEL PERFORMANCE EVALUATION 

9 

Figure 1-1. Time series of wind speed, direction, and temperature for the Bay Area 

region over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

10 

Figure 1-2. MM5 performance for winds and temperature in the Bay Area region. 

MM5 Temperature Performance in Bay Area 

July 7 - 13, 1999

0

1

2

3

4

5

6

-1 0 1 2 3 4 5 6

Bias (K)

Error (K)

c108

c109

c110

Benchmark

MM5 Wind Performance in Bay Area 

July 7 - 13, 1999

0

10

20

30

40

50

60

70

0 0.5 1 1.5 2 2.5

WS RMSE (m/s)

WD Gross Error (degree)

c108

c109

c110

Benchmark
MODEL PERFORMANCE EVALUATION 

11 

Figure 1-3. Time series of wind speed, direction, and temperature for the Sacramento 

region over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

12 

Figure 1-4. MM5 performance for winds and temperature in the Sacramento region. 

MM5 Wind Performance in Sacramento 

July 7 - 13, 1999

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5

WS RMSE (m/s)

WD Gross Error (degree)

c108

c109

c110

Benchmark

MM5 Temperature Performance in Sacramento 

July 7 - 13, 1999

0

0.5

1

1.5

2

2.5

3

3.5

-1 -0.5 0 0.5 1 1.5 2

Bias (K)

Error (K)

c108

c109

c110

Benchmark
MODEL PERFORMANCE EVALUATION 

13 

Figure 1-5. Time series of wind speed, wind direction, and temperature for the Central 

San Joaquin Valley over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

14 

Figure 1-6. MM5 performance for winds and temperature in the Central San Joaquin 

Valley. 

MM5 Wind Performance in SJV Central 

July 7 - 13, 1999

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5

WS RMSE (m/s)

WD Gross Error (degree)

c108

c109

c110

Benchmark

MM5 Temperature Performance in SJV Central 

July 7 - 13, 1999

0

0.5

1

1.5

2

2.5

3

3.5

-1 -0.5 0 0.5 1

Bias (K)

Error (K)

c108

c109

c110

Benchmark
MODEL PERFORMANCE EVALUATION 

15 

Figure 1-7. Time series of wind speed, wind direction, and temperature for the 

Southern San Joaquin Valley over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

16 

Figure 1-8. MM5 performance for winds and temperature in the Southern San Joaquin 

Valley. 

MM5 Wind Performance in SJV Kern 

July 7 - 13, 1999

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5

WS RMSE (m/s)

WD Gross Error (degree)

c108

c109

c110

Benchmark

MM5 Temperature Performance in SJV Kern 

July 7 - 13, 1999

0

0.5

1

1.5

2

2.5

3

-1.5 -1 -0.5 0 0.5 1

Bias (K)

Error (K)

c108

c109

c110

Benchmark
MODEL PERFORMANCE EVALUATION 

17 

Figure 1-9. Time series of wind speed, direction, and temperature for the Northern San 

Joaquin Valley over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

18 

Figure 1-10. MM5 performance for winds and temperature in the Northern San Joaquin 

Valley. 

MM5 Wind Performance in SJV North 

July 7 - 13, 1999

0

10

20

30

40

50

60

70

0 0.5 1 1.5 2 2.5

WS RMSE (m/s)

WD Gross Error (degree)

c108

c109

c110

Benchmark

MM5 Temperature Performance in SJV North 

July 7 - 13, 1999

0

0.5

1

1.5

2

2.5

3

-1.5 -1 -0.5 0 0.5 1 1.5

Bias (K)

Error (K)

c108

c109

c110

Benchmark
MODEL PERFORMANCE EVALUATION 

19 

1.2.2 July-August 2000 Episode (CCOS Episode) 

Under the CCOS program, the meteorological modeling group at the National Oceanic 

and Atmospheric Administration (NOAA) was selected as a contractor to study the July 

29, 2000 12Z – Aug 3, 2000 12Z ozone episode that occurred during CCOS. Under this 

CCOS contract, NOAA studied the meteorology of this episode using the MM5 

numerical model with various model options and initial and boundary conditions. 

After extensive internal simulations, NOAA produced and distributed an MM5 model 

output in 2003 that is referred to as the “NOAA placeholder” simulation. Subsequently, 

NOAA produced several additional MM5 outputs. Three of these other simulations as 

well as the placeholder simulation were selected by ARB as candidates for SIP 

modeling purposes. The last two of these simulations are considered by NOAA to be 

their ‘best available’ runs. The model setups in all of these simulations are identical 

except as noted in the first four rows of the table below (Table 1-3). 

Table 1-3 July-Aug 2000 CCOS Episode. Four 5-day NOAA simulations and two 

7-day ARB simulations are considered.

Simulation 

Number Abbreviation Description 

1 NOAA placeholder 5 day simulation using 5 layer soil model and 

observational FDDA file prepared by the Bay 

Area AQMD 

2 NOAA FDDA1 5 day simulation; Same as NOAA placeholder 

(#1), except using NOAH land-surface model 

3 NOAA FDDA2 5 day simulation; Same as NOAA FDDA1 (#2), 

above, except using observational FDDA file 

prepared by NOAA and with roughness length 

doubled. 

4 NOAA FDDA3 5 day simulation; Same as NOAA FDDA1 (#2), 

above, except using observational FDDA file 

prepared by NOAA and with 5 times the 

roughness length. 

5 ARB NO FDDA 7 day simulation 

6 ARB FDDA 7 day simulation 

As indicated in the last two rows of the table above, model simulations were also 

conducted at ARB. In these two ARB simulations, different model options were used: 

the Gayno Seaman boundary layer scheme was used; a larger radius of influence was 

selected; and the model was started approximately two days earlier, on July 27, 00Z, in 

order to provide additional days for Relative Reduction Factor calculations performed for 

air quality analyses. Thus, both ARB runs are for 7 days. 
MODEL PERFORMANCE EVALUATION 

20 

The four five-day NOAA outputs along with two seven-day ARB outputs, called ARB NO 

FDDA and ARB FDDA, were analyzed and compared against observational data using 

the comparison methods discussed previously. 

Model performance statistics provided in Figures 1-10 (a-g) and 1-11(a-g) point out that 

the statistical error of simulation NOAA FDDA3 are within the acceptable limits of EPA 

standards for wind speed and direction while temperature predictions are not very good. 

The temporal comparison of model variables at the Angiola site is illustrated in 

Figures 1-12 (a and b). Comparisons at other stations were also made and can be 

made available (see the reference to other 2000 meteorology information in the 

Appendix). While previous figures show the station averaged model performance 

statistics within each subregion, these give a detailed perspective of model performance 

at an individual observation station. As can be seen from the examination of these 

temporal comparisons, model performance can vary dramatically from one station to the 

next. While all NOAA and ARB FDDA simulations appear to adequately produce the 

observed wind field, the NOAA Placeholder model appears to produce observed 

temperatures better than the other model runs do. 

Figures 1-13(a-f) compare horizontal wind vectors against observations at 21Z on 

July 29, 2000 (2 PM local time) when the flow field is expected to play an important role 

in maximum ozone concentrations. Each model has slightly different wind flows, 

however the flows in the NOAA FDDA2, FDDA3 and Placeholder simulations seem to 

be more organized than in the other models. 

Since NOAA FDDA2 and NOAA FDDA3 are sensitivity tests generated by playing with 

roughness length, these results will not be considered in air quality simulations until the 

effects of these tests are further understood and accepted. Presently, it appears that 

this has an adverse impact on temperature performance. ARB is currently working with 

NOAA to better understand these runs. The figures indicate that the overall 

performance of NOAA’s placeholder model for all variables is generally as acceptable 

as all other MM5 results that were considered. Therefore, the NOAA placeholder model 

output is used in air quality simulations. 
MODEL PERFORMANCE EVALUATION 

21 

Figures 1-11 (a-g): Model performance statistics of wind speed and direction created 

for subregions 3, 6, 7, 8, 9, 10 and 11, respectively. 

MM5 Wind Performance Within Bay Area

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5 3 3.5 4

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(a) 

MM5 Wind Performance Within Sacramento Region

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5 3 3.5 4

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(b) 
MODEL PERFORMANCE EVALUATION 

22 

MM5 Wind Performance Within SJV Central

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5 3 3.5 4

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(c) 

MM5 Wind Performance Within SJV Kern

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

0 1 2 3 4

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(d) 
MODEL PERFORMANCE EVALUATION 

23 

MM5 Wind Performance Within SJV North

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5 3 3.5 4

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(e) 

MM5 Wind Performance Within Central Mountain Counties

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(f) 
MODEL PERFORMANCE EVALUATION 

24 

MM5 Wind Performance Within SJV Above 3000 ft

July 29-Aug 3, 2000

0

10

20

30

40

50

60

70

80

90

0 0.5 1 1.5 2 2.5 3 3.5 4

WS RMSE (m/s)

WD Gross Error (degree)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(g) 
MODEL PERFORMANCE EVALUATION 

25 

Figures 1-12 (a-g): Same as Figures 1-11, except for temperature. 

MM5 Temperature Performance Within Bay Area 

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-2 -1 0 1 2 3 4 5

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(a) 

MM5 Temperature Performance Within Sacramento Region 

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-5 -4 -3 -2 -1 0 1 2 3

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(b) 
MODEL PERFORMANCE EVALUATION 

26 

MM5 Temperature Performance Within SJV Central 

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-5 -4 -3 -2 -1 0 1 2 3

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(c) 

MM5 Temperature Performance Within SJV Kern

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-5 -4 -3 -2 -1 0 1 2

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(d) 
MODEL PERFORMANCE EVALUATION 

27 

MM5 Temperature Performance Within SJV North 

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-4 -3 -2 -1 0 1 2 3 4

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(e) 

MM5 Temperature Performance Within Central Mountain Counties

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-4 -3 -2 -1 0 1 2 3

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(f) 
MODEL PERFORMANCE EVALUATION 

28 

MM5 Temperature Performance Within SJV Above 3000 ft

July 29-Aug 3, 2000

0

1

2

3

4

5

6

7

-4 -3 -2 -1 0 1 2 3

Bias (K)

Error (K)

Benchmark

ARB NO FDDA

ARB FDDA

NOAA FDDA1

NOAA FDDA2

NOAA FDDA3

NOAA Placeholder

(g) 
MODEL PERFORMANCE EVALUATION 

29 

Figures 1-12 (a and b): Temporal comparisons of wind speed, direction and 

temperature at Angiola station for ARB (a) and NOAA (b) model results 

(a) 
MODEL PERFORMANCE EVALUATION 

30 

(b) 
MODEL PERFORMANCE EVALUATION 

31 

Figures 1-13(a-f): Horizontal variation of wind vectors on July 29, 2000 21Z (2 PM local 

time) compared to the observations for ARB NO FDDA (a), ARB FDDA (b), NOAA 

FDDA1 (c), NOAA FDDA2 (d), NOAA FDDA3 (e) and NOAA Placeholder models (f). 

(a) 
MODEL PERFORMANCE EVALUATION 

32 

(b) 
MODEL PERFORMANCE EVALUATION 

33 

(c) 
MODEL PERFORMANCE EVALUATION 

34 

(d) 
MODEL PERFORMANCE EVALUATION 

35 

(e) 
MODEL PERFORMANCE EVALUATION 

36 

(f) 
MODEL PERFORMANCE EVALUATION 

37 

2 Air Quality Model Performance 

2.1 Air Quality Model Performance Metrics 

Air quality model results are used to develop strategies for attaining the federal 8-hour 

ozone standard. The development of these strategies relies on the use of relative 

reduction factors (RRFs). More detailed discussion of RRFs is provided in other 

documents. However, the use of RRFs requires an evaluation of relative air quality 

model response at specific monitoring sites in the base year(s), a baseline year, and a 

future year. 

Adequate model performance is a requirement for use of modeled results. The lack of 

acceptable performance greatly increases uncertainty in the use of the modeling results, 

and casts doubt on conclusions based on the modeling. Although it is desirable to 

include as many days as possible in the RRF calculations, our experience has 

demonstrated that not all modeled days meet the minimum performance standards, and 

are thus not suitable for use. Therefore only those days that satisfy the following model 

performance criteria will be utilized in subsequent RRF calculations. 

The USEPA (1991) and ARB (1990) outline a number of procedures for analysis of 

base year, air quality model performance. These include spatial and time-series plots, 

and statistical analyses, comparing simulated and observed pollutant concentrations, as 

well as sensitivity analysis of selected input fields. The purpose of the performance 

analysis is to provide some confidence that the air quality simulations – which are the 

basis of future-year ozone concentration estimates – are performing properly. 

The application of air quality modeling results to demonstrate attainment of the federal 

1-hour ozone standard emphasized the simulated unpaired peak ozone concentration. 

Three statistical measures were recommended to evaluate model performance: 

unpaired peak ratio (UPR), paired mean normalized bias (NB), and paired gross error 

(GE). These statistical measures were calculated for the modeling domain as a whole, 

and the NB and GE were calculated from all hourly concentrations in excess of 60 ppb 

(to avoid biasing the statistical measures with low concentrations). To meet 

performance guidelines, recommendations were that the UPR should be within ± 20%, 

NB should be within ± 15%, and the GE less than 35%. However, California’s 

geography is very complex and modeling domains have evolved to cover large 

geographic areas. Thus it is recommended that the domains be divided into 

subregions, and that the performance measures be calculated independently for each 

subregion. The configuration of these subregions is somewhat arbitrary; however, they 

should be configured to isolate "common" regions of higher ozone. Figure 2-1 

illustrates the proposed subregions for the CCOS domain. 

The USEPA (2005) recommends that model performance be evaluated for 8-hour 

concentrations as well . The recommended statistical measures to assess simulated 

versus observed maximum 8-hour ozone concentrations include paired (in space, but 
MODEL PERFORMANCE EVALUATION 

38 

not time) peak prediction accuracy (PPPA), paired mean normalized bias (NB), and 

paired gross error (GE). Although limited performance analysis has been completed for 

8-hour ozone modeling in California, it seems prudent at this point to carry forward the 

1-hour statistical goals and apply them for the 8-hour standard (UPR within ± 20%, NB 

within ± 15%, and the GE less than 35%). However, these limits may need to be 

revised as 8-hour SIP modeling progresses and rigorous model performance 

evaluations are completed. 

While statistical measures for 1-hour model performance were typically calculated 

independently for each modeled day available, the USEPA also suggests that PPPA, 

NB, and GE be calculated for each site over all modeled days. However, because the 

number of episode days available may be very limited, the statistical uncertainties in 

these latter calculations would be large and they are not recommended or used herein. 

In order to have confidence in future year estimates from air quality models, there must 

be confidence in the air quality modeling for the base year. That is, days not meeting 

model acceptance criteria provide high uncertainty, and should not be used for the 

modeled attainment test. 

In addition to the issue of model performance, analyses conducted by the USEPA 

(2005) suggest that air quality models respond more to emission reductions at higher 

predicted ozone values. Correspondingly, the model predicts less benefit at lower 

concentrations. This is consistent with preliminary modeling in support of the 8-hour 

ozone standard conducted by the ARB and the districts. These results imply that RRF 

calculations should be restricted to days with predicted high ozone concentrations. It is 

thus reasonable to establish a minimum threshold for predicted peak 8-hour ozone 

concentrations in the baseline year. Days for which the predicted daily peak 8-hour 

ozone concentrations at a site are less than the threshold, would not be used for 

calculating RRFs at that site. Consistent with USEPA’s recommendation, we propose 

to use a value of 85 ppb for the baseline year threshold. However, USEPA guidelines 

allow the use of the maximum 8-hour concentrations within 15km of a site for this 

purpose. 

Based on the above discussion, we propose the following model performance based 

methodology for determining sites and modeled days to be used in the RRF 

calculations: 
MODEL PERFORMANCE EVALUATION 

39 

Only those modeled days meeting the following criteria will be used 

to calculate site-specific RRFs: 

1) The modeled daily 8-hour peak ozone concentration within 15 km 

of the site for the base year of the modeling must be within ±20% 

of the observed value at the site. 

2) The modeled daily 8-hour peak ozone concentration within 15 km 

of the site in the baseline year must be 85 ppb or greater. 

3) The subregional 1-hour and 8-hour statistical measures of NB and 

GE must fall within the thresholds of ± 15% and 35%, respectively. 

Of these three criteria, only the third is considered in this document. 

Along with the statistical measures discussed above, the graphical and statistical tests 

recommended by the USEPA (1991 and 2005) and shown in Tables 2-1 and 2-2 will be 

used to assess overall model performance. Several sensitivity tests recommended by 

the USEPA (1991) will also be used (Table 2-3) for qualitative evaluation. While the 

results of these sensitivity analyses are inherently subjective, they are designed to 

provide confidence that the air quality model is not only performing well, but is also 

properly responding to changes in inputs. 
MODEL PERFORMANCE EVALUATION 

40 

Figure 2-1 Sub-regions of air quality model performance evaluation (3: Bay area 

region, 6: metro Sacramento region, 7: central San Joaquin valley region , 8 

southern San Joaquin valley region, 9: northern San Joaquin valley region). 

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15
MODEL PERFORMANCE EVALUATION 

41 

Table 2-1. Statistics for evaluating base year air quality model performance for 

all sub-regions. 

− mean normalized bias for all 1-hour ozone concentrations (60 ppb), unpaired in time 

and space for all sites 

− mean normalized gross error for all 1-hour ozone concentrations (≥60 ppb), 

unpaired in time and space for all sites 

− peak 1-hour ozone concentration ratio, unpaired in time and space 

− mean normalized bias for all 8-hour ozone concentrations (≥60 ppb), unpaired in 

time for all sites 

− mean normalized gross error for all 8-hour ozone concentrations (≥60 ppb), 

unpaired in time for all sites 

− peak 8-hour ozone concentration ratio, unpaired in time and space 
MODEL PERFORMANCE EVALUATION 

42 

Table 2-2. Graphical tools for evaluating base year air quality model 

performance. 

− time-series plots comparing 1-hour measured and simulated concentrations of 

ozone, NO, NO2, and CO for each site. 

− hourly spatial plots of 1-hour measured and simulated concentrations of ozone, NO, 

NO2, and CO for the CCOS modeling domain. 

− scatter plot of 1-hour ozone concentrations for each day, and for each subregion of 

the modeling domain. 

Table 2-3. Sensitivity tests for evaluation of Base Year air quality simulations. 

The results of these analyses will be tabulated by subregion. 

1 Minimize vertical diffusivity based on land cover 

2 Zero anthropogenic emissions 

3 Zero biogenic emissions 

4 Set lateral ozone boundary conditions to 50 ppb 

5 Set lateral ozone boundary conditions to 90 ppb 

6 Set initial ozone conditions to 40 ppb everywhere 

7 Set initial conditions to 0.1 ppb NO2 and 0.0 NO (run with all emissions) 

8 Set initial conditions to 0.1 ppb NO2 and 0.0 NO (run with biogenic emissions only)

9 Double biogenic emissions 

10 Remove wildfires 

11 Zero mobile emissions 

12 Set top ozone boundary conditions to 135ppb at 15km
MODEL PERFORMANCE EVALUATION 

43 

2.2 Air Quality Model Performance Results 

The following two sections present the results of air quality model performance for the 

two modeling episodes, based on the criteria discussed in the previous section. For 

illustration purposes, only a portion of the graphics that were actually produced are 

presented. All of the graphics that have been generated are available via ftp per the 

table in the Appendix. 

2.2.1 July 1999 Episode (Routine Episode) 

The July 1999 air quality model simulation covers the 7-day period from July 7, 1999, 

through July 13, 1999. However, the model performance assessment only covers the 

5-day, non-spin-up period from July 9 to July 13, 1999. As discussed previously in 

Section 1.2.1, the ARB c109 MM5 meteorological simulation is used for these air quality 

simulations. 

Tables 2-4 and 2-5 summarize the 1-hour and 8-hour statistical model performance 

assessment in terms of identifying the days for which simulated results fall within 

acceptable statistical performance thresholds in each model performance region 

(performance regions were shown in Figure 2-1). Each cell in the tables represents 

whether model-simulated results, on a region-wide basis, are statistically acceptable. 

The cell is assigned a value of 1 if the model-simulated results pass the statistical model 

performance criteria; while a value of 0 means that the model-simulated results for the 

region do not meet the criteria. If all simulated ozone concentrations in the region are 

below 60 ppb, then the region-day cell is assigned -99 and the modeling results cannot 

be used for that region. Total days for each episode day and region are provided at the 

bottom row and far right column of the table, respectively. 

For the Bay Area region, 4 days meet the 1-hour criteria and only 1 day meets the 8-

hour criteria. For the Sacramento region, all 5 days meet both the 1-hour and 8-hour 

criteria. For the Central San Joaquin Valley region, 5 days meet the 1-hour criteria and 

3 days meet the 8-hour criteria. For the Southern San Joaquin Valley region, only 1 day 

meets the 1-hour criteria and 4 days meet the 8-hour criteria. For the Northern San 

Joaquin Valley region, 2 days meet the 1-hour criteria and 3 days meet the 8-hour 

criteria. 

Figures 2-2 through 2-6 show the site-averaged time series of modeled versus 

predicted CO, ozone, NO and NO2 for the Bay Area, Sacramento, Central San Joaquin 

Valley, Southern San Joaquin Valley, and Northern San Joaquin Valley regions, 

respectively. The orange and blue lines represent observations and model predictions, 

respectively. The time series for individual stations for these five regions have also 

been plotted and are available via the ftp site and filename indicated in the Appendix. 
MODEL PERFORMANCE EVALUATION 

44 

Figure 2-2 shows the hourly averaged CO, ozone, NO and NO2 for the Bay Area region. 

Predicted CO concentrations are slightly under-predicted for most of the simulation 

period. However, the simulated ozone is generally over-predicted for the entire 

simulation period. The model captures the magnitude and diurnal variation of the NO 

and NO2 reasonably well. 

Figure 2-3 shows the hourly averaged CO, ozone, NO and NO2 for the Sacramento 

region. Predicted CO concentrations are over-predicted for most of the simulation 

period. Ozone concentrations perform well for the first three days and the last day of 

the simulation period, but are under-predicted from July 10 to July 12. Predicted NO 

and NO2 concentrations generally agree with the observations. 

Figure 2-4 shows the hourly averaged CO, ozone, NO and NO2 for the Central San 

Joaquin Valley region. Predicted CO concentrations are under-predicted for the entire 

simulation period. The simulated ozone is generally under-predicted during the day, but 

over-predicted in the morning. Both predicted NO and NO2 concentrations are underpredicted for the entire simulation period. 

Figure 2-5 shows the hourly averaged CO, ozone, NO and NO2 for the Southern San 

Joaquin Valley region. Predicted CO concentrations are generally under-predicted in 

the morning, but over-predicted in the afternoon. The simulated ozone is generally 

under-predicted during the day, but over-predicted in the morning. NO concentrations 

are under-predicted for the entire simulation period. In general, NO2 concentrations are 

also under-predicted. However, the NO2 concentrations are over-predicted at night on 

July 9 and July 11. 

Figure 2-6 shows the hourly averaged CO, ozone, NO and NO2 for the Northern San 

Joaquin Valley region. Predicted CO concentrations are under-predicted for the entire 

simulation period. The simulated ozone is generally predicted well for all days. The 

model captures the diurnal variation of the NO, but predicted concentrations are 

generally less than the observed values. NO2 concentrations also generally agree with 

the observations. 

Plots of model performance statistics for each region are provided in Figures 2-7 and 2-

8, which show the predicted 1-hour and 8-hour unpaired peak ratio and normalized bias 

in graphical format for each station in each of the five regions. 
MODEL PERFORMANCE EVALUATION 

45 

Table 2-4. 1-hour ozone performance by each region over the July 9-13, 1999 

modeling period. 

Region ID Region Name 7/9/1999 7/10/1999 7/11/1999 7/12/1999 7/13/1999 Total 

2 North Coast 1 0 0 1 -99 2

3 BAAQMD 1 1 1 1 0 4

4 MBAQMD 1 0 0 1 0 2

5 Sacramento Valley North 1 0 1 1 0 3

6 Sacramento Region 1 1 1 1 1 5

7 SJVAPCD Central 1 1 1 1 1 5

8 SJVAPCD Kern 0 0 0 0 1 1

9 SJVAPCD North 1 0 1 0 0 2

10 Sierra Nevada Central 0 1 0 0 0 1

11 SJVAPCD Above 3000 ft 0 0 1 1 1 3

12 South Central Coast 1 1 0 1 0 3

13 Sierra Nevada North 1 0 0 1 1 3

14 Desert 0 0 0 0 0 0

15 Nevada 1 0 -99 1 0 2

Total: 10 5 6 10 5 36

Table 2-5. 8-hour ozone performance by each region over the July 9-13, 1999 

modeling period. 

Region ID Region Name 7/9/1999 7/10/1999 7/11/1999 7/12/1999 7/13/1999 Total 

2 North Coast -99 -99 0 1 -99 1

3 BAAQMD 0 0 1 0 0 1

4 MBAQMD 1 -99 0 0 -99 1

5 Sacramento Valley North 1 1 1 1 0 4

6 Sacramento Region 1 1 1 1 1 5

7 SJVAPCD Central 1 1 0 1 0 3

8 SJVAPCD Kern 1 1 1 0 1 4

9 SJVAPCD North 1 0 1 0 1 3

10 Sierra Nevada Central 1 1 0 0 0 2

11 SJVAPCD Above 3000 ft 0 0 1 1 1 3

12 South Central Coast 0 1 0 1 1 3

13 Sierra Nevada North 1 0 0 1 1 3

14 Desert 0 0 0 1 0 1

15 Nevada 1 -99 -99 1 1 3

Total: 9 6 6 9 7 37
MODEL PERFORMANCE EVALUATION 

46 

Figure 2-2. Hourly averaged of CO, ozone, NO and NO2 for the Bay Area region over 

the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

47 

Figure 2-3. Hourly averaged CO, ozone, NO and NO2 for the Sacramento region over 

the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

48 

Figure 2-4. Hourly averaged CO, ozone, NO and NO2 for the Central San Joaquin 

Valley over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

49 

Figure 2-5. Hourly averaged CO, ozone, NO and NO2 for the Southern San Joaquin 

Valley over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

50 

Figure 2-6. Hourly averaged CO, ozone, NO and NO2 for the Northern San Joaquin 

Valley over the July 7-13, 1999 modeling period. 
MODEL PERFORMANCE EVALUATION 

51 

1-ho ur O3 Statistics, B ay A rea

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

1-ho ur O3 Statistics, Sacramento

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

1-ho ur O3 Statistics, C entral SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

52 

Figure 2-7. Unpaired peak ratio vs. normalized bias for 1-hour ozone for the July 9-13, 

1999 modeling period. Each dot represents one-day results for an individual site. 

 

1-ho ur O3 Statistics, So uthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

1-ho ur O3 Statistics, N o rthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

53 

8-ho ur O3 Statistics, B ay A rea

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

8-ho ur O3 Statistics, Sacramento

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

8-ho ur O3 Statistics, C entral SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

54 

Figure 2-8. Unpaired peak ratio vs. normalized bias for 8-hour ozone for the July 9-13, 

1999 modeling period. Each dot represents one-day results for an individual site. 

8-ho ur O3 Statistics, So uthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

8-ho ur O3 Statistics, N o rthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

55 

2.2.2 July-August 2000 Episode (CCOS Episode) 

The air quality model simulation covers the period from July 27, 2000, to August 2, 

2000. The first two days of the simulation are treated as a model spin-up period for 

which model performance is not considered. As a result, the statistical model 

performance assessment only covers the non-spin-up period, from July 29th through 

August 2nd. As described previously in Section 1.2.2, the NOAA placeholder MM5 

meteorological simulation is used for these air quality simulations. 

Performance statistics for 1-hour and 8-hour model performance were calculated for 

each region and are listed in Table 2-6 and 2-7, respectively. For days that a region 

meets the criteria a value of 1 is assigned. A value of 0 means that region doesn’t meet 

the criteria for the respective day and, if there is no model simulated concentrations 

above 60ppb, then -99 is assigned. The following paragraph summarizes the 

subregional statistical results. 

For the Bay Area region, the model performance meets both the 1-hour and 8-hour 

criteria on July 29, 2000 and August 2, 2000, but fails on all the other days. For the 

Sacramento region, 4 days meet the 1-hour criteria and 3 days meet the 8-hour criteria. 

For the Central San Joaquin Valley region, all 5 days meet both the 1-hour and 8-hour 

criteria. For the Southern San Joaquin Valley region, 4 days meet the 1-hour criteria 

and all 5 days meet the 8-hour criteria. For the Northern San Joaquin Valley region, only 

2 days meet both the 1-hour and 8-hour criteria. 

Figures 2-9 through 2-13 show the site-averaged time series of observed versus 

predicted CO, ozone, NO and NO2 for the Bay Area, Sacramento, Central San Joaquin 

Valley, Southern San Joaquin Valley and Northern San Joaquin Valley regions, 

respectively. The orange and blue lines represent observations and model predictions, 

respectively. Take note that the start hour of the simulation is 0600 PDT, so the first 

day has only 18 data points. Also, note that the 1-hour and 8-hour region-wide statistics 

are not directly calculated from site-averaged time series, but from the statistics of each 

station and then arithmetically averaged. The time series for individual stations for 

these five regions have also been plotted and are available via the ftp site and filename 

indicated in the Appendix. 

Figure 2-9 shows the hourly averaged CO, ozone, NO and NO2 for the Bay Area region. 

Predicted CO concentrations are over-predicted for the entire simulation period. Ozone 

concentrations are significantly over-predicted for the entire simulation period although 

predictions gradually begin to match observations by the last day. Predicted NO 

concentrations generally match the observed concentrations but are under-predicted for 

the last two days of the simulation period. The model predicts NO2 concentrations that 

have a significant diurnal variation, where the NO2 increases in the morning traffic hours 

and in the evening hours when the atmospheric boundary layer becomes less turbulent. 

However, the regional averaged observation does not exhibit this NO2 variation. 
MODEL PERFORMANCE EVALUATION 

56 

Figure 2-10 shows the hourly averaged CO, ozone, NO and NO2 for the Sacramento 

region. Predicted CO concentrations generally agree with the observations. Ozone 

concentrations show excellent agreement with observations for all days. The NO 

concentrations generally agree with the observations and clearly reproduce the early 

morning traffic peak. The model captures the magnitude as well as the diurnal variation 

of the NO2. 

Figure 2-11 shows the hourly averaged CO, ozone, NO and NO2 for the Central San 

Joaquin Valley region. Predicted CO concentrations are under-predicted for the entire 

simulation period. Ozone concentrations show excellent agreement with observations 

for all the days. The NO concentrations also generally agree with the observations and 

clearly reproduced the early morning traffic peak. The model captures the diurnal 

variation of the NO2, but predicted concentrations are generally less than the observed 

values.

Figure 2-12 shows the hourly averaged CO, ozone, NO and NO2 for the Southern San 

Joaquin Valley region. Predicted CO concentrations are under-predicted for the entire 

simulation period. Ozone concentrations show excellent agreement with observations 

for all the days. The NO concentrations also generally agree with the observations and 

clearly reproduced the early morning traffic peak. The model captures the magnitude 

as well as the diurnal variation of the NO2.

Figure 2-13 shows the hourly averaged CO, ozone, NO and NO2 for the San Joaquin 

Valley North region. Predicted CO concentrations generally match the observed 

concentrations through July 29th, but, the sharp peaks in the observations later in the 

episode are not captured in the simulated concentrations. These sharp peaks, 

however, are likely from some high-emitters near the observation site and cannot be 

captured by a regional scale model. Ozone concentrations generally agree with the 

observations, but some days are over-predicted. The NO concentrations also generally 

agree with the observations and clearly reproduced the early morning traffic peak. The 

predicted NO2 concentrations are generally higher than the observations. 

Figures 2-14 and 2-15 show the correlation between the unpaired peak and the mean 

normalized bias for each individual site in a region. Through experience it has been 

observed that, if the NB statistical metric is satisfied, then the GE statistic is also 

satisfied. Hence, GE statistical results are not presented. 

The results shown above were generated using NOAA’s ‘placeholder’ MM5 

meteorology. Since the first ‘placeholder’ version of the NOAA meteorology, CARB and 

NOAA have been working together to improve the meteorological model performance. 

A more recent NOAA meteorology, as described in the previous meteorology section, 

was developed and used as an alternative input to air quality modeling simulations. The 

model performance results using this alternative meteorological field are provided in 

Tables 2-8 and 2-9 for 1-hour and 8-hour averaging periods, respectively. Using the 

updated meteorology inputs slightly degrades the 1-hour model performance, but 
MODEL PERFORMANCE EVALUATION 

57 

improves the 8-hour model performance. However, as mentioned in section 1.2.2, this 

alternative wind field as well as the associated meteorological model options and inputs 

are still in draft form and continue to be investigated. 
MODEL PERFORMANCE EVALUATION 

58 

Table 2-6. 1-hour ozone performance by each region over the July 29-August 2, 2000 

modeling period. 

Region ID Region Name 7/29/2000 7/30/2000 7/31/2000 8/1/2000 8/2/2000 Total 

2 North Coast -99 0 -99 -99 -99 0

3 BAAQMD 1 0 0 0 1 2

4 MBAQMD 0 0 0 1 1 2

5 Sacramento Valley North 1 1 1 1 1 5

6 Sacramento Region 1 1 0 1 1 4

7 SJVAPCD Central 1 1 1 1 1 5

8 SJVAPCD Kern 1 1 1 1 0 4

9 SJVAPCD North 1 0 0 0 1 2

10 Sierra Nevada Central 0 1 0 1 1 3

11 SJVAPCD Above 3000 ft 1 1 1 1 1 5

12 South Central Coast 1 1 0 0 0 2

13 Sierra Nevada North 1 1 1 1 1 5

14 Desert 1 1 0 0 0 2

15 Nevada -99 -99 -99 -99 -99 0

Total: 10 9 5 8 9 41

Table 2-7. 8-hour ozone performance by each region over the July 29-August 2, 2000 

modeling period. 

Region ID Region Name 7/29/2000 7/30/2000 7/31/2000 8/1/2000 8/2/2000 Total 

2 North Coast -99 -99 -99 -99 -99 0

3 BAAQMD 1 0 0 0 1 2

4 MBAQMD -99 0 0 1 1 2

5 Sacramento Valley North 1 1 1 1 0 4

6 Sacramento Region 1 0 0 1 1 3

7 SJVAPCD Central 1 1 1 1 1 5

8 SJVAPCD Kern 1 1 1 1 1 5

9 SJVAPCD North 1 0 0 0 1 2

10 Sierra Nevada Central 1 1 0 1 0 3

11 SJVAPCD Above 3000 ft 1 1 1 1 0 4

12 South Central Coast 0 0 0 0 1 1

13 Sierra Nevada North 1 1 1 1 1 5

14 Desert 1 1 1 1 1 5

15 Nevada -99 -99 -99 -99 -99 0

Total: 10 7 6 9 9 41
MODEL PERFORMANCE EVALUATION 

59 

Figure 2-9. Hourly averaged CO, ozone, NO and NO2 for the Bay Area region over the 

July 27 – August 2, 2000 modeling period. 
MODEL PERFORMANCE EVALUATION 

60 

Figure 2-10. Hourly averaged CO, ozone, NO and NO2 for the Sacramento region over 

the July 27 – August 2, 2000 modeling period. 
MODEL PERFORMANCE EVALUATION 

61 

Figure 2-11. Hourly averaged CO, ozone, NO and NO2 for the Central San Joaquin 

Valley region over the July 27 – August 2, 2000 modeling period. 
MODEL PERFORMANCE EVALUATION 

62 

Figure 2-12. Hourly averaged CO, ozone, NO and NO2 for the Southern San Joaquin 

Valley region over the July 27 – August 2, 2000 modeling period. 
MODEL PERFORMANCE EVALUATION 

63 

Figure 2-13. Hourly averaged CO, ozone, NO and NO2 for the Northern San Joaquin 

Valley region over the July 27 – August 2, 2000 modeling period. 
MODEL PERFORMANCE EVALUATION 

64 

1-ho ur O3 Statistics, B ay A rea

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

1-ho ur O3 Statistics, Sacramento

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

1-ho ur O3 Statistics, C entral SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

65 

Figure 2-14. Unpaired peak ratio vs. normalized bias for 1-hour ozone for the July 29 – 

August 2, 2000 modeling period. Each dot represents one-day results for an individual 

site. 

1-ho ur O3 Statistics, So uthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

1-ho ur O3 Statistics, N o rthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

66 

8-ho ur O3 Statistics, B ay A rea

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

8-ho ur O3 Statistics, Sacramento

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

8-ho ur O3 Statistics, C entral SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

67 

Figure 2-15. Unpaired peak ratio vs. normalized bias for 8-hour ozone for the July 29 – 

August 2, 2000 modeling period. Each dot represents one-day results for an individual 

site. 

8-ho ur O3 Statistics, So uthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio

8-ho ur O3 Statistics, N o rthern SJV

-0.4

-0.3

-0.2

-0.1

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.4 0.6 0.8 1.0 1.2 1.4 1.6 1.8 2.0

Unpair Peak Ratio
MODEL PERFORMANCE EVALUATION 

68 

Table 2-8. 1-hour ozone performance by each region over the July 27-August 2, 2000 

modeling period using the most recent NOAA meteorology (under ARB investigation) 

Region ID Region Name 7/29/2000 7/30/2000 7/31/2000 8/1/2000 8/2/2000 Total 

2 North Coast -99 0 -99 -99 -99 0 

3 BAAQMD 1 1 1 1 1 5 

4 MBAQMD 1 1 1 1 1 5 

5 Sacramento Valley North 1 1 1 1 1 5 

6 Sacramento Region 0 1 1 0 1 3 

7 SJVAPCD Central 0 1 1 1 1 4 

8 SJVAPCD Kern 0 0 1 0 0 1 

9 SJVAPCD North 1 1 1 1 1 5 

10 Sierra Nevada Central 0 1 1 0 0 2 

11 SJVAPCD Above 3000 ft 0 0 0 1 1 2 

12 South Central Coast 1 1 0 0 0 2 

13 Sierra Nevada North 1 1 1 0 0 3 

14 Desert 0 0 0 0 0 0 

15 Nevada -99 -99 -99 -99 -99 0 

 

Total: 6 9 9 6 7 37 

Table 2-9. 1-hour ozone performance by each region over the July 27-August 2, 2000 

modeling period using the most recent NOAA meteorology (under ARB investigation) 

Region ID Region Name 7/29/2000 7/30/2000 7/31/2000 8/1/2000 8/2/2000 Total 

2 North Coast -99 -99 -99 -99 -99 0 

3 BAAQMD 1 0 1 1 1 4 

4 MBAQMD -99 1 1 1 1 4 

5 Sacramento Valley North 1 1 1 1 1 5 

6 Sacramento Region 1 1 1 1 1 5 

7 SJVAPCD Central 1 1 1 1 1 5 

8 SJVAPCD Kern 1 1 1 1 1 5 

9 SJVAPCD North 1 0 0 1 1 3 

10 Sierra Nevada Central 1 1 1 0 0 3 

11 SJVAPCD Above 3000 ft 0 1 1 1 1 4 

12 South Central Coast 1 1 1 1 1 5 

13 Sierra Nevada North 1 1 1 1 0 4 

14 Desert 1 0 1 1 1 4 

15 Nevada -99 -99 -99 -99 -99 0 

 

Total: 10 9 11 11 10 51 
MODEL PERFORMANCE EVALUATION 

69 

3 Conclusion – Cumulative 1-hour and 8-hour Days 

Per the prior discussion of performance statistics and analyses of model performance, 

Tables 3-1 and 3-2 provide a summary of 1-hour and 8-hour episode days that meet 

model performance criteria for both episodes. As noted previously with regard to these 

types of tables, for days that a region meets the associated performance criteria a value 

of 1 is assigned. A value of 0 means that region doesn’t meet the criteria for the 

respective day and, if there is no model simulated concentrations above 60ppb, then 

-99 is assigned. 

As is illustrated in the tables, of 10 possible days per region (5 per episode), 2-10 days 

are available for each region based on 1-hour metrics and, with the exception of the 

North Coast (1 day), 1-9 days are available based on 8-hour metrics. 
MODEL PERFORMANCE EVALUATION 

70 

Table 3-1. Combined Number of Available Days Per Subregion Under 1-hour Metrics 

Region Name 

July 

1999 

July-Aug 

2000 Total 

North Coast 2 0 2 

BAAQMD 4 2 6 

MBAQMD 2 2 4 

Sacramento Valley North 3 5 8 

Sacramento Region 5 4 9 

SJVAPCD Central 5 5 10 

SJVAPCD Kern 1 4 5 

SJVAPCD North 2 2 4 

Sierra Nevada Central 1 3 4 

SJVAPCD Above 3000 ft 3 5 8 

South Central Coast 3 2 5 

Sierra Nevada North 3 5 8 

Desert 0 2 2 

Nevada 2 0 2 

Total 36 41 77 

Table 3-2. Combined Number of Available Days Per Subregion Under 8-hour Metrics 

Region Name 

July 

1999 

July-Aug 

2000 Total 

North Coast 1 0 1 

BAAQMD 1 2 3 

MBAQMD 1 2 3 

Sacramento Valley North 4 4 8 

Sacramento Region 5 3 8 

SJVAPCD Central 3 5 8 

SJVAPCD Kern 4 5 9 

SJVAPCD North 3 2 5 

Sierra Nevada Central 2 3 5 

SJVAPCD Above 3000 ft 3 4 7 

South Central Coast 3 1 4 

Sierra Nevada North 3 5 8 

Desert 1 5 6 

Nevada 3 0 3 

Total 37 41 78 
MODEL PERFORMANCE EVALUATION 

71 

4 Results for Model Sensitivity Tests 

(Work in Progress.) 
MODEL PERFORMANCE EVALUATION 

72 

5 References 

Air Sciences, Inc. (2002) 1996 Fire Emissions Inventory. Draft Final Report. 

Lakewood, CO. At: 

http:/www.wrapair.org/forums/FEJF1/emissions/FEJF1996EIReport_021208.pdf 

Battye, W. and Battye, R. 2002. Development of emissions inventory methods for 

wildland fires. U.S. EPA Contract No. 68-D-98-046. On-line link: 

http://www.epa.gov/ttn/chief/ap42/ch13/related/c13s01.html

Benjamin, M., Sudol, M., Bloch, L. and A. Winer. 1996. Low-emitting urban forests: a 

taxonomic methodology for assigning isoprene and monoterpene emission rates. 

Atmospheric Environment. 30 (9): 1437-1452. 

Benjamin, M., Sudol, M., Vorsatz, D. and A. Winer. 1997. A spatially and temporally 

resolved biogenic hydrocarbon emissions inventory for the California south coast air 

basin. Atmospheric Environment. 31 (18): 3087-3100.

CARB. 1995. "Sacramento Area Modeling Analysis for the 1994 State Implementation 

Plan". California Air Resources Board. Technical Support Division. Sacramento, CA 

95814. April, 1995. 

CARB. 2005 ARB Speciation Profiles: (a) California Air Resources Board, 

"Identification of Volatile Organic Compound Species Profiles: ARB Speciation Manual, 

Second Edition, Volume 1 of 2," August 1991; and (b) subsequent revisions. The latest 

ARB speciation profiles are available from ARB's web site at 

http://www.arb.ca.gov/emisinv/speciate/speciate.htm. 

CARB. 2005. http://www.arb.ca.gov/airways/ccos/ccos.htm

CARB. “Extension and input refinement to the ARB wildland fire emissions estimation 

model,” ARB agreement number 00-729 

Carter, W.P.L, 2000. “Documentation of the SAPRC-99 Chemical Mechanism for VOC 

Reactivity Assessment,” Final Report to California Air Resources Board, Contract No. 

92-329, and (in part) 95-308,May 8, 2000, http://pah.cert.ucr.edu/~carter/reactdat.htm

Chang, J. S., J. Shengxin, L. Yinghong, M. Beauharnois, L., Cheng-Hsuan, and H. HoChun. 1997. The SARMAP Air Quality Model. Planning and Technical Support Division. 

California Air Resources Board. Sacramento, CA. April, 1997. 

Coe, D. 2003. “Assisstance to Rural Counties with Development of Area Source 

Emission Inventories.” Technical Memoranda. Contract No. 00-24CCOS. San Joaquin 
MODEL PERFORMANCE EVALUATION 

73 

Valleywide Air Pollution Study Agency and California Environmental Protection Agency 

– Air Resources Board. At: www.sonomatech.com/ccosii/; user name: “ccosii”; 

password: “emissions” 

Cofer, W., Levine, J., Winstead, E. and Stocks, B. 1991. Trace gas and particulate 

emissions from biomass burning in temperate ecosystems. In: Global Biomass Burning: 

Atmospheric, Climatic and Biospheric Implications. Levine, J., editor. MIT Press. 

Davis, F. W., P. A. Stine, D. M. Stoms, M. I. Borchert and A. D. Hollander. 1995. Gap 

analysis of the actual vegetation of California –1. The southwestern region. Madrono 42: 

40-78. 

Earth Tech. 2000. A User's Guide for the CALMET Meteorological Model (Version 5). 

Earth Tech. Concord, MA. 01742. January, 2000. 

Einfeld, W., Ward, D. and Hardy, C. 1991. Effects of fire behavior on prescribed fire 

smoke characteristics: a case study. In: Global Biomass Burning: Atmospheric, Climatic 

and Biospheric Implications. Levine, J., editor. MIT Press. 

ENVIRON. 2001. The program 'mm5camx_v3'. Personal communication. ENVIRON. 

Novato, CA. 95945. August, 2001. 

ENVIRON. 2004. User's Guide. Comprehensive Air Quality Model With Extensions 

(CAMx) v4.10S. ENVIRON. Novato, CA 95945. August, 2004. 

ENVIRON. 2005a. Personal communication. ENVIRON. Novato, CA. 95945. February, 

2005. 

ENVIRON. 2005b. The Development of a Photochemical Modeling System for the Bay 

Area Air Quality Management District to Support Ongoing Air Quality Analysis. 

BAAQMD. San Francisco, CA. January, 2005. 

Fairley, D. and DeMandel, R. 1996. An Analysis of SARMAP Episode Day 

Representativeness. Final report. Prepared for the SARMAP Data Analysis Project by 

the Bay Area Air Quality Management District, San Francisco, CA, Faust, B. C., 

Photochemistry of clouds, fogs and aerosols, Environ. Sci. Technol., 28A, 217, 1994. 

Fujita, E., D. Campbell, R. Keilar, and J. Bowen. 2001. Central California Ozone Study – 

Volume III: Summary of Field Operations. Final Report. DRI. California Air Resources 

Board. Sacxramento, CA 95814. February, 2001. 

Fujita, E., D. Campbell, R. Keisler, and J. Bowen. 1999. "Central California Ozone Study 

– Volume III. Summary of Field Operations". Planning and Technical Support Division, 

California Air Resources Board. Sacramento, CA. 95814. November, 1999. 

Funk, T., Stiefer, P., Chinkin, L. 2001 “Development of Gridded Spatial Allocation 

Factors for the State of California”. Final Report. Contract No. 00-24CCOS. San 
MODEL PERFORMANCE EVALUATION 

74 

Joaquin Valleywide Air Pollution Study Agency and California Environmental Protection 

Agency – Air Resources Board. 

Grell, A. G., J. Dudhia, and D. R. Stauffer. 1994. "A Description of the Fifth-Generation 

Penn State/NCAR Mesosclae Model (MM5)". NCAR Tecnical Note NCAR/TN398+STR. National Center for Atmospheric Research. Boulder, CO. June, 1994. 

Griffith, D., Mankin, W., Coffey, M., Ward, D., Riebau, A. 1991. FTIR remote sensing of 

biomass burning emissions of CO2, CO, CH4, CH2O, NO, NO2, NH3, and N2O. In: 

Global Biomass Burning: Atmospheric, Climatic and Biospheric Implications. Levine, J., 

editor. MIT Press. 

Guenther, A. B., P. R. Zimmerman, P. C. Harley, R. K. Monson and R. Fall. 1993. 

Isoprene and monoterpene emission rate variability – model evaluations and sensitivity 

analyses. Journal of Geophysical Research. 98(D7): 12609-12617. 

Guenther, A. B. R. K. Monson and R. Fall. 1991. Isoprene and monoterpene emission 

rate variability: observations with eucalyptus and emission rate algorithm development. 

Journal of Geophysical Research. 96: 10799-10808. 

Hayes, T. P., J. J. R. Kinney, and N. J. M. Wheeler. 1984. "California Surface Wind 

Climatology". Planning and Technical Support Division. California Air Resources Board. 

Sacramento, CA. 95812. June, 1984. 

Karlik, J. and A. McKay. 1999. Development of methodology and databases for 

estimating leaf masses in California airsheds. Final Report. Contract No. 97-719. State 

of California Air Resources Board. Sacramento, CA. 

Karlik, J. 2002. Validation of databases for modeling biogenic volatile organic compound 

emissions in central California. Draft Final Report. Contract No. 00-16CCOS. San 

Joaquin Valleywide Air Pollution Study Agency and California Environmental Protection 

Agency – Air Resources Board. 

Harley, P., V. Fridd-Stroud, J. Greenberg, A. Guenther and P. Vasconcellos. 1996. 

Emission of 2-methyl-3-buten-2-ol by pines: A potentially large natural source of 

reactive carbon to the atmosphere. Journal of Geophysical Research. 103: 25479-

25486. 

Horie, Y., Sidawi, S. and R. Ellefsen. 1990. Inventory of leaf biomass and emission 

factors for vegetation in California’s south coast air basin. Final Technical Report III-C. 

South Coast Air Quality Management District. Diamond Bar, CA. 

Hunman, R.C., D. J. Jacob, O.R. Cooper, M. J. Evans, C.L. Heald, R.J. Park, F. 

Fehsenfeld, F. Flock, J. Holloway, G. Hubler, K., Kita, M. Koike, Y. Kondo, A. Neuman, 

J. Nowak, S. Oltmans, D. Parrish, J. M Roberts, abd T. Ryerson. 2004. Ozone 
MODEL PERFORMANCE EVALUATION 

75 

production in transpacific Asian pollution plumes and implications for ozone air quality in 

California. J. Geophys Research, 109:D23S10. 

Kleeman, M. 2000. “Updating Point Source Emissions Inventories in the Sacramento 

Valley and Mountain counties Using Student Assistants” Contract No. 00-22CCOS. 

Lam, T. Niemeier, D., Jierranaitanakit, K. 2002. “Estimation of Hourly Allocation Factors 

for the Central California Ozone Study Using State-wide Model Data and Real Time 

Traffic Data.” Final Draft Report. Contract No. 00-04PM. San Joaquin Valleywide Air 

Pollution Study Agency and California Environmental Protection Agency – Air 

Resources Board. 

Lerhman, D., B. Knuth, and D. Fairly. 2003. "Characterization of the CCOS 2000 

Measurment Period". T&B Systems Contract No. 01-2CCOS. California Air Resources 

Board -- Planning and Technical Support Division. Sacramento, CA 95814. September, 

2003. 

Lobert, J., Scharffe, D., Hao, W.-M., Kuhlbusch, T., Seuwen, R., Warneck, P., and 

Crutzen, P. 1991. Experimental evaluation of biomass burning emissions: nitrogen and 

carbon containing compounds. In: Global Biomass Burning: Atmospheric, Climatic and 

Biospheric Implications. Levine, J., editor. MIT Press. 

McRae, G.J, W.R. Goodin, and J.H. Seinfeld. 1982, Mathematical Modeling of Air 

Pollution. EQL Report No. 18. Planning and Technical Support Division, CARB. 

Sacramento, CA 95814. April, 1982. 

McPherson, E. G. 1998. Structure and sustainability of Sacramento’s urban forest. 

Journal of Arboriculture. 24 (4): 174-190. 

Mesinger, F., Z. I. Janjic, S. Nickovic, D. Gavrilov, and D. G. Deaven, 1988: The stepmountain coordinate: model description and performance for cases of Alpine lee 

cyclogenesis and for a case of an Appalachian redevelopment. Mon. Wea. Rev., 116, 

1493-1518. 

Myrup, L.O., R.G. Flocchini, and D. Ewell (1981) Transport of Atmospheric Aerosols 

Above the Sierra Nevada Slopes, Final Report, Contract A4-127-32. Prepared by 

University of California Davis, January 15, 1989. 

Neff, W.D., J. Jorden. J. Gaynor, D. Wolfe, W. Ecklund, D. Carter, and K. Gage (1991) 

The use of 915 MHz radar wind profilers in complex terrain and regional air quality 

studies. Preprints, Seventh Joint Conference on Applications of Air Pollution 

Meteorology with AWMA. 14-18 January 1991, New Orleans, LA, American 

Meteorological Society. Boston, MA., J230-J233. 
MODEL PERFORMANCE EVALUATION 

76 

Newchurch, M. J., M. A. Ayoub, S. Oltmans, B. Johnson, and F.J. Schmidlin. 2003. 

Vertical distribution of ozone at four sites in the United States. J. of Geophys. Research, 

108(D1):ACH 9-1,17. 

Nikolov, N. T. 1999. 1-km resolution database of vegetation leaf area index and canopy 

clumping factor for the western U.S.A. Final Report, U.S.D.A. Forest Service Agreement 

No. PSW-99-001-RJVA. N&T Services. Oak Ridge, TN. 

Nowak, D. J. 1991. Urban forest development and structure: Analysis of Oakland, 

California. PhD dissertation. University of California, Berkeley, CA. 

Pielke, R. A., W. R. Cotton, R. L. Walko, C. J., Tremback, W. A. Lyons, L. D. Grasso, 

M. E. Nicholls, M. D. Moran, D. A. Wesley, T. J. Lee, and J. H. Copeland, 1992: A 

comprehensive meteorological modeling system – RAMS. Meteorology and 

Atmospheric Physics, 49, 69-91. 

Pielke, R. A. and M. Uliasz. 1998. Use of meteorological models as input to Regional 

and mesoscale air quality models -- limitations and strengths. 1998. 

Atmospheric Environment 32:1455-1466. 

Pun, B. K., J. F. Louis, and C. Seigneur. 1998. "A Conceptual Model for ozone 

formation in the San Joaquin Valley". AER Document No. CP049-1-98. Pacific Gas and 

Electric Co. San Ramon, CA 94583. December, 1998. 

Radke, L., Hegg, D., Hobbs, P., Nance, D., Lyons, J., Laursen, K., Weiss, R., Riggan, 

P., and Ward, D. 1991. Particulate and trace gas emissions from large biomass fires in 

north America. In: Global Biomass Burning: Atmospheric, Climatic and Biospheric 

Implications. Levine, J., editor. MIT Press. 

Reinhardt, E., Keene, R. and Brown, J. 1997. First Order Fire Effects Model: FOFEM 

4.0 User’s Guide. USDA Forest Service, Intermountain Research Station, General 

Technical Report INT-GTR-344. 

Seaman, N.L., D. R. Stauffer, and A. M. Lario-Gibbs. 1995. A multiscale fourdimensional data assimilation system applied to the San Joaquin Valley during 

SARMAP. Part I: Modeling design and basic model performance characteristics. J. of 

Applied Meteorology 34:1739-1761. 

SCAQMD. 2003. "2003 Air Quality Management Plan -- Modeling and Attainment 

Demonstrations." Final Report -- Appendix V. South Coast Air Quality Management 

District. Diamond Bar, CA 91765. August, 2003. 

Scire J.S., R.J. Yamartino, S.R. Hamma, G.R. Carmichal, and Y.S. Chang. 1989. 

CALGRID: A Mesoscale Photochemical Grid Model. Volume I: Model Formulation 
MODEL PERFORMANCE EVALUATION 

77 

Document. Planning and Technical Support Division, CARB. Sacramento, CA 95814. 

June, 1989. 

Sidawi, S. and Y. Horie. 1992. Leaf biomass density for urban, agricultural and natural 

vegetation in California’s San Joaquin Valley. Final Report. San Joaquin Valley Air 

Pollution Study Agency. 

SJVAQMD. 1994. "The Ozone Attainment Plan". San Joaquin Valley Unified Air 

Pollution Control District. Fresno, CA. November, 1994 

Skaramarock, W. C., J. B. Klemp, J. Dudhia, G. O. Gill, D. M. Barker, W. Wang, and J. 

G. Powers, 2005: A description of the Advanced Research WRF Version 2. NCAR 

Technical Note NCAR/TN-468+STR, June 2005. 

Smith, T.B., D.E. Lehrman, D.D Reible, and F.H. Shair (1981) The origin and fate of 

airborne pollutnats within the San Joaquin Valley, Rep. MRI FR-1838, Meteorology 

Research, Inc., Altadena, CA. Prepared for the California Air Resources Board. 

Smith, T. B. (1994) Ozone Episode Forecasting in the San Joaquin Valley. Planning 

and Managing Regional Air Quality Modeling and Measurement Studies: A Perspective 

Through the San Joaquin Valley Air Quality Study and AUSPEX, ed. by P. A. Solomon. 

Published by Lewis Publishers, Chelsea, MI in conjunction with Pacific Gas and Electric 

Company, San Ramon, CA, pp. 507-528. 

Susott, R., Ward, D., Babbitt, R. and Latham, D. 1991. The measurement of trace gass 

emissions and combustion characteristics for a mass fire. In: Global Biomass Burning: 

Atmospheric, Climatic and Biospheric Implications. Levine, J., editor. MIT Press. 

Tonneson, G. 2003. Personal Communication. CE-CERT. U.C. Riverside, Riverside, 

CA. April, 2003. 

Ward, D. and Hardy, C. Smoke emissions from wildland fires. 1991. Environment 

International. V17. Pp.117-134. 

Wilkinson, J. 2003, “Development of the California Integrated Transportation Network 

(ITN)”, Draft Final Report. Contract No. 93-2PM. San Joaquin Valleywide Air Pollution 

Study Agency and California Environmental Protection Agency – Air Resources Board. 

Winer, A., Karlik, J. and J. Arey. 1998. Biogenic hydrocarbon inventories for California: 

generation of essential databases. Final Report. Contract No. 95-309. State of 

California Air Resources Board. Sacramento, CA. 

Winer, A. and Karlik, J. 2001. Development and validation of databases for modeling 

biogenic hydrocarbon emissions in California’s airsheds. Final Report. Contract No. 97-
MODEL PERFORMANCE EVALUATION 

78 

320. California Environmental Protection Agency – Air Resources Board. Sacramento, 

CA. 

USEPA. 1990. Carbon Bond IV: Morris, R.E. and Meyers, T. E., "User's Guide for the 

Urban Airshed Model," Volume I, Appendix 1: "The Carbon Bond IV Chemical Kinetics 

Mechanism for Urban and Regional Scale Computer Modeling," EPA-450/4-90-007A, 

June 1990. 

USEPA. 1991. Guideline for the Regulatory Application of the Urban Airshed Model. 

OAQPS, USEPA. Research Triangle Park, NC 27711. 

USEPA, 1991. "Guideline for Regulatory Application of the Urban Airshed Model". EPA450/4-91-013. USEPA, OAQPS. Research Triangle Park, NC 27711. July, 1991. 

USEPA. 1999. "Science Algorithms of the EPA Models-3 Community Multiscale Air 

Quality Model (CMAQ) Modeling System. USEPA. EPA/600/R-99/030. Office of 

Research and Development. Washington, D. C. 20460. March, 1999. 

USEPA, 2005. "Guidance on the Use of Models and Other Analysis in the Attainment 

Demonstration for the 8-Hour Ozone NAAQS". USEPA, OAR/OAQPS, Research 

Triangle Park, NC 2771. October, 2005. 
6 Appendix – Information Available for Downloading 

Anonymous ftp to eos.arb.ca.gov, then change directories to /pub/outgoing/model_protocol2 

Model Episode Task/Item Complete? File Name on eos.arb.caAQ 1999Regional daily tabulation of 1-hour performance results (1,0,-99) Y Included in document AQ 1999Station-specific tabulation of 1-hour performance results (1,0,-99) Y 1999 ozone performance AQ 1999Station-specific tabulation of 1-hour performance STATISTICS Y 1999.050c.1hO3.doc AQ 1999Station-specific time-series plots of 1-hour ozone Y 1999.pdf.zip AQ 1999

Station-specific time-series plots of 1-hour precursors (CO, NO, 

NO2); ok to combine w/ ozone Y 1999.pdf.zip 

AQ 1999Regional daily tabulation of 8-hour performance results (1,0,-99) Y Included in document AQ 1999Station-specific tabulation of 8-hour performance results (1,0,-99) Y 1999 ozone performance AQ 1999Station-specific tabulation of 8-hour performance STATISTICS Y 1999.050c.8hO3.doc AQ 2000Regional daily tabulation of 1-hour performance results (1,0,-99) Y Included in document AQ 2000Station-specific tabulation of 1-hour performance results (1,0,-99) Y 2000 ozone performance AQ 2000Station-specific tabulation of 1-hour performance STATISTICS Y 2000.050c.1hO3.doc AQ 2000Station-specific time-series plots of 1-hour ozone Y 2000.pdf.zip AQ 2000

Station-specific time-series plots of 1-hour precursors (CO, NO, 

NO2); ok to combine w/ ozone Y 2000.pdf.zip 

AQ 2000Regional daily tabulation of 8-hour performance results (1,0,-99) Y Included in document AQ 2000Station-specific tabulation of 8-hour performance results (1,0,-99) Y 2000 ozone performance AQ 2000Station-specific tabulation of 8-hour performance STATISTICS Y 2000.050c.8hO3.doc 
MODEL PERFORMANCE EVALUATION 

80 

Anonymous ftp to eos.arb.ca.gov, then change directories to /pub/outgoing/model_protocol2 

Model Episode Task/Item Complete? File Name on eos.arb.caMet 1999

Wind Speed Statistics per Performance Region (RMSE < 2 m/s; Bias::< ±0.5 m/s; IOA:3 0.6) Y Included in document 

Met 1999

Wind Direction Statistics per Performance Region(Gross Error:< 30 deg; Bias:< ±10 deg) Y Included in document 

Met 1999

Temperature Statistics per Performance Region(Gross Error:< 2 K; Bias:< ±0.5 K; IOA3 0.8) Y Included in document 

Met 1999Station-specific, time-series plots of hourly mean air temperature Y July1999.met.regionN.pdfMet 1999Station-specific, time-series plots of hourly mean wind speeds. Y same as above Met 1999Domain-wide spatial plots of hourly wind vectors Y 

July1999_surface_hourly_EEEE is simulation ID 

Met 1999Domain-wide, spatial plots of hourly air temperatures N Met 2000

Wind Speed Statistics per Performance Region (RMSE < 2 m/s; Bias::< ±0.5 m/s; IOA:3 0.6) Y Included in document 

Met 2000

Wind Direction Statistics per Performance Region(Gross Error:< 30 deg; Bias:< ±10 deg) Y Included in document 

Met 2000

Temperature Statistics per Performance Region(Gross Error:< 2 K; Bias:< ±0.5 K; IOA3 0.8) Y Included in document 

Met 2000Station-specific, time-series plots of hourly mean air temperature Y <<to be posted>> Met 2000Station-specific, time-series plots of hourly mean wind speeds. Y <<to be posted>> Met 2000Domain-wide spatial plots of hourly wind vectors Y Included in document Met 2000Domain-wide, spatial plots of hourly air temperatures N