Model Performance Evaluation

Introduction

This section documents an operational analysis of the CMAQ model runs used to support

the attainment demonstration for Philadelphia. The purpose of this effort is to validate the

model’s ability to replicate observed concentrations of ozone. This analysis was prepared in

accordance with EPA guidance1 and the protocol document previously submitted to the

EPA.2

Descriptive Statistics

Pennsylvania has calculated the following measures comparing the hourly ozone and

8-hourly maxima for each ozone monitor location of interest from May 15th through

September 30th, 2002.3

• Mean Normalized Bias (MNB): “This performance statistic averages the

model/observation residual, paired in time, normalized by observation, over all

monitor times and locations.”4

MNB =

 Xn

i=1

 

(Modeli − Obsi)

Obsi

!! ×

100%

n

(1)

• Mean Normalized Gross Error (MNGE): “This performance statistic averages the

absolute value of the model/observation residual, paired in time, normalized by

observation, over all monitor times and locations.”5

MNGE =

 Xn

i=1 

|(Modeli − Obsi)|

Obsi

!! ×

100%

n

(2)

• Normalized Mean Bias (NMB): Because of the tendency of the MNB statistic,

defined above, to inflate the importance of biases at low observed values, the US EPA

recommends using NMB for calculations that have no threshold.6

1US EPA, Guidance on the Use of Models and Other Analyses for Demonstrating Attainment of Air

Quality goals for Ozone, PM2.5, and Regional Haze, Draft 3.2, 176–201 (Sept., 2006).

2PA DEP, Final Draft Protocol for Regulatory Photochemical Air Quality Modeling for the Philadelphia

Eight-Hour Ozone Nonattainment Area, (June, 2006).

3While equations 1 and 2 differ from those printed in the EPA’s draft guidance (EPA 2006, at 184), the

DEP believes the versions used here accurately reflect the correct methodology.

4US EPA 2006, at 184.

5

Id.

6US EPA 2006, at 184 n.47.

1
NMB =

Pn

i=1 (Modeli − Obsi)

Pn

i=1 (Obsi)

× 100% (3)

• Normalized Mean Error (NME): Because of the tendency of the MNGE statistic,

defined above, to inflate the importance of errors at low observed values, the US EPA

recommends using NME for calculations that have no threshold.7

NME =

Pn

i=1 |Modeli − Obsi

|

Pn

i=1 (Obsi)

× 100% (4)

• Average Peak Prediction Bias and Error (APPB & APPE): This set of statistics uses

the same equations as the calculation for NMB and NMGE, listed above, but only

consider the daily maximum 8-hour ozone prediction at each monitor. Because there

is no implicit threshold in this calculation, Pennsylvania applied calculations as

described in equations 3 and 4.

The error statistics provide an overall assessment of model performance and can be

interpreted as precision. The bias parameter measures a model’s ability to reproduce

observed spatial and temporal patterns and can be interpreted as accuracy. These statistics

were calculated using to two different datasets. The first is for all observed 8-hour ozone

concentrations during the 2002 ozone season. The second set uses the same methodology,

but it is limited to only those hours where the observed concentrations exceed 60ppb.

In addition to the calculations for each monitor location, the statistics are also calculated

over all of the sites and time period in the region. In addition to the region-wide data, this

document also provides a separate analysis where only the monitors in Pennsylvania are

averaged.

Data

Table 1 lists the monitors used in this analysis along with the model i-cell and j-cell

associated with their locations:

The ozone monitor data were obtained from the EPA Air Quality System (AQS) national

database using the standard retrieval report “Raw Data NAAQS Averages”

(AMP350NW). Calculations to create the running 8-hour ozone averages are based on EPA

guidelines. The data submitted to AQS by state and local agencies is expected to have

passed internal quality control checks and been certified by the senior air quality control

manager for the submitting agency.

Model data was provided by the NY DEC. This consisted of the

“OTC2002 BaseB1 12CCTM v451 cb4 ae3.ACONC.2002 BaseB1.*” series of data sets

calculated using the SMOKE/CMAQ modeling system using 2002 emissions data and

MM5 meteorology.8

7

Id.

8See generally: NY DEC, CMAQ Model Performance Assessment, 8-Hr OTC Modeling, TSD-1e, (Feb.,

2006).

2
Table 1: Monitor locations

Site ID Location State I-cell J-cell

10-001-0002 Killens Pond DE 131 83

10-003-1003 River Road Park DE 130 90

10-003-1007 Lums Pond DE 128 88

10-003-1010 Brandywine DE 129 91

10-003-1013 Bellefonte DE 129 90

10-005-1002 Seaford DE 131 80

10-005-1003 Lewes DE 134 82

24-015-0003 Fair Hill MD 127 89

34-001-0005 Nacote Creek NJ 137 90

34-007-0003 Camden NJ 132 92

34-007-1001 Ancora St. Hos NJ 134 91

34-011-0007 Millville NJ 133 88

34-015-0002 Clarksboro NJ 132 91

34-021-0005 Rider Univ. NJ 133 96

34-029-0006 Colliers Mills NJ 136 95

42-017-0012 Bristol PA 133 94

42-029-0050 West Chester PA 128 92

42-029-0100 New Garden PA 127 90

42-045-0002 Chester PA 130 91

42-091-0013 Norristown PA 130 94

42-101-0004 Frankford PA 132 93

42-101-0014 Northwest PA 131 93

42-101-0024 Northeast PA 132 94

42-101-0136 Southwest PA 131 92

Operational Evaluation

Table 2 (page 5) shows the Normalized Mean Bias and Error, which were calculated using

equations 3 and 4. The results are given for each monitor location, averaged for all monitor

locations, and averaged across the monitors within Pennsylvania. In addition, this table

shows the Mean Normalized Bias and Gross Error, calculated using equations 3 and 4,

when the recommended9 60ppb threshold is applied. Table 3 (see page 6) presents the

Average Peak Prediction Bias and Gross Error in a similar fashion.

Figure 1 (page 7) compares monitor data with actual modeled results. The “monitor” data

are averages of the maximum 8-hour ozone concentration recorded at each of the

twenty-four monitor locations listed in Table 1. The “model” data uses a similar approach,

averaging the maximum 8-hour ozone concentration calculated for the grid cells occupied

by those monitors. Figure 2 presents the same data, but limits the set to the nine monitors

located in Pennsylvania.

Figures 3 and 4 (page 8) compare the mean normalized bias with the mean normalized

9EPA 2006, at 185.

3
g

g q

on page 1, above. Except that, rather than depicting the mean seasonal data, the selected

monitor data was averaged for each day of the season.

Figures 5 and 6 (page 9) are scatter plots that compare the monitor data to the model’s

predictions. In those figures, the solid horizontal bar is the 65ppb threshold—the dividing

line between “Green” and “Yellow” on the Air Quality Index (AQI) scale. The center

diagonal dashed lines represents a 1:1 relationship, while the outer lines represent a range

of (±34.1%) of the 1:1 line’s value—that is, one standard deviation assuming a normal

distribution. For values above 65ppb recorded at all monitor locations, the coefficient of

determination (r2) for the data, is 35%.

Figure 7 (page 10) depicts the range of model to monitor variation. On that graph, the

solid line shows the average variation of all of the monitors. In addition, vertical error bars

shows the range of variation found at individual monitors.

Summary of Model Performance

Figures 1 and 2 show that CMAQ’s prediction of daily maximum ozone concentrations

corresponds very well with the observed data. Table 3 (page 6) supports this, showing that

the predictions fall within the EPA’s recommended range for bias (±15%) and error (less

than 30%.)

While Figures 3 and 4 show spikes in the calculated bias and error, further analysis

indicates that these will not impact the attainment demonstration. On the 27 days where

the bias exceeds the of 30% limit, the average daily maximum 8-hour ozone concentration

averages 36ppb and never exceeds 56ppb.

Pennsylvania also calculated the normalized bias and error for those episode days which

could potentially have impacted the design value at any of the Commonwealth’s nine

monitors—that is, the four highest values in 2002. For these episodes, the model’s

performance was even better with a resulting bias of 7.1% and and error of 12.2%.

For these reasons, Pennsylvania feel the modeling system does an adequate job of

estimating the 8-hour surface ozone concentrations throughout the Philadelphia area.

4
Table 2: Bias and error for predicted hourly 8-hr ozone concentrations.

NMB NME MNB ≥60 MNGE ≥60

Site ID (%) (%) (%) (%)

10-001-0002 34.70 39.21 49.90 54.36

10-003-1003 28.52 35.69 29.36 32.70

10-003-1007 54.85 56.91 67.76 70.14

10-003-1010 18.86 30.62 14.49 26.38

10-003-1013 39.70 43.26 45.81 48.10

10-005-1002 24.30 32.80 30.66 43.17

10-005-1003 31.02 34.35 37.76 40.53

24-015-0003 17.13 25.89 17.49 27.16

34-001-0005 45.13 47.24 70.42 72.34

34-007-0003 13.34 29.35 11.94 24.11

34-007-1001 12.68 23.38 7.13 18.87

34-011-0007 27.16 33.63 32.61 39.93

34-015-0002 19.20 30.49 21.16 31.96

34-021-0005 17.83 29.47 4.17 18.62

34-029-0006 12.50 25.31 4.01 19.43

42-017-0012 31.27 39.86 15.01 26.69

42-029-0050 17.17 28.80 20.38 34.90

42-029-0100 5.51 21.30 6.14 24.58

42-045-0002 19.11 30.06 21.14 33.05

42-091-0013 26.63 33.73 26.57 32.89

42-101-0004 66.76 68.69 73.18 73.80

42-101-0014 21.36 31.20 22.43 30.61

42-101-0024 15.04 27.34 9.76 24.23

42-101-0136 42.56 47.72 45.05 48.69

All Sites 25.53 34.28 28.88 37.68

PA Sites 25.01 34.79 25.02 35.54

5
Table 3: Bias and error for predicted daily maximum 8-hr ozone concentrations.

APPB APPGE APPB ≥60 APPGE ≥60

Site ID (%) (%) (%) (%)

10-001-0002 16.76 20.99 18.23 22.86

10-003-1003 15.53 19.15 15.74 18.81

10-003-1007 21.02 23.20 24.26 26.11

10-003-1010 10.36 17.77 10.32 17.45

10-003-1013 21.43 23.51 23.09 25.00

10-005-1002 1.87 14.93 -1.69 14.58

10-005-1003 20.10 22.79 23.46 25.97

24-015-0003 7.39 14.66 9.39 16.53

34-001-0005 22.54 24.70 25.52 27.44

34-007-0003 7.03 14.85 7.23 14.93

34-007-1001 3.72 11.77 2.78 11.58

34-011-0007 7.76 15.00 6.74 14.37

34-015-0002 4.95 13.05 4.36 12.72

34-021-0005 3.24 14.12 1.64 14.20

34-029-0006 3.17 14.53 2.24 14.81

42-017-0012 10.09 17.34 8.95 16.89

42-029-0050 0.42 12.49 -0.86 11.66

42-029-0100 -2.15 13.38 -1.68 14.61

42-045-0002 5.56 13.99 4.31 12.71

42-091-0013 10.43 15.85 11.19 16.16

42-101-0004 48.81 49.26 51.85 52.16

42-101-0014 11.17 17.21 12.74 18.67

42-101-0024 3.24 13.67 2.66 13.10

42-101-0136 26.54 28.54 27.16 29.14

All Sites 10.91 18.06 11.94 19.15

PA Sites 11.02 18.98 12.02 19.97

6
 0

 20

 40

 60

 80

 100

 120

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10

O3 (ppb)

Model

Monitor

Figure 1: Comparison of modeled vs. observed mean daily maximum 8-hr ozone concentrations: All Sites

 0

 20

 40

 60

 80

 100

 120

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10

O3 (ppb)

Model

Monitor

Figure 2: Comparison of modeled vs. observed mean daily maximum 8-hr ozone concentrations: PA Sites

7
-50

 0

 50

 100

 150

 200

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10

Percent

Bias

Error

Figure 3: Mean bias vs. error for the daily maximum 8-hr O3: All Sites

-50

 0

 50

 100

 150

 200

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10

Percent

Bias

Error

Figure 4: Mean bias vs. error for the daily maximum 8-hr O3: PA Sites

8
 0

 20

 40

 60

 80

 100

 120

 0 20 40 60 80 100 120

Model: O3 (ppb)

Monitor: O3 (ppb)

8Hr Avg.

Figure 5: Predicted vs. observed hourly 8-hr O3: All Sites

 0

 20

 40

 60

 80

 100

 120

 0 20 40 60 80 100 120

Model: O3 (ppb)

Monitor: O3 (ppb)

8Hr Avg.

Figure 6: Predicted vs. observed hourly 8-hr O3: PA Sites

9
-40

-30

-20

-10

 0

 10

 20

 30

 40

 50

 60

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10

O3 (ppb)

Range: (model-monitor)

Average (model-monitor)

Figure 7: Variation in (Model - Monitor) data between monitors

10