## Model Performance Evaluation

Introduction This section documents an operational analysis of the CMAQ model runs used to support the attainment demonstration for Philadelphia. The purpos e of this effort is to validate the model’s ability to replicate observed concentrations of oz one. This analysis was prepared in accordance with EPA guidance 1 and the protocol document previously submitted to the EPA. 2


### Descriptive Statistics Pennsylvania has calculated the following measures compar ing the hourly ozone and 8-hourly maxima for each ozone monitor location of interest from May 15 th through September 30 th , 2002. 3

• Mean Normalized Bias (MNB): “This performance statistic averages the model/observation residual, paired in time, normalized by observation, over all monitor times and locations.” 4

MNB = n

X
i =1 ( Model i − Obs i )

Obs i !! × 100%

n (1)

• Mean Normalized Gross Error (MNGE): “This performance stat istic averages the absolute value of the model/observation residual, paired i n time, normalized by observation, over all monitor times and locations.” 5

MNGE = n

X
i =1 | ( Model i − Obs i ) |

Obs i !! × 100%

n (2)

• Normalized Mean Bias (NMB): Because of the tendency of the MN B statistic, defined above, to inflate the importance of biases at low observed values, the US EPA recommends using NMB for calculations that have no threshol d. 6

1 US EPA, Guidance on the Use of Models and Other Analyses for Demonstr ating Attainment of Air Quality goals for Ozone, PM 2.5 , and Regional Haze, Draft 3.2, 176–201 (Sept., 2006). 2 PA DEP, Final Draft Protocol for Regulatory Photochemical Air Quality Modeling for the Philadelphia Eight-Hour Ozone Nonattainment Area , (June, 2006). 3 While equations 1 and 2 differ from those printed in the EPA’s draft guidance (EPA 2006, at 184), the DEP believes the versions used here accurately reflect the co rrect methodology. 4 US EPA 2006, at 184. 5 Id. 6 US EPA 2006, at 184 n.47.

1NMB = P n i =1 ( Model i − Obs i ) P n i =1 ( Obs i ) × 100% (3)

• Normalized Mean Error (NME): Because of the tendency of the M NGE statistic, defined above, to inflate the importance of errors at low observed values, the US EPA recommends using NME for calculations that have no threshol d. 7

NME = P n

i =1 | Model i − Obs i | P n i =1 ( Obs i ) × 100% (4)

• Average Peak Prediction Bias and Error (APPB & APPE): This set of statistics uses the same equations as the calculation for NMB and NMGE, liste d above, but only consider the daily maximum 8-hour ozone prediction at each monitor. Because there is no implicit threshold in this calculation, Pennsylvania applied calculations as described in equations 3 and 4.
The error statistics provide an overall assessment of model performance and can be interpreted as precision. The bias parameter measures a mod el’s ability to reproduce observed spatial and temporal patterns and can be interpret ed as accuracy. These statistics were calculated using to two different datasets. The first is f or all observed 8-hour ozone concentrations during the 2002 ozone season. The second set uses the same methodology, but it is limited to only those hours where the observed conce ntrations exceed 60ppb. In addition to the calculations for each monitor location, t he statistics are also calculated over all of the sites and time period in the region. In additio n to the region-wide data, this document also provides a separate analysis where only the mo nitors in Pennsylvania are averaged.



### Data
Table 1 lists the monitors used in this analysis along with the model i-cell and j-cell associated with their locations: The ozone monitor data were obtained from the EPA Air Quality System (AQS) national database using the standard retrieval report “Raw Data NAAQ S Averages” (AMP350NW). Calculations to create the running 8-hour ozone averages are based on EPA guidelines. The data submitted to AQS by state and local agencies is expected to have passed internal quality control checks and been certified by the senior air quality control manager for the submitting agency. Model data was provided by the NY DEC. This consisted of the “OTC2002 BaseB1 12CCTM v451 cb4 ae3.ACONC.2002 BaseB1.*” series of data sets calculated using the SMOKE/CMAQ modeling system using 2002 emissions data and MM5 meteorology. 8

7 Id. 8 See generally: NY DEC, CMAQ Model Performance Assessment, 8-Hr OTC Modeling, TSD- 1e , (Feb., 2006).

2Table 1: Monitor locations Site ID Location State I-cell J-cell 10-001-0002 Killens Pond DE 131 83 10-003-1003 River Road Park DE 130 90 10-003-1007 Lums Pond DE 128 88 10-003-1010 Brandywine DE 129 91 10-003-1013 Bellefonte DE 129 90 10-005-1002 Seaford DE 131 80 10-005-1003 Lewes DE 134 82 24-015-0003 Fair Hill MD 127 89 34-001-0005 Nacote Creek NJ 137 90 34-007-0003 Camden NJ 132 92 34-007-1001 Ancora St. Hos NJ 134 91 34-011-0007 Millville NJ 133 88 34-015-0002 Clarksboro NJ 132 91 34-021-0005 Rider Univ. NJ 133 96 34-029-0006 Colliers Mills NJ 136 95 42-017-0012 Bristol PA 133 94 42-029-0050 West Chester PA 128 92 42-029-0100 New Garden PA 127 90 42-045-0002 Chester PA 130 91 42-091-0013 Norristown PA 130 94 42-101-0004 Frankford PA 132 93 42-101-0014 Northwest PA 131 93 42-101-0024 Northeast PA 132 94 42-101-0136 Southwest PA 131 92



### Operational Evaluation Table 2 (page 5) shows the Normalized Mean Bias and Error, whi ch were calculated using equations 3 and 4. The results are given for each monitor location, averaged for all monitor locations, and averaged across the monitors within Pennsylvania. In addition, this table shows the Mean Normalized Bias and Gross Error, calculated u sing equations 3 and 4, when the recommended 9 60ppb threshold is applied. Table 3 (see page 6) presents the Average Peak Prediction Bias and Gross Error in a similar fas hion. Figure 1 (page 7) compares monitor data with actual modeled r esults. The “monitor” data are averages of the maximum 8-hour ozone concentration reco rded at each of the twenty-four monitor locations listed in Table 1. The “model ” data uses a similar approach, averaging the maximum 8-hour ozone concentration calculat ed for the grid cells occupied by those monitors. Figure 2 presents the same data, but limit s the set to the nine monitors located in Pennsylvania. Figures 3 and 4 (page 8) compare the mean normalized bias with the mean normalized

9 EPA 2006, at 185.

3gross error of the model’s calculations. The data were calculated using equations 1 and 2

on page 1, above. Except that, rather than depicting the mean seasonal data, the selected monitor data was averaged for each day of the season. Figures 5 and 6 (page 9) are scatter plots that compare the monitor data to the model’s predictions. In those figures, the solid horizontal bar is the 65ppb threshold—the dividing line between “Green” and “Yellow” on the Air Quality Index (AQI) scale. The center diagonal dashed lines represents a 1:1 relationship, while the outer lines represent a range of ( ± 34.1%) of the 1:1 line’s value—that is, one standard deviati on assuming a normal distribution. For values above 65ppb recorded at all monito r locations, the coefficient of determination (r 2 ) for the data, is 35%. Figure 7 (page 10) depicts the range of model to monitor variation. On that graph, the solid line shows the average variation of all of the monitors . In addition, vertical error bars shows the range of variation found at individual monitors.


### Summary of Model Performance Figures 1 and 2 show that CMAQ’s prediction of daily maximum o zone concentrations corresponds very well with the observed data. Table 3 (page 6 ) supports this, showing that the predictions fall within the EPA’s recommended range for bias ( ± 15%) and error (less than 30%.) While Figures 3 and 4 show spikes in the calculated bias and er ror, further analysis indicates that these will not impact the attainment demonst ration. On the 27 days where the bias exceeds the of 30% limit, the average daily maximum 8-hour ozone concentration averages 36ppb and never exceeds 56ppb. Pennsylvania also calculated the normalized bias and error for those episode days which could potentially have impacted the design value at any of the Commonwealth’s nine monitors—that is, the four highest values in 2002. For these episodes, the model’s performance was even better with a resulting bias of 7.1% and and error of 12.2%. For these reasons, Pennsylvania feel the modeling system do es an adequate job of estimating the 8-hour surface ozone concentrations throughout the Philadelphia area.

4Table 2: Bias and error for predicted hourly 8-hr ozone conce ntrations.
NMB NME MNB ≥ 60 MNGE ≥ 60 Site ID (%) (%) (%) (%) 10-001-0002 34.70 39.21 49.90 54.36 10-003-1003 28.52 35.69 29.36 32.70 10-003-1007 54.85 56.91 67.76 70.14 10-003-1010 18.86 30.62 14.49 26.38 10-003-1013 39.70 43.26 45.81 48.10 10-005-1002 24.30 32.80 30.66 43.17 10-005-1003 31.02 34.35 37.76 40.53 24-015-0003 17.13 25.89 17.49 27.16 34-001-0005 45.13 47.24 70.42 72.34 34-007-0003 13.34 29.35 11.94 24.11 34-007-1001 12.68 23.38 7.13 18.87 34-011-0007 27.16 33.63 32.61 39.93 34-015-0002 19.20 30.49 21.16 31.96 34-021-0005 17.83 29.47 4.17 18.62 34-029-0006 12.50 25.31 4.01 19.43 42-017-0012 31.27 39.86 15.01 26.69 42-029-0050 17.17 28.80 20.38 34.90 42-029-0100 5.51 21.30 6.14 24.58 42-045-0002 19.11 30.06 21.14 33.05 42-091-0013 26.63 33.73 26.57 32.89 42-101-0004 66.76 68.69 73.18 73.80 42-101-0014 21.36 31.20 22.43 30.61 42-101-0024 15.04 27.34 9.76 24.23 42-101-0136 42.56 47.72 45.05 48.69 All Sites 25.53 34.28 28.88 37.68 PA Sites 25.01 34.79 25.02 35.54

5Table 3: Bias and error for predicted daily maximum 8-hr ozone concentrations.
APPB APPGE APPB ≥ 60 APPGE ≥ 60 Site ID (%) (%) (%) (%) 10-001-0002 16.76 20.99 18.23 22.86 10-003-1003 15.53 19.15 15.74 18.81 10-003-1007 21.02 23.20 24.26 26.11 10-003-1010 10.36 17.77 10.32 17.45 10-003-1013 21.43 23.51 23.09 25.00 10-005-1002 1.87 14.93 -1.69 14.58 10-005-1003 20.10 22.79 23.46 25.97 24-015-0003 7.39 14.66 9.39 16.53 34-001-0005 22.54 24.70 25.52 27.44 34-007-0003 7.03 14.85 7.23 14.93 34-007-1001 3.72 11.77 2.78 11.58 34-011-0007 7.76 15.00 6.74 14.37 34-015-0002 4.95 13.05 4.36 12.72 34-021-0005 3.24 14.12 1.64 14.20 34-029-0006 3.17 14.53 2.24 14.81 42-017-0012 10.09 17.34 8.95 16.89 42-029-0050 0.42 12.49 -0.86 11.66 42-029-0100 -2.15 13.38 -1.68 14.61 42-045-0002 5.56 13.99 4.31 12.71 42-091-0013 10.43 15.85 11.19 16.16 42-101-0004 48.81 49.26 51.85 52.16 42-101-0014 11.17 17.21 12.74 18.67 42-101-0024 3.24 13.67 2.66 13.10 42-101-0136 26.54 28.54 27.16 29.14 All Sites 10.91 18.06 11.94 19.15 PA Sites 11.02 18.98 12.02 19.97

60 20 40 60 80 100 120

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10O3 (ppb) Model Monitor

Figure 1: Comparison of modeled vs. observed mean daily maxi mum 8-hr ozone concentrations: All Sites

0 20 40 60 80 100 120

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10O3 (ppb) Model Monitor

Figure 2: Comparison of modeled vs. observed mean daily maxi mum 8-hr ozone concentrations: PA Sites

7-50 0 50 100 150 200

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10Percent Bias Error

Figure 3: Mean bias vs. error for the daily maximum 8-hr O 3 : All Sites

-50 0 50 100 150 200

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10Percent Bias Error

Figure 4: Mean bias vs. error for the daily maximum 8-hr O 3 : PA Sites

80 20 40 60 80 100 120

0 20 40 60 80 100 120Model: O3 (ppb)

Monitor: O3 (ppb) 8Hr Avg.

Figure 5: Predicted vs. observed hourly 8-hr O 3 : All Sites

0 20 40 60 80 100 120

0 20 40 60 80 100 120Model: O3 (ppb)

Monitor: O3 (ppb) 8Hr Avg.

Figure 6: Predicted vs. observed hourly 8-hr O 3 : PA Sites

9-40 -30 -20 -10 0 10 20 30 40 50 60

05/09 05/23 06/06 06/20 07/04 07/18 08/01 08/15 08/29 09/12 09/26 10/10O3 (ppb) Range: (model-monitor) Average (model-monitor)

Figure 7: Variation in (Model - Monitor) data between monito rs

10