Appendix G Attainment Modeling Demonstration and Documentation.

Attachment 10 was submitted in Electronic Format Only. Some portions have been copied but 

large data files in Attachment 10 are available from the EPA Region III office by request.

Listing of Large data Files in Attachment 10 available from the EPA Region III office by 

request:

● “obs+pred.8hr_O3_BaseB1.tar.gz”

 Read.me.txt

 Castnet.txt

 State09.txt

 State10.txt

 State11.txt

 State23.txt

 State24.txt

 State25.txt

 State33.txt

 State34.txt

 State36.txt

 State42.txt

 State44.txt

 State50.txt

 State51.txt

 Special,txt

● diurnals_BaseB1.xls: 

● stats_max8hr_o3_BaseB1.xls

● stats_max8hr_o3_naa_BaseB1.xls

● stats_frm+teom_BaseB1.xls

● stats_stn_BaseB1.xls

● stats_improve_BaseB1.xls

● stats_gases_BaseB1.xls: 

● stats_voc_BaseB1.xls: 

● wetdep_BaseB1.xls
OTC_statistics.pdf
1

Statistics for the OTC model evaluation

August 10, 2006

The statistical formulations presented in each spreadsheet are as follows: Pi and Oi are the 

individual (daily maximum or average, depending on the parameter) predicted and 

observed concentrations, respectively;

P

and 

O

are the average concentrations, 

respectively; and N is the sample size.

(1) Coefficient of determination, R2:

 

R

P P O O

P P O O

i i

i i

2

2

2 2



 

 



 

( )( )

( ) ( )

(2) Normalized mean error, in %:

NME

P O

O

i i

i











| |

100%

(3) Root mean square error, in ppb or μg m

-3

:

RMSE

N

Pi Oi 











 

1 2

1 2

( )

/

(4) Fractional gross error, in %:

FE

N

P O

P O

i i

i i





  

2

100%

(5) Mean absolute gross error, in ppb or μg m-3:

MAGE

N

Pi Oi  

1

(6) Mean normalized gross error, in %:

MNGE

N

P O

O

i i

i





 

1

100%

(7) Mean bias, in ppb or μg m-3:

MB

N

Pi Oi  

1

( )

(8) Mean normalized bias, in %:

MNB

N

P O

O

i i

i



  

1

100%

( )
2

(9) Mean fractionalized bias, in %:

MFB

N

P O

P O

i i

i i

















  

2

100%

(10) Normalized mean bias, in %:

NMB

P O

O

i i

i











( )

100%
README_BaseB1.pdf
README_BaseB1.txt

-----------------

Evaluation of the Base B1 CMAQ simulation, May 15 - September 30, 2002

New York State DEC, Division of Air Resources

September 27, 2006

The New York State DEC completed the latest ("Base B1") base case 

simulation

covering the period May 15 - September 30, 2002. The files within this

directory reflect the most up-to-date results over the OTR plus all of

Virginia. Each spreadsheet contains a metadata page that lists the 

number

of stations, assumptions made to compute statistics, etc.

1. obs+pred.8hr_O3_BaseB1.tar.gz: gzip'd archive containing time series 

of 8-hour

average observed and predicted O3; files are organized by different 

states;

format of each file is site ID, day, time (end of hour EST), observed O3,

predicted O3; this archive also includes a small "readme.txt" file; 

CASTNet

and other "special" sites are also included

2. OTC_statistics.doc: lists the different statistical metrics used in 

the

model evaluation

3. diurnals_BaseB1.xls: observed and predicted composite diurnal 

variations

of selected species - O3 at SLAMS/NAMS sites, O3 at CASTNet & other 

sites,

continuous PM2.5, CO, ethene, isoprene, NO, NO2, and SO2

4. stats_max8hr_o3_BaseB1.xls: statistical evaluation of daily maximum 

8-hour O3 at

SLAMS/NAMS sites and CASTNet/other sites; statistics are computed using 

two

different thresholds for observed daily max O3 - 40 and 60 ppb; 

statistics

are computed by date (all sites on a given day) and by site (one site 

over

all days)

5. stats_max8hr_o3_naa_BaseB1.xls: statistical evaluation of daily 

maximum 8-hour O3

at SLAMS/NAMS sites that fall within non-attainment counties (EPA 2003

designations); statistics are computed by non-attainment area

6. stats_frm+teom_BaseB1.xls: statistical evaluation of daily average 

PM2.5 mass,

including FRM sites and continuous PM2.5 sites; statistics are computed

by date and by site
7. stats_stn_BaseB1.xls: statistical evaluation of daily average PM2.5, 

SO4, NO3,

NH4, EC, OM (1.8*blank-corrected OC), and crustal/soil mass at EPA STN

sites; statistics are computed by date and by site

8. stats_improve_BaseB1.xls: statistical evaluation of daily average 

PM2.5, SO4, NO3,

EC, OM (1.8*blank-corrected OC), and crustal/soil mass at IMPROVE sites;

statistics are computed by date and by site; also includes Dolly Sods

9. stats_gases_BaseB1.xls: statistical evalution of daily average CO, 

NO, NO2, and

SO2 at SLAMS/NAMS and other sites; statistics are computed by date and by 

site

10. stats_voc_BaseB1.xls: statistical evalution of daily average ethene, 

isoprene, and

formaldehyde at SLAMS/NAMS and other sites; statistics are computed by 

date 

and by site

11. wetdep_BaseB1.xls: comparison of SO4, NO3, and NH4 accumulated wet 

deposition

at NADP/NTN, AIRMoN, and NYSDEC acid deposition monitors

12. temporal_plots_BaseB1.ppt: plots of composite time series (daily max 

8-hour O3,

FRM PM2.5 mass, STN speciation) and RMSE & mean bias - for illustrative 

purposes

only!

13. spatial_plots_BaseB1.ppt: spatial plots of observed daily max 8-hour 

O3, FRm PM2.5 mass,

and STN speciation, as well as RMSE & mean bias - for illustrative 

purposes only!
spatial_plots_BaseB1.pdf
1

• A more complete statistical evaluation, including diurnal 

variations, of the summer portion (May 15 – September 30, 

2002) is available on the NYSDEC FTP site 

ftp://ftp.dec.state.ny.us/dar/air_research/kevin

• Here we present a subset of the analysis of selected 

species on a spatial basis

• Daily maximum 8-hour O3, FRM PM2.5, and major speciation 

(EPA STN) data

• Only data from AQS are considered here; analysis of 

IMPROVE, CASTNet & special data are on the FTP site

Preliminary evaluation of the 2002 

Base B1 CMAQ simulation: Spatial 

Analysis
2

Daily maximum 8-hour average O3 with a minimum 

observed threshold of 40 ppb at 208 SLAMS/NAMS 

sites across the OTR and Virginia: (A) observed 

averages, (B) root mean square error (RMSE), and (C) 

mean bias (MB=P-O). At least 10 days were required for 

displaying the statistics at a given site. Data from the 

July 6-9 period are excluded since they may be affected 

by the Canadian forest fires.
3

Daily maximum 8-hour average O3 with a minimum 

observed threshold of 60 ppb at 208 SLAMS/NAMS 

sites across the OTR and Virginia: (A) observed 

averages, (B) root mean square error (RMSE), and (C) 

mean bias (MB=P-O). At least 10 days were required for 

displaying the statistics at a given site. Data from the 

July 6-9 period are excluded since they may be affected 

by the Canadian forest fires.
4

Daily average FRM PM2.5 mass at 257 SLAMS/NAMS 

sites across the OTR and Virginia: (A) observed 

averages, (B) root mean square error (RMSE), and (C) 

mean bias (MB=P-O). At least 10 days were required for 

displaying the statistics at a given site, and only those 

days when the observed and predicted mass agreed to 

within a factor of 25 are included. Data from the July 6-9 

period are excluded since they may be affected by the 

Canadian forest fires.
5

Daily average SO4 mass at 49 EPA/STN sites across 

the OTR and Virginia: (A) observed averages, (B) root 

mean square error (RMSE), and (C) mean bias (MB=PO). At least 10 days were required for displaying the 

statistics at a given site, and only those days when the 

observed and predicted mass agreed to within a factor of 

25 are included. Data from the July 6-9 period are 

excluded since they may be affected by the Canadian 

forest fires.
6

Daily average NO3 mass at 49 EPA/STN sites across 

the OTR and Virginia: (A) observed averages, (B) root 

mean square error (RMSE), and (C) mean bias (MB=PO). At least 10 days were required for displaying the 

statistics at a given site, and only those days when the 

observed and predicted mass agreed to within a factor of 

25 are included. Data from the July 6-9 period are 

excluded since they may be affected by the Canadian 

forest fires.
7

Daily average NH4 mass at 49 EPA/STN sites across 

the OTR and Virginia: (A) observed averages, (B) root 

mean square error (RMSE), and (C) mean bias (MB=PO). At least 10 days were required for displaying the 

statistics at a given site, and only those days when the 

observed and predicted mass agreed to within a factor of 

25 are included. Data from the July 6-9 period are 

excluded since they may be affected by the Canadian 

forest fires.
8

Daily average EC mass at 49 EPA/STN sites across the 

OTR and Virginia: (A) observed averages, (B) root mean 

square error (RMSE), and (C) mean bias (MB=P-O). At 

least 10 days were required for displaying the statistics at 

a given site, and only those days when the observed and 

predicted mass agreed to within a factor of 25 are 

included. Data from the July 6-9 period are excluded 

since they may be affected by the Canadian forest fires.
9

Daily average OM (=1.8×blank-corrected OC) mass at 

49 EPA/STN sites across the OTR and Virginia: (A) 

observed averages, (B) root mean square error (RMSE), 

and (C) mean bias (MB=P-O). At least 10 days were 

required for displaying the statistics at a given site, and 

only those days when the observed and predicted mass 

agreed to within a factor of 25 are included. Data from 

the July 6-9 period are excluded since they may be 

affected by the Canadian forest fires.
10

Daily average soil/crustal mass at 49 EPA/STN sites 

across the OTR and Virginia: (A) observed averages, (B) 

root mean square error (RMSE), and (C) mean bias 

(MB=P-O). At least 10 days were required for displaying 

the statistics at a given site, and only those days when 

the observed and predicted mass agreed to within a 

factor of 25 are included. Data from the July 6-9 period 

are excluded since they may be affected by the 

Canadian forest fires.
temporal_plots_BaseB1.pd

f
1

Preliminary evaluation of the 2002 

Base B1 CMAQ simulation: 

Temporal Analysis

• A more complete statistical evaluation, 

including diurnal variations, of the summer 

portion (May 15 – September 30, 2002) is 

available on the NYSDEC FTP site 

ftp://ftp.dec.state.ny.us/dar/air_research/kevin

• Here we present a subset of the analysis of 

selected species on a temporal basis

• Daily maximum 8-hour O3, FRM PM2.5, and 

major speciation (EPA STN) data

• Only data from AQS are considered here; 

analysis of IMPROVE, CASTNet & special 

data are on the FTP site
2

Daily maximum 8-hour average O3 with a minimum observed threshold of 40 ppb 

at 208 SLAMS/NAMS sites across the OTR and Virginia. Top panel shows observed and 

predicted composite averages, bottom panel shows the root mean square error and 

mean bias (P-O). At least 10 monitors per day were required for displaying the statistics 

on a given day. Data from the July 6-9 period are excluded since they may be affected 

by the Canadian forest fires.

Composite Daily Maximum 8-hour Ozone - 40 ppb threshold

0

10

20

30

40

50

60

70

80

90

100

2002051520020521200205272002060220020608200206142002062020020626200207022002070820020714200207202002072620020801200208072002081320020819200208252002083120020906200209122002091820020924

Ozone (ppb)

obs avg, ppb CMAQ avg, ppb

Root mean square error and Mean bias - 40 ppb threshold

-25

-20

-15

-10

- 5

0

5

10

15

20

25

30

2002051520020521200205272002060220020608200206142002062020020626200207022002070820020714200207202002072620020801200208072002081320020819200208252002083120020906200209122002091820020924

RMSE or MB (ppb)

RMSE, ppb MB, ppb
3

Daily maximum 8-hour average O3 with a minimum observed threshold of 60 ppb

at 208 SLAMS/NAMS sites across the OTR and Virginia. Top panel shows observed and 

predicted composite averages, bottom panel shows the root mean square error and 

mean bias (P-O). At least 10 monitors per day were required for displaying the statistics 

on a given day. Data from the July 6-9 period are excluded since they may be affected 

by the Canadian forest fires.

Composite Daily Maximum 8-hour Ozone - 60 ppb threshold 

0

10

20

30

40

50

60

70

80

90

100

2002051520020521200205272002060220020608200206142002062020020626200207022002070820020714200207202002072620020801200208072002081320020819200208252002083120020906200209122002091820020924

Ozone (ppb)

obs avg, ppb CMAQ avg, ppb

Root mean square error and Mean bias - 60 ppb threshold

-30

-20

-10

0

10

20

30

2002051520020521200205272002060220020608200206142002062020020626200207022002070820020714200207202002072620020801200208072002081320020819200208252002083120020906200209122002091820020924

RMSE or MB (ppb)

RMSE, ppb MB, ppb
4

Daily average FRM PM2.5 mass at 257 SLAMS/NAMS sites across the OTR and 

Virginia. Top panel shows observed and predicted composite averages, bottom panel 

shows the root mean square error and mean bias (P-O). At least 10 monitors per day 

were required for displaying the statistics on a given day, and only those days when the 

observed and predicted mass agreed to within a factor of 25 are included. Data from the 

July 6-9 period are excluded since they may be affected by the Canadian forest fires.

Composite Daily Average PM2.5

0

5

10

15

20

25

30

35

40

45

200205152002052120020527200206022002060820020614200206202002062620020702200207082002071420020720200207262002080120020807200208132002081920020825200208312002090620020912200209182002092420020930

PM2.5 (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

-20

-15

-10

- 5

0

5

10

15

20

25

200205152002052120020527200206022002060820020614200206202002062620020702200207082002071420020720200207262002080120020807200208132002081920020825200208312002090620020912200209182002092420020930

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
5

Daily average SO4 mass at 49 EPA STN sites across the OTR and Virginia. Top panel 

shows observed and predicted composite averages, bottom panel shows the root mean 

square error and mean bias (P-O). At least 10 monitors per day were required for 

displaying the statistics on a given day, and only those days when the observed and 

predicted mass agreed to within a factor of 25 are included. Data from the July 6-9 

period are excluded since they may be affected by the Canadian forest fires.

Composite Daily Average SO4

0

5

10

15

20

25

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

SO4 (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

- 6

- 4

- 2

0

2

4

6

8

10

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
6

Daily average NO3 mass at 49 EPA STN sites across the OTR and Virginia. Top panel 

shows observed and predicted composite averages, bottom panel shows the root mean 

square error and mean bias (P-O). At least 10 monitors per day were required for 

displaying the statistics on a given day, and only those days when the observed and 

predicted mass agreed to within a factor of 25 are included. Data from the July 6-9 

period are excluded since they may be affected by the Canadian forest fires.

Composite Daily Average NO3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

3.5

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

NO3 (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

3.0

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
7

Daily average NH4 mass at 49 EPA STN sites across the OTR and Virginia. Top panel 

shows observed and predicted composite averages, bottom panel shows the root mean 

square error and mean bias (P-O). At least 10 monitors per day were required for 

displaying the statistics on a given day, and only those days when the observed and 

predicted mass agreed to within a factor of 25 are included. Data from the July 6-9 

period are excluded since they may be affected by the Canadian forest fires.

Composite Daily Average NH4

0

1

2

3

4

5

6

7

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

NH4 (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

2.0

2.5

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
8

Daily average EC mass at 49 EPA STN sites across the OTR and Virginia. Top panel 

shows observed and predicted composite averages, bottom panel shows the root mean 

square error and mean bias (P-O). At least 10 monitors per day were required for 

displaying the statistics on a given day, and only those days when the observed and 

predicted mass agreed to within a factor of 25 are included. Data from the July 6-9 

period are excluded since they may be affected by the Canadian forest fires.

Composite Daily Average EC

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

1.6

1.8

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

EC (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

-0.2

0.0

0.2

0.4

0.6

0.8

1.0

1.2

1.4

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
9

Daily average OM (=1.8×blank-corrected OC) mass at 49 EPA STN sites across the 

OTR and Virginia. Top panel shows observed and predicted composite averages, 

bottom panel shows the root mean square error and mean bias (P-O). At least 10 

monitors per day were required for displaying the statistics on a given day, and only 

those days when the observed and predicted mass agreed to within a factor of 25 are 

included. Data from the July 6-9 period are excluded since they may be affected by the 

Canadian forest fires.

Composite Daily Average OM

0

2

4

6

8

10

12

14

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

OM (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

- 9

- 6

- 3

0

3

6

9

12

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
10

Daily average soil/crustal mass at 49 EPA STN sites across the OTR and Virginia. 

Top panel shows observed and predicted composite averages, bottom panel shows the 

root mean square error and mean bias (P-O). At least 10 monitors per day were 

required for displaying the statistics on a given day, and only those days when the 

observed and predicted mass agreed to within a factor of 25 are included. Data from the 

July 6-9 period are excluded since they may be affected by the Canadian forest fires. 

The data from July 4 are also excluded (substantial effects from fireworks).

Composite Daily Average Crustal

0.0

0.5

1.0

1.5

2.0

2.5

3.0

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

crustal (ug/m3)

obs avg, ug/m3 CMAQ avg, ug/m3

Root mean square error and Mean bias

0.0

0.5

1.0

1.5

2.0

2.5

3.0

2002051720020523200205292002060420020610200206162002062220020628200207042002071020020716200207222002072820020803200208092002081520020821200208272002090220020908200209142002092020020926

RMSE or MB (ug/m3)

RMSE, ug/m3 MB, ug/m3
Ozone_8_May_2002s_BaseB.pdf































Ozone_8_June_2002s_BaseB.pdf






























Ozone_8_July_2002s_BaseB.pdf































Ozone_8_August_2002s_BaseB.pdf