### 1 UNITED STATES ENVIRONMENTAL PROTECTION AGENCY Office of Air Quality Planning and Standards (OAQPS) Research Triangle Park, North Carolina 27711



### April 7, 2011

MEMORANDUM

SUBJECT: Assessment of PM 2.5 

### FEMs Compared to Collocated FRMs

FROM: Tim Hanley and Adam Reff, OAQPS /s/

TO: PM NAAQS Docket, EPA - HQ - OAR - 2007 - 0492

Objective

To assess the quality of recently approved PM 2.5 

### continuous Federal Equivalent Method (FEM) data as operated in r outine monitoring networks.

Background

On October 17, 2006, EPA published Revisions to th e Ambient Air Monitoring Regulations that included new performance criteria for acceptance of continuous PM 2.5 

### monitors as Class III Federal Equivalent Methods (FEM). PM 2.5 

### Class III Federal Equivalent Methods (FEMs) are analyzers capable of providing ambient air measurements represen tative of one-hour or shorterperiod integrated PM 2.5 

### concentrations as well as 24-hour measurements determined as, or equivalent to, the mean of 24 consecutive one-hou r measurements. The testing and performance

criteria for Class III FEMs can be found in 40 CFR Part 53.

Since finalizing these new testing and performa nce criteria several instruments have received designation as an FEM. EPA maintains a li st of approved FEMs on its web site at: http://www.epa.gov/ttn/amtic/criteria.html

Available Data

In the fall of 2010, we retrieved collocated PM 2.5 

### FRM and continuous FEM data that had been submitted to the Air Quality System (AQS) database. As of that time, the majority of available data were associated with the following two methods:

1. Met One BAM-1020 Monitor as an automate d equivalent method EQPM-0308-170 (61 sites collocated with an FRM); and

2. Thermo Scientific Ambient Particulate Monitor with Series 8500C FDMS as an automated equivalent method EQPM-0609-181 (17 sites collocated with an FRM)

### 2 

### We also had sufficient data to evaluate a third method, but only at two locations; therefore we are including the two sites assessed for this method in the attachments, but are not analyzing the data

across sites for this method.

3. Thermo Scientific Model 5030 SHARP Monitor as an automated equivalent method EQPM-0609-184 (2 sites collocated with an FRM)

Methods used to Assess the Collocated Data

Since the FRM measurement is a 24-hour integrated average, we can only evaluate the FEM for

the coincident 24 consecutive hours where both the FEM and FR M are operating (i.e., midnight to midnight local standard time for days when th e FRM is operated). In these assessments, we are using data from sites with collocated FR M and FEM as there are often high numbers of sample pairs available to assess the FEM perfor mance. Therefore, the data used in this

assessment are from the collocated FRMs operated by the same monitoring agency 1



### .

The statistics used in this assessment are based on the performance criteria for approval of Class III FEMs from 40 CFR Part 53, which are derived from the related PM 2.5 

### data quality objectives 2



### . There are related data quality goals defined in the Quality Assurance Requirements for Air Monitoring (Appendix A to 40 CFR Part 58). The Appendix A goals are for comparison to the measurement uncertainty for automated and manual PM 2.5 

### methods. The goal for this acceptable measurement uncertainty is define d as 10 percent coefficient of va riation (CV) for total precision and plus or minus 10 percent total bias. We c hose to use the Part 53 performance criteria as these statistics provide more detailed information that has the potential to relate to instrument set-up, operation, and maintenance procedures.

We used two approaches to assess the data. Fi rst, one-page assessments were produced for each FEM collocated with an FRM; thes e are presented in the attachments. Second, we took some of the key statistics (slope, intercept, and mean con centration) and compared the results across sites for the two FEMs most commonly available (i.e., Met-One BAM 1020 and Thermo 8500C FDMS). These are presented further below.

The following steps were taken to prepare a nd assess the data for use in the one-page assessments presented in the attachments:

1. Data were retrieved from EPA’s TTN web site ( http://www.epa.gov/ttn/airs/airsaqs /detaildata/downloadaqsdata.htm ) on October 6,

1 There is an audit program known as the Performance Ev aluation Program, where inde pendent auditors conduct FRM measurements using samplers that are temporarily collocated at stations with PM 2.5 FRMs or FEMs; however, these audits are conducted only a few times a year (5-8 samples per year) per monitoring organization.

2 A report on this subject was reviewed by CASAC’s technical subcommittee on Ambient Air Monitoring and Methods at a meeting in Durham, NC on September 21-22, 2005. The report is available on the web at: http://www.epa.gov/ttn/amtic/files/ambient/pm25/casac/att4casac.pdf The report and comments from CASAC subcommittee members, in addition to public comments fro m the proposed rule, were all incorporated in the final performance criteria published in Part 53.

### 3 

### 2010. The TTN provides publically accessible data files from EPA’s Air Quality System (AQS) database.

2. The dataset included all valid data, includi ng data flagged with an exceptional event

code.

3. PM 2.5 

### continuous FEM data were used if the following criteria were met:

a. There were at least 18 valid hours in a day from the FEM

b. There was a valid PM 2.5 

### FRM and FEM measurement for a given day.

4. Assessments were conducted using the perf ormance criteria from Ta ble C-4 and Figures

C-2 and C-4 of 40 CFR Part 53

5. Assessment periods for each site included:

a. All data

b. Each calendar year

c. Seasons as defined by calendar (e.g., Winter is December 21 to March 20). Note: this is not prescribed in regulation, but is a useful way to look at the instruments performance according to season.

6. Assessments by station included:

a. A scatter plot with a linear regression. A one-to-one line is also drawn on each figure.

b. A time series chart illustrating the di fference between the FRM and continuous

FEM data.

c. An illustration of the additive and multip licative bias as compared to the acceptable limits for these sta tistics (Part 53 table C-4).

d. An illustration of the correlation as compar ed to the acceptable correlation. Note:

the statistic used here is r and not r 2



### .

e. A summary of the number of data poin ts, means of the data, and ratio of continuous to FRM is presente d in the middle of the page.

7. While every site with collocated FRM and continuous FEM was analyzed 3 

### , we are only presenting data from those sites with at la st 23 sample pairs, which is the minimum number of sample pairs required for each test campaign, (described in Table C-4 to Part 53).

Results

Attachments 1, 2, and 3 provide each of the one-page assessments.

A summary of the slope, intercep t, and mean of the FEM and FR M statistics from the one-page assessments is presented below. In each case, the X-axis is organized by site from low to high with the Y-axis being the slope, intercept, or mean concentration value. The orange boxes represent the acceptance criteria for the applicable metric (i.e., slope or intercept). The red arrows represent the median site for the slope and intercept figures. Note: the slope and

3 All available sites in AQS were used except for one site in South Dakota (AQS ID: 46-033-0132). This site was impacted by a substantial PM event around September 4, 2009. The PM 2.5 measured during this event was in the hundreds of micrograms per cubic meter and would therefore not be representative of the performance of the instrument in the range of interest.

### 4 

### intercepts are presented independently here, wh ile the one-page assessments in the attachments appropriately consider them together.

0.7 0.8 0.9

1 1.1 1.2 1.3 1.4 1.5

1 2 3 4 5 6 7 8 9 101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960 

## Slope 

# Met One BAM 1020 ‐ FEM

Average slope for all sites = 1.081
Average slope for sites with an intercept within +/

‐ 2 μg/m 3 = 1.066



### Figure 1 – Met One BAM 1020 with accepta ble slopes illustrated in the box.

‐ 6 ‐ 4 ‐ 2 0 2 4 6 8 10

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 

## Intercept (μg/m 3 

## )

Average intercept for all sites = 1.12 μg/m 3

Average intercept for sites with a slope within +/

‐ 10% of 1 = 1.34 μg/m 3 

# Met One BAM 1020 ‐ FEM

Figure 2 – Met One BAM 1020 with acceptabl e intercepts illustrated in the box

### 5 

# Met One BAM 1020 ‐ FEM

0 5 10 15 20 25 30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41 43 45 47 49 51 53 55 57 59microgramspercubicmeter Mean Concentrations ‐ All Seasons

FRM

FEM



### Figure 3 – Met One BAM 1020 with mean concen tration by site for both the FRM and

FEM



# Met One BAM 1020 ‐ FEM

0 5 10 15 20 25 30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39microgramspercubicmeter Mean Concentrations for Winter

FRM

FEM 0 5 10 15 20 25 30

1 3 5 7 9 111315171921232527293133353739414345microgramspercubicmeter Mean Concentrations for Spring

FRM

FEM

0 5 10 15 20 25 30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35microgramspercubicmeter Mean Concentrations for Fall

FRM

FEM

0 5 10 15 20 25 30

1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37microgramspercubicmeter Mean Concentrations for Summer

FRM

FEM



### Figure 4 – Met One BAM 1020 with mean concentr ation by site by season for both the

FRM and FEM

### 6 

# 8500C FDMS ‐ FEM

0.4 0.5 0.6 0.7 0.8 0.9

1 1.1 1.2 1.3 1.4

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 

# Slope

Average slope for all sites = 0.937
Average slope for sites with an intercept within +/

‐ 2 μg/m 3 = 0.926



### Figure 5 – Thermo 8500C FDMS – FEM with acceptable slopes illustrated in the box



# 8500C FDMS ‐ FEM

‐ 3 ‐ 2 ‐ 1 0 1 2 3 4 5

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 Intercept (μg/m 3 )

Average intercept for all sites = 1.40
Average intercept for sites with a slope within +/

‐ 10% of 1 = 0.68



### Figure 6 – Thermo 8500C FDMS – FEM with acceptable intercep ts illustrated in the box

### 7 

# 8500C FDMS – FEM

Mean Concentration – All Seasons

0 5 10 15 20 25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 FRM

FEM



### Figure 7 – Thermo 8500C FDMS – FE M with mean concentration by site for both the FRM

and FEM



# 8500C FDMS ‐ FEM

0 5 10 15 20 25

1234microgramspercubicmeter Mean Concentrations for Fall

FRM

FEM

0 5 10 15 20 25

1 2 3 4 5 6 7 8 9 10 11 12 13 14microgramspercubicmeter Mean Concentrations for Winter

FRM

FEM 0 5 10 15 20 25

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16microgramspercubicmeter Mean Concentrations for Spring

FRM

FEM

0 5 10 15 20 25

1234microgramspercubicmeter Mean Concentrations for Summer

FRM

FEM



### Figure 8 – Thermo 8500C FDMS – FEM with mean concentration by site by season for

both the FRM and FEM

### 8 

### Discussion

There are a wide variety of results for each method. In some cases the collocated FEM is producing data that is meeting th e performance criteria described in 40 CFR Part 53; however, in other cases it is reading differently a nd often higher than the collocated FRM.

A summary of each of the major statistics fo r the two methods most widely deployed is

described here:

Slope for the Met One BAM 1020

About 2/3 of the sites had acceptable slopes; how ever, on average these acceptable slopes were reading about 6-7% high. This may be due to a number of issues under i nvestigation. Some of the possible explanations might include under performance of the “Smart Heater” sample conditioning, inaccurate calibration of the relative humidity sensor, and inconsistent control of the air conditioning during warm months of the year.

Intercept for the Met One BAM 1020

About 2/3 of the sites had an acceptable inte rcept; however on average these acceptable intercepts were a little over a microgram per c ubic meter higher than zero. Given that the Met One BAM 1020 has a zero testing procedure as pa rt of its installation, we might expect the average to fall closer to zero, with some scatter above and below zero.

Mean Concentration (all data) for the Met One BAM 1020

Most sites either predicted about the same mean as the colloca ted FRM or were 2-5 micrograms per cubic meter higher than the FRM data. On ly a few sites had FEM data with a mean

concentration lower than the FRM and in those fe w cases these values were usually within two micrograms per cubic meter of the FRM mean concentration.

Mean Concentration (by season) for the Met One BAM 1020

For most sites each of the seasons still shows a positive bias (the FEM reading higher than the FRM); however, the biases appear more pronounced during the spring and summer. This could be due to the differences in ambient temp erature between winter and spring/summer.

Slope for the 8500C FDMS

About half of the FEMs had acceptable slopes; ho wever, figure 5 clearly illustrates that the middle of the dataset has a consistent slope of about 0.9. The average slope for sites with an acceptable intercept was 0.926, thus the slop e is lower than may be expected.

Intercept for the 8500C FDMS

### 9 

### About 2/3 of the sites had an acceptable interc ept and on average these acceptable intercepts were about 0.68 micrograms per cubic meter higher th an zero. There is al so a noticeable tail of sites with large and unacceptable intercepts (>2 μg/m 3 

### ) possibly indicating a set-up or operational problem for these sites.

Mean Concentration (all data) for the 8500C FDMS

Most sites either predicted the same mean as the collocated FRM or were 2-4 micrograms per cubic meter higher than the FRM data. Only a few sites had FEM data with a mean

concentration lower than the FRM and in those fe w cases these values were usually within one microgram per cubic meter compared to the FRM.

Mean Concentration (by season) for the 8500C FDMS

The seasonal data was very similar for fall and winter. The spring data showed substantially higher (4-6 μg/m 3 

### ) concentrations for 4 sites and very si milar data for the others. The summer had mixed results, but there were only 4 sites with enough data points to analyze. The higher data in the spring at 4 sites may be a result of volatile PM 2.5 

### being readily measured by the 8500C FDMS FEM, which incorporates measurement of volatile PM as part of its design.

Conclusions

The assessments presented illustrate that a sub-se t of monitoring sites are generating data that are meeting the performance criteria used to approve PM 2.5 

### FEMs. However, there are clearly a

number of sites with continuous FEMs that are not meeting these pe rformance criteria. In these cases, agencies should reexamin e their set-up and operational procedures, including data reporting, to attempt to maximize th eir data quality. To assist in the effort to maximize data quality, EPA ambient air monitoring staff are working with the monitoring committee of the National Association of Clean Air Agencies, EP A Regional Offices, EPA’s Office of Research and Development, and instrument manufactures to use these results and other assessments in combination with set-up and maintenance information to document best practices for the set-up, operation, and reporting of data from approved FE Ms. Ideally, a best practices document will be

used in combination with this and future assessments to target troubleshooting of

instrumentation.

The lack of acceptable performance data at some sites with FEMs when compared to collocated FRMs, on a 24-hour basis, calls in to question the use of these continuous FEMs. For the PM 2.5 

### primary standard, monitoring agencies ha ve the option of continuing to use FRMs 4 

### , or where applicable, using a well performing continuous FEM. The annual monitoring network plan (described in § 58.10), due to the applicable EPA Regional Office by July 1 of each year, is the appropriate place for monitoring agencies to id entify the methods and sampling frequencies it will operate in its network. In cases where a PM 2.5 

### continuous FEM is not meeting the part 53

4 The data quality of each monitoring agencies FRM operation is available in assessments on the EPA web site at: http://www.epa.gov/ttn/amtic/anlqa.html. National statistics on the FRM data quality are described in the PM

ISA.

### 10 

### performance criteria, we recommend keeping the PM 2.5 

### FRM as the Primary monitor 5 

### while working towards improvements in FEM data qualit y. For those agencies with well performing

PM 2.5 

### continuous FEMs, we support the use of th ese instruments in the agencies network.

The lack of more uniform and acceptable performance data for PM 2.5 

### continuous FEMs compared to FRMs on a 24-hour basis clearly calls into question the use of FEM data to support a possible secondary standard w ith a sub-daily averaging time (e.g., one-hour or four-hour). While some agencies are achieving good data quality with PM 2.5 

### FEMs on a 24-hour basis, others are not, at this time. Since a possible secondary standard with a sub-daily averaging time would necessitate having monito ring agencies use continuous PM 2.5 

### FEMs, we are concerned that such a requirement could not be implemented for all agencies. Until such time as there are

continuous PM 2.5 

### FEMs operated across the nation in State and local networks that can be demonstrated to meet the acceptable performance criteria defined in 40 CFR Part 53 (which is based on 24-hour measurements), we should move cautiously about cons idering wider use of these instruments for sub-daily measurement needs.

Attachments:

1. Met One BAM 1020 Assessments – 61 Sites.

2. Thermo Scientific Ambient Particulate Moni tor with Series 8500C FDMS Assessments -

17 sites.

3. Thermo Scientific Model 5030 SHARP Assessments - 2 sites.

5 A Primary monitor is the monitor designated by the State, local or Tribal agency in their annual monitoring

network plan to be compared to the NAAQS. Primary monitors must be FRMs, FEMs, or ARMs.