Milwaukee PM2.5 Clean Data Determination

Technical Support Document

Michael Rizzo

USEPA Region 5

Air and Radiation Division

December 2011
Technical Support Document

Table of Contents

Background ...................................................................................................................................................3

Review of Historical Monitoring Data...........................................................................................................5

2010 Data Imputation for 55-079-0043 and 55-079-0099 ...........................................................................7

Conclusions.................................................................................................................................................13

References ..................................................................................................................................................13
Background

As part of the 2010 annual network review, the Wisconsin Department of Natural Resources 

requested to discontinue two sites (AQS ID: 550790043 and AQS ID: 550790099) within the 

Milwaukee, WI metropolitan area to save resources. The State cited two other violating 

monitors (AQS ID: 550790026 and AQS ID: 550790010) within the area which could be used for 

attainment designation purposes (Figure 1). At the time of the request four monitors within 

Milwaukee were violating the 24-hour PM2.5 National Ambient Air Quality Standards (NAAQS) 

including the two sites the State requested to shut down. During the review, Region 5 

mistakenly agreed that the State could shut down the two sites even though 40 CFR 58.14 does 

not allow any violating site to be discontinued. In 2010, the remaining two violating sites in 

Milwaukee had data showing that they attained the standard for the 2008-2010 time period. 

Wisconsin DNR requests a clean data finding based on those two sites so that the area can be 

redesignated from nonattainment for the 24-hour PM2.5 NAAQS to attainment. However, 

because the two sites which were allowed to be terminated at the end of 2009 were also 

violating, the clean data determination cannot be granted without first determining whether or 

not those two sites could have met the NAAQS if they had been allowed to continue operating.
Figure 1. Violating PM2.5 Sites within the Milwaukee WI CSA. Sites with blue markers are 

the sites which were discontinued.

Past issues with incomplete PM2.5 data have been settled using statistical techniques to impute 

missing data, calculate the three year design value, and determine if the monitor in question 

meets the NAAQS within some margin of error. Usually, these tests are only allowed after all 

due diligence has been exhausted to assure that all measures have been taken to confirm that 

the data were collected and monitoring networks were maintained according to the 

requirements prescribed in 40 CFR. That did not happen in this case. The Region believes, 

however, that the use of historical data along with the statistical techniques to impute the 

missing data to determine whether or not the two sites would attain the 24- hour PM2.5 

NAAQS can provide enough evidence to make a definitive decision on the redesignation of the 

Milwaukee area. Region 5 also will require Wisconsin DNR to re-establish the Milwaukee Fire 

Department Headquarters site (550790099) beginning January 1, 2012 so that further 

measurements can be collected for an amount of time to be negotiated between the State and 

EPA.
Review of Historical Monitoring Data

The Milwaukee area had a total of four monitors which were violating the 24-hour PM2.5 

NAAQS at the time the two fire station sites were discontinued. Figure 2 shows the 98th

percentiles for all four sites from their inception in 1999 to the present. For many years, the 

two discontinued sites had 98th percentile values (blue lines in Figure 2) which were greater 

than those seen at either 550790010 or 550790026. One important observation that needs to 

be noted is that starting around 2007 and continuing through 2009, the 98th percentiles from 

the two discontinued sites tracked the other two violating sites well in that all four sites had 

98th percentiles that rose and fell with each other. If this trend continued into 2010, it is a good 

possibility that the two discontinued sites would have had 98th percentile values that would 

have been less than those seen in 2009. The three year design values at the four sites (Figure 3)

show that the two discontinued sites had values that were greater than the level of the NAAQS 

almost every year they were in operation with none of the four sites consistently having a 

higher set of design values over the remaining three. Because three consecutive years worth of 

data area required to determine attainment, 2010 98th percentile critical values show what 

each sites concentrations could have been to still attain the 24-hour NAAQS. Those 

concentrations were 37.1 ug/m3 and 36.9 ug/m3, respectively, for site 550790043 and site 

550790099 (gray lines in Figure 2). Table I shows the 98th percentiles for the four sites from 

2007 through 2009. For all three years, the 98th percentile concentrations were within 2-3

ug/m3 of each other. If 2010 were consistent with 2007 through 2009, the 98th percentiles for 

the two missing sites would most likely be below the maximum critical values necessary for the 

2008-2010 design values to be in attainment for the two sites. 

Table I: Annual 98th Percentile Concentrations (ug/m3) for Violating Sites within Milwaukee WI

Year Virginia 

Street Fire 

Station 

(550790043)

Milwaukee Fire 

Department HQ 

(550790099)

Wisconsin 

DNR SE HQ 

(550790026)

16th Street 

Health Clinic 

(550790010)

2007 38 40.7 39.8 40.6

2008 27.4 29 27.5 27.3

2009 41.7 40.3 39 39.1

2010 31.9 30.9
Without some more definitive estimate of the 2010 data, however, a determination of each 

monitor’s attainment status still cannot be made. Thus, a set of statistical regression 

techniques have been used to provide a larger weight of evidence of the two sites’ attainment 

status. 

Figure 2. Annual 98th Percentile Concentrations for Violating Sites within Milwaukee WI
Figure 3. 24-hour PM2.5 Design Values for the Violating Sites within Milwaukee WI

2010 Data Imputation for 55-079-0043 and 55-079-0099

An analysis has been completed to determine the potential of two discontinued monitors 

within the Milwaukee, WI CSA of violating the 24 hour PM2.5 National Ambient Air Quality 

Standards (NAAQS) for the three year period covering 2008 -2010. The monitors were shut 

down at the end of 2009 and have two complete years of Federal Reference Method (FRM) 

data from 2008-2009. Without 2010 data, the two violating sites cannot be compared to the 

NAAQS to determine if they are, in fact, meeting the 24-hour standard. Comparing the two 

discontinued sites to nearby sites allows for the 2008-2010 design values estimation which 

indicates whether or not the two sites could have met the standard if they had remained 

operational for 2010. Three slightly different techniques were utilized to predict the 2010 98th

percentiles for the two sites and then compute the 24-hour NAAQS. All three techniques use 
linear regression to relate the concentrations from the discontinued sites to a base site 

containing 2010 data. The residuals from the regression provide an indication as to the overall 

error in the calculated design values. 

The Office of Air Quality Planning and Standards (OAQPS) developed the first technique to 

originally aid in predicting annual PM2.5 design values in areas which did not meet specific data 

completeness requirements. A linear relationship between the monitor missing data and one 

that can be used as a comparison site is created using ordinary least squares (OLS). This type of 

relationship can be expressed in the form:

 (1) 

with

 (2)

The comparison monitor is chosen from among sites within the general vicinity of the site with 

missing data and is usually the highest correlated site based on historical data. Equation 1 

predicts the 2010 concentrations at the two sites by taking 2010 data from the comparison site 

and multiplying those concentrations by the slope and adding the intercept from the linear 

regression. The linear regression yields a set of residuals (the error term in Equation 2) which is 

the difference between what the concentration the model predicts and what the site actually 

measured. Through a process called bootstrapping, the residuals are randomly sampled 1000 

times with replacement and then added to the prediction. The residual addition accounts for 

the random error from the regression. In the case of Milwaukee, the comparison site had a 

total of 177 observations for 2010. A total of 1000 sets of 177 observations were therefore 

produced for each missing site. A 2010 98th percentile and, subsequently, a 2008-2010 design 

value for each set were calculated yielding 1000 individual design values. OAQPS accepts a 

monitor as meeting the standard when at least approximately 90% of the bootstrapped design 

values meet the standard which would mean that around 900 of the 1000 bootstraps would 

have to meet the standard. Depending on how the residuals are randomly chosen, the number 

of bootstrapped design values greater than the standard could vary by analysis. Nonparametric 

statistical tests are employed to demonstrate that the majority of the bootstrap samples meet 

the NAAQS (Conover, 1999). 

The bootstrapped residuals can also be used to determine the variability in the slope and 

intercept of the relationship between the discontinued and comparison sites. The resampled 

residuals are added to the historical predictions and then the regression is recalculated 

resulting in a slightly different slope and intercept. The residuals from this regression are then 

sampled as described above to give a set of 98th percentiles and design values. The whole 
process is repeated 1000 times and the final results are processed in the same manner as 

described above. Region 5 used this technique for the two discontinued Milwaukee sites and 

showed that it is comparable to the OAQPS technique. 

The two previously described analyses assume that there is only error in the measurements 

made at the two discontinued sites and that the concentrations at the comparison monitor are 

essentially error free. Because both the discontinued and comparison monitors are from the 

same monitoring network and the measured PM2.5 concentrations are subject to the same 

operating and quality assurance procedures, an error free assumption for the comparison 

monitor is unrealistic. To account for the error in both the concentrations from the 

discontinued and comparison sites, Region 5 also imputed the missing data from the two 

discontinued sites using Deming regression which is also known as orthogonal regression. This 

type of regression is similar to the OLS regression displayed in Equations 1 and 2 with the 

exception that the PM2.5comparison site term also has an error term associated with it. The Deming 

regression model can be expressed in the form:

 (3) 

with

 (4)

 (5)

Equations 3 through 5 are solved iteratively to get estimates for the slope and intercept terms. 

The residuals (the error terms in Equations 4 and 5) are resampled or bootstrapped similarly to 

what was done to the error term for the Region 5 OLS analysis. The bootstrapping is repeated 

1000 times like the processes described above and the results are similarly analyzed. 

When building the relationships between each discontinued site and the potential comparison 

sites, censoring criteria determine which potential sites could be utilized as comparison 

monitors. To be eligible to be a comparison site, there has to be at least 100 paired samples 

available over the course of 20 calendar quarters (5 years of data) and no set of calendar 

quarters over the 5 year period can have less than 20 paired samples. The correlation 

coefficient (R2) was calculated for each candidate missing/comparison site pair and the 

comparison site with the highest correlation to the missing site was chosen to be used for the 

final analysis. In both cases for Milwaukee, the comparison site which had the highest 

correlation to each of the discontinued sites was the Wisconsin DNR Southeast Headquarters 

site (AQS ID: 550790026). The results of the correlation analysis along with the regression 

results for each site are shown in Figures 4 and 5. 
Figure 4. PM2.5 Relationship between the Virginia Street Fire Station Site (550790043) and 

the Wisconsin DNR Southeast Headquarters Site (550790026)
Figure 5. PM2.5 Relationship between the Milwaukee Fire Department Headquarters Site 

(550790099) and the Wisconsin DNR Southeast Headquarters Site (550790026)
Table II: Results from the data imputation techniques for the two discontinued PM2.5 sites in 

Milwaukee, WI

Monitor Comparison 

Monitor R

2 Method

Number of 

bootstraps 

below 

NAAQS

Number of 

bootstraps 

above 

NAAQS

% of 

bootstraps 

greater 

than 

NAAQS

3 Year 

Design 

Value

What is the chance that 

we could incorrectly

conclude that the 

monitor violates the 

24-hour PM2.5 

NAAQS?

What is the chance that we 

could incorrectly conclude 

that the monitor does not 

violate the 24-hour PM2.5 

NAAQS?

550790043 550790026 0.91 Region 5 995 5 1% 34 100% 0%

(Virginia 

Street Fire 

Station)

(Wisconsin 

DNR SE HQ) OAQPS 999 1 0% 34 100% 0%

Deming 1000 0 0% 34 100% 0%

550790099 550790026 0.86 Region 5 898 102 10% 34 44% 60%

(Milwaukee 

Fire 

Department 

HQ)

(Wisconsin 

DNR SE HQ) OAQPS 903 97 10% 34 64% 39%

Deming 920 80 8% 34 98% 2%

Table II shows the monitor ID for the discontinued site and the comparison monitor with the 

highest correlation to the discontinued site. The correlation coefficient (R2) is provided in the 

third column with the fourth column giving the method used for the analysis. The three year 

design value is the estimate directly predicted from the regression equations (Equations 1 and 

3). The last two columns display the results of the nonparametric quantile tests and are 

displayed in a way that reflects the probability of making an erroneous conclusion which would 

cause the null hypothesis to be rejected and the acceptance of the alternative. A monitor that 

attains the standard from the bootstrap results would have a very high probability for Test #1 

(>=95%) and a very low probability for Test #2 (<=5%). In cases where the probabilities from 

the two tests do not meet these requirements, the monitor is either leaning more toward 

attainment or nonattainment. This “gray” zone signifies that other evidence may be necessary 

to aid in deciding whether or not the monitor is attaining the standard. 

The OAQPS results listed in Table II represent an independent running of the OAQPS regression 

technique by Region 5. OAQPS’ results showed both discontinued sites having three year 

design values of 34 ug/m3. The Virginia Street Fire Station site (550790043) had 2 bootstrap 

simulations that produced a failing design value (Test #1 probability: 100% and Test #2 

probability: 0%) while the Milwaukee Fire Department Headquarters site (550790099) had 101 

simulations that produced a failing design value (Test #1 probability: 48% and Test #2 

probability: 56%).
For both sites, the regression results predict attainment with all techniques returning three 

year design values of 34 ug/m3. The relationship between a discontinued site’s concentrations 

and its comparison site is high for both sites. The number of bootstraps that are below the 

NAAQS for the Virginia Street fire Station indicates attainment of the 24-hour standard. The 

Milwaukee Fire Department Headquarters site, however, falls more into the “gray” zone for 

both the Region 5 and OAQPS methods making it difficult to determine whether the site is 

attaining the NAAQS. The Deming regression results, however, do indicate attainment and may 

be considered more reliable because the error in the comparison site’s measurements is also 

accounted for. 

Conclusions

 The trends in the annual 98th percentiles show that the two discontinued sites were 

tracking the other two violating sites for 2007-2009 and that the 98th percentile 

concentration differences were around 2-3 ug/m3 between the four sites. If this trend 

continued for 2010, the two discontinued sites would probably have had 98th percentile 

concentrations less than the calculated 2010 critical values of approximately 37 ug/m3 

and the two sites would have attained the standard.

 If we use OAQPS’ criterion to determine whether or not a monitor passes using the 

regression methods, the Virginia Street Fire Station (550790043) attains the standard. 

The Milwaukee Fire Department HQ site’s status lies more in a “gray” zone, but the 

Deming regression which accounts for the error in both the discontinued and 

comparison sites’ data shows that the site attains the 24-hour NAAQS.

 With the weight of evidence from both the trends in the 98th percentile concentrations 

and the results from the regression analyses, AMAS would support a clean data 

determination for the Milwaukee area.

 However, the Milwaukee Fire Department Headquarters site must be re-established as a 

means to confirm attainment and monitor ambient PM2.5 concentrations within 

Milwaukee for a period consistent with the requirements of 40 CFR Part 58. The site 

should be operational by January 1, 2012 and operate on a one in three day schedule.

References

Conover, W. J.; Practical Nonparametric Statistics; John Wiley and Sons; New York; 1999; third 

edition; pp. 135-143.