U.S. ENVIRONMENTAL PROTECTION AGENCY OFFICE OF INSPECTOR GENERAL

Catalyst for Improving the Environment 

## Evaluation Report



# EPA Needs an Oversight Program for Protocol Gases

Report No. 09-P-0235 September 16, 2009Report Contributors: Jim Hatfield Geoff Pierce Rick Beusse

Abbreviations

CAA Clean Air Act CAIR Clean Air Interstate Rule CEMS Continuous Emissions Monitoring System CFR Code of Federal Regulations CO Carbon monoxide CO 2 Carbon dioxide EPA U.S. Environmental Protection Agency GMIS Gas Manufacturer’s Intermediate Standard NAAQS National Ambient Air Quality Standards NIST National Institute of Standards and Technology NO Nitric oxide NO x Nitrogen oxides NO 2 Nitrogen dioxide NTRM NIST Traceable Reference Material OAP Office of Atmospheric Programs OAQPS Office of Air Qualit y Planning and Standards OAR Office of Air and Radiation OIG Office of Inspector General ORD Office of Research and Development PGVP Protocol Gas Verification Program ppm parts per million SO 2 Sulfur dioxide SRM Standard Reference Material

Cover photo: Gas cylinders stored at the Nationa l Institute of Standards and Technology laboratory in Gaithersburg, Maryland. (P hoto courtesy National Institute of Standards and Technology)U.S. Environmental Protection Agency 09-P-0235 September 16, 2009 Office of Inspector General 

# At a Glance

Catalyst for Improving the Environment
Why We Did This Review The accuracy of continuous emissions monitors is critical to the U.S. Environmental Protection Agency’s (EPA’s) Acid Rain Program because data from these monitors determine the number of allowances a utility can bank, sell, or trade. Specialized gases known as “EPA Protocol Gases” are used to calibrate and assure the quality of these monitors. We sought to determine whether certified concentrations of these gases are accurate. Background Vendors produce and certifyEPA Protocol Gases in highpressure cylinders according to EPA procedures. EPA regulations require the use of these gases, or National Institute of Standards and Technology (NIST)-certified reference materials, when conducting quality assurance for continuous emissions and ambient air monitoring systems. EPA’s Acid Rain Program requires that the certified concentration of the gases be within ± 2 percent of the true concentration.
For further information, contact our Office of Congressional, Public Affairs and Management at (202) 566-2391. To view the full report, click on the following link: www.epa.gov/oig/reports/2009/ 20090916-09-P-0235.pdf 

### EPA Needs an Oversight Pr ogram for Protocol Gases

What We Found We purchased 87 cylinders of EPA Prot ocol Gases and had NIST analyze each cylinder to determine whether the three gaseous mixtures contained in each cylinder (carbon dioxide, nitric oxide, an d sulfur dioxide) met the Acid Rain Program’s accuracy criterion. We found that 89 percent (233 components) met the Acid Rain Program’s accuracy crite rion and 11 percent (28 components) did not. Of the 28 components that did not meet the criterion, 17 were within 3.0 percent of the NIST-determined true con centration; 7 were within 3.0 to 5.0 percent; and 4 exceeded the true concentr ation by more than 5.0 percent. Our sample was not designed to estimate the impact of the test results on the Acid Rain Program. However inaccurately certif ied concentrations could cause system operators to unknowingly calibrate their monitoring systems to record inaccurate measurements. For example, if a utility overstates its emissions, it could lose the opportunity to sell allowances to other utilities. If a utility understates its emissions, the utility and regulators may incorrectly conclude that the source is complying with emissions standards. With respect to ambient air monitoring, the accuracy of these monitors is important because the data are used to determine whether areas are in compliance with the Nation’s ambient air quality standards. EPA has conducted only two tests of the accuracy of EPA Protocol Gases since 1997, when EPA’s Office of Research and Development discontinued its annual testing program. Thus, EPA does not have reasonable assurance that the gases used to calibrate emissions monitors fo r the Acid Rain Program and continuous ambient monitors for the Nation’s air monitoring network are accurate.

What We Recommend We recommend that the Office of Air and Radiation (OAR) implement oversight programs to assure the quality of EPA Prot ocol Gases used to calibrate continuous emissions monitoring systems and ambient air monitors. We also recommend that the Office of Research and Development (ORD) update and maintain the protocol gas procedures to ensure that the protoc ol meets the objectives of the Acid Rain, ambient air, and stationary source air programs. OAR and ORD concurred with our recommendations. OAR has initiated effo rts to conduct another gas audit later this year. OAR also plans to propose a rule later in 2009 to establish a largely self-supported, annual gas audit program of protocol gases used for the Acid Rain Program. Further, OAR plans to implement a separate verification program to address the lower concentration protocol gases used to calibrate continuous ambient air monitors. ORD will update and maintain the protocol gas procedures. EPA’s planned actions meet the intent of our recommendations.UNITED STATES ENVIRONMENTAL PROTECTION AGENCY WASHINGTON, D.C. 20460

OFFICE OF INSPECTOR GENERAL

September 16, 2009

MEMORANDUM

SUBJECT: EPA Needs an Oversight Pr ogram for Protocol Gases Report No. 09-P-0235

Assistant Inspector General for Program Evaluation FROM: Wade T. Na jjum

TO: Gina McCarthy Assistant Administrator for Air and Radiation

Lek G. Kadeli Acting Assistant Administrato r for Research and Development

This is our report on the subject evaluation cond ucted by the Office of Inspector General (OIG) of the U.S. Environmental Protec tion Agency (EPA). This report contains findings that describe the problems the OIG has identified and correc tive actions the OIG recommends. This report represents the opinion of the OIG and does not necessarily re present the final EPA position. Final determinations on matters in this report will be made by EP A managers in accordance with established audit resolution procedures.
The estimated cost of this report – calculat ed by multiplying the project’s staff days by the applicable daily full cost billing rates in effect at the time, including the costs of purchasing EPA Protocol Gases and having them analyzed by th e National Institute of Standards and Technology

–

is $665,846.
Action Required
In accordance with EPA Manual 2750, EPA’s Audit Management Process , the Office of Air and Radiation should provide a writte n response within 90 calendar days. The Office of Air and Radiation’s response should include a corrective action plan and planned completion dates for Recommendations 2-1 and 2-2. The Office of Re search and Development submitted a corrective action plan that sufficiently addresses Reco mmendation 2-3. As such, we are “closing” Recommendation 2-3 in our tr acking system upon issuance of this report. These recommendations will be tracked to completion in the Agency’s tracking system. No furtherresponse is required for Recommendation 2-3. As outlined in EPA Manual 2750, the Agency is responsible for tracking the impl ementation of these actions in its Management Audit Tracking System. We have no objections to the further rel ease of this report to the public. This report will be available at http://www.epa.gov/oig

. If you or your staff have any questions regard ing this report, please contac t me at (202) 566-0832
or najjum.wade@epa.gov , or Rick Beusse at (919) 541-5747 or beusse.rick@epa.gov

.EPA Needs an Oversight Program for Protocol Gases 09-P-0235 

# Table of Contents

Chapters

1 Introduction

........................................................................................................... 1 Purpose

.......................................................................................................... 1 Background

.................................................................................................... 1 Noteworthy Achievements

.............................................................................. 5 Scope and Methodology

................................................................................. 5

2 EPA Needs an Oversight Program to Assure the Accuracy of Protocol Gases

...................................................................................................... 7 EPA Does Not Have an Oversight Program to Assure the Accuracy of Protocol Gases

................................................................... 7 Low Concentration NO and SO 2 Gases and Lower-Cost Cylinders Calibration Gases Play an Important Role in Assuring Data Quality Performed Worst in Our Tests

............................................................... 8 for EPA Air Programs

............................................................................ 10 Conclusions

.................................................................................................... 13 Recommendations

......................................................................................... 13 Agency Comments and OIG Evaluation

......................................................... 13 Status of Recommendations and Potential Monetary Benefits

................................. 15 

## Appendices

A Details on Scope and Methodology

.................................................................... 16

B Test Results with Uncertainty Factored into Analyses

.................................... 18
C OAR’s Response to OIG Draft Report

............................................................... 20

D ORD’s Response to OIG Draft Report

................................................................ 24

E Distribution

............................................................................................................ 2709-P-0235 

# Chapter 1 Introduction
Purpose The U.S. Environmental Protection Agency (EPA) Protocol Gases are used to calibrate and test continuous emissions monitoring systems (CEMS) and ambient air quality monitoring systems. EPA requi res the use of CEMS for large utilities covered by the Acid Rain Program and ot her clean air programs. Consequently, the accuracy of the EPA Protocol Gases is critical to ensuring the integrity of emissions trading and other EPA programs. We conducted this evaluation to determine whether the certified concentr ations of EPA Protocol Gases were within acceptable limits for accuracy. 

### Background
The integrity and effectiveness of EPA’s air programs depend upon collecting accurate pollutant emissions and air quality data. EPA requires that the systems collecting these data periodically undergo certain quality assurance procedures to assure their accuracy. Thes e procedures include daily calibration and accuracy tests of the monitoring systems. Calibration tests check the accuracy of these monitoring systems by running a gas mixt ure of a known concentration through the systems and comparing the systems’ r eadings with the known concentration. The systems are adjusted accordingly to eliminate measurement inaccuracies.
The certified concentrations of gaseous mixtures used to calibrate and test monitoring systems should accurately reflect the true gaseous concentration. EPA regulations require that the gases used for calibrating a nd testing the accuracy of ambient air quality analyzers a nd continuous emissions monitors 1 be traceable to either a National Institute of Sta ndards and Technology (NIST) Standard Reference Material (SRM) or a NIST Traceable Reference Material (NTRM). EPA has established an EPA Traceability Pr otocol that specialty gas producers must follow when preparing calibration gases to meet the NIST traceability requirement. The gases produced in accord ance with these standards are referred to as EPA Protocol Gases.
This protocol was designed by EPA, NIST , and specialty gas producers so that a laboratory analyst can analyze and certify compressed gas calibration standards with low uncertainty and with traceability to NIST. The protocol specifies a general analytical procedure for determining the concentration of the standards, but it does not specify the analytical instrumentation or the gas-handling

1 Continuous emissions monitors continuously measure pollutants emitted into the atmosphere in exhaust gases from combustion or industrial processes.

109-P-0235

apparatus that the analyst mu st use during the analysis. In summary, the protocol requires that (1) a NIST-certified gaseous mixture be used as the analytical reference standard, (2) a monthly calibra tion curve for the instrumentation be established, and (3) both the NIST-certifi ed gaseous mixture and the candidate EPA Protocol Gas be measured at least three times, with reactive gas mixtures measured at least three additional times at least 7 days after the first set of measurements. The protocol has detailed st atistical procedures for estimating the total uncertainty of these standards. Although it includes procedures for estimating uncertainty, the protocol does not specify that EPA Pr otocol Gases meet any given level of uncertainty and does not establish specific accuracy criteria for the standards’ uncertainty. However, the Acid Rain Program has prescribed an uncertainty standard for EPA Protocol Gases used to calibrate CEMS as described in the following section. Acid Rain Program Requirements for EPA Protocol Gases
The Acid Rain Program implements Title IV of the Clean Air Act (CAA) to reduce emissions of sulfur dioxide (SO 2 ) and nitrogen oxides (NO x ), the primary causes of acid rain. The program uses an allowance trading system to reduce SO 2 emissions. Under this system, affected utility units are allocated allowances (one allowance = one ton of SO 2 ) that permit a unit to emit SO 2 during or after a specified year. Utilities can buy, sell, or bank allowances. The program began in 1995. For each ton of SO 2 emitted in a given year, one allowance is retired and can no longer be used. The program is phased in, with the final SO 2 cap set at 8.95 million tons in 2010. The Acid Rain Program does not use a trading system 2 to reduce emissions for NO x , but instead establishes emissions limits for covered units that are designed to achieve the program’s goal of reducing NO x emissions by 2 million tons from 1980 levels.
Continuous emissions monitoring is instru mental in ensuring that the mandated reductions of SO 2 and NO x under the Acid Rain Program are achieved. Accordingly, EPA has established requi rements for continuously monitoring SO 2 and NO x (and other parameters) for units regul ated under the Acid Rain Program. Appendices A and B to 40 Code of Federa l Regulations (CFR) Part 75 describe specification and test procedures for these monitors, which include daily calibration tests and periodic accuracy test s. The regulations require that the calibration gases used to conduct these tests must be NIST-certified reference standards or that their concentrations must be traceable to NIST-certified reference standards according to the EPA Traceability Protocol for Assay and Certification of Gaseous Calibration Standards . In January 1993, EPA’s Office of Atmospheric Programs (OAP) included a 2.0 percent accuracy criterion in its final Acid Rain Program emissions mon itoring rule (58 Federal Register 3731).

2 NO x emissions are traded under other air programs, including the NO x Budget Trading Program, which is designed to reduce NO x emissions in order to reduce ambient ozone levels.

209-P-0235

Appendix A to 40 CFR Part 75 3 specifies that the EPA Protocol Gases must have a certified uncertainty (95 percent confiden ce interval) that must not be greater than plus or minus (±) 2.0 percent of the ce rtified concentration (tag value) of the gas mixture.
In January 2008, EPA revised the Acid Ra in Cap and Trade Program regulations to require that producers of EPA Protocol Gases partic ipate in EPA’s Protocol Gas Verification Program in order to certify their gases as EPA Protocol Gases. National Ambient Monitoring Pr ogram Requirements for EPA Protocol Gases
Appendix A to 40 CFR Part 58 4 describes quality assurance procedures for national ambient air monitors. States a nd EPA use these monitors to determine whether an area’s air quality meets or exceeds the National Ambient Air Quality Standards (NAAQS). The qua lity control procedures in clude biweekly one-point checks of carbon monoxide (CO), SO 2 , and nitrogen dioxide (NO 2 ), and annual performance evaluations that test the monitors against gases of known concentrations for at least three different ranges of concentrations. Appendix A to this regulation requires that cylinders of compressed gas used to obtain test concentrations must be traceable to eith er a NIST Traceable Reference Material or a Gas Manufacturer’s Intermediate St andard, certified in accordance with EPA’s Traceability Protocol for Assay and Ce rtification of Gaseous Calibration Standards. Further, vendors advertising cer tification with the procedures and distributing gases as “EPA Protocol Gas” must partic ipate in the EPA Protocol Gas Verification Program or not use “EPA” in any form of advertising.
Unlike the Acid Rain Program, the ambien t monitoring regulations do not specify accuracy criteria for the EPA Protocol Gases used to calibrate ambient air monitors. Stationary Source Performance Standards Requirements for EPA Protocol Gases The CAA required EPA to establish emissions standards 5 for certain stationary sources that cause or contri bute significantly to air po llution. These standards are intended to promote use of the best air pollution control technol ogies and apply to sources that have been constructed or modi fied since the proposal of the standard. These standards are published in 40 CFR Part 60, and may require the source to install CEMS to demonstrate compliance.

3 Appendix A to Part 75—Specifications and Test Procedures. 4 Appendix A to Part 58—Quality Assurance Requirements for State and Local Air Monitoring Stations, Special Purpose Monitors, and Prevention of Significant Deterioration Air Monitoring. 5 Referred to as New Sour ce Performance Standards.

309-P-0235

Part 60 contains varied requirements for using calib ration gases depending upon the EPA method used to conduct emissions testing. Specifically, the methods for determining SO 2 (Method 6C) and NO x (Method 7E) emissions from stationary sources require that the calibration ga ses be prepared according to the EPA protocol or certified by the tester using a specified wet chemical test method. The electric utility, petroleum, and munici pal waste incinerati on industries are the largest industries (in terms of emissions and the number of sources) subject to continuous emissions monitoring requiremen ts and the use of EPA Protocol Gases for Part 60 emissions standards. EPA’s Protocol Gas Verification Program
From 1985 to 1997, EPA’s Office of Research and Development (ORD) conducted a series of analytical tests 6 of EPA Protocol Gases sold by specialty gas producers. In 1996, ORD issued a final dr aft exit strategy th at transferred the administration of this and other quality assurance programs to the Office of Air Quality Planning and Standards (OAQPS) of the Office of Air and Radiation (OAR). ORD provided OAR with resour ce support during the transition, but provided no further resources after the transfer was completed in 1998. The annual tests were discontinued after 1 997, and no additional tests occurred until 2003. Due to a lack of analytical test s and a concern that EPA Protocol Gas quality may have declined, the OAP conduc ted a test in 2003. OAP’s contractor purchased 42 EPA Protocol Gas cylinders containing tri-blen d mixtures of carbon dioxide (CO 2 ). nitric oxide (NO), and SO 2 from a total of 14 specialty gas vendors nationwide. A third party purchased the cylinders for EPA so that the gas vendors would not know that EPA was analyzing th e cylinders. A gas passed the test if the “true” concentration of the gas differe d from the certified concentration by ± 2 percent or less. Overall, 89 percent of the gases passed on a gas component basis; 11 percent did not pass the test.

OAP conducted another test in 2006 but decided not to publish the results because of concerns about the repres entativeness of some of the sample gases. Instead of purchasing gases directly from vendor s, OAP arranged for end users (i.e., utilities) to provide EPA with the Prot ocol Gases for testing. However, EPA encountered several problems with th is approach, including over- and under-representation of manufacturers and production sites, as well as old and used samples.
Approximately 6 years ago, EPA starte d developing an ongoing EPA Protocol Gas Verification Program (PGVP). After a development process involving U.S. specialty gas companies, NIST, the Institute of Clean Air Companies, and EPA, tentative agreement was reached on a specialty gas company-funded, ongoing draft PGVP. EPA promulgate d a requirement for producers to participate in this

6 EPA calls these tests EPA Protocol Audits . We refer to these audits as tests in this report to avoid confusion with the term “audit” as defined by the Government Accountability Office’s Government Auditing Standards , and applicable to work conducted by th e Office of Inspector General (OIG).

409-P-0235

program in a January 24, 2008, final rule. However, in March 2008, one specialty gas vendor petitioned EPA for reconsiderati on of the final rule. The vendor cited several concerns including the nature of the funding for the PGVP. Since that time, EPA has been working to resolve th ese concerns, but sti ll had not funded the program as of May 2009. EPA had planne d to have EPA Protocol Gas vendors fund the verification program through payments of fees to a third party. However, EPA has determined such a pr ocess could violate the Miscellaneous Receipts Act. 7 

### Noteworthy Achievements
Recognizing the importance of EPA Pr otocol Gases, the OAP conducted two protocol gas assessments, one in 2003 and another in 2006. Although the sample size in each of these assessments was not st atistically significant and market share information was not available, both assessments indicated that some specialty gas vendors were not meeting the EPA Protocol Gas accuracy criterion of ± 2.0 percent. OAP posted the results, in cluding the names of the specialty gas companies, on an Agency Website. Based on comments by specialty gas companies, the 2003 accuracy check resulted in some vendors making improvements in personnel, training, quality, and equipment. When we originally approached EPA with our proposal to asse ss the accuracy of EPA Protocol Gases, OAR supported our plans and cooperated fully with our evaluation team to develop a useful assessment of the EPA Protocol Gas program. 

### Scope and Methodology

We conducted this evaluation to test the accuracy of EPA Protocol Gases used to calibrate continuous emissions monitoring equipment. We designed our sampling methodology to test at least one set of th ree EPA Protocol Gas cylinders from every specialty gas manufacturing location in the country and to maintain the confidentiality of the tests so that the gases we purchased would represent those sold to utilities and other customers. 8 In order to maintain the confidentiality of the tests, we contracted with a third pa rty to purchase the gases and deliver them to NIST for analysis

. Our contractor purchased tr i-blend mixtures of NO, CO 2 , and SO 2 , since tri-blend gases are often used by utilities. We identified 11 gas producers nationwide using 18 manufacturing locations to produce EPA Protocol Ga ses. Our contractor purchased at least one set of EPA Protoc ol Gas cylinders produced from each of the 18 manufacturing locations. In al l, we purchased 87 EPA Protocol Gas cylinders from 14 specialty gas vendors (i.e., 11 producers and 3 distributors), and

7 The Miscellaneous Receipts Act was pa ssed in order to ensure that govern ment agencies did not bypass the appropriations authority of Congress by augmenting their budgets via other means, such as user fees, fees for training courses, parking fees, contract and lease fees and revenues, monetary awards in court cases involving the agencies, court costs and fees, or civil penalties. 8 A set consisted of three cylinders in low-, medium-, and high-range concentrations.

509-P-0235

18 manufacturing locations. Ou r contractor delivered th e sample gases to NIST for analysis. We selected NIST because of its extensive experience in gas metrology and its reputation for analytical accuracy. NIST analyzed the gaseous concentrations of the 87 tri-blend cy linders (i.e., 261 separate component analyses) to determine whether the “true” concentrations were within ± 2 percent of the concentration certified by the vendor. 9

We reviewed documentation related to EP A’s past assessments of EPA Protocol Gases, including past results, progra m funding and implementation, and current plans for implementing a revised EPA Pr otocol Gas Verifica tion Program. We reviewed federal regulations to identify requirements fo r the use of EPA Protocol Gases. We interviewed staff and mana gers from ORD, OAQPS, and OAP.

We conducted this evaluation in accordan ce with generally accepted government auditing standards. Those standards require that we obtain suffi cient, appropriate evidence to provide a reasonable basis fo r our findings and conclusions based on our evaluation objectives. We believe th at the evidence obtained provides a reasonable basis for our findings and c onclusions based on our objectives. We conducted our work from June 2007 to July 2009.
Appendix A provides more detailed info rmation on our scope and methodology.

9 Vendors typically certify gaseous concentrations using either a NIST Traceable Reference Material (NTRM), a Standard Reference Material (SRM), or a Gas Manufact urer’s Intermediate Standard (GMIS). NTRMs are reference material produced by a commercial supplier with a well-defined traceability linkage to NIST. These reference materials are allowed to bear the NIST NTRM ce rtification mark and were developed to allow NIST to respond to increasing needs for high-quality reference materi als. SRMs are reference ma terials produced by NIST and are certified to have specific chemical or physical properties. A GMIS is a NIST traceable standard produced by the gas vendor. A NIST traceable st andard is certified with an instrume nt calibrated through direct traceability with a NIST standard such as an SRM or NTRM.

609-P-0235 

# Chapter 2 EPA Needs an Oversight Program to Assure the Accuracy of Protocol Gases

EPA does not have reasonable assurance that EPA Protocol Gases used to calibrate CEMS for the Acid Rain Program and continuous ambient monitors for the Nation’s air monitoring network are accurate. EPA has conducted only two assessments since 1997, when ORD discontinue d its program to test the accuracy of EPA Protocol Gases. We analyzed 261 gaseous components and found that 89 percent (233 components) met the AcidRain Program’s accuracy criterion for calibration gases, while 11 percent (28 co mponents) did not meet the accuracy criterion. Calibration gases must be accura te because inaccurate calibration gases can cause a system operator to calibra te a measurement system to produce inaccurate measurements. In turn, inaccura te CEMS measurements can affect the integrity of the computed allowances available for trade on the open market. 

### EPA Does Not Have an Oversight Progr am to Assure th e Accuracy of Protocol Gases
EPA does not have a viable oversight program in place to assure the quality of EPA Protocol Gases. Since 1997, EPA overs ight to assure th e accuracy of these gases has consisted of two analytical tests conducted by OAP in 2003 and 2006. However, EPA did not publish the 2006 test results because of concerns over the sampling methodology. Without an overs ight program, EPA does not have reasonable assurance that EPA Protocol Gases meet the accuracy criterion certified to by the vendors and required for the Acid Rain Program.
In January 2008, EPA promulgated a fi nal rule requiring EPA Protocol Gas producers to participate in an EPA P GVP. EPA anticipated implementing a verification program to assess the acc uracy of Protocol Gases on an ongoing basis. However, implementation was delayed when a specialty gas vendor petitioned EPA to reconsider the rule . As of May 2009, EPA’s OAP was still working to resolve legal issu es raised in the petition and deciding how to fund the program. As of May 2009, OAR’s OAQPS wa s developing a similar but separate verification program to test the accuracy of the EPA Protocol Gases used to calibrate continuous ambient air monitors.
Even though the Acid Rain Program and the ambient air monitoring program generally use different concentrations of EPA Protocol Gases in their programs, the gases must be produced and certifie d under the same EPA Protocol. Although ORD originally developed the Protocol, a co-author of the Protocol informed us that it was not clear who currently has re sponsibility for maintaining the Protocol. Although the 2003 EPA Protocol Gas a ssessment report and specialty gas

709-P-0235

producers have identified seve ral analytical issues not addressed by the current Protocol, the Protocol has not been revised since 1997.


### Low Concentration NO and SO 2 

### Gases and Lower-Cost Cylinders Performed Worst in Our Tests We purchased and analyzed 87 cylinders of tri-blend EPA Protocol Gases to determine whether each of the three gaseous concentrations (per cylinder) met the Acid Rain Program’s accuracy criterion. Of the 261 gaseous components analyzed, 89 percent met the Acid Rain Program’s accuracy criterion for EPA Protocol Gases, while 28 (11 percent) did not meet the accuracy criterion.
Of the 28 components that did not meet the criterion, 17 were within 3.0 percent of the NIST-determined true concentratio n; 7 were within 3.0 to 5.0 percent; and 4 exceeded the true concentration by more than 5.0 percent. All exceptions were for gaseous components from lower-cost cylinders (i.e., under $380). Twelve of the gas components’ “true” concentra tions were higher than the certified concentration, and 16 were lower than the certified concentration.
The gaseous components with the most exceptions were low concentration NO and SO 2 . The certified concentratio ns for low range NO and SO 2 gases exceeded the NIST analyzed or “true” concentration by more than ± 2 percent for 24 and 17 percent of the samples, respectively. The following table summarizes the test results for each range of gaseous components analyzed. Table 2-1: Results by Gas and Concentration

No. of Met the Accuracy Criterion Did Not Meet the Accuracy Criterion Gas Range Samples No. Percentage No. Percentage

NO Low 29 22 76 7 24 Med 29 29 100 0 0 High 29 25 86 4 14

CO 2 Low 29 26 90 3 10 Med 29 28 97 1 3 High 29 29 100 0 0

SO 2 Low 29 24 83 5 17 Med 29 25 86 4 14 High 29 25 86 4 14 Total 261 233 89 a 28 11 a Source: OIG, developed from data in NIST Report of Analysis, December 4, 2008
a Represents the overall percentage, not the column total.
We also assessed the test results on a cost-per-cylinder and production-facility basis. This analysis revealed that all gaseous components not meeting the accuracy criterion were from cylinders costing under $380. All of the gaseous components not meeting the accuracy criteri on came from 7 of the 18 (39 percent) facilities. These seven facilities accounted for 108 of the 261 components we tested, and 28 of these 108 components di d not meet the Acid Rain Program’s

809-P-0235

accuracy criterion. All 153 gaseous compone nts associated with cylinders costing over $380 met the accuracy criterion. The following table shows the total tests conducted, the cost per cylinder, and the test results for each production facility. Table 2-2: Test Results by Cost and Production Facility

Production

Facility Cost Per Cylinder Total Tests No. of Tests Meeting the

Accuracy Criterion No. of Tests Not Meeting the Accuracy Criterion A <380 18 15 3 B <380 18 17 1 N <380 18 9 9 D <380 a 18 15 2 K <380 9 7 2 Q <380 18 10 8 R <380 9 6 3

Subtotal < 380 108 80 28 C >380 9 9 0 D >380 a 9 9 0 E >380 9 9 0 F >380 9 9 0 G >380 18 18 0 H >380 18 18 0 I >380 9 9 0 J >380 9 9 0 L >380 9 9 0 M >380 18 18 0 O >380 18 18 0 P >380 18 18 0

Subtotal >380 153 153 0 Totals 261 233 28 Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
a Three sets of cylinders (a set consists of a low-, a medium-, and a high-range cylinder) were produced at this location but were purchased from two different vendors. Two sets of cylinders cost under $380 per cylinder and one set cost over $380 per cylinder. We further analyzed the test results, ta king into account the vendor’s analytical technique and the reference standard us ed by the vendor to certify its gaseous concentrations. This analysis showed that low-cost cylinders certified with a Gas Manufacturer’s Intermediate Standard (GMI S) performed the worst in our tests. Specifically, 24 of the 28 (86 percent) ga seous components that did not meet the Acid Rain Program’s accuracy criterion we re from lower-cost cylinders that were certified using GMIS. Although most of the gaseous components not meeting the accuracy criterion were certified usi ng GMIS, many GMIS-certified components did meet the accuracy standard. However, all of these gaseous components were from cylinders costing over $380. The fo llowing table shows the test results by certification standard and cylinder cost.

909-P-0235

Table 2-3: Pass/Fail Rates By Analytical Reference Standard

Reference Standard Used and Cost of Gas Cylinder No. of Gaseous Components Certified with this Standard Components Not Meeting the Acid Rain Program’s Accuracy Criterion No. Percentage NTRM/SRM < $380 8 2 25 NTRM/SRM > $380 103 0 0 GMIS < $380 96 24 25 GMIS > $380 50 0 0 Unknown a < $380 4 2 50 Totals 261 28 11 b Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008.
a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration

. b Represents the overall percentage, not the column total. 

### Calibration Gases Play an Important Role in Assuring Data Quality for EPA Air Programs Calibration gases play an important role in helping to assure the quality of data used in EPA programs. Inaccurate calib ration gases can negatively affect EPA programs that rely upon accurate emissions or ambient air quality measurements by causing system operators to calibrate their measurements systems inaccurately. For example, if a calibration gas used by a utility was certified to contain 100 parts per million (ppm) of SO 2 , but only contained 96 ppm, the system operator would unknowingly calibrate the CEMS to read 96 ppm as 100 pp m. This would result in the CEMS overestimating emissions. Conversely, if the true concentration of the calibration gas were more than the certified concentration, the system operator would unknowingly calibrate the CEMS to underestimate emissions.
The number of monitoring systems poten tially using EPA Protocol Gases is significant. For example, over 3,500 electr ic generating units were covered by the SO 2 allowance trading program in 2007. A subset of almost 1,000 of these 3,500 units was also covered by the NO X provisions of the Acid Rain Program. Further, approximately 1,300 ambient air monitors collect data on CO, SO 2 , and NO x concentrations. An EPA contractor survey ed Protocol Gas use in preparation for EPA’s analytical tests of Protocol Gases in 2003. The contractor estimated that utilities purchased from 37,500 to 75,000 Pr otocol Gas cylinders annually and missions testing companies purchased from 16,500 to 55,000 cylinders annually. The survey did not estimate annual purchas es by State and local agencies, but one large State reported using about 75 Protocol Gases a year for its ambient monitoring network.
Due to the nature of our sample selecti on and a lack of data on vendor market shares, we were unable to develop a sampling methodology to ensure that the number of samples selected from each producer represented the number of cylinders produced and sold to consumers. Consequently, we did not project our test results to the universe of EPA Protocol Gases, or estimate the impact that

1009-P-0235

these results could have on the Acid Rain Program. Despite this limitation, our tests represent the largest one-time assessment of the Protocol Gas industry, and revealed some potential areas of concer n with respect to the quality of lessexpensive gases certified using GMIS.
The following sections describe the importa nce of accurate monitoring data to the Acid Rain and Air Monitoring programs. These data are collected by measurement systems whose accuracy is a ssured by quality assurance activities such as calibrations using EPA Protocol Gases. Calibration Gases Help Assure Importance of Quality Assurance Accuracy of Emissions Data for Tests for CEMS Trading Programs “Sources are required to conduct For sources participating in emissions stringent quality assurance tests of trading programs, overestimating their monitoring systems, such as emissions can result in lost opportunities daily and quarterly calibration tests to sell or bank allowances. and a semiannual or annual relative Underestimating emissions could result accuracy test audit. These tests ensure in banking or selling allowances that did that sources report accurate data and not really exist, and provide an provide assurance to market inaccurate picture of a facility’s progress participants that a ton of emissions measured at one facility is equivalent in reducing emissions. to a ton measured at a different facility.” The volume of SO 2 allowances traded under the Acid Rain Cap and Trade Source: NO X Budget Trading Program Compliance and Environmental Results Program is significant. According to 2007, EPA-430-R-08-008, December 2008 EPA’s Acid Rain and Related Programs: 2007 Progress Report , the total value of the SO 2 allowance market was over $5.1 billion in 2007. 10 In that same year, 4,700 private transactions involved 16.9 million allowances.
In addition to the Acid Rain Cap and Trade Program, utilities and other large combustion sources participate in EPA’s NO X Budget Trading Program. This program is designed to reduce emissions of NO X to improve air quality in the eastern part of the United St ates. Under this program, facilities in the eastern States can buy and sell NO X emissions allowances. As noted in the NOX Budget Trading Program Compliance and Environmental Results report for 2007, 99 percent of the NO X emissions measured under the NO X Budget Program were measured by CEMS. In 2007, approximately 700,000 NO X allowances were traded in private transactions. 11 The price of NO X allowances varied from about $500 to $1,000 per ton during 2007, with the year-end closing price at $825 per ton. The total value of NO X allowances traded in 2007 exceeded $350 million.

10 Based on total annual volume of 15,776,130 tons at an average nominal price of $325 per ton. 11 Private transactions include all tran sfers initiated by authorized account representatives for any compliance or general account purposes.

1109-P-0235

The importance of calibration gases could increase as EPA expands the use of emissions trading programs. On Ma rch 10, 2005, EPA announced the Clean Air Interstate Rule (CAIR). 12 CAIR caps emissions for NO X and SO 2 for 28 eastern States and the District of Columbia. States must achieve required emissions reductions by requiring power plants to par ticipate in an interstate cap and trade program or meet individual State em issions budget by implementing its own measures. Under CAIR, States may choose to participate in an EPA-administered regional trading program.
In order to address climate change, C ongress is considering a cap and trade program to control greenhouse gas emissions. The primary greenhouse gases include CO 2 , methane, nitrous oxide, and s ynthetically produced fluorinated gases. 13 Implementing a greenhouse gas emissions cap and trade program could greatly expand the use and importance of EPA Protocol Gases if such a program included monitoring and calibration requi rements similar to the Acid Rain Program. Calibration Gases Help Assure th e Accuracy of Ambient Monitoring Measurements
EPA Protocol Gases also help assure the accuracy of ambient air quality monitors used to collect ambient air quality data. EPA Protocol Gases are needed to calibrate the monitoring systems for three National Ambient Air Quality Standards – CO, NO 2 , and SO 2 . Based on the ambient monitoring data, EPA determines whether the air quality in an ar ea meets or exceeds the standard. If the air quality exceeds the standard, the economic cost to the community for control measures needed to reduce emissions and co mply with the standard can be in the hundreds of millions of dollars. Conversel y, if EPA were to determine that an area met the standard because of monitori ng data that produced inaccurately low data, control measures may not be impl emented to protect the public health. Consequently, it is extremely important th at the monitoring data are accurate so that EPA can make the pr oper attainment decision.

The EPA Protocol Gases we tested were tri-blend cylinders with gaseous concentration levels generally used by u tilities to calibrate CEMS. Ambient air monitoring systems operators would genera lly use EPA Protocol Gases of lower concentrations to calibrate their monito rs, and perform other quality assurance tests. For example, low concentration tri-blend EPA Protocol Gas cylinders are used by OAQPS when they conduct the National Performance Audit Program to assess the accuracy of ambient air quality m onitors. Since we did not test gases of the lower concentration range used by OAQPS for the National Performance Audit Program, we cannot co mment on their quality.

12 On December 23, 2008, the DC Circuit Court of Appeals reversed an earlie r decision and granted EPA’s petition to remand the rule to EPA without vacatur . This ruling means that the regulation remains in effect but EPA must remedy CAIR’s flaws in accordance with the Co urt’s July 11, 2008, op inion in the case. 13 Hydrofluorocarbons, perfluorocarbons, and sulfur hexafluoride.

1209-P-0235



### Conclusions
Acid Rain regulations specify accuracy criterion for EPA Pr otocol Gases and require vendors to participate in an EPA verification program. Ambient monitoring regulations also require the use of EPA Prot ocol Gases and gas vendor participation in a verification program. However, EPA has not implemented a program to provide reasonable assurance th at EPA Protocol Ga ses are accurate. In addition, the Protocol for preparing EP A Protocol Gases has not been revised since 1997 and programmatic responsibiliti es for maintaining the Protocol are unclear.


### Recommendations

We recommend that the EPA Assistant Administrator for Air and Radiation:

2-1 Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous emissions monitoring systems for EPA’s Acid Rain Cap and Trade Program, and other stationary source air programs.

2-2 Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous ambient air monitors for the NAAQS monitoring program.

We recommend that the EPA Assistan t Administrator for Research and Development:

2-3 Update and maintain the EPA Traceability Protocol to meet the defined objectives of the Acid Rain, NAAQS, and other stationary source air programs. 

### Agency Comments and OIG Evaluation
OAR and ORD agreed with the report’s recommendations.

• For Recommendation 2-1, OAR plans to propose a rule for public comment in fall 2009 to establish a largely self -supported, annual gas audit program of protocol gases used for the Acid Rain program. In the interim, OAR has already initiated efforts to conduct an other OAR-funded test of EPA Protocol Gases gas later in 2009.

• For Recommendation 2-2, OAR plans to implement a separate verification program to address the lower concentrat ion protocol gases used to calibrate continuous ambient air monitors. Using analytical support from EPA Regions 2 and 7, OAR plans to verify EP A Protocol Gases supplied to State,

1309-P-0235

local, and tribal agencies for calib ration of continuous, ambient gaseous pollutant monitors. OAR plans to launch this program by March 2010.

• For Recommendation 2-3, ORD will update and publish, as guidance, a revised protocol to reduce the percentage of EPA Protocol Gases that do not attain the Acid Rain Program’s accuracy criterion. ORD plans to complete and issue the revised protocol by Oc tober 1, 2010. ORD also provided a corrective action plan in response to Recommendation 2-3 (see Appendix D).
The general actions outlined in OAR’s and ORD’s responses meet the intent of our recommendations. Recommendations 2-1 and 2-2 will remain open in our tracking system pending our receipt and approval of OAR’s final corrective action plan. ORD’s proposed corrective action plan suffi ciently addresses R ecommendation 2-3. As such, we are closing R ecommendation 2-3 in our tr acking system upon issuance of this report. These recommendations will be tracked to completion in the Agency’s tracking system. With respect to the ambient monitoring program, OAR commented that “the report demonstrates that the gases tested met the expectations of the EPA Traceability Protocol with 95 percent confidence limit. We point out that the gases used for the ambient air program rely mainly on the traceability protocol. Thus, there are no findings demonstrating a concern with using protocol gases for that program.” We disagree with this characte rization. Our tests found that the low-range concentration gases performed the worst in meeting the 2 percent accuracy criterion for the Acid Rain Program. The ambient monitoring program uses even lower concentration gases than the ones we tested during our evaluation. The past two Protocol Gas assessments conducted by EPA have not incl uded these lower concentration gases. We believe the lack of test data to assu re the accuracy of the lower concentration gases used for the ambient monitoring program is a concern that should be addressed by EPA. After receiving OAR’s response to the dr aft report, the OIG met with OAR to discuss its response. OAR requested that we revise Recommendation 2-2 to state that OAR should implement an oversight pr ogram to provide reasonable assurance of the quality of EPA Protocol Gases. We made this change to Recommendation 2-2. OAR and ORD also provided several technica l clarifications and comments to the report. We made changes to the fina l report based on these comments, as appropriate. OAR and ORD’s responses to the draft report are contained in Appendices C and D.

1409-P-0235


## Status of Recommendations and Potential Monetary Benefits

POTENTIAL MONETARY RECOMMENDATIONS BENEFITS (in $000s) 2

Rec. No. Page No. Subject Status 1 Action Official Planned Completion Date Claimed Amount Agreed To Amount

2-1

2-2

2-3 13

13

13 Implement an oversi ght program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous emissions monitoring systems for EPA’s Acid Rain Cap and Trade Program, and other stationary source air programs. Implement an oversight program to provide reasonable assurance of the quality of EPA Protocol Gases used to calibrate continuous ambient air monitors for the NAAQS program. Update and maintain the EPA Traceability Protocol to meet the defined objectives of the Acid Rain, NAAQS, and other stationary source air programs. O

O

O Assistant Administrator for Air and Radiation

Assistant Administrator for Air and Radiation

Assistant Administrator for Research and Development 10/01/2010

1 O = recommendation is open with agreed -to corrective actions pending; C = recommendation is closed with all agreed-to actions completed; U = recommendation is undecided with resolution efforts in progress.
2 Identification of potential monetary benefits was not an objective of this evaluation.

1509-P-0235

Appendix A 

## Details on Scope and Methodology

To test the accuracy of EPA Protocol Gases, we contracted with a thir d party to purchase EPA Protocol Gases directly from the specialty gas pro ducers. Our intent was to purchase at least one set of three EPA Protocol Gas cylinders from each gas producer and manufacturing location in the country. After purchasing at least one cylinder from each identified producer 14 or manufacturing location, we purchas ed additional cylinders from each producer to maximize our sample size and obtain a better representation of the facilities’ production capability. Our contractor purchased 87 EPA Prot ocol Gas tri-blend cylinders from 11 different producers. NIST analyzed these cylinders and compared the re sults with each cylinder’s certified tag values. Chain of custody procedures were employed to account for each sample cylinder throughout theprocess. Both our contractor and NIST conducted their work in accordance with the OIGapproved quality assu rance project plans. Procedures for Purchasing Sample Gases
We tasked our contractor with surveying the in dustry to identify EPA Protocol Gas vendors and manufacturing locations. To accomplish this surve y, the contractor developed a set of keywords to electronically search the In ternet and the Thomas Register ® for potential vendors of EPA Protocol Gases. Our contract or also obtained industry inform ation through conversations with gas vendor personnel during the process of obtaini ng vendor quotes. The survey identified a total of 11 gas producers and 18 manufacturing locations.

We authorized our contractor to purchase gas cylinders from each of the identified vendors with an initial goal of purchasing at least three tri-blend cylinders of a lo w-, medium-, and highconcentration range from each vendor manufactur ing location. After accomplishing this goal, we purchased additional cylinde rs from vendors with a goal of obtaining additional cylinders from a cross section of vendor types taking into account such factors as cylinder cost and potential market share. An initial set of thr ee cylinders was purchased from all 18 manufacturing locations, an additional set of cylinders was purch ased from 9 of the 18 locations, and a third set was purchased from one of the 18 locations. In al l, 87 cylinders (29 sets of 3) were purchased. Our contractor purchased the gases from January 30, 2008, to April 29, 2008. The following table shows the number of cylinders by pollutant and concentration range.
Table A-1: Summary of Sample Gases Purchased

No. of Cylinders Range Type CO 2 (%) NO (ppm) SO 2 (ppm)
29 High 18.0 900 1000 29 Mid 12.0 400 500 29 Low 5.00 50.0 50.0 Source: EPA Protocol Gas Industry Survey and Blind Audit, Final Report, February 2009

14 To ensure cylinders were purchased from every producer, sets of cylinders were purchased from three distributors.

1609-P-0235

NIST Analysis
Under an interagency agreement with the OI G, NIST was tasked with analyzing the concentrations of the sample cylinders and comp aring the test results to the manufacturers’ certified value. NIST’s primary objective in co nducting its analysis was to achieve a calculated uncertainty of ± 0.5 percent or be tter for its analysis to achieve a 4:1 or better ratio between the acceptance criterion (i.e., the Acid Rain Program ’s ± 2 percent accuracy criterion) being tested and the uncertainty of NIST’s analysis. A genera l standard of practice in metrology is for this ratio – referred to as TAR or test to accuracy ratio – to be greate r than or equal to 4:1. NIST’s analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent at the 95 percent confidence level, depending upon the gas analyzed and the concentration.
To be consistent with the reporting of past EP A Protocol Gas tests conducted by EPA, Chapter 2 of this report presents the analytical results without adjustment for the uncertainty of the analyses. Appendix B presents NI ST’s results with uncertainty.

Limitations
We instructed all parties involved in this evaluation not to disc uss our plans with specialty gas vendors or persons not needing such information to implement the evaluation. However, several months prior to purchasing the gas cylinders, we were informed that at least one specialty gas producer had become aware our intended evaluation. We do not know whether additional vendors also became aware of our planned evaluati on, or whether they would have been able to identify the orders intended for the tests. If a specialty gas vendor was aware of our evaluation, the results may not necessarily represent the qu ality of gases such a vendor routinely produces and sells to users, but only indicates its ab ility to produce quality EPA Protocol Gases.

Because of a lack of data on th e specialty gas vendor market share, we were unable to design a sample that would allow us to project our result s to the entire universe of EPA Protocol Gases with an acceptable level of c onfidence and precision. Since we were unable to select gas cylinders for testing in a manner that would ensu re a statistically representative sample of the EPA Protocol Gas market, we did not perform any inferential statistical analysis to project the sample test results to the overa ll EPA Protocol Gas market.

1709-P-0235

Appendix B 

## Test Results With Uncertainty Factored into Analyses

NIST calculated the total uncertainty of its analysis at the 95 percent confidence level for each of the component ranges it tested. NIST’s calcu lated analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent. To assess the results taking into account the uncertainty of NIST’s analysis, we adjusted the test limits to account for uncertainty using the following formula:

Test limit = Specification (Acid Rain Program ± 2 percent cr iterion) – ± Uncertainty (NIST’s analytical uncertainty) The following table shows how the test limits were adjusted using a NIST analytical uncertainty of ± 0.42 percent as an example.
Table B-1: Example of Test Limits Adjusted for Uncertainty Test Limit (Expressed as

Percentage Difference) Conclusion if Sample Falls in This Range < - 2.42 Failed with 95 percent confidence

-2.42 to -1.58 Within the range of uncertainty

-1.57 to 1.57 Passed with 95 percent confidence

1.58 to 2.42 Within the range of uncertainty

> 2.42 Failed with 95 percent confidence Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. When the test results are adjusted for uncer tainty, the number of components meeting or exceeding the Acid Rain Program’s accuracy cr iterion is smaller than the unadjusted test results presented in Chapter 2. For example, when we factor in uncertainty, we can only say with 95 percent confidence that 82 percent (214 of 261) of the components met the criterion, whereas the unadjusted test result s show that 89 percent of the components met the accuracy criterion. Similarly, when we fact or in uncertainty, we can only state with 95 percent confidence that 5 per cent (12 of 261) of the compone nts did not meet criterion. The remaining 13 percent (35 of 261) of the co mponents were within the range of NIST’s analytical uncertainty. Table B- 2 on the next page summarizes the test results adjusted for uncertainty.

1809-P-0235

Table B-2: Summary of Test Results Test Results Category No. Percentage Met the Acid Rain Accuracy Criterion 214 82 Did Not Meet the Acid Rain Accuracy Criterion 12 5 Test Results Within the Range of Analytical Uncertainty 35 13
Total 261 100 Source: Developed from data in NIST Report of Analysis, December 4, 2008. Both the unadjusted and adjusted test results showed that the lower-cost cylinders in our sample performed worse than high er-cost cylinders. The adjust ed results show that 11 of the 12 gaseous mixtures not meeting the Acid Rain Program’s accuracy criterion were lower-cost cylinders (< $380) certified with GMIS. The fo llowing table summarizes the tests meeting the accuracy criterion by cost a nd reference standard used to certify the concentration.
Table B-3: Results by Vendor’s Analytical Reference Standard and Cost

Reference Standard Used and Cost of Gas No. of Components Certified with this Standard Components Meeting the Acid Rain Accuracy Criterion

No. Percentage NTRM/SRM < $380 8 6 75 NTRM/SRM > $380 103 100 97 GMIS < $380 96 59 61 GMIS > $380 50 47 94 Unknown a < $380 4 2 50 Totals 261 214 82 b Source: Developed from data in NIST Report of Analysis, December 4, 2008

a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration. b Represents the overall percentag e, not the column total.

1909-P-0235

Appendix C 

## OAR’s Response to OIG Draft Report

August 5, 2009

MEMORANDUM

SUBJECT: Comments on Draft IG Report: “EPA N eeds an Oversight Program for Protocol Gases” Assignment number 2007-000877

FROM: Gina McCarthy /s/ Assistant Administrator for Air and Radiation

TO: Wade T. Najjum Assistant Inspector General for Program Evaluation

The Office of Air and Radiation (OAR) appr eciates the Office of Inspector General’s (OIG) effort to work with our staff on this uni que testing program to produce the subject Draft Final Report “EPA Needs an Oversight Program for Protocol Gases” to improve the quality assurance of protocol gases. We generally agree with the data described in the draft final report. However, the issues raised apply differently to the Acid Rain Program’s (ARP’s) source emissions monitoring systems and National Ambient Air Quality measurement network systems. Therefore, this memorandum is structured in tw o parts – an Acid Rain Program section and an Ambient Air Program section.
Acid Rain Program
Our major comments are presented below. We include both major and detailed comments in the attached Word document in Track Changes. After you’ve reviewed our comments, we’d like to have another meeting with you as soon as possible.

• At the end of the first paragraph in “At a Gl ance”, please add: “However, 96 percent were within 3 percent of the true value; and there was no bias associated with the 28 gas components that did not meet the ARP performance specification. Based on the analysis results, there are no anticipated nationwide e nvironmental consequen ces of the calibration gases used in the ARP. However, the ARP is addressing potential sour ce-level problems in two ways: (1) by performing an EPA-funded gas audit in 2009; and (2 ) by proposing a rule establishing a largely self-supported, ongoing, annual gas audit program.”

• Add the following paragraph to the “At a Glance” section:
“Due to the nature of our samp le selection and a lack of da ta on vendor market shares, we were unable to develop a sampling methodology to ensure that the num ber of samples selected from each producer represented the number of cylinders pr oduced and sold to cons umers. Consequently,

2009-P-0235

we could not project our test results to the universe of EPA Protocol Gases, or estimate the potential impact that gases not meeting the Acid Rain Pr ogram’s accuracy criteria could have on that program. Despite this limitation, our tests represen t the largest one-time assessment of the Protocol Gas industry. The final results of this study identi fied some potential areas of concern with respect to the quality of less expensive gases certified using GMIS.”

• Add the following paragraph to the “N oteworthy Achievements” section:
“The ARP is currently conducting a third ga s audit, covering all known producers of EPA Protocol gas. As in past ARP audits, the results w ill be released to the public. We expect results by early 2010. To provide a cost-effective, long te rm audit capability, OAR is also developing a proposed rule package to implem ent a largely self-supporting, a nnual gas audit program. This proposed rule is expected to be published for public comment in the Federal Register in the fall of 2009.”

ARP concurs with all of the OIG recomme ndations. However, we suggest a slight rewording for recommendation #2-1 as follows:
2-1 Implement a better oversight program to assure the qua lity of EPA Protocol Gases used to ensure the accuracy of continuous emissi ons monitoring systems for EPA’s Acid Rain Program, and other stationary source air programs.
Ambient Air Programs
[ OIG NOTE : The comments initially submitted by the Ambient Air Program on August 5, 2009 were superseded by the amended comments provided September 2, 2009, in the following memorandum.]

Thank you again for the opportunity to commen t on the draft final re port. If you have questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up Coordinator, at 202-564-1228.

2109-P-0235

September 2, 2009 15

MEMORANDUM

SUBJECT: Amended Comments on Draft IG Report: “EPA Needs an Oversight Program for Protocol Gases” Assi gnment Number 2007-000877

FROM: Gina McCarthy /s/ Assistant Administrator for Air and Radiation

TO: Wade T. Najjum Assistant Inspector General for Program Evaluation
The Office of Air and Radiation (OAR) appr eciates the Office of In spector General’s (OIG) effort to work with our staff on this unique tes ting program to verify th e quality assurance of protocol gases as published in the subject Draf t Final Report “EPA Need s an Oversight Program for Protocol Gases”. In a follow-up meeting between our staffs, your office explained that recommendation 2-2 was directed toward the program under development and outlined by the Office of Air Quality Planning and Standards (O AQPS) during the audit. Consequently, we agree with recommendation 2-2 and provide the following amended paragraph titled “Ambient Air Program”, along with new recommendations to amend our August 5, 2009 response concerning the ambient air program’s protocol gases.
Ambient Air Program
The report demonstrates that the gases testedmet the expectations of the EPA Traceability Protocol with 95 percent confidence limit. We poi nt out that the gases used for the ambient air program rely mainly on the traceability protocol . Thus, there are no findings demonstrating a concern with using protocol gases for that program . Determination of comp liance with an ambient air standard is completed with c onsideration of many factors includi ng the quality of the data. We believe updating the traceability prot ocol is the most efficient a nd effective corrective action to improve the quality of gases and subsequently th e data collected for the ambient air program. Revising the protocol will also benefit stationary source emissions measurements. We, therefore, agree with recommendation 2-2 and have already started creation of an appropriate oversight program to reasonably assure the quality of protocol gases for th e ambient air monitoring program. We amend our August 5, 2009 response:

1. Replace the sentence in the “At A Glance” sec tion that says, “However, as of April 2009, EPA had not implemented a verification program , primarily due to indecision over how to fund the program.” with “However, as of April 2009, OAQPS, with commitments by EPA Regions 2 and 7 for analytical support, had begun development of a program to provide independent verification of EP A Protocol Gas cylinders used for continuous ambient air monitors.”

15 The OAR memorandum was undated but was received by OIG on September 2, 2009.

2209-P-0235

2. Add the following paragraph to the “Notew orthy Achievements” section: “OAQPS has planned and begun development of the Protocol Gas Verification Program to independently verify “EPA Protocol Gases” supplied to state, local and Tribal agenci es for calibration of continuous, ambient gaseous pollutant monitors. EPA Regions 2 and 7 have committed to providing analytical support. The program is expected to be launched in March 2010.
Finally, we suggest the following rewording for r ecommendation 2-2: “Implement an oversight program, which reasonably assures the quality of EP A Protocol Gases used to calibrate continuous ambient air monitors for the National Ambient Air Quality Standards monitoring program as specified in 40 CFR part 58 Appendix A.”
Thank you again for the opportunity to commen t on the draft final re port. If you have questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up Coordinator, at 202-564-1228.

2309-P-0235

Appendix D 

## ORD’s Response to OIG Draft Report

July 30, 2009

MEMORANDUM

SUBJECT: ORD Response to OIG’s Draft Evaluati on Report: EPA Needs an Oversight Program for Protocol Ga ses (Assignment No. 2007-00877)

FROM: Lek G. Kadeli /s/ Acting Assistant Administrator (8101R)

TO: Wade T. Najjum Assistant Inspector Genera l for Program Evaluation (2460T)

Thank you for the opportunity to provide comments on the draft Office of the Inspector General (OIG) report entitled “EPA Needs an Oversight Program for Protocol Gases.”
Attached are the Office of Research and Development’s (ORD) specific comments on the draft report. In general, ORD concurs with the recommendation that ORD should update and maintain the EPA Traceability Protocol for Gaseous Calibration Standards . We appreciate your consideration of our comments. If there are any questions please do not hesitate to call me at 202­ 564-6620 or Jorge Rangel at 202-564-1606.

Attachment

2409-P-0235 

## Office of Research and Development Comments on Office of Inspector General Draft Audit Report, "EPA Needs an Oversight Program for Protocol Gases", Assignment No. 2007- 000877, July 8, 2009

This document is comprised of four sections:

1. General comments regarding the OIG recommendations 2. Table of ORD Response to OIG recommendations 3. Proposed detailed schedule of corrective actions 4. Specific comments by page number 

## 1. General Comments regardin g the OIG Recommendations
Thank you for the opportunity to comment on the draft Office of the Inspector General (OIG) report entitled “EPA Needs an Oversight Pr ogram for Protocol Gases.” EPA’s Office of Research and Development (ORD) concurs with the recommendation that ORD should update and maintain the EPA Traceability Protocol for Gaseous Calibration Standards . We believe that the scientific expertise within ORD puts it in a unique position to understand the gas metrological issues that need to be addressed in the update and to prepare the update. 

## 2. Table of ORD Response to OIG Recommendations

Rec. No. OIG Recommendation Action Official ORD Corrective

Actions Planned Completion Date 2-3 Update and maintain the EPA Traceability Protocol to meet the defined objectives of the Acid Rain, NAAQS, and other stationary source air programs. Assistant Administrator for Research and Development Update and publish (as guidance) a revised protocol to reduce the percentage of EPA Protocol Gases that do not attain the CAMD accuracy criteria 10/01/2010



## 3. Proposed Detailed Schedul e of Corrective Actions

Corrective Action Taken Planned Completion Date Final audit report disseminate d to public October 1, 2009 List of potential protoc ol changes* sent to NIST and specialty gas producers for October 15, 2009

2509-P-0235

Corrective Action Taken Planned Completion Date review with request for their comments, suggested revised text, and any concentration stability data supporting changes to allowable gas composition, concentration ranges, minimum cylinder pressures, and certification periods Receive all comments and stab ility data November 15, 2009 Complete internal review dr aft of protocol May 15, 2010 Receive EPA comments on internal draft June 1, 2010 Complete external revi ew draft of protocol by NIST and specialty gas producers July 1, 2010

Receive external comments on revisions August 15, 2010 Revised protocol published as guidance October 1, 2010 * In 2005, EPA sent the following list of informal change suggestions to NIST and specialty gas producers:

- include a procedure specifically for the use of Fourier Transform infrared (FTIR) spectroscopy;

- improve statistical procedures for uncer tainty calculations and revise the Excel spreadsheet;

- include mercury calibration standards in the protocol;

- eliminate (or retain) gas manufacturers intermediate standards (GMISs);

- lengthen the certification periods for EPA Protocol Gases;

- change the cylinder pressure limitations for EPA Protocol Gases;

- include a procedure for analyzing and certifying zero gases;

- include gas dilution systems in the protocol;

- tighten documentation requirements;

- develop audit participant identification number for regulatory data reporting purposes;

- require ISO 17025 accreditation for producers of environmental calibration gases;

- include provision for preparation and analysis of batches of EPA Protocol Gases;

- require on-site visits of spec ialty gas producers by EPA representatives; and

- provide for technical assistance/outreach to specialty gas producers by NIST representatives; 

## 4. Specific comments by page number

[OIG NOTE : ORD also provided several technical clarifications and comments to the draft report. We made changes to the final repor t based on these comments, as appropriate.]

2609-P-0235

Appendix E 

## Distribution

Office of the Administrator Assistant Administrator for Air and Radiation Acting Assistant Administrator for Research and Development Deputy Assistant Administrator for Air and Radiation Deputy Assistant Administrator for Research and Development Director, Office of Air Quality Planning and Standards, Office of Air and Radiation Director, Office of Atmospheric Prog rams, Office of Air and Radiation Director, National Risk Manageme nt Research Laboratory, Office of Research and Development Agency Follow-up Official (the CFO) Agency Follow-up Coordinator General Counsel Associate Administrator for Congressi onal and Intergovernmental Relations Associate Administrator for Public Affairs Audit Follow-up Coordinator, Office of Air and Radiation Audit Follow-up Coordinator, Offi ce of Research and Development Acting Inspector General

27