U.S. ENVIRONMENTAL PROTECTION AGENCY 

OFFICE OF INSPECTOR GENERAL 

Catalyst for Improving the Environment 

Evaluation Report 

EPA Needs an Oversight Program 

for Protocol Gases 

Report No. 09-P-0235 

September 16, 2009
 
Report Contributors:	 Jim Hatfield 

Geoff Pierce 

Rick Beusse 

Abbreviations 

CAA Clean Air Act 

CAIR Clean Air Interstate Rule 

CEMS Continuous Emissions Monitoring System 

CFR Code of Federal Regulations 

CO Carbon monoxide 

CO2 Carbon dioxide 

EPA U.S. Environmental Protection Agency 

GMIS Gas Manufacturer’s Intermediate Standard 

NAAQS National Ambient Air Quality Standards 

NIST National Institute of Standards and Technology 

NO Nitric oxide 

NOx Nitrogen oxides 

NO2 Nitrogen dioxide 

NTRM NIST Traceable Reference Material 

OAP Office of Atmospheric Programs 

OAQPS Office of Air Quality Planning and Standards 

OAR Office of Air and Radiation 

OIG Office of Inspector General 

ORD Office of Research and Development 

PGVP Protocol Gas Verification Program 

ppm parts per million 

SO2 Sulfur dioxide 

SRM Standard Reference Material 

Cover photo:	 Gas cylinders stored at the National Institute of Standards and Technology 

laboratory in Gaithersburg, Maryland. (Photo courtesy National Institute of 

Standards and Technology) 
U.S. Environmental Protection Agency 09-P-0235 

September 16, 2009 Office of Inspector General 

At a Glance 

Catalyst for Improving the Environment 

Why We Did This Review 

The accuracy of continuous 

emissions monitors is critical 

to the U.S. Environmental 

Protection Agency’s (EPA’s) 

Acid Rain Program because 

data from these monitors 

determine the number of 

allowances a utility can bank, 

sell, or trade. Specialized

gases known as “EPA

Protocol Gases” are used to 

calibrate and assure the quality

of these monitors. We sought

to determine whether certified 

concentrations of these gases 

are accurate. 

Background 

Vendors produce and certify 

EPA Protocol Gases in highpressure cylinders according 

to EPA procedures. EPA 

regulations require the use of

these gases, or National 

Institute of Standards and 

Technology (NIST)-certified 

reference materials, when 

conducting quality assurance 

for continuous emissions and 

ambient air monitoring 

systems. EPA’s Acid Rain 

Program requires that the 

certified concentration of the 

gases be within ± 2 percent of 

the true concentration. 

For further information, contact 

our Office of Congressional, 

Public Affairs and Management 

at (202) 566-2391. 

To view the full report, 

click on the following link: 

www.epa.gov/oig/reports/2009/ 

20090916-09-P-0235.pdf 

EPA Needs an Oversight Program for Protocol Gases 

What We Found 

We purchased 87 cylinders of EPA Protocol Gases and had NIST analyze each 

cylinder to determine whether the three gaseous mixtures contained in each 

cylinder (carbon dioxide, nitric oxide, and sulfur dioxide) met the Acid Rain 

Program’s accuracy criterion. We found that 89 percent (233 components) met 

the Acid Rain Program’s accuracy criterion and 11 percent (28 components) did 

not. Of the 28 components that did not meet the criterion, 17 were within 3.0 

percent of the NIST-determined true concentration; 7 were within 3.0 to 5.0 

percent; and 4 exceeded the true concentration by more than 5.0 percent. 

Our sample was not designed to estimate the impact of the test results on the Acid 

Rain Program. However inaccurately certified concentrations could cause system 

operators to unknowingly calibrate their monitoring systems to record inaccurate 

measurements. For example, if a utility overstates its emissions, it could lose the 

opportunity to sell allowances to other utilities. If a utility understates its 

emissions, the utility and regulators may incorrectly conclude that the source is 

complying with emissions standards. With respect to ambient air monitoring, the 

accuracy of these monitors is important because the data are used to determine 

whether areas are in compliance with the Nation’s ambient air quality standards. 

EPA has conducted only two tests of the accuracy of EPA Protocol Gases since 

1997, when EPA’s Office of Research and Development discontinued its annual 

testing program. Thus, EPA does not have reasonable assurance that the gases 

used to calibrate emissions monitors for the Acid Rain Program and continuous 

ambient monitors for the Nation’s air monitoring network are accurate. 

What We Recommend 

We recommend that the Office of Air and Radiation (OAR) implement oversight 

programs to assure the quality of EPA Protocol Gases used to calibrate continuous 

emissions monitoring systems and ambient air monitors. We also recommend that 

the Office of Research and Development (ORD) update and maintain the protocol 

gas procedures to ensure that the protocol meets the objectives of the Acid Rain, 

ambient air, and stationary source air programs. OAR and ORD concurred with 

our recommendations. OAR has initiated efforts to conduct another gas audit later 

this year. OAR also plans to propose a rule later in 2009 to establish a largely 

self-supported, annual gas audit program of protocol gases used for the Acid Rain 

Program. Further, OAR plans to implement a separate verification program to 

address the lower concentration protocol gases used to calibrate continuous 

ambient air monitors. ORD will update and maintain the protocol gas procedures. 

EPA’s planned actions meet the intent of our recommendations. 
UNITED STATES ENVIRONMENTAL PROTECTION AGENCY 

WASHINGTON, D.C. 20460 

OFFICE OF 

INSPECTOR GENERAL 

September 16, 2009 

MEMORANDUM 

SUBJECT:	 EPA Needs an Oversight Program for Protocol Gases 

Report No. 09-P-0235 

Assistant Inspector General for Program Evaluation 

FROM:	 Wade T. Najjum 

TO: Gina McCarthy 

Assistant Administrator for Air and Radiation 

Lek G. Kadeli 

 Acting Assistant Administrator for Research and Development 

This is our report on the subject evaluation conducted by the Office of Inspector General (OIG) 

of the U.S. Environmental Protection Agency (EPA). This report contains findings that describe 

the problems the OIG has identified and corrective actions the OIG recommends. This report 

represents the opinion of the OIG and does not necessarily represent the final EPA position. 

Final determinations on matters in this report will be made by EPA managers in accordance with 

established audit resolution procedures. 

The estimated cost of this report – calculated by multiplying the project’s staff days by the 

applicable daily full cost billing rates in effect at the time, including the costs of purchasing EPA 

Protocol Gases and having them analyzed by the National Institute of Standards and Technology 

– is $665,846. 

Action Required 

In accordance with EPA Manual 2750, EPA’s Audit Management Process, the Office of Air and 

Radiation should provide a written response within 90 calendar days. The Office of Air and 

Radiation’s response should include a corrective action plan and planned completion dates for 

Recommendations 2-1 and 2-2. The Office of Research and Development submitted a corrective 

action plan that sufficiently addresses Recommendation 2-3. As such, we are “closing” 

Recommendation 2-3 in our tracking system upon issuance of this report. These 

recommendations will be tracked to completion in the Agency’s tracking system. No further 
response is required for Recommendation 2-3. As outlined in EPA Manual 2750, the Agency is 

responsible for tracking the implementation of these actions in its Management Audit Tracking 

System. We have no objections to the further release of this report to the public. This report will 

be available at http://www.epa.gov/oig. 

If you or your staff have any questions regarding this report, please contact me at (202) 566-0832 

or najjum.wade@epa.gov, or Rick Beusse at (919) 541-5747 or beusse.rick@epa.gov. 
 

 

 

EPA Needs an Oversight Program for Protocol Gases 09-P-0235 

Table of Contents 

Chapters 

1 Introduction ........................................................................................................... 1 

Purpose .......................................................................................................... 1 

Background .................................................................................................... 1 

Noteworthy Achievements.............................................................................. 5 

Scope and Methodology................................................................................. 5 

2 EPA Needs an Oversight Program to Assure the Accuracy of 

 Protocol Gases...................................................................................................... 7 

EPA Does Not Have an Oversight Program to Assure the 

 Accuracy of Protocol Gases ................................................................... 7
 

Low Concentration NO and SO2 Gases and Lower-Cost Cylinders 

Calibration Gases Play an Important Role in Assuring Data Quality 

Performed Worst in Our Tests............................................................... 8 

for EPA Air Programs ............................................................................ 10 

Conclusions.................................................................................................... 13 

Recommendations ......................................................................................... 13 

 Agency Comments and OIG Evaluation ......................................................... 13 

Status of Recommendations and Potential Monetary Benefits................................. 15 

Appendices 

A Details on Scope and Methodology.................................................................... 16
 

B Test Results with Uncertainty Factored into Analyses .................................... 18
 

C OAR’s Response to OIG Draft Report ............................................................... 20 

D ORD’s Response to OIG Draft Report ................................................................ 24
 

E Distribution............................................................................................................ 27 
09-P-0235 

Chapter 1

Introduction 

Purpose 

The U.S. Environmental Protection Agency (EPA) Protocol Gases are used to 

calibrate and test continuous emissions monitoring systems (CEMS) and ambient 

air quality monitoring systems. EPA requires the use of CEMS for large utilities 

covered by the Acid Rain Program and other clean air programs. Consequently, 

the accuracy of the EPA Protocol Gases is critical to ensuring the integrity of 

emissions trading and other EPA programs. We conducted this evaluation to 

determine whether the certified concentrations of EPA Protocol Gases were 

within acceptable limits for accuracy. 

Background 

The integrity and effectiveness of EPA’s air programs depend upon collecting 

accurate pollutant emissions and air quality data. EPA requires that the systems 

collecting these data periodically undergo certain quality assurance procedures to 

assure their accuracy. These procedures include daily calibration and accuracy 

tests of the monitoring systems. Calibration tests check the accuracy of these 

monitoring systems by running a gas mixture of a known concentration through 

the systems and comparing the systems’ readings with the known concentration. 

The systems are adjusted accordingly to eliminate measurement inaccuracies. 

The certified concentrations of gaseous mixtures used to calibrate and test 

monitoring systems should accurately reflect the true gaseous concentration. EPA 

regulations require that the gases used for calibrating and testing the accuracy of 

ambient air quality analyzers and continuous emissions monitors1

 be traceable to 

either a National Institute of Standards and Technology (NIST) Standard 

Reference Material (SRM) or a NIST Traceable Reference Material (NTRM). 

EPA has established an EPA Traceability Protocol that specialty gas producers 

must follow when preparing calibration gases to meet the NIST traceability 

requirement. The gases produced in accordance with these standards are referred 

to as EPA Protocol Gases. 

This protocol was designed by EPA, NIST, and specialty gas producers so that a 

laboratory analyst can analyze and certify compressed gas calibration standards 

with low uncertainty and with traceability to NIST. The protocol specifies a 

general analytical procedure for determining the concentration of the standards, 

but it does not specify the analytical instrumentation or the gas-handling 

1

 Continuous emissions monitors continuously measure pollutants emitted into the atmosphere in exhaust gases from 

combustion or industrial processes. 

1 
09-P-0235 

apparatus that the analyst must use during the analysis. In summary, the protocol 

requires that (1) a NIST-certified gaseous mixture be used as the analytical 

reference standard, (2) a monthly calibration curve for the instrumentation be 

established, and (3) both the NIST-certified gaseous mixture and the candidate 

EPA Protocol Gas be measured at least three times, with reactive gas mixtures 

measured at least three additional times at least 7 days after the first set of 

measurements. 

The protocol has detailed statistical procedures for estimating the total uncertainty 

of these standards. Although it includes procedures for estimating uncertainty, 

the protocol does not specify that EPA Protocol Gases meet any given level of 

uncertainty and does not establish specific accuracy criteria for the standards’ 

uncertainty. However, the Acid Rain Program has prescribed an uncertainty 

standard for EPA Protocol Gases used to calibrate CEMS as described in the 

following section. 

Acid Rain Program Requirements for EPA Protocol Gases 

The Acid Rain Program implements Title IV of the Clean Air Act (CAA) to 

reduce emissions of sulfur dioxide (SO2) and nitrogen oxides (NOx), the primary 

causes of acid rain. The program uses an allowance trading system to reduce SO2 

emissions. Under this system, affected utility units are allocated allowances (one 

allowance = one ton of SO2) that permit a unit to emit SO2 during or after a 

specified year. Utilities can buy, sell, or bank allowances. The program began in 

1995. For each ton of SO2 emitted in a given year, one allowance is retired and 

can no longer be used. The program is phased in, with the final SO2 cap set at 

8.95 million tons in 2010. The Acid Rain Program does not use a trading system2 

to reduce emissions for NOx, but instead establishes emissions limits for covered 

units that are designed to achieve the program’s goal of reducing NOx emissions 

by 2 million tons from 1980 levels. 

Continuous emissions monitoring is instrumental in ensuring that the mandated 

reductions of SO2 and NOx under the Acid Rain Program are achieved. 

Accordingly, EPA has established requirements for continuously monitoring SO2 

and NOx (and other parameters) for units regulated under the Acid Rain Program. 

Appendices A and B to 40 Code of Federal Regulations (CFR) Part 75 describe 

specification and test procedures for these monitors, which include daily 

calibration tests and periodic accuracy tests. The regulations require that the 

calibration gases used to conduct these tests must be NIST-certified reference 

standards or that their concentrations must be traceable to NIST-certified 

reference standards according to the EPA Traceability Protocol for Assay and 

Certification of Gaseous Calibration Standards. In January 1993, EPA’s Office 

of Atmospheric Programs (OAP) included a 2.0 percent accuracy criterion in its 

final Acid Rain Program emissions monitoring rule (58 Federal Register 3731). 

2

 NOx emissions are traded under other air programs, including the NOx Budget Trading Program, which is designed 

to reduce NOx emissions in order to reduce ambient ozone levels. 

2 
09-P-0235 

Appendix A to 40 CFR Part 753 specifies that the EPA Protocol Gases must have 

a certified uncertainty (95 percent confidence interval) that must not be greater 

than plus or minus (±) 2.0 percent of the certified concentration (tag value) of the 

gas mixture. 

In January 2008, EPA revised the Acid Rain Cap and Trade Program regulations 

to require that producers of EPA Protocol Gases participate in EPA’s Protocol 

Gas Verification Program in order to certify their gases as EPA Protocol Gases. 

National Ambient Monitoring Program Requirements for EPA 

Protocol Gases 

Appendix A to 40 CFR Part 584 describes quality assurance procedures for 

national ambient air monitors. States and EPA use these monitors to determine 

whether an area’s air quality meets or exceeds the National Ambient Air Quality 

Standards (NAAQS). The quality control procedures include biweekly one-point 

checks of carbon monoxide (CO), SO2, and nitrogen dioxide (NO2), and annual 

performance evaluations that test the monitors against gases of known 

concentrations for at least three different ranges of concentrations. Appendix A to 

this regulation requires that cylinders of compressed gas used to obtain test 

concentrations must be traceable to either a NIST Traceable Reference Material 

or a Gas Manufacturer’s Intermediate Standard, certified in accordance with 

EPA’s Traceability Protocol for Assay and Certification of Gaseous Calibration 

Standards. Further, vendors advertising certification with the procedures and 

distributing gases as “EPA Protocol Gas” must participate in the EPA Protocol 

Gas Verification Program or not use “EPA” in any form of advertising. 

Unlike the Acid Rain Program, the ambient monitoring regulations do not specify 

accuracy criteria for the EPA Protocol Gases used to calibrate ambient air 

monitors. 

Stationary Source Performance Standards Requirements for EPA 

Protocol Gases 

The CAA required EPA to establish emissions standards5

 for certain stationary 

sources that cause or contribute significantly to air pollution. These standards are 

intended to promote use of the best air pollution control technologies and apply to 

sources that have been constructed or modified since the proposal of the standard. 

These standards are published in 40 CFR Part 60, and may require the source to 

install CEMS to demonstrate compliance. 

3 Appendix A to Part 75—Specifications and Test Procedures.
 4

 Appendix A to Part 58—Quality Assurance Requirements for State and Local Air Monitoring Stations, Special 

Purpose Monitors, and Prevention of Significant Deterioration Air Monitoring. 

5

 Referred to as New Source Performance Standards. 

3 
09-P-0235 

Part 60 contains varied requirements for using calibration gases depending upon 

the EPA method used to conduct emissions testing. Specifically, the methods for 

determining SO2 (Method 6C) and NOx (Method 7E) emissions from stationary 

sources require that the calibration gases be prepared according to the EPA 

protocol or certified by the tester using a specified wet chemical test method. The 

electric utility, petroleum, and municipal waste incineration industries are the 

largest industries (in terms of emissions and the number of sources) subject to 

continuous emissions monitoring requirements and the use of EPA Protocol Gases 

for Part 60 emissions standards. 

EPA’s Protocol Gas Verification Program 

From 1985 to 1997, EPA’s Office of Research and Development (ORD) 

conducted a series of analytical tests6 of EPA Protocol Gases sold by specialty gas 

producers. In 1996, ORD issued a final draft exit strategy that transferred the 

administration of this and other quality assurance programs to the Office of Air 

Quality Planning and Standards (OAQPS) of the Office of Air and Radiation 

(OAR). ORD provided OAR with resource support during the transition, but 

provided no further resources after the transfer was completed in 1998. The 

annual tests were discontinued after 1997, and no additional tests occurred until 

2003. Due to a lack of analytical tests and a concern that EPA Protocol Gas 

quality may have declined, the OAP conducted a test in 2003. OAP’s contractor 

purchased 42 EPA Protocol Gas cylinders containing tri-blend mixtures of carbon 

dioxide (CO2). nitric oxide (NO), and SO2 from a total of 14 specialty gas vendors 

nationwide. A third party purchased the cylinders for EPA so that the gas vendors 

would not know that EPA was analyzing the cylinders. A gas passed the test if 

the “true” concentration of the gas differed from the certified concentration by ± 2 

percent or less. Overall, 89 percent of the gases passed on a gas component basis; 

11 percent did not pass the test. 

OAP conducted another test in 2006 but decided not to publish the results because 

of concerns about the representativeness of some of the sample gases. Instead of 

purchasing gases directly from vendors, OAP arranged for end users (i.e., 

utilities) to provide EPA with the Protocol Gases for testing. However, EPA 

encountered several problems with this approach, including over- and 

under-representation of manufacturers and production sites, as well as old and 

used samples. 

Approximately 6 years ago, EPA started developing an ongoing EPA Protocol 

Gas Verification Program (PGVP). After a development process involving U.S. 

specialty gas companies, NIST, the Institute of Clean Air Companies, and EPA, 

tentative agreement was reached on a specialty gas company-funded, ongoing 

draft PGVP. EPA promulgated a requirement for producers to participate in this 

6

 EPA calls these tests EPA Protocol Audits. We refer to these audits as tests in this report to avoid confusion with 

the term “audit” as defined by the Government Accountability Office’s Government Auditing Standards, and 

applicable to work conducted by the Office of Inspector General (OIG). 

4 
09-P-0235 

program in a January 24, 2008, final rule. However, in March 2008, one specialty 

gas vendor petitioned EPA for reconsideration of the final rule. The vendor cited 

several concerns including the nature of the funding for the PGVP. Since that 

time, EPA has been working to resolve these concerns, but still had not funded the 

program as of May 2009. EPA had planned to have EPA Protocol Gas vendors 

fund the verification program through payments of fees to a third party. 

However, EPA has determined such a process could violate the Miscellaneous 

Receipts Act.7 

Noteworthy Achievements 

Recognizing the importance of EPA Protocol Gases, the OAP conducted two 

protocol gas assessments, one in 2003 and another in 2006. Although the sample 

size in each of these assessments was not statistically significant and market share 

information was not available, both assessments indicated that some specialty gas 

vendors were not meeting the EPA Protocol Gas accuracy criterion of ± 2.0 

percent. OAP posted the results, including the names of the specialty gas 

companies, on an Agency Website. Based on comments by specialty gas 

companies, the 2003 accuracy check resulted in some vendors making 

improvements in personnel, training, quality, and equipment. When we originally 

approached EPA with our proposal to assess the accuracy of EPA Protocol Gases, 

OAR supported our plans and cooperated fully with our evaluation team to 

develop a useful assessment of the EPA Protocol Gas program. 

Scope and Methodology 

We conducted this evaluation to test the accuracy of EPA Protocol Gases used to 

calibrate continuous emissions monitoring equipment. We designed our sampling 

methodology to test at least one set of three EPA Protocol Gas cylinders from 

every specialty gas manufacturing location in the country and to maintain the 

confidentiality of the tests so that the gases we purchased would represent those 

sold to utilities and other customers.8 In order to maintain the confidentiality of 

the tests, we contracted with a third party to purchase the gases and deliver them 

to NIST for analysis. 

Our contractor purchased tri-blend mixtures of NO, CO2, and SO2, since tri-blend 

gases are often used by utilities. We identified 11 gas producers nationwide using 

18 manufacturing locations to produce EPA Protocol Gases. Our contractor 

purchased at least one set of EPA Protocol Gas cylinders produced from each of 

the 18 manufacturing locations. In all, we purchased 87 EPA Protocol Gas 

cylinders from 14 specialty gas vendors (i.e., 11 producers and 3 distributors), and 

7

 The Miscellaneous Receipts Act was passed in order to ensure that government agencies did not bypass the 

appropriations authority of Congress by augmenting their budgets via other means, such as user fees, fees for 

training courses, parking fees, contract and lease fees and revenues, monetary awards in court cases involving the 

agencies, court costs and fees, or civil penalties. 

8

 A set consisted of three cylinders in low-, medium-, and high-range concentrations. 

5 
09-P-0235 

18 manufacturing locations. Our contractor delivered the sample gases to NIST 

for analysis. We selected NIST because of its extensive experience in gas 

metrology and its reputation for analytical accuracy. NIST analyzed the gaseous 

concentrations of the 87 tri-blend cylinders (i.e., 261 separate component 

analyses) to determine whether the “true” concentrations were within ± 2 percent 

of the concentration certified by the vendor.9 

We reviewed documentation related to EPA’s past assessments of EPA Protocol 

Gases, including past results, program funding and implementation, and current 

plans for implementing a revised EPA Protocol Gas Verification Program. We 

reviewed federal regulations to identify requirements for the use of EPA Protocol 

Gases. We interviewed staff and managers from ORD, OAQPS, and OAP. 

We conducted this evaluation in accordance with generally accepted government 

auditing standards. Those standards require that we obtain sufficient, appropriate 

evidence to provide a reasonable basis for our findings and conclusions based on 

our evaluation objectives. We believe that the evidence obtained provides a 

reasonable basis for our findings and conclusions based on our objectives. We 

conducted our work from June 2007 to July 2009. 

Appendix A provides more detailed information on our scope and methodology. 

9

 Vendors typically certify gaseous concentrations using either a NIST Traceable Reference Material (NTRM), a 

Standard Reference Material (SRM), or a Gas Manufacturer’s Intermediate Standard (GMIS). NTRMs are 

reference material produced by a commercial supplier with a well-defined traceability linkage to NIST. These 

reference materials are allowed to bear the NIST NTRM certification mark and were developed to allow NIST to 

respond to increasing needs for high-quality reference materials. SRMs are reference materials produced by NIST 

and are certified to have specific chemical or physical properties. A GMIS is a NIST traceable standard produced 

by the gas vendor. A NIST traceable standard is certified with an instrument calibrated through direct traceability 

with a NIST standard such as an SRM or NTRM. 

6 
09-P-0235 

Chapter 2

EPA Needs an Oversight Program to 

Assure the Accuracy of Protocol Gases 

EPA does not have reasonable assurance that EPA Protocol Gases used to 

calibrate CEMS for the Acid Rain Program and continuous ambient monitors for 

the Nation’s air monitoring network are accurate. EPA has conducted only two 

assessments since 1997, when ORD discontinued its program to test the accuracy 

of EPA Protocol Gases. We analyzed 261 gaseous components and found that 

89 percent (233 components) met the Acid Rain Program’s accuracy criterion for 

calibration gases, while 11 percent (28 components) did not meet the accuracy 

criterion. Calibration gases must be accurate because inaccurate calibration gases 

can cause a system operator to calibrate a measurement system to produce 

inaccurate measurements. In turn, inaccurate CEMS measurements can affect the 

integrity of the computed allowances available for trade on the open market. 

EPA Does Not Have an Oversight Program to Assure the Accuracy of 

Protocol Gases 

EPA does not have a viable oversight program in place to assure the quality of 

EPA Protocol Gases. Since 1997, EPA oversight to assure the accuracy of these 

gases has consisted of two analytical tests conducted by OAP in 2003 and 2006. 

However, EPA did not publish the 2006 test results because of concerns over the 

sampling methodology. Without an oversight program, EPA does not have 

reasonable assurance that EPA Protocol Gases meet the accuracy criterion 

certified to by the vendors and required for the Acid Rain Program. 

In January 2008, EPA promulgated a final rule requiring EPA Protocol Gas 

producers to participate in an EPA PGVP. EPA anticipated implementing a 

verification program to assess the accuracy of Protocol Gases on an ongoing 

basis. However, implementation was delayed when a specialty gas vendor 

petitioned EPA to reconsider the rule. As of May 2009, EPA’s OAP was still 

working to resolve legal issues raised in the petition and deciding how to fund the 

program. As of May 2009, OAR’s OAQPS was developing a similar but separate 

verification program to test the accuracy of the EPA Protocol Gases used to 

calibrate continuous ambient air monitors. 

Even though the Acid Rain Program and the ambient air monitoring program 

generally use different concentrations of EPA Protocol Gases in their programs, 

the gases must be produced and certified under the same EPA Protocol. Although 

ORD originally developed the Protocol, a co-author of the Protocol informed us 

that it was not clear who currently has responsibility for maintaining the Protocol. 

Although the 2003 EPA Protocol Gas assessment report and specialty gas 

7 
09-P-0235 

producers have identified several analytical issues not addressed by the current 

Protocol, the Protocol has not been revised since 1997. 

Low Concentration NO and SO2 Gases and Lower-Cost Cylinders 

Performed Worst in Our Tests 

We purchased and analyzed 87 cylinders of tri-blend EPA Protocol Gases to 

determine whether each of the three gaseous concentrations (per cylinder) met the 

Acid Rain Program’s accuracy criterion. Of the 261 gaseous components 

analyzed, 89 percent met the Acid Rain Program’s accuracy criterion for EPA 

Protocol Gases, while 28 (11 percent) did not meet the accuracy criterion. 

Of the 28 components that did not meet the criterion, 17 were within 3.0 percent 

of the NIST-determined true concentration; 7 were within 3.0 to 5.0 percent; and 

4 exceeded the true concentration by more than 5.0 percent. All exceptions were 

for gaseous components from lower-cost cylinders (i.e., under $380). Twelve of 

the gas components’ “true” concentrations were higher than the certified 

concentration, and 16 were lower than the certified concentration. 

The gaseous components with the most exceptions were low concentration NO 

and SO2. The certified concentrations for low range NO and SO2 gases exceeded 

the NIST analyzed or “true” concentration by more than ± 2 percent for 24 and 17 

percent of the samples, respectively. The following table summarizes the test 

results for each range of gaseous components analyzed. 

Table 2-1: Results by Gas and Concentration 

No. of 

Met the Accuracy 

Criterion 

Did Not Meet the 

Accuracy Criterion 

Gas Range Samples No. Percentage No. Percentage 

NO 

Low 29 22 76 7 24 

Med 29 29 100 0 0 

High 29 25 86 4 14 

CO2 

Low 29 26 90 3 10 

Med 29 28 97 1 3 

High 29 29 100 0 0 

SO2 

Low 29 24 83 5 17 

Med 29 25 86 4 14 

High 29 25 86 4 14 

Total 261 233 89 a 28 11a 

Source: OIG, developed from data in NIST Report of Analysis, December 4, 2008 a Represents the overall percentage, not the column total. 

We also assessed the test results on a cost-per-cylinder and production-facility 

basis. This analysis revealed that all gaseous components not meeting the 

accuracy criterion were from cylinders costing under $380. All of the gaseous 

components not meeting the accuracy criterion came from 7 of the 18 (39 percent) 

facilities. These seven facilities accounted for 108 of the 261 components we 

tested, and 28 of these 108 components did not meet the Acid Rain Program’s 

8 
09-P-0235 

accuracy criterion. All 153 gaseous components associated with cylinders costing 

over $380 met the accuracy criterion. The following table shows the total tests 

conducted, the cost per cylinder, and the test results for each production facility. 

Table 2-2: Test Results by Cost and Production Facility 

Production 

Facility 

Cost Per 

Cylinder 

Total 

Tests 

No. of Tests 

Meeting the 

Accuracy 

Criterion 

No. of Tests 

Not Meeting the 

Accuracy 

Criterion 

A <380 18 15 3 

B <380 18 17 1 

N <380 18 9 9 

D <380a 

18 15 2 

K <380 9 7 2 

Q <380 18 10 8 

R <380 9 6 3 

Subtotal < 380 108 80 28 

C >380 9 9 0 

D >380a

 9 9 0 

E >380 9 9 0 

F >380 9 9 0 

G >380 18 18 0 

H >380 18 18 0 

I >380 9 9 0 

J >380 9 9 0 

L >380 9 9 0 

M >380 18 18 0 

O >380 18 18 0 

P >380 18 18 0 

Subtotal >380 153 153 0 

Totals 261 233 28 

Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. a Three sets of cylinders (a set consists of a low-, a medium-, and a high-range cylinder) were 

produced at this location but were purchased from two different vendors. Two sets of cylinders 

cost under $380 per cylinder and one set cost over $380 per cylinder. 

We further analyzed the test results, taking into account the vendor’s analytical 

technique and the reference standard used by the vendor to certify its gaseous 

concentrations. This analysis showed that low-cost cylinders certified with a Gas 

Manufacturer’s Intermediate Standard (GMIS) performed the worst in our tests. 

Specifically, 24 of the 28 (86 percent) gaseous components that did not meet the 

Acid Rain Program’s accuracy criterion were from lower-cost cylinders that were 

certified using GMIS. Although most of the gaseous components not meeting the 

accuracy criterion were certified using GMIS, many GMIS-certified components 

did meet the accuracy standard. However, all of these gaseous components were 

from cylinders costing over $380. The following table shows the test results by 

certification standard and cylinder cost. 

9 
09-P-0235 

Table 2-3: Pass/Fail Rates By Analytical Reference Standard 

Reference Standard 

Used and Cost of Gas 

Cylinder 

No. of Gaseous 

Components 

Certified with this 

Standard 

Components Not Meeting 

the Acid Rain Program’s 

Accuracy Criterion 

No. Percentage 

NTRM/SRM < $380 8 2 25 

NTRM/SRM > $380 103 0 0 

GMIS < $380 96 24 25 

GMIS > $380 50 0 0 

Unknowna < $380 4 2 50 

Totals 261 28 11b 

Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. a

 Vendor documentation did not specify type of reference standard used to certify the gaseous 

concentration. b Represents the overall percentage, not the column total. 

Calibration Gases Play an Important Role in Assuring Data Quality for 

EPA Air Programs 

Calibration gases play an important role in helping to assure the quality of data 

used in EPA programs. Inaccurate calibration gases can negatively affect EPA 

programs that rely upon accurate emissions or ambient air quality measurements 

by causing system operators to calibrate their measurements systems inaccurately. 

For example, if a calibration gas used by a utility was certified to contain 100 

parts per million (ppm) of SO2, but only contained 96 ppm, the system operator 

would unknowingly calibrate the CEMS to read 96 ppm as 100 ppm. This would 

result in the CEMS overestimating emissions. Conversely, if the true 

concentration of the calibration gas were more than the certified concentration, 

the system operator would unknowingly calibrate the CEMS to underestimate 

emissions. 

The number of monitoring systems potentially using EPA Protocol Gases is 

significant. For example, over 3,500 electric generating units were covered by the 

SO2 allowance trading program in 2007. A subset of almost 1,000 of these 3,500 

units was also covered by the NOX provisions of the Acid Rain Program. Further, 

approximately 1,300 ambient air monitors collect data on CO, SO2, and NOx 

concentrations. An EPA contractor surveyed Protocol Gas use in preparation for 

EPA’s analytical tests of Protocol Gases in 2003. The contractor estimated that 

utilities purchased from 37,500 to 75,000 Protocol Gas cylinders annually and 

missions testing companies purchased from 16,500 to 55,000 cylinders annually. 

The survey did not estimate annual purchases by State and local agencies, but one 

large State reported using about 75 Protocol Gases a year for its ambient 

monitoring network. 

Due to the nature of our sample selection and a lack of data on vendor market 

shares, we were unable to develop a sampling methodology to ensure that the 

number of samples selected from each producer represented the number of 

cylinders produced and sold to consumers. Consequently, we did not project our 

test results to the universe of EPA Protocol Gases, or estimate the impact that 

10 
09-P-0235 

these results could have on the Acid Rain Program. Despite this limitation, our 

tests represent the largest one-time assessment of the Protocol Gas industry, and 

revealed some potential areas of concern with respect to the quality of lessexpensive gases certified using GMIS. 

The following sections describe the importance of accurate monitoring data to the 

Acid Rain and Air Monitoring programs. These data are collected by 

measurement systems whose accuracy is assured by quality assurance activities 

such as calibrations using EPA Protocol Gases. 

Calibration Gases Help Assure Importance of Quality Assurance Accuracy of Emissions Data for Tests for CEMS Trading Programs 

“Sources are required to conduct For sources participating in emissions stringent quality assurance tests of trading programs, overestimating their monitoring systems, such as emissions can result in lost opportunities daily and quarterly calibration tests to sell or bank allowances. and a semiannual or annual relative 

Underestimating emissions could result accuracy test audit. These tests ensure 

in banking or selling allowances that did that sources report accurate data and 

not really exist, and provide an provide assurance to market 

inaccurate picture of a facility’s progress participants that a ton of emissions 

measured at one facility is equivalent in reducing emissions. to a ton measured at a different 

facility.” The volume of SO2 allowances traded 

under the Acid Rain Cap and Trade Source: NOX Budget Trading Program 

Compliance and Environmental Results Program is significant. According to 2007, EPA-430-R-08-008, December 2008 EPA’s Acid Rain and Related 

Programs: 2007 Progress Report, the 

total value of the SO2 allowance market was over $5.1 billion in 2007.10 In that 

same year, 4,700 private transactions involved 16.9 million allowances. 

In addition to the Acid Rain Cap and Trade Program, utilities and other large 

combustion sources participate in EPA’s NOX Budget Trading Program. This 

program is designed to reduce emissions of NOX to improve air quality in the 

eastern part of the United States. Under this program, facilities in the eastern 

States can buy and sell NOX emissions allowances. As noted in the NOX Budget 

Trading Program Compliance and Environmental Results report for 2007, 

99 percent of the NOX emissions measured under the NOX Budget Program were 

measured by CEMS. In 2007, approximately 700,000 NOX allowances were 

traded in private transactions.11 The price of NOX allowances varied from about 

$500 to $1,000 per ton during 2007, with the year-end closing price at $825 per 

ton. The total value of NOX allowances traded in 2007 exceeded $350 million. 

10 Based on total annual volume of 15,776,130 tons at an average nominal price of $325 per ton.
 11 Private transactions include all transfers initiated by authorized account representatives for any compliance or
 

general account purposes.
 

11 
09-P-0235 

The importance of calibration gases could increase as EPA expands the use of 

emissions trading programs. On March 10, 2005, EPA announced the Clean Air 

Interstate Rule (CAIR).12 CAIR caps emissions for NOX and SO2 for 28 eastern 

States and the District of Columbia. States must achieve required emissions 

reductions by requiring power plants to participate in an interstate cap and trade 

program or meet individual State emissions budget by implementing its own 

measures. Under CAIR, States may choose to participate in an EPA-administered 

regional trading program. 

In order to address climate change, Congress is considering a cap and trade 

program to control greenhouse gas emissions. The primary greenhouse gases 

include CO2, methane, nitrous oxide, and synthetically produced fluorinated 

gases.13 Implementing a greenhouse gas emissions cap and trade program could 

greatly expand the use and importance of EPA Protocol Gases if such a program 

included monitoring and calibration requirements similar to the Acid Rain 

Program. 

Calibration Gases Help Assure the Accuracy of Ambient Monitoring 

Measurements 

EPA Protocol Gases also help assure the accuracy of ambient air quality monitors 

used to collect ambient air quality data. EPA Protocol Gases are needed to 

calibrate the monitoring systems for three National Ambient Air Quality 

Standards – CO, NO2, and SO2. Based on the ambient monitoring data, EPA 

determines whether the air quality in an area meets or exceeds the standard. If the 

air quality exceeds the standard, the economic cost to the community for control 

measures needed to reduce emissions and comply with the standard can be in the 

hundreds of millions of dollars. Conversely, if EPA were to determine that an 

area met the standard because of monitoring data that produced inaccurately low 

data, control measures may not be implemented to protect the public health. 

Consequently, it is extremely important that the monitoring data are accurate so 

that EPA can make the proper attainment decision. 

The EPA Protocol Gases we tested were tri-blend cylinders with gaseous 

concentration levels generally used by utilities to calibrate CEMS. Ambient air 

monitoring systems operators would generally use EPA Protocol Gases of lower 

concentrations to calibrate their monitors, and perform other quality assurance 

tests. For example, low concentration tri-blend EPA Protocol Gas cylinders are 

used by OAQPS when they conduct the National Performance Audit Program to 

assess the accuracy of ambient air quality monitors. Since we did not test gases of 

the lower concentration range used by OAQPS for the National Performance 

Audit Program, we cannot comment on their quality. 

12 On December 23, 2008, the DC Circuit Court of Appeals reversed an earlier decision and granted EPA’s petition 

to remand the rule to EPA without vacatur. This ruling means that the regulation remains in effect but EPA must 

remedy CAIR’s flaws in accordance with the Court’s July 11, 2008, opinion in the case. 13 Hydrofluorocarbons, perfluorocarbons, and sulfur hexafluoride.
 

12 
09-P-0235 

Conclusions 

Acid Rain regulations specify accuracy criterion for EPA Protocol Gases and 

require vendors to participate in an EPA verification program. Ambient 

monitoring regulations also require the use of EPA Protocol Gases and gas vendor 

participation in a verification program. However, EPA has not implemented a 

program to provide reasonable assurance that EPA Protocol Gases are accurate. 

In addition, the Protocol for preparing EPA Protocol Gases has not been revised 

since 1997 and programmatic responsibilities for maintaining the Protocol are 

unclear. 

Recommendations 

We recommend that the EPA Assistant Administrator for Air and Radiation: 

2-1 Implement an oversight program to provide reasonable assurance of the 

quality of EPA Protocol Gases used to calibrate continuous emissions 

monitoring systems for EPA’s Acid Rain Cap and Trade Program, and 

other stationary source air programs. 

2-2 Implement an oversight program to provide reasonable assurance of the 

quality of EPA Protocol Gases used to calibrate continuous ambient air 

monitors for the NAAQS monitoring program. 

We recommend that the EPA Assistant Administrator for Research and 

Development: 

2-3 Update and maintain the EPA Traceability Protocol to meet the defined 

objectives of the Acid Rain, NAAQS, and other stationary source air 

programs. 

Agency Comments and OIG Evaluation 

OAR and ORD agreed with the report’s recommendations. 

•	 For Recommendation 2-1, OAR plans to propose a rule for public comment 

in fall 2009 to establish a largely self-supported, annual gas audit program of 

protocol gases used for the Acid Rain program. In the interim, OAR has 

already initiated efforts to conduct another OAR-funded test of EPA Protocol 

Gases gas later in 2009. 

•	 For Recommendation 2-2, OAR plans to implement a separate verification 

program to address the lower concentration protocol gases used to calibrate 

continuous ambient air monitors. Using analytical support from EPA 

Regions 2 and 7, OAR plans to verify EPA Protocol Gases supplied to State, 

13 
09-P-0235 

local, and tribal agencies for calibration of continuous, ambient gaseous 

pollutant monitors. OAR plans to launch this program by March 2010. 

•	 For Recommendation 2-3, ORD will update and publish, as guidance, a 

revised protocol to reduce the percentage of EPA Protocol Gases that do not 

attain the Acid Rain Program’s accuracy criterion. ORD plans to complete 

and issue the revised protocol by October 1, 2010. ORD also provided a 

corrective action plan in response to Recommendation 2-3 (see Appendix D). 

The general actions outlined in OAR’s and ORD’s responses meet the intent of our 

recommendations. Recommendations 2-1 and 2-2 will remain open in our tracking 

system pending our receipt and approval of OAR’s final corrective action plan. 

ORD’s proposed corrective action plan sufficiently addresses Recommendation 2-3. 

As such, we are closing Recommendation 2-3 in our tracking system upon issuance 

of this report. These recommendations will be tracked to completion in the 

Agency’s tracking system. 

With respect to the ambient monitoring program, OAR commented that “the report 

demonstrates that the gases tested met the expectations of the EPA Traceability 

Protocol with 95 percent confidence limit. We point out that the gases used for the 

ambient air program rely mainly on the traceability protocol. Thus, there are no 

findings demonstrating a concern with using protocol gases for that program.” We 

disagree with this characterization. Our tests found that the low-range concentration 

gases performed the worst in meeting the 2 percent accuracy criterion for the Acid 

Rain Program. The ambient monitoring program uses even lower concentration 

gases than the ones we tested during our evaluation. The past two Protocol Gas 

assessments conducted by EPA have not included these lower concentration gases. 

We believe the lack of test data to assure the accuracy of the lower concentration 

gases used for the ambient monitoring program is a concern that should be addressed 

by EPA. 

After receiving OAR’s response to the draft report, the OIG met with OAR to 

discuss its response. OAR requested that we revise Recommendation 2-2 to state 

that OAR should implement an oversight program to provide reasonable assurance of 

the quality of EPA Protocol Gases. We made this change to Recommendation 2-2. 

OAR and ORD also provided several technical clarifications and comments to the 

report. We made changes to the final report based on these comments, as 

appropriate. OAR and ORD’s responses to the draft report are contained in 

Appendices C and D. 

14 
09-P-0235 

Status of Recommendations and 

Potential Monetary Benefits 

POTENTIAL MONETARY 

RECOMMENDATIONS BENEFITS (in $000s) 2 

Rec. 

No. 

Page 

No. Subject Status1 Action Official 

Planned 

Completion 

Date 

Claimed 

Amount 

Agreed To 

Amount 

2-1 

2-2 

2-3 

13 

13 

13 

Implement an oversight program to provide 

reasonable assurance of the quality of EPA 

Protocol Gases used to calibrate continuous 

emissions monitoring systems for EPA’s Acid 

Rain Cap and Trade Program, and other 

stationary source air programs. 

Implement an oversight program to provide 

reasonable assurance of the quality of EPA 

Protocol Gases used to calibrate continuous 

ambient air monitors for the NAAQS program. 

Update and maintain the EPA Traceability 

Protocol to meet the defined objectives of the 

Acid Rain, NAAQS, and other stationary source 

air programs. 

O 

O 

O 

Assistant Administrator for 

Air and Radiation 

Assistant Administrator for 

Air and Radiation 

Assistant Administrator for 

Research and 

Development 

10/01/2010 

1 O = recommendation is open with agreed-to corrective actions pending; 

 C = recommendation is closed with all agreed-to actions completed; 

 U = recommendation is undecided with resolution efforts in progress. 

2 Identification of potential monetary benefits was not an objective of this evaluation. 

15 
09-P-0235 

Appendix A 

Details on Scope and Methodology 

To test the accuracy of EPA Protocol Gases, we contracted with a third party to purchase EPA 

Protocol Gases directly from the specialty gas producers. Our intent was to purchase at least one 

set of three EPA Protocol Gas cylinders from each gas producer and manufacturing location in 

the country. After purchasing at least one cylinder from each identified producer14

 or 

manufacturing location, we purchased additional cylinders from each producer to maximize our 

sample size and obtain a better representation of the facilities’ production capability. Our 

contractor purchased 87 EPA Protocol Gas tri-blend cylinders from 11 different producers. 

NIST analyzed these cylinders and compared the results with each cylinder’s certified tag values. 

Chain of custody procedures were employed to account for each sample cylinder throughout the 

process. Both our contractor and NIST conducted their work in accordance with the OIGapproved quality assurance project plans. 

Procedures for Purchasing Sample Gases 

We tasked our contractor with surveying the industry to identify EPA Protocol Gas vendors and 

manufacturing locations. To accomplish this survey, the contractor developed a set of keywords 

to electronically search the Internet and the Thomas Register® for potential vendors of EPA 

Protocol Gases. Our contractor also obtained industry information through conversations with 

gas vendor personnel during the process of obtaining vendor quotes. The survey identified a 

total of 11 gas producers and 18 manufacturing locations. 

We authorized our contractor to purchase gas cylinders from each of the identified vendors with 

an initial goal of purchasing at least three tri-blend cylinders of a low-, medium-, and highconcentration range from each vendor manufacturing location. After accomplishing this goal, 

we purchased additional cylinders from vendors with a goal of obtaining additional cylinders 

from a cross section of vendor types taking into account such factors as cylinder cost and 

potential market share. An initial set of three cylinders was purchased from all 18 manufacturing 

locations, an additional set of cylinders was purchased from 9 of the 18 locations, and a third set 

was purchased from one of the 18 locations. In all, 87 cylinders (29 sets of 3) were purchased. 

Our contractor purchased the gases from January 30, 2008, to April 29, 2008. The following 

table shows the number of cylinders by pollutant and concentration range. 

Table A-1: Summary of Sample Gases Purchased 

No. of 

Cylinders 

Range 

Type CO2 (%) NO 

(ppm) 

SO2 

(ppm) 

29 High 18.0 900 1000 

29 Mid 12.0 400 500 

29 Low 5.00 50.0 50.0 

Source: EPA Protocol Gas Industry Survey and Blind Audit, 

Final Report, February 2009 

14 To ensure cylinders were purchased from every producer, sets of cylinders were purchased from three distributors. 

16 
09-P-0235 

NIST Analysis 

Under an interagency agreement with the OIG, NIST was tasked with analyzing the 

concentrations of the sample cylinders and comparing the test results to the manufacturers’ 

certified value. NIST’s primary objective in conducting its analysis was to achieve a calculated 

uncertainty of ± 0.5 percent or better for its analysis to achieve a 4:1 or better ratio between the 

acceptance criterion (i.e., the Acid Rain Program’s ± 2 percent accuracy criterion) being tested 

and the uncertainty of NIST’s analysis. A general standard of practice in metrology is for this 

ratio – referred to as TAR or test to accuracy ratio – to be greater than or equal to 4:1. NIST’s 

analytical uncertainty ranged from ± 0.42 percent to ± 0.68 percent at the 95 percent confidence 

level, depending upon the gas analyzed and the concentration. 

To be consistent with the reporting of past EPA Protocol Gas tests conducted by EPA, Chapter 2 

of this report presents the analytical results without adjustment for the uncertainty of the 

analyses. Appendix B presents NIST’s results with uncertainty. 

Limitations 

We instructed all parties involved in this evaluation not to discuss our plans with specialty gas 

vendors or persons not needing such information to implement the evaluation. However, several 

months prior to purchasing the gas cylinders, we were informed that at least one specialty gas 

producer had become aware our intended evaluation. We do not know whether additional 

vendors also became aware of our planned evaluation, or whether they would have been able to 

identify the orders intended for the tests. If a specialty gas vendor was aware of our evaluation, 

the results may not necessarily represent the quality of gases such a vendor routinely produces 

and sells to users, but only indicates its ability to produce quality EPA Protocol Gases. 

Because of a lack of data on the specialty gas vendor market share, we were unable to design a 

sample that would allow us to project our results to the entire universe of EPA Protocol Gases 

with an acceptable level of confidence and precision. Since we were unable to select gas 

cylinders for testing in a manner that would ensure a statistically representative sample of the 

EPA Protocol Gas market, we did not perform any inferential statistical analysis to project the 

sample test results to the overall EPA Protocol Gas market. 

17 
09-P-0235 

Appendix B 

Test Results With Uncertainty Factored into Analyses 

NIST calculated the total uncertainty of its analysis at the 95 percent confidence level for each of 

the component ranges it tested. NIST’s calculated analytical uncertainty ranged from ± 0.42 

percent to ± 0.68 percent. To assess the results taking into account the uncertainty of NIST’s 

analysis, we adjusted the test limits to account for uncertainty using the following formula: 

Test limit = Specification (Acid Rain Program ± 2 percent criterion) – ± Uncertainty (NIST’s analytical uncertainty) 

The following table shows how the test limits were adjusted using a NIST analytical uncertainty 

of ± 0.42 percent as an example. 

Table B-1: Example of Test Limits Adjusted for Uncertainty 

Test Limit (Expressed as 

Percentage Difference) Conclusion if Sample Falls in This Range 

< - 2.42 Failed with 95 percent confidence 

-2.42 to -1.58 Within the range of uncertainty 

-1.57 to 1.57 Passed with 95 percent confidence 

1.58 to 2.42 Within the range of uncertainty 

> 2.42 Failed with 95 percent confidence 

Source: OIG table developed from data in NIST Report of Analysis, December 4, 2008. 

When the test results are adjusted for uncertainty, the number of components meeting or 

exceeding the Acid Rain Program’s accuracy criterion is smaller than the unadjusted test 

results presented in Chapter 2. For example, when we factor in uncertainty, we can only 

say with 95 percent confidence that 82 percent (214 of 261) of the components met the 

criterion, whereas the unadjusted test results show that 89 percent of the components met 

the accuracy criterion. Similarly, when we factor in uncertainty, we can only state with 95 

percent confidence that 5 percent (12 of 261) of the components did not meet criterion. 

The remaining 13 percent (35 of 261) of the components were within the range of NIST’s 

analytical uncertainty. Table B-2 on the next page summarizes the test results adjusted for 

uncertainty. 

18 
09-P-0235 

Table B-2: Summary of Test Results 

Test Results 

Category No. Percentage 

Met the Acid Rain Accuracy Criterion 214 82 

Did Not Meet the Acid Rain Accuracy Criterion 12 5 

Test Results Within the Range of Analytical Uncertainty 35 13 

Total 261 100 

Source: Developed from data in NIST Report of Analysis, December 4, 2008. 

Both the unadjusted and adjusted test results showed that the lower-cost cylinders in our 

sample performed worse than higher-cost cylinders. The adjusted results show that 11 of 

the 12 gaseous mixtures not meeting the Acid Rain Program’s accuracy criterion were 

lower-cost cylinders (< $380) certified with GMIS. The following table summarizes the 

tests meeting the accuracy criterion by cost and reference standard used to certify the 

concentration. 

Table B-3: Results by Vendor’s Analytical Reference Standard and Cost 

Reference Standard 

Used and Cost of Gas 

No. of Components 

Certified with this Standard 

Components Meeting the Acid Rain 

Accuracy Criterion 

No. Percentage 

NTRM/SRM < $380 8 6 75 

NTRM/SRM > $380 103 100 97 

GMIS < $380 96 59 61 

GMIS > $380 50 47 94 

Unknowna < $380 4 2 50 

Totals 261 214 82b 

Source: Developed from data in NIST Report of Analysis, December 4, 2008 

a Vendor documentation did not specify type of reference standard used to certify the gaseous concentration. b Represents the overall percentage, not the column total. 

19 
 

09-P-0235 

Appendix C 

OAR’s Response to OIG Draft Report 

August 5, 2009 

MEMORANDUM 

SUBJECT: Comments on Draft IG Report: “EPA Needs an Oversight Program for Protocol 

Gases” Assignment number 2007-000877 

FROM: Gina McCarthy /s/ 

Assistant Administrator for Air and Radiation 

TO: Wade T. Najjum 

Assistant Inspector General for Program Evaluation 

The Office of Air and Radiation (OAR) appreciates the Office of Inspector General’s 

(OIG) effort to work with our staff on this unique testing program to produce the subject Draft 

Final Report “EPA Needs an Oversight Program for Protocol Gases” to improve the quality 

assurance of protocol gases. We generally agree with the data described in the draft final report. 

However, the issues raised apply differently to the Acid Rain Program’s (ARP’s) source 

emissions monitoring systems and National Ambient Air Quality measurement network systems. 

Therefore, this memorandum is structured in two parts – an Acid Rain Program section and an 

Ambient Air Program section. 

Acid Rain Program 

Our major comments are presented below. We include both major and detailed 

comments in the attached Word document in Track Changes. After you’ve reviewed our 

comments, we’d like to have another meeting with you as soon as possible. 

•	 At the end of the first paragraph in “At a Glance”, please add: “However, 96 percent were 

within 3 percent of the true value; and there was no bias associated with the 28 gas 

components that did not meet the ARP performance specification. Based on the analysis 

results, there are no anticipated nationwide environmental consequences of the calibration 

gases used in the ARP. However, the ARP is addressing potential source-level problems in 

two ways: (1) by performing an EPA-funded gas audit in 2009; and (2) by proposing a rule 

establishing a largely self-supported, ongoing, annual gas audit program.” 

•	 Add the following paragraph to the “At a Glance” section: 

“Due to the nature of our sample selection and a lack of data on vendor market shares, we 

were unable to develop a sampling methodology to ensure that the number of samples selected from 

each producer represented the number of cylinders produced and sold to consumers. Consequently, 

20 
09-P-0235 

we could not project our test results to the universe of EPA Protocol Gases, or estimate the potential 

impact that gases not meeting the Acid Rain Program’s accuracy criteria could have on that 

program. Despite this limitation, our tests represent the largest one-time assessment of the Protocol 

Gas industry. The final results of this study identified some potential areas of concern with respect 

to the quality of less expensive gases certified using GMIS.” 

• Add the following paragraph to the “Noteworthy Achievements” section: 

“The ARP is currently conducting a third gas audit, covering all known producers of EPA 

Protocol gas. As in past ARP audits, the results will be released to the public. We expect results by 

early 2010. To provide a cost-effective, long term audit capability, OAR is also developing a 

proposed rule package to implement a largely self-supporting, annual gas audit program. This 

proposed rule is expected to be published for public comment in the Federal Register in the fall of 

2009.” 

ARP concurs with all of the OIG recommendations. However, we suggest a slight 

rewording for recommendation #2-1 as follows: 

2-1 Implement a better oversight program to assure the quality of EPA Protocol Gases used 

to ensure the accuracy of continuous emissions monitoring systems for EPA’s Acid Rain 

Program, and other stationary source air programs. 

Ambient Air Programs 

[OIG NOTE: The comments initially submitted by the Ambient Air Program on August 5, 2009 

were superseded by the amended comments provided September 2, 2009, in the following 

memorandum.] 

Thank you again for the opportunity to comment on the draft final report. If you have 

questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up 

Coordinator, at 202-564-1228. 

21 
 

09-P-0235 

September 2, 200915 

MEMORANDUM 

SUBJECT: Amended Comments on Draft IG Report: “EPA Needs an Oversight Program for 

Protocol Gases” Assignment Number 2007-000877 

FROM: Gina McCarthy /s/ 

Assistant Administrator for Air and Radiation 

TO: Wade T. Najjum 

Assistant Inspector General for Program Evaluation 

 The Office of Air and Radiation (OAR) appreciates the Office of Inspector General’s (OIG) 

effort to work with our staff on this unique testing program to verify the quality assurance of 

protocol gases as published in the subject Draft Final Report “EPA Needs an Oversight Program 

for Protocol Gases”. In a follow-up meeting between our staffs, your office explained that 

recommendation 2-2 was directed toward the program under development and outlined by the 

Office of Air Quality Planning and Standards (OAQPS) during the audit. Consequently, we 

agree with recommendation 2-2 and provide the following amended paragraph titled “Ambient 

Air Program”, along with new recommendations to amend our August 5, 2009 response 

concerning the ambient air program’s protocol gases. 

Ambient Air Program 

The report demonstrates that the gases tested met the expectations of the EPA Traceability 

Protocol with 95 percent confidence limit. We point out that the gases used for the ambient air 

program rely mainly on the traceability protocol. Thus, there are no findings demonstrating a 

concern with using protocol gases for that program. Determination of compliance with an ambient 

air standard is completed with consideration of many factors including the quality of the data. We 

believe updating the traceability protocol is the most efficient and effective corrective action to 

improve the quality of gases and subsequently the data collected for the ambient air program. 

Revising the protocol will also benefit stationary source emissions measurements. We, therefore, 

agree with recommendation 2-2 and have already started creation of an appropriate oversight 

program to reasonably assure the quality of protocol gases for the ambient air monitoring program. 

We amend our August 5, 2009 response: 

1.	 Replace the sentence in the “At A Glance” section that says, “However, as of April 2009, 

EPA had not implemented a verification program, primarily due to indecision over how to 

fund the program.” with “However, as of April 2009, OAQPS, with commitments by EPA 

Regions 2 and 7 for analytical support, had begun development of a program to provide 

independent verification of EPA Protocol Gas cylinders used for continuous ambient air 

monitors.” 

15 The OAR memorandum was undated but was received by OIG on September 2, 2009. 

22 
09-P-0235 

2.	 Add the following paragraph to the “Noteworthy Achievements” section: “OAQPS has 

planned and begun development of the Protocol Gas Verification Program to independently 

verify “EPA Protocol Gases” supplied to state, local and Tribal agencies for calibration of 

continuous, ambient gaseous pollutant monitors. EPA Regions 2 and 7 have committed to 

providing analytical support. The program is expected to be launched in March 2010. 

Finally, we suggest the following rewording for recommendation 2-2: “Implement an oversight 

program, which reasonably assures the quality of EPA Protocol Gases used to calibrate continuous 

ambient air monitors for the National Ambient Air Quality Standards monitoring program as 

specified in 40 CFR part 58 Appendix A.” 

Thank you again for the opportunity to comment on the draft final report. If you have 

questions regarding our comments, please contact Dawn Roddy, OAR Audit Follow-up 

Coordinator, at 202-564-1228. 

23 
 

09-P-0235 

 Appendix D 

ORD’s Response to OIG Draft Report 

July 30, 2009 

MEMORANDUM 

SUBJECT:	 ORD Response to OIG’s Draft Evaluation Report: EPA Needs an Oversight 

 Program for Protocol Gases (Assignment No. 2007-00877) 

FROM:	 Lek G. Kadeli /s/ 

Acting Assistant Administrator (8101R) 

TO:	 Wade T. Najjum

 Assistant Inspector General for Program Evaluation (2460T) 

Thank you for the opportunity to provide comments on the draft Office of the Inspector 

General (OIG) report entitled “EPA Needs an Oversight Program for Protocol Gases.” 

Attached are the Office of Research and Development’s (ORD) specific comments on the 

draft report. In general, ORD concurs with the recommendation that ORD should update and 

maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We appreciate your 

consideration of our comments. If there are any questions please do not hesitate to call me at 202­

564-6620 or Jorge Rangel at 202-564-1606. 

Attachment 

24 
09-P-0235 

Office of Research and Development Comments on 

Office of Inspector General Draft Audit Report, "EPA 

Needs an Oversight Program for Protocol Gases", 

Assignment No. 2007- 000877, July 8, 2009 

This document is comprised of four sections: 

1. General comments regarding the OIG recommendations 

2. Table of ORD Response to OIG recommendations 

3. Proposed detailed schedule of corrective actions 

4. Specific comments by page number 

1. General Comments regarding the OIG Recommendations 

Thank you for the opportunity to comment on the draft Office of the Inspector General 

(OIG) report entitled “EPA Needs an Oversight Program for Protocol Gases.” EPA’s Office of 

Research and Development (ORD) concurs with the recommendation that ORD should update and 

maintain the EPA Traceability Protocol for Gaseous Calibration Standards. We believe that the 

scientific expertise within ORD puts it in a unique position to understand the gas metrological issues 

that need to be addressed in the update and to prepare the update. 

2. Table of ORD Response to OIG Recommendations 

Rec. 

No. 

OIG Recommendation Action 

Official 

ORD 

Corrective 

Actions 

Planned 

Completion 

Date 

2-3 Update and maintain the 

EPA Traceability Protocol 

to meet the defined 

objectives of the Acid 

Rain, NAAQS, and other 

stationary source air 

programs. 

Assistant 

Administrator 

for Research 

and 

Development 

Update and 

publish (as 

guidance) a 

revised protocol 

to reduce the 

percentage of 

EPA Protocol 

Gases that do not 

attain the CAMD 

accuracy criteria 

10/01/2010 

3. Proposed Detailed Schedule of Corrective Actions 

Corrective Action Taken Planned Completion Date 

Final audit report disseminated to public October 1, 2009 

List of potential protocol changes* sent to 

NIST and specialty gas producers for 

October 15, 2009 

25 
09-P-0235 

Corrective Action Taken Planned Completion Date 

review with request for their comments, 

suggested revised text, and any 

concentration stability data supporting 

changes to allowable gas composition, 

concentration ranges, minimum cylinder 

pressures, and certification periods 

Receive all comments and stability data November 15, 2009 

Complete internal review draft of protocol May 15, 2010 

Receive EPA comments on internal draft June 1, 2010 

Complete external review draft of protocol 

by NIST and specialty gas producers 

July 1, 2010 

Receive external comments on revisions August 15, 2010 

Revised protocol published as guidance October 1, 2010 

* In 2005, EPA sent the following list of informal change suggestions to NIST and specialty gas producers: 

- include a procedure specifically for the use of Fourier Transform infrared (FTIR) spectroscopy; 

- improve statistical procedures for uncertainty calculations and revise the Excel spreadsheet; 

- include mercury calibration standards in the protocol; 

- eliminate (or retain) gas manufacturers intermediate standards (GMISs); 

- lengthen the certification periods for EPA Protocol Gases; 

- change the cylinder pressure limitations for EPA Protocol Gases; 

- include a procedure for analyzing and certifying zero gases; 

- include gas dilution systems in the protocol; 

- tighten documentation requirements; 

- develop audit participant identification number for regulatory data reporting purposes; 

- require ISO 17025 accreditation for producers of environmental calibration gases; 

- include provision for preparation and analysis of batches of EPA Protocol Gases; 

- require on-site visits of specialty gas producers by EPA representatives; and 

- provide for technical assistance/outreach to specialty gas producers by NIST representatives; 

4. Specific comments by page number 

[OIG NOTE: ORD also provided several technical clarifications and comments to the draft 

report. We made changes to the final report based on these comments, as appropriate.] 

26 
09-P-0235 

Appendix E 

Distribution 

Office of the Administrator 

Assistant Administrator for Air and Radiation 

Acting Assistant Administrator for Research and Development 

Deputy Assistant Administrator for Air and Radiation 

Deputy Assistant Administrator for Research and Development 

Director, Office of Air Quality Planning and Standards, Office of Air and Radiation 

Director, Office of Atmospheric Programs, Office of Air and Radiation 

Director, National Risk Management Research Laboratory, Office of Research and Development 

Agency Follow-up Official (the CFO) 

Agency Follow-up Coordinator 

General Counsel 

Associate Administrator for Congressional and Intergovernmental Relations 

Associate Administrator for Public Affairs 

Audit Follow-up Coordinator, Office of Air and Radiation 

Audit Follow-up Coordinator, Office of Research and Development 

Acting Inspector General 

27