System Audit of the Ambient Monitoring Program:

Bay Area Air Quality Management District

May 26 – June 4, 2009

Conducted by:

Air Quality Analysis Office Quality Assurance Office 

Air Division Management and Technical 

Services Division

US EPA Region 9

75 Hawthorne Street

San Francisco, California 94105

 Final Report: November 2010
ii

INDEX

PAGE

GLOSSARY OF ACRONYMS........................................................................iii

EXECUTIVE SUMMARY................................................................................1

INTRODUCTION..............................................................................................3

A. GENERAL PROGRAM....................................................................................5

B. NETWORK MANAGEMENT.........................................................................7

C. FIELD OPERATIONS......................................................................................9

D. LABORATORY OPERATIONS....................................................................14

E. DATA AND DATA MANAGEMENT............................................................22

F. QUALITY ASSURANCE AND QUALITY CONTROL.............................26

APPENDIX A, Summary of Findings.............................................................32

APPENDIX B, Good Practice Recommendations.........................................35
iii

GLOSSARY OF ACRONYMS

APCD.....................................Air Pollution Control District

AQS....................................... Air Quality System

BAAQMD..............................Bay Area Air Quality Management District

BAM...................................... Beta Attenuation Mass Monitor (continuous PM2.5)

CARB.....................................California Air Resources Board

CARE.....................................Community Air Risk Evaluation

CO..........................................Carbon Monoxide

CFR........................................Code of Federal Regulations

DMS.......................................Data Management System

DQO.......................................Data Quality Objective

EC.......................................... Elemental Carbon

EPA....................................... Environmental Protection Agency

FEM....................................... Federal Equivalent Method

FRM....................................... Federal Reference Method

GC-MS...................................Gas Chromatography-Mass Spectrometry

MDL.......................................Method Detection Limit

MET/PE................................. Meteorology and Performance Evaluation

NAAQS..................................National Ambient Air Quality Standard

NAMS....................................National Air Monitoring Station

NIST.......................................National Institute for Standards and Technology

NOx/NOy...............................Nitrogen Oxides

O3........................................... Ozone

OC..........................................Organic Carbon

ODAMN................................ Operations Data Action Management Monitoring 

Notification

ORD.......................................Office of Research and Development

PAMS.....................................Photochemical Assessment Monitoring Stations

PE...........................................Performance Evaluation

PM..........................................Particulate matter

PM2.5...................................... Particulate matter 2.5 microns or less in aerodynamic 

diameter

PM10....................................... Particulate matter 10 microns or less in aerodynamic 

diameter

PQAO.....................................Primary Quality Assurance Organization

QA..........................................Quality Assurance
iv

QAPP..................................... Quality Assurance Project Plan

QC..........................................Quality Control

SPM....................................... Special Purpose Monitor

SOP........................................ Standard Operating Procedure

SLAMS.................................. State or Local Air Monitoring Station

SO2......................................... Sulfur Dioxide

SRP........................................ Standard Reference Photometer

STN........................................Speciation Trends Network 

TAD....................................... Technical Assistance Document

TSA........................................Technical System Audit

USEPA or EPA......................United States Environmental Protection Agency

VOC.......................................Volatile Organic Compound
1

EXECUTIVE SUMMARY

This document is a report on the findings made by the U.S. EPA Region 9 while 

conducting a Technical Systems Audit (TSA) on the air monitoring program of the Bay Area Air 

Quality Management District (BAAQMD). A TSA is an on-site review and inspection of a state 

or local ambient air monitoring program to assess its compliance with established regulations

governing the collection, analysis, validation, and reporting of ambient air quality data. This 

TSA meets the requirements for EPA Regional 9 audits of the BAAQMD monitoring 

organization required by 40 CFR Part 58, Appendix A, Section 2.5.

The TSA was conducted by U.S. EPA Region 9 staff during the weeks of May 26 – June 

4, 2009. The audit team interviewed management and staff on specific aspects of the air 

monitoring program including network design, field operations, laboratory operations, data 

handling, quality assurance and quality control procedures. The audit team also inspected some 

of the monitoring sites operated by BAAQMD. The site inspections consisted of an interview 

with the site operator (when possible), reviewing station and instrument logbooks, and evaluating 

the station siting with respect to EPA requirements for probe siting1. The laboratory inspection 

included a review of the particulate matter program for mass determinations, laboratory analysis 

for volatile organic compounds (VOC), OC/EC and carbonyl sample analysis. 

The TSA is one of the ways that EPA provides oversight to ensure that data collected by 

state and local agencies meets certain minimum data quality objectives. Other assessments such 

as network reviews and performance evaluations are also used to collect information on the 

overall quality of ambient air monitoring data. These assessments also enable agencies to 

identify and correct those program elements which may be adversely affecting the quality of 

ambient air data. The results of the TSA are summarized here and fully described in this report, 

along with recommended actions to address the findings. The specific actions to be taken by 

BAAQMD will be determined through negotiations between EPA and BAAQMD and will be 

documented in a corrective action plan prepared by BAAQMD. 

EPA would like to thank all the staff and management of Bay Area Air Quality 

Management District for their support and cooperation during the audit.

A. Program Strengths:

BAAQMD has a very robust ambient air monitoring program, as evidenced by the very high 

quality data with very little missing data. This can be attributed to a combination of skilled and 

dedicated field and PE/QA staff, sound analytical work in the laboratory, highly competent 

validators, and committed managers. Some specific examples of the strength of BAAQMD’s 

ambient air monitoring program include:

 BAAQMD managers exhibit a solid awareness of air quality issues within the San 

Francisco Bay Area along with a strong understanding of EPA network requirements and 

recommendations. The result is very effective network planning.

 

1 40 CFR, Part 58, Appendix E, “Probe and Monitoring Path Siting Criteria for Ambient 

Air Quality Monitoring”
2

 BAAQMD has developed an exemplary network plan.

 Field station operators have primary responsibility for at most three sites and therefore 

keep the instruments and sites in good working condition. 

 The District has a ‘Station Operator SOP’ which provides operators with an overview of 

their many responsibilities in running a monitoring station.

 The troubleshooting SOP for station operators compiles technical insights from other 

operators and gives direction for troubleshooting problems previously encountered.

 The laboratory has established an effective system of tracking and documenting 

instrumental quality control results using control charting in the OC/EC analysis area.

B. Program Major Findings:

During the TSA, EPA identified areas where the BAAQMD monitoring program should be 

strengthened. The major findings are: 

1. BAAQMD has had significant quality issues with the Thermo Scientific Partisol FRM, 

Model 2000. 

2. BAAQMD lacks a QA Officer position whose sole responsibility is to oversee quality 

management of the laboratory.

3. The BAAQMD QAPP’s approval is conditioned on establishing DQOs for data uses that 

are not defined by EPA.

4. Most quality management functions are being performed; however, QA authority is not 

clearly defined.

5. BAAQMD does not have a formal corrective action system outside of the performance 

audit program.

The individual findings, along with minor findings, are reported in the topic sections of this 

document and are also summarized in Appendix A. In addition to those areas which should be 

strengthened, EPA also identified areas which could be strengthened and those recommendations 

and suggested best practices are included in Appendix B for the information and use of the 

BAAQMD’s management.
3

INTRODUCTION

The Bay Area Air Quality Management District (BAAQMD or the District) has been in 

existence for more than 40 years and has primary responsibility for ensuring the health and 

welfare of citizens of the San Francisco Bay Area under the Clean Air Act of 1970 and its 

Amendments of 1990. BAAQMD operates the ambient monitoring network for seven full 

counties (Marin, Napa, Contra Costa, Alameda, Santa Clara, San Mateo, San Francisco) and two 

partial counties (southern Sonoma and western Solano). 

From May 26 through June 4, 2009, EPA Region 9 conducted a Technical System Audit 

(TSA) of the ambient monitoring program operated by BAAQMD for all program areas except 

certain aspects of data management. BAAQMD is in the process of developing and 

implementing a new data management system (DMS) and therefore issues specifically pertaining 

to the DMS have been deferred until the next TSA. In its entirety, the TSA covers the following 

program areas:

Network Management

Network Design and Monitoring Station Siting

Organization, Staffing and Training

Facilities

Field Operations

Routine Operations

Field Quality Control

Preventive Maintenance

Record Keeping

Data Acquisition and Handling

Laboratory Operations

Routine Operation

Laboratory Quality Control

Preventive Maintenance

Record Keeping

Data Acquisition and Handling

Specific Pollutants:

PM

VOCs

Carbonyls

OC/EC

Data and Data Management

Data Handling

Software Documentation

Data Validation and Correction

Data Processing

Internal Reporting

External Reporting

Quality Assurance and Quality Control

Status of Quality Assurance Program
4

Audits and Audits System Traceability 

National Performance Audit Program (NPAP) 

Audit Procedure Review

Documentation and Data Processing Review

Corrective Action System

Audit Result Acceptance Criteria

Participating managers and supervisors of the Technical Services Division and positions 

at the time of the TSA were:

Eric Stevenson, Air Monitoring Manager

Jim Hesson, Laboratory Services Manager

Dick Duker, Meteorology/Performance Evaluation Manager

Mark Stoelting, QA Officer

Jeff Robbins, Laboratory QA Officer

Bob Franicevich, Performance Evaluation Supervisor

Stan Yamaichi, Air Monitoring Supervisor

Glen Colwell, Air Monitoring Supervisor

The EPA regional staff members conducting the TSA were Meredith Kurpius, Catherine 

Brown, and Michael Flagg of the Region 9 Air Division’s Air Quality Analysis Office and 

Mathew Plate, Roseanne Sakamoto and Steve Remaley of the Region 9 Quality Assurance 

Office. Eugenia McNaughton, Manager of the Quality Assurance Office attended the exit 

meeting representing EPA Management.

This report is divided into ten sections. The first section is an executive summary that 

describes the purpose of the TSA and a summary of the major findings. The body of the report 

includes both major and minor findings. Minor findings are those that have a smaller impact on 

the integrity of the program but still require corrective action. The second section is an 

introduction that provides a brief description of the organization of BAAQMD’s air monitoring 

program, the EPA audit team, and the report organization. The third section discusses our 

general findings on the BAAQMD monitoring program as a whole. The remaining five sections 

address specific aspects of the air monitoring program: network management, field operations, 

laboratory operations, data management and quality assurance/quality control. Appendix A is a 

summary listing of the findings contained in this report. Appendix B is a list of recommended 

good practices that are not required by Federal regulation but would strengthen BAAQMD’s 

monitoring program, nonetheless. The findings and recommendations in this report are listed in 

priority order within each section. Recommendations to resolve findings are provided to give 

some indication of the Region’s expectations as to how findings can be addressed. If BAAQMD 

has other approaches or alternatives to address the concerns identified, the Region will consider 

them, provided the corrective action adequately addresses the finding.
5

A. GENERAL PROGRAM

The Bay Area Air Quality Management District has been in existence for more than 40 

years and has primary responsibility for ensuring the health and welfare of citizens of the San 

Francisco Bay Area under the Clean Air Act of 1970 and its Amendments of 1990. The District 

conducts ambient air quality monitoring in support of its role in developing and enforcing air 

pollution control regulations. BAAQMD is an independent agency with more than 300 

employees, governed by a 21-member Board of Directors. The Board is made up of publiclyelected officials apportioned according to the population of the represented county.

To ensure that the policies and regulations adopted by the Board are working to improve 

the quality of the air in the San Francisco Bay Area, BAAQMD continuously monitors air 

quality at 24 SLAMS locations and eight SPM locations throughout the nine-county area. In 

addition, CARB operates one non-FEM BAM PM2.5 monitor in Point Reyes and Monterey Bay 

Unified Air Pollution Control District operates one FRM PM2.5 monitor in the city of Hollister 

which is in Northern San Benito County. This network also allows BAAQMD to notify the 

public whenever air quality reaches levels regarded as unhealthy. 

The BAAQMD sections that are the subject of this TSA are Air Monitoring, Laboratory 

Services, and Meteorology and Performance Evaluation (MET/PE), which are all in the 

Technical Services Division and are under the direction of Technical Services Director. At the 

time this TSA was conducted Gary Kendall was the Director of Technical Services (see attached 

organization chart1). Source Test operations, which are also within the Technical Services 

Division, were not part of this audit.

The Air Monitoring Section, managed at the time of TSA by Eric Stevenson, oversees 

field station operations. Within this section there are two supervisors, who were Stan Yamaichi 

and Glen Colwell at the time of the TSA, and 16 technical staff.

The Laboratory Services Division, managed by Jim Hesson, has five chemists and an 

additional vacancy that is not currently being filled. One of the chemists, Jeff Robbins, also 

serves as the laboratory QA Officer.

Meteorology and Performance Evaluation (PE) are grouped into one section (MET/PE),

managed at the time of the TSA by Dick Duker. The QA Officer responsibility for ambient air 

monitoring resides within this section. Mark Stoelting, who is also an air and meteorology 

specialist, was recently assigned to this position. Performance Evaluation (e.g., field audits) is 

also within this section, but has been separated from other QA activities and is supervised by 

Bob Franicevich. In addition to the manager, this section has one QA Officer/Air and 

Meteorology Monitoring Specialist, one PE Supervisor, four PE technical staff, and four 

meteorologists.

 

1

Since this TSA was conducted numerous management changes have occurred: Gary Kendall has retired and been 

replaced by Eric Stevenson; Eric Stevenson was replaced by Glen Colwell; and Dick Duker retired and has not 

yet been replaced.
6

The organization of BAAQMD’s Technical Services Division divides QA responsibility among 

various sections and various personnel: Laboratory QA Officer, QA Officer within the MET/PE 

Section, and PE Supervisor within the MET/PE Section. All QA functions are embedded within 

the sections and rather than having a formal QA role to oversee all ambient air monitoring 

operations at the management level.

FINDING A1 (Major):

BAAQMD has had significant quality issues with the Thermo Scientific Partisol FRM, Model 

2000. 

Discussion:

Due to the discontinuation of support for the Andersen PM2.5 samplers, BAAQMD 

converted their monitoring network to Partisol monitors. For conversion of a large 

monitoring network, BAAQMD found that the Partisol was the only FRM instrument 

available. This is due to a deficit of currently available models of filter-based reference 

methods. BAAQMD subsequently has had problems with instrument performance and 

vendor support.

 Recommendation:

USEPA Region 9 needs to follow-up with OAQPS and ORD on the issues of lack of an 

adequate number of viable FRMs and vendor support for FRMs.

FINDING A2:

BAAQMD keeps multiple hard copies and electronic copies of many documents, making it 

unclear which is the official copy. 

Discussion:

Many of the BAAQMD documents are generated electronically and subsequently printed. 

This procedure generates both an electronic record and one or more hard-copy records. 

Generally BAAQMD retains both records. However, at a later time the hard copy 

records may be annotated and the electronic records can be modified. In some 

circumstances it was unclear to staff interviewed which are the preferred records to be 

referred to and retained.

 Recommendation:

For records that have multiple copies BAAQMD should designate the official copy and 

handle it appropriately. Staff should understand which records are official documents 

and how they should be handled.
7

B. NETWORK MANAGEMENT

The ambient air monitoring network in San Francisco Bay Area consisted of 32 

monitoring sites at the time this TSA was conducted (25 State/Local Air Monitoring Stations 

(SLAMS) plus nine Special Purpose Monitoring (SPM) sites) that utilize a variety of air 

pollutant measuring instruments. The following table summarizes the SLAMS air monitoring 

sites in the San Francisco Bay Area.

SITE O3 CO NOx SO2 PM10 FRM 

PM2.5

BAM* 

PM2.5

Bethel Island X X X X X

Concord X X X X X X

Fremont X X X

Livermore X X X X X X

Napa X X X X X

Oakland X X X

Pittsburg X X X X X

Redwood City X X X X X X

San Francisco X X X X X X X

San Jose – Jackson X X X X X X

San Jose – Tully X

San Pablo X X X X X

San Rafael X X X X

Santa Rosa X X X X X

Vallejo X X X X X X X

Crockett X

Fairfield X

Gilroy X X

Hayward X

Los Gatos X

Martinez X

San Leandro X

San Martin X

Sunnyvale X

Hollister+ X

* Non-FEM BAM

+

operated by Monterey Bay Unified Air Pollution Control District

Source: BAAQMD’s “Annual Air Quality Monitoring Network Plan 2008”

BAAQMD also operates eight special purpose monitoring sites (SPM) and CARB operates one 

within the San Francisco Bay Area:

 Point Reyes (CARB) – Non-FEM BAM PM2.5

 Oakland, West Grand – PM2.5 speciation

 Point Richmond – H2S
8

 Richmond – SO2, H2S

 Rodeo – H2S

 Benicia – O3, CO, NOx, SO2, PM10, Non-FEM BAM PM2.5

 Berkeley – O3, CO, NOx, SO2, CH4, PM10, Non-FEM BAM PM2.5, VOCs, metals

 Cupertino – Hexavalent chromium

 Hunter’s Point – asbestos 

In addition, the District operates a network of air toxics stations, including a NATTS at 

San Jose – Jackson. This audit addressed air toxics activities that support the NATTS, including 

both field and laboratory operations. The San Jose – Jackson site is also the NCore site (National 

Core multipollutant monitoring station). Other air monitoring activities include PM2.5 speciation 

monitoring at San Jose – Jackson, Vallejo, Livermore, and Oakland – West Grand. BAAQMD is 

also establishing PAMS-like monitoring at Livermore and two new sites in eastern Contra Costa 

County.

The air monitoring network within the San Francisco Bay Area meets or exceeds the 

requirements for the minimum number of monitoring sites designated as SLAMS for all of the 

criteria pollutants. BAAQMD managers exhibit broad knowledge of air quality issues within the 

San Francisco Bay Area along with a strong understanding of EPA network requirements and 

recommendations. The result is very effective network planning. In addition, the monitoring 

sites are all properly located in terms of monitoring objectives and spatial scales. 

The District submits an annual monitoring network plan. The most recent submittal, in 

2008, includes all of the required information about the monitoring stations. 40 CFR 58.20 

requires States to perform an annual review of their existing ambient monitoring networks to 

ensure that they continue to meet the monitoring objectives of 40 CFR 58, appendix D. The 

current BAAQMD document meets the requirements of describing the network and the intended

monitoring objectives of each site and is, in fact, an exemplary document that has been provided 

to other agencies as a model. 

According to 40 CFR 58.10 (d), “The State, or where applicable local, agency shall 

perform and submit to the EPA Regional Administrator an assessment of the air quality 

surveillance system every 5 years to determine, at a minimum, if the network meets the 

monitoring objectives defined in appendix D to this part, whether new sites are needed, whether 

existing sites are no longer needed and can be terminated, and whether new technologies are 

appropriate for incorporation into the ambient air monitoring network.” The first network 

assessment is due July 1, 2010. BAAQMD is aware of this requirement and the deadline for 

submission and is preparing to undertake this effort.

There are no findings that require corrective action related to network management.
9

C. FIELD OPERATIONS

Field operation duties for monitoring are primarily managed by the Air Monitoring 

Section with support for instrument audits handled by the MET/PE Section. Both sections are 

within the Technical Services Division but are separate sections with different managers. Air 

Monitoring Section staff handle all day-to-day operations including: instrument installation, 

repair and maintenance; sample collection; first-level data review; some quality control 

activities; and general field site maintenance. Most field operators are responsible for 2-3 sites 

and visit each site multiple times per week. Generally we found that the station operators have a 

very high level of oversight and a comprehensive understanding of their field sites. As a result, 

BAAQMD has very robust ambient air quality data with very little missing data.

The main function of the Performance Evaluation Group is to assess quality control in the 

Air Monitoring Section by conducting performance evaluations of gas analyzers and particulate 

samplers at the District’s air-monitoring sites. The group also performs calibrations and 

verifications of pressure and temperature standards for the Air Monitoring Section as a service 

for the division. Other than pressure and temperature standards, PE staff are not involved in any 

Air Monitoring Quality Control activities. These activities are more thoroughly addressed in the 

Quality Assurance section.

As part of this TSA, managers and supervisors for the Air Monitoring and MET/PE 

Sections were interviewed. We also spoke with station operators at all the stations visited and 

reviewed logbooks. Station logbooks are archived on site whereas instrument logbooks remain 

with the instrument. All station operators demonstrated a thorough knowledge of the monitoring 

equipment for which they were responsible and expressed a sense of ownership of the data. 

Current copies of QAPPs and SOPs were available to all station operators electronically at most 

field stations via secure access to the District’s central computing network. 

The field sites visited were: Bethel Island, Concord, Livermore, Napa, Oakland 

(International Blvd.), San Francisco, San Jose (Jackson St.), Vallejo, Fairfield, Gilroy, and 

Berkeley (SPM site). Most sites were generally in pre-existing commercial buildings rented by 

the District with inlets or instruments located on the roof. Some sites with one or two monitors

were outfitted simply with mounted containers to house instruments and supplies. Each station 

operator has a “home station” that serves as the operator’s office. The home stations contain 

basic office functions such as computing, networking, and printing capabilities along with tools, 

supplies, storage capability, and other field necessities.

Our evaluation at these sites included inspection of the inlet manifolds, examination of 

station and instrument logbooks, and a specific check of whether appropriate QC checks and QA 

audits were being performed for dates in 2008 selected for review prior to the audit. All sites 

were well-operated and met the siting criteria of 40 CFR 58, Appendix E. For example, 

instruments were sited appropriately away from each other and obstacles/trees, sites had clean 

inlet manifolds, calibration standards were current, and station logbooks and instrument 

logbooks were generally up-to-date and contained relevant information on operations and 

maintenance activities. 
10

Overall we found the field station operations to be very strong and we identified many 

positive findings, including:

 Very competent and dedicated field station operators and managers.

 Field station operators have primary responsibility for at most three sites and therefore 

keep the instruments in good working order.

 The District has a ‘Station Operator SOP’ which provides operators with an overview of 

their many responsibilities in running a monitoring station.

 Most long-term monitoring stations are housed in permanent structures with sufficient 

amenities and space, which ensures smooth, efficient and convenient field operations. 

 The troubleshooting SOP for station operators compiles technical insights from other 

operators and gives direction for troubleshooting problems previously encountered.

FINDING C1:

PM10 flow checks may not be occurring at the required frequency at all sites.

Discussion:

As of December 2006 flow checks for PM10 became a regulatory requirement. The 

requirement is once a month for low volume samplers (40 CRF Part 58, Appendix A, 

Section 3.2.3) and once a quarter for high volume samplers (40 CRF Part 58, Appendix 

A, Section 3.3.2). One of the station operators noted that PM10 flow checks are 

performed annually and the PM10 instrument logbook confirmed that these checks were 

being performed no more frequently than annually.

Recommendation:

BAAQMD should ensure that flow rate checks are conducted and documented at the 

appropriate frequency.

FINDING C2:

BAAQMD operates many of their gaseous monitors using sample pumps that deviate from the 

reference method.

Discussion:

USEPA defines methods for establishing Federal Reference and Equivalent Methods 

(FRMs and FEMs) in 40 CFR Part 53. All the monitors being used by BAAQMD for 

compliance with the NAAQS are either FRM or FEM monitors. It must be demonstrated 

that any modification to FRM or FEM monitoring equipment does not impact the 

equivalency of the instrument and approved by USEPA.

BAAQMD has indicated that changing to the sampling pumps has resulted in significant 

cost and energy savings. USEPA Region 9 is supportive of these changes and, based on 

monitoring network performance, has confidence that these deviations could be 

approved. USEPA Region 9 and BAAQMD have consulted USEPA Office of Research 

and Development (ORD) to determine if using different sample pumps qualifies as a 
11

change to a FRM. USEPA ORD indicated that this change would qualify as a 

modification requiring USEPA approval.

Recommendation:

Develop and submit documentation that demonstrates that changing the instrument 

sample pumps does not significantly change instrument performance.

FINDING C3:

Several different SOPs appear to apply to instrument calibrations that are not consistent. 

Additionally, documentation of calibration was hard to follow and incomplete.

Discussion:

USEPA defines how gaseous instruments must be calibrated in 40 CFR Part 50. 

BAAQMD has defined gaseous calibration procedures in both the instrument SOPs and 

in SOP 220. However, these procedures do not appear to be consistent. This particularly 

evident when comparing SOP 711, for NOx monitors, which is detailed and follows 40 

CRF Part 50, Appendix F very closely, to SOP 220.

The only documentation of the calibration procedure appears to be included on the sheet 

titled “TECO 146C Calibration System Control Settings.” However, this worksheet is 

hard to follow and does not include evaluations of linearity and converter efficiency for 

NO2. Additionally, the low calibration level for ozone does not adequately address 

linearity at concentrations near or below the NAAQS of 75 ppb. It was also noted that 

the worksheet is annotated with additional information by site operators between 

calibrations.

Recommendation:

BAAQMD needs to reconcile its gaseous calibration SOPs, ensure that the calibration 

procedures used conform or 40 CFR Part 50, and improve documentation of calibration.

FINDING C4:

The level of information recorded in the station logbooks is not consistent and at some sites the 

logbooks are not frequently used.

Discussion:

Station logbooks are generally used to record all pertinent information regarding station 

operations. This could include: days the site operator was present at the site; site visitors; 

instrument or site maintenance; instrument QC checks; instrument repair or replacement;

instrument calibration; weather conditions; unexpected problems; and other information 

not included in another logbook.

The level of information noted to be included in station logbooks, in BAAQMD SOP 

101, is minimal. Additionally, the level of information found in the field varied by 
12

operator. In some cases most logbook entries are not made by the routine site operator. 

The following specific examples were noted:

 One of the items required by SOP 101, “Dates that instrumentation were repaired 

or changed out and serial numbers of replacement instruments” was not included 

in the logbook.

 At the Napa site it was noted that the bi-weekly instrument filter changes were not 

being documented in the instrument logbook but instead on a piece of masking 

tape on the back of the instrument rack. This is a good practice to help the site 

operator remember when the filter was changed last. However, a record of each 

filter change should also be included in the instrument logbook.

 Reagent changes and maintenance to the zero air scrubber at the Fairfield site are 

not being documented.

 Some archived logbooks did not contain a closeout comment. For example, an 

appropriate closeout comment would include date, last entry, and new logbook 

with control number XXXXX to be used, followed by hatch marks. Whenever a 

new logbook is employed, the old logbook should be closed out with an entry 

such as noted above. This is to ensure that no additional entries are made in the 

old logbook. Any entries made below the statement and within the hatch marks 

are not considered official documentation.

 The calibration of the NOx instrument, at the Vallejo site, conducted on 12/29/08 

was not found in the instrument logbook.

Standardizing logbook entries to include the following may be helpful. This could 

include: 

 Date, time and initials of the person(s) who have arrived at the site

 Visitors

 Brief description of the weather (e.g., clear, breezy, sunny, raining)

 Brief description of exterior of the site. Any changes that might affect the data 

should be recorded – for instance, if someone is parking a truck or tractor near the 

site, this may explain high NOX values

 Any unusual noises, vibrations, or anything out of the ordinary

 Records of any station maintenance or routine operations performed

 Description of the work accomplished at the site (calibrated instruments, repaired 

analyzer)

 Dates that instrumentation were repaired or changed out and serial numbers of 

replacement instruments

 Detailed information about the instruments that may be needed for repairs or 

troubleshooting

 Other pertinent information recorded in other logbooks

Source: QA Handbook Volume II, Section 11.2.3, Instrument and Site Logs

Recommendation:

BAAQMD should have a more consistent policy on what station operators enter into the 

station logbook, including routine maintenance, instrument repair, audits and calibrations, 

and logbook closeout comments. Logbooks should be routinely reviewed to ensure that 

pertinent information is being recorded.
13

FINDING C5:

Toxics samplers are not routinely verified by checking with blanks and/or standards for 

contamination or losses.

Discussion:

USEPA Method TO-14A, Sections 11.2.2 and 11.2.3 and other USEPA documents 

describe methods for sampling system certifications. In general, these certifications are 

completed to verify that the sampling system is capable of adequately recovering the 

analyte of interest from ambient air. For example, TO-14A Section 11.2.2 describes 

humid air zero certification for VOC sampling. Under this section certification is defined 

as “evaluating the sampling system with humid zero air and humid calibration gases that 

pass through all active components of the sampling system. The system is ‘certified’ if 

no significant additions or deletions (<0.2 ppbv of targeted compounds) have occurred 

when challenged with the test gas stream.” 

BAAQMD does not have a program to certify the toxic sampling systems. Additionally, 

the toxic sampling systems are not routinely checked using blank air to verify that they 

are not introducing contaminants into the sampling containers/media.

Recommendation:

To produce definitive toxics data, BAAQMD should use sampling systems that are 

routinely certified and screened for blank contamination.

FINDING C6:

Instrument SOPs were not available at some sites with limited monitors (e.g., Fairfield and 

Gilroy).

Discussion:

There is no remote access computer installed at some sites to access SOPs or other 

information. Neither were hard copies of SOPs maintained at the site or carried by the 

station operator.

Recommendation:

Ensure that field operators have access to information in relevant SOPs at the field 

station, including remote field stations.
14

D. LABORATORY OPERATIONS

Analytical laboratories provide support for measurement methods that are either too 

complex or sensitive to perform in the field environment. In order to provide these services, 

laboratories employ complex instrumentation and staff with highly specialized training. 

For ambient air samples to provide useful information or evidence, laboratory analyses 

must meet the following four basic requirements:

1. Equipment must be frequently and properly calibrated and maintained

2. Personnel must be qualified to make the analysis

3. Analytical procedures must be in accordance with accepted practice

4. Complete and accurate records must be kept

Laboratory operations are managed by Jim Hesson, Manager of the Laboratory Services

Section, within the Technical Services Division. BAAQMD’s laboratory facility is located in the 

District’s headquarters office in San Francisco, California. Most analyses are performed inhouse by BAAQMD laboratory staff. Those compounds for which BAAQMD does not have 

staff or laboratory resources to perform are sent to support laboratories such as ERG and DRI for 

analysis. In addition, BAAQMD provides laboratory support services for other districts, such as 

PM2.5 weighing for North Coast Unified AQMD and Monterey Bay Unified APCD. One of the

primary responsibilities of laboratory is the handling of PM filters which includes preparation, 

weighing, tracking, and storing PM2.5 and PM10 filters. In addition to PM responsibilities, the 

laboratory is also responsible for sample analyses for air toxics (including NATTS), PAMS-like 

monitoring, and special projects run by BAAQMD such as monitoring near the Port of Oakland 

as part of the BAAQMD CARE project. The compounds that BAAQMD routinely analyzes for 

include:

- VOCs (currently by TO-14 but expanding to include TO-15)

- Carbonyls, except acrolein (TO-11)

- Acrolein for NATTS (modified TO-15)

- OC/EC

BAAQMD collects but does not analyze samples for speciated PM2.5, PM10 metals, PAHs, and 

Chrome(VI). There are a number of additional activities the laboratory undertakes to support the 

collection and analysis of air pollutants. These include canister cleaning and preparation, data 

validation, and sample storage. 

The BAAQMD laboratory staff is one of the most important assets of the laboratory. The 

analysts/chemists were knowledgeable, talented, and clearly dedicated. It was generally found 

that BAAQMD laboratory is in compliance with respect to performing analyses according to 

EPA methods but lacks a dedicated laboratory Quality Assurance Officer to oversee the review 

and documentation of quality assurance results and implementation of corrective actions on a 

consistent basis. While the District has assigned a QA Officer for the laboratory, the 

laboratory’s current staffing situation (i.e., short-staffed with an unfilled vacancy) prevents the 

appropriate level of effort required to fulfill this role. Internal quality assurance audits are not 
15

performed and many critical Standard Operating Procedures (SOPs) do not reflect current 

laboratory practice. The laboratory has established an effective system of tracking and 

documenting instrumental quality control (QC) results using control charting in the OC/EC 

analysis area and it would be of considerable benefit to expand implementation of a similar 

system throughout all departments of the laboratory as originally planned. The laboratory would 

benefit from assigning dedicated logbooks for instrument maintenance, standards preparation, 

instrument injection run logs and MDL studies in each area in order to simplify review. The 

laboratory has purchased a new Gas Chromatograph/Mass Spectrometer which it plans to 

dedicate for volatile organic carbon (VOC) analyses according to EPA method TO-15. 

Additional facility space and possibly a separate air handling system may be needed to 

accommodate the new system.

TOXICS LABORATORY FINDINGS

FINDING D-T1 (Major):

BAAQMD lacks a QA Officer dedicated to overseeing quality management of the laboratory.

Discussion:

The laboratory has identified a chemist to serve as the QA Officer for the laboratory and 

the Technical Services Division has assigned a Quality Assurance Officer for ambient air 

monitoring; however, neither of these positions has been fully developed to address the 

quality management needs of the laboratory. For example, SOPs are not up-to-date 

(Finding D-T2), and logbook and data reviews are not done on a consistent basis 

(Findings D-T6 and D-T7). The laboratory currently has a vacant position and staffing 

issues were cited as the primary reason that the laboratory QA role has not been fully 

developed.

Recommendation:

Fully develop and allocate adequate time for a laboratory QA Officer role in the 

laboratory.

FINDING D-T2:

Many of the SOPs do not reflect actual laboratory practices.

Discussion:

SOPs document the formal laboratory procedures that are adhered to in generating 

sample results by all analysts and by doing so represent a critical component of data 

defensibility. There were numerous instances where SOPs do not reflect current 

laboratory practices, including:

 SOPs describe in-house preparation of the TO-14 standards but BAAQMD 

has moved to purchasing a gas cylinder standard from a vendor. 

 Sample matrix spikes for carbonyls are being performed at a desirable 

frequency but the frequency is inconsistent with the laboratory's written SOP.
16

 A secondary control standard is not currently being analyzed for TO-14 by the 

laboratory as stated in the BAAQMD SOP.

 BAAQMD SOP specifies that 1 out of every 8 canisters in each batch should 

undergo certification but in reality only 1 in 16-20 is certified.

 Laboratory staff indicated that the OC/EC analyses have changed but these 

changes are not reflected in the SOP.

Recommendation:

It is recommended that the laboratory follow through with current plans to update all 

laboratory SOPs so they reflect actual laboratory practices.

FINDING D-T3:

The TO-14 standard certification and stability are uncertain.

Discussion:

Calibration directly affects sample quantitation and certification of standards is a critical 

QA step. Laboratory staff stated that calibration gas standards are purchased from one of 

two commercial vendors but the concentration certification is not purchased because the 

cost is prohibitive. In place of the vendor certification, the laboratory relies on analysis 

of a sample from the new gas cylinder by CARB and by Air Toxics Laboratory. CARB 

purchases a fully certified gas standard directly from NIST and so the CARB analysis is 

regarded as a definitive certification analysis. Current District practice differs from 

vendor certification in some important ways, for example, cylinders are analyzed 

multiple times by the vendor but only once by CARB and the transport of the cylinders 

between BAAQMD to CARB could introduce problems if proper container and/or 

system surface conditioning is not utilized. This procedure may be adequate but the 

robustness of the procedure needs to be demonstrated. 

In addition, there is no expiration date assigned to the gas standard. The stability of the 

standard has not been formally studied and the laboratory is unaware of any outside 

studies that establish an expiration date. Therefore, periodic checks at suitable intervals 

are needed to ensure components of the standard have not cross reacted or degraded. It is 

necessary to document the ongoing integrity of standards as this forms the basis for 

sample contaminant quantitation in order for data results to be considered defensible.

Recommendation:

Review and justify the reliability of standard certification procedure. Included in the 

review, EPA suggests that the laboratory determine: the impact of transporting cylinders, 

and if new cylinders can feasibly be directly analyzed by CARB; how many times the 

cylinder or sample should be analyzed by CARB; if the CARB values are to be used as 

the certified values or if the values provided by the vendor are to be used; and, 

acceptance limit for allowable percent difference between the CARB results and the 

vendor supplied results. Studies should be performed to assess the stability of the 

standards over time and to establish an appropriate expiration/recertification schedule.
17

FINDING D-T4:

Canister certification for cleanliness is not performed according to the "Technical Assistance 

Document for Sampling and Analysis of Ozone Precursors" and the BAAQMD SOP. 

Discussion:

Laboratory staff were unclear as to the number of canisters in each batch that are selected 

for certification of cleanliness. Laboratory staff estimated that one out of 20 or possibly 

one out of 16 canisters in every batch are certified. This practice is outside the 

recommendation in the “Technical Assistance Document for Sampling and Analysis of 

Ozone Precursors” (Ozone Precursor TAD) and the BAAQMD SOP specification that 

one out of every eight canisters in each batch should undergo certification. A greater 

level of assurance that contamination reported in samples did not originate from sources 

extraneous to the sample or from a previous sample in the canister would be provided by 

adhering to the procedures specified in the BAAQMD SOP and TAD of certifying one

out of eight canisters in every cleaned batch. This would strengthen the defensibility of 

data.

Canisters are selected at random for certification following cleaning. Laboratory staff 

stated they do not take into consideration which canisters had the highest concentrations 

of contaminants when selecting the canister in each batch for cleanliness certification. 

Staff further clarified the cleaning process has been very effective in their experience and 

therefore they believed a random selection of canisters for certification has been 

sufficient. The current random selection practice is outside the recommendation in the 

Ozone Precursor TAD, page 105 and may not be adequate with the planned transition to 

the TO-15 GC-MS method which has lower detection limits.

Recommendation:

It is recommended that the District review its canister cleanliness certification process 

and ensure compliance with the BAAQMD SOP on all issues including the number of 

canisters selected for certification. In addition, BAAQMD should consider adopting 

recommendations outlined in the Ozone Precursor TAD, such as tracking the 

contamination history and selecting canisters that are the most highly contaminated for 

certification.

FINDING D-T5:

The laboratory has not established a retention time for certified canisters after which they must 

be recertified. 

Discussion:

Canisters can be subject to contamination following certification. The District monitors 

the pressure gauge after canister shipment to the field following cleaning and 

certification. While this is a good quality assurance measure for ensuring significant 

vacuum loss has not occurred, it has been the experience of the EPA Region 9 Laboratory 
18

that canisters may become contaminated over time through small leaks or micro leaks 

that may not be obvious from monitoring canister gauge readings. Laboratory staff 

agreed that canisters have on occasion exhibited leaks. Although it was stated that 

canisters are generally used within two or three weeks of certification, a procedure is not 

in place to ensure they are not used beyond this time, which could result in 

contamination. Establishing a retention time policy for canisters stored at the laboratory 

after they have been cleaned and certified would add additional assurance that canisters 

have not become contaminated over time through smaller leaks. 

Recommendation:

It is recommended that the laboratory establish a retention time policy for clean canisters 

after which date they will be re-cleaned and certified as an added quality assurance 

measure they have not become contaminated. A retention time of 30 days is used in 

some other laboratories and would be reasonable. 

FINDING D-T6:

Documentation and record-keeping in the laboratory does not fully capture laboratory activities, 

was found to have errors, and is not organized for easy tracking and retrieval of information.

Discussion:

Accurate record documentation serves an important QA function in the defensibility of 

data and becomes critically important if data are ever questioned. Specific issues 

identified include:

 Dedicated, separate logbooks are currently not maintained for instrument 

maintenance, instrument analyses and minimum detection limit studies. All 

information is currently recorded in the same laboratory instrument logbook. 

Using separate logbooks dedicated for instrument analyses, instrument 

maintenance activities, standards preparation, and MDL studies would 

facilitate the tracking and retrieval of information. 

 Computer generated instrument batch sequence lists or raw data printouts 

constitute the current method of documenting analytical analyses. 

Supplementing these records with handwritten, bound instrument logbooks 

would allow a permanent record of notes concerning everything that happened 

during an analytical run, including the analysis of highly contaminated 

samples with potential carryover would allow the entire instrument analysis 

run to be reconstructed when necessary. 

 Errors were identified for dates entered in the standard preparation logbook. 

For example, numerous entries spanning a year's duration appeared to fall 

under a single August, 2008 date caption, which the analyst agreed was in 

error.

 A system of tracking and documenting the receipt and monitoring of 

expiration dates of standards is lacking. New gas standard cylinders are not 

logged in upon arrival. Logbooks providing the time of receipt of new 

standards and certifications or ongoing verification analysis by outside 
19

laboratories should document the integrity of standards used for sample 

quantitation.

 Laboratory staff stated that standards used for the purpose of establishing 

contaminant peak retention times are analyzed but not documented in the 

computer sequence lists of analyses that serve as the laboratory's instrument 

run logbook. There is no documentation of the date and time that peak 

retention times were updated. Since the identification of sample contaminants 

is based on the retention time comparison of sample peaks to known standard 

contaminant peaks, the analysis of standards used to determine, update or reverify retention times should be documented. 

Recommendation:

Improve documentation of some quality assurance and routine laboratory activities. 

FINDING D-T7:

Secondary review of records was found to be irregular. Approximately six month intervals 

between secondary reviews were observed in some instances.

Discussion:

Regular review of instrument logbooks by a supervisor or QA Officer helps to ensure and 

document that proper analysis protocol is being followed, e.g. calibrations, blanks 

analyses etc. Repeated failures or attempts to pass calibrations noted in logbooks can be 

an indication that instrument maintenance or other corrective actions need to be 

performed.

Recommendation:

It is recommended that a policy of regular secondary review and signature of logbooks or 

other records of analyses be adopted. A weekly or greater frequency would be desirable.

FINDING D-T8:

Laboratory space is constrained and may be inadequate once the additional capabilities such as 

GC-MS that are on-line.

Discussion:

Currently, the laboratory facilities are crowded and barely adequate for the current 

analyses. For example, liquid nitrogen Dewars have to be stored in the hallway, which 

partially blocks egress, because there is no room in the laboratory. Moreover, there is no 

dedicated air handling system for the VOC analyses which may result in high background 

contamination, potentially interfering with the performance of the new high-sensitivity 

GC-MS.

Recommendation:

Assess and address the laboratory facilities needs.
20

FINDING D-T9:

The laboratory is not tracking the retention time of sample cartridges to ensure they are used 

within six months of the manufacturer's certification analysis as recommended by the 

manufacturer.

Discussion:

The laboratory is not tracking the retention time of sample cartridges to ensure they are 

used within six months of the manufacturer's certification analysis as recommended by 

the manufacturer. Representative cartridges which have exceeded the six month 

expiration date are not being analyzed for contamination as specified in the BAAQMD 

SOP, page 8. Furthermore, the laboratory has changed its practice of shipping the exact 

number of cartridges needed to the field for sampling events by sending a full box of 

cartridges which can result in some cartridges not being used and increasing the potential 

for cartridges to exceed the six month expiration time. Certification that cartridges are 

free of contamination prior to sampling is an important quality assurance measure to 

document that contamination reported in samples did not originate from an extraneous 

source.

Recommendation:

It is recommended that the laboratory formalize a procedure for tracking the expiration 

date of cartridges to ensure the date has not been exceeded.

PARTICULATE MATTER LABORATORY FINDINGS

FINDING D-PM1:

BAAQMD does not perform periodic field blanks for PM10.

Discussion:

The use of blank controls determine the potential for contamination of filter and help

evaluate the consistency of the balance readings over the filter handling period. 

Additionally, field (i.e., filter) blanks are required by regulation as a mechanism for 

evaluating filter media, see 40 CFR Part 50, Appendix J, Section 7.2.3, and EPA 

Compendium Method IO-3.1 notes in Section 5.4, “Provide one blank sample with every 

10 actual samples.” BAAQMD performs laboratory blanks for PM10 on a routine basis. 

By adding field blanks BAAQMD would provide for control of field as well as laboratory 

handling.

Recommendation:

It is recommended that BAAQMD collect periodic PM10 field blanks.

FINDING D-PM2:

The records of the Polonium strips were not retained and they could not be verified.
21

Discussion:

Polonium strips have a set shelf life and should be tracked as a laboratory standard.

Recommendation:

Retain documentation of the polonium strips purchase, use, and expiration date.

FINDING D-PM3:

The PM2.5 filter associated with the data point tracked for January 4, 2008 at Livermore was 

invalidated because it was outside the holding time.

Discussion:

The data was flagged in the database as a laboratory error. However, it appears that the 

filter was not delivered to the laboratory until over three weeks after the sampling day. 

This resulted in the laboratory to have less than 1 week to perform the filter weighing.

Recommendation:

BAAQMD should ensure that filters are transported to the laboratory on a timely basis to 

avoid holding time violations.
22

E. DATA MANAGEMENT

Data management generally involves both a data validation process and a data 

management system. The District is currently developing a new data management system 

(DMS) which should be operational within the next few months. For this reason we have 

deferred review of the data management system an all related components (e.g., software 

documentation, data processing, etc.) until our next TSA in 2012. This data management 

section, therefore, focuses on data validation.

Data validation is a systematic process for reviewing a body of data against a preestablished set of quality control acceptance criteria. When data do not meet the acceptance 

criteria, they are flagged with a qualifier identifying the associated problem and possibly 

invalidated. Data validation should occur at the earliest opportunity for optimizing cost 

effectiveness. This allows for corrective actions to take place early in data collection and 

analytical processes. One of the primary objectives of data validation is to determine if data are 

adequate to meet the intended purposes, such as Air Quality System (AQS) reporting. 

The MET/PE, Laboratory Services, Air Monitoring, and Planning program staff were 

interviewed regarding their involvement in data validation for criteria pollutants, PM2.5, PM10, 

toxics and meteorology results. Interviewees, with their respective areas of review responsibility 

in parenthesis, were: 

Air Monitoring

Nancy Balberan (initial review of field operator submitted data for

gaseous and PM)

Laboratory Services 

Mary Beacon (PM analysis and review)

MET/PE

Mike Basso (continuous gaseous, toxics)

Ken Chrysler (PM10)

Dick Duker (PM2.5)

Planning Office

Jim Cordova (meteorology)

Jeff Matsuoka (meteorology and computer support)

Each office has specific validation responsibilities as outlined in the BAAQMD SOP 101, and 

601 through 609. 

Data points for BAAQMD's ambient air monitoring program are generated either in the 

field for continuous monitoring data (e.g., gaseous criteria pollutants and BAM PM2.5) or in the 

laboratory for time-integrated samples collected in the field (e.g., PM filters, VOC canisters, 

carbonyl cartridges, etc.). Continuous data are logged by computer with strip chart backup and 
23

are telemetered to the District office from each site once per day with the exception of ozone and 

continuous PM2.5 which are telemetered hourly for AQI reporting. At the field station, operators 

review and evaluate strip chart recordings and compare the information to the data loggers; for 

example, they check the zero and span results to ensure the instruments have operated within 

established limits. The Senior Supervising Technician reviews the data daily. The precision and 

accuracy data is reviewed by Nancy Balberan with the final review and entry into AQS by the 

Monitoring and Meteorology Manager. Time-integrated samples are collected by field station 

operators and either delivered to the District laboratory or sent to a contract laboratory for 

analyses. 

Once data arrive at the District office, a working copy of all collected data is produced 

from the master copy. Validation is performed on the working copy while the master is 

maintained separately in the BAAQMD database in case the working copy becomes corrupted or 

a need arises to review the master copy. No one has “write” access to the master. 

The BAAQMD validation process excels in that they have assigned very skilled 

individuals to address specific areas of validation as identified. In summary, all gaseous and PM 

data is initially reviewed by the Air Monitoring office; PM data is also reviewed by the 

Laboratory Services office. Subsequent review of both gaseous and PM data is performed by the 

MET/PE office and is then uploaded into AQS. Initial validation for toxics data is performed by 

the Laboratory Services Office and then forwarded to the MET/PE office for further validation. 

The MET/PE staff occasionally consult the Laboratory Services Office for clarification when 

performing validation. Meteorology data is initially reviewed by the MET/PE office and then 

transferred to the Planning office for further validation to maintain objectivity in review since the 

MET/PE Office has responsibility for collecting the meteorology data. Any data points voided 

through any of the validations identified above is reviewed by the manager of the MET/PE office 

prior to upload into AQS reporting.

Taking these individual efforts in aggregate, the validation program is one of great 

strength, as each validator performs a focused review of their area of responsibility. The 

challenge with this level of expertise is that it cannot easily be replicated by other staff in the 

event that a validator is absent and the process may not be fully transparent to others. 

FINDING E1:

Finding: SOPs 601-609 for data management are general and do not provide step-by-step 

instruction of how data are actually validated by the data reviewers (validators) in Ambient 

Monitoring, Laboratory Services, MET/PE, and Planning Sections.

Discussion:

Although SOPs for data management are contained in the Quality Assurance Project Plan 

(QAPP), they are descriptions rather than true SOPs that inform one how to perform a 

particular function in a consistent, reproducible manner. More detailed SOPs or 

checklists have been developed by some, but not all, reviewers to help themselves 

perform validation and to ensure they do not miss any important steps. 
24

It is important to prepare SOPs that cover all the activities expected to be performed by 

the validator: 1) to enable tracking of what actually is being checked and the criteria used 

to determine acceptability of data; and 2) to provide guidance to new, back-up or crosstrained personnel unfamiliar with validation or a particular step in validation. The SOPs 

should cover validation activities taking place by Laboratory Services, Ambient 

Monitoring, and MET/PE staff.

Recommendation:

Develop and update SOPs to explicitly identify the specific activities that should be 

performed following EPA’s Guidance for Preparing Standard Operating Procedures

(QA/G-6) EPA600/B-07/001/April 2007. 

The following are suggested to be covered in BAAQMD SOPs for all phases of 

validation: 

 identification of the location and specific databases accessed 

 the checks performed 

 criteria used to determine data acceptability 

 points of contact when questions arise 

 documentation of the communications 

 whether validated data is approved by another party or the individual performing 

validation 

 where validated data is saved 

 how subsequent individuals are notified that review is completed by the previous 

validator 

 identification of all voided data with explanation

 identification of the individuals receiving final evaluation 

 how the organization knows validation is complete 

The SOPs should be complete enough such that additional SOPs do not need to be 

developed separately by the individuals validating data. 

Cross-training for validators is also suggested for consistency in reviews and to act as 

back-ups, when necessary. 

FINDING E2:

Questions raised by data validators in the MET/PE to the Ambient Monitoring or Laboratory 

Services are not uniformly documented. Some validators are not documenting the discussions of

data quality or the reason for changing data points.

Discussion:

In the majority of cases, validators are going beyond the scope of the data management 

SOPs in a commendable manner by documenting their discussions with Ambient 

Monitoring or Laboratory Services staff whenever they need clarification for determining 

data quality and whether data will be kept or voided. Communications were saved in 
25

various forms, e.g., on individual or network computer drives, individual forms or 

meeting logbooks. These were accessible only by the individual documenting the 

communication and not attached to the data that had been validated. All communications

and decisions made regarding data points of concern should identify the:

 the party raising the question* 

 the date the question was raised

 the parties who were contacted to reach resolution* 

 the party approving the resolution* 

 the date of resolution 

*there should be a signature associated with each step identified with asterisk 

Where resolution impacts field operations, they should be communicated to field station 

operators and their supervisors.

Recommendation:

Standardize the documentation process for validation communications and ensure they 

are taken to completion, such as is done for ODAMNs. Include all relevant personnel on 

communications and final decision on questioned data, including field station operators,

so that field staff are aware of particular or recurring conditions that impact data quality 

(e.g., partisols).

RECOMMENDATIONS FOR NEW DMS

Although we did not specifically address the data management system during this phase 

of the TSA, we did identify some recommendations for the new DMS. These include:

 Incorporate a tool to allow field station operators and validators to verify the accuracy of 

the daily printouts. These could be control charts or some other tool.

 Provide validators with access to instrument performance data to allow a secondary check 

on instrument performance.

 Initiate use of electronic chain of custody for data validation with “write” capability lock 

out activated after each validator completes his/her review. Once lock out is activated, it 

is recommended that a program be written that initiates a chain of custody and sends 

notice to subsequent reviewers that the data is ready for review. 

 Use one system for meteorological data validation. For example, perform meteorological

data validation on FOXPRO. Export validated data to DMS and have a program or 

additional staff verify that the DMS contents mirror that of FOXPRO prior to complete 

conversion to DMS.

 Flag data reported below MDL (e.g., ozone).

 Data should be easily retrievable both in the field and in the office.

 Time of filter weighing should be documented and visible in the electronic database.

 A system should be put in place to alert the PM lab analyst if the temperature goes out of 

range for short time periods (i.e., five minutes or less).
26

F. QUALITY CONTROL (QC) AND QUALITY ASSURANCE (QA)

Quality management is the system put in place that oversees quality assurance, quality 

control, and quality improvement activities. EPA requires that ambient air monitoring agencies 

have a quality management system that conforms to 40 CFR Part 50 Appendix A and the EPA 

quality policy (EPA Order CIO 2105.0). Additionally, USEPA grant regulations specifically 

require each grantee to provide for QA activities (40 CFR Part 31.45).

Specifically, 40 CFR Part 50 Appendix A Section 2 requires that each ambient air 

monitoring PQAO conforms to certain quality management practices. These include:

 Having a documented quality system that meets EPA requirements for QMPs and

QAPPs.

 Having a quality management function that is independent of air monitoring operations.

 Developing or adopting DQOs, or equivalent systematic planning procedures, for all 

monitoring programs.

 Participating in National Performance Evaluation Programs, which consist of 

performance audits used to independently determine program adequacy, national 

monitoring network performance, and national consistency.

 Submitting to Technical Systems Audits by EPA Regional Offices at a frequency of 

every three years or less.

 Using certified reference materials to standardize monitoring equipment.

EPA views the application of these quality management system components as integral to 

monitoring programs. Insufficient quality management and control has been cited as rationale to 

support legal challenges to NAAQS designation decisions. 

The BAAQMD quality management system meets the basic EPA requirements. 

BAAQMD has a QMP and QAPP that are approved by EPA Region 9. BAAQMD has 

implemented mechanisms to independently manage its quality system by assigning these duties 

to staff in the MET/PE section that are independent from data collection activities (with the 

exception of meteorological instrument calibrations). BAAQMD conforms to the data quality 

objectives and/or the systematic planning developed by EPA for criteria pollutants. EPA and

CARB perform national performance and technical evaluations of the monitoring network. 

Quality control and quality assurance are the two components of quality management 

within a monitoring program that support the assumption that the data they collect represent the 

true air quality of the area. They are the means by which an organization manages its quality 

aspects in a systematic, organized manner and provides a framework for planning, implementing, 

and assessing work performed by an organization. A properly developed QA/QC program 

encompasses a variety of technical and administrative elements, including policies and 

objectives, organizational authority, responsibilities, accountability, and procedures and 

practices.

Quality assurance is a management or oversight function; it deals with setting policy and 

running an administrative system of management controls that cover planning, implementation, 

the review of data collection activities, and the use of data in decision making. Quality control is 
27

a technical function that includes all the scientific precautions, such as calibrations and 

duplications that are needed to acquire data of known and adequate quality. 

The organization of BAAQMD’s Technical Services Division divides QA responsibility 

among various sections and various personnel: Laboratory QA Officer, QA Officer within the 

Meteorology/PE Section, and PE Supervisor within the MET/PE Section. All QA functions are 

embedded within the sections rather than having a formal QA role that oversees all ambient air 

monitoring operations at the management level. Most quality assurance functions, including the 

QA Officer for ambient air monitoring and the PE group, reside in the MET/PE Section, 

managed by Dick Duker. The QA Officer for ambient air monitoring, Mark Stoelting, also has 

responsibilities as an air and meteorology specialist and does not have authority to direct 

activities in the Laboratory Operations or Air Monitoring sections. The BAAQMD MET/PE

Section has a PE Group with three performance evaluation auditors that perform audits of the 

particulate and gaseous monitoring networks. The PE Group has also explored performance 

audits of BAAQMD Laboratory operations. The Meteorological and Performance Evaluation 

Section also performs audits of the meteorological monitoring stations periodically with an 

instrument specialist and an independent contractor. Audits of gaseous pollutants are generally 

conducted at a frequency that exceeds the minimum specified by regulation.

GENERAL QA/QC FINDINGS

FINDING F-QA1 (Major):

The BAAQMD QAPP’s approval is conditioned on establishing DQOs for data uses that are not 

defined by USEPA.

Discussion:

The USEPA has developed DQOs, or equivalent quality planning, for data uses that 

support NAAQS decisions and several other data uses (including NATTS and the STN). 

It was identified in the review of the QAPP that BAAQMD collects data that supports 

additional uses. In order to meet EPA’s requirements for systematic planning, 

BAAQMD must determine the data quality required for each data use and ensure that the 

data collected is of sufficient quality to support the manner in which it is being used.

Recommendation:

Work with data users inside BAAQMD to determine the quality needs to support their 

data uses and/or decisions. Revise the QAPP to reflect these uses and modify data 

collection activities as needed. 

 

FINDING F-QA2 (Major):

Most quality management functions are being performed; however, QA authority is not clearly 

defined. 

Discussion:
28

All ambient air monitoring agencies must demonstrate Independence of Quality 

Assurance Activities as outlined in 40 CFR Part 58, Appendix A, Section 2.2. The 

MET/PE Section manages the QMP, QAPP, SOPs, and data review. However, there is 

the perception that quality management responsibility and authority for other aspects of 

the air monitoring program are not the responsibility of the MET/PE Section. 

The MET/PE Office manager’s position is not identified as a QA Manager in the QAPP 

organization chart. Although it is implied that he is a QA Manager because he oversees 

the QA Officer’s position, his management authority is limited to the MET/PE Office. 40 

CFR Part 58, Appendix A, Section 2.2 states “the quality assurance management function 

must have sufficient technical expertise and management authority to conduct 

independent oversight and assure the implementation of the organization’s quality system 

relative to the ambient air quality monitoring program and should be organizationally 

independent of environmental generation activities.” This section also states “the 

monitoring organization’s quality system must have adequate resources both in personnel 

and funding to plan, implement, assess and report on the achievement of the requirements 

of this appendix and its approved QAPP.”

A QA Officer’s position is identified in the BAAQMD’s organization chart, however, 

QA functions are carried out by several individuals e.g., QA Officer, PE group, and 

Laboratory Services. While the organizational structure may remain the same, a QA 

manager needs to be designated to provide leadership to the previously mentioned 

positions and offices. The designated individual should have overall QA authority to 

conduct independent oversight and require corrective action. In order to enable the QA 

manager to function effectively, a QA manager should oversee rather than perform staff 

level duties (e.g., PM2.5 validation). 

In addition, the QA Manager should be given the authority to perform:

 Periodic audits of the quality management system (management system reviews), 

routine procedures, and data quality to identify areas of improvement and to 

ensure that monitoring programs continue to consistently follow sound and 

documented procedures, 

 A quality assurance training program and oversight of training for air monitoring 

and laboratory staff training, and

 Audits of data quality, including data tracking, especially following deployment 

of the new DMS.

In general, EPA QA/R-5, EPA Requirement for Quality Assurance Project Plans, 

discusses assessments that should be elements of a quality management program. These 

include, “surveillance, management systems reviews, readiness reviews, technical 

systems audits, performance evaluations, audits of data quality, and data quality 

assessments.”

Recommendation:

The MET/PE Section should be identified as the lead group for ambient air monitoring 

QA and all monitoring activities should include independent oversight either directly by 
29

MET/PE Section or with their concurrence. Further, it is recommended that BAAQMD 

identify a QA Management position in the QAPP and provide this position leadership 

authority and resources to conduct:

 independent reviews of the District’s overall Quality System i.e., Laboratory 

Service, Ambient and Meteorological Monitoring (including Field Operations), 

and Data Validation; 

 overall QA oversight for the District including corrective actions whether 

uncovered during internal audits or otherwise, e.g., through staff innovation that 

improves upon data defensibility. 

FINDING F-QA3 (Major):

BAAQMD does not have a formal corrective action system for corrective actions outside of the 

performance audit program.

Discussion:

EPA quality management standards (EPA QA/R-2, Quality Improvement Section) 

require that management and staff “ensure that conditions adverse to quality are” 

prevented, identified promptly, fully defined, corrected, prevented from recurring, and 

documented as corrective actions which are tracked to closure. There is a corrective 

action processes for the performance audit program. There should also be a formal, 

documented mechanism for elevating potentially significant corrective actions originating 

from the laboratory staff or field operators.

Recommendation:

The BAAQMD air monitoring program should include a formal corrective action process

for actions originating from the laboratory or field, in addition to the performance audit 

process. 

 

FINDING F-QA4:

BAAQMD should have a documented system to ensure that staff are trained on appropriate 

procedures and are familiar with the general QA system as defined in the QMP and QAPP(s).

Discussion:

BAAQMD staff appear to be trained and proficient with the procedures they use to 

conduct their monitoring activities. However, it was noted that this training and/or 

demonstration of proficiency was not adequately documented for either laboratory or 

field staff.

Copies of the QMP and QAPP are available on the BAAQMD network to all staff. 

However, there are no trainings or other mechanisms to ensure that staff understand the 

QA system as outlined in these documents.

Recommendation:
30

BAAQMD should develop a formal system to ensure and document that all staff are 

familiar with the quality management system and are trained and proficient at the 

monitoring tasks that they are performing.

 

FINDING F-QA5:

BAAQMD does not have a standard procedure to ensure that the performance contractors used to 

conduct or support monitoring are evaluated by BAAQMD QA staff.

Discussion:

In order to ensure that quality objectives are met, quality requirements for monitoring 

data must also apply to contractors involved in the data collection process. This is 

achieved by including quality needs in contracts and by evaluating the quality of the 

contractor’s work product.

BAAQMD uses a minimal amount of contract support for laboratory analysis and quality 

assurance tasks (e.g., meteorological audits). Normally BAAQMD does provide quality 

assurance oversight to these contractors. However, there is no procedure in place to 

assure that contract oversight is provided. Additionally, USEPA or other agencies may 

be already doing parallel oversight of these contractors and may be able to work jointly 

with BAAQMD (i.e., conduct audits and share contractor evaluations).

Recommendation:

BAAQMD should establish a procedure to ensure that the performance contractors used 

to conduct or support monitoring are evaluated by BAAQMD QA staff.

 

PERFORMANCE EVALUATION FINDINGS

FINDING F-PE1:

Audits are performed for NO2 by using ozone as the reference concentration instead of NO. 

Additionally, NO provided is in excess of recommendations.

Discussion:

No specific NO2 titration procedure is specified in 40 CRF Part 58, Appendix A, Section 

3.2. However USEPA specifies that NO2 should be calibrated using the NO original and 

NO residual concentrations in 40 CFR Part 50, Appendix F, Section 1.5.9. While this 

procedure is only required for calibrations, alternative audit procedures should be 

validated. Additionally, sub-section 3.2.2.2 states that “NO concentrations substantially 

higher than 0.08 ppm, as may occur when using some gas phase titration (GPT) 

techniques, may lead to evaluation errors in chemiluminescence analyzers due to 

inevitable minor NO–NOx channel imbalance.”

Recommendation:

BAAQMD should evaluate the NO2 audit procedure to ensure and document that the 

procedure being used is equivalent or better than the procedure recommended by EPA. 
31

FINDING F-PE2:

It was noted during data tracking that the manufacturer’s certification for Delta Cal 249 had the 

wrong ID.

Discussion:

Traceability of standard certification is a key component of the air monitoring program. 

A review of the certification documentation indicated that the manufacturer had 

inadvertently provided the wrong instrument identification in 2008 for Delta Cal 249. 

The PE Group confirmed that this had been noted when the instrument returned from 

certification. However, this error was not corrected on the certification documentation 

and the manufacturer was not requested to provide a “clean” certification document.

Recommendation:

The certification documentation should be annotated to explain and correct this error. In 

the future errors in standard certification should be corrected by the certifying entity.

 

FINDING F-PE3:

The 2008 calibration for the DPI 70524767 pressure monitor was not in the instrument logbook.

Discussion:

During the data tracking process USEPA noted that the logbook for the DPI 70524767 

pressure standard did not include the calibration performed in 2008. It was indicated that 

this information was available on the auditors’ computer.

Recommendation:

Instrument calibration information should be entered into the logbook when the 

calibration is performed, or when calibration information is received to avoid inadvertent 

omissions.

 
32

APPENDIX A: SUMMARY OF FINDINGS

MAJOR FINDINGS

Finding A1: BAAQMD has had significant quality issues with the Thermo Scientific Partisol 

FRM, Model 2000. 

Finding D-T1: BAAQMD lacks a QA Officer dedicated to overseeing quality management of 

the laboratory.

Finding F-QA1: The BAAQMD QAPP’s approval is conditioned on establishing DQOs for 

data uses that are not defined by USEPA.

Finding F-QA2: Most quality management functions are being performed; however, QA 

authority is not clearly defined.

Finding F-QA3: BAAQMD does not have a formal corrective action system for corrective 

actions outside of the performance audit program.

MINOR FINDINGS

Finding A2: BAAQMD keeps multiple hard copies and electronic copies of many documents, 

making it unclear which is the official copy.

Finding C1: PM10 flow checks may not be occurring at the required frequency at all sites.

Finding C2: BAAQMD operates many of their gaseous monitors using sample pumps that 

deviate from the reference method.

Finding C3: Several different SOPs appear to apply to instrument calibrations that are not 

consistent. Additionally, documentation of calibration was hard to follow and incomplete.

Finding C4: The level of information recorded into the station logbooks is not consistent and at 

some sites the logbooks are not frequently used.

Finding C5: Toxics samplers are not routinely verified by checking with blanks and/or 

standards for contamination or losses.

Finding C6: No instrument SOPs were available at some sites with limited monitors (e.g., 

Fairfield and Gilroy).

Finding D-T2: Many of the SOPs do not reflect actual laboratory practices.

Finding D-T3: The TO-14 standard certification and stability are uncertain.
33

Finding D-T4: Canister certification for cleanliness is not performed according to the 

"Technical Assistance Document for Sampling and Analysis of Ozone Precursors" and the 

BAAQMD SOP. 

Finding D-T5: The laboratory has not established a retention time for canisters after they have 

been certified beyond which they must be recertified. 

Finding D-T6: Documentation and record-keeping in the laboratory does not fully capture 

laboratory activities, was found to have errors, and is not organized for easy tracking and 

retrieval of information.

Finding D-T7: Secondary review of records was found to be irregular. Approximately six 

month intervals between secondary reviews were observed in some instances.

Finding D-T8: Laboratory space is constrained and may be inadequate once the additional 

capabilities such as GC-MS that are on-line.

Finding D-T9: The laboratory is not tracking the retention time of sample cartridges to ensure 

they are used within six months of the manufacturer's certification analysis as recommended by 

the manufacturer.

Finding D-PM1: BAAQMD does not perform periodic field blanks for PM10.

Finding D-PM2: The records of the Polonium strips were not retained and they could not be 

verified.

Finding D-PM3: The PM2.5 filter associated with the data point tracked for January 4, 2008 at 

Livermore was invalidated because it was outside the holding time.

Finding E1: Finding: SOPs 601-609 for data management are general and do not provide stepby-step instruction of how data are actually validated by the data reviewers (validators) in 

Ambient Monitoring, Laboratory Services, MET/PE, and Planning Sections.

Finding E2: Questions raised by data validators in the MET/PE to the Ambient Monitoring or 

Laboratory Services are not uniformly documented. Some validators are not documenting the 

discussions made for determining data quality or for changing data points.

Finding F-QA4: BAAQMD should have a documented system to ensure that staff are trained 

on appropriate procedures and are familiar with the general QA system as defined in the QMP 

and QAPP(s).

Finding F-QA5: BAAQMD does not have a standard procedure to ensure that the performance 

contractors used to conduct or support monitoring are evaluated by BAAQMD QA staff.

Finding F-PE1: Audits are performed for NO2 by using ozone as the reference concentration 

instead of NO. Additionally, NO provided is in excess of recommendations.
34

Finding F-PE2: It was noted during data tracking that the manufacturer’s certification for Delta 

Cal 249 had the wrong ID.

Finding F-PE3: The 2008 calibration for the DPI 70524767 pressure monitor was not in the 

instrument logbook.
35

APPENDIX B: GOOD PRACTICE RECOMMENDATIONS

NETWORK RECOMMENDATION 1:

The BAAQMD has not developed a schedule for implementation of the Lead monitoring 

program.

Discussion:

USEPA establish new monitoring requirements for Lead beginning in January, 2010 and 

January, 2011. Sites within BAAQMD’s jurisdiction will need to be operational by 

January, 2011. In order to meet this schedule, agencies will need to establish programs 

including establishing approved QAPPs and FEM for laboratory analysis prior to the 

commencement of monitoring.

 Recommendation:

BAAQMD should develop a schedule that ensures that pre-monitoring testing and the 

appropriate supporting documentation and approvals are in place prior to the 

commencement of monitoring.

FIELD RECOMMENDATION 2:

The trees adjacent to the Napa site (behind the building to the east) are starting to encroach on 

the probe and will need to be trimmed soon. 

Discussion:

USEPA siting criteria require that trees be at least 10 meters from instrument inlets and at 

least 20 meters when the trees act as an obstruction. The trees at the Napa site are 

currently just below the height of the probe at a distance of approximately five meters.

 Recommendation:

BAAQMD should continue to ensure that the trees at the Napa site are routinely trimmed.

FIELD RECOMMENDATION 3:

The San Jose Jackson Street site requires an electrical upgrade.

Discussion:

The San Jose Jackson site is nearing the limit of the electrical capacity for that site. The 

current system may not accommodate the addition of all NCore instruments.

Recommendation:

Upgrade the San Jose Jackson Street electrical system to accommodate additional NCore 

requirements.

FIELD RECOMMENDATION 4:
36

There is no system for tracking and controlling station and instrument logbooks.

Discussion:

Field procedures require that logbooks be kept. However, these logbooks were not 

tracked, identified, and archived in a manner to ensure that the critical documentation 

they contain will be accessible and defensible.

Recommendation:

Develop a standardized procedure for creating, labeling, and archiving logbooks.

FIELD RECOMMENDATION 5:

The Ambient Monitoring SOP 101, Station Operator, Section IV, Data Retrieval and Review, 

Section 5.e instructions does not contain example calculations.

Discussion:

The captioned SOP states that the percentage chart values should be converted to 

concentrations (PPM) using the most recent calibration curve. No information is 

provided as to how to perform this conversion.

 Recommendation:

Provide one or two example calculations to ensure calculations are properly performed.

FIELD RECOMMENDATION 6:

Not all meteorological stations have sufficient power to store and transmit collected data.

Discussion:

Meteorological stations need sufficient power to be provided so that data capture can 

meet 75% completeness criteria by January, 2011 at the NCore site. This would be a 

good practice at all the meteorological sites.

Recommendation:

Ensure sufficient power to Meteorological stations to ensure data capture of 75% 

completeness.

FIELD RECOMMENDATION 8:

Strip charts and daily printouts were not present at the Fairfield site.

Discussion:

Charts and printouts for Fairfield are currently maintained at the Concord monitoring 

station. Charts should be maintained at the site, similar to instruments logs (i.e., 

instrument logs follow the instrument), to compare current results from historical 

responses.
37

 Recommendation:

Maintain charts and daily printouts at each site.

LAB RECOMMENDATION 1:

Inter-laboratory comparison sample studies are being conducted with CARB, but a formal 

schedule and frequency for reviewing and documenting the results and a follow up guideline has 

not been established.

Discussion:

The analysis of samples collected every twelve days as concurrent samples analyzed by 

CARB and BAAQMD provides useful quality assurance information concerning the 

accuracy of analyses. The review of results is informal and irregular due to time 

constraints. Laboratory staff stated that the results for samples collected from the same 

manifold and analyzed by CARB and BAAQMD usually compare well but occasionally 

they do not. A guideline criterion for apparent outliers has not been established that

would trigger follow up and communication with the CARB laboratory and internal 

analyst staff and any corrective action and documentation. Divergent results should 

trigger review of the analytical system by both laboratories to ensure they are in control. 

Method TO-11 specifies that results should fall within a ± 20% relative percent 

difference. The results of investigations into outliers should be documented.

 Recommendation:

It is recommended that a policy of timely and formal review of inter-laboratory sample 

comparison results be established and that conclusions from the review and any 

corrective actions be documented in a dedicated QA file along with documentation of any 

follow up communications with CARB or the BAAQMD analyst.

LAB RECOMMENDATION 2:

The expiration date for laboratory standards provided by the manufacturer is used to determine 

when standards used to calibrate the HPLC instrument need to be replaced. An expiration date 

for standards is not currently based on the date the sealed standard vial was first opened. In 

addition, the standard vial in current use was not initialed by the analyst who opened it.

Discussion:

A new time clock for expiration should be established whenever a sealed standard vial 

from the manufacturer is first opened and exposed to the air and it becomes a working 

standard. This expiration date calculated from the time the vial is first opened will be 

shorter than the expiration date provided by the manufacturer which only applies to 

sealed, ampulated materials and not opened vial materials that are exposed to air and 

more subject to breakdown and solvent loss.

 Recommendation:
38

It is recommended that a policy for calculating expiration dates for standards from the 

date the standard vials are opened be established. The date the standard is opened and the 

(new) expiration should be documented in a dedicated standards logbook. Vials should 

be initialed by the person opening the standard along with the date and new expiration 

date.

LAB RECOMMENDATION 3:

BAAQMD does not have a formal system to notify the laboratory when PM filter sample 

custody has been transferred from the Air Monitoring Section to the Laboratory Services 

Section.

Discussion:

Sample custody is a process that ensures the integrity of samples and sampling media. 

Under a formal custody procedure, samples must be in the control of a single individual 

until formally relinquished to the next individual. This process is documented and 

verifiable. Because ambient air samples are not normally intended to be used to support 

enforcement, custody can be somewhat less formal. However, the custody process 

should still ensure sample integrity and help to manage sample processing.

The Air Monitoring Section routinely delivers exposed samples to the PM Laboratory by 

placing them in a shared freezer in the laboratory area. However, there is no formal 

procedure for notifying the laboratory of when and what samples are available to begin 

the weighing process. The PM analyst indicated that normally there is a verbal 

notification of sample delivery.

Recommendation:

The Air Monitoring Section and the Laboratory Services Section should develop a more 

formal mechanism to transfer samples ready for laboratory analysis.

DATA RECOMMENDATION 1:

There is a significant time lag for meteorological data validation. 

Discussion:

Most of the meteorological data validation responsibilities reside with the Planning 

Office’s validator, who is not in the Technical Services Division. The Planning Office’s 

meterological data validator’s primary responsibilities and priorities are with planning, 

not validation, and so he can only validate meteorological data on a time-permitting 

basis. This has resulted in a two year lag in meteorological data validation. According to 

BAAQMD’s Quality Management Plan, meteorological data is used by the District and 

public for forecasting, photochemical modeling, source modeling, and data analysis. It is 

not clear how accurate forecasting, modeling, analysis can be performed using data of 

unknown quality (i.e., unvalidated data). While users of the data are warned of the risk, 

the delay in review of data by two years is a significant concern. 
39

Per 40 CFR Part 58.13(a) The network of NCore multipollutant sites must be physically 

established no later than January 1, 2011, and at that time, operating under all of the 

requirements of this part, including the requirements of appendices A, C, D, E, and G to 

this part. Data submittal and archiving requirements for meteorological data is quarterly 

as established at §58.16. The ability to meet completeness criteria in 2011 will suffer if 

validation continues to be performed on a time permitting basis. Further, meteorological 

monitoring is a requirement of NCore and validated data will have to be submitted to 

AQS quarterly.

 Recommendation:

In order to prepare for NCore reporting in January 2011, establish a process to 

accomplish timely and accurate meteorological data validation and reporting.

DATA RECOMMENDATION 2:

Validators do not always have access to field notes (PM2.5 and PM10) and initial meteorological

QC checks for making data quality determinations.

Discussion:

Readily available field notes makes for efficient data validation. Validators currently 

contact relevant individuals who maintain field notes or initial QC check info. Where the 

field station operators notes are insufficient to clarify a concern, there was no hesitation 

by validators to make contact with field station operators or their Supervisors to address 

concern.

 Recommendation:

Provide copies of field station operators chain of custodies and initial meteorological QC 

checks to validators who have a need for this information in performing validation. If 

several parties need access, it may be cost effective for completed chain of custody (e.g., 

PM) and other relevant information to be scanned into the mainframe for access by 

interested parties.

DATA RECOMMENDATION 3:

The evaluation of data quality trends by the MET/PE Section does not include evaluation of data 

quality over three year intervals.

Discussion:

The primary decisions made with criteria pollutant data is comparison to the NAAQS. 

Three years of data is used to compare to the NAAQS. Therefore, evaluation of the 

DQOs and the quality assurance measures that support the DQOs should be made over 

the same time interval.

The BAAQMD MET/PE Section reviews quality assurance data and trends. However, 

staff do not specifically evaluate trends over three years.
40

 Recommendation:

The MET/PE Section should specifically evaluate quality assurance data in intervals of 

three years in addition to other intervals.

DATA RECOMMENDATION 4:

Null code flags are not always being identified in AQS.

Discussion:

Quality Assurance Handbook for Air Pollution Measurement Systems, Volume 2, Section 

17.3 http://www.epa.gov/ttn/amtic/files/ambient/pm25/qa/QA-Handbook-Vol-II.pdf), 

states: " NOTE: it is strongly suggested that flags, specifically the appropriate null data 

code flags, be used in place of any routine values that are invalidated. This provides some 

indication to data users and data quality assessors to the reasons why data that were 

expected to be collected were missing."

Recommendation:

Provide null codes for gaseous data that have been invalidated and removed when 

reported to AQS.

QA RECOMMENDATION 1:

BAAQMD performance audit corrective action process should include a section that documents 

how re-occurrence of problems will be prevented.

Discussion:

The primary purpose of the quality system corrective action process is to correct quality 

deficiencies and to prevent future deficiencies. The BAAQMD uses an ODAMN report 

request to initiate and track corrective action resulting from a performance audit. 

However these forms do not have a specific place for the discussion as to how a 

deficiency will be prevented in the future.

 Recommendation:

Modify the ODAMN form to include a specific discussion of how future deficiencies will 

be prevented.

QA RECOMMENDATION 2:

ODAMNs do not record the assessment of the individuals performing review of corrective 

actions performed. Superceded ODAMNs were still in use, despite revision on February 10, 

2009. 

Discussion:

ODAMNs were recently revised to include a summary assessment by the Air Monitoring 

Manager and MET/PE Manger of the corrective actions taken to address any findings 
41

identified (adequate, inadequate, etc.). Entries were not completed by these individuals 

for ODAMN dated 02/10/09 for Livermore #1023. Superceded ODAMNs dated 

02/09/09 and older had signatures by several individuals, including Air Monitoring and 

MET/PE manager, but no assessment of the corrective actions taken were documented. 

ODAMN dated 02/26/09 was recorded on superceded format. 

 Recommendation:

Record the assessments of individuals performing review of corrective actions 

performed. Include the revision date and the version on recently revised ODAMN. 

Archive superceded ODAMN as historical document.

QA RECOMMENDATION 3:

The Director of Technical Services resolutions to QA related concerns are not documented.

Discussion:

Decisions made by the Director related to QA are undocumented. Whether the Director’s 

decisions are the result of resolving differences in opinion between the data generating 

and reviewing offices of his directive, they should be documented.

 Recommendation:

Develop a form similar to ODAMN that identifies date, concern, resolution, date of 

resolution or issuance of new directive by Director. Should changes be necessary to any

part of the data collection operations, information documented in these forms should be 

communicated to relevant, affected staff.

PE RECOMMENDATION 1:

Due to differences in sampler design the PM2.5 Partisol sampler’s temperature can only be 

audited at a single point.

Discussion:

The procedure of PM2.5 sampler audits, written for an Anderson sampler, indicates that 

multi-point temperature sensor audit will be performed. USEPA agrees that this is a 

good practice.

 Recommendation:

BAAQMD should take steps to evaluated Partisol temperature sensor performance across 

the operating range (e.g., track the temperatures at which the audits were performed).

PE RECOMMENDATION 2:

During the field audit it was noted that, as expected, the delivery of audit gases produced a 

slightly positive pressure on the station manifold.
42

Discussion:

USEPA has observed that, when delivering gases through the probe, or to the manifold of 

a monitoring station, excess positive pressure can bias audit results. Each BAAQMD 

station manifold is equipped with a pressure monitor (Magnehelic) and the auditor and 

station operator monitor manifold pressure during audits. However, there is no criterion 

for pressurizing the manifold.

 Recommendation:

The PE Group should have a criterion to prevent over-pressurizing the monitoring 

system.