Office of Inspector General

Recent Additions | Contact Us | Print Version Search:

EPA Home > Office of Inspector General > Audits, Evaluations & Other Publications >Land

OIG Home

Our Organization

Audits, Evaluations

& Public Liaison

Reports

Reports to Congress

Planning &

Performance

Results

Ombudsman

& Hotline

Office of Audit

Office of

Investigations

Office of Program

Evaluation

Compendium of

Environmental

Programs

Freedom of

Information Act

Career Opportunities

Helpful Links

How to Contact Us

Environmental Data Quality at DOD Superfund Sites in

Region 9

TABLE OF CONTENTS

Executive Summary

CHAPTERS

1 - Introduction

Purpose

Background

Scope and Methodology

Prior Audit Coverage

2 - Laboratory Quality Assurance Oversight at DOD Sites Needs Improvement

3 - Sacramento Army Depot

4 - March Air Force Base

5 - Travis Air Force Base

6 - Hunters Point Naval Shipyard

7 - Luke Air Force Base

APPENDICES

Appendix A - Region 9 Response to Draft Report

Appendix B - OIG Evaluation of Region 9 Response

Appendix C - Superfund Cleanup Process

Appendix D - Regulations, Orders, Directives, and Guidance

Appendix E - Activities Contacted During the Audit

Appendix F - EPA Data Analytical Levels

Appendix G - Definitions of Quality Assurance Techniques

Appendix H - Acronyms

Appendix I - Report Distribution

Chapter 1

Introduction

PURPOSE

EPA has an important role in overseeing Department of Defense's (DOD) Superfund cleanups. This role

includes ensuring that laboratory data is of acceptable quality. EPA is also responsible for expediting

cleanups at closing and downsizing bases by resolving critical technical issues, such as those with

1 of 53 06/17/13 07:36 AM
laboratory data quality problems, before they become impediments to cleanup and reuse.

In 1992, serious problems began surfacing with the quality of laboratory data used in the cleanup of DOD

Superfund sites located in Region 9 (the Region). These problems caused millions of dollars of laboratory

data to be rejected and cleanups to be delayed.

The objective of the audit was to determine if the Region was ensuring that laboratory data was of known

and acceptable quality under Federal facility agreements with DOD.

BACKGROUND

Characterized as the nation's largest industrial organization, DOD has 100 bases on the EPA's Superfund

National Priorities List (NPL), a register of the nation's worst contaminated hazardous waste sites.

Twenty-four of these bases are located within the Region. DOD bases received $2.7 billion in Fiscal 1995

to cleanup their contaminated sites.

Nearly half of the NPL bases in the Region are scheduled for closure. However, hazardous waste

contamination on many of these bases must be cleaned up before the property can be returned to the

community for economic development.

For decades, DOD facilities generated, stored, and disposed of hazardous wastes, which often

contaminated nearby soil and groundwater. Hazardous waste contamination can significantly contribute to

serious illness or death, or pose a hazard to the environment. The types of hazardous wastes found at

most DOD installations include solvents and corrosives; paint strippers and thinners; metals, such as lead,

cadmium, and chromium; and unexploded ordnance. Contamination usually results from disposal, leaks, or

spills

Under the Comprehensive Environmental Response, Compensation, and Liability Act and its amendments,

commonly known as Superfund, DOD is required to carry out its hazardous waste cleanups according to

the same guidelines as other facilities. Executive Order 12580 further delegates certain cleanup

authorities under Superfund to DOD.

Federal Facility Agreements

Once a base is placed on the NPL, EPA, DOD, and the state must negotiate a Federal facility agreement

to govern the cleanup. The state is a party to the agreement because Superfund requires Federal

agencies to consult with states regarding cleanups. Also, Federal agencies must comply with state

environmental laws, which sometimes are more stringent than Federal rules.

Federal facility agreements set requirements and enforceable schedules for completing studies, reports,

and cleanup records of decision. EPA is responsible for overseeing these agreements and has final

approval authority over fundamental documents, such as the risk assessments and the feasibility studies.

It also has final decision-making authority for selecting, on the basis of environmental data, the cleanup

remedy. EPA also oversees cleanups at closing bases not on the NPL. (The Superfund cleanup process is

described in Appendix C of this report.)

Environmental Data Quality Requirements

Environmental data is obtained primarily by sampling contaminated water, soil, air, and other materials,

and having the samples analyzed by a laboratory. EPA Order 5360.1 requires environmental

measurements to be of known quality, verifiable, and defensible. The quality of environmental data may

be adversely impacted by weaknesses in sampling, laboratory analysis, and the validation of results. Poor

quality data can negatively impact or delay the decision making process. Further, incorrect decisions can

lead to inadequate health protection or expenditures for unneeded cleanup remedies.

The primary goal of the QA program is to ensure that all environmentally related measurements...

[Laboratory Analysis] produce data of known quality. The quality of the data is known when all

components...are thoroughly documented, such documentation being verifiable and defensible.

-EPA Order 5360.1

In focusing our audit on laboratory data quality activities, we determined that, unlike EPA, DOD does not

have a centrally-managed contract laboratory program for monitoring laboratory quality. Typically, DOD's

cleanup efforts are carried out by contractors, who procure the laboratory services.

Steps to Ensure Laboratory Analyses Quality

There are two major steps to ensure the quality of laboratory analyses for a site.

2 of 53 06/17/13 07:36 AM
First, data quality objectives must be determined. Such objectives define how data will be used, and

establish corresponding quality objectives before data is collected, thereby resulting in a defensible

decision-making process.

Second, a quality assurance project plan (QAPP) must be developed in accordance with 40 CFR

300.430. The quality assurance activities necessary to achieve the data quality objectives are

incorporated into a QAPP. This plan is a blueprint for ensuring the laboratory analyses produce data of

sufficient quality and quantity for decision-making.

EPA's Oversight Responsibilities

EPA's regulations and EPA Office of Solid Waste and Emergency Response (OSWER) directives define

regional oversight responsibilities in the Federal facility area. (The applicable guidance is listed in

Appendix D of this report.)

In overseeing a Superfund cleanup, EPA is responsible for:

Ensuring environmental data is of known and acceptable quality for decision making;

Approving the QAPP and ensuring that it is acceptable; and

Verifying that the QAPP is implemented.

The Hazardous Waste Management Division is responsible for implementing the Superfund program in the

Region. The Division's Federal Facilities Cleanup Office works with Federal, state and local agencies to

oversee investigations and cleanups at Federal facilities. It is also responsible for initiating enforcement

actions.

Regional

Administrator

Hazardous Office

Waste Of Policy

Management Division And Management

Federal Environmental

Facilities Services

Cleanup Office Branch

The Region's Environmental Services Branch, under the Office of Policy and Management, helps ensure

that "key environmental decisions...are based on environmental measurements of the correct type and of

sufficient quality and quantity." It reviews QAPPs and provides technical assistance to program offices.

SCOPE AND METHODOLOGY

We performed our audit in accordance with Government Auditing Standards issued by the Comptroller

General. Our field work was conducted between November 17, 1994 and June 30, 1995. Our audit

covered the period Fiscal 1991 to 1994 and management procedures in effect as of December 31, 1994.

We reviewed the Region's oversight of laboratory data quality assurance activities at 5 of the 24 DOD

bases on the NPL. This included the following facilities:

March Air Force Base;

Travis Air Force Base;

Hunters Point Naval Shipyard;

Luke Air Force Base; and

Sacramento Army Depot.

The first four facilities were selected because of known or possible problems with the quality of laboratory

data. The Sacramento Army Depot was selected because some data validation procedural problems

were noted in our audit survey. In addition, some limited audit survey work was done at Mather Air Force

Base.

Internal Controls

We reviewed the internal controls associated with the regional oversight of laboratory data used by DOD

3 of 53 06/17/13 07:36 AM
facilities, including the Federal facility agreements, the QAPPs, and implementing guidance. The internal

control weaknesses noted in our audit are described in Chapters 2 and 3 of this report, along with

recommendations for corrective action.

FMFIA

In planning our audit, we reviewed EPA's 1994 Integrity Act Report to The President and Congress,

issued in December 1994. This report, which documents EPA's compliance with the Federal Managers'

Financial Integrity Act, identified environmental data quality as an existing material weakness. It also

stated that previously reported problems with EPA's oversight of Federal facility cleanups had been

corrected. Our audit report identifies additional problems related to EPA's oversight of Federal facility

cleanups.

To accomplish our audit objective, we obtained:

Applicable laws, regulations, directives, and other guidance, including DOD laboratory quality assurance

guidance;

Federal facility agreements;

QAPPs and related review comments; and

Evidence of quality assurance oversight.

We interviewed responsible EPA, DOD, and state officials. (A complete list of the entities contacted

during the audit is included in Appendix E of this report.)

To evaluate the effectiveness of laboratory quality assurance, we identified quality assurance activities

used in EPA's Contract Laboratory Program, and those activities that detected laboratory data problems

at DOD sites.

These activities were used to evaluate the adequacy of the QAPPs. We determined compliance with key

quality assurance activities in plans by reviewing supporting documentation. The audit also identified other

key quality assurance activities that were not called for in the plans.

We also evaluated regional procedures for improving laboratory data quality at Federal facilities.

PRIOR AUDIT COVERAGE

There have been no prior audits of environmental laboratory quality at DOD by either the EPA Office of

Inspector General, the Department of Defense Office of Inspector General, the General Accounting

Office, or the military service audit agencies. However, in a related audit, the Air Force Audit Agency

evaluated environmental contract oversight at selected bases. Its report, issued on November 16, 1994,

determined that Air Force oversight of environmental compliance contracts was not adequate.

Chapter 2

Laboratory Quality Assurance Oversight At DOD Sites Needs Improvement

SUMMARY

In 1992, serious problems began surfacing with laboratory analyses related to Superfund cleanups at

DOD sites in Region 9. At the same time, a regional survey identified quality assurance deficiencies at

Federal facilities in the Region. However, at the time of our review, the Region had not significantly

strengthened its laboratory data quality oversight program at its DOD NPL sites.

The integrity of the data used to make cleanup decisions cannot be compromised if we are to

successfully fulfill our joint responsibilities to protect public health and the environment.

-Region 9's Federal Facilities Cleanup Office in a May 1994 letter to DOD

QAPPs, the primary tool for controlling laboratory quality, were generally not designed to detect the

laboratory quality problems found at the five DOD sites reviewed. We also found that QAPPs were not

always officially approved or fully implemented. Further, the Region did not effectively monitor compliance

with QAPPs or perform sufficient oversight to determine the quality of laboratory data. In addition, QAPP

inadequacies were not corrected after data problems were found. We found many of these problems

manifested at Sacramento Army Depot, resulting in serious data quality deficiencies. In our opinion, the

extent of these deficiencies compromise the validity of the determination that a portion of the Depot had

been cleaned up.

4 of 53 06/17/13 07:36 AM
We also believe that a more effective quality assurance oversight program would have avoided a portion

or all of the $5.5 million of rejected laboratory work and cleanup delays of up to 2 years.

DOD'S LABORATORY PROBLEMS

In 1992, the Region became aware of serious problems with the quality of laboratory work performed by

DOD contractors. In May 1992, the Navy informed the Region that $2.5 million of data analyzed for the

Hunters Point Naval Shipyard cleanup had been rejected. In September 1992, the Air Force informed the

Region that there were significant problems with the laboratory analyses for Travis Air Force Base (AFB).

Subsequently, EPA determined that $2 million of data was of unknown quality, setting the Travis AFB

cleanup back 2 years.

Problems with Eureka Laboratories

In January 1993, the Region identified significant problems with laboratory analyses for the March AFB

cleanup, eventually resulting in the rejection of $1 million of data. The laboratory performing the poor

analyses for March Air Force Base, Eureka Laboratories, Inc., was eventually convicted of fraud based

on its analyses for March AFB and another DOD site in Region 8.

Commendably, the Region took aggressive action to help prevent further data problems once the Eureka

analyses deficiencies were discovered. At the Region's request, EPA suspended Eureka Laboratories in

December 1993 from all future Federal direct procurements. In May 1994, the Region wrote to DOD

asking for a "...comprehensive analysis of the impacts of the Eureka case...in Region 9..." In July 1994,

Eureka Laboratories and two of its managers were indicted on 71 counts of fraud.

In August 1994, DOD responded to the Region's request, and identified 24 DOD installations that used

Eureka Laboratories in the Region since 1989. However, DOD's response was not a complete analysis of

the use of Eureka Laboratories' data. For example, DOD reported that Tracy Defense Distribution Region

used Eureka Laboratories. However, DOD did not identify the extent of the laboratory's work done for

Tracy or steps taken to ensure the reliability of data.

Eureka confirmed, by letter to ... the prime contractor, that none of the analyses were under question by

EPA or were subject to fraudulent practices.

-Letter from DOD to Region 9,

August 12, 1994

The steps DOD reported it took to determine the reliability of data at other sites were clearly inadequate.

For example, DOD said the Naval Station Guam and Pearl Harbor Naval Shipyard were not affected by

the Eureka Laboratories problems. They indicated that Eureka Laboratories confirmed that none of the

analyses for these bases were under question by EPA or were subject to any fraudulent practices. It is

logical to assume that a laboratory under investigation for fraud would not admit to any fraudulent

practices.

Also, DOD's response for these two sites did not indicate what types of cleanup decisions the data was

used for. Additionally, DOD did not identify how it verified that the data was valid. Further, DOD failed to

report a least three installations that used Eureka Laboratories.

In spite of DOD's incomplete response to the Eureka concerns, the Region did not request any further

data from DOD. In our opinion, the impacts of the Eureka fraud need to be accurately determined to

ensure that the laboratory data at affected sites is of the quality needed for decision making.

RELIABILITY OF DOD'S LABORATORY QA PROGRAM UNKNOWN

We also found that the Region's remedial project managers were unfamiliar with DOD's specific

laboratory quality assurance procedures. The Region explained that DOD had the principal responsibility

for data quality assurance. However, the Region is ultimately responsible for ensuring data is of known

quality. Thus, the Region must perform sufficient oversight to ascertain the quality of the laboratory data.

It shall be the policy of all EPA organizational units to ensure that ... environmentally related

measurements are of known quality.

-EPA Order 5360.1

The degree of regional oversight of DOD sites should depend on the effectiveness of DOD's laboratory

quality assurance program. If DOD has an effective quality program, the Region can reduce its oversight

procedures. Conversely, if DOD has a marginal quality assurance program, the Region needs to increase

5 of 53 06/17/13 07:36 AM
its oversight efforts.

We found the Region accepted at face value the description of laboratory quality assurance procedures

provided by the DOD activities. However, our review disclosed that these procedures were not accurately

portrayed. For example, DOD purported that a laboratory must successfully analyze performance

evaluation samples before it receives samples. In actuality, performance evaluation samples were not

always used. Further, laboratories were provided more than two opportunities to "pass" a performance

evaluation sample.

We also question whether DOD's quality assurance procedures can be relied upon. This was illustrated

by the results of laboratory audits of Eureka Laboratories.

Standard operating procedures...are available throughout...[Eureka Laboratories]. However, most of them

are incomplete (with regards to QC and calibration requirements and acceptance criteria) and

unapproved... Metals preparation, sample custody, and GC [gas chromatography]-pesticide SOPs do not

represent what is actually being done...

-Air Force contractor,

February 11, 1991

An Air Force laboratory audit found major problems with Eureka Laboratories in January 1991 and again

in October 1992. Apparently unaware of these problems, the Army Corps of Engineers (the Corps)

validated Eureka Laboratories for an 18-month period starting January 1992. The validation was made

after determining that performance evaluation samples successfully passed without any other major or

minor deficiencies.

A Navy contractor also reviewed Eureka Laboratories in April 1991 and determined that Eureka

Laboratories:

Has both the technical ability and sample capacity to meet the analytical and reporting requirements for

the volume of work proposed for Navy...projects.

Subsequently, both the Army and Navy used Eureka Laboratories. In our opinion, the obvious difference in

quality of laboratory audits between the military services impacts the degree of oversight the Region must

perform.

No Tracking System

In 1986, EPA's Office of Solid Waste and Emergency Response issued a directive which required the

oversight and monitoring of all Superfund laboratories, not just those under EPA contract. However, to

date, the Region does not have a data base or tracking system to identify and oversee laboratories used

at DOD NPL sites. To illustrate, when the Eureka Laboratories fraud was discovered, the Region could

not determine the extent that Eureka's laboratories had been used at DOD NPL sites.

We believe this is a serious deficiency that warrants the development of some type of data base tracking

system for Federal sites which identifies the laboratory used, the quality assurance measures performed,

and any data quality deficiencies.

REGIONAL SURVEY IDENTIFIES LABORATORY QUALITY ASSURANCE DEFICIENCIES

In a 1992 survey of laboratory quality assurance oversight practices, the Region found vulnerabilities in

Federal facility Superfund cleanups. The survey found that data validation, split sampling and performance

evaluation sampling were used to a much lesser degree at Federal facility sites than at EPA lead sites.

The survey concluded that:

...the ultimate quality of the data generated was not known at a number of sites. As demonstrated by

recent events at Hunters Point, this is a program vulnerability.

In spite of this acknowledged vulnerability, we were advised, at the beginning of the audit, that no actions

had been taken as a result of the survey. To illustrate, at Sacramento Army Depot, we found that since

1992 the Region had not performed any independent oversight of laboratory quality using procedures

such as data validation, performance evaluation samples, or magnetic tape audits.

QAPPS LACKING EVIDENCE OF EPA APPROVAL

We found the Region frequently could not provide evidence that the QAPPs were approved. Both 40 CFR

300 and EPA Order 5360.1 require EPA approval of the QAPPs. Regional guidance document no.

9QA-03-89 calls for both the Region's remedial project manager and quality assurance officer to approve

6 of 53 06/17/13 07:36 AM
each QAPP.

We reviewed 12 QAPPs for the 5 DOD bases in our audit. The review identified that only two QAPPs had

documented approvals by both the regional program office and the Environmental Services Branch's

Quality Assurance Section, which was responsible for quality assurance. Both of these QAPPs were for

Sacramento Army Depot. Also, a QAPP for the Groundwater Operable Unit at the Depot was not

prepared.

Unlike other investigations currently being conducted at [Sacramento Army Depot], ...the ground water

investigation therefore lacks a Quality Assurance Project Plan.

-Letter from Region 9's Environmental Services Contractor, December 1989

The Region advised us that it did not need formal documentation to approve QAPPs. It pointed out that

under Federal facility agreements, QAPPs will automatically be considered "final" documents if the Region

did not provide comments to DOD within its 30-day review period.

In our opinion, formal approval is essential for a couple of reasons. First, without formal approval, it is not

clear if the QAPP being used is the most current version. In fact, a recently issued EPA quality assurance

regulation (EPA QA/R-5) calls for making sure all personnel involved in the cleanup work have copies of

the approved QAPP. We found several versions of QAPPs during our reviews, but it was not apparent

which one was the "approved" version. In fact, one regional remedial project manager provided us an

"approved" QAPP that turned out to be a draft version.

Second, formal approval from the Region's Quality Assurance Section would help ensure its comments

were solicited, received, and considered by the remedial project managers. EPA QA/R-5 provides that

the appropriate content in the QAPP may be best achieved by having the QAPP reviewed and confirmed

by the EPA remedial project manager, with the assistance and approval of the quality assurance

manager.

However, it did not appear that the Quality Assurance Section recommendations were always given full

consideration. For example, the March AFB QAPP was not approved by the Region's Quality Assurance

Section. The section had previously recommended that the QAPP address the selection of subcontract

laboratories and include more detail on the conduct of laboratory oversight. However, these

recommendations were not included in the QAPP. In this case, DOD's laboratory selection process for

March AFB had vulnerabilities, which were fully manifested by the selection of Eureka Laboratories. This

issue is further discussed in Chapter 4 of this report.

It is recommended that the basewide plan [for March Air Force Base] address the selection of

subcontract labor-atories and how laboratory oversight...will be conducted throughout the duration of the

project.

-Regional quality assurance

staff recommendations,

September 13, 1991 QAPPS NOT WELL DESIGNED

Our audit disclosed that the QAPPs for the five DOD bases evaluated were typically not designed to

detect poor laboratory work. To determine the most effective quality assurance techniques, we identified

those that detected problems at the sites included in our report. As shown in the chart below, we found

that certain techniques were highly effective in detecting poor or fraudulent laboratory data.

Quality Assurance Techniques Used

To Identify Data Quality Problems

Quality

Assurance

Technique

March Air

Force Base

Hunters Point

Naval Shipyard

Travis Air

Force Base

Sacramento

Army Depot

Luke Air

Force

Base

Full Data Validation X X X

"Double Blind" PE

Samples X

Magnetic Tape

Audits X X

7 of 53 06/17/13 07:36 AM
Laboratory Audits X X

As discussed below and in Chapters 3 through 7 of this report, we found that the above four quality

assurance techniques were generally not required by the QAPP. In those instances where the techniques

were required, they were not sufficiently monitored by DOD or EPA.

Except for Travis AFB, we found that the QAPPs were not revised after laboratory data analyses

problems were found. To illustrate, the QAPP for March AFB was not revised to require double-blind

performance evaluation samples or magnetic tape audits, even though the Region used these techniques

to identify data problems.

We believe the QAPPs could be better designed to detect poor quality data by incorporating techniques

that have been successful in identifying prior data problems. This should be done at the time of the annual

QAPP review required by EPA QA/R-5:

For programs or projects of long duration,...the QAPP shall be reviewed at least annually by the Project

Manager, revised if necessary to reflect current needs, and resubmitted for review and approval.

DATA VALIDATION REQUIREMENTS

Data validation is used to of ensure that laboratory data is of known quality. It involves reviewing data

against a set of criteria to provide assurance that data is adequate for its intended use.

KEY DECISION POINTS

Requiring Data Validation

Determining contaminants of concern.

Identifying the extent of contamination.

Determining a cleanup is complete.

-"Basic Requirements for Quality Assurance at Region 9 Superfund Sites,"

Region 9, July 1988

Data validation is absolutely essential at key decision points, such as determining the boundaries of

groundwater contamination.

EPA has data validation guidelines, known as national functional guidelines, for its own contract laboratory

program. Generally, the QAPPs reviewed called for data validation that corresponded with EPA data

validation guidelines. Data validation is typically performed on 100 percent of the data produced by EPA's

contract laboratories. However, for DOD NPL sites, typically a much lower percentage is required.

According to EPA guidelines, data validation includes a review of documentation such as raw data,

instrument printouts, chain of custody records, and instrument calibration logs. In order to perform an

EPA-type validation, the data package should be an EPA "Level IV" package that includes the necessary

documentation. (See Appendix F for description of data levels.) Without a Level IV package, it is not

possible to evaluate many aspects of a laboratory's analyses or even verify the analyses were

performed.

...a percentage of data representing all matrices, analysis types, and lab contractors for decision points

should be selected for ... documentation and data validation. A bare minimum would be 10 percent.

-Region 9 Quality Assurance Management Section

To illustrate how effective data validation can be in determining the quality of laboratory data, we noted

that the results of the data validation at Hunters Point and Sacramento Army Depot caused data to be

rejected, making it unusable for most significant decision-making, such as risk assessments. In fact, the

problems with data quality were so serious at the Sacramento Army Depot that, in our opinion, they

compromised the final cleanup decision.

Limited Data

Our review of 12

QAPPs at 5 DOD facilities showed that 8 QAPPs did not require a specific percentage of data to be

validated using EPA's national functional guidelines (or an equivalent).

8 of 53 06/17/13 07:36 AM
For example, two Luke Air Force QAPPs required data validation only when "substantial quality control

problems" were found. Since no quality control problems were found, data validation was not performed

until fraud was alleged at the laboratory.

Sacramento Army Depot's Basewide QAPP also required data validation according to EPA national

functional guidelines, and called for a percentage to be set in each operational unit QAPP. However, the

percentage requirement was incorporated into only one of the three operable unit QAPPs.

Data Validation Was Deficient

We found when data validation was performed, it was not performed according to EPA national functional

guidelines, completed promptly, or performed by an independent third party.

Our review noted serious problems were overlooked when data validation did not use procedures

comparable to EPA national functional guidelines. For example, the engineering contractor for Sacramento

Army Depot advised us that some of its laboratory data was being validated. However, the OIG's

Engineering and Science staff found that data validation, comparable to EPA national functional guidelines,

had not been completed.

In our opinion, this was a material quality assurance weakness. Therefore, we asked the Region to

perform data validation using EPA functional guidelines for two operable units at Sacramento Army Depot.

Both of these validations found major problems. The validation for the Burn Pits Operable Unit rejected all

analyses for volatile organic compounds for one round of sampling.

The validation for the Groundwater Operable Unit found serious deficiencies, such as missed holding

times, which compromised the laboratory's conclusions that contaminants of concern were not present.

As discussed in Chapter 3, we believe that any decision that cleanup actions are completed at the Depot

should be deferred until a positive determination can be made to the quality of the environmental data

used in decision-making.

Although the Hunters Point QAPP required data validation according to EPA national functional guidelines,

it was not performed promptly.

A large number of the Hunters Point samples were taken between October and December of 1990. In

January 1991, the engineering contractor started performing "cursory" validation of this data and decided

the laboratory analytical results were, "at best," estimates. In May 1992, the engineering contractor

rejected all organic analyses for one phase of the project after performing data validation according to

EPA guidelines. When data is rejected, the presence or absence of the analyte cannot be verified and the

data is unusable.

In this case, well over one year passed before the data validation determined data was unusable. In our

opinion, much of the delay in determining the usability of data could have been avoided if this validation

would have been done promptly.

Another problem observed at Travis AFB was a lack of independence by the party performing the data

validation. In this case, the engineering contractor used its own laboratory. When the engineering

contractor performed data validation, it found most of the data was usable. When the Air Force's quality

assurance contractor reviewed the validation, it determined the same data was unusable. Thus, it

appears the lack of independence may have adversely affected the reliability of the data validation. A

similar situation occurred at Mather AFB where the contractor performing the data validation had used its

own laboratory to complete the laboratory analytical work.

Data validation is one of the most important quality assurance techniques in a QAPP. We are

recommending that the Region ensure the QAPP requirements for independent and prompt data validation

of DOD laboratory data, in accordance with EPA guidelines, be established at a minimum of 20 percent.

This percentage was established in the Sacramento Army Depot QAPP for the Burn Pits Operable Unit.

We also noted over 20 percent data validation was performed at Travis AFB. This validation should

include all matrices, analysis types, and laboratories.

Computer Assisted Data Validation Should Be Implemented

EPA has developed a computerized data validation program called Computer-Aided Data Review and

Evaluation (CADRE). Computerized data validation is a relatively new quality assurance technique that is

more efficient than traditional manual data validation.

CADRE More Efficient

According to regional data, CADRE is much more efficient than manual data validation as shown below:

9 of 53 06/17/13 07:36 AM
COMPARISON OF DATA VALIDATION METHODS

CADRE Manual

Time to Validate 4 hours 35 hours

Turn Around Time 1 week 1 month

Cost to Validate $150 $1,200

CADRE More Effective

During an in-house test, the Region found that CADRE not only identified the same problems that manual

data validation did, but was more objective and consistent. A drawback to CADRE was that, as a

computer program, it could not visually inspect raw data to identify anomalies.

We believe an electronic data validation program, such as CADRE, could be of great assistance in

accelerating DOD cleanups. Under the President's Fast Track Cleanup Program for closing and

down-sizing military bases, EPA is charged with forging a partnership with DOD that hastens cleanups.

One of EPA's missions in this program is to provide the environmental expertise to streamline the cleanup

process. Consistent with its mission, we believe the Region should ensure that DOD uses electronic

validation techniques to the maximum extent possible.

In this regard, the Region conducted CADRE training for DOD staff in April 1995. However, no one from

the Air Force attended. Further, CADRE was only distributed to the individuals attending the training.

In our opinion, QAPPs for the DOD sites should be modified to require the use of electronic validation

techniques. We recognize that an upfront investment in software may be necessary. However, we believe

the advantages of efficiently performing data validation outweigh any additional implementation costs.

"DOUBLE-BLIND" PE SAMPLES NOT REQUIRED

As discussed in Chapter 4, "double-blind" performance evaluation (PE) samples were used by the Region

at March AFB to identify poor quality laboratory data. However, only 2 of the 12 QAPPs reviewed

required "double-blind" PE sample analyses.

The PE sample results provide a point-in-time evaluation of data quality related to the program quality

assurance objectives.

-Travis AFB Model QAPP

PE samples are prepared by "spiking" a known concentration of chemicals into a contaminate-free media,

such as water or soil. A laboratory's analyses of PE samples are used to evaluate its ability to produce

accurate results.

PE samples can be administered by two methods: "blind" or "double-blind". When a PE sample is blind,

the laboratory is aware the sample is a PE, but does not know the chemical concentration levels. Since

the laboratory is aware the sample is a PE, it can make an extra effort to perform a thorough analysis.

When a sample is double-blind, the PE sample is submitted as part of a field sample shipment. In this

situation, the laboratory is not only unaware of the concentration levels, it is also unaware that the sample

is a PE. Double-blind PEs involve more logistical considerations, since the PEs must be inserted into a

batch of samples in the field, using the same sample containers and sample tags.

The [double-blind] PE samples are made to look as similar to field samples as possible, and are

submitted as part of a field sample shipment so that the laboratory is unable to distinguish between

them.

-Travis AFB Model QAPP

In addition to double-blind PEs generally not being required, they were not always evaluated by a party

independent of the laboratory.

After data quality problems were found at Travis AFB, one of the QAPPs was revised to require

double-blind PE samples for each analytical method and field sampling program. According to this QAPP,

The concentrations reported for the PE samples will be compared to the known or expected

concentrations...The percent recovery will be calculated and the results assessed according to the

accuracy criteria...If the accuracy criteria are not met, the cause of the discrepancy will be investigated

and a second PE sample will be submitted.

We believe this is an excellent quality assurance technique that should be incorporated into each DOD site

QAPP as early in the project as possible.

10 of 53 06/17/13 07:36 AM
MAGNETIC TAPES SHOULD BE AVAILABLE

None of the QAPPs required magnetic tape audits.

Magnetic tape audits are routinely conducted by EPA in monitoring its contract laboratories to assist in

determining if the laboratory is complying with the contract, the integrity of the laboratory's computer

systems, and the appropriateness of any software editing.

At the time of our review of the five DOD bases, we found that only March AFB used magnetic tape

audits, as described in Chapter 4. In this case, the laboratory provided the magnetic tapes, although not

required by the contract.

Although not suitable for all types of analytical data, we believe magnetic tape audits should be performed

if major deficiencies are found by other methods, such as data validation or performance evaluation

samples. However, in order to do so, DOD must be able to obtain the magnetic data. This means

including the requirement in the QAPP. To illustrate the importance of the requirement, we noted that for

Travis AFB, the laboratory would not provide the necessary magnetic tapes so that the Region could

evaluate data quality.

LABORATORY AUDIT RESULTS NOT SHARED

All but 2 of the 12 QAPPs required some type of laboratory audit. However, none of the QAPPs required

the results of the laboratory audits to be shared with the Region.

On-site audits are designed to identify technical areas which may cause laboratories to improperly identify

or quantitate chemicals. The audits normally evaluate a laboratory's technical expertise, standing

operating procedures, facility and equipment sufficiency, and possible sources of sample contamination.

As discussed earlier in this chapter, an Air Force laboratory audit of Eureka Laboratories in January 1991

found major problems with the Laboratory. It recommended that no further samples be sent to Eureka

Laboratories. This audit recommendation was not shared with the Region. In the future, the Region should

ensure that it obtains the most recent laboratory audits from each service as part of its oversight function.

We believe such audits are a good management technique and should be incorporated into QAPPs. In

summary, our review identified the following weaknesses in laboratory audit requirements:

WEAKNESSES IN LABORATORY AUDIT

REQUIREMENTS FOR 5 DOD BASE QAPPS

Weakness Sites Affected

No laboratory audit required Hunters Point, Travis East Operable Unit

Audit results not required to be shared with Region 9 All

Audit not required before work is started Sacramento, March, Luke, Travis

Audit not conducted by firm independent of laboratory Travis

QAPPS NOT IMPLEMENTED OR CURRENT

As described in the following chapters of this report, we found that the QAPPs were not fully implemented

by DOD. According to EPA Order 5360.1, EPA project managers are responsible for making sure QAPPs

are implemented and adhered to.

To illustrate, the QAPP for the Burn Pits Operable Unit at Sacramento Army Depot required 20 percent of

the data to be validated according to EPA national functional guidelines. We found that no data was

validated. In another example, the QAPP for Luke AFB required the engineering contractor to do a

laboratory audit at least semi-annually. We found the engineering contractor did not perform any

laboratory audits.

We also found that some QAPPs were not current. For example, the Sacramento Army Depot QAPP

named a quality assurance officer that was no longer employed by the depot's engineering contractor.

It is absolutely essential that the QAPP be kept current and that all personnel involved in the work effort

have a current version of the QAPP available.

-EPA QA/R-5

11 of 53 06/17/13 07:36 AM
DATA QUALITY OBJECTIVES WERE DEFICIENT

Data quality objectives were not sufficiently defined for four of the five DOD bases reviewed. Thus, it was

difficult to determine whether data of sufficient quality and quantity was collected to support decision

making process at these sites.

According to OSWER Directive 9355.9-01, EPA developed the data quality objective process because:

...it is the goal of EPA and the regulated community to minimize expenditures related to data collection by

eliminating unnecessary, duplicative, or overly precise data. At the same time, they would like to collect

data of sufficient quantity and quality to support defensible decision making. The most efficient way to

accomplish both of these goals is to begin by ascertaining the type, quality, and quantity of data

necessary to address the problem before the study begins.

The data quality objectives process was developed to avoid the collection of data that is inconsequential

to the decision making process. The process allows decision makers to define their data requirements

and acceptable levels of decision errors during planning, before any data is collected.

Data quality objectives are the basis for preparing the QAPP. The QAPP should show how the intended

measurements or data acquisition methods are appropriate for achieving the data quality objectives.

Thus, when data quality objectives are inadequately specified, the project runs the risk of not collecting

sufficient quality data or expending too much on sampling.

EPA guidance concerning data quality objectives was updated in 1993. However, we evaluated the

adequacy of data quality objectives using guidance prepared in 1987, since all of the QAPPs reviewed,

except for the Travis Model QAPP, were prepared before the new guidance was issued. With the

exception of this QAPP and the QAPP for Luke AFB, we found data quality objectives were not

developed for each site investigation as required.

QUALITY ASSURANCE OFFICERS NOT APPOINTED

We found that DOD did not appoint the quality assurance officers in a timely fashion and appointed one

quality assurance officer that was a contractor.

Generally, the Federal facility agreements or QAPPs required DOD to appoint a quality assurance officer

to monitor project quality. As discussed in the chapters on Travis AFB and Hunters Point Naval Shipyard,

these officers were sometimes not appointed for periods up to 3 years after the agreement was signed.

In addition, we found there was no quality assurance officer at Sacramento Army Depot.

We also noted that, in one instance, the quality assurance officer appointed was an engineering

contractor. Since the engineering contractors performed field sampling and sometimes were involved in

laboratory analyses, we do not believe that they were in a position to independently evaluate data quality.

The quality assurance officer position should be assigned to a government employee.

BEST PRACTICES

We identified some best practices with a Travis AFB QAPP which should be considered for inclusion in all

DOD QAPPs. The Travis "Model" QAPP required double-blind performance evaluation samples. It also

developed a very usable format for the quality assurance report that showed findings, corrective actions

required, and the overall effect on data quality assurance.

We also identified a Federal facility agreement for a site not included in our audit, that did a good job

specifying quality assurance requirements. This agreement, for the Marine Corps Air Station Yuma, called

for the following:

In order to provide quality assurance and maintain quality control regarding all field work and sample

collection performed pursuant to this Agreement, the Marine Corps agrees to designate a Quality

Assurance Officer (QAO) who will ensure that all work is performed in accordance with approved work

plans, sampling plans, and QAPPs. The QAO shall maintain for inspection a log of quality assurance field

activities and provide a copy to the Parties upon request.

...To ensure compliance with the QAPP, the Marine Corps shall, upon request by EPA or the State, use

its best efforts to obtain access to all laboratories performing analysis on behalf of the Marine Corps

pursuant to this Agreement. If such access is not obtained for any laboratory, EPA or the state may

reject all or portions of the data generated by such laboratory and require the Marine Corps to have the

same or comparable data analyzed by a laboratory that will grant such access.

RECOMMENDATIONS

12 of 53 06/17/13 07:36 AM
We recommend that the Regional Administrator, Region 9:

1. Fully identify the impacts of the Eureka Laboratories fraudulent practices on Federal facility

investigations, studies, and cleanups in the Region.

2. Determine the reliability of DOD's laboratory quality assurance programs and perform sufficient

oversight to assure the quality of the laboratory data.

3. Develop a data base or tracking system for laboratory data quality at DOD facilities that identifies the

laboratory used, the quality assurance measures performed, and data quality deficiencies identified.

4. Ensure that a QAPP is prepared for each DOD site and that the QAPPs are kept current.

5. Require QAPPs to be formally approved by the Region, after review and concurrence of the quality

assurance staff. In addition, the remedial project managers should be required to document the reasons

quality assurance staff's recommendations have not been included in final QAPPs.

6. Review the DOD QAPPs at least annually and require appropriate revisions to include, as a minimum,

the following quality assurance measures:

a. The use of EPA national functional guidelines for at least 20 percent of the data representing all

matrices, analysis types, and laboratories for decision points.

b. The completion of data validation requirements by a party independent of both the laboratory and its

parent company.

c. The use of electronic data validation techniques.

7. Require the QAPPs to be amended to include the requirement for independent, double-blind PE

samples.

8. Revise the QAPPs to require that magnetic data is maintained and made available to the Region. In

addition, magnetic tape audits should be required if major deficiencies are found by other quality

assurance methods, such as data validation or performance evaluation samples.

9. Improve the use of laboratory audits by amending the QAPPs to:

a. Require laboratory audits to be done before work is started and periodically throughout the project.

b. Specify that the audits will be conducted by an activity independent of the laboratory.

c. Require the laboratory audit results to be provided promptly to the Region.

10. Ensure the QAPPs are fully implemented by DOD facilities. In making this determination, the Region

should obtain appropriate supporting documentation from the Federal facilities to substantiate their

actions.

11. Make sure reasonable data quality objectives are included for each site investigation.

12. Ensure quality assurance officers are appointed in a timely manner and that they are government

employees.

13. Publicize best practices used in DOD QAPPs and Federal facility agreements to make remedial

project managers and quality assurance managers aware of them.

REGION 9 COMMENTS AND OIG EVALUATION

The Region did not respond to the specific recommendations, but said it was pursuing the following

actions:

1. Region 9 has recently and will continue to seek to require federal facilities, through our comments on

their Quality Assurance Project Plans (QAPjPs), to follow the Agency's Data Quality Objective Process

for Superfund Interim Final Guidance dated September 1993. Region 9 will also ensure that, whenever

possible, numeric data quality objectives on precision, accuracy, completeness, and detection limits are

set prior to the approval of each QAPjP.

2. The OIG draft report found that data validation comparable to the EPA National Functional Guideline

was not performed at DOD sites. The Region has requested that several NPL federal facilities submit

representative data packages to EPA. Region 9 will perform our standard data validation on these

packages using EPA's Functional Guidelines for Data Validation. The data packages not containing

13 of 53 06/17/13 07:36 AM
sufficient analytical deliverables and all other data quality problems associated with the other data

packages will be identified to DOD for action.

3. The Region has been working with DOD for the last 3 years to raise awareness of the quality

assurance problem and to help define an improved DOD quality assurance program that will meet or

exceed our minimum standards. We will continue to participate in these efforts. As part of this effort, the

Region will continue to encourage DOD facilities to develop and implement, as part of their own quality

systems, a self sufficient performance evaluation program, and will seek to require DOD to submit PE

study results to EPA. The Region plans to score and validate as many PE sample results as our limited

resources allow and will make recommendations to DOD on follow up corrective actions when

necessary.

4. In addition to recommending that the Federal facilities conduct PE samples, recently, Region 9 QAMS

has conducted performance evaluation samples for on-site and off-site labs at MCA Tustin (monthly),

Yuma MCA (twice), George AFB, McClellan AFB (twice), and March AFB. Additional performance

evaluation activities are planned.

5. Through the review of work plans and QAPjPs, the Region will seek to require DOD facilities to

perform both pre-award and routine audits of their contract labs. We will also seek to require federal

facilities to furnish all audit reports to EPA.

6. Through the review of work plans and QAPjPs, the Region will seek to require DOD facilities to

address how they will ensure the authenticity of the data. At a minimum, we will seek to require that all

organic data for GC and GC/MS magnetic tapes be archived and available to DOD and EPA upon

request. This action would provide a strong deterrence to the fraudulent activities and any poor quality

laboratory performance that have been observed in the past. Region 9 QAMS has conducted magnetic

tape audits on March AFB, Travis AFB, Luke AFB, and Yuma MCA. Additional tape audits will be

conducted as resources allow. The Region is committing resources for purchasing equipment necessary

to develop a regional tape audit capability, and will suggest DOD do the same.

7. Region 9 and OERR provided training of the use of the Computer Assisted Data Review and

Evaluation (CADRE) software on April 10-13, 1995 to the U.S. Army Corps of Engineers, the U.S. Navy

and their contractors. An additional CADRE training sessions will be provided to the U.S. Air Force later

this fall. Since the first training, QAMS has continued to distribute copies of CADRE and technical

assistance to the attendees. QAMS received confirmation that the Army and Navy will begin to

implement data validation using CADRE on a number facilities. Implementation of CADRE would help

remediate vulnerability regarding data quality deficiencies identified in the IG report.

8. As appropriate, the Region will seek modifications by DOD of their previously approved QA Plans still

in use to incorporate the IG comments regarding data quality objectives, data validation, the use of PE

samples, etc.

9. One of the IG concerns is that federal facilities have not followed the requirements set forth in FFAs

and approved QAPjPs. The Region's QAMS has developed a QA check list to assist the remedial

project managers in their oversight of the quality system at federal facilities. The QA checklist can be

used during the Technical Review Committee meetings to ensure that federal facilities comply with the

requirements in the approved QAPjPs and appropriate oversight follow up is conducted.

10. The Region is exploring the idea of providing training for federal facilities and their contractors that

would aim at improving DOD's overall quality systems, their reporting, development of expertise in the

data quality objectives process, and managing DOD's contractors' laboratories.

The Region also indicated that:

Its staff were informed and directed to discuss with DOD officials the extent of Eureka's use at DOD

sites, and to assess the criticality of this particular data to remedy selection at the site.

It agreed that "some" tracking system for data quality at Federal sites should be in place.

The Navy was revising the Hunters Point Naval Shipyard QAPP for use in the groundwater monitoring

program.

OIG Evaluation

We accept the Region's comments as a strong commitment to improve the QAPPs, and related data

quality oversight at DOD activities. Since the response to the draft report did not specifically address

each recommendation in the draft report, the recommendations must be addressed during the final audit

resolution process.

14 of 53 06/17/13 07:36 AM
Chapter 3

Sacramento Army Depot

SUMMARY

We found the Region had performed virtually no data quality oversight at the Sacramento Army Depot

(the Depot) since 1992. As a result, the environmental data obtained for the Depot Superfund site was of

unknown quality because: (i) the data packages the laboratories provided were insufficient to verify the

quality of the analyses; and (ii) analyses were not validated using EPA national functional guidelines. Also,

in some instances, environmental data used in the decision-making process was subsequently rejected

due to unacceptable quality.

The overall quality objectives for the...QAPP...[are] to obtain data that are scientifically and legally

defensible...

-Sacramento Army Depot Quality Assurance Project Plan

As a result, we believe any decision that cleanup actions are completed should be deferred until a positive

determination can be made to the quality of the environmental data used in decision-making. This includes

the Region reevaluating its June 1994 certification that the Depot's Tank 2 Operable Unit was cleaned up.

Until the reevaluation is completed, the Army should be informed the certification is being withdrawn.

Further, we recommend that the Depot's QAPP be revised to better oversee laboratory quality for the

remainder of the cleanup effort.

BACKGROUND

The Depot, located in Sacramento, California, is a nearly 500-acre former military operation that was

established in 1945. The Depot handled communications equipment and supplies. Operations at the

Depot involved the use of solvents, oils and grease, fuels, caustic solutions, and metal-plating baths. The

Depot was closed in March 1995. Part of the Depot property has already been transferred to the City of

Sacramento.

The Depot was added to the NPL in 1987. EPA, the Depot, and the State of California entered into a

Federal facility agreement to investigate and cleanup the site in 1988. In January 1995, a Basewide

record of decision was agreed to by the Army, State of California, and EPA to cleanup both the soil and

groundwater at the Depot. The record of decision was completed more than a year ahead of schedule.

Depot Includes Four Operable Units

The schedule below shows the Depot's four operable units and the type of contaminants found.

Operable Unit Contaminants of Concern

Burn Pits Volatile organic compounds, metals

Tank 2 Volatile organic compounds

Groundwater Volatile organic compounds

Oxidation Lagoons Metals

NO QAPP FOR GROUNDWATER OPERABLE UNIT

QAPPs were prepared for 3 of the 4 operable units, and a Basewide QAPP was prepared. However, a

QAPP was not prepared for the Groundwater Operable Unit, even though one was required by EPA

Order 5360.1. While there was a "monitoring plan" for the Groundwater Operable Unit, it was not subject

to the same requirements as a QAPP. For example, EPA has responsibility for ensuring that a QAPP is

implemented, but there is no similar requirement for a monitoring plan.

QAPP APPROVAL PROCESS DEFICIENT

We noted there were no documented regional approvals for the QAPPs for the Oxidation Lagoons or

Burn Pits Operable Units. Such approvals are important to ensure that the approved version of the QAPP

is being used. We also found that recommendations made by the Region's Quality Assurance Section,

including one calling for the preparation of a QAPP for the Groundwater Operable Unit, were not

followed. The Region's files did not document why this recommendation was not followed.

QAPPS DATA VALIDATION REQUIREMENT NOT MET

15 of 53 06/17/13 07:36 AM
The Depot's Basewide QAPP required data validation according to EPA national functional guidelines. The

percentage of samples requiring validation was to be incorporated into the operable unit QAPPs. Our

review disclosed that three of four operable units had no percentage requirement for data validation in

the QAPP or monitoring plan. Only the Burn Pits Operable Unit had a percentage requirement.

A percentage of all samples will be subjected to a complete statistical data validation procedure. The

samples selected for the statistical analysis will be those that are nearest to the anticipated boundaries of

contamination. Valid data will be used to evaluate whether the [data quality objectives] of the QAPP have

been met.

-Basewide QAPP,

Sacramento Army Depot

DATA OF UNKNOWN QUALITY

EPA Order 5360.1 requires data to be of known quality, verifiable, and defensible. In our opinion, the

laboratory data at the Depot was of unknown quality for several reasons.

Data Packages Are Insufficient

The OIG Engineering and Science Staff (ESS) determined the data packages that the laboratory provided

to the Depot were insufficient to perform data validation in accordance with EPA national functional

guidelines. One of the key items not provided by the laboratory were detailed equipment calibration

records. These records are needed to perform data validation using EPA guidelines.

Although planned, data validation comparable to [EPA's] National Functional Guideline data validation has

not ever been completed at the Depot. In addition, the data packages that the contractor required of the

analytical laboratories are insufficient to perform EPA data validation.

-EPA OIG's Engineering and Science Staff, June 2, 1995

The data packages needed for data validation are normally called EPA Level IV data packages. (See

Appendix F for an explanation of data package levels.) Without a Level IV package, it is not possible to

confirm that the laboratory even performed the analyses.

Data Not Validated

The Depot's engineering contractor advised us that it had validated some of its contract laboratory data.

However, the OIG ESS found it had not. In fact, we were not able to document any data validation

performed by the Army or EPA that was comparable to EPA national functional guidelines for any of the

operable units.

Samples Related to Tank 2 Cleanup

The Region accepted that the Tank 2 cleanup was complete in June 1994. As noted above, we

determined that there was no documentation that data validation was performed in accordance with EPA

national functional guidelines. However, we were advised that 12 "confirmation" samples were used to

determine the Tank 2 Operable Unit cleanup had been completed. We subsequently determined that

these samples had not been validated.

According to 1988 regional guidance, data validation is required for determining if cleanup is complete. In

a May 1993 technical review committee meeting, the Depot agreed to validate 6 of the 12 confirmation

samples.

Both the Corps, who administered the Depot's contracts, and the Region's project manager were under

the impression that data validation had been performed by the Depot engineering contractor. However,

when we asked for the data validation report, we found that the data validation was never completed.

The OIG ESS found the data package submitted by the laboratory was incomplete for data validation

using EPA national functional guidelines. For example, the package did not include critical information for

calibrations and performance check samples. Further, the OIG ESS determined that the concentration

units were consistently reported wrong. Specifically, the soil sample results were incorrectly reported in

liquid concentrations.

The Region's determination that the Tank 2 Operable Unit cleanup was complete only referenced the 12

confirmation samples. However, in response to our concerns about the lack of the data validation report,

the Region advised us it had considered other data, such as contaminant removal rates, to determine the

cleanup was complete. Further, the Region stated that it was working with the Depot to validate the

16 of 53 06/17/13 07:36 AM
confirmation samples to the "extent feasible."

The data must be evaluated per EPA's functional guidelines. It is not acceptable to take the data at face

value without additional validation.

-Region 9 discussing the requirements for cleanup completion of the Tank 2 site with Sacramento Army

Depot officials, May 25, 1993

In our opinion, and consistent with the requirements agreed to by the Depot, the sample analyses were of

unknown quality without data validation. Consequently, its analyses did not meet the requirement for

determining that the cleanup is complete.

Request for Regional Review

Because of the lack of data validation and its impact on the status of this NPL site, we asked the Region

to perform data validation using EPA guidelines for two operable units at Sacramento Army Depot, the

Burn Pits and Groundwater Operable Units.

Burn Pits Operable Unit Data Rejected

Data validation for the Burn Pits Operable Unit found that volatile organic compound laboratory analyses

had to be rejected. In this respect, the Region's quality assurance staff performed a validation of 33

selected samples from the 137 samples included in the March 1991 round of sampling at the Burn Pits. In

a letter to us dated May 19, 1995, the Region's quality assurance staff's validation found that:

The sample results [for volatile organic compounds] are rejected due to serious deficiencies in the

ability to analyze the sample and meet quality control criteria. The presence or absence of the analyte

cannot be verified.

The 33 samples selected from the March 1991 sampling round were critical to the cleanup decision

making process at the site. These samples were used in the public health risk assessment, the remedial

investigation, the feasibility study, and the record of decision. The data was also used to decide the

contaminants of concern, determine the cleanup levels for the contaminants, and to select the cleanup

remedy. Further, the OIG ESS concluded that all 137 samples taken in March 1991 should be rejected

because of a defect in the sampling technique. Specifically, the samples were not segregated into

appropriate containers.

Groundwater Operable Unit Data Validation Finds Problems

The Region's quality assurance section found serious problems in its validation of the volatile organic

compound data for the Groundwater Operable Unit including:

Missed holding times on three samples, possibly causing false assumptions that contaminants were

absent ("false negatives"), or biasing the minimum values of all compounds; and,

Unacceptable calibrations impacting numerous compound analyses, including one contaminant of concern

in all samples, and another contaminant of concern in 10 samples.

As a result, the samples could not be used to determine how much contamination was present or whether

some contaminants of concern were absent. This was significant, since most of the sample analyses

showed that contaminants were not present. Because the samples were used in the decision-making

process, including determination of the cleanup levels and selection of the cleanup remedy, it is important

that accurate sample analyses be obtained.

QUALITY ASSURANCE PROJECT PLANS CAN BE STRENGTHENED

In addition to correcting the weaknesses relating to data validation, the Depot's QAPPs could be

strengthened to improve the opportunity for quality data. This could be done by: (i) using double-blind

performance evaluation (PE) samples; (ii) requiring that laboratory audit results be provided to the

Region; (iii) requiring the laboratories to provide electronic data information; and (iv) identifying site

specific data quality objectives.

Double-Blind PE Samples Not Used

Neither the Region nor the Army used double-blind PEs at Sacramento Army Depot to evaluate the quality

of data. Double-blind PE samples were successfully used to detect laboratory analyses problems at the

March Air Force Base NPL site, as discussed in Chapter 4. Further, the Travis Air Force Base Model

QAPP required the Air Force to use double-blind PEs.

17 of 53 06/17/13 07:36 AM
Laboratory Audit Results Not Provided To Region 9

We believe laboratory audit results should be available to the Region. According to the Basewide QAPP,

a laboratory audit was to be performed at the beginning of the project, and each time a data set was

received. However, the QAPP did not require the audit results to be provided to the Region, and the

Region was not receiving the results. In our opinion, this is not consistent with EPA guidance which

requires that the Region verify the QAPP implementation. Further, had these results been made available

to EPA, it would have detected these to problems at the laboratories performing analysis of Depot

samples.

Corps Laboratory Inspections Find Problems

Generally, we found the Corps, who administered the engineering contract, performed a laboratory

inspection before a laboratory was used. These inspections were normally not as thorough as audits.

While they did disclose deficiencies, they were not a substitute for audits. However, the results of these

inspections were not provided to EPA, although they disclosed major deficiencies that may have impacted

data quality.

For example, a Corps May 1992 inspection of the laboratory that analyzed the Tank 2 Operable Unit

confirmation samples found:

There was one major and 3 minor deficiencies noted during the lab inspection. The major deficiency

noted during the lab inspection pertained to traceability. There was a consistent absence of initialed and

dated reagents, log book entries and standards and spiking solutions ...minor

deficiencies...included...lack of consistent measurement and recording of storage temperatures and a

lack of consistent calibration of analytical balances.

The laboratory inspector recommended that any further validation for tank work be handled by the Corp's

Sacramento District.

[The] lab has documentation problems...[and] QC [quality control] problems with temperatures and

balances.

-Army Corps Inspection Report,

May 27, 1992

A Corps August 1992 inspection of the laboratory that performed the groundwater analyses found:

One major deficiency that would adversely affect [the laboratory's] ability to conduct the required

chemical determinations was noted during the inspection.

This major deficiency dealt with a retention time window being incorrectly set. Nine minor deficiencies

were also disclosed noting problems such as retention time calculations not performed in accordance with

method protocols. The report stated that these deficiencies could adversely affect the laboratory's ability

to conduct the required analyses.

[Retention time] windows have been corrected, ...74 previously reported samples may have been

affected by this error.

-Army Corps of Engineers Inspection,

September 1, 1992

We also could not find any evidence that the Corps performed a laboratory inspection each time a data

set was received, as required by the QAPP.

Magnetic Tapes Not Required

The QAPPs did not require the laboratory to provide electronic data, such as magnetic tapes of raw data,

needed to perform audits. Magnetic tape audits were instrumental in detecting laboratory fraud at March

Air Force Base, as discussed in Chapter 4.

...[Data quality objectives] were not adequately addressed in either the basewide QAPP or in either of the

[other] QAPPs reviewed.

-OIG Engineering and Science Staff, June 2, 1995

Site Specific Data Quality Objectives Not Provided

18 of 53 06/17/13 07:36 AM
The QAPPs did not present any site specific data quality objectives. Data quality objectives are the basis

for determining the level of quality assurance activities.

QUALITY ASSURANCE OFFICER NOT REQUIRED

The Federal facility agreement did not require a quality assurance officer. We recognize this is not a

mandatory requirement, but believe it would help affix DOD responsibility for data quality.

While the QAPP identified a quality assurance officer, the individual named worked for a subcontractor

that was no longer employed by the Depot's engineering contractor.

IMPACTS AT SACRAMENTO ARMY DEPOT

According to State and Army personnel, it is anticipated that the Depot will be taken off the NPL within

two years. OSWER Directive 9320.2-3A, specifies that a Superfund site is considered completed when

all cleanup actions identified in the record of decision have been successfully implemented and cleanup

levels have been achieved. Only after the satisfaction of this requirement can a close out report be

prepared. A closeout report is required before a site can be deleted from the NPL.

In view of the data validation problems noted in our review, and the general absence of other validated

data, it is our opinion the Depot cleanup should not be considered complete until a positive assurance can

be provided that the data was of sufficient quantity and quality to support the decisions made.

RECOMMENDATIONS

We recommend that the Regional Administrator, Region 9:

1. Inform the Army that the cleanup certification for the Tank 2 Operable Unit is being withdrawn until the

required data validation is performed.

2. Require all environmental data used for decision making to be validated using EPA national functional

guidelines.

Note: Additional recommendations are addressed in Chapter 2 of this report.

REGION 9 COMMENTS AND OIG EVALUATION

The Region generally agreed with the recommendations and provided the following comments:

Sacramento Army Depot has agreed to resample the Tank 2 Operable Unit and perform the necessary

data validation on the new samples. We propose advising the Army that if resampling indicates that

cleanup levels have not been achieved, EPA will withdraw its cleanup certification and additional

remediation will be required.

Concur with the recommendation; however, we would prefer that the recommendation would be flexible

enough to allow for a cost-effective approach to the validation of all previous data. We suggest that the

recommendation read: "Require the Army to develop and implement an-EPA approved plan to validate

previous data used for decision making using EPA national functional guidelines or equivalents."

OIG Evaluation

The Region's response meets the intent of the recommendations.

Chapter 4

March Air Force Base

SUMMARY

Despite objections raised by the Region, the engineering contractor for the AFB cleanup hired Eureka

Laboratories in 1992 to analyze samples for part of the NPL site cleanup. Subsequently, in late 1993, the

Region rejected $1 million worth of Eureka Laboratories analyses. This action delayed the project by

about 1 years. In December 1993, EPA suspended the laboratory from further Federal procurement

actions, and in May 1995, the laboratory pleaded guilty to criminal charges that its test results were

falsified.

Our audit noted that the QAPPs for March AFB did not contain the primary techniques eventually used by

the Region to detect the laboratory fraud. These techniques included a laboratory audit before the start of

sampling and that magnetic tapes be provided upon request.

19 of 53 06/17/13 07:36 AM
We believe that Eureka Laboratories probably would not have been used, if the laboratory audit had been

required before the start of sampling. This belief is based on the fact that a 1991 Air Force laboratory

audit of Eureka Laboratories under another project recommended that samples not be sent to Eureka

Laboratories.

BACKGROUND

March AFB covers 7,100

acres near Riverside, California. It served as a training, bomber and air-to-air refueling operations base.

Contaminants include heavy metals and volatile organic compounds, such as benzene. The base was

listed on the NPL in 1989. The Air Force, the Region, and the State of California signed the Federal

facility agreement for the cleanup in September 1990. Under the agreement, there is a Basewide QAPP

and QAPPs for the base's three operable units. A Basewide remedial investigation and feasibility study

report was due in August 1995. The Air Force Center for Environmental Excellence (AFCEE), in San

Antonio, Texas issued most of the NPL site related contracts for March AFB.

DATA PROBLEMS FOUND BY REGION 9

Although the AFCEE decided to use Eureka Laboratories in spite of warnings from the regional staff, the

Region deserves credit for taking aggressive actions to minimize project set backs caused by the

laboratory's deficient work.

The Region learned that the Air Force planned to use Eureka Laboratories in early 1992, after EPA

received the Air Force's QAPP for Operable Unit 2. The QAPP identified Eureka Laboratories as the

contractor for the sample analyses.

Use of Eureka Laboratories Discouraged

Because another EPA Region had experienced problems with Eureka Laboratories, the Region verbally

recommended that Eureka not be used. However, the Region's advice was not heeded. Instead, the

AFCEE decided that it would continue to use Eureka Laboratories, along with two other laboratories, for

laboratory analyses at Operable Unit 2.

The Region, sensitive to the potential problems associated with Eureka Laboratories, approved the

Operable Unit 2 QAPP in August 1992 after deciding to use double-blind PE samples as part of its quality

assurance oversight. The use of PE samples and subsequent regional audits of magnetic tapes from

Eureka Laboratories disclosed that the laboratory work was deficient and fraudulent work was pervasive.

This led to Eureka Laboratories pleading guilty to falsifying test results and two of its chemists being

convicted of fraud in May 1995.

A Sacramento laboratory [Eureka Laboratories] admitted it routinely falsified results of water, soil and

air tests done for the government to determine levels of pollutants and toxins at some of the worst

hazardous waste sites in the country.

-Sacramento Bee

May 2, 1995

Meanwhile, EPA rejected all sample analyses work performed by Eureka Laboratories at the base. By

January 1994, the Air Force had developed a corrective action plan for replacing the rejected data.

QUALITY ASSURANCE PROJECT PLANS WERE INADEQUATE

While the Region's oversight at March AFB was successful in detecting the problems with Eureka

Laboratories, we concluded that the Basewide QAPP and Operable Unit 2 QAPP were inadequate to

detect the problems. Specifically, the QAPPs did not require:

the use of PE samples;

the submission of magnetic tapes of raw data;

a laboratory audit before any analysis work began or that subsequent laboratory audits be provided to the

Region; and,

the performance of data validation in accordance with EPA's national functional guidelines.

Further, we found that the QAPPs were not revised after the data problems were found. In addition, there

was no record that the Basewide QAPP had been approved by the Region.

20 of 53 06/17/13 07:36 AM
We also found that data quality objectives, which were the basis for determining the level of necessary

data quality, were not sufficiently established in the QAPPs.

Performance Evaluation Samples Not Required

Neither of the QAPPs required the Air Force to use PE samples to judge laboratory quality. Instead, the

QAPPs said that the laboratory will "participate" in PE sample programs, and that the results will be made

available during audits.

Magnetic Tapes Not Provided

The QAPPs did not require the laboratory to provide electronic data, such as magnetic tapes of raw data,

needed to perform audits of the validity of the data. Magnetic tape audits were instrumental in the

Region's detection of the Eureka Laboratories fraud.

Laboratory Audits Not Required

The Region did not require a laboratory audit before a laboratory was used. While laboratory audits are

not always required by QAPPs, they have been effectively used in reviewing a laboratory's procedures. In

our opinion, Eureka Laboratories probably would not have been selected if the Region would have

required an audit before the laboratory was used.

Instead of requiring an audit before selection of a laboratory, the QAPPs required the engineering

contractor to perform one laboratory systems audit during the project. We consider this a weak quality

assurance measure because it does not ensure that a laboratory is capable of the required analyses

before work was performed. In addition, the QAPPs did not require the results of the audit to be

submitted to the Region.

As previously noted, AFCEE contracted for a laboratory audit of Eureka Laboratories in 1991 under

another Air Force project. The audit found major problems and recommended that no samples be sent to

Eureka Laboratories for analysis. In February 1992, and again in October 1992, followup reviews by the

AFCEE contractor found

many of the problems discovered in the 1991 audit remained unresolved. It was during this time frame

that the QAPP for Operable Unit 2 was submitted to EPA for approval. The QAPP named Eureka

Laboratories as the laboratory which would perform the sample analyses for Operable Unit 2. The sample

analyses commenced in August 1992. During our discussions, AFCEE representatives were unable to

explain why Eureka Laboratories was used at March AFB, when the laboratory audit recommended

against sending samples to Eureka.

[The contractor] recommends that no further samples be sent to Eureka until the aforementioned

corrective actions have been completed.

-AFCEE contractor

February 11, 1991

No Data Validation

The QAPPs did not require that data validation be performed in accordance with EPA's national functional

guidelines. The QAPPs merely cited that there would be data validation "where applicable." However,

they did not define where it was applicable or what percentage of the total samples analyzed would be

validated. In addition, the QAPPs did not require the submission of detailed supporting sample

documentation, known as "Level IV" data packages. These packages were necessary to perform data

validation according to EPA national functional guidelines.

Failure to Establish Data Quality Objectives

Adequate data quality objectives were not established in the QAPPs, although they are the basis for

determining the level of quality assurance activities. Specifically, the data quality objectives were

necessary to:

Establish objectives for each type of data collection and analysis effort; and

Identify when "Level IV" data packages were needed. The Region recognizes that some "Level IV" data is

a necessary prerequisite for data validation.

Additional Costs and Delays Associated With Rejected Work

21 of 53 06/17/13 07:36 AM
March AFB officials advised us that it paid Eureka Laboratories about $1 million for the laboratory

analyses which were rejected. An additional $250,000 was expended for resampling. All but $60,000 of

this amount was paid for by the engineering contractor. Further, because of the deficient and fraudulent

laboratory analyses, the due date for the Remedial Investigation Report slipped from January 11, 1994 to

April 10, 1995. Thus, the remedial investigation was delayed at least 1 years.

RECOMMENDATIONS AND REGION 9 RESPONSE

Our recommendations are addressed in Chapter 2 of this report. The Region's complete response to this

chapter and our evaluation of the response are included as Appendices A and B of this report.

Chapter 5

Travis Air Force Base

SUMMARY

After the Federal facility agreement was signed in 1990, the Air Force selected Roy F. Weston, Inc.

(Weston) as the engineering contractor for the basewide remedial investigation and feasibility study. In

1991 and 1992, Weston completed three rounds of sampling to determine the extent of contamination at

Travis AFB. The samples were sent to its own laboratories for analyses.

In January 1992, an Air Force laboratory audit of one of Weston's laboratories found major problems with

its analytical practices. Because of the analytical problems, the Region determined that about $2 million of

Weston's analyses were of unknown quality. In addition, because of these problems, one of the operable

units has experienced a cleanup delay of more than 2 years.

We found the QAPP was not designed to detect the laboratory problems. After the data problems were

disclosed, the Air Force developed two new QAPPs for Travis AFB. These QAPPs included many of the

key data quality assurance ingredients missing from the original QAPP. However, in our opinion, the new

QAPPs could be strengthened to improve the quality of laboratory analyses.

BACKGROUND

Travis AFB is located about 45 miles southwest of Sacramento, California, and covers over 5,000 acres.

Its primary mission is to provide worldwide airlift capacity for cargo and troops.

Contaminants include volatile organic compounds, heavy metals, and polynuclear aromatic hydrocarbons.

The base was listed on the NPL in 1990.

After the Federal facility agreement was signed in September 1990, the base was set up as one operable

unit, and a basewide QAPP was prepared. The Air Force Center for Environmental Excellence (AFCEE)

in San Antonio, Texas contracted with Weston to perform the remedial investigation and feasibility study.

A subsequent AFCEE laboratory audit of one of Weston's laboratories disclosed major problems, which

led to the Region's July 1993 determination that the data analyses were of unknown quality.

Operable Units

In 1993, after data problems were found, Travis AFB was divided into four operable units covered by two

QAPPs as shown below:

REVISED QAPPS

Operable Unit QAPP

East Insustrial East Industrial QAPP

North Model QAPP

West Industrial Model QAPP

West, Annexes, and Basewide Model QAPP

A remedial investigation has not been completed for any of the operable units as of June 1995.

Weston is the prime contractor for the East Industrial Operable Unit, but no longer uses its own

laboratories for sample analyses. Two other engineering contractors are doing investigative work at the

22 of 53 06/17/13 07:36 AM
other three operable units.

AFCEE administers the contracts for both the North and West Industrial operable units. The Sacramento

District of the U.S. Army Corps of Engineers administers the contracts for the East Industrial, and West,

Annexes, and Basewide operable units.

DATA PROBLEMS ENCOUNTERED

Weston performed sampling in June 1991, November 1991, and March 1992. Weston's laboratories in

Stockton, California and Lionville, Pennsylvania were used to analyze the samples. The following events

led to the Region's determination that the laboratory analyses were of unknown quality:

In January 1992, an AFCEE laboratory audit of the Stockton laboratory found major problems. The

laboratory audit identified that instruments were not properly calibrated and acceptable laboratory quality

control procedures were not followed.

It was not until September 1992, nearly 9 months later, that the Air Force notified the Region of the

potential data problems. This was about the same time that Air Force officials met with Weston to discuss

the data problems and determine what should be done to salvage the data. One of the decisions that

came out of this meeting was that Weston would validate its own data according to EPA national

functional guidelines, instead of having an independent third party do this. As discussed in this chapter, we

consider this approach inadequate.

The consensus was to have Weston...conduct a validation process...A validation by another third party

would be too long, too costly, too involved, and unnecessary because of the integrity expected from

Weston.

-Minutes of September 25, 1995 meeting between the Air Force and Weston

In July 1993, the Region determined that the quality of the data was unknown and could not be used for

its intended purpose.

QUALITY ASSURANCE PROJECT PLANS ARE DEFICIENT

Our review disclosed that the original QAPP was inadequate to detect the above cited laboratory

problems.

After the base was divided into four operable units, two new QAPPs were developed to replace the

original QAPP. In our opinion, the East Industrial QAPP omitted several important quality assurance

features. We concluded that the Model QAPP is the most comprehensively designed QAPP for any of the

five bases included in our audit. The strengths and weaknesses of the original QAPP and the new QAPPs

are summarized below and discussed in detail in the ensuing paragraphs.

DATA QUALITY ASSURANCE ITEMS IN QAPPS

Data Quality Original East OU Model

Assurance Item QAPP QAPP QAPP

Double-Blind Performance X

Evaluation Samples

Independent

Laboratory Audits

Data Quality Objectives X

23 of 53 06/17/13 07:36 AM
Data Validation According X X

to EPA Guidelines

Magnetic Data Required

Upon Request

Format for Quality X

Assurance Report

X - Item is adequately described in QAPP.

ORIGINAL QAPP INADEQUATE

As explained previously, the original QAPP was not adequately designed to detect serious data problems.

In addition, we found Weston made changes to the QAPP without the approval of either the Air Force or

EPA.

Laboratory Audit Requirement Was Insufficient

The original QAPP did not contain an adequate requirement for laboratory audits. Specifically, the QAPP

did not require an independent laboratory audit, but instead provided that the audit would be performed by

Weston. In our opinion, this resulted in a conflict of interest situation, since Weston would be reviewing its

own data. Further, the QAPP did not establish any procedures for the Air Force to review the Weston

audits. Moreover, we could find no evidence the audits were performed.

Even though external laboratory audits were not required under the QAPP, the Air Force contracted for

audits of Weston's Lionville, Pennsylvania and Stockton, California laboratories in the summer of 1990.

The audits were initiated before the sampling started at Travis AFB in June 1991.

The audits found the laboratories had not fully implemented in-house control limits. These control limits are

needed to ensure that data is of known quality, and is produced throughout the sampling and analysis

programs.

The laboratory audits concluded that the in-house control limits must be in use by the fourth quarter of

1990. The contractor reaudited both laboratories in 1992, and found the control limits were still not being

used for all methods of analyses. Even though the control limits were not established at either laboratory,

the Air Force continued to submit samples to both Weston laboratories.

In-house control limits for both precision and accuracy must be in use by the fourth quarter of 1990 as

projected.

- Air Force contractor

August 13, 1990

The Air Force contractor's reaudit of Weston's Stockton laboratory in January 1992 also found other

major problems. The contractor recommended that no more samples be sent to the laboratory until

Weston implemented an effective laboratory quality control system, and had demonstrated that it was

capable of ensuring the quality of data. The Air Force did not follow the contractor's audit

recommendations, and Weston initiated its third and final round of sampling in March 1992.

Since the Air Force was not providing the results of its laboratory audits to EPA, the Region was unaware

of the problems. If the information had been available, the Region would have had a sufficient basis to

prevent Travis AFB from submitting the final round of sampling analyses to Weston's Stockton laboratory.

Split Sampling Used By EPA to Find Problems

The original QAPP did not require split sampling, a technique where identical samples are sent to two

different laboratories for analysis and the results are compared. However, the Region performed some

24 of 53 06/17/13 07:36 AM
split sampling in June 1991, during the first round of Weston's sampling. Unfortunately, the Region did not

compare the results of the split sampling until June 1993. The results found major problems.

The results reported by EPA and Travis AFB for both soil and groundwater samples show almost no

correspondence, particularly for metals.

- Report on Comparison

of Split Samples

June 18, 1993

If the sample comparison had been done promptly, the Region would have known about the data

problems early enough to have avoided sending one or both of the remaining two rounds of samples to

Weston's laboratory for analysis.

Data Validation Not Done

The original QAPP did not have a firm requirement for independent data validation. The QAPP stated the

data would be reviewed according to EPA national functional guidelines, but did not:

Establish how often the data validation would be performed;

Require an independent third party to perform the validation; and

Require the necessary analysis documentation, known as EPA Level IV data packages. These packages

are necessary to perform data validation in accordance with EPA national functional guidelines.

As a result, the initial data validation, performed by Weston's in-house staff in October 1992, determined

that the data was usable for its intended purpose. The Air Force quality assurance contractor reviewed

Weston's validation report and found major problems that made much of the data unusable.

The analytical data from Weston are insufficient to conduct an EPA Level IV Review.

- EPA contractor's review

of Weston's data validation

January 27, 1993

Data Quality Objectives Not Specific

Data quality objectives were not established for each type of data collection and analysis effort. Further,

the objectives did not specify the data package requirements necessary to complete data validation by

EPA national functional guidelines.

Performance Evaluation Samples Not Done

The original QAPP requirement for PE samples called for Weston's laboratory quality assurance

coordinator to provide double-blind PE samples to its own laboratories at least semi-annually. While

Weston did not provide the PE samples to its laboratories, we also consider this quality assurance

procedure inadequate for two reasons.

First, PE sample evaluations should have been provided and judged by an activity independent of the

laboratory. Second, there should have been a requirement to submit the PE results to either the Air Force

or the Region.

No Requirement to Provide Magnetic Tapes

The original QAPP did not require Weston to provide electronic data, such as magnetic tapes of raw

data, needed to perform audits. We were advised that Weston refused to provide the magnetic tapes

when they were requested by the Region. It should be noted that magnetic tape audits were instrumental

in detecting the fraud at Eureka Laboratories, as discussed in Chapter 4.

QAPP Revised Without Approval

We also noted that Weston made changes to the original QAPP without Air Force or the Region's

approval.

Weston requested QAPP deviations from the Air Force involving evaluating data for usability instead of

25 of 53 06/17/13 07:36 AM
rerunning data and widening criteria. Weston apparently assumed the deviations had been approved

based upon discussions with AFCEE chemists. However, deviations from the approved QAPP had not

been formally approved.

Variances requested by Weston have not been approved in writing and can't be until they can be justified

to the regulators.

- Minutes of meeting between

Air Force and Weston

October 6, 1992

MODEL QAPP IMPROVED

Travis AFB finalized, in May 1994, a Model QAPP for three operable units. The Region did an excellent

job of ensuring most of the key ingredients for data quality assurance were both included in this QAPP,

and complied with. The only two quality assurance activities that we believe could have been added to

ensure the data quality were:

provisions for independent laboratory audits before samples are provided, and

access to magnetic tapes.

Best Practices

The Model QAPP incorporated most of the items that we believe are critical to ensuring data quality. It

adequately defined data quality objectives, and required double-blind PE samples and data validation. It

also required that data quality assurance reports be provided for EPA review.

Our review of PE samples for the North Operable Unit showed Travis AFB performed the required PE

samples. We also reviewed data validation requirements for the North and West Industrial Operable Units

and found about 20 percent of the samples were being validated. Further, an independent third party was

performing the data validation.

Quality Assurance Report Is Meaningful

The Model QAPP established an effective format for the quality assurance report. The report showed the

results of the PE samples, laboratory audits, and data validation. The report included information on the

findings, corrective actions required, and the effects on data quality assurance. The report was included

with the remedial investigation reports for EPA's review.

EAST INDUSTRIAL QAPP DEFICIENT

After the data problems became known, Travis AFB developed a QAPP for the East Industrial Operable

Unit. Although the QAPP satisfied the documentation requirements for data validation using EPA national

functional guidelines, many of the problems with the original QAPP remained.

The East Industrial Operable Unit QAPP, like the original QAPP, did not require:

Independent laboratory audits;

PE samples (although it noted that PE samples are an additional type of independent quality control);

The laboratory to provide electronic data, such as magnetic tapes of raw data, needed to perform audits.

We also noted the QAPP did not identify data quality objectives for each data collection activity.

However, the QAPP did require third-party data validation according to EPA national functional guidelines

on 10 percent of the data, and our review found that the data validation requirements were met.

QUALITY ASSURANCE OFFICER NOT APPOINTED

The Federal facility agreement, signed in September 1990, required the Air Force to appoint a quality

assurance officer. This officer has the responsibility to ensure that work is performed in accordance with

the QAPP. However, a quality assurance officer was not appointed until 1993.

To date, nearly one and a half years after the effective date of the [federal facility agreement], the Air

Force has not designated a QAO [for Travis Air Force Base].

26 of 53 06/17/13 07:36 AM
-EPA Region 9 letter to Air Force, April 15, 1992

COSTS AND DELAYS IN CLEANUP

The Air Force informed us that it paid Weston about $2 million for its laboratory data analyses. However,

the quality and usability of this data is unknown. Further, because of the laboratory analyses problems,

the due date for the original remedial investigation report for one of the operable units slipped from May

1993 to February 1996, a delay of more than 2 years.

RECOMMENDATIONS AND REGION 9 RESPONSE

Our recommendations are addressed in Chapter 2 of this report. The Region's complete response to this

chapter and our evaluation of the response are included as Appendices A and B of this report.

Chapter 6

Hunters Point Naval Shipyard

SUMMARY

Between October 8, 1990 and December 18, 1990, the Navy's engineering contractors collected over

1,200 soil samples and dozens of groundwater samples to initiate remedial investigations at Hunters Point

Naval Shipyard. These samples were sent to two laboratories. Nearly 1 years later, in May 1992, all

organic analyses from these samples were rejected due to poor quality laboratory data.

Throughout the RI/FS process, the [Region 9 remedial project manager] will be primarily responsible for

overseeing that the Navy's decisions have been based on data of sufficient quality.

-Region 9's Data Quality Oversight Plan,

March 1991

According to the engineering contractor, the cost of the rejected data was $2.5 million. It cost the Navy

another $1 million to replace the rejected data and the cleanup was set back 2 years.

We found that the Region had prepared a data quality oversight plan to help it assess the quality of the

data and the performance of the laboratories. However, the plan was not implemented. If it had been, and

had the QAPP been better designed and implemented, we believe the 2-year delay could have been

substantially reduced.

BACKGROUND

Hunters Point Naval Shipyard is located in San Francisco and covers 965 acres, of which about half is on

land and the remainder is on San Francisco Bay. It has continuously operated as a shipyard or ship repair

facility since 1869. Contaminants include paints, solvents, fuels and oils, acids and bases, metals,

polychlorinated biphenyl (PCB), and asbestos.

The base was listed on the NPL in 1989 and a Federal facility agreement to cleanup the site was signed

in 1990. Hunters Point is divided into five operable units covered by one Basewide QAPP. In addition to

the QAPP, the Region developed an oversight plan for monitoring data quality.

DATA PROBLEMS

From October 8, 1990 to December 18, 1990, the Naval Facilities Engineering Command had an

engineering contractor collect over 1,200 soil samples and dozens of groundwater samples to initiate

remedial investigations at Hunters Point. These samples were sent to two laboratories: Eagle Picher

Environmental Services and National Environmental Testing (NET) Pacific Laboratories.

Problems were first noted by the engineering contractor in November 1990, when preliminary data

received from both laboratories indicated that holding times were missed. By December 1990, the

contractor concluded that the laboratories' analyses reports were going to be significantly late, and

stopped work on this phase of the field work.

"Cursory" data reviews found more problems and the engineering contractor decided to discontinue

sending samples to the two laboratories. However, the engineering contractor thought that previously

analyzed data could be used for the remedial investigation.

When the contractor finished data validation using EPA national functional guidelines in March 1992, it

found serious data quality deficiencies, such as instrument calibration problems.

27 of 53 06/17/13 07:36 AM
Laboratory problems identified included difficulties meeting holding times because of sample overloading,

difficulties in delivering results in electronic format, and problems submitting ...full validation packages.

Additional lab problems... included instrument calibration problems, and calculation errors.

-Hunters Point Technical Meeting,

June 2, 1992

Consequently, in May 1992, the contractor recommended and the Navy agreed, that all organic compound

analyses be rejected. At the May 1992 technical review committee meeting, the Naval Facilities

Engineering Command informed the Region and the State of California of the data quality problems and

its decision to reject the data.

REGIONAL OVERSIGHT PLAN NOT IMPLEMENTED

In March 1991, the Region developed a data quality oversight plan to help it assess the quality of the data

and laboratory performance. These plans are not required and were not developed for the other bases

included in our review. We believe that the plan could have been a good tool for overseeing data quality;

however, it was never implemented.

Data Validation Oversight Not Performed

One of the major oversight areas in the plan was data validation. According to the plan, the following

steps were to be accomplished:

The Navy would validate 100 percent of the data.

The Region's data validation contractor would review at least 10 percent of the Navy's validation.

If validation problems arose, the Region's contractor could review another 10 percent of the Navy's

validation to identify the causes of problems.

If the Region finds major problems with data validation, it should meet with the Navy and discuss the

problems.

As discussed later in this chapter, the QAPP did not require the 100 percent data validation specified in

the oversight plan. It only required that 10 percent of the data be validated. Also, there was no evidence

that the Region's contractor reviewed 10 percent of the Navy's data validation. Had either of these levels

of data validation been performed, we believe the quality control problems that led to the rejected data

could have been detected about one year earlier than May 1992.

Laboratory Audits Not Performed

The second major oversight area in the plan pertained to laboratory audits. According to the plan, the

Region's project manager was responsible for implementing laboratory audits. The plan included a

detailed audit checklist and recommended an audit before the start of analytical work and during the

course of the program. However, the Region did not perform any laboratory audits.

An audit of the laboratories used for the Navy's Hunters Point Annex Remedial Investigation/Feasibility

Study activities will assist the U.S. EPA in verifying the quality of the data.

-Hunters Point's Data

Quality Oversight Plan

QUALITY ASSURANCE PROJECT PLANS DEFICIENT

We reviewed the Basewide QAPP and found it included some good features such as requiring data

validation using EPA national functional guidelines. However, it did not include any data quality objectives,

an important tool for deciding data quality.

Further, it did not establish time limits for submitting data packages, completing data validation, or

submitting data validation reports. In addition, it did not require laboratory audits, double-blind PE

samples, or magnetic tapes availability. We also noted the Basewide QAPP was not formally approved

by the Region.

Data Quality Objectives Not Defined

The OIG ESS found the QAPP did not adequately identify data quality objectives. The Region's data

28 of 53 06/17/13 07:36 AM
quality oversight plan also commented that the QAPP did not:

Clearly present data quality objectives;

Establish the analytical (data package) levels needed to meet the data quality objectives; or,

Establish acceptable error rates or confidence requirements for determining the sample size to reach the

data quality objectives.

The lack of data quality objectives may have caused many of the data problems because the objectives

are used to determine the type, quantity, and quality of data to be collected. No samples should be

collected until data quality objectives have been determined.

...the allowable error or confidence needs to be documented in the...QAPP. The acceptable error and/or

confidence is used to establish the number of samples required and the number of [quality control]

samples required.

-Data Quality Oversight Plan,

March 1991

Data Validation Requirements Deficient

The QAPP required data to be validated using EPA national functional guidelines. It was through the use

of data validation that the data quality problems at Hunters Point were found. However, the QAPP did not

include time limits for providing data packages or performing the data validation.

Data was not rejected until 1 years after sampling because data validation was not performed promptly. A

large number of samples were taken between October and December of 1990. In January 1991, the

engineering contractor started performing "cursory" validation of this data and found "serious, but not

fatal," holding time deficiencies. Holding times are time limits for analyzing samples. After limits are

exceeded, results are considered less accurate.

Cursory validation was not completed until August 1991 because laboratory data was missing. Based on

this validation, the engineering contractor decided the laboratory analytical results were, "at best,"

estimates. The engineering contractor erroneously assumed that the "degree of confidence in data

qualitywould not get worse after completion of full validation." It did.

Data validation, in accordance with EPA national functional guidelines or the equivalent, did not start until

about 10 months after sampling was completed. After completing the data validation in March 1992, the

engineering contractor decided, in May 1992, to reject all organic analyses for three of the operable units.

Problems Found by

Data Validation

. Missed holding times

. Incomplete data validation packages

. Instrument calibration problems

. Calculation errors

At a June 1992 meeting, the State requested that, "...all future reports include only fully validated data..."

While this was an appropriate request, we believe that timely validation should also have been required. In

this case, well over one year passed before full data validation determined that all organic analyses had

to be rejected.

Matrix Spike Results Excluded

Although the QAPPs generally required the use of EPA national functional guidelines, data validation

reports did not contain all required elements. Our review of two data validation reports found that matrix

spike results were not included in either report.

A matrix spike is prepared in the laboratory by adding a known amount of target analytes into the field

samples prior to laboratory analysis. They are generated to determine long-term accuracy of the

analytical method and to demonstrate acceptable compound recovery by the laboratory at the time of

sample. We believe that matrix spikes and all other items included in the national functional guidelines

should be included in the data validation reports to ensure the data is of known quality.

29 of 53 06/17/13 07:36 AM
Data Validation Percentage Not Met

We also found that the percentage requirement for data validation was not fully met. The QAPP required

data validation according to national functional guidelines and that validation be performed on 10 percent

of the samples. We found just 8.3 percent of the 12,437 samples taken from 1991 to 1994 were

validated.

Laboratory Audits Not Required

The QAPP did not require the Navy to perform laboratory audits. As a result, one of the two initial

laboratories, NET Pacific, was not audited. Further, one of the primary replacement laboratories that

analyzed nearly 3,000 samples, Kennedy/Jenks/Chilton Laboratory, was not audited.

Nonetheless, Navy contractors did some audits of laboratories performing analyses for the Hunters Point

site. However, these audits did not ensure problems were found or resolved.

In April and May 1991, after the Navy realized there was a data problem, the engineering contractor

audited two of the three replacement laboratories: Enseco, Inc.-California Analytical Laboratories and

Anametrix. We concluded that these laboratory audits were of marginal value because:

They appeared to be cursory. The audits took only three hours to complete compared to the three days

generally needed for a thorough audit.

The engineering contractor performing the audit did not report any problems. This was the same

contractor that audited Eureka Laboratories on April 25, 1991 and did not find any major problems,

although the laboratory later confessed to falsifying test results.

Moreover, when deficiencies were identified in a more thorough audit, they apparently remained

uncorrected. The Navy contracted to have an audit of Eagle Picher Environmental Services laboratory

conducted one month before the Navy started sending samples to the laboratory. The audit found several

deficiencies related to calibration and acceptance criteria. Based on written comments received from the

laboratory, the Navy's contractor felt the deficiencies were satisfactorily addressed. Subsequent data

problems at the laboratory tend to contradict this conclusion.

Eagle Picher Audit Findings

. Need acceptance criteria for calibration of balances.

. Need acceptance criteria for water system.

. Calibration standards for gas chromatography/mass spectrometry volatile analysis cannot be traced to

standards.

. No policy for documenting out of control events.

-September 1990 Navy Audit

The QAPP should be modified to require laboratory audits, and to ensure that they are thorough,

completed before the work starts on the project, and performed periodically throughout the project. The

QAPP also should require the results of the audits be provided to the Region. We believe that laboratory

audits are a key piece of the laboratory quality picture and must be considered in defensible decisionmaking.

No Double-Blind PE Samples

Double-blind PE samples were not required by the QAPP and were generally not used.

Magnetic Tapes Not Required

The Basewide QAPP did not require the laboratories to provide electronic data, such as magnetic tapes

of raw data, needed to perform audits.

QUALITY ASSURANCE OFFICER

Although the September 1990 Federal facility agreement required the appointment of a quality assurance

officer, the Navy did not appoint one until July 1994, over 3 years after the agreement was signed.

Further, the quality assurance officer, who oversaw 15 other Navy facilities, did not start work until

September 1994.

30 of 53 06/17/13 07:36 AM
COST AND DELAY CAUSED BY REJECTED DATA

According to the Navy's engineering contractor and a corresponding newspaper report, the Navy paid

about $2.5 million for the laboratory work that was rejected. Another $1 million was spent to replace the

rejected data. In addition, we believe the data problems caused at least 2 years of the 3-year delay in

completing the basewide remedial investigation.

RECOMMENDATIONS AND REGION 9 RESPONSE

Our recommendations are addressed in Chapter 2 of this report. The Region's complete response to this

chapter and our evaluation of the response are included as Appendices A and B of this report.

Chapter 7

Luke Air Force Base

SUMMARY

Since the Federal facility agreement was signed in 1990, Luke Air Force Base (AFB) has used Analytical

Technologies Laboratory, Inc.'s (ATI) Phoenix laboratory to perform most of its laboratory analyses. In

September 1994, the Air Force submitted a Remedial Investigation (RI) Report to EPA that was based on

analyses performed by the ATI-Phoenix laboratory. Just before the submission of the RI Report, EPA

suspended the ATI-Phoenix laboratory from further Federal work for allegedly using faulty test equipment

and reporting false results under another EPA program.

The Region subsequently determined that the RI Report is not sufficiently supported. As a result, cleanup

has been delayed by nearly one year thus far, and may require a major resampling effort. In our opinion,

the delay could have been avoided if the QAPPs had been more effectively designed and implemented.

...We believe that the quality of the data from the remedial investigation at Luke Air Force Base cannot

be determined to the extent that a Superfund Basewide Record of Decision...can be signed and

finalized.

-Region 9's letter to

Luke Air Force Base, June 2, 1995

BACKGROUND

Luke AFB is a 4,200-acre base near Phoenix, Arizona, with a primary mission of advanced fighter training.

The contaminants found at the site include petroleum products and volatile organic compounds, such as

benzene. The base was listed on the NPL in 1990, and has been divided into two operable units (1 and

2). Three QAPPs were prepared; a Basewide QAPP and one for each operable unit. The base is using

the services of the Omaha District of the U.S. Army Corps of Engineers (the Corps) to issue and monitor

its contracts for the environmental investigation and cleanup.

The base's engineering contractor contracted with the ATI-Phoenix laboratory in 1991 to perform most of

the analyses for the RI Report. In early 1994, a former ATI-Phoenix employee alleged that fraud was

occurring at the laboratory. On September 1, 1994, EPA suspended ATI-Phoenix from performing any

additional Federally funded work. The suspension was due to the laboratory's alleged use of faulty testing

equipment and reporting false test results for analyses performed for the Glendale, Arizona Water

System under the Safe Drinking Water Act.

The quality of laboratory data is of the utmost concern to us.

-EPA's Press Release

on ATI's Suspension

September 1, 1994

Although Luke AFB was aware of the suspension, it submitted the RI Report for Operable Unit 1 to EPA

on September 2, 1994. However, neither the Air Force, the Corps, or the Region have been able to fully

ascertain the usability of ATI-Phoenix's analyses. In August 1995, the Region determined that some of

ATI-Phoenix's data was manipulated and therefore is of unknown quality. The Region, the Air Force and

the state have agreed on the outline of a risk-based management approach that will determine the quality

of the data. Because data is of unknown quality, the Region has been unable to approve the RI Report,

delaying the cleanup by nearly one year thus far.

31 of 53 06/17/13 07:36 AM
QUALITY ASSURANCE PROJECT PLANS WERE DEFICIENT

We reviewed the Basewide and Operable Unit 1 QAPPs at Luke AFB. Although these QAPPs included

some good quality assurance features, such as split sampling and laboratory audits, they did not require:

Double-blind PE samples;

Data validation, according to EPA national functional guidelines (unless problems were found);

Magnetic tapes to be provided upon request; and

Disclosure of the primary laboratory used at the site.

In our opinion, all of the above are essential features for an effective laboratory quality assurance

program and should have been included in the QAPPs. We also noted that none of the QAPPs were

approved by the Region.

Performance Evaluation Samples Not Fully Effective

The QAPPs provided that PE samples may be used if data problems were found. We noted the Corps

used single-blind PE samples as part of its process to validate laboratories. The Corps sent single-blind

PE samples to ATI-Phoenix in August 1991 and again in March 1993. However, we do not believe that

single-blind PE samples are fully effective because the laboratory is aware it is evaluating a specific PE

sample.

The use of double-blind PE samples would be a more effective way of detecting problems, since the

laboratory would not be aware that it was being evaluated. A good example of the effectiveness of this

evaluation approach occurred at March AFB, as discussed in Chapter 4 of this report.

Data Validation Not Required

The QAPPs did not routinely require data validation according to EPA's national functional guidelines. The

Region, in a review of quality assurance procedures in 1991, advised the Corps that:

...The minimum percentage of data package selection for validation...has not been proposed by the

Corps. EPA recommends that the Corps outline a routine approach for identifying a minimum

percentage of data validation plus a strategy for selecting data.

Although this was good advice, the Corps did not incorporate a validation percentage into the QAPPs.

Instead, the QAPPs stated that data validation was required only when "substantial quality control

problems" were detected. According to the Region's Quality Assurance Management Section, data

validation should be done for a minimum of 10 percent of the samples, whether quality control problems

are found or not.

Further, the QAPPs did not require the laboratory to provide data packages needed for data validation,

unless problems were found. As a result, Luke AFB has experienced difficulty in obtaining the necessary

data packages. Without these packages, the quality and useability of ATI Phoenix's data could not be

determined, thereby delaying decisions on how the cleanup should proceed.

Even after learning of the potential laboratory analyses problems with ATI-Phoenix, the Corps performed

data validation on only 3 of 2,073 samples analyzed by the laboratory. The validation, performed in mid

1994, did not disclose any problems. However, we consider the sample size too small to effectively

evaluate data quality.

I do not believe ATI is responding in a timely manner with our/EPA's request for raw data.

-Luke Air Force Base

Restoration Section Chief

February 25, 1995

In September 1994, after suspending ATI-Phoenix, the Region requested ATI-Phoenix's data packages

from Luke AFB. ATI-Phoenix did not furnish these packages promptly even though they were required to

be provided within 30 days. In a February 22, 1995 letter, Luke AFB's cleanup manager told the Corps

that:

I will not tolerate any further delays in ATI's responsiveness to this request. Our data is only usable if EPA

determines it so via an audit of the ENTIRE list of accession numbers which I compiled. Failure to provide

32 of 53 06/17/13 07:36 AM
this entire list in a timely matter will result in a default determination that our data is not usable. I have

determined that the definition of "a timely manner" runs out 10 March 95.

Although data packages were delivered by March 10, 1995, some of the packages were incomplete and

could not be used to perform data validation in accordance with EPA guidelines. Consequently, the

cleanup has been delayed because the majority of the work cannot proceed until the useability of the data

has been determined.

Magnetic Tapes Not Available

The QAPPs did not require the laboratory to provide electronic data, such as magnetic tapes of the raw

data, needed to perform magnetic tape audits.

ATI-PHOENIX LABORATORY NOT IDENTIFIED IN QAPP

The QAPP for Operable Unit 1 included the organizational structure and selected resumes for the ATI

laboratories in San Diego and Pensacola. However, it did not identify or discuss the ATI-Phoenix

laboratory. According to Luke AFB personnel, the QAPP was written by the engineering contractor's

Tampa, Florida office, and it may have been unaware that ATI had a Phoenix laboratory. In order for the

Region to effectively evaluate a QAPP, it must have complete information about the laboratory and its

capabilities.

LACK OF QUALITY ASSURANCE OFFICER

Luke AFB, the Region, and the State of Arizona signed the Federal facility agreement in September 1990.

We believe that one of the weaknesses in the agreement was that it did not require the appointment of a

quality assurance officer. Such an appointment could be effective in helping to affix responsibility for

oversight of laboratory quality.

DELAY IN CLEANUP

The potential data quality problems have already delayed the cleanup by nearly one year. To illustrate, the

Region's review of the RI Report for Operable Unit 1, originally scheduled for completion by November

1994, has slipped to October 1995.

Further, it appears resampling will be required. According to a June 2, 1995 letter from the Region to

Luke Air Force Base:

...we believe at this time that results for volatile organic compounds..., semi-volatile organic

compounds..., and pesticides are, at best qualified...Qualified data alone is not sufficient to support a

baseline risk assessment...Based on available information, we strongly anticipate that confirmatory

sampling will be needed to support the Basewide ROD [Record of Decision].

RECOMMENDATIONS AND REGION 9 RESPONSE

Our recommendations are addressed in Chapter 2 of this report. The Region's complete response to this

chapter and our evaluation of the response are included as Appendices A and B of this report.

APPENDIX A

Region 9 Response to Draft Report

Region 9's complete response to the draft report is attached. The Region's specific comments were

referenced to the page numbers of our draft report. These page numbers have changed in the final

report. Our evaluation of the Region's comments is included in Appendix B.

NOTE: Please see the hardcopy for page references.

UNITED STATES ENVIRONMENTAL PROTECTION

AGENCY

REGION 9

75 Hawthorne Street

San Francisco, CA 94105

SEP 20, 1995

MEMORANDUM

33 of 53 06/17/13 07:36 AM
SUBJECT: Response to Draft Audit Report No. E1SKF5-09-0031 Environmental Data Quality at DOD

Superfund Sites in Region 9

FROM: Nora L. McGee, Assistant Regional Administrator

for Policy and Management (P-1)

TO: Truman R. Beeler

Divisional Inspector General for Audits

Western Division (I-1)

The attached memo, dated September 3, 1995, titled "Audit of EPA Region 9's Oversight Of Data Quality

Assurance at Federal facilities, Draft Audit Report No. E1SKF5-09-0031," and signed by Julie AndersonRubin, Chief, Federal Facilities Cleanup Office, constitutes our official management decision on the

subject audit. We feel confident that this memo is responsive to your concerns but understand from the

exit conference that we still need to better correlate our planned corrective actions to the draft audit

report's recommendations. We will provide an improved corrective action plan in our response to the final

audit report.

In the interim, should you or your staff have any comments or questions, please contact Rich Hennecke,

Regional Audit Followup/Management Controls Coordinator at (415) 744-1630.

Attachment

cc: Geary Pena, Audit Manager (I-2)

Keith Takata (H-1-S)

Julie Anderson-Rubin (H-9)

Terry Stumph (P-3)

Vance Fong (P-3-2)

Rich Hennecke (P-2-1)

UNITED STATES ENVIRONMENTAL PROTECTION AGENCY

REGION IX

75 Hawthorne Street

San Francisco, CA 94105

September 3, 1995

MEMORANDUM

SUBJECT: Audit of EPA Region 9's Oversight of Data Quality

Assurance at Federal Facilities; Draft Audit Report

No. E1SKF5-09-0031

FROM: Julie Anderson, Director

Federal Facilities Cleanup Office

TO: Geary H. Pena

Acting Divisional Inspector General for Audit

Thank you for the opportunity to provide comments on the Draft Audit Report No. E1SKF5-09-0031,

Environmental Data Quality at DOD Superfund Sites in Region 9. Included in this response are general

clarifications and comments about the report, followed by comments on specific findings and

recommendations in the report and updates on EPA Region 9's follow up to earlier findings.

Draft Report Overview

34 of 53 06/17/13 07:36 AM
As we stated in our previous written comments, we in EPA Region 9 have always recognized the

essential role that data quality assurance plays in our environmental programs. We work hard to instill in

the federal facilities we oversee an awareness of their lead agency responsibilities.

EPA's oversight responsibilities at Federal facilities are substantial and diverse. They encompass scoping

and approval of all investigations and other studies (workplans, risk assessments, ecological

assessments, removal plans, feasibility studies, treatability studies), community involvement (including

attendance at frequent Restoration Advisory Board meetings,), environmental justice awareness,

resolution of the numerous issues that often arise in remedy selection (cleanup levels, unexploded

ordnance, endangered species protection, natural resource protection, historical preservation), general

compliance with CERCLA and the National Contingency Plan, and data quality assurance. These oversight

responsibilities are compounded by the vast acreage and, frequently, hundreds of contaminated areas

involved at each facility. At closing bases, additional efforts must be directed to land use considerations,

to quickly identify properties most suitable for early reuse, and make those properties available to the

community as expeditiously as possible. Given this considerable workload at Federal facilities, EPA does

not (and likely never will) have sufficient resources to oversee all aspects of Superfund cleanup at each

facility we ideally would, let alone to actually perform DOD's lead agency responsibilities. Instead, EPA

must leverage our very limited oversight resources to provide maximum benefit with minimum investment.

One way we have historically done this is to identify to DOD issues we have observed that may effect

remedy selection -- particularly if these issues have the potential to cross multiple facilities. We then work

with these lead agencies to help them address these problems systematically.

To date, our general approach in overseeing DOD's quality assurance responsibilities has been to work

with DOD to establish basic quality assurance plans for each activity, and require DOD to ensure these

plans are implemented through establishment of quality assurance officers dedicated to this function. We

still believe this fundamental approach, well executed, is sound. However, improvements to the actual

quality assurance plans and to DOD's execution are obviously necessary. Recent data quality problems at

DOD sites (in some cases discovered by EPA Region 9) confirm that these have been breakdowns in this

approach, leading to data losses or data of limited usability. In some cases, DOD did not assign quality

assurance officers, and as a result did not implement all aspects of the approved quality assurance plans.

While EPA needs to check in more routinely on DOD's implementation of the program, simply increasing

our "policing" of DOD won't prevent or solve the fundamental problem. Instead, DOD must assign a high

priority to developing and implementing a sound Department-wide approach to assuring data quality from

its contract laboratories, as EPA has done with our Contract Lab Program. Since EPA Region 9 became

aware of this DOD problem, we have worked closely with all Services to assist DOD in building an

improved program that will fulfill their lead agency responsibility to provide data of acceptable quality.

The report does not provide the reader a clear differentiation of the EPA's role and DoD's role at DoD

Superfund sites. The report does not reference or explain the effect Executive Order 12580, which

delegated lead-agency authority and responsibility to DOD at DOD CERCLA sites. In the absence of

explanation, many readers will assume incorrectly that EPA's role at federal facilities Superfund sites is

analogous to its role at enforcement-lead Superfund sites, where potentially responsible parties are

performing Superfund work. At enforcement-lead Superfund sites, EPA is the lead agency with primary

responsibility for assuring proper implementation of NCP requirements, even though the work is being

performed by a PRP. At DOD sites, DOD retains this primary responsibility. This intrinsically places EPA

in a different, secondary position with regard to our role and responsibility at Federal sites.

While EPA retains certain authorities at federal facility sites (listing and delisting sites on the NPL,

selecting remedies jointly with DOD and DOE, entering into interagency agreements), the resources we

are provided to carry out our responsibilities are commensurate with very broad-brush oversight rather

than a lead role. The lead agency functions of carrying out all aspects of the National Contingency Plan,

preparing documents, performing fieldwork and construction, and ensuring data quality remain with DOD.

In several locations in the draft report, problems and recommendations are described in terms that imply

that EPA is the lead agency (e.g. "The Region did not...revise the QAPPs"). We recognize that the focus

of this audit was to evaluate EPA Region 9, but where a perceived deficiency is attributable to DoD it

should be so stated. Also, we suggest that the report recommend actions DoD should take to improve its

QA programs, which EPA could recommend to DOD in our capacity as overseer. For instance, EPA

should recommend to DOD that their contract documents be improved to establish QA requirements that

assure adequate prime contractor review of subcontracted laboratories despite the inherent disincentives.

While the report addresses in great detail the perceived vulnerabilities in Region 9's oversight of DOD's

quality assurance program, it fails to acknowledge our many competing responsibilities or the substantial

effort and resource investment Region 9 has made over the last 2 1/2 years to detect and prosecute lab

fraud at DOD labs, raise awareness of DOD's QA problem, and assume a national leadership role in

working with DOD to attempt to improve this situation. We have found ourselves greatly frustrated in our

many attempts to influence this larger issue, and were hopeful the Inspector General's review would

assist in assigning a higher DOD priority to improving the situation. We are disappointed that the review

35 of 53 06/17/13 07:36 AM
doesn't appear to acknowledge the basis of the problem; as written it instead seems to portray DOD's

failure to provide data of adequate quality as a problem caused by EPA's limited policing powers.

Executive Summary

1. Page i, paragraph 1

We recommend the following language changes:

At one DOD base, the Region uncovered extensive laboratory fraud was found.

Also, it's the source of the $5.5 Million/2.5 year loss should be referenced and supported.

2. Page ii, paragraph 3:

It is not EPA's role to directly oversee DOD laboratories as is implied in this paragraph; it is EPA's role to

oversee DOD's quality assurance program, which may include occasional direct EPA involvement in

reviewing laboratories under contract to DOD, or data produced by those labs. However, DOD has

generally refused to sign FFAs that provide EPA independent authority to conduct audits and inspections

of DOD contract laboratories.

We recommend the following language changes:

Although serious laboratory problems were identified, the Region had not significantly strengthened its

oversight program over DOD laboratories. Nor had it only required that DOD modify the quality assurance

project plans at one of the five sites audited ...

Further, it should be stated that the Region has increased its federal facility QA/QC oversight efforts in

other ways. The Region has instituted a PE sampling program for DOD sites, increased the number of

field and lab audits conducted (with DOD concurrence), has used several occasions to highlight the

importance of QA/QC to DOD's contractor community, has raised the DOD QA/QC problem at the

highest organizational levels of DOD and to all Service Branches, has worked collaboratively with DOD to

develop and begin utilizing electronic data validation methods, and has participated in DOD's triservice

data quality workgroup, recommending improvements to DOD's departmental data quality program.

Further, the Region's debarment of fraudulent labs and subsequent criminal prosecution of laboratory

personnel have sent effective messages to DOD and the laboratory community.

3. Page iii, paragraph 2

DQOs were a required element in all QAPPs. Often the DQOs stated in QAPPs were only statements of

method requirements or laboratory capabilities, but this element had to be addressed in some fashion in

all QAPPS undergoing QAMS review. Prior to 1994, there was no Agency guidance for defining

quantitative project data quality goals (and there is still no practical method of deriving laboratory quality

goals from statistical confidence levels for overall (field + lab) data quality). The 1987 Superfund DQO

guidance stopped at defining data quality parameters (PARCC) and 5 categories of analytical data, which

were erroneously termed as levels of data quality. We acknowledge that site-specific numerical data

quality objectives (DQOs) have not been prepared. However, the 1987 data quality objective guidance

document was considered by Region 9 staff as being not helpful in this regard.

4. Page iii, paragraph 3

We recommend the following language changes:

While the Region took action to reject data, address impacts of lost data, and increase certain QA

oversight activities such as expanding its PE sample and lab audit programs, The Region did not

effectively monitor compliance with QAPPs, or recommend that DOD revise the QAPPs after laboratory

data problems were found.

5. Page iv, paragraph 4

The intent of this paragraph is unclear. Eureka laboratory was audited several times before being used by

March AFB, and while one of the audits identified serious problems with the lab, none of the audits

detected fraud. Further, this paragraph doesn't acknowledge that data quality problems as a result of lab

fraud were finally detected and prosecuted by EPA Region 9. While it is easily said in hindsight that

stronger provisions in the QAPP to detect fraud could have prevented this problem before the fact, the

Region feels that our identification of the problem and insistence on taking corrective actions should

receive equal billing with the finding that the QAPP was not sufficient.

Chapter One

36 of 53 06/17/13 07:36 AM
1. Page 2, paragraph 1: The report should note that the Region is also responsible for oversight of twelve

closing bases which are not on the NPL, and may soon be expected to provide oversight to 8-10 more

closing bases added as a result of the fourth round of base closures.

2. Page 3, paragraph 2: The report would benefit by specifying what the IG learned about DOD's own

data quality assurance program -- i.e., you found they don't have a CLP and use multiple contractors.

What did you learn about DOD's program to fulfill its responsibility to provide data of acceptable quality

from these many contract and subcontract labs?

3. Page 5, paragraph 1: The report should indicate that the first four facilities were identified by EPA

Region 9 as sites which had experienced known data quality problems. The report should also mention

that, in the initial IG review, when asked which issues were of greatest concern to Region 9 with regard to

pace and quality of DOD's cleanup program, Region 9 expressed our continuing frustration with repeated

data quality problems, some of which we had uncovered.

Chapter Two

1. Page 9 & 10, Reliability of DOD's Laboratory QA Program Unknown

The statement in Chapter 2 of the audit report that the Region is "unfamiliar with DOD's laboratory quality

assurance program," is incorrect. It is recognized that some confusion exists concerning the individual

laboratory QA programs for the Navy, Army, and Air Force. The text in Chapter 1 (Introduction,

Background, Environmental Data Quality Requirements, Page 3) acknowledges that "unlike EPA, DOD

does not have a centrally-managed contract laboratory program for monitoring laboratory quality." As a

result, all three services operate somewhat differently.

However, it is unfair to conclude that the Region lacks expertise in understanding DOD's laboratory quality

assurance program. The Region's Remedial Project Managers (RPMs) and Quality Assurance

Management Section (QAMS) staff routinely handle issues related to DOD's program. Any criticisms of

the Region's expertise in this matter should be accompanied by acknowledgement of problems with

DOD's internal consistency. In addition, DOD staff are themselves often unfamiliar with their own QA

organizations and methods for quality assurance.

2. Page 9, paragraph 3: The report should indicate that, wherever the Rivers response indicated Eureka

lab had been used at an NPL site in Region 9, EPA staff were informed and directed to discuss the extent

of Eureka's use at the site and to assess the criticality of this particular data to remedy selection at the

site. While it may be true that EPA didn't request in writing more data, this misrepresents our awareness

and interest in the data generated by this lab, and implications on decision making.

3. Page 10, Laboratory Quality Assurance Oversight Activities at DOD Sites Need Improvement

The text in one section of Chapter 2 (Reliability of DOD's Laboratory QA Program Unknown) criticizes the

Region for providing laboratories with more than one opportunity to "pass" a performance evaluation (PE)

sample. However, the text in another section of Chapter 2 ("Double-Blind" PE Samples not Required,

page 21) implies that submission of a second PE sample, after a laboratory has failed to meet the

accuracy criteria for a first PE sample, is considered to be acceptable. The audit report provides an

example of language which was added to a QAPP for Travis Air Force Base (AFB) that requires the

analysis of double-blind PE samples. The text in this example states that "If the accuracy criteria are not

met, the cause of the discrepancy will be investigated and a second PE sample will be submitted." This

apparent discrepancy should be addressed.

4. Page 11, paragraph 2: We agree "some tracking system" should be in place, but it should be the

primary responsibility of DOD as lead agency to track its own laboratory use and the quality assurance

measures employed by those labs to validate data. For EPA to create and maintain a similar data base is

duplicative and wasteful of limited national resources.

5. Pages 11 and 15

A figure compares the 100% data validation and PE sample use at EPA-lead sites to the % at federal

sites, implying that the lower % at federal sites is inadequate. 100% data validation is not a requirement,

and a question remains as to whether it is in fact desirable. The data user, when considering the site or

project-specific need may decide, through using the data quality objective process, an adequate

percentage of data to be validated. A uniform 20% data validation may be adequate for one site and not

adequate for another.

6. Page 16, Laboratory Quality Assurance Oversight Activities at DOD Sites Need Improvement, Data

Validation was Deficient, Page 16

37 of 53 06/17/13 07:36 AM
The discussion of data for Hunter's Point Superfund Site states that data rejected in validation, generally,

are unusable. It should be noted rejected data are always considered to be unusable.

7. Pages 18 & 19, Laboratory Quality Assurance Oversight Activities at DOD Sites Need Improvement,

Computer Aided Validation Should be Implemented

The text in Chapter 2 of the audit report recommends that QAPPs for DOD sites be modified to require

the use of the EPA-developed computerized data validation program, Computer-Aided Data Review and

Evaluation (CADRE). The use of computerized data validation is recognized to provide a more efficient,

consistent, and cost effective method of data review than traditional manual validation, and its use should

be encouraged whenever feasible.

However, the audit report fails to address critical implementation problems associated with the use of

CADRE for validation of data generated for DOD sites. CADRE was expressly developed for application

to data generated in Agency Standard Format, which is the electronic data reporting format used by the

EPA Contract Laboratory Program (CLP). In order for CADRE to be useful to DOD, DOD contract

laboratories would need to generate data deliverables in Agency Standard Format. It is unclear whether

EPA has the authority to prescribe specific data reporting requirements for such laboratories. On April 10,

1995, QAMS offered the first CADRE training for Federal facility representatives. QAMS subsequently

request approval from OERR to release CADRE to Federal facilities and their contractors. QAMS also

marketed CADRE at the California Base Closure Committee meeting as a recommendation to better

assess data quality and saving resources.

At this time, it is considered to be more realistic to recommend, rather than require, that QAPPs for DOD

sites incorporate the use of CADRE. Alternatively, the specific model of electronic data validation

software that will be used should be left to the discretion of DOD.

6. Page 19

The report indicated that the Region conducted CADRE training for DOD staff on April 1995. However, no

one from the Air Force attended. QAMS has since scheduled a CADRE training for the Air Force for Fall,

1995.

7. Page 21, Laboratory Quality Assurance Oversight Activities at DOD Sites Need Improvement,

Magnetic Tapes Should be Available

The discussion of magnetic tapes in Chapter 2 of the audit report should be expanded to describe the

limited application of tape audits. It is agreed that magnetic tapes provide a valuable tool for disclosing

analytical deficiencies. However, it should be clarified that, in general, primarily data generated by

analytical methods which involve the use of gas chromatography and gas chromatography/mass

spectrometry [GC/MS] (i.e., volatile and semivolatile organic compound and dioxin/furan data, most

commonly) are suitable for tape auditing purposes. The information presented in the audit report is

misleading in that it implies that magnetic tape audits have application to all types of analytical data.

8. Pages 25-27: Recommendations See our conclusions at end of this document.

Chapter 3, Sacramento Army Depot:

1. Data Validation

Sacramento Army Depot has agreed to resample the Tank 2 Operable Unit and perform the necessary

data validation on the new samples. We propose advising the Army that if resampling indicates that

cleanup levels have not been achieved, EPA will withdraw its cleanup certification and additional

remediation will be required.

2. Recommendation

Concur with the recommendation; however, we would prefer that the recommendation be flexible enough

to allow for a cost-effective approach to the validation of all previous data. We suggest the

recommendation read: "Require the Army to develop and implement an EPA approved plan to validate

previous data used for decision making using EPA national functional guidelines or equivalents."

Chapter 4, March AFB:

1. Page ii states, "...the Region had not significantly strengthened its oversight program..."

This statement is untrue. Four other laboratories in use at March AFB were sent PE samples following the

identification of the Eureka problem. Due to either problems in preparation of the PE samples or analysis

of the samples by our laboratory, the results were not conclusive and additional PE samples could not be

38 of 53 06/17/13 07:36 AM
prepared by EPA QAMS in a timely manner.

2. Page iii states, "...the lack of site-specific data quality objectives was one reason serious problems

were found with environmental data quality."

This finding is a non-sequitur for March AFB. There is no relationship with data quality objectives and

fraud. This also would be the case on at least one other base reviewed by the IG (Luke).

3. Page iv, second paragraph under March AFB, which begins, "We found..."

The intent of this paragraph is unclear. None of the QAPPs for any of the FF in the past have measures to

protect against fraud. So this finding is not unique to March. There were two "laboratory audits" of Eureka

by the Air Force technical support. While those audits certainly identified problems with use of the lab,

those audits did not detect fraud, nor would were they designed to detect fraud.

4. Page vi, second to the last bullet.

I do not know of any evidence to support the concept that the quality of QAPPs could be assured by the

use of only government employees as quality assurance officers. Such a statement diminishes the quality

of the argument you wish to promote. EPA has had excellent technical support from a number of

contractors, and there is no reason why this will not continue to be the case. Also, "...independent

laboratory audits..." needs definition.

5. Page 39, first paragraph

Add the word, "criminal" to the sentence ending, "...laboratory pleaded guilty to criminal charges that its

test results were falsified."

6. Page 39, second paragraph

This paragraph is unclear. If the intent is to say that the QAPP did not contain a requirement to do a

laboratory audit nor did it require magnetic tape audits, this is true. However, it is also not unusual. None

of the FF QAPPs have required either of these functions. In fact, magnetic tape audits by EPA of PRP

work are not SOP in any of EPA guidance. They are done only if fraud is suspected, and the submittal of

magnetic tape is probably enforceable only through Consent Decrees or Section 104 action. It is doubtful

we could require it at FF unless it is in our guidance (The FF agrees to follow EPA guidance in the model

FFA). Perhaps the IG wished to recommend that the guidance is strengthened, but this is not the

impression I have when reading the draft.

7. Page 42, first full paragraph

The basewide QAPP was approved as part of the approval of the basewide Workplan and Sampling and

Analysis Plan. Approval of the Workplan and the Sampling and Analysis Plan were given in our letter of

December 21, 1991, in which we gave additional comments on the draft final. The Air Force agreed to our

comments in their letter dated December 30, 1991. Copies of both those letters were sent to Dan Cox of

the IG's office by mail on August 22, 1995.

8. Page 42, first paragraph under Laboratory Audits Not Required

EPA Region IX QAMS has not seen much value in "laboratory audits" in the past. Furnishing PE samples

probably has more value on a dollar-for-dollar basis. Also, the Eureka Lab problem was fraud. Fraud, as

we carefully explained to the IG staff can not be detected by a "laboratory audit". The type of fraud

performed by Eureka would not have been detected by a "laboratory audit".

9. Page 43, first full paragraph

It would appear that the real candidate for an audit would be the Air Force. Why was a sub-contract given

to Eureka when the Air Force contract technical auditor recommended, after two lab audits, that samples

not be sent to that lab? It would appear that the largest contribution the EPA IG could make to this story

is to push the Air Force to conduct an internal audit to discover the full extent of their internal QA problem.

10. Page 43, No Data Validation

This paragraph is a non-sequitur. Laboratory fraud will not be detected by Level IV data packages.

11. Page 44, Costs

Eureka Labs was paid about $845,000 for analyses of March AFB samples. An additional $144,000 was

expended for all costs associated with resampling and analysis of samples. All of the $144,000 for

resampling and analysis was paid by the engineering contractor.

39 of 53 06/17/13 07:36 AM
Chapter 5, Travis Air Force Base

No additional comments.

Chapter 6, Hunters Point Annex:

1. Rejected Data Cost Estimates

The Navy engineering contractor, PRC, informed EPA Region 9 that it estimates that the rejected data

cost approximately $1.21M, and the cost to replace this data was $945,906, not $2.5M and $1.1M as

stated in the report. Please see the attached documentation provided by PRC, the Navy's engineering

contractor.

2. Federal Facility Agreement. On page 56, Position Paper for Hunters Point, the following quotation is

inaccurate: "Although the base was listed on the NPL in 1989, a federal facility agreement to cleanup the

site was not signed until] 1992."

The original FFA for Hunters Point was signed September 1990. In January 1992, the FFA was revised

with new schedules and the addition of the Regional Water Quality Control Board as a signatory. A copy

of this earlier FFA is enclosed for your records.

3. Data Validation. On page 58 of the Report the following statement is made: "Also, there was no

evidence that the Region's contractor reviewed 10 percent of the Navy's validation. If this had been done,

we believe the quality control problems that led to the rejected data could have been detected by May

1991, about one year earlier."

We believe it is important to note that according to the Navy's engineering contractor, the sample holding

time exceedence problems were detected early enough in the process, during the contractor's cursory

review of the Phase 1A data, to result in termination of use the offending labs and initiation of use of new

labs in the Summer of 1991 for the Phase 2A data analysis.

4. Page 58, Laboratory Audits Not Performed.

The Navy is the lead agency for the project. As the IG report acknowledges, the Navy did perform some

lab audits. These audits were already part of the Navy's and their contractor's procedures and therefore,

the Region did not use its oversight resources to perform its own lab audits which would have been

duplicative of the Navy's efforts.

5. Page 61, Laboratory Audits Not Required.

The Region 9 staff are uncomfortable with the language used in the first sentences of this page: "The

QAPP did not require the Navy to perform laboratory audits. Nonetheless, Navy contractors did some

audits of laboratories doing analyses for the Hunters Point site." This statement could be rephrased:

"Although the QAPP did not include a requirement that the Navy conduct lab audits, the Navy, in

accordance with their own internal procedures, did perform audits of laboratories doing analyses for the

Hunters Point site. It is for this reason, that EPA Region 9 did not use its oversight resources to perform

its own lab audits which would have been duplicative of the Navy's efforts."

6. Page 61, Laboratory Audits Not Required. The last sentence of the first bulleted item reads, "This

conclusion appears to be contradicted by the subsequent data problems at the laboratory."

This statement supports our belief that laboratory audits cannot be relied upon to detect all quality

assurance problems that may arise and calls into question the Report's recommendations for increased

emphasis on laboratory audits.

7. Estimated Schedule Delay

EPA does not agree with the estimated 2 year delay in the overall project schedule. It would be more

appropriate to state that full characterization of the sites in question was delayed. Again, there have been

many factors which have contributed to delays in the overall cleanup schedule at Hunters Point, most

particularly, availability of sufficient capacity and funds for Navy contracts.

8. Split Sampling

During the most recent sampling of groundwater from a parking lot spring on Parcel A, a Field Sampling

Plan was prepared by EPA Region for the Navy with complete review and approval of the Region 9

Quality Assurance Management Section. This sampling occurred in late 1994 and early 1995. In addition,

EPA collected and analyzed split groundwater samples to confirm the validity of the Navy's sampling

effort.

40 of 53 06/17/13 07:36 AM
9. Revised Basewide QAPP

Remedial investigation fieldwork is near completion. The Navy, with the assistance of its engineering

contractor, is in the process of revising its QAPP for use in the long term groundwater monitoring program

at Hunters Point.

Chapter 7, Luke AFB:

1. Page vi, 1st paragraph: "To date, neither the Region or the Air Force have been able to determine if

environmental data is of sufficient quality to support remedial action."

The statement should be updated by the fact that to date, the State, EPA, and Air Force have agreed

(7/11/95) on the outline of a risk-based management approach that will determine the quality of data. The

plan, that will implement the approach, is going through regulatory review and should be finalized by

October 1995.

2. Chapter 7, Page 63, Summary: "The Region has not yet

determined whether the RI report is sufficiently supported, or if the report should be rejected."

The statement should be updated. To date, evidence has been (EPA Region 9, 8/4/95) acquired indicating

that some of the Luke AFB RI data have been manipulated by ATI-Phoenix to the extent that the quality of

the data is unknown, and therefore cannot be used for remedial decisions. Taking the data manipulation

into account with the EPA lab suspension and State of Arizona suspension of contracts, the Region has

determined that the RI report is not sufficiently supported. Negotiations are currently underway to put

together the Plan (with enforceable dates) to address the data quality issues and for the Air Force

resubmit the RI report.

3. Chapter 7, Page 64, Summary, Last paragraph: "..Neither the Air Force,...or the Region have been

able to substantiate the quality of ATI-Phoenix's analyses...Consequently, the Region has been unable to

approve the RI report, delaying the cleanup by nearly one year thus far."

The statement is not accurate and may give the impression that the Region has been primarily

responsible for delaying the cleanup for one year. The statements also implies that if the Region had

approved the RI report, the cleanup would not have been delayed. Lastly, the statements make it appear

that the Region has done nothing to determine the quality of the RI data. The statement should be

rewritten as follows:

However, the Air Force, the Army Corps of Engineers, and the Region have yet to fully ascertain the

usability of ATI's analyses. Consequently, the Region has not approved the RI report more than one year

after its submittal, which will ultimate delay the cleanup. While laboratory fraud was a primary cause of

delays at this site, lack of cooperation on the part of ATI and the Air Force with EPA and State efforts to

determine whether Luke data was impacted by fraud have further exacerbated the delays.

The Region should be given appropriate credit for its efforts to identify problems with the quality of data

as appropriate.

The statements should be revised to be more accurate. First, it should be made clear that the Region is

not primarily responsible for the delay in the cleanup. Primary responsibility for the delay should be given

to the Air Force as the lead agency, which sub-contracted the analyses work with ATI-Phoenix through the

Army Corps of Engineers. Second, EPA Region 9 approval of the RI report (for the purpose of not

delaying the cleanup), would have been an irresponsible action that would have compromised EPA's

mission to protect public health. The latest evidence (indicating direct manipulation of data by

ATI-Phoenix) shows that the Region made the right decision to withhold approval of the RI report.

The Region did provide preliminary comments the RI report (EPA Region 9, 11/7/94). The Region's

comments focused only on the metals analyses, since no evidence of fraud in that area of analyses was

found.

Between the State of Arizona (AZ) and Air Force, the Region has taken the lead in identifying and

determining that the RI data for Luke AFB was affected by ATI-Phoenix fraud (EPA Region 9, 6/2/95 and

8/4/95). In addition, the Region has taken the lead in starting the process of auditing the tapes to

determine the extent of manipulation (Region 9, 6/8/95 & 6/23/95). As indicated in the draft report, data

for the audit was not provided by the Air Force in a timely manner. Most importantly, the Region was the

lead in proposing a risk-based management approach to address the data-quality issue to the Air Force

and State of AZ (Project Manager meeting 7/11/95). The Plan outline of the approach was submitted on

8/10/95 and is under review at this time.

41 of 53 06/17/13 07:36 AM
Lastly, the Region has conducted a preliminary review of the Army Corps of Engineer (ACE) QA/QC

program (8/9/95). Our review focused on the ACE QA/QC program's ability to detect fraud with its split

sampling program. Our initial review concludes that the ACE split samples cannot be used to discriminate

between manipulated and untainted accession numbers (see comment #8).

4. Page 64, Last paragraph: "We also noted that none of the QAPPs were approved by the Region."

The statement is inaccurate. The statement should be revised to indicate that the Base-wide QAPP is a

primary document under the Luke Air Force Base Federal Facilities Agreement (FFA) and was reviewed

by the Region in accordance with the FFA.

5. Page 66, 4th paragraph: "Consequently, the cleanup has been delayed because work cannot proceed

until useability of the data is determined.

This statement is inaccurate because cleanup work is progressing at some sites at Luke AFB. For

example, a bioventing treatability study is being conducted at site SS-42, and the remedy for the OU-2

ROD (ex-situ biological treatment) is in the remedial action phase at site DP-23.

6. Page 66, last paragraph: The QAPPs did not require the laboratory to provide electronic data...to

perform laboratory audits.

The statement should be clarified. The QAPP indicates that tapes will be provided upon request from the

Air Force at any time. In, fact the Region has acquired tapes through the Air Force for the purpose of

conducting a tape Audit.

7. Page 64, QAPP features that were lacking.

It should be noted that the type of fraud committed by ATI Phoenix would not have been detected by the

features identified as lacking from the QAPP. Based on meetings with QAMs Region 9 (Steve Remaley),

the best features that should be added to the QAPP, and conducted by the Air Force should be to: 1)

conduct random tape audits (with no prior warning to the lab); and 2) The use of more than one lab for

Remedial Investigation sample analyses (to minimize impact to decision making).

8. Page 67, QAPP Has Good Practices

The section should be updated to the fact that EPA Region 9 has conducted a preliminary review of the

Army Corps of Engineer's (ACE) QA/QC Program in support of the Installation Restoration Program

(IRP) at Luke Air Force Base. The review (EPA Region 9 8/9/95), focused on the split samples and on

the program's ability to identify manipulated IRP data Luke Air Force Base.

Region 9 concluded that the criteria used to compare ACE split sample data and ATI-Phoenix data is too

broad to discriminate differences caused by manipulation (fraud). Therefore, ACE split sampling program

was not able to detect fraud.

Appendix B

These tables should include dates that the regulations, orders, directives, and guidances were finalized to

clarify which were available at different stages in the Superfund process for the sites audited. Executive

Order 12580 which delegates lead agency authority to DOD should be included.

Ongoing and Planned Corrective Actions

1. Region 9 has recently and will continue to seek to require federal facilities, through our comments on

their Quality Assurance Project Plans (QAPjPs), to follow the Agency's Data Quality Objective Process

for Superfund Interim Final Guidance dated September 1993. Region 9 will also ensure that, whenever

possible, numeric data quality objectives on precision, accuracy, completeness, and detection limits are

set prior to the approval of each QAPjP.

2. The OIG draft report found that data validation comparable to the EPA National Functional Guideline

was not performed at DOD sites. The Region has requested that several NPL federal facilities submit

representative data packages to EPA. Region 9 will perform our standard data validation on these

packages using EPA's Functional Guidelines for Data Validation. The data packages not containing

sufficient analytical deliverables and all other data quality problems associated with the other data

packages will be identified to DOD for action.

3. The Region has been working with DOD for the last 3 years to raise awareness of the quality

assurance problem and to help define an improved DOD Quality assurance program that will meet or

exceed our minimum standards. We will continue to participate in these efforts. As part of this effort, the

Region will continue to encourage DOD facilities to develop and implement, as part of their quality

42 of 53 06/17/13 07:36 AM
systems, a self sufficient performance evaluation program, and will seek to require DOD to submit PE

study results to EPA. The Region plans to score and validate as many PE sample results as our limited

resources allow and will make recommendations to DOD on follow up corrective actions when necessary.

4. In addition to recommending that the Federal facilities conduct PE samples, recently, Region 9 QAMS

has conducted performance evaluation samples for on-site and off-site labs at MCA Tustin (monthly),

Yumas MCA (twice), George AFB, McClellan AFB (twice), and March AFB. Many other performance

evaluation activities are planned.

5. Through the review of work plans and QAPjPs, the Region will seek to require DOD facilities to

perform both pre-award and routine audits of their contract labs. We will also seek to require federal

facilities to furnish all audit reports to EPA.

6. Through the review of work plans and QAPjPs, the Region will seek to require DOD facilities to

address how they will ensure the authenticity of the data. At a minimum, we will seek to require that all

organic data for GC and GC/MS magnetic tapes be archived and available to DOD and EPA upon

request. This action would provide a strong deterrence to the fraudulent activities and any poor quality

laboratory performance that have been observed in the past. Region 9 QAMS has conducted magnetic

tape audits on March AFB, Travis AFB, Luke AFB, and Yumas MCA. Additional tape audits will be

conducted as resources allow. The Region is committing resources for purchasing equipment necessary

to develop a regional tape audit capability, and will suggest DOD do the same.

7. Region 9 and OERR provided training of the use of the Computer Assisted Data Review and Evaluation

(CADRE) software on April 10-13, 1995 to the U.S. Army Corps of Engineers, the U.S. Navy and their

contractors. An additional CADRE training sessions will be provided to the U.S. Air Force later this fall.

Since the first training, QAMS has continued to distribute copies of CADRE and technical assistance to

the attendees. QAMS received confirmation that the Army and Navy will begin to implement data

validation using CADRE on a number facilities. Implementation of CADRE would help remediate

vulnerability regarding data quality deficiencies identified in the IG report.

8. As appropriate, the Region will seek modifications by DOD of their previously approved QA Plans still

in use to incorporate the IG comments regarding data quality objectives, data validation, the use of PE

samples, etc.

9 . One of the IG concerns is that federal facilities have not followed the requirements set forth in FFAs

and approved QAPjPs. The Region's QAMS has developed a QA check list to assist the remedial project

managers in their oversight of the quality system at federal facilities. The QA check list can be used

during the Technical Review Committee meetings to ensure that federal facilities comply with the

requirements in the approved QAPjPs and appropriate oversight follow up is conducted.

10. The Region is exploring the idea of providing training for federal facilities and their contractors that

would aim at improving DOD's overall quality systems, their reporting, development of expertise in the

data quality objectives process, and managing DOD's contractors' laboratories.

APPENDIX B

OIG Evaluation of Region 9 Response

A draft report was provided to the Region on July 20, 1995 and the Region responded on September 5,

1995. This response is included as Appendix A. An exit conference was held with regional officials on

September 18, 1995.

We agreed with many of the Region's specific comments, and have modified our report accordingly. The

modifications did not impact our overall conclusions.

The Region's response to our draft report (see Appendix A) included a Draft Report overview, and

comments on the report's executive summary and individual chapters. Our reaction to the Region's Draft

Report overview is incorporated into the Executive Summary section of our final report. For the balance of

the Region's response, the following chart (referenced to note numbers in the Region's response)

summarizes those comments where we did not agree with the Region. Reasons for our disagreement are

discussed in the ensuing paragraphs.

Chapter Note

Executive Summary 2,5

1 2,3

2 1,2,3,4,5

43 of 53 06/17/13 07:36 AM
4 1,2,3,4,6,7,8,9,10,11

6 1,3,4,5,6,7,8,9

7 4,6,7

Executive Summary, Note 2

The Region's actions to elevate the data quality problems to the highest levels of DOD are commendable.

However, at the time of our audit, we do not agree that the Region had significantly strengthened its own

oversight program.

Our review of regional actions for the five sites included in the audit showed that the Region had not

intensified its oversight of these DOD sites, since the discovery of fraud at March AFB in January 1993.

To illustrate, the Region did not use available oversight tools such as performance evaluation samples or

laboratory audits in 1993 or 1994 at Sacramento Army Depot, Hunters Point Naval Shipyard, or Luke Air

Force Base. Further, the Region did not require the Air Force to revise the QAPP for March AFB to

require performance evaluation samples, although this activity detected the original fraud.

Executive Summary, Note 5

The Region's actions to detect the fraud at March AFB were outstanding, and have been so recognized in

our audit report. The thrust of our report was to recommend ways that the QAPPs could be improved to

readily identify other poor performing laboratories. Specifically, it is our opinion that the coordination of

laboratory audit results between EPA and DOD would be an effective, yet inexpensive, management

technique.

Chapter 1, Note 2

Chapter 2 generally describes what we learned about DOD's quality assurance program. The objective of

our audit was to determine if the Region was ensuring that laboratory data was of acceptable quality

under Federal facilities agreements with DOD. As such, we only evaluated DOD's program to the extent

necessary to evaluate the Region's oversight.

Chapter 1, Note 3

This audit started as a special review of two Federal facility agreements: Sacramento Army Depot and

Mather Air Force Base. During our entrance conference for this review, the Region advised us that the

data quality was not being compromised during the accelerated cleanup process.

Although the Region admitted data quality assurance was "spotty" during this meeting, it did not inform us

about problems with Eureka Laboratories. In this regard, Eureka had been suspended, at the Region's

request, 6 months prior to our meeting. In fact, we first learned of problems with Eureka Laboratories

from the local newspaper. Moreover, the Region did not identify any other Federal facility data quality

problems during this meeting.

Chapter 2, Note 1

We believe our assertion that the Region is unfamiliar with DOD's laboratory program is correct.

The Region advised us, during the course of the audit, that it accepted DOD's description of its quality

assurance procedures "at face value." As detailed in Chapter 2, we found that the procedures described

by DOD were sometimes inaccurate. The Region also acknowledged that "some confusion exists

concerning the individual QA programs for the Navy, Army, and Air Force..."

Chapter 2, Note 2

The Region's stated action was not adequate to fully assess the impacts of the Eureka Laboratories

fraud. As noted in this chapter, one of the problems in assessing the effect of Eureka analyses was the

identification of the sites that were impacted. DOD has yet to adequately identify all the NPL sites using

Eureka Laboratories, on the NPL, including McClellan AFB and Sacramento Army Depot. Further, the

Region was responsible for overseeing other closing DOD sites not on the NPL, including Mare Island

Naval Shipyard and the Presidio of San Francisco. Some of these sites may have also used Eureka

Laboratories, and the impacts of such usage would also need to be determined.

Chapter 2, Note 3

The performance evaluation samples discussed in Chapter 2 are of two types. The first type, discussed in

44 of 53 06/17/13 07:36 AM
the DOD laboratory QA section, is a preaward, single-blind sample. In other words, the laboratory

knows it is analyzing a performance sample. DOD purported its procedures were the same as EPA's

contract laboratory program procedures which allow a laboratory one opportunity to pass this known test.

However, DOD procedures were not consistent with EPA procedures. To the contrary, some laboratories

were given more than two opportunities to "pass" these tests.

The performance evaluation samples discussed later (in the double-blind PE samples section), are

post-award, double-blind samples. Both EPA's contract laboratory program and DOD allow a laboratory

another opportunity to "pass" these evaluations.

Chapter 2, Note 4

Since the DOD activities have not maintained a tracking system, we disagree that it would be duplicative

or wasteful for EPA to maintain a data base of laboratories used by DOD. The establishment of a

tracking system would be a valuable tool for helping the Region determine where to focus its resources.

Further, as noted by EPA's Office of Solid Waste and Emergency Response (OSWER), there is a need to

monitor "all Superfund analytical activities, not just those analyses provided by [EPA]... laboratories."

OSWER further pointed out that non-EPA laboratory work played a significant role in supporting

Superfund.

Chapter 2, Note 5

The referenced graph compares the percentage of sites using data validation, not the percentage of data

validated. In contrast, the subsequent chart shows the percentage of data that is required to be validated

based on our review of 12 QAPPs. As noted in our recommendations, we believe that 20 percent of the

data representing all matrices, analysis types, and laboratories for decision points should be validated, at

a bare minimum. We agree that the data user may require a higher percentage.

Chapter 4, Note 1

It remains our position that the Region did not take adequate action to strengthen its oversight program

after problems with Eureka Laboratories were discovered. As noted in the Region's response, there were

problems with the EPA laboratory's preparation or analysis of data and the laboratory results were not

conclusive. Further, the Region did not submit any additional PE samples to evaluate data quality or

perform any other data quality assurance activities. Also, the QAPP was not changed to add or

strengthen the data quality assurance activities.

Chapter 4, Note 2

We agree data quality objectives do not necessarily prevent fraud. However, they help ensure that data is

of adequate quality. The adequacy of data quality objectives at specific sites is discussed in Chapter 2.

Chapter 4, Note 3

We disagree with the Region's comments. PE samples, magnetic tape audits, and laboratory audits can

be used to help detect fraud, as is evidenced by the March AFB situation. PE samples at March AFB led

to detection of the fraud, by identifying major data problems. As a result of the PE samples, magnetic

tape audits were performed and the fraud was confirmed. Also, a laboratory audit performed before the

start of the analyses found major problems that warranted further investigation. Unfortunately, none of

these data quality activities were required by the QAPP and the QAPP has not been changed to include

any of these items.

Chapter 4, Note 4

As indicated in Chapter 2, we believe that the Quality Assurance Officer position should be assigned to

government employees, since they are in a position to independently evaluate data quality.

Chapter 4, Note 6

Our recommendations concerning magnetic tape audits are fully discussed in Chapter 2. We disagree that

the Region cannot require specific quality assurance activities in QAPPs. The Region's authority to

approve QAPPs is clearly defined in 40 CFR Chapter 1, 300 and EPA Order 5360.1. The model Federal

facility agreement requires a QAPP, and QA/R-5 defines the QAPP requirements.

The Region is also incorrect in stating the QAPP did not require laboratory audits. As discussed in this

chapter, laboratory audits were required by the QAPP. The weakness was that it did not require an audit

before sampling started. It is also incorrect that none of the Federal facility QAPPs required laboratory

audits. As discussed in Chapter 2 of this report, 60 percent of the QAPPs we reviewed required

45 of 53 06/17/13 07:36 AM
laboratory audits. It should be noted that, in response to our draft report, the Region agreed to require

DOD facilities to perform both pre-award and routine audits.

Chapter 4, Note 7

The QAPP was not approved as part of the Basewide Sampling and Analysis Plan approval. The Region

gave the Air Force additional comments to be incorporated into the QAPP, after it reviewed the Plan. The

Air Force then replied in a letter that it had incorporated all of the suggested changes. However, the

Region did not provide us with any documentation showing it approved the QAPP after the Air Force

made the final changes, nor that it had otherwise ensured that the changes were made to the QAPP.

Chapter 4, Note 8

While we would agree that a laboratory audit, in itself, may not normally detect fraud, it can provide

indication of poor laboratory practices. It is our understanding that a DOD laboratory audit of Eureka

Laboratories detected fraud before problems surfaced at March AFB. However, due to inadequate

communication within DOD, no action was taken to suspend or debar the laboratory.

Chapter 4, Note 9

The purpose of our audit was to evaluate Region 9's oversight of data quality. Making recommendations

to the Air Force was outside the scope of the audit. We have suggested that the DOD Inspector General

evaluate DOD's contract laboratory program.

Chapter 4, Note 10

Again, we agree that data validation, in itself, would not normally detect fraud. It was never our contention

that it would. However, data validation is clearly a valuable tool for detecting unreliable data as evidenced

at Hunters Point, Travis AFB, and Sacramento Army Depot. Data validation is essential at key decision

points, if EPA is to make sound, supportable environmental decisions.

Chapter 4, Note 11

The Region provided different costs for the laboratory analyses and resampling work in its response to

the draft report than it previously presented. However, the Region did not provide us with any additional

documentation to support these revised cost amounts, although this information was requested. Our

report, therefore, remains unchanged.

Chapter 6, Note 1

The Region's response relating to the estimated cost of rejected data and the cost to replace data are

acknowledged. However, since the Region's response also represents an unsupported estimate, we have

chosen to not revise the report.

Chapter 6, Note 3

Our report acknowledges that "cursory" data validation found problems, and that the same laboratories

were dropped from the project. The purpose of this part of the report was to show that full data

validation, although delayed, resulted in the rejection of all organic analyses. Thus, if full data validation

had been done promptly, the rejected data could have been found up to one year earlier.

Chapter 6, Notes 4 and 5

A regional laboratory audit would not be duplicative, since the Navy did not audit NET Pacific, one of

laboratories that had serious problems. It also did not audit one of the three primary replacement

laboratories that performed 24 percent of the analysis.

Chapter 6, Note 6

Good quality, comprehensive laboratory audits can be relied on to detect serious quality assurance

problems, as was illustrated, in Chapter 4, by the Air Force's audit of Eureka Laboratories. Also, when a

laboratory audit recommends not using a laboratory, a potential problem could be avoided.

Nonetheless, our point is that the Region must obtain laboratory audit reports, and consider the depth and

scope of the audit before deciding not to conduct its own laboratory audit.

Chapter 6, Note 7

The due date for the Basewide remedial investigation has been delayed three years, from January 30,

1993 to February 15, 1996. We estimate that bad data contributed to two years of this delay due to:

46 of 53 06/17/13 07:36 AM
rejecting data (17 months); and resampling (7 months). The Region did not provide us an estimate of the

length of delays due to the bad data, although this information was requested.

We recognize that other factors may have impacted the three-year delay. The Region's comment that the

lack of available laboratory capacity was another major issue may be part of the problem with bad data.

According to the Navy contractor, "Laboratory nonperformance started impacting [Hunters Point]...in

November 1990 when, first, the inferred capacities [of both laboratories]...were exceeded." These two

laboratories provided data that was eventually rejected 17 months later.

We believe an effective laboratory audit would have prevented samples being sent to laboratories without

sufficient capacity. It is unclear why an engineering contractor would contract with a laboratory without

sufficient capacity.

Chapter 6, Notes 8 and 9

The report did not mention the split samples referenced in the Region's response because they were

done in 1995. Our audit scope covered quality assurance activities performed as of December 31, 1994.

Further, it is our understanding the QAPP will be modified by October 2, 1995. We commend the Region

for any improvements in quality assurance activities, such as split sampling and performance evaluation

samples, which are now being used at this site.

Chapter 7, Note 4

Even though the Basewide QAPP went through the same review process as a primary document, the

Region did not provide any documentation showing that it approved the final version of the QAPPs for

Luke AFB.

Chapter 7, Note 6

Section 9.3.3.4 of the Basewide QAPP is the only part of the QAPP that addresses magnetic data.

However, it did not specifically require the laboratory to provide the magnetic data needed to perform the

tape audits. Instead, it called for providing electronic data for maintaining a data base of sample results. It

should be noted that the Air Force used the provisions of its laboratory contracts, not QAPP provisions, to

obtain the magnetic tapes from ATI-Phoenix.

Chapter 7, Note 7

It is debatable whether the features lacking from the QAPP would detect fraud. However, they would

certainly detect unreliable data and provide indicators of fraud. The purpose of this audit was to

determine if the Region was ensuring that laboratory data was of known and acceptable quality. The

QAPP was prepared to ensure the quality of laboratory data. This is the goal of EPA's quality assurance

program, not just to detect fraud.

We agree that conducting random tape audits and requiring the use of more than one laboratory are good

quality assurance activities and encourage their inclusion in the QAPP.

APPENDIX C

Superfund Cleanup Process

Preliminary Assessment

The initial stage of the cleanup program is an installation-wide study to determine if sites are present that

pose hazards to public health or the environment. Available information is collected on the source, nature,

extent, and magnitude of actual and potential hazardous substance releases at sites on the installation.

Site Assessment

The next step consists of sampling and analysis to determine the existence of actual site contamination.

Information gathered is used to evaluate the site and determine the response action needed.

Uncontaminated sites do not proceed to later stages of the process.

Remedial Investigation

Remedial investigation may include a variety of site investigative, sampling, and analytical activities to

determine the nature, extent, and significance of the contamination. The focus of the evaluation is

determining the risk to the general population posed by the contamination.

Feasibility Study

47 of 53 06/17/13 07:36 AM
Concurrent with the remedial investigations, feasibility studies are conducted to evaluate remedial action

options for the site to determine which would provide the protection required.

Remedial Design

Detailed design plans for the remedial action option chosen are prepared.

Remedial Action

The implementation of the chosen remedial option is implemented.

Interim Remedial Action

Remedial actions can be taken at any time during the cleanup process to protect public health or to

control contaminant releases to the environment.

Remedy in Place and Functioning

The remedial action is functioning properly and performing as designed. These include such actions as the

operation of pump and treat systems that will take decades to complete cleanup.

APPENDIX D

Regulations, Orders, Directives, and Guidance

OSWER Directives and Publications

OSWER Directive Title

9240.0-2 Analytical Support for Superfund

9240.0-2A Further Guidance on OSWER Directive 9240.0-2

9240.0-2B Extending the Tracking of Analytical Services to Potentially Responsible

Party-Lead Superfund Sites

9320.2-3A Procedures for Completion and Deletion of National Priorities List Sites

9355.0-07 Data Quality Objectives For Remedial Response Activities

9355.1-1 Superfund Federal-Lead Remedial Project Management Handbook

9355.3-01 Guidance for Conducting Remedial Investigations and Feasibility Studies Under

CERCLA

9355.9-01 Guidance on Implementing the Data Quality Objectives Process for Superfund

9830.2 Regional Oversight of Federal Facility Cleanups Under CERCLA

9837.2B Enforcement Project Management Handbook

9992.4 Federal Facilities Hazardous Waste Compliance Manual

OSWER

Publication

9240.1-05 US EPA Contract Laboratory Program National Functional Guidelines for

Inorganic Data Review

9240.1-05-01 US EPA Contract Laboratory Program National Functional Guidelines for Organic

Data Review

Regulations, Orders, Directives, and Guidance

Number Title

40 CFR Chapter 1,

300 National Oil and Hazardous Substances Pollution Contingency Plan

48 of 53 06/17/13 07:36 AM
EPA Order 5360.1 Policy and Program Requirements to Implement the Mandatory Quality

Assurance Program

EPA QA/G-4 Guidance for Planning for Data Collection in Support of Environmental Decision

Making Using the Data Quality Objectives Process

EPA QA/R-5 EPA Requirements for Quality Assurance Project Plans for Environmental Data

Operations

QAMS-005/80 Interim Guidelines and Specifications for Preparing QA Project Plans

(No Number) Guidance on Accelerating CERCLA Environmental Restoration at Federal

Facilities, August 1994

Region 9 Guidance

Number Title

9QA-03-89 US EPA Region 9 Guidance for Preparing Quality Assurance Project Plans for Superfund

Remedial Projects

(No

Number) Basic Requirements for Quality Assurance at Region 9 Superfund Sites, July 1988

APPENDIX E

Activities Contacted During the Audit

Activity Location

Environmental Protection Agency, Region 9

Federal Facilities Cleanup Office San Francisco, CA

Quality Assurance Management Section San Francisco, CA

Environmental Protection Agency, Headquarters

Office of Solid Waste and Emergency Response,

Analytical Operations Branch

Washington, DC

Office of Research and Development,

Quality Assurance Management Division Research Triangle Park, NC

Sacramento Army Depot

Remedial Project Manager Sacramento, CA

Engineering Contractor Sacramento, CA

March Air Force Base

Remedial Project Manager Riverside, CA

Travis Air Force Base

Remedial Project Manager Fairfield, CA

Engineering Contractor Sacramento, CA

Hunters Point Naval Shipyard

Remedial Project Manager

San Bruno, CA

49 of 53 06/17/13 07:36 AM
APPENDIX F

EPA Data Analytical Levels

Level I

This level of data is typically used for screening data in the field. It is characterized by the use of portable

instruments which provide real-time data to assist in determining sampling locations. Results are often not

compound specific or quantitative. It is the least costly of the analytical levels, but data is not accurate

enough to be used for public health risk assessments, evaluating cleanup alternatives, or remedial design.

Level II

Typically employing portable instruments or on-site mobile laboratories, this level of analysis is used for

analyzing data in the field. A wide range of data quality can be generated, depending on the types of

contaminants, calibration standards used, reference materials, and personnel skills. This type of analytical

data is limited mostly to volatiles and metals and is not considered accurate enough for public health risk

assessments or remedial design.

Level III

This analytical level is laboratory analysis using standard EPA-approved procedures. Some procedures

may be equivalent to EPA's contract laboratory program but validation and documentation procedures are

normally not equivalent to EPA's program. This level of analysis is used for engineering studies and, in

specific cases, can be used to provide data for public health risk assessments.

Level IV

Level IV is laboratory analysis using EPA's contract laboratory program protocols. This data level is

characterized by rigorous quality assurance and quality control protocols and specific documentation

requirements. Level IV data packages contain information on initial and continuing calibration, gas

chromatography/mass spectrometry tuning, surrogate recovery percentages, matrix spike duplications,

internal chain of custody, and holding times. Level IV documentation is needed for data validation by EPA

national functional guidelines.

Without Level IV documentation, it is not possible to (1) evaluate many aspects of laboratory analysis,

such as the effect of interferences; (2) confirm the accuracy of quality control summaries provided by the

laboratory; or (3) verify that the analyses were actually performed.

This level of analytical data is relatively expensive and is commonly used for confirming low levels of

contamination, making public health risk assessments, and obtaining highly documented data.

Level V

Level V is special laboratory analyses obtained for EPA's contract laboratory program. Data is analyzed

by non-standard methods and method development or method modification may be required. Data quality

is method specific and data can be used in public health risk assessments.

APPENDIX G

Definitions of Quality Assurance Techniques

Computerized Data Validation

Computerized data validation is a relatively new quality assurance measure that is more efficient than

traditional manual data validation. EPA has developed an automated data validation program called

Computer-Aided Data Review and Evaluation (CADRE) that has been issued to the field.

Data Validation

Data validation is a method for ensuring laboratory data is of known quality. It involves reviewing data

against a set of criteria to provide assurance that data is adequate for its intended use.

EPA has data validation guidelines, known as national functional guidelines, for its own contract lab

program. According to EPA guidelines, data validation includes a review of documentation such as raw

data, instrument printouts, chain of custody records, and instrument calibration logs.

Double-Blind Performance Evaluation Samples

50 of 53 06/17/13 07:36 AM
Performance evaluation samples can be administered by two methods: "blind" or "double-blind". Both

methods are prepared by "spiking" a known concentration of chemicals into a contaminate-free media,

such as water or soil. When a PE sample is blind, the laboratory is aware the sample is a PE, but does

not know the chemical concentration levels.

When a sample is double-blind, the PE sample is submitted as part of a field sample shipment, so that the

laboratory is not only unaware of the concentration levels, it is also unaware that the sample is a PE. A

laboratory's analysis of PE samples are used to evaluate its ability to produce accurate results.

Laboratory Audits

Laboratory audits are on-site audits designed to identify technical areas which may cause laboratories to

improperly identify or quantitate chemicals. Audits normally evaluate a laboratory's technical expertise,

standing operating procedures, facility and equipment sufficiency, and possible sources of sample

contamination.

Magnetic Tape Audits

Electronic data, often in the form of magnetic tapes, are an output of laboratory analyses. By obtaining

magnetic tapes (or other electronic data) from a laboratory, audits can be conducted to help determine:

If the laboratory is complying with its contract;

The integrity of the laboratory's computer systems; and,

The appropriateness of any software editing.

Split Samples

Split samples are samples collected in the field which are divided into two samples. One sample is sent to

the contract laboratory and the other one is sent to an independent laboratory. The results from the two

laboratories are then compared and the differences are analyzed. Split samples can be used to verify the

use of proper analytical methodology and to detect unusual data trends.

APPENDIX H

Acronyms

Acronym Name

AFCEE Air Force Center for Environmental Excellence

CERCLA Comprehensive Environmental Response, Compensation, and Liability Act

CLP EPA's Contract Laboratory Program

DOD Department of Defense

ESS (OIG) Engineering and Science Staff

NPL National Priorities List

OSWER Office of Solid Waste and Emergency Response

PE Performance Evaluation (Samples)

QA Quality Assurance

QAPP Quality Assurance Project Plan

QAO Quality Assurance Officer

QC Quality Control

RI Remedial Investigation

(This page intentionally left blank.)

APPENDIX I

Report Distribution

51 of 53 06/17/13 07:36 AM
Distribution Individual or Activity

Office of Inspector

General

Inspector General (2410)

Deputy Inspector General (2410)

Assistant Inspector General for Audit (2410)

EPA Headquarters

Assistant Administrator for Enforcement and Compliance Assurance (2211)

Assistant Administrator for Research and Development (8101)

Assistant Administrator for Solid Waste and Emergency Response (5101)

Associate Administrator for Regional Operations and State/Local Relations

(1501)

Associate Administrator for Congressional and Legislative Affairs (1301)

Associate Administrator for Communication, Education, and Public Affairs

(1701)

Headquarters Library (3304)

Director, National Center for Environmental Research and Quality

Assurance (8201)

Director, Office of Federal Facilities (2261)

Director, Federal Facilities Restoration and Reuse Office (5101)

Agency Followup Official, Attn: Director, Resource Management Division

(3304)

Region 9

Director, Hazardous Waste Management Division (H-1)

Assistant Regional Administrator, Office of Policy and Management (P-1)

Chief, Federal Facilities Cleanup Office (H-9)

Chief, Environmental Services Branch (P-3)

Audit Management Integrity Officer (P-2-1)

External

General Accounting Office

DOD Inspector General

Air Force Audit Agency

Executive Summary

EPA OIG Home Page

EPA Home Page

Created January 6, 1997

Top of Document | OIG Home

EPA Home | Privacy and Security Notice | Contact Us

Last updated on Monday, June 17th, 2013

URL: http://web.archive.org/web/20040713194058/http://www.epa.gov/oigearth/reports/1995/ffqar9rp.htm

52 of 53 06/17/13 07:36 AM
53 of 53 06/17/13 07:36 AM