NBSIR 82-251 1 (O

Review of Material Provided by

ERA on the Analysis for Organic

Chemicals in the ERA Love Canal

Monitoring Study

u S DEPARTMENT OF COMMERCE

NATIONAL bureau of standards

NBSIR 82-2511

REVIEW OF MATERIAL PROVIDED BY

EPA ON THE ANALYSIS FOR ORGANIC

CHEMICALS IN THE EPA LOVE CANAL

MONITORING STUDY

May 1982

U.S. DEPARTMENT OF COMMERCE, Malcolm Baldrige, Secretary

NATIONAL BUREAU OF STANDARDS. Ernest Ambler, Director

Review of Material Provided by EPA on the Analysis

for

Organic Chemicals in the EPA Love Canal Monitoring Study

NBS Review Panel

H. S. Hertz

H. H. Ku

W. S. Liggett

L. T. McClendon

J. K. Taylor

E. White V

W. H. Kirchhoff, Chairman

May 10, 1982

Table of Contents

Page

Foreword 1

Introduction 3

Review of the Analysis for Organic Chemicals in

the EPA Love Canal Monitoring Study 4

Appendix A: Responses to the Template Questions 11

I. Goals and Objectives for the Acquisition of Organic

Analytical Data from the Love Canal Study 11

II. Monitoring Protocols . 16

III. Quality Assurance Protocols 21

IV. Performance of the Monitoring Program 26

V. Performance of the Quality Assurance Program 32

VI. Data Reduction and Analysis 38

VII. The EPA Audit of the GC-MS Computer Records 41

Appendix B: Documents Received by NBS 44

FOREWORD

On July 20, 1981, EPA requested NBS to review the analysis for organic chemicals

performed by EPA in its Love Canal Monitoring Study. In response to that

request, this review has been prepared by a panel of NBS scientists with

expertise in organic analytical chemistry, quality assurance and statistics.

The review is based upon written material, provided by EPA to NBS, related to

the acquisition of organic analytical data for samples collected from the Love

Canal vicinity by EPA contractors.

The charge from EPA, accepted by NBS on August 17, 1981, was to carry out the

following seven tasks:

1. Review the sample collection and analytical protocols for organic

chemicals in air, water, soils, sediments and biota to determine 'their

appropriateness for identifying and measuring the substances of interest

to EPA at Love Canal.

2. Review the adequacy of the quality assuranc e/qual ity control protocols

for all of the media in Task 1 to assure the validity of substance

identification and analytical measurements.

'3. Review the results of the quality assurance program presented in the EPA

Love Canal report to assess the performance of the analytical program.

4. Review as necessary any of the reports of the on-site laboratory audits

carried out by EPA.

5. Review the results of the EPA audit of GC-MS tapes to evaluate the

quality of the analytical program.

6. Review any of the monitoring data as necessary to help in the overall

review of the program.

7. Prepare a report on the overall adequacy of the EPA analytical and

quality assurance protocols to meet the organic chemical monitoring

goals (accuracy and precision) as set forth in the EPA Love Canal

Monitoring PI an.

On August 18, 1981, NBS received an initial set of dociments for review, and on

August 25, NBS was given an oral briefing by officials of EPA and

representatives of the prime contractor, the GCA Corporation, on the conduct of

the Love Canal Monitoring Study. At the time NBS initiated its review, no final

report on the Love Canal Monitoring Study was available. As a result of the

unavailability of such a report. Tasks 3 and 7 were modified to:

3. Review the results of the quality assurance program as revealed in

documents provided to NBS describing the performance of the analytical

program.

1
and

7. Based upon written material provided by EPA, prepare a report

summarizing the reviews conducted by NBS in Tasks 1 to 6, paying

particular attention to conclusions drawn by EPA concerning the

precision and accuracy of organic analytical data from the Love Canal

Monitoring Study.

In order to give structure and specificity to the review, the NBS panel devised

a set of questions to serve as guidance for the evaluation of the organic

analytical portion of the EPA Love Canal Monitoring Study. This set of

questions, designated in correspondence between EPA and NBS as the template for

the review, was transmitted to EPA on October 13.

On December 9, 1981, NBS submitted its review of material received from EPA.

That material consisted of a variety of documents including contractor plans,

tables of data derived from samples taken from the Love Canal vicinity, tables

of data derived from internal and external performance check samples, quality

assurance audit reports, miscellaneous letters and reports, and a preliminary

draft of portions of a final report. This latter document was designated as OEM-LC-2 by NBS. The December 9 review identified a number of deficiencies in

the Love Canal Monitoring Study on the basis of the written material provided by

EPA. In the cover letter to EPA accompanying the December 9 review, NBS stated

that the responses to the template questions would be submitted to EPA in

January 1982.

On December 15, EPA requested NBS to review additional material, and on

December 17, EPA provided NBS with a copy, designated #9, of a draft report.

NBS was informed that those sections of the draft report which dealt with the

quality assurance aspects of the study were still in the process of being

revised. NBS was requested to work with the revised versions. Since the

additional material influenced the NBS responses to the template questions, NBS

decided not to transmit the responses until the additional material had been

reviewed. By February 17, 1982, the last of the additional material provided by

EPA, which included revisions of the draft report and a copy of the final report

of the prime contractor, was received. At the request of EPA, NBS briefed EPA

on the contents of this review on April 15, 1982.

The present Review covers all material received, with the exception of the

aforementioned OEM-LC-2, which was replaced by the draft report. All documents

received by NBS are catalogued at the end of this Review.

Finally, the authors of this Review wish to acknowledge the advice, and

assistance of Prof. Michael Gross, Mr. Hugh Huffman, and Dr. James Sphon.

I

1

2
INTRODUCTION

This is a review by the National Bureau of Standards of the analysis for organic

chemicals conducted by the U. S. Environmental Protection Agency in the Love

Canal Area of Niagara Falls, New York. In conducting this review, NBS has been

guided by the seven tasks specified by EPA in the charge given to NBS.

To give additional structure to the review and to ensure that all appropriate

aspects of the Organic Analytical Program of the Love Canal Monitoring Study

were considered, NBS devised a series of questions concerning various aspects of

the program to serve as a detailed guide for the review. This set of questions

has been designated in correspondence between NBS and EPA as the "template" for

the review and the responses to the template questions are included as Appendix

A of this Revi ew.

In this Review, NBS has commented upon deficiencies or limitations that NBS has

identified in the Love Canal Monitoring Study. Not all of these deficiencies or limitations influence the conclusions drawn by EPA in its draft report and some have already been described and interpreted by EPA. The purpose of including

such comments is to call to the attention of readers particular limitations

which might not be immediately apparent or which should be considered by those

who wish to use the EPA data to draw independent conclusions. When deficiencies

or limitations are identified which might affect the conclusions drawn by EPA in

its draft report, they are so noted.

Finally, since the NBS review was to be concerned solely with the analysis for

organic chemicals, other aspects of EPA's Love Canal Monitoring Study have been

excluded, by agreement, from this review. Specifically, these are:

1. The choice of sample locations and media sampled. NBS did review,

however, how the sample collection protocols might have affected the

chemical integrity of the samples being collected.

2. The reduction of acquired data leading to contamination level simmaries

and distribution patterns. NBS did review, however, those statistical

procedures associated with the determination of precision, accuracy,

and limits of detection in the analysis of environmental samples for

organic chemicals.

3. The choice of compounds which would be determined in the samples

collected from the Love Canal vicinity. NBS did consider, however, the

effect of selecting a list of targeted substances on the overall

analytical program.

4. The analytical programs for inorganic chemicals and radioactivity

.

5. The conclusions drawn by EPA from the Love Canal Monitoring Study. NBS

did consider, however, whether or not the quality and reliability of

the organic analytical data might affect the conclusions.

3
REVIEW OF THE ANALYSIS FOR ORGANIC CHEMICALS

IN THE EPA LOVE CANAL MONITORING STUDY

EPA STUDY GOALS

The EPA Love Canal Monitoring Study was undertaken to accomplish three

expl ici t goals:

1. Determine the current extent and degree of chemical contamination

in the area defined by the emergency declaration order.

2. Assess the near-term and long-term implications of groundwater

contamination in the general vicinity of Love Canal.

3. Provide an assessment (from an environmental contamination

perspective) of the habitability of residences included in the

emergency declaration order.

EPA STUDY DESIGN FOR THE ANALYSIS FOR ORGANIC CHEMICALS

The monitoring effort involved the analysis of environmental samples for

more than 100 targeted substances. The substances targeted for analysis

were selected from those known to have been buried in Love Canal, those

previously observed in the environment near Love Canal, and the EPA list of

priority pollutants. In addition to the targeted substances, the EPA draft

report cites as one of the major safeguards in the monitoring study the

identification of the 20 most abundant non-targeted substances in each

environmental sample.

Four media—air, water, soil and sediments, and biota--were to be

independently studied using gas chromatography-mass spectrometry (GC-MS) as the primary analytical technique. A detailed quality assurance program was designed for which the number of quality assurance samples to be analyzed

equaled the number of field samples. From the analytical data, EPA was togenerate a validated data base for the environmental samples.

SUMMARY OF THE NBS REVIEW

In response to a request from EPA, NBS has reviewed the analysis for

organic chemicals in the EPA Love Canal Monitoring Study. The review is

based upon written documentation provided to NBS during the period

August 18, 1981 to February 17, 1982. (See Appendix B for the list of

documents received.) The review was confined to those subjects discussed

in the Foreword and the Introduction to this report.

4
• EPA's stated goals and objectives, as they applied to the analysis for

organic chemicals, were appropriate in that media, substances, potential

patterns of contamination, and temporal variation were considered. They

were not, however, quantitatively explicit. EPA did not define at the

outset of the study the limits of detection or the precision and accuracy

required to permit comparison of levels of contamination in the Declaration

Area with levels which constitute an environmental hazard or with levels

currently found in U.S. cities.

In the implementation of the monitoring program, the requirement for the

identification of the 20 most abundant non-targeted substances was

generally not met. Consequently, the possibility of contamination by nontargeted substances has not been conclusively excluded. This deficiency

does not invalidate, however, the investigation of targeted substances.

The quantitation limits realized by the water monitoring laboratories for

individual analytes varied by more than a factor of ten. Few laboratories

reported quantitation limits as good as those reported by the EPA

laboratory responsible for the water monitoring effort. Comparisons of

quantitation limits with data provided by EPA from other studies indicated

that the contract laboratories were not uniformly performing at the state

of the art in the water monitoring program.

In any study involving measurements, conclusions are bounded by measurement

error and limits of detection. EPA has not fully used its quality

assurance data to derive estimates of precision, accuracy, and limits of

detection. EPA has not incorporated estimates of precision, accuracy, and

limits of detection into its validated data base. Unless measured values,

including "none detected," are accompanied by estimates of uncertainty,

they are incomplete and of limited usefulness for further interpretation

and for drawing conclusions.

SPECIFIC FINDINGS

In the remainder of this Review, specific comments are grouped according to

the seven tasks in the charge to NBS from EPA. Because of the variations

in the approach taken in the analysis of samples from each of the four

different media, these comments often pertain to only one of the four

media. Additional detailed comments can be found in the answers to the

template questions appearing in Appendix A.

1. "Review the sample collection and analytical protocols for organic

chemicals in air, water, soils, sediments and biota to determine their

appropriateness for identifying and measuring the substances of

interest to EPA at Love Canal."

The approach of identifying a list of targeted substances and

requiring a search for the 20 most abundant non-targeted substances

5
was sound. Targeted substances were chosen from appropriate lists and

from the results of earlier Love Canal investigations. The criteria

established for the selection of targeted substances were reasonable.

However, EPA has not explicitly related included or excluded

substances to specific criteria. The substances sought were

representative of broad classes of organic chemicals.

Chemical identification and quantitation by gas chromatography-mass

spectrometry (GC-MS) was appropriate to the general goals and

objectives of the study and represented the best overall technique for

monitoring organic chemicals in environmental samples. NBS is not

aware of any fundamentally different analytical techniques for the

identification and quantitation of organic compounds which would have

been more suitable for the study as planned. For some individual

substances, alternative analytical methods might have been more

suitable (for example, the use of a collection medium other than TENAX

for benzene and toluene--a problem recognized in EPA's draft report).

The sample collection and analytical protocols were generally complete

for the air and water monitoring efforts and have been, to varying

degrees, widely used in environmental monitoring for several years.

The protocols for soil and sediments and for biota were modifications

of the water methods and were to be evaluated during the course of the

study. (The biota monitoring effort was experimental in nature and

the data resulting from this effort were not used by EPA to draw

conclusions from its study.) Some ambiguity existed in the details of

the protocols which could have affected the quality of the analytical

data.

The monitoring program design did not anticipate the possibility that

a significant fraction of the samples might not contain any detectable

levels of analytes. Consequently, the significance of the limits of

detection and quantitation and how these limits might affect the

conclusions drawn from the study were not adequately considered.

2. "Review the adequacy of the quality assurance/quality control

protocols for all of the media in Task 1 to assure the validity of

substance identification and analytical measurements."

The number, nature, and frequency of analysis of quality assurance

samples specified by EPA should have been adequate to maintain quality

control.

The quality assurance plans for the four media (air, water, soil and

sediments, and biota) were similar in scope but varied in detail. The

quality assurance plans for the participating laboratories were, in

general, complete and followed the outline given in the overall

Quality Assurance Plan. These plans did, however, vary in detail from

one laboratory to another. The variations in detail could lead to

6
difficulties in comparing data obtained from different laboratories

and different medi a.

The Quality Assurance Plan encouraged but did not require the use of

control charts. Uniform reporting of corrective actions taken

when quality control was lost was not required. It will be difficult

at best to evaluate further any data reported by a laboratory without

an accompanying quality control chart (or equivalent data).

The Quality Assurance Plan did not address how the quality assurance

data would be used to qualify the environmental data, that is, to

assign uncertainty limits to each measured value, including "none

detected."

Estimates of the limits of detection are critical to the conclusions

which can be drawn when the majority of samples contain no measurable

concentration of analytes. The Quality Assurance Plan did not

anticipate such a possibility.

3. "Review the results of the quality assurance program as revealed in

documents provided to NBS describing the performance of the analytical

program."

The quantitation limits realized by the water monitoring laboratories

for individual analytes varied by more than a factor of ten. Few

laboratories reported quantitation limits as good as those reported by

the EPA laboratory responsible for the water monitoring effort.

Comparisons of quantitation limits with data from other studies (where

available) indicated the laboratories were not uniformly performing at

the state of the art.

The limits of quantitation to be achieved by the air monitoring

laboratories were set at concentration levels comparable to levels

observed for some U.S. cities.

As demonstrated by the audit of the GC-MS computer records, the water

and the soil and sediments monitoring laboratories did not generally

identify non-targeted substances. This aspect of the performance of

the air monitoring laboratories was not evaluated by EPA.

In isolated instances, contamination of the samples during sample

collection or analysis may have obscured possible trends and patterns

in the environmental data.

The transcription of the data and the verification of the accuracy of

the transcription and transmittal of the data into the final data base

were adequately demonstrated.

The documentation provided by EPA indicated that the quality control

limits set in the Quality Assurance Plan widened during the study.

7
The degree of change was greater for the water program than for the

air program. The reasons for the observed changes have not been

explained in the documents provided by EPA.

The estimation of the precision and accuracy of the nunerical values

of concentrations of targeted substances has not been adequately

docunented. Only a portion of the available quality assurance and

environmental data has been used in arriving at these estimates.

The estimates are generally unconfirmed and estimates for some of the

analytes have not been given.

The estimation of the limits of detection and quantitation are also

not adequately documented. Because the conclusions of the study rest

on comparisons of the frequency of compound detection, the values

obtained for the limits of detection of individual substances can influence significantly such conclusions.

The rejection of data has neither been clearly described nor

adequately justified.

4. "Review as necessary any of the reports of the on-site laboratory

audits carried out by EPA."

The reports of on-site laboratory audits reviewed by NBS described

audits conducted by EPA and by the prime contractor during the early

stages of the Love Canal Monitoring Study. These reports indicate

that all laboratories were having some difficulties and some laboratories were having great difficulty at the start of the study.

In particular, some of the laboratories were experiencing difficulties

in adjusting to the use of capillary colunns.

The prime contractor maintained telephone contact with participating

laboratories during the course of the study, but there is no evidence

of further monthly on-site visits as prescribed by the Quality

Assurance Plan. Such on-site audits would have confirmed whether or not initial difficulties had been overcome, whether or not laboratory

contamination of samples was a problem, whether or not good laboratory

practices were being followed, and whether or not analog data (e.g.,

chromatograms) from the GC-MS runs indicated laboratories were

operating at the expected level of performance.

5. "Review the results of the EPA audit of GC-MS tapes to evaluate the

quality of the analytical program."

The EPA audit of GC-MS computer records was not an evaluation of the

quality of the entire analytical program, but only an evaluation of

one aspect of that program, namely the interpretation of the computer

8
records. The audit evaluated a sample of the computer records

generated in the water and in the soil and sediments monitoring

programs. No audit of the air or biota monitoring programs was

performed.

The conclusions of the audit have not been completely justified, e.g.,

the claim that missed compounds were predominantly in heavily

contaminated samples is not supported.

The implications of the audit--that significant difficulties existed

at the limits of detection--to the conclusions of the study are not

discussed in documents provided by EPA.

Eighty samples (water, soil and sediments) were audited for the

identification of non-targeted substances. In 58 of these samples,

both the audit laboratory and the analytical laboratory identified no

non-targeted substances. In the remaining 22 samples, the audit

laboratory identified 84 non-targeted substances while the analytical

laboratories found only 1. This result indicated that, for the

samples audited, most laboratories did not identify non-targeted

substances.

"Review any of the monitoring data as necessary to help in the overall

review of the program."

Precision, accuracy, limits of detection, and limits of quantitation

have not been incorporated into the validated data base. That is, the

measured values, including "none detected," in the validated data base

are not accompanied by estimates of their uncertainty.

The information given is not sufficient to assist others in the

interpretation of the Love Canal data. Information on' 1 aboratory-tolaboratory variability is incomplete. In particular, recovery factors

have not been given nor confirmed for most analytes in water and in

soil and sediments samples.

In the presentation of information, the distinction between plans and

attained performance is not always clear. Examples selected to

support the conclusions of the study have not always been confirmed as being truly typical of the data in general.

Conclusions of no discernible patterns or differences or of no

significant c on tami nation are bounded by the limits of detection and

quantitation. In the presentation of its conclusions, EPA has not

addressed explicitly how those conclusions are influenced by the

limitations in its analytical data. Unless measured values, including

"none detected," are accompanied by estimates of uncertainty, they are incomplete and of limited usefulness for further interpretation and

for drawing conclusions.

9
7. "Based upon written materrial provided by EPA, prepare a report

summarizing the reviews conducted by NBS in Tasks 1 to 6, paying

particular attention to conclusions drawn by EPA concerning the

precision and accuracy of organic analytical data from the Love Canal

Monitoring Study."

The foregoing discussion, along with additional, detailed comments

appearing in Appendix A, constitute the report summarizing the reviews

conducted by NBS in Tasks 1 to 6.
APPENDIX A

RESPONSES TO THE TEMPLATE QUESTIONS

Template questions were designed by NBS to serve as a guide for its review, to

give structure to the review, and to ensure that all appropriate aspects of the

analysis for organic chemicals in the Love Canal Monitoring Study were

considered. NBS has not always answered each subquestion but has used each

group of questions as guidance for its review. All questions are included for

completeness.

I. Goals and Objectives for the Acquisition of Organic Analytical Data from the

Love Canal Monitoring Study.

Scope: To evaluate the consistency of the goals and objectives for the

acquisition of organic analytical data with existing state-of-the-art

methodology.

A. What were the goals and objectives of the Organic Analysis Project? Has

EPA addressed in the written documentation whether or not these goals

were met?

The goals and objectives of the Love Canal Monitoring Study were presented

on Page 1 of the draft report reviewed by NBS. As stated therein, these

were:

1. Determine the current extent and degree of chemical contamination in

the area defined by the emergency declaration order.

2. Assess the near-term and long-term implications of groundwater

contamination in the general vicinity of Love Canal.

3. Provide an assessment (from an environmental contamination

perspective) of the habitability of residences included in the

emergency declaration order.

These goals were defined in somewhat more detail on Pages 53 and 54 of the

draft report as follows:

11
1.

To characterize in each environmental mediun the incremental extent

and degree of chemical contamination in the Declaration Area

directly attributable to Love Canal.

2.

To determine potential temporal variability in contamination and

infer the causal mechanisms (for example, changes in climate)

influencing the observed contamination patterns.

3. To determine if swales, sewer lines, and other geological features

(for example, sandy soil deposits in the form of sand lenses) had a significant effect on the migration of toxic substances from the

canal

.

4. To determine the presence and direction of ground-water flow in the

area, and evaluate the effectiveness of the remedial construction

performed at Love Canal.

5. To investigate the use of locally available biological systems as potential indicators of toxic substances present in the environment.

6.

To obtain integrated multimedia measurements of environmental

contami nation.

7.

To provide from an environmental perspective an assessment of the

relative habitability of Declaration Area residences and the shortterm and long-term implications of observed environmental

contami nation.

In the section on implementation beginning on Page 54, immediately following

the statement of the objectives of the study, it is stated:

"The EPA studies were initiated by first identifying the data

requirements of the overall objectives and then designing data

collection mechanisms appropriate for such activities."

These identified data requirements have neither been described further nor

have they been presented in the report beyond the non-specific statements on Page 61:

"The common objective of these plans was to collect and analyze a statistically adequate number of samples to characterize accurately

Declaration Area contamination caused by Love Canal, and to minimize the

effects and uncertainties associated with the constrained .samp 1 ing

period."

and Page 62:

"...a primary goal of qualitative accuracy for organic analyses (that

is, correct identification of detected substances) was established."

12
These statements do not address in quantitative terms the data requirements

of the goals and objectives of the study. For example, the data

requirements could have been made more quantitative by defining the

concentrations of targeted compounds which had to be detected and/or

quantified in order to compare levels of contami nation in the Declaration

Area with levels known to constitute an environmental hazard and/or with

levels currently found in U.S. cities.

On Page 70 of the draft report, statements concerning the expected

performance of analytical methodology are made:

"Precisions better than 50 percent RSD [relative standard deviation]

were expected in water and air; precisions better than 100 percent were

expected in other media."

"Furthermore, it is also known that in water, the minimum method

quantitation limits expected for the methods used are in the range of 1 to 10 micrograms per liter (parts per billion) Quantitation limits

below these values were neither required nor expected of the analytical

subcontractors, except as noted previously for 2,3,7,8-TCDD and certain

pesticides."

With the exception of the 2,3,7,8-TCDD, these were expectations rather than

data requirements. Without more quantitative specificity, it is difficult

to see how EPA could design a monitoring program to meet its requirements

.

In the presentation of its conclusions, EPA does not address explicitly how

limitations in the analytical data affected the conclusions. Whether or not this is a direct consequence of the absence of quantitative precision

and accuracy goals cannot be determined. Nevertheless, those who wish to

use the Love Canal data or wish to interpret the conclusions of the Love

Canal Monitoring Study must read and understand the entire report in detail,

including the Appendices, to gain even a qualitative sense of the

limitations of the data gathered in the study.

The "primary goal of qualitative accuracy" is partially addressed in the GCMS audit performed by EPA and is discussed in more detail in Section VII of

this Appendix.

B. Would the approach of selecting a list of targeted compounds in any way

hinder EPA from observing, identifying, or quantifying significant

quantities of other compounds? Should EPA review all of the GC-MS tapes

to determine if significant chemicals might have been missed?

The approach of selecting a list of targeted compounds should not have

hindered EPA from observing, identifying, or quantifying other compounds

which are amenable to GC-MS. Furthermore, this approach is appropriate to

the study and the list is reasonable. By restricting the analytical

13
methodology mainly to GC-MS, those compounds which are either too polar or too nonvolatile to analyze without derivati zation would not be detected. To derive the list of targeted compounds, EPA selected from a list of chemicals

which were known to have been dumped in the Love Canal, chemicals found in

previous studies of the Love Canal, and certain chemicals generally

recognized as toxic, associated with industrial production, or observed in

other studies as environmental contaminants—the so-called "priority

pollutants". EPA should state in its report which compounds were selected

from which lists. The list of targeted compounds was not all-inclusive.

For example, a previous study of the Love Canal area (J. Barkley, et al .

,

Biomed. Mass Spectrom. 7(4), 139-146 (1980)) cites estimated levels of

1,1,1-trichloroethane and trichloroethylene in air inside homes in the Love i Canal area, but these compounds were not in the list of targeted air

compounds in Table 1-2, Page 1-4 of the draft report.

All laboratories were instructed to identify the 20 most abundant substances

beyond those appearing on the target list. The audit of the GC-MS computer

records (discussed in Section VII below) indicated that the requirement for

identifying non-targeted compounds was generally not met. Further review of

the GC-MS computer records for non-targeted compounds missed in the initital

analyses would be useful only if it were to become necessary to evaluate the

significance of these compounds.

In the description of the development of the list of targeted compounds, it

was noted:

I "...the intentional inclusion of specific substances on the target list i that were known to serve as effective and efficient tracers of

subsurface migration of leachate was designed to permit a comprehensive

determination and assessment of migration patterns from the canal

source."

Nowhere in the report are these compounds identified or justified as efficient tracers, nor were any results presented which give the reader of

the report an indication of whether or not these efficient tracers migrated

through the subsurface area in the Love Canal region.

C. Were the accuracy and precision goals for the organic data clearly

established at the outset and have the criteria for these goals been

clearly expl ained?

Section 3.3.2, Pages 68 to 70, of the draft report contains a description of

the precision and accuracy goals for the study. This section deals

primarily with the accuracy of compound identification and provides some

references to the precision which might be expected for quantitative

analyses such as those conducted by EPA in its Love Canal study. Precisions,

of 2 to 13 percent relative standard deviation (RSD) for water analyses, 50

percent for air analyses and 100 percent for other environmental media were

14
quoted in the draft report as generally being acceptable. These estimates

of precision did not serve as limits to be used in a quality control

program, but as guidelines for what could be expected from a study such as this. In its analyses of the data acquired from the Love Canal samples, EPA

accepted, in some cases, data which had uncertainties as high as one or two

orders of magnitude (cf. Table 1 1 1- 5 , Page 1 1 1—28 of the draft report).

Criteria (reviewed in Sections II and III below) were established for the

acceptance or rejection of data and were based on laboratory analyses of

quality assurance samples. However, accuracy and precision goals for the

organic data were not established at the outset. The draft report includes

statements of precision and accuracy which varied from one procedure to

another and from one laboratory to another.

0. If inadequacies in the precision and accuracy goals are identified in

answering the above questions, will these inadequacies affect our

judgment of the quality of the organic analytical data?

The establishment of precision and accuracy goals, the training of

laboratory personnel to meet those goals, and the use of quality control

charts based on those predetermined goals would have improved data quality.

Without precision and accuracy goals which have been formulated on the basis

of the overall goals of the study, it is difficult to determine if these

overall goals were actually met. Moreover, the lack of clarity in relating

the precision and accuracy estimates of the acquired data to the overall

conclusions of the report will make decisions based on those conclusions

difficult to justify. This lack of clarity is illustrated by the following

excerpt from the discussion of air contamination on Page 183 of the draft

report:

“Even though maximum concentration levels are often of considerable

interest to individuals, because in some way they may be thought to

represent 'worst case' estimates of environmental contamination,

problems of interpretation exist. The reason for this is because both

the occurrence and reliability of the obtained maximum values may be

plagued by measurement problems. To illustrate this point, it is often

the case that maximum concentrations are reported by only one analytical

laboratory and on one particular date, whereas other analytical

laboratories do not report concentration levels anywhere near such

maxima (and in some cases do not even report concentration levels above

the minimum detection value)."

Statements such as these allude, in a general way, to the lack of precision

and accuracy in the data but are not supported by quantitative information

in the draft report. They give little guidance to the user of the data on how the lack of precision and accuracy affects conclusions that might be

drawn.

15
II. Monitoring Protocols

Scope: To evaluate the consistency of the monitoring protocols with the

goals and objectives of the Organic Analysis Project and with existing

state-of-the-art methodology.

A. Have the sources of the protocols used by EPA been identified? Are

these protocols well documented? Had they been evaluated and did EPA

consider these evaluations in their selection? Are there additional

references EPA should have considered in the selection of these

protocol s?

The analytical methodology used in the study is described in the docunent

"Qua! ity. Assurance Plan: LC- 1-619-206, Appendix B" by the prime contractor

for the EPA study, the GCA Corporation.

Those methods which were used in the water analyses. Methods 608, 624, and

625, had been, prior to the initiation of the study, published in the

Federal Register. These methods were in the process of being validated by

interlaboratory tests, and such validation may now be complete. To the best

of our knowledge, such validation was not available at the outset of the

Love Canal Monitoring Study and hence could not be used as a guide for

quality control. When Methods 608, 624, and 625 were published in the

Federal Register in December 1979, EPA asked for and received comments on the methods. These comments have been considered by EPA, were found in some cases to be germane, and if incorporated would have led to improvement in

the clarity of presentation or in the actual performance of the methods.

These modifications were not incorporated into the methods used in the Love

Canal Monitoring Study. Nevertheless, Methods 608, 624, and 625 have been

widely used and in our opinion represent a reasonable choice for the study.

The methods used for the air analyses are well referenced in the draft

report and are described in detail in the aforementioned Appendix B of the

Love Canal Quality Assurance Plan. To our knowledge these methods have not

been formally validated.

The sample preparation steps in the methods used for soil, sediments and

biota samples were experimental in nature and had to be developed as part of

the Love Canal Monitoring Study. They were, consequently, unevaluated

beyond the evaluation appearing in the draft report. A nunber of the

protocols which, taken together, comprised the methods for the soil,

sediments and biota were still optional at the time Appendix B of the

Quality Assurance Plan was released. Although the methodology was fixed by

the end of the study, it is unclear whether all the data acquired for the

soil, sediments and biota were obtained using the methodology described in

the draft report. 9 ]’

16
There are no fundamentally different analytical techniques for the detection

and quantitation of organic compounds which would have been more suitable

for the study as planned.

B. Does EPA discuss the precision and accuracy to be expected from the

chosen protocols? Does EPA document the minimum detectable limits for

their targeted compounds? Are the precision, accuracy and minimun

detectable limits claimed by EPA reasonable and consistent with our

knowledge of or experience with the methodology? Are these claims

supported by reference to available literature?

Since none of the methods used in the study had been subjected to a formal

inter! aboratory validation prior to the study, expectations concerning the

precision, accuracy and minimum detection limits had to be drawn from the

reported experience of analysts who had used the various methods.

For air methods, EPA established limits of quantitation for targeted

compounds; limits of detection were to be determined by each laboratory.

For water methods, the limits of quantitation for targeted compounds were to

be determined by each laboratory; expected limits of detection were reported

in the Federal Register description of the methods.

For many of the protocols, expectations of precision, accuracy, and

detection limits were unavailable and were to be determined during the

course of the study from the quality assurance data. The protocols did not,

in all cases, explain how such determinations were to be made. Moreover,

such guidance as was given varied from one environmental medium to another.

For example, precision was to be determined for air monitoring methods from

data on duplicate measurements and sample splits and the accuracy from data

on spiked TENAX tubes and polyurethane foam plugs, while for total organic

carbon determinations in water, the methods called for the analysis of

blanks and replicates but had no requirement for the determination of

accuracy.

Not all claims on expected precision, accuracy, and detection limits were

supported by appropriate references. The requirement that they be

determined by the individual laboratories during the course of the study is

correct and always necessary.

C. Can we identify targeted or non-targeted compounds which the monitoring

protocols would be likely to miss? Can we recommend ways in which EPA

could justify the completeness of its identification of compounds which

it has not presented in the written documentation supplied to NBS?

17
In general, the protocols were capable of detecting all targeted compounds.

Because of the composition of TENAX, and the allowed lapse time between

collection and analysis of samples, the use of TENAX for the analysis of

benzene and toluene in air is inappropriate at the levels encountered. This

problem has been recognized by EPA and is commented upon in its draft

report.

The protocols are likely to miss certain non-targeted compounds. In

particular, compounds which are too polar or not sufficiently volatile will

not be detectable using GC-MS.

D. Are the protocols as written complete? Do they omit steps or

precautions that would be expected to affect the identification or quantification of certain organic compounds? Were all of the following

steps clearly described in the protocols, were they complete?

1. Sample Collection Techniques

2. Sample Storage and Transportation

3. Chemical and Physical Manipulation of the Samples Prior to Analysis

4. Calibration

5. Analysi s

6. Records

The protocols were sufficiently complete in the sense that they could be

followed by competent analytical chemists with experience in organic

analysis. The protocols for the sample collection, for storage and

transportation, and for record keeping were satisfactory. Some ambiguities

in the remaining steps could have resulted in actual differences in how the

protocols were executed. Such differences could introduce difficulties when

comparing data from the various laboratories.

Instances of incompleteness or ambiguity in the protocols are listed below, j

(1) The analytical procedure for volatile organics on TENAX allows for the use of an internal standard, dependent on the availability of the

apparatus needed to perform spiking (Quality Assurance Plan, Appendix B,

Section 1.2.2C). This apparatus is not defined in the procedure and the

addition of an internal standard is optional. Comparability of data

between laboratories requires comparable methods of internal standard

addition.

(2) Mass spectrometri c confirmation of gas chromatography-electron capture

(GC-EC) and high performance liquid chromatography (HPLC) measurements ; for pesticides collected on polyurethane foam plugs are to be conducted on composited or selected samples. The choice of these samples is not

18
specified. Furthermore, Subsection F, Section 1.3.1, Appendix B of the

Quality Assurance Plan of this analytical procedure directs the analyst

to “confirm results as required by combined GC-MS" but does not specify

what constitutes “as required."

(3) The analyst is given the option of choosing from four methods, involving

different extraction schemes and packed or capillary columns for the

analysis of semi -volatile organic compounds in water.

(4) The analytical procedure for the determination of total organic carbon

in water samples requires a chemical preservation step to be performed

in the laboratory within two hours of collection. The QA procedure

submitted by JRB Associates specifies that samples to be analyzed for

total organic carbon and total organic halogen content are to be

transferred to the GCA sample bank within four hours of collection.

(5) The analytical method for the determination of pesticides in sediments

calls for the use of a procedure entitled “Sample Preparation and

Analysis of Bottom Sediments" with the exception that part 5 of the

method has been modified and part 4 of the method is to be disregarded.

There is no statement of what is to be done in place of part 4. (Part 4

is the section dealing with sample preparation and extraction

procedures.) One could assune that Method 608 is the replacement, but

this is not specified. In this same bottom sediments procedure, the gas

chromatographic step calls for proceeding as described in Section 11. A.

There is a Section 11 of Method 608 which deals with gas chromatogr aphy

but there is no subsection 11. A.

E. Do the protocols adequately address the prevention of sample

contamination from impure reagents, the sampling process, contaminated

sample containers, etc.?

With few exceptions, the protocols dealt with the problems of analytical

contamination in a reasonable manner. These exceptions included the

problems of benzene and toluene contamination of TENAX mentioned in II.

above and problems with vagueness or inconsistency in instructions on how to

correct for common sources of analytical contamination. For example, the

quality assurance procedure for the analysis of semi-volatile compounds in

soil and sediment required that the method blank not show any signal

corresponding to compounds on the list of targeted compounds, but if common

laboratory contaminants such as phthalate esters (which were on the targeted

list) are found, the reported values for the corresponding compounds in the

environmental samples must be reduced by the amount found in the blank.

This instruction is vague in that levels of contamination for which method

blank corrections are valid are not given.

19
F. In the documentation provided by EPA, is the intended use and purpose of

the data from the control samples described? Is this intended use

reasonable? Could it have been expected to achieve its intended

purpose?

The control samples, that is, samples from the Control Area, were designet

for comparison of levels of contamination in the Declaration Area with

levels outside that area. (As such, they were not intended as analytical

controls and should not be interpreted as such. They are, within the

context of a review of the analytical program, to be treated as environmental samples, indistinguishable from any other environmental

samples.) Their use and purpose are described and the intended use is

reasonable. Determination of whether or not they were sufficient in number

and kind to have achieved their intended purpose is beyond the charge from

EPA to NBS.

G. Are there any means, not addressed above but available to EPA at the

beginning of the study, by which EPA could have improved upon the

selection, description or justification for use of the selected

monitoring protocols?

Improvements in the methodology have been addressed in the sections above,

20
[II. Quality Assurance Protocols

Scope: To evaluate the consistency of the quality assurance protocols with

the accuracy and precision objectives of the study and to evaluate the

adequacy of these protocols to maintain quality control over the reported

data.

A. Was the quality assurance plan sufficient to maintain quality control?

If not, what were the major flaws in the plan? Were there any necessary

elements missing?

A major feature of the Love Canal Monitoring Study was the quality assurance

program for the study. The prime contractor, GCA, prepared an extensive

quality assurance plan, and all but one of the participating subcontractors

prepared quality assurance plans in response to the GCA plan. These plans

contained a large variety of quality assurance measures including document

and report control, laboratory control standards, replicate analyses,

internal standards, calibration checks, and surrogate standard additions.

Quality assurance must be recognized as consisting of two factors: quality

control and quality assessment. The quality control aspects of the plans

were generally adequate, especially those portions which dealt with the

collection and handling of samples and maintenance of records, including

chain of custody. The Quality Assurance Plan was deficient in the manner in

which it addressed quality assessment. The interrelation of individual

items and the manner in which quality assurance data was to be analyzed and

used were not clearly described.

Protocols varied between media. Examples of this variability from Section 7

of the Quality Assurance Plan will be described in the following paragraphs.

(1) For air samples collected on TENAX cartridges, the use of an internal

standard was optional and this standard when used was to be added to the

TENAX cartridge? TnFernal standards should have been mandatory . For

air samples collected on polyurethane foam plugs, an internal standard

was mandatory, but was added to the sample extract. Moreover, the

procedure stated "if a suitable compound can be determined a single

internal standard will be added to each sample extract." The standard

should have been specified and should have been added as early as possible in the analytical procedure. For the determination of dioxin

(TCDO) in air samples, chlorine-37 labeled TCDO was added to each sample

as an internal standard. This was appropriate.

(2) For those air samples collected on polyurethane foam plugs, a three

point calibration curve of all compounds was required each day; a single-concentration, mixed standard was required after every fifth

21
sample; and one spiked foam plug was to be analyzed per analysis batch

per day. For those air samples collected on TENAX, only a single

calibration check was required per eight hours of analysis. No

discussion is presented describing why the two protocols were so different.

(3) For the determination of semi -vol ati 1 e organic compounds in water,

deuterated internal standards were required (but the draft report

contains no mention of their use). For the determination of volatile

organic compounds in water, internal standards (non-deuterated) were optional. For the determination of pesticides in water, the analyst is

referred to Section 7 of Method 608 which recommends the use of

fortified samples. The use of internal standards should have been

required.

(4) For the determination of volatile and semi -vol ati 1 e organic compounds in water, surrogate compound additions were required. Recoveries of

surrogate compounds were to fall within specified ranges. (The ranges

described in the draft report were substantially different from the

ranges in the Quality Assurance Plan.) For pesticides, the use of

fortified samples was recommended (by reference to Method 608).

(5) For the determination of pesticides, volatile organic compounds, and

semi -vol ati le organic compounds in water, laboratory control standards

were required. Control limits were established for the recoveries of

the laboratory control standards. For pesticides, the control limits

were specified as "±2 standard deviations." For volatile and semivolatile organic compounds, the control limits were specified as ranges

j of "percent of true value" for individual compounds.

(6) For the analysis of pesticides in soils and house dust, a recovery check on a spiked sample using a list of specified pesticides was required.

The same recovery check was not part of the sediment procedure and

should have been.

j (7) For the analysis of total organic carbon and total organic halogen

content in water, specifications were presented for method blanks and

replicate analyses, but none for a laboratory control standard, an internal standard or a surrogate standard; hence, there was no accuracy

check.

In addition, control limits associated with many of the quality assurance procedures were to be determined during the course of the study. These

limits, if they were determined, have not been reported. Performance as determined from calibration check samples and performance evaluation samples was to be evaluated as the study progressed. This evaluation process was not described in the Quality Assurance Plan.

i The above remarks indicate the quality assurance plans were quite extensive

and detailed, but were lacking in consistency from one medium to the next

and lacking in detail on how the quality assessment was to be performed.

22
B. Were criteria established for the rejection or acceptance of data? Were

these criteria reasonable, and if not, how could they have been

improved?

Two levels of review for rejection or acceptance of data were included in

the Love Canal Monitoring Study. The Quality Assurance Plan called for

contract laboratories to monitor their own performance and to invalidate

data according to certain criteria. Laboratory performance was to be

reviewed by the prime contractor and by EPA. Criteria for acceptance or rejection by the prime contractor or by EPA of data already validated by the

contract laboratories were not part of the Quality Assurance Plan, but were

described in the draft report. Furthermore, these criteria were not

consistent with those in the Quality Assurance Plan, as will be described in

the following paragraph. In contrast to the statement on page 64 of the

draft report, the Quality Assurance Plan did not require the contract

laboratories to maintain and submit quality control charts.

The criteria used by EPA for acceptance or rejection of data differed from

the criteria specified for the contract laboratories in the Quality

Assurance Plan. For example. Table 7.2.1 of the Quality Assurance Plan

specifies control limits for the recovery of surrogate compounds in water of

81 to 100 percent for f 1 uorobenzene, 88 to 118 percent for p-bromof luorobenzene, 18 to 58 percent for 2-f luorophenol , 33 to 8l? percent for 1-

f luoronaphthalene, and 34 to 98 percent for 4

,4' -dibromooctaf 1 uor o- biphenyl. Table III-5, Page III- 28 of the draft report indicates there were

no upper control limits for the recovery of these five surrogates and that

EPA invalidated all analyses for which more than one of the surrogates was

lower than the lower control limits of 68 percent, 60 percent, 1 percent,

2.8 percent, and 8.3 percent respectively for the five compounds mentioned

above.

The EPA criteria for validation of data were less stringent, in practice,

than the criteria in the Quality Assurance Plan. Whether this indicates the

plans were unrealistic or the implementation of the Quality Assurance Plan

was inadequate cannot be determined.

C. Were the quality assurance plans of the individual contractors uniform

and consistent with each other and with the overall Quality Assurance

Plan?

The quality assurance plans of the individual contractors followed an essentially consistent outline which was appropriate to the study. The

individual plans, however, differed in detail.

Not all quality assurance plans were available or complete when NBS began

its review. In particular, to our knowledge, no plan was filed by Research

23
Triangle Institute (RTI) which was one of the three QA contractors and which

had responsibility for the spiking of the TENAX cartridges used in the QA

program for the analyses of air samples. The QA plans from TRW, one of the

laboratories performing analyses for organic compounds in water, and Wright

State University, the only laboratory performing dioxin analyses, were

received late by NBS and it is not clear whether these plans had received

approval from EPA prior to the initiation of the Love Canal Monitoring

Study. The plans received from CompuChem/Mead and Ac curex Corp. contained

notations that additional information had been requested. This information was received by NBS well after completion of the chemical analyses. It is

not clear whether these additions were included in their QA plans or were descriptions of practices which had been followed. The plans from PJB and

Advanced Environmental Systems were lacking in a nunber of elements required

by the overall QA plan of the prime contractor, but no indication was given

whether revised plans or additional information had been requested. The QA

plan for Southwest Research Institute provided no information on the

analytical procedure or the quality control procedure for the preparation of

the polyurethane foam plugs which were used in the air analyses. The plan

from Battel le Columbus Laboratories lacked a summary of analytical methods,

equipment maintenance, and trouble-shooting procedures. Finally, none of

the QA plans addressed the identification of the 20 most abundant, non- targeted compounds.

The quality assurance plans should always be signed and dated by the

submitting 1 aboratory and the approving authority before work begins. This was not done consistently in the Love Canal Monitoring Study.

D. Was the number, nature and frequency of quality control samples or performance audit samples appropriate to achieve monitoring goals, to

take corrective action, and/or to maintain quality control?

The number, nature, and frequency of quality control samples as required by

the Quality Assurance Plan should have been adequate to maintain quality

control. Evidence exists, however, that quality control was not maintained

in some cases. The Quality Assurance Plan did not anticipate the large

number of samples which would be found to contain no levels of contamination

above the limits of detection. When a large number of samples are expected

to contain concentrations below the detection limit, sample splits and

duplicate samples are of limited usefulness. These difficulties will be

discussed further in Section V dealing with implementation.

24
E. Did the plan adequately and clearly prescribe corrective action?

Instructions from EPA and the prime contractor to the contract laboratories

for corrective action (CA) were clearly and adequately described in the

overall Quality Assurance Plan with two exceptions: (1) The overall plan

did not require contract laboratories to maintain and submit quality control

charts. (2) There existed ambiguity in the description of the mechanism for

reporting corrective action to the prime contractor. Figure 11-1, Page 11-3

of the Quality Assurance Plan, is an example of a form which was to be

completed and submitted describing corrective action taken whenever loss of

quality control was identified. On Page 11-4, however, it was stated

"subcontractors are not required to follow this system exactly or to use the

pictured CA form." For such a large and complex study conducted in such a short period of time, a uniform system should have been required. On the

basis of the written, corrective-action procedures appearing in the

individual subcontractor QA plans, it is apparent that corrective-action

procedures were not consistently adequate. For example, 1 1 T stated

corrective action (unspecified) would be implemented as required but did not

state that it would report the need for corrective action to GCA. PEDCO

also gave only a vague statement of the nature of corrective-action

procedures and the documentation of corrective action.

25
IV. Performance of the Monitoring Program

Scope: To evaluate, from the information provided by EPA, whether or not

the implementation of the monitoring program met the goals and objectives of

the Organic Analysis Project and were consistent with proper implementation

of the monitoring protocols and with commonly accepted good laboratory

practices.

A. What evidence does EPA present to support claims of proper

implementation of the monitoring protocols? Is the evidence presented

that the contract laboratories were capable of performing their assigned

tasks convincing?

Based on the material reviewed, the sample collection, storage and

transportation, document control/chain of custody procedures, and data

management were generally implemented according to plan. Field and quality

assurance samples were distributed according to plan. An acceptable

fraction of the samples was analyzed. Some difficulties with the analysis

of the samples were encountered and these difficulties are discussed in the

remainder.of this Section and in Section V below.

B. Is there any reason to believe that significant concentrations of

targeted or non-targeted compounds were missed in the analyses of the

samples? ' What additional information should EPA provide to justify

claims that compounds were correctly identified and that significant

concentrations of targeted or non-targeted compounds were not missed?

The GC-MS audit of computer records indicated that most non-targeted

compounds were probably missed. Pages VI-12 to 13 of the draft report

state:

"In most of the 22 samples [of the 80 samples audited] containing non- target compounds, the audit laboratory reported finding 1 or 2 compounds

while the analytical laboratory reported none. There were 5 samples

audited where 6 to 20 compounds were reported by the audit laboratory

but none were reported by the analytical 1 aboratory. . . ."

The audit also indicated inconsistency between laboratories in the

determination of the presence of targeted compounds when the concentration

of those compounds was near the detection limit.

Pages II 1-14 to II 1-16 contain a discussion of the limits of detection and

quantitation for organic compounds in water. Table III-l, Page 1 11-15,

contains a summary of the limits of quantitation, and the accompanying

discussion states:
"There was considerable variance among the analytical

laboratories in method quantitation limits for a given analyte,

and the data suggest that some laboratories were not operating

consistently at the state-of-the-art possible with the methods.

This is neither unusual nor unexpected."

Comparison of the numbers in Table III-l with:

(a) Table II-l, Page 1 1-5 which contains the range of median values of

compounds found in drinking and surface waters in various industrial

locations,

(b) Table 1 1-4, Page 1 1-8 which contains values of volatile organic

compounds found in drinking water in a number of cities, and

(c) Table 1 1-9, Page 11-34 which contains values of selected volatile

organic compounds found in national surveys of water supplies,

indicates that the limits of quantitation for volatile organics, determined

by three of the five laboratories performing water analyses in the Love

Canal Monitoring Study, were an order of magnitude higher than values

reported by laboratories participating in the studies quoted in Appendix II

of the EPA draft report. Also, for the volatile organics, the limits of

quantitation for the five contract laboratories were 1, 2, 4, 8 and 9 times

higher than those of EPA's own laboratory in Cincinnati. For the

semivolatile organics, little data exist against which to compare the

results of this study, but the limits of quantitation for the four contract

laboratories were 1, 8, 9, and 14 times higher than those of the EPA

laboratory. In its draft report, EPA recognizes the variability in limits

of quantitation reported by the contract laboratories. This variability

will make the Love Canal monitoring data difficult to interpret.

Tlie limits of quantitation set for the air monitoring laboratories appear to

be adequate for quantifying levels of compounds which have been reported for

some U.S. cities.

C. Is there any reason to believe that significant contamination of the

samples from impure reagents, the sampling process, or contaminated

tools, or containers may have occurred? Does EPA present evidence that

this has not occurred? Is it convincing and if not, what additional

documentation is needed?

EPA has presented evidence that artifactual contamination, that is,

contamination of the samples from impure reagents, the sampling process, or from contaminated tools or containers, was a problem for certain samples and

for identified analytes.

(1) For those air samples collected on TENAX, significant blank problems

associated with benzene, toluene, and 1,1,2,2-tetrachloroethyl ene were

encountered. In Table V-4, Page V-13 of the draft report, EPA indicates

27
'• y v

•

the frequency and concentrations of these compounds found on blank TENAX

cartridges. The accompanying text states:

"To be relatively certain that an obtained single value

was not due to blank contamination, the field concentration

should be greater than three standard deviations above the mean

values reported in Table V- 4 for these three compounds."

From the data in Table V- 4, "three standard deviations above the mean values" were 7, 29, and 22 micrograms/cubic meter for benzene, toluene

and 1,1 ,2 ,2-tetrachloroethylene respectively. According to the draft

report analytical values obtained for these compounds below these levels

would be suspect. (Averages of several analyses would have somewhat

lower limits depending on the number of samples included in the

analyses.) In Section 4.2.6, Pages 167 to 195 of the draft report, EPA

describes the results of the air monitoring program. The only

significant differences between the Declaration, Canal and Control Areas are the concentrations of o-chlorotoluene, o-dichlorobenzene, and

chlorobenzene. Yet, for tFe remainder of tHe discussion, only benzene,

toluene and 1,1,2,2-tetrachloroethylene are discussed. We question the

appropri ateness of using these three compounds, for which blank problems

call into question the reliability of their determination, as "typical"

examples. The median concentration plots in Figures 54, 55 and 56 on Pages 188 to 190 were chosen to illustrate the lack of pattern in the

distribution of these compounds, but the maximum values for these

medians are below the above-mentioned limits on TENAX blanks.

(2) On Page 1 1 1-29. the contamination of bedrock B wells by use of ordinary j hydrant water during the drilling process is described. This led to the

exclusion of "28 ground-water , Method 624 samples" from the validated

data base. Chloroform was identified as the contaminant and its source was attributed by EPA to improper purging of the wells, but no quantitative data were presented to support this claim. A total of 340

bedrock well samples were included in the validated sample set. The

effect of excluding the 28 samples on the conclusions drawn by EPA is

not discussed in the draft report.

(3) On Pages 1 1 1-29 to 31, the contamination of water samples by methylene

chloride, b i s( 2-ethyl hexyl ) phthalate, and dibutyl phthalate is

described. This contamination resulted from the presence of these

chemicals in the atmosphere of the analytical laboratories and from

plastic containment materials. As a result, all reports of these three

analytes were removed from the validated data set. The same problem was encountered with the sediment and soil analyses.

With the exception of item (1) above, the incidence of artifactual

contamination should present no problem so long as conclusions drawn

concerning Love Canal are based on patterns observed for groups of chemicals

and not for individual substances. The above examples of contamination were discovered because so many of the samples which were collected had no concentrations of analytes above the detection limit. For the same reason,

28

.
it is unlikely there are other cases of widespread, but as yet undetected,

artif actual contamination by targeted compounds.

0. Is there any evidence to indicate that participating laboratories were

incapable of adequately following the protocols or did not adequately

follow the protocols? Is there any evidence of carelessness in

execution of any of the steps from sample collection to analysis?

As part of the quality assurance program, site visits were made to the

participating laboratories and reports of these visits were reviewed by NBS.

The reports indicated that most laboratories were having some difficulties

at the outset, and a few were having great difficulties. In particular,

some of the laboratories were experiencing difficulties in adjusting to the

use of capillary columns. No follow-up visits were made, or if they were,

reports of such follow-up visits were not provided. It is not known if

appropriate corrective action was taken in all cases.

The data in Table 6 on Page 110 of the draft report are inconsistent with

the number of targeted compounds which were actually sought in water,

sediment and soil samples. Table 6 contains a summary of the frequency of

detection of substances in the validated Love Canal samples. The table also

contains the "number of substances (analytes) measured" and the number of

samples which were analyzed. Dividing the number of analytes by the number

of samples gives the average number of analytes per sample. These average

numbers are, for the air analyses, consistent with the number of targeted

analytes listed in Table 1-2 on Page 1-4. For the water and sediment

analyses, these average numbers range from 20 to 30 for almost all of the

samples analyzed, whereas Table 1-1 on Pages 1-1 and 1-2 indicates that 135

analytes were to be sought. Similarly, the average number of analytes per

sample of drinking water obtained from Table 12, Page 196, is 25 to 28.

This inconsistency should be corrected.

Performance qualification of all the laboratories prior to initiation of the

study, as was done for air monitoring subcontractors, would have diminished

the number of problems encountered but could have delayed the initiation of

the study. This has been acknowledged by EPA.

E. Does EPA, in its report, adequately document the performance of the

contract laboratories? If not, what additional documentation should EPA

provide?

Appendix III of the draft report does not adequately document the

performance of the water monitoring laboratories. Four types of performance

checks were used. Laboratories analyzed laboratory control standards (LCS)

and quality control (QC) samples daily, performance evaluation (PE) samples

monthly and surrogates for every analysis. The laboratory control standards

will be discussed in Section V.B. The QC and PE samples were concentrates

of selected compounds in organic solvents which were added to reagent water

29
and analyzed. The identities of the compounds in the QC samples may or may

not have been known to the performing laboratories; the concentrations were unknown. The identities and concentrations of compounds in the PE samples

were unknown to the performing 1 aboratories. The surrogates were known

amounts of known compounds added to each sample. No sunmary of laboratory

performance on QC samples is given in the draft report and should be. The

performance on QC and PE samples was not used to determine precision and

accuracy for the methods but should have been used, at least, to confirm the

estimates of precision and accuracy.

Performance on PE samples is summarized in Table II 1-4 on Page II 1-22 of the

draft report as percentages of analyses submitted by the contract

laboratories which were judged "acceptable." The definition of "acceptable"

is not given and should be. The percentage of all analyses of PE samples

found to be unacceptable cannot be determined because the numbers of

analyses corresponding to the entries in Table 1 1 1- 4 are not given. Thus,

it is not possible to compare the performance on PE samples with the overall

performance, i.e., the percentage of Love Canal data which was validated.

The performance on PE samples should be consistent with the data validation

process and this consistency should be documented and discussed. As Table

II 1-4 now stands, it appears that the number of PE sample analyses judged as unacceptable (ranging from 0 to 75%) is considerably larger than the number

of samples invalidated on the basis of surrogate recoveries (17 samples).

This apparent inconsistency should be clarified.

i The documentation of the performance of the air monitoring laboratories

appearing in Appendix V of the draft report is generally acceptable though

Table V-5 on Page V-20 would be more readily interpretable if it contained

the ranges of the concentrations of the compounds to which the standard

deviations pertained. Inconsistencies exist in the main body of the draft j report in the description of the performance of the air monitoring

laboratories. On Page 174, is the statement:

"The analytical accuracy. . .for the TENAX samples was within ±10 percent

(as indicated by analytical recovery of substances purposely introduced

into samples at concentration levels known only to EPA) ."

On Page 184 is the statement:

"As can be seen from the results displayed in Figures 52 and 53,

variability in concentration levels were observed over time. As noted

above, however, much of the variability observed in the sample results

could be accounted for by the systematic differences in analytical

laboratories and by measurement error."

The variability in Figures 52 and 53 ranges from 0 to 40 yg/m 3

, well beyond

the ±10% "accuracy" claimed on Page 174. Moreover, there is no quantitative

justification given of the statement "...could be accounted for by the

systematic differences in analytical laboratories and by measurement error."

These statements, as they stand, appear to be in conflict in their

description of the performance of the air monitoring laboratories. i

30
The documentation of laboratory performance would be improved if the

performance of individual laboratories on all of the different classes of

quality assurance samples were sunmari zed, tabulated, and compared with

criteria established for performance in the Quality Assurance Plan.

31
V. Performance of the Quality Assurance Program

Scope: To evaluate from the information provided by EPA whether or not the

implementation of the quality assurance program was adequate to maintain

quality control of the Organic Analysis Project and whether or not accuracy

and precision claims, if given, have been convincingly justified.

A. Were the quality assurance samples sufficient in kind, nunber and

reliability to test adequately the performance of the participating

laboratories? What sources of measurement error, if any, might EPA have

missed or have had the most difficulty in evaluating?

In general, we believe the quality assurance samples were sufficient in

kind, number and reliability to test adequately the performance of the

participating 1 aboratori es . They should also have been sufficient to

maintain quality control provided appropriate corrective action was taken.

The large nunber of samples containing no analytes above the limits of

quantitation limited the usefulness of replicates for evaluating the

comparability of analyses performed in different laboratories (e.g.,

estimating i nterl aboratory precision).

* The measurement errors most difficult to evaluate are those associated with

the soil, sediment and biota samples. The quality control samples used for

the monitoring of these samples were water samples which do not duplicate

the matrix effects presented by soils, sediments and biota. Thus, not all

of the steps in the analytical methodology were tested by the quality

assurance program. This is acknowledged by EPA in its draft report.

B. Has the rejection or acceptance of data from a participating laboratory

been clearly explained and justified? Can we suggest ways in which the

rejection of data might distort the representativeness of the data?

The rejection of data has not been clearly described and adequately

justified.

m i The fractions of samples rejected for various reasons in arriving at the

final set of validated samples are sunmarized in Table 1 on Page 11 of the

draft report.

(1) In Table 1 EPA indicates 410 samples out of 2457 water samples which

were analyzed were rejected. The extensive discussion on the data

validation process for water samples on Pages II 1-25 to 29 states 17 of i* the Method 624 and 625 samples were rejected. Discussions for the

‘ 32
other analytes are vague but indicate similarly small numbers of

rejected samples. Therefore, it appears about 300 samples were

rejected without a corresponding, satisfactory explanation in the draft

report.

(2) The percentage figures in the fourth and fifth columns of Table 1 do

not agree with the totals listed in the left three columns. Moreover,

they do not agree with the numbers of rejected samples appearing in the

appendices. For example, from Table 1, a comparison of samples

analyzed with samples validated for soil, sediments and biota give 27,

4, and 8 samples rejected, presumably on the basis of the validation

procedure. Appendix IV, Page IV-23 reports these numbers as 9, 7, and

7 respectively.

(3) The footnote to Table 1 should apply to the column headed "Percent

Other" and not "% Rejected by QA/QC."

Beyond these observations, the following difficulties with the

justifications of the rejection or acceptance of data were encountered:

(1) Page II 1-24 contains a discussion of the errors which could result

from holding samples to be analyzed by Method 624 beyond the

maximum allowable holding time of 14 days. Experiments to determine

the effects of holding time were performed using samples containing 100

micrograms/liter concentrations of selected analytes. The report

should address and justify the significance of these experiments to

samples containing 10 micrograms/liter, that is, concentrations near

the limits of quantitation.

(2) The discussion on data validation procedures (Pages II 1-25 to 33) and

on estimates of data accuracy (Pages 1 11-36 to 38) for the water

monitoring effort should contain a description of the percent

recoveries for the analytes studied. The reporting of data without

correction for the lack of complete recovery of particular analytes has

been justified on the basis that the analytical methods do not provide

a procedure to correct for these losses. Recovery information is

critical for proper assessment of data resulting from the Love Canal

Study. If the mean recovery for a particular analyte were 50%, then,

on the average, all data for that particular analyte would tend to be

low by a factor of two. This could be important when comparing the

distributions of two different analytes. The only indication of such

recoveries is given in Table II 1-5 on Page II 1-28. These data were

derived by EMSC and are probably not representative of the performance

of the contract laboratories.

(3) The description of the rejection of the 28 ground-water. Method 624

samples from bedrock B wells should be clarified. How did the

Environmental Research Laboratory in Ada, Oklahoma determine these

wells were not purged adequately prior to sampling?

33
(4) As discussed in Section III.B above, criteria were established by EPA

for validation of data by the water monitoring laboratories based on recoveries of surrogate compounds. EPA used the recoveries of the same compounds but substantially wider acceptance limits for its

validation of sample data which should have been rejected (as required

by the Quality Assurance Plan) by the contract laboratories. These

wider limits were based on a statistical analysis of the recoveries

measured by EMSC in 5% of the water, soil and sediments samples. The

reasons why wider acceptance limits were used are not discussed in the

draft report. A comparison between the number of samples, for which

the data would have been validated had the original control limits been , used, and the final number of validated samples would likely give

greater insight into the quality of the water data than the tabular

data presented in the draft report. As mentioned in Section III.E

above, laboratory performance on PE and QC samples should be consistent

with the sample validation procedure.

i

Similar, though not as serious, concerns exist for the air monitoring

data.

(5) Table 1 1 1-6 on Page II 1-35 of the draft report presents relative

standard deviations for replicate measurements of Method 624 and Method

625 analytes in laboratory control standards (LCS). Table 7.2.1, Pages

7.2.2 to 7.2.4 of the Quality Assurance Plan lists control limits for

the analyses of laboratory control standards. If the analysis of the

laboratory control standard yielded values outside the control limits,

the laboratory was directed to check analytical procedures, analyze a second reference sample and, if still out of control, invalidate

results. A comparison of Table II 1-6 with the control limits for the

LCS's indicates the performance for certain of the laboratories was out of control for a significant fraction of the time. For example, the

control limit for benzene (item 6 of Table 7.2.1 on Page 7.2.2) was 79

to 107 percent recovery, a range of 15% around the average of 93 percent recovery. The relative standard deviations of the recovery of

benzene for two of the six laboratories included in Table 1 1 1-6 were well in excess of this ±15% control limit. Similar statements can be

made about other compounds listed in Table 1 1 1- 6 . EPA should, in its

report, explain this apparent discrepancy in the quality control -ll

(6) The quality assurance program for the air samples addressed the problem

of artifactual contamination of TENAX by benzene and toluene if samples

were analyzed more than 30 days after final cleaning of the TENAX. Hie

significance of such contamination could be better assessed if

experimental results were presented showing variability in background

levels of these compounds as a function of time after cleaning.

Finally, EPA has not used the full range of quality assurance data available j to it to justify its acceptance of data. The incomplete use of quality

assurance data is discussed in other sections of this Appendix. j

34
C. What conclusions has EPA drawn from the sample splits and the

intercomparison of data from single sampling sites as to the precision

of the data? Are such conclusions justified? If not, what conclusions

could be drawn and/or what justification should be presented to support

such conclusions?

EPA has drawn few conclusions concerning the precision of the data from the

sample splits and intercomparison of data from single sampling sites. This

is primarily a consequence of the small fraction of samples containing

concentrations of analytes above their detection limits.

Triplicate samples were collected by the EPA contractors in the water

monitoring program. Five percent of the samples taken consisted of such

triplicates. Seventy-five percent of the samples analyzed with Method 625

contained no analytes above the quantitation limit. This would imply that

only 25 of the triplicate samples had quantifiable concentrations of

analytes present (25% of 5% of the total number of validated water samples

listed in Table 1 on Page 11 of the draft report). In the draft report, EPA

states that similar observations were made with the Method 624 and the

Method 608 analytes. EPA concluded that this nunber was too small to give a reliable estimate of method precision. The actual nunber of triplicate

samples containing quantifiable concentrations of analytes should be

reported by EPA along with estimates of precision derived therefrom and

these estimates compared with the estimates in Table 1 1 1-6 on Page 1 1 1-35

and with additional estimates made from quality control and performance

evaluation samples.

Duplicate samples collected in the air monitoring program were used to

estimate analytical precision. The precision was based on the standard

deviation of the observed differences in reported values. This procedure

has the desirable feature of taking into account all aspects of the

monitoring process. The three compounds listed in Table V- 5. on Page V- 20 of

the draft report for which a large nunber of duplicates were obtained were

the three compounds identified in this study to be artifacts on the TENAX

columns: benzene, toluene and tetrachloroethyl ene. The standard deviations

appearing in Table V-5 for these three compounds reflect both the

variability in analysis and in the actual concentrations of the three

compounds as artifacts on TENAX. Therefore precision estimates deduced from

Table V-5 for these compounds are not transferable to other compounds.

These difficul ities in interpreting the significance of data from duplicate

samples have been pointed out by EPA on Page V - 12 of the draft report.

No data are presented on those replicates which might give insight into the

measurement process near the limit of detection, i.e., for cases where the

detected compounds in at least one of the replicates are reported at trace

concentrations. Comparison of replicates for which at least one of the

compounds was reported as below the quantitation limit should be tested for

consistency with the limits of quantitation reported by EPA.

35
D. What are some possible limitations of the quality assurance program?

The major difficulty which will be encountered in further analyses of the

data acquired at Love Canal will be coping with 1 aboratory-to-1 aboratory

variability in performance. No additional limitations in the program have

been found beyond those discussed in the answers to the other questions of

this section. In summary, EPA has not used all of the quality assurance

data it has collected in assessing the performance of the analytical

1 aboratories, i.e., tables of precision, accuracy, and detection limits are based on limited sets of quality assurance data and have not been

conf i rmed.

E. Were the abilities of the participating laboratories to detect trace

levels of targeted and non-targeted compounds adequately tested by the

qual.ity assurance program? What evidence does EPA present to justify

claims of laboratory performance? Is additional evidence necessary to

support such claims and, if so, what?

The quality assurance program did not adequately address the question of

minimum detection limits. Each of the participating laboratories was asked

to estimate its limit of quantitation (for the water program) or detection

(for the air program). These estimates have been tabulated in the draft

report. The ability of the participating laboratories to detect trace

concentrations (that is concentrations between the limit of detection and

the limit of quantitation) was not formally evaluated as part of the Love

Canal Monitoring Study. As stated in Section IV. B above, the participating

laboratories did not appear to be achieving the levels of detection with

Methods 624 and 625 which had been achieved in other studies. Greater

attention should be paid to the estimates of the limits of detection. These j estimates should be confirmed with other quality assurance data wherever

possible. The methods used to estimate and confirm the limits of detection

should be clearly described. This point is critical because all conclusions

of the draft report are bounded by the limits of detection achieved in the

study.

The ability to detect and identify non-targeted compounds was evaluated by

the subsequent GC-MS audit (see Section VII below).

F. Beyond the acceptance or rejection of data, what conclusions has EPA

drawn from the analysis of quality control samples, field blanks, and

samples from control areas as to the precision and accuracy of the

reported data, the overall performance of the participating

laboratories, or the minimum detectable limits of targeted or nontargeted compounds? Are these conclusions justified? If not, what

36
conclusions could be drawn and/or what evidence should be presented to

justify such conclusions?

EPA has used a portion of the quality assurance data to estimate precision,

accuracy, and limits of detection for the acquired Love Canal data.

Estimates of these quantities were supplied by EPA or by the contract

laboratories with little documentation as to how the estimates were made.

The conclusions of the study may be sensitive to patterns and comparisons

near the limits of detection, and therefore the determination of these

limits must be described with greater clarity and the limits confirmed.

Precision estimates were based on different types of data for air, water,

soil and sediments, and biota samples. For the air samples, the precision

estimates were based on comparisons of duplicate samples (see Section V.C

above). For soil and sediments and for water samples, the estimates of

precision were based on the analysis of laboratory control standards (LCS)

Not all targeted analytes were included in the laboratory control standards.

Table III-6 on Page III- 35 lists the relative standard deviations (RSD) for

the recovery of compounds in the LCS's. Some entries are missing,

suggesting lack of adherence to quality assurance protocols. Table III-6

would be improved if it included the number of LCS's for each compound for

which relative standard deviations were calculated. As noted above, EPA

should also use the data from quality control and performance evaluation

samples to confirm the claims of precision and accuracy appearing in the

draft report. These same comments apply to the precision estimates for the

analyses of soil and sediments.

The accuracy of air methods was estimated from extensive studies of

calibration check samples. The accuracy of water methods was estimated from

the data validation procedures (discussed elsewhere in this review) and the

laboratory reports on the analyses of LCS samples. EPA did not estimate the

accuracy of soil, sediments, and biological data because the relative matrix

effects of spiked versus natural samples were not known.

Given the imprecision of the analyses as tabulated in the draft report, it

is inappropriate to comment further on the presence or absence of bias.

Because EPA has not fully utilized its quality assurance data, it cannot

substantiate the claim on Page 67 of the draft report:

"As a result, the Love Canal data are among the most carefully

validated environmental measurements, and (given the constraints

previously mentioned) are representative of the current state-of-theart in environmental measurement methodology in terms of precision,

accuracy and specificity."

37
VI. Data Reduction and Analysis

Scope: To determine if the data reduction and analysis are consistent with

and supported by the quality of the reported data.

A. Has evidence been presented to demonstrate the accuracy of

transcription, collection and tabulation of data? If not, what

additional supporting documentation is needed?

The report of the prime contractor, GCA, presents compelling evidence that

sufficient care was taken in the transcription of the data and in the

verification of the accuracy of the transcription and transmittal of the

data into the final data base. We have no reason to believe any

significant errors were introduced into the validated data base through

errors in the transcription and transmittal of data.

B. Has EPA assigned accuracy and precision limits to the reported data?

Have these precision and accuracy claims been adequately justified by

the quality assurance program and the available literature on the

monitoring methods used? If accuracy and precision limits have not

been assigned, does the documentation provided by EPA indicate that

they could be assigned? If so, what documentation should EPA provide

to justify precision and accuracy claims?

EPA has not incorporated precision and accuracy estimates into the

val idated data base.

EPA has used data generated by the quality assurance program to estimate

precision and, in some cases, accuracy for the determination of particular

analytes in particular media. It has been noted in several sections of

this Review that EPA has not used, as fully as it could, the data generated

by its quality assurance program to confirm its reported estimates of

precision, accuracy, and limits of quantitation. EPA has not compared the

precision and accuracy estimates derived from the quality assurance program

of the Love Canal Monitoring Study to estimates derived in other studies

and published in the scientific literature.

C. Has EPA adequately interpreted the significance of "none detected" or "below detection limit"? Has EPA listed and supported detection limits

for its targeted compounds? Has it supported "below detection limit"

claims with documentation from its quality assurance program? Are

there ways in which EPA could present the organic analytical data to

38
make "below detection limit" more useful or should it continue to

report such data in such a manner?

EPA has provided, in its draft report, estimates of the limits of

quantitation and, in some cases, limits of detection for particular

analytes in particular media. Methods used to arrive at these limits are

not described and should be. Wherever possible, additional data from the

quality assurance program should be used to support the reported estimates.

Additional comments on the limits of detection and quantitation can be

found in Sections IV and V of this Appendix.

EPA has not discussed in its draft report how these limits of detection

could have affected its conclusions, a central issue since most of EPA's

conclusions are based on comparisons of "detects" with "non-detects If

the limits of detection are well below the level at which EPA would

recommend some alternative actions to those appearing in its

recommendations, then EPA should so state. If, on the other hand, the

conclusions would be altered if more sensitive or more accurate data were

available, EPA must consider this in its report. In any case, the

conclusions drawn in the EPA report must include a statement of the

concentration levels at which the conclusions are valid.

D. What conclusions has EPA drawn from intercomparison of sample splits

and sample campaigns (the collection of a number of samples from a given location over the duration of the study)? Are the conclusions

justified?

The use of sample replicates has been addressed in Section V.C above.

Because of the small fraction of samples containing levels of analytes

above the detection limit, the conclusions drawn were limited and were

used to estimate levels of precision for some of the methods and some of

the sampled media. EPA has not compared variability at a given site over a period of time with analytical variability. Such comparisons may not prove

to be i 11 uminating, but would be well worth investigating.

E. Has EPA adequately justified the rejection of data and, if not, should

some data which have been rejected be reincluded in the validated data

base?

The rejection of data has not always been clearly explained (see Section

V.B). For data whose rejection has been clearly explained, the rather wide

tolerances allowed for validation make it unlikely that acceptable data

were rejected by EPA.

F. Has EPA adequately justified the inclusion of data in the validated

data base and, if not, what kinds of analyses should EPA perform on the

39
sample data and the quality assurance data to provide such

justification?

EPA has not addressed, in its draft report, the probability of false "nondetects" in its validated data base. The GC-MS audit has given some

insight into the fraction of compounds missed or misidentified at

concentrations near the detection limit. Such considerations become

important only when the conclusions of the study are likely to be affected.

The statements and recommendations made in Section VI. C addressing the

significance of limits of detection apply here as well.

G. Has EPA incorporated precision and accuracy statements into its

validated data base? If not, does it have sufficient information to do

so? If it has, has it been done correctly?

Precision and accuracy statements have not been incorporated into the

validated data base. Some precision and accuracy statements have been

given in the appendices to the EPA draft report and those wishing to use the Love Canal data for further study must refer to these estimates when

interpreting the data.

As indicated earlier in this Review, EPA should use to the fullest extent

the available data to confirm these estimates of precision and accuracy.

H. How have recovery factors been included in the validated data file? Is it clearly stated how this was done? Was the procedure justified?

Data in the validated data base have not been 'corrected' by division by

recovery factors, which is consistent with current accepted practice.

Recovery factors have been presented for some of the targeted compounds for

the air monitoring program but not the water monitoring program. Recovery

factors should be presented for all targeted compounds whenever possible.

Recovery factors for the air monitoring program were calculated from

performance on calibration check samples. In a similar manner, recovery

factors for the water monitoring program analytes should be obtainable from

laboratory control standards, and from performance evaluation and quality

control samples. Recovery factors are invaluable when comparing

environmental data from several laboratories.

I. Should any of the reduced data tables be published as they now stand?

In discussions with EPA, following submission of the study template to EPA,

it was determined and mutually agreed upon that this question was beyond

the scope of the review requested by EPA.

40
/II. The EPA Audit of the GC-MS Computer Records

Scope: To determine if the EPA audit of the GC-MS computer records

provides usable information on the quality of the organic analysis data

and to suggest ways in which information obtained from this audit might be

used.

A. What sources of error would have and/or has such an audit uncovered?

Does the audit suggest that significant nunbers of such errors might

be present in the validated data file? Could such errors affect

substantially any conclusions which might be drawn by EPA or anyone

using the data?

The EPA audit reviewed only the interpretation of GC-MS computer records

(tapes) and was not a review of all the steps of the analyses conducted

by the contract laboratories. EPA reviewed the computer records for 5% of

the water and sediment samples. The two indices developed for the audit

give a measure of the agreement between two laboratories. They do not

distinguish between mi sidentif ications and missed identifications. For

example, if only one compound was present and was detected by both

laboratories, but was identified incorrectly by one laboratory, the same

indices would be obtained as in the situation where two compounds were

present but each laboratory correctly identified only one compound and

each identified a different compound. EPA gives no additional indication

of the nature of the differences uncovered by the audit. For example,

were differences primarily in the identification of two targeted compounds

of similar chemical structure?

The use of only three comparisons between EMSC and AERL to establish the

standard for agreement against which all other agreements will be compared

does not seem reasonable.

These concerns aside, the GC-MS audit provided valuable insight into

discrepancies in compound identification and into the effectiveness of the

identification of non-targeted compounds.

The discussion on Pages VI-11 and 12 concerning the concentration levels

at which disagreenent occurred is convincing. No evidence is presented,

however, to support the claim "the great majority of discrepancies

...[were]... in samples that were identified as heavily contaminated."

Assertions concerning whether missed identifications were primarily in

samples already identified as contaminated as opposed to samples

considered free of contamination, if false, could influence the

conclusions of the report. The seriousness of this concern would also

depend on the concentration levels at which EPA is basing its conclusions

and recommendations as discussed in Section VI. C above.

The EPA draft report cites as one of the major safeguards in the

monitoring study, the identification in each environmental sample of the

41
20 most abundant non-targeted substances. Eighty samples (water, soil and

sediments) were audited for the identification of non-targeted substances.

In 58 of these samples, both the audit 1 aboratory and the analytical

laboratory identified no non-targeted substances. In the remaining 22

samples, the audit laboratory identified 84 non-targeted substances while

the analytical laboratories found only 1. This result indicated that, for

the samples audited, most laboratories did not identify non-targeted

substances.

Finally, the results of the GC-MS audit of water, soil, and sediment data are not applicable to other media. An independent audit of air data is

necessary if similar information is sought.

B. Does the audit indicate there are major differences in the ways

the laboratories analyzed their GC-MS tapes? How would such

differences manifest themselves? Was the audit adequate to detect

such differences? Should it have been?

The audit showed that laboratories differed in the identification of

compounds near the detection limit. In its analysis of the audit, EPA

attributed these differences to differences in computer algorithms and

analysts' judgements.

The differences would manifest themselves as mi sident if ications or missed identifications by the analytical and/or audit laboratories.

The audit appears to have been adequate to detect these differences.

C. Should all of the GC-MS tapes corresponding to the validated data

file be reanalyzed using a single program? Can we specify which

program? Should GC-MS tapes corresponding to rejected data also

be reanalyzed? Could faults in the analysis of the GC-MS tapes

have led to incorrect rejection of data from or inclusion of data

into the validated data file?

Variability introduced by use of different programs is likely to be less

important than the variability introduced by the different analysts who

reviewed the data.

Further audit of the GC-MS records, including review by a team of analysts

using identical criteria, would appear to be necessary, so long

as conclusions drawn by EPA in its Love Canal Monitoring Study are sensitive to the presence of undetected contaminants at concentrations

near the detection limit.

42
Faults in the analysis of the GC-MS tapes are not likely to have led to

incorrect rejection of data from or inclusion of data into the validated

data file because the criteria for rejection were basically unrelated to

possible problems with compound identifications.

43
APPENDIX B

DOCUMENTS RECEIVED BY N8S

Item Doc unen t Title Date Received

at NBS

Bound and Titled

1 . Quality Assurance Plan Love Canal Study 18 Aug. 81

2. Quality Assurance Plan Love Canal Study

Appendix A (Sampling Procedures) 18 Aug. 81

3. Quality Assurance Plan Love Canal Study

Appendix B (Analytical Procedures) 18 Aug. 81

4. •Quality Assurance Plan Love Canal Study

Appendix Q (Subcontractors' QA Plans) 18 Aug. 81

5. Love Canal Monitoring Progran

GCA QA/QC Summary Report 28 Jan. 82

Unbound and Titled •

6. Environmental Monitoring at Love Canal, 1980

(Designated by NBS as OEM-LC-2) 18 Aug. 81

7. Data and Control Charts by Level

[Air] — 'Four Documents' 21 Oct. 81

8. Audit of Gas Chromatography/Mass Spectrometry

Data (GC/MS) Provided by Love Canal Project

Analytical Laboratories 21 Oct. 81

9. GCA Report Volume I, Proposed Work Plan Love

Canal Monitoring Project 21 Oct. 81

10. Additional EPA Love Canal Materials for NBS Review

(Correspondence) 21 Oct. 81

11. Supplemental Documentation of EPA Love Canal Quality

Assurance Program (Correspondence) 19 Nov. 81 1

12. Revised Version of Audit of

Gas Chromatography/Mass Spectrometry Data

Provided by Love Canal Project Analytical

Laboratories (Item 8 above)

I

19 Nov. 81

i

1

1

44
13. U.S. EPA Environmental Monitoring at Love

Canal (Draft Report, Designated Copy #9) 17 Dec. 81

14. Appendix I List of Substances Monitored at Love Canal 17 Dec. 81

15. Appendix II

Comparative Data and Existing Standards for

Substances Monitored at Love Canal 17 Dec. 81

16. Revised Sections 3.3, 3.3.1, and 3.3.2 for

EPA Draft Report 28 Jan. 82

17. Quality Assurance for Water Samples

Revised Appendix III for Draft Report 28 Jan. 82

18. Quality Assurance for Air Samples

Revised Appendix V for Draft Report 28 Jan. 82

19. Report on the Audit of Gas Chromatography/Mass

Spectrometry Data Provided by Love Canal

Project Analytical Laboratories

New Appendix VI for Draft Report 28 Jan. 82

20. Quality Assurance Plan, Love Canal Study,

Appendix Q, Revision 1 28 Jan. 82

21. Quality Assurance for Soil, Sediment,

and Bi ota Samples

Revised Appendix IV for Draft Report 17 Feb. 82

Unbound and Untitled

22. Laboratory Audits - Trip Reports

Love Canal Study

(Designated by NBS as OEM-LC-1) 18 Aug. 81

23. Love Canal Study Area: Sampling Sites

(Designated by NBS as OEM-LC-3) 18 Aug. 81

24. Aggregate of Data from Various Sampling Areas

(Designated by NBS as OEM-LC-4) 18 Aug. 81

25. Aggregate of Data from Various Sampling Areas

(Designated by NBS as OEM-LC-5) 18 Aug. 81

26. Aggregate of Data from Control, Canal, and

Declaration Areas

(Designated by NBS as OEM-LC-6) 18 Aug. 81

45
27. Aggregate of Data from Control, Canal, and

Declaration Areas

(Designated by NBS as 0EM-LC-7) 18 Aug.

28. Graphical Displays of Air Monitoring Data as

a Function of Sampling Date 6 Oct.

29. Quality Assurance Control Charts & Tables

for Water Samples 6 Oct.

30. Quality Assurance Data (Tables) for Love Canal

Soil, Sediment, and Biota 21 Oct.

31. Soil Holding Times (Internal EPA Correspondence) 21 Oct.

32. Holding Time Study for Purgeable Compounds

in Water (Internal EPA Correspondence) 21 Oct.

33. Report on Audit of PJB Laboratories (Internal

GCA Correspondence) 21 Oct.

34. Report on Audit of Battel! e and PEDCO

Laboratories (Internal GCA Correspondence) 21 Oct.

35. Collection of Public Comments Received by

EPA Regarding Methods 624 and 625 and EPA

Response to those Comments 28 Jan.

81

81

81

81

81

81

81

81

82

!« yI

IH 46
8S-VNA IREV. 2-eOI U.S. OEPT. OF COMM. 1. PUBLICATION OR 2. Performing Organ. Report No. 3. Publication Date

BIBLIOGRAPHIC DATA

REPORT NO.

May 1982 SHEET (See instructions) NBSIR 82-2511

l. TITLE AND SUBTITLE

5. AUTHOR(S)

PERFORMING ORGANIZATION (If joint or other than N BS, see instructions) 7. Contract/ Grant No.

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, O.C. 20234

t. Typo of Report & Period Covered

}. SPONSORING ORGANIZATION NAME AND COMPLETE AOORESS (Street. City. State, ZIP)

U.S. Environmental Protection Agency

401 M Street, SW

Washington, D.C. 20460

L0. SUPPLEMENTARY NOTES

Document describes a computer program; SF-185, FIPS Software Summary, is attached.

11. ABSTRACT (A 200-word or less factual summary of most si gnificant information. If document includes a significant

Oibhograohy or literature survey, mention it here)

ABSTRACT

This report is a review conducted by the National Bureau of Standards

at the request of the U.S. Environmental Protection Agency (EPA) of

the analysis for organic chemicals conducted by EPA in the Love

Canal Area of Niagara Falls, New York.

12. KEY WORDS (Six to twelve entries; alphabetical order; capitalize only proper names; and separate key words by semicolon s) Love Canal, Monitoring, Organic Analysis, Environment, Pollution

13. AVAILABILITY

' j Unlimited

fXl P°r Official Distribution. Do Not Release to NTIS

1 Order F'om Superintendent of Documents, U.S. Government Printing Office, Washington, D.C.

20402.

Order From National Technical Information Service (NTIS), Springfield, VA. 22161

14. NO. OF

PRINTED PAGES

15. Price

• i « «n < i.o«n