United States Office of W ater DRAFT Reviews & Responses Environmental Protection Agency (4303) April 2010 

# Peer Review of the Method 1668A InterlaboratoryValidation Study

April 2010United States Office o f W ater Environmental Protection Agency (4303) April 2010

ii 

# U.S. Environmental Protection Agency Office of Water (4303T) 1200 Pennsylvania Avenue, NW Washington, DC 20460

EPA 820
- R
- 10
- 007Method 1668A Interlab Study Peer Review

April 2010 iii Table of Contents DISCLA IMER ................................ ................................ ................................ ............................ IV
INTRODUCTION ................................ ................................ ................................ ....................... 1
EPA’S CHA RGE TO THE FIRST TWO PE ER REVIEWERS ................................ ..................... 2
EPA’S CHA RGE TO THE THIRD PEER REVIEWER ................................ ................................ 3
PEER REVIEW ................................ ................................ ................................ .......................... 5
ISSUES RA ISED BY BOT H REVIEWERS A ND EPA’ S RESPONSES ................................ ..... 5
PEER REVIEW COMMENTS FROM M ICHA EL C. NEW MA N ................................ ................. 7
PEER REVIEW COM MENTS FROM CHA RLES S , WONG ................................ ..................... 17
PEER REVIEW COMMENTS FROM KURUNTHACHA LA M KA NNA N ................................ .... 21
ORIGINA L PEER REVIEW COMM ENTS ................................ ................................ ................. 35Method 1668A Interlab Study Peer Review

April 2010 iv Disclaimer Mention of trade names or commercial products does not constitute endorsement or recommendation for use. Contacts The first two peer reviews in this document were conducted under EPA Contract # 68

- C
- 02
- 091 by Versar, Inc., 6850 Versar Center, Springfield, Virginia 22151. The third and final peer review in this document was conducted under Contract No. EP
- C
- 07
- 059 by Eastern Research Group, Inc. 110 Hartwell Avenue. Lexington, MA 02421. Please direct your questions c oncerning this document to: Cl ean Water Act methods Group , Brian Englert or Richard Reding Engineering & Analytical Support Branch Engineering and Analysis Division (4303T) Office of Science and Technology, Office of Water U.S. Environmental Protection Agency 1200 Pennsylvania Avenue NW Washington, DC 20460 OSTCWAMethods@epa.gov , englert.brian@epa.gov or reding.richard@epa.govMethod 1668A Interlab Study Peer Review

April 2010 1 Introduction This document contains the comments and EPA’s responses to comments of a peer review of the Method 1668A Interlaboratory Validation Study Report. In 2007, a contractor obtained three independent peer reviewers and conducted this review for EP A using EPA’s Science Policy Council Peer Review Handbook (EPA/100/B

- 06/002, 3 rd Edition, the “Peer Review Handbook”). For the review, the contractor sent the Study Report to each reviewer with a “charge

. ” Two reviewers provided written comments and sugge stions for improvement of the Study Report on the Method 1668A Interlaboratory Validation Study. A fter a third reviewer dropped out for personal reasons , EPA solicited a third revi ewer in April of 2010. In our responses to peer reviewer comments and sugg estions , we either accept a suggestion or provide an explanation why the suggestion was not accepted. Based on the suggestions and on further data gathered, EPA has revised Method 1668A and the “Method 1668A Interlaboratory Validation Study Report” ( March 2010 , EPA
- 820
- R
- 10
- 004 .) This Study Report and Method 1668C are posted on the CWA methods website at 

### http://www.epa.gov/waterscience/methods/method/other.html

.Method 1668A Interlab Study Peer Review

April 2010 2 EPA’s Charge to the first two Peer Reviewers EPA developed the following charge for the first two peer reviewer s : You are requested to peer review “Report of the Method 1668A Interlaboratory Validation Study”. Please prepare a summary of comments based on the EPA speci fied Charge. The Comments Summary should include a cover page with the title of the document, your name and affiliation and comments organized by the format described in the cover letter. Note: You may use this draft to become familiar with Method 1668 A Interlaboratory Validation Study, but please do not distribute it. We shall provide you with a revised copy after changes resulting from comments have been incorporated. EPA Method 1668A was developed in the late 1990s and a draft version was peer reviewed in 1999. Results of that peer review were published in February, 2000, in Peer Review of Draft EPA Method 1668, Revision A: Chlorinated Biphenyl Congeners in Water, Soil, Sediment, and Tissue by HRGC/HRMS. Draft EPA Method 1668A was revised as a res ult of the peer review and the revised version, dated December 1999 (EPA

- 821
- R
- 00
- 002) was published in January of 2000. After publication of the December 1999 draft of EPA Method 1668A, EPA distributed the method extensively and planned inter

- laboratory validation. In preparation for the inter

- laboratory validation, EPA updated, revised, and released Method 1668A on August 20, 2003. A copy of that revision is included with this letter. The interlaboratory validation study was conducted in 2003

- 2004. Results of the study were analyzed and the enclosed Report of the Method 1668A Interlaboratory Validation Study was drafted in 2006. The major result of the interlaboratory study is a revision to the quality control acceptance criteria (QCAC) in EPA Meth od 1668A based on interlaboratory study results. These QCAC are performance specifications within which Method 1668A must be operated and performance yardsticks against which a method modification must measured. EPA received comments as a result of the interlaboratory study and other comments between release of the August 20, 2003, revision and the present. EPA has included with this letter a list of changes that EPA plans to make to EPA Method 1668A based on the revised QCAC and comments received. Thes e changes are listed in the document titled: Changes to be made to EPA Method 1668A from the 8/20/03 revision based on the Method 1668A interlaboratory validation study and comments and corrections received from laboratories, included with this letter. B ecause of changes to Method 1668A, primarily as a result of the revised QCAC, and based on your peer review, EPA plans to revise EPA Method 1668 to Revision B (EPA Method 1668B). In your review, please respond to the following questions: 1. Please review EPA's Report of the Method 1668A Interlaboratory Validation Study for the objectives of the study, the soundness of the technical approach, and the results produced. 2. Please render an opinion as to suitability for use of the data from the study for revision of the QCAC in Method 1668 to make it Method 1668B. 3. If you have a suggestion for an improvement of the interlaboratory study report, please provide suggested text by section number. 4. If you believe that a given item should be clarified, p lease provide the clarification by section number or define the clarification needed to a sufficient level of detail that we can make the necessary changes. 5. Please point out errors or omissions.Method 1668A Interlab Study Peer Review

April 2010 3 EPA’s Charge to the third Peer Reviewer EPA developed the following charge for the third and final peer reviewer: BACKGROUND EPA Method 1668A was developed in the late 1990s and a draft version was peer reviewed in 1999. Results of that peer review were published in February, 2000, in “Peer Review of Draft EPA Method 1668, Revision A: Chlorinated Biphenyl Congeners in Water, Soil, Sediment, and Tissue by HRGC/HRMS.” A copy of the peer

-review report is included. Draft EPA Method 1668A was revised as a result of the peer review and the revised version, date d December 1999 (EPA

- 821
- R
- 00
- 002) and was published in January of 2000. After publication of the December 1999 draft of EPA Method 1668A, EPA distributed the method extensively and planned an inter

- laboratory validation. In preparation for the inter

- lab oratory validation, EPA updated, revised, and released Method 1668A on August 20, 2003. The interlaboratory validation study was conducted in 2003

- 2004 to validate the performance of EPA Method 1668A in municipal wastewater, fish tissue, and biosolids matrices. EPA received comments as a result of the interlaboratory study and other comments after the release of the August 20, 2003 revision of 1668A. Results of the study were analyzed and the “Report of the Method 1668A Interlaboratory Validation Study was drafted in 2006 and peer reviewed in 2008.” The major result of the interlaboratory study is a revision to the quality control acceptance criteria (QAQC) in EPA Method 1668A based on interlaboratory study results. EPA used the results of the study to evaluate and revise Method 1668A quality control (QC) acceptance criteria for initial precision and recovery, ongoing precision and recovery, and labeled compound recovery from real world samples. EPA subsequently revised 1668A to 1668B based on the Int erlaboratory study, and comments received on 1668A. EPA published 1668B with a list of changes in November 2008. These interlab criteria (Table 5

- 1 of the study report) replace the single

- lab criteria, and are published in Table 6 of the revised working draft version of PCB

- congener method, EPA Method 1668B (Method VIII).

“ Method 1668A Interlaboratory Validation Study Report,” presents the results of EPA’s interlaboratory validation of EPA Method 1668A Chlorinated Biphenyl Congeners in Water, Soil, Sediment, and Tissue by HRGC/HRMS. Objectives of this study were to 1) characterize the performance of Method 1668A in multiple laboratories and matrices, and 2) evaluate and, if appropriate, revise the QC acceptance criteria in the method. EPA designed the Method 1668A Interlaboratory Validation Study in accordance with guidelines published by EPA and ASTM International (ASTM). These guidelines recommend a minimum of six complete data sets for evaluation of a method. To allow for some loss of data due to error, lost samples, outlier removal, or other unforeseen causes, EPA included 14 participant laboratories in the study. The study design and report are detailed in the document, “Method 1668A Interlaboratory Validation Study Report. ” The QC acceptance criteria developed in the interlaboratory method validation study did not allow the upper recovery limit for some congeners to be above 100 percent. The QC acceptance criteria were recently revised based on data from the interlaboratory study and data from two laboratories with extensive experience in use of Method 1668A. Data were provided by TestAmerica, Knoxville, Tennessee and AXYS Analytical Services, Ltd., Sidney, BC, Canada and are included in the attached excel file. Method 1668B was subsequently revised. A draft version of that as yet unpublished revision toMethod 1668A Interlab Study Peer Review

April 2010 4 1668B is attached (See Method VIII). The revisions to that method are summarized in the introduction of the method. The addendum describing the revised QC acceptance criteria, “ Addendum to the Method 1668A Interlaboratory Validation Study Report Draft, August 2009” is included in this review. Data supporting the addendum can be found in the file, “OPRresults_Labs1&2.” EPA peer reviewed the “Method 1668A Interlaboratory Validation Study Report” in late 2007 but was unable to obtain three peer reviews and is attempting to obtain a third peer review now. CHARGE QUEST IONS Please refer to the Review Guidelines and Formatting Instructions found on pages 4 and 5 of this Charge. Please review the att ached documents “Method 1668A Interlaboratory Validation Study Report” and
“ Addendum to the Method 1668A Interlaboratory Validation Study Report Draft, August 2009 .” In your review, please respond to the following questions: 1. Please review EPA's Report of the Method 1668A Interlaboratory Validation Study and its addendum . Comment on the objectives of the study, the soundness of the technical approach, and the results produced. 2. Please render an opinion as to suitability for use of the data from the study and the addendum for revision of the QCAC in Method 1668 to make it Method 1668B. 3. If you have a suggestion for an improvement of the interlaboratory study report or addendum, please provide suggested text by section number. 4. If you believe that a given item should be clarified, please provide the clarification by section number or define the clarification needed to a sufficient level of detail that we can make the necessary changes. 5. Please point out errors or omissions. 6. Please pro vide any additional comments you would like to make regarding the documents under review.Method 1668A Interlab Study Peer Review

April 2010 5 Peer Review Dr. Michael Newman, Dr. Charles Wong and Dr. Kurunthachalam Kannan peer -review ed the Study Report
. This document has three parts. The first section is EPA’s responses to some common issues form more than one reviewer

. The s econd section is EPA’s responses to specific comments from each reviewer. The last section contains the complete review of each reviewer

. Section One: Selected Issues raised by more than one review er and EPA’s responses Comment: Itwould have been useful if the laboratories were asked to provide measurement data on

QC Check samples (Method 1668A, Section 7.15) from standardized reference materials (e.g., NIST SRMs or Canadian NRC S RMs). Response: EPA asked laboratories to analyze spiked and typical Clean Water Act matrices, i.e.,

wastewater, tissue, and biosolids (sewage sludge.) These samples better represented the intended application of this method for environmental monitoring. In addition, laboratories used to using the available QC check samples prepared in reagent waters or solvents might be overly familiar with their content. Method 1668A has been available for use by laboratories since 2000. Because it contains the sugge stion for a QC Check sample in Section 7.15, many labs had been using a NIST SRM or NRC CRM, and might recognize the sample from having analyzed it previously. Comment: Laboratory participation was disappointing and this was the source of numerous issues.

For example, six of 14 laboratories declined to participate in the biosolids analysis. Response: The high failure

-to -report rate is attributable to (1) the extensive effort needed to set up

the extraction, concentration, cleanup, and HRGC/HRMS systems for determination of 209 PCB congeners, (2) a lack of familiarity with the biosolids matrix, and (3) the voluntary nature of participation

. Comment: Outlier deletion, as discussed in Grubbs' paper and many others, should be done very

thoughtfully acco rding to clear rules. For example, the "outliers" detected based on the assumption of a normal distribution might come from a population more appropriately described as lognormal. Response: Outlier tests were applied to both blind sample data (as presen ted in Table 4

- 1 of the report) and QC data. In both cases, Grubbs' test was applied conservatively, following guidelines for maximum outlier removal as specified by ASTM D2777

- 98. Comment: Round all numbers to 3 significant figures. Response: We rounded all results to three significant figures maximum in the final report

. Comment: On page 4, paragraph 2, please consider replacing "properly homogenized" with

"homogenized." Response: We have removed "properly." Comment: Itis extremely important in judging the generalizability of the report conclusions that the

laboratory participation in generating the final QC criteria is very clearly discussed. At present, it is not easy to follow. Response: Where possible, we revised the report to clarify this discussion. EPA discusse s e xclusion

of a laboratory's data is in Section 3 of the report. EPA generally rejected a laboratory because the laboratory submitted incomplete data s ets , insufficient raw data to verify a laboratory's results, or did not fo llow the method correctly.Method 1668A Interlab Study Peer Review

April 2010 6 Comments: Move the definition of "Youden pair" from Section 2.5.1 to its first use in Section 2.4. Response: We move d the definition to its first use.Method 1668A Interlab Study Peer Review

April 2010 7 Section Two : Comments by each reviewer and EPA’s responses 

## P EER RE VIEW COMMENTS FRO M MICHA EL C. NEWMA N NA ME : Michael C. Newman , Ph.D. AF FILIA TION : 121 York Point Dr., Seaford, VA, also Professor at the College of William and Mary, Virginia Institute of Marine Science DA TE : September 18, 2007 COMMENTS ON : Method 166 8A Interlaboratory Validation Study I. GENERA L IMPRESSIONS Comment 1: Overall, this Report of the Method 1668A Interlaboratory Validation Study is a

sufficiently accurate, clear, and thorough analysis discussing the results of a very difficult task. The study objectives were to (1) characterize the performance of Method 1668A for multiple laboratories and matrices (wastewater, biosolid, fish tissue), and (2) evaluate and possibly revise the QC acceptance criteria. These objectives were met with some qualification. It is important to emphasize prior to discussing the specific qualifications that this effort does appear to be generally well

- intended,

-planned and

- interpreted. Response: No response needed

. Comment 2: Some challenges to applying th e results do exist. The laboratory participation was

disappointing and this was the source of numerous issues. For example, six of 14 laboratories declined to participate in the biosolids analysis. An additional four produced unusable numbers, resulting in only four of 14 potential sets of results being applicable to the task at hand. For wastewater and fish tissue, only six laboratories produced the data used in this exercise. Four laboratories successfully analyzed all three sample types and, therefore, have a very strong influence on the results. The candidate laboratories included government and private organizations which might be expected to have differences in response; however, it is unclear from the report if the final subset was biased toward one group or another. The very high failure

-to -report rate for this process suggests that failure to achieve useful analyses with 1668A warrants more discussion. Was the high failure rate normal for such voluntary validation exercises or a feature to be anticipated for Method 1668A as currently written? Response: The high failure

-to -report rate is attributable to (1) the extensive effort needed to set up

the extraction, concentration, cleanup, and HRGC/HRMS systems for determination of 209 PCB congeners, (2) a lack of familiarity with the biosolids matrix, and (3) the voluntary nature of participation. When participating voluntarily, a laboratory has no financial incentive to complete the work and submit a complete set of results. Thus, when the laborato ry encounters difficulty, as with the biosolids matrix, the laboratory may simply stop the analysis rather than time

- consuming corrective actions

. Comment 3: Some laboratories deviated from 1668A for some analyses. Also, outlier tests that require

extremely thoughtful application were applied to cull away some data. Together, these features do not permit as high confidence in the study ’ s generalizations as might be desired. Response: Within limits, laboratories were allowed to deviate from prescrib ed procedures under the

performance

-based provisions of Method 1668A. EPA applied the same outlier removal procedures that we have applied to EPA wastewater methods for the past approximatelyMethod 1668A Interlab Study Peer Review

April 2010 8

20 years
. These procedures are based on ASTM D2777

- 98 1 , which ensures that outlier removal be limited to a small percentage of the data, and therefore the removal would have a small effect on the study results. These outlier procedures remove true outliers yet retain valid data. Additionally, some of the QC sample results removed as outliers were
“better” (i.e., closer to 100% recovery) than the remaining results, which means that their removal would not portray method performance as unrealistically good. Comment 4: The studies conclusions are not unequivocal at this point. The resulting, central concern

can be articulated with a quote from Stuart ’s Basic Ideas of Scientific Sampling (1968):
“ the credentials of a sample are not only important for the interpretation of results, but are in principle the only informati on of value we can have in this respect .” Response: Since EPA completed this study and peer review, EPA has obtained additional data that

demonstrate that laboratories can overcome blank contamination, and otherwise perform Method 1668 under tighter speci fications than we observed in this study

. Comment 5: My general conclusion is that the results are adequate for meeting the study objectives but

should be augmented in the near future in order to enhance confidence in the general validity of the conclus ions. Detailed below are specific examples from within the report and associated recommendations. Response: EPA has since obtained additional data that demonstrate that laboratories can perform

Method 1668 under tighter specifications than we observed in this study . Subsequent to this study, EPA has also obtained MDL data from several contract laboratories

. In a study of data comparability between two laboratories on samples collected from the Passaic River in New Jersey, in which 151 PCB congeners were identified and measured, accuracy as measured by analysis of a NIST SRM was 15% or better. Recoveries of the PCB congeners ranged from 90% to 124% and averaged 105%; precision ranged from 4.2 to 23%. II. RESPONSE TO CHA RGE QUESTIONS 1. Please review the EPA's Report of the Method 1668A Interlaboratory Validation Study for the objectives of the study, the soundness of the technical approach, and the results produced. Comment 6: The Interlaboratory Validation Study attempts to characterize the performance of Method

1668A in multiple laboratories and matrices (wastewater, biosolid, fish tissue), and to evaluate and possibly revise the QC acceptance criteria. As stated above, my overarching opinion is that the study met these goals but further work is required to enhance confidence in the generality of the conclusions. The goals for this complicated method were extremely challenging so the implied incremental generation of sufficient characterization/validation information is, in my opinion, reasonable. My suggestion that more work is required is not an oblique inference about the work quality of the study leaders. This was a very difficult task. Response: We appreciate your understanding of the difficulty of conducting the study and the

analyses. 2. Please render an opinion as to suitability for use of the data from the study for revision of the QCAC in Method 1668 to make it Method 1668B. 1 American Society for Testing and Materials. 1998. Annual Book of ASTM Standards, Vol. 11.01. Standard Practice for Determination of Precision and Bias o f Applicable Test Methods of Committee D

- 19 on Water, ASTM D2777
- 98, October 1998.Method 1668A Interlab Study Peer Review

April 2010 9 Comment 7: Yes. The study data are adequate, although lacking in some ways, for revision of the QC

in Me thod 1668. Response: EPA has since obtained additional data and revised the QC data in the 2009 version of

Method 1668. 3. If you have a suggestion for an improvement, please provide suggested text by section number. Comment 8: A wide range of sugg estions and insights will be addressed below regarding the

improvement of the study or existing study presentation. Response: Our respon ses to this group of comments follo w. Comment 9: Page 6, Top paragraph and first full paragraph

- This is the f irst place in the report that the issue of sample homogeneity emerged. It is also discussed in Section 4.1. Described here is the process in which a “ sample processing laboratory was required to perform

background and homogeneity analyses of both the bio solids and tissue matrices. ” The text does not convince me that the samples were “homogeneous ” in a strict sense relative to the analytes. For example, some fatty tissues might contain more analyte than others and similar

- sized pieces of fatty tissues mi ght not be homogeneously distributed in a well

- mixed homogenate. The biosolid to which sand was added might experience density

driven sample component segregation (i.e., sinking sand) so that the sample aliquot analyzed by the volunteer laboratory might n ot have been homogeneous (well

- mixed?) relative to the analyte. There are very specific methods of determining this sample quality (well

- or poorly

- mixed) yet they are not applied by the sample processing laboratory. It is not clear how this mi ght or might not have contributed to differences in results among laboratories. This concern is also inherently linked to sample aliquot size which 1668 prescribes versus and the actual sample sizes used by the volunteer laboratories. (I assume that the recommendations of sample sizes were based only on anticipated analyte concentrations in typical samples and the mechanics of the sample processing.) The point being made above can be illustrated easily using the conventional explanation as depicted in the figure to the left. The analyte (dot) is homogeneously distributed in this figure = s top panel; therefore, sample aliquot size illustrated in the figure as different size boxes will not influence the measured analyte concentration in an analyzed aliquot. (Homog eneity is a material quality in which the analyte is distributed in a completely homogeneous manner throughout the material.) In the bottom panel, a heterogeneous material is depicted in which sample aliquot size becomes critical depending on whether the heterogeneous material is well

- or poorly

- mixed. If sample aliquot sizes are too small for a poorly

- mixed, heterogeneous material (small squares in figure bottom panel), variation among reported analyte concentrations will be higher than that derived wit h aliquot sizes of sufficient size (large square in the bottom panel) (Newman 1995). To conclude that a heterogeneous sample is sufficiently well

- mixed to permit use of a specified aliquot size (e.g., 20 g of fish tissue) requires some conventional calcul ations. The homogeneity analyses described in this report do not include the associated statistics permitting such decisions. This is especially relevant to the biosolid and tissue samples, and with sufficiently high solids content, also to wastewater samples. The pertinent approach to this problem and associated statistical computations are described in classicMethod 1668A Interlab Study Peer Review

April 2010 10

papers by Visman (1969), Ingamells (1973) Ingamells and Switzer (1974), and Wallace and Kratochil (1987). Response: We are confident that the samples were homogeneous

. although using the standard that

you have developed, those that you reference, and existing data, it is impossible to prove. The sample

-processing laboratory has extensive experience in sample homogenization, and homogenizati on procedures are provided in Section 11.7 of Method 1668A. Comment 10: Page 8, Top Bullets

- The laboratories were told to report all “ >nm” values as quantified

concentrations and the statistical analysis of these measured values was treated such that all “ >nm” values possessed the same quality of measurement. It is incorrect to assume that all values above “nm” had the same measurement quality (e.g., Keith et al. (1983)). The relative uncertainty of a measured value can be estimated relative to the standard deviation of the analytical process baseline noise (See also figure on page 6),



### N Z y Uncerta lative 2 100 (%) int Re  where Z= Z

- statistic value for a confidence level of  (conventionally 95%), and N is the measured signal expressed in units of multiples of the standard deviation of the baseline noise, e.g., N=3 and 10 are associated with the signal at the detection limit and limit of quantification, respectively. Interpretation of the study results is very difficult if this fact is not addressed in the discussions and shown in graphs of results for the various samples and sample types. An example of this will be discussed below relative to the figures in the report. Response: An analysis of this type would require knowledge of the error function for the analytical

system in each laboratory in the interlab study. D ifferences in sample amounts and volumes, and final extract volumes, used by the laboratories, as these amounts and volumes would shift the results upward or downward on the concentration

- error fu nction would be further complicate the suggested analysis . Therefore, this type of analysis is impractical and beyond the scope of the study. Comment 11: Page 12, First paragraph after bullet list and also last sentence

- Here and elsewhere, my

assump tion is that the study results are intended to provide measurement quality characteristics anticipated for “ in

- control ” analyses of the specified sample types by a laboratory represented by the initial 14 laboratories. It is critical that the population for which these final QC criteria are generated be clearly defined given the variations in reporting, method application, and data selected for final QC calculations. Response: EPA intends that any single laboratory using Method 1668, not a hypothetical population

derived from the laboratories in the study, will use the final QC acceptance criteria

. Our experience is that QC acceptance criteria generated from inter

- laboratory data are approximately a factor of 2 broader than the same criteria generated from data from a single laboratory. Note that EPA derived the QC acceptance criteria for the recovery of labeled compound s from results from all three matrices in the study . And EPA derived the QC acceptance criteria for IPR and OPR sample recovery from re sults of spiked water and solid matrices. Therefore, although the reduced data set may narrow the criteria somewhat, the QC acceptance criteria are sufficiently generous that future l aboratory will not have difficulty meeting them. Comment 12: Here and elsewhere, Grubbs ’ single outlier test (Grubbs 1969) is applied to cull away

apparent outliers. Outlier deletion, as discussed in his paper and many others, should beMethod 1668A Interlab Study Peer Review

April 2010 11

done very thoughtfully according to clear rules. For example, the “ outliers ” detected bas ed on the assumption of a normal distribution might come from a population more appropriately described as lognormal. The “ outlier” would be falsely removed based on the assumption of the wrong distribution. The rules for handling outliers in Grubbs (196 9) and other papers should be considered prior to applying any outlier tests. The results of such application should then be described in the report. Classic papers that mi ght be useful are Anscombe (1960) and Rosner (1983). As multiple outliers might b e masking each other in these data sets, Rosner ’ s method (1983) for multiple outliers might be useful. Personally, I think that Woodworth ’ s (2004) classification of outliers and the associated approach based on inter

- quartile ranges is more relevant as it is more open to intelligent application than the above hypothesis

-based, parametric tests. This point will be repeated later in discussing the absence of quartile (Q1 and Q3) values in tables of summary statistics. Response: Outlier tests were applied to both blind sample data (as presented in Table 4

- 1 of the

report) and QC data. In both cases, EPA applied the Grubbs test conservatively, following guidelines for maximum outlier removal as specified by ASTM D2777

- 98. In the case of sample data, these results were only used to calculate summary statistics of the sample results by chlorination level, and therefore outlier removal would not strongly affect conclusions about method performance. For QC data, outlier removal was limited to one result per cong ener. For labeled compound samples, there were a greater number of results, and a maximum of two results were removed per congener. As stated elsewhere, EPA had no strong evidence that these data did not follow a normal distribution, and therefore EPA’s us e of the Grubbs test is appropriate. We are familiar with outlier tests based on the interquartile range, but these tests tend to classify an unrealistically large percentage of data as outliers. Comment 13: Page 13, Section 4.2.1, line 4

- The “homogen eity” was found acceptable

. Respectfully,

my guess would be that this statement is correct; however, I disagree that enough information was extracted to make a definitive statement. I base this disagreement on the material already presented above. Respons e: We agree that the samples could have been homogeneous but we lack sufficient data to

prove it. At this point, we must assume homogeneity because further testing is not possible. Comment 14: Page 14, Table 4

- 1
- Here and elsewhere, it would be very h elpful to have the first and

third quartile values (Q1 and Q3) or, minimally, the inter

- quartile range (|Q3

- Q1|) in the table. One immediate use would be to apply inter

- quartile range

-based methods to explore potential outliers or outlying laboratories. The inclusion of these statistics is highly recommended. Response: Application of these ranges would alter the results or conclusions significantly or

appreciably alter the QC acceptance criteria to justify the effort to do so

. Comment 15: As an aside, the differences between means and medians for congeners with 3 to 8

chlorination levels (biosolids and tissues) suggest a skewed sample set. Are these data skewed for any reason or is this simply a spurious trend in the tabulated data? Response: We agree that the distribution of results presented in Table 4

- 1 were often skewed. This

was most likely due to the difference in prevalence and coelution of the various congeners within each chlorination level. This skewness was not observed with a single congener in the QC sample data, and therefore this skewness would not reflect a departure from the normality assumption used in the developing QC acceptance criteria.Method 1668A Interlab Study Peer Review

April 2010 12 Comment 16: Page 29, Last line of text

- I generally agree but feel that a less definitive , and more

conditional, conclusion can be reached. Response: Because you did not provide an alternative conclusion, we continue to conclude that QC

acceptance criteria developed from multi

- laboratory data more accurately reflect the performance of an an alytical method than QC acceptance criteria developed from single

laboratory data. The QC acceptance criteria in Method 1668A were estimates resulting from testing the method in a single laboratory. The acceptance criteria in revise Method 1668 are based on data from several laboratories. Comment 17: Appendix A. Section 1.0., 95% Confidence Intervals

- The section begins by describing

conventional, parametric calculations for all congeners in all matrices at all concentrations. Was the assumption of normality tested and met, or found to be unimportant? Some discussion seems warranted. Response: While a statistical test of normality is not extremely powerful for a small set of data, the

spiked QC sample results for nearly all congeners passed the Shapiro -Wilk test at alpha = 0.05. Those that did not pass generally had one or two extreme results, or systematic differences in recovery between matrices. A normal distribution of QC spike recoveries is also consistent with experience with past environmental studies. While results of environmental samples tend to follow a skewed distribution, percent recoveries of QC samples spiked at approximately the same concentration level tend to follow a more symmetric distribution. Comment 18: Appendix A Section 1.0, A pplication of Grubbs ’ outlier test

- Again, outlier tests should be used very carefully and interpretation of any consequent data sets based on truncated distribution statistics (see Grubbs (1969) and other sources for discussion). Response: Outlier te sts were applied to both blind sample data (as presented in Table 4

- 1 of the

report) and QC data. In both cases, EPA applied the Grubbs ’ test conservatively, following guidelines for maximum outlier removal as specified by ASTM D2777

- 98. In the case of sample data, these results were only used to calculate summary statistics of the sample results by chlorination level, and therefore outlier removal would not strongly affect conclusions about method performance. For IPR and OPR data, outlier removal was limited to one result per congener. For labeled compound data, there were more results, and a maximum of two results were removed per congener. As stated elsewhere, EPA had no strong evidence that these data did not follow a normal distribution, and therefore EPA’s use of the Grubbs ’ test is appropriate. Comment 19: As a simple example, what is the n in the equation given in Section 2.0? This is not as

easy a question to answer as might be suspected since some observations were thrown out. Response: For most analytes, the n in the equation given in Section 2.0 was 24, as stated in Section

5.5 on page 27 of the report. Two outliers each were removed for labeled congeners 111 and 178, so n would be 22 for these two congeners. Comment 20: Also, was the assumption of a normal distribution (upon which Grubbs ’ test depends)

appropriate? Response: T he assumption of normality was appropriate because the testing was performed in the

center of the analytical range. See also the response to the comment on Appendix A. Section 1.0., 95% Confidence Intervals (Comment 17), above. Comment 21: Finally, what Type I error rate was selected? Likely, the typical 0.05 was used yet one

could argue for a more conservative error rate.Method 1668A Interlab Study Peer Review

April 2010 13 Response: Y es, EPA selected a Ty pe I error rate of 0.05. A more conservative Type I error rate would yield QC inappropriately wide acceptance criteria, and would not be useful in detecting laboratories that are not performing the method appropriately or are out of control. 4. If you be lieve that a given item should be clarified, please provide the clarification by section number or define the clarification needed to a sufficient level of detail that we can make the necessary changes. Comment 22: A wide range of issues will be addres sed below as suggestions for clarifying presentation. Response: Our responses to this group of comments follow. Comment 23: Section 1.2, Line 3

- Change “ seen ” to “ seven ” ? Response: We have corrected this

. Comment 24: Page 4, Paragraph 2

- Please consider replacing “properly homogenized” with

“homogenized. ” The reasons are given above for removing the vague word from this sentence. One could easily argue that the samples were not shown to be “properly homogenized .” Response: We have removed "properly." Comment 25: Page 5, Paragraph 2 of Section 2.4

- The term, “ Youden pairs ” is first used here but not

defined until Section 2.5.1. Please move the definition up to the first use of the term. Response: We have move d the definition to its first use. Comment 26: Table 2

- 1 (page 4) and Table 3

- 1 (page 12)

- Itis extremely important in judging the

generalizability of the report conclusions that the laboratory participation in generating the final QC criteria is very clearly discussed. At present , it is not easy to follow. Generally, one starts with 14 laboratories (government and private) and ends up with data from 4 to 6 labs. It isn ’ t easy to judge how (1) withdrawal of the various laboratories from providing results, (2) the study author ’ s j udgment about reported data usefulness, and (3) the final data inclusion influences the statistical population of laboratories for which inferences about QC are being made. Several issues to be addressed include the following. Are the QC conclusions relevant to only the best or best equipped laboratories? Are they representative of primarily government laboratories, private laboratories, or both? A very clear and concise summary addressing this linchpin issue is needed. In developing this summary, discu ssion of the influence of outlier rejection is also warranted. Response: Where possible, we revised the report to clarify this discussion. EPA discusses exclusion

of a laboratory's data is in Section 3 of the report. EPA generally rejected a laboratory because the laboratory submitted incomplete data sets, insufficient raw data to verify a laboratory's results, or did not follow the method correctly. Exclusion of these laboratories does not mean that the conclusions are relevant only to the best laboratories, but to laboratories that perform the method correctly. Comment 27: Page 8, Section 2.7, first paragraph

- The requirements were given for calibration curves

but the laboratories interpreted the requirements differently. I saw no obvious reason for the misinterpretation; however, some thought should be put into rewriting the associated text so it is even clearer. Also, relative to Section 2.7.1. and 2.7.2, were the consequences of any of the various deviations quantifiable? For example, the differences due toMethod 1668A Interlab Study Peer Review

April 2010 14

different aliquot weights used by the various laboratories might be quantified with the methods cited above for Page 6, Top paragraph and first full paragraph. Response: We were unable to establish any clear relationship between the variou s deviations and the

results. The primary reason is that neither we nor the laboratories linked a specific deviation to specific data. If a laboratory chose to calibrate its analytical system different from the calibration suggested in Method 1668A, or cho se not to follow a procedure, we had no way to force the laboratory to re

- calibrate or to repeat the analysis. Comment 28: Page 9, Section 2.7.2

- Different problems were encountered by different laboratories.

Different weight normalizations were done by the different laboratories. It isn ’ t clear to me how this directly influenced statistical estimates and associated inferences. Also, the numerous deviations suggest that even more clarity needs to be placed into 1668A. Perhaps more discussion is needed about what deviations should not be considered. Response: In principle, we agree. Although Method 1668A is clear about the amount of material to

be extracted and the final extract volume , in revising the method we have provided more clarity about where flexibility is allowed and not allowed in conducting the method. We control laboratories under contract to us, so these deviations would not occur with our routine data gathering nor should they with others who contract for lab work

. In this interlab study, although we asked the participants to follow Method 1668A, we had no control over the deviations that these volunteer s eventually chose. Therefore, we used the data as provided except where we could justify finding it lacking, inconsistent with other data, or outlying. Comment 29: Page 11, Section 3, first paragraph

- This section begins by describing three laboratories

that did not submit data. It is important to know if non

- reporters are considered a biased or random subset of the population of p ossible responders. Was there a risk of biased sampling of possible laboratories using this method as a result of biased non

-response? Response: No. The laboratories that did not submit data were government laboratories , but we have

no evidence that the ir skill level differs from other private or commercial laboratories. Comment 30: Page 11, Section 3, fourth bullet

- How was this issue handled during the analysis of

results? Would omission of an outlier laboratory create a biased sample of laboratories? Response: Results from the laboratory were not considered in the analysis for the reasons given in

this bullet in the report. The omission of this laboratory would not bias the results. We could not question the laboratory about poor recoveries or other problems. In addition, because the laboratory is no longer in business, the laboratory is no longer part of the population of laboratories performing this method. This laboratory is not an "outlier laboratory

. " We did not have any such laboratories because there were not enough laboratories in the study with which to perform an outlying laboratory test. Comment 31: Page 15, Table 4

- 2 and all other tables

- Univariate statistics are applied throughout these

tables. Some graphs suggest skewed distributions for some data. Was the assumption of normality or the robustness of conclusions to violations of this assumption assessed? Response: Many of the univariate statistics and graphs combine congeners, and as such the

distribution of results would be affected by differences in the prevalence of congeners in the environment and in coelution. While a statistical test of normality is not extremely powerful for a small set of data, the spiked QC sample results of most congeners passed the Shapiro

-Wilk t est at alpha = 0.05. Those that did not pass generally had one or two extreme results, or systematic differences in recovery between matrices. A normal distribution of QC spike recoveries is also consistent with experience with past environmental studies. While results of environmental samples tend to follow a skewedMethod 1668A Interlab Study Peer Review

April 2010 15

distribution, percent recoveries of QC samples spiked at approximately the same level tend to follow a more symmetric distribution. Comment 32: Page 16, first lines

- This discussion is too speculative in my opinion. Response: The sample

-preparation lab prepared a solution containing all of the congeners. The

laboratory then lined up all of the sample bottles, filled each with POTW effluent, and spiked an aliquot of the solution into each bottle. The lab then analyzed two of the samples and found all congeners to be present in the concentrations spiked (within experimental error). The lab shipped the bottles to the participant labs. The mean of the concentrations found by the participant labs for the mono

- and di

- CB congeners was 3 and 54 %, respectively, whereas the mean for the tri

- through deca

- CB congeners ranged between 68 and 96 %. The most logical explanation is that the congeners were lost during transit or when the bottles were op ened at the participant laboratories. Remember, EPA recovery corrected all results using the labeled compounds as reference, and labeled compound recoveries were within the normal range, so the losses were real and occurred prior to the time the samples were spiked with labeled compounds at the participant labs. Comment 33: Page 17, Figure 4

- 1 (and also 4

- 3, 4
- 5, and 4

- 7)

- As dictated by the generally

- accepted

equation given above, the relative uncertainty of a measurement will rapidly increase as concentration decreases. The following figure gives the general shape of such a relationship. Depending on the positions of the detection limit (LOD in this figure) and limit of quantitation (LOQ in this figure) (from Newman (1995)), the variation in the y dire ction as a function of measured concentration would be totally expected based on this relationship. Would it be possible to insert vertical lines at the LOD and LOQ in these figures so a reader could judge this issue? As a good example of how this would help, it is difficult to decide if the explanation for the Figure 4

- 5 outliers (Page 20, first few lines) is reasonable or whether the variations at low concentrations (in lower part of the Region of Less
- Certain Quantitation? are what you would expect for any measurement process. Response: Itwould not be possible to insert lines at the LOD and/or LOQ in Figure 4

- 1 or the other

figures, because different participant labs used different sample volumes and different final extract volumes. Further, and mos t unfortunate, the congeners at low spike concentration were also those at low chlorine level. Therefore, the apparent resemblance to the Newman model is misleading. Comment 34: Page 18, Figure 4

- 2 (and also 4.4, 4.6)

- These figures do not convey any useful

information. Relative Standard Deviations (e.g., 4

- 3) are much more useful for conveying the associated information (that variation can change with concentration). Response: These figures can be useful in determining where in the concentration s cale absolute

variability begins to increase linearly with concentration. Comment 35: Page 21, Table 4

- 4
- A simple mistake was made here such that the “ # pairs ” column has

two digits after the decimal point, e.g., 32.00 pairs. Response: We have corrected the number of pairs to a whole number.Method 1668A Interlab Study Peer Review

April 2010 16 Comment 36: Page 29
- The conventional “ This page intentionally blank” statement is made but, in fact,

there is text on the page. Response: We have corrected the offset between this page and the previous page. 5. Please point out errors or omissions. Comment 37: Please see comments above. Response: We responded to these in the preceding section

. III. SPECIFIC OBSERVA TI ON S [none submitted]
IV. AD DITIONA L REFERENCES 1. Anscombe, F.J. 1960. Rejec tion of outliers. Technometrics 2: 123

- 147. 2. Grubbs, F.E. 1969. Procedures for detecting outlying observations in samples. Technometrics 11: 1
- 21. 3. In gamells, C.O. 1974. New approaches to geochemical analysis and sampling. Talanta 21: 141

- 155. 4. In gamells, C.O. and P. Switzer. 1973. A proposed sampling constant for use in geochemical analysis. Talanta 20: 547

- 568. 5. Keith, L.H., W. Crummett, J. Deegan, Jr., R.A. Libby, J.K. Taylor, and G. Wentler. 1983. Principles of environmental analysis. Anal. Chem. 55: 2210

- 2218. 6. Newman, M.C. 1995. Quantitative Methods in Aquatic Ecotoxicology. Lewis/CRC Publishers, Boca Raton, FL, pp. 426. 7. Rosner, B. 1983. Percentage points for a generalized ESD many

- outlier procedure. Technometrics 25: 165
- 172
. 8. Stuart, A. 1968. Basic Ideas of Scientific Sampling, Hafner Publishing Co., New York, pp. 99. 9. Visman, J. 1969. A general sampling theory. Mat. Res. Stand. 9:9

- 64. 10. Wallace, D. and B. Kratochvil. 1987. Visman equations in the design of sampling plans for chemical analysis of segregated bulk materials. Anal. Chem. 59: 226

- 232. 11. Woodworth, G.G. 2004. Biostatistics. A Bayesian Introduction. John Wiley & Sons/Method 1668A Peer Review

October 2007 17



## PEER REVIEW COMMENTS FROM CHA RLES S, WONG NA ME : Charles S. Wong , Ph.D. AF FILIA TION : University of Alberta, Dept. of Chemistry DA TE : September 18, 2007 COMMENTS ON : Method 1668A Interlaboratory Validation Study I. GENERA L IMPRESSIONS Comment 1: This document is a synopsis of the interlaboratory validation study intended to improve

EPA method 1668A, a protocol for congener

- specific measurement of polychlorinated biphenyls (PCBs) in water, soil, sediment, biosolids, and tissue by high resolution gas chromatography/high resolution mass spectrometry. Fourteen laboratories were invited to participate, and samples as well as instructions and required reporting information was provided to them. Reported data was compiled and summarized. Comment 2: The document provides a useful but succinct summary of the interlaboratory validatio ns

very large dataset. Sufficient detail is provided on the background on Method 1668A and the methodology used to invite participating laboratories to ascertain the effectiveness of the validation study in characterizing measurement of PCBs in the sample s chosen. Methods for selection of samples for the laboratories are clear and reasonable, including explicit discussion of limitations for logistical and technical purposes. A clear and detailed discussion of deviations from the required protocol is evident, so that the reader can clearly understand limitations in the reported data. For the most part, the discussion of the usable data is clear and pertinent to the intended goal of determining possible changes to the protocol and to acceptance criteria. Response : No response needed

. II. RESPONSE TO CHA RGE QUESTIONS 1. Please review the EPA’s Report of the Method 1668 Interlaboratory Validation Study for the objectives of the study, the soundness of the technical approach, and the results produced. Comment 3: The objectives of the study are sound. Should Method 1668A be revised, and if so, what

protocol changes and/or acceptance criteria should be modified? This can only be done with an interlaboratory validation of a standard set of samples. Co mment 4: Sample selection is reasonable in my opinion. Biosolids and tissues were prepared by

EPA at two distinct concentrations, properly homogenized, and shipped to each laboratory with appropriate safeguards against decompositions to ensure that tha t the integrity of the samples upon receipt by the participating laboratory was maintained. While it would have been ideal to have wastewater samples with PCBs shipped to the laboratories, it is reasonable to assume that testing of Method 1668 capabilitie s would not be feasible. Thus, providing instructions to spike the wastewater effluent shipped to the laboratory was reasonable. Response: The wastewater samples were spiked with PCBs at a sample

-preparation laboratory. The

participant laboratories spike d the 13 C
- labeled PCBs in their l aboratory.Method 1668A Interlab Study Peer Review

April 2010 18 Comment 5: As previously stated, deviations from the method were characterized in a succinct but

meaningful manner. This allowed the flagging of suspect data as unusable for further purposes. Many of the deviations noted had details on the underlying rationale, e.g., problems with cleanup of samples encountered by some of the laboratories. However, not all such deviations were detailed in this manner, which I would have found useful. It is troubling that there were sufficient problems in sample processing, measurement, and even data reporting (e.g., no calibration reporting from one laboratory???) that less than half the data from the participating laboratories was usable (Table 3

- 1), as part of the point of having a large number of independent participants was to assess the robustness of both the overall Method as well as the validation for improving the Method. That having been said, I feel that the most problematic deviations were flagged unusable, and that the usable data is sufficient for the intended purpose. Response: You have verified our conclusions concerning problems encountered with the samples

and data reporting. Comment 6: Itwould have been useful if the laboratories were asked to provide measurement data on

QC Check samples (Method 1668A, Section 7.15) from standardized reference materials (e.g., NIST SRMs or Canadian NRC SRMs) in addition to the other QC data, e.g., OPR samples in reagent water, and blanks. This is important because as noted in the report, PCB concentrations in the shipped samples were unknown, and having this check would be useful. If this was actually done, this was not clear from the report. Response: EPA asked laboratories to analyze spiked and typical Clean Water Act matrices, i.e.,

wastewater, tissue, and biosolids (sewage sludge.) These samples better represented the intended application of this method for environmental monitoring. In addition, laboratories used to using the available QC check samples prepared in reagent waters or solvents might be overly familiar with their content. Method 1668A has been available for use by laboratories since 2000. Because it contains the suggestion for a QC Check sample in Section 7.15, many labs had been using a NIST SRM or NRC CRM, and might recognize the sample from having analyzed it previously. Comment 7: The reported results provided a good summary of the variability observed with PCB

measurement in the various matrices supplied to the participating laboratories. This information is most useful for resetting IPR/OPR guidelines. The summary and recommendations appear reasonable, e.g., using average relative responses under specific circumstances, noting poorer and more variable recoveries for lower chlorinated congeners, and somewhat tighter recovery windows particularly for heavier congeners. Response : No response needed. 2. Please render an opinion as to suitability for use of the data from the study for revision of the QCAC in Method 1668 to make it Me thod 1668B. Comment 8: I feel that this data, despite limitations noted above and in the report, are of sufficient

quality to revise the method for the next version of this Method. Response: We agree that EPA should move forward with the revised QC acceptance criteria and

other changes to Method 1668A. 3. If you have a suggestion for an improvement, please provide suggested text by section number. Comment 9: In Method 1668, Section 23.0, page 61: It is useful to provide the PCB numbering

system used, to avoid confusion. I realize that CAS registry numbers are provided, whichMethod 1668A Interlab Study Peer Review

April 2010 19

eliminate ambiguity; however, most discussion of PCBs is by the numbering system, so having that clarified explicitly in the document is very useful. See comments below on references. Response: Itis important to link the PCB numbers to the widely

-used and internationally

-recognized

CAS Registry numbers to minimize ambiguity about the identity of specific congeners

. 4. If you believe that a given item should be clarif ied, please provide the clarification by section number of define the clarification needed to a sufficient level of detail that we can make the necessary changes. Comment 10: Itwould be useful to provide more details on the instructions to the particip ating

laboratories as how to spike the shipped wastewaters with PCBs. Response: The wastewater samples were spiked at the sample

-processing laboratory. See Section

2.5.2 on page 6 of the report. Comment 11: Also, as stated previously, additional QC data from the laboratories should be given to

help ensure that the usable data was of high quality. Response: None of the participant laboratories submitted data for a QC Check sample. Because the

participant laboratories were volunteers, EPA could not req uire analysis of a QC Check sample. 5. Please point out errors or omissions: Comment 12: In the document “ Changes to be made to EPA Method 1668A from the 8/20/03 revision

based on the Method 1668A interlaboratory validation study and comments and corre ctions received from laboratories “, Under Quality control acceptance criteria, Table 5
- 1 from the interlaboratory report should replace Table 6 not Table 5 in Method 1668A. Comment 13: In Method 1668A, Section 13.5.1.4, page 48, typo: should read If calibration is not

verified not If calibration is not verified. Comment 14: In Method 1668A, Section 22.0, page 59: the title for reference 4 should be provided for

consistency. Comment 15: In Method 1668A, Section 22.0, page 60: the title for reference 18 should be provided

for consistency. Comment 16: In Method 1668A, Appendix A, Section 2.1, page 110: typo, should be individual

congeners not individual congene4rs. Response: We have made these corrections.
III. SPECIFIC OBSERVA TI ONS These comments refer to the report on the interlaboratory study. Comment 17: Page 14, Section 4.2, Table 4

- 1
- I do not feel that reporting of concentrations to up to 5

significant figures is justifiable. The GC used must have either a splitless or on

- column injector (Method 1668A, Section 6.9). The latter in particular does not have sufficient precision to justify 5 significant figures. Three or perhaps four significant figures is the maximum that should be present.Method 1668A Interlab Study Peer Review

April 2010 20 Response: We have rounded all results to three significant figures maximum. Comment 18: Page 14, Section 4.2, Table 4

- 1
- In addition, while providing a mean, median, and

maximum for concentrations of usable data is useful, it is difficult to assess the distribution of data. This is important, as the fairly large differences between means and medians suggest skewed data distributions, which are suggested by the various figures of variability. It would be useful to provide a box plot of this data to make assessment much easier for the reader. Respons e: A box plot would be so busy that the reader would not be able to discern useful

information. Comment 19: Page 21, Section 4.6, Table 4

- 4
- The same comments regarding significant figures from

my comments on Section 4.2, page 14, Table 4

- 1 hold here as well. Response: We have rounded all results to three significant figures maximum

. Comment 20: Inclusion of the interlaboratory reports Appendix A, Statistical Procedures Used to

Develop QC Acceptance Criteria in the Method as an Appendix may be use ful. Response: This is not necessary because the interlaboratory study report has been posted on the

CWA methods website. It would serve no useful purpose to detail how the QC acceptance criteria were developed in an Appendix to the method. EPA plans to provide the report as a part of the rulemaking package so the information would be available to readers interested in how EPA developed the QC acceptance criteria. Comment 21: The following comment is suggested for improvement of Method 1688A, and not th e

interlaboratory report: Comment 22: In Section 5.0 on Safety, pages 5

- 7, some explicit mention should be made about the use

of appropriate biohazard containment for samples which are biosolids, as these are likely to host pathogens which are a potential danger until rendered harmless by disinfection. This would include preparation of raw samples in a biosafety cabinet until pathogens are killed (e.g., by soaking in solvents), appropriate safety garments and emergency procedures for accidental spills of pathogenic material, and warning about covering all materials containing solvents in biosafety cabinets to prevent solvent fumes that can pose an explosion hazard in biosafety cabinets that do not vent to exterior air. Response: We agree and have added a sub

- section on biohazards to Section 5.
IV. AD DITIONA L REFERENCES 1. Mills, S.A., III; Thal, D. A., Barney J. (2007)

-A summary of the 209 PCB congener nomenclature, Chemosphere 60, 1603

- 1612.Method 1668A Peer Review

October 2007 21



## PEER REVIEW COMMENTS FROM KURUNTHACHA LA M KA NNA N NA ME : Kurunthachalam Kannan , Ph.D. AF FILIA TION : Research Scientist. Wadsworth Center, New York State Department of Health & Professor, Department of Environmental Health Sciences. State University of New York at Albany. Empire State Plaza PO Box 509. Albany, NY 12201
- 0509 DA TE : March 24, 2010 COMMENTS ON : Method 1668A Interlaboratory Validation Study Review of “Method 1668A Interlaboratory Validation Study Report” and “ Addendum to the Method 1668A Interlaboratory Validation Study Report Draft, August 2009 . Method validation is an integral part of a good analytical laboratory practice. The method 1668A validation study was intended to provide the Environmental Protection Agency (EPA) with sufficient amount of data to evaluate the performance of the method. The method 1668A was originally validated in 2000, in a single laboratory study. However, the guidelines published by the EPA and American Society for Testing and Materials (ASTM) recommend a minimum of six complete data sets for validating a method. The obj ectives of the validation study were to characterize the performance of the Method 1668A in multiple laboratories and matrices and to evaluate and revise the Quality Control (QC) acceptance criteria. The objectives of the study are sound and logical. The multi

- laboratory validation study is a requirement to meet the national and international criteria to ensure reliability, consistency, and accuracy of the analytical data generated using the Method, especially when the method is used in the EPA’s Clean Water Act programs. The major outcome of the validation study was a revision to the quality control/quality acceptance (QA/QC) criteria in EPA Method 1668A. Itis agreed that method validation studies are time and resource consuming and EPA’s effort to perform this validation is highly commendable. Coordination of activities among several participating, volunteer, laboratories is a daunting task. EPA’s efforts on this method validation are significant; nevertheless, data generated from this study are under

-utilized and with the existing data it is possible to generate additional information required for validation. 1. Please review EPA's Report of the Method 1668A Interlaboratory Validation Study and its addendum. Comment on the objectives of the study , the soundness of the technical approach, and the results produced. Comment: The objectives of the study are sound and logical. The multi

- laboratory validation study

is a requirement to meet the national and international criteria to ensure reliability , consistency, and accuracy of the analytical data generated using the Method, especially when the method is used in the EPA’s Clean Water Act programs. The major outcome of the validation study was a revision to the quality control/quality acceptance (QAQC) criteria in EPA Method 1668A. The overall technical approach , ranging from selection of laboratories, sample selection, sample analysis, data reporting, data review, and validation, is good, although there are some potential areas for improvements. A major drawback with the report is on the biosolids data set. The objective of the study was to have 6 complete data sets, as per the EPA/ASTM guidelines, to validate the method. Only 4 complete data sets have been available for biosolids (Table 4

- 1), although validation for biosolids appears to be an important goal for the method, as indicated on the title. Furthermore, the data submitted by the 4 laboratories for biosolids are highly variable and no ‘in

- depth’ statistical analysis of data set has been made to determine accuracy and precision of the data submitted forMethod 1668A Interlab Study Peer Review

April 2010 22

biosolids. In other words, the Method has not adequately met the objectives to validate the analysis of PCBs in biosolids. Many international organizations such as the International Org anization for Standardization (ISO) require at least eight complete data sets to validate the methods. Because the data set available for biosolids does not meet the EPA/ASTM guidelines, validation of the Method 1668A for biosolids is incomplete/inadequat e. Response: Reasons for receiving only four data sets for biosolid samples are in Section 3 of the

Report. EPA believes that the high failure

-to -report rate may be attributable to (1) the extensive effort needed to set up the extraction, concentration, cleanup, and HRGC/HRMS systems for determination of 209 PCB congeners, (2) the difficulty and lack of familiarity with the biosolids matrix, and (3) the voluntary nature of participation in the study. Given that four labs were able to analyze the biosoli ds samples successfully, EPA chose to use the data

. Although the ASTM guideline , D2777 , generally requires six complete data sets for method validation, ASTM has approved a means for taking exceptions to the requirement for six complete data sets. The As sociation for Official Analytical Chemists, AOAC, has similar requirements and exceptions. Agency guidelines for validation of chemical methods were developed by the Forum for Environmental Measurement, FEM , as guidance , not requirement s . These guidelines do not specify a minimum number of labs for a validation, and refer to D2777 as an example protocol

. Each program office (e.g., EPA’s Office of Water) may apply the guidelines as appropriate to specific situations , and t o the extent practicable, our vali dations are consistent with the F E M guidelines and other guidelines. In the United States, validation principles developed AOAC and ASTM are more widely followed than the international ISO guidelines

. EPA did not desire to validate Method 1668 internationally, as was done with EPA Method 1613 for chlorinated dioxins and furans

. Note: The four lab comment is repeated as comment “m” in section 5 , and comment “a” in section 6 of this review. Comment: The second major concern with the validation study re sults is that the data

analysis/review is focused heavily (or only) on precision , by measuring variability in concentrations reported by the laboratories for PCB congeners. Little attention has been paid to the accuracy of the results. Only for wastewater samples, recovery values are presented as a measure of accuracy (only for wastewater, the spiked concentrations were known and therefore the recovery could be measured). Accuracy is an important criterion in method validation. There are several ways to measure accuracy, without increasing the burden on the laboratories. Accuracy could have been assessed by analyzing a sample of known concentration, as a laboratory control material, and comparing the measured value to the true value. National Institute of Standards and Technology (NIST) reference materials are available for PCBs and laboratories could have been supplied or asked to analyze NIST reference materials, along with samples supplied by the EPA. It has been stated that ‘to minimize the burden on volunteer participants’, laboratories were not required to analyze initial precision and recovery (IPR) samples. Nevertheless, accuracy is an important parameter for validating accuracy of a method, a well

- characterized matrix such as NIST SRMs can be u sed. Alternatively, the technique of standard additions, which is used to determine recovery of spiked congeners from on

- going precision and recovery (OPR) samples (sand/oil and water) could have been used (Results from Table 4

- 4) to validate the accuracy of the method.Method 1668A Interlab Study Peer Review

April 2010 23 Response: Table 4

- 4 presented results for accuracy (as recovery) for the 27 labeled compounds

spiked into all matrices (water, biosolids, and tissue) by level of chlorination (homolog). Mean recoveries ranged from 57 to 85 percent. We believe that these recoveries are reasonable for these analyses given the number of analytes and matrices covered by 1668C. To address your concern about recoveries for the dioxin

- like (Toxic) congeners, we have expanded Table 4

- 3 of the Report to list precision and recovery for the 27 individual labeled congeners from wastewater and have expanded Table 4

- 4 to list precision and recovery for all 27 labeled congeners (including the Toxic congeners) spiked into all wastewater, biosolids, and tissue samples. Revised Tables 4

- 3 and 4

- 4 are at the end of these responses. Regarding use of a Standard Reference Material (SRM) from NIST or a certified reference material (CRM) from another source, EPA did not use such materials to address previous criticisms for no t using “real

-world” sample matrices that the Agency normally encounters in its data

- gathering activities . In addition, SRMs and CRMs are often dried or do not contain the mixture of pollutants found in real world samples . Therefore, we tested real

-world wastewater, biosolid, and tissue samples, in the forms typically submitted to laboratories for analysis (i.e., whole

-volume water samples, wet biosolids, and tissues)

. Comment: The third issue on results : The method 1668A is intended for congener specif ic analysis of PCBs with an emphasis on toxic congeners (twelve non

- and mono

- ortho substituted PCBs). The scope of the method 1668A indicates that twelve PCB congeners designated as ‘toxic’ by the World Health Organization (WHO) are determined. Neverthe less, the validation study is focused on ten homolog groups. The data analyses involved summation of all isomer data within a congener group. Precision and accuracy of the Method 1668A for the analysis of individual toxic congeners, are not validated. In other words, the method is not validated for 12 toxic PCB congeners. Response: Method 1668A was validated in wastewater for all 209 congeners, as individual

congeners and coeluted congener groups, and in all matrices for the labeled analogs of the dio xin

- like (“Toxic”) congeners and congeners at the earliest and latest retention time at each level of chlorination (27 labeled congeners total). Presenting results by level

- of

chlorination (homolog) was a means of condensing results so that they could be understood more easily. However, to respond to your concern about presenting results for the Toxics, we have produced a table of recovery and precision for the 27 labeled compounds individually, and shall replace Tables 4

- 3 and 4

- 4 in our revision of the Report to show recoveries of the labeled compounds spiked into wastewater and into all matrices, respectively. The tables are attached for your use at the end of these responses. Comment: Furthermore, the suitability of this method for the analysis of mono - and di

chlorobiphenyls (CBs) is questionable for the reasons given below. First, the precision and accuracy for mono

- and di

- CBs are very poor, if not dismal. For example, mean recovery of mono

- and di

- CBs in wastewater samples were 3 and 54%, respe ctively. Furthermore, the concentration mono

- CB found in wastewater sample is similar to what was found in blanks (mean for wastewater is 27 pg/L [Table 4

- 1] and that for water blank is 26 pg/L [Table 4

- 2]). Obviously, the QC criteria have been relaxed to reflect the poor recoveries (Addendum to the method validation study). Overall, the method did not perform well for mono

- and di

- CBs. If these two congener groups are removed, then the QC criteria for the method would be tighter and more reliable/repro ducible. These two congeners are easily lost in the solvent evaporation steps, as has been stated in several places of the interlaboratory study report.Method 1668A Interlab Study Peer Review

April 2010 24 Response: As explained in the second paragraph of Section 4.4 of the Report, low recoveries at

levels of chlorination (LOCs) 1 and 2 for wastewater (Table 4

- 3) may be due to loss during transport from the sample preparation laboratory to the participant laboratories because data reported by the sample preparation laboratory showed that the congeners were present immediately after shipment in replicates of the samples held by the sample preparation lab. Because the labeled compounds at LOCs 1 and 2 were recovered in the normal range by the participant labs, losses during extraction and/or cleanup can be ruled out. Therefore, and as explained in Section 4.4, the only mechanism for loss we could think of is loss to the headspace of the sample bottle during shipment, with subsequent loss to the atmosphere when the bottle was opened. Resources were not and are not available to verify this theory. Also, we have added labeled compound recoveries for wastewater by LOC in Table 4

- 3, so that it is clear that the labeled compounds were recovered at LOCs 1 and 2, even though the native compounds were not. We reco gnize that the performance of this method will vary among the 209 congeners, and in different matrices. This is typical of multi

- analyte methods because not all chemicals respond identically to extraction and clean up techniques, or have identical instrument responses. In a study of data comparability between two laboratories on samples collected from the Passaic River in New Jersey, in which 151 PCB congeners were identified and measured, accuracy as measured by analysis of a NIST SRM was 15% or better. Recoveries of the PCB congeners ranged from 90% to 124% and averaged 105%; precision ranged from 4.2 to 23%. Comment: A few other issues on the results , such as robustness, ruggedness, and limit of detection

have not been discussed or analyzed in detail. A method validation would require that the above mentioned parameters are tested and validated. Some of the information presented on the validation report is not clear (e.g., section 4.2). For example, how were the outliers detected? How many outliers were found in samples? Rationale for removal of the outliers from data set is missing. A column showing number of outlier values should be presented on Table 4

- 1. The reproducibility variation coefficients (between

- laboratory precision and within

- laboratory precision) for the method were not reported. Overall, data have not been fully utilized to describe several parameters that are required in validating a method (based on the comment 2 above and this comment). Response: Robustness is indicated by method performance on multiple matrices, the results of which

are presented in revised Table 4

- 4 for the 27 labeled compounds spiked into all samples. These results show that the labeled congeners are recovered from wastewater, biosolids, and tissue within ranges expected for these types of analyses. W e did not examine ruggedness directly in the interlab study . Ruggedness testing is done in a single

- laboratory study when a method is developed. As stated previously, we expanded the performance data gathered during the validation study to show recovery and precision on an individual congener basis for the 27 labeled congeners, and have include d an updated table in the next revision to the Report. EPA provides an updated table at the end of these responses . In addition, evaluation of ruggedness (the effect of changes to variables in the sample preparation, extraction, cleanup, and instrument steps) f or a method such as Method 1668A, which is applicable to a variety of sample matrices, can be counter productive. EPA’s historical experience suggests that the sample extraction procedures used for the various matrices are likely to have the greatest effect on the results . Thus, both the method and the study

- specific instructions sought to minimize deviations from the extraction steps described in the method.Method 1668A Interlab Study Peer Review

April 2010 25

Regarding limit of detection (LOD), evaluation of LOD was beyond the scope of the study. An LOD study would require 7 replicate analyses minimum of spikes at low concentrations to produce method detectio n limits (MDLs). Regarding outliers, as stated in the final paragraph in Section 3 of the Report, “Study samples were assessed for outlying results using Grubbs’ outlier test, performed in accordance with Standard Practice for Determination of Precision and Bias of Applicable Test Methods of Committee D

- 19 on Water (ASTM D2777

- 98) . Details on the outlier analyses are presented in Appendix A.” In Appendix A, we give further detail on removal of outliers. As stated in Appendix A, one outlier was removed for the initial and on - going precision and recovery test, and two outliers were removed for the test of labeled compound recoveries from samples. Because only three outliers were removed, we did not list these three values. Regarding presentation of between - and within

- laboratory precision, pooled within

-pair RSDs were presented by LOC in Table 4

- 4 for the 27 labeled congeners spiked into all samples. We have expanded the presentation to all 27 labeled congeners in the next revision to the Report, and h ave attached the revised table at the end of these responses. 2. Please render an opinion as to suitability for use of the data from the study and the addendum for revision of the QA/QC in Method 1668 to make it Method 1668B. Comment: I believe that con siderable effort has been put forward in validating the method 1668A.

Nevertheless, the data from the validation study are under

-utilized and more important validation parameters such as accuracy of the method, reproducibility, ruggedness, etc should be e valuated. These issues and corrections should be corrected/clarified. Once these revisions are made, the method may be revised to 1668B. However, I believe that there is lot more work to be done. Response: As stated above, accuracy (as recovery), and reproducibility and ruggedness (as RSD)

data were presented by LOC in Tables 4

- 3 and 4

- 4. EPA's Quality System defines accuracy to include precision and recovery (bias), but most use accuracy and recovery synonymously. Table 4

- 4 of the Report gave reco veries across all labs and matrices. Reproducibility is indicated by the RSDs presented in Table 4

- 4 and, although we did not evaluate ruggedness, per se, in the study, how rugged a method is indicated by the precision of the measurements (if the method i sn't rugged, differences in the way different labs practice the method will cause great variability in results). Also as stated above, we have expanded the data in Tables 4

- 3 and 4

- 4 to show recovery and precision on an individual congener basis for the 27 labeled congeners, so that reproducibility (as precision) can be viewed on a per

- congener basis. The revised tables are at the end of these responses. 3. If you have a suggestion for an improvement of the interlaboratory study report or addendum, please provide suggested text by section number & 4. If you believe that a given item should be clarified, please provide the clarification by section num ber or define the clarification needed to a sufficient level of detail that we can make the necessary ch anges & 5. Please point out errors or omissions. I feel that the charge questions 3, 4 and 5 are inter

-related. The following items would help improve the study report and addendum Comment:Method 1668A Interlab Study Peer Review

April 2010 26

a. Page 3, Section 2.4 (Sample Selection): Some description of samples is needed. For example, does the fish tissue represent wholebody homogenate or fillet? Also, please specify when these fish samples were collected? I assume that these are wet samples (i.e., not freeze

- dried). Similarly for biosolids, some sam ple description is needed. After reading the entire document, I realized that these are wet samples. It would be useful to indicate on page 3 (section 2.4), what biosolids really are and when they were collected; also state that they are wet and if possible indicate the moisture content? Response: As stated in Section 2.4 of the Report, the tissue and biosolids samples were generated

from excess sample collected during two unrelated EPA studies. The biosolids samples were collected in 2001 as part of EPA’s 2001 National Sewage Sludge Survey. The fish tissue samples were collected in 1999 and 2000 during the early phases of EPA’s multi

year National Lake Fish Tissue Study (NLFTS). All of the tissues used in the Method 1668A validation study were fro m bottom

- dwelling fish species and were originally prepared as whole

- fish composite samples. The samples for the NLFTS were prepared as finely ground tissue by a single laboratory. Excess sample beyond that shipped to the laboratories during the NLFTS wa s archived in 500

- mL j ars and stored frozen. (The NLFTS also collected samples of predator species, from which fillets were taken and composited for analysis. The bottom

- dweller samples provided more tissue than the predator samples, and thus a greater ex cess than was available for other purposes such as a Method 1668A validation study.) The tissue samples used for the validation study were prepared from the 500

- mL archive jars. We have added this information to our revision to the Report. Regarding your concern about the fish tissue type (whole; fillet), the fish tissue homogenate was from a mixture of ground whole fish and ground fish fillets. Results of analyses of hundreds of fish tissue samples have demonstrated that the performance of Method 1668 is independent of tissue type. Results were reported on a wet

-weight (as

received) basis, not a lipid basis. Both the biosolids and tissue samples had been stored in freezers at an EPA sample repository. In order to have a sufficient amount of each samp le available to support the study, EPA identified several samples of each matrix type that could be combined to produce large volumes of Youden pairs with the desired congener distribution. Data for the PCBs from those original studies were used to select samples that contained PCB congeners in the range of interest for the validation study (e.g., neither the cleanest nor the most contaminated samples were selected, but rather samples that would contain PCBs in the general operating range of Method 1688A). Because of the chance that biosolids samples could lose low

- molecular weight PCB congeners during any drying procedure, the biosolids samples were not dried. Thus, EPA submitted biosolids samples to the laboratories in the form that EPA believes is typi cal for this matrix: wet, with varying moisture contents that reflect the source of the material. The original biosolids samples used to prepare the study samples were collected as solids, as opposed to the pourable liquid sewage sludges that may be produced at some wastewater treatment facilities. Each laboratory that analyzed the biosolids samples determined the percent solid contents. The reported values were in the range from 30 to 40% for the two samples, amounts that are typical for many biosolids produced in the U.S. Results for the biosolids samples were reported on a dry

-weight basis. Comment: b. Did the wastewater effluent samples contain suspended solids? When aliquoting

wastewater samples, how was the homogeneity in the distribution of suspended solidsMethod 1668A Interlab Study Peer Review

April 2010 27

between aliquots ensured? Wastewater samples have been spiked with PCB congeners. Itappears that there was no PCB present in wastewater sample prior to spiking. Was that true? I suspect that PCBs can be detected in wastewater samples at several tens to hundreds of pg/L level. The information on the spike concentrations listed on Table 2

- 2 is not clear. What was the solvent that was spiked into water? Response: The wastewater samples were prepared from a final effluent sample collecte d from a

publicly owned treatment works (POTW) near the sample preparation laboratory. NPDES permit limits typically restrict the total suspended solids (TSS) content of final effluents to values around 10 mg./L. Thus, the parent wastewater did not contain appreciable suspended solids; and there was no difficulty in aliquoting the wastewater samples. The unspiked wastewater from divided into 1

- L aliquot in suitable glass bottles and each aliquot was spiked separately (rather than spiking a bulk volume of wastewater and then subdividing the spiked sample into replicate aliquots). The sample preparation laboratory analyzed several spiked aliquots to ensure homogeneity. In addition, the study

- specific instructions provided to each participating laboratory required that the laboratory filter the wastewater sample prior to extraction and treat both the filtrate and any solids on the filter in the manner described in Method 1688A. This instruction was included to prevent problems in which some laboratories fo llowed the method as written, and others deciding to skip the filtration step if the wastewater did not appear turbid, as detailed in Method 1668A. This information has been added to the Report. Regarding the presence of a PCB background in the wastewater, the unspiked wastewater sample was analyzed by the sample preparation laboratory using Method 1668A. Out of the 209 PCB congeners, 39 congeners were detected in the sample. All of those congeners were between PCB 001 and PCB 168, and all of the concentrations were between 17 and 247 pg/L, well below the concentrations of the spikes of the congeners into the wastewater (see Table 2

- 2 of the Report for the spiking levels). Results for the blanks were all below the calibration range of Method 1668A as practiced by the sample preparation laboratory. The sample prep lab thus flagged the results for the blanks as estimates. In preparing the actual study samples, EPA decided that these background levels were low enough that adjustments need not be made to the amount of each analyte spiked into the samples. The solvent used for spiking the congeners into wastewater was acetone. The amount of acetone used to spike the Youden pair samples was either 0.5 or 0.6 mL per 1
- L volume of wastewater. This informati on has been added to the Report. Comment:
c. Page 5, Section 2.6: Did all the participating laboratories use HRMS? The

Method 1668A calls for the use of HRMS and the method validation requires the use HRMS. The deviations from the Method 1668A have been listed for certain parameters (section 2.7 on page 6), but not sure if all of the laboratories used HRMS for the analysis. Also, on page 5, it has been stated that the laboratories were instructed to follow the Method 1668A, it will be helpful to mention/confirm that on Section 3 (Data Review and Validation) that all laboratories indeed followed the Method 1668A (other than the deviations noted on section 2.7). Response: All participating labs used HRMS, and all labs claimed that they followed Method 1 668A

other than for the deviations noted in Section 2.7 of the Report. Comment: d. Page 6, line 3: Expand “SOW”; the abbreviation is used here for the first time.Method 1668A Interlab Study Peer Review

April 2010 28 Response: We have spelled out “SOW” to indicate Statement of Work. Comment: e. Page 6; Laboratory reporting of rules: One important item that is missing is the

conversion of data from ‘wet weight’ to ‘dry weight’ and vice versa (for biosolids). Response: Section 17 of Method 1668A states how to calculate the concentration of a congener on a

wet - or dry

-weight basis, as appropriate. The study

- specific instructions stipulated that each laboratory determine the percent solids of the samples, using no more than 2.5 g for that purpose, to ensure that sufficient material was available for s everal analyses by Method 1668A. The reporting instructions also stipulated that the biosolids results be reported on a dry

-weight basis. Other reporting requirements may be used when required by State or Federal regulations and it is not EPA’s intent to preclude such approaches in an analytical method. We have added the information on the amount used for percent solids determination and the basis on which results were to be reported to the Report. Comment: f. A major issue with the method is lies in the weight of samples used for extraction

(especially biosolids) and final extract volume. I think the method should be modified to incorporate some flexibility because the weight used for extraction can vary depending on the concentrations, presence of interferences, etc. Also, concentration to a final volume of 20 uL prior to injection can introduce large variability in analytical results. I would prefer to use a sample weight of 15 g for biosolids and a final volume of 100 or 200 uL. I hope that the M ethod 1668B incorporates these changes to reflect some amount of flexibility in the weight of the samples used for extraction and the final extract volume. Response: We believe that the most significant problem the labs encountered with the biosolids

sample was the inability to separate the PCB congeners from the biosolids matrix using the extraction and cleanup procedures in the method. In part, this problem is likely caused by overload of the cleanup columns. The study

- specific instructions directed the laboratories to determine the percent solids content of the biosolids samples before any other steps, and then to use an aliquot of wet sample sufficient to yield about 10 g of dry biosolid material. These instructions were consistent with the instructions in Section 11.5.1 of the method and were based on EPA’s experience during the 2001 National Sewage Sludge Survey. The use of a consistent dry

weight aliquot across all of the participating laboratories simplified the comparison of results across lab oratories and between samples. Section 11.5.1 of the method instructs the laboratory to weigh out “a well

- mixed aliquot of each sample (of the same matrix type) sufficient to provide 10 g of dry solids,” but Section 11 of the method also states that “ For samples known or expected to contain high levels of the CBs, the smallest sample size representative of the entire sample should be used.” Regarding concentration of the sample extract to 20 μL, the addition of an n -nonane “keeper” in Section 12.7.7 of Method 1668A reduces the possibility of loss when the extract is concentrated to 20 μL. Moreover, EPA’s experience with Method 1613B, for PCDDs/PCDFs, does not suggest that the HRGC/HRMS laboratory community has difficulty with the 20 μL final extract volume. Finally, any option for changes to the final extract volume would necessitate other changes in the method to ensure that the final sample extract and the calibration standards contain similar amounts of the labeled compounds used for isotope dilution quantitation. Without clear evidence of a problemMethod 1668A Interlab Study Peer Review

April 2010 29

with the current final extract volume, EPA does not believe that such changes are warranted at this time

. Comment: g. Page 11, First paragraph, line 9: Expand “ML” Response: We have spelled out “minimum level of quantitation” in the Report. Comment: h. Table 4

- 1, Page 12, column #7 (Mean), third row from the bottom; 241

- ???

Something is missing here. It looks like a typo. Is there a value after ‘241’? Al so, footnote 1 for the table: specify whether biosolid and fish data are given on wet weight or dry weight basis. For footnote 2; insert “sum of” between ‘represent’ and ‘all’. Response: We have made your suggested changes to the Report. Comment: i . Section 4.3.; Were the blanks analyzed similar to samples in all aspects, especially

the concentration factor (final extract volume)? This should be mentioned in section 4.3. Response: Yes, to the extent that the matrices could be simulated. For the soil/tissue blank, a

mixture of playground sand and corn oil was used to simulate soil and tissue. This mixture was chosen to minimize the number of blanks that would need to be analyzed. The method requires that blanks and other QC samples be treated in the same manner as field samples, including concentration of extracts to 20 μL. There is no allowance in the method for using a different final extract volume. We have added a statement to Section 4.3 of the Report clarifying this information. Com ment: j. Page 13, Table 4

- 2: “# Labs” for sand/oil should be “6”; “10” is a misleading

number here because “10” is the number of datasets available, not ‘# Labs”. Response: We have revised the header to column 2 in Table 4

- 2 to state #Blank samples. Comment: k. Section 4.4: Low recoveries of mono

- and di

- CBs have been reported to be due

to loss during transport or as ‘unknown’. I would expect that it is mainly due to loss of these volatile congeners in the solvent evaporation steps in Method 166 8A. Response: As stated in response to the issue on results under item 1 above, the loss could not have

occurred in the solvent evaporation steps because the mono

- and di

- labeled compounds were recovered in the normal range. We have revised Table 4

- 1 in the Report to show this information and have attached revised Table 4

- 1 at the end of these responses for your use. Comment: l. Section 5.5: “(Table 5

- 1).” Move the ‘.” Out of the parentheses. Response: We could not find a period within parentheses in Table 5

- 1. Comment:Method 1668A Interlab Study Peer Review

April 2010 30

m. Section 6. I am little concerned about the statement that "the result of this interlaboratory study met our objectives to characterize......”. The results have not satisfied the criteria to validate the method for biosolids, if we follow the guidelines that we need dataset from 6 laboratories. Response: This comment is the same as your comment in item one of section one, technical

approach . Comment:
n. Corrections on addendum: Table 4

- 1, footnote Biosolids concentrations in ng/kg,

wet weight, not ‘ng/L’. Please correct. Also, the labeled compound recoveries are too low for a good method. The low recoveries are only for mono and di

- CBs and if these two congeners are removed or reported separately, the recoveries are more tha n 50% for all other congeners. I suggest that recoveries for mono

- and di

- CB congeners are reported separately and tri

- deca CB are reported separately. A 15 or 20% recovery is too low to consider a method to be a good method

. Response: We have made th e correction to the biosolids concentrations.

Regarding low recoveries for the mono

- and di

- CB congeners, we responded to this issue in the section on results under item 1 above. 6. Please provide any additional comments you would like to make regarding the documents under review. Comment:
a. Although 14 laboratories were invited to participate (on a voluntary basis), only 6

labs have provided results for fish and wastewater and 4 labs for biosolids. Participation of a minimum of six labs is required f or a method validation study. It is disappointing to note that several laboratories have expressed dissatisfaction with method and problems in analysis. Inviting some laboratories for participation with some (minimum) payment, and use of NIST SRM sample s for accuracy analysis would enhance the quality of method validation. Because the method is intended for use in EPA’s Clean Water Act programs, I would strongly recommend that contract laboratories analyzing clean water act samples should participate in this validation study. Response: As suggested, we have obtained data, including MDLs, from contract laboratories

experienced with this method. This data is in the current version of this method, 1668C. We answered your comment about the number of labs and use of SRMs in response to your first two comments in section 1 of this review. Again, the final data sets of 6 , 6 and 4, respectively, for the t

hree sample types, water, tissue and biosolids, resulted from review of laboratory results against our data validation guidelines and from admissions and narratives from the labs. Comment: b. I would suggest that mono and di

- CBs are reported separately or removed from

the method. The QC criteria for these two congener groups should be reported separately from the other congeners. Response: As explained in the section on results under item 1 above, we believe that the low

recoveries for wastewater may be the result of losses during sample transport and not the result of a method deficiency.Method 1668A Interlab Study Peer Review

April 2010 31 Comment:
c. Participating laboratories should report twelve toxic congeners separately and

QC criteria for these congeners should be established; in other words, the method validation is needed for these 12 toxic congeners. Response: There is no prohibition in Method 1668 to reporting results for the toxic congeners

separately. And QC acceptance criteria were given for the toxic congeners, as shown in Table 5

- 1 of the Report and Table A

- 1 of the Addendum Comment: d. A major issue with biosolids analysis lies in the use of 30 g of sample weight for

extraction. Similarly, concentration of samples to a final volume of 20 uL, can impose serious variability issues. I would recommend that the method be amended to incorporate some flexibility in the weight of the sample s taken for extraction. For biosolids, it can be from 10 to 30 g (wet weight). Also, concentration of extract to 20 uL, will introduce large variability. I would suggest that the final volume can be from 20 uL to 200 uL. I would recommend using sediments for method validation. Response: As noted in response to earlier comments, the method does not require the use of a 30

- g

sample. Section 11.5.1 of the method instructs the laboratory to weigh out “a well

- mixed aliquot of each sa mple (of the same matrix type) sufficient to provide 10 g of dry solids,” Section 11 of the method also states that “ For samples known or expected to contain high levels of the CBs, the smallest sample size representative of the entire sample should be used.” Regarding concentration of an extract to a final volume of 20 to 200 uL, a lab may concentrate to a higher volume under the equivalency provisions in Section 9 of Method 1668 and under 40 CFR 136.6, provided that all QC requirements in the method are met. However, and as noted in an earlier response, any option for changes to the final extract volume would necessitate other changes in the method to ensure that the final sample extract and the calibration standards contain similar amounts of the labeled compounds used for isotope dilution quantitation. Without clear evidence of a problem with the current final extract volume, EPA does not believe that such changes are warranted at this time. Regarding use of sediments for method validation, EPA may use sediments in future validation studies depending on program needs and requirements. However, at the time of the validation study, EPA felt that biosolids were a more likely candidate for PCB congener regulations than sediments. Revised tables 4

- 3 and 4

- 4, described in EPA’s responses above, are shown below for your convenience.Method 1668A Interlab Study Peer Review

April 2010 32 Table 4 - 3 Wastewater Sample Recovery and Precision b y Level of Chlorination

LOC Percent Recovery (% ) Within - pair Relative Standard Deviations (% ) # Results Mean Median Min. Max. # Pairs Pooled* Median Min. Max. Native congeners spiked by sample prep lab 1 25 3.15 2.71 0.49 11.8 11 29.7 17.5 5.8 80.8 2 118 54.2 44.6 2.63 162 57 12.2 4.42 0.17 62.4 3 223 89.5 82.8 34.2 164 111 7.62 5.06 0.02 24 4 356 95.6 91.4 38.1 201 178 7.36 6.16 0 20.6 5 344 81.4 72.2 30.6 182 170 10.8 7.99 0.06 40.5 6 362 75.3 68.8 8.14 196 178 12.1 8.64 0.16 46.8 7 235 72.3 64.4 10.4 155 114 9.63 5.24 0.14 39.3 8 116 68 59.3 18.1 135 56 11.3 7.26 0.12 32.3 9 35 70.8 57. 8 44.4 126 17 8.91 6.66 0.57 18.5 10 12 70 59.1 49.3 118 6 5.67 4.23 0.37 9.05 Labeled congeners spiked by participant labs 1 24 51.4 48.9 21.0 84 12 19.7 13.8 3.45 42.9 2 24 58.5 55.1 25.0 90 12 15.7 11.9 0 29.2 3 36 67.4 62.6 26.0 108 18 12.5 5.32 0.516 33.3 4 33 60.5 57.5 35.0 101 15 8.24 3.34 0.873 17.7 5 83 77.2 81.0 41.0 110 41 11.0 2.61 0 28.7 6 50 75.6 74.3 38.6 106 25 9.83 7.13 1.32 21.8 7 36 76.9 77.0 5.00 123 18 30.4 3.72 0 126 8 23 76.6 79.5 38.4 94 11 6.20 5.12 2.29 12.4 9 24 71.2 7 0.0 49.1 98 12 6.33 4.24 0 9.92 10 12 73.1 74.5 52.8 98.2 6 6.81 5.26 4.07 12.0 * Pooled RSD calculated as the square root of the mean of the squared within

-pair RSDsMethod 1668A Peer Review

October 2007 33 Table 4 -4 Recovery and Precision of Labeled Compounds Spiked into Samples 1

LOC 2 L abeled

Congener3 Recovery (% ) Within - pair RSD (% ) # Results Mean Median Min. Max. # Pairs Pooled 4 Median Min. Max. 1 1L 32 52.9 48.5 13.0 95.0 16 22.0 17.4 0.81 43.4 1 3L 32 60.4 61.1 15.9 107 16 18.9 10.9 2.70 47.6 2 4L 32 60.8 56.1 32.0 120 16 16.7 11.9 1.30 31.6 2 15L 32 64.8 69.0 25.0 96.3 16 13.7 2.8 0 31.4 3 19L 32 59.0 59.0 4.5 112 16 12.0 5.4 0.69 26.5 3 28L 32 78.8 87.8 25.0 118 16 11.7 7.8 1.40 33.3 3 37L 32 77.2 77.6 35.0 110 16 10.0 2.4 0.52 23.6 4 54L 31 61.6 59.1 25.5 109 15 10.3 5.4 0.87 22.0 4 77L 31 72.7 67.3 43.0 106 15 9.1 5.7 1.10 19.8 4 81L 31 74.9 71.0 26.0 129 15 9.0 6.6 1.30 18.8 5 104L 32 76.5 80.0 41.0 102 16 10.1 4.8 0 24.4 5 105L 31 81.9 86.3 52.7 103 15 10.9 5.2 0.50 25.2 5 111L 29 85.2 86.3 63.0 110 13 5. 4 3.7 0.29 12.7 5 114L 32 83.0 86.3 48.0 119 16 10.8 5.2 1.00 28.3 5 118L 32 83.9 87.5 53.8 120 16 10.8 6.4 0.36 26.6 5 123L 32 84.7 88.8 53.8 116 16 11.4 6.5 0.48 23.4 5 126L 32 81.0 80.8 59.0 123 16 11.8 4.5 0 28.7 6 155L 32 77.3 81.3 33.0 106 16 7. 8 3.6 0 21.8 6 156L+157L 32 87.8 76.5 44.0 216 16 12.9 8.0 1.10 30.3 6 167L 31 84.3 82.0 57.3 110 15 9.9 8.6 1.30 22.6 6 169L 32 78.3 75.3 30.5 112 16 14.8 8.0 2.40 37.5 7 178L 30 89.1 93.8 5.0 126 14 34.6 4.7 0.69 126 7 188L 32 78.0 85.2 32.0 136 16 8.5 4.6 0.22 17.4 7 189L 32 82.8 82.0 54.0 118 16 8.1 4.8 0 18.2 8 202L 31 84.9 87.8 38.4 145 15 9.2 5.4 1.30 19.9 8 205L 32 82.3 81.5 51.0 118 16 7.9 4.8 0.25 17.3 9 206L 32 80.8 81.3 50.0 115 16 9.2 7.7 0 19.8Method 1668A Interlab Study Peer Review

April 2010 34 9 208L 32 80.4 84.7 49.0 129 16 7.6 4.9 0 17.2 10 209L 32 78.7 79.8 47.7 110 16 9.2 6.3 2.20 18.9 1 Wastewater, biosolids, and tissue 2 Level of chlorination 3 Labeled analog of World Health Organization dioxin

- like (Toxic) congener shown in bold 4 Pooled RSD calculated as the square root of the mean of the squared within

-pair RSDsMethod 1668A Interlab Study Peer Review

April 2010 35 Section Three: Original Peer Review Comments on Draft EPA Method 1668A Interlaboratory

Validation Study

Prepared for:

U.S. Environmental Protection Agency

Office of Water Office of Science and Technology Healt h and Ecological Criteria Division 1301 Constitution Ave., N.W. Washington, D.C. 20004

First Two Sets of Comments Prepared by:

Versar, Inc. 6850 Versar Center Springfield, Virginia 22151 September 2007
Third Set of Comments Prepared by: Eastern Rese arch Group, Inc. 110 Hartwell Avenue. Lexington, MA 02421 March 2010.Method 1668A Interlab Study Peer Review

April 2010 36 PEER REVIEW COMMENTS FROM MICHAEL C. NEWMAN Peer Review Comments on: Draft EPA Method 1668A Interlaboratory Validation Study NAME: Michael C. Newman, PhD AFFILIAT ION: 121 York Point Dr., Seaford, VA, also Professor at the College of William and Mary, Virginia Institute of Marine Science DATE: Sept. 18, 2007 I. GENERAL IMPRESSIONS Overall, this Report of the Method 1668A Interlaboratory Validation Study is a sufficiently accurate, c lear, and thorough analysis discussing the results of a very difficult task. T he study objectives were to (1) characterize the performance of Method 1668A for multiple laboratories and matrices (wastewater, biosolid, fish tissue), and (2) evaluate and pos sibly revise the QC acceptance criteria. These objectives were met with some qualification. Iti s important to emphasize prior to discussing the specific qualifications that this effort does appear to be generally well

- intended,

-planned and

- interpreted. Some challenges to applying the results do exist. The laboratory participation was disappointing and this was the source of numerous issues. For example, six of 14 laboratories declined to participate in the biosolids analysis. An additional four produced “unusable” numbers, resulting in only four of 14 potential sets of results being applicable to the task at hand. For wastewater and fish tissue, only six laboratories produced the data used in this exercise. F our laboratories successfully analyzed all three sample types and, therefore , have a very strong influence on the results. T he candidate laboratories included government and private organizations which might be expected to have differences in response; however, it is unclear from the report if the f

inal subset was biased toward one group or another. The very high failure

-to -report rate for this process suggests that failure to achieve useful analyses with 1668A warrants more discussion. Was the high failure rate normal for such voluntary validation e xercises or a feature to be anticipated for Method 1668A as currently written? Some laboratories deviated from 1668A for some analyses. Also outlier tests that require extremely thoughtful application were applied to cull away some data. Together, these fe atures do not permit as high confidence in the study’s generalizations as might be desired. T he studies conclusions are not unequivocal at this point. T he resulting, central concern can be articulated with a quote from Stuart’s Basic Ideas of Scientific Sampling (1968): “... the credentials of a sample are not only important for the interpretation of results, but are in principle the only information of value we can have in this respect.” My general conclusion is that the results are adequate for meeting the study objectives but should be augmented in the near future in order to enhance confidence in the general validity of the conclusions. Detailed below are specific examples from within the report and associated recommendations. II. RESPONSE TO CHARGE QUESTIONS 

### 1. Please review the EPA's Report of the Method 1668A Interlaboratory Validation Study for the objectives of the study, the soundness of the technical approach, and the results produced . COMMENT: The Interlaboratory V alidation Study

attempts to characterize the performance of Method 1668A in multiple laboratories and matrices (wastewater, biosolid, fish tissue), and to evaluate and possibl y revise the QC acceptance criteria. As stated above, my o verarching opinion is that the study met these goals but further work is required to enhance confidence in the generality

o f the conclusions. The goals for this complicated method were extremely challenging so the implied incremental generation of sufficient characterization/validationMethod 1668A Interlab Study Peer Review

April 2010 37 

### information is, in my opinion, reasonable. M y suggestion that more work is required is not an oblique inference about the work quality of the study leaders. This was a very difficult task. 2. Please render an opinion as to suitability for use of the data from the study for revision of the QCAC in Method 1668 to make it Method 1668B. COMMENT: Yes. The study data are adequate, although lacking in some ways, for revision of the QC in Method 1668. 3. If you have a suggestion for an improvement, please provide suggested text by section number . COMMENT: A wide range of suggestions and insights will be addressed below regarding the improvement of the study or existing study presentation. Page 6, Top paragraph and first full paragraph : This is the first place in the report that the issue of sample homogeneity emerged. Iti s also discussed in Section 4.1. D escribed here is the process in which a “sample processing laboratory w as required to perform background and homogeneity

analyses of both the biosolids and tissue matrices. ” The text does not convince me that the samples were “homogeneous” in a strict sense relative to the analy tes. F or example, some fatty tissues might contain more analy te than others and similar

- sized pieces of fatty tissues might not be homogeneously di stributed in a well

- mixed homogenate. The biosolid to which sand was added might experience density

- driven sample component segregation (i.e., sinking sand) so that the sample aliquot anal yz ed by the volunteer laboratory might not have been homogeneous (w ell

- mixed?) relative to the analyte. There are very specific methods of determining this sample quality (well

- or poorl y- mixed) yet they are not applied by

the sample processing laboratory . It is not clear how this might or might not have contributed to differences in results among laboratories. This concern is also inherentl y linked to sample aliquot size which 1668 prescribes versus and the actual sample sizes used by the volunteer laboratories. (Ia ssume that the recommendations of sample sizes were ba sed only o n anticipated anal yt e concentrations in ty pical samples and the mechanics of the sample processing.) The point being made above can be illustrated easily u sing the conventional explanation as depicted in the figure to the left. The anal yt e (do t) is homogeneousl y distributed in this figure’s top panel; therefore, sample aliquot size illustrated in the figure as different size boxes will not influence the measured anal yt e concentration in an analyzed aliquot. (Homogeneity is a material quality in which the analy

te is distributed in a completely

h omogeneous manner throughout the material.) In the bottom panel, a heterogeneous material is depicted in which sample aliquot size becomes critical depending on whether the heterogeneous material is well

- or poorly

- mixed. I f sample aliquot sizes are too small for a poorl y- mixed, heterogeneous material (small squares in figure bottom panel), variation among reported anal yt e concentrations will be higher than that derived with aliquot sizes of sufficient s ize (large square in the bottom panel) (Newman 1995). T o conclude that a heterogeneous sample is sufficientl y well

- mixed to permit use of a specified aliquot size (e.g., 20 g of fish tissue) requires some conventional calculations. The homogeneity anal ys es described in this report do not include the associated statistics permitting such decisions. This is especially relevant to the biosolid and tissue samples, and with sufficiently high solids content, also toMethod 1668A Interlab Study Peer Review

April 2010 38 

### wastewater samples. The pertinent approach to this problem and associated statistical computations are described in classic papers by V isman (1969), In gamells (1973) In gamells and Switzer (1974), and Wallace and Kratochil (1987). Page 8, Top Bullets : The laboratories were told to report all “>nm” values as quantified concentrations and the statistical anal ys is of these measured values was treated such that all “>nm” values possessed the same quality o f measurement. It is incorrect to assume that all values above “nm” had the same measurement qu ality ( e.g., Keith et al. (1983)). The relative uncertainty of a measured value can be estimated relative to the standard deviation of the anal ytical process baseline noise (See also figure on page 6), Relati ve Un certainty ( % )  10 0 Z 2 N 

### where Z= Z
- statistic value for a confidence level of % (conventionall y 95%), and N is the measured signal expressed in units of multiples of the standard deviation of the baseline noise, e.g., N=3 and 10 are associated with the signal at the detection limit and limit of quantification, re spectivel y. Interpretation of the study results is very

difficult if this fact is not addressed in the discussions and shown in graphs of results for the various samples and sample ty pes. An example of this will be discussed below relative to the figures in the report. Page 12, First paragraph after bullet list and also last sentence : H ere and elsewhere, my assumption is that the study r esults are intended to provide measurement quality characteristics anticipated for “in

- control” analyses of the specifie d sample ty pes by a laboratory

represented by the initial 14 laboratories. It is critical that the population for which these final QC criteria are generated be clearl y defined given the variations in reporting, method application, and data selected for f

inal QC calculations. Here and elsewhere, Grubbs’ single outlier test (Grubbs 1969) is applied to cull away apparent outliers. O utlier deletion, as discussed in Grubbs’ paper and many o thers, should be done very thoughtfull y according to clear rules. For example, the “outliers” detected based on the assumption of a normal distribution might come from a population more appropriatel y described as lognormal. The “outlier” would be falsel y removed based on the assumption of the wrong distribution. The rul es for handling outliers in Grubbs (1969) and other papers should be considered prior to applyingany outlier tests. The results of such application should then be described in the report. C lassic papers that might be useful are Anscombe (1960) and Rosner (1983). As multiple outliers might be masking each other in these data sets, Rosner’s method (1983) for multiple outliers might be useful. P ersonally , I think that Woodworth’s (2004) classification of outliers and the associated approach based on interquartile ranges is more relevant as it is more open to intelligent application than the above hy pothesis

-based, parametric tests. This point will be repeated later in discussing the absence of quartile (Q1 and Q3) values in tables of summary

statistics. P age 13, Section 4.2.1, line 4 : The “homogeneity

... w as found acceptable...” Respectfull y, my guess would be that this statement is correct; however, I disagree that enough information was extracted to make a definitive statement. Ib ase this disagreement on the material already presented above.Method 1668A Interlab Study Peer Review

April 2010 39 

### Page 14, Table 4
- 1 : H ere and elsewhere, it would be very h elpful to have the first and third quartile values (Q1 and Q3) or, minimall y, the interquartile range (|Q3

- Q1|) in the table. One immediate use would be to apply interquartile range -based methods to explore potential outliers or outlying laboratories. The inclusion of these statistics is highly r ecommended. A s an aside, the differences between means and medians for congeners with 3 to 8 chlorination level s (biosolids and tissues) suggest a skewed sample set. Are these data skewed for any reason or is this simply

a spurious trend in the tabulated data? Page 29, Last line of text : Igenerally agree but feel that a less definitive, and more conditional, conc lusion can be reached. Appendix A. Section 1.0 ., 95% Confidence Intervals : The section begins by describing conventional, parametric calculations for all congeners in all matrices at all concentrations. Was the assumption of normality tested and met, or found to be unimportant? Some discussion seems warranted. Appendix A Section 1.0, Application of Grubbs’ outlier test : Again, outlier tests should be used very c arefully and interpretation of any consequent data sets based on truncated distribution statistics (see Grubbs (1969) and other sources for discussion). As a simple example, what is the n in the equation given in Section 2.0? This is not as easy aquestion to answer as might be suspected since some observations were thrown out. A lso, was the as sumption of a normal distribution (upon which Grubbs’ test depends) appropriate? F inally , what Ty pe I error rate was selected? Likel y, the ty

pical 0.05 was used yet one could argue for a more conservative error rate. 4. If you believe that a given item should be clarified, please provide the clarification by section number or define the clarification needed to a sufficient level of detail that we can make the necessary changes. COMMENT: A wide range of issues will be addressed below as suggestions for clarifying presentation. Section 1.2, Line 3 : Change “seen” to “seven”? Page 4, Paragraph 2 : Please consider replacing “properl y homogenized” with “homogenized.” The reasons are given above for removing the vague word from this sentence. One could easil y argue that the samples were not shown to be “properly homogenized.” Page 5, Paragraph 2 of Section 2.4 : The term, “Youden pairs” is first used here but not defined until Section 2.5.1. Please move the definition up to the first use of the term. Table 2
- 1 (page 4) and Table 3
- 1 (page 12 ) : Iti s extremely

important in judging the generalizability

o f the report conclusions that the laboratory p articipation in generating the final QC criteria is very

c learly discussed. At present, it is not easy to follow. Generally , one starts with 14 laboratories (government and private) and ends up with data from 4 to 6 labs. It isn’t eas y to judge how (1) withdrawal of the various laboratories from providing results, (2) the study author’s judgment about reported data us efulness, and (3) the final data inclusion influences the statistical population of laboratories for which inferences about QC are being made. S everal issues to be addressed include the following. Are the QC conclusions relevant to only the best orMethod 1668A Interlab Study Peer Review

April 2010 40 

### best equipped laboratories? Are they representative of primarily g overnment laboratories, private laboratories, or both? A very clear and concise summary addressing this linchpin issue is needed. In developing this summary , discussion of the influence of outli er rejection is also warranted. Page 8, Section 2.7, first paragraph : The requirements were given for calibration curves but the laboratories interpreted the requirements differentl y. I saw no obvious reason for the misinterpretation; however, some thought s hould be put into rewriting the associated text so it is even clearer. A lso, relative to Section 2.7.1. and 2.7.2, were the consequences of any of the various deviations quantifiable? For example, the differences due to different aliquot weights used b y the various laboratories might be quantified with the methods cited above for “ Page 6, Top paragraph and first full paragraph. ” Page 9, Section 2.7.2
. Different problems were encountered by different laboratories. Different weight normalizations were do ne by the different laboratories. It isn’t clear to me how this directly influenced statistical estimates and associated inferences. Also, the numerous deviations suggest that even more clarity needs to be placed into 1668A. P erhaps more discussion is ne eded about what deviations should not be considered. Page 11, Section 3, first paragraph : This section begins by d escribing three laboratories that did not submit data. It is important to know if non - reporters are considered a biased or random subset of the population of possible responders. W as there a risk of biased sampling of possible laboratories using this method as a result of biased non

- response? Page 11, Section 3, fourth bullet : How was this issue handled during the anal ys is of results? Would omission of an outlier laboratory create a biased sample of laboratories? Page 15, Table 4
- 2 and all other tables : Univariate statistics are applied throughout these tables. Some graphs suggest skewed distributions for some data. W as the assumption of normality o r the robustness of conclusions to violations of this assumption assessed? Page 16, first lines : This discussion is too speculative in my opinion. Page 17, Figure 4
- 1 (and also 4
- 3, 4
- 5, and 4
- 7) : As dictated by the generally

- accepted equatio n given above, the relative uncertainty of a measurement will rapidly increase as concentration decreases. The following figure gives the general shape of such a relationship. Depending on the positions of the detection limit (LOD in this figure) and limi t of quantitation (LOQ in this figure) (from Newman (1995)), the variation in the y direction as a function of measured concentration would be totally expected based on this relationship. W ould it be possible to insert vertical lines at the LOD and LOQ in these figures so a reader could judge this issue? As a good example of how this would help, it is difficult to decide if the explanation for the Figure 4
- 5 outliers (Page 20, first few lines) is reasonable or whether the variations at low concentrations ( in lower part of the Region of Less
- Certain Quantitation?) are what you would expect for any measurement process.Method 1668A Interlab Study Peer Review

April 2010 41 

### Page 18, Figure 4
- 2 (and also 4.4, 4.6): These figures do not convey any u seful information. Relative Standard Deviations (e.g., 4
- 3) are much more useful for conveying the associated information (that variation can change with concentration). Page 21, Table 4

- 4 : A simple mistake was made here such that the “# pairs” column has two digits after the decimal point, e.g., 32.00 pairs. Page 29 : The conventional “This page intentionally blank” statement is made but, in fact, there is text on the page. 5. P lease point out errors or omissions. COMMENT: Please see comments above. III. SPECIFIC OBSERVATIO NS [NONE SUBMITTED] IV. ADDITIONAL REF ERENCE S Anscombe, F.J. 1960. Rejection of outliers . Technometrics 2: 123

- 147. Grubbs, F.E. 1969. Procedures for detecting outlying observations in samples. Technometrics 11: 1

- 21. In gamells, C.O. 1974. New approaches to geochemical analysis and sampling. Talanta 21: 141
- 155. In gamells, C.O. and P. Switzer. 1973. A proposed sampling constant for use in geochemical analysis. Talanta 20: 547

- 568. Keith, L.H., W. Crummett, J. Deegan, Jr., R.A. Libby, J.K. Taylor, and G. Wentler. 1983. Principles of environmental analysis. Anal. Chem. 55: 2210

- 2218. Newman, M.C. 1995. Quantitative Methods in Aquatic Ecotoxicology. Lewis/CRC Publishers, Boca Raton, FL, pp. 426. Rosner, B. 1983. Percentage points for a generalized ESD many

- outlier procedure. Technometrics 25: 165
- 172. S tuart, A. 1968. Basic Ideas of Scientific Sampling, Hafner Publishing Co., New York, pp. 99. Visman, J. 1969. A general sampling theory. Mat. Res. Stand. 9:9

- 64. Wallace, D. and B. Kratochvil. 1987. Visman equations in the design of sampling plans for chemical analysis of segregated bulk materials. Anal. Chem. 59: 226

- 232. Woodworth, G.G. 2004. Biostatistics. A Bayesian Introduction. John Wiley & Sons/Method 1668A Interlab Study Peer Review

April 2010 42 PEER REVIEW COMMENTS FROM

CHARLES WONG Peer Review Comments on: Draft EPA Method 1668A Interlaboratory Validation Study NAME: Charles S. Wong, Ph.D. AFFILIATION: University of Alberta, Department of Chemistry DATE: 18 September 2007

I. GENERAL IMPRESSIONS This document is a synopsis of the interlaboratory validation study intended to improve EPA method 1668A, a protocol for congener

- specific measurement of polychlorinated biphenyls (PCBs) in water, soil, sediment, biosolids, and tissue by high resolution gas chromatography/high resolution mass spectrometry. Fourteen laboratories were invited to participate, and samples as well as instructions and required reporting information was provided to them. Reported data was compiled and summarized. The document provides a useful but succinct summary of the interlaboratory validation’s very large dataset. Suff

icient detail is provided on the background on Method 1668A and the methodology used to invite participating laboratories to ascertain the effectiveness of the validation study in characterizing measurement of PCBs in the samples chosen. Methods for selection of samples for the laboratories are clear and reasonable, including explicit discussion of limitations for logistical and technical purposes. A clear and detailed discussion of deviations from the required protocol is evident, so that the reader can clearly understand limitations in the reported data. For the most part, the discussion of the usable data is clear and pertinent to the intended goal of determining possible changes to the protocol and to acceptance criteria. 1. Please review the EPA’s Report of the Method 1668 Interlaboratory Validation Study for the objectives of the study, the soundness of the technical approach, and the results produced. COMMENT: The objectives of the study are sound. Should Method 1668A be revised, and if so, what protocol changes and/or acceptance criteria should be modified? This can only be done with an interlaboratory validation of a standard set of samples. Sample selection is reasonable in my opinion. Biosolids and tissues were prepared by EPA at two distinct concentrations, properly homogenized, and shipped to each laboratory with appropriate safeguards against decompositions to ensure that that the integrity of the samples upon receipt by the participating laboratory was maintained. While it would have b een ideal to have wastewater samples with PCBs shipped to the laboratories, it is reasonable to assume that testing of Method 1668 capabilities would not be feasible. Thus, providing instructions to spike the wastewater effluent shipped to the laboratory was reasonable. As previously stated, deviations from the method were characterized in a succinct but meaningful manner. This allowed the flagging of suspect data as unusable for further purposes. Many of the deviations noted had details on the underlying rationale, e.g., problems with cleanup of samples encountered by some of the laboratories. However, not all such deviations were detailed in this manner, which I would have found useful. It is troubling that there were sufficient problems in sample pro cessing, measurement, and even data reporting (e.g., no calibration reporting from one laboratory???) that less than half the data from the participating laboratories was usable (Table 3

- 1), as part of the point of having a large number ofMethod 1668A Interlab Study Peer Review

April 2010 43 independent participants was to assess the robustness of both the overall Method as well as the validation for improving the Method. That having been said, I feel that the most problematic deviations were flagged unusable, and that the usable data is sufficient for the intended purpose. Itwould have been useful if the laboratories were asked to provide measurement data on QC Check samples (Method 1668A, Section 7.15) from standardized reference materials (e.g., NIST SRMs or Canadian NRC SRMs) in addition to the other QC data, e.g., OPR samples in reagent water, and blanks. This is important because as noted in the report, PCB concentrations in the shipped samples were unknown, and having this check would be useful. If this was actually done, this was not clear from the report. The reported results provided a good summary of the variability observed with PCB measurement in the various matrices supplied to the participating laboratories. This information is most useful for resetting IPR/OPR guidelines. The summary and recommendations appear reasonable, e.g., using average relative responses under specific circumstances, noting poorer and more variable recoveries for lower chlorinated congeners, and somewhat tighter recovery windows particularly for heavier congeners. 2. Please render an opinion as to suitability for use of the data from the study for revision of the QCAC in Method 1668 to make it Method 1668B. COMMENT: I feel that this data, despite limitations noted above and in the report, are of sufficient quality to revise the method for the next version of this Method. 3. If you have a suggestion for an improvement, please provide suggested text by section number. COMMENT: In Method 1668, Section 23.0, page 61: It is useful to provide the PCB numbering system use d, to avoid confusion. I realize that CAS registry numbers are provided, which eliminate ambiguity; however, most discussion of PCBs is by the numbering system, so having that clarified explicitly in the document is very useful. See comments below on ref erences. 4. If you believe that a given item should be clarified, please provide the clarification by section num ber of define the clarification needed to a sufficient level of detail that we can make the necessary changes. COMMENT: It would be useful to provide more details on the instructions to the participating laboratories as how to spike the shipped wastewaters with PCBs. Also, as stated previously, additional QC data from the laboratories should be given to help ensure that the usable data was of high quality. 5. Please point out errors or omissions: COMMENT: In the document “Changes to be made to EPA Method 1668A from the 8/20/03 revision based on the Method 1668A interlaboratory validation study and comments and corrections received from laborato ries”: Under Quality control acceptance criteria, Table 5

- 1 from the interlaboratory report should replace Table 6 not Table 5 in Method 1668A. In Method 1668A, Section 13.5.1.4, page 48, typo: should read “If calibration is not verified” not “If calibration I s not verified”. In Method 1668A, Section 22.0, page 59: the title for reference 4 should be provided for consistency. In Method 1668A, Section 22.0, page 60: the title for reference 18 should be provided for consistency.Method 1668A Interlab Study Peer Review

April 2010 44 In Method 1668A, Appendix A, Section 2.1, page 110: typo, should be “individual congeners” not “individual congene4rs”. III. SPECIFIC OBSERVATIONS These comments refer to the report on the interlaboratory study. Section 4.2, page 14, Table 4

- 1. I do not feel that reporting of concentrations to up to 5 significant figures is justifiable. The GC used must have either a splitless or on

- column injector (Method 1668A, Section 6.9). The latter in particular does not have sufficient precision to justify 5 significant figures. Three or perhaps four significant figures is the maximum that should be present. Section 4.2, page 14, Table 4

- 1. In addition, while providing a mean, median, and maximum for concentrations of usable data is useful, it is difficult to assess the distribution of data. This is important, as the fairly large differences between means and medians suggest skewed data distributions, which are suggested by the various figures of variability. It would be useful to provide a box plot of this data to make assessment m

uch easier for the reader. Section 4.6, page 21, Table 4

- 4. The same comments regarding significant figures from my comments on Section 4.2, page 14, Table 4

- 1 hold here as well. Inclusion of the interlaboratory report’s Appendix A, Statistical Procedures Used to Develop QC Acceptance Criteria” in the Method as an Appendix may be useful. The following comment is suggested for improvement of Method 1688A, and not the interlaboratory report: In Section 5.0 on Safety, pages 5

- 7, some explicit mention shoul d be made about the use of appropriate biohazard containment for samples which are biosolids, as these are likely to host pathogens which are a potential danger until rendered harmless by disinfection. This would include preparation of raw samples in a bi osafety cabinet until pathogens are killed (e.g., by soaking in solvents), appropriate safety garments and emergency procedures for accidental spills of pathogenic material, and warning about covering all materials containing solvents in biosafety cabinets to prevent solvent fumes that can pose an explosion hazard in biosafety cabinets that do not vent to exterior air. IV. ADDITIONAL REFERENCES Mills, S.A., III; Thal, D. A., Barney J. (2007) “A summary of the 209 PCB congener nomenclature”, Chemosphere 60, 1603
- 1612. #####Method 1668A Peer Review

October 2007 45 PEER REVIEW COMMENTS FROM KURUNTHACHALAM KANNAN Peer Review Comments on: Draft EPA Method 1668A Interlaboratory Validation Study NAME: Kurunthachalam Kannan, Ph.D . AFFILIATION: Wadsworth Center, New York State Department of Health & P rofessor, Department of Environmental Health Sciences. State University of New York at Albany. Empire State Plaza PO Box 509. Albany, NY 12201 - 0509 . DATE: March 24, 2010
Review of “Method 1668A Interlaboratory Validation Study Report” and “ Addendum to th e Method 1668A Interlaboratory Validation Study Report Draft, August 2009 . Method validation is an integral part of a good analytical laboratory practice. The method 1668A validation study was intende d to provide the Environmental Protection Agency (EPA) with sufficient amount of data to evaluate the performance of the method. The method 1668A was originally validated in 2000, in a single laboratory study. However, the guidelines published by the EPA and American Society for Testing and Materials (ASTM) recommend a minimum of six complete datasets for validating a method. The objectives of the validation study were to characterize the performance of the Method 1668A in multiple laboratories and matrices and to evaluate and revise the Quality Control ( QC ) acceptance criteria. T he objectives of the study are sound and logical. T he multi

- laboratory validation study is a requirement to meet the national and international criteria to ensure reliability, consistency , and accuracy of the analytical data generated using the Method, especially when the method is used in the EPA’s Clean Water Act programs. The major outcome of the validation study wa s a revision to the quality control/quality acceptance (QAQC) criteria in EPA Method 1668A

.

Itis agreed that me thod validation studies are time and resource consuming and EPA’s effort to perform this validation is highly commendable. Coordination of activities among several participating, volunteer, laboratories is a daunting task. EPA’s efforts on this method validation are significant; nevertheless, data generated from this study are under

-utilized and with the existing data it is possible to generate additional information required for validation.Method 1668A Interlab Study Peer Review

April 2010 46 1. Please review EPA's Report of the Method 1668A Interlabo ratory Validation Study and its addendum. Comment on the objectives of the study, the soundness of the technical approach, and the results produced.

T he o bjectives of the study are sound and logical. T he multi

- laboratory validation study is a requirement to meet the national and international criteria to ensure reliability, consistency , and accuracy of the analytical data generated using the Method, especially when the method is used in the EPA’s Clean Water Act programs. The major outcome of the validation study wa s a revision to the quality control/quality acceptance (QAQC) criteria in EPA Method 1668A

. The overall technical approach , ranging from selection of laboratories, sample selection, sample analysis, data reporting, data review , and validation , is good , although there are some potential areas for improvements. A major drawback with the report is on the biosolids dataset. The objective of the study wa s to have 6 complete datasets, as per the EPA/ASTM guidelines, to validate the method. Only 4 complete datasets have been available for biosolids (Table 4

- 1), although validation for biosolids appears to be an important goal for the method, as indicated on the title. Furthermore, the data submitted by the 4 laboratories for biosolids are highly variable and no ‘ in

- depth ’ statistical analysis of data set has been made to determine accuracy and precision of the data submitted for biosolids . In other words , the M ethod has not adequately met the objectives to validate the analysis of PCBs in biosolids. Many international organizations such as the International Organization for Standardization ( ISO) require at least eight com plete datasets to validate the methods. Because the dataset available for biosolids does not meet the EPA/ASTM guidelines, validation of the Method 1668A for biosolids is incomplete/inadequate

.
The second major concern with the validation study results is that the data analysis/review is focused heavily (or only) on precision , by measuring variability in concentrations reported by the laboratories for PCB congeners. Little attention has been paid to the accuracy of the results. Only for wastewater samples, recovery values are presented as a measure of accuracy (only for wastewater , the spiked concentrations were known and therefore the recovery could be measured). Accuracy is an important criterion in method validation. There are several ways to measure accuracy, without increasing the burden on the laboratories. Accuracy could have been assessed by analyzing a sample of known concentration, as a laboratory control material, and comparing the measured value to the true value. National Institute of Standards and Technology (NIST) reference materials are available for PCBs and laboratories could have been supplied or asked to analyze NIST reference materials , along with samples supplied by the EPA . It has been stated that ‘ to minimize the burden on volunteer participants ’ , laboratories were not required to analyze initial precision and recovery ( IPR) samples. Nevertheless,Method 1668A Interlab Study Peer Review

April 2010 47 accura cy is an important parameter for validating accuracy of a method, a well

- characterized matrix such as NIST SRMs can be used . Alternatively, the technique of standard additions, which is used to determine recovery of spiked congeners from on

- going precisio n and recovery (OPR) samples (sand/oil and water) could have been used (Results from Table 4

- 4) to validate the accuracy of the method.

The third issue on results : T he method 1668A is intended for congener specific analysis of PCBs with an emphasis on toxic congeners (twelve non

- and mono

- ortho substituted PCBs)

. The scope of the method 1668A indicates that twelve PCB congeners designated as ‘toxic’ by the World Health Organization (WHO) are determined. Nevertheless, the validation study is focused on ten homologue groups. The data analyses involved summation of all isomer data within a congener group. Precision and accuracy of the Method 1668A for the analysis of individual toxic congeners, are not validated. In other words, the method is not validated for 12 toxic PCB congeners. Furthermore, the suitability of this method for the analysis of mono

- and di

- chlorobiphenyls (CBs) is questionable for the reasons given below. First, the precision and accuracy for mono

- and di

- CBs are very poor, if not dismal. For example, mean recovery of mono

- and di

- CBs in wastewater samples were 3 and 54%, respectively. Furthermore, the concentration mono

- CB found in wastewater sample is similar to what was found in blanks (mean for wastewater is 27 pg/L [Table 4

- 1 ] and that for water blank is 26 pg/L [Table 4

- 2]). Obviously, the QC criteria have been relaxed to reflect the poor recoveries (Addendum to the method validation study). Overall, the method did not perform well for mono

- and di

- CBs. If these two congen er groups are removed, then the QC criteria for the method would be tighter and more reliable/reproducible. These two congeners are easily lost in the solvent evaporation steps, as has been stated in several places of the method validation report.

A few other issues on the results , such as robustness, ruggedness, and limit of detection have not been discussed or analyzed in detail. A method validation would require that the above mentioned parameters are tested and validated. Some of the information presented on the validation report is not clear ( e.g., section 4.2). For example, how were the outliers detected? How many outliers were found in samples? Rationale for removal of the outliers from dataset is missing. A column showing number of outlier values should be presented o n Table 4

- 1. The reproducibility variation coefficients (between

- laboratory precision and within

- laboratory precision) for the method were not reported. Overall , data have not been fully utilized to describe several parameters that are required in validating a method (based on the comment 2 above and this comment)

.Method 1668A Interlab Study Peer Review

April 2010 48 2. Please render an opinion as to suitability for use of the data from the study and the addendum for revision of the QCAC in Method 1668 to make it Method 1668B. I believe that considerable effort has been put forward in validating the method 1668A. Nevertheless, the data from the validation study are under

-utilized and more important validation parameters such as accuracy of the method, reproducibility, ruggedn ess, etc should be evaluated. These issues and corrections should be corrected/clarified. Once these revisions are made, the method may be revised to 1668B. However, I believe that there is lot more work to be done.

3. If you have a suggestion for an improvement of the interlaboratory study report or addendum, please provide suggested text by section number & 4. If you believe that a given item should be clarified, please provide the clarification by section num ber or define the clarification needed to a sufficient level of detail that we can make the necessary changes & 5. Please point out errors or omissions. I feel that the charge questions 3, 4 and 5 are inter

-related. The following items would help improve the study report and addendum
a. Page 3, Section 2.4 (Sample Selection): Some description of samples is needed. For example, does the fish tissue represent wholebody homogenate or fillet? Also, please specify when these fish samples were collected? I assume that these are wet sample s (i.e., not freeze

- dried). Similarly for biosolids, some sample description is needed. After reading the entire document, I realized that these are wet samples. It would be useful to indicate on page 3 (section 2.4), what biosolids really are and when they were collected; also state that they are wet and if possible indicate the moisture content? b. Did the wastewater effluent samples contain suspended solids? When aliquoting wastewater samples, how was the homogeneity in the distribution of suspended s olids between aliquots ensured? Wastewater samples have been spiked with PCB congeners. It appears that there was no PCB present in wastewater sample prior to spiking. Was that true? I suspect that PCBs can be detected in wastewater samples at several tens to hundreds of pg/L level. The information on the spike concentrations listed on Table 2

- 2 is not clear. What was the solvent that was spiked into water? c. Page 5, Section 2.6: Did all the participating laboratories use HRMS? The Method 1668A call s for the use of HRMS and the method validation requires the use HRMS. The deviations from the Method 1668A have been listed for certain parameters (section 2.7 on page 6), but not sure if all of the laboratories used HRMS for the analysis. Also, on page 5, it has been stated that theMethod 1668A Interlab Study Peer Review

April 2010 49 laboratories were instructed to follow the Method 1668A, it will be helpful to mention/confirm that on Section 3 (Data Review and Validation) that all laboratories indeed followed the Method 1668A (other than the deviations noted on section 2.7). d. Page 6, line 3: Expand “SOW”; the abbreviation is used here for the first time. e. Page 6; Laboratory reporting of rules: One important item that is missing is the conversion of data from ‘wet weight’ to ‘dry weight’ and vice versa ( for biosolids). f. A major issue with the method is lies in the weight of samples used for extraction (especially biosolids) and final extract volume. I think the method should be modified to incorporate some flexibility because the weight used for extraction can vary depending on the concentrations, presence of interferences, etc. Also, concentration to a final volume of 20 uL prior to injection can introduce large variability in analytical results. I would prefer to use a sample weight of 15 g for bioso lids and a final volume of 100 or 200 uL. I hope that the Method 1668B incorporates these changes to reflect some amount of flexibility in the weight of the samples used for extraction and the final extract volume. g. Page 11, First paragraph, line 9: Expand “ML” h. Table 4

- 1, Page 12, column #7 (Mean), third row from the bottom; 241

- ??? Something is missing here. It looks like a typo. Is there a value after ‘241’? Also, footnote 1 for the table: specify whether biosolid and fish data are given on wet weight or dry weight basis. For footnote 2; insert “sum of” between ‘represent’ and ‘all’. i. Section 4.3.; Were the blanks analyzed similar to samples in all aspects, especially the concentration factor (final extract volume)? This should be mentioned in section 4.3. j. Page 13, Table 4

- 2: “# Labs” for sand/oil should be “6”; “10” is a misleading number here because “10” is the number of datasets available, not ‘# Labs”. k. Section 4.4: Low recoveries of mono

- and di

- CBs have been reported to be due to loss during transport or as ‘unknown’. I would expect that it is mainly due to loss of these volatile congeners in the solvent evaporation steps in Method 1668A. l. Section 5.5: “(Table 5

- 1).” Move the ‘.” Out of the parentheses. m. Section 6. I am little concerned a bout the statement that ‘the result of this interlab study met our objectives to characterize......”. The results have not satisfied the criteria to validate the method for biosolids, if we follow the guidelines that we need dataset from 6 laboratories. n. Corre ctions on addendum: Table 4

- 1, footnote’ Biosolids concentrations in ng/kg, wet weight, not ‘ng/L’. Please correct. Also, the labeled compound recoveries are too low for a good method. The low recoveries are only for mono and di

- CBs and if these two co ngeners are removed or reported separately, the recoveries are more than 50% for all other congeners. I suggest that recoveries for mono

- and di

- CB congeners are reported separately and tri

- deca CBMethod 1668A Interlab Study Peer Review

April 2010 50 are reported separately. A 15 or 20% recovery is too low to consider a method to be a good method.

6. Please provide any additional comments you would like to make regarding the documents under review.

a. Although 14 laboratories were invited to participate (on a voluntary basis), only 6 labs have provided re sults for fish and wastewater and 4 labs for biosolids. Participation of a minimum of six labs is required for a method validation study. It is disappointing to note that several laboratories have expressed dissatisfaction with method and problems in analysis. Inviting some laboratories for participation with some (minimum) payment, and use of NIST SRM samples for accuracy analysis would enhance the quality of method validation. Because the method is intended for use in EPA’s Clean Water Act programs, I would strongly recommend that contract laboratories analyzing clean water act samples should participate in this validation study. b. I would suggest that mono and di

- CBs are reported separately or removed from the method. The QC criteria for these two congener groups should be reported separately from the other congeners. c. Participating laboratories should report twelve toxic congeners separately and QC criteria for these congeners should be established; in other words, the method validation is needed for t hese 12 toxic congeners. d. A major issue with biosolids analysis lies in the use of 30 g of sample weight for extracti on. Similarly, concentration of samples to a final volume of 20 uL, can impose serious variability issues. I would recommend that the method be amended to incorporate some flexibility in the weight of the samples taken for extraction. For biosolids, it can be from 10 to 30 g (wet weight). Also, concentration of extract to 20 uL, will introduce large variability. I would suggest that the f inal volume can be from 20 uL to 200 uL. I would recommend using sediments for method validation.