EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

1 

### Verification and Validation of WIPP Data Packages

July 17, 2014 Table of Contents 1.0 Overview
...................................................................... 2 2.0 Verification Summary....................................................... 2 3.0 Validation Summary......................................................... 3 Table 1 Sample Result Qualifiers...................................... 4 4.0 Package

- by

- Package Notes
.................................................. 5 5.0 Observation.................................................................... 9EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

2 1.0 Overview EPA’s National Analytical Radiation Environmental Laboratory received six data packages from the Department of Energy’s (DOE) WIPP Laboratories on May 13, 2014, and performed verification and validation of the americium and plutonium analysis data, which were contained in five of the packages. EPA did not review the gross alpha/beta analysis data. The data came from air particulate samples collected during February and March 2014 at various locations around the WIPP. The packages were for the following sample delivery groups (SDGs): SDG 2014
- 079 18 samples Am and Pu; ( also received gross alpha/beta, not reviewed ) SDG 2014
- 080 11 samples Am and Pu SDG 2014
- 085 11 samples Gross alpha/beta only, not reviewed SDG 2014
- 098 2 samples Am and Pu; ( also received gamma, not reviewed ) SDG 2014
- 101 13 samples Am and Pu SDG 2014
- 108 13 samp les Am and Pu; ( also received U, Sr
- 90 and gamma, not reviewed ) EPA followed the WIPP lab’s own internal standard operating procedure 1 for data verification and validation, double

- checking the operations previously performed by lab personnel. EPA also rev iewed the lab’s calculations and attempted to reproduce calculated analysis results for a few of the sample batches. This report contains the outputs from EPA ’s review. A completed verification checklist for each SDG and a table (spreadsheet) of all the me asurement results with qualifiers can be found in EPA’s WIPP Docket A

- 98
- 49, Item II
- B1
- 28 . The verification identified a number of exceptions, defined as deviations from stated require ments or expectations. EPA ’s review of the calculations also identifi ed apparent flaws in how results are calculated. The calculation errors should have minimal impact on the usability of the data that were reviewed, but for good metrology and to provide the most defensible data possible the lab should make the necessary co rrections
. 2.0 V erification Summary A number of exceptions, defined as deviations from stated requirements or expectations, were noted. These included:
 nine pages were missing from the package for SDG 2014

- 079
 typographical errors in case narratives
 discrepancies in sample collection dates/times between chain

- of

- custody form ( COCs ) , case narratives, raw data, etc.
 inconsistencies in providing spectra and channel by channel printouts
 a few cases where EPA suspects pages should have been marked “VOID” but were not For specific details pl ease see the Package

- by

- Package Notes in 4.0
.

1 Nuclear Waste Partnership, LLC. Radiochemistry Laboratory Data Validation and Verifica tion, Technical Procedure. WP 12
- RL3002, Revision 9. Carlsbad, NM: Effective date November 26 2012.EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

3 3.0 Validation Summary EPA checked the following elements from DOE’s SOPs :
 DOE’s reagent laboratory control samples and dupli cates (RLCS and RLCSD) recoveries between 80% and 120% and within ±3σ limits on control charts o No deficiencies found.
o The RLCS and RLCSD samples in SDG 2014

- 079 were analyzed without blank correction. This seems to represent a deviation from normal practice, but the impact on the results is inconsequential.

 RLCS/RLCSD replicate error ratios (RER) less than 1 o In all the packages only one RER exceeded 1, and only by a small amount. It was duly noted in the DOE package’s case narrative.
o The SOP and pr inted forms indicate that the RER should be less than 2, but since the WIPP lab reports all uncertainties as 2σ, it actually checks that the RER is less than 1. The SOP and forms should be revised to reflect the actual practice.

 T racers and standards not past their expiration dates o No deficiencies found.

 S ample tracer recoveries between 30% and 110% o The tracer recoveri es for several samples in SDG 2014
- 079 exceeded 110%. Although the lab report stated that these samples were “reprocessed,” giving acceptable recoveries, EPA concluded that whatever caused the original high recoveries could still have impacted the reprocessed results, since in each cas e the entire sample had been spiked with tracer and analyzed, leaving no remaining sample for a second independent analysis. EPA assigned a J qualifier (“estimated”) to each of the associated plutonium resu lts
. (See Table 1 for a list of the sample result qualifiers used by EPA. ) Note that all the affected plutonium results were undetectable (less than 2σ).
o The americium analysis on blank sample C7897 also had a high tracer yield, which the WIPP laboratory apparently failed to notice during its internal re view of the data. EPA assigned a J qualifier to the Am

- 241 result for C7897 only.

 R eagent blank results within ±3σ limits on control charts o No deficiencies found.
o NAREL observed that the r eagent blank results were never blank

- corrected, even in batches w here ordinary sample results were blank

- corrected. Presumably this represents the lab’s standard practice, which allows reagent blank results from blank

- corrected and non - blank

- corrected batches to be control charted together.

 N o trends on control charts o No deficiencies found.EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

4 o Although the SOP requires the validator to check for trends on control charts, a trend on the U

- 234 reagent blank control chart had not been noted. EPA found no such mistakes in the americium or plutonium data. Documentation in the packages was insufficient to perform useful checks of instrument calibra tions and quality control. Note that the efficiency calibrations for analyses based on alpha

- particle spectrometry using alpha

- emitting tracers affect only the chemical yields, or tr acer recoveries, not the final results of the analyses. However, instrument background data would have been useful information. The WIPP L ab oratories never assigned a D qualifier ( “detected”) to any americium or plutonium result, although many of the results exceeded 10 times the standard uncertainty of the measurement, making them not only detectable but quantifiable. In two cases (the Pu

- 238 results for samples C7911 and C7912 in SDG 2014
- 080) EPA assigned a D qualifier where the WIPP lab had assigned a U qualifier, “undetectable.” Since the D qualifier had not been used, NAREL assumed the WIPP lab considered any result without a U qualifier to be detected. In cases where the WIPP lab had assigned no explicit qualifier, NAREL assigned a “D,” indicating a detectable result. Table 1: Sample Result Qualifiers D Detected E Exceptions J Estimated; unusually uncertain N Negative R Rejected; unusable U UndetectableEPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

5 4.0 P ackage - by - Package Notes: EPA’s Findings When Reviewing DOE’s SDGs SDG 2014 - 079 8 samples collected February 17 and 18; 10 blanks Verification
 Typo on page 4

– Am
- 241 detected for samples C7892, C7893, and C7896 instead of C7893, C7893, and C7896.
 Report pages 17 through 25 were missing.
 On page 4, C7900 is listed in both batches, SP14

- 03
- 09 and SP14

- 03
- 10.
 On page 3, C7896 client ID is incorrect and is also incorrect on page 12.
 On page 232, C7909 was omitted from the first block of the tabl e.
 Batch SP14

- 03
- 09 did not provide as much alpha spec raw data as SP14

- 03
- 10 such as spectra and channel by channel printouts.
 On page 319, the collection date/time for matrix blanks ( MBLK ) is not the same as listed on page 3. It is correct on page 326.
 P ages 322 and 323 appear to be the same as 324 and 325. There are slight differences in MDC and DLC values. The signatures on 322 and 323 have a later date. Data on pages 322 and 323 were reported, but 324 and 325 were not voided.
 On page 429, the collectio n date/time for MBLK is not the same as listed on page 3.
 On page 430, C7909 was omitted from the first block of the table.
 Some Am raw data have spectra and channel by channel printouts; and others do not. Validation
 Sample result summaries for samples C 7901

– C7909 were missing from the data package.
 EPA detected Am

- 241 in 5 samples, although the case narrative by WIPP Laboratories claimed only 3. Two of the detections occurred in samples whose result summaries were missing from the package.
 Samples C7896

— C7900 originally had high Pu

- 242 yields. It is unclear how they could be “reprocessed” to get valid results. EPA assigned a J qualifier to each of the associated plutonium results.
 Blank sample C7897 also had a high Am

- 243 yield (greater than 110 %), which the lab failed to note during its own data review. EPA assigned a J qualifier to the Am

- 241 result for C7897 only, not to the samples in the batch that were blank

- corrected using C7897.
 RLCS, RLCSD, and reagent blanks ( RBLK ) were obviously calculated with out blank correction. Other samples were blank corrected. In other SDGs it appeared that the RLCS and RLCSD were calculated with blank correction, but the effects of blank correction are so small for spiked samples that they may not be detectable.EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

6 SDG 20 14 - 080 10 samples collected February 15 to 19; 1 blank Verification
 Collection dates/times on page 79 are not the same as those listed on page 3. Page 3 does not compare to the COC on page 53. The low

- volume airborne particulate sample data sheets were not included with this SDG as they were in SDG 2014

- 079; so collection data cannot be verified
 The uncertainty for the Am

- 241 result for sample C7916 is presented on page 15 first as 102 dpm and then as 10.2 dpm . According to the gamma analysis report, the actual uncertainty is 102 dpm.
 Pages 142, 212, and 265 also have discrepancies in collection date/times.
 Page 264 which is the Am Raw Data separation page is in the wrong place in the package. It should be page 212 instead which is where the Am raw data begin.
 On page 68, the Am

- 243 tracer source ID is recorded as SP2

- 177 which is an Am

- 241 standard not an Am
- 243 standard.
 Am and Pu raw data sheets have the same collection date/time discrepancies as mentioned ab ove.
 Inconsistency in providing spectra and channel by channel printouts occurs in this data package as in SDG 2014

- 079. Validation
 No samples in SDG 2014

- 080 were blank

- corrected. However, Am

- 241 and Pu

- 239/240 activities were significantly elevated in a ll the non

- blank samples, making the blank correction essentially irrelevant.
 All samples in the SDG were counted for 60 minutes, although the case narrative states that they were counted for 1000 minutes (the routine procedure). Since all of the non

- blank samples in this group contained elevated levels of Am

- 241 and Pu

- 239/240, the rationale for the shorter counts was clear.
 WIPP Laboratories detectable Pu

- 238 in samples C7910 and C7916 only. EPA also assigned a D qualifier to the Pu

- 238 results for C7911 and C7912 since both those results exceeded 2σ (as well as the DLC).
 WIPP Laboratories Pu
- 239/240 in all the non

- blank samples.
 WIPP Laboratories Am
- 2 41 in all the non

- blank samples.EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

7 SDG 2014 - 098 1 sample collected February 21; 1 blank Verification
 Client ID incorrect on pages 3 and 8.
 Pages 57 and 58 are voided but not pages 62 through 68. Validation
 WIPP Laboratories detected Am

- 241 and Pu

- 239/240 in the non

- blank sample.
 The sample and blank in SDG 2014

- 098 were counted for 60 minute s, although the case narrative states that they were counted for 100 minutes. One sample was a blank, but Pu

- 239/240 was easily detectable in the other sample and the Am

- 241 result was quantifiable (greater than 10 times its standard uncertainty), implying that the 60

- minute count times were adequate. SDG 2014 - 101 11 samples collected March 11; 2 blanks Verification
 There are no Pu

- 238 counts on page 109, yet counts were entered on page 98. Validation
 No issuesEPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

8 SDG 2014 - 108 11 samples collected March 18; 2 blanks Verification
 On page 7, the 4th table should be for U not Pu since only U was recounted on 3/24/2014.
 The collection date/time for the RBLK on page 330 is different than that used on page 360.
 The Pu and Am control chart filter matrix blank activity data sheets and control charts are missing for SP14

- 06
- 02. Therefore the value used on pages 364 and 431 for the “Blank Standard Deviation of the Activity Charts” cannot be verified for the matrix blank values.
 The sample net counts and background counts for Pu

- 239/240 cannot be verified because Pu

- 239 data are not listed on page 368.
 Pages 468 and 473 were in the wrong places and should be switched. Validation
 There was an apparently unnoticed trend on the U

- 234 RBLK control chart, but the uraniu m analyses were outside the scope of EPA ’s review.
 The replicate error ratio (RER) for the Am

- 241 RLCS/RLCSD pair exceeded 1 by a small amount and was duly noted in the case narrative.EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

9 5.0 Observations All of the data are considered usable for decision

- m aking; however, EPA makes the following observations that need to be addressed in order to provide the highest quality, most defensible data possible at all times: 1. It appears that the calculation spreadsheets perform blank corrections using the absolute v alue of the blank count rate (MBLK). Using a measured blank value, positive or negative, for blank correction is common, but converting negative blanks to positive introduces a negative bias into the measurement process. When the blank result is positive, the sample result is reduced, as expected; but when the blank result happens to be negative, the sample result is also reduced by this approach. 2. The calculation spreadsheets appear to double

- correct for both Am

- 243 and Pu

- 242 “abundances” in the tracer regions of interest. Fortunately, both abundances are close to 100%, making the errors small (less than 1%). 3. Detection decisions appear to be implemented incorrectly. The case narrative for 2014

- 079 states that Am

- 241 was detec ted in three samples, while the data show it was detected in five. (Two of the detections occurred in samples whose sample result summaries were missing from the data package.) Pu

- 238 was flagged as “U” (undetected) in samples C7911 and C7912, apparently b ecause the result was less than the minimum detectable concentration (MDC) , although it was greater than the 2σ uncertainty and greater than the decision level concentration ( DLC ) . EPA flagged Pu

- 238 as “D” (detected) in both of these samples. 4. According to its SOP for “D ata R eduction and R eporting ” 2 , the WIPP lab incorrectly propagates the uncertainty of the detector efficiency when calculating the uncertainties of the final results of americium and plutonium analyses when using an alpha

- emittin g tracer as a yield monitor. The efficiency effectively cancels out of the activity equation, so that its uncertainty does not contribute to the total uncertainty of the calculated activity. Note that the efficiency does affect the calculated value of the yield, or tracer recovery, and its uncertainty does contribute to the uncertainty of the yield although it does not contribute to the uncertainty of the analyte activity. Note also that the uncertainty of the efficiency is relatively small and should make a negligible contribution to the total propagated uncertainty even when it is propagated (incorrectly). 5. There is no obvious statistical rationale for the expression used for the uncertainty of the average background correction, as described in WP 12

- RL300 3 (the SOP for data reduction and report ing) 3
. The expression appears to be for a single Poisson background measurement of duration equal to that of the sample count, not for an average of several measurements. If the distribution is truly Poisson, the un certainty of the average should be smaller than calculated; if not, it would seem

2 Nuclear Waste Partnership , LLC . Data Reduction and Reporting, Technical Procedure. WP 12
- RL3003, Revision 12. Carlsbad, NM: Effective date June 19 2013.EPA Air Docket A - 98 - 49, Item II - B1 - 31 ; July 2014

10 more logical to make use of the experimental standard deviation of the multiple measured backgrounds. None of these observations calls into question the usability of the dat a in the packages that were reviewed. Of all the observations, observation 1 seems most significant. It tends to hide small positive results, converting what might sometimes be a barely detectable result into a “non

- detect.” It also tends to increase the f

raction of results that are “too negative” (i.e., more than 2 or 3 sigma below zero). Observa tions 1 and 2 suggest that the calculation spreadsheets should be re

- validated, or replaced by fully validated in

- house software for data reduction. Observation 3 describes a common mistake made by many laboratories and their clients across the U nited S tates . Observations 4 and 5 are offered for the lab’s consideration.