EPA Air Docket A-98-49, Item II-B1-31; July 2014

1

Verification and Validation of WIPP Data Packages

July 17, 2014

Table of Contents

1.0 Overview...................................................................... 2

2.0 Verification Summary....................................................... 2

3.0 Validation Summary......................................................... 3

Table 1 Sample Result Qualifiers...................................... 4

4.0 Package-by-Package Notes.................................................. 5

5.0 Observation.................................................................... 9
EPA Air Docket A-98-49, Item II-B1-31; July 2014

2

1.0 Overview

EPA’s National Analytical Radiation Environmental Laboratory received six data packages from the 

Department of Energy’s (DOE) WIPP Laboratories on May 13, 2014, and performed verification and 

validation of the americium and plutonium analysis data, which were contained in five of the packages. 

EPA did not review the gross alpha/beta analysis data. The data came from air particulate samples 

collected during February and March 2014 at various locations around the WIPP. The packages were 

for the following sample delivery groups (SDGs):

SDG 2014-079 18 samples Am and Pu; (also received gross alpha/beta, not reviewed)

SDG 2014-080 11 samples Am and Pu

SDG 2014-085 11 samples Gross alpha/beta only, not reviewed

SDG 2014-098 2 samples Am and Pu; (also received gamma, not reviewed)

SDG 2014-101 13 samples Am and Pu

SDG 2014-108 13 samples Am and Pu; (also received U, Sr-90 and gamma, not reviewed)

EPA followed the WIPP lab’s own internal standard operating procedure1for data verification and 

validation, double-checking the operations previously performed by lab personnel. EPA also reviewed 

the lab’s calculations and attempted to reproduce calculated analysis results for a few of the sample 

batches. This report contains the outputs from EPA’s review. A completed verification checklist for 

each SDG and a table (spreadsheet) of all the measurement results with qualifiers can be found in 

EPA’s WIPP Docket A-98-49, Item II-B1-28.

The verification identified a number of exceptions, defined as deviations from stated requirements or 

expectations. EPA’s review of the calculations also identified apparent flaws in how results are 

calculated. The calculation errors should have minimal impact on the usability of the data that were 

reviewed, but for good metrology and to provide the most defensible data possible the lab should make 

the necessary corrections.

2.0 Verification Summary

A number of exceptions, defined as deviations from stated requirements or expectations, were noted. 

These included:

 nine pages were missing from the package for SDG 2014-079

 typographical errors in case narratives

 discrepancies in sample collection dates/times between chain-of-custody form (COCs), case 

narratives, raw data, etc.

 inconsistencies in providing spectra and channel by channel printouts

 a few cases where EPA suspects pages should have been marked “VOID” but were not

For specific details please see the Package-by-Package Notes in 4.0.

 

1 Nuclear Waste Partnership, LLC. Radiochemistry Laboratory Data Validation and Verification, Technical Procedure. WP 

12-RL3002, Revision 9. Carlsbad, NM: Effective date November 26 2012.
EPA Air Docket A-98-49, Item II-B1-31; July 2014

3

3.0 Validation Summary

EPA checked the following elements from DOE’s SOPs:

 DOE’s reagent laboratory control samples and duplicates (RLCS and RLCSD) recoveries 

between 80% and 120% and within ±3σ limits on control charts

o No deficiencies found.

o The RLCS and RLCSD samples in SDG 2014-079 were analyzed without blank 

correction. This seems to represent a deviation from normal practice, but the impact on 

the results is inconsequential.

 RLCS/RLCSD replicate error ratios (RER) less than 1

o In all the packages only one RER exceeded 1, and only by a small amount. It was duly 

noted in the DOE package’s case narrative.

o The SOP and printed forms indicate that the RER should be less than 2, but since the 

WIPP lab reports all uncertainties as 2σ, it actually checks that the RER is less than 1. 

The SOP and forms should be revised to reflect the actual practice.

 Tracers and standards not past their expiration dates

o No deficiencies found.

 Sample tracer recoveries between 30% and 110%

o The tracer recoveries for several samples in SDG 2014-079 exceeded 110%. Although 

the lab report stated that these samples were “reprocessed,” giving acceptable 

recoveries, EPA concluded that whatever caused the original high recoveries could still 

have impacted the reprocessed results, since in each case the entire sample had been 

spiked with tracer and analyzed, leaving no remaining sample for a second independent 

analysis. EPA assigned a J qualifier (“estimated”) to each of the associated plutonium 

results. (See Table 1 for a list of the sample result qualifiers used by EPA.) Note that all 

the affected plutonium results were undetectable (less than 2σ).

o The americium analysis on blank sample C7897 also had a high tracer yield, which the 

WIPP laboratory apparently failed to notice during its internal review of the data. EPA

assigned a J qualifier to the Am-241 result for C7897 only.

 Reagent blank results within ±3σ limits on control charts

o No deficiencies found.

o NAREL observed that the reagent blank results were never blank-corrected, even in 

batches where ordinary sample results were blank-corrected. Presumably this represents 

the lab’s standard practice, which allows reagent blank results from blank-corrected and 

non-blank-corrected batches to be control charted together.

 No trends on control charts

o No deficiencies found.
EPA Air Docket A-98-49, Item II-B1-31; July 2014

4

o Although the SOP requires the validator to check for trends on control charts, a trend on 

the U-234 reagent blank control chart had not been noted. EPA found no such mistakes 

in the americium or plutonium data.

Documentation in the packages was insufficient to perform useful checks of instrument calibrations 

and quality control. Note that the efficiency calibrations for analyses based on alpha-particle 

spectrometry using alpha-emitting tracers affect only the chemical yields, or tracer recoveries, not the 

final results of the analyses. However, instrument background data would have been useful 

information.

The WIPP Laboratories never assigned a D qualifier (“detected”) to any americium or plutonium 

result, although many of the results exceeded 10 times the standard uncertainty of the measurement, 

making them not only detectable but quantifiable. In two cases (the Pu-238 results for samples C7911 

and C7912 in SDG 2014-080) EPA assigned a D qualifier where the WIPP lab had assigned a U 

qualifier, “undetectable.” Since the D qualifier had not been used, NAREL assumed the WIPP lab 

considered any result without a U qualifier to be detected. In cases where the WIPP lab had assigned 

no explicit qualifier, NAREL assigned a “D,” indicating a detectable result.

Table 1: Sample Result Qualifiers

D Detected

E Exceptions

J Estimated; unusually uncertain

N Negative

R Rejected; unusable

U Undetectable
EPA Air Docket A-98-49, Item II-B1-31; July 2014

5

4.0 Package-by-Package Notes: EPA’s Findings When Reviewing DOE’s SDGs

SDG 2014-079

8 samples collected February 17 and 18; 10 blanks

Verification

 Typo on page 4 – Am-241 detected for samples C7892, C7893, and C7896 instead of C7893, 

C7893, and C7896.

 Report pages 17 through 25 were missing.

 On page 4, C7900 is listed in both batches, SP14-03-09 and SP14-03-10.

 On page 3, C7896 client ID is incorrect and is also incorrect on page 12.

 On page 232, C7909 was omitted from the first block of the table.

 Batch SP14-03-09 did not provide as much alpha spec raw data as SP14-03-10 such as spectra and 

channel by channel printouts.

 On page 319, the collection date/time for matrix blanks (MBLK) is not the same as listed on page 

3. It is correct on page 326.

 Pages 322 and 323 appear to be the same as 324 and 325. There are slight differences in MDC and 

DLC values. The signatures on 322 and 323 have a later date. Data on pages 322 and 323 were 

reported, but 324 and 325 were not voided.

 On page 429, the collection date/time for MBLK is not the same as listed on page 3.

 On page 430, C7909 was omitted from the first block of the table.

 Some Am raw data have spectra and channel by channel printouts; and others do not.

Validation

 Sample result summaries for samples C7901–C7909 were missing from the data package.

 EPA detected Am-241 in 5 samples, although the case narrative by WIPP Laboratories claimed 

only 3. Two of the detections occurred in samples whose result summaries were missing from the 

package.

 Samples C7896—C7900 originally had high Pu-242 yields. It is unclear how they could be 

“reprocessed” to get valid results. EPA assigned a J qualifier to each of the associated plutonium 

results.

 Blank sample C7897 also had a high Am-243 yield (greater than 110 %), which the lab failed to 

note during its own data review. EPA assigned a J qualifier to the Am-241 result for C7897 only, 

not to the samples in the batch that were blank-corrected using C7897.

 RLCS, RLCSD, and reagent blanks (RBLK) were obviously calculated without blank correction. 

Other samples were blank corrected. In other SDGs it appeared that the RLCS and RLCSD were 

calculated with blank correction, but the effects of blank correction are so small for spiked samples 

that they may not be detectable.
EPA Air Docket A-98-49, Item II-B1-31; July 2014

6

SDG 2014-080

10 samples collected February 15 to 19; 1 blank 

Verification

 Collection dates/times on page 79 are not the same as those listed on page 3. Page 3 does not 

compare to the COC on page 53. The low-volume airborne particulate sample data sheets were not 

included with this SDG as they were in SDG 2014-079; so collection data cannot be verified

 The uncertainty for the Am-241 result for sample C7916 is presented on page 15 first as 102 dpm 

and then as 10.2 dpm. According to the gamma analysis report, the actual uncertainty is 102 dpm.

 Pages 142, 212, and 265 also have discrepancies in collection date/times.

 Page 264 which is the Am Raw Data separation page is in the wrong place in the package. It should 

be page 212 instead which is where the Am raw data begin.

 On page 68, the Am-243 tracer source ID is recorded as SP2-177 which is an Am-241 standard not 

an Am-243 standard.

 Am and Pu raw data sheets have the same collection date/time discrepancies as mentioned above.

 Inconsistency in providing spectra and channel by channel printouts occurs in this data package as 

in SDG 2014-079.

Validation

 No samples in SDG 2014-080 were blank-corrected. However, Am-241 and Pu-239/240 activities 

were significantly elevated in all the non-blank samples, making the blank correction essentially 

irrelevant.

 All samples in the SDG were counted for 60 minutes, although the case narrative states that they 

were counted for 1000 minutes (the routine procedure). Since all of the non-blank samples in this 

group contained elevated levels of Am-241 and Pu-239/240, the rationale for the shorter counts 

was clear.

 WIPP Laboratories detectable Pu-238 in samples C7910 and C7916 only. EPA also assigned a D 

qualifier to the Pu-238 results for C7911 and C7912 since both those results exceeded 2σ (as well 

as the DLC).

 WIPP Laboratories Pu-239/240 in all the non-blank samples.

 WIPP Laboratories Am-241 in all the non-blank samples.
EPA Air Docket A-98-49, Item II-B1-31; July 2014

7

SDG 2014-098

1 sample collected February 21; 1 blank

Verification

 Client ID incorrect on pages 3 and 8.

 Pages 57 and 58 are voided but not pages 62 through 68.

Validation

 WIPP Laboratories detected Am-241 and Pu-239/240 in the non-blank sample.

 The sample and blank in SDG 2014-098 were counted for 60 minutes, although the case narrative 

states that they were counted for 100 minutes. One sample was a blank, but Pu-239/240 was easily 

detectable in the other sample and the Am-241 result was quantifiable (greater than 10 times its 

standard uncertainty), implying that the 60-minute count times were adequate.

SDG 2014-101

11 samples collected March 11; 2 blanks

Verification

 There are no Pu-238 counts on page 109, yet counts were entered on page 98.

Validation

 No issues
EPA Air Docket A-98-49, Item II-B1-31; July 2014

8

SDG 2014-108

11 samples collected March 18; 2 blanks

Verification

 On page 7, the 4th table should be for U not Pu since only U was recounted on 3/24/2014.

 The collection date/time for the RBLK on page 330 is different than that used on page 360.

 The Pu and Am control chart filter matrix blank activity data sheets and control charts are missing 

for SP14-06-02. Therefore the value used on pages 364 and 431 for the “Blank Standard Deviation 

of the Activity Charts” cannot be verified for the matrix blank values.

 The sample net counts and background counts for Pu-239/240 cannot be verified because Pu-239 

data are not listed on page 368.

 Pages 468 and 473 were in the wrong places and should be switched.

Validation

 There was an apparently unnoticed trend on the U-234 RBLK control chart, but the uranium 

analyses were outside the scope of EPA’s review.

 The replicate error ratio (RER) for the Am-241 RLCS/RLCSD pair exceeded 1 by a small amount 

and was duly noted in the case narrative.
EPA Air Docket A-98-49, Item II-B1-31; July 2014

9

5.0 Observations

All of the data are considered usable for decision-making; however, EPA makes the following 

observations that need to be addressed in order to provide the highest quality, most defensible data 

possible at all times:

1. It appears that the calculation spreadsheets perform blank corrections using the absolute value of 

the blank count rate (MBLK). Using a measured blank value, positive or negative, for blank 

correction is common, but converting negative blanks to positive introduces a negative bias into the 

measurement process. When the blank result is positive, the sample result is reduced, as expected; 

but when the blank result happens to be negative, the sample result is also reduced by this 

approach.

2. The calculation spreadsheets appear to double-correct for both Am-243 and Pu-242 “abundances” 

in the tracer regions of interest. Fortunately, both abundances are close to 100%, making the errors 

small (less than 1%).

3. Detection decisions appear to be implemented incorrectly. The case narrative for 2014-079 states 

that Am-241 was detected in three samples, while the data show it was detected in five. (Two of 

the detections occurred in samples whose sample result summaries were missing from the data 

package.) Pu-238 was flagged as “U” (undetected) in samples C7911 and C7912, apparently 

because the result was less than the minimum detectable concentration (MDC), although it was 

greater than the 2σ uncertainty and greater than the decision level concentration (DLC). EPA

flagged Pu-238 as “D” (detected) in both of these samples.

4. According to its SOP for “Data Reduction and Reporting”

2

, the WIPP lab incorrectly propagates 

the uncertainty of the detector efficiency when calculating the uncertainties of the final results of 

americium and plutonium analyses when using an alpha-emitting tracer as a yield monitor. The 

efficiency effectively cancels out of the activity equation, so that its uncertainty does not contribute 

to the total uncertainty of the calculated activity. Note that the efficiency does affect the calculated 

value of the yield, or tracer recovery, and its uncertainty does contribute to the uncertainty of the 

yield although it does not contribute to the uncertainty of the analyte activity. Note also that the 

uncertainty of the efficiency is relatively small and should make a negligible contribution to the 

total propagated uncertainty even when it is propagated (incorrectly).

5. There is no obvious statistical rationale for the expression used for the uncertainty of the average 

background correction, as described in WP 12-RL3003 (the SOP for data reduction and reporting)3. 

The expression appears to be for a single Poisson background measurement of duration equal to 

that of the sample count, not for an average of several measurements. If the distribution is truly 

Poisson, the uncertainty of the average should be smaller than calculated; if not, it would seem 

 

2 Nuclear Waste Partnership, LLC. Data Reduction and Reporting, Technical Procedure. WP 12-RL3003, Revision 12. 

Carlsbad, NM: Effective date June 19 2013.
EPA Air Docket A-98-49, Item II-B1-31; July 2014

10

more logical to make use of the experimental standard deviation of the multiple measured 

backgrounds.

None of these observations calls into question the usability of the data in the packages that were 

reviewed. Of all the observations, observation 1 seems most significant. It tends to hide small positive 

results, converting what might sometimes be a barely detectable result into a “non-detect.” It also tends 

to increase the fraction of results that are “too negative” (i.e., more than 2 or 3 sigma below zero). 

Observations 1 and 2 suggest that the calculation spreadsheets should be re-validated, or replaced by 

fully validated in-house software for data reduction. Observation 3 describes a common mistake made 

by many laboratories and their clients across the United States. Observations 4 and 5 are offered for the 

lab’s consideration.