DISCLAIMER: This is a working document provided by the Centers for Disease Control and Prevention (CDC) 

technical support contractor, SC&A for use in discussions with the National Institute for Occupational Safety and 

Health (NIOSH) and the Advisory Board on Radiation and Worker Health (ABRWH), including its Working Groups or 

Subcommittees. Documents produced by SC&A, such as memorandum, white paper, draft or working documents are 

not final NIOSH or ABRWH products or positions, unless specifically marked as such. This document prepared by 

SC&A represents its preliminary evaluation on technical issues.

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

Memorandum

To: SEC Issues Work Group

From: SC&A, Inc.

Date: October 17, 2023 

Subject: SC&A Comments on NIOSH White Paper, “A Discussion of Completeness in 

Co-Exposure Models” 

Background

In March 2023, the National Institute for Occupational Safety and Health (NIOSH) issued a 

white paper, “A Discussion of Completeness in Co-Exposure Models” (NIOSH, 2023), to 

address comments by the Sandia National Laboratories Work Group during its April 2022 

meeting. The discussion during that meeting centered around the potential for creating a more 

quantitative framework for evaluating data completeness, as it is a near-universal issue for the 

majority of sites evaluated under the Energy Employees Occupational Illness Compensation 

Program Act (EEOICPA). As noted in appendix A of NIOSH (2023), a member of the Advisory 

Board on Radiation and Worker Health (ABRWH, Board) commented, 

there’s this thing that’s always hanging over one of these discussions, and it’s the 

data completeness. 

This kind of brings me then into my second comment. Bob mentioned that, I think 

expressed a desire for having a quantitative way to look at completeness, and boy 

that would really be helpful, because these discussions that take place in other, on 

other Sites too, are quite frustrating because SC&A will say well, we don’t feel 

that there’s data completeness, but it’s left sort of vague. 

It’s very hard for NIOSH to answer that and to come back then without some 

specific direction on, as to how to answer that question. Of course quantitative, if 

there a quantitative way, then that would make it much easier. [ABRWH, 2022a, 

p. 49] 

This potential line of inquiry was also discussed at the subsequent meeting of the ABRWH on 

April 28, 2022 (ABRWH, 2022b, pp, 40, 54-56), at which time NIOSH was tasked with 

evaluating the feasibility of a uniform quantitative approach to evaluating completeness. 

Approximately 1 year later, at the meeting of the ABRWH on April 20, 2023, SC&A was tasked 

to review the ensuing NIOSH white paper (NIOSH, 2023). This memorandum contains SC&A’s 

commentary on completeness as interpreted in NIOSH (2023). 
SC&A, Inc.

Data Completeness in Co-exposure Modeling, page 2 October 17, 2023 

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

SC&A Comments on NIOSH 2023 Conclusions

Section 9 of NIOSH (2023) gives seven bulleted conclusion points (pp. 16–17). The following 

subsections discuss these seven conclusions. 

NIOSH conclusion 1

Making a bounding co-exposure model does not require all the data, just a 

significant portion of the data from the most highly exposed workers. [NIOSH, 

2023, p. 16] 

SC&A agrees with this concept in principle. However, it also must be noted that establishing that 

the bounding co-exposure model actually contains a significant portion of the most highly 

exposed worker population is likely more difficult than it initially sounds. For example, how can 

we establish what the exposure potential for the missing portion of data (whether because the 

records are unavailable or the workers were not monitored) would have reflected? This is, and 

has always been, a professional judgment decision that resides with the Board. SC&A 

acknowledges that certain lines of inquiry can help inform that judgment, such as examining job 

titles, work areas, and specific work duties in the available dataset and making a qualitative 

determination if the dataset is representative of the different working conditions and/or clearly 

bounding of various exposure potentials experienced. 

Furthermore, SC&A notes that “a significant portion” indicates a quantitative threshold when, in 

reality, a quantitative metric is simply not plausible for every site and situation. SC&A maintains 

a quantitative metric is not likely to be universally established, and this is acknowledged in 

NIOSH conclusion 5. 

NIOSH conclusion 2 

Even the best radiation protection monitoring programs are seldom 100% 

effective. However, if a radiation protection program is working properly, 

monitoring missingness is most likely to affect workers with low doses, whereas 

data missingness can exhibit practically any pattern. [NIOSH, 2023, p. 16] 

SC&A agrees that as part of a weight-of-evidence evaluation, proper characterization of the 

monitoring program becomes paramount, including standard procedures, audit findings, and 

actual interviews with the workers both administering the monitoring program and those 

included (or should have been included) in the monitoring program. This type of evaluation is 

obviously subjective in determining whether the monitoring program would have included a 

“significant portion” of the most highly exposed workers. As this memo noted for NIOSH 

conclusion 1, a ”significant portion” is not an established quantitative measure and thus will 

likely always be a subjective, qualitative judgment. 

SC&A further cautions that it believes it is inappropriate to assume a priori that a given radiation 

protection program was “working properly.” Rather, it must be proven via the appropriate 

procedure reviews, available audits, and interviews with both the radiation protection staff and 

those included (or not included) in the monitoring program to affirm the efficacy of the program. 
SC&A, Inc.

Data Completeness in Co-exposure Modeling, page 3 October 17, 2023 

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

NIOSH conclusion 3 

To demonstrate that the dataset is “complete enough” to construct a co-exposure 

model one must, to some degree, vet the radiation protection programs that 

generated the data and the recordkeeping systems that store and report the data. 

[NIOSH, 2023, p. 17] 

SC&A agrees that such an analysis is a necessary aspect of evaluating whether the dataset 

underlying any subsequent co-exposure model is representative or bounding of the exposed 

population. Such analyses are already a standard part of Special Exposure Cohort (SEC) 

deliberations involving co-exposure modeling feasibility. That is, the dataset cannot be evaluated 

for completeness in a vacuum, and programmatic performance and administration issues must be 

carefully considered. 

NIOSH conclusion 4 

After vetting the programs and starting under the presumption that the data are 

complete, a limited number of internal and external checks of the datasets can be 

performed to look for signs of significant missingness or monitoring missingness. 

[NOISH, 2023, p. 17] 

SC&A agrees that internal and external checks of the datasets should be performed. These 

include intercomparison of the full dataset provided by a given site to other data sources. These 

might include: 

• dose tracking reports designed to demonstrate compliance 

• periodic documentation of the health physics department activities (e.g., 

monthly/quarterly reports that include significant incidents and, where available, the 

number of workers and samples requested/submitted/and analyzed during the given 

timeframe) 

• program performance assessments, both internal self-assessments and independent 

external reviews, that are directed at the effective and adequate implementation of site 

monitoring and recordkeeping in accordance with site procedures and departmental 

requirements

• comparison with the NIOSH DCAS Claims Tracking System (NOCTS) to assure that 

data transmitted for individual claims also appears in the full electronic dataset supplied 

by the site for co-exposure modeling 

However, SC&A does not agree that after vetting appropriate documentation and interviews of 

the monitoring program that the available dataset should “start under the presumption that the 

data are complete.” SC&A notes that neither should it be assumed that the dataset is incomplete, 

SC&A believes the evaluation of a given dataset using “internal and external checks” should 

always start with the neutral assumption that it is not known whether the data are complete or 

incomplete until such checks are performed. 
SC&A, Inc.

Data Completeness in Co-exposure Modeling, page 4 October 17, 2023 

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

NIOSH conclusion 5 

It may not be feasible to establish universally applicable, technically based 

quantitative limits for a dataset being complete enough. This is primarily a 

qualitative discussion based on the preponderance of evidence. [NIOSH, 2023, 

p. 17] 

Per previous discussions, SC&A agrees that a universal quantitative measure is not plausible to 

develop and agrees that each situation must be evaluated individually on a subjective, qualitative 

basis. SC&A notes that evaluating the feasibility of a quantitative measure was the original 

purpose of the tasking by the ABRWH in April 2022. 

NIOSH conclusion 6 

Regulatory compliance with a monitoring program or lack thereof cannot be used 

by itself to decide if a dataset is complete enough to construct an acceptable coexposure model. [NIOSH, 2023, p. 17] 

SC&A agrees that regulatory compliance issues, by themselves, do not demonstrate the 

infeasibility of a co-exposure model. However, evaluating compliance issues is in the same vein 

as vetting the programmatic aspects of a given site’s radiation protection program in that it 

provides weight-of-evidence indicators as to how effective and comprehensive the monitoring 

program is in practice. If any program assessment, whether performed internally or by an 

external independent source, establishes the incompleteness or inadequacy of a site monitoring 

program, that should be considered significant and require further review regarding the 

feasibility of co-exposure model development. 

NIOSH conclusion 7 

Stratification of datasets cannot be used to correct for data missingness or 

monitoring missingness and can be very time consuming and resource intensive to 

perform. Furthermore, stratification of the datasets will likely not provide any 

significant benefit to the unmonitored worker to whom the co-exposure model is 

applied. [NIOSH, 2023, p. 17] 

NIOSH (2023) provides its justification to remove stratification from co-exposure formulation in 

Section 8. NIOSH postulates that stratification is ill-advised because: 

• It is time consuming

• Workers would have to be appropriately categorized. 

• It reduces the amount of data in each stratum (thus the co-exposure models are less 

accurate). 

• Some workers would get more dose and some workers would get less dose when 

stratifying than with a single co-exposure model. 

• It does not correct for data and monitoring missingness. 
SC&A, Inc.

Data Completeness in Co-exposure Modeling, page 5 October 17, 2023 

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

SC&A recognizes that the first three of these arguments are certainly aspects of stratification that 

should be considered. The fourth argument echoes the old adage, “robbing Peter to pay Paul”; 

SC&A finds this immaterial. If an unmonitored worker is indeed one of the higher exposed 

workers, then it only seems appropriate they would be assigned higher doses. Similarly, if a 

worker is a lesser exposed worker, then it seems appropriate they would get a lower dose 

assignment than the higher strata. In SC&A’s opinion, this is entirely the purpose of 

stratification, if needed and feasible, not a downfall. 

SC&A dose agree with the fifth argument, that stratification alone would not solve potential 

completeness concerns. If there is a group of the most highly exposed workers who have 

significant data completeness issues from a qualitative standpoint (whether it is data missingness 

or monitoring missingness), then this is when questions of dose reconstruction feasibility are 

most poignant and an SEC should be considered by the Board. 

Section 8 concludes: 

if the radiation protection program at a facility was mature and functional, 

stratification is not needed to construct a bounding co-exposure model and can 

result in less accurate predictions of dose compared to unstratified co-exposure 

models. [NIOSH, 2023, p. 16] 

Whether a particular radiation protection program is mature and functional is often a matter of 

professional judgment and should not be presumed a priori. Whether or not stratification may be 

warranted in a given situation is a site-specific issue, and SC&A does not believe the future 

application of the stratification concept should be discarded out of hand. SC&A notes that the 

concept of stratification was developed by NIOSH in concert with SC&A and the SEC Issues 

Work Group, discussed, and accepted by the Board as documented in DCAS-IG-006, 

revision 00, “Criteria for the Evaluation and Use of Co-Exposure Datasets” (NIOSH, 2020). This 

document discusses stratification as follows: 

The distribution of a potentially more highly exposed population should be 

evaluated as a separate standalone distribution in situations where: 1) accurate job 

categories and/or descriptions can be obtained for all workers making up the 

general co-exposure dataset; 2) there is reason to believe that one of the job 

categories is more highly exposed; and, 3) there were unmonitored workers in this 

job category. If it can be demonstrated, however, that there were no unmonitored 

workers with the potential for exposure in this more highly exposed population, 

then stratification would not be necessary. [NIOSH, 2020, p. 11] 

SC&A believes this is the correct and appropriate approach to handling potential stratification, 

and its stance remains unchanged as that documented in DCAS-IG-006 and approved by the 

Board. 

SC&A Conclusion

SC&A notes that data completeness has been an oft-discussed topic with the general concept 

clearly defined in DCAS-IG-006 (NIOSH, 2020). That document states the following concerning 

data completeness:
SC&A, Inc.

Data Completeness in Co-exposure Modeling, page 6 October 17, 2023 

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

the amount of available monitoring data must be evaluated to determine if there 

are sufficient measurements to ensure that the data are either bounding or 

representative of the exposure potential for each job/exposure category at the 

facility. . . . 

If the number of potentially exposed workers in each category is unknown, a 

useful starting point is to look at the distribution of samples among the various 

categories of workers represented in the claimant population at that site. . . . In 

[the analysis of workers at the Nevada Test Site], the radiation safety staff was 

monitored to a larger extent than workers directly involved in site activities. Thus, 

a co-exposure model based on these data would not necessarily reflect the 

exposure conditions of the unmonitored production/process workers. If, in fact, it 

can be established that the categories of workers were potentially exposed, yet 

inadequately monitored, it could preclude the development of a sufficiently 

accurate co-exposure model, unless it can be established that the exposures to 

another, adequately monitored category of workers reliably bounds the initial 

category’s exposures.” [NIOSH, 2020, pp. 6–7] 

SC&A is including this information because it maintains that this is the correct interpretation of 

data completeness under EEOICPA as developed in concert with NIOSH, the SEC Issues Work 

Group, and SC&A and that was unanimously approved by the full Board on December 11, 2019 

(ABRWH, 2019, pp. 145–147). 

With this in mind, SC&A does not materially disagree with the general logic and philosophical 

content of NIOSH (2023), with the noted exceptions of presuming completeness a priori and 

removing potential stratification from consideration. SC&A also believes that each site is 

distinctly different regarding the implications of incomplete datasets used in co-exposure 

modeling. As such, SC&A agrees with NIOSH that an actual quantitative uniform approach 

would be inappropriate and likely not plausible. SC&A continues to assert that each site must be 

taken individually and that the only way to evaluate completeness is to apply qualitative weightof-evidence arguments to determine whether a co-exposure model can be constructed that is 

reasonably bounding for all affected energy employees.

It is SC&A’s interpretation of NIOSH (2023) that NIOSH appears to agree with this position. In 

fact, most of the concepts discussed in NIOSH (2023) have already been introduced to varying 

degrees in SEC discussions and co-exposure model development and so do not necessarily 

present a novel approach to completeness evaluations. For example, programmatic 

documentation is already reviewed to determine the efficacy of the radiation protection program 

(refer to NIOSH conclusion 3). This includes relevant information on regulatory compliance 

audits and citations that serve to illustrate the efficacy of the program, though SC&A recognizes 

that these indicators alone do not serve to inform data completeness (refer to NIOSH conclusion 

6). Sources such as NOCTS and alternate datasets are already considered when evaluating full 

datasets for co-exposure feasibility and appropriateness (refer to NIOSH conclusion 4). 

Regarding the NIOSH (2023) objections concerning stratification, SC&A maintains that the 

concept has merit and should continue to be a consideration when evaluating co-exposure model 

feasibility and applicability as was outlined in DCAS-IG-006 (refer to NIOSH conclusion 7). 
SC&A, Inc.

Data Completeness in Co-exposure Modeling, page 7 October 17, 2023 

NOTICE: This report has been reviewed to identify and redact any information that is protected by the 

Privacy Act 5 U.S.C. § 552a and has been cleared for distribution.

Finally, and perhaps most importantly, SC&A and NIOSH are in agreement that a quantitative 

threshold or metric is not possible to develop. Answering this question was the whole purpose of 

this investigation at the outset (refer to NIOSH conclusion 5). 

References

Advisory Board on Radiation and Worker Health. (2019). 132nd meeting Wednesday, December 

11, 2019 [Transcript]. https://www.cdc.gov/niosh/ocas/pdfs/abrwh/2019/tr121119-508.pdf

Advisory Board on Radiation and Worker Health. (2022a). Sandia National Laboratories 

Working Group Monday, April 11, 2022 [Transcript of teleconference meeting]. 

https://www.cdc.gov/niosh/ocas/pdfs/abrwh/2022/wgtr041122-508.pdf

Advisory Board on Radiation and Worker Health. (2022b). 145th meeting Thursday, April 28, 

2022 [Transcript]. https://www.cdc.gov/niosh/ocas/pdfs/abrwh/2022/tr042822-508.pdf

National Institute for Occupational Safety and Health. (2020). Criteria for the evaluation and use 

of co-exposure datasets (DCAS-IG-006, rev. 00). https://www.cdc.gov/niosh/ocas/pdfs/dr/dc-ig006-r0-508.pdf

National Institute for Occupational Safety and Health. (2023). A discussion of completeness in 

co-exposure models [White paper]. SRDB Ref. ID 196229