Review of NIOSH Responsiveness to Worker

Comments for Los Alamos National Laboratory

Lynn Ayers, SC&A

Teleconference Meeting of the Worker Outreach Work Group

June 5, 2014 
Background – Work Group Charter

Mission statement (excerpt):

...to evaluate the effectiveness of NIOSH activities in obtaining and making use of 

information from current and former workers and their representatives...

Evaluation Objective 3: 

Determine whether [DCAS] is giving thorough consideration to information received from 

workers through the worker outreach efforts, incorporating consideration of that material 

into its work products, as appropriate, and adequately communicating the impact of 

substantive comments to workers.

Two “Objective 3” reviews have focused on inputs received over the full 

course of site-specific outreach activities, from site profile rollout through 

SEC deliberations: 

– Pilot Review of Rocky Flats Plant (RFP), 2011-2012

– Follow-up review of Los Alamos National Laboratory (LANL), 2013-2014
General Approach to Objective 3 Reviews

• Documentation practices

o Completeness of records in Outreach Tracking System (OTS)

o Site expert interviews – completeness, validation reviews

o Action Items and follow-up on issues discussed with commenters

• Consideration and incorporation into technical documents

o Evidence from work documents, transcripts, databases, data capture

o Reflection of most recent NIOSH position on issue

o Consideration of general issues, not specific inputs

o Opportunity for NIOSH response 

• Response to commenters

o Pilot RFP review: direct response within 6 months

o LANL review: considered beyond scope of ABRWH charter
Pilot Review Recap – Rocky Flats Plant (2011-2012)

• Scope and Time Frame

o Venues included Advisory Board and Work Group meetings

o Topics included general EEOICPA concerns (e.g., COI, Burden of Proof)

o Cutoff of Nov-2010; Bulk of comments received 2004-2007.

o Identified 549 comments; reviewed 101; statistical sampling plan

• Consideration of issues

o 94% of comments showed some consideration of issues

o 85% adequate/complete consideration

o 15% unidentified or incomplete consideration

o Some missed opportunities (e.g., tritium addressed in later SEC petition)

• Communication with commenters

o 50% of comments received direct response in 6 months (varied by venue)

o Site expert review of interview notes was not well documented

o Lack of response, discussion, rationale, and closure may appear dismissive, 

regardless of actual consideration and/or correctness of NIOSH’s position
Recommendations from RFP Pilot Review

Seek clarification and/or action on the following issues:

1. Current expectations, practices, and procedures affecting outreach 

communications 

2. NIOSH self-evaluation and continual improvement efforts

3. Institutional or management system for tracking and following up on 

worker comments in an integrated fashion to assure responsiveness, 

timeliness, and accountability 

4. Staff training and guidance in effective worker outreach 

communications 

5. Solicitation of worker feedback on outreach communications 

6. Consider follow-up review of more recent site-specific worker 

outreach experience to gauge the effectiveness of programmatic 

improvements 
NIOSH Response to RFP Review

(Work Group meeting November 8, 2012)

Considering specific venues where direct responses had been lacking, NIOSH 

clarified expectations and process improvements (transcript pages 79-82):

o Public Comments at Advisory Board meetings – Process has been developed 

for tabulating comments and providing information to the Board.

o SEC Petition - Information and issues submitted in SEC Petition are addressed 

in ER. No expectation of direct response to individuals. Information is 

addressed as determined by ER and subsequent Work Group discussion. 

o Work Group meetings – Comments are addressed through Work Group 

discussion. Interested parties can follow Work Group discussion to learn how 

information is handled. 

o NIOSH facilitate access to Work Group discussions by:

• Posting transcripts on website (long-standing practice)

• Posting white papers and other discussion items in advance of meetings (recent 

enhancement)
NIOSH Response to RFP Review

(Work Group meeting November 8, 2012)

DCAS clarified current expectations and practices for receiving 

and responding to worker comments (transcript pages 84-86)

Respond to correspondence that requires a response as soon as possible, using the same method 

of communication as the original correspondence (e-mail, conventional mail, etc.).

DCAS tasks ATL to review Advisory Board meeting transcripts to identify and list public comments. 

DCAS prepares responses as appropriate and forwards comments and responses to the Board. 

Comments related to specific Subcommittee and Work Group activities are forwarded to the 

respective Chairs.

When DCAS staff members attend activities where claimants, advocates, and other members of 

the public are present, expectations for documentation and responsiveness include: 

1. Provide immediate response. No specific documentation or follow-up is required if question or 

comment is addressed to the individual’s satisfaction. 

2. Encourage individual to send question by mail or e-mail. This places comment in existing tracking 

system and assures that it is not misstated by DCAS staff. 

3. DCAS staff carry note cards for documenting comments, questions, and contact information to 

facilitate responses. When a computer application is developed for tracking comments and 

questions, the note cards will be used as a source for data entry. 

4. Minutes of meetings, when prepared, are entered into the SRDB and are procedurally required to be 

considered during any subsequent revisions to technical documents. 
Other responses and developments

• Procedure for formal Worker Outreach 

Program, DCAS-PR-012 (Rev 1) was presented 

at the Work Group meeting in Nov-2012 and 

issued in Dec-2012.

• DCAS intends to develop a comment tracking 

application to provide a method for managing 

comments and responses. [This initiative is 

responsive to both PR-012 and RFP reviews.]
Follow-up Review – Los Alamos (2013-2014)

• Scope and Time Frame

o Seeking to assess impact of programmatic changes over time 

o Recent SEC activity, revised Evaluation Report and Site Profile TBDs

o SC&A did not review comments from Advisory Board and Work Group 

meetings – now subject to tracking process

o Topics impacting dose reconstruction and SEC evaluations

o Cutoff Aug-2012; bulk of comments received 2005-2008.

o Identified and reviewed 78 comments.

• Elements of review

o Documentation practices

o Consideration/investigation of issues

o Incorporation into technical documents

o Did not evaluate feedback to commenters – “quality of service”
Documentation Practices

• Worker Outreach Meetings (issue from 2010 review of OCAS-PR-012)

o Minutes initially available for 12 of 15 sessions

o Missing sessions artifact of transition from ORAUT to OCAS contract

o Remaining sessions added 12/05/13 (not included in review)

• Site Expert Interviews (issue from Rocky Flats Pilot Review)

o Few interviews found in SRDB or SEC Viewer databases (8)

o Predominantly focused on dosimetry data; one discussion regarding 

facility operations during SEC-109 review period (2008-2012)

o Some indicators of additional interviews; insufficient detail to assess 

completeness of interview records

• Action Items (issue from 2010 review of OCAS-PR-012)

o LANL Outreach meetings (2005 and 2008) pre-date formal AI guidance

o One Action Item was recorded from 15 Worker Outreach sessions

o Several NIOSH responses indicate interest in further follow-up that 

were not recorded as Action Items in WISPR/OTS. Follow-up was not 

identified.
Consideration of Technical Information

• Incorporation into Technical Documents

o 86% incorporated to some degree (“yes,” general, partial)

o 14% not incorporated (5% responsive, 3% not responsive, 6% uncertain)

o Responsive example: exposure from operating screening X-ray equipment

o Unresponsive example: impact of wearing dosimeter under lead apron

o Support service worker issues not directly addressed in SEC-109 ER. Not 

primary basis for qualifying petition. Workers included in recommended class.

• Investigation of Technical Issues

o 82% positive investigation of issue (yes, general)

o 6% partially investigated, some aspects not investigated

o 4% not investigated (2 with adequate rationale; 1 missed opportunity)

o 8% uncertain/inconclusive
Consideration of Technical Information

• NIOSH Evaluation Comments 

o Concurred with SC&A Observation for 50 comments

o 28 substantive Evaluation Comments

• 18 Responsive – new info, explanation, rationale

• 6 Partially Responsive – aspects not addressed; investigation implied

• 4 Non-Responsive – no additional info or attempt to obtain

• Overall Responsiveness

o 90% of comments addressed to some extent (Yes, General, Partial)

o 15% with some negative observations (Partial, No, Uncertain)

o Less than 3% assessed as non-responsive (No)
Lessons Learned

• Comprehensive reviews of site-specific outreach

+ Appreciation of process, inputs, challenges

+ Identify big-picture strengths and weaknesses

+ Practical application of procedure review findings

– Cumbersome, resource-intensive

– Retrospective approach complicates review – may skew results (+ or -)

– Unable to assess impact of programmatic changes

• Unresolved questions

o Current status of comment documentation and follow-up

o Impact of recent guidance (e.g., Action Items)

o Current documentation of site expert interviews