**NRC INSPECTION MANUAL**	IRAB

		INSPECTION PROCEDURE 95003 ATTACHMENT 02

GUIDANCE FOR CONDUCTING AN INDEPENDENT 

NRC SAFETY CULTURE ASSESSMENT

PROGRAM APPLICABILITY:  IMC 2515 B, IMC 2600 B

INTRODUCTION

The purpose of this assessment is for the NRC to assess the licensee’s safety culture.  Safety culture is defined by the NRC as “the core values and behaviors resulting from a collective commitment by leaders and individuals to emphasize safety over competing goals to ensure protection of people and the environment.”  Therefore, an organization’s core values and behaviors (i.e., safety culture traits that comprise the visible aspects of a safety culture) can be assessed by evaluating the extent to which its policies, programs, and processes ensure that nuclear safety issues receive the attention warranted by their significance.  For example, the effectiveness of the licensee’s corrective action program at identifying, prioritizing, and resolving issues with nuclear safety impacts provides important insights into the licensee’s safety culture.  An organization’s members’ shared attitudes and behaviors with respect to nuclear safety also provide important insights into a licensee’s safety culture and can be assessed through behavioral observations, interviews, and focus groups.  

The guidance in this attachment is intended to enable inspectors to identify those consistencies in attitudes and behavior that are indicative of safety culture.

Relevant attitudes with respect to nuclear safety include:

·  specific attitudes (i.e., an individual’s tendencies to react favorably or unfavorably) towards different characteristics of the organization, which may include general attitudes about the organization as a whole, the effectiveness of the job performance evaluation system in encouraging taking responsibility for nuclear safety, the effectiveness of the work control system in scheduling work activities safely and efficiently, or the procedures and work packages the individual is given to assist in performing tasks;

·  perceptions, which are how an individual interprets information about the organization to form beliefs; and

·  values, which are an individual’s judgments about what is important, meaningful, and worthwhile at work both to the individual and to the organization.

Consistencies in attitudes are typically identified by interviewing an organization’s members to elicit their specific attitudes, perceptions, and values, and their views of the organization’s values and attitudes as they relate to nuclear safety.  Consistencies in attitudes may also be identified by asking individuals to provide examples of situations and experiences that are consistent with the attitudes, perceptions, and values they describe.  If a large proportion of an organization’s members share the same specific attitudes, perceptions, and values, these become a social “fact” within the safety culture of the organization and can influence individuals’ subsequent decisions and actions. 

Behavioral consistencies with respect to nuclear safety include:

·  observable behaviors (e.g., how often, as indicated by the frequency or proportion of occurrences,  supervisors mention safety considerations during pre-job briefs for jobs that may impact nuclear safety, how often personnel use procedures in the manner required), 

·  written communications (e.g., how often, as indicated by the frequency or proportion of occurrences, do annual performance reviews mention individuals’ decisions and actions related to nuclear safety in the past year; how often do the forms from a management feedback program note unsafe acts or favorably record desirable safety behaviors; how often do email communications from managers and supervisors emphasize production or safety goals), and 

·  verbal communications (e.g., how often, as indicated by the frequency or proportion of occurrences, do supervisors and managers mention deadlines and schedules during conversations or in meetings compared to the number of times they mention nuclear safety considerations).

Consistencies in behavior with respect to nuclear safety are typically identified by observing an organization’s members going about their daily work as well as reviewing written records of decisions made and work that was previously performed.  Behavioral consistencies can also be identified by asking an organization’s members questions that focus on their perceptions of the organization’s norms.  Norms are an organization’s underlying, usually unwritten, rules for behavior that establish “how we do things around here,” and they may or may not coincide with the organization’s stated policies and procedures.

Therefore, the safety culture assessment will focus on the use of tools and instruments 

(i.e., information-collection methods) that will help the inspection team identify these consistencies in organizational attitudes and behavior through the use of interviews, structured behavioral observations, document reviews and analysis, and case studies.

1.  Preparation for an Independent NRC Safety Culture Assessment 

a.	From the list of inspection requirements in Enclosure 95003.02-A, select the requirements that relate to the performance deficiencies that prompted this inspection, and, to the extent possible, adapt the selected requirements to focus on those performance deficiencies.  Ensure that the selected requirements include at least one requirement associated with each safety culture trait.  A specific performance deficiency may or may not be the result of a weak safety culture.  As input to the safety culture assessment, inspectors/(senior) safety culture assessors (SCAs) should independently determine whether weaknesses in one or more safety culture traits played a causal or contributing role in each performance deficiency.

b.	Identify the senior SCA lead and SCA sub-team who will conduct the safety culture assessment, and ensure that at least two different SCAs (or other inspection team members as appropriate) independently collect information from each 

Issue Date:  04/01/19	2	95003.02

	functional/organizational group.  The senior SCA and SCAs should be qualified per IMC 1245, Appendix C-12. The safety culture traits (detailed in NUREG-2165) describe organizational characteristics and consistencies in attitudes and behavior with respect to nuclear safety that are indicative of safety culture.  

(1)	When developing the tools and instruments and selecting the methods for this assessment, include opportunities to look for evidence of all of these traits.

(2)	When planning focus group interviews, assign two different SCAs to lead the interviews with participants randomly selected by the NRC from the same functional area.  If there is an insufficient number of licensee staff within a functional area to form two separate focus groups (e.g., chemistry), consider combining focus group participants from more than one functional area or use individual interviews instead of focus groups.  If it is difficult to conduct a focus group with personnel required to be on-shift (i.e., control room operators or on-shift security officers), consider conducting interviews with these participants one on one or two at a time.  

(3)	Assign two different SCAs to conduct structured behavioral observations of the selected work activities within a functional area.  

(4)	Establish a plan for communication between and coordination among the senior SCA and SCAs and with the other inspection team members.

c.	Obtain access to a sample of the following documents, including information the team needs for its targeted review (ensure they are not duplicates of that which has already been requested by team leader):

(1)  	The root cause investigations of the performance deficiencies that prompted this inspection.

(2) 	Reports of any self or third-party organizational/safety culture types of assessments from the past five years.  These reports may include:

	-safety culture assessments completed either by the licensee or a third-party.

	-safety conscious work environment assessments.

	-leadership assessments (redacted, as necessary).  Information gathered about an individual manager that provides evidence of leadership ineffectiveness or staff dislike of a particular leadership style should not be considered when developing safety culture insights.  However, if there is evidence of generally ineffective leadership or style which demonstrates organizational attitudes and behaviors that are inconsistent with those described in the safety culture traits, this should be noted.

	-employee morale/job satisfaction assessments.  Information gathered that provides evidence of low employee morale or low job satisfaction should not be considered when developing safety culture insights, except if the low morale or job satisfaction significantly contributes to weaknesses in the safety culture.

Issue Date:  04/01/19	7	95003.02

	-Nuclear Quality Assurance/Oversight assessments related to the traits of safety

           culture.

	-any focused or broad-scope organizational performance assessments. 

(3)	Review corporate and site safety policy statements as they relate to safety culture.

(4)	A sample of redacted job performance reviews from each functional group in the organization (e.g., operations, maintenance, security, engineering) and any redacted agreements or documents related to the bases for management compensation and incentives.  Review the sample to determine if there is evidence that the actual reviews implemented the licensee’s guidance for conducting performance reviews, especially with regard to the balance between safety and production.

(5)	Meeting minutes from the past year for site senior management team meetings, nuclear safety culture monitoring panels, nuclear oversight review group meetings, and corrective action review group meetings; meetings to develop and amend site financial plans and budgets, including operating, maintenance, and capital improvement plans; meetings that focus on decision-making with nuclear safety implications; and other meetings held to plan and discuss mitigating any potential or actual chilling effects from disciplinary actions.  Review for evidence that safety has priority, e.g., for safety over costs in making improvements or maintaining systems.

(6)	Documents describing any reward or incentive programs focused on promoting nuclear safety behaviors and documentation pertaining to the implementation of the program(s) (redacted, if necessary).

(7)	Lesson plans used to train site personnel on safety culture and/or safety conscious work environment, and records that show when the training was presented and attendance.  Review training lesson plans and records to determine what was presented, when it was presented, and who attended.

(8)	Summaries of documents from the corrective action program that relate to the traits of safety culture and were identified or resolved within the previous year.  Review and code them to the applicable safety culture traits.

(9)	Complete and current organizational charts, including the names and site contact information for the individuals listed.

(10)    Written communications (e.g., memoranda, e-mails) between management and staff related to any significant organizational changes within the past year.  Significant organizational changes could include changes in organizational structures and functions, leadership, policies, programs, procedures, and resources.

(11)  Documents that correspond to the selected inspection requirements in Enclosure 95003.02-A.

d.	From NRC sources, obtain and review:

(1)	Allegations related to the site’s safety culture received within the previous year, at a minimum; and

(2)	NRC inspection reports. 

e.	Review the results of the licensee’s third-party safety culture assessment to determine if particular functional/organizational groups were identified by the SCAs as having problems in any of the safety culture traits or if the results indicated that there were weaknesses in any of the safety culture traits across the site.

f.	From the review of the documentation, determine if any functional groups, management levels, or safety culture traits should receive more emphasis in the assessment based on the licensee’s assessment having identified safety culture or performance issues related to them.

g.	Determine the assessment methods that are best suited to the perceived situation at the site, ensuring that each safety culture trait will be assessed with at least two different methods, and develop sampling plans for each method.  Assessment methods shall include individual and/or focus group interviews (see guidance in Enclosure 95003.02-B and Enclosure 95003.02-C); structured behavioral observations (see guidance in Enclosure 95003-D.02); and event follow-up studies (see guidance in Enclosure 95003.02-E).

h.	Prepare the selected data-collection tools, which may include interview and focus group guides and behavioral observation checklists.

i.	Work with the licensee to identify the appropriate means to disseminate a communication plan to site personnel that addresses the purpose of the assessment; states the team will meet with groups, observe meetings and work activities, and talk with individuals; states anyone who wants to talk with the NRC should contact the team (provide appropriate instructions); describes, to the extent possible, information obtained during the assessment will not be attributed to individual participants who are interviewed or observed by NRC inspectors; and requests sensitivity to information shared between personnel during participation in NRC activities, such as focus groups.

2.  Conduct the NRC’s Independent NRC Safety Culture Assessment **

a.	Complete this assessment by applying the methods and sampling plans, using the data-collection tools developed during the preparation phase.

b.	As the assessment progresses, the lead senior SCA should adjust the assessment plan as required to: 

(1) 	ensure that the information collected will be adequate to reach valid and reliable insights about the safety culture traits at each management level and in each functional group that falls within the scope of the assessment;

(2) 	resolve inconsistencies identified in NRC assessment results; and 

(3) 	address emergent issues identified during other inspection or assessment activities.  

		Adjustments to the assessment plan may include increasing the number of interviewees or focus group participants, conducting additional individual interviews, adding documents to be reviewed, increasing the number and focus of behavioral observations, re-directing resources to complete a case study, and/or increasing the number of team members collecting related information.

c.  	Ensure that each safety culture trait is assessed using at least two data-collection methods, by at least two SCAs (or other inspection team members as appropriate) independently, and that the data-collection methods are applied consistently.  In this context, valid insights are based on consistent results from applying multiple information-gathering methods, and reliable insights are based on consistent results from multiple team members who are independently collecting related information.

d.  	Identify weaknesses in any safety culture trait and the functional groups and management levels in which any weaknesses appear.  As examples, identify any performance deficiencies for which safety culture weaknesses were a root or contributing cause.

e.  	As results from the various data-collection tools are being compiled, and particularly after all results have been obtained, aggregate those results to determine whether any consistency regarding a particular safety culture trait exists among results obtained from various data-collection tools and inspectors.  From this determination, develop insights about the various safety culture traits and how they are reflected in the attitudes and behaviors within the various licensee functional groups.  

	The insights of most concern include the following:

		●	Corporate and/or senior site management demonstrates attitudes and behavior with respect to nuclear safety that are substantively inconsistent with the expectations in any of the safety culture traits.

		●	A single critical functional group, including operations, engineering, maintenance, or security, demonstrates weaknesses across multiple safety culture traits.

		●	The majority of functional groups demonstrate some weaknesses in multiple safety culture traits (i.e., weaknesses are widespread throughout the organization).

If any of these statements are supported by consistent results obtained through application of multiple data-collection tools in conjunction with inspector/SCA insight, then:

		●	Use this information to inform the assessment of the contributors to degraded performance in the affected strategic performance areas (SPAs).  For instance, the results of this assessment can be reflected in statements such as ”weaknesses in a safety culture trait (or group of traits) contributed to deficiencies in (some SPA Key Attribute),” or ”weaknesses exhibited by the operations organization related to (some specific list of safety culture traits) contributed to the degraded Reactor Safety SPA (or Key SPA Attribute).”

		●	Document each such statement as a key result of this assessment.  Follow the requirements in Section 02.12. and associated guidance in Section 03.12. to perform the documentation in coordination with the rest of the inspection team.

END

Attachment 95003.02 Appendices:

Appendix 95003.02-A:  “Sample Inspection Requirements for Safety Culture Traits.”  

	[ADAMS ML082670440]

Appendix 95003.02-B:  “Sample Questions for Safety Culture Traits.” 

 [ADAMS ML082670848]

Appendix 95003.02-C:  “Guidance for Focus Groups and Individual Interviews.”  

	[ADAMS ML082630691]

Appendix 95003.02-D:  “Guidance for Structured Behavioral Observations.” 

 [ADAMS ML082670874]

Appendix 95003.02-E:  ”Guidance for Event Follow-up Studies.”

[ADAMS ML082630714]

Appendix 95003.02-F:  “Guidance for Evaluating Safety Culture Surveys.”

[ADAMS ML082630765]

Issue Date:  04/03/14	1	95003.02

Appendix 95003.02-A

Sample Inspection Requirements for Safety Culture Traits

This enclosure contains sample inspection requirements from which safety culture assessors may select and adapt inspection requirements related to performance deficiencies, as required by section 95003.02.08.a.  This enclosure also identifies documents which correspond to the requirements, to assist safety culture assessors in compiling a list of documents to request from the licensee.  Hence, this enclosure is a resource that safety culture assessors may use to develop the assessment section of the inspection plan.

In this enclosure, sample inspection requirements and corresponding documents are listed in the two-column table below, under each of the safety culture traits:  column one includes the inspection requirements associated with each trait, while the second column describes the corresponding documents which should be requested from the licensee.

Issue Date:  04/01/19	AppA-1	95003.02

Issue Date:  04/01/19	AppA-2	95003.02

Issue Date:  04/01/19	AppA-3	95003.02

Issue Date:  04/01/19	AppA-4	95003.02

Issue Date:  04/01/19	AppA-5	95003.02

Issue Date:  04/01/19	AppA-6	95003.02

Issue Date:  04/01/19	AppA-7	95003.02

Issue Date:  04/01/19	AppA-8	95003.02

Issue Date:  04/01/19	AppA-9	95003.02

Issue Date:  04/01/19	AppA-10	95003.02

Issue Date:  04/01/19	AppA-11	95003.02

Issue Date:  04/01/19	AppA-12	95003.02

Issue Date:  04/01/19	AppA-13	95003.02

Issue Date:  04/01/19	AppA-14	95003.02

Issue Date:  04/01/19	AppA-15	95003.02

Issue Date:  04/01/19	AppA-16	95003.02

Issue Date:  	95003.02

Appendix 95003.02-B

Sample Questions for Safety Culture Traits

The following questions are examples of the types of questions which may be asked during focus group or individual interviews.  Safety culture traits and questions selected for inspection should be based upon specific site circumstances, i.e., not all safety culture traits and/or all questions will necessarily be applicable.  The questions are not all-inclusive and the following list is not intended to be used in its entirety.  The interviewer shouldn't simply read the questions to the focus group or interviewees as written; instead, the interviewer should understand the audience and reword/tailor the questions/topics for the audience.

The questions related to each safety culture trait are organized here in the same order as presented in Appendix A of this inspection procedure, but the list of questions to be used in any particular interview or focus group can be organized differently.  However, when developing the list of questions to be asked in interviews or focus groups, it is easier for respondents to answer groups of related questions than questions that skip from one topic to another.  One way to introduce a group of related questions is with a statement such as, “Now let’s talk about decision making" (or continuous learning, work processes, etc.).  

The first step in developing a good question is to identify exactly what kind of information you want the respondents to provide.  This is important because it is easy to receive one type of information when you really want another.  Virtually all questions are either open-ended or close-ended.  

Open-ended questions are generally more demanding to answer, typically produce many different responses, and often produce information that cannot be compared across respondents.  These questions may be used when the safety culture assessor doesn't want to specify response choices or wants to (1) to give respondents a chance to state opinions, (2) let respondents vent frustrations, and (3) hear what has been overlooked, or (4) further explore issues.  Open-ended questions are helpful when they follow a close-ended question and ask respondents to explain details about their particular answer.  Also, open-ended questions are desirable when respondents are being asked about behaviors. 

Close-ended questions are those that provide specific choices and the respondent selects from the choices.  These questions are generally less demanding to answer and the responses are easier to analyze and aggregate.  

In summary, well worded questions asked in the appropriate format will make it easier for the respondent to answer.  Using the opened and closed ended questions appropriately will provide useful information for the safety culture assessor. 

Enclosure 95003.02-C provides additional guidance for conducting individual interviews and focus groups.

Leadership Safety Values and Actions

a. Has any change occurred in the nuclear-safety orientation of the site during the last several years?  If so, has this had any effect on your work?  How?  Please describe.

Issue Date:  04/01/19	AppB-1	95003.02

b. What messages have you received from various people in this organization regarding the priority of safety?  Please describe any different messages.  

c. How does your management treat errors?  Does your management consider errors as negative to the business, as learning opportunities, or both?  Can you describe any examples of errors that were used as learning examples and/or errors that were considered to be negative for the business?  If an error was used as a learning example, how was it discussed?  Would you be more or less likely to report an error you or a co-worker made as a result?  If an error was considered to be negative for the business, what happened?  Would you be more or less likely to report an error you or a co-worker made as a result?

d. How do your managers balance production and safety?  Can you give an example of a good balance between production and safety?  An unacceptable example?  If your work group has production goals, are these goals communicated in a manner that reinforces safety?

e. Please describe your line of authority.  Who revises your line of authority if necessary?

f. Does the plant have any kind of program to reward staff for improving safety, such as spot awards for excellent work practices or awards for the best suggestions to improve safety?  If so, please describe the program(s).  Do you think they make difference in what people do around here?  If not, what would be more effective?

g. Can you think of an instance from the past year in which your supervisor or a manager spoke directly to you about a nuclear safety issue?  If so, please describe the circumstances.  What was the nature of the discussion?  Was a decision made to change anything about how you perform your work?  If so, please describe the decision.

h. Can you think of an instance from the past year in which you approached a supervisor or manager about a nuclear safety issue?  If so, please describe the circumstances.  What was the nature of the discussion?  Was a decision made to change anything about how work is performed as a result of the discussion?  If so, please describe.  If not, what was the reason(s) the decision was made not to make any changes?

i. Do you get feedback on your nuclear safety performance from your supervisors?  Peers?  Can you describe a recent (past year) situation in which you received feedback from your supervisor/peers?  What was said or done to give you the feedback?  Did you change anything as a result?  Are you able to provide feedback on safety performance to your peers or supervisors?  Please explain.

j. Based on your experiences, does your manager fully understand technical and nuclear safety issues that you or members of your work group raise?  How does your manager handle if s/he doesn’t understand your issue?  Is your input solicited during the problem resolution if you enter an issue into the CAP? 

k. In the past year, have you lacked the type of tools, equipment, and other resources you need to perform your job?  Can you think of an instance in the past year when you needed additional tools/equipment/resources to perform your work and were able/unable to obtain them?  Please describe the request.  If it was turned down, what reasons were you given for the decision?  What impact did the lack of resources have on your work?

l. During the past year are you aware of situations in which the lack of staff, equipment, or facilities impacted the safe completion of a job?  Are the people available qualified to perform the work?  If no, why do you think that is?  In the past year, can you think of any instances in which a lack of qualified staff, equipment, or facilities delayed the completion of work tasks?  If so, please describe the circumstances.  Are there any barriers to adding the qualified staff you need to get work done around here?  Please describe.

Issue Date:  04/01/19	AppB-2	95003.02

m. Have you worked overtime under a deviation from the work hours policy in the past year?  If so, which of the work hour limits was/were exceeded?  What were the circumstances that required the extra work hours?  How often does this occur? 

n. Have you ever turned down overtime because you believed you were too tired to be safe?  What response did your supervisor (or whoever you reported being too tired to) give you?

o. Describe your fitness for duty program.  Based on your experience, is it effective?  Please explain.  Based on your experience, is management supportive of the program?

p. Do managers observe your work?  If so, how often?  What do they do when they are observing?  Are you aware of any changes to how work is performed at the site that have resulted from management observations?  If so, please describe.

q. Does the site provide oversight of contractors?  If so, how?  In your experience, has the oversight been adequate?  Can you think of any examples where lack of oversight of contractors led to a safety issue?  Please describe.

Problem Identification and Resolution

a. Are issues entered into the Corrective Action Program (CAP) prioritized correctly? Please explain.  Is there any difference in the handling of an immediate nuclear safety issue vs. a more long-term nuclear safety issue at this site?  Can you describe the types of nuclear safety issues that are immediately addressed if they arise?  What types of nuclear safety issues are associated with longer response times?  Can you think of any long-term nuclear safety issues which exist that in the past year either have not been appropriately evaluated or whose corrective actions have been inappropriately extended?  Please describe the situation.  Do you know the reasons for delay in resolving the issue?

b. Is there a threshold for items identified for entry into the CAP?  Is this threshold at the appropriate level?  Are you aware of any abnormal or unusual or questionable conditions that may not be entered into the CAP but could potentially be indications of or precursors to nonconforming or degraded conditions?

c. Does your management encourage condition report (CR) initiation?  How?  Where?  Can concerns be filed anonymously through the CR process?  What about the Employee Concerns Program process?

d. Describe how CAP trends information in the aggregate to identify common cause problems.

e. Please describe an example from the past year in which your supervisor or manager stopped work or delayed completing a task because of a nuclear safety issue or concern.  What was the nature of the issue?  How was it identified?  How was it resolved?  Were you personally satisfied with the resolution?  If not, what would you have liked to see happen? 

f. Do you believe that the site’s corrective action program is successful in addressing issues that are submitted?  Can you provide an example related to your answer?  Is the effectiveness of the implemented corrective actions evaluated?  How?  How frequently? 

g. Are issues raised from CAP tracked to completion?  Are initiators informed of the result?  Can you describe any examples when this was true?  If not, please describe what happened to the issue and why it wasn’t completed or the initiators weren’t informed, if you know? 

h. Have you ever submitted an issue to the corrective action program?  Was the issue adequately addressed?  If not, did you pursue the issue?  Please explain.

Issue Date:  04/01/19	AppB-3	95003.02

i. In your experience, are issues in the CAP addressed in a timely manner?  Were the issues addressed in a manner consistent with their importance? 

j. Are you aware of any specific instances in which another employee (or contractor) submitted an issue to the corrective action program and considered the response s/he got to be incomplete or unacceptable?  Please describe the situation.

k. Do you see any changes in the amount of time necessary to resolve corrective action issues over the past year?  If so, do you have any ideas about the reasons for the changes? 

l. How is the CAP assessed?  What have been the results?  Are such reviews effective?  Please explain.  How does CAP address causes and extent of conditions?  Are such reviews complete?  Please explain.  

m. Describe any unexplainable change in the number or nature of issues raised by employees to the CAP.

n. How and at what point are employees who raised issues included in the corrective action process?  Is this necessary?

o. What corrective action tracking systems exist that are not part of the official CAP?  If so, why do they exist?  Who uses them? What types of issues are entered into them?  Do any of these tracking systems include items that are potentially important to nuclear safety or that should be in the official CAP?  Please give examples.  Do these tracking systems get periodically audited to verify that any items that should be in the official CAP are put into that program?

p. Can anyone enter an issue into the CAP?  When someone enters an issue into the CAP, does the entry have to be approved by a supervisor?  Does anyone higher up also have to approve the entry before it goes through the remainder of the CAP process?  Are you aware of any issues that someone wanted to enter into the CAP, but a supervisor or higher-level manager disapproved the entry?  Please describe the situation.

Personal Accountability

a. Who is responsible for nuclear safety at this site?  Explain.

b. Do you have nuclear safety responsibilities in your job?  If so, please describe what they are.  Is nuclear safety incorporated into your job performance review?  If so, how?

c. In the past year, have you experienced any challenges or barriers to meeting your nuclear safety responsibilities?  If so, please describe them.  Have you had any particular successes in meeting your nuclear safety responsibilities?  If so, please describe them.

d. In terms of safety, what is your personal approach to your own work?  Who do you look to for guidance on nuclear safety issues

e. Are there any nuclear safety initiatives or programs that your work group or team is currently involved in?  If so, please describe the initiative/program.  What is its purpose and goal(s)?  What are the actions you are taking to accomplish the goal(s)?  Do you have a sense of whether the program or initiative is being effective?  Please describe.  Does the program include ways to measure its effectiveness? 

f. Can you think of an instance from the past year in which you approached a co-worker or a co-worker approached you about a nuclear safety issue?  If so, please describe the circumstances.  What was the nature of the discussion?  Was a decision made to change anything about how work is performed as a result of the discussion?  If so, please describe.  If not, what was the reason(s) the decision was made not to make any change(s)?

Issue Date:  04/01/19	AppB-4	95003.02

g. When there is a problem to be solved that will affect several work groups or other departments, how is that handled?  Who is usually involved with the resolution (e.g. employees or managers)?  If the solutions are identified by managers, do employees have input?  If so, how much?  What would you do if a solution is identified that ins unworkable for you or your department?

Work Processes

a. Does the site plan work such that job site conditions are adequate, including environmental, which may impact human performance; plant structures, systems, and components; human-system interface; and radiological safety?  (Divide this question in pieces as appropriate for the respondent) Please describe.  

b. Has your supervisor provided you with “abort criteria” for situations or conditions in which you should stop work?  What are they?  Are you aware of any guidance with respect to “compensatory actions?”

c. How does the site schedule emergent work?  Can you give an example where it was not scheduled appropriately?	

d. In the past year, have you found that the work you were assigned to do couldn’t be performed because it was scheduled in a way that it interfered or conflicted with other work being performed?  Did the work have to be stopped or delayed?  Over the past year, about how often would you estimate that this has happened to you? 

e. In the past year, have you had to make a decision with nuclear safety implications and did not have any procedure to follow?  Please explain.  Have you ever had to deviate from procedure in order to assure the nuclear safety of the site?  Please explain.

f. Are you aware of an example of a repeat maintenance issue that occurred in which the licensee had previously resolved the issue with temporary modifications?  Please explain. 

g. Is preventive maintenance scheduled in a manner that promotes long term equipment reliability?  Please provide an example.

h. Is your work impacted by “operator work-arounds?”  If so, please describe how?

i. On a scale from 1 to 5, how reactive (rather than preventive) do you rate the plant site maintenance scheduling?  When equipment failures occur, are the maintenance activities sufficient to address all aspects of the system which could have been impacted or are subject to the same failure types?  Please explain.

j. What methods does your site use to maintain plant safety long term?  (e.g., maintenance of design margins, minimization of long-standing equipment issues, minimizing preventive maintenance deferrals, ensuring maintenance and engineering backlogs are kept low enough to support safety) Can you provide any examples or how it has (or has not) had an effect on resources?

k. Have any instances occurred in which backlogs impacted the site’s ability to respond to nuclear safety issues?  Please describe.

l. Are there adequate resources to ensure that procedures, work packages, and design documentation are complete, accurate, and up-to-date?  If no to any of these attributes, please describe.

m. Do you use the simulator for any of your work or training?  How would you describe the difference in the performance of the simulator with the plant?

n. Are you aware of an emergency drill where the adequacy of the emergency facility impacted performance of personnel?  If so, please describe.

Issue Date:  04/01/19	AppB-5	95003.02

o. In the past year are there any long standing equipment issues at the site that were not addressed, such as deferred maintenance and/or PM’s, deferred outage work or deferred emergent work, unaddressed operator burdens and control room deficiencies, or long-standing temporary mods?  If so, please explain.

p. Is there an appropriate balance between outage and on-line maintenance at this site?  Are outages typically long enough to do all the necessary work?  Is work sometimes moved from the outage to on-line maintenance to keep outages as short as possible?

q. What is the first thing that happens here when an event has occurred that seems to have been caused by human error?  Can you give an example?  Based on your experience, what are the most common reasons behind human errors?

r. Are self and peer checking procedures used at the site?  If so, how?  Please describe your personal experiences.  Can you think of any situations where they should be used but are bypassed?  If so, please explain.  Have you received any training on human error reduction techniques?  How is the use of human error reduction techniques reinforced?  Are you held accountable for using human error reduction techniques in your work?

s. Do you participate in pre-job briefs?  If so, are pre-job briefs routinely held?  For what types of work?  Are they effective?

t. What approach does this site take towards preparing for new and infrequently performed tasks?  Is just-in-time training conducted for infrequently performed tasks?  If so, how often?  Have you ever participated in one of these training sessions?  If so, please describe your experience.  Do supervisors and/or managers typically provide oversight when new or infrequently performed tasks are being conducted?

Continuous Learning

a. On average, what is the proportion of staff time in your department that is devoted to training?  What do you think about this proportion of time (e.g., too little, too much, right amount)?  How does this compare with other departments at the site?  How does this compare with other plants that you are aware of?  

b. Is it ever necessary to cancel or reschedule training due to operational requirements?  For what types of reasons does this occur?  About how often does it occur?  In the past several years, have you had planned training cancelled due to operational requirements?  Were there any actions taken to reschedule the cancelled training?  Who took these actions?  Please describe what happened.

c. In the past several years, have you been involved in any benchmarking activities?  If so, please describe them.  Were any of your benchmarking ideas implemented?  Please describe the ideas and how they were implemented or why they weren’t implemented.  Does it seem that the changes have made a difference?  If so, how?

d. Have other departments been involved in benchmarking?  Were any changes made as a result of what they learned?  Does it seem that the changes have made a difference?  If so, how?  (Limit this question to managers, supervisors or others who would know about benchmarking activities in other sections.)

e. Have you been involved in a major organizational change (e.g. reorganization, layoff, voluntary staff reduction/buy-outs, retirements), where knowledge was transferred to persons who might be new to a position?  If so, was this done in a timely way (i.e., before the knowledgeable person leaves)?  How well did this work?

f. Are you able to find out what's going on relative to operating experience in the rest of the industry?  from the NRC?  Please describe how.

Issue Date:  04/01/19	AppB-6	95003.02

g. Does the site have a program for the collection of operating experience (OE) information?  What about for evaluation?  How are the programs communicated to the affected staff?  How effective are the programs? 

h. Can you think of an instance in the past year where external operating experience (OE) resulted in changes in how things are done here?  If so, what was the event/problem/OE?  Do you think the changes solved any problems or improved things?

i. Do you read relevant internal OE information?  What about external OE information?  How often?  Is the information provided to you in a timely manner?  Is it useful?  Is it accessible?  Can you quickly find what you need (without having to wade through everything)?  (If a supervisor) Can you quickly find relevant information related to specific jobs to include in pre-job briefs?  Do you have sufficient time to effectively use OE (i.e., is it an organizational priority?)?  If no to the any of the above, please explain. 

j. Can you think of an instance in the past year where OE information resulted in change to procedure(s)?  If so, how often has this happened?  Please provide an example and describe what happened.  

k. Have there been any changes to training as a result of OE information in the past year?  If so, Please provide an example and describe what happened.

l. Can you think of an instance in the past year where OE information did not receive attention?  Was it something you thought was relevant?  What was the topic?  What would you have liked to see changed because of it?  Please explain.

m. Is self-assessment and improvement important at this site?  Explain.

n. Have you ever been involved in a selfassessment or independent assessment?  If so, what was the subject?  When was it done?  Did you have the resources you needed to complete the task (e.g., time, qualified personnel to address the technical issues involved)?  If not, please explain.  Were the schedule and due dates for completing the assessment consistent with the complexity of the issue?  Was the assessment of sufficient depth to address the complexity of the issue?  Were the results and recommendations from your effort used?  How?  Did the outcomes from the selfassessment lead to positive changes in how things are done?  Did you experience any challenges while conducting the assessment?  Please explain.  Would you volunteer to become involved in one again?  Please, describe your reasons.  If not, do you know someone who has been involved in a self-assessment who we could talk to?

o. Do you know what typically happens to results from self-assessments or independent assessments?  If so, please explain.  If the assessment makes recommendations, do you know what happens to those recommendations?  Please explain.  Is there any tracking done on the recommendations?  Please explain.  Has there been a recent self-or independent assessment of your department?  Were the results communicated to you?  If so, what were the results/findings?  What changes resulted from the findings of the assessment?  Were there any results/findings that were not addressed?  Please explain.

p. Are you aware of an assessment of the effectiveness of the corrective action program?  If so, did any of the findings impact your work?  How?

q. In your job, do you track or trend performance indicators?  What are those indicators?  How are they used?  Based on your experience, how would you describe the usefulness of the performance indicators?  Are there any indicators you believe should be used but are not currently? 

Issue Date:  04/01/19	AppB-7	95003.02

r. Are the results from self-or independent assessments typically communicated to you or affected personnel?  How?  If the results typically aren't communicated or you do not know, please describe the various assessments you know have been done in the past year.  What do you think about not receiving this information?  Is not receiving this type of information a problem?  Whose responsibility do you think it is to deal with communicating that information (management, yours, or both)?  If you are aware of assessment results, could this communication be improved?  How?

s. Have there been instances in which the results of some self- or independent assessment resulted in changes in how you do your job?  What were the changes?  Have the changes solved the problem(s) and/or improved things?  Was there something else that could have been done that would have worked better or been more effective?

t. What’s the reaction to independent assessments done by your QA department?  By external groups?  By INPO?  NRC inspections?  Is this input valued by the organization?  by management?  by your peers?  Is it used to improve?

Environment for Raising Concerns (for working level personnel)

a. Are you willing to raise a safety concern?  Are there any conditions under which you would be hesitant to raise a safety concern?  If yes, does that condition exist here at (Insert Plant Name)?  Please elaborate.

b. Are you aware of situations in the past year, where any employee or contractor may have been hesitant to raise concerns, internally or externally?  If yes, please explain.  

(If the NRC safety culture assessor is aware of a specific incident that may have caused such hesitation, then ask about it.  Focus on whether or not the interviewee or others may be less likely to report concerns since that incident).

c. Where would you go to raise a safety issue?  [The NRC safety culture assessor should be aware of the following avenues for raising concerns, but not prompt the interviewee:    supervisor, corrective action program (CAP), alternative program (Employee Concerns Program (ECP)/Ombudsman), NRC or other avenue.]  Why would you pick this avenue?  Have you or others had any experiences, or know of any situations, that have influenced your decision to pick this avenue?  If so, please describe.

d. Are there other avenues available to you for raising safety issues?  Ask each of the questions listed in the following table for each avenue available:

Issue Date:  04/01/19	AppB-8	95003.02

e. Explain how to use the ECP/Ombudsman program.  How did you learn about the ECP/Ombudsman program?  Is the ECP/Ombudsman office accessible?  Too visible?  Do you (routinely) see the ECP/Ombudsman Manager/Investigators around the plant?  If yes, give examples.  Is your management supportive of the ECP/Ombudsman program?  If yes, how is such support demonstrated?  If no, please describe what has led you to believe that they are not supportive.

f. Has your confidentiality been breached when a safety concern was raised to the ECP/Ombudsman?  Describe.  Do you know if anyone else’s confidentiality has been breached when a safety concern was raised to the ECP/Ombudsman?  If yes, describe.

g. Would you say that your management is supportive of the SCWE policy? If yes, how is such support demonstrated?  If no, please describe what has led you to believe they are not supportive.

h. Have events or circumstances occurred in the past year that have reduced:  Your willingness to identify or raise safety issues?  Your confidence in the corrective action program?  Your willingness to challenge actions or decisions you believe are wrong?  Your comfort level in voicing your viewpoints and opinions?

i. Have you received training concerning SCWE?  If yes, describe what it covered.  When did you last receive SCWE training?  Is there periodic SCWE refresher training?  If yes, how often?

j. Did your supervisor receive any SCWE training?  If yes, did you notice a difference in the way he or she handled concerns after the training?

k. If there has been an assessment of SCWE, were the results effectively shared with you?  Were any actions taken with the results?  If yes, please describe.

l. If actions have been taken by management to maintain or improve the SCWE, have they been effective?  Why or why not?

Environment for Raising Concerns (For supervisors and managers)

a. Are you willing to raise a safety concern?  Are there any conditions under which you would be hesitant to raise a safety concern?  If yes, does that condition exist here at (Insert Plant Name)?  Please elaborate.

b. Are you aware of situations in the past year, where any employee or contractor may be hesitant to raise concerns, internally or externally?  If yes, please explain.  (If NRC safety culture assessor is aware of specific incident that may have caused such hesitation, then ask about it.  Focus on whether or not the interviewee or others may be less likely to report concerns since that incident).

Issue Date:  04/01/19	AppB-9	95003.02

c. Where would you go to raise a safety issue?  [The NRC safety culture assessor should be aware of the following avenues for raising concerns, but not prompt the interviewee:    supervisor, corrective action program (CAP), alternative program (Employee Concerns Program (ECP)/Ombudsman), NRC or other avenue.]  Why would you pick this avenue?  Have you or others had any experiences, or know of any situations, that have influenced your decision to pick this avenue?  If so, please describe.

d. What are your management’s expectations of you regarding handling employee (safety) concerns (e.g. responsiveness, timeliness, availability, confidentiality)?  How has your management documented these expectations?  Please explain.  How are these expectations measured/assessed (e.g., performance appraisal)? 

e. What are management’s expectations regarding employee behavior when raising safety concerns (e.g. peertopeer retaliation)?  How are these expectations communicated? 

f. How do you actively encourage your employees to bring concerns to you?  Give examples (e.g. reward/incentive programs; communications). 

g. Describe what an employee could do if he/she were not satisfied with how their concern was handled.  For instance, is there an appeal process they may use?  If yes, have you ever been through such a process?  Describe your experience.  How do you advertise this process to employees?

h. Have you been requested to protect the confidentiality of an employee raising safety concerns?  If yes, how did you protect that?  Are you aware of any instances where an employees’ confidentiality was breached?  If yes, describe.

i. As a manager, explain how you use the ECP/Ombudsman.  Do you believe that the ECP/Ombudsman program is sufficiently independent from management?  How do you show support for the ECP/Ombudsman?  Give an example.  (Is there survey data that shows what percentage of employees believe their manager supports the ECP?) 

j. How are contractors covered by your SCWE Policy?  Are all contractors covered?  How do you ensure that contractors working under your direction foster a SCWE?  Ask if guidance exists for evaluating contractor SCWE programs. 

k. Have you received training concerning SCWE?  If yes, describe what it covered. If you have received SCWE training, when did you last receive it?  Describe any changes in the way you handled safety concerns brought to you after you took the SCWE training.  Is there periodic SCWE refresher training? If yes, how often?  What SCWE training do contractors receive? If none, why not?

l. Describe the tools that management uses to monitor overall SCWE performance.  How are the results of these tools (e.g., SCWE surveys, etc.) made available to the workforce?

m. Have actions been taken to maintain or improve the SCWE?  Have they been effective?  Why or why not? 

n. Do you know what is expected of you in handling employee concerns?  Do you feel able to effectively handle employee concerns?  Are you held accountable for your handling of employee concerns?  How?

Effective Safety Communication

a. Is there a vision/mission statement/policy that addresses safety?  Is it clearly visible and understood?  Where is that statement/policy located?  What goals does that statement/policy specify? 

Issue Date:  04/01/19	AppB-10	95003.02

b. Is there a separate policy on safety culture?  If so, what do you think is the most important aspect of this policy?  What is your reason?  Is there a separate policy on safety conscious work environment?  If so, what is the most important aspect of this policy?  What is your reason?  Specifically, how does management communicate these policies to the staff?  How effective is this communication?

c. Other than policies, what tells you about the priority of nuclear safety at this site?  What is the message you receive?  Have any additional nuclear safety goals have been formulated for your work?  What are they?  Have you used them to guide your work?  If so, how?

d. Do you get sufficient and timely information about what’s going on at the site and in your department from your supervisor?  In the past year, can you think of an instance in which timely communication or lack of timely communication made a difference in how you perform your job?  Please describe.

e. Do decisions and actions at your level incorporate the safety goals we just discussed?  How?

f. Have you received training concerning safety policies?  Describe what it covered.  What did you think of this training (e.g. useful/not useful, effective/not effective)?  When did you last receive such training?  Have any of your other training courses referred to the priority of nuclear safety?  Which courses and which policies?   

g. How are the safety policies reinforced? (e.g., by management?, goals?, incentives?, accountability?)

h. In your experience, how well do the managers at this site communicate in a clear way that nuclear safety is a high priority?  Please provide an example of an effective communication.  Did it change anything about how you think about your work or how you do it?  If so, what changed?

i. If another department makes a change in how they perform their work that affects your work, are you informed of the change?  How?  Have there been instances where changes were made that directly impacted your job that you were not made aware of?  If so, please describe.

j. Is it communicated to personnel which procedures require verbatim compliance?  Are such procedures followed?  If not, please explain.

Respectful Work Environment

a. What are the organization’s policies regarding preventing and detecting retaliation and/or chilling effect?

b. Are you aware of any actions taken by your management to prevent and detect retaliation and/or a chilling effect?  If so, were their actions effective in addressing the situation?  Do you believe that management’s handling of the issues is consistent?

c. Are you aware of any instances in which another individual experienced a negative reaction for raising a safety issue?  If yes, please describe the incident, including any information conveyed by management concerning the incident. 

d. Are you aware of any specific instances in which another employee (or contractor) submitted an issue to the corrective action program or ECP and was retaliated against for pursuing the issue?  Please describe the situation.

e. Are you aware of your company’s policy with regard to protecting employees against retaliations/discrimination for raising nuclear safety concerns?  Does management tolerate retaliation of any kind for raising concerns?  Please explain.

Issue Date:  04/01/19	AppB-11	95003.02

f. How do you/your management prevent retaliation or the perception of retaliation associated with disciplinary actions or changes to the plant organization to ensure actions do not chill others (e.g., communicate reasons for discipline)?  (This question is for managers)

g. How do you ensure that you don’t discourage the reporting of issues when there is pressure to meet production goals?  (This question is for managers)

h. What actions can you take if an allegation of employment discrimination involving a contractor is made?

Questioning Attitude

a. Can you think of an instance where you raised a concern about a work project, work task, or organizational policy with your supervisor?  Where a coworker raised a concern with a work project, work task, or organizational policy with his/her supervisor?

b. If so, was your concern addressed to your satisfaction?

c. What is your organization’s policy when work is being performed and the work procedure cannot be executed as written?

d. Have you ever performed work and the work procedure could not be followed as written?  What did you do?  What did your supervisor direct you to do?

e. Are human error prevention tools used in your work?  If so, what are they?

f. Over the past year, have you encountered a situation where you or someone else was preparing to perform a job or was in the middle of performing it, and unexpected or uncertain conditions arose that were different from what you or the other person were led to expect?  If so, about how often?  Please provide examples and describe what you or the other individual did.

g. Do you personally have stop-work authority?  Have you used it or considered using it at any time within the past year?  If so, please describe the situation.  If you did stop work, what kind of reactions did you receive from co-workers?  Your immediate supervisor/manager?  Higher-level management, if they became involved?  Were you satisfied with your decision?  Would you do it again?

Decision Making

a. When a situation arises that requires a choice between nuclear safety and production how is the decision handled and who decides?  Can you give any examples of situations in which there may be a trade-off or even a conflict between nuclear safety and production in your job (e.g., there was pressure to meet a schedule goal, but you or someone you know identified a problem which would delay the work)?  Have you ever run into a situation like this?  If so, what did you do?  How did your supervisor react?  How did your management react?  How did it work out?  Would you do the same thing next time?  If not, what would you do differently?

b. In your own words, what does the term, “conservative decision-making,” mean?  Based on your definition, can you give any examples where you have demonstrated “conservative decision-making?”  If so, please describe.  What about your supervisor?  Your management?  Please describe.

c. What do you take into consideration when making a decision on whether a situation is safe (in terms of nuclear safety) to continue operation?  What about your supervisor?  Your management?

Issue Date:  04/01/19	AppB -12	95003.02

d. Do you receive information on the basis of management decisions related to safety-significant or risk-significant decisions?  If so, how would you characterize the level of information you receive (e.g., not enough, right amount, too much detail).  Please explain.  

e. What is the site process for making decisions related to safety-significant or risk-significant decisions such as whether a component is inoperable or whether a proposed design change assures safety?  Does the process use a systematic approach?  Is this process consistently followed?  If not, please explain.

f. Can you recall an instance in which decisions were not made at the appropriate organizational level, and, as a result, a nuclear safety issue arose?  Please describe.

g. Do management decisions regarding operational issues (such as changes to the scope of work or response to operational events) reflect the appropriate focus on safety?  If so, please give examples.

h. Do management decisions related to deferred outage items, corrective vs. elective maintenance, and treatment of emergent outage items reflect the appropriate focus on safety?  If so, please give examples.

Issue Date:  04/01/19	AppB-13	95003.02

Issue Date:  	95003.02

Appendix 95003.02-C

Guidance for Focus Groups and Individual Interviews

This enclosure provides information about the strengths and limitations of different interview types and methods.  It also provides guidance for conducting individual and group interviews.

This enclosure is arranged in the following five parts:

Note that both individual and group interviews may be either structured or unstructured.

The first four parts are subdivided into Overview, Strengths, and Limitations subsections.  Parts 1 and 2 also include a Guidance subsection.  Part 5 presents considerations in selecting samples of participants for interviewing.

1.	Individual Interviews

a.	Overview

Individual interviews consist of collecting information in a face-to-face, one-on-one setting where an interviewer poses a series of questions/topics to the interviewee and records the information provided.  The degree of structure in the questions/topics can vary.  (See parts 3 and 4.)

b.	Strengths

(1) Permits detailed exploration of an individual’s values, perceptions, attitudes, and views about the organization’s norms.

(2) Preferred method when information is more sensitive or personal and anonymity can be assured.

(3) Useful when additional probing on answers is required.

(4) Interviewer has good control over the session in terms of topics discussed and detail provided.

Issue Date:  04/01/19	AppC-1	95003.02

c.	Limitations

(1) Time-consuming and resource-intensive.

(2) Quality of interview data strongly influenced by interviewer skills.

(3) Interviews with approximately 10% of the population are needed to adequately understand existing issues.

(4) Information from a single interviewee cannot be considered unless validated through other sources of information.

d.	Guidance

Complete individual interviews as follows:

(1)	Note that the main objectives of the interview is to gain opinions on:

(a) the licensee’s actual awareness of nuclear safety issues;

(b) the safety-related attitudes of management and staff;

(c) the degree of compliance with policies and procedures;

(d) the possible reasons for observed inconsistencies or contradictions between actual and expected behavior, thus gathering data on social norms, beliefs, and values relevant to safety culture.

(2)	Prepare an interview plan in advance of the interview, and follow that plan to conduct the interview.

(3)	Conduct the interview in 3 stages, as follows:

(a) Establish a relationship of trust and cooperation with the interviewee

(b) Gather interview data

(c) Tell the interviewee that the results of all the interviews are captured and considered along with other results, and that observations, findings, and conclusions are documented in the Inspection Report.

(d) Discuss and evaluate issues raised by the interviewee

(4)	After the interview, prepare a written summary of the interview results, with emphasis on the issues raised by the interviewee and insights gained by the safety culture assessor.

Issue Date:  04/01/19	AppC-2	95003.02

2.	Group Interviews

a.	Overview

Group interviews (i.e., focus groups) consist of collecting information in a face-to-face, group setting, where an interviewer poses a set of questions/topics to the participants and records the information provided.  The degree of structure in the questions/topics can vary.

b.	Strengths

(1) Group interaction can prompt/sustain discussions without a high level of interviewer input.

(2) Efficient - requires fewer team resources than individual interviews to obtain adequate sample size.

c.	Limitations

(1) Should not be used (1) for sensitive or personal topics, (2) when there is evidence of conflict within or between work groups, or (3) when the participants have concerns about anonymity and confidentiality of responses.

(2) Interviewer does not have a high degree of control over the session; time may be lost on irrelevant issues.  Participants will react to others’ statements in addition to the interviewer’s questions.

(3) The interview session may be dominated by the views of a minority of the participants.

(4) Participants may feel pressure towards consensus.

(5) Qualitative data may be difficult to analyze.  

d.	Guidance

(1)	The typical amount of time to conduct the focus group interview will range from 90 – 120 minutes.  Open each focus-group interview by introducing the safety culture assessors who are in the focus group.  Ask the participants to introduce themselves (first names should be sufficient) and describe how long they have worked in their current organization, as well as, on site in total.  This information (time in current organization and time on site) will provide context to the data collected during the focus group interview.  Verify that the attendees are as selected by the safety culture assessors.  Question the inclusion of substitutes.  If supervisory or management personnel are among the working-level attendees, ask them to leave and invite them to meet privately with the safety culture assessors at another time or check if they are in the supervisory focus groups.  Then tell participants:

Issue Date:  04/01/19	AppC-3	95003.02

(a) that the purpose of the focus group is to determine whether and how underlying issues contributed to the performance deficiencies which prompted this inspection;

(b) how interview participants were selected;

(c) that the focus group will consist of discussions prompted by questions;

(d) indicate that, to the extent possible, information obtained during the inspection will not be attributed to individual participants who are interviewed by the NRC.  Remind the participants that the discussion is occurring in a group so others will hear what is said.  Also, state that if an individual provides details about a specific event in which he or she was uniquely involved, and if the NRC includes those details in their writeup of this inspection, then those details may identify the participant who provided the details;

(e) all records held by the NRC of people participating will be destroyed after the focus group; that the NRC expects that cooperating with the NRC inspection, including participating in the focus group should not be used in any way to threaten, punish, or retaliate against an individual.  The safety culture assessor should explain that if the individual wishes to discuss issues in a separate private interview, the safety culture assessors or residents if the safety culture assessors are no longer onsite, will make themselves available.  Such concerns may be handled as allegations (e.g., claims of retaliation or wrongdoing) or included as input into the ongoing inspection, as appropriate.

(f) that the results of all the interviews are captured and considered along with other results, and that observations, findings, and conclusions are documented in the Inspection Report.

(2)	From the set of questions developed or selected for this interview (see parts 3 and 4), ask one of the questions and invite participants to respond.  Ask followup questions as necessary to clarify responses and encourage discussion from other participants.  Continue followup questions until the group’s response is complete and understood, and the team has obtained from the participants as many insights into safety culture traits as are reasonably associated with that question.  After that, ask another question.

(3)	Continue question-and-answer conversations as described above, for as long as participants willingly respond, or until the planned questions are all asked, subject to the following:

(a) Encourage all participants to speak up.  Ask the same question or a variation of it to at least one other person.

Issue Date:  04/01/19	AppC-4	95003.02

(b) If any participant(s) does not answer any question directed to the group, direct a question or two to that participant, and encourage him or her to say what he or she thinks.

(c) If any participant(s) appears reluctant to answer a question or if his or her answers suggest an underlying issue or concern, make note of the question and the answers (if provided), for later follow-up.  Do not persist if someone shows continued reluctance to speak.

(4)	Close the focus group by thanking the participants for their participation.  Tell them that if:

(a) they have anything else to say that they did not feel comfortable saying during the session, 

(b) they later want to clarify or revise something they said during the focus group,

(c) they later remember something they wish they had said during the focus group, or

(d) they want to talk about something that wasn’t discussed during the focus group, then they should contact the safety culture assessors, and tell them how to do that.

(5)	After closing the focus group, review the focus group notes, and add relevant safety culture assessor observations and comments.

(6) If any participant’s response identified a concern or issue related to a safety culture trait, or if any discussion suggested or otherwise indicated a concern or issue related to a safety culture trait, then address that concern or issue as described in 02.09.	

		NOTE:  Information that reflects negatively on licensee performance or safety culture, if provided in the context of the purpose of the inspection, is not an allegation.  However, information that describes an inadequacy in licensee performance, which is specific and outside the scope of the inspection, may be an allegation if the NRC has not already assessed the validity of the issue.  Any issues related to wrongdoing, whether provided by licensee management, licensee employees or contractors, or NRC staff, are allegations.  For more specific guidance, refer to Management Directive 8.8, “Management of Allegations,” or contact the appropriate regional allegation coordinator.

Issue Date:  04/01/19	AppC-5	95003.02

3.	Structured Interviews

a.	Overview

Structured interviews consist of using a pre-defined set of questions that are consistently asked of each interviewee or of subsets of interviewees.  Can be done in a face-to-face or group setting.

b.	Strengths

(1) Ensures similar topic areas are explored across multiple interviewees.

(2) Reduces differences in the process followed across interviewees and interviewers that could bias the results.

(3) Semi-structured interviews still allow follow-up questions and more in-depth probing of a topic.

(4) Ensures all key topics are addressed during the allocated interview time.

(5) Provides a basis for comparison between respondents.

c.	Limitations

(1) If conducted as a fully structured interview, does not allow for additional follow-up on issues that arise.

(2) The question sets used require careful consideration.

4.	Unstructured Interviews

a.	Overview

Unstructured interviews consist of an interviewer asking interviewees a series of questions that are developed as the interview is conducted.  They can be done in a face-to-face or group setting.

b.	Strengths

(1) Gives interviewer complete freedom to fully explore topics of interest that arise.

(2) The interviewee largely guides the interview in terms of discussion topics.  This may lead to the identification of previously unknown issues.

c.	Limitations

(1) Provides no basis for comparison between respondents.

Issue Date:  04/01/19	AppC-6	95003.02

(2) The interview process is more likely to be influenced by the style and biases of interviewer.

(3) Information collected may not be highly relevant to the assessment.

(4) No framework is available to guide the interviewer.

5. 	Sampling Guidance

a.	Develop a sampling plan that is informed by the findings of the licensee’s independent safety culture assessment and review of background material.

b. 	Include all functional groups at the site and possibly some corporate functional groups in the sampling plan.

c. 	Select licensee and contractor participants using a stratified random distribution (as described in (1) and (2) below) of personnel from organizational rosters, focused on specific groups as appropriate.  Nominally: 

(1) For the work group(s) most closely associated with the performance deficiencies, plan to interview approximately 20% of the working- and supervisory-level personnel.

(2) For the major functional groups that were not closely associated with the performance deficiencies, plan to interview approximately 10% of the working- and supervisory-level personnel.

(3) Plan to interview all heads of functional groups within the licensee’s management structure, and all managers organizationally above those individuals.

(4) Work with the licensee to select permissible substitutes for selected participants, based on unavailability due to shift work, vacation, sickness, or press of duty.  

Issue Date:  04/01/19	AppC-7	95003.02

Issue Date:  		95003.02

Appendix 95003.02-D

Guidance for Structured Behavioral Observations

This enclosure provides guidance for developing and using structured behavioral observation checklists to identify patterns of behavior related to the safety culture traits.  Note that behavioral observation checklists may not be appropriate for all of the safety culture traits because some are process-type traits and don’t lend themselves well to behavioral observation.

1.	Overview

Behavioral Observation Checklists involve the use of a structured format to record observational data.  Key observable attributes of behaviors associated with safety culture are listed in checklist fashion, which ensures structured collection of data associated with observations.  The structure also allows quantification of observational information.  Behavioral Observation Checklists may also be used to guide and focus observations without quantifying the information collected.

2.	Strengths

a. Data collected reflect real activities (versus respondent opinions or perceptions).

b. Multiple observations of similar activities (e.g., turnovers) allow quantification of information across multiple occurrences of the activity.

c. Observer is non-intrusive and does not interrupt activity.

d. Checklist format ensures similar information will be collected across multiple observers.

e. When quantitative data are not obtained or cannot be reported due to limited observations, qualitative data can be useful.

3.	Limitations

a. Observer’s presence may affect the manner in which the activity is conducted.

b. In some cases, multiple observations of a similar activity are not possible.

c. Unless multiple observations of a similar activity are conducted, quantitative data cannot be reported.

d. Those observed may avoid discussing any sensitive topics in the presence of the observer.

4.	Applications

To be completed when observing:

a. licensee decision-making processes, including goal-setting, oversight, and work planning sessions; 

Issue Date:  04/01/19	AppD-1	95003.02

b. the actual performance of work activities, including activities for which formal procedures and standards of behavior exist; 

c. communications, including interactions between managers and staff, between peers, as well as interdepartmental, intradepartmental and external communications; and

d. training.

5.	Guidance:

a. Identify the categories of activities that will be observed.  Select activities to observe based on their relevance to specific safety culture traits to be assessed with this data-collection method, as defined in the assessment plan.

b. Through discussion with knowledgeable licensee personnel, identify the frequency with which the selected activities typically occur and determine the number of observations to be scheduled for each category of activity.  If structured behavioral observation will be the primary method of collecting data about a specific safety culture trait (e.g., decision making), plan to observe a minimum of 25 activities of interest over the course of a one-week inspection.  A minimum of 15 observations may be sufficient if behavioral observation will be used as a supplement to other information-gathering methods.

c. To develop consistency in using the checklists among different observers, 

1. discuss the checklist items in advance and determine how they will be used; 

2. jointly observe several of the same activities;

3. compare the results obtained by the different observers when observing the same activity;

4. discuss and resolve any differences in how the checklist items were interpreted;

5. revise the checklist items, as necessary.

d. For each category of activity to be observed, select a subset of the checklist items below or develop additional items, based on the nature of the activity and the safety culture traits to be assessed.  Do not plan to collect data about all of the safety culture traits from any one observation, because the behaviors associated with some safety culture traits do not occur with sufficient frequency to be provide an adequate sample of observations (e.g., budget planning meetings involving corporate and site management that might provide insights related to the Leadership Safety Values and Actions trait).  

e. Include no more than 15 items on a single checklist.  Longer checklists are difficult to use and searching for items on the checklist can distract the observer.

f. Use the same checklist items when observing activities that fall into the same category of activities, so that the frequencies of the behaviors of interest can be determined.

Issue Date:  04/01/19	AppD-2	95003.02

g. For activities performed frequently during the inspection (e.g., shift turnover, pre-job briefs, and surveillance and maintenance activities), plan to observe up to 25 of the activities during the inspection.  

h. For infrequently performed activities (e.g. weekly management/staff meetings, all-hands meetings, personnel action meetings) plan to observe a sample of convenience (i.e., perform the observation if one occurs during the inspection and if safety culture assessors are available at the time.)

i. Maintain the checklists used for each observation, even if no data were collected, in order to document the sample size.

j. For each checklist created, the safety culture assessor should note: 

1. the date and time of the observation; 

2. the activity observed (e.g., pre-job briefing, shift turnover, plan-of-the-day meeting, department meetings, a maintenance job, corrective action review meeting); 

3. the levels of management and staff involved (e.g., senior management, functional area management, middle management, first-line supervisors, staff or contractors);  

4. the functional area(s) involved (e.g., operations, maintenance, radiation protection, engineering);

5. the number of individuals involved, and 

6. other characteristics of the activity that can be used to compare and contrast data collected from different activities.

k. Provide space on the checklist for the safety culture assessor to add notes that record more details about the interactions observed.  For example, one of the checklist items below asks, "Was risk or nuclear safety discussed?"  If the answer is yes, the safety culture assessor should add a description of the context in which risk or safety was discussed, the extent of the discussion, and an assessment of it.  However, the additional information should be recorded only after the observation is completed, in order to ensure that the safety culture assessor is not distracted from observing.

l. Following the observation, the safety culture assessor should also document any qualitative assessment of the interaction or work activity observed, related to the safety culture traits.  This information will be necessary to ensure that the observation data are appropriately interpreted.

m. When all observations have been completed, summarize the following:

1. The number of observations made of each category of activity; 

Issue Date:  04/01/19	AppD-3	95003.02

2. The extent to which behaviors were observed that are consistent with the safety culture traits;

3. The extent to which behaviors were observed that are inconsistent with the safety culture traits; and

4. Any qualitative information necessary to interpret properly the quantitative data.

This information can then be used to assess how the safety culture traits are integrated into day-to-day activities.  This information is useful in assessing the overall safety culture as well as the safety culture of individual functional groups.

Example checklists are included on the following pages.

Issue Date:  04/01/19	AppD-4	95003.02

Leadership Safety Values and Actions Checklist

(Observed during ongoing work activities.)

Are the personnel who are performing the activities given specific success criteria that define organizational expectations before beginning the work? 

 Yes  No  N/A____

If yes, nuclear safety was was not  among the expectations.

Is performance feedback timely, so that corrections in performance can be achieved?  

Yes  No  N/A	

If yes, did any feedback concern nuclear safety?  Yes  No  N/A 

Is performance feedback available from verbal communication or performance evaluation reports generated at a later date ?

If yes, did any feedback concern nuclear safety?  Yes  No  N/A 	

Did any supervisor offer performance feedback related to nuclear safety?  

Yes  No N/A

Did any manager offer performance feedback related to nuclear safety?  

Yes  No N/A 

Did any peers offer performance feedback related to nuclear safety?

Yes  No N/A

If it was necessary to deviate from the originally planned activities, did the personnel performing the activities have the authority to approve the deviation?  

Yes  No  N/A 

If yes, did the deviation have nuclear safety implications? 

 Yes  No  N/A

If the work is being performed by a crew, is there an obvious structure to the group (i.e., there is a clearly identified group leader and specified roles and responsibilities for each of the other group members)?  Yes  No N/A 

Were the personnel selected to perform the activities familiar with the task requirements  or was there obvious uncertainty regarding the tasks to be performed?

	Yes  No N/A ____

Did personnel have problems reading the work package (legibility)?  

	Yes  No N/A ____

Issue Date:  04/01/19	AppD-5	95003.02

Did personnel have problems interpreting the information in the work package?

	Yes  No N/A____

Was any information missing from the work package?  Yes  No N/A ____

Were an adequate number of staff available to perform the work? 

	Yes  No N/A ____

Were the procedures adequate to perform the work?  Yes  No N/A 

Did personnel have the equipment necessary to perform the work safely?

	Yes  No N/A ____

(To be observed during meetings.)

Were the specific individuals responsible for implementing the initiative, project, or program under discussion present?  Yes  No  N/A

Was the individual given an opportunity to present discuss or defend his or her position?  

Yes  No  N/A	

If the responsible individual was present, did s/he receive any feedback related to nuclear safety?  Yes  No N/A

If yes, was the feedback provided by (check all that apply):	

 Peers

 Supervisor

 Manager

If the responsible individual was present, did s/he receive any feedback related to deadlines, costs, quality or other performance criteria?  Yes  No N/A 	

If yes, was the feedback provided by (check all that apply):	

 Peers

 Supervisor

 Manager

Issue Date:  04/01/19	AppD-6	95003.02

Problem Identification and Resolution Checklist

(Typically observed during issue screening, management screening or closure meetings)

Which of the following individuals participated in the meeting?

Corporate management ____

Senior management ____

Functional area management ____

Middle management ____

Licensee staff ____

Contractor ____

Other (describe)?  ____

Were screening criteria used?  Yes ____ No ____ N/A ____

Were the screening criteria conservatively applied for every issue?  

Yes ____ No ____ N/A ____

Did anyone challenge how any of the criteria were being applied?	

Yes ____ No ____ N/A ____

Did anyone challenge the prioritization of any issues?  

Yes ____ No ____ N/A ____

Were any issues upgraded or downgraded in priority?  Yes ____ No ____ N/A ____

How many upgraded?  ____ Downgraded?  ____

If so, did anyone challenge the change(s)?  Yes ____ No ____ N/A ____

Were issues thoroughly discussed?  Yes ____ No ____ N/A ____

Was safety, as applicable, considered for every issue?  Yes ____ No ____ N/A ____

Were there any issues where it was decided not enough information was available to make the prioritization?  Yes ____ No ____ N/A ____

If so, were any individuals directly involved with the issue consulted or plans made to consult the individuals involved?  Yes ____ No ____ N/A ____

Did the reviewers have an understanding of the evaluation (i.e., they reviewed the evaluation prior to the meeting)?  Yes ____ No ____ N/A ____

Did the reviewers place safety as the highest priority?  Yes ____ No ____ N/A ____

Was there any discussion about the scope of the evaluation (i.e., what areas the evaluation covered)?  Yes ____ No ____ N/A ____

Issue Date:  04/01/19	AppD-7	95003.02

Were the issues investigated to an appropriate extent? 

Yes ____ No ____ N/A ____

Did any reviewer raise any concerns about problems not being adequately investigated in the evaluation?  Yes ____ No ____ N/A ____

Was there any discussion on if the corrective action(s) presented could resolve all the problems identified in the evaluation?  Yes ____ No ____ N/A ____

Did any reviewer interact with the evaluator(s) of the issue?  

	Yes ____ No ____ N/A ____

If so, did the reviewer(s) behave at any point in a way that could potentially discourage the evaluator from performing a thorough/in-depth investigation in the future?  Yes ____ No ____ N/A ____

Were there any evaluations not accepted by the reviewers? 

	 Yes ____ No ____ N/A ____

Is yes, was resolution on what to do about the evaluation reached?

Yes ____ No ____ N/A ____

If yes, was it through consensus-seeking ____ or top-down direction from management ____?

If no, was it decided to push the decision up the management hierarchy ____ or not ____?

If it was determined that the evaluation should have any rework done: 

Was guidance provided on how to improve the evaluation?

Yes ____ No ____ N/A ____

Were any additional resources (e.g., training, additional evaluators, management assistance) offered to the evaluator(s)?  

Yes ____ No ____ N/A ____

Were there any concerns raised about the new deadline?  

Yes ____ No ____ N/A ____

If yes, who raised the concern?  Reviewer(s) ____ Evaluator(s) ____

Issue Date:  04/01/19	AppD-8	95003.02

Personal Accountability Checklist

(observed during meetings, work tasks, work preparation)

Which of the following individuals participated?

Corporate management ____

Senior management ____

Functional area management ____

Middle management ____

Licensee staff ____

Contractor ____

Other (describe)?  ____

Did everyone involved with the activity demonstrate an understanding of the safety significance of the activity?  Yes      No

Did everyone demonstrate that they were aware of their role in the activity and its impact on safety?  Yes     No

Did different organizations/work groups effectively communicate their roles in coordinating the activity?  Yes     No

Were differences of opinion satisfactorily resolved?  Yes    No

Issue Date:  04/01/19	AppD-9	95003.02

Work Processes Checklist

(Observed during ongoing work activities or a work planning session.)

When planning a work activity, were the following issues discussed (check all that apply)?

risk insights

defense in depth

job site conditions that could impact human performance

task sequencing to optimize system availability

potential impacts on nuclear safety of performing the activity at the same time as other activities are performed

contingencies

compensatory actions

conditions under which the work would need to stop for nuclear safety reasons

the impact on nuclear safety of any temporary modifications to be installed

the impact on human performance of any operator work-arounds to be created

any relevant internal or external operating experience

A pre-job briefing was  was not conducted.  If it was conducted, were the following issues discussed (check all that apply)?

risk insights and/or nuclear safety considerations

defense in depth

job site conditions that could impact human performance and means to mitigate       their potential effects

contingencies for mitigating the effects of mistakes and/or possible worst-case    scenarios

procedure usage requirements

other work activities that have the potential to interact with this one

conditions under which work would be stopped for safety reasons

communications requirements

applicable lessons learned from internal or external operating experience

When performing a work activity simultaneously with other work activities that had the potential to interact, communications were/were notmaintained between the individuals/groups performing the different activities.

When performing the work activity, unexpected conditions      did/did not arise.

Are there obvious time pressures for work completion?  Yes  No N/A ____

If obvious time pressures exist:

Do they appear reasonable given the activities to be performed?  

	Yes  No N/A____

Issue Date:  04/01/19	AppD-10	95003.02

	  Is there evidence that those pressures compromised the quality of the work performed in 

      any way?  Yes  No N/A____

Is there evidence that those pressures compromised the safety of the work performed 	in any way?  YesNo N/A____

Were time constraints for the work activities clearly communicated to all individuals involved in the activity?  YesNo N/A ____

The reason for the time constraints is related to (check all that apply):

nuclear safety concerns

limited personnel resources

other scheduled work activities

pressure to get the facility back on-line

other/unknown

Human error prevention techniques were  were not  used.

Human error prevention techniques were  were not  discussed during the pre-job brief.

Were procedures used in performing the activity?  Yes  No N/A____

If procedures were used, were they conveniently located and easily accessible?  

Yes  No N/A____

Verbatim compliance with the procedures was  was not  required.

	If verbatim compliance was required, was it achieved?  Yes  No N/A ____

If verbatim compliance was not achieved, 

(Note - these items relate to Resources.)

was it because the activities described by the procedure could not be performed as written, given the conditions (e.g., time constraints, personnel resources, unexpected conditions)?  Yes   No N/A ____

was it because the procedures not well understood or understandable?  

		Yes No N/A ____

The formal process for deviating from a procedure was  was not  followed.

Were any problems encountered during performance of the work activities?  

Yes  No N/A____

If yes, did the problems have any nuclear safety implications?  

YesNoN/A ___

Issue Date:  04/01/19	AppD-11	95003.02

	  Work was  was not  stopped until the problem was resolved.

If a management decision or additional expertise was required to solve the problem, were the necessary individuals made available within a reasonable time period? 

  Yes NoN/A____

Did any personnel point out *conditions*that could adversely impact nuclear safety?

Yes  No N/A____

Did any personnel point out *behaviors* that could adversely impact nuclear safety?

Yes  No N/A____

Were any work-arounds used?  Yes  No N/A____

If yes, was the work-around long-standing or created for the current work 

activity? 

Was it proceduralized?  Yes        No       N/A ____

If the work activity was considered critical, was management present?

Yes  No N/A____

If yes, did management offer direction  or feedback  ?

		Was the direction or feedback related to nuclear safety? 

		Yes  No N/A____

Issue Date:  04/01/19	AppD-12	95003.02

Continuous Learning Checklist

(When observing training.)

Is the training a result of an event or incident that occurred at the facility due to a human performance problem?    Yes  NoN/A ____

Do trainees appear hesitant to ask questions or seek clarification?  

	Yes  NoN/A ____

Do trainees appear to be engaged?  Yes  No N/A ____

Do trainees have an opportunity to offer feedback about the training?  

	Yes  No N/A ____

Are trainees evaluated at the completion of training?  

	Yes  No N/A____

Are trainees provided with feedback while the training is ongoing?  

	Yes No  N/A____

Are lessons learned from internal or external operating experience incorporated into the training? 

	 Yes  No  N/A ____

Is nuclear safety addressed during the training?  Yes  No  N/A ____

Issue Date:  04/01/19	AppD-13	95003.02

Environment for Raising Concerns Checklist

(Observed during any interactions among site personnel.)

Did a subordinate(s) ask any questions of a superior during the interaction?  

Yes  No  N/A____

If yes, did the superior answer the question(s)?  Yes  No  N/A ____

Did a subordinate(s) raise any concerns to a superior during the interaction?   

Yes  No  N/A____

If yes, did the concerns involve (check all that apply):

nuclear safety

radiological or industrial safety

resources (e.g., staff, expertise)

scheduling or deadlines

other

If yes, did the superior address the concerns?  Yes  No  N/A ____

If yes, did the superior resolve the concerns?  Yes  No  N/A____

If yes, was the supervisor’s response open and non-defensive?  

	Yes  No  N/A____

Did a subordinate offer any suggestions to a superior during the interaction?

	Yes  No  N/A ____

If yes, did the superior discuss the suggestion(s)?  Yes  No  N/A ____

Was the interaction obviously strained , obviously pleasant , or was there no apparent affect ?

Was the interaction related to a safety issue , regulatory requirement(s) , production issue(s) , personal conflict , other ?

Did the interaction include discussion of ways to improve the facility performance?  

	Yes  No  N/A ____

Did the interaction include discussion of ways to improve personnel performance?

	Yes  No  N/A____

Did any staff member self-report an error?  YesNo N/A____

If yes, did peers react favorably?  Yes NoN/A ____

If yes, did supervisor(s) react favorably?  YesNo N/A____

Issue Date:  04/01/19	AppD-14	95003.02

Effective Safety Communication Checklist

(Typically observed in scheduled meetings.)

Was nuclear safety discussed as a goal?  Yes  No  N/A____

Were goals other than nuclear safety discussed?  Yes  No  N/A____

Goals     were/were not  prioritized?

Nuclear safety    was/was not assigned the highest priority.

Were any target levels attached to the goals?  Yes  No  N/A____

If goals were being set on a departmental level, were overall organizational goals factored in?  Yes  No  N/A ____

If yes, nuclear safety      was/was not  one of the goals.

If goals were being set on an organizational level, were corporate goals factored in?  Yes  No  N/A 

If yes, nuclear safety was  was not  one of the goals.

Was there overall agreement among the individuals setting the goals on what the goals and priorities should be?  Yes  No N/A ____

Was there any indication that the goals of different departments were in conflict?  

	Yes  No  N/A____

If nuclear safety goals were discussed, the following individuals brought them up:

 Corporate management

 Senior management

 Functional area management

 Middle management

 Licensee staff

Contractor

Other (describe)

If production goals were discussed, was the potential impact on nuclear safety mentioned?

	Yes  No  N/A ____

Issue Date:  04/01/19	AppD-15	95003.02

Respectful Work Environment Checklist

(Observed during management or oversight meetings.)

Was there a rigorous investigation of the potential issue?  Yes No N/A____

Did the disposition seem appropriate?  YesNo_N/A____

Was the potential for the action to discourage the reporting of concerns discussed?

	YesNo_N/A____

If yes, mitigation actions werewere notassigned.

Issue Date:  04/01/19	AppD-16	95003.02

Questioning Attitude Checklist

(observed during meetings, work tasks, work preparation)

Which of the following individuals participated?

Corporate management ____

Senior management ____

Functional area management ____

Middle management ____

Licensee staff ____

Contractor ____

Other (describe)?  ____

Did anyone involved in the activity challenge the assumptions of the activity leader?  Yes   No   N/A

Were the risks associated with the activity discussed?  Yes   No   N/A

Were abort criteria discussed?  Yes   No   N/A

Did anyone raise a concern about proceeding with the activity?  Yes   No   N/A

If so, were the concerns adequately addressed by the participants prior to conducting the activity?  Yes     No     N/A

Were error prevention techniques discussed prior to the activity and implemented during the activity?  Yes    No   N/A

Issue Date:  04/01/19	AppD-17	95003.02

Decision Making Checklist

(May be observed in scheduled or informal meetings or during ongoing work activities.)

**

Did the decision involve technical , policy, or personnel issues? 

Were any uncertainties discussed?  Yes  No  N/A ____

Alternatives     were/were not     generated.

Was “risk” or nuclear safety discussed?  Yes  No N/A ____

Were conservative assumptions used?  Yes____No____ N/A ____

Were any alternatives rejected because of risk or nuclear safety considerations?  

	Yes  No  N/A____

Resolution      was/was not      reached.

If resolution was reached, was it through consensus-seekingor top-down direction from management ?

If resolution was not reached, was it decided to push the decision up the management hierarchy  or not ?

If resolution was not reached, was it decided to seek more information 

or not ?

If nuclear safety was involved, was the decision based on sufficient evidence that it was safe to proceed?  YesNoN/A____

If nuclear safety was involved, was the decision based on sufficient evidence that it was unsafe to proceed?  YesNoN/A____

If the decision concerned policies, rules, and goals, did the manager consult with his/her immediate subordinates?  Yes No N/A ____

If the decision concerned staffing, did the manager consult with his/her immediate subordinates?  Yes  No  N/A____

If the decision concerned a technical issue, did the manager consult with any technical staff?  	Yes  No N/A____

If the decision concerned how to solve a work-related problem, did the individual consult his/her superior?  Yes No  N/A ____

Issue Date:  04/01/19	AppD-18	95003.02

Was a plan made for communicating the results of the decision? 

 	YesNo  N/A____

If yes, was communicating with the affected individuals discussed?  

	 Yes No  N/A ____

If yes, was communicating with a higher management level discussed?  

Yes No  N/A____

Were any previous, similar decisions discussed?  Yes  No  N/A____

If yes, was the effectiveness of the previous decision discussed?  

		Yes  No  N/A ____

(Observed during ongoing work activities.)

Are the personnel who are performing the activities given specific success criteria that define organizational expectations before beginning the work? 

 Yes  No  N/A____

If yes, nuclear safety was was not  among the expectations.

Is performance feedback timely, so that corrections in performance can be achieved?  

Yes  No  N/A	

If yes, did any feedback concern nuclear safety?  Yes  No  N/A 

Is performance feedback available from verbal communication or performance evaluation reports generated at a later date ?

If yes, did any feedback concern nuclear safety?  Yes  No  N/A 	

Did any supervisor offer performance feedback related to nuclear safety?  

Yes  No N/A

Did any manager offer performance feedback related to nuclear safety?  

Yes  No N/A 

Did any peers offer performance feedback related to nuclear safety?

Yes  No N/A

If it was necessary to deviate from the originally planned activities, did the personnel performing the activities have the authority to approve the deviation?  

Yes  No  N/A 

If yes, did the deviation have nuclear safety implications? 

 Yes  No  N/A	

Issue Date:  04/01/19	AppD-19	95003.02

If the work is being performed by a crew, is there an obvious structure to the group (i.e., there is a clearly identified group leader and specified roles and responsibilities for each of the other group members)?  Yes  No N/A 

Were the personnel selected to perform the activities familiar with the task requirements  or was there obvious uncertainty regarding the tasks to be performed ?

Did personnel have problems reading the work package (legibility)?  

	Yes  No N/A ____

Did personnel have problems interpreting the information in the work package?

	Yes  No N/A____

Was any information missing from the work package?  Yes  No N/A ____

Were an adequate number of staff available to perform the work? 

	Yes  No N/A ____

Were the procedures adequate to perform the work?  Yes  No N/A 

Did personnel have the equipment necessary to perform the work safely?

	Yes  No N/A ____

(To be observed during meetings.)

Were the specific individuals responsible for implementing the initiative, project, or program under discussion present?  Yes  No  N/A

Was the individual given an opportunity to present discuss or defend his or her position?  

Yes  No  N/A	

If the responsible individual was present, did s/he receive any feedback related to nuclear safety?  Yes  No N/A

If yes, was the feedback provided by (check all that apply):	

 Peers

 Supervisor

 Manager

If the responsible individual was present, did s/he receive any feedback related to deadlines, costs, quality or other performance criteria?  Yes  No N/A 

If yes, was the feedback provided by (check all that apply):	

 Peers

 Supervisor

 Manager

Issue Date:  04/01/19	AppD-20	95003.02

Appendix 95003.02-E

Guidance for Safety Culture Event Follow-up Studies

This enclosure provides guidance for selecting and performing event follow-up studies to identify consistencies in attitudes and behaviors related to the safety culture traits.

1.	Overview

An event follow-up study is an in-depth investigation and analysis of an organizational event (e.g., a high-visibility disciplinary action, a significant management change, a human performance problem that resulted in an operational event) or organizational condition (e.g., weaknesses in the safety culture traits).  Event follow-up studies provide an opportunity to trace the progression of a single event, or the development of an organizational condition, using multiple methods, to observe how organizational behaviors impact the facility’s ability to cope with that event or condition.

2.	Strengths

a. Allows for a thorough examination of a particular situation.

b. Results are documented in a narrative format providing valuable examples to support the overall findings of the assessment.

c. Most effective when the activity to be tracked is identified early in the assessment and at the beginning stages of the activity, although retrospective analyses are possible.

3.	Limitations

a. Results cannot be generalized beyond the single situation studied.

b. Requires sufficient time devoted by one to two safety culture assessors, detracting from time available for other assessment activities.

c. Detailed information on the organization’s assessment of the activity or event may not yet be available.

d. If the event follow-up study requires retrospection, biases may be introduced by the effects of intervening events on individuals’ memories.

4.	Example Applications

a. Understanding the history of a particular functional group or specific work unit that may be demonstrating weaknesses in one or more safety culture traits to identify the causes of the weaknesses and the effectiveness or ineffectiveness of the licensee’s corrective actions.

Issue Date:  04/01/19	AppE-1	95003.02

b. As part of evaluating the licensee’s decision-making processes by identifying the patterns of thinking and behaving that led to a specific decision.

c. As part of evaluating the licensee’s effectiveness in maintaining an environment for raising concerns by studying organizational events that did or did not create a chilling effect.

5.	Guidance

a.	Identify the organizational event or condition to be studied.

(1)	Significant events and conditions that will provide useful information about the safety culture traits can be identified from the team’s other assessment and inspection activities, including individual and group interviews, the review of issues entered into the corrective action program, as well as the review of allegations, previous inspection reports, and licensee self-assessments.

(2)	Operational events are also typically organizationally meaningful and understanding the management, organizational, and human performance causes and contributors to the events, as well as the event’s organizational consequences, often provides useful information about the safety culture traits.

b.	Use a combination of interviews, document reviews, and observations, if possible, to obtain a complete understanding of why and how the event or condition occurred and its relationship to the safety culture traits.  Investigating and analyzing a single event or condition often provides information related to multiple safety culture traits.

c.	Ensure that the information obtained that is related to the safety culture traits is shared within the team.

4

Issue Date: 10/26/06	95003

Issue Date:  04/01/19	AppE-2	95003.02

Appendix 95003.02-F

Guidance for Evaluating Safety Culture Surveys

This enclosure provides safety culture assessors with guidance for evaluating a safety culture survey that was administered by a licensee.  (The NRC’s safety culture assessment will not include the use of surveys.)

Method:  Quantitative surveys**are structured, written questionnaires, administered to respondents.  Questions are close-ended (require a single answer with no explanation) and require respondents to select the best answer from the several options provided.  Answers given can be transformed into numerical information for statistical analysis.

Strengths:  

•	Can be administered to a very large sample or entire population.

•	Can provide precise and quantitative data.

•	Usually quick and easy to complete, depending on questionnaire length.

•	Data can be rapidly analyzed.

•	Respondents remain anonymous while information on general demographic characteristics can be collected.

•	When completed by a representative sample can provide precise and reliable information on total population and subpopulations.

•	Some reliable and valid surveys already exist in the survey and public opinion research industries.

Limitations: 

•	Not effective for exploring complicated/ambiguous issues.

•	Managers can be strongly influenced by statistics. 

•	Results can be misleading, especially if the design, application, or interpretation of the questionnaire is less than satisfactory.

•	Requires large sample sizes to draw valid conclusions, make valid comparisons, and assure statistical validity across the population and subpopulations selected.

Guidance

1. Review the questions used, as follows, to determine whether:

(Note - the criteria listed for this step can also be used to evaluate questions used by the licensee for interviews or focus groups.)

14

Issue Date: 10/26/06	95003

Issue Date:  04/01/19	AppF-1	95003.02

a. Question wording is simple.  Questions avoid technical or specialized words, unless the participants are highly familiar with them.

b. Sentences are short.

c. No ambiguous words or equivocal sentence structures.

d. Times and places and frequencies are specified, even if they are usually assumed.

e. Questions do not include double negatives.

f. Questions address only one topic at a time; questions are not embedded within questions.

g. Questions are unbiased and not leading (i.e., wording does not lead the respondent to answer one way rather than another or place the respondent in a double-bind where no answer accurately reflects his or her situation).

h. Each question is necessary and provides additional, useful information.

i. Related questions are grouped.

j. Questions are sequenced so that one question or line of questioning does not influence responses to subsequent questions.

k. Questions flow from the general to the more specific.

l. Questions flow from the least sensitive to more sensitive topics.

m. Initial questions address screening and rapport-building topics before specific questions.

n. Unique or unusual questions are prefixed with an explanation to avoid confusion.  For example, terms used in the questions, such as ”your supervisor“ or ”management“ should be defined, as well as any terms that may be unfamiliar to the participants, such as “SCWE.”

2. Through interviews and document reviews, evaluate whether the survey was developed in accordance with standard practices.  Determine whether:

a. The survey questions were pilot-tested with respondents who were representative of the intended participants.

b. Problematic survey questions were revised, on the basis of pilot test results.

c. The revisions were again pilot-tested with representative respondents.

d. The survey developer assessed test-retest or split-half reliability of the survey instrument.

e. The survey has been previously used at the licensee’s facility, or in other organizations.  Evaluate any evidence provided by the licensee that indicates whether the previous results were valid and accurately identified strengths and weaknesses that could be verified from other sources of information.

3. Evaluate the procedures used to administer the survey to determine whether they were systematic and were unlikely to have biased the responses.  Determine whether:

a. The methods used to select the sample of participants assured representativeness.

b. Questionnaires were administered in a consistent location under a consistent set of conditions.  If the survey was administered at different locations or an online survey was used, determine whether the instructions to participants and other means were used to minimize the potential spurious effects of such differences on the data.

c. Participants were monitored while taking the survey by the survey administrators and survey administrators were available to answer questions.

d. Participants had sufficient time to complete the survey.

e. All individuals in the sample had an equal opportunity to participate (e.g., accommodations were made to permit backshift personnel to participate).

f. Licensee supervisors or management personnel were present only to introduce the survey team or not at all.

Issue Date:  04/01/19	AppF-2	95003.02

g. The introduction to the survey clearly describes the purpose(s) of the survey, whether responses will be maintained anonymous, who will have access to the raw data, and how the information will be used.

h. Introductory information and instructions encourage the respondents to answer truthfully, indicate that there are no right or wrong answers, and avoid statements that may bias the responses.

i. The same introductory information and instructions were provided to all survey respondents.

j. Anonymity and confidentiality were discussed.

4. Evaluate the statistical methods used to analyze the results.  Determine whether:

a. Sufficient responses were received to ensure statistical validity.

b. The statistical techniques applied were appropriate for the types of data collected (i.e., nominal, ordinal).

c. Any differences in responses between functional groups or levels of management were appropriately tested to determine whether the differences were likely due to chance or appear to be statistically reliable.

d. The probability level established for comparisons between responses to individual questions, question sets, and among different subgroups was sufficiently low to reduce the likelihood of “false positives,” in which differences appear to be statistically reliable but are, in fact, due to chance.

e. Any analyses were performed to verify that scales or sets of grouped questions are internally consistent and so appear to be measuring related constructs, and that the results confirm the item groupings.

f. The conclusions drawn from the survey are supported by the results of the analyses.

5. Determine whether the quantitative survey results were supplemented with any of the following to enhance the interpretation of the results:

a. Interviews or focus groups were conducted to gain additional information, as needed, to interpret ambiguous results or gain greater insights related to any issues identified in the survey.

b. The survey provided opportunities for respondents to write-in comments, clarifications, explanations, and additional, more detailed information.

c. Additional information related to any global organizational conditions that could affect the results, such as recent reductions in force, acquisitions or mergers, incentive buy-outs leading to large-scale retirements, or other factors, was used to evaluate differences between subpopulations or responses to the same survey administered at different times.

6. Evaluate participants’ responses to the survey:

a. Determine whether any issues related to the survey were entered into the CAP, raised to the Employee Concerns Program/Ombudsman or other alternate means of raising concerns, or to the NRC in allegations.

b. Elicit individuals’ perceptions of the survey, the manner in which the survey was administered, the integrity of the results, the manner in which results were communicated, and the manner in which the results were used.

Issue Date:  04/01/19	AppF-3	95003.02

Revision History for IP 95003.02

Issue Date:  04/01/19	Att1-1	95003.02