April 23, 2014 

MEMORANDUM TO: Ho K. Nieh, Director 

 Division of Inspection and Regional Support 

 Office of Nuclear Reactor Regulation 

FROM: Daniel J. Merzke, Acting Chief /RA/ 

Performance Assessment Branch 

Division of Inspection and Regional Support 

Office of Nuclear Reactor Regulation 

SUBJECT: EFFECTIVENESS REVIEW OF SUBSTANTIVE 

CROSS-CUTTING ISSUES 

 

In the summer of 2013, you requested the staff from the Performance Assessment Branch 

(IPAB) to conduct an effectiveness review of the substantive cross-cutting issue (SCCI) process 

as part of the Reactor Oversight Process (ROP) Enhancement Project. In an e-mail dated 

August 19, 2013, you requested that we review and compare what the SCCI process was 

intended to accomplish versus what it has actually accomplished, and to determine if there were 

any implementation issues or concerns. 

The ROP Independent Assessment report (Agency Document Access and Management 

System Accession Number ML14035A571), dated February 18, 2014, also included a 

recommendation to perform a comprehensive analysis to determine whether the use of crosscutting issues and safety culture provides regulatory value in terms of licensee safety 

performance for the resources expended. The report also recommended that the staff should 

clarify and document the goals, purposes, uses, and desired outcomes associated with the 

inclusion of cross-cutting issues and safety culture in the ROP. The purpose of this memo is to 

provide to you the results of the effectiveness review. 

Background

In the Staff Requirements Memorandum to SECY-04-0111, “Recommended Staff Actions 

Regarding Agency Guidance in the Areas of Safety Conscious Work Environment and Safety 

Culture,” the Commission directed the staff to “enhance the ROP treatment of cross-cutting 

issues to more fully address Safety Culture.” The staff developed and implemented the current 

SCCI process in response to that direction. 

CONTACT: Daniel Merzke, NRR/DIRS 

 (301) 415-1457 
H. Nieh - 2 - 

The purpose for assigning SCCIs, as stated in Inspection Manual Chapter (IMC) 0305, 

“Operating Reactor Assessment Program,” is to inform licensees that the U.S. Nuclear 

Regulatory Commission (NRC) has a concern with a licensee’s performance in a cross-cutting 

area, and to encourage the licensee to take appropriate action before more significant 

performance issues emerge. 

Cross-cutting aspects (CCAs) are assigned to inspection findings if the most significant causal 

factor associated with the performance deficiency matches one of the CCAs described in IMC 

0310, “Aspects Within Cross-Cutting Areas.” For the CCAs of problem identification and 

resolution (PI&R) and human performance, a cross-cutting theme exists when at least four 

inspection findings are assigned the same CCA during a mid-cycle or end-of-cycle assessment 

period. A cross-cutting theme exists in the area of safety conscious work environment (SCWE) 

if at least one of the following three conditions exists in an 18-month period: (1) a finding with a 

documented CCA in SCWE and the impact on SCWE was not isolated, or (2) the licensee has 

received a chilling effect letter, or (3) the licensee has received correspondence from the NRC 

that transmitted an enforcement action with a Severity Level I, II, or III, and that involved 

discrimination, or a confirmatory order that involved discrimination. An SCCI is assigned when 

a licensee has a cross-cutting theme and the NRC staff has a concern with the licensee’s scope 

of efforts or progress in addressing that theme. 

Desired outcomes from identifying SCCIs include improved licensee performance in one or 

more of the following areas: decision-making; the use of resources; work control; work 

practices; the corrective action program; the use of operating experience; the conduct of 

self-assessments; the environment for raising concerns; and preventing, detecting, and 

mitigating perceptions of retaliation. This improvement would be demonstrated by fewer 

inspection findings with the same CCA. Another desired outcome is for the NRC to have 

increased confidence in the licensee’s ability to implement effective corrective actions. 

Effectiveness Review 

The effectiveness review was performed by staff from IPAB, supported by the IMC 0305 

working group, consisting of members from each Region. Enclosure 1 describes the data 

obtained for the effectiveness review. Since the purpose of assigning SCCIs is to ensure 

licensees take corrective action before more significant performance issues emerge, the 

measure of effectiveness used in the review was principally licensee performance as 

determined by position in the Action Matrix prior to, coincidental with, and after identification of 

an SCCI. Significant performance issues manifest themselves through safety significant 

performance indicators and findings, which move licensees in the Action Matrix. The data was 

evaluated to determine if licensee performance was steady, improving, or declining before and 

after the time an SCCI (or multiple SCCIs) was identified. Data was gathered from the 2006 

end-of-cycle meetings to the 2013 mid-cycle meetings because this period utilized the current 

categorization and definition of cross-cutting aspects. 

Additionally, data was gathered to support two measures of effectiveness suggested in the ROP 

independent review report, i.e., the number of sites/plants that have had SCCIs identified, 

corrected the SCCIs, and subsequently had additional SCCIs; and the number of assessment 

cycles required for a site/plant to clear an SCCI. However, staff felt that those more closely 

measured the effectiveness of licensee corrective actions to address SCCIs than the 
H. Nieh - 3 - 

effectiveness of the SCCI process. For those licensees that had the same SCCI identified after 

being previously closed, one can conclude that the desired outcome of improved performance 

was not sustained. 

In addition to evaluating licensee performance before and after an SCCI was identified, staff 

also gathered data on licensee performance for when cross-cutting themes were identified, but 

an SCCI was not assigned because the staff did not have a concern with the licensee’s scope of 

efforts or progress in addressing the cross-cutting theme. Licensees would still be expected to 

enter the identification of a cross-cutting theme into their corrective action programs such that 

appropriate corrective action would be taken before more significant performance issues 

emerge, much like for SCCIs. The data was evaluated to determine if identification of a 

cross-cutting theme was effective in preventing more significant performance issues from 

emerging. 

The staff was unable to perform a cost-benefit analysis to determine if the use of cross-cutting 

issues and safety culture provides regulatory value in terms of licensee safety performance for 

the resources expended. Staff does not track the number of hours dedicated to the 

determination or identification of CCAs or SCCIs. However, anecdotal information from regional 

stakeholders indicates a significant amount of time is devoted to ensuring that inspection 

findings are assigned the proper CCAs, when warranted. In addition, one can conclude that a 

substantial amount of time has been expended on assigning CCAs, as there have been 5400 

CCAs assigned during the period evaluated out of approximately 6900 inspection findings. Of 

those findings,97.7 percent were of very low safety significance, i.e., green findings. Based on 

observations of mid-cycle, and end-of-cycle meetings, staff spent a significant amount of time 

preparing for and discussing CCAs, cross-cutting themes, and determining whether or not to 

issue an SCCI, again based primarily on issues of very low safety significance. Based on these 

facts, one can conclude that the resource cost of the SCCI process is significant. However, 

because the vast majority of these CCAs are assigned to findings of very low safety 

significance, one cannot conclude that the regulatory value is commensurately significant. 

Approximately 39 percent of SCCIs were closed out within one year of identification. Over 74 

percent of SCCIs were closed out within two years of identification. These figures would 

suggest that the NRC was successful in communicating a concern with the licensee’s 

performance in cross-cutting areas such that licensees could implement effective corrective 

actions to reverse the potential adverse trends. They also highlight potential weaknesses in 

licensee corrective action programs where SCCIs remained open for more than two years, such 

that the NRC could focus additional resources inspecting those programs. 

Conference calls with the working group were held on December 16, 2013, and January 13, 

2014, to solicit ideas for other potential measures of effectiveness, and to share the data 

gathered, as well as to discuss preliminary conclusions. 

Based on the data gathered, the staff has made the following conclusions regarding the 

effectiveness of the SCCI process: 

1. Identification of SCCIs was effective in communicating to licensees that there was a 

concern in a CCA, so that licensees focused corrective actions that might preclude more 

significant performance issues. 
H. Nieh - 4 - 

2. It is very difficult to prove that licensee corrective actions resulting from identification of 

an SCCI prevented more significant performance issues, especially for those licensees 

whose performance was steady before and after an SCCI was identified. Therefore, the 

staff cannot say definitively that the SCCI process is either effective or ineffective. 

3. SCCIs are not a leading indicator for declining licensee performance. A plant moved 

right in the Action Matrix without identification of an SCCI 86 times. (7 of those times 

were caused by security reintegration into the ROP. 10 of those times a cross-cutting 

theme was identified at the same time, but no SCCI was assigned). A plant was already 

exhibiting declining performance prior to or coincidental with identification of an SCCI. 

Conversely, only 14 plants moved right in the Action Matrix within a year of identification 

of an SCCI 17 times. 

4. The resource cost for implementing the SCCI process is significant without an apparent 

commensurate increase in the safety benefit. 

Enclosure: 

SCCI Statistics (EOC 2006 – Present)
H. Nieh - 4 - 

2. It is very difficult to prove that licensee corrective actions resulting from identification of 

an SCCI prevented more significant performance issues, especially for those licensees 

whose performance was steady before and after an SCCI was identified. Therefore, the 

staff cannot say definitively that the SCCI process is either effective or ineffective. 

3. SCCIs are not a leading indicator for declining licensee performance. A plant moved 

right in the Action Matrix without identification of an SCCI 86 times. (7 of those times 

were caused by security reintegration into the ROP. 10 of those times a cross-cutting 

theme was identified at the same time, but no SCCI was assigned). A plant was already 

exhibiting declining performance prior to or coincidental with identification of an SCCI. 

Conversely, only 14 plants moved right in the Action Matrix within a year of identification 

of an SCCI 17 times. 

4. The resource cost for implementing the SCCI process is significant without an apparent 

commensurate increase in the safety benefit. 

Enclosure: 

SCCI Statistics (EOC 2006 – Present) 

DISTRIBUTION: 

RidsNrrDirsIpab HNieh AHowe RFranovich KKennedy 

RidsRgn1MailCenter CRegan RLorson JRogge TPruett 

RidsRgn2MailCenter MScott RCroteau WJones RPowell 

RidsRgn3MailCenter TReis KObrien GShear SRose 

RidsRgn4MailCenter PLouden TBlount JClark JLara 

 

ADAMS ACCESSION NO: ML14099A171 Concurrence by email* 

OFFICE NRR/DIRS/IPAB RGN I* RGN II* RGN III* RGN IV* 

NAME DMerzke RPowell SRose JLara RKellar 

DATE 04/23/2014 04/21/2014 04/15/2014 04/14/2014 04/14/2014

OFFICIAL RECORD COPY
Enclosure 1 

Substantive Cross-Cutting Issue Statistics (EOC 2006 – MOC 2013)

269 total substantive cross-cutting issues (SCCIs) identified 

180 SCCIs in Human Performance (66.9%) 

89 SCCIs in Problem Identification and Resolution (33.1%) 

0 SCCIs in Safety Conscious Work Environment 

Numbers by 

Action Matrix 

column 

Region I Region II Region III Region IV Total Percentage

Column 1 11 7 42 54 114 42.4 

Column 2 4 4 24 34 66 24.5 

Column 3 4 6 15 19 44 16.4 

Column 4 0 12 0 33 45 16.7 

Total 19 29 81 140 269 100 

To determine if identification of an SCCI was effective at having licensees implement corrective 

actions prior to more significant performance issues emerging, staff used licensee position in 

the Action Matrix as a measure of licensee performance. Staff reviewed licensee position in the 

Action Matrix for the two assessment letters prior to, after, and coincident with identification of 

an SCCI. 

Prior to identification of an SCCI: (Performance trends determined from the Action Matrix 

column from the two assessment letters before identification of the SCCI) 

• 6 times a plant exhibited declining performance in the Action Matrix. 

• 11 times a plant exhibited declining performance in the Action Matrix coincident with 

identification of an SCCI. 

• 30 times a plant exhibited steady performance in the Action Matrix. 

• 3 times a plant exhibited improving performance in the Action Matrix. 

• The remaining SCCIs were repetitive or the licensee had multiple SCCIs identified 

simultaneously. 

After identification of an SCCI: (Performance trends determined from the Action Matrix 

columns for the two assessment letters after identification of the SCCI) 

• 6 times a plant exhibited improving performance immediately after (Immediately is 

defined as the next assessment letter). 

• 37 times a plant exhibited steady performance immediately after. 

• 14 times a plant exhibited declining performance immediately after. 

• 8 times a plant exhibited improved performance one year after. 

SCCIs identified to encourage licensees to take corrective action before more significant 

performance issues emerge 
- 2 - 

• 17 times a plant transitioned to column 3 with no SCCIs identified (two of those times 

were due to security reintegration back into the action matrix). 

• 68 times a plant transitioned from column 1 to 2 with no SCCIs identified (five of those 

were due to security reintegration back into the action matrix). 

• 1 time a plant transitioned to column 4 with no SCCIs identified (Ft. Calhoun). 

Licensees demonstrated more significant performance issues 86 times without identification of 

an SCCI. This would indicate that SCCIs are not a reliable leading indicator of performance 

issues. 

Simultaneous SCCIs issued (used to determine if there is a correlation between performance 

and SCCIs) 

131 assessment letters issued since EOC 2006 identifying SCCIs

69 assessment letters issued with 1 SCCI 

30 assessment letters issued with 2 SCCIs 

14 assessment letters issued with 3 SCCIs 

8 assessment letters issued with 4 SCCIs 

4 assessment letters issued with 5 SCCIs (Perry, San Onofre, Wolf Creek) 

1 assessment letter issued with 6 SCCIs (Palo Verde) 

1 assessment letter issued with 7 SCCIs (San Onofre) 

3 assessment letters issued with 8 SCCIs (Palo Verde, San Onofre) 

1 assessment letter issued with 9 SCCIs (Palo Verde) 

Results indicate that plants with performance problems generally have more SCCIs identified. 

However, this may result from the greater scrutiny of increased inspection hours resulting from 

supplemental inspections. 

Length of time SCCIs remained open (indicative of licensee’s success in closing SCCIs) 

One suggested measure of effectiveness identified in the Reactor Oversight Process (ROP) 

Independent Assessment report was the number of assessment cycles required for a site/plant 

to clear an SCCI. The following data was gathered to support that measure of effectiveness: 

Consecutive 

assessment 

letters 

1 2 3 4 5 6 7 8 10 

SCCIs 20 43 18 10 5 2 2 1 1 

Results show several licensees required a long time to close out SCCIs, as long as five years. 

This could indicate weakness in some licensee corrective action programs in that their 

corrective actions do not achieve the desired results. Success of the ROP is predicated on 

licensees having effective corrective action programs. Industry perspective is that the closure 

criteria are sometimes subjective, and that there are times when the wrong problem was 

addressed, or the initial cross-cutting problem evolves into a different problem. Therefore, some 

licensees have difficulty closing out the SCCIs in a timely manner. 
- 3 - 

Repetitive SCCIs (measure of effectiveness suggested in draft ROP independent review 

report) 

Another suggested measure of effectiveness identified in the ROP Independent Assessment 

report was the number of sites/plants that have had SCCIs identified, corrected the SCCIs, and 

subsequently had additional SCCIs. The following data was gathered to support that measure 

of effectiveness:

• 10 times a plant closed an SCCI, and had the same SCCI identified later. 

• 7 plants have had an SCCI opened, closed, and subsequently had the same SCCI 

identified again: 

o Browns Ferry – P.1.d 

o Palisades – H.2.c 

o Perry – H.2.c 

o Prairie Island – H.1.b 

o Arkansas Nuclear One – H.1.b 

o Cooper – H.2.c and H.1.b 

o Wolf Creek – P.1.a, P.1.d, and H.2.c 

P.1.a - corrective action program with a low threshold for identifying issues 

P.1.d - appropriate corrective actions 

H.1.b - conservative assumptions in decision making 

H.2.c - Complete, accurate and up-to-date design documentation, procedures, and work 

 packages, and correct labeling of components 

This measure more closely reflects the effectiveness of licensee’s corrective actions in resolving 

the initial SCCI. 

Cross-Cutting Themes 

Staff reviewed licensee performance in the Action Matrix based on identification of a 

cross-cutting theme, but not an SCCI, to determine if there was a correlation between the two. 

During the same period reviewed for SCCIs, 107 cross-cutting themes were identified with no 

SCCI, communicated in 95 assessment letters 

• 14 times a cross-cutting theme was identified coincidentally with a decline in licensee 

performance in the action matrix. 

• 9 times a cross-cutting theme was identified; licensee performance declined in the action 

matrix within one year. 

• 35 times a cross-cutting theme was identified; licensee performance was steady in the 

action matrix prior to identification. 

• 7 times licensee performance improved in the action matrix after identification of a crosscutting theme. 
- 4 - 

Summary of Observations 

• For steady performers before and after identification of an SCCI, it is difficult to prove 

that identification of an SCCI ensured licensees took action which prevented more 

significant performance issues. 

• 17 times a plant was already exhibiting declining performance in the Action Matrix before 

or coincident with identification of an SCCI. 

• 86 times a plant moved right in the Action Matrix without identification of an SCCI (7 of 

those times were due to security reintegration into the ROP; 10 of those times a 

cross-cutting theme was identified at the same time, but no SCCI was identified). 

• The staff was unable to attribute cause and effect of licensee performance to the SCCI 

process.