POLICY ISSUE

INFORMATION

April 17, 2003 SECY-03-0057

FOR: The Commissioners

FROM: William D. Travers

Executive Director for Operations

SUBJECT: FY 2002 RESULTS OF THE INDUSTRY TRENDS PROGRAM FOR

OPERATING POWER REACTORS AND STATUS OF ONGOING

DEVELOPMENT

PURPOSE:

To inform the Commission of the results of the Nuclear Regulatory Commission’s (NRC)

Industry Trends Program (ITP) for FY 2002, and the status of ongoing development. The ITP

supports the NRC’s performance goals of maintaining safety and enhancing public confidence

in the agency’s regulatory processes.

SUMMARY:

The NRC staff implemented the ITP in 2001, and is continuing to develop the program as a

means to confirm that the nuclear industry is maintaining the safety of operating power plants

and to increase public confidence in the efficacy of the NRC’s processes. The NRC uses

industry-level indicators to identify adverse trends, evaluate them using agency databases, and

take appropriate actions. One important output of this program is to report to Congress each

year on the performance goal measure of “no statistically significant adverse industry trends in

safety performance” as part of the NRC’s Performance and Accountability Report. Based on

the information currently available from the industry-level indicators originally developed by the

former Office for Analysis and Evaluation of Operational Data (AEOD) and the Accident

Sequence Precursor (ASP) Program implemented by the Office of Nuclear Regulatory

Research (RES), no statistically significant adverse industry trends have been identified through

FY 2002.

CONTACT: Tom Boyce, NRR/DIPM

301-415-1130
-2-

The staff is continuing to use the AEOD and ASP indicators while it develops additional

industry-level indicators that are more risk-informed and better aligned with the cornerstones of

safety in the Reactor Oversight Process (ROP). These additional indicators will be developed

in phases and qualified for use in the ITP and the annual Performance and Accountability

Report to Congress. The results of this program, along with any actions taken or planned, are

reviewed annually during the Agency Action Review Meeting (AARM) and reported to the

Commission.

BACKGROUND:

This paper is the third annual report to the Commission on the ITP. The previous Commission

reports were SECY-01-0111, “Development of an Industry Trends Program for Operating

Power Reactors,” and SECY-02-0058, “Results of the Industry Trends Program for Operating

Power Reactors and Status of Ongoing Development.” Additional information on the ITP

program and the process for identifying and addressing adverse trends is in Attachment 1.

The NRC provides oversight of plant safety performance on a plant-specific basis as part of

implementing the ROP. The ROP uses both plant-level performance indicators (PIs) and

inspections. The ITP provides a means to assess overall industry performance using industrylevel indicators. Issues that are identified from either the ROP or the ITP are evaluated using

information from agency databases, and those assessed as having generic safety significance

are addressed using existing NRC processes, including generic safety inspections in the ROP,

the generic communications process, and the generic safety issue process.

The purposes of the ITP are to provide a means to assess whether the nuclear industry is

maintaining the safety performance of operating reactors and, by clearly communicating that

performance, to enhance stakeholder confidence in the efficacy of the NRC’s processes. The

specific objectives of the ITP are as follows:

(1) Collect and monitor industry-wide data that can be used to assess whether the nuclear

industry is maintaining the safety performance of operating plants and to provide

feedback on the ROP.

(2) Assess the safety significance and causes of any statistically significant adverse

industry trends, determine if the trends represent an actual degradation in overall

industry safety performance, and respond appropriately to any safety issues that may be

identified.

(3) Communicate industry-level information to Congress and other stakeholders in an

effective and timely manner.

The NRC currently uses the results of the ITP in the following ways.

(1) The NRC reports the industry indicators to Congress annually in the NRC’s

“Performance and Accountability Report, Fiscal Year 200X” (NUREG-1542 series) and

in the NRC’s “Budget Estimates and Performance Plan Fiscal Year 200X” (NUREG1100 series). The indicators demonstrate how the agency has met the measure of “no
-3-

statistically significant adverse industry trends in safety performance” for the

performance goal of maintaining safety.

(2) The NRC communicates overall industry performance to stakeholders by publishing the

ITP indicators on the Nuclear Reactors portion of the agency’s public Web site at

http://www.nrc.gov/reactors/operating/oversight/industry-trends.html. The staff believes

that communication of the industry-level indicators, when added to the information on

individual plants from the ROP, enhances stakeholder confidence in the efficacy of the

NRC’s oversight of the nuclear industry.

(3) The results of the ITP are a key element of the review by senior NRC managers of the

agency’s oversight of operating facilities in the annual AARM.

(4) The staff informs the Commission of the results of the ITP in an annual report in the

same timeframe as the AARM.

(5) The Commission uses the ITP indicators when presenting the status of industry

performance to the NRC’s oversight committees and at major conferences with industry.

(6) NRC managers use the ITP indicators to provide an overview of industry performance at

various conferences with industry, such as the NRC’s Regulatory Information

Conference.

DISCUSSION:

The ITP is intended to monitor trends in industry safety performance so that the staff can

identify and address adverse industry trends. The ITP accomplishes this using indicators of

known conditions and issues that are compiled from the best available data. The staff monitors

a comprehensive set of indicators; however, the staff recognizes that the ITP has limits as to

what can be tracked and trended. Oversight of plant-specific conditions and events is provided

by the ROP. The staff’s approach recognizes that new and unforseen events will inevitably

occur, such as the significant erosion of the reactor vessel head at Davis-Besse. The staff is

mindful of the need to respond promptly to these events, as well as the need to review its

regulatory processes in light of the issues revealed by these events.

RESULTS OF FY 2002 TREND ANALYSES

A key output of the ITP is that it provides the basis for agency monitoring and reporting to

Congress against the performance goal measure of “no statistically significant adverse industry

trends in safety performance,” as established by the NRC’s Strategic Plan. In early FY 2001,

NRR assumed responsibility from RES for reporting on this measure as part of NRR’s overall

responsibilities in the Reactor Safety arena. The current bases for assessing performance

against this measure are trends in the industry indicators developed by the former AEOD

(henceforth referred to as the “AEOD indicators”) and trends identified by the ASP Program. 

Notably, these indicators were among those cited as demonstrating improvements in industry

safety performance that contributed to the agency’s decision to revise the oversight process for

operating power reactors.
-4-

Based on the AEOD indicators and the ASP Program results, no statistically significant adverse

trends in industry safety performance were identified through the end of FY 2002. Graphs of

the trends for each of these indicators are presented in Attachment 2. The results and graphs

were also included in the NRC’s Performance and Accountability Report for FY 2002.

In addition, as discussed in SECY-01-0111, the staff adopted a statistical approach using

“prediction limits” to provide a consistent method to identify potential short-term year-to-year

emergent issues before they manifest themselves as long-term trends. Although the ASP

indicator in Figure A2-8 did not cross a prediction limit, there appears to be a relatively low

number of precursors between 1997 and 1999 and an increasing number of potential

precursors in 2000 and 2001. For 2000 and 2001, the staff’s analyses are only preliminary and

the number of ASP events may decrease as the analyses are finalized. Nonetheless, the staff

intends to follow the ITP process discussed in Attachment 1 to analyze these short-term

variations in the number of ASP events. The evaluation will be a joint project between NRR and

RES as part of the ITP. The evaluation may include a review of the risk significance of the

events, types of facilities involved, a categorization of causes and factors for analyzed events

and conditions, the time period for analysis, and whether any additional actions are appropriate.

In SECY-02-0058, the staff analyzed two indicators that had exceeded their prediction limits for

FY 2001, but did not identify any safety issues associated with exceeding those limits. In early

FY 2002, the staff reviewed its methods for setting prediction limits and modified them to use

data starting in about 1996 instead of 1988, since this more recent timeframe was assessed to

be more representative of current industry performance. In general, the modification had the

effect of raising the prediction limits for FY 2002 to about the level of industry performance in

1996. No AEOD indicators exceeded their prediction limits during FY 2002.

ITP PROGRAM DEVELOPMENT

1. Incorporation of Additional Industry Operating Experience

The NRC’s Davis-Besse Lessons Learned Task Force (DBLLTF) provided a number of

recommendations to improve NRC utilization of operating experience information. The staff is

currently assessing its processes for handling this information, including the ITP, to determine

what changes or enhancements may be appropriate, and will highlight any necessary changes

to the ITP in the next annual report to the Commission.

For example, the staff is assessing whether operating experience information from various

agency databases can be used to develop additional indicators that can provide improved

feedback to the ROP, particularly the inspection program. The staff is updating the data that

have been published in various NUREG-series reports for system reliability studies, component

reliability studies, common-cause failure studies, and other special studies for which industrywide trends were reported. This work is anticipated to be completed over the next 1–2 years. 

In addition, as a means of providing greater stakeholder access to this information, the staff is

developing a Web-based system to replace the current paper-based system of NUREG-series

reports.

In addition, partly in response to a recommendation by the DBLLTF, the staff intends to assess

how to more effectively incorporate foreign operating experience information into the ITP. The
-5-

staff will build on the international work on PIs described in SECY-02-0030, “Summary Report

on NRC’s Historical Efforts To Develop and Use Performance Indicators.” This would include

enhanced participation with efforts of the Nuclear Energy Agency of the Organization for

Economic Cooperation and Development, principally through its Committee on the Safety of

Nuclear Installations and its Committee of Nuclear Regulatory Authorities.

2. Development of Additional, More Risk-informed Indicators

The staff has continued to develop additional indicators that are more risk-informed and better

aligned with the cornerstones of safety in the ROP. For example, the staff has continued

development of industry-level indicators from the data that licensees submit for the plant-level

ROP PIs. However, since the ROP was only implemented in April 2000, there is still insufficient

data for reliable long-term trending (<4 years) of these indicators. Nonetheless, based on a

review of the indicator data submitted to date, no significant short-term trends have emerged.

As discussed in SECY-02-0058, the staff developed additional risk-informed indicators for the

initiating events cornerstone, consisting of multiple indicators of initiating events for both

pressurized water reactors (PWRs) and boiling water reactors (BWRs). This effort involved

updating the data that was most recently published in NUREG-5750, “Initiating Events at U.S.

Nuclear Power Plants: 1987–1995.” In general, the number of initiating events has continued to

decline over the past decade. During FY 2002, the staff built on this work in developing an

overall industry-level indicator for the initiating events cornerstone. An overall indicator can

provide a better representation of the overall risk from initiating events than multiple individual

indicators of initiating events with varying degrees of risk significance. For example, it is

possible that there could be an increase in loss of feedwater events in any given year, but the

overall risk from all initiating events may actually have declined if the contribution to risk from

that single indicator is low and the contribution to risk from all other initiating events has

declined.

This overall indicator of initiating events, called the Industry Initiating Events Performance

Indicator (IIEPI), consists of an index of the most risk significant industry initiating events. This

set of events is defined in NUREG-1753, “Risk-Based Performance Indicators: Results of

Phase 1 Development,” as those events that contribute >1 percent to industry core damage

frequency and that have occurred at least once during the period 1987–1995. An index was

developed for BWRs that has 9 risk-significant initiating events, and a similar index was

developed for PWRs that has 10 events (the additional category of events is steam generator

tube ruptures). Each initiating event is weighted in the index based on its relative contribution

to industry core damage frequency. Although the indicator is being developed to monitor

industry-level performance, it could potentially be adapted to monitor plant-level performance as

well, similar to the Mitigating Systems Performance Index (MSPI) that is currently being

assessed in a pilot program as part of the ROP PIs. The IIEPI is discussed in more detail in

Attachment 3.

The staff has given briefings on the IIEPI concept during periodic ROP working group public

meetings, and has briefed two subcommittees of the Advisory Committee on Reactor

Safeguards (ACRS). The staff has received valuable feedback during these meetings, and no

major concerns have been identified to date. The staff intends to increase its interactions with
-6-

stakeholders on the IIEPI during this fiscal year, working towards a pilot program and possible

implementation within 1–2 years.

3. Risk-Informed Response Thresholds

In SECY-01-0111, the staff stated that it was developing risk-informed thresholds, to the extent

practicable, which would be used to determine the appropriate agency response to trends in

indicator data. This could enable the staff to establish a more objective and predictable agency

response in a manner analogous to the ROP’s Action Matrix. The use of thresholds would also

preclude a scenario in which the improving trends in indicators have leveled off, presumably

when the practical performance limits of operating plants have been reached, and then a small

decline in performance results in an adverse trend that would necessitate a report to Congress

as an adverse trend, notwithstanding the fact that industry performance is better than it was

before the trends leveled. In the staff requirements memorandum (SRM) related to

SECY-01-0111 dated August 2, 2001, the Commission directed the staff to develop riskinformed thresholds for the industry-level indicators “as soon as practicable.”

The staff formed an interoffice working group to examine the feasibility of developing thresholds

for indicators in all cornerstones of safety. For the initiating events and mitigating systems

cornerstones, risk-informed thresholds appeared to be possible since NRC risk analysis tools,

such as the Standardized Plant Analysis Risk (SPAR) models, were available. However, in light

of the development of more risk-informed indicators for the initiating events cornerstone

(e.g., the IIEPI) and the mitigating systems cornerstone (e.g., the MSPI as part of the plantspecific ROP PIs), the staff prioritized its resources on developing those potential replacement

indicators rather than on developing risk-informed thresholds for the current set of ITP

indicators.

In addition, the working group examined development of thresholds for indicators in other

cornerstones of safety and for the AEOD indicators, which do not lend themselves to

development of risk-informed thresholds due to lack of risk models. The working group used a

statistical approach based on prediction limits for ITP indicators to explore whether thresholds

could be established. Although thresholds using a statistical approach were developed, the

staff encountered some difficulty in determining the appropriate level for regulatory action. For

example, a statistical approach to setting the threshold for the alert notification system reliability

indicator in the emergency protection cornerstone resulted in a threshold of about 97 percent,

compared to the current actual industry performance of about 99 percent. However, if the

reliability dropped below the threshold to 96 percent, the staff would likely not take regulatory

action on an industry-wide basis because the reliability is still high and appears acceptable. 

The staff intends to seek stakeholder input, including input from the ACRS, while continuing

staff actions to develop thresholds.

4. Potential Single Integrated Indicator of Industry Performance

Attachment 4 shows an integrated indicator, known as the Action Matrix Summary. It is a

histogram that shows the number of plants in each column of the NRC’s Action Matrix since

initial implementation of the revised ROP. This single indicator provides a representative

picture of industry performance because it effectively rolls up both performance indicators and

inspection findings from the ROP in all cornerstones of safety.
-7-

The staff is considering establishing a new strategic plan performance goal measure that would

count the number of plants with significant performance issues, and would use the Action Matrix

Summary indicator for reporting against the measure. For example, the measure could be “no

more than 5 plants in the multiple/degraded cornerstone column or above.” Consistent with the

intent of the Government and Performance Results Act to tie agency actions to performance

measures, this measure has the advantage that the staff is already taking actions to address

safety issues for these plants in an objective and predictable manner in accordance with the

Action Matrix for the ROP.

This measure is in addition to the existing performance goal measure of “no adverse industry

trends.” The staff is pursuing this performance goal measure for use in the NRC’s Budget

Estimates and Performance Plan for FY 2005 (also known as the Blue Book and Green Book). 

However, in accordance with reporting guidelines from the NRC’s Office of the Chief Financial

Officer (OCFO), reporting for this measure would commence in the NRC’s Performance and

Accountability Report for FY 2004.

5. Improved Data Collection and Reporting 

Licensees report operating experience data to the NRC, including data for the ROP plant-level

indicators, licensee event reports (LERs), and monthly operating reports. Licensees also report

additional data to other organizations such as the Institute of Nuclear Power Operations (INPO). 

The staff uses all of these reports as data sources for the indicators used in the ITP, and the

databases are essential to investigating safety issues when trends in the ITP indicators are

identified. The databases are also used as part of the bases for changes to ROP PIs and

inspection procedures.

The staff continues to seek improvements and efficiencies in data collection and reporting by

industry for both the ITP and the ROP. For example, RES coordinated with NRR to consolidate

coding of LERs by two Department of Energy laboratories into one laboratory, thereby saving

the agency about $500K per year. In addition, the staff is currently working extensively with

industry to develop a consistent set of data elements, definitions, and reporting guidelines for

reliability and unavailability data that would encompass the needs of all stakeholders. Finally,

the staff is working with the industry Consolidated Data Advisory Committee of INPO to develop

a common collection system for industry operating experience data that can be used by both

the NRC and INPO. These development efforts are expected to continue over the next several

years.

RESOURCES:

For FY 2003, NRR has budgeted 1.5 full-time equivalent (FTE) and $254K for the continued

development and implementation of the ITP. For FYs 2004 through 2006, NRR estimates

resource requirements of 1.5 FTE per year, with estimated contract assistance funding

requirements of about $254K per year. NRR has included these requirements in its budget

request submittals. For FY 2003, RES has budgeted 0.5 FTE and $270K for the development

of risk-informed thresholds for indicators in the ITP. For FY 2004, RES budgeted 0.5 FTE and

$240K, and for FY 2005 and FY 2006 estimates 0.5 FTE and $50–100K will be required. 

RES is developing the operational experience data and risk models as part of existing programs

that are currently budgeted. Should additional resources be needed to accomplish these tasks,
-8-

the staff will reprogram the resources from within the current budget using the NRC’s Planning,

Budgeting, and Performance Management process.

COORDINATION:

The Office of the Chief Financial Officer has reviewed this paper and concurs with the resource

estimates.

The Office of the General Counsel has reviewed this paper and has no legal objection.

/RA/

William D. Travers

Executive Director 

 for Operations

Attachments: 1. Description of the ITP process

2. FY 2002 Trend Results Based on AEOD and ASP Indicators

3. Description of the Industry Initiating Events Performance Indicator

4. Action Matrix Summary
-8-

the staff will reprogram the resources from within the current budget using the NRC’s Planning,

Budgeting, and Performance Management process.

COORDINATION:

The Office of the Chief Financial Officer has reviewed this paper and concurs with the resource

estimates.

The Office of the General Counsel has reviewed this paper and has no legal objection.

/RA/

William D. Travers

Executive Director 

 for Operations

Attachments: 1. Description of the ITP process

2. FY 2002 Trend Results Based on AEOD and ASP Indicators

3. Description of the Industry Initiating Events Performance Indicator

4. Action Matrix Summary

WITS 200100034

ADAMS Accession Number: ML030870815

*via email

OFFICE IIPB:DIPM IIPB:DIPM IIPB:DIPM Tech Editing RGN 1

NAME TBoyce:kig MSatorius CCarpenter PKleene GCobey*

DATE 04/01/03 04/01/03 03/28/03 03/11/03 04/01/03

OFFICE RGN 2 RGN 3 RGN 4 REXB:DRIP SPSB:NRR

NAME LPlisco* JDyer* AHowell* TReis* MJohnson*

DATE 03/11/03 03/27/03 03/15/03 03/11/03 03/19 /03

OFFICE PPRB:PMAS OCFO DIPM:NRR OGC RES

NAME RCorreia BFicks* BBoger LChandler AThadani*

DATE 03/19/03 03/27/03 03/28/03 03/14/03 04/02/03

OFFICE ADIP NRR EDO

NAME RBorchardt SCollins WTravers

DATE 04/03/03 04/03/03 04/17/03

OFFICIAL RECORD COPY
Description of the Industry Trends Program (ITP) Process

1. Background

The U.S. Nuclear Regulatory Commission (NRC) provides oversight of plant safety

performance on a plant-specific basis using both inspection findings and plant-level

performance indicators (PIs) as part of its Reactor Oversight Process (ROP). Individual issues

that are identified as having generic safety significance are addressed using other NRC

processes, including the generic communications process and the generic safety issue process.

As discussed in SECY-01-0111, “Development of an Industry Trends Program for Operating

Power Reactors,” the NRC's Office of Nuclear Reactor Regulation (NRR) initiated the ITP to

complement these processes by monitoring and assessing industry-level trends in safety

performance.

The purposes of the ITP are to provide a means to confirm that the nuclear industry is

maintaining the safety performance of operating reactors and, by clearly demonstrating that

performance, to enhance stakeholder confidence in the efficacy of the NRC's processes. The

objectives of the ITP are as follows:

 Collect and monitor industry-wide data that can be used to assess whether the nuclear 

industry is maintaining the safety performance of operating plants and to provide

feedback on the ROP.

 Assess the safety significance and causes of any statistically significant adverse

industry trends, determine if the trends represent an actual degradation in overall

industry safety performance, and respond appropriately to any safety issues that may be

identified.

 Communicate industry-level information to Congress and other stakeholders in an

effective and timely manner. 

A key output of the ITP is that it provides the basis for agency monitoring and reporting in the

Nuclear Reactor Safety arena against the performance goal measure of “no statistically

significant adverse industry trends in safety performance,” as defined by the NRC's Strategic

Plan. The agency reports these results annually to Congress in the Performance and

Accountability Report, Fiscal Year 200X” (NUREG-1542 series). In early FY 2001, NRR

assumed responsibility from the NRC’s Office of Nuclear Regulatory Research (RES) for

reporting on this measure as part of NRR’s overall responsibilities in the Reactor Safety arena. 

The current bases for assessing performance against this measure are trends in the industry

indicators developed by the former NRC Office of Evaluation of Operational Data (AEOD) and

trends identified by the ASP Program. Notably, these indicators were among those cited as

demonstrating improvements in industry safety performance that contributed to the agency's

decision to revise the ROP.

ATTACHMENT 1
-2-

In developing the ITP, the staff used the following general concepts for its approach:

 The indicators were developed using information available from current NRC programs. 

In the future, indicators will be developed in stages, and will provide information for each

ROP cornerstone of safety.

 Industry trend information is derived from quantitative, industry-wide data.

 Trends are identified on the basis of long-term data, rather than short-term data. This

minimizes the impact of short-term variations in data, which may be attributable to such

factors as operating cycle phase, seasonal variations, and random fluctuations.

 Trends and contributing factors are assessed for safety significance. The results of

inspections, analyses of significant events and abnormal occurrences, and other

analyses may be used to facilitate an evaluation of the trends. The agency’s response

is commensurate with the safety significance.

 While additional indicators are being developed, a subset of high-level indicators may be

used for the report on adverse trends to Congress in the NRC’s Performance and

Accountability Report. For reporting on the performance measure of “no statistically

significant adverse industry trends in safety performance,” indicators will be qualified for

use in phases. Until they are qualified, the staff will continue to use the AEOD indicators

and ASP results. Additional indicators from the ITP will be incorporated for use in

accordance with a controlled process for making such changes to the NRC’s

Performance Plan. In addition, the staff intends to consider refinements to the

performance measure as the indicators and more risk-informed methods of assessing

their safety significance are developed.
-3-

2. ITP Process

A flowchart of the ITP process is shown below

Industry Trends Program

Report to Congress

Communications

Senior 

Management 

Review

Agency Action 

Review Meeting

Analyses of Issues

Agency Response

Identify Short Term Issues

• Exceed prediction limits

Identify Adverse Trends

• Few plants => No adverse trend

• Plant comparison groups

• Examine agency databases

• Attempt to establish causes

• NRR Technical branch review

• Assess Safety Significance

• Apply statistically significant trend line

Declining slope => adverse

• Exceed thresholds

• NRC Performance & 

Accountability Report

• Green/Blue Books

• Oversight Committees

ROP PIs (18)

• PI submittals

In all 7 cornerstones

Of safety

IIEPI (2)

• LERs

• EPIX

• Other data

• SPAR

AEOD PIs (7)

• LERs

• MORs

• 50.72s

ASP (1)

• Various data

• SPAR models

Collect Indicator Data

• NRC web page for ITP

• Annual report to 

Commission

• NRC Info Digest

• Industry conferences

• No action/continue to monitor

• Engage industry

• Generic communications

• Generic safety inspection

• Generic safety issue

Report to Congress

Communications

Senior 

Management 

Review

Agency Action 

Review Meeting

Analyses of Issues

Agency Response

Identify Short Term Issues

• Exceed prediction limits

Identify Adverse Trends

• Few plants => No adverse trend

• Plant comparison groups

• Examine agency databases

• Attempt to establish causes

• NRR Technical branch review

• Assess Safety Significance

• Apply statistically significant trend line

Declining slope => adverse

• Exceed thresholds

• NRC Performance & 

Accountability Report

• Green/Blue Books

• Oversight Committees

ROP PIs (18)

• PI submittals

In all 7 cornerstones

Of safety

IIEPI (2)

• LERs

• EPIX

• Other data

• SPAR

AEOD PIs (7)

• LERs

• MORs

• 50.72s

ASP (1)

• Various data

• SPAR models

Collect Indicator Data

• NRC web page for ITP

• Annual report to 

Commission

• NRC Info Digest

• Industry conferences

• No action/continue to monitor

• Engage industry

• Generic communications

• Generic safety inspection

• Generic safety issue

Figure 1-1 Industry Trends Program process flowchart
-4-

Collect Indicator Data

In developing the ITP, the staff used information currently available from existing NRC

programs to develop an initial set of indicators for identifying adverse industry trends. The

indicators consisted of the 7 indicators in the AEOD indicator program and the results of the

ASP program. In addition, the staff is developing more risk-informed industry-wide indicators

using data from the 18 plant-level indicators submitted by licensees for the ROP PI program. 

The staff also identified potential indicators for initiating events that are anticipated to be

available from RES operating experience data. These indicators are being consolidated into an

Industry Initiating Events Performance Indicator (IIEPI).

Identify Short-Term Issues

In addition, as discussed in SECY-01-0111, in FY 2001 NRR adopted a statistical approach

using “prediction limits” to provide a consistent method to identify potential short-term emergent

issues before they manifest themselves as long-term trends. The prediction limits are values

established at the beginning of a FY that set an upper bound on expected performance for that

year for each indicator. Actual indicator values during the year can then be monitored and

compared to the prediction limits. Indicators that cross the prediction limits are investigated to

determine the factors contributing to the data. These factors are assessed for their safety

significance and used to determine an appropriate agency response. However, should very

obvious adverse trends emerge in the short-term data, the staff does not wait until the end of

the annual reporting period to initiate a review.

Identify Adverse Trends

For purposes of assessing whether there are any statistically significant adverse industry

trends, only long-term data is used. The trending of long-term data minimizes reacting to

potential “false positive” indications that may emerge in short-term data. “Short-term” was

defined to be less than four years to ensure that sufficient data (i.e., data for at least two typical

nuclear plant operating cycles) is available so that valid trends can be distinguished from

operating cycle effects such as refueling outages and from random fluctuations in the data and

to allow sufficient data for the use of statistical methods. The staff expects that any variations

beyond these will result from plant-specific issues which can be addressed under the ROP.

The staff applies common statistical techniques to the long-term indicator data to identify

trends. These techniques have been previously adapted and used extensively by the former

AEOD and by RES in reactor operating experience analyses over the past several years. In

general terms, a trendline is fitted to each indicator using regression techniques. Once a

statistically significant fit of a trendline is made to each indicator, the slope of the trendlines is

examined. Improving or flat trendlines are not considered adverse and need not be

investigated further. Degrading trendlines are considered adverse.

Analyses of Issues

Once an adverse trend is identified, the staff conducts an initial analysis of information readily

available in the databases used to compile the indicator data to determine whether the trend is

unduly influenced by a small number of outliers and to identify any contributing factors. If the
-5-

trend is the result of outliers, then it is not considered a trend requiring generic actions, and the

agency will consider any appropriate plant-specific actions using the ROP. For example, the

affected plants unduly influencing the adverse trend may have already exceeded plant-level

thresholds under the ROP, and the NRC regional offices would conduct supplemental

inspections at these plants to ensure the appropriate corrective actions have been taken. If the

plants did not exceed any thresholds, while the NRC would not take regulatory actions beyond

the ROP, the NRC would gather additional information on the issue within the scope of the ROP

using risk-informed baseline inspections. The results of these inspections would be examined

to determine if a generic issue existed requiring additional NRC review or generic inspections.

If no outliers are identified, the staff conducts a broader review to assess whether larger groups

of facilities are contributing to the decline and to assess any contributing factors and causes.

For example, the data review is expanded to include a review of various plant comparison

groups, contributing factors such as the operational cycle stage of the facilities (shutdown,

at-power, startup from refueling, etc.), and the apparent causes for the data (equipment

failures, procedure problems, etc.). The staff will also conduct a more detailed review of

applicable licensee event reports. Should a group of plants be identified, the staff will examine

the results of previously conducted inspections at these plants, including any root causes and

the extent of the conditions.

Once this information is reviewed, the staff assesses the safety significance of the underlying

issues. The staff is mindful that trends in individual indicators must be considered in the larger

context of their overall risk significance. For example, a hypothetical increase in automatic

scrams from 0.4 to 0.7 per plant per year over several years may be a statistically significant

trend in an adverse direction. However, it may not represent a significant increase in overall

risk since the contribution of a small number of scrams is relatively low, and it is possible that

overall risk may actually have declined if there were reductions in the frequency of more

risk-significant initiating events or the reliability and availability of safety systems had improved.

Depending on the issues, the staff may perform an additional evaluation using the most current

risk analysis tools or an evaluation by the ASP Program.

Agency Response

Until thresholds for ITP indicators are developed to establish the significance of indicator data,

should a statistically significant adverse trend in safety performance be identified or an indicator

cross a prediction limit, the staff will determine the appropriate response using the NRC’s

established processes for addressing and communicating generic issues. These processes are

described in SECY-99-143, “Revisions to Generic Communications Program.”

In general, the issues will be assigned to the appropriate branch of NRR for initial review. The

branch will engage NRC senior management and initiate early interaction with the nuclear

power industry. Depending on the issue, the process could include requesting industry groups

such as NEI or various owners groups to provide utility information. As discussed in

SECY-00-0116, “Industry Initiatives in the Regulatory Process,” industry initiatives, such as the

formation of specialized working groups to address technical issues, may be used in lieu of, or

to complement, regulatory actions. This can benefit both the NRC and the industry by

identifying mutually satisfactory resolution approaches and reducing resource burdens.
-6-

Depending on the issues, the NRC may consider generic safety inspections at plants. In

addition, the issues underlying the adverse trend may also be addressed as part of the generic

safety issue process by RES. After this interaction, the NRC may consider additional regulatory

actions as appropriate, such as issuing generic correspondence to disseminate or gather

information, or conducting special inspections for generic issues. The process also includes

consideration of whether any actions proposed by the NRC to address the issues constitute a

backfit.

Senior Management Review

The industry trends program, results, and agency response are reviewed annually during the

Agency Action Review Meeting (AARM). In general, the AARM is intended to review the

appropriateness and effectiveness of staff actions already taken, rather than to make decisions

on agency actions. NRC senior managers review the industry trends information and, if

appropriate, recommend any additional actions beyond those implemented by the staff.

Communications With Stakeholders

The NRC communicates overall industry performance to stakeholders by publishing the ITP

indicators on the Nuclear Reactors portion of the agency’s public Web site at

http://www.nrc.gov/reactors/operating/oversight/industry-trends.html. The staff believes that

communication of the industry-level indicators, when added to the information on individual

plants from the ROP, enhances stakeholder confidence in the efficacy of the NRC’s oversight of

the nuclear industry.

The staff informs the Commission of the results of the ITP in an annual report in the same

timeframe as the AARM. The indicators are also published annually in the NRC’s “Information

Digest 200X” (NUREG-1350 series). In addition, NRC managers have historically presented

industry indicators and trends at major conferences with industry.

Reports to Congress

The NRC reports the industry indicators to Congress annually in the NRC’s “Performance and

Accountability Report, Fiscal Year 200X” (NUREG-1542 series), and in the NRC’s “Budget

Estimates and Performance Plan Fiscal Year 200X” (NUREG-1100 series). The indicators

demonstrate how the agency has met the measure of “no statistically significant adverse

industry trends in safety performance” for the performance goal of maintain safety. Adverse

trends would be reported, but indicators that exceeded prediction limits need not be included in

these reports since these are tools to monitor industry performance rather than desired

thresholds of performance.

In addition, the Commission has historically used the ITP indicators when presenting the status

of industry performance to the NRC’s oversight committees.
FY 2002 Trend Results 

Based on AEOD and ASP Indicators

Indicators Originally Developed by the Former Office of AEOD and

Accident Sequence Precursor (ASP) Indicators

ATTACHMENT 2
-2-

Safety System Failures

3.25

3.53 3.58 3.44

3.78

3.09

2.32

2.03

2.89

2.71 2.76

1.67

1.37

0.81 0.86

y = -0.1944x + 4.0945

R2

 = 0.7743

0

1

2

3

4

5

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Failures per Plant

Industry Average

Industry Average Trend

Automatic Scrams While Critical

2.41

1.88

1.61 1.57 1.52

1.18

1.05 1.04

0.80

0.54 0.48

0.64

0.52 0.57

0.44

y = 2.4519e-0.12x

R2 = 0.9301

0

1

2

3

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Scrams per Plant

Industry Average

Industry Average Trend

Figure A2-1

Figure A2-2
-3-

Safety System Actuations

1.22

1.38

0.99

1.06

0.81 0.81

0.62

0.46

0.39 0.35 0.31 0.29 0.29

0.19 0.18

y = 1.6736e-0.1487x

R2 = 0.9733

0.0

0.5

1.0

1.5

2.0

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Actuations per Plant

Industry Average

Industry Average Trend

Forced Outage Rate (%)

7.86

10.13

7.60 7.90

8.89

7.79

9.40

6.76

7.54

10.21

10.73

5.20

4.24

3.00

1.70

y = 12.274e-0.0776x

R2 = 0.4597

0

5

10

15

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Forced Outage Rate (%)

Industry Average

Industry Average Trend

Figure A2-3

Figure A2-4
-4-

Equipment Forced Outages/

1000 Commercial Critical Hours

0.46 0.45

0.38 0.36 0.35

0.24 0.26

0.23 0.24 0.24

0.18 0.16

0.13 0.12 0.12

y = 0.5297e-0.1001x

R2 = 0.9578

0.0

0.2

0.4

0.6

0.8

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Equipment Forced Outages/

1000 Comm. Crit. Hours

Industry Average

Industry Average Trend

Collective Radiation Exposure

364 374

320

286 277

244

215 202

178 176

140 128 115 123 111

y = 422.43e-0.0936x

R2 = 0.9829

0

150

300

450

600

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Average Exposure per Plant

(Person-REM)

Industry Average

Industry Average Trend

Figure A2-5

Figure A2-6
-5-

Significant Events

0.77

0.90

0.45

0.40

0.25 0.26

0.21

0.17

0.08 0.10

0.04 0.03 0.02 0.05 0.04

y = 1.0817e-0.2584x

R2 = 0.9104

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002

Fiscal Year

Events per Plant

Industry Average

Industry Average Trend

Figure A2-7
-6-

All precursors—occurrence rate per reactor-calendar year, by fiscal year. No trend was

identified during the FY 1993–2001 period. The results for 2000 and 2001 are preliminary. 

A trend line is not shown in the figure because the trend is not statistically significant.

Accident Sequence Precursor Trends

Figure A2-8 below shows the occurrence rate per reactor-year for all Accident Sequence

Precursor (ASP) events by fiscal year. No statistically significant trend was observed in the

occurrence rate for all precursors (CCDP or CDP >10-6) during the 1993–2001 period (shaded

bars in the figure).

The trend is based on the number of all precursors starting in FY 1993. Data prior to FY 1993

are shown in the figure to provide historical perspective. Analyses for FY 2000 are preliminary

with final analyses nearing completion (pending resolution of peer review comments). The data

for FY 2001 are based on ongoing analyses that have undergone internal staff review. The

number of ASP events may decrease as the analyses are finalized. RES has provided

additional information on the ASP Program in SECY-03-0049, “Status of the Accident

Sequence Precursor (ASP) and the Development of Standardized Plant Analysis Risk (SPAR)

Models.”

Nonetheless, the staff will investigate the nature of the precursors to determine if there is an

explanation for the relatively low number of precursors between 1997 and 1999 and the

increasing number of potential precursors in 2000 and 2001. This evaluation will occur after

RES completion of the preliminary analyses of FY 2001 events. This evaluation will be a joint

project between NRR and RES as part of the Industry Trends Program. The evaluation may

include a review of the risk significance of the events, types of facilities involved, a

categorization of causes and factors for analyzed events and conditions, the time period for

analysis, and whether any additional actions are appropriate.

Figure A2-8
CDF Bi i

i

m

=

=

∑ λ

1

Description of the

Industry Initiating Events Performance Indicator

As discussed in SECY-02-0058, “Results of the Industry Trends Program for Operating Power

Reactors and Status of Ongoing Development,” RES developed about 10 additional riskinformed indicators for significant initiating events for both PWRs and BWRs. The staff

developed these indicators by updating data that were most recently published in NUREG5750, “Initiating Events at U.S. Nuclear Power Plants: 1987–1995.” These indicators were

selected because NUREG-1753, “Risk-Based Performance Indicators: Results of Phase 1

Development,” identified them as events that contributed >1% to industry core damage

frequency. The list of risk-significant initiating events is provided below.

1. Loss of Offsite Power

2. Loss of Safety-related Vital AC Bus

3. Loss of Safety-related Vital DC Bus

4. Small/Very Small Loss of Coolant Accident

5. Loss of Feedwater

6. Loss of Instrument Air/Control Valve

7. General Transients

8. Stuck Open Safety/Relief Valve

9. Loss of Heat Sink

10. Steam Generator Tube Rupture (PWRs only)

During FY 2002, RES and NRR built on this work by developing an overall industry-level

indicator for the initiating events cornerstone. An overall indicator can provide a better

representation of the overall risk from initiating events than multiple individual indicators of

initiating events with varying degrees of risk significance. For example, it is possible that there

could be an increase in loss of general transient events in any given year, but the overall risk

from all initiating events may actually have declined if the contribution to risk from that single

indicator is low and the contribution to risk from all other initiating events has declined.

This overall initiating events indicator, tentatively called the Industry Initiating Events

Performance Indicator (IIEPI), consists of an index of these risk-significant industry initiating

events. An index was developed for BWRs that has 9 risk-significant initiating events, and a

similar index was developed for PWRs that has 10 events. Each initiating event is weighted in

the index based on its contribution to industry core damage frequency (CDF).

The contribution of each initiating event to CDF is determined by multiplying the frequency of

occurrence by a risk weighting factor. The contribution of the 9 or 10 terms is then summed to

get the overall index for initiating events. Mathematically, this can be shown for an individual

plant by the following equation:

ATTACHMENT 3
-2-

where the risk weighting factor is represented by Bi

, and the initiating event frequency is

represented by i

. The risk weighting factor is a common measure used in probabilistic risk

assessments (PRAs) called the Birnbaum importance measure.

An industry average CDF can be calculated by using average industry values for the Birnbaum

importance measures and the initiating event frequencies. This is illustrated in Figure 1. 

B

N j ij B i

N

= ∑=

1

1 λ j

ij i

N

i i

N

C

t

= =

=

∑

∑

1

1

IIEPI Bj j j

m

= ∑ = λ 1

Figure A3-1 Integrated initiating event performance indicator calculations

In addition, should the concept be demonstrated successfully at the industry level, the indicator

could potentially be adapted to monitor plant-level performance as well by using plant-specific

values for either or both terms in the equation as appropriate. This approach would be similar

to the Mitigating Systems Performance Index (MSPI) that is currently being assessed in a pilot

program as part of the ROP PIs.

An interesting characteristic of the IIEPI is that it need not require any additional submission of

data from licensees, even at the plant level. The staff currently receives all required information

from existing data, including Licensee Event Reports (LERs) and Monthly Operating Reports

(MORs), as shown in Figure 2.
-3-

Figure A3-2 IIEPI data sources

An example of the IIEPI for PWRs is shown in Figure 3 for illustration only. These example

calculations show the feasibility of the indicator. No attempt has been made to validate or verify

these results.

Figure A3-3 Example calculation for the IIEPI for PWRs
82

79 79

82

77

74 73

70

73

78 76

16 16 16 15

22

25 25

28

21 19

23

2

5 5 333 4 4 5 4 2 11111111 3 1 1

0

10

20

30

40

50

60

70

80

90

100

2Q2000 3Q2000 4Q2000 1Q2001 2Q2001 3Q2001 4Q2001 1Q2002 2Q2002 3Q2002 4Q2002

Number of Plants

Licensee Response Column

Regulatory Response Column

Degraded Cornerstone Column

Multiple/Repetive Degraded Cornerstone Column

Action Matrix Summary

The NRC provided oversight of 103 operating power reactors using the Reactor Oversight

Process (ROP). On average, approximately 75% of the plants were listed in the Licensee

Response column of the ROP Action Matrix, which corresponds to the baseline level of NRC

oversight. The chart below shows trends in the numbers of plants that are listed in the

Regulatory Response, Degraded Cornerstone, Multiple/Repetitive Degraded Cornerstone, and

Unacceptable Performance columns of the Action Matrix, which correspond to increasing levels

of regulatory engagement with the licensee.

Figure A4-1

Notes for Figure A4-1:

1. This chart includes DC Cook units 1 and 2, beginning in 2Q/2001.

2. Davis-Besse is not included beginning in 2Q/2002 since under IMC 0350.

3. Data current as of January 2003.

ATTACHMENT 4
-2-

The chart appears to show a slight migration of plants from the Licensee Response Column to

the other columns in the Action Matrix. This can be attributed to several factors associated with

the initial start up of the ROP. First, the staff has continued to work with industry to improve the

ROP since initial implementation. These improvements include enhancements to its riskinformed inspection procedures, improved SDP Phase 2 notebooks, and improvements to the

guidance for performance indicators. A second factor is that the staff is much more familiar

with applying these risk-informed ROP tools and with the ROP processes. These factors have

likely enhanced the ability of both the NRC and licensees to identify the most risk-significant

aspects of licensee performance.

In addition, inspection findings that are determined to have greater than very low safety

significance (green) are counted for 4 quarters when determining the appropriate column of the

Action Matrix for licensees. Thus, for at least the first 4 quarters from the date of initial

implementation of the ROP on April 2, 2000, the number of plants moving out of the Licensee

Response Column has increased as inspection findings are accrued by plants under the ROP.