**NRC INSPECTION MANUAL**	IRAB

INSPECTION MANUAL CHAPTER 0308 ATTACHMENT 1

TECHNICAL BASIS FOR PERFORMANCE INDICATORS

Effective Date:  01/01/2021

Issue Date: 11/08/07		0308, Attachment 1

ii

Issue Date: 11/08/07		0308, Attachment 1

TABLE OF CONTENTS

0308.01-01  INTRODUCTION	1

0308.01-02  BASIS FOR SELECTING INITIAL SET OF PIs	1

0308.01-03  BASIS FOR SELECTING PI THRESHOLDS	2

0308.01-04  GENERIC PIs AND THRESHOLDS vs. PLANT SPECIFIC	5

0308.01-05  BENCHMARKING OF INITIAL SET OF PIs	5

0308.01-06  BASIS FOR EACH CURRENT PI AND THRESHOLD	7

06.01  Drill/Exercise Performance PI	7

06.01.01  Drill/Exercise Performance PI Thresholds	8

06.02	 ERO Drill Participation PI	9

06.02.01	 ERO Drill Participation PI Thresholds	10

06.03	Alert and Notification System Reliability PI	10

06.03.01	Alert and Notification System Reliability PI Thresholds	12

0308.01-07  OTHER PI PROGRAM ASPECTS CONSIDERED BUT NOT USED	12

0308.01-08  SECURITY CORNERSTONE	12

FIGURES

Figure 1 	Unplanned Scrams per 7000 Critical Hours Basis Summary Sheet	13

Figure 2 	Unplanned Scrams with Complications Basis Summary Sheet	14

Figure 3 	Unplanned Power Changes per 7000 Critical Hours Basis Summary Sheet	15

Figure 4 	Safety System Functional Failures Basis Summary Sheet	16

Figure 5 	Mitigating System Performance Index Basis Summary Sheet	17

Figure 6 	Reactor Coolant System Specific Activity Basis Summary Sheet	20

Figure 7 	Reactor Coolant System Leakage Basis Summary Sheet	21

Figure 8 	DEP Basis Summary Sheet	22

Figure 9 	ERO Drill Participation Basis Summary Sheet	23

Figure 10	ANS Reliability Basis Summary Sheet	24

Figure 11	Occupational Exposure Control Effectiveness Basis Summary Sheet	25

Figure 12	RETS/ODCM Radiological Effluent Occurrence Basis Summary Sheet	26

ATTACHMENTS

Attachment 1     PI Program Aspects Considered but Not Used	Att1-1

Attachment 2     Revision History For IMC 0308, Attachment 1	Att2-1

Issue Date:  10/16/20	2	0308 Att 1

0308.01-01  INTRODUCTION 	

Performance indicators (PIs), together with risk-informed baseline inspections, are intended to provide a broad sample of data to assess licensee performance in the risk-significant areas of each cornerstone.  They are not intended to provide complete coverage of every aspect of plant design and operation.  It is recognized that licensees have the primary responsibility for ensuring the safety of the facility.  Objective performance evaluation thresholds are intended to help determine the level of regulatory engagement appropriate to licensee performance in each cornerstone area.  Furthermore, based on past experience it is expected that a limited number of risk-significant events may occur with little or no advanced indication of declining performance.  Follow up inspections will be conducted to ensure that the cause of these events are well understood and that licensee corrective actions are adequate to prevent recurrence.

As described in Commission paper SECY-99-007, the Agency established a task group to identify appropriate PIs.  The PIs selected for each cornerstone, along with performance thresholds, are described in Figures 1 through 12 of this Attachment.  These thresholds were selected for consistency with the performance threshold conceptual model provided in Exhibit 12 of Inspection Manual Chapter 0308, “Reactor Oversight Process Basis Document.”  They correspond to levels of performance requiring no additional regulatory oversight (the "Licensee Response Band"), performance that may result in increased oversight (the "Increased Regulatory Response Band" across the Green/White threshold), performance that will result in specific NRC actions (the "Required Regulatory Response Band" across the White/Yellow threshold), and performance that represents an unacceptable loss of safety margin (across the Yellow/Red threshold).  For some PIs, White/Yellow or Yellow/Red thresholds were not identified, because the indicators could not be directly tied to risk data.  Should licensee performance result in a PI crossing the Yellow/Red threshold, margin would still exist before undue risk to public health and safety would be present.

Once the PIs and corresponding thresholds were selected, a task group performed a benchmarking analysis to compare the indicators against several plants that had been previously designated by the Agency as having either poor, declining, average, or superior performance.  The analysis indicated that the PIs could generally differentiate between poor and superior plants, but were not as effective at differentiating average levels of performance.  In some instances, the cause of the poorly rated plants was due to design or other issues for which valid PIs have not been developed.  Issues such as these are within the scope of the risk-informed baseline inspection program.

0308.01-02	BASIS FOR SELECTING INITIAL SET OF PIs 	 

Where possible, the task group sought to identify PIs as a means of measuring the performance of key attributes in each of the cornerstone areas.  In selecting PIs, the task group tried to select indicators that:  (1) were capable of being objectively measured; (2) allowed for the establishment of a risk-informed threshold to guide NRC and licensee actions; (3) provided a reasonable sample of performance in the area being measured; (4) represented a valid and verifiable indication of performance in the area being measured; (5) would encourage appropriate licensee and NRC actions; and (6) would provide sufficient time for the NRC and licensees to correct performance deficiencies before the deficiencies posed an undue risk to public health and safety.

Issue Date: 11/08/07		0308, Attachment 1

Issue Date:  10/16/20	2	0308 Att 1

0308.01-03.	BASIS FOR SELECTING PI THRESHOLDS 	 

The concept for setting performance thresholds includes consideration of risk and regulatory response to different levels of licensee performance.  The approach is intended to be consistent with other NRC risk-informed regulatory applications and policies (e.g. Regulatory Guide [RG] 1.174) as well as consistent with regulatory requirements and limits.  The thresholds were selected to be risk-informed to the extent practical, but also accommodate defense-in-depth and indications based on existing regulatory requirements and safety analyses.  Thresholds were established so that sufficient margin exists between nominal performance bands to allow for licensee initiatives to correct performance problems before reaching escalated regulatory involvement, and sufficient margin exists to allow for both NRC and licensee diagnostic and corrective actions to be effectuated in response to declining performance.  Thresholds have been established sufficiently above the point of unsafe plant operation to allow the NRC sufficient opportunity to take appropriate action to preclude operation in this condition.

The four performance bands and their general performance characteristics are as follows:

∙	The Green band is characterized by acceptable performance in which cornerstone objectives are fully met; nominal risk with nominal deviation from expected performance. Performance problems would not be of sufficient significance that escalated NRC engagement would occur.  Licensees would have maximum flexibility to "manage" corrective action initiatives.  The threshold for this band would involve performance that would be outside the normal range of industry historical performance and risk.

∙	The White band would be entered when licensee performance is outside the normal performance range, but would still represent an acceptable level of performance.  Cornerstone objectives met with minimal reduction in safety margin; outside bounds of nominal performance; within Technical Specification Limits.  Degradation in performance in this band is typified by changes in risk of up to Δ10-5 Core Damage Frequency (CDF) or Δ10-6 Large Early Release Frequency (LERF).  The CDF and LERF threshold characteristics were selected to be consistent with RG 1.174 applications.

∙	The Yellow band involves a level of licensee performance that is still acceptable with cornerstone objectives met, but with significant reduction in safety margin; Technical Specification limits reached or exceeded.  Degradation in performance in this band is typified by changes in risk of up to Δ10-4 CDF or Δ10-5 LERF.  These threshold characteristics and required regulatory response are also selected to be consistent with risk-informed regulatory applications and mandatory actions for regulatory compliance.

∙	The Red band is typified by changes in performance that are indicative of changes in risk greater than Δ10-4 CDF or Δ10-5 LERF.  Plant performance represents an unacceptable loss of safety margin.  It should be noted that should licensee’s performance result in a PI reaching the Red band, margin would still exist before an undue risk to public health and safety would be presented.

As described in Commission Paper SECY-99-007, Attachment 2, Appendix H, PI thresholds in some instances could be directly tied to probabilistic risk assessment data, such as those for scrams and safety system unavailability.  A sample of plants with probabilistic risk analysis (PRA) models available was selected to cover a spectrum of "typical" designs.  Normal 

Issue Date:  10/16/20	2	0308 Att 1

performance ranges were identified, and core damage frequency sensitivity analyses were performed to evaluate the effects of departures from normal performance.  This information was used to set PI threshold values that corresponded to the nominal and declining performance bands.

PRA models were used to provide a risk-perspective on the thresholds for the Initiating Events and Mitigating Systems cornerstones.  This was done by performing sensitivity studies to investigate how the CDF of the plants varies as the values of the PIs change.  The analyses were performed by NRC staff or their contractors with the SAPHIRE code, using seven NRC-developed simplified models (SPAR models) and six licensee PRA models that were available at what was then called Idaho National Engineering and Environmental Laboratory.  In addition, results from twelve licensee PRA models were provided by the Nuclear Energy Institute (NEI).  While, for most cases, the PRA results were able to provide information relevant to establishing the White/Yellow and Yellow/Red thresholds, in some cases, the CDF results were insensitive to large changes in the parameters corresponding to the PIs.  For these cases, an alternate approach to choosing thresholds was required.

To determine the Green/White threshold, it was necessary to define what was acceptable performance.  The Green/White threshold for the PI was chosen to be commensurate with a generically achievable level of performance and takes into consideration the statistical variability arising from the random nature of the contributing events as seen across the entire population of plants.  For the purpose of establishing the Green/White threshold, histograms were provided by NEI of the maximum value recorded for each PI for all the plants.  The threshold was determined by the simple approach of choosing a value to no more than two significant figures that is such that about 95% of the plants have observed data values that would be in the Green band, and is therefore established on a generic basis.  This method depends only on the number of plants with less than acceptable performance, but not on determining by how much their performance exceeds the norm.  Alternative approaches, such as using the mean plus two standard deviations of the PI values to set the threshold puts more weight on the actual values of the PIs, and could be biased by the poor performers in a non-conservative direction.  This threshold value may be higher or lower than the value of the corresponding parameter used in licensee’s PRAs.  That the threshold is reasonable from a risk standpoint was demonstrated by the fact that use of the threshold in the sample of PRA models used for the sensitivity studies would have resulted in an increase in CDF of less than 10-5/reactor year.

There is no clear regulatory definition of unacceptable risk in numerical terms that can be used to define unacceptable performance.  However, in RG 1.174, the NRC has established acceptance guidelines for allowing changes to the licensing basis that relate to changes in CDF and LERF.  Specifically, for CDF, an increase in the range of 10-6 to 10-5/reactor year would be acceptable, under certain conditions and with staff review and approval, while changes resulting in an increase greater than 10-5/reactor year would not be acceptable.  While these acceptance guidelines are intended for permanent changes to the licensing basis, it was consistent to also apply these to changes resulting from operating practices, using the argument that if the degradation in performance were uncorrected, it would lead to a permanent increase in CDF.  Furthermore, a change in CDF of 10-5/reactor year is used in the staff’s regulatory analyses as one element in determining the requirement for a backfit.  Thus, it was decided that the White/Yellow threshold should be determined on the basis of sensitivity analyses to identify that mean value of the PRA parameter associated with the PI that would increase CDF by an amount that corresponds to a substantially declining performance, which has been chosen as 10-5/reactor year.  For the PI to be a meaningful indicator, this increase must be significant compared with the expected statistical variation captured by the setting of the Green/White 

Issue Date:  10/16/20	2	0308 Att 1

threshold.  In comparison with the way the Green/White threshold is determined, this approach is somewhat conservative in that it does not increase the value to compensate for the expected statistical variation.  However, since this is only an indicator of performance rather than a criterion for regulatory action, this is considered appropriate.

A truly unacceptable performance would likely correspond to a change in CDF well in excess of 10-5/reactor year, and is chosen as corresponding to a change in CDF of 10-4/reactor year.  The Yellow/Red thresholds were determined by identifying the PI values that would correspond to increases in CDF of 10-4/reactor year.

Other PI thresholds could not be specifically tied to probabilistic risk data.  In such cases, the PI thresholds were tied to regulatory requirements or were based on the professional judgement of the NRC staff.  For example, under the barrier integrity cornerstone, reactor coolant system (RCS) activity is a good measure of the integrity of the fuel cladding, but the performance thresholds chosen were based on technical specifications.

For two PIs (Unplanned Power Changes and Safety System Functional Failures [SSFFs]), no thresholds have been identified for the Yellow and Red Bands because the indicators could not be directly tied to risk data.  These two indicators have provided good correlation with plant performance in the past and they are considered to be leading indicators of the more risk-significant indicators: (Unplanned Scrams, Scrams with Complications, and Mitigating System Performance Index (MSPI)). The Barrier Integrity cornerstone PIs (RCS Activity and RCS Leak Rate) do not have thresholds identified for the Red Band because their lower thresholds are based on regulatory requirements (technical specifications).  Individual plant technical specifications would require plant shutdown within a short time after the regulatory limits were exceeded.  The Emergency Preparedness, and Occupational and Public Radiation Safety cornerstones do not have thresholds identified for the Red Band.  There is no risk basis for a determination that a certain degraded level of performance reflected by these indicators can be correlated into mandatory plant shutdown.  It is expected that declining performance in the areas monitored by these indicators would be arrested by increased licensee corrective actions and by increased NRC attention up to and including the issuance of orders.

The Unplanned Scrams with Complications PI does not have Yellow or Red bands because the PI is not tied directly to risk significance.  However, it does monitor the cumulative effect of scrams that have the potential to present additional challenges to plant operations staff and therefore may be more risk significant than uncomplicated scrams.  During development of this PI it was benchmarked against significant events tracked by the industry trends Accident Sequence Precursor program for data available from 2003 through mid-2004, and MD 8.3, “Incident Investigation Program” for data available for 2005 through mid-2006.  The PI was triggered for all ASP events and all MD 8.3 reactive inspections involving reactor scrams when there was sufficient information provided.  This indicated that the PI had the ability to detect and trigger on events the NRC considered risk significant and probably lower threshold precursor events as well.  The PI was also benchmarked against industry plant scram data provided by NEI from 1995 to 2000.  This benchmarking showed that the PI would result in approximately 5% of the industry with a white indicator.  The PI was also based on a rolling 4 quarters, representing more current performance than the 12 quarters used by the previous Loss of Normal Heat Removal PI. 

Issue Date:  10/16/20	2	0308 Att 1

0308.01-04	GENERIC PIs AND THRESHOLDS vs. PLANT SPECIFIC 	 

As described in Section 3 above, the thresholds were selected to be risk-informed to the extent practical.  Because of significant differences among plants in both Nuclear Steam Supply System (NSSS) and balance-of-plant equipment, and operations, the change in risk associated with a particular PI value may vary considerably from one plant to another.  The MSPI is a more risk-informed performance indicator that replaced the safety system unavailability indicators. 

0308.01-05	BENCHMARKING OF INITIAL SET OF PIs   

An initial benchmarking analysis was performed by NEI on a set of eight plants that they categorized as excellent, average, or declining performers, plus eight NRC watch list plants.  The indicators they used were the ones originally proposed in their draft white paper, (RCS Activity, RCS Leakage, Containment Leakage, Unplanned Scrams, Safety System Actuations [SSAs], and Transients) except the Reliability and Availability of Risk-Significant systems, structures, and components and Shutdown Operating Margin.  Since NEI did not have unavailability data at the time, they used SSFs from the NRC PI program as a surrogate.  They used monthly or quarterly data from July 1995 through June 1998 for RCS activity, RCS leakage, and containment leakage provided by the plants.  NEI also used annual data from 1990 to 1997 on Scrams, SSAs, and SSFs from the Office for Analysis and Evaluation of Operational Data (AEOD) annual reports, and data from 1990 to 1995 on Transients from an Nuclear Utilities Service (NUS) database of licensee monthly reports.  NEI documented insights from their analysis of these data, including typical PI characteristics for each plant performance category which showed a correlation between the PIs and performance.  These insights were obtained primarily from the SSF and Transients indicators.  They concluded that the set of indicators provided an overall perspective of safety performance, and that the indicators do distinguish between levels of performance in enough of the indicators simultaneously to be a viable assessment tool.

Once the PIs and corresponding thresholds were selected for the Initiating Events, Mitigating Systems, and Barrier Integrity cornerstones by the NRC task group, the NRC staff performed additional independent benchmarking analysis to answer the following questions:

	1.	Do the PIs as a set differentiate between superior, average, declining trend, and watch list plants as designated by the Senior Management Meeting (SMM) process?

2.	How effective are individual PIs at differentiating between plants with different levels of performance as designated by the SMM process?

3.	Do the PIs demonstrate timely response (i.e., do not go directly from green to red)?

4.	Do the PIs show declining trends for plants in SMM designated performance categories prior to SMM actions?  If so, which ones are most effective?  If not, would they be expected to show a declining trend?

5.	Do the PIs show declining trends prior to accident sequence precursor (ASP) events?  If so, which ones are most effective?

6.	How well does the set of PIs conform to those selected by Arthur Andersen for use in the trending methodology that was used in the SMM process?

7.	Do small decreases in the Green/White thresholds capture more of the watch list and declining trend plants (sensitivity analyses)?

Issue Date:  10/16/20	2	0308 Att 1

To perform its analysis, the NRC compared the indicators against the following set of 17 plants that had been previously designated by the NRC SMM as superior performers, average, trending, and watchlist plants:

Superior:		Callaway			Average:		Davis-Besse

				Vogtle 1&2						Point Beach 1&2

											TMI 1

Trending:		Cooper			Watch List:		Crystal River

				D.C. Cook 1&2					Indian Point 3

				Hope Creek						Maine Yankee

											Millstone 1, 2, 3

This independent analysis confirmed that the PIs could generally differentiate between poor and superior plants, but were not as effective at differentiating average levels of performance.  The unplanned power changes and SSF PIs appeared to be the most closely tied with prior NRC judgements about performance.  In some instances, the cause of the plants rated poorly by the agency was due to design or other issues for which valid PIs have not been developed.  It is expected that these plants would continue to be identified by the inspection program.  The results of this benchmarking by the industry and the NRC is described in more detail in Commission Paper SECY-99-007, Attachment 2, Appendix I.

Benchmarking analysis was conducted to evaluate the Drill/Exercise Performance (DEP) and Alert and Notification System (ANS) Reliability PIs for the Emergency Preparedness cornerstone.  For the DEP analysis, data was collected from 70 plants from 1994 through 1997.  For the ANS Reliability analysis, siren data was obtained from 20 plants from 1995 through 1997.  Benchmarking results showed that in general, the plants identified by this analysis were consistent with those identified as having a deteriorating trend in emergency preparedness (EP) performance.

To benchmark the Occupational Exposure Control Effectiveness PI, 14 sites were identified whose performance in occupational radiation protection activities was considered to be below or declining from industry standards.  Additionally, 12 sites were identified who were considered to be good performers in occupational radiation protection activities.  NEI provided data from 1996 through 1998 on 9 of the 14 poor performers and 7 of the 12 good performers.  The staff also collected the systematic assessments of licensee performance (SALP) categories in Plant Support for these plants, since plants with a SALP score of 2 or 3 in that functional area normally have poor radiation protection programs.  The PI data for the 16 plants was analyzed by the staff to compare the highest PI values to the thresholds and to identify the corresponding performance band.  The benchmarking analysis showed reasonable agreement with the perceived performance of the plants.

To benchmark the Radiological Effluent Technical Specifications (RETS)/ Offsite Dose Calculation Manual (ODCM) Radiological Effluent Occurrence PI, the NRC and NEI both identified 15 sites whose performance in effluent monitoring and offsite releases was considered to be below or declining from industry standards.  The staff also identified 12 sites considered to be good performers.  NEI provided data from 1995 through 1997 on 11 of the 15 poor performers and 6 of the 12 good performers.  The PI data for the 17 plants was analyzed by the staff to compare the highest PI values to the thresholds and to identify the corresponding performance band.  The benchmarking analysis showed some agreement with the perceived 

Issue Date:  10/16/20	2	0308 Att 1

performance of the plants.  The plants considered to be good performers had generally low PIs and none of them entered the White band; 4 of the 11 plants considered to be poor performers had PIs in the White band.

0308.01-06	BASIS FOR EACH CURRENT PI AND THRESHOLD 	  

Figures 1 through 15 provide detailed information regarding each PI, including it’s objective, the cornerstone key attributes it measures, the calculational method, the current performance thresholds and their basis, and the significant changes to the PI and/or threshold and their bases.  NEI 99-02 also describes the data and calculations for each PI and describes the quarterly indicator reports that are to be submitted for use in the assessment process.

Additional detail regarding the background and development of some of the PIs are as follows.

06.01	Drill/Exercise Performance PI 	 

The concept for the DEP PI began as three separate indicators:

•	Accurate and timely classifications

•	Accurate and timely notifications

•	Accurate and timely protective action recommendations (PARs).

The percentage success rate of these would be measured in drills, exercises and actual events.  This would largely be accomplished through licensee self assessment programs (i.e., the critique program).  The definition of a "drill" was problematic as many sites use many different types of drills.  The broadest definition of a drill that would be acceptable to NRC was sought.  The drill would require a formal assessment of the measured activities and documentation suitable for inspection.  It should be noted that while industry acceptance was obtained, several programs had to make significant changes to meet the criteria.

An analysis of the typical number of opportunities for the above three PIs proved informative:

Assuming:

•	one unit plant

•	six shifts of operators

•	two training cycles per year would include EP drill elements

•	seven full scale drills in two years

This results in about 60 opportunities/year:

8 opportunities per full scale drill (3 classifications, 3 notifications, 1 PAR and 1 PAR notification) for a total of 56.

2 opportunities (1 classification and 1 notification) per shift training evolution for each of 6 crews, twice per year for a total of 96 over 2 years.

This results in about 60 opportunities per year as follows: 25 classifications, 25 notifications, 5 PARs and 5 PAR notifications.

Issue Date:  10/16/20	2	0308 Att 1

The results of this analysis indicated that this number of opportunities would not support 3 separate PIs.  Separating the PIs was an attempt to diagnose performance problems, rather than create a licensee response band, wherein the licensee could diagnose and correct performance problems.  There was no need to separate the PIs because these measures indicate the status of the same underpinning elements (e.g., training, qualifications, equipment, procedures, correction of weaknesses identified in drills).  These EP program elements must be adequate for a high level of success in performing the risk significant activities measured by the PI.  It was determined that defining 3 separate PIs would be too fine a measurement.  Rather, the focus should be on licensee success in implementing the most risk significant elements of EP, which would allow the measurements to be combined into one PI.  This became the DEP PI.  When combined, the projection of 60 opportunities per year, measured over two years, appeared to provide a good sample of meaningful performance.

The creation of "unintended consequences" was a concern during PI development.  The working group developing the PIs remained conscious of the possibility that PI measures would drive performance or resources in a manner that was not risk-informed.  The DEP PI was recognized as driving resource allocation in an appropriate manner.  NRC regulations establish a minimum number of drills and this was the level of DEP PI reporting that was mandated by the PI.  However, most licensees perform many more drills than required by the regulatory minimum, and in any case, the number of drills are not limited by regulation or the PI.  The result is that if the licensee saw declining performance, additional drills could be scheduled to improve the PI value.  There was some initial concern within the NRC that this could detract from the meaningfulness of the PI.  A limit on the number of opportunities counted in the PI was discussed.  However, the objective of the EP Cornerstone provided insight in that it is licensee proficiency that is important, not the number of drills.  Implementation of additional drills to enhance proficiency is in keeping with the objective and was eventually seen as an advantageous design feature.  Said another way, there is no regulation to require additional drills when performance declines, but the PI encourages licensees to take this action voluntarily to keep the PI Green.

06.01.01	Drill/Exercise Performance PI Thresholds 	  

The 90% Green band threshold was selected by a group of subject matter experts including NRC, State and industry personnel.  It was based on a proposal from NRC staff from data collected from EP exercise inspection reports for the period 1994 through 1997.  While licensees conduct many additional drills, NRC inspection report data was only available for the exercises.  Success rates for the DEP measured activities could be inferred from inspection reports if it is assumed that inspectors would have identified any significant problems with classification, notification or PAR development.  The absence of findings was considered successful performance of the DEP activities.  It was estimated that each exercise represented 10 DEP opportunities (4 classifications, 4 notifications and 2 PARs.)  Given these assumptions, the data included some 1410 opportunities with 51 failures for a success rate of 96%.

The standard deviation of the success rate was calculated as the square root of the average success rate, or 9.7%.  It was recognized that this may not be a mathematically rigorous use of the standard deviation.  However, the results suggested a range of performance that would be expected given natural variations in such performance.  Said another way, if the success rate was worse than about 96% minus 9.7%, or 87%, the performance is not within "normal" variation and should be investigated further.  Given the lack of rigor in this analysis, it was 

Issue Date:  10/16/20	2	0308 Att 1

decided to conservatively round up the estimate to 90% (i.e., performance at less than a 90% success rate warrants additional NRC oversight of corrective actions associated with Emergency Response Organization [ERO] performance).  Performance above a 90% success rate defines the licensee response band and corrective actions may be left to the licensee with NRC oversight through the baseline inspection program.

The group of subject matter experts agreed with the 90% threshold as a high standard appropriate for EP programs and yet flexible enough to provide a viable licensee response band.  It should be noted that the calculational method of the DEP PI threshold encourages a higher number of opportunities (e.g., if performance failures cluster in a single drill, the PI value is less likely to cross the threshold if there are a larger number of total opportunities than if there are a smaller number of opportunities).  This feature encourages more drills and contributes to the objective of the EP Cornerstone.

The Yellow band threshold was chosen in a similar manner.  A success rate less than three standard deviations (3 x 9.7%) below the average was seen as performance that would require NRC involvement with corrective actions.  This results in a success rate of about 67% which was conservatively rounded up to 70%.

The NRC staff proposed that a Red band threshold was not appropriate.  Performance in the Red band is best determined by inspection and not voluntary PIs.

Finally, there was an initial NRC proposal to split the White and Yellow band thresholds into short-term (six month) and long-term (24 month) thresholds.  When this was discussed with project management and NRC PI experts, they suggested a split threshold was too fine an analysis given the available data set.  Additionally, none of the other PI were considering splitting short-term and long-term thresholds and the lack of consistency was a concern.  The short-term threshold was therefore eliminated.

06.02	ERO Drill Participation PI 	  

Development of this PI flowed from the Performance Assessment Workshop outcome that some measure of ERO readiness would make an appropriate PI, and proceeded through public meetings between NRC and industry representatives.  The PI was configured by NRC staff and presented to the industry.  It met with some resistance because it appeared to be an activity measure, was not required by regulation, and penalized sites with large EROs that exceeded regulatory requirements.  On the other hand, NRC staff considered it a necessary compliment to the DEP PI because:

•	DEP requires only a small number of opportunities and there are no requirements regarding the participants in these opportunities.  A small number of participants performing in one exercise/year could meet the Green band threshold, as long as their failure rate was less than 10%.  There are no requirements to rotate participants and the same team could participate every year.  They could be coached through numerous "dress rehearsals" and only then would their performance be assessed for DEP.  This would not be representative nor sufficient for NRC oversight (i.e., it would not create a licensee response band).

•	DEP measures the most risk significant areas of EP.  However, there are several other areas that are important, and were previously inspected but are not covered by the DEP PI (e.g., damage control, worker protection, accident assessment, procedure 

Issue Date:  10/16/20	2	0308 Att 1

	quality, training program, facility readiness).  A premise of the ERO PI is that broad participation in drills by ERO members will result in their identifying weaknesses in areas not covered by the DEP PI.  Inspection of the licensee corrective action program ensures that these weaknesses are getting corrected and inspection of drill critiques ensures weaknesses are getting identified.  The premise contends that broad ERO participation and the focused NRC inspections noted above are more effective in the identification and correction of weaknesses than the previous NRC oversight through infrequent direct inspection.

The ERO PI was defined in a manner that provided licensee flexibility.  Only key ERO positions were to be tracked, a generous Green band threshold was established and flexibility in configuring drills for training purposes was provided.

A key aspect of ERO is that it is linked to DEP (i.e., for participation to contribute to the ERO PI, the success rate of classification, notification and PAR development must also contribute to the DEP PI).  In a sense, DEP and ERO are a PI set.  An attempt was made to combine the two PI, but this complicated the system and added little value.  This linkage and industry acceptance of the PI creates a robust licensee response band.  The PI allows the following statement for the Green band for the EP program:

At least 80% of the ERO participates in drills and they have a success rate of at least 90% in the most risk significant aspects of the EP Cornerstone.

This is a strong statement about the EP program that could not be made under the previous oversight program.  Under the old Core Inspection Program, a team of inspectors would observe one exercise every two years and directly judge performance.

06.02.01	ERO Drill Participation PI Thresholds 	  

There was no historical data available to NRC for drill participation.  Some licensees did keep such data, but the standards used were not universal and even if the data could be obtained, it would not be standardized between sites.  Given the purpose of the PI and the fact that there is no regulatory requirement for anyone to perform in a drill, a generous threshold was considered adequate.  The 80% White band threshold was proposed by NRC and accepted by industry but questioned by internal NRC stakeholders.  Some inspectors felt it was too generous to be meaningful.  It was agreed to test the thresholds through the pilot program and initial implementation to determine if adjustments were warranted.

The Yellow band threshold was set at 60% based on the similar spread of the DEP PI thresholds.  However, many industry stakeholders questioned whether any Yellow band was appropriate for this PI, for the reasons discussed above.  The threshold was sufficiently generous to allay concerns that any licensee could cross the threshold without significant degradation of the EP program.  There was no consideration of a Red band threshold as it was thought that inspection was the proper mechanism to identify findings of such significance.

06.03	Alert and Notification System Reliability PI 	  

This PI was developed out of the recognition that some measure of licensee performance in the maintenance of EP related equipment was appropriate.  When the spectrum of EP related 

Issue Date:  10/16/20	2	0308 Att 1

equipment is considered, the ANS manifests as the most risk significant.  The objective of the EP Cornerstone can not be met unless there is a mechanism to rapidly notify the public of the need to take protective actions.  That mechanism is the ANS and the emergency alert system (the system that uses local radio channels to alert the public of emergencies.)  Generally, the licensee maintains the ANS and local authorities activate it.

Licensees must annually report ANS testing results to the Federal Emergency Management Agency (FEMA), usually via the State.  The data was readily available and it was thought that this PI could be a simple extension of the report to FEMA and perhaps replace the need for that report.  A few stakeholders stated that the annual siren test data almost never showed a problem and thus the PI would never indicate the need for additional action.  However, one of the premises of the ROP development process was that a PI should not be rejected just because the industry does a good job in the area covered.  Some PIs measure important performance even though that performance is excellent (e.g., routine radiological releases).

The definition of the PI turned out to be problematic since the licensee annual reports to FEMA are not standardized.  The guidance for reporting is contained in FEMA/REP-10, "Guide for the Evaluation of Alert and Notification Systems for Nuclear Power Plants."  The guidance asks for a simple average of all regularly conducted tests, which is interpreted to be successful tests divided by tests times 100, for a percentage which represents the simple average.  FEMA considers 90% to be acceptable.  However, many licensees report availability (e.g., days the sirens are operable divided by siren days times 100) for a percentage availability.  Some licensees count the full time between siren tests as days lost when a siren is found inoperable, some count only the day of the test and the time to repair as lost.  Under the guidance of the NRC Maintenance Rule, the time period of T/2, or half the time between tests is used to calculate availability.  This last method is statistically correct, but is not known to be used by any EP programs for ANS statistics.  The consensus view was that NRC could not ask for a PI definition that used a method different from the FEMA guidance, and so the FEMA guidance was used for the PI.  This measurement is defined as "reliability" under the language of the NRC Maintenance Rule and is a measure of successful tests.

Stakeholders have expressed concern over the use of reliability because periods of known outage are not captured by the PI (e.g., when a siren is found out of service but repaired before the next test).  It may be noted that availability is not conservative in all cases, for example:

When a siren fails and is repaired the same day, only 1 day of availability (out of 365) is lost.  Where as with reliability, when a siren fails during a test, 1 of 26 annual tests is lost.

Overall it would seem that availability is a better measure because it more accurately reflects the status of the ANS.  However, a change in Federal guidance is considered necessary before a change to the PI would be appropriate and the minor nature of the flaw in the current PI definition did not seem to justify the level of effort necessary for the change.

A four-quarter rolling average was chosen for the PI to align with the rest of the ROP PIs.  This creates a difference with FEMA reporting which is on an annual basis.  Further, many licensees do not report reliability to FEMA.  This PI complicated their reporting because they would have differing numbers for the PI and annual FEMA report.  Even for those using the FEMA guidance definition, only the fourth quarter report matches the report to FEMA.

Issue Date:  10/16/20	2	0308 Att 1

06.03.01	Alert and Notification System Reliability PI Thresholds   

An analysis of FEMA ANS reporting data was performed.  The reported percentages were used irrespective of the method of calculation used.  Twenty plants submitted 3 years of data.  The average was 98%.  A reliability rate lower than 90% would be unacceptable to FEMA, per the FEMA/REP-10 guidance.  The 90% rate appeared to define the "required regulatory response band," or the Yellow band.  It was thought appropriate to approximate the midpoint between the average reliability rate and the rate unacceptable to FEMA as the White band threshold and 94% was chosen.  Most ANS systems operate well above 94%.  In the 60 plant-years of data that were used to develop this threshold, only one plant was in the White band and no plants were in the Yellow band.

There was consensus with industry that these thresholds were appropriate because crossing a threshold indicated a program for which additional NRC inspection was appropriate.  The historically excellent performance of industry in ANS maintenance contributed to a lack of concern regarding these thresholds.

Issue Date:  10/16/20	2	0308 Att 1

0308.01-07.	OTHER PI PROGRAM ASPECTS CONSIDERED BUT NOT USED 	  

Table 1 lists several aspects of the PI program that were considered during the development of the ROP, but not used, and the basis for not including them in the new oversight process.

0308.08.	SECURITY CORNERSTONE 	

Although the NRC is actively overseeing the security cornerstone, the Commission has decided that the description of this PI and its results will not be publicly available to ensure that potentially useful information is not provided to a possible adversary. 

Issue Date: 11/08/07		0308, Attachment 1

Issue Date:  10/16/20	2	0308 Att 1

Figure 1 	Unplanned Scrams per 7000 Critical Hours Basis Summary Sheet 	

Figure 2 	Unplanned Scrams with Complications Basis Summary Sheet 	

Figure 3 	Unplanned Power Changes per 7000 Critical Hours Basis Summary Sheet 

Issue Date: 11/08/07		0308, Attachment 1

Issue Date:  10/16/20	2	0308 Att 1

Figure 4 	Safety System Functional Failures Basis Summary Sheet 

Issue Date: 11/08/07		0308, Attachment 1

Issue Date: 11/08/07		0308, Attachment 1

Issue Date: 11/08/07		0308, Attachment 1

20

Issue Date: 11/08/07		0308, Attachment 1

Figure 5 	Mitigating System Performance Index Basis Summary Sheet 

Issue Date: 11/08/07		0308, Attachment 1

Issue Date:  10/16/20	2	0308 Att 1

Figure 6 	Reactor Coolant System Specific Activity Basis Summary Sheet 	

Figure 7 	Reactor Coolant System Leakage Basis Summary Sheet 	

Figure 8 	DEP Basis Summary Sheet 	

Figure 9 	ERO Drill Participation Basis Summary Sheet 	

Figure 10	ANS Reliability Basis Summary Sheet 	

Figure 11	Occupational Exposure Control Effectiveness Basis Summary Sheet 	

Figure 12	RETS/ODCM Radiological Effluent Occurrence Basis Summary Sheet 	

Attachment 1:  PI Program Aspects Considered but Not Used

Issue Date: 11/08/07		0308, Attachment 1

Issue Date:  10/16/20	Att 1-2	0308 Att 1

Attachment 2:  Revision History 

IMC 0308, Attachment 1 

Issue Date:  10/16/20	Att 2-2	0308 Att 1