Independent Evaluation of the Reactor Oversight and 

Incident Response Program 

December 31, 2008 

This report was prepared by FocalPoint Consulting Group based in part on information provided by the NRC, 

constituents and stakeholders of the NRC, and others with relevant knowledge. The information was evaluated but 

not independently verified by FocalPoint. The assessment contained in this report was developed independent of the 

individuals that provided the information. 
Table of Contents 

Executive Summary ........................................................................................................................ 1 

1. Introduction ................................................................................................................................. 8 

2. Contribution to Safety ............................................................................................................... 15 

3. Budget Performance Integration ............................................................................................... 37 

4. Internal Evaluation and Improvement ...................................................................................... 52 

5. Public Communications ............................................................................................................ 60 

6. Interaction with Other Agencies ............................................................................................... 66 

7. Financial Management .............................................................................................................. 70 

8. Enforcement and Allegations .................................................................................................... 78 

9. Staff Development .................................................................................................................... 84 

Appendix ....................................................................................................................................... 91 

A. Bibliography............................................................................................................................. 92 

List of Figures 

Figure 1 – Key Program Activities and Objectives ........................................................................ 9 

Figure 2 - Locations of Commercial Operating Reactors ............................................................. 12 

Figure 3 - Reactor Oversight Framework ..................................................................................... 22

Figure 4 - Number of Inspection Findings per Plant .................................................................... 24 

Figure 5 - Number of PI Findings per Plant ................................................................................. 25

Figure 6 - Sample Illustration of Performance Framework .......................................................... 50 

Figure 7 - Resident Inspector Turnover ........................................................................................ 87 

List of Tables 

Table 1 - Program Strategic Outcome Targets ............................................................................. 19 

Table 2 - Industry Trends Indicators ............................................................................................. 20 

Table 3 - ROP Action Matrix Summary ....................................................................................... 23 

Table 4 - Selected Annual ROP Self-Assessment Performance Metrics ...................................... 27 

Table 5 - Budget Program Costs per MWh .................................................................................. 33 

Table 6 - Reactor Oversight and Incident Response Program Budget Authority and FTE .......... 38 

Table 7 - Direct vs. Indirect Cost Analysis of Nuclear Reactor Safety Program Budget ............. 48 

Table 8 - Example of Crosswalk of Program Budget ................................................................... 49 

Table 9 - Selected Evaluation Efforts ........................................................................................... 52 

Table 10 - Timeliness of Resolving Enforcement Actions ........................................................... 80 

Table 11 - Timeliness of Allegations Resolution ......................................................................... 81 
Executive Summary 

Background 

In August 2008, the Nuclear Regulatory Commission (NRC) hired FocalPoint Consulting Group, 

a management consulting firm, to perform an independent evaluation of the NRC’s Reactor 

Oversight and Incident Response (RO-IR) Program (the Program). This report presents the 

results of that evaluation effort. The Program plays a key role in fulfilling the NRC’s mission of 

ensuring adequate protection of public health, safety and security, and protection of the 

environment. The Program’s activities are authorized under the Atomic Energy Act of 1954, as 

Amended, and the Energy Reorganization Act of 1974. The Program’s overarching objectives, 

as set out in the legislation, include oversight of the nation’s commercial nuclear power plants 

and other regulated licensees and coordination of the federal government’s response to 

radiological incidents occurring in licensed facilities, in order to maintain adequate protection of 

public health and safety. 

Key activities in support of those objectives include inspections, performance assessments, event 

assessments, and response related to incidents occurring in licensed facilities. The Program also 

initiates enforcement actions under certain conditions when licensees are not conforming to the 

applicable regulations. 

In carrying out its mission, the Program faces broad ranging expectations from a number of 

stakeholders and constituents. These stakeholders and constituents include the public, the 

Administration, the Congress, nuclear power plants, industry and advocacy groups, and 

international nuclear organizations. The general public desires high safety standards but also 

wants electricity available at a reasonable cost. While there is public interest in increased 

openness and access to information, the NRC may best serve the public interest by not publicly 

releasing some secure information that may pose a threat to safety. The Administration and 

Congress have expectations of high safety standards, but also require that regulatory burdens 

take into consideration cost-benefit analyses and that views of all affected parties are taken into 

account when regulations are developed. Industry groups and public interest advocacy groups 

may differ on views of what level of oversight leads to an appropriate standard for safety. 

In setting standards for the oversight and incident response activities for which the Program has 

legal authority, the NRC develops regulations. The authorizing legislation contains broad 

language with phrases like “adequate protection of public health and safety” and “no undue risk 

to common defense and security.” The Program’s primary mechanism for defining standards 

and processes under its authority is through Title 10 of the Code of Federal Regulation. The 

standards are developed through a rulemaking process in which input from all stakeholder 

groups is solicited and evaluated. 

1 

 
Project Objectives and Approach 

The objective of this project was to perform an independent evaluation of the Program and 

develop recommendations to strengthen program performance. The scope of the study included 

issues relevant to how the Program contributes to the NRC’s goals of safety and security, 

including program purpose and design, strategic planning, program management, and program 

results and accountability. To address this scope, FocalPoint reviewed eight evaluation areas 

including contribution to safety, budget performance integration, internal evaluation and 

improvement, public communications, interaction with other agencies, financial management, 

enforcement and allegations, and staff development. 

The evaluation included quantitative and qualitative input from a variety of sources. Semistructured, confidential interviews were conducted with 46 employees and stakeholders of the 

NRC. Given the range of expectations and objectives of various stakeholders discussed above, 

our goal was to develop standards that reflect a balance of all such views. Accordingly, 

interviewees included representatives from industry, public advocacy, trade publications, and 

NRC staff and management. We studied best practices from relevant public and private sector 

organizations. We examined current program processes and policies including oversight 

framework, incident response documentation, planning, budgeting, financial management, 

operations, communications, and staff development. We also conducted a review of the 

literature relevant to the eight evaluation areas. 

Summary of Findings 

A summary of key findings and recommendations from this evaluation are outlined below. 

Overall, we found the Program is effective in accomplishing its mission of providing reactor 

oversight and incident response. The details of these and other findings and recommendations 

are provided in the body of the report. 

Contribution to Safety. Overall, FocalPoint found the Program to be effective in contributing 

to the NRC’s goals of safety and security. The Program has met the strategic goal of zero 

radiological events in the industry. The industry trends data, which provide broad indicators of 

industry-wide reactor safety as tracked by the NRC, have shown improvement. The Program has 

a robust framework and the activities that comprise that framework are executed in a timely and 

effective fashion. The Program also effectively responds to licensee events and prepares for 

emergency response through exercises, drills, and other activities. 

We did find areas that need improvement. The Program has had some challenges resolving longstanding problems associated with fire protection. The Program also needs to continue efforts to 

improve monitoring of safety culture within the licensee organizations. We believe the 

Program’s post-exercise evaluations would benefit from increased standardization to help track 

progress in readiness for an emergency. Also, the performance indicators, as currently 

2 

 
established, provide limited visibility to variations in plant performance. There are alternate 

interpretations of this lack of variation, as described in Section 2 of the report. The Program 

should ensure that the lack of variation in the reported indicators does not undermine their utility 

as an indicator of declining performance. 

Budget Performance Integration. Based our review, we found the Program is effective in 

some aspects of its performance budgeting process but needs improvement in other areas. It has 

an effective top-down budget formulation process that is informed by the agency’s strategic 

direction, influenced by Program considerations and priorities, and coordinated with Program 

support offices. The Program is effective in providing justification for requested resources and 

in basing its budget estimates on reasonable assumptions about factors affecting program costs 

and resources. However, improvement is needed to define the linkage between outputs and 

outcomes, demonstrate the impact of funding on performance, and ensure accurate accounting 

for both the direct and indirect costs needed to meet performance targets and achieve program 

goals. The Program is working with the Office of the Chief Financial Officer (OCFO) to 

improve budget performance integration and transparency in these areas. 

Internal Evaluation and Improvement. The Program is effective in conducting internal selfassessments of efficiency and effectiveness and in using the results of those and other 

independent evaluations to modify strategic, operational, and financial plans. In some areas, we 

noted that program guidance and operations documentation have not been kept up-to-date. 

Improvements in timeliness and clarity in some of the operations documentation may improve 

understanding of roles and responsibilities among Program staff. 

Public Communications. The Program is effective in disseminating information to stakeholders 

in most areas while balancing the need for security with openness and accountability. The 

Program provides information such as stakeholder feedback, policy and procedures, industry 

trends, performance indicators, licensee inspection performance, and assessment results, related 

to the reactor oversight function. For the most part, information is complete and up-to-date. 

Interaction with Other Agencies. In most cases, roles and responsibilities for working with 

other federal agencies and State and local government have been established but some gaps 

remain. In 2007, an Inspector General (IG) audit identified recurring coordination problems with 

States in preparing for, executing, and evaluating incident response exercises. For example, the 

IG found the Program has not clearly defined or communicated its coordination role. In 

interviews, we also found some program staff had different characterizations of the role of NRC 

3 

 
as compared to other agencies in response to an incident. The Program is taking steps to address 

these issues. 

Financial Management. We found the Program has strong financial management practices. 

Funds are obligated consistently with the overall program plan and a limited amount of 

unobligated funds remains at the end of the year. Adequate procedures exist for reporting actual 

expenditures, comparing them against the intended use, and taking timely and appropriate action 

to correct single audit findings when funds are not spent as intended. Procedures are in place to 

ensure that payments are made properly for the intended purpose to minimize erroneous 

payments. The Program is not in violation of the Anti-Deficiency Act. As reported by the 

auditors, the Program received a clean audit opinion and is free of material internal control 

weaknesses. The Program’s financial management systems meet statutory requirements with the 

exception of one non-compliance that is agency-wide and not program-specific. 

Enforcement and Allegations. The Program is effective in resolving enforcement actions and 

allegations in a timely manner. The Program has had some challenges in establishing a 

transparent regulatory context for enforcement of fire protection rules. Concerns regarding noncompliance in this area have been expressed by stakeholders. The NRC has modified its fire 

protection regulations to allow licensees to adopt, on a voluntary basis, a risk-informed approach, 

in lieu of their existing fire protection licensing basis. The NRC recently clarified procedures for 

transitioning to the new rule. 

Staff Development. The Program has met its overall staffing objectives, including maintaining 

a stable, experienced resident inspector base, but a recent increase in inspector turnover is 

resulting in coverage challenges. While not as severe as the high level of inspector turnover, the 

Agency, as a whole, is experiencing increased turnover as its staff size increases. This has 

contributed to staff and managers with less tenure in their positions. In an effort to maintain 

quality staff, the NRC has several recruiting initiatives, has effectively launched knowledge 

management initiatives, and continues to deliver high quality training. 

Summary of Recommendations 

To address areas that need improvement, we have developed a set of recommendations that are 

put forward in this report. A summary of key recommendations is provided below. 

4 

 
Contribution to Safety 

Train regional staff as subject-matter experts for safety culture assessment. 

The evaluation of safety culture is an important part of the assessment process, but based on 

input from interviewees and internal stakeholders, we believe the uniqueness and subjectivity of 

safety culture assessments are challenging and time consuming for inspection staff. The 

Program should recruit and train a group of individuals to have specialized knowledge in the area 

of safety culture. These individuals would be a resource for inspectors making safety culture 

assessments and could facilitate knowledge sharing and consistency. 

Conduct a lessons learned study to assess the history of and causes for the fire protection 

issues. 

Fire protection is an important part of safety in nuclear reactors. At the time the fire protection 

rules were established back in 1980, there were a significant number of exemptions approved for 

certain vintage plants. Approval of the exemptions established compliance for those licensees. 

However, today there is still significant reliance among licensees on interim measures, 

enforcement discretion, and there is non-compliance among some licensees. The Program 

should conduct a lessons learned study to assess the history of and evaluate the reasons for the 

problems with fire protection. Such a study may help to identify ways to avoid similar problems 

in the future. It should be noted that the Program has already established a plan to conduct a 

lessons learned study in calendar year 2009. We concur with this plan and encourage that 

lessons learned be applied where appropriate to other aspects of the Program’s regulations. 

Enhance exercise evaluations through improved standardization; incorporate a postexercise survey of respondents and input from participating government agencies. 

To ensure that post-exercise evaluations are consistently performed, provide opportunity for and 

documented evidence of participant (including State, local, Federal organizations) feedback, and 

ensure that clear documentation of the process is evident, the post-exercise/event critique process 

should be clearly described in incident response procedures. 

Continue to focus on assessing the reasons for lack of variation in performance indicators 

in order to ensure that this does not continue to undermine their utility as indicators of 

potential problems. 

The performance indicators provide limited visibility to variations in plant performance. As 

such, it is more difficult to confirm their utility as indicators of potential problems. The Program 

should continue to assess the reasons for lack of variation in the performance indicators. 

Budget Performance Integration 

Establish published crosswalk of the Program’s budget to demonstrate alignment of costs 

to Program outputs and annual and long-term outcomes. 

5 

 
The program budget request lacks transparency in that it does not clearly define the relationship 

between resources and annual and long-term performance goals. The Program should coordinate 

with OCFO to present the Program’s budget in a more transparent manner in the NRC 

Performance Budget. This may include aligning the program budget with the Agency’s strategic 

goals, safety and security, and linking the Program’s strategic outcomes, performance measures, 

and output measures. 

Develop logic model or performance framework that clarifies linkage between outputs and 

outcomes. 

The Program’s complexity may make it difficult for some individuals to understand the 

relationship between its outputs and outcomes. A logic model or redesigned framework might 

facilitate understanding of the linkage between outputs and outcomes (e.g., oversight activity and 

safety). The Program may consider developing a logic model or revising the ROP performance 

framework to clarify the linkage between the oversight activities and program outcomes. 

Develop marginal cost model to conduct sensitivity analyses. 

Under the current budget process, the Program cannot answer the question, “If funding were 

increased/decreased by X, this program would be able to achieve Y more/less outcomes.” Given 

the Program’s goal of zero nuclear accidents and the public’s intolerance for a nuclear incident, 

developing a programmatically relevant marginal cost model represents a challenge. The 

Program may consider developing a marginal cost analysis to demonstrate the impact of funding 

on output (vs. outcome) performance and extrapolating these results to provide a qualitative 

explanation of the impact on annual and long-term outcomes. 

Internal Evaluation and Improvement 

Complete the process of updating and maintaining current key program documentation. 

Documentation was up-to-date in most areas, but there were gaps in some areas. Standardization 

and consistency of procedures documentation promotes consistent understanding and supports 

measurement for continuous improvement and achievement of excellence. The Program should 

continue efforts underway to update documentation. 

Enforcement and Allegations 

Address issues and uncertainties regarding compliance with the fire safety requirements. 

There has been uncertainty among some stakeholders regarding some aspects of enforcement of 

fire safety rules. The NRC has made recent clarifications, which should address at least some of 

the questions. The Program should continue to clarify licensee requirements to help establish a 

transparent regulatory context for enforcement of fire protection rules, where it is clear when 

licensees are not in compliance, thus making it easier to enforce. 

6 

 
Staff Development 

Analyze root causes of increased inspection turnover and take actions to address this issue. 

High turnover among resident inspectors is making it difficult to meet site staffing objectives. 

Turnover has been high overall within the NRC and this also affects the Program. The Program 

should further analyze the root cause of the increased turnover. Based on this analysis, the 

Program should make appropriate changes to address causes. 

7 

 
1. Introduction 

1.1. Objectives and Approach 

In August 2008, the Nuclear Regulatory Commission (NRC) hired FocalPoint Consulting Group, 

a management consulting firm, to perform an independent evaluation of the NRC’s Reactor 

Oversight and Incident Response (RO-IR) Program (the Program). This report presents the 

results of that evaluation effort. The Program plays a key role in fulfilling the NRC’s mission of 

ensuring adequate protection of public health, safety and security, and protection of the 

environment. 

The Program’s activities are authorized under the Atomic Energy Act of 1954, as Amended, and 

the Energy Reorganization Act of 1974. The Program’s major objectives, as set out in the 

legislation, include oversight of the nation’s commercial nuclear power plants and other 

regulated licensees and coordination of the federal government’s response to radiological 

incidents occurring in licensed facilities, in order to maintain adequate protection of public health 

and safety. The overarching goal is to ensure that licensees operate with no undue risk to 

common defense and security. 

Key activities in support of those objectives include inspections, performance assessments, event 

assessments, and response to incidents occurring in licensed facilities. The Program also 

initiates enforcement actions under certain conditions when licensees are not conforming to the 

applicable regulations. 

Project Objectives 

The objective of this project was to perform an independent evaluation of the Program, and 

develop recommendations to strengthen program performance. The scope of the study included 

issues relevant to how the Program contributes to the NRC’s goals of safety and security, 

including program purpose and design, strategic planning, program management, and program 

results and accountability. 

Project Approach 

To develop the approach for the study, we conducted an initial review of program activities and 

objectives. This initial review included a preliminary review of program documentation, 

authorizing legislation, and relevant regulations. We also conducted a set of preliminary 

interviews with program staff and stakeholders. The purpose of this initial review was to 

identify key categories of program performance based on the Program’s legislative mandate, the 

NRC’s rules and regulations, and the expectations and objectives important to the Program’s 

stakeholders, constituents, and program staff. From this initial review, we identified the key 

activities and objectives for the Program. These are summarized in the figure below. 

8 

 
Figure 1 – Key Program Activities and Objectives 

Stakeholders

•Public

•Media

•Industry Groups

•Government Agencies

•State & local Gov

•Int’l Agencies

•Committees

Operations

Key Objectives

•Contribute to safety through oversight and incident 

response

•Develop budget requests linked to performance that 

will provide adequate resources to execute plan

•Interface with federal and international organizations 

for cooperation and knowledge sharing

•Provide communications to public and other 

stakeholders on safety and security issues

•Continuously improve oversight and incident response 

activities

•Take appropriate enforcement actions and evaluate 

allegations 

•Maintain adequate high quality staff expertise

•Conduct financial management and budget 

execution in a timely and accurate manner

Constituents

• Licensees

• Regions

• NRC Offices

• Commission 

• OEDO

Communicate 

with the public

Coordinate with 

federal and 

international agencies

Conduct 

inspections

Collect and analyze 

performance 

indicators

Provide internal 

correspondence / 

reporting

Ensure 

qualifications and 

provide training

Evaluate and 

manage performance

Recruit qualified 

staff

Report significant 

items and take 

appropriate actions

Follow-up and 

evaluate allegations

Key Activities

Develop and maintain 

performance plans 

and budget

Conduct nuclear 

reactor oversight

Provide response and 

coordination in an 

incident

Conduct 

exercises and 

drills

Conduct incident 

response

Source: FocalPoint 

Based on the key objectives we identified, we developed eight evaluation areas that defined the 

scope of the study: 

• Contribution to safety; 

• Budget performance integration; 

• Internal evaluation and improvement; 

• Public communications; 

• Interaction with other agencies; 

• Financial management; 

• Enforcement and allegations; and 

• Staff development. 

The evaluation included quantitative and qualitative input from a variety of sources. We 

conducted semi-structured, confidential interviews with 46 employees and stakeholders of the 

NRC. Given the range of expectations and objectives of various stakeholders, our goal was to 

develop standards that reflect a balance of all views. Accordingly, interviewees included 

representatives from industry, public advocacy, trade publications, and NRC staff and 

management. We examined best practices from relevant public and private sector organizations. 

9 

 
We studied current program processes and policies including oversight framework, incident 

response documentation, planning, budgeting, financial management, operations, 

communications, and staff development. We conducted a review of the literature relevant to the 

eight evaluation areas. The overall approach for assessing each evaluation area consisted of four 

key components as set out in the chart below: 

Develop 

Assessment 

3. Review leading 

and best practice 

organizations 

2. Conduct 

interviews 

4. Analyze 

current 

performance 

1. Review 

relevant 

documentation 

and published 

materials 

1. Review relevant documentation and published material. We studied internal program 

documentation and conducted a brief literature review of published material related to nuclear 

safety, regulatory oversight, evaluation of regulatory programs, safety evaluation, emergency 

preparedness and response, and other program management functions (e.g., budget formulation, 

financial management, communications, staff development). A list of documents reviewed is 

contained in Appendix A. 

2. Conduct interviews. We conducted interviews with stakeholders, constituents, and program 

staff. We developed interview guides and conducted semi-structured, confidential interviews to 

obtain information related to the evaluation areas. Interviews included program staff in the 

Office of Nuclear Reactor Regulation (NRR), the Office of Nuclear Safety and Incident 

Response (NSIR), the Regions, the Office of Enforcement (OE), the Office of the Executive 

Director for Operations (OEDO), and the Office of the Chief Financial Officer (OCFO). We 

also interviewed external stakeholders from advocacy groups, industry, and publications. 

3. Review leading organizations and their best practices. We identified leading organizations 

and reviewed their best practices. To identify organizations, we used a combination of 

FocalPoint databases and literature and Web searches. We reviewed regulatory-based programs 

that have demonstrated success in the evaluation areas. 

4. Analyze current performance. Based on the interviews, literature review, and analysis, we 

established standards on which we based our assessment. The standards are based on program 

goals as set forth in the authorizing legislation, NRC rules and regulations, and expectations of 

key stakeholders and constituents. In carrying out its mission, the Program faces broad ranging 

and high priority expectations from a number of stakeholders and constituents. These 

stakeholders and constituents include the public, the Administration, the Congress, nuclear 

power plants, industry and advocacy groups, and international nuclear organizations. The 

general public desires high safety standards but also wants electricity available at a reasonable 

cost. While there is public interest in increased openness and access to information, the NRC 

may best serve the public interest by not publicly releasing some secure information that may 

10 

 
pose a threat to safety. The Administration and Congress have expectations of high safety 

standards, but also require that regulatory burdens take into consideration cost-benefit analysis 

and that views of all affected parties are taken into account when regulations are developed. 

Industry groups and public interest advocacy groups may differ on views of which level of 

oversight leads to appropriate standards for safety. Given the range of expectations and 

objectives of various stakeholders, our goal was to develop standards that reflect a balance of 

those views. To analyze current performance with respect to those standards, we assessed 

internal processes, evaluated program performance, and compared the Program’s current 

performance against goals and stakeholder expectations. 

1.2. Program Overview 

The Program is responsible for oversight of the 104 commercial nuclear power plants in the 

United States and response to radiological incidents. The Program focuses on the oversight, 

enforcement, and incident response activities of the U.S. commercial nuclear power industry. 

Through these activities, the Program ensures that licensees are conforming to the applicable 

regulations and the conditions of their licenses. The Program also ensures that licensees provide 

timely and appropriate event assessment and response. The Program is authorized under the 

Atomic Energy Act of 1954, as Amended, and the Energy Reorganization Act of 1974. 

The U.S. produces the most nuclear energy of any country, with 806 billion megawatt hours 

(MWh) produced at 104 plants in 2007. Today, nuclear power provides 20% of the electricity 

the U.S. consumes (NEI). 

11 

 
Figure 2 - Locations of Commercial Operating Reactors 

Source: NRC 

Of the 104 reactors, 69 are pressurized water reactors (PWRs) and 35 are boiling water reactors 

(BWRs). The PWRs maintain pressure in the reactor vessel at approximately 2,200 pounds per 

square inch (psi) so the coolant water does not boil and exits at a temperature of approximately 

310° Celsius. The PWRs have a primary loop and a secondary loop. The heat from the primary 

(radioactive) loop is transferred to the secondary loop in a steam generator, which is inside the 

containment building. The secondary loop, which now contains steam, goes outside containment 

to drive the turbines, which produce electricity. The secondary water then returns to the steam 

generator to be turned back into steam. In a BWR, the pressure is significantly lower 

(approximately 1,000 psi), so the coolant water boils. Above the core are a steam separator and 

dryer; the separated and dried steam goes outside containment to the turbine before returning to 

the reactor as condensed water to again cool the reactor. 

Components of a plant’s safety systems (e.g., pumps, backup water supplies) all require 

electricity. If offsite power is lost, diesel generators are turned on to supply the necessary 

electricity to power the equipment needed to cool the core. The most serious type of accident is 

one that results in a loss of coolant to the core which can lead to a core meltdown. Part of the 

NRC’s oversight task is to ensure that rules and regulations are followed such that no single 

mishap can cause a serious accident. The NRC accomplishes this task by establishing rules and 

regulations and conducting assessments of reactors to determine if the rules and regulations are 

12 

 
being followed. In the event of an incident, the NRC has rules and regulations that govern their 

response to make sure that the incident is managed appropriately to minimize risk to the public. 

The Reactor Oversight and Incident Response Program is the program through which NRC 

provides oversight for commercial operating reactors. The safety performance of each licensed 

plant is monitored through inspections performed by the Program and performance indicators 

provided by the licensee. Any findings of deficiency are evaluated to determine a level of 

significance, based on safety consequences, whether the deficiency was willful, and other 

considerations. Based on this assessment, the deficiency can be handled either through the 

reactor oversight significance determination process (SDP) or enforcement. 

The Program also coordinates the overall NRC incident response to radiological incidents and 

emergency events involving licensees. Through its incident response function, the Program 

maintains NRC readiness for response to incidents by planning and preparedness activities such 

as plan and procedure maintenance, training, exercises, interagency liaison/coordination, 

stakeholder outreach, and program assessments. Additionally, the Program conducts inspections 

and performance-based emergency preparedness evaluations with licensees, in coordination with 

offsite (State/local/Tribal) preparedness program inspections and evaluations performed by 

FEMA, ensuring that regulatory compliance is maintained and programs demonstrate acceptable 

performance capabilities. 

The Program’s FY 2009 annual budget is approximately $280M with 1,165 full-time equivalent 

(FTE) staff operating both at headquarters and in the Regions. The oversight function maintains 

centralized oversight management and has resident inspectors located at each licensee in the 

Regions. The incident response function provides central oversight of the Program and regional 

response programs including response activities from the NRC Operations Center. 

Regulatory Framework 

In order of hierarchy, the regulatory framework consists of the authorizing legislation, 

regulations in Title 10 of the Code of Federal Regulation, associated regulatory guides, NRC 

policies and procedures, and program documentation and guidance. The authorizing legislations 

contain broad language with phrases like “adequate protection of public health and safety” and 

“no undue risk to common defense and security.” This gives the Program some latitude in 

defining the other parts of the framework (Kamabi). This, in part, has enabled the Program to 

implement the ROP, which is a more risk-informed approach, replacing the earlier, Systematic 

Assessment of Licensee Performance (SALP). The objective of implementing a more riskinformed approach is to more closely align activities with areas of assessment that pose greater 

risk. This approach can potentially reduce costs and gives licensees more options in meeting 

safety goals. 

In addition, the nuclear power industry has itself formed an organization, the Institute of Nuclear 

Power Operations (INPO), whose mission is to “promote the highest levels of safety and 

13 

 
reliability, and to promote excellence in the operation of nuclear electric generating plants.” 

INPO provides a system of personnel training and qualification for all key positions at nuclear 

power plants; workers undergo both periodic training and assessment. INPO also conducts 

periodic evaluations of operating plants, focusing on plant safety and reliability, in the areas of 

operations, maintenance, engineering, radiological protection, chemistry, and training. 

14 

 
2. Contribution to Safety 

2.1. Objectives / Key Activities 

Reactor Oversight 

A key goal of reactor oversight is to contribute to the NRC’s safety mission in three strategic 

performance areas: reactor safety, radiation safety, and safeguards. To this end, the Program 

provides oversight of commercial operating reactors. The safety performance of each licensed 

plant is monitored through inspections performed by the NRC inspectors and performance 

indicators provided by the licensee. The findings of deficiency are evaluated to determine a level 

of significance, based on safety consequences, whether the deficiency was willful, and other 

considerations. Based on this assessment, the deficiency can be handled either through the 

reactor oversight significance determination process (SDP), or through the enforcement process. 

This section provides an evaluation of the Program’s contribution to safety through the reactor 

oversight process. Section 8 provides an evaluation of the enforcement process. 

When any of the oversight activities (e.g., inspections, review of performance indicators) yields a 

finding of safety significance, SDP is used to assess the safety impact and assign a color code 

that indicates the risk significance of the finding. Green findings indicate low risk significance, 

while White, Yellow, and Red findings indicate higher risk. 

Incident Response 

The incident response function coordinates the NRC’s planning and preparation for, response to, 

and recovery from radiological incidents and/or emergency events involving licensees. The 

Program encompasses all incidents for which the NRC has a response role. Specific 

responsibilities include developing and maintaining program documentation; staffing, operating 

and maintaining the Headquarters Operations Center (HOC); coordinating the staffing of 

response teams; conducting training, drills, and exercises; and carrying out a process of agencywide continuing improvement for incident management. Each of the four regional offices 

manages and administers similar functions at a regional level. 

The HOC and the regional incident response centers are equipped with communications, 

information display, and analysis systems. Communications systems provide direct linkages, 

including secure telephone lines with licensees and connections with government entities. The 

Emergency Response Data System (ERDS) displays real-time plant parameters and safety 

system data from all nuclear power reactor plants. Radiological analysis and consequence 

assessment processes provide the capability for predicting radiological consequences to the 

public and/or the environment. 

The NRC is an integral part of the Federal incident management community and actively 

participates in interagency policy and planning and preparedness activities with the Homeland 

15 

 
Security Council (HSC), Department of Homeland Security (DHS), FEMA, and other 

departments/agencies. 

In the event of an incident, NRC activates the response function at its HOC and one of its four 

Regional Incident Response Centers (Region I in King of Prussia, PA; Region II in Atlanta, GA; 

Region III in Lisle, IL; and Region IV in Arlington, TX). During an incident, NRC conducts an 

independent assessment of the reactor plant’s condition, operator response, and protective action 

recommendations. If necessary, the NRC Chairman can direct licensee actions to ensure public 

health and safety and protection of the environment. The NRC also provides expert consultation, 

support, and assistance to the State and local public safety officials responding to the event. 

As the Coordinating Agency, NRC has technical leadership for the Federal government’s 

response to the event. If the severity of an event rises to the level of an Incident of National 

Significance DHS will take on the role of coordinating the overall Federal response to the event, 

while NRC would retain a technical leadership role. The Secretary of Homeland Security, in 

consultation with other departments and agencies as appropriate, will determine when it will take 

a lead coordination role in an accident. Based on the Program’s experience with tabletop and 

full-scale exercises this will most likely occur at the General Emergency level for a technical 

emergency and Alert level for a terrorist event. 

2.2. Approach 

The approach for assessing this evaluation area consisted of three key components as explained 

below. 

1. Review of relevant documentation and published material. We examined internal program 

documentation and conducted a brief literature review of published material related to nuclear 

safety, regulatory oversight, evaluation of regulatory programs, safety evaluation, and emergency 

preparedness and response. The following documents were reviewed: 

• Inspection Reports 

• Inspection Findings Summaries 

• Performance Indicator Summaries 

• CY 2007 Annual ROP Self-Assessment Report 

• FY 2007 Annual Industry Trends Report 

• CY 2007 Consolidated Response to ROP Feedback 

• IMC 0307 ROP Self-Assessment Program 

• IMC 0305 Operating Reactor Assessment Program 

• IMC 2515 Inspection Program 

• IMC 0608 Performance Indicator Program 

• IMC 0609 Significance Determination Process 

16 

 
• MD 8.13 Reactor Oversight Process 

• Reactor Oversight Process (ROP) Homepage 

• Incident Response Homepage 

• OIG Audit of the Incident Response Program 

• GAO Report: NRC: Oversight of Nuclear Power Plant Safety Has Improved, but 

Refinements Are Needed 

• Davis-Besse Lessons Learned Task Force (DBLLTF) Report 

• DBLLTF Final Status Of Recommendations 

• Public Version of Palo Verde Lessons Learned 

• Lochbaum: How Palo Verde Made the NRC’s Naughty List 

• Gunter, Warren, & Lochbaum: Fire When Not Ready 

• Bureau of Labor Statistics: Employment Cost Index – September 2008 

• Kadambi: Performance-Based (Risk Informed) Regulation: A Regulatory Perspective 

• Klien, Brandenburg, Atas, & Maher: The Use of Trained Observers as an Evaluation Tool for 

a Multi-Hospital Bioterrorism Exercise 

• Feinstein: The Safety Regulation of U.S. Nuclear Power Plants: Violations, Inspection, and 

Abnormal Occurrences 

• Marks & Potter: Drilling for Results: The Quest for Objective Exercise Evaluations 

• Sorensen: Safety Culture: A Survey of the State-of-the-Art 

• Kadak & Matsuo: The nuclear industries transition to risk-informed regulation and operation 

in the United States 

• Ford & Schmidt: Emergency response training: Strategies for Enhancing Real-World 

Performance 

2. Conduct staff interviews. We conducted interviews with stakeholders, constituents and 

program staff. We developed interview guides and conducted semi-structured, confidential 

interviews to obtain information related to contribution to safety. This process included program 

staff in NRR, NSIR, and the Regions. We also interviewed external stakeholders from advocacy 

groups, industry, and publications. 

3. Analyze current performance. Based on the interviews, literature review, and research, we 

established standards or performance metrics on which we based our analysis. To analyze 

current performance with respect to those standards, we assessed internal processes, evaluated 

program performance, compared the Program’s current performance against goals and 

stakeholder expectations, and analyzed causes for gaps.

To evaluate the Program’s contribution to safety, we assessed the Program on the following 

standards: 

17 

 
• Ensures that an appropriate level of safety is being maintained by licensees; 

• Takes appropriate actions to prevent degradation of safety and to promote safety 

improvements; and 

• Performs oversight and incident response activities in cost-effective manner. 

2.3. Findings 

Standard 1: Ensures that an appropriate level of safety is being maintained 

by licensees 

For this standard, we assessed whether the Program was ensuring that the appropriate level of 

safety was being maintained by the licensees. A number of factors must be considered in setting 

a threshold for “appropriate” level of safety. These factors include, but are not limited to: 

• Requirements set out by the authorizing legislation; 

• The Program’s stated strategic mission and goals; 

• Stakeholder (e.g., public, OMB, GAO) expectations; and 

• Constituent (e.g., licensee, industry groups) expectations. 

As discussed in Section 1.2, the authorizing legislation contains broad language with phrases like 

“adequate protection of public health and safety” and “no undue risk to common defense and 

security.” The Program has established a set of overall targets. Some stakeholder groups and 

some constituent groups would disagree on what “appropriate” level of safety the NRC should 

work to ensure, versus what level of safety should be the industry’s responsibility to ensure. To 

evaluate performance with regard to this standard, we considered all of the above. 

Finding: The Program’s strategic outcome targets (e.g. mission and goals) are being 

adequately met 

An indirect measure of the effectiveness of the Program’s regulatory activities is the extent to 

which the ultimate outcome targets are achieved. While these measures are only part of the 

picture as they do not measure the direct impact of program activities, nor do they always 

provide visibility to variations in safety levels over time, they do focus on the ultimate goals of 

the Program. The Program has a range of safety targets that are outcome measures in its budget, 

strategic planning, and operations planning processes. These measures and the associated targets 

are set out in the table below. As the table shows, the Program has met its own targets 100% of 

the time for these measures over the past five years. 

18 

 
Table 1 - Program Strategic Outcome Targets 

Performance Measure Target

% Met

(2003-2007)

No nuclear reactor accidents 0 100% 

No inadvertent criticality event 0 100% 

No acute radiation exposures resulting in fatalities 0 100% 

No releases of radioactive materials that result in significant radiation exposures 0 100% 

No releases of radioactive materials that cause significant adverse environment 

impacts 0 100% 

No instances where licensed radioactive materials are used domestically in a 

manner hostile to the security of the United States 0 100% 

No significant licensing or regulatory impediments to the safe and beneficial uses of 

radioactive materials 0 100% 

Number of new conditions evaluated as Red by the Reactor Oversight Process 

(ROP) <=3 100% 

Number of significant accident sequence precursors (ASPs) of a nuclear reactor 

accident 0 100% 

Number of operating reactors whose integrated performance entered the Inspection 

Manual Chapter 0350 process, the multiple/repetitive degraded cornerstone 

column, or the unacceptable performance column of the ROP Action Matrix 

<=3 100% 

Number of significant adverse trends in industry safety performance <=1 100% 

Number of events with radiation exposures to the public or occupational workers 

from reactors that exceed A.O. Criteria 1.A 0 100% 

Number of radiological releases to the environment that exceed applicable 

regulatory limits 0 100% 

Unrecovered losses or thefts of risk-significant radioactive sources 0 100% 

Number of significant unauthorized disclosures of classified and/or safeguards 

information 0 100% 

Number of security events and incidents that exceed the Abnormal Occurrence 

Criteria I.C 2-4 <=4 100% 

No more than one instance per program where licensing or regulatory activities 

unnecessarily impede the safe and beneficial uses of radioactive materials (Annual 

Target) 

0 100% 

Source: NRC 

We also reviewed the data, from the past five years, from the Program’s Industry Trends 

initiative. The Program tracks long-term trends using a 10-year rolling time period. We 

examined the last five years to focus on more recent trends. The table below shows the 

compound average growth rate (CAGR) for the 13 indicators. A declining trend (i.e., negative 

19 

 
growth rate) translates to improved performance for the first 10 indicators. For the last three 

indicators, an increasing trend indicates improved performance. 

Table 2 - Industry Trends Indicators 

Indicator CAGR (2003 -

2007)

Automatic Scrams While Critical -11%

Safety System Actuations -12%

Significant Events -27%

Safety System Failures -9%

Forced Outage Rate (%) -17%

Equipment Forced Outages / 1,000 Commercial Critical Hours -9%

Collective Radiation Exposure -3%

Unplanned Power Changes -6%

Reactor Coolant System Activity 1%

Reactor Coolant System Leakage -3%

Drill / Exercise Performance 0.1%

ERO Drill Participation 0%

Alert and Notification System Reliability 1%

Source: NRC and FocalPoint analysis 

Of the 13 industry indicators, 11 show improvement between 2003 and 2007. One indicator, 

Emergency Response Organization (ERO) drill participation, is unchanged. One indicator, 

Reactor Coolant System Activity, showed a 1% increase. This indicator measures maximum 

monthly reactor coolant system activity expressed as a percentage of the technical specification 

limit. The indictor increased from only 0.41% of technical limit, to only 0.43% technical limit 

from 2003 to 2007. In addition, there have been no safety significant indictors as reported by 

licensees during this timeframe. As such, this increase is not safety significant. 

Finding: The ROP framework is an effective approach for guiding the Program’s oversight 

activities 

The purpose of our review of the design of the ROP framework was to assess the extent to which 

the inspections and performance indicators collectively covered the areas important to safety and 

generally identified performance deficiencies needed to maintain an appropriate level of safety. 

In the late 1990s, NRC made the decision to incorporate risk-informed elements into the 

oversight process, where appropriate. The NRC defines risk-informed as an approach in which 

risk insights, engineering analysis and judgment, and performance history are used to focus 

attention on the most important activities. These activities include establishing objective criteria 

based on risk insights for evaluated performance, developing measurable parameters for 

20 

 
monitoring systems and licensee performance, and focusing on results as the primary basis of 

regulatory decision-making (SECY-98-144). This decision led to implementation of the ROP. 

Prior to the ROP, the oversight process was based on the Systematic Assessment of Licensee 

Performance (SALP). 

The ROP provides a number of improvements over SALP. For example, the ROP incorporated 

risk-informed assessment, which gives the licensee more opportunity to innovate to find more 

efficient or effective ways of meeting safety goals. SALP evaluations were held less frequently, 

which meant the safety issues were more likely to go uncorrected for a longer period of time. 

ROP uses more objective assessments, which enables more consistent assessment of risk. In 

addition, SALP had been criticized by stakeholders and constituents for its lack of clear 

guidelines for reacting to deficiencies and inconsistent implementation. 

The ROP makes use of both inspection information and monitored performance indicators (PIs). 

The risk significance of the findings from the indicators and inspections is evaluated using the 

SDP, which, where appropriate, uses probabilistic risk assessment (PRA) methods to evaluate 

the impact of a finding. If the aggregate risk exceeds predetermined thresholds, the NRC 

subjects the licensee to increased oversight. In areas where PRA methods are used, the NRC 

went from prescribing operational requirements and instead began evaluating the risk associated 

with potential faults or failures, and taking action when the risk threshold was exceeded. In 

addition, ROP increased the scope of the assessment to respond to new security objectives. After 

the 9/11 terrorist attacks for example, assessments were added to the physical protection (also 

called, security) cornerstone to provide for additional safeguards. 

The Program’s framework for its reactor oversight function is shown in the figure below. At the 

foundation of this framework are seven cornerstones that reflect the safety aspects of facility 

operation. The cornerstones contain both risk-informed and prescriptive elements. Satisfactory 

licensee performance in the cornerstones is intended to provide reasonable assurance of 

appropriately safe facility operation. 

21 

 
Figure 3 - Reactor Oversight Framework 

Source: NRC 

Within this framework, the Program: collects information about licensee performance; assesses 

the information for its safety significance; and provides for appropriate licensee and NRC 

response. 

There are more than 30 baseline inspection procedures and 16 performance indicators across the 

seven cornerstones. The inspections are conducted by the resident inspector(s) assigned to the 

facility and regional specialists. The performance indicators are collected to provide a broad 

spectrum of data that is also used to assess the licensee’s performance. Collectively, these are 

not intended to cover every aspect of plant design and operation, so it is important that the 

elements of the assessment cover the optimal number of risk-significant aspects without undue 

overlap. Excessive overlap would reduce the number of aspects of operation evaluated at a given 

cost, which would make the oversight process less cost effective. 

When there are findings of safety significance, the SDP is used to assess the safety impact and 

assign a color code that indicates the risk significance of the finding. Green findings indicate 

low risk significance, while White, Yellow, and Red findings indicate higher risk. Depending on 

the number and significance of the findings, the plant may be subject to additional oversight. 

The amount of oversight is determined by the ROP Action Matrix, which provides the basis for 

NRC oversight action to be taken in the event of findings from the oversight processes. The 

responses are set forth in the ROP Action Matrix, which calls for a response on the basis of 

increased risk as implied by the finding. 

22 

 
An example of additional oversight that might be conducted in accordance with the ROP Action 

Matrix is supplemental inspections. Some supplemental inspections are conducted by a 

multidisciplinary team of regional inspectors and may take place over several months. The ROP 

Action Matrix is summarized in the figure below. 

Table 3 - ROP Action Matrix Summary

Source: NRC 

23 

 
In reviewing the inspection data from the past 4 years, we looked to see if there was enough 

variation in the inspection findings to provide visibility to differences in performance among 

plants. The figure below shows the number of plants with a given number of inspection findings. 

The number is adjusted to reflect severity of findings. Green findings are consolidated within the 

finding’s cornerstone as a single finding. White findings count as two, Yellow as three, and Red 

findings count as four. 

Figure 4 - Number of Inspection Findings per Plant 

10

28

32

22

8

0

3

0

5

10

15

20

25

30

35

1234567

Number of Plants

Number of Findings

Number of Plants with X Findings: 3Q2008

Source: NRC and FocalPoint analysis 

As the chart above shows, there is enough variation in the number of findings per plant to 

provide some visibility to differences in performance. In addition, the inspections continuously 

result in findings that were not identified by the licensees. External and internal stakeholders 

were mostly positive about the effectiveness of the inspection program. However, many external 

and internal stakeholders who expressed positive views regarding the inspection program also 

expressed caveats. The Program has demonstrated a commitment to learning from feedback and 

experience gained. Based on our review, our finding is that the inspection program is generally 

effective at covering all of the safety-related areas and identifying degrading performance. 

The figure below shows the number of plants with a given number of performance indicators 

greater than Green. There is little visibility to variation in performance. It should be noted that 

all of these plants are operating within columns 1-4 of the ROP Action Matrix and are therefore 

considered to be safe by the NRC. 

24 

 
Figure 5 - Number of PI Findings per Plant 

2

000000

0

1

2

3

1234567

Number of Plants

Number of Greater‐than‐Green PIs

Number of Plants with X Greater‐than‐Green PIs:

3Q2008

Source: NRC and FocalPoint analysis 

Unlike the inspection findings, the performance indicators show very little visibility to 

differences in performance among plants. In 3Q2008 only 2 plants had a greater-than-green 

performance indicator. In total, less than 0.1% of the performance indicators are greater than 

Green. As a result they provide little discrimination. There are a number of potential 

explanations for the lower number of greater-than-Green performance indicators (compared to 

the number of safety-significant inspection findings). For example: licensees may be closely 

monitoring the areas covered by the performance indicators and therefore operating more safely 

in those areas, than the areas covered by inspections. In such cases, the indicators would be 

contributing to and reflecting good performance by licensees. However, possible alternative 

explanations include: the thresholds between Green and White may be too high; there may be 

issues with data collection and analysis of the indicators; or there may be some fatigue in the 

indicators as licensees have learned how to obtain Green indicators. 

The Program continues to focus its efforts to improve the performance indicators. Recent efforts 

such as replacing the Unplanned Scrams with Loss of Normal Heat Removal (USwLONHR) PI 

with the Unplanned Scrams with Complications (USwC) PI and review of MSPI indictors are 

examples of such efforts. Making such changes have been beneficial, but are also necessarily a 

time-consuming and expensive process for both the NRC and licensees. However, we believe 

the Program’s improvement efforts should continue to focus on assessing the reasons for lack of 

variation in the PIs and ensuring that this does not undermine their utility as indicators of 

potential problems. 

The SDP, described above, has been the subject of concern among some NRC staff and external 

stakeholders. Concerns expressed were difficulty in application; issues with timeliness and 

25 

 
determinations that were sometimes subjective and prescriptive, rather than risk-informed; and 

undue influence by licensees. 

The Program has made recent improvements in the SDP. For example, in early 2008, the 

Program reviewed the SDP with stakeholders and received feedback on various aspects of the 

SDP to improve its effectiveness and efficiency. The scope of the review consisted of an 

evaluation of: 

• The current criteria for a White finding to ensure consistency with risk-informed goals of the 

ROP; 

• The entry conditions into the radioactive effluent release program branch of the SDP 

flowchart for spills and leaks; and 

• The extent to which the SDP reflects the NRC Organizational Excellence objective. 

Such improvement efforts demonstrate the Program’s commitment to self-evaluation and 

improvement. 

Overall, our finding is that the ROP framework and its components are effective. Issues such as 

the PIs can be tweaked within the framework; ongoing internal evaluation and improvement 

activities demonstrate the Program’s commitment to continuous improvement. 

Finding: Assessment-related activities are completed in accordance with the planned schedule 

As discussed above, we believe the inspections, combined with the Program’s improvement 

efforts (such as the ROP realignment) are effective in identifying problems and addressing the 

areas of safety significance. In order for the Program to ensure an appropriate level of safety, it 

must execute that framework in accordance with the planned schedule. The Program maintains a 

set of self-assessment metrics that present the extent to which various aspects of the licensee 

oversight activities are completed in accordance with the Program’s plan. We reviewed data 

from the past 2 years on performance of assessment-related activities including inspections, 

timely PI data reporting and dissemination, completion of temporary instructions, final 

significance determinations, and assessment program results. 

For each of these areas, we reviewed the metrics tracked, whether the targets were sufficiently 

ambitious, and whether or not the targets were met. In addition, in cases involving subjective 

metrics, we reviewed the underlying data and indicated whether we agreed or disagreed with the 

Program’s self-assessment. Overall, we concur with the results of the Program’s selfassessment. The table below sets out the key metrics that track the extent to which planned 

assessment-related activities are completed in accordance with the planned schedule. 

26 

 
Table 4 - Selected Annual ROP Self-Assessment Performance Metrics 

Metric (Target)

Target Sufficiently 

Ambitious

Performed in 

Accordance with Plan

Concur with Assessment 

/ Methodology

Documenting inspection finding in 

accordance with requirements (stable 

or improving trend)

Yes Yes Yes

Completion of inspection reports on a 

timely basis (90% or inspection reports 

issued on time)

Yes Yes Yes

Temporary Instructions are completed 

on a timely basis (All TI completed on 

time)

Yes Yes Yes

Public availability of inspection

information is timely (stable or 

declining trend)

Yes Yes Yes

Inspection reports are relevant useful 

and written in plain language (trend 

average level of agreement)

Yes, but target 

should be clarified

Yes Yes, but there were an 

insufficient number of 

external respondents

Final significance determinations are 

timely (90% processed in 90 days)

Yes Yes Yes

Source: NRC and FocalPoint analysis 

Finding: Emergency exercises are planned and carried out effectively 

The Program conducts exercises or drills throughout the year to practice mock scenarios that 

would be faced in a real incident. The objectives of these exercises include increasing 

preparedness and the ability to respond effectively to an incident in a way that assists in 

mitigating consequences to the public and the environment, and helping to provide assurances 

that the Program could apply the necessary resources to respond. 

In general, there are three key challenges that organizations face in training for emergencies 

through drills or exercises that are not factors in training for routine operations. 

• The first challenge is retention of training knowledge and skills over time, given limited 

opportunities to perform emergency response skills during normal operations. 

• The second is effective generalization of skills learned in training and ability to transfer 

that knowledge to the significantly different demands that could arise in an actual emergency. 

• The third challenge is effective assimilation of individual efforts into a coordinated 

emergency response (Ford and Schmidt, 2000). 

To assess the Program in these areas, we interviewed program staff and external stakeholders, 

reviewed program documents, toured the Headquarters operation center, and observed a licensee 

exercise. 

With regard to the first challenge, retention of training knowledge and skills over time, we 

reviewed the frequency of the drills/exercises and the extent to which the format of the exercise 

facilitated learning by the participants. The Program conducts four or five licensee exercises in 

27 

 
which NRC headquarters participates, along with NRC Regions, and licensees. In addition, each 

region participates in two to four exercises with licensees each year. This provides a sufficient 

number of drills/exercises since each of the approximately 300 responders is involved in a 

drill/exercise at least once a year, and the Agency routinely participates in Federal Agency 

exercises involving various aspects of Homeland Security-related areas of focus. In addition to 

the exercises, a variety of other training is conducted. These efforts include classroom training, 

read-and-sign training, and Web-based training. In addition to the incidence response drills 

conducted by Headquarters, licensees are evaluated in an exercise every two years and routinely 

in licensee-specific drills, actual events, and planned baseline inspections as part the emergency 

preparedness cornerstone in the reactor oversight framework. The NRC’s incidence response 

performance-based training activities are aligned with selected licensee emergency preparedness 

exercise evaluations. 

We found that the exercise we observed involved a well-developed scenario that addressed with 

sufficient realism the range of challenges that would be faced in a radiological emergency, plus 

additional challenges associated with response to a hostile action. The structure of the exercise 

included features such as handling responsibility broader than what responders would typically 

face, fixing problems created by others, developing new directions for the team, and managing 

pressures from external stakeholders. These types of challenges enhance learning (McCauley, 

Ruderman, Ohlott, and Morrow, 1994). Accordingly, we believe the Program addresses the first 

challenge effectively. The Program should also consider conducting post-exercise surveys of 

responders to obtain additional insight to knowledge gain and likelihood of retention. 

To assess whether the second challenge was addressed, we reviewed the extent to which the drill 

forced responders to explore the situation to perform diagnostics and understand underlying 

issues. Drills that develop problem-solving skills improve the ability of learners to generalize 

what they have learned (McDaniel and Schlager, 1990). We found the drill effectively used 

techniques like providing updates to the problem without giving solutions, unexpected 

developments, and mitigation of problems when correct measures were taken. 

Finally, with regard to the third challenge, effective assimilation of individual efforts into a 

coordinated emergency response, we reviewed the interaction of the teams (executive team, 

reactor safety team, protective measures team, safeguards team, liaison team, public affairs team, 

base team, site team and other federal agencies participating). As an observer, it was difficult to 

monitor the content of all communications. For the most part, there was good fidelity in 

communication between teams. The Program use of team chronologies, for example, which 

other teams could access, facilitated this communication. However, in a 2007 report, the 

Inspector General (IG) found weakness in NRC’s coordination with State and local government 

authorities. The IG stated: 

NRC has repeatedly demonstrated problems coordinating and communicating with 

State authorities during EP exercises. This weakness recurs because (1) NRC has not 

28 

 
clearly defined and communicated its coordination role to State and local authorities, 

and (2) has not followed a consistent approach for working with the States during these 

exercises. Inadequate coordination and communication adversely affects NRC’s 

emergency operations with State agencies and could diminish the public’s confidence 

in NRC.

The Program is taking steps to address this although we did not detect this weakness in the 

exercise we observed. We did find however, that the approach for evaluating exercises did not 

provide a good basis for documenting and communicating improvements in emergency 

preparedness resulting from exercises. Additional standardization in the evaluation process 

would help ensure consistency in evaluation, which could provide insights to overall trends in 

preparedness. 

Finding: The Program works with licensees to complete performance indicators as planned 

As described above, the performance indicators are part of the assessment process that uses 

indicators based on data collected from licensees and analyzed by NRC. As such, the overall 

assessments are integrated with licensees. We reviewed the Program’s data on timeliness of 

performance indicator data reporting and dissemination. The Program has a goal of publishing 

the PI data within 5 weeks of the end of each calendar quarter, which is a reasonable goal. There 

were no late postings in the 24-month period that we reviewed. 

Finding: Program documents are published in a timely fashion and are generally 

comprehensive and clear 

We reviewed inspection and event notification reports, conducted interviews, and reviewed 

comments collected by the Program from internal and external stakeholders. We found the 

inspection reports were published in a timely fashion, well-written, and comprehensive. We 

believe the inspection reports would be reasonably clear to a reader with knowledge of the 

reactor oversight program. We believe a reader without working knowledge of the ROP would 

have difficulty understanding the inspection reports. That said, we found the “ROP Inspection 

Finding Summary” information that is available on the Web site to contain similar information 

and would be understandable to most readers even without knowledge of the ROP. Stakeholders 

providing comments to the Program regarding inspection reports noted some areas where 

improvements could be made. In particular, there were comments regarding the need for more 

detail on why a finding had risk significance. The Program is working to improve consistency in 

documentation of inspection findings, which should enable improved discussion of findings in 

the reports. We reviewed the log of event notification reports that document events reported by 

licensees. We found the event notification reports to be sufficiently comprehensive for the initial 

recording of events. 

29 

 
Standard 2: Takes appropriate actions to prevent degradation of safety and 

to promote safety improvements 

For this standard, we evaluated the extent to which the Program was effective in taking action to 

prevent degradation of safety when plant performance declines and to promote improvements to 

safety. As with Standard 1, we evaluated standard 2 in the context of the Program’s stated goals, 

authorizing legislation and stakeholder and constituent expectations. 

Finding: The Program is effective in finding safety deficiencies, and issues are corrected, but 

there have been challenges resolving a long-standing issue related to fire protection 

As discussed earlier, our finding is that the inspection process is effective at identifying safety 

deficiencies. We also examined the inspection findings data to identify the findings that were 

corrected (corrected being defined as not recurring within 1 year of the initial finding) as a 

percentage of total findings that occurred. On that basis, for the sample we examined, over 99% 

of the findings were corrected. 

There have been some concerns expressed by some stakeholders about the amount of time some 

plants have remained in columns 3 and 4 of the ROP Action Matrix. We examined inspection 

findings for plants placed in column 4 of the ROP Action Matrix over the past five years to 

assess effectiveness in resolving safety issues identified at those plants. There were five such 

plants: Palo Verde 3, Perry 1, Point Beach 1, Point Beach 2, and Cooper. On average, these 

plants remained in column 4 for 9.6 quarters. We examined the inspection findings data to 

assess the extent to which the long period of time reflected challenges in getting deficiencies 

resolved, or instead reflected a conservative approach by the Program of keeping the findings 

open and maintaining increased oversight associated with column 4 until sufficient assurances 

could be made that issues have been resolved. Based on our review, we conclude the latter is the 

case. Take the case of Palo Verde 3, for example. 

Palo Verde 3 was placed in column 4 in the fourth quarter of 2006 because of one Yellow 

finding in the Mitigating Systems cornerstone originating in the fourth quarter of 2004, and one 

White finding in the same cornerstone originating in the fourth quarter of 2006. There have been 

no new findings in this cornerstone in the last two quarters. In the most recent quarter there are 

no findings in any of the cornerstones. The Program keeps the original findings open and 

therefore the plant remains in column 4, in accordance with IMC 0305, which states: 

Due to the depth and/or breadth of performance issues reflected by a plant being in the 

Multiple/Repetitive Degraded Cornerstone column of the Action Matrix, it is prudent to 

ensure that actual performance improvements (which typically take longer than several 

quarters to achieve) have been made prior to closing out the inspection findings and 

exiting the Multiple/Repetitive Degraded Cornerstone column of the Action Matrix. 

30 

 
In order to continuously monitor the rate at which deficiencies are corrected, the Program should 

consider a metric that presents correction rate. This would be the percentage of findings that do 

not recur within one year of the initial finding. 

While there has been overall success in the resolution of safety deficiencies, there has been a 

long-standing issue associated with fire protection. Fires pose a significant risk to operating 

reactors. According to the NRC, approximately one-half of core damage risk results from 

accident sequences that initiate with fire events. The fire protection rules allow for manual 

actions or interim compensatory measures when the fire protection features in place do not meet 

requirements. The manual actions must be approved and in some cases interim measures have 

been in use for extended periods. While the NRC’s regulatory positions on these issues are set 

out in Regulatory Issue Summaries 2006-10 and 2005-7, the NRC has not resolved several longstanding issues that have lead to insufficient transparency in the regulatory context with respect 

to fire protection. In addition, the NRC lacks a comprehensive database on the status of 

compliance. The regulations are intended to provide prevention, suppression, and protection of 

plant shutdown capability. Non-compliance can affect this defense-depth-sequence. Issues that 

remain unresolved include: 

• Licensees’ use manual actions to ensure fire safety rather than fire protection features in 

place, such as fire barriers and automatic fire detection and suppression; 

• Licensees’ use of interim compensatory measures for extended periods of time, rather than 

making repairs; 

• Uncertainty regarding the effectiveness of fire wraps as barriers; and 

• Mitigating the impacts of short circuits that can cause simultaneous, malfunctions of safetyrelated equipment and complicate the safe shutdown. 

In the near term, the impact of these issues could be mitigated through a centralized database on 

the use of exemptions from regulations, manual actions, or compensatory measures. This would 

facilitate tracking compliance trends. Issues such as these and others have also led to a lack of 

transparency in the regulatory context that has made it more difficult to enforce the fire 

protection rules (see Section 8). Given that the fire protection rules were established in 1980 and 

there is still significant reliance among licensees on interim measures and enforcement 

discretion, and there is non-compliance among some licensees, the Program plans to establish a 

lessons learned task force in calendar year 2009. We concur with this plan and encourage that 

lessons learned be applied where appropriate to other aspects of the Program’s regulations. 

Finding: The Program effectively records, analyzes, and responds to licensee incidents 

The overall response to an incident is under the direction of the NRC Chairman or his/her 

designee. In an incident, response personnel are organized into teams, including an executive 

team, reactor safety team, protective action team, liaison team, safeguards team, base team, site 

team and public affairs team. The response to an event is tailored to the nature of the event. 

Upon receiving a notification of an incident, the HOC authenticates the source and screens the 

31 

 
incident. A record of the event is entered into a log. Information entered includes data such as 

the licensee, location, event time, person reporting event, emergency class, status of reactor 

operation, and description of the event. We reviewed this log and found the entries provided 

were clear and concise and provided a good base for initial analysis of the incident. The 

Headquarters Operations Officer (HOO) and the Headquarters Emergency Response Officer 

(HERO) notify designated regional and headquarters decision makers and a decision is then 

made on the level of response to address the incident. The response modes include: 

• Monitoring mode: Initial level in which the responsible region leads the NRC’s response; 

• Activation mode: A higher level of escalation, in which NRC headquarters leads the 

response; and 

• Expanded activation mode: A higher level of escalation involving deployment of a NRC site 

team to the licensee and delegation of specific authorities to a NRC site team director. 

Based on our interviews of program staff and management, review of program documentation, 

tour of the HOC, and observations of activities of program staff responsible for recording 

licensee events, we believe the Program is effective in ensuring prompt and accurate recording of 

licensee events, developing a decision on whether to escalate, and maintaining an accurate and 

timely database of licensee events. 

Finding: The Program needs to continue efforts to improve monitoring of safety culture 

within licensee organizations 

In a risk-informed regulatory context, the NRC and licensees must adopt a role in which risk is 

managed in the overall context of safe plant operation. This requires a safety culture in which 

the emphasis shifts from only assuring compliance with rules to operating with an appropriate 

level of risk. This requires an open culture of communication and willingness to raise concerns 

about practices even if they are within the operating envelope of the technical specifications. 

With the use of risk-informed initiatives that provide more flexibility, there is more 

responsibility in assuring that the plant actions are well controlled and monitored (Kadat). 

Although an accident in a plant may be the result of an action or inaction by an individual, many 

accidents are a result of conditions in the organization. The NRC evaluates whether a safety 

culture-related issue, which the Program calls substantive cross-cutting issue (SCCI), exists at 

each operating reactor twice a year. Among the substantive cross-cutting areas are human 

performance, safety-conscious work environment (which includes environment for raising 

concerns), and problem identification and resolution. These are called cross-cutting areas 

because they affect all of the ROP cornerstones. Each of the cross-cutting areas contains subareas the Program calls cross-cutting aspects. As part of the inspection process, safety culture 

aspects are assigned to inspection findings. A SCCI may be indicated when there are the 

requisite number of inspection findings that have been assigned cross-cutting aspects, causal 

factors with a common theme, and the Program has a concern about the licensee’s progress in 

addressing the deficiency. 

32 

 
Based on input from interviewees and internal stakeholders, the uniqueness and subjectivity of 

safety culture assessments are challenging for inspection staff. Consideration should be made for 

augmenting regional staff who have specialized knowledge in the area of safety culture. 

Effective partnership and cooperation with licensees is important for inspection staff to obtain 

the necessary information for effective safety culture assessments. We believe the Program is on 

the right track with its approach and the recommendations under consideration from the lessons 

learned with Palo Verde. 

Standard 3: Performs oversight and incident response activities in costeffective manner 

Finding: Budgeted program costs per unit of output are increasing. This is in part due to 

increasing scope of the Program and higher staffing costs at the NRC. 

We compared enacted budget figures for program support costs from the NRC Performance 

Budget to aggregate power output of licensed facilities as an indicator of change in program 

costs for 2005 through 2007. Ideally, actual costs should be used, and the Program should 

collect this information and track this metric. As the table shows, the budgeted cost per 

megawatt-hour has increased over the past 2 years. This is due in part to broad-based increases 

in employment costs. In the U.S., compensation costs among civilian workers increased 3.0% in 

2005, 3.3% in 2006, and 3.3% in 2007 (Bureau of Labor Statistics, 2008). In addition, there has 

been some expansion in the scope of the Program during this time period. For example, the 

Program’s oversight activities include increased assessments of licensees’ safety culture and 

additional aspects of security. It is also a result of generally higher costs in the NRC stemming 

from expansion of staff and the associated recruiting, training, and turnover costs. 

Table 5 - Budget Program Costs per MWh 

Year

Power Generation 

(MWh)

Program 

Support 

Costs (M)

Cost/ 

MWh

2005 781,986,365 $122.1 $0.16 

2006 787,218,636 $142.3 $0.18 

2007 806,486,978 $150.8 $0.19 

Source: NRC, NEI and FocalPoint analysis 

2.4. Recommendations 

Train regional staff as subject-matter experts for safety culture assessment 

The evaluation of safety culture is an important part of the assessment process, but based on 

input from interviewees and internal stakeholders, we believe the uniqueness and subjectivity of 

safety culture assessments are challenging and time consuming for inspection staff. The 

33 

 
Program should recruit and train a group of individuals to have more knowledge in the area of 

safety culture. These individuals would be a resource for inspectors making SCCI assessments 

and could facilitate knowledge sharing and consistency in assessment of safety culture. In 

addition, they could provide guidance and standards, as needed, for the independent safety 

culture assessments performed by licensees in column 4 of the ROP Action Matrix. Training 

staff to develop specialized knowledge in the area safety culture would yield benefits including 

more high quality safety culture assessments and increased consistency that would lead to higher 

credibility with constituents and stakeholders. 

Continue to focus on assessing the reasons for lack of variation in performance indicators 

in order to ensure that this does not continue to undermine their utility as indicators of 

potential problems 

The performance indicators provide limited visibility to variations in plant performance. As 

such, it is more difficult to confirm their utility as indicators of potential problems. The Program 

should continue to assess the reasons for lack of variation in the performance indicators. To the 

extent that the lack of greater-than-Green indicators does not reflect good performance on the 

part of the licensee, the Program should consider options such as: reviewing and modifying 

thresholds for White, Yellow, and Red indicators; reviewing inspection procedures for validating 

licensee methodology in collecting and reporting data for the performance indicators to ensure an 

accurate assessment of performance; and identifying options for modifying indicators on a 

regular basis in order to reduce fatigue of the indicators. 

Enhance exercise evaluations through improved standardization; incorporate a postexercise survey of respondents and input from participating government agencies 

To ensure that post-exercise evaluations are consistently performed, provide opportunity for and 

documented evidence of participant feedback and self-critique, include non-NRC response 

stakeholders (e.g. State, local, Federal organizations) involved in the drill/exercise/event, and 

ensure that clear documentation of the process is evident, the post-exercise/event evaluation 

process should be clearly described in IR procedures. These procedures should include: 

• Preparatory actions for coordination with offsite response organizations (OROs) regarding 

their participation or observance; 

• Post-exercise/event critique review forms used in soliciting and documenting participant 

comments; 

• Format and timeliness schedules for post-action reports; and 

• Instruction and retention guidance for evaluation materials. 

34 

 
Adopt performance metric to reflect safety deficiency correction rate 

To track the extent to which issues identified in inspections or through PIs are corrected, the 

Program should consider adopting a metric that tracks safety deficiency correction rate, such as: 

The percentage of findings that do not recur within 12 months of the original finding. 

This metric could replace the existing metric: 

Number of operating reactors whose integrated performance entered the Inspection 

Manual Chapter 0350 process, the multiple/repetitive degraded cornerstone column or the 

unacceptable performance column of the ROP Action Matrix. 

OMB has criticized the existing metric as not being ambitious enough and lacking in the ability 

to show trends. We believe the proposed metric provides an opportunity to set a metric with an 

aggressive target, e.g., over 99%. The value would increase as the number of overall findings 

increases, and the number of findings are resolved and do not recur. In addition, since there are a 

large number of findings, tracking of this metric would provide visibility to trends. 

Adopt metric to track program efficiencies (Program Cost per MWh) 

In order to monitor the program costs, the Program should adopt efficiency metric, program cost 

per aggregate power output of licensed facilities. 

Program Cost per MWh = 

Direct program costs 

Aggregate power output of licensed facilities 

This measure reflects increased benefits to the public as a result of risk-informed regulation, 

underlying research, and other activities supporting power uprates. To the extent that the scope 

of security-related assessment has increased over time, NRC may want to limit this metric to 

safety-related assessment and incident response cost to facilitate comparison over time. 

Conduct a lessons learned study to assess the history of and causes for the fire protection 

issues 

Given that the fire protection rules were established in 1980 and there is still significant reliance 

among licensees on interim measures and enforcement discretion, and there is non-compliance 

among some licensees, the Program plans to establish a lessons learned study in calendar year 

2009. We concur with this plan and encourage that lessons learned be applied where appropriate 

to other aspects of the Program’s regulations to help to identify ways to avoid similar problems 

in the future. 

35 

 
Implement a database to track the status of fire protection exemptions, compensatory 

measures, and manual actions in place at plants in order to facilitate the assessment of 

compliance trends 

Given the reliance among licensees on interim measures, the Program should implement a 

central database of interim compensatory measures being used in place of permanent fire 

protection features. Such a database could help the NRC track trends of compliance to the fire 

protection rules. Such information could help the NRC make informed decisions as it works to 

resolve this long-standing issue. 

36 

 
3. Budget Performance Integration 

3.1. Objectives / Key Activities 

For over a decade, the NRC has evolved its Planning, Budgeting, and Performance Management 

(PBPM) process, which integrates the agency’s strategic planning, budgeting, and performance 

management processes. The NRC created the PBPM process in the fall of 1997 in response to 

the enactment of the Government Performance and Results Act (GPRA), which provided for the 

establishment of strategic planning and performance measurement in the Federal Government as 

well as an increased emphasis on accountability and results. Since that time, the NRC has 

evolved its PBPM process to meet internal agency requirements as well as changing mandates 

from the Congress, the executive branch, and other external stakeholders. 

Under the current administration, improving the integration of budget and performance is one of 

the high-priority initiatives in the President’s Management Agenda (PMA). A part of this 

initiative is the Office of Management and Budget’s (OMB) diagnostic tool, the Program 

Performance Assessment Rating Tool (PART), which is designed to provide a consistent 

approach to reviewing program design, planning, and goals development as well as program 

management and results. Other significant legislation that is pertinent to budget performance 

integration includes the CFO Act of 1990, which provides for long-range planning, requires 

audited financial statements and modern financial systems, and strengthens accountability 

reporting. Title 31 of the United States Code provides the legal foundation for the federal budget 

process in the executive branch; OMB Circular No. A-11, “Preparing and Submitting Budget 

Estimates,” is the guidance issued to agencies by OMB to ensure compliance with these budget 

laws and to obtain information needed to formulate the President’s Budget. 

In support of its efforts to be open and transparent, the NRC communicates to the Congress, 

OMB, NRC staff, and other stakeholders its strategic direction, required resources and activities, 

and performance measures through the Strategic Plan, Performance Budget, and the Performance 

and Accountability Report (PAR). Some recent accomplishments of the Agency in this area 

include receiving the Certificate of Excellence in Accountability Reporting (CEAR) award for 

the seventh straight year from the Association of Government Accountants (AGA) for the FY 

2007 PAR. This PAR was also ranked fourth out of 24 agencies by the Mercatus Center, which 

rates Federal agency PARs according to how well they inform the public. The GAO also 

recently recognized the NRC for the improvements in recent survey results regarding the 

agency’s use of performance information; and the CFO was invited to provide Senate testimony 

on how the NRC is using performance information to improve management of NRC programs in 

late July 2008. 

The budget performance integration efforts of the Program support the agency’s organizational 

excellence objectives of openness, effectiveness, and operational excellence as identified in 

NRC’s current Strategic Plan for FY 2008-2013. Each year, the Office of the Chief Financial 

37 

 
38 

 

Officer (OCFO), in coordination with the Office of the Executive Director of Operations 

(OEDO), issue guidance to the program and support offices for developing the performance 

budget. NRR is the lead program office for the Reactor Oversight program budget and NSIR is 

the lead program office for the Incident Response program budget. The Budget Formulation 

Team within PMDA of NRR and NSIR are responsible for coordinating the budget process for 

the RO and IR program, respectively, and interacting with the OCFO throughout the PBPM 

process. In the FY 2009 Performance Budget, the budget request for the Program totaled $279.0 

million, including 1,165 FTE. The table below provides additional detail on the program budget 

for Reactor Oversight and Incident Response from the NRC Performance Budget for FY 2009: 

Table 6 - Reactor Oversight and Incident Response Program Budget Authority and FTE 

$M FTE $M FTE $M FTE $M FTE $M FTE

RO Program Support $139.0 847 $139.5 838 $148.6 848 0.4% -1.1% 6.5% 1.2%

RO Infrastructure and Support $95.4 241 $99.8 238 $106.8 247 4.6% -1.2% 7.0% 3.8%

Subtotal RO $234.4 1,087 $239.3 1,076 $255.4 1,094 2.1% -1.0% 6.7% 1.7%

IR Program Support $11.8 50 $12.9 56 $16.0 53 9.3% 12.0% 24.0% -5.4%

IR Infrastructure and Support $5.6 12 $6.7 16 $7.6 18 19.6% 33.3% 13.4% 12.5%

Subtotal IR $17.4 62 $19.6 72 $23.6 71 12.6% 16.1% 20.4% -1.4%

Total RO-IR Program Support $150.8 897 $152.4 894 $164.6 901 1.1% -0.3% 8.0% 0.8%

Total RO-IR Infrastructure and Support $101.0 253 $106.5 254 $114.4 265 5.4% 0.4% 7.4% 4.3%

Total RO-IR $251.8 1,150 $258.9 1,148 $279.0 1,165 2.8% -0.2% 7.8% 1.5%

FY08-FY09 

FY 2007 % Change

FY 2008 

Enacted

FY 2009 

Request

FY07-FY08 

% Change

Source: NRC Performance Budget for FY 2009 and FocalPoint analysis 

The key components of the PBPM process are: 1) setting the strategic direction; 2) determining 

the planned accomplishments and budgeting resources; 3) executing the budget and monitoring 

performance; and 4) assessing the performance and providing feedback for adjusting strategy and 

the budget. Our assessment of this evaluation area touches on all four components of the PBPM 

process; however, our primary focus was on the second and third components related to budget 

formulation and performance management. 

3.2. Approach 

The approach for assessing this evaluation area consisted of four key components as outlined 

below. 

1. Review of relevant documentation and published material. We reviewed both internal 

program documentation and conducted a brief literature review of published documentation 

related to budget performance integration. The following internal documents were reviewed: 

• FY 2008 - FY 2013 Strategic Plan 

• FY 2004 - FY 2009 Strategic Plan 
• Components of FY 2010 Performance Budget 

• FY 2009 Performance Budget 

• FY 2008 Performance Budget 

• FY 2007 Performance Budget 

• FY 2007 PAR 

• 2003 OMB PART Assessment of Reactor Inspection and Performance Assessment 

• 2007 OMB PART Assessment of Decommissioning and Low Level Waste 

• 2007 OMB PART Assessment of High-Level Waste Repository 

• MD 4.2 Administrative Control of Funds (2008) 

• MD 4.3 Financial Management Systems (2005) 

• MD 4.7 Policy and Practices Governing NRC Long-Range Planning, Budget Formulation, 

and Resource Management (1989) 

• OCFO Budget Call Guidance for FY 2010 PBPM 

• OCFO Budget Call Guidance for FY 2009 PBPM 

• OCFO Budget Call Guidance for FY 2008 PBPM 

• Components of FY 2010 Office budget submissions 

• FY 2009 Office budget submissions 

• FY 2008 Office budget submissions 

• Office operating plans 

• Office performance monitoring reports 

• Office operating reports 

• Advice of Allowances and Financial Plans 

• Office instructions and guidance 

• PBPM process documentation from OCFO 

• CY 2007 Annual ROP Self-Assessment Report 

• ROP Budget History – Significant Events Impacting ROP Budgets 

• U.S. Nuclear Regulatory Commission Charter for the Performance Improvement Panel 

Reactor Oversight Process Initial Implementation Evaluation Panel Final Report (2001) 

• OIG, Audit of the Budget Formulation Process (2005) 

We also reviewed the following documents: 

39 

 
• Government Accounting Office (GAO) results-oriented budget practices in federal agencies 

and other GAO studies related to budget performance integration and performance 

improvement 

• OMB PART Guidance 2008 

• President’s Management Agenda (PMA) criteria for getting to green on Budget Performance 

Integration 

• Federal Deposit Insurance Corporation Strategic Plan, Budget, PAR, OMB PART and other 

organizational documents 

• Department of Interior Budget Request, PAR and OMB PART for Office of Surface Mining 

– Federal Managed Regulation of Surface Coal Mining and Abandoned Mine Land 

Reclamation 

• International Federation of Accountants: Costing to Drive Organizational Performance 

• Mihm: Implementing GPRA: Progress and Challenges 

• Skelly: Improving Integration of Performance Information in Congressional Budget 

Justifications 

• Hoge & Martin: Linking Accounting and Budget Data 

• Beck & Davis: President’s Management Agenda: Performance Improvement Initiative 

From the literature review, we identified a set of best practices and criteria to be used as 

standards for the assessment. 

2. Conduct staff interviews. We conducted interviews with stakeholders, constituents and 

program staff. We developed interview guides and conducted semi-structured, confidential 

interviews to obtain information related to program planning, budgeting, and performance 

management. 

3. Review organizations successful at budget performance integration. We identified two 

organizations that we deemed exhibited relevant best or leading practices. To identify 

organizations, we used a combination of FocalPoint databases and literature and Web searches. 

We reviewed regulatory-based programs that have demonstrated success in the area of budget 

performance integration, including: 

• Department of the Interior – Office of Surface Mining, Federal Managed Regulation of 

Surface Coal Mining and Abandoned Mine Land Reclamation; and 

• Federal Deposit Insurance Corporation – Regulation and Examination Program. 

4. Analyze current performance. Based on the standards established from our literature review 

and examination of successful organizations, we performed financial and operational analysis to 

compare the Program’s current performance against these criteria.

40 

 
To evaluate the Program’s budget performance integration, we assessed the Program on the 

following standards: 

• Performance informs budget formulation; 

• Reliable estimates of costs and resources are produced; and 

• Program can relate performance, budget, spending, and workforce information. 

3.3. Findings 

Standard 1: Performance informs budget formulation 

For this first evaluation standard, we based our assessment on a combination of GAO’s 

framework for results-oriented budget practices, OMB Circular A-11 guidance, and OMB PART 

criteria. We performed our evaluation by reviewing program documentation, conducting 

interviews with program officials, and assessing the extent to which the Program meets the 

specific criteria for this standard. These criteria include: 

• Receives general guidance on agency goals, performance issues, and resource constraints for 

budget formulation; 

• Considers relative priorities, performance issues, and other factors to weigh competing needs 

and decide funding levels; 

• Coordinates with other entities to achieve common goals and avoid duplication; 

• Defines the relationship between resources and annual and long-term performance goals; 

• Demonstrates the impact of funding on expected performance; and 

• Provides justification that the requested resources enable the program to achieve its 

performance goals. 

At the end of August 2007 after the FY 2009 PBPM process was completed, the Commission 

directed the CFO to provide options for improving the budget formulation process. As a result, 

the budget formulation process changed for the FY 2010 budget and more changes are planned 

for the FY 2011 budget. Although we reviewed both past and current documentation related to 

the PBPM process, when assessing the performance budgeting process, our review primarily 

focused on the current process used for the FY 2010 budget since the FY 2011 process was still 

in the early development stages. To evaluate the presentation and transparency of the program 

budget to stakeholders, we reviewed the FY 2009 and FY 2008 Performance Budgets. When 

assessing budget versus actual results, we reviewed financial and operating performance 

information for FY 2006 through FY 2008 depending on data availability. 

Based on our assessment, we found the Program has an effective budget process that is informed 

by the agency’s strategic direction, influenced by program considerations and priorities, and 

coordinated with program support offices. However, improvement is needed to define the 

41 

 
linkage between outputs and outcomes and demonstrate the impact of funding on performance. 

Below are our detailed findings for this evaluation standard. 

Finding: The Program’s budget process has effective top-down strategic direction and 

guidance for budget formulation; program considerations and priorities to weigh competing 

needs and decide funding levels; and coordination with other entities to achieve common goals 

and avoid duplication 

Using the Strategic Plan, actual expenditures from prior years and current estimated resource 

needs, the Program develops an annual budget, identifying the planned activities and resources 

necessary for accomplishment. The Commission provides high-level planning objectives for 

budget development and prioritization of planned activities. Using these planning objectives 

from the Commission and financial data from OCFO, the lead program offices coordinate with 

all of the offices that support the Program to develop the budget proposal. 

As described in the budget formulation process documentation and the budget call guidance, the 

budget process begins with the request from the CFO and EDO to lead program offices to 

develop planning assumptions, a workload analysis, and out-year workload trends for the 

program budget. These planning products identify the key cost drivers, external factors, and 

internal influences affecting program activities and support development of the program budget 

request. Offices are then asked to prioritize the most significant planning assumptions while 

OCFO develops the financial considerations, which may include expected OMB cuts based on 

historical data and an assessment of unobligated and unliquidated funds. The CFO and EDO 

then brief the Chairman on financial and program considerations and priorities. The Chairman 

then develops planning objectives and shares these with fellow Commissioners, resulting in a 

Staff Requirements Memo (SRM) that provides Commission planning guidance to the program 

staff. 

The CFO issues budget guidance to the lead program offices and communicates the planning 

objectives and priorities issued by the Chairman in consultation with the Commission. The 

program offices use the planning objectives and priorities to develop their budget proposals. 

Additional program direction is provided by the Deputy EDO (DEDO), including specific 

guidance for the Program and workload expectations for planned activities, to ensure the budget 

developed by the offices is aligned with a top-down strategic direction to achieve common goals 

and avoid duplication. Based on the guidance, the lead program offices coordinate with all of the 

offices that support the Program to develop workload and resource estimates, as well as 

performance and output measures for the program budget. Program support staff expressed 

concerns regarding how ‘major drivers’ (i.e., that level in the budget structure between the 

planned activity and subprogram levels) were defined by OCFO for the FY 2010 budget cycle. 

Staff suggested increased communication and coordination between the program offices and 

OCFO in defining major drivers that best reflect significant program activities. Program support 

staff also identified the need to better define roles and responsibilities of all parties involved in 

the budget process, which include OEDO, OCFO, lead offices and supporting offices. While the 

42 

 
changes to the FY 2010 budget process were regarded as positive improvements, the staff felt 

that clearly established roles and responsibilities would support improved coordination and 

collaboration among the offices resulting in the development of a robust budget proposal for the 

Program. 

Finding: Improvement is needed in defining the relationship between resources and annual 

and long-term performance goals 

The program budget as presented in the FY 2009 Performance Budget as well as the internal 

program budget submission does not define the relationship between annual and long-term 

performance goals and resources for the Program. The presentation of the program budget 

request is linked to a limited number of output measures but the funding request is not tied to the 

Program’s performance measures, strategic outcomes, or strategic goals. There is a separate 

section on performance measures that aligns the agency’s two main programs with its two 

strategic goals; however, there is no breakdown of the RO-IR program budget request between 

the safety and security strategic goals to demonstrate the Program’s contribution. Also, there is a 

crosswalk in the appendix that shows which programs contribute to each strategic goal and 

performance measure, but there is no linkage to resources. 

The program offices that support the RO-IR program have reporting that provides some insight 

to the budget performance linkage but these reports are limited to each individual office and do 

not provide a transparent and comprehensive account of the Program. For example, NRR, the 

lead office for RO, has a Performance Monitoring Report (PMR) that links a portion of the 

office’s budget to some of the Program’s annual performance goals; however, this report does 

not account for all of the office’s program resources and it does not tie resources to long-term 

performance goals. NSIR, the lead office for IR, has the Division of Preparedness and Response 

(DPR) OPS Plan Performance Indicators Report that ties each annual performance goal to a 

long-term goal of safety or security or an organizational excellence objective; however, there are 

no linkages to resources in this report. 

From an analysis of the Program’s budget structure and the operating plans for the program 

support offices, it appears that the program budget request could be broken down by strategic 

goals, safety and security, based on the planned activity code; however, the Program would need 

to coordinate with OCFO, who leads the budget formulation process and development of the 

annual performance budget to present the program budget in this way. In addition, the Program 

needs to define the relationship of all program resources to annual performance goals to improve 

the transparency of the annual performance budget. 

Finding: The Program provides justification that the requested resources enable the Program 

to accomplish its planned activities and output targets but needs improvement in 

demonstrating the impact of funding on annual and long-term outcomes 

The Program’s funding request goes through various levels of internal and external review that 

requires the Program to provide adequate budget justification prior to approval. These levels of 

43 

 
review and approval include CFO/EDO, Chairman, Commission, OMB, and the Congress. In 

addition, the agency’s budget process includes a Scenario Planning component, which requires 

the Program to explain the impact of a reduction in resources and benefits of additional 

resources. However, the explanation provided by the Program primarily demonstrates the impact 

on outputs rather than outcomes. In some instances, the Program provides details of the impact 

on a specific activity, or references an improvement or reduction in the agency’s goals for safety 

or security, but the explanations are limited in demonstrating the quantitative and qualitative 

impact on annual and long-term outcomes. The Program does not perform rigorous marginal 

cost analysis to accurately estimate the marginal cost (e.g., +/ - 10%) of changing performance 

goals. 

Although the Program is able to provide justification for resource estimates based on historical 

performance and risk-informed assessment of emerging needs, the program budget request does 

not answer the question, “If funding were increased/decreased by X, this program would be able 

to achieve Y more/less outcomes.” With the Program’s goal of zero nuclear accidents and the 

public’s intolerance for a nuclear incident, developing a programmatically relevant marginal cost 

model represents a challenge to the Program. The Program may consider developing a marginal 

cost analysis to demonstrate the impact of funding on output performance and extrapolating 

these results to provide a qualitative explanation of the impact on annual and long-term 

outcomes. 

Standard 2: Reliable estimates of costs and resources are produced 

For this second evaluation standard, we based our assessment on a combination of GAO’s 

framework for results-oriented budget practices and OMB Circular A-11 guidance. We 

performed our evaluation by reviewing program documentation, analyzing office and program 

budget submissions, reviewing financial and operating performance reports, and conducting 

interviews with program officials. Based on our review and analysis, we assessed the extent to 

which the Program meets the specific criteria for this standard, including: 

• Bases its budget estimates on reasonable assumptions about factors affecting program costs 

or budgetary resources; and 

• Looks back to assess the accuracy of previous estimates and makes appropriate adjustments 

to estimating methods. 

Because of the changing program budget structure and process in recent years and limited 

availability of data we were unable to perform a budget vs. actual assessment for the entire 

program budget on this standard. As an alternative approach, we evaluated the budget estimation 

methodology of the Reactor Oversight Process (ROP) which accounts for more than half of the 

RO program budget. We also reviewed the performance and budget planning products and 

budget requests submitted by NRR and NSIR for the RO and IR program, which included the 

Environmental Scan, Methodology for Determining Resources, Budget Resource Worksheet, 

44 

 
45 

 

Budget Request by Program, and Scenario Planning. Below are our findings for this evaluation 

standard. 

Based on our assessment for this standard, we found the Program produces reliable estimates of 

costs and resources for the ROP. Below are our detailed findings for this evaluation standard. 

Finding: The Program bases its budget estimates on reasonable assumptions about factors 

affecting program costs and resources, and looks back to assess the accuracy of previous 

estimates, and makes appropriate adjustments to estimating methods 

Each year, as a part of the annual ROP self-assessment, the Program reviews the inspection 

procedures and resources expended for the ROP to better understand the reasons for regional 

variations and differences in resources required to complete a procedure at different sites. This 

review also helps to determine if adjustments to the frequency and scope of individual 

procedures were appropriate and resulted in an accurate and realistic resource estimate for the 

ROP. The results of this analysis are used to make any necessary adjustments to the ROP 

Resource Model, which is used to formulate future ROP resource requirements. This annual 

assessment ensures the ROP resource model produces reliable estimates for ROP resource 

requirements. In 2007, RO also implemented a Biennial Inspection Process Review to 

streamline inspection procedures and focus resources on highest value efforts. The reallocation 

of resources to highest value activities based on risk-informed assessment through the Annual 

ROP Self-Assessment and the Biennial Inspection Process Review demonstrates how the 

Program uses actual operating performance to produce reliable cost estimates. 

In addition, as a part of the budget formulation process each year, the Program is required to 

document key planning assumptions and develop a preliminary assessment of workload strategy 

and out-year workload trends. Key planning assumptions are the external factors and internal 

influences that significantly affect the Program’s work activities and resource requirements. The 

Program is also required to provide an explanation of the budget estimation methodology in the 

Methodology for Determining Resources attachment that accompanies the Budget Request by 

Program submission. NSIR also prepares Background/Issue Papers for some key line items in 

the IR budget request, which provide additional justification for IR budget estimates.1

 OCFO 

performs an independent financial analysis of the budget submissions while OEDO conducts 

programmatic analysis on the integrated budget. 

Standard 3: Program can relate performance, budget, spending, and 

workforce information 

For this evaluation standard, we based our assessment on a combination of GAO’s framework 

for results-oriented budget practices, OMB Circular A-11 guidance, and OMB PART criteria. 

We performed our evaluation by reviewing program documentation, analyzing office and 

program budget submissions, reviewing financial and operating performance reports, and 

 

1

 Background/Issue Papers were developed by NSIR but are not required for the FY 2010 budget process 

and are not likely to be required for the FY 2011 process. 
conducting interviews with program officials. We assessed the extent to which the Program 

meets the specific criteria for this standard, including: 

• Relates budget, workforce, accounting, and performance information; and 

• Accounts for both the direct and indirect costs needed to meet performance targets and 

achieve program goals. 

Based on our review and analysis, the Program needs improvement in relating performance, 

budget, spending, and workforce information. Below are our detailed findings for this evaluation 

standard. 

Finding: The Program needs improvement in relating budget, workforce, accounting, and 

performance information. 

As part of the budget development, the program staff identifies measures and targets that 

supplement those in the annual performance budget to track expected performance in the areas of 

quantity, cost, quality, and timeliness of planned activity products. These measures are included 

in the Operating Plans of the program offices. During the budget execution year, performance is 

monitored and reported to various levels of agency management. For example, in NRR, the lead 

office for the RO program, senior management meets monthly to review the Performance 

Monitoring Report, which reports the office performance on its annual Operating Plan. This 

monthly report relates a portion of the budget and FTE allocations to actual obligations and FTE 

utilization as well as to performance targets and results. On a quarterly basis, offices are 

required to submit performance reports to OEDO via the SharePoint site, which serves as a data 

collection and reporting system. Then, office directors meet with the respective DEDOs to 

review office performance in meeting program goals and objectives each quarter. In NSIR, the 

lead office for IR, there is the DPR OPS Plan Performance Indicators Report that tracks the 

performance of the IR program against the operating plan. Both NRR and NSIR have financial 

reports, such as the Financial Plan tracking spreadsheet and FTE utilization reports in NSIR and 

the Budget Execution Status Report in NRR, which track budget vs. actual financial 

performance. The Funds Control Team in OCFO also issues the monthly and quarterly Budget 

Execution Report (BER) that relates budget, workforce, accounting, and performance 

information for the agency overall and allowance holders. 

While there are budget execution reports at the agency, allowance holder, and office levels and 

other financial and performance reporting at the office level, there is no comprehensive 

monitoring and reporting at the program level that provides a complete account of performance 

for resources budgeted vs. expended, which are tied to outputs/outcomes targeted vs. achieved 

with those resources. As noted in the first standard above, the Program needs improvement in 

relating budget to performance. In addition, the agency needs to improve its information systems 

to allow for efficient and timely data collection, analysis, and reporting of this information at the 

program level and must also assign responsibility for financial and performance management at 

the program level.

46 

 
Finding: The Program accounts for both the direct and indirect costs needed to meet 

performance targets and achieve program goals but further assessment is recommended to 

ensure that the full cost budget allocation for infrastructure and support is accurate 

The program budget as presented in the NRC Performance Budget for FY 2009 accounts for 

both direct and indirect costs of the Program. As explained in Appendix III, Explanation of the 

Full Cost Budget Allocation, of the Performance Budget, the indirect costs, (also referred to as 

infrastructure and support costs) for the agency were identified and distributed to programs as a 

portion of the total program cost consistent with the allocation methodology used for preparing 

the agency’s financial statements. The agency’s infrastructure and support involve centrally 

managed activities that are necessary for the staff and agency programs to achieve their goals. 

These activities include rent and facilities management, approved space acquisition, physical and 

personnel security, administrative support services, acquisition of goods and services, human 

resources management, training and development, matters involving small and disadvantaged 

businesses and civil rights, information technology, information resources management, planning 

and budget analysis, accounting and finance, and policy support services to the Commission and 

program area staff in performing regulatory mission activities and achieving their performance 

goals. From our interviews with program and support office staff, we found some concerns 

regarding the allocation methodology and that the indirect cost allocation to the Program may be 

inaccurate. 

In response to our interview findings, we conducted a direct vs. indirect cost analysis of the 

Nuclear Reactor Safety Program (NRS) and compared RO and IR percentages with the other 

subprograms. Direct costs are the Program Support costs and indirect costs are the Infrastructure 

and Support costs from the NRC Performance Budget for FY 2009. We found that the indirect 

portion of total RO funding was significantly higher by 10 percentage points on average when 

compared with other subprograms in NRS. The table below shows the indirect percentage for 

RO in FY 2009 is almost 42% versus 32% for NRS overall. The IR program is consistent with 

NRS at 32%. We also calculated the indirect cost percentage for the Nuclear Materials and 

Waste Program, which was lower at 29%. We recognized that the total FTE for RO is relatively 

higher; however, when we applied a pro-rata share of RO indirect costs based on RO FTE, we 

came up with an indirect cost percentage of 36%. We attempted to meet with the manager 

responsible for the full-costing allocation to better understand the variance but because of timing 

constraints, we were unable to further investigate and provide an explanation for this report. 

There may be a reasonable programmatic explanation for this difference, but because we have 

limited information, we included a recommendation for the Program to assess the full-costing 

allocation methodology to ensure the infrastructure and support costs that are allocated to the 

RO-IR program are indeed accurate. 

47 

 
48 

 

Table 7 - Direct vs. Indirect Cost Analysis of Nuclear Reactor Safety Program Budget 

Nuclear Reactor Safety (NRS)

Program Support (Direct) $M FTE $M FTE $M FTE $M FTE $M FTE $M FTE

New Reactors $93.2 355 $178.9 667 $175.0 668 75.2% 79.2% 76.3% 83.5% 73.7% 82.3%

Reactor Licensing Tasks 146.6 645 150.5 642 162.5 648 71.3% 81.3% 70.5% 81.1% 72.1% 81.7%

Reactor Oversight 139.0 847 139.5 838 148.6 848 59.3% 77.8% 58.3% 77.9% 58.2% 77.4%

Reactor License Renewal 19.4 93 14.3 88 23.8 108 71.1% 81.6% 63.6% 81.5% 71.5% 83.1%

Incident Response 11.8 50 12.9 56 16.0 53 67.8% 80.6% 65.8% 77.8% 67.8% 74.6%

International Activities 5.9 32 8.2 31 8.3 30 68.6% 82.1% 72.6% 81.6% 73.5% 81.1%

Subtotal Program Support $415.9 2,022 $504.3 2,322 $534.2 2,355 67.4% 79.5% 68.1% 80.5% 67.9% 80.2%

Infrastructure & Support (Indirect)

New Reactors $30.8 93 $55.5 132 $62.5 144 24.8% 20.8% 23.7% 16.5% 26.3% 17.7%

Reactor Licensing Tasks 59.0 148 63.0 150 63.0 145 28.7% 18.7% 29.5% 18.9% 27.9% 18.3%

Reactor Oversight 95.4 241 99.8 238 106.8 247 40.7% 22.2% 41.7% 22.1% 41.8% 22.6%

Reactor License Renewal 7.9 21 8.2 20 9.5 22 28.9% 18.4% 36.4% 18.5% 28.5% 16.9%

Incident Response 5.6 12 6.7 16 7.6 18 32.2% 19.4% 34.2% 22.2% 32.2% 25.4%

International Activities 2.7 7 3.1 7 3.0 7 31.4% 17.9% 27.4% 18.4% 26.5% 18.9%

Subtotal Infrastructure & Support $201.4 522 $236.3 563 $252.4 583 32.6% 20.5% 31.9% 19.5% 32.1% 19.8%

Total NRS Program $617.3 2,544 $740.6 2,885 $786.6 2,938 

FY 2009 

% Indirect

FY 2007 FY 2008 FY 2009 FY 2007 FY 2008

Enacted Request % Direct

Note: Numbers may not add due to rounding. 

Source: NRC Performance Budget for FY 2009 and FocalPoint analysis 

3.4. Recommendations 

Establish published crosswalk of the program budget to demonstrate alignment of costs to 

program outputs and annual and long-term outcomes

The program budget request lacks transparency in that it does not clearly define the relationship 

between resources and annual and long-term performance goals. In order to demonstrate that 

performance informs budget formulation and resource allocation decisions reflect desired 

performance levels, the Program should coordinate with OCFO, who leads the PBPM process 

and development of the annual Performance Budget and Performance and Accountability Report, 

to present the RO and IR program budget in a more transparent manner. This may include 

aligning the program budget with the agency’s strategic goals, safety and security, and linking 

the Program’s strategic outcomes, performance measures, and output measures. For example, 

the Program may consider presenting its budget request as shown in the table below with 

resources linked to strategic goals and long-term outcomes and then also providing an illustration 

of the performance management framework (see recommendation below) that links the program 

outputs and annual outcomes to these long-term outcomes: 
Table 8 - Example of Crosswalk of Program Budget Ensure adequate protection of public health and safety and the environment. Prevent the occurrence of any nuclear accidents. Prevent the occurrence of any inadvertent 

criticality events.Prevent the occurrence of any acute radiation exposure resulting in fatalities.Prevent the occurrence of any releases of radioactive materials that result in significant radiation exposures.Prevent the occurrence of any releases of radioactive materials that cause significant adverse environmental impacts.Ensure adequate protection in the secure use and management of radioactive materials.Prevent any instances where licensed radioactive materials are used domestically in a manner hostile to the security of the United States.

Total

Reactor Oversight $233.2 $22.2 $255.4

Incident Response $23.6 $23.6

Total $256.8 $22.2 $279.0

Safety Security

Develop logic model or performance framework that clarifies linkage between outputs and 

outcomes

The relationship between resources and annual and long-term performance goals is not clearly 

defined. The Program is complex for the uninitiated reader and a logic model or redesign 

framework might facilitate understanding of the linkage between outputs and outcomes (e.g., 

oversight activity and safety). The Program may consider developing a logic model or revising 

the ROP performance framework to clarify the linkage between the oversight activities and 

program outcomes for someone who is not indoctrinated in the NRC mission. The logic model 

development will be an opportunity to add/retain a limited number of measures that are 

appropriate, meaningful, and reflective of desired program outcomes, and provide 

comprehensive and quality measurement. This will also present an opportunity to eliminate any 

measures that do not meet the criteria. Below is a sample illustration of a performance 

framework that links program objective, activity, outputs, and annual and long-term outcomes. 

This illustration is for demonstration purposes only and is not an actual representation of the 

Program. 

49 

 
50 

 

Figure 6 - Sample Illustration of Performance Framework 

Key Objectives

Stakeholders

Constituents

Operations

Activities Outputs

Annual 

Outcomes

Long-term 

Outcomes

Reactor 

inspections

• Baseline/Reactive/ 

Supplemental/Other 

inspections

• Inspections 

reports and safety 

assessment 

• SDP 

determinations

• New conditions 

evaluated as red by 

ROP is <= 3.

• Significant ASPs of 

a nuclear reactor 

accident is zero.

• Operating reactors 

in degraded or 

unacceptable 

cornerstone of ROP 

Action Matrix is <=3.

Ensure 

completion of 

inspections, 

performance 

assessments, 

and reporting on 

safety status of 

licensees.

• No nuclear reactor 

accidents.

• No inadvertent criticality 

events.

• No acute radiation 

exposures resulting in 

fatalities.

• No releases of 

radioactive materials that 

result in significant 

radiation exposures.

• No releases of 

radioactive materials that 

cause significant adverse 

environmental impacts.

Develop marginal cost model to conduct sensitivity analyses

The Program needs improvement in demonstrating the impact of funding on annual and longterm outcomes. The Program may consider developing a model that analyzes the impact of 

reducing and increasing funding by a given percentage on annual and long-term outcomes. With 

the Program’s goal of zero nuclear accidents and the public’s intolerance for a nuclear incident, 

developing a programmatically relevant marginal cost model represents a challenge to the 

Program. The Program may consider developing a marginal cost analysis to demonstrate the 

impact of funding on output performance and extrapolating these results to provide a qualitative 

explanation of the impact on annual and long-term outcomes. 

Assess the full costing methodology to ensure accurate representation of full program costs 

Our analysis of direct vs. indirect costs for the Program found a relatively higher indirect cost 

percentage for the RO program than the other programs under the Nuclear Reactor Safety 

Program. There may be a reasonable programmatic explanation for the higher indirect costs, but 

due to our inability to obtain an explanation of the full costing methodology and how it was 

applied to the RO program, we recommend the Program assess the full costing allocation 

methodology to ensure the infrastructure and support costs that are allocated to the Program are 

accurate and representative of its full program costs. 
Improve performance reporting to provide integrated and comprehensive monitoring of 

financial and operating results against plan in a timely manner

The Program needs improvement in relating budget, workforce, accounting, and performance 

information. Improved performance reporting that provides a complete and timely account of 

program resources budgeted vs. expended which are tied to outputs/outcomes targeted vs. 

achieved enables management to monitor ongoing performance and to ensure program targets 

are achieved. The annual results of this report can be used to develop the Performance Budget 

and the Performance and Accountability Report as well as reporting to OMB’s expectmore.gov 

site each year. Implementation of this recommendation requires system enhancements to support 

efficient and timely data collection, analysis, and reporting. In addition, responsibility should 

also be assigned for financial and performance monitoring and reporting at the program level. 

Our assessment does not include a separate recommendation for systems improvements since the 

Agency already has a number of initiatives underway. Currently, NRC is implementing a new 

core financial management system that combines the functionality of the core accounting, license 

fee billing, cost accounting, allotment/allowance financial plan, and the capitalized property 

systems. Some key benefits sought from this new system include improved access, timeliness, 

efficiency, and overall quality in financial and performance reporting. The agency’s time and 

labor system is also undergoing a major upgrade which is expected to improve the Program’s 

capability to collect information for fee billing, and cost accounting and provide a wider range of 

management reports. 

51 

 
4. Internal Evaluation and Improvement 

4.1. Objectives / Key Activities 

There are several mechanisms through which the Program evaluates itself and seeks continuous 

improvement. These include a variety of self-assessments and support for independent 

assessments. The Program’s objectives in these efforts are to evaluate its performance, solicit 

input from stakeholders and constituents, and improve effectiveness and efficiency. The table 

below lists some of the Program’s evaluation efforts, past and present. 

Table 9 - Selected Evaluation Efforts 

SelfAssessments 

• ROP Self-Assessment (annual) 

• Inspection Procedure Review (every 2 years) 

• Lessons Learned Task Forces (ad hoc, situation specific) 

• Internal staff surveys (biennial) 

• IR Self Assessment Process (IRMC 0210) 

• Corrective Action Process (IRMC 0220) 

• After Action Reports 

Independent 

Assessments 

• The Pilot Program Evaluation Panel (1999) 

• The Initial Implementation Evaluation Panel (2001) 

• External Stakeholder Survey (biennial) 

• OIG Audits of specific RO elements (2002, 2005, 2005) 

• OMB PART Assessment (2003) 

• GAO Report on Nuclear Reactor Oversight (2006) 

• Acton Burnell Organizational Assessment (2004) 

• FEMA review of IR continuity plans 

• OIG Audit of NRC’s Incident Response Program (2004) 

• OIG Audit of NRC’s Office of Nuclear Security and Incident Response 

(2006) 

• OIG Audit of Emergency Preparedness (2007) 

• Advisory Committee on Reactor Safeguards (ACRS) (2003) 

52 

 
4.2. Approach 

The approach for assessing this evaluation area consisted of three key components as outlined 

below. 

1. Review of relevant documentation and published material. We reviewed both internal 

program documentation and conducted a brief literature review of published documentation 

related to program evaluation and performance improvement. 

The following internal documents were reviewed: 

• IMC 0307 ROP Self-Assessment Program 

• Annual ROP Self-Assessment Reports 

• IRMC 0210 IR Self Assessment Procedure 

• IRMC 0220 Corrective Action Procedure 

• MD 8.2 NRC Incident Response Program 

• The NRC Incident Response Plan (IRP), NUREG-0728, Revision 4 

• Davis-Besse Lessons Learned Task Force (DBLLTF) Report 

• DBLLTF Final Status Of Recommendations 

• Public Version Of Palo Verde Lessons Learned 

• ROP Budget History – Significant Events Impacting ROP Budgets 

• Pilot Program Evaluation Final Report (1999) 

• U.S. Nuclear Regulatory Commission Charter for the Performance Improvement Panel 

Reactor Oversight Process Initial Implementation Evaluation Panel Final Report (2001) 

• SECY-03-0104 Organizational Effectiveness Assessment for the Office of Nuclear Security 

and Incident Response (2003) 

• OIG: Audit of the Significance Determination Process (2002) 

• OIG: Audit of the Baseline Inspection Program (2004) 

• OIG: Audit Nuclear Regulation: NRC Needs to More Aggressively and Comprehensively 

Resolve Issues Related to the Davis-Besse Nuclear Power Plant's Shutdown (2004) 

• OIG: Audit of NRC’s Incident Response Program (2004) 

• OIG: Audit of the Reactor Program System (2005) 

• OIG: Audit of NRC’s Office of Nuclear Security and Incident Response (2006) 

• OIGL Audit of NRC’s Emergency Preparedness Program (2007) 

• GAO: NRC: Oversight of Nuclear Power Plant Safety Has Improved, but Refinements are 

Needed (2006) 

53 

 
• Marks, & Potter: “Drilling for Results: The quest for objective exercise evaluations,” 

Homeland First Response, July/August 2004. 

• Power & Martin: “Using Evaluation Criteria to Organize the Planning, Conduct, and 

Oversight of Readiness Assurance Activities,” DOE. 

• American Nuclear Society: Criteria for Planning, Development, Conduct, and Evaluation of 

Drills and Exercises for Emergency Preparedness, American Nuclear Society. ANS-3.8.7-

1998 

• U.S. DHS: Corrective Action Program System Overview 

• Homeland Security Exercise and Evaluation Program, Volume III: Exercise Evaluation and 

Improvement Planning, Department of Homeland Security, 2007. 

• Ford & Schmidt: “Emergency Response Training: Strategies For Enhancing Real-World 

Performance,” Journal of Hazardous Materials 75_2000.195–215 

• An Analysis of the Office of Management and Budget’s Program Assessment Rating Tool 

(PART) for Fiscal Year 2008, Mercatus Center. 

From the literature review, we identified a set of best practices and criteria to be used as 

standards for the assessment. 

2. Conduct staff interviews. We conducted interviews with stakeholders and Program staff. 

We developed interview guides and conducted semi-structured, confidential interviews to obtain 

information related to program evaluation and performance improvement. 

3. Analyze current performance. Based on our literature review, interviews, and study of 

successful organizations, we established standards or performance metrics on which we based 

our analysis. We performed operational analysis to compare the Program’s current performance 

against these criteria, including assessment of internal procedure documentation, application of 

processes, use of performance metrics, tracking of recommendations, development and 

recommendation of action plans, and internal and external stakeholder feedback. To evaluate the 

Program’s internal assessment and improvement, we assessed the Program on the following 

standards: 

• Conducts internal assessments of efficiency and effectiveness that are used to modify 

strategic, operational and financial plans; 

• Uses independent evaluations to modify strategic, operational and financial plans; and 

• Applies system for making improvements within appropriate timeframes. 

54 

 
4.3. Findings 

Standard 1: Conducts internal assessments of efficiency and effectiveness 

that are used to modify strategic, operational and financial plans 

For this standard, we evaluated the extent to which the Program has processes in place to 

evaluate its progress in achieving goals. We reviewed documentation on the Program’s selfassessment processes including the methodologies used and the performance reporting. The 

criteria used were a combination of OMB PART, best practices garnered from our literature 

review, and staff and stakeholder interviews. We assessed the extent to which the Program 

meets criteria such as: 

• Collects high quality performance data on critical aspects of its operation; 

• Performance information is used to adjust strategic, operational and financial plans; and 

• Meaningful performance targets are established based on adequate baseline data. 

Finding: The Program is effective in conducting periodic internal self-assessments 

The Program has processes and measures in place for self-assessments (IMC 0307). The selfassessment process for the reactor oversight was first published in 2002 and has since been 

revised five times to reflect changing safety and security objectives and streamlining. The 

historical data provides an adequate baseline that enables the Program to set meaningful targets. 

The self-assessment process includes 49 performance measures with targets that cover key 

program areas. As part of this process, the Program also consolidates feedback from internal 

staff and external stakeholders. 

The Program also uses incident response performance measures in the operating plan that tracks 

the Program’s performance in preparing for, executing, and evaluating exercises and drills. 

These measures cover performance in exercises, response times, center operations and systems, 

and staffing levels, skills, training, and participation. The Program also provides a summary 

Emergency Response Performance Index as part of the annual Performance Budget submission. 

In 2005, the Program documented its incident response self-assessment procedures under 

Incident Response Manual Chapter but not all appendixes were included until 2008. In 2008, the 

Program is documenting its process to manage identified recommendations and track the 

completion of associated corrective actions (IMC 0220) for implementation in early January 

2009. 

In addition, Lessons Learned Task Forces have been applied to risk-significant inspection 

findings to identify scope, cause, and corrective actions. Identified actions are incorporated into 

the annual ROP self-assessment process for implementation and tracking. Examples of such 

efforts include taskforces for Davis-Besse, Palo Verde, mitigating systems performance index, 

and safety culture evaluation. After Action reports from incident response are used for all 

exercises, drills, and incidents to document lessons learned and prioritize opportunities for 

improvement. 

55 

 
Finding: The Program is effective in applying lessons learned from internal assessments to 

improve strategic, operational and financial plans 

The Program applies findings from self-assessments to adjust its activities. For example, the 

Program performs its reactor oversight assessment annually (ROP Annual Self- Assessments) 

and reviews performance against specified targets. It reviews how effectively it executes its key 

program areas (e.g., performance indicators, inspection, significance determination process, 

assessment). It then determines the allocation of resources based on historical performance to 

ensure resources are adequately aligned to highest value efforts. The Program has regularly 

performed procedure reviews to align procedures with changing safety and security objectives. 

For example, the Program conducts a biennial realignment process to streamline baseline 

inspection efforts and focus them on the highest value procedures. Every 2 years, this process 

reviews existing baseline inspection procedures to identify areas where depth, scope, and 

frequency of each of the inspection procedures should be modified. This has allowed the 

Program to streamline procedures and focus inspection resources on areas with higher safety 

risk. In addition, the Program periodically reviews and refines its incident response procedures 

(NUREG-0728) to meet internally identified issues. 

Standard 2: Uses independent evaluations to modify strategic, operational 

and financial plans 

For this standard, we assessed the extent to which the Program uses the results of independent 

evaluations to adjust its strategic, operational and financial plans. We reviewed previous 

evaluations and the Program’s improvement activities. The criteria used for this standard were a 

combination of OMB PART, best practices from our literature review, and staff and stakeholder 

interviews. The primary criteria were: 

• Evaluations were of sufficient quality and scope; and 

• Extent to which the Program had addressed the findings in the evaluations. 

Finding: Independent evaluations to modify strategic, operational and financial plans are 

used effectively 

The Program has used the results of independent evaluations, such as those conducted by the 

Government Accountability Office (GAO), Inspector General (IG), OMB, and others, to guide 

implementation of program improvements. For example, implementation of recommendations 

from IG assessments have led to improvements in areas such as the Significance Determination 

Process (2002), the Baseline Inspection Program (2004), the Incident Response Program (2004), 

the Reactor Program System (2005), and NRC’s Emergency Preparedness Program (2007). The 

Program tracks implementation of IG recommendations and reports quarterly to the Office of the 

Executive Director of Operations. Examples of implemented incident response improvements 

include establishment of a formal emergency response organization qualification program, 

improved use of technology and consideration of human factors to improve communications and 

56 

 
facility infrastructure, and inclusion of additional performance-based training opportunities for 

response teams. 

The Program has also made improvements based on GAO evaluations. For example, in 2004, 

the GAO reviewed the Davis-Besse nuclear power plant shutdown. In 2006, it undertook an 

assessment of the oversight of nuclear power plant safety. The latter found that “NRC has 

improved its oversight process in various areas, but it has been slow to act on needed 

improvements, particularly in improving the agency’s ability to identify and address early 

indications of declining safety performance.” In response to this feedback, the Program has 

developed action plans to address these recommendations and tracks the timeliness of 

completion. 

The Program has had several other independent evaluations performed. Oversight reviews were 

conducted by advisory committees, including the Advisory Committee on Reactor Safeguards 

(ACRS). Specific panels were established, by charter under the rules of the Federal Advisory 

Committees Act, to evaluate ROP effectiveness independently. In 2003, one year after the 

creation of the incident response Program within NSIR, Acton Burnell, a consulting firm, 

performed an independent assessment that developed performance measures and made 

recommendations for NSIR’s organizational effectiveness, public relations policy, internal 

processes and procedures, and overall effectiveness of internal and external communications. 

However, per an OIG report in 2006, many of these recommendations have not yet been 

sufficiently evaluated, in part because of the large level of emergent work arising from 9/11. 

Also, FEMA performed an assessment of the operational continuity of the Program’s 

Headquarters Operations Center, in which the Center was highly rated. 

Standard 3: Is systematic in making program improvements within 

appropriate timeframes 

For this standard, we assessed the extent to which the Program is systematic in making 

improvements when weaknesses are identified. We reviewed the Program’s consistency and 

success in implementing improvements. The criteria used for this standard were a combination 

of OMB PART and best practices identified from our literature review. The primary criteria 

were: 

• Extent to which identified improvement plans were consistently implemented; 

• Extent to which there were processes in place to implement improvement plans; and 

• Program uses clear and consistent program documentation to improve processes. 

Finding: The Program has effectively and consistently refocused its efforts in response to 

changing safety and security objectives 

After the 9/11, terrorist attacks, the scope of the Program expanded to address additional security 

objectives. Working with licensees, NRC supplemented its Design Basis Threat (DBT) in 2003 

57 

 
and 2006 to incorporate additional security requirements. In addition to mandatory directives 

and advisories issued after the 2001 attacks, the Program required nuclear power plants to 

implement security-related enhancements into their emergency preparedness programs in 2005. 

Among these enhancements were trained and armed security officers, equipment and structure 

intrusion detection and surveillance systems, and access controls. 

In addition to its core nuclear power plant operational risk assessments, the Program has 

incorporated these additional security elements into its risk-informed oversight and emergency 

preparedness plans. The Program has also expanded its coordination of threat information and 

response efforts with other federal agencies, including DHS, FBI, intelligence agencies, the 

departments of Defense and Energy, States, and local law enforcement. The Program increased 

annual security inspection hours at reactor facilities from 1,600 hours to over 8,000 hours 

between 2001 and 2006. Force-on-force annual inspection activity increased from 2,000 to 

7,700 hours between 2001 and 2006. 

The Program has also increased its focus on safety culture. The Davis-Besse Lessons Learned 

Task Force reemphasized the importance of safety culture. Especially in a risk-informed 

regulatory context, staff at nuclear facilities must be encouraged to raise safety concerns without 

fear of retaliation by licensee management or the NRC. The Program enhanced ROP treatment 

of cross-cutting issues to identify safety culture problems. The Program modified its inspection 

manual chapters, inspection procedures, and inspector training. 

Finding: The Program is effective in using a systematic approach for implementing program 

improvement but completeness and timeliness of some documentation needs improvement 

The Program applies the documented procedures to consolidate and take action on 

recommendations in a timely manner. Recommendations from all internal and external 

assessments are integrated into improvement plans through either the annual ROP SelfAssessment Program or the draft IR Corrective Action procedure. The Program uses 

performance metrics in the operating plan with a target of timely completion of OEDO ticketed 

items of 95 percent. The Program has achieved its target for all fiscal year 2008 quarters. In 

2008, the Program implemented two new performance measures to track the timely completion 

of priority-1 and priority-2 incident response action items. 

Prior to 2007, the annual ROP Self-Assessment Report contained an enclosure that listed 

recommendations and status of implementation plans to address each identified gap. The 

Program is improving its execution of its incident response improvement plans. Upon 

implementation in early January 2009, the IR Corrective Action procedure, consistent with the 

DHHS Corrective Action Program System, will support integrated tracking of recommendations 

and implementation plans. Recommendations from After Action Reports, OIG audits, other 

audits, self-assessments, and issues documented on Problem Identification Forms will be 

consolidated and prioritized in one database and assigned to action plans for implementation. 

58 

 
The new procedure calls for each corrective action to be tracked for extensions, completion 

timeliness, and effectiveness of implementation. 

Based on our review, we found some of the documentation for the incident response function 

was not current, incomplete, or inconsistent. For example, the published NUREG 0728, Rev. 4 

is stamped as “interim release mode” since its April 15, 2005 issue date and is pending Rev. 5 in 

2009. We also found that regions and headquarters have different documented procedures for 

exercise coordination. These procedures would benefit from more consistent maintenance, 

increased alignment between regions and headquarters organizations, and increased level of 

detail to ensure consistent response actions. The main documents for IRMC 0410 Drill and 

Exercise Standards, 0420 Drill and Development Coordination, and 0810 Outreach Programs are 

not available from the IRMC published Web site although Appendix A of each was available. 

The Program has taken steps to address these issues. For example, the Program has performed 

regional reviews to identify and adopt best practices between regions. It has standardized its 

manual chapter formatting. Also, it has updated its program improvement and corrective action 

procedures. 

4.4. Recommendations 

Complete, implement, and maintain timeliness of procedure documentation 

In some areas, the procedure documentation was out of date. The main documents for IRMC 

0410 Drill and Exercise Standards, 0420 Drill and Development Coordination, and 0810 

Outreach Programs are not available from the IRMC published Web site although Appendix A of 

each was available. In some cases, regions and headquarters have different documented 

procedures for exercise coordination. The published NUREG 0728, Rev. 4 is stamped as 

“interim release mode” since its April 15, 2005 issue date and is pending Rev. 5 in 2009. IRMC 

0210 would benefit from greater specificity of procedures. Going forward, the Program should 

focus on more consistent document maintenance, increased alignment between regions and 

headquarters organizations, and increased level of detail to ensure consistent response actions. 

Identify and document objectives from implementation actions in terms of outcome 

improvements, timeliness, process and/or cost efficiencies 

While improvements are tied to specific recommendations, clear objectives and their impacts are 

not always developed. The Program should identify objectives of each action plan as it is 

developed and approved and then confirm the impact of improvement actions. The objectives 

should identify the problems or issues being addressed, expected benefits, and criteria for 

success. 

59 

 
5. Public Communications 

5.1. Objectives / Key Activities 

The objective of the Program’s public communications activities is to support the NRC’s 

organizational excellence objectives of openness and transparency. With regard to reactor 

oversight, the Program performs a number of communications activities. It provides background 

information as well as detailed procedure and outcome information on the NRC public Web site. 

It provides public forums with key stakeholders through its monthly ROP Working Group 

meetings, annual end-of-cycle regional meetings, and annual assessment meetings with each 

licensee. The Program also monitors and provides periodic announcements to media channels. 

To gather feedback, it performs an annual ROP external survey, which is incorporated into its 

annual ROP self-assessment. 

With regard to incident response, the Program also provides information through the NRC public 

Web site. In order to gather public and stakeholder input into its procedures, the Program 

publishes draft documents in the Federal Register and uses public meetings focused on specific 

topics. 

5.2. Approach 

The approach for assessing this evaluation area consisted of three key components as explained 

below. 

1. Review of relevant documentation and published material. We reviewed internal program 

documentation and conducted a brief literature review of published documentation related to 

public communications. 

The following internal documents were reviewed: 

• ROP Homepage 

• ROP Program Evaluations and Feedback (independent evaluations, internal and external 

surveys, annual self-assessments, etc) 

• Consolidated Response to ROP External Surveys 

• ROP Program Documents 

• NRC Emergency Preparedness Web site 

• Management Directive 8.2 NRC Incident Response Program 

• The NRC Incident Response Plan (IRP), NUREG-0728, Revision 4 

• Incident Response Public Meetings Web site 

• OIG: Audit of NRC’s Emergency Preparedness Program (2007) 

60 

 
• OIG: Audit of NRC’s Office of Nuclear Security and Incident Response (2006) 

• OIG: Inspector General's Assessment of the Most Serious Management Challenges Facing 

NRC (2008) 

• NRC Management Directive 3.14 U.S. Nuclear Regulatory Commission Public Web Site, 

2008 

• SECY-03-0104 Organizational Effectiveness Assessment for the Office of Nuclear Security 

and Incident Response (2003) 

• SECY-02-0036 Formation of NSIR and Communications Mandate from Commission (2002) 

From the internal documentation and literature review, we identified a set of leading practices 

and criteria to be used as standards for the assessment. 

2. Conduct staff interviews. We conducted interviews with the media, stakeholders, and 

program staff. We developed interview guides and conducted semi-structured, confidential 

interviews to obtain information related to public communications. 

3. Analyze current performance. Based on the interviews, literature review, and research, we 

established standards or performance metrics on which we based our analysis. We performed 

analysis to assess the Program’s provision of program information to all affected parties 

including federal, state, and local entities and the general public, as well as its solicitation of 

feedback to these stakeholders. We then analyzed the program’s current performance against 

these criteria. 

To evaluate the Program’s performance in the area of public communications, we assessed the 

program on the following standards: 

• Provides program information to and solicits the views of all appropriate stakeholders; and 

• Adequately reviews and incorporates received suggestions and comments and provides 

feedback on actions taken. 

5.3. Key Findings 

Standard 1: Provides program information to and solicits the views of all 

appropriate stakeholders 

For this standard, we assessed the extent to which the Program has established two-way 

communication with its stakeholders. We interviewed NRC staff and stakeholders, and reviewed 

data from the Program’s stakeholder surveys. The criteria used were based on NRC policy, OIG 

reports, best practices from our literature review, and stakeholder and staff interviews. NRC 

Management Directive 3.14 defines the objectives of communications as contributing to 

openness and public confidence by enhancing the public’s understanding of NRC’s mission, 

goals, programs, and activities and enhancing the ability of stakeholders to participate effectively 

in the regulatory process. At the end of FY 2008, the IG’s Assessment of the Most Serious 

61 

 
Management and Performance Challenges Facing NRC identified as NRC-wide challenge #2, 

“Managing information to balance security with openness and accountability.” As stated in the 

report, the three specific issues related to this challenge include: 1) Manage information in 

accordance with new Federal Government policies for designating, marking, safeguarding, and 

disseminating controlled unclassified information (CUI); 2) Ensure that sensitive information is 

handled in accordance with agency policies and procedures for public disclosure; and 3) Provide 

external stakeholders with clear and accurate information about regulatory programs, and 

facilitate public participation in the regulatory process. 

Finding: The Program provides significant information to all affected parties but needs to 

increase the information provided in some areas while balancing the need for security with 

openness and accountability 

The Program successfully uses the Internet and public meetings to provide the public with 

information, but there are gaps. There is a broad set of documents related to reactor oversight on 

the NRC public Web site. This includes policy and direction, procedures, industry trends, 

performance indicators, licensee inspection performance, and assessment results. For the most 

part, the information is complete and up-to-date. 

As described in Section 2, the Program is effective in creating and disseminating incident 

information in the event log. The log entries are clear and concise and provide a good base for 

initial analysis of the incident. The IG found weaknesses in other aspects o the Program’s 

communications in this area. With regard to incident response, per an OIG audit in 2006, 

“weaknesses exist in NSIR’s process for interacting with internal and external stakeholders and 

the office has performed limited assessments of the success of its communications” and “NSIR’s 

ability to share sensitive or classified information is limited to those who have a need to know, as 

well as by NRC policy, however their unclassified communication needs to be increased to 

enhance public confidence.” Since then, Emergency Preparedness has held a number of public 

meetings with stakeholders. While the current Emergency Preparedness section of the NRC 

Web site includes information on incident response objectives, roles and activities, the 

information could be expanded to increase stakeholder confidence in and understanding of the 

incident response function.. 

After the 9/11 terrorist attacks, NRC revised its information dissemination policy to limit 

information available to the public on regulatory decisions or actions involving security 

inspections, assessment, exercises, and enforcement. In mid-2006, NRC began making summary 

results of its security inspection program for nuclear power plants available to the public and has 

held three public meetings in 2008 to gather external stakeholder and the public comments on the 

appropriate level of openness and transparency of information associated with NRC security and 

performance assessments of NRC licensees. The objective is to allow the exchange of 

information with external stakeholders and the public without disseminating information that 

could pose a threat. The NRC has also started providing summary information on security 

62 

 
allegations. Additional changes to increase openness without risking security are currently under 

assessment by the Commission, with expected release of a revised policy in 2009. 

Finding: The Program should improve efforts to solicit information from stakeholders in 

some areas 

The Program uses public meetings to facilitate public access and comment, including monthly 

ROP Working Group Meetings and regional public meetings. Actions to facilitate stakeholder 

participation at public meetings have been and are being implemented. The Program has further 

facilitated participation by extending meeting access through conference call bridges and predistributing meeting materials. The regional offices conduct quarterly meetings to disseminate 

periodic assessment information for each licensee. The Program is providing more detailed 

agendas for reactor oversight meeting announcements and modifying RO Category 2 meeting 

processes to permit public comments at the end of each topic. However, some interviewees 

noted logistics problems with meetings including difficulty getting through security to attend 

meetings, insufficient handouts and seating for attendees, and the need for a microphone for 

speakers. We attended a public meeting and had difficulty getting through security. 

Since the start of 2005, the Program has been presented at over 20 public meetings on the topic 

of emergency preparedness/incident response. These have been used to communicate 

information to stakeholders as well as solicit input on open issues such as appropriate disclosure 

of secure information. 

The Program performs a biennial external survey related to its oversight function via Federal 

Register notice to collect comments and suggestions on its performance. The survey requests 

stakeholder input on a variety of program areas. This input is incorporated into the annual ROP 

self-assessment process. All received comments are included in an integrated report with the 

Program’s feedback and response to each received comment. In 2007, 700 external stakeholders 

from state, local, tribal, university, and public groups were solicited. One issue is the declining 

level of participation. In 2007, there were only 7 responses (down from 21 in 2004). A related 

issue is that some stakeholders who do not respond indicated they are skeptical that their input 

will be considered fairly and acted upon given their previous historical experience. Although we 

noted that the Program addresses all comments in its response. 

Standard 2: Adequately reviews and incorporates received suggestions 

and comments and provides feedback on actions taken 

For this standard, we interviewed NRC staff and stakeholders, and review data from the 

Program’s stakeholder surveys. The criteria used were from best practices from our literature 

review. We assessed the extent to which the Program met criteria such as: 

• Thoroughly evaluates the concerns and suggestions raised by stakeholders; and 

• Incorporates suggestions by stakeholders, where appropriate into program plans. 

63 

 
Finding: The program is effective in reviewing and incorporating received suggestions and 

comments and providing feedback on actions taken 

The Annual ROP Self-Assessment Report provides survey feedback to stakeholders, which 

includes discussion of and proposed actions for every received comment. This annual report also 

provides a consolidated response to comments received to its biennial external survey in which it 

publishes the received comments as well as provides feedback on if and how the ROP is 

incorporating the comments in its efforts. Stakeholder input has resulted in several program 

enhancements that are incorporated into the annual ROP self-assessment. It should be noted that 

the Program is perceived as discounting input received by some stakeholders. Although, we did 

note that the Program systematically addressed all comments in its responses. The Program also 

publishes the results of its incident response public meetings and participation at other industry 

meetings on the NRC public site. 

5.4. Recommendations 

Improve logistics of public meetings 

Interviewees and respondents from the biennial external survey mention some shortfalls in the 

logistics of public meetings. The Program has made improvements, including providing more 

detailed agendas in reactor oversight meeting announcements and modifying Category 2 meeting 

processes to permit public comments at the end of each topic. Several participants at Working 

Group meetings also noted some logistical problems. The Program should ensure participants 

have access through security up until the time of the meeting, sufficient handouts and chairs are 

provided, and a microphone for speakers is available. 

Evaluate external survey program and include incident response activities in the survey 

One issue is the declining level of the reactor oversight survey participation. In 2007 there were 

only 7 responses (similar to 2006 levels). Another issue is that some stakeholders who do not 

respond are skeptical that their input will be considered fairly and acted upon by the Program. 

The Program should consider options to increase responsiveness and encourage greater 

participation in the external survey. The Program should also evaluate reasons for the decrease 

in responses through outreach to previous responders. Given findings, options to increase 

participation should be evaluated including modifying the survey instrument, increasing 

outreach, working with an organization like INPO, or other outreach campaigns. Also, the 

Program should solicit stakeholder input on incident response activities in the survey. 

64 

 
Increase publicly available Web-based information on incident response function and 

security cornerstone 

The Program should provide more information on the progress of its incident response efforts. 

As discussed above, the Program is assessing its approach for the dissemination of non-sensitive 

and unclassified security information and should continue to do so to enhance confidence with 

regard to activities associated with the security cornerstone. While the current Emergency 

Preparedness section of the NRC Web site includes information on incident response objectives, 

roles and activities, the information should be expanded to increase stakeholder understanding of 

and confidence in the incident response function. For example, information on incident response 

public meetings should include a post-meeting summary of received comments and 

feedback/actions taken. Also a periodic update on activities and improvement actions related to 

incident response would demonstrate progress. 

65 

 
6. Interaction with Other Agencies 

6.1. Objectives / Key Activities 

Success in many aspects of the Program requires interaction with other agencies. These include 

state and local, federal, and international organizations. The Program works with such 

organizations for purposes that include sharing of resources and expertise, knowledge sharing, 

and conducting joint operations. The Program coordinates emergency response support with 

other Federal agencies, including DHS/FEMA, DOE, FBI, and others. For example, 

collaboration for incident response takes place through several key coordinating/working groups. 

The Federal Radiological Preparedness Coordinating Committee (FRPCC) is an interagency 

body consisting of the Program and other Federal programs, chaired by DHS/FEMA. The 

FRPCC coordinates planning and validating requirements of each agency; reviewing integration 

requirements; and incorporating agency-specific plans, procedures, and equipment, into the 

response system to ensure minimum duplication and maximum benefits. NRC staff members are 

assigned to the Domestic Nuclear Detection Office, the National Counterterrorism Center, and 

the DHS Infrastructure Protection Office to enhance inter-organizational communication and 

support the integrated assessment of security-related information. Within NRC, the Program’s 

emergency preparedness function specifies intra-NRC roles and responsibilities for safety and 

security inspections of licensee emergency preparedness plans and running exercises. 

The Program shares inspection assessment results with state and local agencies through quarterly 

meetings and phone calls conducted by the regional offices. The nature of the communications 

varies between Regions, in response to the level of interest from region-specific state and local 

governments. The Program also coordinates with nuclear organizations in other countries for 

knowledge sharing. In this area, the Program works with the NRC’s Division of Policy and 

Rulemaking to interact and establish bilateral agreements with organizations in other countries. 

6.2. Approach 

The approach for assessing this evaluation area consisted of three key components as outlined 

below. 

1. Review of relevant documentation and published material. We reviewed internal program 

documentation and conducted a brief literature review of published material related to 

emergency response, implementation of Federal mandates and inter-agency coordination and 

communication. 

The following documents were reviewed: 

• Management Directive 8.2 NRC Incident Response Program 

• The NRC Incident Response Plan (IRP), NUREG-0728, Revision 4 

66 

 
• OIG: Audit of NRC’s Emergency Preparedness Program (2007) 

• International Atomic Energy Agency programs: IAEA meetings, Convention on Nuclear 

Safety and Integrated Regulatory Review Service 

• U.S. DHS: the National Response Framework 

• U.S. DHS: National Incident Management System 

• U.S. DHS: Target Capabilities List: A companion to the National Preparedness Guidelines 

(2007) 

• National Fire Protection Association: “NFPA 1600: Standard on Disaster/Emergency 

Management and Business Continuity Programs” (2007) 

• American Nuclear Society: Criteria for Planning, Development, Conduct, and Evaluation of 

Drills and Exercises for Emergency Preparedness, American Nuclear Society. ANS-3.8.7-

1998 

2. Conduct staff interviews. We conducted interviews with stakeholders and program staff. 

We developed interview guides and conducted semi-structured, confidential interviews to obtain 

information related to exercise and drill coordination within NRC and with stakeholders and 

implementation of Federal mandates. 

3. Analyze current performance. Based on the standards established from our literature review 

and research on best and leading practices, we assessed internal documentation on roles, 

responsibilities, and processes, federal emergency response requirements and guidance, and 

outcomes from working relationships with stakeholders.

To evaluate the Program’s interaction with other agencies, we assessed the Program on the 

following standards: 

• Has clearly defined roles, responsibilities, and procedures for working across agency 

boundaries; and 

• Tracks outcomes and performance of working relationships. 

6.3. Key Findings 

Standard 1: Has clearly defined roles, responsibilities and procedures for 

working across agency boundaries 

For this standard, we evaluated the extent to which the roles and responsibilities for other 

relevant agencies was established and documented. We based the assessment on a combination 

of standards from OMB, National Incident Management System (NIMS), best practices review, 

and staff and stakeholder interviews. 

Finding: Clearly defined roles, responsibilities, and procedures for working with federal 

agencies have been established effectively 

67 

 
The Program works extensively with other Federal agencies (e.g. DHS, DOE, EPA, USDA, 

DOT, and DOJ) and state and local entities in incident response efforts. The scope and 

responsibilities for these incident response interworking relationships are defined in published 

procedures and memoranda of understanding between agencies. The Program was a contributor 

to DHS’s NIMS and the National Response Framework (NRF). Individual memoranda of 

agreement/understanding have been established with other Federal agencies. Management 

Directive 8.2 and NUREG-0728 define NRC’s Incident Response Plan. The Program reviews 

and modifies its internal procedures to ensure conformance with evolving procedures of its 

partners (e.g., DHS). NRC participates in 5 of 15 DHS planning scenarios. NRC processes are 

documented in NUREG 0728. However, roles and responsibilities are less detailed for the 

intermediate and ingestion phase of an incident than they are for the emergency/plume phase. 

Additional detail and clarity would improve preparedness for such a situation. 

Each year, the Program successfully conducts a number of incident response exercises with 

licensees and federal, state, and local partners to apply these procedures, as described in Section 

2. 

Finding: The Program has effectively established international agreements to support its 

activities 

The Program works with international regulatory programs to bring about knowledge sharing to 

support the Program’s activities. The Program participates through multiple programs of the 

International Atomic Energy Agency (IAEA) and through OECD’s Nuclear Energy Agency. 

Each IAEA program defines specific roles and procedures to enable interworking. As well, NRC 

has a number of bilateral agreements with other countries’ nuclear regulatory agencies. There 

are 10-12 bilateral agreement meetings per year organized around jointly developed topics. NRC 

raises topics based on technical issues itemized in NRC’s Task Action Plans that address U.S. 

technical safety and procedure issues (for the investigation or confirmation of NRC plans). 

Standard 2: Tracks outcomes and performance of working relationships 

For this standard, we assessed the extent to which the Program evaluates the effectiveness of its 

working relationships. We based the assessment on a combination of standards from OMB, 

National Incident Management System (NIMS), and others from our best practices review. 

Finding: Outcomes and performance of Federal working relationships are tracked effectively 

The Program has implemented a number of incident response performance measures to track its 

participation in interworking efforts with other agencies. It manages its coordination with other 

agencies through a number of forums (e.g. the Quarterly NRC/DHS Steering Committee 

meetings, the FEMA Working Group, and the FEMA/NRC Exercise Taskforce). NRC staff 

members are assigned to the Domestic Nuclear Detection Office, the National Counterterrorism 

Center, and the DHS Infrastructure Protection Office to enhance inter-organizational 

communication and support the integrated assessment of security-related information. 

68 

 
Finding: The Program is improving the effectiveness of its working relationship with State 

stakeholders 

Under the Nuclear/Radiological Incident Annex and for incidents below the threshold of an 

Incident of National Significance, NRC is assigned the coordinating role with other Federal, 

state, and local stakeholders. A major emphasis in the NRC incident response is providing 

offsite authorities with an evaluation of license protective action recommendations. A 2007 OIG 

audit of emergency preparedness identified recurring coordination problems with States in 

preparing for, executing, and evaluating incident response exercises. Per the report, 

The weakness in NRC’s coordination with State authorities recurs because (1) NRC has 

not followed a consistent approach for working with the States during these exercises and 

(2) has not clearly defined or communicated its coordination role.... In January 2006, 

NSIR proposed tentative plans for updating internal guidance, conducting tabletop 

exercises, and hosting regional conferences designed to help NRC, Federal, and State 

government officials better understand mutual EP roles and responsibilities. 

The Program is taking steps to address issues with exercise communication with State 

stakeholders. NSIR has implemented plans for updating internal guidance, inviting State 

participants to observe exercises prior to an upcoming scheduled exercise, conducting tabletop 

exercises, and hosting regional conferences. We did not detect this weakness in the exercise we 

observed, however, the Program should enhance its exercise evaluations as described in Section 

2 to confirm the effectiveness of these efforts. 

6.4. Recommendations 

Clarify procedures for coordination and handoff after the emergency/plume phase of an 

incident 

The incidence response documents are clear and fairly detailed with regard to roles and 

responsibilities during the emergency and plume phase of an incident. However, they are less 

specific about what happens during the intermediate and ingestion phase. Additional detail and 

clarity would improve preparedness for such a situation. The Program should raise this topic 

through working group meetings with partners, update documentation, and test and evaluate 

these procedures through tabletop discussions or drills. 

69 

 
7. Financial Management 

7.1. Objectives / Key Activities 

To ensure proper stewardship of Federal resources, the NRC has policies and procedures to 

comply with federal financial management legislation, including the Federal Managers' Financial 

Integrity Act (FMFIA), CFO Act, GPRA, Government Management Reform Act, Federal 

Financial Management Improvement Act, Clinger-Cohen Act, Anti-Deficiency Act, and 

Improper Payments Information Act. These policies and procedures are documented in NRC 

Management Directives (MD), Volume 4 – Financial Management, which cover such areas as 

accounting, administrative control of funds, financial systems, management controls, license 

fees, lapsed appropriation periods, strategic planning, and budget formulation. 

NRC management is responsible for establishing and maintaining effective internal controls and 

financial management systems that are compliant with these laws and regulations and meet their 

objectives. A key piece of legislation is the FMFIA, which requires the agency to establish 

controls that ensure (1) obligations and costs comply with applicable law; (2) assets are 

safeguarded against waste, loss, unauthorized use, or misappropriation; and (3) revenues and 

expenditures are properly recorded and accounted for. 

NRC also has goals to address the PMA initiative for improved financial management including 

providing information that is reliable, transparent, useful, and timely to stakeholders and 

management for making decisions; maintaining adequate controls; and implementing integrated 

and flexible systems to meet the agency’s reporting needs. 

The key financial management activities performed by the program offices include budget 

execution, funds control, internal controls, resource management, and contracts management. 

NRR is the lead program office for the reactor oversight function and NSIR leads the incident 

response function. The program support offices include: NRR, NSIR, FSME, HR, OE, OGC, 

OI, FSME, REG I, REG II, REG III, REG IV, and RES. 

7.2. Approach 

The approach for assessing this evaluation area consisted of three key components as explained 

below. 

1. Review of relevant documentation and published material. We reviewed internal program 

documentation and conducted a brief literature review of published material related to financial 

management. The following internal documents were reviewed: 

• FY 2007 PAR 

• FY 2008 Performance Budget 

• FY 2007 Performance Budget 

• FY 2008 Budget Execution Report (as of September 30, 2008) 

70 

 
• FY 2007 Budget Execution Report (as of September 30, 2007) 

• FY 2006 Budget Execution Report (as of September 30, 2006) 

• RPS Funds Report (as of September 30, 2008) 

• RPS Funds Technical Assistance Report (as of September 30, 2008) 

• Advice of Allowances and Financial Plans 

• Office monthly budget execution reports 

• Office contract management reports 

• Advanced Procurement Plan User Guide and office submission 

• MD 4.1 Accounting Policy and Practices (September 9, 2005) 

• MD 4.2 Administrative Control of Funds (April 11, 2008) 

• MD 4.3 Financial Management Systems (July 7, 2005) 

• MD 4.4 Management Controls (May 18, 2004) 

• MD 4.6 License Fee Management Program (August 13, 2002) 

• MD 4.7 Policy and Practices Governing NRC Long-Range Planning, Budget Formulation, 

and Resource Management (October 1, 1989) 

• 2003 OMB PART Assessment of Reactor Inspection and Performance Assessment 

• 2007 OMB PART Assessment of Decommissioning and Low Level Waste 

• 2007 OMB PART Assessment of High-Level Waste Repository 

• FY 2008 Office budget submissions 

• Office operating plans 

• Office performance monitoring reports 

• Office operating-level reports 

• Office instructions, guidance and process documentation 

• FY 2008 - FY 2013 Strategic Plan 

• FY 2004 - FY 2009 Strategic Plan 

From the literature search, we identified a set of best practices and criteria to be used as 

standards for the assessment by reviewing published material, including: 

• GAO studies on financial management; 

• OMB PART Guidance 2008; 

• PMA initiative for improving financial management performance; and 

• U.S. Chief Financial Officers Council and metric tracking system financial management 

indicators. 

71 

 
2. Conduct staff interviews. We conducted interviews with stakeholders, constituents and 

program staff. We developed interview guides and conducted semi-structured, confidential 

interviews to obtain information related to financial management, including budget execution, 

funds control, internal controls, resource management, and contracts management. 

3. Analyze current performance. Based on the standards established from our literature review 

and research on best and leading practices, we assessed internal processes, performed financial 

analysis, and reviewed financial management indicators to compare the Program’s current 

performance against these criteria.

To evaluate the Program’s financial management, we assessed the Program on the following 

standards: 

• Funds are obligated in a timely manner, spent for the intended purpose, and accurately 

reported; and 

• Program uses strong financial management practices. 

7.3. Key Findings 

Standard 1: Funds are obligated in a timely manner, spent for the intended 

purpose, and accurately reported 

For this first evaluation standard, we based our assessment on OMB PART criteria which are 

supported by GAO studies, PMA criteria, and CFO Council standards. We performed our 

evaluation by reviewing program documentation, reviewing operating and financial plans, 

analyzing office budget execution reports, analyzing financial and operating performance 

reports, and conducting interviews with program officials. We assessed the extent to which the 

Program meets the specific criteria for this standard, including: 

• Funds are obligated consistently with the overall program plan and schedule of resource 

requirements; 

• Limited amount of unobligated funds remain at the end of the year; 

• Adequate procedures exist for reporting actual expenditures, comparing them against the 

intended use, and taking timely and appropriate action to correct single audit findings when 

funds are not spent as intended; and 

• Program awards are reported promptly and accurately. 

Based on our review and analysis, we found the Program is effective in obligating funds 

consistently with the financial and operating plans and achieving a limited amount of 

unobligated funds that remain at the end of the year. Below are our detailed findings for this 

evaluation standard. 

Finding: Funds are obligated consistently with the overall program plan and schedule of 

resource requirements 

72 

 
Program support offices, Division of Contracts in the Office of Administration, and OCFO have 

a role in obligating funds for the Program and ensuring obligations are consistent with the overall 

program plan. Once budget authority is received and the funds are apportioned by OMB, OCFO 

issues the Advice of Allowances and Financial Plan (AAFP), which provides authority to 

Allowance Holders to incur commitments and obligations. The AAFP enables the CFO to assign 

accountability to Allowance Holders, such as the Office Director of NRR and NSIR, to ensure 

that obligations and unobligated commitments incurred do not exceed an appropriation, 

apportionment, allowance, financial plan, footnote, or any other administrative subdivision. Any 

restrictions to the use of resources are noted in the footnotes of the AAFP and monitored by both 

financial management staff in PMDA as well as OCFO. Changes to the AAFP are tracked by 

OCFO in the AAFP Monthly Status Report as well as by the program office financial 

management staff. OCFO also monitors commitments, obligations, and expenditures for the 

agency and allowance holders on a monthly basis and reports the results in the Budget Execution 

Report (BER) each month and quarter. The BER is a tool that senior management and offices 

use to monitor, manage, and communicate financial performance against established goals and 

objectives. An example of how program offices track spending against plan is the monthly NRR 

Budget Execution Status Report. In NSIR, the PMDA staff manages the NSIR Financial Plan 

report that tracks commitments and obligations against plan. 

To ensure the schedule of obligations is consistent with the resource needs of the program plan, 

funds are incrementally obligated throughout the year, typically on a quarterly basis, when 

program requirements are better established. Obligations are informed by the Advanced 

Procurement Plan, which identifies the purpose, budget amount, and expected expenditure rate 

per quarter for each procurement action. Any reprogramming of funds that exceed established 

thresholds requires the approval of the Leadership Team prior to submitting an AAFP change 

request to OCFO. 

Finding: Limited amount of unobligated funds remain at the end of the year

To ensure that a limited amount of funds remain unobligated at the end of the fiscal year, the 

OCFO tracks financial performance on a number of financial management measures, including 

utilization of funds; commitments, obligations, and expenditures as a percentage of total 

financial plan; and number of months funding available. As reported in the year-end FY 2008 

BER, NRR and NSIR, the lead program offices for the RO and IR program, had obligated 100% 

and 92%, respectively, of their financial plan. Collectively, 98% of the financial plan for these 

two allowance holders was obligated at the end of FY 2008. 

Upon further review of the budget execution reports from OCFO and the program offices, we 

also found that some of the program support offices, including NRR and NSIR, did not meet the 

FY 2008 carryover target of 4 months, which is the number of months of available funding that 

remains at the end of the fiscal year for contract support and travel. Due to the extended period 

of the continuing resolution in recent years, the CFO has established a carryover target for FY 

2009 and beyond of no more than 6 months. Office operating plan measures will be adjusted to 

73 

 
reflect the 6-month standard. According to NRR’s FY 2008 Budget Execution Status Report, 

NRR/DIRS had 5.6 months of funding remaining at the end of the fiscal year. Program support 

staff explained that the remaining funds were necessary to fund inspections scheduled for 

November. For NSIR/DPR, the year-end carryover figure had not yet been reported by the 

office but it was expected to exceed the target. Although the BER for year-end FY 2008 

reported 11 months of funding available for NSIR overall, this may not be representative of 

DPR’s performance which is responsible for the IR program. The program staff in NSIR 

reported that efforts are underway to work with OCFO to improve performance in this area as 

well as other financial management measures. 

Finding: Adequate procedures exist for reporting actual expenditures, comparing them 

against the intended use, and taking timely and appropriate action to correct single audit 

findings when funds are not spent as intended 

Agency policy and procedures are documented in NRC Management Directive Volume 4 –

Financial Management. Program support offices also have process documentation for program 

staff responsible for budget execution and funds control to ensure compliance with reporting and 

intended use requirements. Agency procedures include assignment of funds control duties to a 

sufficient number of adequately trained program staff who are designated in writing. The staff 

responsible for fund certification is different from staff responsible for fund commitment and 

obligation, ensuring an appropriate check and balance in fund management. 

OCFO as well as the program offices monitor commitments, obligations, and expenditures on a 

monthly basis and reports results in budget execution reports. An example of how program 

support offices track actual spending against plan is NRR’s monthly Budget Execution Status 

Report and NSIR’s Financial Plan report that tracks commitments and obligations against plan. 

Also, in NRR, the Contracts Management Team (CMT) provides monitoring and reporting of 

commitments, obligations, and expenditures against budget as well as the Advanced 

Procurement Plan for each NRR contract. CMT staff also review invoices prior to payment to 

ensure services are provided as intended pursuant to the contract. 

Each year, to ensure the agency’s compliance with FMFIA, the program offices are required to 

certify that internal controls are achieving their intended results, resources are being used 

consistently with the agency mission and applicable laws and regulations, and resources are 

protected from waste, fraud, and abuse. In the event that funds are not spent as intended 

resulting in an audit finding, OCFO leads the effort to work with the Program on corrective 

action. As reported in the most recent PAR and interviews with the financial management staff 

of the program office, the Program has not been subject to any singe audit findings regarding 

funds not spent as intended. 

Finding: Program awards are reported promptly and accurately 

Based on interviews with the contracts management staff of the program office, we found that 

program awards are reported promptly and accurately. For example, the Contracts Management 

74 

 
Team in NRR, the lead office for RO, reports Department of Energy awards within 30-45 days 

from initial receipt of requirement from program staff. Division of Contracts in the Office of 

Administration is responsible for commercial contract awards. 

Standard 2: Program uses strong financial management practices 

For this evaluation standard, we based our assessment on OMB PART criteria which are 

supported by GAO studies, PMA criteria, and CFO Council standards. We performed our 

evaluation by reviewing program documentation, reviewing operating and financial plans, 

analyzing office budget execution reports, analyzing financial and operating performance 

reports, and conducting interviews with program officials. We assessed the extent to which the 

Program meets the specific criteria for this standard, including: 

• Procedures are in place to ensure that payments are made properly for the intended purpose 

to minimize erroneous payments; 

• Compliance with the Anti-Deficiency Act; 

• Clean audit opinion and no material internal control weaknesses reported by auditors; and 

• Financial management systems meet statutory requirements. 

The agency’s strong financial management practices are evidenced by its high performance on 

the Financial Management Indicators reported in the Metric Tracking System (MTS) established 

by the CFO Council. The MTS is a performance measurement system that captures key financial 

management indicators across the federal government. From May to July 2008, the agency 

achieved a Green rating on all 9 metrics that are tracked. Less than half of the 25 agencies 

tracked in MTS achieved all Green ratings. 

Based on our review and analysis, we found the Program is effective in its financial management 

practices. Below are our detailed findings for this evaluation standard. 

Finding: Procedures are in place to ensure that payments are made properly for the intended 

purpose to minimize erroneous payments 

According to the FY 2007 PAR, improper payments continue to be at low risk for the Agency. 

NRC continues to evaluate its internal controls to guard against improper payments and monitors 

and reports on improper payments within its programs. Based on the results of the annual risk 

assessment conducted in FY 2007, the number of and amount of improper payments fell below 

what is considered to be a significant risk by OMB and subject to the external reporting 

requirement, which is 2.5% of program payments and $10 million for high risk. 

Financial and contract management training is provided to appropriate staff based on their 

responsibilities. Staff responsible for the Program's contract and project management activities 

must attend acquisition training. Also, on a semi-annual basis, the Agency conducts a financial 

management seminar that provides an overview of the Federal and agency budget process, 

financial management policies and procedures, and applicable laws and regulations. 

75 

 
Finding: The Program is not in violation of the Anti-Deficiency Act 

As reported in the most recent PAR and interviews with the financial management staff of the 

program office, the Program is not and has not been in violation of the Anti-Deficiency Act. 

Finding: The Program received a clean audit opinion and is free of material internal control 

weaknesses as reported by the auditors 

NRC received a clean audit opinion on its FY 2007 financial statements and the auditors did not 

identify any material internal control weakness affecting the Program specifically. The auditors 

found that the Agency had effective internal control over financial reporting and was compliant 

with laws and regulations; the only exception was one material weakness related to information 

systems security controls, which also represents a substantial noncompliance with the Federal 

financial management system requirements under FFMIA. This material weakness is further 

explained in the next finding below. 

In FY 2007 as required by the FMFIA, the Agency provided a qualified statement of assurance 

that the internal controls were operating effectively and compliant with applicable laws and 

regulations in accordance with OMB Circular A-123, Management’s Responsibility for Internal 

Control. No material weaknesses were found in the design or operation of the internal controls, 

except for the one material weakness explained below. 

Finding: Financial management systems meet statutory requirements with the exception of 

one non-compliance that is agency-wide and not program specific 

The Agency reported in the FY 2007 PAR and in the FMFIA qualified statement of assurance 

that NRC is in compliance with the FMFIA with the exception of one material weakness, which 

pertains to the agency’s overall information systems security controls related to NRC’s 

implementation of the Federal Information Security Management Act (FISMA). The FISMA 

report identified two significant deficiencies related to a lack of contingency plan testing for 

information security systems, and a lack of certification and accreditation for most of the 

agency’s major information systems. The Agency has an initiative underway to coordinate with 

the Office of Information Services and the EDO to address this material weakness by ensuring 

that the security vulnerabilities of the general support systems and the financial management 

systems are resolved. 

The Agency is also in the process of integrating and modernizing its financial systems to 

enhance further controls, reporting, and decision-making. As explained in section 4, Budget 

Performance Integration, NRC is implementing a new core financial management system that 

combines the functionality of the core accounting, license fee billing, cost accounting, 

allotment/allowance financial plan and the capitalized property systems. Some key benefits 

sought from this new system include improved access, timeliness, efficiency, and overall quality 

in financial and performance reporting. The agency’s time and labor system is also undergoing a 

76 

 
major upgrade which is expected to improve the Program’s capability to collect information for 

fee billing and cost accounting, and provide a wider range of management reports. 

7.4. Recommendations 

Improve program performance on corporate measures related to financial management, 

specifically carryover target 

Over the past year, the Agency has made significant efforts to improve and standardize the 

corporate performance measures, which include measures for financial management. The 

OEDO tracks office performance on these corporate measures on a quarterly basis. Some of the 

program support offices for the Program are not meeting the Green targets for the measures 

related to financial management. Program support offices need to improve their performance for 

the measures that are out of standard, especially the carryover measure, which has a target of no 

more than 6 months starting in FY 2009 and beyond. Other out-of-standard measures include: 

Time and Labor (T&L) certification timeliness, T&L correction rate,’ purchase order/purchase 

card invoice certification timeliness, contract funding commitment timeliness, and contract 

invoice timeliness. For any corporate performance measure that does not have an appropriate 

target for Red/Yellow/Green status, the program support office should coordinate with the 

OCFO and/or OEDO to revise the targets so that they are meaningful and representative of the 

desired performance levels. 

As discussed above under the financial management systems criteria, our assessment does not 

include a separate recommendation for systems improvements since the Agency already has a 

number of initiatives underway. 

77 

 
8. Enforcement and Allegations 

8.1. Objective / Key Activities 

The enforcement process begins when potential violations meeting certain conditions are 

identified in the inspection process. Most deficiencies identified in the inspection process are 

handled through the reactor oversight process (ROP). Depending on the nature of the deficiency, 

for example, if the deficiency was willful, the deficiency may be handled through enforcement 

instead of the ROP. Enforcement is used with violations that may impact the NRC’s ability for 

oversight such as those associated with deliberate misconduct. Violations are designated a 

severity level (I-IV). These correspond roughly to Red, Yellow, White, and Green findings used 

with the ROP. 

The NRC considers many factors when deciding whether or not to issue a civil penalty to a 

licensee and the appropriate amount of the civil penalty. As documented in the NRC 

Enforcement Policy, these factors include the licensees’ enforcement history, whether or not the 

licensee identified the violation, and what corrective actions the licensee has taken. Regulatory 

tools such as the risk of agency intervention, ensuing stakeholder visibility, and civil penalties 

serve as deterrents. 

The NRC has an allegation handling process in place with timeliness metrics and administrative 

mechanisms to gauge performance including audits via the Agency Allegations Advisor (AAA), 

involvement through an Allegations Coordinator (AC), and monthly calls involving HQ and the 

regional offices. When an allegation is submitted, the process is activated absent of definitional 

threshold. In other words, all allegations are immediately considered for processing and closure 

regardless of significance or criticality. 

Furthermore, the allegation process accommodates sources originating from licensees including 

employees, vendors, contractors, and external stakeholders (public, etc.) but also routes 

allegations as necessary to other agencies (e.g., FEMA, OSHA). Wrongdoing allegations 

identified by NRC employees and contractors are also routed through a separate channel (e.g., 

OIG, DOJ). 

8.2. Approach 

The approach for assessing this evaluation area consisted of three key components as explained 

below. 

1. Review of relevant documentation and published material. We reviewed internal program 

documentation and conducted a brief literature review of published material related to 

enforcement and risk management. The following documents were reviewed: 

• Enforcement Program Annual Reports 

• Enforcement Manual, Sept 28, 2006 Revision #5 

78 

 
• Enforcement Guidance Memoranda 

• Enforcement Policy, January 14, 2005 

• MD 8.8 Management of Allegations (February 4, 1999) 

• MD 9.19 O/F, Office of Enforcement (May 9, 1989) 

• NUREG 0800, NRC Standard Review Plan 

• NUREG/BR0240, Rev. 3, Reporting Safety Concerns to the NRC, April 2005 brochure 

• FY 2007 PAR Highlights 

• OIG-08-A-17, Audit of NRC’s Enforcement Program, September 26, 2008 

• OIG, NRC’s Response to Security-Related Concerns at Peach Bottom Atomic Power Plant, 

August 22, 2008 

• OIG: 96-01S, Office of the Inspector General Event Inquiry: Implementation of 

Recommendations to Improve NRC's Program for Protecting Allegers against Retaliation 

• Backgrounder – Allegation Process 

• Allegation Program – 2007 Annual Performance Report, September 29, 2008 

• Technical Quality of Incident and Allegation Activities, June 13, 2007 

• Reviewing the Common Performance Indicator, Technical Quality of Incident and Allegation 

Activities Appendix B Allegation Casework Review Summary Sheet 

• NEI: 04-02, Guidance for Implementing a Risk-Informed, Performance-Based Fire 

Protection Program Under 10 CFR 50.48(c) 

• National Fire Protection Agency: (NFPA) 805, Performance-Based Standard for Fire 

Protection for Light Water Reactor Electric Generating Plants (2006) 

• Professional Reactor Operator Society Web site 

• Atomic Energy Act of 1954, Chapter 18 Enforcement 

• CRS Report September 17, 2004 

• Federal Register/Volume 73, No.176, Wednesday, September 10, 2008, Notices 52705 

• GAO Reports October 3, 2007; March 10, 2003, September, 2003, respectively 

• Buehler & Freeman: The Risk Revolution – Owning the Right Risks, September, 2008 

2. Conduct staff interviews. We conducted interviews with customers, stakeholders, and 

program staff. We developed interview guides and conducted semi-structured, confidential 

interviews to obtain information related to enforcement and allegations. 

3. Analyze current performance. Based on the standards established from our literature review 

and research on best and leading practices, we assessed internal processes and performed 

operational analysis to compare the Program’s current performance against these criteria.

79 

 
To evaluate the Program’s enforcement and allegations function, we assessed the Program on the 

following standards: 

• Resolves enforcement actions in a timely manner; 

• Resolves allegations in a timely manner; and 

• Ensures quality and consistency in its resolution of enforcement. 

8.3. Findings 

Standard 1: Resolves enforcement actions in a timely manner 

To evaluate this standard we interviewed program staff and reviewed relevant program and other 

internal documentation. 

Finding: The Program is effective in resolving enforcement actions in a timely manner 

The Program sets targets for non-investigation and investigation cases. The table below sets out 

the Program’s statistics in meeting its timeliness objectives for calendar years 2004 - 2007. For 

calendar year 2007, the Program missed its goal for investigations due to the complexity of 

several cases and the difficulty of the issues involved in the reactor and materials areas. In 

addition alternative dispute resolution (ADR) was used to settle some disputes. 

Table 10 - Timeliness of Resolving Enforcement Actions 

Initial 

escalated

enforcement 

action

Initial 

escalated

enforcement 

action

Administrative 

goals 

Administrative 

goals

Year

Noninvestigation 

cases within 180 

calendar days

Investigation 

cases are made 

within 360 

process days

Noninvestigation 

cases within 

avg. 120 

calendar days

Investigation 

cases within 

avg. 180 

process days

2007 100% 100% 100% Not met

2006 100% 100% 100% 100%

2005 100% 100% 100% 100%

2004 100% 100% 100% 100%

Source: Enforcement Program Annual Reports 

Standard 2: Resolves allegations in a timely manner 

To evaluate this standard we interviewed program staff and reviewed relevant documentation 

including the Enforcement Annual Report (CY 2007). 

Finding: Allegations are processed and closed on a timely basis 

80 

 
The Program requires an initial Allegation Review Board (ARB) to be held within 30 days of the 

receipt of an allegation in 100 percent of the cases. 568 of the 569 initial ARBs held agencywide in CY 2007 met this goal. With regard to closing allegations that have technical concerns, 

but do not have wrongdoing, the Program met all those targets. The following table contains 

statistics on timeliness of allegation resolution for NRR, NSIR, and the four Regions. Given the 

actual performance relative to the targets, the targets do not seem sufficiently ambitious. The 

Program should consider also tracking average closure time and track trends over time. 

Table 11 - Timeliness of Allegations Resolution 

Time to Close 150 days 180 days 360 days

Target Closure 70% 90% 100% 

Actual Closure 96% 100% 100% 

Source: 2007 Allegation Program Annual Performance Report 

Standard 3: Ensures quality and consistency in its resolution of 

enforcement 

To evaluate this standard we interviewed program staff and reviewed relevant documentation 

including the Enforcement Annual Report (CY 2007) and the OIG Audit of NRC’s Enforcement 

Program. 

Finding: The Program needs improvement in resolving enforcement with quality and 

consistency 

The OIG report states that the enforcement program lacks clear and comprehensive guidance 

needed to ensure consistent program implementation. As a result, there were inconsistencies in 

the way enforcement was carried out among the four Regions. In addition, the OIG report states 

that enforcement decisions may not be based on complete and reliable data. The OIG report 

indicated that “In general, Agency officials agreed with the Report’s findings.” We reviewed the 

methodology used in the OIG study, reviewed program documentation, and interviewed program 

staff. We did not audit the underlying information sources, but based on our review, we agree 

with its findings. In addition, we make two observations: 

• The OIG study was agency-wide and therefore was broader in scope than the reactor 

oversight and incident response program. The finding regarding incomplete and unreliable 

data was in part due to lack of a standardized enforcement tracking system. However, the 

Reactor Program System (RPS) is used to support enforcement related to reactors in the 

Regions, so this finding may apply less to reactor oversight-related enforcement. 

• The OIG report findings applied primarily to non-escalated enforcement. 

81 

 
Enforcement standards for fire protection are an area of concern expressed by some stakeholders. 

The NRC has modified its fire protection regulations to allow licensees to adopt, on a voluntary 

basis, National Fire Protection Association (NFPA) Standard 805, Performance-Based Standard 

for Fire Protection for Light-Water Reactor Electric Generating Plants (NFPA 805), in lieu of 

their existing prescriptive fire protection licensing basis. This initiative is part of a broader NRC 

effort to incorporate risk information within regulations to enhance safety. Forty-seven licensees 

have indicated they intend to make the transition. Others have begun developing the requisite 

fire probabilistic risk assessments (PRAs). 

The NRC recently revised the “Interim Enforcement Policy Regarding Enforcement Discretion 

for Certain Fire Protection Issues” which will allow a licensee the option to request an extended 

enforcement discretion period if they are pursuing transition to NFPA 805. The Program also 

recently clarified some related stakeholder concerns, including one regarding whether the 

enforcement discretion period would end if an amendment request was rejected if the licensee 

had sufficient corrective action in place. Some stakeholders in the past have indicated concerns 

that additional enforcement discretion will further delay getting licensees into compliance. The 

recent clarification provides additional specificity on the window for enforcement discretion. 

Some stakeholders have expressed concerns that the revised policy is problematic because there 

are limited industry resources that will make it difficult to complete license amendment requests 

in an orderly fashion. 

8.4. Recommendations 

Add additional level of review in non-escalated enforcement actions to review outcomes 

and procedures to help ensure that standardized procedures are used 

The IG found that the enforcement program lacks clear and comprehensive guidance needed to 

ensure consistent program implementation. As a result, there were inconsistencies in the way 

enforcement was carried out among the Regions. The Office of Enforcement is considering 

calling for adding an additional level of review, e.g., review by a Branch Chief before 

enforcement actions are submitted. We agree with this approach. In addition, we recommend 

clarifying the guidance with additional examples and adding additional training for Branch 

Chiefs on standard criteria to apply in enforcement decisions. We also recommend that a 

Community of Practice be established in the Agency’s Knowledge Management System for staff 

involved in enforcement to make use of this system as a way of sharing practices. 

Standardize procedures for tracking of enforcement cases 

The IG report states that enforcement decisions may not be based on complete and reliable data. 

This study was agency-wide and therefore was broader in scope than the reactor oversight and 

incident response program. The finding regarding incomplete and unreliable data was in part 

due to lack of a standardized enforcement tracking system. However, since the RPS is used to 

82 

 
support enforcement related to reactors in the Regions, this finding may apply less to reactor 

oversight-related enforcement. We recommend that guidance be updated with additional 

examples to clarify requirements and data quality standards for information to be used in 

enforcement cases. 

83 

 
84 

 

9. Staff Development 

9.1. Objectives / Key Activities 

In recent years the NRC has gone through a significant transition. There have been increasing 

retirements, mid-career transfers, and hiring of younger staff. NRC projects that its workforce 

size will need to grow from about 3,100 employees in early fiscal year 2006 to nearly 4,000 

employees by 2010 to accommodate workload demands associated with new nuclear reactors. 

An aging workforce in the Agency has meant increasing retirements, and a rise in mid-career 

transfers. To replace retiring employees and expand its workforce, NRC must hire 300 to 400 

new employees per year through at least 2010.2

A related challenge the Agency is facing is the number of staff changing jobs internally. In 

2008, approximately 930 staff changed jobs. Some of this is the result of realignment to support 

new reactors. The result is many staff members have been in their current positions for a short 

period of time. 

These trends pose significant challenges to the Program. Last year, turnover of resident 

inspectors, a key component of the Program’s staff, was 46%, up from 20% in 2006. A 

significant portion of the headquarters management staff we interviewed had been in their 

positions for less than one year. 

To address these challenges, the agency stepped up its recruiting process, increased training, and 

launched a knowledge management initiative. There have been successes on these fronts. Over 

the last two years, the Agency has exceeded it recruiting goals. This has been aided by the 

NRC’s reputation as a good place to work. In addition the NRC has staff development initiatives 

in place. Among these are the Nuclear Safety Professional Development Program (NSPDP), a 

program that involves recent graduates at the undergraduate or graduate level and strong 

academic records in health physics, earth sciences, or engineering. New employees in this 

program participate in on-the-job training, formal classroom training, and rotational assignments. 

There is also the Student Career Experience Program. In this program, college students pursuing 

degrees in science, engineering, and other disciplines related to the NRC mission have the 

opportunity to alternate periods of academic study and work experience, or perform the two in 

parallel.. 

NRC has been cited as the government agency with the highest employee satisfaction ratings in a 

study conducted by the Office of Personnel Management (OPM). In addition, in recent years 

there has been an increase in enrollment in nuclear engineering programs. The training 

initiatives have been important and NRC staff indicates high levels of satisfaction with training 

programs. In 2006, the Agency launched a knowledge management initiative to help address the 

impact of knowledge loss resulting from the transition the Agency is undergoing. 

 

2

 Human Capital Retirements and Anticipated New Reactor Applications Will Challenge NRC’s Workforce, 

GAO, January 2007 
9.2. Approach 

To evaluate the Program’s ability to maintain a quality workforce, our review consisted of 

assessing whether the Program is meeting the following standards: 

• Maintains the appropriate staffing levels to execute plan; and 

• Delivers staff training effectively. 

9.3. Findings 

Standard 1: The Program maintains the appropriate staffing levels to 

execute plan 

For this standard, we evaluated the Program’s ability to maintain adequate numbers of 

experienced resident inspectors. We conducted staff interviews, reviewed internal survey 

information, and reviewed resident inspector demographic data. 

Finding: The Program is maintaining a stable, experienced resident inspector base, but recent 

turnover is resulting in coverage challenges 

A key objective is to maintain a stable and experienced base of resident inspectors. A 

recommendation developed after the Davis-Bessie incident was to establish a site staffing metric. 

The Davis-Bessie plant was shut down in 2002 after it was discovered that boric acid had nearly 

eaten through a 6 1⁄2-inch reactor pressure vessel head. The site staffing metric reflects the 

percentage of days a site is staffed by an inspector who meets certain requirements. The criteria 

defining inspectors who meet requirements are set out in IMC 0307, ROP Self-Assessment:

NOTE: Inspectors assigned to the site permanently or through a rotation with a 

minimum duration of 6 weeks shall be counted. Inspectors on 6 week or longer 

rotational assignments will be identified as such. Inspectors assigned to the site for 

less than 6 weeks will not be counted, but should be indicated as such. Additionally, 

the regions shall indicate sites where permanently assigned resident or senior resident 

inspectors are away from the site for an extended period of time (one continuous time 

period which is greater than 6 weeks). Only inspectors, who have attained at least a 

basic inspector certification status, as defined by Appendix A to Inspection Manual 

Chapter 1245, shall be counted. Data will indicate the number of days a qualified 

resident and senior resident inspector are permanently assigned to the site during the 

85 

 
year divided by the number of days in the year. Number of days spent on training; 

meetings away from the site; participation in team inspections; leave; or other 

temporary duties (e.g. acting for branch chiefs in his/her absence) will not be counted 

against the metric unless the absence exceeds 6 continuous weeks. 

In 2007, nine sites were below the 90% target in part due to turnover. The lowest site was 74%, 

but the overall average was 96% (Resident Inspector Demographics, 2007). Inspector turnover 

was a significant contributor to the site staffing challenges. It was also noted in 2007 that the 

turnover rates for both resident inspectors (RI) and senior resident inspectors (SRI) had been 

increasing over the past few years. Even though a significant portion (40 percent) of the RIs 

were promoted to an SRI position, an equal share of the RIs were either promoted or reassigned 

outside the RI program, as indicated by the “combined” line in the figure 7 below. “Combined” 

is RI turnover plus SRI turnover, less RIs promoted to SRIs. 

This turnover has resulted in decreases in the median number of years an inspector has been 

either a resident or senior resident. Similarly the median time at a current site has decreased 

slightly. The national median value for total resident time decreased 20 percent from 2006 to 

2007. However, despite the fact that total resident time has gone down, overall relevant 

experience remains stable. The national data from 2003 to 2007 shows that the resident 

inspectors have maintained an average of 10 years relevant non-NRC experience and 4 years of 

NRC experience. This demonstrates stability in the overall level of experience despite the high 

turnover rates in recent years. The high turnover, however, has presented challenges in 

maintaining continuity of oversight. 

86 

 
Figure 7 - Resident Inspector Turnover 

Source: NRC and FocalPoint analysis 

Filling the gaps required extra management and staff time and resulted in some sites having 

lower staffing levels than the ideal. While the shortages were covered, they put some strain on 

staff and management. 

Standard 2: Delivers training to staff effectively 

Finding: The Program has launched knowledge management initiatives effectively 

As discussed above, the Agency is undergoing a transition, including increasing retirement, midcareer transfers, and recruitment of a new, younger workforce, which is resulting in a loss of 

knowledge. The loss in knowledge has the potential for decreases in efficiency, increased risk of 

errors, slower delivery of work products, and overall decrease in effectiveness. Several 

managers we interviewed acknowledged that this transition was having an impact on efficiency 

and effectiveness. 

As part of its effort to address these rising challenges, the Agency began a knowledge 

management (KM) initiative in 2006. In our review of the knowledge sharing effort we assessed 

the extent to which: 

• There was a clear plan established and is being implemented; 

• Senior management support is visible; 

• There is effective management/coordination of knowledge sharing function; and 

• Information and knowledge are valuable commodities. 

87 

 
When the initiative was launched, a plan was established to provide staff tools in four categories 

of interest, including: 

• Human resources, processes, policies, and procedures: including succession planning, 

strategic workforce planning, hiring incentives, double encumbering, retention incentives, 

and rehired annuitants. 

• Knowledge sharing practices: including documentation, mentoring communities of 

practice, brown bag lunch discussions, employee-led seminars, and videotaping of seminars 

and subject-matter interviews. 

• Knowledge recovery practices: including guest lecturers, reemployed annuitants, 

effectiveness reviews, and procedures and documentation. 

• Information technology applications: including collaborative workflow software, etraining, videoconference, updating and enhancing ADAMS, and NRC knowledge center. 

The Agency has a clear plan and is implementing accordingly. A steering committee is active in 

identifying ongoing office- and regional-level efforts to capture knowledge in each of these 

areas. 

With regard to management support, for knowledge management efforts to succeed, senior 

management must lead by example. Based on interviews conducted, we found that program 

senior management has been visible and active supporters of the initiative. 

The Program is also demonstrating effective management and coordination of the KM function. 

The Program produces a quarterly KM update that provides information on activities in each of 

the four categories. For each activity it lists successes, challenges, other comments, and a point 

of contact. We also reviewed the knowledge center application. It contains a set of forums, 

called “Communities of Practice.” The number of postings shows the system is being used, with 

postings in over 600 topics being discussed overall. 

The final criterion applies to the importance of information and knowledge. Based on our 

observations, we found that knowledge and information are valued commodities given the nature 

of the Program’s work. Some interviewees expressed concerns about a cultural shift in the NRC. 

The concern expressed was that in the past there was a deeper commitment to professional 

development to obtain deep expertise in a given functional area. This meant the NRC could 

bring significant subject-matter expertise to solving tough problems. Those experts were also a 

source of knowledge sharing for other staff. The concern was that newer staff seemed less 

motivated to develop that same deep expertise, preferring instead to gain a broader range of 

experience in different areas. 

Overall, we believe the knowledge management effort is on track and with continued effort and 

outreach will help the Program deal with the transition challenges. 

88 

 
Finding: The Program delivers high quality training, but the nature of its challenges requires 

improvements in evaluating the effectiveness of training 

The critical nature of the NRC’s work and the high turnover means the Program has a 

particularly high need for quality training for new staff. In addition, as the Program makes 

improvements or changes policies and procedures, it requires high quality training for those 

employees. All resident inspectors, for example, go through initial training as set out in IMC 

1245, Appendix A and B. Additional initial training is set out in Appendix C and varies by 

specialty. Over the last 2 years, training was developed to teach staff about new inspection 

procedures, such as the recently implemented safety culture assessment and the new Unplanned 

Scrams with Complication performance indicator. The Program has a number of efforts in place 

to evaluate its training programs. Among these are: 

• Feedback loops; 

• Internal surveys (every 2 years); and 

• Self-assessments. 

While training at the NRC is considered to be the best in the federal government according to a 

recent OPM study, there are requirements for more effective training in some specialized areas, 

such as safety culture and the significance determination process, according to feedback 

provided in an internal survey. The OIG initiated a review of the NRC’s Training and 

Development Program. Although their findings applied to the entire Agency, not just the 

Program, one finding was that access to training is difficult and there was insufficient ability to 

measure the effectiveness of training in supporting the wide range of skills needed for the 

programs currently in place. 

Currently post-class surveys are conducted immediately after the class, but they are not 

conducted systematically after a period of time when the employee can better assess the impact 

on his or her job performance. There should be follow-up surveys sent to attendees immediately 

after a class and then again 3 months later. The Federal Aviation Administration (FAA) for 

example also sends surveys to inspectors and their supervisors 90 to 180 days after course 

completion to obtain their perspectives on whether the course was needed and the extent to 

which the inspector is applying new skills and knowledge to the job. FAA reports that since the 

inception of post-course surveys, the return rate from inspectors and supervisors has ranged from 

49 to 50 percent. The FAA post-course survey results from the six most-highly-attended 

technical courses in the last 2 years reflected generally positive responses (GAO 2005). 

9.4. Recommendations 

Analyze root cause of increased turnover and take actions to address 

High turnover is making it difficult to meet site staffing objectives. The Program should analyze 

the root cause of the increased turnover by conducting exit interviews of staff leaving, 

89 

 
conducting a market study on salaries and benefits, and conducting an employee satisfaction 

survey. While some of the turnover is structural, in sympathy with the overall challenges the 

Agency is having with turnover, the results of this analysis should be applied to make the work 

context changes available to the Program. 

Conduct follow-up surveys three months after training 

Currently post-class surveys are conducted immediately after the class, but they are not 

conducted systematically after a period of time when the employee can better assess the impact 

on his or her job performance. The Program should also conduct follow-up surveys three months 

after the course is taken. Other agencies have used such follow-up surveys in similar contexts 

and have had good results. For example, the FAA also sends surveys to inspectors and their 

supervisors 90 to 180 days after course completion to obtain their perspectives on whether the 

course was needed and the extent to which the inspector is applying new skills and knowledge to 

the job. 

90 

 
Appendix 

91 

 
A. Bibliography 

American Nuclear Society. (1998). ANSI/ANS-3.8.7-1998: Criteria for Planning, Development, 

Conduct, and Evaluation of Drills and Exercises for Emergency Preparedness. Available 

from http://www.new.ans.org/store/i_240230/r_a 

Atomic Energy Act of 1954. (1954, August 30). Public Law 83-703, 68 Stat. 919. Available from 

http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr0980/ml022200075-

vol1.pdf#pagemode=bookmarks&page=14 

Barrett, R. J. (2005, November 18). NRC Standards for Nuclear Regulation. Presentation to 

FERC Technical Conference, Washington, DC. 

Beck, R., & Davis, M. President’s Management Agenda: Performance Improvement Initiative 

(Previously Budget and Performance Integration). Available at 

http://www.usbr.gov/doiconference/presentations/Providing_Increasing_Programmatic_Per

formance.ppt 

Behrens, C., & Holt, M. (2004, September 17). CRS Report for Congress: Nuclear Power 

Plants: Vulnerability to Terrorist Attack. Order Code RS21131. Available at 

http://digital.library.unt.edu/govdocs/crs/permalink/meta-crs-10058:1 

Buehler, K., Freeman, A., & Hulme, R. (2008, September). The Risk Revolution – Owning the 

Right Risks. Harvard Business Review. Available at 

http://hbr.harvardbusiness.org/2008/09/owning-the-right-risks/ar/1 

Crichton, M. T., & Flin, R. (2004, May). Identifying and Training Non-Technical Skills of 

Nuclear Emergency Response Teams. Annals of Nuclear Energy, 31, 1317–1330. 

Available from www.sciencedirect.com 

Federal Deposit Insurance Corporation. (2008). 2008–2013 Strategic Plan. Available at 

http://www.fdic.gov/about/strategic/strategic/index.html 

Federal Deposit Insurance Corporation. (2008, February 15). 2007 Annual Report (also Referred 

to as the Performance and Accountability Report). Available at 

http://www.fdic.gov/about/strategic/report/2007annualreport/intro.pdf 

Federal Emergency Management Agency. National Incident Management System Resource 

Center. Available at http://www.fema.gov/emergency/nims/ 

Feinstein, J. S. (1989, February). The Safety Regulation of U.S. Nuclear Power Plants: 

Violations, Inspections, and Abnormal Occurrences. The Journal of Political Economy, 

92 

 
97(1), 115–154. Chicago: The University of Chicago Press. Retrieved October 19, 2008, 

from http://www.jstor.org/stable/1831057

Ford, J. K., & Schmidt, A. M. (2000, June 28). Emergency Response Training: Strategies for 

Enhancing Real-World Performance. Journal of Hazardous Materials 75, 195–215. 

Gunter, P., Warren, J., & Lochbaum, D. (2008, October). Fire When Not Ready. Available at 

http://www.ucsusa.org/assets/documents/nuclear_power/Fire-When-Not-Ready.pdf 

Hoge, J., & Martin, E. (2006, June). Linking Accounting and Budget Data: A Discourse. Public 

Budgeting & Finance, 26(2), 121–142. Available from 

http://www3.interscience.wiley.com/journal/118728554/abstract 

Iftikhar, Z., Eriksson, I. V., & Dickson, G. W. (2003). Developing an Instrument for Knowledge 

Management Project Evaluation. Electronic Journal of Knowledge Management, 1(1), 55–

62. 

International Federation of Accountants, Professional Accountants in Business Committee. 

(2008, June 23). Costing to Drive Organizational Performance. New York: Author. 

Available from http://www.ifac.org/guidance/EXD-Details.php?EDID=0114 

Jones, C. G. (2000, August). The U.S. Nuclear Regulatory Commission and How It Works

(NUREG/BR-0256). Washington, DC: U.S. Nuclear Regulatory Commission, Office of 

Nuclear Material Safety and Safeguards. Available at http://www.nrc.gov/reading-rm/doccollections/nuregs/brochures/br0256/ml031400642.pdf

Kadak, A. C., & Matsuo, T. (2007). The Nuclear Industry’s Transition to Risk-Informed 

Regulation and Operation in the United States. Reliability Engineering and System Safety, 

92, 609–618. 

Kadambi, N. P. (2005, January). Performance-Based (Risk-Informed) Regulation: A Regulatory 

Perspective. Nuclear Technology, 149, 110–121. 

Klien, K. R., Brandenburg, D. C., Atas, J. G., & Maher, A. (2005, May-June). The Use of 

Trained Observers as an Evaluation Tool for a Multi-Hospital Bioterrorism Exercise. 

Prehospital and Disaster Medicine, 20(3). Available at http://pdm.medicine.wisc.edu/20-

3%20PDFs/Klein.pdf 

Lochbaum, D. (2006, November 22). How Palo Verde Made the NRC’s Naughty List.

Washington, DC: Union of Concerned Scientists. Available at 

http://ucsusa.wsm.ga3.org/clean_energy/nuclear_safety/how-palo-verde-made-thenrcs.html 

Marks, L. K., & Potter, M. (2004, July-August). Drilling for Results: The Quest for Objective 

Exercise Evaluations. Homeland First Response, 2(4). Available at 

93 

 
http://www.altarum.org/files/pub_resources/04_studies_papers_hsd_drilling_for_results.pd

f 

McCauley, C. D., Ruderman, M. N., Ohlott, P. J., & Morrow, J. E. (1994). Assessing the 

Developmental Components of Managerial Jobs. Journal of Applied Psychology, 37, 46–

67. 

McDaniel, M. A., & Schlager, M. S. (1990). Discovery Learning and Transfer of Problem 

Solving Skills. Cognition and Instruction, 7(2), 129–159. Hillsdale, NJ: Lawrence Erlbaum 

Associates, Inc. 

Mihm, J. C. (2001, September). Implementing GPRA: Progress and Challenges. In D. W. 

Forsythe (Ed.), Quicker, Better, Cheaper: Managing Performance in American 

Government (pp. 101–112). Albany: Rockefeller Institute Press. 

National Fire Protection Association. NFPA 1600: Standard on Disaster/Emergency 

Management and Business Continuity Programs. Available from 

http://www.nfpa.org/index.asp 

National Fire Protection Association. NFPA 805: Performance-Based Standard for Fire 

Protection for Light Water Reactor Electric Generating Plants. Available from 

http://www.nfpa.org/index.asp 

Norcross, E., & Adamson, J. (2007, July 30). An Analysis of the Office of Management and 

Budget’s Program Assessment Rating Tool (PART) for Fiscal Year 2008. Arlington, VA: 

George Mason University Mercatus Center Working Paper 

Nuclear Energy Institute. Key Issues: Electricity Supply. Available at 

http://www.nei.org/keyissues/reliableandaffordableenergy/electricitysupply/ 

Nuclear Energy Institute. NEI 04-02: Guidance for Implementing a Risk-Informed, PerformanceBased Fire Protection Program under 10 CFR 50.48(c). Available from 

http://www.nei.org/ 

Powers, J., Jamison, J, & Martin, G. (2008). Using Evaluation Criteria to Organize the 

Planning, Conduct, and Oversight of Readiness Assurance Activities. Available at 

http://orise.orau.gov/emi/events/recent/2008/presentations/WK1_Powers_Jamison.ppt 

President’s Management Agenda. PMA Criteria for Getting to Green on Budget Performance 

Integration. Available at http://www.whitehouse.gov/omb/circulars/a11/2002/S225.pdf 

President’s Management Agenda. PMA Initiative for Improving Financial Management 

Performance. Available at http://georgewbushwhitehouse.archives.gov/results/agenda/financialperformance.html

94 

 
Professional Reactor Operator Society Web Site. Enforcement News. Available at 

http://www.nucpros.com/index.php?q=search/node/enforcement 

Skelly, T. (2006). Improving Integration of Performance Information in Congressional Budget 

Justifications. Available at http://georgewbushwhitehouse.archives.gov/results/agenda/q406_skelly.ppt 

Sorensen, J. N. (2002). Safety Culture: A Survey of the State-of-the-Art. Reliability Engineering 

and System Safety, 76, 189–204. 

U.S. Chief Financial Officers Council. Metric Tracking System Financial Management 

Indicators. Available from http://www.fido.gov/mts/cfo/public/200812/Indicators-200812-

00.htm 

U.S. Department of Homeland Security. Corrective Action Program System Overview. Available 

at https://hseep.dhs.gov/support/CAPSOverviewFAQs.pdf 

U.S. Department of Homeland Security. (2007, March). Target Capabilities List: A Companion 

to the National Preparedness Guidelines. Available from 

http://www.dhs.gov/xlibrary/assets/National_Preparedness_Guidelines.pdf 

U.S. Department of Homeland Security. (2007, February). Homeland Security Exercise and 

Evaluation Program, Vol. III: Exercise Evaluation and Improvement Planning. Available 

from https://hseep.dhs.gov/support/VolumeIII.pdf 

U.S. Department of Homeland Security. (2008, January). National Response Framework. FEMA 

Publication P-682 (Catalog number 08011-1). Washington, DC: Author. Available at 

http://www.fema.gov/pdf/emergency/nrf/nrf-core.pdf 

U.S. Department of Labor, Bureau of Labor Statistics (2008, October). Employment Cost Index – 

September 2008. Available at http://www.bls.gov/news.release/archives/eci_10312008.pdf 

U.S. Department of the Interior. (2008, November 15). Fiscal Year 2008 Performance and 

Accountability Report. Available at http://www.doi.gov/pfm/par/par2008/ 

U.S. Department of the Interior, Office of Surface Mining Reclamation and Enforcement. 

(2008). Budget Justifications and Performance Information Fiscal Year 2008. Available at 

http://www.doi.gov/budget/2008/data/greenbook/FY2008_OSM_Greenbook.pdf 

U.S. Government Accountability Office. (2001, August). GAO-01-1084SP: Results-Oriented 

Budget Practices in Federal Agencies. Available at 

http://www.gao.gov/new.items/d011084sp.pdf 

95 

 
U.S. Government Accountability Office. (2002, January). GAO-02-236: Managing for Results: 

Agency Progress in Linking Performance Plans with Budgets and Financial Statements.

Available at http://www.gao.gov/new.items/d02236.pdf

U.S. Government Accountability Office. (2002, December). GAO-03-258: Managing for 

Results: Efforts to Strengthen the Link between Resources and Results at the Nuclear 

Regulatory Commission. Available at http://www.gao.gov/new.items/d03258.pdf

U.S. Government Accountability Office. (2003, September). GAO-03-1166T: Results-Oriented 

Government: Using GPRA to Address 21st Century Challenges. Available at 

http://www.gao.gov/new.items/d031166t.pdf

U.S. Government Accountability Office. (2004, March). GAO-04-432: Coast Guard: 

Relationship between Resources Used and Results Achieved Needs to Be Clearer.

Available at http://www.gao.gov/new.items/d04432.pdf 

U.S. Government Accountability Office. (2004, May 17). GAO-04-415: Nuclear Regulation: 

NRC Needs to More Aggressively and Comprehensively Resolve Issues Related to the 

Davis-Besse Nuclear Power Plant's Shutdown. Available at 

http://www.gao.gov/products/GAO-04-415 

U.S. Government Accountability Office. (2005, September 7). GAO-05-728 Safety inspector 

training: Aviation Safety: FAA Management Practices for Technical Training Mostly 

Effective; Further Actions Could Enhance Results. Available at 

http://www.gao.gov/products/GAO-05-728 

U.S. Government Accountability Office. (2006, September). GAO-06-1029: Oversight of 

Nuclear Power Plant Safety Has Improved, But Refinements Are Needed. Available at 

http://www.gao.gov/new.items/d061029.pdf 

U.S. Government Accountability Office. (2007, March). GAO-07-542T: Federal Financial 

Management: Critical Accountability and Fiscal Stewardship Challenges Facing Our 

Nation. Available at http://www.gao.gov/new.items/d07542t.pdf 

U. S. Government Accountability Office. (2007, January 17). GAO-07-105: Human Capital 

Retirements and Anticipated New Reactor Applications Will Challenge NRC’s Workforce. 

Available at http://www.gao.gov/products/GAO-07-105 

U.S. Nuclear Regulatory Commission. Advanced Procurement Plan User Guide (and office 

submission). 

U.S. Nuclear Regulatory Commission. Advice of Allowances and Financial Plans.

U.S. Nuclear Regulatory Commission. Consolidated Response to the 2007 Reactor Oversight 

Process External Survey. Available at 

96 

 
http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

81440771 

U.S. Nuclear Regulatory Commission. Emergency Preparedness and Response. Available at 

http://www.nrc.gov/about-nrc/emerg-preparedness.html 

U.S. Nuclear Regulatory Commission. Enforcement Manual. Available at 

http://adamswebsearch2.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML

062710589 

U.S. Nuclear Regulatory Commission. Fiscal Year 2006 Budget Execution Report (as of 

September 30, 2006). 

U.S. Nuclear Regulatory Commission. Fiscal Year 2007 Budget Execution Report (as of 

September 30, 2007). 

U.S. Nuclear Regulatory Commission. Fiscal Year 2008 Budget Execution Report (as of 

September 30, 2008). 

U.S. Nuclear Regulatory Commission. Fiscal Year 2008 Office Budget Submissions. 

U.S. Nuclear Regulatory Commission. Fiscal Year 2010 Office Budget Submissions. 

U.S. Nuclear Regulatory Commission. Fiscal Year 2010 Performance Budget. 

U.S. Nuclear Regulatory Commission. Inspection Reports. Available at 

http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/listofrpts_body.html 

U.S. Nuclear Regulatory Commission. IRMC 0210 IR Self Assessment Procedure. 

U.S. Nuclear Regulatory Commission. IRMC 0220 Corrective Action Procedure. 

U.S. Nuclear Regulatory Commission. NRC Enforcement Policy. Available from 

http://www.nrc.gov/about-nrc/regulatory/enforcement/enforce-pol.html 

U.S. Nuclear Regulatory Commission. OCFO Budget Call Guidance for Fiscal Year 2008 

PBPM. 

U.S. Nuclear Regulatory Commission. OCFO Budget Call Guidance for Fiscal Year 2009 

PBPM. 

U.S. Nuclear Regulatory Commission. OCFO Budget Call Guidance for Fiscal Year 2010 

PBPM. 

97 

 
U.S. Nuclear Regulatory Commission. Office contract management reports. 

U.S. Nuclear Regulatory Commission. Office instructions, guidance, and process documentation. 

U.S. Nuclear Regulatory Commission. Office monthly budget execution reports. 

U.S. Nuclear Regulatory Commission. Office operating plans. 

U.S. Nuclear Regulatory Commission. Office operating-level reports. 

U.S. Nuclear Regulatory Commission. Office performance monitoring reports. 

U.S. Nuclear Regulatory Commission. Palo Verde NRC Lessons Learned. Available at 

http://adamswebsearch2.nrc.gov/idmws/doccontent.dll?library=PU_ADAMS%5ePBNTAD

01&ID=081010296 

U.S. Nuclear Regulatory Commission. Planning, Budgeting and Performance Management 

(PBPM) process documentation from Office of the Chief Financial Officer. 

U.S. Nuclear Regulatory Commission. Resident Inspector Demographics, 2007. Available at 

http://www.nrc.gov/reading-rm/doc-collections/commission/secys/2008/secy2008-

0046/atch6.pdf 

U.S. Nuclear Regulatory Commission. ROP Budget History – Significant Events Impacting ROP 

Budgets. 

U.S. Nuclear Regulatory Commission. RPS Funds Report (as of September 30, 2008). 

U.S. Nuclear Regulatory Commission. RPS Funds Technical Assistance Report (as of September 

30, 2008). 

U.S. Nuclear Regulatory Commission. Standard Review Plan for the Review of Safety Analysis 

Reports for Nuclear Power Plants (NUREG-0800). Available at 

http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr0800/ 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (1996, March 5). 

OIG/96-01S: Office of the Inspector General Event Inquiry: Implementation of 

Recommendations to Improve NRC's Program for Protecting Allegers against Retaliation. 

Available at http://www.nrc.gov/reading-rm/doc-collections/insp-gen/1996/96-01s.html 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2002, August 21). OIG02-A-15: Audit Report: Review of NRC’s Significance Determination Process. Available at 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/2002/02a-15/02a-15.pdf 

98 

 
U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2004, September 16). 

OIG–04-A-20: Audit Report: Audit of NRC’s Incident Response Program. Available at 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/2004/04-a-20.pdf

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2004, December 22). 

OIG-05-A-06: Audit Report: Audit of the Baseline Inspection Program. Available at 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/2005/05-a-06.pdf 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2005, January 31). OIG05-A-09: Audit Report: Audit of the Budget Formulation Process. Available at 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/2005/05-a-09.pdf 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2005, April 13). OIG05-A-11: Audit Report: Audit of the Reactor Program System. Available at 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/2005/05-a-11.pdf 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2006, February 16). 

OIG-06-A-09: Audit Report: Audit of NRC’s Office of Nuclear Security and Incident 

Response. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

60470432 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2007, June 25). OIG-07-

A-13: Audit Report: Audit of NRC’s Emergency Preparedness Program. Available from 

http:/www.nrc.gov/reading-rm/doc-collections/insp-gen/ 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2008, August 22). 

NRC’s Response to Security-Related Concerns at Peach Bottom Atomic Power Plant. 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2008, September 26). 

OIG-08-A-17: Audit Report: Audit of NRC’s Enforcement Program. Available at 

http://www.nrc.gov/reading-rm/doc-collections/insp-gen/2008/oig-08-a-17.pdf 

U.S. Nuclear Regulatory Commission, Office of the Inspector General. (2008, September 30). 

OIG-08-A-20: Evaluation Report: Inspector General's Assessment of The Most Serious 

Management Challenges Facing NRC. Available at http://www.nrc.gov/reading-rm/doccollections/insp-gen/2008/oig-08-a-20.pdf 

U.S. Nuclear Regulatory Commission. (1987, August 21; amended 1991, June 27). NRC 

Regulations Title 10, Code of Federal Regulations 952 FR 31602, Aug. 21, 1987, as 

Amended at 56 FR 29407, June 27, 1991). Available at http://www.nrc.gov/readingrm/doc-collections/cfr/

99 

 
U.S. Nuclear Regulatory Commission. (1989, May 9). Management Directive 9.19: O/F, Office 

of Enforcement. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

41400086 

U.S. Nuclear Regulatory Commission. (1989, October 1). Management Directive 4.7: Policy and 

Practices Governing NRC Long-Range Planning, Budget Formulation, and Resource 

Management. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

41400125 

U.S. Nuclear Regulatory Commission. (1999). Pilot Program Evaluation Final Report. 

Available at http://www.nrc.gov/NRR/OVERSIGHT/ROP/ppepfinalreport.pdf 

U.S. Nuclear Regulatory Commission. (1999, February 4). Management Directive 8.8: 

Management of Allegations. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

41730152 

U.S. Nuclear Regulatory Commission. (2000, July). Reactor Oversight Process (NUREG-1649, 

Rev. 3). Washington, DC: U.S. Nuclear Regulatory Commission, Office of Public Affairs. 

Available from http://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1649/r3/#pub-info

U.S. Nuclear Regulatory Commission. (2001). Charter for the Performance Improvement Panel 

Reactor Oversight Process Initial Implementation Evaluation Panel Final Report. 

U.S. Nuclear Regulatory Commission. (2001, March 27). Management Directive 8.3: NRC 

Incident Investigation Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

31250592 

U.S. Nuclear Regulatory Commission. (2002). SECY-02-0036: Formation of NSIR and 

Communications Mandate from Commission. 

U.S. Nuclear Regulatory Commission. (2002, August 13). Management Directive 4.6: License 

Fee Management Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

41410572 

U.S. Nuclear Regulatory Commission. (2002, June 19). Management Directive 8.13: Reactor 

Oversight Process. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

41770519 

100 

 
U.S. Nuclear Regulatory Commission. (2003, June 23). SECY-03-0104: Organizational 

Effectiveness Assessment for the Office of Nuclear Security and Incident Response.

Available at http://www.nrc.gov/reading-rm/doccollections/commission/secys/2003/secy2003-0104/2003-0104scy.html 

U.S. Nuclear Regulatory Commission. (2004, August). FY 2004-2009 Strategic Plan (NUREG1614, Vol. 3). Available from http://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1614/v3/

U.S. Nuclear Regulatory Commission. (2004, May 18). Management Directive 4.4: Management 

Controls. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

41770522 

U.S. Nuclear Regulatory Commission. (2005, April). NUREG-0728, Rev. 4. Washington, DC: 

U.S. Nuclear Regulatory Commission, Office of Nuclear Security and Incident Response. 

Available at http://www.nrc.gov/about-nrc/emerg-preparedness/respond-to-emerg/incidentresponse.pdf 

U.S. Nuclear Regulatory Commission. (2005, April). Reporting Safety Concerns to the NRC 

(NUREG-BR0240, Rev. 3). Washington, DC, U.S. Nuclear Regulatory Commission, Office 

of Public Affairs. Available at http://www.nrc.gov/reading-rm/doccollections/nuregs/brochures/br0240/r3/br0240r3.pdf

U.S. Nuclear Regulatory Commission. (2005, February 23). Enforcement Program Annual 

Report – Fiscal Year 2004. Washington, DC: U.S. Nuclear Regulatory Commission, Office 

of Enforcement. Available at http://www.nrc.gov/reading-rm/doccollections/enforcement/annual-rpts/04report.pdf 

U.S. Nuclear Regulatory Commission. (2005, July 7). Management Directive 4.3: Financial 

Management Systems. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

52100100 

U.S. Nuclear Regulatory Commission. (2005, June). Backgrounder on Allegation Process. 

Available at http://www.nrc.gov/reading-rm/doc-collections/fact-sheets/allegation-processbg.html

U.S. Nuclear Regulatory Commission. (2005, November 22). NRC Inspection Manual, Chapter 

0609: Significance Determination Process. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

52790205 

101 

 
U.S. Nuclear Regulatory Commission. (2005, September 9). Management Directive 4.1: 

Accounting Policy and Practices. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

52650463 

U.S. Nuclear Regulatory Commission. (2005, September 29). Semiannual Report – Status of 

Davis-Besse Lessons Learned Task Force Report Recommendations. Available at 

http://www.nrc.gov/reactors/operating/ops-experience/vessel-head-degradation/lessonslearned/lessons-learned-files/lltf-web-page-status083105.pdf

U.S. Nuclear Regulatory Commission. (2006). Enforcement Program Annual Report - Calendar 

and Fiscal Years 2005. Washington, DC: U.S. Nuclear Regulatory Commission, Office of 

Enforcement. Available at 

http://adamswebsearch2.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML

061380394

U.S. Nuclear Regulatory Commission. (2006, February). Performance Budget: Fiscal Year 2007 

(NUREG-1100, Vol. 22). Available at http://www.nrc.gov/reading-rm/doccollections/nuregs/staff/sr1100/v22/sr1100v22.pdf 

U.S. Nuclear Regulatory Commission. (2006, June). Nuclear Regulatory Legislation: 109th 

Congress (Vol. 1, No. 7, Rev. 1, 2nd Session and Vol. 2, No. 7, 1st Session). Washington, 

DC: U.S. Nuclear Regulatory Commission, Office of the General Counsel. Available at 

http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr0980/

U.S. Nuclear Regulatory Commission. (2006, June 16). Management Directive 8.2: NRC 

Incident Response Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

61860066 

U.S. Nuclear Regulatory Commission. (2006, March 29). Management Directive 8.14: Agency 

Action Review Meeting. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

60940095 

U.S. Nuclear Regulatory Commission. (2006, November 16). NRC Inspection Manual, Chapter 

0313: Industry Trends Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

62890438 

U.S. Nuclear Regulatory Commission. (2007). Integrated Regulatory Review Service SelfAssessment Results and Recommendation to Invite an Integrated Regulatory Review 

Service Mission to the United States.

102 

 
U.S. Nuclear Regulatory Commission. (2007, February). Performance Budget Fiscal Year 2008 

(NUREG 1100, Vol. 23). Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

72910359 

U.S. Nuclear Regulatory Commission. (2007, February 27). NRC Inspection Manual, Chapter 

0608: Performance Indicator Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

70360605 

U.S. Nuclear Regulatory Commission. (2007, January 10). NRC Inspection Manual, Chapter 

0307: Reactor Oversight Process Self-Assessment Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

73520141 

U.S. Nuclear Regulatory Commission. (2007, January 26). NRC Inspection Manual, Chapter 

2515: Light-Water Reactor Inspection Program – Operations Phase. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

61580550 

U.S. Nuclear Regulatory Commission. (2007, June 13). Reviewing the Common Performance 

Indicator, Technical Quality of Incident and Allegation Activities; Appendix B of the 

Allegation Casework Review Summary Sheet. SA-105. Available at http://nrcstp.ornl.gov/procedures/sa105.pdf 

U.S. Nuclear Regulatory Commission. (2007, November 15). Performance and Accountability 

Report – Fiscal Year 2007 (NUREG-1542, Vol. 13). Available at 

http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1542/v13/ 

U.S. Nuclear Regulatory Commission. (2007, November 27). NRC Inspection Manual, Chapter 

0305: Operating Reactor Assessment Program. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

72770496 

U.S. Nuclear Regulatory Commission. (2008). Calendar Year 2007 Analysis of the Reactor 

Oversight Process Self-Assessment Metrics (in Accordance with IMC 0307, Appendix A). 

Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

80350368 

U.S. Nuclear Regulatory Commission. (2008, April 2). Policy Issue Notation Vote: SECY-08-

004): Reactor Oversight Process Self-Assessment For Calendar Year 2007. Available at 

http://www.nrc.gov/reading-rm/doc-collections/commission/secys/2008/secy2008-

0046/2008-0046scy.html

103 

 
U.S. Nuclear Regulatory Commission. (2008, April 10). Cross-Cutting Issue Review Effort.

Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

81010229 

U.S. Nuclear Regulatory Commission. (2008, April 10). Lesson Learned Inputs for the Reactor 

Oversight Program Safety Culture Evaluation for Use at 4/17/08 Public Meeting.

Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

81010229

U.S. Nuclear Regulatory Commission. (2008, April 11). Management Directive 4.2: 

Administrative Control of Funds. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

81120200 

U.S. Nuclear Regulatory Commission. (2008, August). Davis-Besse Reactor Vessel Head 

Degradation Lessons Learned Task Force (NUREG/BR-0353, Rev. 1). Available at 

http://www.nrc.gov/reading-rm/doc-collections/nuregs/brochures/br0353/br0353r1.pdf 

U.S. Nuclear Regulatory Commission. (2008, August). Information Digest, 2008-2009 (NUREG1350, Vol. 20). Washington, DC: U.S. Nuclear Regulatory Commission, Office of Public 

Affairs. Available at http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1350/

U.S. Nuclear Regulatory Commission. (2008, August 6). Management Directive 3.14: U.S. 

Nuclear Regulatory Commission Public Web Site. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

80600112 

U.S. Nuclear Regulatory Commission. (2008, August 13). ROP Program Evaluations and 

Stakeholder Feedback. Available at 

http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/program-evaluations.html

U. S. Nuclear Regulatory Commission. (2008, February). Fiscal Year 2008–FY 2013 Strategic 

Plan. Available at http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1614/ 

U.S. Nuclear Regulatory Commission. (2008, February 4). Performance Budget (NUREG-1100).

Available at http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1100/

U.S. Nuclear Regulatory Commission. (2008, February 14). Strategic Plan (NUREG-1614).

Available at http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1614/

U.S. Nuclear Regulatory Commission. (2008, March 27). SECY-08-0041: Fiscal Year 2007 

Results of the Industry Trends Program for Operating Reactors and Status of the Ongoing 

Development of the Program. Available at 

104 

 
105 

 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

80510574 

U.S. Nuclear Regulatory Commission. (2008, September 10). NRC Enforcement Policy: 

Extension of Discretion Period of Interim Enforcement Policy. Federal Register, 73(176), 

Notices. Available at http://edocket.access.gpo.gov/2008/pdf/E8-20972.pdf 

U.S. Nuclear Regulatory Commission. (2008, September 15). Reactor Oversight Process. 

Available at http://www.nrc.gov/NRR/OVERSIGHT/ASSESS/index.html

U.S. Nuclear Regulatory Commission. (2008, September 29). Allegation Program – Calendar 

Year 2007 Annual Performance Report. Available at 

http://adamswebsearch.nrc.gov/idmws/ViewDocByAccession.asp?AccessionNumber=ML0

82730153 

U.S. Nuclear Regulatory Commission. Performance and Accountability Report (NUREG-1542).

Available at http://www.nrc.gov/reading-rm/doc-collections/nuregs/staff/sr1542/

U.S. Nuclear Regulatory Commission.. Enforcement Guidance. Available at 

http://www.nrc.gov/about-nrc/regulatory/enforcement/guidance.html 

U.S. Office of Management and Budget. (2006). Program Assessment: Office of Surface Mining 

– Federal Managed Regulation of Surface Coal Mining and Abandoned Mine Land 

Reclamation. Available at 

http://www.whitehouse.gov/omb/expectmore/summary/10002364.2006.html 

U.S. Office of Management and Budget. Program Assessment Rating Tool Guidance No. 2008-

01. 

U.S. Office of Management and Budget. Program Assessment: 2003 Reactor Inspection and 

Performance Assessment. Available at 

http://www.whitehouse.gov/omb/expectmore/summary/10001174.2003.html 

U.S. Office of Management and Budget. Program Assessment: 2007 Decommissioning and Low 

Level Waste. Available at 

http://www.whitehouse.gov/omb/expectmore/summary/10009039.2007.html

U.S. Office of Management and Budget. Program Assessment: 2007 High-Level Waste 

Repository. Available at 

http://www.whitehouse.gov/omb/expectmore/summary/10009040.2007.html