OFFICE OF 

THE INSPECTOR GENERAL

U.S. NUCLEAR

REGULATORY COMMISSION

Audit of NRC’s Baseline Inspection Program

OIG-05-A-06 December 22, 2004

AUDIT REPORT

All publicly available OIG reports are accessible through 

NRC’s Web site at:

http://www.nrc.gov/reading-rm/doc-collections/ionsp-gen/
December 22, 2004

MEMORANDUM TO: Luis A. Reyes

Executive Director for Operations

FROM: Stephen D. Dingbaum/RA/

Assistant Inspector General for Audits

SUBJECT: AUDIT OF NRC’S BASELINE INSPECTION PROGRAM 

(OIG-05-A-06)

Attached is the Office of the Inspector General’s (OIG) audit report titled, Audit of NRC’s

Baseline Inspection Program. 

The report reflects the results of our audit to determine whether NRC’s baseline inspection

program (1) is based on a sound methodology, (2) is carried out by sufficient, qualified staff,

and (3) is completed at all operating commercial nuclear power plants. Overall, the audit

found that NRC staff, licensees, and stakeholders view the Reactor Oversight Process,

which includes the baseline inspection program, as a significant improvement over the

previous plant assessment program. However, OIG identified several weaknesses

associated with the baseline program. Specifically, NRC’s baseline inspection program is

generally sound, but needs improvement; impacts on the baseline program from resource

challenges are likely to continue; the resident inspector training program needs

improvement; the report of 100 percent baseline inspection program completion in CY 2002

is not fully supported by documentation; and guidance is unclear for new baseline

completion criteria.

This report makes 10 recommendations to improve the efficiency and effectiveness of the

baseline inspection program.

During an exit conference on August 30, 2004, NRC officials provided informal comments

concerning the draft audit report. Subsequent to that meeting, OIG met with NRR senior

managers to address issues and comments needing further clarification and/or explanation. 

Comments your office provided at the exit meeting, during subsequent discussions, and in

your November 23, 2004, written response to the draft report have been incorporated, as

appropriate, in our final report. Appendix F contains your written response in its entirety. 

Appendix G contains our point-by-point analysis of the agency’s formal comments. 

If you have any questions, please call Russ Irish at 415-5972 or me at 415-5915.

Attachment: As stated
Distribution List

B. John Garrick, Chairman, Advisory Committee on Nuclear Waste 

Mario V. Bonaca, Chairman, Advisory Committee on Reactor Safeguards

John T. Larkins, Executive Director, Advisory Committee on Reactor

 Safeguards/Advisory Committee on Nuclear Waste

G. Paul Bollwerk, III, Chief Administrative Judge, Atomic Safety and

 Licensing Board Panel

Karen D. Cyr, General Counsel

John F. Cordes, Jr., Director, Office of Commission Appellate Adjudication 

Jesse L. Funches, Chief Financial Officer

Janice Dunn Lee, Director, Office of International Programs 

William N. Outlaw, Director of Communications

Dennis K. Rathbun, Director, Office of Congressional Affairs

Eliot B. Brenner, Director, Office of Public Affairs

Annette Vietti-Cook, Secretary of the Commission

Patricia G. Norry, Deputy Executive Director for Management Services, OEDO 

William F. Kane, Deputy Executive Director for Homeland Protection 

 and Preparedness, OEDO

Martin J. Virgilio, Deputy Executive Director for Materials, Research 

 and State Programs, OEDO

Ellis W. Merschoff, Deputy Executive Director for Reactor Programs, OEDO

William M. Dean, Assistant for Operations, OEDO

Jacqueline E. Silber, Chief Information Officer

Michael L. Springer, Director, Office of Administration

Frank J. Congel, Director, Office of Enforcement

Guy P. Caputo, Director, Office of Investigations 

Paul E. Bird, Director, Office of Human Resources

Corenthis B. Kelley, Director, Office of Small Business and Civil Rights 

Jack R. Strosnider, Director, Office of Nuclear Material Safety and Safeguards 

James E. Dyer, Director, Office of Nuclear Reactor Regulation 

Carl J. Paperiello, Director, Office of Nuclear Regulatory Research

Paul H. Lohaus, Director, Office of State and Tribal Programs

Roy P. Zimmerman, Director, Office of Nuclear Security and Incident Response 

Samuel J. Collins, Regional Administrator, Region I

William D. Travers, Regional Administrator, Region II

James L. Caldwell, Regional Administrator, Region III

Bruce S. Mallett, Regional Administrator, Region IV

Office of Public Affairs, Region I

Office of Public Affairs, Region II

Office of Public Affairs, Region IV
Audit of NRC’s Baseline Inspection Program

i

EXECUTIVE SUMMARY

BACKGROUND

The Nuclear Regulatory Commission (NRC) is responsible for developing,

implementing, and enforcing rules and regulations that govern licensed,

commercial nuclear power plants. To carry out that responsibility, NRC

conducts inspections at the Nation’s 103 operating commercial nuclear

power plants to ensure adequate protection of the public health and safety,

the common defense and security, and the environment in the civilian use of

nuclear materials. 

Beginning in April 2000, the NRC’s Reactor Inspection Program and the

Reactor Performance Assessment program were combined into a single

program for commercial nuclear power plants. This combined program

implements the revised reactor oversight process (ROP). An integral part of

the ROP is the baseline inspection program, which was developed using a

risk-informed approach to determine a comprehensive list of areas to inspect

(i.e., inspectable areas). The baseline inspection program evaluates licensee

performance in areas not measured, or not fully measured, by licenseereported performance indicators. 

NRC resident inspectors and senior resident inspectors located at each plant

site carry out a major part of the baseline inspection program. Their primary

role is to observe, evaluate, and verify the adequacy of licensees' nuclear

safety activities, concentrating on day-to-day licensee operational and event

follow-up activities, and licensee activities and processes that are important

to safety and reliability. In addition, region-based inspectors with specific

qualifications have primary responsibility for performing a number of the

baseline inspection procedures related to specialty areas, such as Radiation

Protection and Physical Security Protection. NRC uses inspectors from

headquarters, as requested by the regions and in exceptional circumstances,

to provide the regions with assistance towards the completion of the baseline

inspection program. 

PURPOSE

This audit was conducted to determine whether NRC’s baseline inspection

program:

< is based on a sound methodology, 

< is carried out by sufficient, qualified staff, and 

< is completed at all operating commercial nuclear power plants.
Audit of NRC’s Baseline Inspection Program

1

 SECY-04-0053, Reactor Oversight Process Self-Assessment for Calendar Year 2003, dated April 6,

2004.

ii

RESULTS IN BRIEF

Overall, NRC staff, licensees, and stakeholders view the ROP, which

includes the baseline inspection program, as a significant improvement over

the previous, more subjective, plant assessment program. NRC's most

recent ROP self-assessment1

 indicates that the ROP was generally effective

in monitoring operating commercial nuclear power plant activities, as well as

focusing NRC resources on significant performance issues in CY 2003. As a

result, the staff believes that plants are receiving an appropriate level of

oversight commensurate with their performance. However, the Office of the

Inspector General (OIG) identified weaknesses associated with the program. 

Specifically, 

A. the baseline inspection program framework is generally sound, but

needs improvement (see page 7),

B. impacts on the baseline inspection program from resource challenges

are likely to continue (see page 25), 

C. the resident inspector training program needs improvement (see page

31), 

D. the report of 100 percent baseline inspection program completion in

CY 2002 is not fully supported by documentation (see page 35), and

E. guidance is unclear for new baseline completion criteria (see page

40).

These issues exist primarily because the agency lacks a mechanism to

assess the overall effectiveness and quality of the baseline program. As a

result, issues such as unclear guidance, inconsistent implementation of the

program, and insufficient documentation of inspection activities have not

been adequately evaluated. Due to its importance in ensuring the Nation’s

safety, additional management focus is warranted to improve the

effectiveness of NRC’s baseline inspection program. Although the current

program provides a framework for managing the agency’s

inspection activities at operating nuclear power plants, the program should be

strengthened to better define expectations regarding implementation,

completion, and training. Addressing the identified weaknesses will improve

the overall effectiveness and efficiency of the baseline program and provide

assurance that all aspects of the program are coordinated.
Audit of NRC’s Baseline Inspection Program

iii

RECOMMENDATIONS

A Consolidated List of Recommendations appears on page 43 of this report.

OIG ANALYSIS OF AGENCY COMMENTS

At an exit conference with agency senior executives held on August 30,

2004, NRC officials generally agreed with most of the report’s findings and

recommendations. Subsequent to that meeting, OIG met with NRR senior

managers to address specific issues and concerns needing further

clarification and/or explanation. On November 23, 2004, the Executive

Director for Operations (EDO) provided a formal response to this report in

which he agreed with nine of the report’s eleven recommendations. 

However, the EDO stated that Recommendation 3 is not appropriate under

the current ROP philosophy and that Recommendation 5 would be better

addressed through the agency’s response to Recommendation 4. The

EDO’s transmittal letter and specific comments on this report are included as

Appendix F. 

This final report incorporates revisions made, where appropriate, as a result

of the subsequent meetings and the agency’s informal and formal written

comments. Because OIG takes exception to the agency’s comments

regarding Recommendations 3 and 5, a point-by-point analysis is presented

in Appendix G. Recommendation 5 has been retracted and will be

addressed, as the agency requested, through the agency’s response to

Recommendation 4. However, upon submittal, OIG will evaluate the

agency’s proposed plan of action for addressing Recommendation 4 to

ensure that the intent of Recommendation 5 is incorporated.
Audit of NRC’s Baseline Inspection Program

iv

[Page intentionally left blank.]
Audit of NRC’s Baseline Inspection Program

v

ABBREVIATIONS AND ACRONYMS

BWR boiling-water reactor

CFR Code of Federal Regulations

CY calendar year

EDO Executive Director for Operations

FTE full-time equivalents

FY fiscal year

GE General Electric

IMC Inspection Manual Chapter

NRC Nuclear Regulatory Commission

NRR Office of Nuclear Reactor Regulation

OIG Office of the Inspector General

OMB Office of Management and Budget

PWR pressurized-water reactor

ROP reactor oversight process

ROP3 reactor oversight process inspection cycle 3
Audit of NRC’s Baseline Inspection Program

vi

[Page intentionally left blank.]
Audit of NRC’s Baseline Inspection Program

vii

TABLE OF CONTENTS

EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . i

ABBREVIATIONS AND ACRONYMS ...................................v

I. BACKGROUND . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1

I. PURPOSE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

II. FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

A. THE BASELINE INSPECTION PROGRAM FRAMEWORK IS GENERALLY

SOUND, BUT NEEDS IMPROVEMENT ...........................7

A.1 Agency Lacks Documentation to Support Rationale for

Sampling Methodology .............................7

A.2 Operating Experience Program Needs Improvement . . . . . 12

A.3 Agency Lacks Clear Guidance on Handling Informal Issues 17

A.4 Performance Measures Need Improvement . . . . . . . . . . . . 21

B. IMPACTS ON THE BASELINE INSPECTION PROGRAM FROM RESOURCE

CHALLENGES ARE LIKELY TO CONTINUE . . . . . . . . . . . . . . . . . . . . . . . 25

C. THE RESIDENT INSPECTOR TRAINING PROGRAM NEEDS 

IMPROVEMENT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

D. THE REPORT OF 100 PERCENT BASELINE INSPECTION PROGRAM

COMPLETION IN CY 2002 IS NOT FULLY SUPPORTED BY 

DOCUMENTATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35

E. GUIDANCE IS UNCLEAR FOR NEW BASELINE COMPLETION CRITERIA . . 40

III. CONSOLIDATED LIST OF RECOMMENDATIONS . . . . . . . . . . . . . . . . . . 43

IV. AGENCY COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

APPENDICES

A. SCOPE AND METHODOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45

B. DISCUSSION OF GUIDANCE ON PERFORMANCE AND OUTPUT

MEASURES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

C. OPERATING PLAN PERFORMANCE MEASURES . . . . . . . . . . . . . . . . . 53

D. DETAILS OF OIG’S REVIEW OF CY 2002 BASELINE INSPECTION

PROGRAM COMPLETION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

E. BASELINE INSPECTION PROGRAM INSPECTION PROCEDURES . . . . 59

F. FORMAL AGENCY COMMENTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .61

G. DETAILED OIG ANALYSIS OF AGENCY COMMENTS. . . . . . . . . . . . . . . 65
Audit of NRC’s Baseline Inspection Program

viii
Audit of NRC’s Baseline Inspection Program

ix

[Page intentionally left blank.]
Audit of NRC’s Baseline Inspection Program

1

 Fort Calhoun nuclear power plant

 (Photo courtesy of NRC Website)

I. BACKGROUND

Authorization and Responsibilities

The NRC is responsible for developing,

implementing, and enforcing rules and regulations

that govern licensed, commercial nuclear power

plants. Under the Atomic Energy Act and the

Commission’s regulations, licensees must allow

NRC representatives to inspect the premises and

facilities where nuclear material is used or stored. 

NRC conducts inspections at the Nation’s 103

operating commercial nuclear power plants to

ensure adequate protection of the public health

and safety, the common defense and security, and

the environment in the civilian use of nuclear

materials. 

Principles of Good Regulation

NRC adheres to the Principles of Good Regulation, which, among other

things, includes elements of independence, openness, and clarity. 

Independence means that NRC decisions will be based on “objective,

unbiased assessments of all information . . .” and documented with reasons

explicitly stated. Openness considers that “Nuclear regulation is the public's

business, and it must be transacted publicly and candidly. The public must

be informed about and have the opportunity to participate in the regulatory

processes as required by law.” And clarity means that “Regulations should

be coherent, logical, and practical. There should be a clear nexus between

regulations and agency goals and objectives whether explicitly [or implicitly]

stated. Agency positions should be readily understood and easily applied.” 

The Reactor Oversight Process Framework 

Beginning in April 2000, the NRC’s Reactor Inspection Program and the

Reactor Performance Assessment program were combined into a single

program for commercial nuclear power plants. This combined program

implements the revised reactor oversight process (ROP). The ROP is

designed to verify that agency licensees operate in accordance with NRC
Audit of NRC’s Baseline Inspection Program

2

Protect Public Health and Safety 

in the Operation of Commercial 

Nuclear Power Plants

Reactor Safety Radiation Safety

Initiating 

Events

Mitigating 

Systems

Barrier 

Integrity

Emergency 

Preparedness

Occupational 

Radiation 

Safety

Public 

Radiation 

Safety

Safeguards

Mission:

Performance Areas:

Cornerstones:

Physical 

Protection

Cross-cutting areas: 

- human performance 

- safety conscious work environment 

- problem identification and resolution

rules and regulations in order to identify and resolve issues before the safety

of plant operations is affected. The ROP includes risk-informed baseline

inspections, use of licensees’ performance indicator data, and a revised

reactor assessment process. Another goal of the ROP is to effectively focus

attention on risk-significant activities while reducing unnecessary regulatory

burden on licensees. 

The ROP assesses three key performance areas: reactor safety, radiation

safety, and safeguards. Each area contains “cornerstones” of safety that

reflect the essential aspects of safe plant operation. The following chart

shows the seven ROP cornerstones and three ROP cross-cutting areas

within which NRC assesses licensee performance.

REACTOR OVERSIGHT PROCESS PERFORMANCE AREAS AND CORNERSTONES
Audit of NRC’s Baseline Inspection Program

3

The proportions for each area are: baseline - 93%; plant-specific - 5%; and

generic issues - 2%.

There are two main principles of the ROP: 

< the agency’s mission of assuring public health and safety is met when

the agency has reasonable assurance that licensees are meeting the

objectives of the seven cornerstones, and 

< there is a level of performance above which the NRC does not need

to engage the licensee beyond some minimum level of oversight. 

The ROP uses both inspection findings developed from selective

examinations and performance indicator data to assess plant performance

within the framework of the seven cornerstones of safety. 

The ROP inspection process is composed of three major elements: 

1. a baseline inspection program, [emphasis added]

2. plant-specific inspections, and 

3. generic issue inspections (address areas of emerging concern or

those requiring increased emphasis because of recurring problems).

The Baseline Inspection Program

The baseline inspection program is an integral part of the agency’s reactor

oversight process. The program was developed using a risk-informed

approach to determine a comprehensive list of areas to inspect (i.e.,

inspectable areas) within each of the seven cornerstones. The baseline

inspection program evaluates licensee performance in areas not measured,

or not fully measured, by licensee-reported performance indicators. 
Audit of NRC’s Baseline Inspection Program

4

 Inspection Planning

 (Photo courtesy of NRC Website)

Baseline Inspection Program Objectives

According to NRC's Inspection Manual Chapter (IMC) 2515, Appendix A,

Risk-Informed Baseline Inspection Program, the baseline program supports

the goals and objectives of the reactor oversight process. The program’s

objectives are: 

• to obtain sufficient inspection information to use in conjunction with

performance indicators to assess the safety performance of power

reactor licensees,

C to determine the licensee’s ability to identify, assess the significance

of, and effectively correct issues commensurate with their risk

significance, 

C to verify the accuracy and completeness of performance indicators

used in conjunction with inspection findings to assess the

performance of power reactor licensees, and 

C to provide a mechanism for the NRC to remain cognizant of plant

status and conditions.

Organizational Responsibilities and Budget

The Office of Nuclear Reactor Regulation (NRR) has the overall

responsibility for providing guidance and direction for the ROP and the

baseline inspection program. 

The agency’s four regional offices are responsible for

implementing the baseline inspection program through

planning and conducting the inspections. 

The FY 2003 costs for the ROP exceeded approximately

336 full-time equivalents (FTE) with an additional $16.4

million spent for contractor support and travel costs. 

Approximately 251 FTE were expended toward

completing the baseline inspection program. The agency

estimated approximately 350 FTE for the ROP in FY 2004

(including contractor support and travel costs) of which

about 271 are for completing the baseline inspection

program. Table 1 shows the allocation and actual usage

of ROP resources dedicated for the baseline inspection program in the four

regions. According to NRR, the actual resources expended on the baseline

inspection program in calendar year (CY) 2003 included time spent on direct

inspections, inspection preparation and documentation, and plant status

efforts. 
Audit of NRC’s Baseline Inspection Program

2 The ROP is implemented on a calendar year basis; however, staff obtains and reports resource data

on a fiscal year basis. The staff asserts that there is no reason to believe that these results would be

significantly different if prepared on a calendar year basis. Nonetheless, OIG requested NRR to

provide actual numbers for the CY 2003 total resource effort.

3 In March 2004, the agency increased its FY 2004 regional allocation by 3 FTE to 271 FTE

for baseline inspections.

5

Table 1: Baseline Inspection Program Resources

Fiscal Year FTE Allocated Actual FTE Expended/calendar year2

2003 273.3 251 /CY 2003 

2004 2713 CY 2004 (to be determined)

In an effort to avoid baseline inspection program resource challenges in

future years, NRC added 15 FTE inspector positions for FY 2004 through FY

2006. The additional FTE is intended to increase inspector resources for

performing plant-specific activities and supplemental inspections.

Baseline Inspection Program Inspectors

Resident inspectors and senior resident inspectors located at each plant site

carry out a major part of the baseline inspection program and also

participate, to a lesser degree, in plant-specific and generic safety issue

inspections. Their primary role is to observe, evaluate, and verify the

adequacy of licensees' nuclear safety activities, concentrating on day-to-day

licensee operational and event follow-up activities, and licensee activities and

processes that are important to safety and reliability.

In addition, region-based inspectors with specific qualifications have primary

responsibility for performing a number of the baseline inspection procedures

related to specialty areas, such as Radiation Protection and Physical Security

Protection. NRC uses inspectors from headquarters, as requested by the

regions and in exceptional circumstances, to provide the regions assistance

towards the completion of the baseline inspection program. 
Audit of NRC’s Baseline Inspection Program

4 SECY-04-0053, Reactor Oversight Process Self-Assessment for Calendar Year 2003, dated 

April 6, 2004.

6

II. PURPOSE

The objectives of this audit were to determine whether the baseline

inspection program:

‚ is based on a sound methodology, 

‚ is carried out by sufficient, qualified staff, and 

‚ is completed at all operating commercial nuclear power plants.

Appendix A provides a detailed description of the audit’s scope and

methodology.

III. FINDINGS

Overall, NRC staff, licensees, and stakeholders view the ROP, which

includes the baseline inspection program, as a significant improvement over

the previous, more subjective, plant assessment program. NRC's most

recent ROP self-assessment4

 indicates that the ROP was generally effective

in monitoring operating commercial nuclear power plant activities, as well as

focusing NRC resources on significant performance issues in CY 2003. As a

result, the staff believes that plants are receiving an appropriate level of

oversight commensurate with their performance. However, OIG identified

weaknesses associated with the program. Specifically, 

A. the baseline inspection program framework is generally sound, but

needs improvement,

B. impacts on the baseline inspection program from resource challenges

are likely to continue, 

C. the resident inspector training program needs improvement, 

D. the report of 100 percent baseline inspection program completion in

CY 2002 is not fully supported by documentation, and

E. guidance is unclear for new baseline completion criteria.
Audit of NRC’s Baseline Inspection Program

7

As stated by one regional administrator, the baseline inspection program

provides a good foundation for licensee oversight, but its effective

implementation needs significant management involvement. Addressing

these weaknesses will improve the efficiency and effectiveness of the

baseline inspection program.

A. THE BASELINE INSPECTION PROGRAM FRAMEWORK IS GENERALLY 

SOUND, BUT NEEDS IMPROVEMENT

In general, NRC staff and industry representatives believe the baseline

inspection program's framework is generally sound, in that focus on safety is

excellent and that, for the most part, resources are appropriately focused

across the seven cornerstones of safety. As required, the agency performs

an annual self-assessment of the ROP which includes a discussion of

program effectiveness. However, the overall effectiveness of the baseline

inspection program cannot be adequately assessed because the agency

lacks a mechanism to measure effectiveness or quality. As a result, the

agency does not critically assess the effectiveness of several aspects of the

baseline inspection program. For example, weaknesses in the following

areas have not been addressed in any evaluation of the baseline inspection

program: 

C insufficient documentation of the rationale for the sampling

methodology,

C inadvertent negative effects from changes to the operating

experience program,

C inconsistent handling of informal issues, and 

C inadequate performance measures. 

A review of the above issues could identify significant weaknesses; therefore,

a critical re-evaluation of the current baseline inspection program is needed

to determine whether changes are warranted. 

 

Agency Lacks Documentation to Support Rationale for Sampling

Methodology

According to agency documents, the ROP is designed to maintain safety

more effectively by focusing NRC and industry attention on risk-significant

activities, while reducing unnecessary regulatory burden on NRC licensees. 

The agency performs a baseline program of inspections at each operating

commercial nuclear power plant by conducting at least a “minimum” level of

effort (i.e., lowest acceptable number of samples) as defined by each

inspection procedure. Yet, agency managers expect the inspection program
Audit of NRC’s Baseline Inspection Program

8

to be completed using a “nominal” (i.e., median) level of samples and

inspection hours even though the agency has not sufficiently documented the

basis for the sample sizes established in each inspection procedure. 

Additionally, there is (1) no clear understanding of whether inspection

samples or inspection hours is the intended focus of the program and (2) no

documented rationale for the use of “minimum” versus “nominal” inspection

effort. As a result, NRC cannot assure that the defined sample sizes are

adequate to assess licensee performance or that NRC resources are being

used as efficiently as possible. In addition, the commercial nuclear power

industry and the public cannot be sure that the agency’s rationale for its

sampling methodology adequately justifies the “minimum” inspection effort.

No Consensus on Basis for Sample Sizes 

Opinions regarding the basis for the determined sample sizes vary among

agency staff responsible for developing and implementing the baseline

inspection program. OIG interviewed 7 of 14 members of the three ROP

development teams regarding the basis of the new ROP. All seven said the

ROP risk insights were provided by risk analysts' using historical knowledge,

but tempered by the team’s experience and past findings information. 

According to the team’s program manager, the idea was to look at only

samples having risk significance in order to achieve a balance between

having nothing to inspect and what has to be inspected. The team also

determined the expectation of “minimum” samples and developed data

regarding how long it took (hours) to inspect an area under the old inspection

system. After the first year of trial and implementation, the agency made

major revisions to all baseline inspection procedures at which point a range

of samples from minimum to maximum was provided to give flexibility and

satisfy regional managers. 

The majority of the development team members interviewed said there was

no statistical basis for the sample sizes identified for each baseline

inspection procedure. In fact, the program manager said the team “did not

get to” the population of activities in the inspectable areas to determine

[statistical-based] sample sizes because the population varies too much from

plant to plant. Other members said that:

C the team looked at how things were done in the past and that

information weighed into how many samples were enough; 

C the team developed "rough estimates" of the samples that could be

done in the estimated hours; and, 

C the sampling was based on expert judgment rather than any rigorous

statistical methodology. 
Audit of NRC’s Baseline Inspection Program

5 According to NUREG-1475, Applying Statistics, by D. Lurie and R. Moore, dated February

1994, “95% confidence” is a probabilistic measure of assurance.

9

One of the program developers indicated that there was some statistical

basis for the sample sizes, citing a 95 percent confidence5

 sampling

methodology from NUREG-1475, Applying Statistics. He added that it would

not be possible to do “random” sampling because inspectors would have to

do “something like 50 or 60 samples per inspection procedure.” A co-author

of the above-referenced book said that the concept cited (i.e., 95 percent

confidence level) does not apply to a risk-informed baseline inspection

program because the samples must be completely random. The underlying

concept of the risk-informed inspection program is that the program “intends”

to examine samples which have a greater probability of impacting safe plant

operations or, in essence, have a higher “priority.” In addition, inspectors

may be able to further inform their inspection sample selection (e.g., picking

“smart” samples) on higher priority areas known to have performance

problems at the plant in question or in other, similar plants. Therefore,

according to the statistics expert, one needs to recognize that the samples

selected by experienced inspectors are far from “random.” 

Opinions on the basis of sample sizes also vary widely among agency

managers responsible for implementing the baseline inspection program. 

For example, many of the senior executives OIG interviewed thought expert

panels developed the sampling basis. Others believed that the samples

were derived from old findings identified under the previous inspection

program, while still others had no idea how the sample sizes were developed. 

According to some of the managers, there was no scientific bases behind the

methodology; it was a "best guess" at what to look at. In fact, they said the

number of samples required is minuscule in many areas compared to the

number of licensee activities that could be sampled. 

Focus of the New Baseline Inspection Program

IMC 2515 states that the inspection activities [prescribed in each procedure]

and “minimum” sample sizes must be completed to provide an adequate

assessment of each cornerstone; hours are estimated for resource planning

only. However, there is some misunderstanding among senior managers

and inspectors in the field regarding whether the intended focus of the

inspection program is samples, hours, or inspection scope. This lack of

common understanding leads to an inconsistent application of the program. 

According to the development team program manager, the team focused on

samples (rather than hours) -- the concept being “here’s how many things to

inspect not how much time should be spent on those things.” Yet another

team member said risk information was tied to the hours needed to do

inspection work in that area, and a senior risk analyst on the team said the

focus was on content of the inspections and not on the sample sizes. The
Audit of NRC’s Baseline Inspection Program

10

analyst added that a well-defined scope of the inspection would determine

the hours needed, and the hours would then determine the sample size

needed to satisfy the defined scope. In fact, the analyst said that doing the

same amount of activity at different plants would not be appropriate, so exact

sample sizes would "not be the way to go." 

Similar to the development team members, there was no consensus among

agency senior managers or inspectors about whether samples or hours, or

both, are most important for assessing licensee performance. Generally,

those interviewed said the focus, right or wrong, is on samples. However,

the hours spent in an inspection area are considered by regional senior

managers as key to completing an inspection procedure. The lack of

common understanding of the actual intent, samples or hours, leads to

inconsistent implementation of the inspection program.

Adequacy of Sample Sizes to Assess Licensee Performance

Overall, those NRC staff interviewed said the minimum level of inspection is

adequate to assess a licensee’s programs. Yet, without giving any specific

reasoning, nearly every senior manager interviewed said that he/she expects

or wants inspectors to inspect more than the minimum number of samples

(e.g., use nominal levels of effort). The following represent typical comments

received from every level of management on the adequacy of sample sizes: 

< A regional administrator said he “wants” inspectors to do nominal

sample sizes, but recognizes there are resource constraints

preventing that. 

< Division directors said that something may get missed if the agency

continues to do just the minimum program. 

< A branch chief admitted a "concern about doing only the minimum

sample sizes" especially for more than a year. (OIG notes that the

minimum has been done for the last two inspection cycles.) 

< In addition, a “minimum” inspection program is adequate only if

certain caveats and assumptions are true, such as resident inspectors

are continually on-site. As one senior executive said, if resident

inspectors were not on-site continually monitoring licensee

performance, the program would not provide adequate information (at

the minimum level of oversight). 

< A division director said that whether the sampling is adequate to

assess licensee programs is a mixed answer, yes and no. Some

sample sizes are so small that even doubling or tripling them would

not be meaningful unless the population of the activity was

correspondingly small. 
Audit of NRC’s Baseline Inspection Program

11

< A branch chief said that the sample size in some procedures is

unrealistic for assessing overall performance. He cited the

application of one sample size regardless of the number of units at a

site (in other words, an inspector could fulfill the required sample size

by inspecting only one of multiple units at a plant). 

< Senior resident inspectors cited specific procedures where the

sample size is out of proportion to the population of activities. For

example, the operability evaluation procedure is "too light on

samples" (i.e., inspect 19-25 samples out of the 1,000-2,000

evaluations generated by the licensee) and a surveillance procedure

requires a sample of six out of a "very large" number of surveillance

activities. 

Use of Minimum Versus Nominal Sample Sizes

The premise of the baseline inspection program is that every plant receives

the same basic level of oversight with the flexibility of adding additional

oversight depending on NRC’s assessment of a licensee’s performance. 

However, the baseline inspection program lacks clarity about "when" to do

which level of sampling (minimum or nominal). In fact, there appears to be

no criteria, aside from available time, for when to do more than the minimum

under a routine baseline inspection. For example, one deputy regional

administrator said that while his inspectors have to do the minimum program,

they are free to perform as many samples as time will allow. Other

managers said that their inspectors are to perform just the minimum because

headquarters has determined that minimum is good enough. Yet, nearly

every senior manager interviewed said it would be preferable to have

inspectors use a nominal sample size if resource shortages do not prevent

that from happening. The number of samples inspected appears dependent

on available resources as opposed to performance-based. Without clear

criteria for when more than a minimum level of effort is necessary, NRC may

experience inefficient use of its inspection staff. In addition, contrary to the

ROP’s goal of reducing unnecessary burden on licensees, accommodating

inspections of more than the required minimum samples without a clear basis

can result in an extra burden. 

It is important to note that OIG is not questioning whether the sample sizes

are adequate, but rather that there is not enough support for the sample

sizes selected as the minimum necessary to adequately assess licensee

performance. The agency went to some lengths to have experts help

develop and design the baseline inspection program and the majority of

people interviewed believe the baseline inspection program is appropriately

focused on safety. Yet, there is no documented basis or sound rationale

behind the determination of sample sizes used to assess licensee

performance in the safe operation of their facilities. The agency also has no

documented rationale and or distinction of when (or why) to use more than a

minimum sample size at good performing plants. 
Audit of NRC’s Baseline Inspection Program

12

Summary

Agency managers have different opinions about the effectiveness of the

baseline inspection program, but most agree it is only effective with certain

assumptions and caveats in place. There are also differing interpretations

regarding whether the focus of the baseline inspection program should be on

the number of samples conducted or the hours spent in an inspection area,

or a combination of both. With or without a rationale for the sample size,

staff agreed that the adequacy of assessing licensee performance through

the baseline inspection program depends on the “quality” of the samples

chosen for inspection. 

The agency draws conclusions about licensee performance in particular

baseline areas based on a relatively small number of samples which are not

randomly selected. According to the agency's statistician, it is acceptable to

preselect the samples in a risk-informed program provided plant-specific risk

knowledge is being used effectively by inspectors. However, he added that

the agency must be clear that the conclusions drawn pertain to the samples

taken and should not necessarily be projected to the whole population. 

Without a documented basis for the sampling, the agency cannot determine

whether there is more (or less) inspection effort applied than necessary,

which impacts the agency’s use of its inspection resources and burden on

the NRC licensees. Additionally, NRC, the nuclear industry, and the public

cannot be assured that the agency’s justification and rationale for it's

sampling methodology is adequate to obtain sufficient inspection information

needed to assess the overall safety performance of power reactor licensees.

RECOMMENDATION

OIG recommends that the Executive Director for Operations (EDO): 

1. Document the basis and rationale for the determined baseline

inspection program sample sizes, including a discussion of when, or

why, to use more than minimum samples.

Operating Experience Program Needs Improvement

NRC made a commitment to use operating experience to maintain safety,

protect the environment, and promote the common defense and security. 

However, over time the agency made changes in the amount and type of

operating experience data it collects and analyzes, most notably in the area

of human factors information. As a result, operating experience information

that may be critical to NRC’s understanding of risks at power plants is being

lost. Because this lost information may be important to focusing inspection

work in risk important areas and assisting inspectors in sample selections,

the agency should reconsider changes previously made to the program.
Audit of NRC’s Baseline Inspection Program

6 Operating experience in this context includes a broad range of information about events and

conditions at commercial nuclear power plants collected from a variety of sources, including

but not limited to, Licensee Event Reports, 10 Code of Federal Regulations (CFR) 50.72

notifications, 10 CFR Part 21 reports, component failure data, and inspection reports, as well

as industry reports and foreign operating experience reports.

13

Historical Handling of Operating Experience at NRC 

NRC’s goal to maintain safety, protect the environment, and promote the

common defense and security is the preeminent agency performance goal. 

Although NRC licensees continue to have the primary role in maintaining

safe operation of the Nation’s commercial nuclear power plants, NRC states

it will take action to improve safety performance before it falls below

acceptable levels and will require plants to shutdown when their safety

performance is unacceptable. This principle is inherent in the NRC's new

ROP. Among the strategies NRC uses to achieve its performance goals is a

commitment to evaluate operating experience and the results of risk

assessments for safety implications.

Operating experience6

 is a broad term that has evolved to describe NRC and

licensee evaluation and use of operational safety data. It has long been

recognized that NRC’s systematic collection and evaluation of operating

experience data is an important part of its mission to ensure that commercial

nuclear power plants are operated in a safe manner. In fact, in July 1979,

the Office of Analysis and Evaluation of Operational Data was created to:

< coordinate operational data collection;

< systematically analyze and evaluate operational experience;

< feed back the lessons of experience to improve the safety of licensed

operations;

< assess the effectiveness of the agency-wide program; and

< act as a focal point for interaction with outside organizations for

operational safety data analysis and evaluation. 

Also in 1979, the nuclear industry took action and created the Institute of

Nuclear Power Operations, in part to provide an independent capability to

evaluate operational experience and feed back lessons learned to licensees. 

In 1998, as a result of further study, the Commission approved the staff’s

proposal for consolidating Office of Analysis and Evaluation of Operational

Data functions into program offices. The Commission noted that these

functions need to continue with a degree of independence. The 
Audit of NRC’s Baseline Inspection Program

7 Memorandum from C. Ader, Manager, Operating Experience Task Force to the Task Force Steering

Committee re: Reactor Operating Experience Task Force Report, dated November 26, 2003.

8 Memorandum to W. Travers from S. Dingbaum, Memorandum Report: Review of NRC’s Reactor

Operating Experience Task Force Report (OIG-04-A-13), dated March 30, 2004.

14

Commission also recognized the importance of sharing lessons learned from

the independent assessments of operational events with the nuclear industry

in an effort to improve the safety of licensed operations and to assess the

effectiveness of agency-wide programs. 

In 2002, agency executives raised questions regarding the effectiveness of

NRC’s operating experience program in the aftermath of a safety significant

event at a U.S. nuclear power plant. In response, NRC formed a task force

to evaluate the agency’s reactor operating experience program and to

recommend specific program improvements. The task force’s November

2003 final report7

 cited an earlier agency finding that a number of specific

NRC operating experience programs had been reduced in scope or

eliminated following previous program evaluations, but the impact of those

changes on the effectiveness of the operating experience program had not

been systematically assessed. The task force report made 21

recommendations for improvements to NRC’s operating experience program,

including periodic assessments of the program’s effectiveness.

Because of the extensive review performed by the task force, OIG limited its

work in the operating experience area. However, due to the importance of

operating experience to the baseline inspection program, OIG reviewed the

task force’s report. OIG found that, if implemented, the recommendations

presented will result in significant needed improvements to the operating

experience program and improve inspection efforts. OIG published the

results of its review of the task force report in a March 2004 memorandum

report,8

 which included six recommendations aimed at strengthening the task

force’s recommendations to ensure that existing weaknesses are addressed. 

Additionally, OIG noted that human factors operating experience was one

area not specifically addressed by the task force report. Therefore, during

this audit, OIG placed a particular focus on this important feature of operating

experience.

Loss of Human Performance Operating Experience Data

Humans are integral to the safe operation of a nuclear power plant. 

Understanding that human performance and error contribute to the root

causes that underlie performance problems in nuclear power plants, NRC

established human performance as an ROP cross-cutting issue. An NRCsponsored study on the risk impact of human performance in operating 
Audit of NRC’s Baseline Inspection Program

9 NUREG/CR-6753, Review of Findings for Human Error Contribution to Risk in Operating Events, Date

Published: March 2002 and NUREG/CR-6775, Human Performance Characterization in the Reactor

Oversight Process, Date Published: September 2002. Both studies were conducted for NRC by the

Idaho National Engineering and Environmental Laboratory.

10 The Commission recognized that some issues should be documented even though they could not be

evaluated under a specific cornerstone and its associated significance determination process. The

ROP contemplated that substantive cross-cutting issues could be documented in inspection reports. 

According to NRR, inspectors can, and do, document these types of findings (formerly called “no-color

findings”) but for clarity have given them a "green" color. IMC 0612 describes this type of finding and

what is required for inspectors to be able to document them in NRC inspection reports. 

15

events confirmed this position. Additional work documented how the ROP

monitors, analyzes, and feeds back information on human performance and

compares it to findings from the review of operating events.9

 

The NRC-sponsored study concluded that, in general, the ROP is likely to

capture important issues through a combination of baseline inspections,

supplemental inspections, performance indicators, cornerstones, and

cross-cutting issues and that the ROP identifies the same-type of human

performance issues that were identified through analyses of operating

events. The study resulted in a number of findings regarding the influence of

human performance on the analyzed sample of 37 significant operating

events, including the following ones considered most important to

probabilistic risk assessment:

C Without exception, the 37 operating events included multiple

contributing factors. On average, the events contained four or more

human errors in combination with hardware failures. Fifty percent of

events contained five or more errors.

C Eliminating “no color findings10” because they are of low risk

significance may make current practices more objective but may

result in the unavailability of raw data if many human performance

insights individually fail to trigger the significance determination

process.

C Trending low safety significant problems and issues can potentially

indicate declining plant performance. Trends in human performance

below established thresholds may be an indicator that is predictive of

future systems performance. 

C Human errors can likely result in failure of risk-significant equipment. 
Audit of NRC’s Baseline Inspection Program

11 Latent errors are those committed pre-event but whose effects are not realized until the event occurs. 

Latent conditions can be present for long periods of time before combining with workplace factors,

including active errors, to produce an event. Latent errors contributed most often to plant events and

caused the greatest increases in plant risk.

12 These types of findings were formally referred to as “no-color findings.”

16

C Latent errors11 were present in every event and more predominant

than active errors. 

According to an NRC specialist in the correlation of human factors to

performance problems at commercial nuclear power plants, in the time

between the two sets of data discussed above the content of inspection

reports was changed due in part to changes in the definition of the threshold

for what constitutes a finding. As a result, the number of findings in

inspection reports was reduced. Although cited as an important source of

information about human performance, findings that are not greater than

minor significance12 have been essentially eliminated from inspection reports,

and thereby, unavailable for trending by agency analysts.

Summary

Research has established human performance as a significant factor in

performance problems at commercial nuclear power plants. In fact, an NRC

human factor’s specialist said that the agency’s preliminary review of recent

information again shows the correlation between human error and poor

performing plants to be positive with a range from +0.3 to +0.6 (on a scale of

-1.0 to +1.0), and an average of +0.5. NRC awareness of human

performance allows for informed decision making about changes that may be

needed in NRC's inspection guidance and contributes to appropriate focusing

of inspection activities for each plant. However, changes made to the

threshold for inspection report content and a reduction in licensees’ event

reporting requirements resulted in a decrease of NRC’s access to important

operating experience information related to potential performance problems

at commercial nuclear power plants. This reduces the staff’s ability to

compile and evaluate human performance data in order to trend performance

issues at the plants.

RECOMMENDATION

OIG recommends that the Executive Director for Operations: 

2. Develop guidance on how to identify human performance trends and

how that information should be integrated into the reactor oversight

process. 
Audit of NRC’s Baseline Inspection Program

13 OIG Report 02-A-15, Review of NRC’s Significance Determination Process, dated August 21,

2002.

17

Agency Lacks Clear Guidance on Handling Informal Issues 

Informal issues are generally viewed as a very important part of NRC's

inspection effort and, as such, NRC managers and the inspection manual

encourage inspectors to discuss this information with licensees. However, as

noted in a 2002 OIG audit report related to the reactor oversight process,13

NRC had not developed written guidance for informal issues. That remains

the case today. Individual regions have provided some guidance about the

handling of informal issues and several managers stated they provide verbal

guidance to inspectors. However, such instruction may be inconsistent and

management expectations are not clear. This results in an important part of

the inspection program being implemented inconsistently. Important

information about plant operations can be lost because of inconsistent or

non-existent documentation. As a result, NRC could miss opportunities to

identify declining plant performance and the public has no access to

information on informal issues considered important to inspectors.

Management Accountability and Control

According to Office of Management and Budget (OMB) Circular No. A-123,

Management Accountability and Control, agency managers are encouraged

to continuously monitor and improve the effectiveness of management

controls associated with their programs. Clear guidance helps to ensure

effective and efficient use of valuable agency resources and consistent

regulation of NRC licensees. 

Agency Guidance

NRC has an established policy regarding the threshold by which inspection

findings are considered significant. IMC 2515 also (1) provides guidance for

its inspectors on inspection report content and (2) conveys that licensees

desire to hear inspector insights, even in instances where the insights do not

reach the threshold for documentation in inspection reports. Staff clarified

that this type of information, typically called “inspector observations” or

“informal issues,” relates to issues that inspectors see as a concern or about

which they want a licensee to know, but do not meet the threshold for

inclusion in inspection reports. Because such information falls below the

prescribed threshold, IMC 2515 does not provide guidance regarding the

handling of informal inspection issues. Nevertheless, varying levels of

attention are being spent by agency managers and inspectors on informal

issues.

NRC inspectors document violations of NRC regulations (findings) in

inspection reports. But, inspectors also identify other informal issues

regarding plant operations which are included with “no-color findings” which

typically get eliminated from inspection reports. NRC and licensee officials
Audit of NRC’s Baseline Inspection Program

18

stated that they view informal issues as very important to improving plant

operations. In fact, inspectors are encouraged to bring informal concerns

about plant operations to licensee management, but these concerns cannot

be included in inspection reports because they fall below a defined threshold

of importance. Despite the importance and widespread use of informal

issues, NRC has not provided adequate guidance because cognizant senior

managers believe that issues which fall below the threshold for inclusion in

inspection reports do not require formal guidance. As a result, the handling

of informal issues has been inconsistent.

NRC inspectors regularly discuss informal issues with licensee officials and

with NRC managers. Many NRC inspectors and managers believe in the

importance of these informal issues, specifically that they can prevent future

violations. In order to ensure licensees resolve the “informal” concerns

properly, most inspectors and some region managers track these issues. In

fact, most of the inspectors and many of the managers interviewed said a

“second set of books” is kept which captures this information. In addition,

some inspectors and managers analyze informal issues to identify trends that

could reflect potential problem areas at plants. For example:

< An NRC inspector performing a Health Physics inspection discussed

a specific situation with the licensee who then put the concern into its

corrective action program. However, this issue won’t be in an NRC

inspection report because it does not meet the criteria for inclusion. 

According to the inspector, if he had not seen the situation and

mentioned it, the result would have been a violation. The significance

was minor -- there was no violation, but there would have been a

violation if not identified by the inspector.

< According to one NRC manager's informal tracking system, an

inspector recognized many examples where the licensee had not

identified the appropriate equipment to be protected during various

maintenance activities. The issues were discussed with the licensee

who was very appreciative and intended to issue a general

communication about the problem.

Licensees Encourage Information Exchange

Licensee officials stated that they appreciate NRC inspectors bringing

informal issues to their attention and that they take action to correct the

issues in virtually all cases. Many licensee officials viewed discussion of

informal issues as being proactive and predictive of potential performance

problems at their plants. In fact, several officials viewed informal issues as

more important to improving plant operations than formal inspection findings

in NRC inspection reports. Some of the licensees said they would not have a

problem with NRC capturing informal issues in the formal inspection reports,

as long as the issues are appropriately characterized.
Audit of NRC’s Baseline Inspection Program

19

Inconsistencies in Implementation and Expectations

During this audit, OIG informed NRR and region managers about

inconsistencies related to handling informal issues. For example, some

region managers specifically tell their inspectors not to document or track

informal issues while others have developed elaborate tracking systems and

encourage their staff to keep records. OIG also found that, despite not

meeting threshold requirements, one region actually documents discussion

of minor, “informal” issues in their formal baseline inspection reports. 

Given their awareness that inspectors and/or managers are developing

tracking systems and spending time collecting and documenting informal

issues, OIG asked why NRR has not issued guidance or communicated

expectations related to handling informal issues. One NRR senior executive

said that when he asked staff about the handling of informal issues, he was

repeatedly told that the ROP would eventually deal with the issue. But,

according to staff who have a vested interest in the development of the ROP,

the topic has yet to be discussed because of the relative newness of the

ROP. 

In disagreement that the ROP, after four years, is too new, a number of

agency senior managers expressed that it is an appropriate time to do a

critical reassessment of the ROP, including the handling of less than

reportable issues. According to one cognizant manager, it is important to

look at the issue now to answer the primary question of "how much effort

should be put into these minor things?" He acknowledged that some of the

minor issues discussed between inspectors and licensee staff would be

considered operating experience, albeit specific to an individual plant. But,

that operating experience will not get captured on the operating experience

websites if the issue does not meet the risk significance threshold, and may

or may not get shared with other inspectors. In his opinion, NRR will have to

look at the resources already expended on informal issue tracking systems in

each region and get each region's expectations before deciding on what

further guidance is needed. Despite the existence of guidance in the agency

inspection manual regarding thresholds, this important part of the inspection

program continues to be inconsistently implemented. 

In the finding titled, Operating Experience Program Needs Improvements,

OIG noted that potentially important information about operating experience

was no longer documented and preserved in NRC inspection reports. 

Because there is no continuity in how informal data is captured or shared,

important information about plant operations which could help inspectors pick

“smart” samples could be lost. As a result, NRC could miss information

critical to NRC’s understanding of risk and identification of declining plant

performance (trends). 
Audit of NRC’s Baseline Inspection Program

20

Certain Information Not Publicly-Available

One of the agency’s goals is to increase public confidence through open

communications. Typically, the public should have access to enough

information to understand how the agency makes decisions, including its

annual rating of a licensee’s performance. However, informal issues and

inspector observations are typically shared with licensees during informal inplant discussions or before/after formal inspection exit meetings. Therefore,

the public is not privy to certain information regarding licensee performance. 

As a result, there may not be enough documentation in inspection reports to

gain public confidence on NRC’s assessment of licensee performance.

According to one senior manager, the baseline inspection program produces

two levels of information: formal inspection findings which are documented

in the reports, and a second tier of informal observations and minor issues

which do not rise to the threshold for inclusion in the reports. He thinks that if

the inspectors (or their management) think any of the "minor" issues might

need follow-up, that information should probably be documented somewhere,

but he is not sure the public needs to be aware of it. The questions of where

informal issues should be documented and to what the public should have

access need to be answered. 

Summary

NRR management is aware of the inconsistencies related to the

documenting and sharing of informal/minor issue information. NRR is also

aware that the public is not privy to a lot of the informal issue information

shared with licensee staff during exit meetings and through other informal

discussions, even though some of this information would be considered

operating experience. NRR managers recognize the need to address this

issue, but before deciding on what guidance is needed, NRR must

determine: 

< how much effort should be put into "these minor things," 

< what resources are already being used in each region, 

< what are each region's expectations regarding internal use, and 

< what information should be documented for public view (e.g., in

inspection reports).

RECOMMENDATION

OIG recommends that the Executive Director for Operations:

3. Develop and implement guidance for documenting, tracking, and

trending informal inspection issues.
Audit of NRC’s Baseline Inspection Program

21

Performance Measures Need Improvement

NRC’s methodology for reviewing the success of the baseline inspection

program is generally sound. However, although baseline inspections are

used to assess whether plants are safely operated, the agency does not

have performance measures which adequately assess the quality of those

inspections or define “effectiveness” of the program.

Guidance from the agency and OMB states that the “effectiveness” of a

program must be measured. Performance measures established in NRR's

Operating Plan for the baseline inspection program do not meet the intent of

guidance because there is no method established to measure

“effectiveness.” An NRC group previously identified the lack of a definition or

method to measure effectiveness as a weakness in the agency’s ability to

assess a program’s success, in this case, the baseline inspection program. 

In addition, the agency lacks a metric regarding the “quality” of its baseline

inspections. Without such measures, the agency cannot adequately

evaluate the effectiveness of baseline procedures nor the quality of the

inspection program itself. As a result, NRC cannot be assured that the

baseline inspection program is adequately contributing to the agency's

mission and, therefore, the Commission does not get a reliable annual

assessment of overall ROP effectiveness.

Performance Goals and Measurement System

A primary objective of the ROP is to help fulfill the agency’s performance

goals, including efficiency and effectiveness of its activities. A performance 

measurement system is a process organizations use to collect data, prepare

performance measurements, analyze performance, and report on and use

the results. Those measures should reflect and align with the mission of the

agency, identify expected outputs, identify what aspect of performance needs

measuring, and establish performance goals or targets. Outputs include

measurable characteristics of program activities such as effectiveness,

efficiency, and quality. (For a detailed description of performance and output

measures, see Appendix B.)

Guidance Requires Measure of Effectiveness 

Guidance from the Office of the Executive Director for Operations’ states:

A major area targeted for improvement in the FY 1999

operating plans is the establishment of effectiveness metrics. 

All other metrics are superfluous if the program is not

achieving its desired results or outcome. Effectiveness is also

at the heart of self assessment. Therefore, effectiveness

metrics are a required element for the FY 1999 operating

plans.
Audit of NRC’s Baseline Inspection Program

14 NRC defines a Planned Accomplishment as a level of discrete activity below the program level, where

FTE and budgetary resources can be identified.

15 OMB Circular No. A-11, Preparation, Submission, and Execution of the Budget, revised July 2004.

16 NUREG-1614, Vol. 3., U.S. Nuclear Regulatory Commission Strategic Plan FY 2004-FY 2009, dated 

 August 2004.

22

As early as October 1998, NRC established guidelines for internal output

measures for Planned Accomplishments.14 A model was provided which

suggested establishing metrics for:

‘ effectiveness [the accomplishment of the desired result]

‘ efficiency [accomplishing the desired result without wasting

resources]

‘ quality [degree of excellence]

‘ quantity [how many]

‘ timeliness [performance of the activity within defined milestones]

For the baseline inspection program, there is one Planned Accomplishment

Code, #103-140 - Baseline Inspections. NRC offices were required to

establish, at a minimum, an effectiveness metric at the Planned

Accomplishment level if there were no performance plan output measures for

that accomplishment. In fact, for the period subject to this review, there was

no performance plan output measure specifically defined for effectiveness in

the baseline inspection program. 

In the President’s budget for FY 2004, OMB describes its initiative to

implement performance budgeting throughout the Federal government in

2005 by using “effectiveness” at influencing outcomes as a justification for

resources.15 As OMB describes for budgeting purposes, NRC must link the

performance of day-to-day activities, like baseline inspections, to long-term,

higher level performance goals in order to build a solid foundation for

managing program performance and achieving performance and strategic

goals. 

Measure of Effectiveness Absent from Agency Plans

NRC’s Strategic Plan presents strategic goals which are consistent with its

mission and supported by performance goals. The performance goals

represent outcomes the NRC plans to achieve. NRC’s FY 2004-2009

Strategic Plan, Performance Goal 116 is Ensure protection of public health

and safety and the environment. There are several performance or outcome

measures associated with this goal which need improvement. For example,

in accordance with OMB’s guidance, one of NRR’s Operating Plan
Audit of NRC’s Baseline Inspection Program

17 IMC 0307, Reactor Oversight Process Self-Assessment Program, dated December 12, 2003.

23

performance measurements asks, “What is the tangible contribution to the

outcome goals... in the core activity from doing this work. (What will success

look like or what will we measure to know that this activity was successful? 

How do baseline inspections contribute tangibly to the agency’s outcome

goals?)” NRR’s answer does not provide specific quantitative terms for

measuring the baseline inspection program’s tangible contribution to the

outcome goals, nor does it describe what should be measured to know that

baseline inspections were effective. 

At the operational level, management should include, when possible,

measures for each of the performance categories; effectiveness, efficiency,

quality, quantity, and timeliness. It is important to have clear drivers or

targets in each area to ensure the program effort is not “pushed” in any one

direction at the expense of others. Specifically, the NRR FY 2004 Operating

Plan describes the planning, budgeting, and performance management

process as implemented by NRR. The process is directed toward achieving

organizational effectiveness by measuring and monitoring performance

against measures and targets, and assessing performance. 

The Operating Plan defines success for all planned accomplishments and

includes planning templates for each accomplishment discussed in the plan. 

Each template establishes the measures of success and targets for NRR

management. OIG reviewed the baseline inspection program's performance

measures and targets and found there are no metrics established to

adequately measure effectiveness or quality. Appendix C presents a table

of OIG's evaluation of the specific leadership and operational performance

measures and targets. In addition, other inconsistencies were found in the

baseline inspection program which are fully discussed in Appendix B. If

inconsistences are not addressed, performance measures and targets may

not be developed in alignment with program objectives.

Efficiency Focus Group 

In 2002, an agency Efficiency Focus Group looked for ways to improve

resource efficiencies in the ROP and concluded that there could be no

comprehensive assessment of the effectiveness of the baseline inspection

procedures until “effectiveness” was defined. The focus group recognized

that, aside from the number and significance of inspection findings, there are

a number of less tangible effects of inspections that will be difficult to

measure directly. The focus group also noted that IMC 0307, Reactor

Oversight Process Self-Assessment Program,17 which was then under

development, would require an annual review of the “effectiveness” of

baseline inspection procedures. The focus group recommended that 
Audit of NRC’s Baseline Inspection Program

18 An “inspectable area lead” is a Division of Inspection Program Management, NRR, staff member

assigned responsibility to oversee and manage the use of individual baseline inspection procedures

or attachments to those procedures.

19 For CY 2003, there were approximately 11 different NRR staff members serving as individual area

leads.

20 The annual review did not include the physical protection procedures which were covered under a

current NRC temporary instruction.

24

measures be developed once “effectiveness” is defined so that an

assessment could be made. However, the group did not attempt to define

“effectiveness.”

Incomplete Annual Review of Baseline Inspection Procedures 

Subsequent to the Efficiency Focus Group’s comments, IMC 0307 was

issued and currently requires certain staff, called inspectable area leads,18 to

monitor and report annually on the effectiveness and efficiency of inspection

activities and program objectives within their assigned areas of responsibility. 

One of the primary objectives of this annual review is a critical ‘big picture’

evaluation of “all of the inspectable areas together to justify retaining them in

the baseline inspection program, or determine if the addition of a new

inspectable area is warranted.” 

In accordance with IMC 0307, multiple inspectable area leads19 recently

completed the CY 2003 annual review of their assigned baseline

procedures.20 As with previous reviews, they did not identify the need for any

significant changes to the overall inspection program (e.g., either the addition

or removal of inspection procedures). In fact, the staff mainly recommended

minor adjustments or “tweaks” to levels of effort (i.e., hours), clarification of

the definition of a sample, or how many samples constitutes completion for

procedures that received any revision. However, there is no evidence that

the required ‘big picture’ review was conducted or, according to many senior

officials, that operating experience is being incorporated into changes to

inspection procedures as anticipated.

As noted by the focus group and as seen in the NRR Operating Plan,

references to “effectiveness” appear in individual inspection procedures. Yet

the agency has not defined what effectiveness means for the baseline

inspection program. As a result, the agency cannot properly evaluate the

effectiveness of the procedures, and thereby, the program. When senior

NRR executives were asked why there was no definition of “effectiveness,”

one said this issue has been on his radar because he too realized the

“effectiveness” of the baseline inspection program had never been

measured. Another said that interim graded measures will be put into effect

as a first step, but NRR needs to get closer correlation between its operating

plan and NRC’s strategic goals.
Audit of NRC’s Baseline Inspection Program

25

Lack of “Quality” Measures in the Baseline Inspection Program

In addition to effectiveness, appropriate “quality” measures should be

developed. Monitoring only the other types of measures and targets can

drive performance in another direction at the expense of quality. For

example, there is no measure showing whether inspectors do quality

inspection work, a critical component of baseline inspection program

success. Such a measure might, for example, reflect the quality of

inspection samples chosen.

IMC 0307 also includes the inspection program metrics used in the annual

ROP self-assessment. There are 11 inspection program metrics in IMC 0307

which go beyond those in the Operating Plan. However, IMC 0307 also has

no measures of effectiveness or quality. 

Summary

The existing measures and targets for the baseline inspection program

capture some important aspects of program activity and provide some useful

information to managers. However, management has not adhered to

performance measure requirements in that critical types of measures are

missing. Specifically, the existing measures and targets do not reflect the

most important aspects of evaluating the activity — effectiveness and

quality. Without such measures, agency managers cannot adequately

evaluate the effectiveness of baseline procedures nor the inspection program

itself. Managers also cannot use the existing performance information to

most effectively manage resources and ensure optimal performance in

attaining the agency’s goals. As a result, NRC cannot be assured that the

baseline inspection program is effectively contributing to the agency's

mission.

RECOMMENDATION

OIG recommends that the Executive Director for Operations:

4. Define “effectiveness” as it pertains to the ROP and establish

performance measures and targets to demonstrate that the baseline

inspection program meets that definition.

B. IMPACTS ON THE BASELINE INSPECTION PROGRAM FROM RESOURCE

CHALLENGES ARE LIKELY TO CONTINUE

NRC must allocate adequate, qualified staff to achieve desired program

outcomes. Over the last two years, NRC has faced numerous challenges to

completing the baseline inspection program. According to senior staff, the

primary cause for the current challenges began with a change to the

inspector staffing policy. Although OIG’s analysis did not support that belief,

a number of other tangible factors impact the agency’s ability to carry out the
Audit of NRC’s Baseline Inspection Program

21 The revised policy did not impact single unit sites which continued to require two resident inspectors.

26

baseline inspection program. Specific challenges include: a hiring policy

change, an increase in supplemental inspections, a loss of qualified

inspectors, and diversion of inspectors from baseline activities. In response

to these challenges, the agency instituted coping strategies including 

changing its expectations for completing the baseline inspection program

from a nominal to a minimum level of effort. While NRC appears to be

addressing these challenges, they persist today and undesirable “short term”

coping measures are still in place. Because these challenges significantly

impact a critical agency program, it is imperative that continual senior

management attention be given to the possible negative impacts of the longterm use of short term coping strategies. 

The Move from N+1 to N

In a memorandum dated January 11, 2000, the Commission approved a

revision to the inspector staffing policy to reduce the number of resident

inspectors at operating commercial nuclear power plants having multiple

units on site from “N+1" (i.e., the number of units plus one additional

inspector) to “N” (an equal number of inspectors to units at a site).21 The

change was temporary pending completion of an impact study due in June

2001. The June study concluded that baseline inspections could be

accomplished by "N" resident inspectors with some assistance from regionbased inspectors. As a result, the staffing reduction at plant sites was

continued, but the staff proposed the development of criteria for the

allocation of additional inspection resources for other than performancerelated issues (e.g., unique design or organization features). 

Staffing Policy Changes

Many agency managers and inspectors assert that the challenges to

completing the baseline inspection program began with the change in staffing

from N+1 to N. Using time charges for CY 2002, OIG tested this theory and

found no evidence that hours allocated for completion of the baseline

inspection program are inadequate. To justify an argument for a return to

N+1 staffing at all plants, analysis should show that significantly more than

two FTE (i.e., closer to three FTE) are needed to complete resident

inspector-performed baseline inspection procedures at plant sites. To the

contrary, the data indicates it took fewer than two FTE to complete the

program, including assistance provided by other NRR staff. Even on a plantspecific basis, no plant in CY 2002 required enough FTE to justify a return to

N+1 in order to complete the routine baseline inspection program. 

Additionally, a rough estimate of travel costs related to providing inspector

assistance to sites provided no support for a return to N+1. (OIG notes that

over the last three years, the agency has granted approval for at least three

plants to return to the N+1 policy due to unique unit designs or performance

issues.)
Audit of NRC’s Baseline Inspection Program

27

An alternative argument could be that challenges will persist and that an

additional inspector at plant sites would relieve the burden of applying coping

strategies to deal with existing resource challenges. However, OIG’s

analysis shows that any additional FTE would be better positioned at the

regions because the additional coverage needed to complete resident

inspector-performed inspections at plant sites is far less than one FTE. 

Analysis shows that, if there are two full-time resident inspectors at each

plant site, as required, baseline inspection procedures can be completed at

the minimum level. Interviews with resident inspectors support the position

that there is more than sufficient time for the baseline inspection program

with the caveat that inspectors are not continually diverted to other activities

(such as, temporary instruction inspections, allegations, and enforcement

activities). 

Challenges to Completing the Baseline Inspection Program

NRC revised its personnel policy for FY 2002 regarding new hires. According

to agency documents, the revision called for one of every four (25 percent)

new hires to be entry level, “recent [college] graduates.” Therefore, the new

hires will have little to no field inspection experience. As one senior manager

said, having such a policy will continue to place a burden on qualified

inspectors because it takes new hires approximately two years to process

into fully qualified inspectors. Additionally, the agency notes that it should be

recognized that “any” new inspector, including those with extensive field

experience, requires some time to understand the NRC’s regulatory role and

management expectations for professionalism in the conduct of inspector

functions. According to NRC, the average time to qualify experienced

personnel is one year. 

In a July 2002 memorandum to the regions, NRR discussed challenges

faced by the regions to complete the baseline inspection program. These

challenges included an unexpected increase in supplemental inspections, the

loss of qualified inspectors, and diversion of inspectors to other initiatives. 

Multiple “coping measures” were provided for consideration, including: 

< deferral of biennial/triennial inspections, 

< increased use of inspectors qualified at the "basic" level,

< delay of personnel moves, 

< deferral of non-required training, and

< utilizing the minimum sample size in each inspectable area.

 

Responses from the regions to headquarters identified the possible negative

impacts of the proposed coping strategies, especially those associated with

reducing the target sample size to minimum (low end) at most plants. For
Audit of NRC’s Baseline Inspection Program

28

example, a Deputy Regional Administrator reported that it would be difficult

to judge what effects the across-the-board reduction in samples would have

(e.g., what problems might be missed, if any) from the reduced effort. He

contended that if done year-after-year, it would erode the agency's ability to

obtain adequate indications of licensee performance and to identify risksignificant issues. In his words, there was a need to recognize "one-time

compromises as just that — measures not suitable for repetition." 

Although all four NRC regions were challenged to complete the baseline

inspection program in CY 2002, two regions were particularly hard pressed

which led to a change in intended performance. For example, one region

intended to continue with nominal sample sizes at six nuclear plants due to

poor performance. Resource shortages eventually led that region to reduce

its level of inspection effort at those six plants and to implement the minimum

level of effort at its other assigned plants. In addition, the subject region had

to request 13 weeks of inspection assistance to meet the minimum

completion requirements. A second region was extremely challenged by a

shortage of staff and fully-qualified inspectors, increased supplemental and

special inspections, and diversionary activities involving problem plants. In

fact, that region requested 46 weeks of inspection assistance to complete the

minimum level baseline inspection program. 

In order to provide assistance to the regions in need, another region

implemented most of the proposed coping measures, particularly the

reduction of inspection effort to minimum levels at all its assigned plants. In

that case, poor performance and risk did not impact the decision to reduce

the level of effort to the minimum; instead it was based on resource

shortages. Nevertheless, in a memorandum to staff regarding management

expectations for the CY 2002 cycle, a region branch chief said the intent and

scope of the inspection procedures was to be met and that “a reduction in

samples and hours does NOT mean you should select the easiest or least

time consuming sample.”

End of CY 2002 Cycle Realizations

In a December 2002 memorandum, headquarters conceded that a number of

unanticipated events occurred during the CY 2002 cycle that would continue

to impact the baseline inspection program through CY 2003. The four

regions responded to the headquarters memo with similar messages;

specifically, that they were only able to complete the CY 2002 cycle by

implementing coping strategies. As stated in a region memorandum,

completion in CY 2002 required "a number of less-than-desirable" coping

measures that would need to extend into CY 2003 due to anticipated

continued shortfalls of inspection resources (from an FTE or technical skill

perspective).
Audit of NRC’s Baseline Inspection Program

29

Agency Attempts to Address Challenges

In an effort to alleviate some of the challenges to complete the baseline

inspection program in CY 2003, the agency took a number of actions, most

notably, double-encumbering of inspector positions, continuation of the

minimum level of inspection effort, and the identification of former inspectors

available from headquarters to assist with baseline inspections. 

Double-encumbering

IMC 2515 was revised in April 2003 to give regional administrators the

authority to "double encumber" the resident and senior resident inspector

positions at operating commercial nuclear power plants. This limited "early

reassignment" was expected to minimize the number and duration of gaps in

inspector assignments and site coverage due to personnel changes. The

new policy allows for assignment of a permanent resident inspector up to 12

months before the planned departure of the incumbent and 6 months for

senior resident inspectors. NRR appropriately cautioned the regions that

close management be exercised to ensure that resources expended to

complete the baseline inspection program were not increased solely as a

result of the additional inspectors on site. 

In practice, double-encumbering is only effective when there is advance

notice of a proposed vacancy. However, according to NRR, resident

vacancies frequently occur with little notice. For example, an inspector

leaving for a promotion may give only two weeks notice. As a result, there is

little opportunity for the regions to implement double-encumbering to address

gaps in resident vacancies and, thereby, assist in completing the baseline

inspection program.

Continuation of Minimum Level of Effort

In a May 2003 memorandum to the regions, NRR established guidance for

completion of the CY 2003 baseline inspection program stating that

completion of the minimum baseline inspection program is a "fundamental

annual objective in support of the [NRC] strategic goal of maintaining safety." 

To ensure the program could be completed, NRR specifically requested the

four regions to perform only the minimum samples, wherever possible, based

on the safety performance at each plant. Headquarters recognized that

some inspectors believed that more than the minimum was warranted at

some plants but cautioned that any additional inspection effort should be

"judiciously applied and focused in those specific areas only where needed." 

In addition, headquarters requested the regions to continue with similar

coping measures to those employed during CY 2002. 

In response, two regions again identified a need for assistance to complete

the minimum baseline inspection program primarily due to a shortage of

qualified inspectors and diversion of inspectors to non-baseline activities. 

Despite applying numerous coping strategies (e.g., reduced preparation and
Audit of NRC’s Baseline Inspection Program

30

documentation effort, deferred training, delay of “full” inspector

qualifications), these regions projected a need for between 43 and 120

weeks of outside assistance to complete the minimum baseline

requirements. Both regions stated a preference for individuals with full or

previous qualifications and those with knowledge of the ROP.

Identification of Former Inspectors

Headquarters responded to the regions’ requests for assistance by providing

a chart of former and current-certified inspectors (currently in positions within

NRR) and what level of qualification they retained. NRR intended that these

individuals would provide help in completing baseline inspection programs

where needed. However, some of the identified inspectors were qualified at

only the “basic” level. The use of “basic” qualified inspectors helps the

agency meet the criteria for site coverage, but the practice places an extra

burden on senior resident inspectors because headquarters instructed that

inspectors having “basic” certification could only inspect under “close

supervision.” Therefore, there is not a 1-for-1 payback from this type

assistance because “basic” certified inspectors cannot independently conduct

baseline inspections. 

Commission notified of impediments to completing the program

In the last quarter of the CY 2003 inspection cycle, the EDO notified the

Commission that the loss and shifting of resources impacted the agency's

ability to have two qualified resident inspectors at each site. As a result,

upon evaluation of inspection resource challenges in future years, the FY

2004-2006 regional budget for plant specific inspection activities was

increased by 15 FTE over the FY 2003 budget. The EDO anticipates that

this additional regional FTE will alleviate resource challenges once the new

staff become a pool of fully qualified inspectors who can relieve resident

inspectors from diversionary activities, such as assisting in supplemental

inspections. However, filling the new positions may prove difficult given the

agency’s recent experiences with the inability to retain the existing

inspectors. For example, 27 of the 74 resident inspector positions were

vacant at some point during CY 2003 and 14 senior resident inspectors left

the program in CY 2003 for various reasons, including promotions,

reassignments, and retirement. In addition, the full benefit of the new FTE

will not be realized until they are fully qualified as inspectors which takes

approximately two years. 

Summary

The regions experienced numerous resource challenges in completing the

baseline inspection program in CYs 2002 and 2003. Coping strategies,

encouraged by headquarters in CY 2002 as one-time measures, continued in

CY 2003. The agency anticipated some relief to the resource challenges by

allowing more flexibility in use of "basic" qualified inspectors and early

assignment of new resident inspectors. Yet the value of both measures was
Audit of NRC’s Baseline Inspection Program

31

limited because there is not a 1-for-1 payback in conducting and completing

baseline inspection procedures using “basic” qualified inspectors and there is

little opportunity to implement the double-encumbering policy. 

The agency anticipates that the allocation of an additional 15 FTE to

inspection activities will alleviate some future problems in meeting the

baseline inspection program by relieving resident inspectors of diversionary

activities. But, the full benefit of a new, fully qualified inspection FTE will not

be realized for at least 18-24 months after those positions are filled. 

Because regions have had a difficult time keeping vacancies filled, despite

aggressive recruiting, challenges will likely persist until FY 2006. The

persistent use of coping strategies and continual shifting of resources that

results from the shortages warrants close management attention. Without

sufficient management oversight, pressure on completing the program could,

among other things, result in:

S inspector “burnout,”

S delayed inspector qualifications (which continues the burden on fully

qualified staff),

S choosing easier rather than “smart” samples to meet the minimum

program, and 

S deferrals of biennial and triennial inspections to sometime later in the

required time frame (which could, according to a region senior executive,

result in some areas going nearly four or six years before being

inspected).

OIG notes that the actual resources needed for completion cannot be

determined until the size of the program is established and justified. 

RECOMMENDATION

OIG recommends that the Executive Director for Operations: 

5. Develop a planned, scheduled approach for phasing out the use of

the coping measures.

C. THE RESIDENT INSPECTOR TRAINING PROGRAM NEEDS IMPROVEMENT

NRC requires specific training for baseline inspection program inspectors

and there is a comprehensive inspector training program in place. However,

there is room for improvement. Specifically, (1) inspectors are occasionally

required to inspect areas outside of their comfort zone, (2) inspectors are

occasionally assigned to plants outside their specific training, and (3) there is

no requirement for inspectors to receive vendor-specific training. As a result,

the agency may not be maximizing the efficiency and effectiveness of its
Audit of NRC’s Baseline Inspection Program

32

Inspector at work

(Photo courtesy of NRC Website)

resident inspectors and those inspectors may miss opportunities to identify

performance problems.

Background

IMC 1245 defines the training requirements for NRC

inspectors. The objective of the program is to ensure

that NRC staff have the knowledge and skills necessary

to successfully implement the agency’s inspection

programs, including the baseline inspection program. 

According to guidance, inspectors must understand the

facilities, equipment, processes and activities of the

program, as well as the criteria, techniques, and

mechanics of implementing the program. 

There are two levels of inspector qualifications: “Basic” and “Full.” A Basic

Inspector Certification allows an inspector to perform limited scope inspection

activities under an appropriate degree of detailed supervision (from fully

qualified inspectors). Fully qualified inspectors can independently perform

the full scope of inspection activities with only routine oversight and

supervision.

Every inspector is expected to understand the technology associated with

their assigned plant and apply concepts in various technical areas to allow

the NRC to carry out its overall responsibilities. Inspectors need to develop

and maintain an understanding of how basic nuclear plant design and

operations provide for protection of public health and safety, and use

knowledge of a specific reactor type to identify, address, and resolve

regulatory issues. In order to receive “full” certification as a resident

inspector, staff are required to take a Reactor Full Series course for one

nuclear technology — either boiling-water reactors (BWR) or pressurizedwater reactors (PWR). 

Inspector Assignments 

The agency’s intent is that inspectors will be assigned to plants which match

the training they have received due to significant differences between the

nuclear technologies. In fact, the majority of resident inspectors have had

the technology training for their assigned plants, as intended. However,

there are at least six instances where an inspector is mismatched to the

technology at the plant to which they are assigned, four of which are senior

resident inspector assignments. For example, a permanently assigned

resident inspector had not received technology series training; a reactor

engineer on a 6-month rotation as a resident inspector at a BWR plant had

not taken the BWR simulator training class; and the senior resident inspector

at a PWR plant is a BWR-trained inspector. 
Audit of NRC’s Baseline Inspection Program

33

Resident inspectors are required to have a general knowledge of plant

operations and systems in order to perform baseline inspections. However,

there are a number of baseline inspection procedures, in areas such as

radiation health and physical security protection, that require or suggest

performance by specialists (e.g., inspectable area experts). Because of

shortages of qualified staff, resident inspectors have been tasked with

performing more of these types of specialized inspections. 

OIG asked more than 20 resident and senior resident inspectors if they felt

“fully qualified to perform all inspection procedures required of your position?” 

Overall, the inspectors responded that they are comfortable that they

possess the knowledge and qualifications to perform required inspections. 

However, nearly half of the inspectors identified at least one area where their

level of comfort could be improved. Five areas, in particular, were cited by

multiple resident inspectors where inspections should more appropriately be

performed by specialists: fire protection, diesel generators, physical security,

motor-operated valves, and inspections related to the Temporary Instruction

on lower reactor vessel heads. OIG is aware of at least one occurrence

where an inspector asked management for expert help to review a situation

within one of the five areas identified above, but outside that inspector’s area

of expertise. The inspector was told that a shortage of resources would

prevent getting the requested assistance. The inspector was tasked, and

agreed, to do the review, but documented the level of discomfort. 

Vendor-specific Differences

There are multiple vendors for PWRs (e.g., Westinghouse, Babcock &

Wilcox, and Combustion Engineering) and multiple vintages of BWRs (e.g.,

General Electric (GE) 2 through 6). Agency training experts attest that there

are not only significant differences between technologies, but also between

specific vendors and models within the same technology. Yet, there is no

requirement for inspectors to receive specific vendor training because there

is a presumption that all PWRs and all BWRs are basically alike. Without

having vendor-specific knowledge, inspectors, especially those with less

experience in the field, could miss opportunities to recognize degraded utility

performance and safety issues. 

There are numerous inspectors currently assigned to plants built by vendors

other than those for which they have received training. It is realistic to

conclude that those numbers will increase as the shortage of qualified

inspectors continues. In particular is a situation identified at one commercial

nuclear power plant where, according to cognizant staff, there was a

complete turnover of the NRC inspection staff at the plant, resulting in a full

complement of new inspectors. At the time of their assignments, none were

trained in the appropriate vendor-specific technology. Because the once-peryear training course for this technology was full, it was possible that the 
Audit of NRC’s Baseline Inspection Program

34

three new inspectors would not get the needed training for more than a year. 

(OIG notes that region management did get two of the three inspectors into

the course in April 2004 — more than six months after being assigned.) 

When the agency’s 17 technical instructors were asked by OIG whether

resident inspectors should have vendor-specific training for the plants where

they are assigned, 15 of the 16 respondents answered “yes.” The instructors

presented compelling arguments to support that vendor-specific training

would benefit the inspection program. Examples of instructor comments

follow:

‚ All PWR vendors use different features, like rod movement, so an

inspector would need vendor-specific training to effectively evaluate a

utility’s performance.

‚ An inspector trained on a plant design different from that where he/she is

assigned would be insufficiently familiar with the assigned plant or would

potentially have to perform a great deal of additional research to develop,

understand, or assess findings in areas for which there are no

counterparts in the plant design for which he/she received training. This

situation is not as efficient or effective as a course specifically targeted to

teaching the specifics prior to assignment. 

‚ The GE design is significantly different from the Westinghouse,

Combustion Engineering and Babcock & Wilcox design so training would

be very important to someone with no experience in the GE design.

‚ Vendor-specific training would enhance the inspectors’ ability to provide

an informed, independent assessment of dynamic plant conditions during

an event.

One instructor, speaking as a former senior resident inspector, said that

information received in the technology classes contributed directly and

substantively to the quality of work performed. The instructor stated that

“had I been required to figure it all out upon arrival on site, as opposed to

having a reference to look to, I could not have been as successful as I was.” 

And finally, an NRR executive said a real concern with inspector interns is

that, although the interns are required to qualify on at least one of the plant

types (e.g., BWR or PWR), they have no experience at all in actual plant

operations. “The lack of vendor-specific training compounds that problem.”

Summary

It has become increasingly important to have inspectors work as efficiently as

possible given the challenges faced by the agency to keep qualified

inspectors available for baseline inspection activities. As discussed in the

previous finding, the agency is relying more on inspectors who have only a

“basic” certification to fill the numerous resident inspector slots which results

in more inspectors with less experience in the field. In addition, resident
Audit of NRC’s Baseline Inspection Program

35

inspectors are being tasked to conduct inspections in areas outside of their

comfort zones. There are many possible negative outcomes from inspecting

outside an area of expertise, being unfamiliar with vendor specifics, or being

mismatched between technology training and assigned plant. These include

(1) missed opportunities to recognize degradation and non-compliance, (2)

inappropriate assessment of inspection findings, (3) impaired evaluation of a

utility’s performance and safety, and (4) unfamiliarity with specific emergency

operating procedure approaches. 

RECOMMENDATIONS

OIG recommends that the Executive Director for Operations: 

6. Provide additional management guidance for assigning inspectors to

perform inspection procedures to ensure that inspectors are

adequately qualified for their assignments.

7. Provide guidance for vendor-specific training in resident inspector

training requirements.

D. THE REPORT OF 100 PERCENT BASELINE INSPECTION PROGRAM 

COMPLETION IN CY 2002 IS NOT FULLY SUPPORTED BY DOCUMENTATION

In CY 2002, NRC guidance required 100 percent completion of the minimum

samples specified in all baseline inspection program inspection procedures. 

Overall, for those plants sampled, the majority of the baseline inspection

program was completed in CY 2002, as required. However, there are

numerous examples of incomplete or missed procedures, due mainly to

inconsistent interpretation of completion requirements and reliance on

inaccurate data, which were not properly identified to headquarters. 

Additionally, available documentation does not support completion of

inspection requirements for a number of baseline procedures, especially in

the areas of Radiation Protection and Physical Security Protection. 

Therefore, agency records did not sufficiently support NRC’s claim that 100

percent of the required baseline inspection program was completed at all

reactor sites in the CY 2002 inspection cycle. As a result, despite the belief

of agency managers that the intent of the program was met, NRC may not

have a comprehensive view of plant operations which could hinder NRC’s

ability to evaluate plant operations. Finally, inaccurate reporting undermines

NRC’s credibility which could impact public confidence. 

Background

In 2000, the agency established the baseline inspection program as the

minimum level of oversight necessary to assess licensee performance. 

According to IMC 2515, baseline inspections constitute an appropriate level

of inspection at plants whose overall performance remains in a good

category. The overall objective of the program is to monitor all power reactor
Audit of NRC’s Baseline Inspection Program

22 NRC Reports (all dated February 26, 2003) - OIG-03-A-06, Management Audit of Region I; OIG-03-A07, Management Audit of Region II; OIG-03-A-08, Management Audit of Region III; OIG-03-A-09,

Management Audit of Region IV; and OIG-03-A-10, Headquarters Action Needed on Issues Identified

from Regional Audits. 

36

licensees with a consistent level of defined requirements to indicate whether

licensees' performance meets the objectives for each cornerstone of safety. 

The baseline inspection program defines the planned activities (i.e.,

inspection procedures) to monitor licensee performance over a 12-month

period (i.e., an ROP cycle).

In 2002, OIG’s audit of various elements of operations at the four NRC

regional offices22 concluded that reporting on completing the baseline

inspection program was a challenge for NRC staff and management. 

Regional managers stated that they received limited guidance on the

definition of terms, results presentation, procedures for data collection and

computations, and expectations for quality control. Due to its importance in

supporting the agency's mission, OIG again sampled the extent of baseline

inspection program completion using data for the CY 2002 inspection cycle. 

(For a full discussion of the results of OIG’s review, see Appendix D.)

Program and Reporting Responsibilities

NRC's four regions constitute the agency's front line in carrying out its

mission to protect the health and safety of the public by ensuring the safe

operation of nuclear power plants. The regions track their accomplishments

in these areas against performance metrics established jointly by

headquarters and regional managers. For example, according to IMC 2515,

each regional office is responsible for developing annual site-specific

baseline inspection plans. However, NRR senior managers are responsible

for the overall program direction, revisions to the program, as well as

overseeing regional implementation.

Baseline inspection results are reported in formal reports, in accordance with

the guidance and requirements of IMC 0612. Two of the objectives of

inspection reports are: (1) clearly communicate significant inspection results

in a consistent manner to licensees, NRC staff, and the public, and (2)

provide inspection results as one input into the Operating Reactor

Assessment Program of the ROP. 

Completion Requirements

Completion of the baseline inspection program is a mechanism for NRC to

remain alert to plant status and conditions at all licensed reactors, and

supports the goals and objectives of the ROP. To determine that the

objectives of the inspection procedures have been met, each baseline 
Audit of NRC’s Baseline Inspection Program

23 SECY-03-0062, Reactor Oversight Process Self-Assessment for Calendar Year 2002, dated April 21,

2003.

37

procedure contains specific inspection requirements. There are more than

40 inspection procedures, each with a specific minimum sample size and/or

level of effort prescribed. (See Appendix E for a list of baseline inspection

program procedures.) 

The inspection activities and minimum sample sizes must be completed to

provide an adequate assessment for each cornerstone. Inspection

procedure completion status is to be documented in the NRC's Reactor

Program System. 

Baseline Performance Goal

The agency recently changed its performance requirements for completion of

the baseline inspection program for an annual inspection cycle. (These

changes will be discussed later in this report.) However, for ROP inspection

cycle 3 (ROP3), the scope of this review, the definition of completion was

performance of 100 percent of all required inspection procedures at each

plant.

Performance Measure for Planned Accomplishment Code 103-140, 

Baseline Inspections

(per NRR Fiscal Years 2002/2003 Operating Plans)

Program Type of

Measure

Describe what needs to

be measured

Target

Reactor

Inspection

Outputs Extent of program completion at

each operating power reactor

annually

100% at end of cycle

In an April 21, 2003, paper to the Commission,23 the EDO reported that the

baseline inspection program was completed during ROP3 (CY 2002) in all

regions. He added that all annual and biennial inspections were completed

by the end of CY 2002. Subsequently, the agency reported to Congress that

the baseline inspection program was completed for CY 2002 (i.e., 100

percent of all required procedures completed per IMC 2515). 

Results of OIG’s Baseline Completion Review 

Documentation is a basic quality control procedure. Therefore,

documentation should be complete and accurate and should facilitate

tracking the transaction or event and related information. Documentation

should also be purposeful and useful to managers in controlling their

operations and to others involved in analyzing operations or decision 
Audit of NRC’s Baseline Inspection Program

24 OIG used the term “double-count” to mean crediting the conduct of samples for one procedure

towards the sample size of another procedure.

38

making. Without adequate documentation, senior management does not

know if metric data is reliable and can be used for making meaningful

decisions. 

Overall, the agency completed the majority of required baseline inspection

procedures for those plants sampled. However, throughout the review of

inspection completion for CY 2002, OIG found occurrences of incomplete

procedures at all four regions due, in large part, to documentation problems

(see partial list of examples below). Therefore, agency records do not

support its reported claim that 100 percent of the baseline inspection

program was completed for CY 2002.

Examples of Incomplete Baseline Inspection Procedures

ROP3 for CY 2002

 Procedure No. Procedure Title Condition Cause

 71114.06 Drill Evaluation incomplete

sample 

double-counted

procedures

71111.14 Personnel Performance

During Nonroutine

Evolutions and Events

1⁄2 the required

samples done

ambiguous

language

71111.23 Temporary Plant

Modifications

incomplete

sample

lack of

documentation

71114.04 Emergency Action Level

& Emergency Plan

Changes

incomplete

sample

inaccurate

documentation

71121, 71122, &

71130 Series

Radiation Protection and

Physical Security

Protection

unable to

discern actual

sample size

weak or

inconsistent

supporting

documentation

The lack of clearly identifiable samples and weak supporting evidence in

inspection reports occurred due, in large part, to insufficient guidance on

how, and what, to document to demonstrate completion. Specific reasons

identified by agency managers for failure to meet the 100 percent

completion target of some procedures in CY 2002 included:

# inconsistent interpretations of completion,

# lack of quality controls related to the Reactor Program System (i.e.,

access, validation of data),

# confusion over which procedures can be “double-counted,24” and 
Audit of NRC’s Baseline Inspection Program

39

# ambiguous language in the procedure resulted in identifying

incorrect sample sizes.

Similar to the information communicated to region managers during our

2002 regional reviews, OIG found insufficient documentation during this

audit to support reported sample sizes by all four regions, especially with the

Radiation Protection and Physical Security Protection inspection

procedures. Some of these weaknesses may have contributed to NRC

managers’ inability to identify an incomplete program. For example, 

< report language does not describe the required inspection activities

necessary to support that the intent of the procedures were met, and

< report language does not consistently identify which inspection is

satisfied (i.e., annual or biennial inspection, “full” or "partial"

inspection activity). 

Reliance on the Reactor Program System

Managers rely on data from baseline tracking tools, like the Reactor

Program System, to determine completion status and they expect their

inspectors, or their designees, to enter samples accurately. However, within

OIG's sample, numerous data reliability and validity issues were identified

with the Reactor Program System data. Given the limited nature of the

sample, the extent of the inaccuracies is unknown. OIG will address issues

with the Reactor Program System in a future audit. 

Summary

The baseline inspection program is the “minimum” inspection oversight that

should be conducted at each plant with a 100 percent completion

performance target. While the majority of the required baseline procedures

were completed in CY 2002, baseline inspection program records could not

sufficiently support the agency’s reported 100 percent completion rate. The

cited weaknesses in documentation and completion data presented

management with challenges for interpreting reported results. Regional

managers and inspectors again stated that they have received limited

guidance on the definition of terms, results presentation, procedures for

data collection and computations, and expectations for quality control. For

example, regional managers expressed confusion about which procedures

could be combined for “double-counting” samples. 

When internal assessment data is included with data reviewed at the

agency level, there must be an expectation that some agency managers will

use the information to make assessments or draw conclusions about the

particular program. Therefore, any data reported must be sufficiently

reliable for that purpose. To meet that goal, the process for compiling,

reviewing and reporting the data must have adequate management controls. 
Audit of NRC’s Baseline Inspection Program

40

Absent such assurances, the usefulness of baseline inspection program

information for decision making is limited. For example, if the completion

data in the Reactor Program System is not accurate, NRC managers cannot

be assured that the “minimum” baseline inspection program has been

accomplished. If the defined “minimum” program is not completed, NRC

may not have a comprehensive view of plant operations which impacts the

agency’s ability to assure safe plant operations. Finally, NRC’s inability to

provide sufficient documentation to support the reported 100% completion

of the baseline inspection program undermines the agency’s credibility

which could compromise public confidence in NRC. 

RECOMMENDATION

OIG recommends that the Executive Director for Operations:

8. Improve baseline inspection program guidance to include:

a) direction for implementing a baseline inspection program

completion tracking process using RPS to collect inspection

completion data at a plant level and in real-time,

b) how to document completion and sample sizes in inspection

reports, and 

c) whether performance of one inspection procedure sample

can be counted as a sample for another.

E. GUIDANCE IS UNCLEAR FOR NEW BASELINE COMPLETION CRITERIA

For CY 2004, baseline inspection program completion has been redefined to

be “not more than four (4) inspection procedures not completed, per

Region.” Agency documents indicate the change provides needed flexibility

in completing required inspections. However, headquarters has provided

little guidance to region managers regarding its expectations regarding the

allowance of four non-completed procedures. This may result in possible

misuse of allowances and/or inconsistent application in and across regions,

insufficient information to assess licensee performance and, thereby, impact

public confidence in NRC’s oversight activities.

Background

Since its inception, completion of the ROP’s baseline inspection program

has been defined as performance of 100 percent of the minimum sample

size for all required inspection procedures at each plant. For the CY 2004

inspection cycle, NRC redefined completion to allow no more than four non-
Audit of NRC’s Baseline Inspection Program

41

completed inspection procedures per region. This annual allowance of

sixteen non-completed inspection procedures equates to an annual

completion rate of “greater than 99.5 percent.” 

Reason for Change

The new allowance for non-completed inspection procedures is intended to

provide for unanticipated disruptions in inspection scheduling that

unavoidably cause an inspection procedure or attachment to not be

completed. It also presumes that at least the minimum inspection

requirements will otherwise be completed as soon as possible within the

quarter immediately following the annual inspection cycle. The agency

asserts that achieving this level of performance still provides a basis to

conclude that the intent of the program has been met and, therefore, can be

reported as complete for each annual cycle. 

Management Uncertainties

NRC’s inspection manual states the new completion requirements, however

headquarters and region managers have differing interpretations of the new

performance measure and its planned usage. While some managers

express different assumptions and expectations, all agree there has been

little, if any, actual discussion regarding the changed parameters, and no

formal guidance provided by headquarters. For example, in a recent

discussion about the new criteria, a cognizant headquarters’ manager stated

that all required procedures are expected to be completed at all plants within

the year cycle, though as many as 16 procedures could be postponed for up

to three months in warranted situations (e.g., samples not available within

the inspection year). Yet, one region executive stated that the actual goal is

still 100 percent completion and the flexibility to miss procedures is the

fallback goal, while another region has already changed its performance

measure to reflect less than 100% completion. 

 Because the guidance and criteria provided regarding this change is

unclear, region managers could not answer important questions about its

application. Specifically, OIG asked: 

C Is it reasonable to miss all four procedures at one plant, or all in risk

significant areas?

C Are there procedures which cannot be among those missed? 

C Are there specific plants where all procedures must be completed,

despite the region-wide allowance?

C What kind of tracking will be maintained to ensure that each plant

does not come in at the end of the cycle with four missed

procedures?
Audit of NRC’s Baseline Inspection Program

42

Without clearly communicating its expectations regarding the new

performance goal, the agency risks inconsistent application, at a minimum. 

Less likely, but potentially more worrisome, would be possible widespread

misuse of the allowance (due to the absence of tracking which procedures

at which plants are not being completed). It is important to note that NRC

previously established that 100 percent completion of all inspection

procedures was the “minimum” oversight necessary to assess licensee

performance. Without a sound basis for justifying any reduction in the

criteria, public confidence could be impacted. 

RECOMMENDATIONS

OIG recommends that the Executive Director for Operations: 

9. Clarify existing guidance regarding the expectations and

requirements for invoking the new completion criteria.

10. Provide a method for tracking inspection procedures not performed

as required by years end in order to ensure that each region stays

within the four-procedure allowance.
Audit of NRC’s Baseline Inspection Program

43

IV. CONSOLIDATED LIST OF RECOMMENDATIONS

OIG recommends that the Executive Director for Operations:

1. Document the basis and rationale for the determined baseline

inspection program sample sizes, including a discussion of when, or

why, to use more than minimum samples.

2. Develop guidance on how to identify human performance trends and

how that information should be integrated into the reactor oversight

process. 

3. Develop and implement guidance for documenting, tracking, and

trending informal inspection issues.

4. Define “effectiveness” as it pertains to the ROP and establish

performance measures and targets to demonstrate that the baseline

inspection program meets that definition.

5. Develop a planned, scheduled approach for phasing out the use of

the coping measures.

6. Provide additional management guidance for assigning inspectors to

perform inspection procedures to ensure that inspectors are

adequately qualified for their assignments.

7. Provide guidance for vendor-specific training in resident inspector

training requirements.

8. Improve baseline inspection program guidance to include:

a) direction for implementing a baseline inspection program

completion tracking process using RPS to collect inspection

completion data at a plant level and in real-time,

b) how to document completion and sample sizes in inspection

reports, and 

c) whether performance of one inspection procedure sample

can be counted as a sample for another.

9. Clarify existing guidance regarding the expectations and

requirements for invoking the new completion criteria.

10. Provide a method for tracking inspection procedures not performed

as required by years end in order to ensure that each region stays

within the four-procedure allowance.
Audit of NRC’s Baseline Inspection Program

44

V. AGENCY COMMENTS

On August 30, 2004, OIG discussed its draft report with agency senior

executives. Subsequent to that meeting, OIG met with NRR senior

managers to address issues and comments needing further clarification

and/or explanation. On November 23, 2004, the EDO provided a formal

response to this report in which he agreed with nine of the report’s eleven

recommendations. However, the EDO stated that Recommendation 3 is not

appropriate under the current ROP philosophy and that Recommendation 5

would be better addressed through the agency’s response to

Recommendation 4. The EDO’s transmittal letter and specific comments on

this report are included as Appendix F. 

This final report incorporates revisions made, where appropriate, as a result

of the subsequent meetings and the agency’s informal and formal written

comments. Because OIG takes exception to the agency’s comments

regarding Recommendations 3 and 5, a point-by-point analysis is presented

in Appendix G. Recommendation 5 has been retracted and will be

addressed, as the agency requested, through the agency’s response to

Recommendation 4. However, upon submittal, OIG will evaluate the

agency’s proposed plan of action for addressing Recommendation 4 to

ensure that the intent of Recommendation 5 is incorporated.
Audit of NRC’s Baseline Inspection Program

45

APPENDIX A

SCOPE AND METHODOLOGY

The reactor inspection and performance assessment program is missioncritical for NRC. Therefore, because all elements of the program must be

effective and efficient in order for NRC to meet strategic and performance

goals, OIG will be reviewing each of the elements of the ROP. In 2002, OIG

reported on the Significance Determination Process, an important part of the

baseline inspection program, and found that a number of improvements

were needed. Review of another element of the ROP (e.g., performance

indicators or the reactor assessment process) is included in OIG’s future

plans. 

The scope of this review was limited to the ROP's baseline inspection

program. The objectives were to determine whether the baseline inspection

program:

(1) is based on a sound methodology, 

(2) is carried out by sufficient, qualified staff, and 

(3) is completed at all operating commercial nuclear power plants.

For the third objective, the auditors examined a sample of program results

for CY 2002, the last full year of program operation at the time of this review.

To address the audit objectives, OIG reviewed relevant management

controls and related documentation, including reviews of:

• Management Directive 8.13, Reactor Oversight Process

• IMC 0102, Oversight and Objectivity of Inspectors and Examiners at

Reactor Facilities

• IMC 0307, Reactor Oversight Process Self-Assessment Program

• IMC 0308, Reactor Oversight Process (ROP) Basis Document

and conducted interviews with more than 100 individuals, including:

C 40+ headquarters and region senior managers,

C 7 of 14 ROP development team members,

C resident inspectors and licensee management at 11 nuclear power

plants,

C region-based senior reactor analysts,
Audit of NRC’s Baseline Inspection Program

46

C representatives from the Nuclear Energy Institute,

C representatives from public interest groups,

C a State resident inspector,

C the NRC statistician, and

C Department of Transportation OIG audit staff

This audit progressed in two phases. OIG began its review of the baseline

inspection program in August 2003. In response to an agency request in

February 2004, OIG reviewed the agency’s Operating Experience Task

Force Report and subsequently issued a memorandum report in March

2004. OIG completed its full assessment of the baseline inspection program

in May 2004. 

This audit was conducted in accordance with generally accepted

Government auditing standards and included a review of management

controls related to the objectives of this audit. The major contributors to this

report were Russ Irish, Team Leader; Cathy Colleli, Audit Manager; and

Robert Moody, former Audit Manager.
Audit of NRC’s Baseline Inspection Program

25 NRC defines a Planned Accomplishment as a level of discrete activity below the program level,

where FTE and budgetary resources can be identified.

47

APPENDIX B

DISCUSSION OF GUIDANCE ON PERFORMANCE AND OUTPUT

MEASURES

Planned Accomplishments

At least as early as October 1998, NRC established guidelines for internal

output measures for Planned Accomplishments.25 A model was provided

which suggested establishing metrics for:

‘ effectiveness [the accomplishment of the desired result]

‘ efficiency [accomplishing the desired result without wasting

resources]

‘ quality [degree of excellence]

‘ quantity [how many]

‘ timeliness [performance of the activity within defined milestones]

NRC offices were required to establish, at a minimum, an effectiveness

metric at the Planned Accomplishment level if there were no performance

plan output measures for that accomplishment. For the period subject to

this review, there was no performance plan output measure specifically

defined for effectiveness in the baseline inspection program. 

Operating Plan

The NRR FY 2004 Operating Plan describes the planning, budgeting, and

performance management process as implemented by NRR. The process

is directed toward achieving organizational effectiveness by:

‘ Setting strategic direction

‘ Determining Planned Accomplishments and resources by identifying

critical activities necessary to achieve outcomes

‘ Measuring and monitoring performance against measures and

targets

‘ Assessing performance
Audit of NRC’s Baseline Inspection Program

26 The NRR Rainbow Report is available at http://nrr10.nrc.gov/rainbow/rbooks.html.

27 Complete Light-water Reactor Inspection Program objectives are provided in NRC IMC 2515.

48

The Operating Plan defines success for all Planned Accomplishments and

includes planning templates for each accomplishment discussed in the plan. 

Each template establishes the measures of success and targets for NRR

management.

For the baseline inspection program, there is one Planned Accomplishments

Code, #103-140 - Baseline Inspections, which includes both Leadership and

Operational measures and targets. Baseline inspection program

performance results for measures in the Operating Plan are reported in the

NRR Rainbow Report,26 along with other performance statistics. OIG

reviewed the baseline inspection program's performance measures and

targets to determine whether they:

< tie to the NRC’s mission and represent essential components of the

program

< provide useful information to managers for decision-making

< provide a basis for comparison against a baseline of performance

< are clearly understandable in terms of measures and comparisons 

< reflect the most important aspects of performing the activity

Inconsistencies in baseline inspection program objectives

In contrast to the objectives defined for the baseline inspection program in

IMC 2515, Appendix A, the NRR Operating Plan planning template for

Planned Accomplishments Code 103-140 describes the “Purpose” of the

baseline inspection program’s Planned Accomplishment as:

To collect information providing objective evidence that power

reactors are operated safely and do not pose an undue risk to

the public, to identify declining safety performance, and to

identify safety-significant issues that may have generic

applicability.

The planning template “Purpose” statement actually reflects the objectives

of the Light-water Reactor Inspection Program–Operations Phase, which

encompasses all aspects of the inspection effort at power reactors, not just

the baseline inspection program.27
Audit of NRC’s Baseline Inspection Program

28 Office of Management and Budget Circular No. A-11, Preparation, Submission, and Execution of the

Budget, revised July 16, 2004.

29 NUREG-1614, Vol. 3, U.S. Nuclear Regulatory Commission Strategic Plan, FY 2004-FY 2009, 

dated August 2004.

30 (1) No more than one event per year identified as a significant precursor of a nuclear accident, (2) No

statistically significant adverse industry trends in safety performance, (3) No events resulting in

radiation overexposures from nuclear reactors that exceed applicable regulatory limits, (4) No more

than three releases per year to the environment of radioactive material from nuclear reactors that

exceed the regulatory limits, and (5) No breakdowns of physical security that significantly weaken the

protection against radiological sabotage or theft or diversion of special nuclear materials in

accordance with abnormal occurrence criteria.

49

The stated Purpose of the baseline inspections Planned Accomplishment

should be aligned with the objectives of the program. If this is not done,

performance measures and targets may not be developed in alignment with

program objectives.

OMB Guidance

In the President’s Budget for FY 2004, OMB28 describes its initiative to

implement performance budgeting throughout the Federal government in

2005 as follows:

The agency should develop a ‘performance budget,’

organized like its Strategic Plan that matches resources with

outputs and justifies resources requested by the

effectiveness at influencing the desired outcomes.

As OMB describes for budgeting purposes, NRC must link the performance

of day-to-day activities, like baseline inspections, to long-term, higher level

performance goals in order to build a solid foundation for managing program

performance and achieving performance and strategic goals. 

Strategic Plan - Goals and Performance Measures

The NRC has developed strategic goals consistent with its mission. These

strategic goals are supported by performance goals, which represent

outcomes the NRC plans to achieve over the period covered by the FY

2004-FY 2009 Strategic Plan. In NRC’s Strategic Plan, Performance Goal

129 is: Ensure protection of public health and safety and the environment. 

There are several performance or outcome measures associated with this

goal.30

Planning Template Performance Measures 

Item #6: In accordance with OMB’s guidance, NRR's operating planning

template Item #6: Tangible Contribution, asks What is the tangible

contribution to the outcome goals...in the core activity from doing this work?
Audit of NRC’s Baseline Inspection Program

50

(What will success look like or what will we measure to know that this

activity was successful? Specific quantitative terms.) Planning template

item #6 asks how baseline inspections contribute tangibly to the agency’s

outcome goals. Item #6 also asks for a specific quantitative measure of

baseline inspections success. The response to item #6 is:

Provides information to allow management to make decisions

regarding licensee safety performance. Maintained safety is

reflected in a technically credible basis for reactor

performance assessments, as reflected in inspection reports

and performance indicator information. Public confidence is

supported by a stable, predictable, and understandable

process.

The answer to item #6 does not provide specific quantitative terms for

measuring baseline inspections’ tangible contribution to the outcome goals

and does not describe what should be measured to know that baseline

inspections were successful.

Items #7 & #9: Planning template items #7 and #9 ask for other

performance measures, at the leadership and operational levels,

respectively, that should be used to ensure that this Planned

Accomplishment is in control. The table presented in Appendix C shows the

results of OIG's analysis of the specific leadership and operational

performance measures and targets. At the operational level, management

should include, when possible, measures for each of the performance

categories: effectiveness, efficiency, quality, quantity, and timeliness. It is

important to have clear drivers or targets in each area to ensure the

program effort is not “pushed” in any one direction at the expense of others. 

Effectiveness

October 1998 Operating Plan guidance from the Office of the Executive

Director for Operations stated that:

A major area targeted for improvement in the FY 1999

operating plans is the establishment of effectiveness metrics. 

All other metrics are superfluous if the program is not

achieving its desired results or outcome. Effectiveness is

also at the heart of self assessment. Therefore,

effectiveness metrics are a required element for the FY 1999

operating plans.
Audit of NRC’s Baseline Inspection Program

31 An “inspectable area lead” is a person in the Division of Inspection Program Management of NRR

assigned responsibility to oversee and manage the use of individual baseline inspection

procedures or attachments to those procedures.

32 For CY 2003, there were approximately 11 different NRR staff members serving as Inspectable area

leads.

33 The annual review did not include the physical protection procedures which were covered under a

current NRC temporary instruction.

51

Annual Review of Baseline Inspection Procedures 

IMC 0307 was issued and currently requires certain staff, called inspectable

area leads,31 to monitor and report on the effectiveness and efficiency of

inspection activities and program objectives within their assigned areas of

responsibility. One of the primary objectives of this annual review is a

critical ‘big picture’ evaluation of “all of the inspectable areas together to

justify retaining them in the baseline inspection program, or determine if the

addition of a new inspectable area is warranted.” 

In accordance with IMC 0307, multiple inspectable area leads32 recently

completed the CY 2003 annual reviews of their assigned baseline

procedures.33 They did not identify the need for any significant changes to

the inspection program overall (i.e., either the addition or removal of

inspection procedures). While there were a couple of procedures that were

revised, the staff mainly recommended minor adjustments to levels of effort

(i.e., hours), clarification of the definition of a sample, or how many samples

constitutes completion. 

Although references to “effectiveness” appear in the individual procedures,

as noted by the focus group, and as seen in the NRR Operating Plan, the

agency has not defined what effectiveness means for the baseline

inspection program. As a result, the agency cannot properly evaluate the

effectiveness of the procedures, and thereby, the program. When a senior

NRR executive was asked why there was no definition of “effectiveness,” he

said this issue has been on his radar because he too realized the

“effectiveness” of the baseline inspection program had never been

measured. Another executive said that as a first step, NRR is putting

interim graded measures into effect, but NRR needs to get closer between

the operating plan and their strategic goals.

Quality Measures

In addition to effectiveness, “Quality” metrics should be developed. 

Monitoring only other types of measures and targets can drive performance

in another direction at the expense of quality. For example, there is no

measure showing whether inspectors do quality inspection work, a critical

component of baseline inspection program success. Such a measure

might, for example, reflect the quality of inspection samples chosen.
Audit of NRC’s Baseline Inspection Program

52

Inconsistency in Performance Measure Guidance

IMC 2515, section 04.07, Baseline Inspection Program Completion states

that:

Baseline Inspection Program completion for an ROP annual

inspection cycle is defined to be not more than four (4)

inspection procedures not completed, per Region. This 16

inspection procedure allowance per year nationwide equates

to an annual completion rate greater than 99.5%.

IMC 2515 thus defines the performance target for completion of the baseline

inspection program as 99.5% completion. However, the completion

performance target in the FY 2004 Operating Plan is 100%. 

IMC 0307 - Inspection Program Metrics

IMC 0307 also includes the inspection program metrics used in the annual

reactor oversight process self-assessment. There are 11 inspection program

metrics in IMC 0307. Of those, the following are also reflected in the

performance measures reviewed above from the NRR Operating Plan for

the baseline inspection program:

< Percentage of inspection findings documented in accordance with

requirements

< Completion of baseline inspection program

< Inspection reports are timely

< Analysis of inspection hours

None of the other seven metrics provides a direct measure of baseline

inspection program performance. Therefore, while IMC 0307 provides

metrics beyond those in the Operating Plan, none change the results of the

above review. For example, there are also no measures of effectiveness or

quality in IMC 0307.
Audit of NRC’s Baseline Inspection Program

53

APPENDIX C 

FY 2004 Operating Plan Baseline Inspection Program 

Performance Measures 

Planned Accomplishment Code 103-140

Type of

measure

What to measure Target OIG Evaluation

 Leadership Team Measures and Targets

Outputs Extent of program

completion at each

operating power

reactor annually

100% (as defined in

IMC 2515 and 0306)

Quantity measure - In the absence of a

quality measure, the emphasis on

outputs/quantities could compromise

quality. 

Resources

used 

Budget (Total hours

expended versus

budget)

± 20% (monitoring

only until appropriate

levels are

determined)

This measure should be used for

trending.

Other Adequacy of site

coverage

100% of power

reactor sites staffed

in accordance with

policy.

Quantity measure - shows compliance

with guidance. However, this is a

secondary measure related to the

output measure above. That is, if the

above output target isn’t met, this could

be a cause.

 Operational Team Measures and Targets

Outputs Baseline program

inspection

100% (as defined in

IMC 2515 and 0306)

Quantity measure - In the absence of a

quality measure, the emphasis on

outputs/quantities could compromise

quality. 
Audit of NRC’s Baseline Inspection Program

Type of

measure

What to measure Target OIG Evaluation

54

Productivity Total hours expended

implementing baseline

inspections per region

and per single-, dual-,

and triple-unit site

Hours expended for

direct inspection effort

on baseline

inspections as a

function of total

baseline hours per

region

(BI/BI+BIP+BID)

Regional variance (per

site average)

Overtime contribution

to direct inspection

effort (non-reg

hrs/total hrs)

Within 20% of

estimate for single-,

dual-, and triple-unit

sites and within 10%

of estimate per region

(This measure used

for trending and

monitoring only)

Within 20%

(This measure used

for trending and

monitoring only)

Efficiency measure - This measure

provides efficiency information which is

useful to management for budgeting

and performance monitoring. 

N/A

Clarify - is this related to the measure

directly above this one?

N/A

Quality Inspection report

findings are

documented in

accordance with

program guidance in

IMC 0612

Ratio of qualified

inspectors to total

number of inspectors

90% of findings in the

audited report sample

are properly

documented

Annual average of

monthly ratios >

(TBD)%

Indirect Quality measure - This shows

compliance with guidance and reflects

the quality of report writing rather than

the quality of the actual inspection

finding.

Indirect Quality measure - provides

assurance that inspectors are qualified

to perform good inspection work.

Timeliness issuance of inspection

reports

90% of reports issued

within time limits (30

days/45 days)

Indirect timeliness measure - good

information but doesn’t directly

measure whether inspections

themselves are performed timely.

Resources

used

Budget (Total hours

expended versus

budget)

± 20% (monitoring

only until appropriate

levels are

determined)

This Efficiency measure should be

used for trending purposes.

At the operational level, management should include, when possible,

measures for each of the performance categories: effectiveness, efficiency,

quality, quantity, and timeliness. It is important to have clear drivers or

targets in each area to ensure the program effort is not “pushed” in any one

direction at the expense of others. 
Audit of NRC’s Baseline Inspection Program

55

APPENDIX D

DETAILS OF OIG’S REVIEW OF CY 2002 BASELINE

INSPECTION PROGRAM COMPLETION 

Program responsibilities 

NRC's four regions constitute the agency's front line in carrying out its

mission to protect the health and safety of the public by ensuring the safe

operation of nuclear power plants. The regions track their

accomplishments in these areas against performance metrics

established jointly by headquarters and regional managers. For

example, according to IMC 2515, each regional office is responsible for

developing annual site-specific baseline inspection plans. However,

NRR senior managers are responsible for the overall program direction,

the revisions to the program, as well as overseeing regional

implementation.

Reporting of inspection results

Baseline inspection results are reported in formal reports, in accordance

with the guidance and requirements of IMC 0612. Two of the objectives

of inspection reports are: (1) clearly communicate significant inspection

results in a consistent manner to licensees, NRC staff, and the public,

and (2) provide inspection results as one input into the Operating

Reactor Assessment Program of the ROP. 

Baseline performance goal

For the CY 2002 ROP3, the scope of this review, the definition of

completion was performance of 100% of all required inspection

procedures at each plant. In an April 21, 2003 paper to the Commission,

the EDO reported that the ROP3 baseline inspection program was

completed by the end of CY 2002 in all regions, including all annual and

biennial inspections. Subsequently, the agency reported to Congress

that the baseline inspection program was completed for CY 2002 (i.e.,

per IMC 2515, 100 percent of required procedures completed). 

Completion requirements

To determine that the objectives of the inspection procedures have been

met, each baseline procedure contains specific inspection requirements. 

There are more than 40 inspection procedures, each with a specific

minimum sample size and/or level of effort prescribed. (See Appendix E 
Audit of NRC’s Baseline Inspection Program

34 The inspection cycle dates for Region II plants varied slightly and began as early as

December 16, 2001 and ended as late January 4, 2003, depending on the plant. 

56

for a full list of baseline procedures.) The inspection activities and

minimum sample sizes must be completed to provide an adequate

assessment for each cornerstone. Completion status is to be

documented in the NRC's Reactor Program System.

To test baseline completion for CY 2002, OIG selected a judgmental

sample of 12 operating nuclear power plants, 11 of which were reviewed

as part of this audit. Generally, the inspection cycle dates for ROP3

encompassed inspections performed from December 30, 2001 through

December 28, 2002.34

 Region I Region II Region III Region IV 

1. Beaver Valley 4. Browns Ferry 7. D.C. Cook 10. Arkansas One

2. Peach Bottom 5. Farley 8. Braidwood 11. San Onofre

3. Vermont Yankee* 6. St. Lucie 9. Kewaunee 12. Palo Verde

(*although Vermont Yankee was originally selected for review, time constraints prevented auditors from

including this plant as part of the audit)

Results of Review

For those plants sampled, the regions completed the majority of the

baseline inspection program for CY 2002. However, the following chart

shows the missed or incomplete procedures which were not properly

identified to headquarters. 
Audit of NRC’s Baseline Inspection Program

57

Results of CY 2002 Baseline Inspection Program Completion Sample

Region(s) Procedure No. Procedure Title Condition Cause

I 71114.06 Drill Evaluation incomplete sample double-counted

procedures

II 71111.14 Personnel Performance

During Nonroutine

Evolutions and Events

incomplete sample ambiguous

language

II 71151.00 Performance Indicator

Verification

incomplete sample misinterpretation

of guidance

II 71114.06 Drill Evaluation incomplete sample double-counted

procedures

III 71111.23 Temporary Plant

Modifications

incomplete sample lack of

documentation

IV 71111.04 Equipment Alignment -

Semiannual

intent not met both inspections

in same quarter

IV 71114.04 Emergency Action Level

& Emergency Plan

Changes

incomplete sample inaccurate

documentation

IV IP 71114.06 Drill Evaluation incomplete sample double-counted

procedures

IV 71151 Performance Indicator

Verification 

no evidence of

completion

lack of

documentation

IV 71152 Identification and

Resolution of Problems

no evidence of

completion

lack of

documentation

I, II, III, IV 71121, 71122, &

71130 Series

Radiation Protection and

Physical Security

Protection

unable to discern

actual sample

sizes

weak or

inconsistent

supporting

documentation

Conclusion

All four regions reported to headquarters that 100% of the ROP3

baseline inspection program was complete for CY 2002. Subsequently,

NRC reported to Congress that this performance measure was fully

completed at all reactor sites in the CY 2002 inspection cycle. However,

NRC inspection program records for the 11 operating nuclear power

plants sampled were not sufficient to allow independent verification that

all CY 2002 required baseline inspection samples were completed. For

example, inspection reports did not always provide sufficient 
Audit of NRC’s Baseline Inspection Program

58

documentation necessary to ensure the intent of the procedures were

met. Also, for a number of procedures it was extremely difficult to

determine from a review of inspection report documentation that at least

the required minimum sample sizes were complete. [emphasis added]

Similar to those reported to each region during OIG’s Regional Reviews,

areas in the inspection reports needing improvement include: 

S the number of samples were not clearly identifiable, especially in

the Radiation Safety and Physical Security areas

S language used did not describe the required inspection activities

necessary to support that the intent of the procedures was met

S reports did not identify whether the inspection satisfied an annual,

biennial, or triennial inspection activity or if it is a semi-annual or

"partial" inspection

S no indication that an inspection was conducted by a specialist, as

required

Managers rely on data from the baseline tracking tools, like the RPS, to

determine completion status and they expect their inspectors to enter

samples accurately. However, within OIG's limited sample, numerous

data reliability and validity issues were identified with the RPS data. 

Because of the limited nature of the sample, the extent of the

inaccuracies in RPS is unknown.
Audit of NRC’s Baseline Inspection Program

59

APPENDIX E

BASELINE INSPECTION PROGRAM INSPECTION PROCEDURES

Procedure # Procedure Title

 71111 Reactor Safety - Initiating Events, Mitigating Systems, Barrier Integrity

71111.01 Adverse Weather Protection 

71111.02 Evaluation of Changes, Tests, or Experiments

71111.04 Equipment Alignment

71111.05 Fire Protection

71111.06 Flood Protection Measures

71111.07 Heat Sink Performance

71111.08 In-service Inspection Activities

71111.11 Licensed Operator Re-qualification Program

71111.12 Maintenance Rule Implementation

71111.13 Maintenance Risk Assessments and Emergent Work Control

71111.14 Personnel Performance During Non-routine Evolutions and Events

71111.15 Operability Evaluations

71111.16 Operator Workarounds

71111.17 Permanent Plant Modifications

71111.19 Post Maintenance Testing

71111.20 Refueling and Outage Activities

71111.21 Safety System Design and Performance Capability

71111.22 Surveillance Testing

71111.23 Temporary Plant Modifications

71114.01 Exercise Evaluation

71114.02 Alert Notification System Testing

71114.03 Emergency Response Organization Augmentation

71114.04 Emergency Action Level and Emergency Plan Changes

71114.05 Correction of Emergency Preparedness Weaknesses and Deficiencies

71114.06 Drill Evaluation

 71121 Occupational Radiation Safety

71121.01 Access Control to Radiologically Significant Areas

71121.02 ALARA Planning and Controls

71121.03 Radiation Monitoring Instrumentation

 71122 Public Radiation Safety

71122.01 Radioactive Gaseous and Liquid Effluent Treatment and 

Monitoring Systems 

71122.02 Radioactive Material Processing and Transportation

71122.03 Radiological Environmental Monitoring Program and 

Radioactive Material Control Program 
Audit of NRC’s Baseline Inspection Program

60

Procedure # Procedure Title

 71130 Physical Protection

71130.01 Access Authorization Program (Behavior Observation Only)

71130.02 Access Control (Search of Personnel, Packages, and Vehicles: 

Identification and Authorization) 

71130.03 Response to Contingency Events (Protective Strategy and 

Implementation of Protective Strategy 

71130.04 Security Plan Changes

 Other Baseline Procedures

71150 Discrepant or Unreported Performance Indicator Data

71151 Performance Indicator Verification

71152 Identification and Resolution of Problems

71153 Event Followup
Audit of NRC’s Baseline Inspection Program

61

APPENDIX F

AGENCY COMMENTS
Audit of NRC’s Baseline Inspection Program

62
Audit of NRC’s Baseline Inspection Program

63
Audit of NRC’s Baseline Inspection Program

64

[Page intentionally left blank.]
Audit of NRC’s Baseline Inspection Program

65

APPENDIX G

DETAILED OIG ANALYSIS OF AGENCY COMMENTS

Recommendation 3: Develop and implement guidance for documenting, tracking, and

trending informal inspection issues.

Staff comments: Programmatic documentation, tracking and trending of minor findings,

which includes what the OIG audit report calls informal inspection issues, is not

appropriate and not required under the current Commission-approved Reactor Oversight

Process (ROP) philosophy of a risk-informed approach. The inspection program was

developed to be objective and includes guidance for establishing a minimum threshold

for documenting and assessing inspection findings. Commission emphasis in Staff

Requirements Memoranda for SECY-99-007A and SECY-00-0049 was to limit

documentation and analysis of findings of lower safety significance. The resources

saved from not having to programmatically document minor findings are intended to be

directed toward areas where there may be findings of greater risk significance. 

Redirecting limited inspector resources toward formal monitoring of minor findings which

are less than very low risk significance would detract from the intended safety focus of

the baseline inspection program. 

Although minor findings are not documented in inspection reports, the ROP explicitly

allows minor findings to be conveyed to a licensee verbally for appropriate resolution. 

The feedback received from NRC regional managers indicates that this communication

process is working well. 

In summary, the additional resources required to develop and implement formal tracking

and trending of minor inspection findings do not justify the benefit from such a program. 

Additionally, it would be contrary to the Commission’s direction to use a risk-informed

approach to focus agency and licensee attention on issues of risk significance. 

We request that OIG consider retracting this recommendation
Audit of NRC’s Baseline Inspection Program

66

Recommendation 5: Establish performance measures and targets which demonstrate

the quality of baseline inspections.

Staff comments: The staff disagrees with this recommendation. From discussions with

OIG staff, we understand this issue to be focused at the quality of individual inspections. 

The quality of day-to-day inspection activities is the responsibility of regional

management, who employ a variety of tools to assess inspection quality, such as daily

telephone discussions with inspectors, debriefs with inspectors after inspections, review

of draft inspection reports, consideration of integrated licensee performance, and

periodic site visits. In addition, licensees are afforded opportunities during their

management interactions with NRC management to express concerns regarding the

effectiveness of NRC inspectors at their sites. We do not believe that defining ?quality”

as it pertains to inspection activities is amenable to quantification through metrics and

OIG Response:

The staff correctly states that SECY-99-007A and SECY-00-0049 provide the Commission’s

direction for a risk-informed ROP with emphasis on documenting information at a defined

threshold based on risk significance. It is also correct that the ROP allows information falling

below the designated threshold to be communicated verbally to licensees. OIG does not

dispute that the verbal communication process works well. The issue is not the disposition of

minor “findings” but rather those issues which fall below the threshold defined in the ROP as

reportable in inspection reports. 

NRC officials stated that they view informal issues as very important to improving plant

operations. Licensee officials stated that they appreciate NRC inspectors bringing informal

issues to their attention and that they viewed discussion of informal issues as being proactive

and predictive of potential performance problems at their plants. In fact, several agency and

licensee officials viewed informal issues as more important to improving plant operations

than those formal inspection findings identified in NRC inspection reports. 

The staff’s assertion that this recommendation would result in “redirecting” inspector

resources towards formal monitoring is misleading. It is important to note that OIG is not

taking exception with the direction given by the Commission in the cited SECY papers. 

However, the reality is that regardless of whether the threshold is met for formal

documentation, agency inspectors and managers are already expending varying levels of

time and energy to track and trend informal issues. Similarly, licensees already take action

to correct informal issues in virtually all cases. 

OIG believes it is important that the agency recognize the current situation as reflected in the

report; specifically, the inconsistent guidance provided by agency managers and the ad hoc

nature used by staff to handle this type of information. This information would be much more

useful to the agency if it were consistently managed and could be analyzed for trends.

Therefore, OIG stands by its recommendation in order to resolve the inconsistent disposition

of informal issues.
Audit of NRC’s Baseline Inspection Program

67

targets. The one suggestion proposed by OIG to illustrate a possible staff response to

this recommendation (after-the-fact revisitation of how an inspector selected his/her

inspection sample) would not be consistent or objective. 

We believe this recommendation is better viewed as subsumed within the intent of

recommendation 4 of the OIG audit. In accepting recommendation 4, the staff intends to

clarify what is meant by effectiveness within the context of the ROP, and to demonstrate

that the performance measures and targets (i.e., metrics), which are used by the ROP

self-assessment process, comprehensively monitors measurable ROP outcomes. The

quality of program activities (including inspection) can be inferred from these

measurable ROP outcomes. The quality of the work of individual inspectors is best

assessed via the existing NRC Performance Appraisal System.

Based on this information, the staff recommends that OIG remove this recommendation

from the draft OIG report.

OIG Response:

The most critical aspects of evaluating an activity, like baseline inspections, would be an

assessment of that activity’s effectiveness and quality, but OIG acknowledges the difficulty

in identifying quantitative “quality” measures. While OIG agrees with the agency that the

appropriate mechanism for assessing the quality of an inspector’s individual work is the NRC

Performance Appraisal System, quality performers are only one facet of how effective the

inspection program is overall. As OIG explained to senior NRR managers, the quality of

other parts of the program need evaluation as well, such as a quality assessment of the

areas identified to receive baseline inspections and the quality of samples selected in the

ROP for mandatory inspection. 

As the report identifies, there are weaknesses in the agency’s process for self-assessing the

effectiveness and quality of the inspection program. The agency accepted the report’s

Recommendation 4 to define effectiveness as it pertains to the ROP and, in doing so,

proposes to demonstrate that the metrics used in the self-assessment process will

comprehensively monitor measurable outcomes. It is the agency’s contention that the

“quality” of inspections can be inferred from these measurable outcomes. OIG accepts the

agency’s premise under a presumption that the conduct of high quality inspection activities 

would be necessary to deem the overall baseline program as “effective.” However, upon its

submittal, OIG will carefully evaluate the agency’s proposed plan of action for addressing

Recommendation 4 to ensure that the intent of this recommendation is incorporated.

Therefore, OIG agrees to retract this recommendation at this time.