POLICY ISSUE

INFORMATION

July 26, 2002 SECY-02-0144

FOR: The Commissioners

FROM: William D. Travers

Executive Director for Operations

SUBJECT: PROCESS USED TO DETERMINE WHEN FURTHER RESEARCH ON A

PARTICULAR ITEM SHOULD BE CLOSED OR TERMINATED

PURPOSE:

To inform the Commission of the process used to determine when further research on a

particular item should be closed or terminated.

BACKGROUND:

On Tuesday, March 19, 2002, the staff briefed the Commission on the status of RES programs,

performance, and plans. Subsequently, a Staff Requirements Memorandum (SRM) was issued

on April 1, 2002, that in part directed the staff to inform the Commission of the process used to

determine when further research on a particular item should be closed or terminated. This

paper has been prepared in response to that direction. 

DISCUSSION:

RES management considers the following guidance and information when making its decisions

on whether further research on a particular item should be closed or terminated.

RES Office Letters

In 1996, RES formalized guidance to the staff regarding research project management through

a series of Office Letters. These Office Letters describe processes for implementing,

managing, and terminating research projects. Although developed prior to the use of the 

Contact:

Frank Cherny, RES

301-415-6786
The Commissioners 2

planning, budgeting, and performance management process (PBPM), these Office Letters

contribute to its implementation within RES. Office Letter 1, “Implementing Research

Projects,” provides criteria for implementing and continuing research projects. The

attachment to this paper contains the criteria from Office Letter 1. It contains the following

criteria posed as questions that projects are to be evaluated against relative to closure: Has

the work been completed? Has it answered the specific question(s) addressed? Was the

work terminated when the initiating question was answered and the value of further information

tapered off to a point beyond what is worthwhile? 

In addition, RES Office Letter 1A, “Role of the RES Project Manager in Contract

Management,” states that “Projects should be terminated whenever altered situations and

facts eliminate the need for the product.” RES Office Letter 1A also specifies “Initiate closeout

of the project when (1) the work scope has been completed, or (2) the knowledge gained from

research is sufficient to satisfy the needs of the NRC user, or (3) the value of incremental

knowledge from further research is no longer cost effective.” 

The Office Letters require Project Managers to provide written recommendations to the

Division Director to initiate, continue, or close out a research project. The Division Director

reviews these recommendations and the results of the review are retained in the official project

file. This process provides the Project Manager, the Branch Chief, and the Division Director

with a definite decision point on continuing or closing a research project, along with specific

decision criteria.

Thus, as part of the contract project management process, established RES procedures

directly address the importance of closing research projects by requiring evaluation against 

specific closure criteria. 

To aid in making the determination called for in the RES Office Letters, input from others is

also sought as described below:

Internal Inputs

Over the last few years, RES has modified and improved its budget formulation process

regarding new initiatives and closure of existing projects. This new process stresses

alignment of staff and management and includes interaction with user offices. RES

management provides policy direction and goal setting and the staff is involved in developing

specific activities and issues to implement this direction and to achieve the goals. Outcomes

are defined in terms of success relative to the agency Performance Goals. In addition, a

prioritization process, discussed in more detail later in this paper, is applied to all ongoing

activities. 

External Inputs

RES budget formulation is aided by input from several external sources. The Advisory

Committee on Reactor Safeguards and the Advisory Committee for Nuclear Waste each

provide their views on the research program, including recommendations on programs that

should be considered for closure. RES also has sought input from ad hoc groups, such as the

“Rogers Committee” and DOE national laboratories. Interactions with DOE, the Electric Power

Research Institute, and the international community through the International Atomic Energy

Agency and the Committee on the Safety of Nuclear Installations (CSNI) under the Nuclear

Energy Agency (NEA) also provide insights into needed activities and those that may be
The Commissioners 3

adequately addressed by external organizations. A new RES initiative seeks specific external

stakeholder input on ‘anticipatory’ research needs; this initiative may provide input on research

activities that have reached the point where they can be closed.

Maintaining Core Capabilities

SECY 97-075 discussed criteria for research program closure. Such criteria have been

incorporated into the RES process used for budget formulation. As part of the process, some

decisions are made to continue projects that have a low priority ranking in order to respond to

particular stakeholder needs or to maintain technical expertise in certain critical areas. 

RES Prioritization

The prioritization process developed by RES is a process where RES uses prioritization factors 

with assigned numerical weights to estimate the importance of each ongoing activity to

applicable Strategic Plan Performance Goal outcome measures. This results in a 1 through n

ranking of all RES activities. 

This prioritization process implicitly addresses the need to continue research efforts as

contrasted to the need for new initiatives. The agency budget guidance and the needs for new

initiatives require that RES identify lower priority activities that can be closed. 

In addition to the day-to-day Project Manager consideration, the budget formulation process

brings together the internal and external inputs and requires explicit consideration by RES

Management of closing projects. Proposals for project termination are made taking into

account information developed from the internal and external inputs noted earlier,

considerations of core capability maintenance, and the results of the RES prioritization

process. These proposals are then reviewed by the RES Director as the Office budget

proposal is developed. This is a deliberate step in the budget formulation and provides the

RES management team a specific decision point on continuing or ending a research project or

activity. 

International Activities

The CSNI has established an ad hoc Group on Regulator-Industry Collaboration (GRIC); one

of their activities is to define project closure criteria. RES is an active participant in GRIC and

has been following the closure criteria discussions closely. A set of questions on research

needs or closure has been proposed that addresses many of the same issues addressed by

the criteria in RES Office Letters 1 and 1A. 

RES will continue to follow this activity and we will adjust our closure criteria to incorporate, as

appropriate, new measures proposed by the CSNI. 

CONCLUSION:

In addition to considering input from both internal and external sources, RES management uses

guidance in various office letters when making management decisions on whether a particular

research item should be closed or terminated. Additional enhancements will be made, as

appropriate, if new information or ideas are developed, for determining when further research

on a particular item should be closed.
The Commissioners 4

COORDINATION:

The Office of the Chief Financial Officer does not need to concur on this Commission paper

because there are no resources implications. This paper was not sent to the Office of the

General Counsel for review because it does not involve rulemaking, interpretation of

regulations, policy matters, or other actions with legal or regulatory implications. 

RESOURCES:

The resources needed for continued implementation of these processes have been included in

the RES operating plan. 

/RA/

William D. Travers

Executive Director

 for Operations

Attachment: As stated
The Commissioners 5

research item should be closed or terminated. Additional enhancements will be made, as

appropriate, if new information or ideas are developed, for determining when further research

on a particular item should be closed.

COORDINATION:

The Office of the Chief Financial Officer does not need to concur on this Commission paper

because there are no resources implications. This paper was not sent to the Office of the

General Counsel for review because it does not involve rulemaking, interpretation of

regulations, policy matters, or other actions with legal or regulatory implications. 

RESOURCES:

The resources needed for continued implementation of these processes have been included in

the RES operating plan. 

/RA/

William D. Travers

Executive Director

 for Operations

Attachment: As stated

Distribution:

Commissioners; OGC OCA OPA CFO CIO EDO SECY

DET r/f; F. Cherny; M. Mayfield; J. Strosnider/AThadani

DOCUMENT NAME: C:\ORPCheckout\FileNET\ML021710061.wpd

OAR in ADAMS? (Y or N) Y ADAMS ACCESSION NO.:ML021710061 TEMPLATE NO. EDO-002 

Publicly Available (Y or N) No DATE OF RELEASE TO PUBLIC SENSITIVE? No 

*SEE PREVIOUS CONCURRENCE

OFFICE DET/RES D/DET/RES

NAME F Cherny /RA/ M Mayfield/RA/

DATE 06/14 /2002 06/14/2002

OFFICE DD/RES D/RES EDO EDO

NAME J Strosnider/RA/ A Thadani/RA/JS for C Paperiello W Travers

DATE 06/19/2002 06/19 /2002 7/26/02 07/26/02

OFFICIAL RECORD COPY
Attachment

CRITERIA FOR EVALUATING RESEARCH PROJECTS

(Adapted from Nuclear Regulatory Research Philosophy in NRC Five-Year Plan)

1. Is the research relevant to an important or risk-significant regulatory issue? To

what regulatory issue does it pertain? How? Does the importance or risk

significance of the issue warrant a high or at least medium priority office ranking?

2. Are the research results expected to affect resolution of the regulatory issue? 

What is the expected benefit of the knowledge that the research will produce?

3. Is the estimated cost justified by the value of the knowledge to be gained?

4. For the particular issue, are the best people doing the work and are the best

laboratories being used?

5. Has there been competition of ideas in selecting the direction and approach for

the research?

6. For near-term needs, will the results be timely? For longer-term exploratory

research, is it a reasonable expectation that the forthcoming results can be

applied to NRC needs?

7. Is the work within budget? On schedule?

8. Has appropriate peer review been incorporated in the research plan? Are its

results taken into account?

9. Has the work been completed? Has it answered the specific question(s)

addressed?

10. Was the work terminated when the initiating question was answered and the value

of further information tapered off to a point beyond what is worthwhile?

11. Have results been effectively communicated to users? To the public?

12. Have the research results been recognized and used, as applicable, by peers,

users, the international community?