NBSIR 80-2086

.

An Assessment of an Experiment in

Accelerating the Development of

Nuclear Standards
CENTER FOR FIELD METHODS

THE EXPERIMENTAL TECHNOLOGY INCENTIVES PROGRAM

The Experimental Technology Incentives Program (ETIP) of the

National Bureau of Standards pursues an understanding of the

relationships between government policies and technology-based

economic growth. The pursuit of this objective is based on

three premises:

• Technological change is a significant contributor to

social and economic development in the United States.

• Federal, State, and local government policies can influence the rate and direction of technological change.

• Current understanding of this influence and its impact

on social and economic factors is incomplete.

ETIP seeks to improve public policy and the policy research

process in order to facilitate technological change in the

private sector. The program does not pursue technological

change per se. Rather, its mission is to examine and experiment

with government policies and practices in order to identify

and assist in the removal of government-related barriers and

to correct inherent market imperfections that impede the innovation process.

ETIP assists other government agencies in the design and conduct

of policy experiments. Key agency decisionmakers are intimately

involved in these experiments to ensure that the results are

incorporated in the policymaking process. ETIP provides its

agency partners with both analytical assistance and funding

for the experiments while it oversees the evaluation function.

Eecause all government activities potentially can influence

the rate and direction of technological change, ETIP works

with a wide variety of agencies, including those that have

regulatory, procurement, R&D, and subsidy responsibilities.

Programs are currently underway with the General Services Adminis

tration, Food and Drug Administration, Veterans Administration,

Securities and Exchange Commission, Department of Energy, Environ

mental Protection Agency, Occupational Safety and Health Administration, and other Federal agencies as well as various State

and local agencies.

The accompanying report was prepared under contract. Statements

contained in this document represent the views of the originating

organization and do not necessarily reflect those of the National

Bureau of Standards.

Director

Center for Field Methods

National Engineering Laboratory

National Bureau of Standards

U.S. Department of Commerce
yWOMAL

or ctancahim

ixbkahy

NBSIR 80-2086

I A # AN ASSESSMENT OF AN EXPERIMENT IN

ACCELERATING THE DEVELOPMENT OF

NUCLEAR STANDARDS

JUL 2 i 1980

n&k. o cc - G<c,

CiUOO

(JiS £>

lQ8o

C-2-

Jane P. Woodward

Performance Development Institute

2550 M Street, NW

Suite 750

Washington, DC 20037

Stephen D. Garrity

National Bureau of Standards

U.S. Department of Commerce

Washington, DC 20234

June 1 980

Prepared for Experimental Technology Incentives Program

NRC D.//1

6/9/80

AN ASSESSMENT OF AN EXPERIMENT IN

ACCELERATING THE DEVELOPMENT OF

NUCLEAR STANDARDS

by

Jane P. Woodward

Stephen D. Garrity

December 5, 1979

This document has been produced as a working paper of the Regulatory Processes and Effects Project supported by the Department of Commerce Contract

Number E0-A01- 78-00-3 603 . The paper has been developed for use in the work

of the Regulatory Processes and Effects Project and in the development of

Regulatory Administrative Experiments. Information from internal partner

agency discussions is often included in these working materials. Without

written permission from the Experimental Technology Incentives Program

and the Performance Development Institute, the document is not for circulation or quotation. Such use may require further review and editing.
*
i

TABLE OF CONTENTS

Chapter . Page

PREFACE iii

EXECUTIVE SUMMARY v

I. INTRODUCTION 1

II. CONTEXT OF THE EXPERIMENT:

AN OVERVIEW OF THE VOLUNTARY STANDARDS PROCESS 4

A. ANSI AND THE VOLUNTARY STANDARDS PROCESS 5

B. STANDARDS PROCESS IN DETAIL 8

III. THE EXPERIMENTAL INTERVENTION 11

IV. PROJECT ACTIVITIES 15

A. PROJECT ORIGINS 17

B. PROJECT PLANNING 18

C. PROJECT PLAN CHANGES 18

D. EXPERIMENTS CONDUCTED 19

E. REPORTS PRODUCED 21

F. PERSONNEL CHANGES 22

G. PROJECT EVALUATION 22

H. SUBSEQUENT FOLLOW-UP 23

V. ASSESSING THE ETIP/NRC PROJECT 24

A. PROJECT ACTIVITIES 24

B. PROJECT IMPACTS 26

VI. MAJOR FINDINGS AND LESSONS LEARNED 35

A. THE KNOWNS 35

B. THE UNKNOWNS 37

C. USEFULNESS OF FURTHER STUDY 38

D. SUMMARY 40
ii

LIST OF FIGURES

Figure Page

1 EXPERIMENTAL EXPECTATIONS 1

2 VOLUNTARY STANDARDS PROCESS 8

3 LOCATION OF THE PROJECT EXPERIMENT 12

4 EXPERIMENTAL ASSUMPTIONS 13

5 RESEARCH PROJECTED IF THE PROJECT PROVED

SUCCESSFUL IN REDUCING DRAFTING TIME 14

6 OVERVIEW OF PROJECT ACTIVITIES 16

7 INTENDED VS. REPORTED EVENTS 25

8 EXPERIMENTS CONDUCTED AND RESULTS REPORTED 34

LIST OF APPENDICES

APPENDIX A: INFORMATION SOURCES

APPENDIX B: ACCELERATING THE ESTABLISHMENT OF STANDARDS

AND THEIR ADOPTION AS REGULATORY GUIDES BY

THE ATOMIC ENERGY COMMISSION (PROJECT PLAN)

APPENDIX C: DETAILED CHRONOLOGY

APPENDIX D: A REVIEW OF THE NUCLEAR REGULATORY EXPERIMENT

APPENDIX E: DESCRIPTIONS OF EXPERIMENTS #1 THRU 4
PREFACE

This paper assesses an experiment conducted by the Experimental

Technology Incentives Program (ETIP) in conjunction with the Nuclear

Regulatory Commission (NFC) . This assessment was carried out as part of

analytic support contracts awarded to The Urban Institute and the Performance

Development Institute^- by ETIP.

The purpose of the assessment is twofold:

(1) To review the experiment and examine whether further detailed

evaluation is warranted, and

(2) To determine what lessons can be learned about the conduct of

ETIP experiments in general.

The underlying intent throughout has been to uncover the facts about the

experiments so that an objective, accurate assessment could be made.

Information for the paper was gathered by reviewing available documentation on the project and on the organizations involved, and by interviewing

individual participants. The original designers of the experiment have been

very helpful in this regard. The results of the study represent the views

of the authors and are meant as a critique of method, not the designers.

The lessons learned from the assessment should help improve the design of

future ETIP experiments.

ETIP's goal is to develop policy guidelines relating to the impact of

government on industry and on technological change as well as a body of

knowledge necessary for their effective use. To achieve these goals, ETIP

has been carrying out experiments with regulatory agencies to develop a

1. Department of Commerce Contract #7-35822 to The Urban Institute,

October 1, 1977 to December 31, 1978. Department of Commerce Contract

#E0-78-3603 to the Performance Development Institute for October 1, 1978

to December 31, 1979.
iv

knowledge base and policy recommendations on how regulatory processes can

be changed to bring about the desired commercial impact and to improve the

environment for technological innovation. Under its contract with ETIP,

The Urban Institute and the Performance Development Institute have provided

analytic and data collection support.

One assumption underlying ETIP experimentation is that the rate and

direction of private sector activity is influenced by the regulatory process.

ETIP also assumes that innovation plays a leading role in economic and social

growth. A final assumption is that administrative experimentation will lead

to better understanding and control of the regulatory processes involved.

A major focus of ETIP's work is on the regulatory development process.

This process often involves lengthy time delays, known as "regulatory lag."

One example of the lag problem exists in the nuclear regulatory field, where

the Nuclear Regulatory Commission (NRC) adopts standards for the licensing

of nuclear reactors.

The NRC's mandate is to assure that non-military uses of nuclear

materials and facilities are consistent with the laws governing the public

health and safety, environmental quality, national security and antitrust.

The NRC implements this mandate through the licensing of nuclear materials

and facilities and the conduct of programs to ensure compliance. Standards

for nuclear plants are included in regulations or regulatory guides for

companies seeking nuclear power plant licenses.

ETIP's selection of the NRC and the standards-setting process was based

upon public concern with energy problems and with the costly delay in the

licensing of nuclear power plants. The absence of standards for nuclear

power plants was thought to contribute to such delays. Thus the goal of the

ETIP project described in this report was to explore ways to hasten both the

establishment of standards and their adoption by the NRC.
V

EXECUTIVE SUMMARY

In March 1974, the Experimental Technology Incentives Program (ETIP)

working in conjunction with the Nuclear Regulatory Commission (NRC) and the

American National Standards Institute (ANSI) completed a project plan to

experiment with accelerating the initial drafting of nuclear power plant

standards by drafting subcommittees. These drafting subcommittees were

staffed by volunteers from private industry and government, as part of the

voluntary standards development process. The plan outlined the following

activities

:

1. ETIP was to prepare a project plan to test the following four

changes from the normal standard drafting process for three or four standards (each standard was to be one experiment): —Hiring a full-time working group chairman, —Providing consultant services for data collection/correlation, —Providing clerical/ technical editing services, —Conducting extended working meetings, possibly up to 14 days.

2. ETIP would obligate funds for the experiments for NRC use.

3. NRC would contract with ANSI to conduct experiments.

4. ANSI would hire a project manager to coordinate the individual

experiments and monitor the effects of the projects on the

development of standards.

5. Each experiment would differ in terms of the variables tested,

(see //I above) , although the committees could choose the variables

to be tested.

6. ETIP would contract with an evaluator to conduct a project evaluation, to assess: —whether standard development schedules were met, —quality of standards produced, —whether a more effective/efficient use of working group

members resulted, —whether drafting time reduction offset project costs.

7. ETIP would conduct further research as needed, depending on results.
vi

The standard-drafting experiments were expected to accelerate the production of draft standards, which in turn was expected to accelerate the production of final standards and their ultimate adoption by the NRC. If the project

was successful, multiple benefits (e.g., reduction of nuclear reactor licensing

delays and industry costs) were anticipated.

The purpose of this report is to describe and assess the experiments

conducted, to report the lessons learned from the project, and to review

the usefulness of further research. It is based on an open, objective inquiry

about the facts of the project. To the extent possible, the findings from

the assessment are included in the report so that readers can understand the

project and draw their own conclusions.

A. THE EXPERIMENTAL INTERVENTION

The actual intervention conducted varied from the initial plan. ANSI

hired two contractors to conduct the experiments. Battelle-Northwest

(contractor #1) undertook experiments with three separate subcommittees.

The variables tested included: —Use of a concentrated five-day working meeting to draft

the standard (instead of several one or two-day sessions,

held over many months)

—Premeeting preparation of a working draft outline. —Provision of an executive secretary/ coordinator

—Provision of clerical/duplicating support. —Payment of working group chairmen. —Payment of travel and living expenses of work group members. —Holding of preliminary organizational meetings before the

five-day working session. —Limiting working group membership to 10 members.
vii

Anderson, the second contractor, limited his experiment to the provision

of secretarial support to the subcommittees involved. Each of the contractors

submitted reports on their project experiments, but these were not intended

to be comprehensive impact evaluations. Impact evaluations were planned by

project designers, but were not funded due to complications in contracting and

program changes in ETIP.

B. ASSESSING THE NUCLEAR REGULATORY EXPERIMENT

Three criteria were identified in the project plan as important in

evaluating project success:

• Length of time required to produce the standard (as compared

to "normal" time requirements) .

e Quality of the standard produced.

• Cost

.

The Battelle and Anderson reports provided some information on these

criteria even though the two reports were not intended to be comprehensive

evaluations. The following sections review the data available on each criterion.

1 . TIME

In the Battelle experiment, the time required to draft the three standards

was considered to be faster than the norm for the production of draft standards.

While the Anderson report provided insufficient detail to assess the results

reported, that report also concluded that the experiment helped to speed production of draft standards.

Neither of the reports associated with the experiment traced the time

required for the draft standard to proceed through the remainder of the

standards development process. This trace would have been needed to determine
the overall time required to produce the finished adopted standard. Original

plans had discussed doing this. However, even if the traces had been carried

out, a comparison to normal conditions would have been difficult since it

appears that little data is available on the normal time required to produce

a standard.

2. QUALITY

No precise criteria exist for evaluating the quality of standards produced

through the voluntary standards process, and none of the reports associated

with the experiment attempted to assess the quality of the standards produced.

The evaluation study which had been projected to assess standard quality was

never contracted out .

3. COST

Neither report indicated exactly how the money was spent or allocated

among the experimental variables. It was thus impossible to identify the cost

of implementing the individual variables.

C. LESSONS LEARNED

Three experimental changes—provision of secretarial assistance, clerical/

duplicating support, and the five-day work week—were reported to have accelerated the initial drafting process. The evidence supporting these claims is weak,

however, and the effect of these changes remains unknown. Nevertheless, there

were important lessons learned from the experiment, and these relate to the

experimental process rather than to the outcomes of the experiments conducted.
ix

1 . THE KNOWNS

There were a number of problems in the experiment:

• As a result of having to transfer funds through the NRC and

ANSI, ETIP was unable to maintain close control over the

design, development and implementation of the experiments.

• Because of planning and funding delays, the evaluation originally planned to assess project outcomes and impacts did not

take place.

• Information provided by contract"’’ 0 on key project variables

and outcomes was incomplete.

• The design initially chosen for the project was limited in

the amount of information it could actually provide.

• The initial design did not clearly lay out how the proposed

experiment fit into the major ETIP research needs, or into

the overall standards development process.

• No background information was collected at the outset to initially check the validity of the experimental assumptions. A

questioning of these assumptions by ETIP staff might have permitted a more defensible targeting of the experimental resources,

and questions about the project's relevancy could have been better

addressed both during the experiment and afterward.

2. THE UNKNOWNS

Because of the problems with experimental design and the lack of a comprehensive, follow-on evaluation, much remains uncertain at this time about

the impacts of the experimental changes. Little or no information was

collected on such factors as:

• average time for standards drafting and processing,

• the role of existing standards in the plant licensing process,

• quality of the standards produced,

• cost of introducing the individual innovations, compared

with normal production costs, and

• the details of the experiments conducted

It would be difficult and costly to obtain information on these factors at

this point in the project.
X

3. USEFULNESS OF FURTHER STUDY

ETIP has several options relative to the conduct of further study on

the voluntary standards program:

Option 1 : Terminate the research at the conclusion of the present

review.

Option 2 : Fund a limited research effort to gather additional background information about the experiments conducted and their impacts (e.g.,

the costs of the individual innovations, the quality of the standards

produced) as well as about the "normal" development process and time requirements in the nuclear standards area, with which to compare the experimental

results

Option 3 : Fund a substantial research effort to identify the role of

standards, if any, in accelerating nuclear plant licensing, using a research

design and a contracting process which will address the difficulties experienced in the initial experiment. Assuming that standards are found to be a

substantial factor in accelerating nuclear plant licensing, ETIP could consider

the usefulness of investigating how and whether the voluntary standards process

can be successfully accelerated to develop those standards.

Option 4 : Fund an evaluability assessment of the usefulness of conducting

further study of the voluntary standards process and the impact of standards on

nuclear power plant licensing. This would involve collecting sufficient background information about the proposed research that the costs and likely results

and benefits from pursuing the various research options could be assessed prior

to committing funds to any particular research design.

It is the conclusion of this assessment that if further research is contemplated, Option 4 is the most highly recommended. The conduct of an evaluability assessment would provide ETIP with guidance as to the areas where
xi

further research is desirable and likely to be fruitful. Option 1 is the

alternative best choice if sufficient resources are not available to make it

worthwhile to undertake an evaluability assessment.

4. SUMMARY

While the drafting process was accelerated, the research design and the

lack of sufficient data to assess the experimental assumptions at the design

stage produced an experiment that was unable to demonstrate whether the

acceleration of the drafting stage accelerated the entire standards development

process and subsequent NRC endorsement.

I. INTRODUCTION

In 1975, the Experimental Technology Incentives Program (ETIP) undertook

an experiment to acccelerate the establishment of standards and their adoption

as regulatory guides by the Atomic Energy Commission (now the Nuclear Regulatory Commission, or NRC) . This particular experiment was selected because of

political interest at that time with reducing nuclear plant licensing delays.

Licensing delays were attributed in part to the absence of nuclear power plant

standards. Accelerating the development of nuclear standards was expected to

reduce such delays. Figure 1 below depicts the assumptions underlying the

experiment

.

ETIP/NRC

ExperimentDraft standardsare produced —

more rapidly

Final standards - are produced and- are adopted by

NRC more rapidly

Standards are -used by NRC

and industry

Regulatory review

required for

reactor licensing

is reduced for

components where

standards are used

Reduction in reactor

licensing delays

Reduction in industry

and government costs

Stimulation of industry

investment in technology

Stimulation of timely

implementation of technology

Reduction of human error

and increased reactor safety

FIGURE 1: EXPERIMENTAL EXPECTATIONS
2

The project was viewed as only the first of a series of ETIP projects to

examine the relationship between voluntary or industry-based standards and

regulatory processes.

The purpose of this report is to describe the experiment conducted and

its findings, and to depict how this particular experiment fit into the overall scheme of ETIP experimentation. It follows up on and in part relies on

information gathered in an ETIP case study of the experiment conducted in 1976

and reported in January, 1977.^ This report also purports to assess the

overall effectiveness of the experiment and to pinpoint the problems encountered and how such problems might be avoided in the future, although it is not

2 an evaluation of the experiment per se . Finally, the report traces the

standards produced as part of the experiments, and presents options for ETIP's

consideration in terms of further study which might be undertaken.

Information for this report was gathered by reviewing available documentation on the project and on the organizations involved, and by interviewing individual participants. Individuals from the American National

Standards Institute (ANSI), the NRC, the Experimental Technology Incentives

Program (ETIP), and the National Bureau of Standards (NBS) were interviewed,

either in person or by telephone. (Appendix A lists the individuals interviewed and the major documentation reviewed.)

1. Garrity, Stephen, "A Review of the Nuclear Regulatory Experiment,"

unpublished draft, Washington, D.C.: Experimental Technology Incentives

Program, National Bureau of Standards, January 11, 1977. See Appendix D.

2. ETIP had planned to contract out an evaluation to trace the impacts

of the experiment. Planning for the evaluation occurred after the experiment

had been initiated, however, and although money for the evaluation was obligated, no proposals submitted for the evaluation were funded.

I
3

The report is organized into several sections. Section II describes

the context within which the experiment was conducted: the main players or

"stakeholders," and their established roles in the voluntary standards process.

Section III outlines the expectations held by the project planners when the

experiment was initially designed: the assumptions which the experiment was

intended to test. Section IV describes the project activities as they were

reported by project documentation and the individuals interviewed, and the

products produced. Section V then contrasts what was expected to occur with

what, in fact, did occur, and describes what was learned from the experiment.

The final section addresses major findings, what remains unknown, options for

further study, and "lessons learned" from the experiment applicable to the

design of future experiments.
II. CONTEXT OF THE EXPERIMENT: AN OVERVIEW OF THE

VOLUNTARY STANDARDS PROCESS

The primary function of government agencies, such as the NRC, is to implement Congressional legislation. A crucial NRC mandate, for example, is to

ensure "a high level of public health and safety and environmental protection

in the design, construction, and operation of nuclear power reactors."^ One

means for carrying out this mandate is through NRC promulgation or adoption of

2 standards for the licensing of nuclear reactors. The NRC then includes these

standards in its regulatory program as regulations or regulatory guides for

3 companies interested in obtaining nuclear power plant licenses.

The NRC's standards have been viewed as serving the following major purposes, to:

1. define information required from an applicant for license,

2. define acceptable levels of safety and environmental protection,

3. provide inspection and enforcement criteria,

4. limit contested issues in hearings.

1. Testimony of Lester Rogers, Director of Regulatory Standards, U.S.

Atomic Energy Commission, "Hearings on Reactor Safety before the Joint Committee on Atomic Energy," September 26, 1973, p. 1.

2. Standards can be defined as written specifications (codes and criteria) establishing minimum requirements for the technical design, construction

and performance of specific products or for the conduct of specific activities.

3. As noted in the Rogers' testimony, regulations "set forth both general

and specific requirements which must be met" by a licensing applicant. Regulatory guides, on the other hand, provide "guidance to applicants and licensees

on ways acceptable to the regulatory staff of complying with the general and

specific regulations" (see page 2). However, compliance with the regulatory

guides is not required, if an alternative and acceptable way of complying with

the regulations is proposed.
5

5. inform the public,

6. provide models and methodology for assessing levels of safety,

safeguards, and environmental protection, and

7. help provide a basis for standardization of plants.

The NRC can develop its own standards for inclusion in the regulations

and regulatory guides. Alternatively, when it recognizes the need for a

standard, the NRC may choose to adopt or adapt an existing standard if the

needed standard already has been developed by an interest group (e.g., a particular industry or a consumer group) . Where the needed standard has not yet

been developed, the NRC can request that a standard be developed by submitting

its request to a group such as the American National Standards Institute (ANSI)

.

That request would be handled by the appropriate committee within ANSI—in

this case of nuclear standards, the Nuclear Standards Management Board (NSMB) .

The NSMB would be responsible for coordinating the standard development effort,

the end product of which would be submitted to the NRC (as was done in the

experiment discussed herein)

.

A. ANSI AND THE VOLUNTARY STANDARDS PROCESS

Thousands of technical, product-related standards have been generated in

the United States; over 8000 have been adopted by ANSI as American National

2 Standards. Many such standards have been generated by industry on a

voluntary basis (as opposed to government-developed standards imposed through

regulation). Over 400 trade associations, professional societies, and other

similar interest groups (including consumer groups) have sponsored and coordinated the standards-development efforts. (Standard-setting may comprise from

1. Testimony of Lester Rogers, p.l.

2. Interview with ANSI official.
10-30 percent of the typical association's activity.) This standards development process is known as the industry-based or voluntary standards system.

ANSI is a federation of standards-developing and standards-using organizations comprising approximately 900 companies and 200 trade, technical, labor,

and consumer organizations. ANSI's main functions include managing and coordinating the voluntary development of national standards by its participating

members, and approving such standards as American National Standards if they

meet ANSI's consensus requirements. ANSI's standards development process

emphasizes procedural fairness, a balancing of the interests represented at

the drafting and review stages, and a consensus approach to standard adoption.

According to an ANSI report:

These (consensus) requirements ensure that all substantially

affected interests have had an opportunity to participate in the

standard's development or to comment on its provisions and that

the standard has achieved general recognition and acceptance for

use. Organizations that submit standards to ANSI for approval

must supply evidence that these and related consensus criteria

have been met. ANSI then conducts independent verification of

the adequacy of consensus before acting on approval.

Under the ANSI consensus approach, something greater than a majority but less

than unanimity, is required. Any substantive objection to the standard must

be reviewed and responded to by the drafting group. If the objecting party

is not satisfied by the response, the objection accompanies the draft standard

through the entire review process to publication. If the substantive objections are numerous and unresolvable , the standard must be withdrawn.

ANSI's public review or "canvassing" process can involve sending a proposed standard up to 14,000 individuals and groups, including media outlets

and governmental agencies. Standards which survive the review process without

1. "ANSI Progress Report, 1977," New York: American National Standards

Institute, March 1977, p. 1.
7

major substantive objections (or where the substantive objections are in the

main resolved through standard revision) are adopted as American National

Standards .

ANSI coordinates the standard- se tting activities of its member groups.

It is organized into a number of committees handling particular subject areas.

For example, the Nuclear Standards Management Board (NSMB) is responsible

for overseeing the nuclear standards program, including policy, financing,

and recruiting Governmental or industry requests for particular

standards (e.g., from the NRC) are funnelled through individual ANSI committees such as the NSMB. Generally at least one federal government employee

is a member of such ANSI committees.

ANSI is dependant upon membership dues and the sale of its published

standards for its main support, although it does receive some special project

support. Less than 3 percent of its operating budget comes from government

2 sources in the form of grants for specified programs. One ANSI employee

indicated that the government estimates the voluntary standardization process

to cost over $500 million per year in volunteer labor.

Figure 2 below depicts the ANSI standard development process in summary

form, showing the participants in the process and their respective roles.

Hie process is described in greater detail in the following pages.

1. Ibid , p. 12.

2. Ibid , pp. 11-13.
8

Federal

request

ANSI

request

Draft

to ANSI

NRC NRC publish

Review—^ via regulation

if adopted

i

t

Standards

-developed by

subcommittee

Draft

standards

[

Publish

I

SellI

• Voluntary

use if not

adopted by

government

Mandated

use

Public

-review

if rejected

FIGURE 2: VOLUNTARY STANDARDS PROCESS

UTILIZED BY NRC

B. STANDARDS PROCESS IN DETAIL

Interviews revealed that in the usual standard-setting process, the

interest group which has "responsibility" over the standard area and which

follows ANSI-prescribed consensus procedures works in cooperation with ANSI.

The typical standard development pattern was described by ANSI staff and other

participants as follows

1. An entity ( individual , group, government) proposes to an interest

group or to ANSI that a standard is needed for an activity. If proposed to ANSI, the appropriate ANSI committee refers the standard to

the appropriate interest group [e.g., the American Nuclear Society

(ANS)] . The interest group is then responsible for seeing that the

draft standard is developed in line with ANSI's procedureal requirements .

2. The interest group refers the matter to the committee responsible for

the particular activity involved. If no committee is appropriate, the

group notifies its membership, requesting those interested to volunteer

to form a committee to draft the standard. If a sufficient number respond, the committee members are selected from those responding. (If

too few respond, no action on the proposal is taken.) An effort is

generally made to balance the various interests on the committee. A

government official may or may not serve on the committee.
3.

A subcommittee chair and members are selected as the writing group

for the committee to prepare an initial draft of the standard. Draft

preparation typically involves the following steps:

a. The first one-day meeting: The initial meeting is mainly

organizational, to discover interests, information needs, and

how to organize knowledge. Sections to be drafted may be

assigned to individual members.

b. Second meeting held several months later: Section drafts are probably not completed; discussion focuses on resolving

various interest conflicts and substantive issues.

c. Third meeting held some months later: Section drafts circulated in advance are discussed, issues/interests worked on.

d. Additional meetings are held.l

2 e. The working group votes on/accepts the draft.

4. The draft standard is submitted to the full committee for voting.

(Objections are responded to by the writing group.)

5. The draft standard (if accepted by the full committee) is submitted

to the interest group membership as a whole for voting. (Objections

are responded to by the writing group.)

6. The draft standard (if accepted by the interest group) is submitted

to ANSI for canvassing (submission for public review and comment by

the 14,000 individuals and organizations on its review list.) Comments are returned to the writing group for response.

1. The number of meetings required and the length of time between

meetings typically depends on such factors as the status of knowledge about

the activity (whether studies need to be conducted or the activity has been

studied and knowledge is fairly well codified) , the extent of controversy

about the activity, practices of the interest group involved, use or non-use

of the consensus approach, the urgency of need for the standard, etc. Meetings

are generally several months apart, since they must be fitted in on a volunteer

basis on time contributed by the individuals involved and expenses underwritten

by the companies or governmental entities employing the individuals. An interest group staff member may or may not be assigned to provide administrative,

logical, and clerical support.

2. An interest group typically has certain required voting procedures

as well as an appeals process at each voting level. For example, the voting

procedures may require that 80% of the members vote, and that 70% of the votes

be positive. In addition, where the voluntary consensus process is used, the

rules generally require that all objections be examined and responded to, and

a rationale provided for why the objection is being overruled if the objecting

party is not satisfied with the writing group's response. If the objecting

member is dissatisfied, the member may appeal the group's decision, with the

final level of appeal being a decision by the board of directors for the

particular interest group. Rejection at any level throws the standard back

to the writing group to respond to the objections raised.
10

7. The writing group responds to the comments by making revisions or providing a rationale for why revision is not appropriate.

8. ANSI reviews the comments and responses and determines whether ANSI

consensus criteria are met, permitting standard adoption.

9. ANSI adopts and publishes the standard as an American National

Standard (if consensus criteria are met). Interested parties

(including the government) may purchase and use the standard.

10. The pertinent government agency (e.g., the NRC) reviews the ANSI

standard. (An optional step for the agency.)

11. The agency endorses the standard, fails to endorse the standard,

or endorses the standard with exceptions. If endorsed or endorsed

with exceptions, the standard is incorporated as part of a regulatory guide or as a regulation. (If not endorsed, the agency may

request revisions or draft its own standard.)

12. Those regulated (industry groups, manufacturers, licensees) are required (if the standard is made a regulation) or encouraged (if

the standard is incorporated in a regulatory guide) to use the

standard endorsed by the government agency.

I

1. ANSI's consensus approach is described on page 5, i.e., the requirements of greater than majority approval and subcommittee response to all

substantive objections raised.

2. The NRC's Director of Regulation has made a commitment that an ANSIapproved nuclear standard will be reviewed and action taken to implement the

standard within 90 days, or ANSI will be informed why adoption did not occur.
III. THE EXPERIMENTAL INTERVENTION

The goal of this project was to accelerate the establishment of standards

and their adoption by the NRC as regulatory guides. The expectation of the

project planners was that by altering the conditions und^” which the interest

group drafting subcommittees wrked, the time needed to draft the initial

standard could be reduced. This time savings, in turn, was expected to reduce

the overall time period required to produce a completed standard and get it

adopted by the NRC. The project plan (see Appendix B) outlined the activities

expected to occur as follows:

1. ETIP prepares project plan to test four changes from the normal

standard drafting process, for 3 or 4 standards: — Hiring full-time drafting subcommittee chairman — Providing consultant services for data collection/correlation — Providing clerical/ technical editing services — Conducting up to 14 -day working meeting

2. ETIP obligates funds for the experiment for NRC use

3. The NRC contracts with ANSI to conduct experiments

4. ANSI hires project manager to coordinate project experiments and

monitor effect of project on development of standards

5. Each experiment differs in terms of variables tested, although

committees may choose variables to be tested

6. ETIP contracts with evaluator to conduct project evaluation

to assess: — Whether standard development schedules were met — Quality of standards produced — Whether a more ef fective/eff icient use of working group

members resulted — Whether drafting time reduction offset project costs

7. ETIP conducts further research as needed, depending on results.
Figure 3 summarizes the typical steps for developing nuclear standards

using the MSI voluntary development process, and illustrates the point in

that process where the project experimental intervention occurred.

FIGURE 3: LOCATION OF THE PROJECT EXPERIMENT
13

The expectations underlying the experiment were most clearly outlined

in the project plan for the proposed experiment (see Appendix B) . These

expectations are depicted in Figure 4. The accuracy of these expectations

was reinforced by subsequent interviews by the author with ETIP, NRC, NBS,

and ANSI staff members.

ETIP obligates

funds for

the project

NRC contracts

with ANSI to - conduct experiments

are produced

more rapidly

-produced and

are adopted

by NRC more

rapidly

ANSI coordinates four

standard-drafting

experiments

-are used

by NRC and

industry

Regulatory review

required for

reactor licensing

is reduced for

components where

standards are used

Reduction in reactor

licensing delays

Reduction in industry

and government costs

Stimulation of industry

investment in technology

Stimulation of timely

implementation of technology

Reduction of human error

and increased reactor safety

FIGURE 4: EXPERIMENTAL ASSUMPTIONS

1. The major assumption underlying the voluntary standards process is

that industry production of standards will produce greater likelihood of

industry compliance. This assumption has not been empirically tested.

I
14

The experiments to accelerate the production of first-draft standards

by altering drafting conditions also were foreseen as only one part of

a larger experimental framework. As the initial project plan discussed,

this was to be "the first of a series of projects" in which ETIP planned

to "examine the relationship between voluntary, or industry-based standards

and the regulatory process."^ Figure 5 illustrates the additional research

contemplated by ETIP and discussed in the project plan at the time of the

initial experiment, of which this project was only a first step.

I

Experiment in adoption \

of comprehensive code >"'T

l ime \

of regulations and reduction?

regulatory guidelines

Yes

Investigate why no private sector

action taken to accelerate process

Investigate how to improve

existing system, use private

resources more effectively

Investigate the public benefits

of code adoption and whether

the government should routinely

sponsor code development

FIGURE 5: RESEARCH PROJECTED IF THE PROJECT PROVED

SUCCESSFUL IN REDUCING DRAFTING TIME

1. "Accelerating the Establishment of Standards and Their Adoption As

Regulatory Guides by the Atomic Energy Commission," Project Plan, Washington,

D.C.: Experimental Technology Incentives Program, March 21, 1974.
IV. PROJECT ACTIVITIES

The current research traced the actual project activities from their

origins as far as direct results could be found. Figure 6 is a model of the

project as it occurred, tracing the money obligated and spent, the activities undertaken, and the resulting products.

A detailed chronology of events and products related to the experiment

was constructed by reviewing project documentation and by interviewing involved parties from the NRC, ETIP, NBS, and ANSI. This is reported as Appendix C. In summary, the main events were as follows:

• 1973-1974 — ETIP, NBS, NRC, and ANSI discussed the joint

project to accelerate the standards process; the project plan

and guidelines were prepared.

• June 1974 — ETIP obligated project funds to the NRC to initiate

the project in accordance with the project plan and guidelines.

• June 1974 — ANSI decided to contract out the experiment instead

of hiring a program manager as specified in the plan, and amended

the interventions to be made.

• October 1974 — ANSI contracted with Battelle-Northwest to coordi

nate three experiments.

• November 1974 — ANSI contracted with Harlan Anderson to conduct

a fourth experiment.

• March, 1975 — ETIP developed a project plan for the project eval

uation

.

• 1975 — Experiments were conducted by Bat telle and Anderson; no

satisfactory contractor for the project evaluation was identified

Battelle submitted its report on the experimental process, with a subjective assessment by the participants.

• February 1976 — Anderson submitted his report.
r 16

u
17

e November 1976 — Funds for the project evaluation were deobligated , and no impact evaluation was conducted.

• 19 76 — A review of the Nuclear Regulatory experiment was con- ducted by ETIP staff and a case study report submitted in January,

1977 . 1

A. PROJECT ORIGINS

Early in 1973, ETIP was searching for viable experimental projects, partially through requests for "Qualifications and Interests" Statements. Dr. Jim

Leiss, Director, Center for Radiation Research at NBS and member of the ANSI

Nuclear Standards Management Board, proposed an experiment on accelerating the

voluntary standards process in response to the requests. ANSI responded to

ETIP's request for "Qualification and Interest" statements by suggesting the

idea that ETIP reimburse their committee members for travel expenses. Although

ETIP was not initially interested in these proposals, the discussions with ANSI

led to contact with Lester Rogers, the Director of Regulatory Standards for NRC.

Rogers wanted to improve the voluntary standards process at ANSI. This interest,

plus an emerging political concern with reducing nuclear plant licensing lag,

eventually stimulated ETIP's interest in the idea. From this time on, ETIP

had continuing contact with Carl Johnson from the Regulatory Standards office.

ETIP's selection of the NRC and the ANSI volunteer standards process as

a regulatory experiment was based on a strong public concern at the time with

the U.S. energy problems and with the costly delay in the licensing of nuclear

power plants. Licensing delays were attributed in part to the absence of

nuclear power plant standards. An experiment which would test ways to speed

up the development of standards was thus particularly attractive.

1. Garrity, Stephen, "A Review of the Nuclear Regulatory Experiment,"

unpublished draft, Washington, D.C.: Experimental Technology Incentives

Program, National Bureau of Standards, January 11, 1977, See Appendix D.
B. PROJECT PLANNING

In March of 1974, ETIP completed a project plan for the NRC experiment.

(See Appendix B.) The plan indicated that the NRC project was the first

in a series of projects examining the relationship between voluntary standards

and the regulatory process. The following four changes in the subcommittee

standards-draf ting process were proposed: paying a full-time committee chairman, using consulting services for information gathering, providing clerical

and editing assistance, and using more lengthy meetings.

Under the plan, the Nuclear Technical Advisory Board (NTAB) in ANSI

(established by the NSMB) would assist the NRC in managing the project. They

were supposed to select standards and hire a program manager. The NRC would

retain overall approval authority. While there was no interagency agreement

between ETIP and the NRC because there was no actual transfer of funds, there

were letters of commitment exchanged between them. ETIP held an account

dedicated to the NRC project. ANSI would bill the NRC and the NRC would then

bill the ETIP/NBS account.

In May, 1974, the project plan was approved, and $100,000 was obligated

for NRC/ ANSI use.

C. PROJECT PLAN CHANGES

Contrary to the project plan, ANSI decided to contract out the experiment,

instead of hiring a program manager to coordinate the intervention and conduct

the experiment. It is unclear who at ANSI or the NRC made this decision and

why it was made. According to the ETIP review of the project, "ANSI ...

reworded the language of the plan specifying the interventions to be used.
19

These interventions became illustrative, leaving ANSI the option to suggest

other interventions it felt important

Thus in June, 1974, ANSI issued a request for proposals for the experiment, and in August, Battelle-Northwest submitted a proposal to look at

different variables in three committees. (The standards chosen for the

experiment were selected partly because of the urgency of need for the particular standards.) A contract for $80,000 was signed with Battelle in

October, 1974. Also in October, Harlan Anderson, from Westinghouse-Hanford

Corporation, submitted a proposal to study the effects of one variable in a

single committee. That proposal was funded in November for $5000.

D. EXPERIMENTS CONDUCTED

Four experiments were run using four different committees, each working

on the drafting of a single standard. The changes from the normal committee

operation included

:

• Use of a concentrated five-day working meeting to draft the

standard (instead of several one or two-day sessions, held over many months)

• Premeeting preparation of a working draft or outline

• Provision of an executive secretary/coordinater

• Provision of clerical/duplicating support

• Payment of working group chairmen

• Payment of travel and living expenses of drafting subcommittee

members

• Holding of preliminary organizational meetings before the fiveday work session

• Limiting working group membership to 10 members

1. Ibid . , p .16
20

Battelle-Northwest conducted experiments with three subcommittees, each

subcommittee working on a different standard. These included:

Two of the subcommittees were subcommittees within the American Nuclear

Society (ANS) . One was a subcommittee of the Institute of Nuclear Materials

Management (INMM) . (Appendix E gives more detailed information on each of

the experiments conducted.)

Each committee chair was permitted to select the variables to be tested;

the variables chosen proved very similar. All three committees tested the

five-day working meeting, use of an executive secretary, provision of clerical and duplicating support, and the limiting of the drafting subcommittee

members to approximately ten. Two of the committees paid the chairmen and the

travel and living expenses of the drafting subcommittee members. Two subcommit<

tees had subcommittee members draft and prepare sections of an initial working

draft, so that a working draft would be ready for use at the beginning of the

five-day work session. One subcommittee had the paid consultant/ chairman prepare a working outline for the work session. And two of the subcommittees had

preliminary organizational meetings before the five-day work session was held.

Harlan Anderson focused on only one variable. Anderson was chairman of

ANSI Committee N-ll, and used the $5000 grant to pay for the provision of secretarial assistance to the very active standard s-writing Committee on Fuel,

Control, and Moderator Materials for Nuclear Reactor Applications (C-26) within

the American Society of Testing and Nuclear Materials (AS1M) . Anderson was

also chair of the experimenting ASTM committee, and could directly observe the

Subcommittee Standard

ANS 2.12

ANS 18.6

INMM-11

Guidelines for Combining Natural Phenomena and

Manmade Hazards at Power Reactor Sites

Discharge of Thermal Effluents Into Surface Waters

Criteria and Standards for the Certification of

Nuclear Materials Managers

working sessions of the drafting subcommittees.
21

E. REPORTS PRODUCED

Battelle Northwest : Battelle-Northwest submitted a report on the experiments in September, 1975. In that report, Battelle provided information on:

• The experiment conducted, including the subcommittee members

and the experimental variables tested,

• Costs of funding each subcommittee's changes, and

• Each subcommittee's subjective ratings, based on a five- point

scale, of: — the usefulness of the specific innovations, — the appropriateness of the subcommittee composition, — skills of the chairman, — difficulty level of the standard, — overall subcommittee performance.

Battelle also furnished copies of the draft standards produced.

No information was provided on:

• Actual time for producing the standards,

• Standard quality,

• Follow-up on what happened to the standards, once drafted.

Harlan Anderson : Anderson submitted a brief report in February, 1976,

including a discussion of:

• The variable tested,

• The background of the consensus standards process,

• His subjective assessment of results, based on his own observations and the comments and reaction of other committee members,

• Recommendations that further funding be provided,

• Project costs.

Copies of meeting minutes were also provided.

No information was provided on:

• Standards drafted, if any, or

• Actual time taken to produce standards.
22

F. PERSONNEL CHANGES

A number of personnel changes occurred at the NRC and ETIP during the

course of the project. In December, 1974, Rogers, the Director of the NRC's

Office of Regulatory Standards and the initial NRC supporter of the experiment,

retired. In September, 1975, the original ETIP project monitor (Phil Harter)

left. Also, Victor Berlin joined ETIP staff, and began discussing the need

for a wider focus on ANSI's standards development process.

The precise impact of these changes is unclear. They did provide a certain discontinuity with previous project plans and implementation. In the

author's opinion, they may have lessened the understanding of and support for

the project both at the NRC and at ETIP. This was because of the uncertainties

created over the value of the original project and the change of interests.

At the minimum, project focus was somewhat changed.

G. PROJECT EVALUATION

1

The initial plan for the project foresaw the conduct of an evaluation

of the proposed experiment. During March of 1975, a project plan for the

proposed evaluation was prepared at ETIP. The plan proposed to develop background information on the standards development process, and to prepare a

detailed review of the experiment, including the postexperiment history of

the standard drafts, the project costs and an identification of what

recommendations could be made.

The evaluation plan was approved in April, 1975, and project funds were

obligated in June. In December, ANSI sent out requests for proposals for the

conduct of the evaluation, and in January, 1976, three proposals were submitted.

1. The details of this discussion were derived in large part from the

ETIP Review of the experiment by Garrity, discussed below. See Appendix D.
23

None of the proposals was, however, accepted by the NRC, and in February,

ANSI submitted a proposal to the NRC to conduct the evaluation itself.

Although the NRC attempted to put through sole-source contracting with

ANSI for the evaluation, there proved to be insufficient justification for a

sole-source contract. Although ANSI submitted a second and revised evaluation

proposal in June, 1976, no NRC action was taken.

H. SUBSEQUENT FOLLOW-UP

In July, 1976, Steve Garrity joined the ETIP staff. At that time, he

was requested to undertake a review of the nuclear regulatory experiment.

After considerable investigation, the report based on his review was completed

in January, 19 77. In this report, Garrity described and then critiqued the

experiments conducted. A copy of the Garrity report is attached as Appendix D.

ETIP requested that the current assessment of the project be conducted

as part of the analytic support contract awarded to The Urban Institute in

October 1977. A major purpose of this additional effort was to determine

if further study was warranted.
V. ASSESSING THE ETIP/NRC PROJECT

Assessing the effectiveness of the nuclear regulatory experiment requires

comparing the expected project activities and impacts (best summarized in Figure

p. 12) with the project activities and impacts reported by the project participai

and project documentation (discussed in Section IV). Figure 7 summarizes the

events intended as contrasted to the events reported to have occurred.

A. PROJECT ACTIVITIES

As the comparison indicates, three basic changes occurred from the expected

events

• ANSI chose to subcontract the experiments rather than to conduct

the experiments through the hiring of an in-house project manager.

• More variables were tested than were originally proposed.

• The evaluation of project impacts was not conducted. As a result, project impacts were not ascertained, and adequate

descriptions of the experiments were not provided.

As a result of the changes in the project, the data actually collected and

reported varied from what was expected in the initial project plan
25

INTENDED REPORTED

1 . ETIP prepares project plan Plan prepared (1973-74)

2. ETIP obligates funds for NRC use Funds obligated June 1974

3. NRC contracts with ANSI NRC contracts with ANSI

4. ANSI hires project manager

to coordinate experiments

according to project plans

MSI contracts with Battelle

(October, 1974) to coordinate

3 experiments

ANSI contracts with Anderson

to conduct 4th (single-variable)

experiment (November, 1974)

5. Experiments conducted to

test four variables

Experiments conducted; the 1-8

variables tested overlapped with

the original 4.

6. ETIP contracts out evaluation

to assess impact of experiments

No evaluation contract awarded. Battelle submits report

on experimental process,

with subjective assessment

by participants (1975)

.

Anderson submits report

with conclusions but no supportive data (1976) .

7. Draft standards are produced

more quickly

Draft standards are produced

more quickly

8. Standards are produced and

adopted by NRC more quickly

Not addressed by experiment

9. Regulatory lag decreases,

operational safety increases

Not addressed by experiment

10. ETIP conducts additional research

in area

ETIP conducts follow-up review

of experiment (1976-1977)

FIGURE 7: INTENDED VS. REPORTED EVENTS
B. PROJECT IMPACTS

Three criteria were identified in the initial project plan as important

in evaluating the success of the experimental project:

o Time required to produce the standard (as compared to "normal"

time requirements) .

• Quality of the standard produced.

• Cost

.

The Battelle and Anderson reports provided little information on these

criteria. However, the reports were not intended from the outset to be impact

evaluations. What information was provided in those reports has been supplemented to the extent feasible by the follow-up conducted as part of the current

review.

1. TIME TO PRODUCE THE STANDARD

I I a. TIME TO PRODUCE INITIAL STANDARD

None of the reports explicitly indicated the exact length of time required

to produce the initial draft standard. It was possible, however, to reconstruc

| the time required for the Battelle standards.

Two of the Battelle subcommittees produced a draft standard by the end of

the scheduled one-week working session. However, the subcommittee where no

preliminary organizational meeting was held was unable to produce a completed

t draft by the end of the working session and subsequent sessions were required

to get agreement on the first draft. The time required to draft the three

Battelle standards, from the point of selecting the chairmen to completion of

the draft standard, was 2+ months, 8 months, and 9 months respectively for the

three standards.'*' The Battelle report as well as individuals interviewed

as part of this review concluded this time to be faster than the norm for

1. See footnote ** on Figure 8, p. 34.
27

I production of draft standards, and thus concluded that the experiment was

successful in terms of having accelerated the production of the first draft

standard. Those NRC, ANSI, and NBS individuals interviewed indicated that the

I time for production of the initial draft standard in the nuclear area typically

averaged from 1 to 2 years, and sometimes longer. One commented that the ANS

in particular found it difficult to produce draft standards in a shorter time

] period, so that the 4 to 9-month time required by the ANS subcommittees to

j produce the two standards in the experiment was particularly distinctive.

It was difficult to tell what happened in the Anderson experiment. The

report submitted was brief and included no description of the experiment conducted except that secretarial assistance had been provided. There was no

evidence that the project traced the effect of providing secretarial assistance

on the production of individual standards. Nonetheless, the Anderson report

concluded that:

It was the opinion of the officers and subcommittee chairmen

of ASTM C-26 Committee that this experiment was very worthwhile

and that it helped to speed up the nuclear standard activities

of the Committee to produce final national consensus standards. ^

b. TIME TO PRODUCE FINAL STANDARD

None of the reports associated with the experiment traced the time required

for the draft standard to proceed through the remainder of the standards development process. When this was done as part of the current review, only one of the

draft standards was found to have survived the review process. The time from

the initial meeting of the drafting subcommittee to the adoption of the standard

by ANSI was three years and ten months. The standard went to the NRC for review

and adoption on August 8, 1978. The NRC did not adopt the standard, but sent

1. Anderson, H. J., "A Research Report, A Single Variable Experiment on

Accelerating the Writing and Completing of National Consensus Standards."

Final Report on part of ETIP Project #47 submitted to American National Standards

Institute, Inc., New York, New York: February 27, 19 77, p. iii.
28

its comments to the ANSI subcommittee on November 8, 1978. The committee's

response has not yet been transmitted to the NRC. The time required for adoption by the NRC (if it is in fact adopted) remains unknown.

c. WEAKNESS OF THE "TIME" CRITERION

The "length of time" test is a difficult test to apply when assessing

the effectiveness of a standard-drafting experiment. Factors other than

the variables being tested may influence the time required to produce a

standard and get it adopted by ANSI or an agency, such as: — the controversial nature of the activity proposed to be

regulated by the standard, affecting the amount and length

of associated debate and thus the length of time required

to get consensus on the standard, — whether the state of knowledge about the activity is already

far advanced, lessening the time required to produce the

draft standard, — political sensitivity toward industry-produced standards,

which may lead to distrust of such standards and a tendency

of government personnel not to consider them, — the experience and skill of the particular interest group

in managing the voluntary standards process, — the degree of prior experience the committee members designated to develop the standard had had in working together

on previous standards, — the urgency of need for the standard (a factor of particular

import to the experiment since ANSI selected high priority

standards for the project).

In addition, little data appears to be available from elsewhere on the

"standard" time to develop a standard. Since each standard is unique and

typically developed by a separate group of people, and since interest groups

follow somevfaat different procedures, there is no single standard development

process or standard development time to use for comparison.
29

2. STANDARD QUALITY

No precise criteria have been adopted for evaluating the quality of

standards produced through the voluntary standards process—a problem for

anyone interested in evaluating the impact of attempting to accelerate the

standards development process. Possible tests proposed in the evaluation

plan and by the interviewees included: — whether the standard is endorsed by the relevant government

agency, and the number of exceptions made to the standard, — whether agency staff find the standard sufficiently comprehensive and as easy to use as practicable, — whether the standard is actually adopted and used by industry, — whether the standard is effective over time in achieving

its goal, e .g . , preventing accidents.

Tests proposed by those interviewed for evaluating the quality of the

first draft standard included: — number of objections raised to the standard when reviewed

at various stages of the standard approval process, —• the length of time required for standard adoption by the

various organizations (e.g., the interest group, ANSI, the

pertinent agency)

.

None of the reports associated with the experiment attempted to assess

the quality of the standards produced using the above-mentioned criteria, or

any others. The evaluation study which had been projected to assess standard

quality and might have possibly used these criteria was not contracted out.

a . FOLLOW-UP FINDINGS

The follow-up review of the standard's progress conducted for this report

included no systematic attempt to assess the standards in terms of all of the

above criteria. Follow-up was conducted by 1) discussing standard quality with

some of the individual participants, and by 2) tracing what happened to the individual standards.
Opinions of interviewees: One NBS official indicated that those

individuals with whom he had spoken following the experiment who had looked

at the draft standards concluded that they were as good drafts as those which

are produced using the normal drafting process.

One NRC drafting committee participant was of similar opinion. He indicated, however, that some individuals on the drafting subcommittees thought

that accelerating the drafting stage was likely to prevent the subcommittee

members from getting as much consideration and input from their companies,

leading to more objections being raised at subsequent review stages and thus

lengthening the post-draft review stage since less time was taken to get a

consensus at the first-draft stage. The NRC official disagreed with this concern, however. He thought that the subcommittee members got as much input from

their companies as through the normal process, since they maintained constant

telephone contact with their home company experts during the five-day working

session.— Tracing the progress of the standards to adoption: Only one of the

standards (ANS 2.12) made it through the entire standards development process.

The INMM-11 standard was sent back to the drafting committee pending further

study. Reviewers of the standard at the MSI canvassing stage concluded that

sufficient information was not available on the appropriate criteria for

materials control to permit the developing of standards for selecting materials

control managers.

The ANS 18.6 standard was held back by ANSI and never sent out for

review. The particular standard involved environmental impacts, and since

recent standards relating to the environment had met considerable resistance

from environmentalists and failed to make it through the ANSI review process,

the ANSI review was postponed pending reevaluation of all the particular

ANSI committee's standards.
31

The third standard (ANS 2.12) was successful. It was approved as an ANSI

American National Standard in July, 1978, and sent to the NRC for comment on

August 8, 1978. The NRC did not adopt the standard, however, but returned its

comments to the drafting committee on November 8, 1978. No further action had

been reported.

b. ASSESSMENT OF THE "QUALITY" DATA AND CRITERION

It is impossible to draw useful conclusions from the amount of data

collected in the current follow-up. It would require a much more extensive

survey of project participants and additional experts, as well as the development of more precise standards for comparing standard quality to enable a

comparative assessment of standard quality. The development of such criteria

would not be a simple task.

Since only one of the standards made it through the process to the NRC

review stage, and factors other than standard quality were responsible for

halting progress of two out of three standards, it would be impossible to

apply several of the criteria proposed for assessing standard quality (e.g.,

endorsement by the government agency) . Nonetheless, the draft standards

evolved could be submitted for assessment to a group of experts familiar

with standards development. It would also be possible to submit a sample

of draft standards evolved through the regular drafting procedure to a

similar assessment, to evolve comparative data.

Similarly, a survey could be conducted to determine the percentage of

draft standards which normally survive the standards review process, to see

if the one-third success rate experienced in the experiment was typical.

None of these paths appear to be desirable at this point in time.

j
32

An assessment of standard quality would be costly and time-consuming in

light of the difficulty in establishing objective criteria and the time and

effort required to trace the standards from the drafting stage through

ultimate disposition.

3. COST OF ACCELERATING DRAFTING PROCESS

The Battelle report indicated the costs of introducing the innovations

in each of the three subcommittess . (Figure 8 illustrates the variables introduced and the costs per subcommittee.) No comparison figures were either

available or necessary, since in the typical drafting subcommittee situation,

all costs are underwritten by the companies or governmental entities where

the participants work. The cost of providing the secretarial services for

the Anderson experiment was reported to be $5000, or equivalent to the grant

amount. However, it is difficult to tell from either of the reports submitted

exactly how the money was spent or to allocate the costs among the experimental

variables. It is thus impossible to identify the cost of implementing the

individual variables.

No additional cost information was gathered as part of the current review.

If detailed records were kept by the two contractors, however, such information

could be retrieved. It would also be possible to project such costs with little

difficulty for most of the variables, e .g . , cost of clerical support.

4. SUMMARY OF EXPERIMENTAL CONCLUSIONS

Authors of the Battelle report reported the following changes as successful, based on their observations and questionnaire data:
33

• Use of 5-day or longer working session,

• Use of preliminary organizational meetings prior to the working meeting,

• Premeeting preparation of an outline or preliminary draft by the

working group members,

• Provision of an executive secretary,

• Provision of clerical and duplicating services.

Payment of the chairman or of the travel or living expenses of the members was

not deemed particularly useful.

The Anderson report concluded that the provision of secretary services

was also worthwhile and speeded up the overall process of getting standards

generated and accepted.

In follow-up discussions, one ANSI staff member reported that in a

January, 1978 meeting of the planning committee of the ANSI Nuclear Standards

Management Board, attention focused on how DOE funding should be spent to

accelerate development of fuel cycle standards. The American Nuclear Society

(ANS) member in attendance reported that ANS had drawn the following conclusions

based on ANS' experience with the nuclear standards project:

• Three innovations were of particular usefulness: provision of the

executive secretary and of clerical and duplicating support, and

use of a longer working session.

• Although in the normal course of events, money would not be available

to support use of these innovations, these innovations would be implemented in instances where an agency or group needed a standard quickly

and was willing to provide the funds needed to accelerate the standard

development process.

Figure 8 summarizes the experimental variables tested in the Battelle and

Anderson projects and the results reported.
34

BATTELLE EXPERIMENTAL ANDERSON

COMMITTEES COMMITTEE

ANS 2.12 ANS 18.6 INMM-11 N-ll

Independent variables tested

5 -day working meeting * xi

Pre-meeting preparation of

X X

Working outline * Working draft * X

X

X

Secretary provided * X X X X

Clerical/copying support * X X X

Preliminary meetings X X

Paid chairman X X

Paid member expenses X X

Limited number X X X

Dependent variables reported

Time 8 mos

.

2+ mos.** 9 mos

.

reported

Qual i ty _2

shorter

than usual

Cost $3825 $13,925 $14,770 $5000

Participant opinions on: — usefulness of

innovations X X X X — overall performance X X X X

* Those marked with an asterisk were reported effective in contributing

to the standards drafting process.

** A partial draft was completed within 2+ months, but some sections

remained to be drafted and had not been completed within the following two

months. No information is available on the first draft completion date.

1. The use of an "X" indicates that the variable was tested in the

particular experiment conducted by that committee and/or findings reported.

2. The placement of an indicates that no information was reported.

FIGURE 8: EXPERIMENTS CONDUCTED AND RESULTS REPORTED
VI. MAJOR FINDINGS AND LESSONS LEARNED

A number of things were learned from the experiment. The individuals

reporting the experimental results reported that at least some of the experimental interventions did accelerate the initial drafting process, particularly

the five-day work week, the provision of secretarial assistance, and the

clerical/duplicating support. Subsequently, an ANSI official interviewed

indicated that several interest group members had convened and decided that

in instances where an accelerated process was required and money was available

to underwrite the costs, they had decided to implement the three interventions

believed to be effective. On the other hand, in the normal course of events,

the costs of the administrative supports were otherwise deemed too expensive

and the money unavailable to sustain them.

These reported learnings are not, however, supported by the evidence

generated in the experiments. Most of the lessons learned from the experiment

are related to the experimental process rather than to the outcomes of the

experiments conducted.

A. THE KNOWNS

An assessment of the experiments conducted indicates the following

weaknesses inherent in the design and conduct of the experiments.
36

1.

The requirement that ETIP fund the project with ANSI by transferring the funds through the NRC diminished the ETIP control over the

design, development and implementation of the experiment. Problems

also occurred in the reimbursement procedures as a result of the NRC's

middleman status.^

2. There was substantial turnover of major personnel figures who were interested in the experiment as originally designed. This may have

caused discontinuity in the project oversight and conceptualization,

and in the project's political status and backing (e.g., the director of the AEC resigned; the original ETIP project monitor left)^

This probably contributed to some of the changes which occurred in the

project design and implementation, and also led to the problems in

implementing the evaluation.

3. Planning and funding delays led to a permanent postponement of the

evaluation planned to assess project outcomes and impacts.

4. The information provided by the existing contractors on key project

variables and outcomes was incomplete. The data which were provided

were either over-analyzed statistically in view of the same size

and lack of experimental controls, or totally unavailable on the

other hand.

Even if the anticipated evaluation had been conducted, the results

probably would have been disappointing. This is because:

1. The initial evaluation planning appeared to be short-sighted in

terms of the scope of study necessary to answer questions about

project impacts. For example, tracing whether the process accelerated NRC adoption required an evaluation spanning numerous years.

Only one of the three standards had reached the NRC as of August,

1978. (See Appendix C for a discussion of what happened to the

individual standards developed.) Major project assumptions could

not be adequately tested within a one or even two-year evaluation

period. In addition, there was insufficient up-front identification

of the measures to be collected.

2. The initial design did not indicate how the proposed experiment

fit into major research needs, or into the overall standards

development process. In addition, the design did not include

1. Garrity, pp. 14-16, 28-29.

2. Ibid . , pp. 27-28.

3. Ibid ., pp. 27, 31-38.
37

tasks to collect background data on the "normal" development process.

Thus, there was no initial plan for collecting the type of objective

comparative data needed to enable evaluators to assess whether the

experiment accelerated the development of nuclear standards.

3. There was no suggestion of the need to collect background data

to examine the accuracy of the experimental assumptions. Those who

proposed and undertook the experiment relied on the assumptions outlined in Figure 4. That is, the project and evaluation designers

presumed that by accelerating the establishment of standards, the

NRC review of proposed nuclear plants would be accelerated, thereby

accelerating the nuclear plant licensing and approval process. No

background information was proposed to be (or was in fact) collected

at the outset about the NRC licensing and plant approval process

to see what role standards actually played—to check whether the

assumptions about the role of standards in accelerating plant liThus , while it is entirely possible that the chain of assumpltions

and expectations outlined in Figure 4 is correct, it is not apparent

from the project documentation or interviews with interested parties

that these assumptions were reviewed or questioned at the outset of

the experiment. If they had been, the questions about the project's

relevancy could have been better addressed both during the experiment

and afterward.

B. THE UNKNOWNS

Because of the experimental design process, there remain many unknowns

and uncertainties about the impacts of the experimental interventions. Little

or no information was collected on such factors as:

• Average time for standard drafting and processing.

• The role of existing standards in the plant licensing process.

• Quality of the standards produced.

• Cost of introducing the individual innovations, compared with

normal production costs.

o The details of the experiments conducted.

It would be difficult and costly to obtain information on these factors at

this point in the project.
38

C. USEFULNESS OF FURTHER STUDY

Further study on the impacts of the experiments does not seem to be

justified. The finding that providing secretarial and clerical support and

longer working sessions accelerate the production of draft standards is not

surprising, and there is sufficient information already available to justify

the conclusion that the standards were drafted more quickly than was the

usual case .

The issue which remains somewhat uncertain is the effect of accelerating

the standard drafting process on the consensus process that follows in the

standards development process. This question cannot be answered solely by

further review of the experiment already completed, because the sample was

too small. Additional research would be necessary, replicating the drafting

stage experiment with a number of standards and tracing the standards through

the remainder of the development process. Since only one of the standards

in the current process proceeded beyond the initial stages of the review

process, further follow-up of the initial standards alone would likely glean

little additional information. Also, comparative information about the

"normal" drafting process is needed to render the experimental data meaningful .

ETIP thus has several options relative to the conduct of further study

on the voluntary standards process:

Option 1 : Terminate the research at the conclusion of the present

review.
39

Option 2 ; Fund a limited research effort to gather additional background

information about the experiments conducted and their impacts (e.g., the costs

of the individual innovations, the quality of the standards produced) as well

as about the ''normal" development process and time requirements in the nuclear

standards area, with which to compare the experimental results.

Option 3 : Fund a substantial research effort to identify the role of

standards, if any, in accelerating nuclear plant licensing, using a research

design and a contracting process which will address the difficulties experienced in the initial experiment and detailed in this and the earlier review.

Assuming that standards are found to be a substantial factor in accelerating

nuclear plant licensing, then ETIP could consider the usefulness of investigating how and whether the voluntary standards process can be successfully

accelerated to develop those standards.

Option 4 : Fund an evaluability assessment of the usefulness of conducting

further study of the voluntary standards process and the impact of standards

of nuclear plant licensing. This would involve collecting sufficient background information about the proposed research that the costs and likely

results of the various research options could be assessed prior to commiting

funds to any particular research design.

ASSESSMENT

Options 2 and 3 could be pursued if the resources are available.

Option 2 would provide ETIP with additional impact information and allow the

experimental results to be compared with the data on the "normal" standards

development process. This would provide a better context for the experiments

conducted. However, it is somewhat questionable whether the information

derived would be worth the costs.
40

Option 3 would permit ETIP to address the larger question of the relevancy

of attempts to accelerate the development of standards. In light of the recent

3-Mile Run nuclear incident, the question of the desirability of accelerating

nuclear plant licensing has become particularly significant, if the acceleration implies safety trade-offs. An investigation of the role of standards

in accelerating nuclear plant licensing while concurrently maintaining high

safety standards should therefore be seriously considered. ETIP's selection

of Option 3 would require a substantial research commitment over a number of

years, however. This would probably extend well beyond the scope of the

original project.

If further research in the standards area is contemplated, however.

Option 4 is the most highly recommended. The conduct of an evaluability

assessment would provide ETIP with guidance as to the areas where further

research is desirable and likely to be fruitful, before significant resources

are committed. Option 1 is the best selection at this particular stage of the

ETIP program, however, if sufficient resources cannot be devoted to pursue

any of the other options.

D. SUMMARY

Despite the design and administration difficulties with the experiment

which was actually conducted, some useful information was gleaned. The drafting

process is believed to have been accelerated. However, the goals of the experiment were not limited to the standards drafting stage.

To have learned whether the acceleration of the drafting stage accelerated

the entire development process and NRC endorsement would have required beginning

with many more standards, tracing them for several years to see what happened

to them, and comparing their progress with the norm. To have learned whether
41

the availability of standards accelerated nuclear plant licensing would have

required yet additional research. Considerably greater resources would have

been required to conduct that additional research.

The current review was useful, however, in highlighting factors which

need to be addressed by ETIP in the design of future regulatory experiments.

1. The process of funding projects through an outside agency can

leave ETIP with less control over the design, development and

implementation of its experimental projects. ETIP should

explore ways to ensure that the integrity of ETIP experimental

designs can be maintained. Policies and mechanisms for dealing

with derailed projects should likewise be developed.

2. ETIP should assess proposed experiments at the project design

stage to determine the probable validity of the underlying

experimental assumptions and the feasibility of evaluating

project impacts. Ibis requires collecting background information about the regulatory process which is the subject of the

experiment, the likelihood that the proposed changes can be

implemented, and the feasibility of collecting the data necessary to evaluate the project's ultimate impacts. Had such a preassessment been conducted with the current project, many of

the problems associated with the project design and data collection might have been foreseen and dealt with. Much of the

misunderstanding and controversy over the value of the experiment

undertaken might have thus been avoided.
NRC D.//2

11/1/78

APPENDIX A

INFORMATION SOURCES
APPENDIX A

INFORMATION SOURCES

PERSONS INTERVIEWED

National Bureau of Standards

Dr. Jim Liess, Director, Center for Radiation Research,

National Measurement Laboratory and National Technical Advisory Group

(also member. Nuclear Standards Management Board and National

Technical Advisory Group)

Experimental Technology Incentives Program

Victor Berlin, Chief of Experimental Methods

Dan Fulmer, Chief of Regulatory Area

Stephen Garrity, Experimental Methods

American National Standards Institute

Mary Crehan Vaca, Secretary, Nuclear Standards Management Board

Mr. Richard Simpson, Washington representative, ANSI (ex-Deputy Assistant

Secretary for Standards, NBS)

Nuclear Regulatory Commission

Mr. Leon Beratan, Chief Site Safety Standards Bureau,

Office of Standards Developmnet

Dr. Carl Johnson, Director of Policy, Office of Standards Development

Mr. George Rivenbark, Branch Chief, Program Support Branch,

Office of Standards Development

DOCUMENTATION

Anderson, H.J., "A Research Report, A Single Variable Experiment on Accelerating the Writing and Completing of National Concensus

Standards." Final report on part of ETIP Project #47 submitted

to the American National Standards Institute, Inc., New York,

New York, February 27, 1976.

Garrity, Stephen, "A Review of the Nuclear Regulatory Experiment,"

unpublished draft, Washington, D.C.: Experimental Technology

Incentives Program, National Bureau of Standards, January 17, 1977.

"ANSI Progress Report, 1977," New York: American National Standards

Institute, March 1977.

ETIP project documentation.
2

Experimental Technology Incentives Program (ETIP) . "Accelerating the

Establishment of Standards and Their Adoption As Regulatory Guides

by the Atomic Energy Commission." Duplicated background paper and

project plan for ETIP Project #4 7, National Bureau of Standards,

Washington, D.C., 1974.

Experimental Technology Incentives Program (ETIP) . "Evaluation of the

ETIP/NRC Standards Development Project." Project plan for ETIP

Project #95, National Bureau of Standards, Washington, D.C., 1975.

Garrity, Stephen, "A Review of the Nuclear Regulatory Experiment,"

unpublished draft, Washington, D.C.: Experimental Technology

Incentives Program, National Bureau of Standards, January 17, 1977.

Rogers, L.R. , "The Role of Regulatory Standards in Nuclear Power

Reactor Safety." Testimony before the Joint Committee on Atomic Energy,

U.S. Congress, Washington, D.C., September 26, 1975.

Wittenbrock, N.G. , "An Experiment on Accelerating the Writing of Nuclear

Standards Conducted for the American National Standards Institute."

Final report on ETIP Project #47 submitted to the American Standards

Institute, Inc., New York, New York, September 1975.
APPENDIX B

ACCELERATING THE ESTABLISHMENT OF STANDARDS

AND THEIR ADOPTION AS REGULATORY GUIDES

BY THE ATOMIC ENERGY COMMISSION

APPENDIX B

i

t * PROJECT 47 PLAN

ACCELERATING THE ESTABLISHMENT OF STANDARDS

AND THEIR ADOPTION AS REGULATORY GUIDES

BY THE ATOMIC ENERGY COMMISSION

I BACKGROUND

This is the first of a series of projects in which ETIP

plans to examine the relationship between voluntary, or industry based standards and the regulatory process. This relationship is varied and complex: It ranges from the adoption

by an agency of voluntary standards so that they then have the

force of law, through the reference by an agency to a standard

as an approved method of doing something, on to an independence

from the regulatory process except as a standard may transgress national policy such as the antitrust laws. Each of

the varying relationships has its own needs and concerns. It

is therefore essential that ETIP experiment with these important relationships between voluntary standards and the regulatory processes in an attempt to optimize them.

This project addresses the need of the Atomic Energy

Commission for the efficient and expeditious development of

standards which may be used in the licensing process for nuclear reactors. In order for the country to timely secure

the benefits of a developing technology and to stimulate investment in it by reducing the extraordinary cost of delay,

the punctual development of hundreds of these standards is

essential

.
2

A. The Need for Standards . The advent of the recent

energy crisis has underscored the need to begin using on a widescale something other than fossil fuels for the generation of electricity. While there is considerable research

underway in an effort to discover commercially viable alternative fuels, currently by far the most promising is an expanded use of nuclear reactors.

But before a nuclear power plant may be used for generating electricity (or any other use), its design, construction,

and operation must meet the stringent standards which have been

established by the Atomic Energy Commission to protect the

public health and welfare. Because the reactors are so complex and because the level of review must be so detailed, this

licensing process is extremely time consuming. Currently it

takes approximately ten years for a plant to begin operation

after the decision was made to build it; in large part, this

is due to the licensing process. This long lead time means

that the country must meanwhile generate its power from fossil fuels and hence has to wait a long time to secure the

benefits of a needed technology. Moreover, and perhaps worse,

the long and expensive licensing process means that a utility

which might be interested in building a nuclear plant must

wait a long time before generating income on its investment,

which in turn drastically raises the effective cost of the
3

plant and forces a utility to tie up capital for a long period.

As a result, the President has declared that it is a national

objective to reduce this process by several years.

One of the main reasons for the extraordinary length of

time for the ultimate licensing is that the Commission must

satisfy itself that all aspects of the plant -- from its location through its reactor design and all support equipment --

are safe and have a high degree of freedom from faults and

errors; and, of course, the total system must have similar

properties. If done on a plant-by-plant basis the individualized assessment of each part, sub-system and total system is

necessarily time consuming. And it is also subject to human

error since at each stage a new human judgment as to safety

and reliability must be made.

In order to diminish the amount of time needed to review

( a proposed plant, as well as to diminish the chance of error,

the Commission has encouraged, supported, and given priority

consideration to activities leading to greater standardization

of nuclear power plants in terms of their design, fabrication,

construction, testing, and operation. Thus if a plant, or

part of a plant, meets a standard, the Commission is able to

approve that part without a de novo and detailed examination

into all its facets -- from theoretical physics to industrial

quality control.

1
4

Another benefit of standardization is that as the available technology advances the standards can be amended to incorporate it. .Thus they not only diminish the amount of time

consumed by the regulatory process and its chance for error,

they also serve to advance the state of technology actually

in use.

B. Regulations . The Commission has established a series

of regulations which must be met before a plant can be licensed

Some of these regulations are general in that' they state

basic policy goals of the Commission or they are performance

criteria which must be met. Others are quite specific and

detail precisely what must be done.

C. Regulatory Guides . While the regulations must be

complied with, often times there is still considerable leeway

as to how that can be accomplished. For example, if a regulation requires that a part meet a certain performance level,

there may be many ways of meeting that level and many ways

of testing the part to demonstrate that it does. Thus, without more, many of the initial problems of an uncodified review process would remain: there would still be individualized

review with its consumption of time and chance for error.

In order to provide a method of standardization without

restricting the opportunity for developing new technologies

which comply with the performance requirements, the staff of
5

the Commission has established a series of "regulatory guides".

These guides inform applicants and licensees of ways acceptable to the Commission's regulatory staff of complying with

the mandatory regulations. They implement the regulations and

are not substitutes for them. They are not mandatory, and an

applicant may meet the regulatory requirements in other ways

if it can demonstrate the appropriate compliance. Thus the

guides are approved methods of compliance.

D. How the Regulatory Guides are Established . The adoption of a regulatory guide begins with the development of a standard by either the AEC staff or by the industry and other

experts -- suppliers, users, theoreticians, government personnel and others -- through the voluntary standards program. As

it does in other areas, the American National Standards Institute (ANSI) acts as a clearinghouse which co-ordinates the

development of standards within the private sector. The Nuclear Technical Advisory Board (NTAB) of ANSI is responsible

for co-ordinating the development of nuclear standards, with

the actual development of standards done by committees established by professional societies and standards groups.

In order to assure that the nuclear standards developed

in the ANSI process will be suitable for use by the Commission

in its regulatory program, the Directorate of Regulatory

Standards co-ordinates standards, priorities, schedules and
6

standards writing activities with NTAB. The Director of Regulatory Standards is a member of NTAB and of its Executive Committee. More than 140 staff members of the Commission participate directly in standards writing groups of ANSI, and the

Commission contributes more than $80,000 annually, in funds

and services, to the ANSI efforts. The Director of Regulation

has made a commitment that an ANSI -approved nuclear standard

will be reviewed by his staff and action will be initiated to

implement the standard within 90 days or ANSI will be informed

as to why the particular standard cannot be adopted as either

a regulation or regulatory guide.

In 1971 the Commission requested ANSI to greatly accelerate

the development of standards pertaining to nuclear plants.

ANSI responded by increasing its staff and expanding the scope

of its activities to include an intensive effort in establishing priorities for needed standards, scheduling the development

of those standards, and recruiting standards writers. Recently over 100 new working committees have been formed, management positions have been filled, and a large number of high

priority projects have been scheduled.

The key elements in producing quality standards which

may be used by the Commission are the efficiency and effectiveness of the standards writing groups. At present, the members

of the individual standards wtiting groups are employees of

utilities, architect -eng ineers , equipment manufacturers, and
7

of the Government, who have expertise and experience in the

field in which the group is working. Their participation is

purely voluntary and they are not separately paid for their

s tandards -development work. For the most part the time spent

on developing standards is only that which can be spared from

their full-time duties with their employers or in their afterduty hours. Because of the need for highly qualified and

experienced persons on working groups, the members are usually

key staff members of their employers and consequently the

demands on their time by their employers is heavy. Thus it

may take a considerable number of weeks, months, or even years

in order to secure enough of the time of these busy men to

fully develop a standard.

The chairman of a committee has similar restrictions on

his time, and yet his duties include the generation of background information which underlies a standard under development and the drafting and circulation of proposed standards.

Since he too has so many demands on his time, this process may

be considerably slower than it would be if the committee'

chairman could give the process more of his attention.

The Commission is vitally concerned with developing

these standards as expeditiously as practicable so that the

country can timely secure the full benefits of the technology

of nuclear power. Hence it is interested in discovering ways
8

to accelerate the development of responsible standards for

use in the regulatory process.

D. What is Proposed Here . In this project at least four

changes will be made in the normal operation of several standard

writing groups to determine if any or all of them have a significant effect on accelerating the establishment of responsible

standards which are approved by ANSI and adopted by the Commission. The four changes will be:

° hiring a full-time working-group chairman

° providing consultant services for collection and correlation of basic data

that is required by the group developing

the standard

0 clerical and technical editing assistance

0 bringing the group together for up to two

weeks to concentrate fully on the development of the standard.

II

PURPOSE

The purpose of this project is to determine whether any

or all of the four changes in the method of developing nuclear

standards is a desirable method for accelerating the establishment of responsible standards which are approved by ANSI and

adopted by the Atomic Energy Commission.
9

III

ANTICIPATED RESULTS

It is anticipated that each of the four changes to be

made in the usual method of developing nuclear standards

will accelerate the development of those standards by enabling the committee to more directly focus its attention on

reaching a consensus. In particular, the four changes are

expected to have the following results:

A. Full-time Chairman . Since it is the chairman who is

responsible for co-ordinating the committee -- from the development of information needed to reach consensus to the preparation and circulation of draft standards -- the demands on his

time for administrative work are considerable. Yet it is he

who must pass initial judgment on proposed standards and change

relevant comment on a proposed standard into a suitable amendment to it; if information is needed he must know where to

get it and, since there is rarely a budget for such things,

he must usually have to know where he can generate it for

free. Thus the chairman must necessarily be an expert in

his field and must devote considerable time to the perfection

of the standard on which his committee is working. By hiring

a full-time chairman* these duties could be performed in a

* All the duties of a committee chairman would probably only

occupy half of the time of a suitable expert; thus "fulltime chairman" means that the chairman will be paid to devote a specified portion of his professional time to the

work of standards development.
10

much shorter period and with considerably more concentration,

which in turn would mean the committee itself would be ready

sooner to progress towards consensus and an acceptable

standard

B. Consultant Services . Many of the high-priority

standards require the collection and correlation of considerable amounts of information before the committee can determine

whether there is enough information on which to reach a consensus. Usually this information is gathered- and correlated

by the committee members themselves, and consequently it takes

a considerable amount of time. If this information can be

generated for the committee by a consultant group, then the

time involved in preparing the background information can be

significantly shortened. This too will reduce the amount of

time required for reaching consensus by shortening the amount

of time consumed in satisfying their need for information.

C. Clerical and Editing Assistance . As previously described, there is a considerable amount of administrative work

in the development of a standard. Clerical and editing assistance will be provided the chairman in order to speed up

the process of preparing initial and revised drafts of standards and the support documents which are distributed to the

committee. This should facilitate the standard -development

process by reducing the amount of administrative work per-
11

formed by the chairman and by providing professional help

experienced in the process of codification.

D. Bringing the Committee Together . At present, most

of the functioning of the committees is by mail or in isolated moments when its members are together for professional

meetings. In this part of the project, a committee will be

brought together for as much as two weeks to wota exclusively

on the development of their standard. The free exchange of

ideas made available by face-to-face discussion, and without

intervening duties, should mean that each member of the

committee can give his full concentration to the development

of the standard and any questions or disagreements which

arise can be focused upon directly by the entire committee.

This should serve to resolve many of the time consuming

questions and disputes which naturally arise during the process

Added together, it is anticipated that these four changes

will mean that responsible standards in very difficult areas

can be developed in a significantly shorter period than would

otherwise be the case.

IV

APPROACH

The Atomic Energy Commission will be the lead agency in

this project, but because of its regulatory obligations it

will not perform its duties any differently than it currently

does with respect to the development of standards in the
12

private sector. The overall management of the project will

be by the Nuclear Technical Advisory Board of the American

National Standards Institute, which occupies a .special, quasiofficial status with respect to the development of standards

-which are ultimately used by the Commission as part of its

regulatory structure.

The Nuclear Technical Advisory Board will select three

(or possibly four) standards which are of extremely high

priority and which are difficult to develop because of a need

for a considerable amount of information. The areas selected

will also have standards developing committees already in

existence and they would be able to be carefully monitored

so that the results of this project can be reliably measured.

The four areas are:

0 site seismic design considerations

° effects of combined natural phenomena -

design considerations (seismic, flooding,

tornado, etc.)

° fuel quality assurance

0 valve and pump qualification standards.

NTAB will hire, on a part-time basis, a program manager

to be responsible to it for the management of this project.

The program manager will be the chief administrative officer

of the project, and will be responsible for hiring the respective chairmen, the clerical and editing assistance, and the
13

resoective consultants; he will be responsible for scheduling and

ensuring that the schedules are met; he will also keep account

of how the three or four committees use their resources and

the effect of this project on the development of the respective

standards

.

No specific program for the use of the four changes

itemized has been determined. It is felt that rather than

establishing a set matrix for their use before the committees

begin actual operation, it would be best to allow the committees to draw upon these resources to enhance their ability

to develop their respective standards. It will be the duty

of the program manager to keep detailed accounts of the use of

these resources and to ensure that they are used by the various

committees in such a manner that the effect of each can be

appropriately determined; that is, the three or four committees will not duplicate each other in their use of the four

changes. For example, one (or more) of the committees might

not have a full-time chairman but would extensively use consultants and editors; another might have a full-time chairman

but use consultants sparingly because he would develop the

information himself. If a committee feels some other related

resource would greatly aid it in developing its standard, then

the. project manager may approve the use of the ^unds of this

project for such other purpose upon the written consent of

ETIP and the Director of Regulatory Standards of the AEC

.
14

It is anticipated that the project will run for approximately one year, although the work of some of the committees

may extend somewhat beyond then. The approximate cost is

anticipated to be broken down as follows:

$20,000 - Program manager (part - t ime)

55.000 - 3 chairmen (each with technical expertise

in his field; each hired on 1/4 - 1/2 time

.basis) plus clerical and editing assistance

25.000 - Consultant services; plus 1 two-week meeting

$ 100,000

It is anticipated that the chairmen will be individuals

who have the relevant technical expertise and who are currently employed on a full-time basis by a firm interested in the

prompt licensing of nuclear power plants; the hope is that

an employer will permit a potential chairman to be hired on

a part-time basis for this standard's development work because

of its importance. Another source for such chairmen are individuals who have recently retired from such an active occupation .

V

EVALUATION

The final evaluation of this project is whether the approaches to standards development which are undertaken are a desirable method of accelerating the establishment of responsible standards which a re adopted by the AEC and which hasten
15

the licensing of power -generating reactors. In large measure,

this evaluation will be based on the subjective opinion of

those who are familiar with the standards development process

by comparing what occurred in this project with what they

would have anticipated would have been the case under normal

circumstances. Thus the evaluation will be based on the following questions:

0 Have the schedules for the standard development

(both the overall schedule already .determined

,

and the particular schedule determined by the

project manager for this project) been met?

0 Are the standards which result from the project

of high-quality, in that they are readily acceptable by the Commission and in the opinion

of the Commission’s staff they are comprehensive

and as easy to use as is practicable.

0 Did the project generate a more effective and

efficient use of the working group members?

° Was the time from initiation to adoption by

the Commission reduced sufficiently to offset

the cost of the project?

If it is determined that aspects of this project have

significantly reduced the amount of time necessary for the

development and adoption of standards, an inquiry will be

made as to how much the time needed to secure ultimatp licensing
16

can be reduced by the adoption of a comprehensive code of

regulations and regulatory guides. If that period can be

significantly reduced by such a code, then an inquiry will be

made as to why the private sector, which would profit financial

ly from the reduction, has not undertaken similar efforts

in the past. An inquiry will also examine what may be done

to improve the existing system and to better use the resources

of the private sector.

Another inquiry will be made as to the public benefits,

such as improved public safety or reduced manpower needs,

which stem from the adoption of such codes, and whether the

Government should routinely sponsor similar efforts.

The evaluation will be conducted by a contractor hired

for the purpose by ETIP.
APPENDIX C

DETAILED CHRONOLOGY

1

APPENDIX C

DETAILED CHRONOLOGY

Early 1973

1. ETIP issues a request for "Qualification and Interest" statement.

ETIP placed a notice in the Commerce Business Daily which broadly described

ETIP's mission and requested Q and I statements as well as suggestions for

work in the regulatory area. This request was to be the first of a two-stage

procurement

.

Early 1973

2. ANSI response. The American National Standards Institute (ANSI)

responded to the request. They suggested the idea that ETIP reimburse their

committee members for travel expenses. According to Garrity, "the idea

didn't impress people" at the time.

Early 1973

3. Second stage of procurement dropped. According to Garrity, the

first phase served to bring ETIP in contact with some qualified firms. The

second stage "died a natural death."

1973

4. Contact with Rogers. The discussions with ANSI led to contact

with Lester Rogers, the Director of Regulatory Standards for AEC. He was

interested in improving the voluntary standards at ANSI. From this time on,

ETIP had continuing contact with Johnson who worked for Rodgers.

March 21, 1974

5. ETIP completes a project plan for the NRC experiment. The plan

indicated that the NRC project was the first in a series of projects examining

the relationship between voluntary standards and the regulatory process. The

plan lacked a broad perspective and did not relate the "experiment" to overall

standards development or to industry. The following four interventions were

proposed: paying a full-time committee chairman, using consulting services

for information gathering, clerical editing assistance, and more lengthy

meetings

.

According to the plan, the Nuclear Technical Advisory Board of ANSI

would manage the project, select standards and hire a program manager. * AEC would retain overall approval authority.

Under the plan, the Nuclear Technical Advisory Board (in ANSI)

would manage the project. They were supposed to select standards and hire

a program manager.

6. No Interagency Agreement. There was no interagency agreement because

there was no actual transfer of funds. By congressional mandate, AEC could

only receive funds appropriated by congress. As a result, ETIP held an account dedicated to AEC. ANSI would bill AEC and AEC would then bill the

NBS account.

April 12, 1974

7. ETIP sends project plan to AEC.
2

May 23, 1974

8.

The assistant secretary for science and technology approves

the plan.

June 12, 1974

9.

ETIP obligates $100,000 for AEC/ANSI use. According to the case study, ETIP lost effective control of the project at this time.

June 1974

10. ANSI activity not in conformance with several plan provisions . Instead of hiring a program manager to coordinate the intervention and

conduct the experiment, ANSI decided to contract out the experiment, con- trary to the plan. It is not known, who at ANSI/NRC made these decisions.

According to Garrity's case study, "ANSI . . . reworded the language of

the plan specifying the interventions to be used. These interventions

became illustrative, leaving ANSI the option to suggest other interventions it felt important." (p. 16)

June 1974

11. ANSI issues a request for proposal for the experiment.

August 28, 1974

12. Battelle-Northwest submits a proposal to ANSI. The proposal

said Battelle would look at different variables in three committees.

September 1974

13. Two committees meet . Two of the four committees to be involved

in the experiment hold their first meeting.

October 15, 1974

14. Contract is signed with Battelle. The contract was funded for

$80,000.

October 29, 1974

15. Harlan Anderson submits a proposal. Anderson, from WestinghouseHanford Corporation, submitted a proposal to study the effects of one variable in a single committee.

November 7, 1974

16. ANSI informs Anderson of proposal acceptance. The contract was funded at $5,000.

December 31, 1974

17 . Rogers, Director of AEC^s Office of Regulatory Standards, retires.

January 19, 1975

18. The AEC becomes the Nuclear Regulatory Commission by executive order.
3

March 25, 1975

19 . Development of a project plan for the evaluation of the experiment.

According to Garrity's case study, "The plan considered backgrounding of

standards development process, a detailed review of the experiment, the postexperiment history of the drafts, the project costs and what recommendations

could be made." One problem with the plan was the requirement that substantial process data be collected, even though the experiment was half over.

April 18, 1975

20. Plan approval. The assistant secretary for science and technology,

Department of Commerce, approved the project plan the evaluation of the

experiment

.

June 16, 1975

21. ETIP funds for the evaluation are obligated to NRC

.

The amount

set aside for the evaluation was $15,000.

September, 1975

22. Berlin joins ETIP . Early in his tenure he became- involved in the

project, arguing for a wider focus on ANSI's standards development process.

September 1975

23. Battelle submits its final report dealing with the three-committee

experiment

.

September 1975

24. Harter leaves ETIP .

December 2, 1975

25. NRC asks ANSI to prepare a proposal for the evaluation.

January 1976

26. ANSI becomes dissatisfied with reimbursement procedure. ANSI

billed NRC which in turn billed NBS. However, NRC held the bills until

a large enough amount was accumulated. According to the case study,

ANSI needed faster processing of bills and also wanted reimbursement of

precontract costs.

Late January 1976

27 . ANSI sends out requests for proposal for conduct of the evaluation.

ETIP was unhappy with the RFP.

January 30, 1976

28. Three proposals submitted to ANSI. The bidders were Stafco,

Rockwell International, and Nuclear Services Corporation. All three proposals

were unacceptable to ETIP.
4

February 11, 1976

29. ANSI sent its own proposal to NRC to do the evaluation. (Thus, ANSI

was in the unusual position of bidding to do a job and issuing RFPs to firms

to do the same job.)

February 27, 1976

30. Anderson submits his final report to ANSI.

Spring 1976

31. NRC attempted to put through sole-source contracting with ANSI.

The Garrity case study says, the attempt "was running into delays because

NRC was uncertain that ANSI was uniquely qualified to conduct the evaluation."

(43)

June 14, 1976

32. ANSI sent a second, revised proposal for evaluation to NRC.

However, NRC failed to act on the second proposal either. A year passed

since ETIP had obligated the evaluation contract funds.

July 21 , 1976

33. NRC asks ETIP to take control of the evaluation. According to

Garrity, Rivenbark and Johnson from NRC's Office of Standards Development,

were simply not interested in the evaluation, partly because NRC was not

directly involved in the experiment itself. Also, Victor Berlin wanted to

expand the scope of the evaluation to include NRC processes. According to

Garrity, "This may have turned them off." In his case study, Garrity

describes NRC as reluctant to have any one study their processes. Finally,

ANSI was annoyed at the rate of reimbursement and NRC was reluctant to

continue its accounting function.

November 29, 1976

34. ETIP deobligates funds for the evaluation of the experiment.
APPENDIX D

A REVIEW OF THE NUCLEAR REGULATORY EXPERIMENT

APPENDIX D

A REVIEW OF THE NUCLEAR REGULATORY EXPERIMENT

Stephen Garrity

Experimental Methods

ETIP

January 17, 1977

TABLE OF CONTENTS

Executive Summary i

1 . 0 Introduction 1

2.0 Background 3

2.1 The ETIP Program 3

2.2 The Nuclear Regulatory Commission 3

2.3 Development of the Experiment 6

3.0 The NRC Experiment 9

3.1 Initiation of the Experiment 9

3.1.1 Planning 9

3.1.2 Contracting 15

3.2 Experimental Design 16

3.2.1 The Approach Used and Variables 16

3.2.2 Analysis of the Design 22

3.3 Implementation of the Experiment 26

3.4 Analysis of Outcomes 29

3.4.1 The Battelle Report ' 29

3.4.2 The Anderson Report 39

3.5 Evaluation of the Experiment • 40

3.5.1 Evaluation Project Plan 41

3.5.2 Contracting the Evaluation 42

4.0 . Conclusions and Recommendations 46

4.1 The NRC Experiments . 46

4.2 ETIP Experiments 47

Reference Notes

Reference List

Appendix A - NRC Organizational Chart

Appendix B - Anderson Experimental Design Papers

Appendix C - Chronology of events

Appendix D - People Involved in the Project

1.0 Introduction

The Experimental Technology Incentives Program (ETIP)

conducted an experiment with the Nuclear Regulatory Commission

(NRC) from 1974 to 1976.* The experiment consisted of two

phases: one which involved changing experimentally some

parts of the process used to develop nuclear regulatory standards,

and another which was supposed to evaluate the effects of these

changes. The purpose of this paper is to review the NRC experiement and discuss the implications for the conduct of future

experiments

.

Approach to the Review

The approach used in developing this paper was to review

activities and documents generated over the life of project

by the groups involved. The majority of the documents reviewed

are included as references in this paper. The major events

of the project are listed chronologically in the appendix.

When the majority of the project literature and information

was examined, it was decided to divide the paper into several

*It should be noted that during the time period of this experiment

the Atomic Energy Commission (AEC) was abolished and the Nuclear

Regulatory Commission was established. This paper will include

both names, even though for purposes of the experiment the

two organizations were essentially the same.
2

parts which could be reviewed and analysed independently of

other parts. The resulting sections represent various time

segments of the project, with some sections having more structured

examinations than others. There are sections on ETIP and NRC

background, initiation, design, implementation, analysis, and

evaluation. A final section presents conclusions and recommendations for the NRC experiment and ETIP experiments in general.
3

2.0 Background

2.1 The ETIP Program

The Experimental Technology Incentives Program (ETIP) seeks

to affect public policy and process to facilitate technological

change in the private sector. ETIP's goal is the development

of a set of policy guidelines on technological change and the

body of knowledge necessary for their effective use. This

goal is being pursued through the examination and experimentation with government policies or practices in regulatory,

economic assistance, procurement, and research and development

areas. ETIP works with government agencies and provides assistance and funding for the conduct of studies and experiments

with policies.

» The regulatory policy area at ETIP is interested in examing the relationship between government regulation and the

rate or direction of technological change in the private

sector. One focus is on the regulatory development process

which often involves excessive time delays. This lag problem

is exemplified in the nuclear regulatory arena where standards

are used by the Nuclear Regulatory Commission in its liscensing

process for power plants.
2.2 The Nuclear Regulatory Commission

The Nuclear Regulatory Commission (NRC) is an independent

government agency concerned with the civilian uses of nuclear

energy. Its purpose is to "assure that the civilian uses

of nuclear materials and facilities are conducted in a manner

consistent with the public health and safety, environmental

quality, national security, and the antitrust laws" (Office

of the Federal Register, 1976, p.588).

The Commission fulfills its purpose through the licensing

and regulation of nuclear materials and facilities. The major

program components of the commission are the Offices of Nuclear

Reactor Regulation, Nuclear Material Safety and Safeguards,

Nuclear Regulatory Research, Standards Development, and Inspec-

. I tion and Enforcement. Some examples of commission activities

» are the (Office of Federal Register, 1976, p. 588):

1. Licensing of nuclear power plant construction and

operation

2. Licensing the use, processing, and transport o£

nuclear materials,

3. Development and implementation of rules and regulations governing licensed nuclear activities.
5

4. Conduct of public hearings on radiological safety,

environmental, and antitrust matters, and

5. The development of effective relationships with the

states regarding the regulation of nuclear material.

The major focus of the commission is on the use of nuclear

energy to generate electric power (Office of the Federal

Register, 1976 ,p588). Before a nuclear plant can be used to

generate electricity, NRC must approve the design, construction, and operation of the plant in accordance with the

Commission's established set of regulations. This can be a

lengthy review due to the complexity of power plants and the

need to consider individual designs and subsystems. In 1974,

it was estimated that it took approximately ten years from the

decision to build a plant to the beginning of its operation,

and a major part of this period was devoted to the approval

process (ETIP, 1974, p.2).

One approach used by the Commission to reduce the time

required is that of encouraging standardization in nuclear

facilities. The Commission issues regulatory guides offering

approved methods of compliance with the mandatory regulations

(ETIP, 1974, pp. 4-5).
6

Regulatory guides are developed either by the commission

or by experts in industry, academia, or government through a

voluntary standards program (ETIP, 1974, p.5). This program

is coordinated within the American National Standards Institute

(ANSI) by the Nuclear Technical Advisory Board (NTAB) . The

Board establishes and manages committees of experts for developing nuclear standards. These standards are submitted

to NRC for adoption. In order to coordinate NTAB output with

NRC needs and ensure suitability of standards, the director

of the Office of Standards Development at NRC participates as

a member of NTAB and its executive committee (ETIP, 1974, p.6).

2 .

Development of the Experiment

The new energy conscious climate of the early 1970's

focused national attention on methods for producing electricity

(ETIP, 1974, pp. 2-3). With nuclear reactors being one of

the more attractive alternatives to fossil fuels, interest

1 in building more nuclear plants increased. However, it was

taking a long period of time to bring plants on-line, in part,

due to the lengthy licensing process of AEC. In 1971 the

Commission requested that ANSI accelerate their voluntary

standards program to help reduce these time delays (ETIP 1974,

p.6). ANSI increased its efforts, but even in 1973 it was

still the Commission's view that more improvements in the
7

process were needed (Rodgers, 1973).

ETIP's involvement with this problem began in early

1973. ETIP had issued an announcement in the Commerce

Business Daily presenting ETIP interests and requesting opinions

from private firms (Harter, Note 1). This was the first stage

of a two-step solicitation for developing some contracts with

qualified firms (Van Wyk, Note 2) . ANSI responded to this

request by proposing an experiment with the voluntary standards

process. In an attempt to accelerate the work of their standards

committees, ANSI proposed to reimburse committee members for

travel expenses to meetings. While initial ETIP interest was

low, further discussion of the idea with two members of NTAB

indicated that there might be more experimental potential than

originally proposed (Harter, Note 1)

.

These initial discussions led to contact with another

NTAB member, Lester Rodgers, the Director of Regulatory

Standards for AEC. Rodgers was very interested in improving

the voluntary standards development process at ANSI

(Rodgers, 1973). He provided the contact ETIP needed to* make

the project relevant to a government policy maker and to fulfill

the ETIP model of working through other government agencies.

Development of a formal plan for an experiment began in
8

early 1974 with discussions between AEC and ETIP . The focus

remained voluntary standards committees of ANSI, even though

AEC itself also had a part in the overall standards process.

These committees depended on volunteer experts from

government, academia, and industry. They lacked sufficient

support services to help construct, produce, and distribute

drafts. Both ETIP and AEC were interested in learning if

alteration of these conditions might help reduce the time needed

to draft a standard.
9

3 . 0 The NRC Experiment

/

3.1 Initiation of the Experiment

The general approach in initiating an experiment from the

ETIP perspective is to first make contact with another government agency and discuss the feasibility of doing an experiment.

If the discussions indicate some potential for collaboration,

work is started on a project plan and other related documents,

which must be approved by both ETIP and the agency for a project

to begin. In the NRC experiment, the project plan was developed

primarily at ETIP and was used in the approval process in the

Department of Commerce to justify a transfer of funds to AEC.

AEC then used the plan to contract the experimental work to

ANSI while AEC maintained overall approval in the project.

3.1.1 Planning

Describing the Problem

The project plan provided an overview of the voluntary

standards process including the need for standards, their de

velopment and use, and how the various groups (ANSI, AEC

and industry) were involved. The main problem focus was on

the ANSI part of the process and hoV it might be accelerated
10

through alteration of conditions under which drafting committees

worked. While the plan was sufficient for an overall perspective, several issues should be noted:

1. The relevant background and interests of the groups

involved with the experiment were not all discussed in enough

first in a series of projects examining the relationship be

tween voluntary standards and the regulatory process (ETIP,

1974, p.l) . Documents available to this writer did not indicate an overall strategy involving a series of projects,

and thus the ETIP objectives for the project, both in substantive and methodolgical areas, were unclear.

enough for one to understand the industry stake in the voluntary

standards program or its acceleration. It was indicated that

industry suffered a long delay to eventual plant operation,

•(ETIP, 1974, p.2)

It is unclear how changes in the ANSI

standards program, which provides input to NRC and involves

industry, would have an impact on industry in the long run,

particularly in terms of technological innovation, a key

ETIP interest.

detail

•For ETIP the plan indicated that work with NRC was

•The perspectives of industry were not discussed
11

•The project plan discussed some of the standards development process at ANSI but avoided an overall perspective

of ANSI functions and problems. Thus, while the focus was on

important administrative problems of Committees, equally important technical issues in standards development were not

discussed (Swisher, Note 3) . Also, the other ANSI functions

in the process, such as coordinating reviews of standards in

the field, were not discussed.

•The standards process at NRC itself was only briefly discussed.

2. While it is unreasonable to expect that extensive

discussions on each group would appear in the plan, the general

indication from these shortcomings is a lack of overall perspective of the problem. There is no way to assess the overall

importance of the process at ANSI, and potential subsequent

impacts due to its acceleration, against the rest of the process

at NRC and in industrial application. This would create

problems later during implementation and evaluation phases of

the experiment.

Proposing Some Changes

With the problem focus on the time delay and working

conditions of committees at ANSI, an experiment in altering
12

some of the working conditions was proposed. The purpose of

the project was "to determine whether any of all of the four

changes in the methods of developing nuclear standards is

a desirable method for accelerating the establishment of responsible standards which are approved by ANSI and adopted

by the Atomic Energy Commission" (ETIP, 1974, p.8)

The

four changes suggested were paying for a full time committee

chairman, utilization of consulting services for information

gathering, clerical and editing assistance, and bringing the

committee together for more lengthy meetings.

Approach to Conducting the Experiment

The experiment involved several agencies and some steps

were proposed in how the project was to be conducted. These

were as follows

1. ETIP, after receiving approval of the project plan

within the Department of Commerce, would transfer funds to

AEC

1

.

2. AEC would contract with ANSI, via a sole source procedure, to conduct the experiment for a period of one year.

^It should be noted that AEC was only allowed to use appropriated

funds. Thus, it could not accept funds from ETIP. The actual

arrangement was to establish an account at MBS dedicated to

the AEC project. AEC would bill this account.

(Penn, Note 7)
13

AEC would still retain overall approval authority.

3. The Nuclear Technical Advisory Board (NTAB) of ANSI

would be responsible for the overall management of the project.

This would include selecting the standards to be used in the

experiment and hiring a program manager to run the project.

4. The program manager would hire chairmen for the test

committees , hire and coordinate support services , and collect

data.

5 . The individual committees to participate in the experiment would have the options to select interventions as

needed in their particular situation, even to the extent

of choosing interventions not in the project plan. The program

manager would in a11 cases be responsible for approving interventions and assuring that their impacts could be assessed

later

.

The most notable factor about these arrangements is that

the organizational/managerial distance between ETIP and the

actual experiment was so large. There are several factors

which facilitated or hindered this structure.

1. The commitment of the Director of Regulatory Standards

at AEC was a powerful and useful factor (even though project
14

monitoring was later passed to someone else) . Specifically:

•The Director (Lester Rodgers at the time of project

initiation) was a champion of reforming the standards process,

the central issue of the experiment. This was clear from

formal contacts between ETIP and AEC (Lewis, Note 4; Harter,

Note 5) and his testimony to Congress (Rodgers, 1973).

•The Director was a member of the Nuclear Technical

Advisory Board (NTAB) of ANSI, the group responsible for

coordinating nuclear standards work at ANSI. He was also

a member of the executive board of NTAB and later took part

in the ad-hoc committee for the experiment established from

ANSI/NTAB membership.

•The Office of Regulatory Standards at AEC was the

immediate user of ANSI nuclear standards work of the type

to be used in the experiment.

2. ETIP is interested in conducting its work through

other government agencies. Contracting the experiment through

AEC (rather than directly with ANSI) assured that the

Commission had a direct stake in the experiment. However, the

control of project funds by AEC also presented some problems

for ANSI (see section on implementation)
15

3. Essentially, ETIP did not have control over the

conduct of the experiment, as this rested with MTAB . This

would be a problem later in contracting (see next section)

and experimental methodology (see section on implementation)

•The project plan left a certain flexibility to the

program manager in controlling experimental changes in the

committees

.

•The project contracted at ANSI was already different

from the one initiated at ETIP.

4. During the course of the entire experiment, personnel

in the chain changed (see sections on implementation and

evaluation) . Also, AEC became the Nuclear Regulatory Commission

(NRC) in 1975.

3.1.2 Contracting

In June, 1974, the funds for the experiment were transferred

to AEC, which then contracted to ANSI on a cost-reimbursable

basis. At this point effective control from ETIP was no

longer possible. ANSI went out for bids from consulting firms

to conduct the experiment, contrary to the plan of hiring a

program manager who would coordinate ANSI committees and in-
16

terventions. ANSI also reworded the language of the plan

specifying the interventions to be used. These interventions

became illustrative, leaving ANSI the option to suggest other

interventions it felt important (Harter, Note 6) . These

changes were not expected by ETIP (Harter, Note 6)

The result of these arrangements was the hiring of

Battelle-Northwest to conduct the experiment. Later a separate

agreement was made with Harlan J. Anderson from WestinghouseHanford Corporation to conduct a similar experiment with one of

his committees, using only one of the proposed interventions.

The proposal by Battelle was accepted as the basis for their work,

3.2 Experimental Design

Some amount of design for the experiment was available

before the experiment began. Battelle discussed independent

variables, the variation of independent variables over the test

committees, and the dependent variables. On the other hand,

Anderson only proposed the one change variable and did not

discuss others. This section of the paper will focus on the

.experimental design utilized by Battelle, since theirs was

more complete and they were the primary contractor. The known

characteristics of the Anderson experiment will be identified

as the Battelle design is presented.
17

3.2.1 The Approach Used and Variables

The basic approach of this experiment was exploratory.

It was hypotesized that certain factors in the drafting process

of the voluntary standards groups, such as clerical support

and prior assignment of responsibilities, would have a significant relationship to the speed at which a draft of a standard

could be produced. Other considerations were the cost of

alternative processes in the group effort and +*he quality

of output. It was unclear what impact new committee procedures

would have on these two variables.

Independent Variables

Battelle considered factors in the process which were most

likely to be contributing to the slow development of standards

(Battelle, 1975, p.5). The independent variables chosen for

%

manipulation in the experiment and the difference over normal

procedure were as follows (Battelle, 1975, pp. 6-7):

1. Working meeting. A five day working meeting held at

Battelle. This was different from the usual practice of having

several periodic meetings and communications (ETIP , 1974, p.ll).

2. Working draft or outline. A working draft of some

form would be distributed before the working meeting. While
18

this appears to be the usual procedure, the differences here

were whether it was a draft or outline.

3. Executive secretary. A support person responsible

for assuring the availability of needed services during

meetings and revising drafts according to group member comments.

This constituted new services for meetings and a reduction in

the chairman's workload in communicating committee work. This

was the main intervention made in the Anderson experiment. It

appears that the person in Anderson's experiment also performed

clerical and duplicating functions, a separate intervention

for Battelle.

4. Clerical and duplicating personnel. Support persons

responsible for preparing drafts and distributing materials

to the committee.* This reduced some of the work normally

done by the chairman.

5. Payment of the chairman. Payment of the chairman

for work performed on the standard. This was different from

the usual practice of the chairman being supported through

his employer.

6. Travel and living expenses. Payment to committee

members for travel and living experses incurred while par-

/
19

ticipating in meetings on standards. This was different

from the usual practice of support from the employers of

committee members.

7. Preliminary meeting. Meeting held before the fiveday sessions in order to begin preliminary discussion and

assign responsibilities. This intervention appears to be

due more to recognition of group meetings before the experiment

began

.

8. Committee membership. Committee sizes were limited

to about ten people. While this also appears- to be a function

of group conditions before the experiment, it was probably

a reasonable idea to keep groups the same size so that

theorectically the workload would be the same for all of the

committee members.

Battelle used these factors in variation on three

committees

1. INMM - 11 for ANSI standard N15.28, Criteria and

Standards for the Certification of Nuclear Materials Managers.

2. ANS - 2.12 for ANSI Standard N635, Guidelines for

Combining Natural Phenomena and Man-made Hazards at Power

I
20

Reactor Sites, and,

3. ANS - 18.6 for ANSI Standard N231, Discharge of Thermal

Effluents Into Surface Water.

Anderson used the one intervention on the ASTM C-26 Committee

on Fuel, Control, and Moderator Materials for Nuclear Reactor

Applications.

The conditions were varied over the groups as follows

(Battelle, 1975, p.8):

TABLE I . Elements of the Experiments

Standard Standard Standard ASTM

N635 N15.28 N231 C-26

Working Meeting X

Outline of Standard Prepared

and Distributed

X . X

X

Draft of Standard Prepared

and Distributed

By Chairman X

By Executive Secretary X

Executive Secretary Provided X

Clerical and Duplication

Assistance Provided X

Chairman Paid

.Travel and Living Expenses Paid

Preliminary Meeting/s X

X

X

X

X

X

X

X

X X

X X

X

X

Committee Size Limited X
21

It should be noted that independent variables were not

assigned to the committees in any apparent procedure other than

by recognizing already existing conditions or needs (ETIP , 1974 ,

p.13) . It was to be the duty of the project manager to assure

that they were "used by the various committees in such a manner

that the effect of each can be appropriately determined"

(ETIP, 1974 , p . 13)

.

Also, some of the variables as listed above were later

ignored in the data analysis. For example, the specification of

draft or outline was reduced to working draft, and a preliminary

meeting was reduced to assignment of responsibilities prior

to the five-day meeting.

Dependent Variables

While the speed of the drafting process was the major

variable of interest in this experiment, there were two other

outcome variables which also needed to be examined. These were

the quality of the draft standard and the cost to produce it.

Because of the many groups involved in the standards process

(e.g., NTAB , ANSI, NRC, the nuclear power community, etc.),

these variables were further defined by reference to the

appropriate group. Thus the independent variables considered

by Battelle in their proposal were (Battelle, 1974, pp. 6-8):
22

1. Time. The time required by the committee to produce

a draft for submission to NTAB

2. Quality of the draft standard. The quality of the

draft standard based on the number arid significance of the

comments received from the field on the first draft distributed

for comment by NTAB.

3. Cost of preparing the standard. The cost of preparation (e.g., salaries, travel, clerical support, publication,

etc.) for submission to NTAB.

3.2.2 Analysis of the Design

In order to evaluate the experimental design chosen

by Battelle, some criteria were needed* One scheme which

appeared to be appropriate was the use of a decision process

developed by Anderson (1972) . In this process experimental

designs are chosen on the basis of six questions about what

information is needed from the experiment and the conditions

involved. This diagram and supporting papers appear in the

appendix.

*Most of the following comments also apply to the experiment

conducted by Anderson for ASTM C-26.
23

The six questions are listed below and the answers, as

determined by the writer, follow:

1. Which type of evaluation is needed?

Although the approach to this experience was exploratory, the independent and dependent variables were known and

some of the expected outcomes were discussed before the

experiment. The evaluation needed to assess the effects of

the independent variables on the outcome variables: time, cost

and quality.

2. Can control groups be used?

Control groups were used for the experiment in that

three standards each requiring a different committee were chosen

and the independent variables varied in the treatments . However

,

the indpendent variables were numerous and not assigned in any

logical order for control purposes. As it turned out, each

of the three Battelle groups had almost all of the possible

changes. This problem was identified to Battelle early in the

experiment by NTAB members, apparently with no effect.

(Nuclear Technical Advisory Board, 1974, p.7) .

Another weakness with the design was that there were no control
24

groups operating under the usual circumstances. This need was

considered in the ETIP project plan (ETIP, 1974, p.15). However, further contact with the committee system at ANSI indicated the difficulty of comparing different people, technology

issues, and priorities (in terms of need by NRC) (Harter, Note 1)

3. Is random assignment possible?

Random assignment of groups to the experimental conditions or to the usual arrangements did not occur. It was

probably not feasible given that NTAB was to ‘select standards

needing priority attention, having established drafting committee

chairman but no previous meetings, covering different fields,

and having similar levels of difficulty (ETIP, 1974, p.12.

Nuclear Technical Advisory Board, 1974, p.6).

*

Random assignment of committees to the different

treatment conditions was not performed either because it was

felt by ETIP and NASI that the committees should have the

flexibility to choose conditions based on need (ETIP, 1974,

p.13 , Note 6)

4. Are repeated measurements possible?

Repeated measurements would have provided some before
25

and after treatment comparisons of committee performance.

However, the committees were typically formed to draft one

specific standard. This prevented any comparisons between

committee performance on two different standards, especially

with main interest being on variables based on committee

output only (time, cost, and quality)

.

Given the answers to these questions, and the flow of

the decision process of Anderson's (1972, p.50), the resulting conclusion is that design 3 from Campbell and Stanley

(1966) would have been the design appropriate to the conditions

of the experiment. This design uses one set of observations

made on two comparable but not randomized groups

X 0^ X - treatment (application of the

independent variables)

O

2

0 = observation of the dependent

variables

>

TIME

The design used by Battelle was weak on two points then:

1. They did not utilize any comparison groups operating

under the normal working committee conditions. This would
26

make it difficult to validity claim improvements over normal

conditions

i

‘

2 . They did not vary the treatment or reduce the number

of treatment options to the point where the three experiment

groups could act as controls to each other. Thus, the independent effects of the different treatment options would be

unidentifiable

Some other problems arising due to these conditions were:

1. Selection.

Since randomized assignment of groups was not possible

and treatments were used according to need, it would be

' difficult to identify varying committee experiences as being

the result of the varying treatment conditions or the selfselection of these conditions by the committees.

i I I 2. Interaction of selection and the treatment.

Without comparisons to groups operating under the usual arrange1 I ments , it would be difficult to claim any generalizability to

the results. Experimental results could be partly or solely

a function of the types of committees chosen for the experiment .
27

3.3 Implementation of the Experiment

Battelle started work on the contract in October, 1974, and

submitted their report one year later in September 1975. A separate agreement was made with Harlan J. Anderson of WestinghouseHanford Corporation in November 1974,. and a final report received

in February 1976. Since all control of the experiment was essentially at ANSI, there is little to discuss about the implementation

from the ETIP perspective*. The implementation by ANSI is covered

in other sections on planning and analysis.

There were, however, several interesting events that occurred

during the contract period which should be noted. These serve to

illustrate the problems with the contractual/organizational setup

of the experiment:

•An ad hoc committee was established from ANSI -NTAB members

to oversee the experiment. No ETIP representative was asked to

participate until January 1975, at which time one person acting as

an ex officio member was requested. This delay meant that ETIP

had no direct input to the proposal-contract award decisions.

•Lester Rodgers, the Director of the Regulatory Standards

Office at AEC, retired at the end of 1974. Rodgers was a

* This lack of control by ETIP later increased ETIP ' s interest

in conducting a more well designed evaluation and re-establishing some control . (Harter , Note 1).
28

champion of standards refroms and helped immensely in getting

the experiment initiated. However, he did not participate

in the day-to-day monitoring during the experiment. There

are mixed feelings at ETIP whether this had an impact or

not. For example, the main contact at the Regulatory Standards

Office remained the same, and the new director had similar

reform ideas to Rodgers' (Minogue, 1974a, 1974b). On the

other hand, negotiations with NRC were not as smooth for the

evaluation contract later, and NRC lost interest in having

contracting responsibility for the evaluation. (See section

on evaluation)

•Also, in January 1975, an executive order from the

White House (based on the Energy Reorganization Act of 1974)

was issued establishing the Nuclear Regulatory Commission

(NRC) from parts of the Atomic Energy Commission (AEC) . It

is felt at ETIP that this changeover had litte impact on the

experiment, since the same people remained on the project

and the new Office of Standards Development was essentially

the same as the previous Office of Regulatory Standards.

•By January 1975, three months after the beginning

of the Battelle contract, problems with mechanics of the

contract-agreement with ETIP, NRC, and ANSI were starting to

arise. NRC by charter was not allowed to use any money except

that appropriated by Congress (Penn, Note 7) . This necessitated
29

an arrangement where ANSI had to bill NRC who, in turn, billed

the NBS account. For ANSI, this procedure was different from

their usual one of operating with grants (Van Wyk, Note 8)

.

ANSI needed faster processing of bills and also wanted reimbursement of precontract costs (possibly arising from misunderstanding the meaning of a cost-reimburseable contract) . NRC

held bills until a large enough amount had accumulated.

3.4 Analysis of Outcomes

•Two final reports were issued for this experiment, one

by Battelle (Wittenbrock , 1975) for three test committees,

and one by Anderson (1976) for one test committee. These

reports are completely different and will be discussed in

separate sections here.

3.4.1 The Battelle Report

Battelle proposed to analyze the experiment on the basis

of time for preparation, quality of standards as determined

by comments from relevant groups, and cost (Battelle, 1974, p.2)

At the same time they indicated that the limitations of the

project, and the large number of variables involved would

provide little more than qualitative results (Battelle, 1974,

p.3)

.

They proposed to issue a questionnaire to the test
30

committee members for their evalution of the experimental

approaches. Also a "check sheet" would be used to summarize

the effectiveness of the three different treatment approaches

(Battelle, 1974, p.8).

X

However, Battelle did not present in their final report

most of what they discussed in their proposal. Again, arguments about the lack of quantitative measures, limited time,

limited data, and many variables were discussed to justify

a different approach to the analysis of the experiment

(Wittenbrock, 1975, p.13). The final evaluation consisted

of ratings from group members on the effectiveness of the

innovations, observations by the Battelle project manager,

and costs for the five-day working meetings. Thus, it should

be noted that the three dependent variables were either not

examined or only superficially so. Specifically:

1. Time to prepare the draft.

Time for preparation originally was defined in terms

of the period up until submission of the draft to NTAB . ' Although, theoretically , this time length was available to

Battelle as they were involved up to these draft submissions,

no discussion of time was presented.

2. Quality of the draft.
31

Originally, draft quality was defined in terms of the

number and significance of the comments received from the field

on the draft submitted to NTAB (Battelle, 1974, p.6). Quality

in these terms was not discussed in the final report.

3. Cost of preparing the draft.

Originally, cost estimates were to include all expenses

of the standard drafting process. The costs discussed in the

final report only include the costs of the executive secretary

and salaries of the chairman (Wittenbrock , 1975, p.18). Travel

and living expenses were not presented. The expenses of other

committee meetings besides the ones at Battelle were also not

included. All of the committees held meetings other than the

five day session at Battelle.

Finally, the evaluation "check sheet" which was to act as

a summary of the effectiveness of the three different approaches

was not included.

Ratings by Committee Members

Questionnaires were distributed to committee members at

or near the completion of the working session at Battelle. It
32

was felt by Battelle that because committee members had no

long term commitment to the committee, their feedback would be

relatively unbiased. While this shouldn't be disputed, the

weakness was that their comments received the most attention

in the report.

The questionnaires had 16 items which were identical for all

test committees, except for tense in order to account for

treatment variations. All members were asked to rate items as

experienced or as they would have experienced them. This was a fairly weak approach given the weight placed on the results and

the analysis techniques used.

The 16 items obtained ratings via Likert scales for the

following issues (Wittenbrock , 1975, p.14):

1. The usefulness of each of the specific innovations

instituted in this project;

2. The extent to which the composition of the committee

was appropriate for the task;

3. The skills of the chairman;

4. The difficulty of the standard in question.
33

(e .g. , scope) ; and

5. The overall performance of the committee.

rt should be noted that only seven innovations were considered in these questionnaires, where eight had originally

been defined (Wittenbrock , 1975, pp.6-7). Committee membership,

while presented as an innovation, was dropped. Also, there was

some modification of innovation definitions. For example,

preliminary meeting was changed to mean prior assignment of

responsibilities, leaving out prior organization and discussion

as originally specified.

The first step of item analysis was to present the average

rating for each innovation. The means for each group and the

groups combined were high (favorable) , except for prior assignment of responsibility, which had a more unfavorable rating.

Battelle neglected to present standard deviations and ranges

on these ratings, common and necessary indicators. There was

no way to gauge the level of agreement within the groups. Also,

they presented a mean of all of the individual innovation means,

a meaningless indicator.

Next, these means were subjected to an analysis of variance

to indicate agreement across groups. Three of the innovations
34

had group mean ratings which were statistically different

significantly from each other. While the mean ratings would

seem to support these results, it would have been useful to

see significance levels. Battelle did not indicate what it

considered significant. Also, Battel'le made no effort to discuss

why these differences existed.

The report then discussed another eight variables,

basically parameters, which reflected the working characteristics of the committees , such as adequacy of committee representation and availability of data needed for writing the standard.

Again, as with the innovation ratings, means are presented for

for each group and overall. There are no standard deviations

and ranges. The analysis of variance using these means did

not provide any significant differences, but the actual results

are not presented. Overall performance, the last variable,

is handled in the same way as these also.

The next analysis discussed were regressions of the innovative and committee characteristic variables on overall performance. First, the innovative ratings were regressed on overall

performance , then the committee characteristic ratings , and then

both of these sets were regressed on overall performance. The

prior assignment of responsibility had the only statistically

significant relationship with overall performance. The inter-
'

i&xa tic" I
36

pretation was that prior assignment was a "good" predictor of

the group performance rating—the more favorable the rating

towards prior assignment, the more favorable the rating of

overall performance. It would have been interesting to examine

this result in light of the previous indications that prior

assignment was, on the average, the lowest rated innovation.

There are several problems common to all of these regressions. These are:

1. Small sample size, too many variables.

The three groups had sizes of 6, 7, and 10, or 23 total.

While these are in general small sizes on which to base conclusions, this weakness is even more critical in the regression

models used, where the number of variables in the analysis

approaches the number of individual cases (Weiss, Note 9; Draper

and Smith, 1966, pp. 62-63). For example, a perfect fit for a

regression line can be made when there are two data points and

one independent variable. Thus, n-1 variables, where n is the

total number of cases, can specify a perfect fit. In the *

Battelle regressions, the three analyses are 7 variables and 23

cases, 8 variables and 23 cases, and 15 variables and 23 cases.

2 The R, factor in regression, a measure of the proportion of variation in data explained by a regression line, is
37

affected by this small sample problem. Since the impact is to

inflate R2

, one is even less confident that Battelle really

found anything significant.

2. Lack of specific regression information.

Many of the indicators normally presented with regression

analyses are not presented in the report. Most importantly,

analysis is presented for the innovation variables only. This

included statistical significance levels for each innovation

and the respective increases in R

2 as they were entered in the

model. The report does not include any discussion of the

regression constants A (intercept) or B (slope) , which indicates

the relative importance of each independent variable. These

are normally tested to see if they are significantly different

from zero. Also, *the standard errors of the B constants were

not included, providing no indication of the variability of

the estimate.

3. No overall regression test.

A standard test of any regression line is the overall

F test of whether R (from R2

) equals zero, or equivalently,

whether all B weights are equal to zero. This test is not

discussed. The "F to enter" statistic presented is not explained.
38

The combined effect of all of these problems in presentation

and analysis is a decrease in confidence with the results. The

two major findings based on these analyses are:

1. That most innovations tended to have positive effects,

but several of them, such as payment of chairmen and living

expenses, did not, and

2. That prior assignment of responsibilities seemed to be

related to overall performance.

These have little support. A more rigorous approach in

the application of changes and comparison to other committees

is needed.

Observations

Observations of the three committees were made by the

project manager at Battelle and apply mostly to the five day

workshop period at Battelle. They support the favorable ratings

given by committee members , but do not add a lot of weight

to the results

Summary

The basic conclusions to be drawn from the Battelle analysis
39

are that:

1. The analysis did not fulfill the proposed effort, and

2. While the experiment met some of the needs for exploratory information, no resolution was possible on the effects

' of the individual innovations. • 1

3.4.2 The Anderson Report

The analysis of the ASTM C-26 committee performance was

done independently of the Battelle report and submitted several

months later (Anderson, 1976). Only one of the innovations

was utilized, that of providing a knowledgeable executive

secretary. This basic result claimed was that the provision

of a secretary sped up the committee activities.

For the dependent variables used by Battelle results were:

1 . Time

.

Two standards were produced within one year instead

of the usual two to three. (Anderson, 1976, p.3). Some examples

were provided to show this, but these were not organized well

enough in proving the point.
40

2. Cost.

The cost of the executive secretary was presented in

detail. However, the salary for the secretary was not included

since it was reasoned that under the ideal system, this salary

would be "donated" by the companies or organizations supporting

the committee members (Anderson, 1976, p.l)

3. Quality.

There was no definitive discussion of quality other

than the aforementioned reference to appendix reports. To the

unfamiliar observer, the evidence provided was unclear and not

persuasive. On the other hand, the C-26 committee appeared to be enthusiastic about the results of using the executive secretary,

implying satisfaction with the product (Anderson, 1976,

pp. 3-4)

While the experiment intended with this committee was

simpler than that of Battelle, it did not offer a level of

comparability to Battelle, whereby commonalities could be

assessed. The parameters briefly considered by Battelle^ do

not appear at all in discussions by Anderson. The overall performance of C-26 is not discernable from the report.
41

3.5 Evaluation of the Experiment

An evaluation of the NRC experiment was first considered

in the plan initiating the project (ETIP, 1974, pp. 14-16).

It was proposed that ETIP hire a separate contractor for this

purpose. The content of an evaluation was only briefly considered. Further delineation of the evaluation needs at this

stage might have evolved into an "evaluability assessment",

possibly guiding the design enough to avoid some of the

problems encountered later.

3.5.1 Evaluation Project Plan

In the original project plan several evaluation issues

were discussed. These included the timeliness of the process

attempted, the quality of the drafts in the opinion of AEC,

the efficient and effective use of committee members, and the

comparison of acceleration accomplished versus costs. Considering the lack of background (at the time of the plan) on

the standards process from ANSI through NRC, it is unclear

whether these questions could have been answered in a reasonable amount of time. For example, the experiment essentially

ended at the draft review process in ANSI, but the evaluation

was to look at impacts in NRC.

During the experiment, it became increasingly clear that
42

the evaluation would have to be more encompassing that the

above considerations. This arose because ETIP and ANSI personnel felt that the design by Battelle was leaving many

factors unconsidered (Harter, Note 1). Also, ETIP had requested the Technical Analysis Division (TAD) of NBS to review

the experiment and propose methods for measuring variables

(Harter, Note 1). TAD commented that the intervention impacts were already obvious, that the design was poor and left

variables such as group dynamics or technical problems unconsidered, and that there were too many variables for effective

control (Swisher, Note 3; Harter, Note 10).

The outcome of these events was a more encompassing project

plan for evaluation that was adopted by the Department of

Commerce in April, 1975 (midway through the experiment). The

plan considered backgrounding of the standards development

process, a detailed review of the experiment, the post-experiment history of the drafts, the project costs, and what

recommendations could be made (ETIP, 1975). The plan still

lacked any emphasis on comparing the process and outcomes of

the experiment with other similar variables in the normal

procedure used at ANSI. Another problem with the plan yas

that it concentrated heavily on process variables. With the

experiment half over already, it could not be expected that

the perishable process data would be obtainable.
43

3.5.2 > Ontracting the Evaluation

With the evaluation depending on capturing the process

experience of the committees, the necessity of contracting the

work as soon as possible was obvious. However, the plan was

seriously delayed and timely access to the participants and

results of the experiment lost.

The basic organizational arrangement for the evaluation

was to have been the same as with the experiment. Funds were

transferred to NRC in June, 1975, but it wasn’t until

December 1975, three months after Battelle issued its final

report, that NRC sent a letter to ANSI to prepare a proposal

for evaluation. This work was designated by NRC as an extension of the experiment contract.

ANSI was to have responded by the end of 1975, but they

took until February, 1976. During this time ANSI sent an RFP

out for the evaluation contract and received three responses

(Stafco, 1976, Rockwell International, 1976, Nuclear Services

Corp., 1976). Also, the Director of Nuclear Programs at

ANSI , who had been the main contact and coordinator for the

experiment, left ANSI.

Before NRC received the ANSI response in February, ETIP

contacted NRC and expressed displeasure with the process

occurring (Berlin, Note 11) . A month later, in March, there
44

was still no contract with ANSI. Also, several new events

began to reorient the project.

•The new coordinator at ANSI for the experiment indicated

that the process studied in the project was a small part of

the entire standards process. He wished to explore the other

parts and develop new ideas during the evaluation project.

•ETIP discussed the evaluation mostly with ANSI personnel,

leaving NRC unaware of the changes being considered.

•ETIP began to push for comparing the tested committees

to others operating under normal procedures. ETIP also

pushed for a system analysis of the entire standards process

to gain some perspective on the experiment.

•NRC attempted to put through a sole-source justification

for contracting with ANSI. This was running into delays because NRC was uncertain that ANSI was uniquely capable to conduct the evaluation .

*This was a reasonable concern given that ANSI was not uniquely

qualified to do evaluation. Also, with more and more emphasis

being placed on follow-up work in the standards process, ANSI

would potentially be investigating the process at NRC. This

would not have been an appropriate arrangement, especially

considering NRC ' s reluctance to have anyone study their

processes
45

The outcome of these events was that ANSI issued a new

proposal to NRC in June, 1976, but NRC could still not contract

the evaluation to them.

At a meeting in July, 1976, NRC expressed the desire to

remove themselves from contracting responsibility. While

they were still interested in evaluation, they felt that

ETIP could more appropriately manage it (Garrity, Note 12)

.

The situation was also further complicated by the fact that

the original contract between NRC and ANSI expired.

ETIP eventually agreed to a return of the funds for

evaluation from NRC. The evaluation project was deobligated

in November 1976, and the original experiment was deobligated

in December 1976.
46

4.0 Conclusions and Recommendations

4 . 1 The NRC Experiment

The results of the experiment are fairly inconclusive.

While the work of the selected committees may have been

accelerated by the various interventions, it is unclear how

the improvements compare to normal circumstances, and what

the overall impacts have been on the drafting process at

ANSI. The analyses presented are insufficient and at times

methodologically weak for determining the individual impacts

of the different changes made. If any conclusion can be made

about the results, it is that it appears reasonable to accelerate the committee work by providing extensive support services in combination with prolonged meetings. What importance this has toother committees, other problems in drafting

standards, other parts of the process, or industrial technology,

is unknown.

The option still exists at this time to contract some

analysis of the experiment in the evaluation phase of the

project. Clearly the process data so important to this experiment is no longer obtainable. However, the resulting draft

standards probably exist somewhere in the standards process

and these may be interesting to examine. For example, the
47

history of each draft as it went through further review may

be able to provide some evidence of the quality of the draft,

one of the original outcome variables. The history may also

show the relative importance of the acceleration of committee

work to the later review processes of ANSI and NRC.

If any work is to be done, some effort should be included

on specifying what constitutes the standards process. This

would at least add some perspective to the experiment and could

also point out other areas where further work could be done.

4.2 ETIP Experiments

It seems clear from this experiment that the backgrounding

and contracting phases of a project are two of the more important considerations. While the hindsight review of these

factors easily demonstrates their weakness, it does need to be

realized that items overlooked at these initial stages can have

impacts on later parts of the experiment that can not be so

easily controlled or adjusted. For example, the apparent

lack of perspective of the ANSI functions in the overall standards

process left some uncertainty as to the importance of the

experiment. Several groups expressed this point during the

project and, in fact, the lack of importance became one of the

lasting conclusions about this work.
48

What is- needed in backgrounding is the careful and comprehensive examination of groups having a stake (either directly

or indirectly) in the experiment. This includes defining the

problem (or lack of) as each group views it and identifying

what they might get out of the experiment. This is especially

important where cooperation of a group is needed. Specification of ETIP contributions and utility in the project should

also receive full attention.

Of equal importance with these analyses is the arrangements made between the groups to accomplish their objectives.

The most obvious problem of the NRC experience was the organizational distance between ETIP and the experiment. The lack

of control for ETIP in this arrangement resulted in some

serious methodological problems in the experiment of large

importance to ETIP. The arrangement was also not suitable for

the two phases of the project. While ANSI control was most

likely appropriate for management of the interventions, it was

less relevant to an evaluation of the changes. This was

especially true from the viewpoint of NRC since the ANSI work

would have involved examining their processes.

The use of two phases , one for the interventions and one

for the evaluation, also needs some discussion. First, it is
49

useful to have a follow-on strategy for evaluation, as this

project did to some extent. However, this should be carried

even further to the point of assessing the evaluability of any

changes before they are made. An "evaluability assessment"

would serve to further tie the interventions and evaluations

to each other. This need was clearly demonstrated in this

case because of the major emphasis on process variables.

Evaluation at the later point could not compensate for the

original design deficiencies.

Furthermore, it would seem useful that ETIP consider an

experiment as consisting of interventions and evaluation

rather than separating the two into several independent

phases. Even a cursory glance at the Battelle work demonstrates the need for this, even if one only considers that

the difference between Battelle' s "experimental analyses" and

the proposed second phase evaluation was small. In principle

there is no difference and the NRC experience demonstrates

the problems of trying to separate them.

REFERENCE NOTES

1 Phone conversation with Philip Harter, former Chief of

Regulatory Programs at ETIP, December 6, 1976.

2 Conversation with Marie Van Wyk , Special Assistant to

the Director, ETIP, December 7, 1976.

3 Handwritten note from R.B . Swisher, National Bureau of

Standards, to Philip Harter, January 25, 1975.

4 Conversation with Jordan Lewis, Director, ETIP, September 21, 1976.

5 Letter from Philip Harter to Lester Rodgers, Director,

Office of Regulatory Standards, Atomic Energy Commission,

February 1, 1974.

6 Notes by Philip Harter, September 11, 1974.

7 Conversation with Richard Penn, Director Special Projects,

ETIP, December 22, 1976.

8 Conversation with Marie Van Wyk, December 7, 1976.

9 Conversation with Roland Weiss, Operations Research Analyst,

ETIP, September 6, 1976.
RN-2

10

11 Notes by Philip Harter, March 10, 1975.

Notes by Victor Berlin, Chief, Experimental Methods, ETIP

,

February 3, 1976.

12

Notes by Stephen Garrity, Policy Analyst, July 21, 1976.
REFERENCE LIST

Anderson, H.J. "A Research Report, A Single Variable Experiment

on Accelerating the Writing and Completing of National Concensus Standards." Final report on part of ETIP Project #47

submitted to the Amercian National Standards Institute, Inc.,

New York, New York, February 27, 1976.

Anderson, J.K. A Systems Analysis Evaluation Framework . Unpublished Master's Thesis, Department of Industrial Engineering and Management Sciences, Northwestern University,

December 1971.

Battelle Memorial Institute, Pacific Northwest Laboratories.

"Proposed Research Program, Research Experiment on Accelerating the Establishment of Standards and their Adoption As

Regulatory Guides by the U.S. Atomic Energy Commission."

A proposal for the conduct of ETIP Project #47 submitted to

the American National Standards Institute, Inc., New York,

New York, August 28, 1974.

Campbell, D.T., Stanley, J.C. Experimental and Ouasi-Experimental

Designs for Research . Chicago: Rand McNally, 1966.

Draper, N.R. , Smith, H. Applied Regression Analysis . New York:

John Wiley and Sons, Inc., 1966.
RL-2

Experimental Technology Incentives Program (ETIP) . "Accelerating the Establishment of Standards and -Their Adoption As

Regulatory Guides by the Atomic Energy Commission." Duplicated background paper and project plan for ETIP Project #47,

National Bureau of Standards, Washington, DC, 1974.

; Experimental Technology Incentives Program (ETIP) . "Evaluation

. of ETIP/NRC Standards Development Project." Project plan

! for ETIP Project #95, National Bureau of Standards, Washington DC, 1975.

Minogue , R.B. "Nuclear Standards - Goals Versus Accomplishments

As AEC-Regulatory Sees It." Paper presented to Executive

Conference on Nuclear Standard?, Washington, DC, no date

i but assumed 1974 (a)

I I I I

Minogue, R.B. "The Regulatory-Industry Interface in Standards

Development." Paper presented AIF Workshop on Reactor

Licensing and Safety, San Diego, California, December

11, 1974 (b)

i

Nuclear Services Corporation "Project Number 95, An Evaluation

of Project #47, Accelerating Establishment of Standards by thi

U.S. Regulatory Commission." Proposal for the conduct

of ETIP Project #95 submitted to the American National

Standards Institute, Inc., New York,
RL-3

New York, January, 30, 1976.

Office of the Federal Register, National Archives and Records

Service, General Services Administration, 1976/77 United

States Government Manual , Washington, DC: Office of the

Federal Register, May 1. 1976.

Rockwell International , Atomics International Division.

"Proposal for Project #95 - Evaluation of ETIP Experimental

Standards Development Project Number 47." .Proposal for the

Conduct of ETIP Project #95 submitted to the American

National Standards Institute, Inc., New York, New York,

January 30, 1976.

Rodgers, L.R. "The Role of Regulatory Standards in Nuclear

Power Reactor Safety." Duplicated paper of testimony before

the Joint Committee on Atomic Energy, U.S. Congress,

Washington, DC, September 26, 1973.

Stafco, Inc. No title. Proposal to conduct ETIP Project #95

submitted to the American National Standards Institute, Inc.,

New York, New York, January 29, 1976.

-Wittenbrock , N.G. "An Experiment on Accelerating the Writing

of Nuclear Standards Conducted for the American National

RL-5

Standards Insitute." Final report on ETIP Project #47

submitted to the American National Standards Institute,

Inc., New York, New York, September 1975.

NUCLEAR

REGULATORY

COMMISSION

APPENDIX A

Nuclear Regulatory Commission / 587

*

This

chart

from

the

Office

of

the

Federal

Register,

National

Archives

and

Records

Service,

C?".eral

Services

Administration,

197h/77

United

States

Government

Manual

,

Washington,

D.C.

Office

of

the

Federal

Register,

May

1,

1976,

p.

587.
APPENDIX B

' 'V a

M3 (/> O -H <V 'P CO to rH 4-» rH 0) +-> 43

<B C o M aj d) rH P P e 4-1 (X O e <u g P O- 4,0 cd a3 (D 0— 1 1 u C- O >\*H 43 32 o (D V 03 o s a> o <u 4J e a) OOO- e g »H pH CO 4-i a o h e e io u rH >N to co T3 O in P P <D <D a3 <D P X) o 3. o e 33 ^ a 33 3 U U Cue--- aj 0D*H V) rH P 3 -H £ CL, 3 -H u o UHTJ 3 13 P •H (O 0-0 CO to C n in P *H H (D 4) C O 4> CO CO ago p ca to (U 3 «

03 4-> 33 > u « e <n to co O o3 03 co p a> o t- 33 e <u o

P CO<D Sue U 003 HH 03 a, U to p <ea <+j cu e a

•H 3 . . e . # # 32 cH pH r-j to LTJ vO

qO

UOa

o.

e'- en

c

o

IAu

V

13c<

s

o

0.

<4H

4> Or0) «

X

h-H QZ

03

a,a.<
A

minus

indicates

a

weakness;

a

plus

indicates

that

the

design

controls

for

that

threat;

a

question

mark

indicates

a

possible

source

of

concern;

and a

blank

indicates

that

the

factor

is

irrelevant.

(from

Campbell

and

Stanley,

1966,

pp.

8,

40,

and

56)

i—» i—‘i—‘i—*1—‘i—‘i—

l O^OOO'~lCT'Ln4>U)NJ

a

3

COH»

003 03

CO rj3

3

fl> rt

(01

CO

+ CL + + I + + + + > + + + +

CD

I I

T3

ro3

CL + co + + I + + + + + + + + "J I I

+ 0T + + + + + + + + + + + +

3

rt

co -o(D + + • -o + + + + ->3 + + + +

T3

+ (D + + + + •'S + + + + + + +

+ 3 + + + + + + + + + + + I +

rtO

~o T3 +

03rtO

or

+ + + + + + + + i +

+ C + I I •'i I + + + + + + ' 'a

rtor

ro

rtH*

03

I + + •'*1 I I + + I

I |_U I + 4- .-O •'J ‘O .>3 --O •'O «^3 "O I I I TO3

+ .>3 -j- **0 •'O *-0 | •'O »-0 **0 »-0

I I

Table 3.1

Strengths and Weaknesses of Research Designs

B-2

Hist.

Mat 'n.

Test ’ g.

n> Instr.

3

0> Regr.

Sel’n.

Mort

Int

Int . - Test ,X

pi

rt Int.-

p Sel. ,X

3

dol-1 React

Mult .
B-3

Campbell and Stanley (1966) discuss 16 social science research

designs and describe which of the above threats to validity each falls

prey to. Their list of research designs is as follows:

1. One shot case study: a single group is studied only once,

after exposure to the treatment of interest —

X 0

treatment observation

2. One group, pretest and posttest: a single group is observed

before and after — 0X0

3.

Static group comparison: two groups are studied (one after

treatment, one after no treatment) —

X o x02

4.

Pretest-Posttest Control group design: two randomly constructed

groups are studied (one before and after treatment, one before

and after no treatment) —

R 0

X X 02

°3 °4

randomized.

5.

The Solomon Four-group design: four randomly constructed groups

are studied (two before and after, one not receiving treatment,

and two after, one not receiving treatment) —

R 0 X X 0 2

R O 3 O4

R X 0 5

R 0 6

6 . The Posttest-only control group design: two randomly constructed

groups are studied (one after treatment, one after no treatment) —

R X 0 X

R 0 2

7. The time-series experiment: one group is measured repeatedly,

a treatment being administered sometime

0 °1 °2 n X
B-4

The equivalent time-samples design: one group is measured with

randomly alternating exposure to similar occasions —

X ]_0 Xo0 X XqO x

0

Occasion 7with treatment present

Occasion without treatment present

The equivalent materials design: one group is measured repeatedly

with exposure to equivalent samples (a-d) of materials —

MaXl0 M^O M

c

X_!0 MdXo0

materiality

sample

10. The nonequivalent control group design: two groups (not pre-experimentally

equivalent) are measured before and after — 0X0

0 0

11. Counterbalanced design: two or more groups are exposed to different

treatments at different times —

Time 1 Time 2 Time 3 Time

Group A xx

o x2 o X3 0 x4 o

Group B X2 0 X4

O xx o X3 O

Group C x3o X-lO x4 0 X2 0

Group D x4 0 X3 0 X2 0 xxo

12. The separate- sample pretest-posttest research design: two randomly

chosen groups are measured (one initially, one after treatment) —

R 0 (X) irrelevant treatment

R X 0

13. The separate-sample pretest-posttest control group design: four

randomly chosen groups are measured (two initially, one after treatment,

one after no treatment) —

R 0

R

R

R

0

0

(X)

X

0
B-5

14.

The multiple time-series design: two similar groups are measured repeatedly, the treatment being administered

to only one —

0

0

0

0

X 0

0

15.

The recurrent institutional cycle design: a "patched

up" design: specific design features are added to

control for variables noticed to be threats to validity

once the study is underway. An example —

X Ox

o2 X o 3

16.

Regression-discontinuity analysis: an "award (scholarship,

admission to university, etc.) is given to certain members

(chosen for high achievement for example) and then only

that group is measured to see if the treatment (the award)

made a difference:

Three tables are included in the original work to show how each of the

designs "scores" on the threats to both internal and external validity.

The three are summarized in Table 3.1, which has plus signs to show the

strengths of the designs and minus signs to indicate weaknesses (or

threats to validity)

At this point, the evaluator has determined which alternative

research designs are possible for his use. Now he must decide how he

will collect data from the group(s) to be studied. He has two main

possibilities: to use data already collected and available in records

or to administer some sort of measurement instrument himself. The

former has an advantage of being non-reactive: that is, it does not

selection

the award those selected of those selected
APPENDIX C

CHRONOLOGY OF EVENTS

DATE EVENT

September 26, 1973 Lester Rodgers, Director of Office of

Regulatory Standards, AEC , testifies

before Joint Committe on Atomic Energy,

U.S. Congress, on the role of regulatory

standards in nuclear safety.

February 1, 1974 Letter from Philp Harter, Chief,

Regulatory Program, ETIP , to Lester

Rodgers confirming interest in an

experiment

March 21, 1974 ETIP completes a project plan for the

NRC experiment.

April 17, 1974 Project plan is sent to AEC.

April 25, 1974 ETIP submits further changes in project

plan to AEC.

April 29, 1974 AEC formally expresses to ETIP interest

in a joint ETIP-AEC project.

May 7, 1974 ETIP informs AEC of intent to proceed

with the project.

May 16, 1974 NBS forwards project plan to Assistant

Secretary for Science and Technology,

Department of Commerce, for approval.

May 23, 1974 The Assistant Secretary for Science

and Technology, Department of Commerce,

approves the plan.

June 12, 1974 ETIP advances project funds to AEC.

August 28, 1974 Battelle-Northwest submits a proposal

to the American National Standards Institute (ANSI) for conducting the

NRC experiment.

September 10, 1974 First meeting of N635 committee.

The committee is organized, the content

of the standard discussed, and the work

assignments made.
DATE

September 23-24, 1974

October 15, 1974

October 29, 1974

November 1, 1974

/

November 7, 1974

November 7&8, 1974

November 26, 1974

December 18, 1974

December 31, 1974

January 9, 1975

January 15, 1975

January 24, 1975

January 25, 1975

C-2

EVENTS

First meeting of N15.28 Committee

Battelle begins work. Scope of standard

written

' Harlan Anderson submits a proposal to

ANSI for conducting a similar experiment

with another stnadards committee.

Nuclear Technical Advisory Board (NTAB)

of ANSI meets. Battelle presents description of the experiment. The ad-hoc

NTAB-ETIP committee given responsibility

to select standards to developed. NTAB

indicates some discomfort with experimenta

design

NTAB advises Battelle of three standards

to be developed.

Second meeting of N635. Committee. The

approach to writing the standard is discussed, assignments made, and a schedule

developed.

ANSI informs Anderson of proposal

acceptance

Battelle report #1. Recruitment of N231

committee in progress. N635 Committee

already organized and assignments made.

Lester Rodgers, Director of the Office

of Regulatory Standards, AEC, retires.

ANSI requests a representative from ETIP

for an ex-officio position on the ad hoc

ANSI-ETIP committee.

Executive order issued establishing the

Nuclear Regulatory Commission (NRC) from

AEC (effective January 19)

ETIP reviews payment system due to ANSI

feedback about delays.

TAD comments on experiment received indicating several weaknesses of experiment.
C-3

DATE EVENT

January 27-31, 1975 Third meeting of N635 committe held at

Battelle. A Draft 1, Revision 0 is

produced

February 3, 1975 ETIP suggests a representative for ad hoc

ANSI-ETIP committee to ANSI.

February 12, 1975 Second meeting of N15.28 committee.

Writing assignments were made and criteria

for the standard discussed.

March 25, 1975 A project plan for ETIP Project #95,

evaluation of the experiment, is completed

at ETIP.

April 7-11, 1975 Third meeting of N15.28 committee held

at Battelle. Draft 0, Revision 3, is

produced

April 18, 1975 The Assistant Secretary for Science and

Technology, Department of Commerce,

approves the project plan for ETIP

Project #95.

April 25, 1975 NRC is informed of Project #95 approval

from ETIP.

April 28-Mav 2, 1975 First meeting of N231 committee held at

Battelle. The outline for the draft

was revised and assignments made.

May 6, 1975 Fourth meeting of N635 committee. Comments

on Draft 1, Revision 0 reviewed and

draft prepared for submission to outside

groups

May 13, 1975 NTAB meeting which was to have discussed

Project #95, but not accomplished.

May 21, 1975 Fourth meeting of N15.28 committee.

Comments on Draft 0, Revision 3, are

reviewed and changes agreed to.

June 16, 1975 ETIP funds for Project #95 are obligated

to NRC
01 •U

DATE EVENTS

Fifth meeting of N15.28 committe. Draft 0,

Revision 4 was discussed. These comments

were reviewed until the end of July.

July 2, 1975 ANSI not yet received the evaluation contract for Project #95.

July 9, 1975 Delivery date for Project #47 experiment

extended to December 30, 1975.

August 1, 1975 Evaluation contract for Project #95 still

in NRC contracting office.

August 20, 1975 Chairman of ad hoc ETIP committee changed.

Ralph Chalker resigns and Robert Davidson

assumes chairmanship.

September 1975 Battelle submits final report to ANSI

for Project #47.

September 1, 1975 N231 committee prepares Draft 1, Revision 0 but does not yet submit for outside

comment

December 2 , 1975 NRC sends invitation to ANSI for the conduct of Project #95, the evaluation.

January 23,28, 1976 ANSI sends out invitation to potential

contractors for conduct of the evaluation.

January 30, 1976 Three proposals submitted to ANSI for

conduct of the evaluation.

February 3, 1976 ETIP contacts NRC to express displeasure

with the process of contracting the evaluation and need to keep ETIP informed.

February 11, 1976 ANSI sends response to NRC invitation

for the Project #95 evaluation.

February 27, 1976 Harlan Anderson submits final report on

C-26 committee to ANSI.

March 26, 1976 NRC still has not signed a contract with

ANSI. New project manager at ANSI.
C-5

DATE EVENT

June 14, 1976 ANSI sends second response to NRC invitation for PROJECT #95 incorporating

new components based on ANSI-ETIP discussions . July 21, 1976 NRC and ETIP staff meet to discuss experiment and evaluation. NRC express

desire that ETIP take control of the

evaluation effort.

September 2, 1976 NRC formally requests that ETIP recall

the evaluation fudns and manage the

effort

November 3, 1976 ETIP informs NRC that the funds obligated

for Project #95 are to be rescinded.

Novmeber 29, 1976 Project #95 deobligated at ETIP.

December 16, 1976 Project #47 deobligated at ETIP.
-
APPENDIX E

EXPERIMENTS #1 THRU 3

APPENDIX E

Experiment #1: ANS 2.12 (ANSI N635) Guidelines for Combining Natural Phenomena

and Manmade Hazards at Power Reactor Sites

Sponsoring interest group: American Nuclear Society (ANS)

Variables tested: Working meeting; draft of standard prepared and

distributed by executive secretary; executive secretary

provided; clerical and duplicating support provided;

preliminary meetings held; committee size limited (to 11)

Sequence of events: — Two meetings conducted before experiment started — First meeting organizational, assignments

made (September, 1974) — Second meeting used to design overall writing

approach; members assigned to task groups,

tentative schedule made (November, 1974) — Five-day working session held in January, 1975 — Four members not in attendance — Secretary compiled draft sections written

by individuals into working paper as starting point — Draft standard completed during course of week — Agreement made to incorporate final day's

changes as the next revision — Fourth meeting held in May, 1975 to review comments

on draft — Group agreed that draft ready for review

outside the group — Draft copies were sent for review to committee

members and to group of outside experts

Outcomes: — Standard cleared by appropriate American Nuclear

Society committee in May, 1977 — Standard sent out for ballot by ANS members in December,

1977 — Standard submitted to ANSI in October, 1977, and sent

out for public review and comment (concurrent with

ANS membership review) — Writing group met in February, 1978 to resolve comments; writing group response out to ANS and ANSI

by April, 1978 — ANS membership approved standard — ANSI approved as American National Standard in July,

1978

NRC received copy for review and comment on August

8, 1978

NRC sent comments back to ANSI committee on November 6

1978.
2

Experiment #2: INMM-11 (ANSI N15.28) Criteria and Standards for the

Certification of Nuclear Materials Managers

Sponsoring interest group: Institute of Nuclear Materials Management (INMM)

Variables tested: Working meeting; outline of standard prepared and

distributed; executive secretary provided; clerical

and duplicating support provided; chairman paid; travel

and living expenses paid; preliminary meetings held;

committee size limited (to 9)

Sequence of events: — One meeting held before experiment started — Two-day meeting held September, 1974: scope

of standard was drafted; discussion — One-day meeting held in February, 1975: discussed

philisophical background; writing assignments

made — Five-day working session held in April, 1975 — Two members not in attendance, and three

were alternates — Rudimentary outline of standard distributed

to members before meeting as starting point — Draft standard completed during course of

week; some sections incomplete — Agreement made to incorporate final day's

changes as the third revision; chair agreed to

draft incomplete sections after received

clean draft — Fourth meeting held in May, 1975 to review comments

on draft (five attended): changes agreed upon for

next draft — Fifth meeting held June, 1975 (seven attended):

distributed/discussed , draft and agreed upon

submission to INMM — Draft sent to INMM Executive Committee to review — Writing group chair and other member review Execitive

Committee comments, incorporate into revised draft

by end of July, 1975 — Revised draft sent to writing group, designated

reviewers and INMM Executive Committee for review — Draft completed by November, 1975

Outcomes: — Standard submitted to ANSI in February, 1976 — Standard withdrawn for revision in July, 1976

Discussion: When the standard was sent out by ANSI for public com- ment, there was so much comment that the standard was premature (because the state of knowledge was insufficiently far advanced) that the standard was withdrawn.
3

Experiment #3: ANS 18.6 (ANSI N231) Discharge of Thermal Effluents Into

Surface Waters

Sponsoring interest group: American Nuclear Society (ANS)

Variables tested: Working meeting; draft of standard prepared and distributed by chairman; executive secretary provided;

clerical and duplicating support provided; chairman

paid; travel and living expenses paid; committee size

limited (to 10)

Sequence of events: — No meetings held before experiment started — Chairman drafted preliminary draft and distributed

to members two weeks before working meeting; working

assignments made before meeting — Five-day working session held in April, 1975 — All members were in attendance, with one an alternate — First step was major revision of preliminary

outline — Two draft revisions written and reviewed — Agreement reached that further work needed

before submission for outside review, and

work assignments made — Assignments to members not received as of August,

1975

Outcomes: — Draft never submitted for public review

Discussion: All environmental standards of ANSI's Committee N-19

(the ANSI recipient committee for this standard) are

in the process of reevaluation, because of problems

getting them through the ANSI public review process.

(Extensive commentary on the environmental issues is

stopping approval of all environmentally-related standards.) ANSI review of the Committee's standards has

thus been suspended.

NBS-114A (REV. 9-78)

U.S. DEPT. OF COMM.

BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO. NBSIR 80-20862.Gov't Accession No. 3. Recipient's Accession No.

4. TITLE AND SUBTITLE

N ASSESSMENT OF AN EXPERIMENT IN ACCELERATING

THE DEVELOPMENT OF NUCLEAR STANDARDS

5. Publication Date June 1980

6. Performing Organization Code

7. AUTHOR(S)

Jane P. Woodward and Stephen D. Garrity

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS 10. Project/Task/Work Unit No.

NATIONAL BUREAU of STANDARDS Performance Development Institute

department of commerce anH 2550 M Street, NW

Washington, DC 20234 Washington, DC 20037

11. Contract/Grant No.

7-35822

12. SPONSORING ORGANIZATION NAME AND COMPLETE ADDRESS (Street, City, state, ZIP)

National Bureau of Standards

Department of Commerce

Washington, DC 20234

13. Type of Report & Period Covered

Final

14. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

| | Document describes a computer program; SF-185, FIPS Software Summary, is attached.

16. ABSTRACT (A 200-word or leaa lactual summary of moat significant information. If document includes a significant bibliography or

literature survey, mention it here.)

m

As part of its program on regulatory experimentation, the Experimental Technology Incentives Progconducted a project with the Nuclear Regulatory Commission on accelerating the standards develop

process. This report assesses the execution and outcomes of the project with a specific focus on whether further work is needed on the project and on how to improve future experimental designs.

The report includes sections on agency and project background, project initiation, project plans,

what actually happened, what was learned about the standards development process, and factors

to consider in the design of other experiments.

anr eni

17. KEY WORDS (six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unleaa a proper name;

separated by semicolons)

Administrative experimentation; evaluation; Experimental Technology Incentives Program;

nuclear regulatory standards; regulatory experimentation

18. AVAILABILITY jXjUnlimited

I I For Official Distribution. Do Not Release to NTIS

Order From Sup. of Doc., U.S. Government Printing Office, Washington, DC

20402, SD Stock No. SN003-003-

|5T! Order From National Technical Information Service (NTIS), Springfield,

VA. 22161

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

21. NO. OF

PRINTED PAGES

155

22. Price

$ 11.00

USCOMM-DC
—