NBSm 73-245

Toward New Concepts of Management

Applicable to the U.S. Bureau of Mines'

Energy Research Program

Robert S. Culter, Mary Ellen Golby

Prtjeet SOAT»

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

July 1973

Status Report

April - June 1973

I- repared for Office of Assistant Director - Energy

U.S. Bureau of Mines

Department of the Interior Washington, D. C. 20240

NBSIR 73 - 245

TOWARD NEW CONCEPTS OF MANAGEMENT

APPLICABLE TO THE U.S. BUREAU OF MINESENERGY RESEARCH PROGRAM

Robert S. Cutler. Mary Ellen Golby

Prtjtei SOAP

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

July 1973

Status Report

April-June 1973

Prepared for Office of Assistant Director - Energy

U. S. Bureau of Mines

Department of the Interior Washington. D. C. 20240

U. S. DEPARTMENT OF COMMERCE, Frederick 8. Dent, Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts, Director
c

I
ACKNOWLEDGEMENTS

A joint task group effort of this nature depends upon the cooperation

and assistance of many people both within and outside the respective

organizations. Appreciation is given to the many individuals who participated

in various parts of the survey for their valuable contributions.

Special gratitude is extended to Mr. William L. Crentz, Assistant

Director- Energy , U, S. Bureau of Mines, whose guidance and sponsorship

of this assignment constituted a noteworthy step towards the introduction

of a results -oriented program management philosophy into an established

government research organization.

The authors wish to thank Dr. Alan J. Goldman for his helpful comments

and encouragement during the review of this report. Acknowledgement is

also due to Mrs. Ruth M. Ciufolo and Mrs. Diane R. Beall, without whose

assistance the report would not have heen possibly.
Hi

.>i v.:
AJlS'l'llXCi'

This report describes the progress to date of an effort to develop

new concepts of management applicable to tlie U.S. Bureau of Mines'

program of Energ>' Researcli. A systems analysis task c^oup was organized

to identify principal problems associated with implementing a missionoriented research program and, in particular, to design a mechanism for

detennining researcli project priorities in terms of stated goals.

The Phase I Exploratory Survey, consisting of 14 group interviews at

the Washington Headquarters and at two operational Research Centers, is

detailed. Results thus far include listings of acknowledged "problem

areas," uiiere reasonable improvements are considered possible. In addition,

a metiiodology analogous to the "peer-review" procedure used elsewhere in

government for evaluating research projects in terms of "technical quality"

and "program re. evance" was introduced and tested m a simulation exercise.

Preliminary results were encouraging and suggest direction for the additional

work needed before this mechanism can become a useful management tool for

budget review purposes. A second exercise is recommended involvirig all six

USBM Energy Resarch Centers in conjunction witJi a forthcoming Mid-Year Budget

Review.

TABLE OF CONTENTS

1. INTRODUCTION 1

1.1 Background Perspective ^

1.2 Scope of Assigrunent

1.2.1 Objectives 8

1.2.2 Approach ' ^

1.2.3 Preliminary Results ^

2. PHASE I: EXPLORATORY SURVEY H

2.0.1 Project SOAP Orientation H

2.0.2 The Initial Task 12

2.1 Washington Staff Interviews 1-^

2.2 Morgantown Energy Research Center Interviews 1^

2.2.1 MERC Visit #1 17

2.2.2 MERC Visit #2 21

2.3 Bartlesville Energy Research Center Interviews 22

2.3.1 BERC Visit #1 22

2.4 Exercise I: Preliminary Assessment of Research

Project Proposals, 27

3. FINDINGS AND CONCLUSIQIS — 30

3.1 MERC "Problem Areas" 31

3.2 BERC "Problem Areas" = 32

3.3 Exercise I Prelir/iinary Results 33

3.4 Conclusions 40

4. APPENDIX

Sections A and B

TOVJARD NEW CONCEPTS OF MANAGEMENT APPLICABLE TO THE U.S. BUREAU OF MINES'

ENERGY RESEARCH PROGRAM

1 . INTRODUCTION

This report describes the first three months of work performed

under an Interagency Agreement by which the National Bureau of Standards

(Department of Commerce) is to assist the Energy Research Group of the

Bureau of Mines (Department of Interior) in developing new concepts of

Management applicable to their program of Energy Research.

This work comprised the initial part of an Exploratory Survey aimed at

clarifying principal issues concerning the newly integrated program of

Energy Research within the Bureau of Mines' research organization. Such

exploration and clarification necessarily precede the formulation of

possible means for improvement. Specifically, the task involved the

identification of actual problems associated with implementing missionoriented research projects within the context of the existing six Energy

Research Centers and Laboratories operated by the Bureau of Mines. To

date, only two of tlie six Research Centers have been visited.

To perform this survey, a joint task group x^^as organized by the

Technical Analysis Division of NBS and requested to make an objective

examination of the functional operating activities and principal information flows at three decision levels within the Bureau of Mines Energy

Research organization. The Task Group (called "Project SOAP"*) conducted

^Project SOAP: A systems analysis task group identified by the acronym

for ^Systems/Operation Analysis of programs. The term "systems analysis"

refers to a method for determining information relevant to managing an organized activity, for the purpose of improving its decision -making process in a more rational way. In the context of this assignment, it is

intended to be used as a prerequisite for inplementing the discipline of

Program Management. i
a series of 14 group interviews in April and May 1973 during visits

to the Washington Headquarters and to two of the six field locations, the

Morgantown Energy Research Center (MERC) in West Virginia and the Bartlesville Energy Research Center (BERC) in Oklahoma.

This status report summarizes the work activities of the Project

SOAP task group from March 26th, through June 30, 1973, as required by the

terms of the Interagency Working Fund Agreement*. It describes the type

of approach employed and identifies the progress made, to date, as well as

some of the difficulties encountered during this attempt to improve the

general understanding of the decision-making process for managing the Energy

Research activities of the Bureau of Mines.

The details of the initial part of this Exploratory Survey (covering

field visits to the Washington Headquarters, MERC and BERC only) are

presented in Section 2 of this report. Preliminary findings from the two

visits to MERC (during April 1973) are presented in Section 3 and were

reported at a briefing to the Washington Staff and Research Directors,

Energy Research Group, at their annual meeting in Washington, May 1, 1973.

Also, an informal briefing following the field trip to BERC was given to the

Project Officer and the Assistant Director -Energy on May 30, 1973.

In order of presentation, this report first outlines the background

for this assignment and identifies its scope and objectives; then, in the

main body, it briefly describes the two field visits (to MERC and BERC);

*In accordance with provisions of Public Law 89-672 (80 Stat, 951);

Public Law 92-369 (86-Stat. 514); and Section 601 of the Federal

Economy Act, APPVD June 30, 1932, as amended (31 USC 686).

2
and, lastly, in Section 5, it summarizes the principal findings and

conclusions to date. No formal analyses or recommendations are included

in this first status report. The ^pendix contains material referenced

in the main text.

3
1.1 Background Perspective

In his first two Energy Messages to Congress,* the President established

guidelines for the development by the Federal government of a National Energy

Policy and Program to assure the country has a sufficient supply of

energy at reasonable costs, while providing adequate incentives for

Industry to sustain a healthy economic growth and improve the quality of

national life.

To implement this Policy, the Secretary of Interior accepted the

responsibility for establishing an Energy and Minerals Mission, and

charged the U.S. Bureau of Mines (USBM) with a portion of the Interior

Department's mission pertaining to Energy Research.

The Energy Research Mission of the U.S. Bureau of Mines was recently

formulated with the following primary goal:**

To develop effective means that will increase the availability

and improve the utilization of the Nation's energy resources at

reasonable costs and in a manner that will help attain desirable

social and environmental improvements.

To meet this important challenge, the existing organizational resources

of USBM are to be combined into an integrated, mission-oriented Energy

Research Program. Currently, specific areas of commodity research

are conducted separately, and somewhat independently, at each of six

*March 10, 1972, and April 18, 1973.

*USBM Energy Research Program Mission Statement, l^y 25, 1972.

4
principal USBM Research Centers and Laboratories* employing about 900

people across the United States.

From the Bureau of Mines' Budget of FY 73, almost $20 million was

spent on research and development allocated to the four separate commodity

research areas: coal, petroleum, natural gas, and oil shale. In order

to respond more effectively to the R§D priorities in the President's

Energy Messages, the USBM has recently initiated a new concept of missionoriented program planning and evaluation. The primary aim of instituting

such explicit management methods for its research activities is to assist

the six Researcli Center Directors in developing a more rational and

coordinated approach for allocating research resources toward recognized

national Energy goals, and to provide a consistent way of monitoring

progress toward those goals.

To help organize the major research activities (and their relative

priorities) to be undertaken by the various Centers and Laboratories,

an integrated Energy Research Mission Statement was prepared in May of

1972. It was hoped that this "Mission Statement" would provide tiie basis

for planning and evaluating the various USBM research projects in terms

of the "Energy Mission" objectives. This document, while useful at the

Washington Headquarters level, was not immediately accepted at the field

level because of the precedent**it established. (It has since been thoroughly

reviewed, however, at both the Washington and field levels, and is expected

to be released shortly.)

*Morgantown Energy Research Center (MERC), Morgantown, West Virginia;

Pittsburgh Energy Research Center (PERC) , Pittsburgh, Pennsylvania;

Bartlesville Energy Research Center (BERC) , Bartlesville, Oklahoma;

Larmie Energy Research Center (LERC) , Laramie, Wyoming; San Francisco

Energy Research Laboratory, San Francisco, California; Grand Forks

Energy Research Laboratory, Grand Forks, North Dakota.

^*That of an integrated research, program with its strategic technological

goals detemined by the Washington Headquarters.

5
From this prolonged review of the Energy Research Mission Statement

document, it appeared there were "communications problems" between the

Headquarters Staff and the Researchers in the Field concerning Research

priorities. In addition, a mechanisr.. for implementing the new management

concepts presented in this Statement was needed.

In February 1973, through the recommendation of Mr. Harry R. Johnson,

Assistant to the Assistant Director -Energy (by way of Mr. Leo A. Schrider

[^'1ERC] and Dr. Jack Byrd, Professor of Industrial Engineering at West

Virginia University) , a report entitled "Toward A Discipline for Program

Management (A Summary of Project SOAP Activities)" came to the attention

of the Bureau of Mines. This report, presented at the 1973 TIMS/ORSA Meeting*

in Atlantic City on November 7, 1972, was the result of a two-year project

conducted by a task group from the Technical Analysis Division of the

National Bureau of Standards. It describes an application of Management

Science by a systems analysis task group to resource allocation problems

within a sister Federal research agency- -the National Institute of Allergy

and Infectious Diseases (NIAID) of the National Institutes of Health. Johnson

and Schrider were of the opinion that the methods used by the SOAP team within

NIAID might be applicable to certain Program Management problems withiii the

USBM Energy groups.

Accordingly, in early fferch 1973, preliminary discussions were held

between members of the Bureau of Mines and the National Bureau of Standards

.

On March 26, 1973, Interagency Agreement WFA-SO 133054 was executed

authorizing the initial part of this four-phased assignment.

Joint Meeting of the Operations Research Society of America and The

Institute of Management Sciences, Atlantic City, N.J., November 7-9, 1972.

6
The following is a detailed description of the Task Group's

objectives, its approach, and findings during the first part of its first

pliase--the Exploratory Survey,

1.2 Scope of Assignment

Tliis Interagency Agreement between the U.S. Bureau of Mines, Department of Interior, and the National Bureau of Standards, Department of

Commerce, was undertaken to assist the Bureau of Mines Energy Group develop

new concepts of management applicable to tlieir program of Energy Research.

(The complete Assignment consists of four piiases: Phase I - Exploratory;

Phase II - Analysis; Phase III - Prescriptive; and Phase IV - Implementation.)

The primary purpose of the initial Phase I Exploratory effort was

to seek the clarification of principal issues associated with the implementation of mission-oriented Program Management concepts within the USBM

Energy Group.

Tlie scope of the agreement was limited to only the first part of

tliat phase: to visits to the Washington Headquarters and to two'^

operational Field Centers (Morgantown, West Virginia, and Bartlesville,

Oklalioma) , Emphasis was directed toward investigating tlie potential

usefulness of an explicit mechanism for evaluating research projects.

It was also expected that this survey would provide the organization with

the broader perspective required to coordinate some of the Bureau of

Mines' other Program Management activities.

7
j

1.2.1 Objectives

Specifically, the joint ta^k group* was organized to perform the

following tasks:

• To improve the general understanding of* the External/ Internal and

Institutional context within which the decision-making

process operates.

• To clarify principal issues and identify "tractable

problems" associated with the implementation of missionoriented R§D projects. '^^

'

• To define a mechanism for evaluating "Quality" and

"Relevance" of research projects in terms of stated

program goals. - •

• ' ' '

1.2.2 Approach '

. ^ r .s • - ;

The method the SOAP team used was the surveying of various peer groups

within each location- -at ^4BRC, BERC, and the Washington Headquarters --to

determine the existing functional information flows, feedbacks, and

decision points in order to identify the nature and extent (and precise

location) of significant problems amenable to solution by improved management methods. Tlie mode of operation was the face-to-face, open-ended group

interview technique, with the above three-point agenda used for focusing

discussion on the objectives of the task group.

*The Project SOAP task group members included: Robert S. Cutler,

Vincent A. Martino, and Dr. Mary Ellen Golby from TAD/NBS, and

Dr. Jerry D, Ham and Harry R. Johnson (Project Officer) from USBM.

(At MERC, the group was assisted by Leo A. Schrider, and at BERC,

by William D. Howell.)

8
1.2.3 Preliminary Results

The results anticipated from this survey included a briefing delivered

on May 1 and this report, describing:

• Details of work performed, to date, under the Interagency

Agreement

• Listing of acknowledged "Problem Areas" where reasonable

improvements are considered possible.

• A recommendation concerning whether the use of quantitative

methods for implementing new concepts of program management

within USBM Energy Research Centers and Laboratories is

operationally feasible.

To date, the Project SOAP task group has followed an orderly procedure

for establishing a firm factual basis concerning the nature of actual

"problem areas" identified at each operating location visited. The preliminary results obtained thus far include statements of acknowledged

"problem areas" considered amenable to solution by available management

methods. These results are contained in Section 3 of this report.

It would be unreasonable to expect that any task group could attain

such insight into the myriad problems of a complex organization after just

two site visits as to identify any single problem as the crux of the

matter and recommend how it should be solved. Only after completing its

exploratory work, which assures that the factual foundation is properly

laid, can the SOAP team engage in a more compreKenfive analysis of the

principal problems associated with this assignment- -the development of a

new concept of management applicable to the entire USBM Energy Research

Program

9
From its findings thus far, from the 14 interviews with groups at

three locations within the USBM Energy Research organization and from the

preliminary results of Exercise I*, the Project SOAP task group concludes

that it is worthwhile to proceed with the additional work needed to

implement an operational "peer-review" mechanism for evaluating individual

research projects in terms of program goals.

*A preliminary assessment of a sample set of Research Projects involving

56 individuals in six groups at three locations (WASH/MERC/ BERC) during

May 1973. See Subsections 2.4 and 3.3 for details.

10
2. PHASE I: EXPLORATORY SURVEY

The following details the activities performed and the observations

made by the Project SOAP team during the initial part of this Exploratory

Survey.

2.0.1 Project SOAP Orientation

A preliminary meeting was held March 7, 1973, at the National Bureau

of Standards between three members of the U. S. Bureau of Mines Program of

Energy Research and five members of the Technical Analysis Division to

discuss more fully the possibility of TAD's people doing a systems analysis

study similar to the one previously conducted by TAD for one of the

National Institutes of Health. It was to be, essentially, a study of

Program Management conditions within the Bureau of Mines' Energy Research

Centers

.

After a lengthy discussion devoted to the managerial structure

of the USBM Energy Program, the USBM people stressed that the main problem

of their Program, as they understood it, was one of communication and the

flow of technical ideas and information- -of vertical communication between

Headquarters and the Field, and of horizontal communication between

various levels of one Research Center and those of the other Centers.

Somewhere- -they didn't know precisely where- -there were barriers to this

vital communications flow.

From USBM: Harry R. Johnson, Dr. Andrew W. Decora, Leo A. Schrider;

from TAD: Louis C. Santone, Robert S. Cutler, Vincent A. Martino,

David M. Clancy, Dr. Mary Ellen Colby.

**Cutler, Robert S., "Project SOAP: A^^ystem,s Approach to Biomedical Research

Program Management -A Case Study," NBS Technical Note 761, CApril 197^).

11
The TAD people noted that, in addition to the "communications" problem,

there also seemed to be a "control" problem and that, before considering

any new management concepts, they ought to understand the nature of the

current resource allocation process used by USBM throughout its Energy

Research Program. However, as was clearly pointed out, there was too

little time to deal effectively with the whole organization. It was finally

agreed that an initial survey covering operations at the Washington

Headquarters and at one or two Field Centers would constitute the initial

part of this Exploratory Phase.

The next step recommended by the TAD/ SOAP team members was that they

be permitted to meet as many of the USBM people knowledgeable about

USBM Energy Research as possible before visiting the Field Stations, because

the team operated then, as always, upon the premise that problem- finding

must necessarily precede problem- solving. Dr. Jerry Ham in the Arlington

office and Mr. Harry Johnson in the Washington office volunteered to help

set up meetings with people they felt the SOAP team should interview,

including Mr. William L. Crentz, the Assistant Director of the USBM program

of Energy Research; Dr. G. Alex Mills, Chief of the Division of Coal;

J. Wade Watkins, Chief of the Division of Petroleum and Natural Gas; and

Jack Phillips, Chief of Shale Oil, and their immediate Staffs.

2.0.2 The Initial Task . ..... ,

Initially there was an "identity problem" to solve. Because the

aim of this assignment and the objectives of this survey were not clearly

understood by everyone in advance, the USBM Energy Program personnel

felt uncertain about what to expect in terms of project results. Part of this

*See letter, Santone, L. C. to Crentz, W. L., dated March 9, 1973 in

Appendix, pp. A- 2.

12
difficulty lay in their skepticism about the usefulness of system analysis

techniques. Similarly, the systems analysis people from TAD/NBS recognized

their lack of familiarity with the technology of Energy Research. Thus,

there were mutually recognized difficulties in achieving desired

results within the allotted time. Fortunately, however, there was also

a sincere resolve on both sides to make a meaningful and productive start.

Dr. Jerry D. Ham, Petroleum Engineer, USBM Washington Staff, was assigned

part-time to the Project SOAP team to act as the liaison with the operating

units of the USBM Energy Group.

2.1 Washington Staff Interviews

A series of meetings was scheduled with members of the Washington

Staff of the USBM Energy Research group to acquaint them with the purpose

of Project SOAP and gain their guidance on things to look for during the

field trips. In these (as in later interviews) the SOAP team was cordially

received, even when healthy skepticism was voiced about what the team was

supposed to do.

Following is a list of the initial Washington Staff meetings held

during this time:

j'^ril 2. Orientation Meeting with Dr. Jerry Ham (USBM/Arlington)

,

at NBS.

April 3. Initial Meeting with Mr. Crentz, Asst. Director-Energy

(USBM), in his office, in Washington, D.C.

April 3. Orientation Meeting with Messrs. Watkins, Phillips,

Gooding, Ramsey, and Dr. Ham, in their office at

Arlington, Va.

13
April 4. Orientation Meetings with Dr. Mills and with Mr. Corey

, (USBM) in their offices, in Washington, D.C.

As a result of these initial discussions, the SOAP team gathered the

following information and impressions to guide its further investigations:

(1) There is no mission "indoctrination" for new researchers

• at the various Centers, either initially or subsequently.

C2} Management pressures (e.g., influence of labor unions) are

extremely important at some Centers, almost non-existent at others

(3) There tends to be too much parochialism (single -commodity

interest) at all of the Centers.

(4) The Research Supervisors appear to be the most inflexible

link in the Centers' managerial chain- of- command.

(5) The general career history of the Centers' research people

is one of migration from universities into business and then

to government.

(6) The presence of a university near a Research Center tends to

intellectually stimulate the atmosphere of the Center.

(7) Washington tends to lose sight of changing technologies,

• while people at the Centers do not.

(8) Washington feels the need for a study of the "relevance" of

the research conducted at the USBM Energy Research Centers.

(9) Researchers in the Field tend to have no concept of completion

time- frame.

(10) If the new Mid-Year Review is to be useful to people other than

just the Asst. Director-Energy, review comments should always be

followed by a feedback to the field ^s was recently done for

FY73)

.
(11) The Bureau of Mines seems to have more difficulty selling

its programs to the Department of Interior and 0MB

than to Congress.

(12) Many line-decisions (directives) formerly made in the Field

are now made by the Washington Staff.

(13) There is a mutual Headquarters/Field need for understanding

the functional decision-making processes at the Centers.

(14) There is something believed wrong with the Bureau of Mines'

present resource allocation methods.

(15) Something is needed to improve the effectiveness of the "New

Idea Referral System."

It was also suggested, later, by Mr. Crentz that the SOAP team

visit not MERC and PERC, as originally planned (they had the virtue of

proximity), but MERC and BERC,* by way of contrast. Located in different

areas of the country- -West Virginia and Oklahoma- -they had unique

environments and might perhaps exhibit contrasting problems.

2.2 Morgantown Energy Research Center Interviews

Before the SOAP team arrived at the Morgantown Center they were

aware of the existence of certain stresses there. For one thing,

there had been a Reduction in Force (RIF) the last fiscal year and the

prospect of another one during the next fiscal year, a situation

which did not help morale. For another, there was an organized Labor

Union Negotiation pending at the time. Then, too,, the Center's former

For acronyms, see p. 5.

15
Director had recently died and his successor, Mr. William C. Eckard,

had barely arrived. Lastly, there was the imminent possibility that

the proposed FY74 budget of $3.5 million was going to be cut to $2.7

million. .

Fortunately, however, the new Center Director had taken positive

steps to understand the precise nature o£ the problems which currently

existed at MERC. He had created several internal task forces from among

his people, thirteen of whom had recently participated in a Leadership

Systems Workshop.* Because of these internal task force efforts, the

SOAP team had an easier time identifying significant "problem areas"

than it might otherwise have had. For example, one MERC task force had

concluded that employees have "great difficulty. .. in getting a fair

and reasonable hearing for their ideas on possible new projects that fit...

[their] mission." They stated that the "idea referral system is pitiful"

and that the "project proposal system is fine in theory, but poor in

practice. The major problem is not the mechanics of these systems, but

rather [the] failure of the supervisory chain to nurture the system and

make it work." They complained that "little information goes below the

Research Supervisor level," including mission statements and budget information, and they called for more visits from the Washington Staff to MERC.***

^^ 1^ ^ All seemed to agree that "communications was a part of all the problems,"

_^*Sponsored by the Bureau of Mines.

"Report of Task Force to Consider Recommendation to Maximize Productivity

and Job Satisfaction in MRED and MERC," February 14, 1973, p. 2.

***Ibid., p. 3.

 "Report of Task Force on Communication and Organization," March 19, 1973,

Introduction.

16
though they believed (contrary to what SOAP was to learn that Washington

Headquarters actually wants) that the job of the Research Supervisor was

to "sell the Washington staff on... [their] i^e.as for new or continuing

programs." They called for two Mission Statements, one from Washington

and one from MERC, wherein guidelines "should be developed to outline areas

in which project proposals are described within the framework of both

mission statements." They also called for the selection of Project

Managers from within the Center,** * in cases where extremely important projects were involved.

It was evident from all this that MERC not only had serious problems

but was keenly aware of them, ivforeover, it was attempting, largely as a

result of the stimulus of their new Director, to do something about them.

2.2.1 MERC Visit #1 (April 10-11, 1975)

The first group interviewed by the Project SOAP team the afternoon

of April 10 was comprised of the Research Center Director, the Deputy

Director, and the Center's five Research Supervisors. These men, in turn,

were later asked to select three men from each of their sections to attend

a similar meeting to be held the following day. Mr. Leo A. Schrider,

Petroleum Engineer, was asked to assist the SOAP team by providing

necessary support to the group while at MERC.

In an attempt to make proper introductions and to dispel suspicion

that the Project SOAP team had been sent down by top management in

Washington to conduct a "witch-hunt," the Director, Mr. William E. Eckard,

* Ibid. , p. 3.

Ibid. , p. 4.

Ibid. , p. 6. ****Robert S. Cutler, Vincent A. Martino, David M. Clancy, Dr. Mary Ellen Colby

(all of TAD) , and Dr. Jerry Ham (USBM, Arlington)

.

17
introduced the individual members o£ the SOAP team before each meeting.

He explained their mission and how they came to be involved in it, and

then he discreetly disappeared (except for the meeting with the Research

Supervisors) in order to promote complete frankness on the part of the others.

The team learned that there was a general feeling within the Center that

nobody up there (in Washington) cared about them- -a feeling the team

attempted to probe, pointing out that their mere presence there to help

them identify "problem areas" was proof that "someone up there" did, indeed,

care.

Early in the course of these meetings, and quite by accident, the

term "Research Monitor" came up. When the men at MERC were asked to

explain who this person is and what he was supposed to do (i.e., what

his functions was), they were unable to be precise.

It was then that Dr. Ham mentioned that he was a Research Monitor;

but he, too, could not give a definitive answer on the subject. (Nor

is this surprising, because, as will be seen, there is great confusion among

the Washington Staff- -among the Research Monitors themselves- -about the

exact function of this person.) The SOAP team then concluded that one

of the first things it would have to do once back in Washington was

find out more about the role of the Research Monitor. He loomed too large

on the horizon to be overlooked; moreover, the uncertainty as to his

function seemed symptomatic of the general lack of communications

between the Field and Headquarters .

' —

18
Back in Washington, the team reviewed their notes and impressions

from the two previous days of meetings and developed a tentative listing

of "identified problem areas" (shown below) believed amenable to solution.

The next step was to report to the Assistant Director what they had

learned, at MERC, about the confusion over the functions of the

"Research Nfonitor." They then asked for permission to schedule a return

trip to Morgantown in order to ascertain whether the identified problems

would in fact be acknowledged as real and salient by the participants in

the initial discussions at MERC.

The following lists the findings from this first visit:

"Problem Areas" Identified frra Init-al Visit To >Torgant(^>-n

Enerp' Rcsearcli Center, .\pril lQ-11, 1971

1. Need for r.xplicit Station Mission Statcmerit.

• Translation of Energy Research Progran "Goals"

into Research Project "Cbjectives."

• Participation in Project -Planning Process.

• Results -Oriented Task Group Approach.

2. Clarif)- Research Nbnitor Function.

• Explicit Role in "Cammicat ions" Process.

• Strategic-Level Plaming and Assessments.

• Program Planning/Evaluation Staff.

3. Inproved Mechanism for Evaluating Projects in Terms of

Stated Progi-an Coals

.

I Project Priority Determination.

• "Quality" (Teer-F^view Judgements)

.

• "Relevance" (Inten.al/Extemal Opinions)

.

» Time-Frame for P^sults.

• Costs O-Kalti-Year , Total Investment).

4. Iniproved Organizational "Feedback."

• Intra-Communications 0^'ithin Center) .

• Inter-Ccnnuriications CBetween Centers, HQTRS/Field)

.

• Participation in Decision Process.

5. Idea Development/Project Proposal System.

• Stirulate "Good" P^search Projects.

• Assure Useful Peseardi Results.

• Consistent Criteria.

19
On April 16 and 18, before returning to MERC, the SOAP team met

all of the ten Washington Research Monitors who were in town and who did

not have prior commitments. The gist of what they learned from these

meetings is that there was as much confusion among the Research Monitors

themselves concerning their role as there was among the men at MERC.

While the men at Morgantovni complained of the lack of efficacious

monitoring of programs by Washington and called for more on-site

visits by Monitors (hopefully, once a month), SOAP received the following

diversity of comments from the Research Monitors themselves: (a) that

there really was no such a person as a Research Monitor; (b) that there was,

indeed, such a person, and he was very valuable to the decision-making process;

(c) that the Monitors, who are scientific experts, make no scientific

judgements on the scientific proposals that are sent to them; (d) that the

present Research Monitor function has replaced the former Program Manager

function- -and the latter never really got off the ground, anyway; (e) that

one of the Research Monitor's functions, in addition to keeping tabs on

the research going on in the Field, is to keep abreast of the state-ofthe-arts in industry and elsewhere; (f) that all of the Washington Staff

serve as Research Monitors, whether or not officially designated as such,

and that frequently one substitutes for another; (g) that the actual

function of the Monitor is to make recommendations to the Assistant DirectorEnergy; (h) that Research Monitors and Research Coordinators are one and

the same; (i) that at least one Monitor was given a list of seven projects

he had never seen before and was asked to give his opinion on them within 48

hours.

Dr. Jerry Ham, Richard Gooding, Larry Burman, Jerry Ramsey, Richard Corey,

Jack Smith, and Brian Harney.

20
2.2.2 MERC Visit #2 (April 25-26, 1975)

On April 25, two of the SOAP team members* returned to the Morgantown

Energy Research Center. The visit coincided with the thirty- first meeting

of the Petroleum Industry Technical Advisory Committee to the Morgantom

Energy Research Center on Fluid Energy Resources. The SOAP team was

invited to attend as observers. Being somewhat ignorant about technical

matters up to now, the NBS members were enlightened by the proposals

for GAS DEVEL, Underground Coal Gasification and reports of Hydraulic

Fracturing and Microseismic Monitoring.

The second day of this visit was devoted to scheduled meetings with

the same four groups of the MERC personnel the team had met with on

April 10-11. After further discussion and feedback of what the team

had learned in Washington, they asked each of the four groups to

re-rank, in order of their concept of priority, the list of the five

identified "problem areas" the team had gleaned from its initial visit.

The results of these rankings are show_ on p. 31, Section 3, of this

report. The visit concluded with a brief informal report to the Center

Director, Mr. William E. Eckard.

Robert S. Cutler and Dr. Mary Ellen Colby. (Harry R. Johnson, from

the USBM Washington office, was also present for the conference.)

21
2.3 Bartlesville Energy Research Center Interviews

Prior to its visit to the Bartlesville Energy Research Center,

the SOAP team knew little about either the Center or the town. It had,

however, been informed that Bartlesville was an "oil" community (''It's a

one-company town," the team was told); that there was no labor union

at the Center; that the University closest to the Center, Tulsa, was

sixty miles away; that one of the best Thermodynamics groups in the country

was employed by the Center.

2.3.1 BERC Visit #1 (May 21-22, 1973)

On Monday, May 21, 1973, two SOP^ team members* met with

Mr. John S, Ball, the Research Director of BERC, for lunch. During this

informal gathering, the team briefed him on what it hoped to accomplish

while at Bartlesville, and he, in turn, briefed the team on the situation

at BERC, as he saw it.

The SOAP team had decided well in advance that they would follow

essentially the same procedure for interviews that they had followed at

MERC. They requested the Center Director schedule his Research Supervisors

for the first meeting with the SOAP team and that the Supervisors, in turn,

nominate additional men for a series of three meetings the following day.

All these requests were granted.

That afternoon, from 1:00-2:30 P.M., the SOAP team met with the

Director, four Research Supervisors, the Superintendent, and the appointed

liaison man, William D. Howell (who attended all the meetings)

.

^Robert S. Cutler and Vincent A. Martino, of TAD.

22
This session was particularly interesting in one respect. Although

the Director introduced the SOAP team by way of remarking that they had

not been sent down by Headquarters on a subversive mission but had, rather,

come to identify and help solve problems, there existed an unmistakable

air of skepticism. Fully two-thirds of this meeting was spent by the

SOAP team in justifying both its credentials and the reason for its

mission. One Si^ervisor bluntly challenged the assumption that there

were any "problems" at BERC*

However, in due time, these "problems," voiced as complaints, began

to surface (and were duly noted by the SOAP team) . At the conclusion of

the meeting, the Staff agreed to select three men each from their groups

to attend the next day's sessions.

The three groups met with the SOAP team on May 22. Again, the

Director, John S. Ball, introduced the SOAP team members to all three groups

and then left, in an attempt to assure frankness on the part of those

participating. And the men were frank. That night, the SOAP team drew

up the following list of "problem areas" they had identified as a result

of these four meetings.

*This remark was all the more interesting in view of the following: As

at MERC, BERC had its own Leadership Task Force on improving productivity

and job satisfaction. While the group found that, on the plus side,

salaries, facilities, and si^port functions were good at Bartlesville,

they called for more "active, attentive management at the Washington level";

the creation of a staff to serve as a liaison between Washington and the

field, spending much of its time in the field; funding of only that research

which the Bureau can best perform; better communication, both vertical and

lateral; and "recognition of work by other peer groups." See Memorandum on "Evaluation of Bureau of Mines Management Operation," Leadership Systems, Inc.

Task Force, Bartlesville Energy Center, March 7-12, 1973.

23
"Problem Areas" Identified From Initial Visit To

Bartlesville Energy Research Center, May 21-23, 1975

1. Feelings of Frustration Concerning Inadequate Funding Supoort for BERC

Research Capabilities in Face of Current National Energy Crisis

• Lack Confidence in V/ashington Salesmanship

e Need for Review of Policy on Outside Agency Funding

e Clarify Budgetary Decision Process

2. Recognition of "Communication Gap" Between Washington Headquarters and

Field Stations

o Clarification of Research Direction - Mission Statement

• Improved Program Planning/Evaluation Staff Capability

« Assignment of Soecific Energy Research Program "Requirements" (and

Priorities) From Headquarters to Centers

3. Clarification of Headquarters Energy Division Staff Function

« Research Monitor/Coordination

© Headquarters- Level Planning and Evaluation

o Liaison with Technical/industry Advisory Groups

4. Improved Project Management Review Mechanism

A Determination cf "Technical Quality" end "Program Relevance"

of Research Activities • Establish Center Priorities -

0 Provide Basis for Program Evaluation

5. New Idea Development/Profect Proposal System

• Present Procedure Too Complicated,, ineffective

6 Lack of Incentive (Funding) to Stimulate innovation

• Require Meaningful Feedback inilLn.^^f^^

RSC/YAM:V5C

23 MAy 73

24
Because they had only the morning of May 23rd. for quick wrap-iq)

sessions, the SOAP team alloted approximately one-half hour to each of

four meetings --with the Research Siqjervisors and the same three groups

of Project Leaders interviewed the preceding day. They asked the Research

Director and the Research Supervisors to review and re -rank the list

of five identified "problem areas" and then return the revised listing to

TAD. A letter* from John S. Ball, Research Director of BERC, describes

the results of the process. Instead of having just the Research Supervisors

participate in Exercise I,** as originally planned, the team requested the

participation of all four groups, with the Research Supervisors and one

group of Project Leaders judging "program relevance" only*** and the two

other groups of Project Leaders judging "technical quality" only. The

men were asked to mail their tally sheets directly to TAD/MBS by June 1.

The next step for the SOAP team was an informal feedback report

to the Washington Staff about their first visit to BERC. On May 30, 1973,

Mr. Crentz, Harry Johnson, Dr. Ham and three members of the SOAP team****

held a long meeting in Mr. Crentz 's office. Because the primary aim of this

session was the recapitulation of the team's visit to Bartlesville, the

team gave a detailed account of their stay at BERC, including a contretemps

between one of their members and one of BERC's Project Leaders, with apologies from the involved team member.

*See letter, John S. Ball, Research Director, to Robert S. Cutler,

Project SOAP, dated June 1, 1973 , in Appendix, pp. A- 4 thru A

**To be explained more fully in the following pages.

***Because certain persons in one group were not qualified to judge on

technical matters, they and the rest of their group were asked to judge

on the basis of their concept of "program relevance".

****Robert S. Cutler, Vincent A. Martino, Dr. Mary Ellen Golby, all of TAD.

25
While discussing the submitted list of "problem areas" identified

during this first visit, Mr. Crentz made several observations, am.ong

them these three, which are very pertinent to SOAP's assessments of this

Center's relationship to its Washington Headquarters:

• Bartlesville Energy Research Center is unique in its funding.

It receives approximately 50% of its funding from agencies

other than DOT --from private and other Government agencies,

as well as from the state of Oklahoma. Recently, there has

been a steady decrease in this outside funding, and BERC

has looked to USBM to fill its diminishing coffer. This the

USBM has been unable to do, simply because there are no dollars

to be had.

• To the charge of one man at BERC- -that one of the reasons

BERC was suffering from insufficient funding was that petroleum research did not have the powerful lobby on the Hill

that coal research had- -Mr. Crentz responded negatively. He

pointed out that the petroleum industry has no difficulty in

making its voice heard in Washington, but differs considerably

from the coal industry in its attitude toward and reliance on

government research. Whereas the coal industry conducts

virtually no research of its own (and is, therefore, always

delighted when the government offers to do so) , petroleum has

a multi -million dollar proprietary research program and tends not

to seek USBM research projects precisely because they are not proprietary and tne results are made public for anyone to

exploit.

• The most fruitful way for a researcher to promulgate a new

idea is to get the Research Director behind it. If he confirms

it is "good" and advances the idea to Washington for funding,

Mr. Crentz believes, "in almost no cases at all," would he run

into a stumbling block. The logic here is simply that if the

new idea is actually "good" it should surely be sponsored by

the Research Supervisor or the Research Director, and it will

therefore eventually receive funding and won't be the victim

of the funding of on-going projects. In fact, if it is

sufficiently good, on-going projects may be its victim. (These

remarks were, in effects ^ reiteration of Mr. Crentz 's statement to the Mid-Year Review Conference for May 1, 1973, that

he wanted two sources of information upon which to base his

budgetary decisions: the Research Director and the Division

Chiefs.

)

26
2.4 Exercise I: Preliminary Assessment of Research Project Proposals

Following its May 1st. briefing to the USBM Research Directors, who

were meeting in Washington for a Mid-Year Review, the SOAP team was urged

to conduct a test of a management tool for evaluating projects in terms

of stated program goals. What was desired was a means for ascertaining:

(a) how consistently various people associated with the USBM Program of

Energy Research would rate on-going Project Proposals, and (b) what their

rationale was.

A preliminary Exercise was then designed to determine the feasibility

of obtaining a meaningful assessment of the "quality" and "relevance" of

individual research projects (and their criteria), by requesting selected

review groups to evaluate a representative sample of research Project

Proposals submitted to the U.S. Bureau of Mines for FY- 74 funding.

Two separate assessment scores were sought: one for "technical

quality," and the other for "program relevance." These scores would later*

be used as a basis for comparing the resulting rank order for the

sample set of projects with the actual experience of the organization in

establishing priorities for the funding of research projects.

Specifically, the objectives of this Preliminary Exercise were:

• To determine the feasibility and acceptability of evaluating

Energy Research Project Proposals separately, for "technical

quality" and "program relevance."

• To analyze the scoring patterns of specific review groups

in determining the "relevance" of particular Energy Research

projects to the new Energy Research mission of the U.S.

Bureau of Mines.

• To identify and explicate the criteria used in assessing the

"quality" and "relevance" of the sample set of projects.

'^At a later stage in the work.

27
Using a modification of the traditional "peer review" procedure,

which calls for the informed judgment of competent individuals tp evaluate

the "scientific merit" of particular research Project Proposals, the SOAP

team sought also to emphasize in this Exercise the concept of Program

Evaluation. They did this by including separate assessments for "technical quality" and "program relevance" of the individual projects in

order to help the U.S. Bureau of Mines determine fufiding priorities for

its Energy Research.

• For this Exercise*, six review groups, or "juries," consisting

each of three to five members, were asked to rate a representative sample

of research Project Proposals submitted to the Assistant DirectQX-Bnejcgy

,

U.S. Bureau of Mines, for FY-74 funding.

With Dr. Ham's help, the team selected, at random, seventeen typical

on-going (FY '73-74) Project Proposals to be evaluated.

Juries selected were:

Jury • ;- Research Review Groups

I. ..V;.. USBM Washington Staff

II. USBM Washington Staff

III. .. . WASH/MERC/ BERC Staff

IV. MERC Research Supervisors

V. ., .. ^ BERC Research Supervisors

VI . Industry Technical Advisory Panel

On May 16, 1973, three of the SOAP team members** met with the

Project Officer, Harry R. Johnson,*** in his office to present their plan

for conducting Exercise I and to gain his approval for administering it.

He was of the opinion that the idea was "great," so the team made arrangements for the USBM Washington Staff to meet the following day at Ballston

Towers, in Arlington, Virginia, in order for them to participate.

-See Project SOAP - Exercise I Memorandum, May 15, 1973, Appendix B-1 Thru B-4.

' both^of^TAc!^

Arlington, and Robert S. Cutler and Dr. Mary Ellen Golby,

•*Leo Schrider was also present that day.

28
2.4 Exercise I: Preliminary Assessment of Research Project Proposals

Following its May 1st- briefing to the USBM Research Directors, who

were meeting in Washington for a Mid-Year Review, the SOAP team was urged

to conduct a test of a management tool for evaluating projects in terms

of stated program goals. What was desired was a means for ascertaining:

(a) how consistently various people associated with the USBM Program of

Energy Research would rate on-going Project Proposals, and (b) what their

rationale was.

A preliminary Exercise was then designed to determine the feasibility

of obtaining a meaningful assessment of the "quality" and "relevance" of

individual research projects (and their criteria), by requesting selected

review groups to evaluate a representative sample of research Project

Proposals submitted to the U.S. Bureau of Mines for FY- 74 funding.

Two separate assessment scores were sought: one for "technical

quality," and the other for "program relevance." These scores would later*

be used as a basis for comparing the resulting rank order for the

sample set of projects with the actual experience of the organization in

establishing priorities for the funding of research projects.

Specifically, the objectives of this Preliminary Exercise were:

• To determine the feasibility and acceptability of evaluating

Energy Research Project Proposals separately, for "technical

quality" and "program relevance."

• To analyze the scoring patterns of specific review groups

in determining the "rele-vance" of particular Energy Research

projects to the new Energy Research mission of the U.S.

Bureau of Mines.

• To identify and explicate the criteria used in assessing the

"quality" and "relevance" of the sample set of projects.

*At a later stage in the work.

27
Using a modification of the traditional "peer review" procedure,

which calls for the informed judgment of con^etent individuals to evaluate

the "scientific merit" of particular research Project Proposals, the SOAP

team sought also to emphasize in this Exercise the concept of Prpgram

Evaluation. They did this by including separate assessments for "technical quality" and "program relevance" of the individual projects in

order to help the U.S. Bureau of Mines determine funding priorities for

its Energy Research. '

For this Exercise*, six review groups, or "juries," consisting

each of three to five members, were asked to rate a representative sample

of research Project Proposals submitted to the Assistant Director -Energy,

U.S. Bureau of Mines, for FY -74 funding.

With Dr. Ham's help, the team selected, at random, seventeen typical

on-going (FY '73-74) Project Proposals to be evaluated.

Juries selected were:

Jury Research Review Groups

I. USBM Washington Staff

II. USBM Washington Staff

III. r WASH/MERC/BERC Staff

IV. MERC Research Supervisors

V. BERC Research Supervisors

VI . Industry Technical Advisory Pai>el

On May 16, 1973, three of the SOAP team members** met with the

Project Officer, Harry R. Johnson,*** in his office to present their plan

for conducting Exercise I and to gain his approval for administering it.

He was of the opinion that the idea was "great," so the team made arrangements for the USBM Washington Staff to meet the following day at Ballston

Towers, in Arlington, Virginia, in order for them to participate.

-See Project SOAP - Exercise I Memorandum, May 15, 1973, Appendix B-1. Thru B-4.

both^of^TAD!TM

Arlington, and Robert S. Cutler and Dr. Mary Ellen Colby,

***Leo Schrider was also present that day.

28

*
2.4 Exercise I: Preliminary Assessment of Research Project Proposals

Following its May 1st. briefing to the USBM Research Directors, who

were meeting in Washington for a Mid-Year Review, the SOAP team was urged

to conduct a test of a management tool for evaluating projects in terms

of stated program goals. IVhat was desired was a means for ascertaining:

(a) how consistently various people associated with the USBM Program of

Energy Research would rate on-going Project Proposals, and (b) what their

rationale was

A preliminary Exercise was then designed to determine the feasibility

of obtaining a meaningful assessment of the "quality" and "relevance" of

individual research projects (and their criteria), by requesting selected

review groups to evaluate a representative sample of research Project

Proposals submitted to the U.S. Bureau of Mines for FY- 74 funding.

Two separate assessment scores were sought: one for "technical

quality," and the other for "program relevance." These scores would later'

be used as a basis for comparing the resulting rank order for the

sample set of projects with the actual experience of the organization in

establishing priorities for the funding of research projects.

Specifically, the objectives of this Preliminary Exercise were:

• To determine the feasibility and acceptability of evaluating

Energy Research Project Proposals separately, for "technical

quality" and "program relevance."

• To analyze the scoring patterns of specific review groups

in determining the "relevance" of particular Energy Research

projects to the new Energy Research mission of the U.S.

Bureau of Mines.

• To identify and explicate the criteria used in assessing the

"quality" and "relevance" of the sample set of projects.

*At a later stage in the work.

27
Using a modification of the traditional "peer review" procedure,

which calls for the informed judgment of competent individuals to evaluate

the "scientific merit" of particular research Project Proposals, the SOAP

team sought also to emphasize in this Exercise the concept of Program

Evaluation. They did this by including separate assessments for "technical quality" and "program relevance" of the individual projects in

order to help the U.S. Bureau of Mines determine funding priorities for

its Energy Research. . ,

,

' ^ ;

For this Exercise*, six review groups, or "juries," consisting

each of three to five members, were asked to rate a representative sajnple

of research Project Proposals submitted to the Assistant Directpr-Energy,

U.S. Bureau of Mines, for FY-74 funding.

With Dr. Ham's help, the team selected, at random, seventeen typical

on-going (FY '73-74) Project Proposals to be evaluated.

Juries selected were: . :

Jury Research Review Groups

I. .. . . USBM Washington Staff

II. USBM Washington Staff

III. : .. -^r: WASH/MERC/ BERC Staff

IV. MERC Research Supervisors

V. BERC Research Supervisors

VI . Industry Technical Advisory Panel

On May 16, 1973, three of the SOAP team members** met with the

Project Officer, Harry R. Johnson,*** in his office to present their plan

for conducting Exercise I and to gain his approval for administering it.

He was of the opinion that the idea was "great," so the team made arrangements for the USBM Washington Staff to meet the following day at Bal^-Ston

Towers, in Arlington, Virginia, in order for them to participate.

JSee Project SOAP - Exercise I Memorandum, May 15, 1973, Appendix B-1 Thru B-4.

both^of'^TA?^

°^ Arlington, and Robert S. Cutler and Dr. Mary Ellen Colby,

**Leo Schrider was also present that day.

28
At this afternoon meeting on May 17,* the details of Exercise I

were explained and each person present was given a packet of materials

which included: the seventeen Project Proposals; a four -page explanatory

memorandum; pertinent material excerpted from the USBM Mission Statement

to serve as a guide for judging the "relevance" or the "quality" of these

same Project Proposals; and a tally sheet.** These participants were asked

to mail their completed tally sheets to TAD/NBS before June 1, 1973.

A similar orientation briefing by the Project SOAP team was given to

four groups at BERC (on May 23) and four groups at MERC (on May 31) . In

all, 67 individuals were asked to participate in this research evaluation

exercise. They were informed of the background and purpose for requesting

tills type of subjective assessnient, and each member of each "jury" was

asked to rate the sample set and indicate a "priority score" based upon

his own judgment of the facts presented in the proposal. A set of guideline criteria was included in the reference materials. It was stressed,

each time, that participation in the Exercise was voluntary and that if

a person participated but wanted to remain anonymous, he might.

The Research Supervisors at both Centers (and one group of Staff)

were asked to rate the seventeen sample project proposals for "program

relevance" only, while the other two groups of Technical Staff were asked to

rate them for "technical quality" only. A table summarizing the participation by the various "juries" is included in Section 3.3*** of this report.

^'Attending were Dick Gooding, Larry Ramsey, Larry Burman, Jerry Ham,

all of USBM, Arlington; Henry Jacobson, John Tosh, of USBM, Washington;

and Bob Cutler and Mary Ellen Golby, of TAD. Leo Schrider (who is not

a Monitor), of MERC, also attended. Because they were not able to be

present at this meeting, Alex Mills, Harry Johnson, Dick Corey, and

Brian Harney, of USBM, Washington, were given their packets the preceding

day.

**See Appendix B.

***See Table A, p. 38.

29
3. FINDINGS AND CONCLUSIONS

The primary purpose of this Exploratory Survey was to seek clarification

of principal issues and identification of some of the actual problems associated with implementing miss ion -oriented projects within the Bureau of Mines

Energy Research organization. Such exploration and clarification of desired

"ends" must of necessity precede the analyses of possible "means" for

improvement. /

The Project SOAP team conducted a series of 14 group interviews at

three different locations within the USBM Energy Research organization.

These included meetings with key personnel at both the Washington Headquarters and at MERC and BERC. In addition^ 67 individuals from these

groups were involved in a preliminary assessment of a sample set of Energy

Researcli Project Proposals. Tlie purpose was to test a program management

mechanism for evaluating the "quality" and "relevance" of individual

projects in terms of stated program goals. (No analysis of the results

of this "Exercise I" is included in this report.)

Tlie follov/ing represents the essence of tlie findings obtained thus far.

Each item listed is the result of many man-hours of discussion and deliberation

by USBM personnel at the Centers involved. These differ from the listings

shown on pages 19 and 24, respectively, in that they represent revisions and

acknowledgements of certain "problem areas" identified by the SOAP team.

Although these findings are not all-inclusive, they clearly indicate a

consensus by knowledgeable people on some of the program management problems

that are believed to be solvable with available methods and personnel.

30
3.1 MERC "Problem Areas"

Acknowledged "Problem Areas" Associated with New Management Concepts

Resulting from Second Visit, Morgantown Energy Research Center,

April 25-26, 1973

1 . Explicit Mission Statement

• Translation of Energy Research Program "Goals"

into Research Project "Objectives"

• Participation in Program Planning Process

• Results -Oriented Approach

2 . Improved Organizational "Feedback"

• Research Supervisor Function Clarified

(Intra -Communications: within the Center)

• Research Monitor Function Clarified

(Inter-communications: between the Centers)

f Strategic -Level Planning/Evaluation Staff

3. Mechanism for Evaluating Projects in Terms of Stated

Program Goals

• Project Priority Determination

. "Quality" (Peer Review Judgments)

. "Relevance" (Internal/External Opinions)

Time -Frame for Results

. Costs (multi-Year, Total Investment)

• Idea Referral/Project Proposal System

. Stimulate "Good" Research Projects

. Assure Useful Research Results

• Projects Proposal /Budget Review Process

. Targeted Research Activities

. Consistent Evaluation Criteria

While the majority of the men at MERC originally felt that an explicit

"Mission Statement" was the primary thing they needed to give direction

to their work (and nearly all agreed that an individual, or Station,

Mission Statement could be subsumed under such guidelines) , they had now come

to recognize that the Research Supervisor's function had greater need for

clarification than that of the Research Mbnitor, though they ranked both

31
"problem areas" as second in importance. And moreover, after consideration of relative priority, they felt that a mechanism for judging the

"quality" and "relevance" of a project was more important than improving

either the New Idea Referral System or the Project Proposal System.

The SOAP team reported these findings at the Nfey 1st. briefing to

the USBM Research Center Directors and others from the USBM Energy group.

3.2 BERC "Problem Areas"

After the team's visit to Bartlesville, the Center Director,

John S. Ball, met with his Research Supervisors to draw up a response

to the list of "problem areas" identified by SOAP and presented to them

for reconsideration. The resulting list emerged:

"Problem Areas" Acknowledged as a Result of

Project SOAP Inifial Visit to Bartlesville Energy Research Center Moy 21-23, 1973

1. Improved Project Manogement Review Mechanism • Determinotion of "Technical Quality" and "Progrom Relevonce"of Research

Activities • Continuing Assessment with Feedb %;k of Center Priorities • Assessment of Program Results as a Basis for Program Evaluation

2. New Idea Development/Project Proposal System • Present Procedure Vague and Ineffective • Lock of Incentive (Funding) to Stimulate Innovation • Requires Meaningful Feedback

3. Clorification of Headquarters Energy Division Staff Function • Improved Program Planning/Evaluation by Staff • Research Monitor Function not Clearly Defined

' • Lioison with Technical/Industry Advisory Groups

4. Recognition of Failure of Communication Between Washington Headquarters and

Field Stations and Between Field Stations • Clarification of Research Direction—Correlation of Policy Positions and

Mission Statement • Acceptance of Specific Energy Research Program (and Priorities) from Heodquorters by Centers • Proliferation of Reports and Lack of Evidence of Their Usefulness • Budgetary Decision Progress • Inadequate Funding Support to BERC Copcbilltles in Face of Current Notional Energy Crisis

32
While at Bartlesville, the SO^P team had learned that BERC differs

as much from MERC as the Oklahoma plains differ from the West Virginia

hills or as oil differs from coal. One sensed that the Staff at BERC

are un.usually strong administrators and are highly individualistic. One

sensed, too, that there existed at BERC the suspicion, frequently iterated,

that the Washington Staff, who ars primarily ''coal men," are not sensitive

to the needs of the "oil men" at Bartlesville. The consensus was that

the top decision-makers at Headquarters are "99% coal and 1% petroleum."

From the basis of a balanced Energy Program, there appears to be some inequality in

the distribution of funds to research in coal and petroleum: petroleum

provides 751 of the total energy used, while coal provides only 25%, yet

coal research has an annual appropriation of $12 million, while petroleum

has an appropriation of only $3 million,, At any rate, the SOAP team learned

that BERC is definitely oriented to one commodity, namely, petroleum,

although they did note that several coal research and Health and Safety

projects are being conducted.

3.3 Exercise I Preliminary Results

Essentially, vAiat the SOAP team hoped to find out from this prelininary exercise was whether the use of a "peer- review" mechanism to aid in

evaluating Energy research projects vms operationally feasible. They tested,

by obtaining actual ratings, the opinions of various people in "the system"

who were asked to become involved in a simulation of this decision process.

Of the 67 professionals involved,* 56 actually participated by submitting

their tally sheets. Each participant rated a sample of 17 Project Proposals

on the basis of his own concept of either "tecbjiical quality" or "program

relevance," but not both. Their ratings were subjected to a preliminary

analysis to determine if there exists a meaningful and consistent concept

underlying each of these two criteria dimensions to warrant their operational

'^See Table A, p. 38.

33
use in a Program Management evaluation tool. (See Figure a., following

page .

)

The findings from a preliminary analysis of this first Exercise

are quite indicative though, of course, not definitive:

1. There was implicit in the participation of 83.5% of the

individuals asked to serve as reviewers (and in their subsequent expressions of curiosity about and interest in the

results) a general agreement that such quantitative methods

for evaluating research project priorities within the two

USBM Energy Research Centers (MERC and BERC) are operationally

feasible.

2. As can be seen from Figure a*, which presents the "modes"

(most frequent rating given each project along each axis),

there is some obvious agreement anwng those reviewers in

applying their concepts of "program relevance," but not

a clear consensus. (The concept of "program relevance" is a complex one requiring greater specificity and communication to assure a common understanding within the organization.)

3. The greater degree of consistency among the reviewers in judging

"technical quality" was partially due to the clearer definition

of this concept and also the fact that the sample set were

on-going projects. The concept underlying "technical quality"

as used in this Exercise was found to include more than simple

technological considerations. These additional criteria appear

to be closely related to priority considerations.

4. If the reviewers had experienced great difficulty in separating

"relevance" from "quality" considerations, the plotted points

would have been clustered around the "45 degree" diagonal line

in Figure a. Tliis is clearly not the case; the distinction

between the two criteria is one which can be rendered operational

in such a rating procedure and, indeed, people at two different

Research Centers are capable of concurring that a particular

project (here, "Project G") ranks quite high on one scale but

low on the other.

5. It is interesting to note that Project M (an "oil" Project

from BERC) was judged by the MERC people to be more "relevant"

than by BERC, and similarly, Project B (a "coal" Project from

MERC) was considered more relevant by the reviewers at BERC

than at MERC. This finding suggests that the reviewers can

indeed broaden their concept of mission to include the goals

of the Energy Research Program as a whole, when evaluating the

particular merits of projects originating from USBM Research

Centers other than their own. (See Figures b. and c, pages 36, 37.)

*And its comparison with Figures b. and c. (See pp. 35,36, and 37.)

34
5 M

W I i

1^ S ^ oil ?r

•H

35

u. C -J

M jy>

^ U rs| 00 c^ ^ 51 53

^

I

(2

« ^ a vM* ij

: I

00

-4--

! !- i , ! i

0

OO

|3,

s

o

5i

HI fi

^

1

O

o

37
TABLE A

Exercise I Results

Summary of Participant Responses

(As of June 10, 1973)

JURY PARTICIPANT RESPONSES PERCEOTAGE

RECEIVED

I. USBM WASH Staff 5 5 ! 1001

II. USm WASH Staff . 4 3 751

"II. WASH/^4ERC/BERC Staff

a. WAon J y.)ucixxL.^j A 3 75%

b. MERC #1 (Relevance) 5 2 40%

c. McKL ffz ^l^ualltyJ 0 3 50%

d. MERC #3 (Quality) 5 4 80%

e. DcKL ffi ij^uaiityj 5 4 80%

f. BERC #2 (Quality) 5 100%

g. BERC #3 (Relevance) 5 5 100%

IV. MERC Supervisors 7 7 100%

V . dckLj oupervisors c0

£:0

T n n Q- 100-5

VI. Industry Advisory Panels

a. Industrial Members 5 5 100%

b. Academic Members 5 4 80%

TOTAL 67 56 83.5%

Other Summary Data:

1. No. Criteria Statements Received 46

2. No. Signatures Received 36

Project SOAI* ! TeehniMi Analysis Division

Nofional Bareau of Standards RSC/RMC

j 10 June 1974

38
The remarks received from participants are noteworthy. In describing his reaction to Exercise I, one BERC Project Leader ivrote:

The most obvious result of participation in this project was the

realization of how little actually useful information is contained

in tlie typical project proposal. Some 16 proposals, totaling over 5 million dollars, averaged less than two pages each, of which less

than a half page is devoted to a very brief outline of the work to

be accomplished with the capabilities of the principal investigator

and his available facilities, arriving at an intelligent appraisal of

the potential for success of a particular project is essentially

impossible with the amount of information supplied. Even allowing for

the briefness of the presentation, the proposals are inadequate for

the most part. As an example, tlie objective of a research project is

not (or should not be) "to study the composition ", it should

be to make specific determinations of some facts as a result of study

and physical experiment. The objectives should state what is intended

for accomplishment during the specific period, not simply broad indefinite, hoped for results such as "increased recovery of oil".

Another participant had difficulty separating his concept of "program

relevance" from that of "technical quality." He wrote:

I found it quite difficult to determine "Program relevance"

for the selected projects without also considering technical

feasibility, as technical feasibility is dependent on other

parameters such as funding, timing, competence, manpower....

Further, he went on to add:

The resulting rating should be recognized as only a partial

evaluation of the priority of a project. Likewise, a technical

feasibility evaluation also is only a partial evaluation....

What I'm leading up to is the difficult and less "clear cut"

subjective judgment that must be used in selecting the

"mix" of short-, intermediate-, and long-term projects as well

as the mix of basic, development, and applied research that

must be maintained for a balanced program. I realize

determining the ratios of the mix may be arbitrary decisions,

but they must be made to assure an adequate background of

fundamental and developmental work and competence necessary for

the pursuit of long-term projects. To change the present situation would require a skillful surgeon, a sharp knife, and

probably a considerable degree of pain. The USBM Energy Group

appears to suffer from a number of illnesses that compound the

problem. For example, there are too many Chiefs and not

enough Indians. Many of the Directors, both in the Field and

at the Washington Office, are required to operate with staffs

that are too small and as a result are believed by some to be

\
incapable of thorough analysis and coordination o£ research

projects before they are implemented. There seems to be

pressure to "get into the field" with projects before adequate laboratory research work is completed, apparently

because of the better visibility of field research.

The basic missing ingredient is specific program direction from

the Washington office. Because of the lack of it, most projects are written on the ba'sis of what the researcher would

like to do rather than what, needs to be done. The simple

suggestion to "improve the efficiency of oil recovery" is

not enough. Field personnel are too restricted in their

viewing area, in their work experience, and in their contacts

with Industry (except in a few special cases) , to effectively

relate their capabilities with broad needs. They either over

estimate the amount of research that can be accomplished at a given funding level and consequently spread themselves too

thinly, or they concentrate all effort on a minuscule portion

of the real problem, either through lack of understanding of the

problem or because of personal reasons of self interest. Only

direct, detailed, and coordinated program direction from an informed and concerned Washington Staff can correct this

situation. . .

.

One person at MERC summed it up when he said: "the Washington

Office has exercised little in the way of guidelines for Energy programs

in the past and has had little or no time to formulate thoughts on

any program or to evaluate what is submitted to them from the field.

Yet, the researchers annually look to them for some relatively defined

goals towards which efforts are to be directed. In this time of

concern over energy shortages, however, perhaps we need a NASA- type

'systems approach' to accomplish some important development."

3.4 Conclusions

Thus far into this exploratory survey, the Project SOAP task group

has followed a planned fact-finding procedure. Based upon interviews at

three locations (WASH, MERC, BERC) , it has focused attention on principal

problems observed (and verified) at each of the operating units visited.

40
Of the many problems the USBM management has been aware of, the

lists of acknowledged "problem areas," identified in Sections 3.1 and 3.2

above, represent those where reasonable improvements are considered possible, both by the SOAP team and by the Field Center personnel themselves.

These lists include a recognition from two Research Centers of the need

for some rational mechanism for evaluating the quality and relevance of

individual projects for use in determining project priorities at the Field

Station level.

At this early stage, the SOAP team (including members of the USBM

Washington Staff) have concluded that two of the key functions of the

decision-making "pipeline" in most need o£ attention are those of the

Research Supervisor and the Research Monitor. There is apparently confusion

at both the Headquarters and Field levels concerning what is or ought to be

their respective roles in bringing about tjhe new concept of miss ion -oriented

program management.

As for the existing "Idea Referral System," the procedure appears to

some at both MERC and BERC to have become an effective process for

killing "good ideas." There is a need for more dialogue between the

Washington Headquarters, who should establish program "requirements," and

the Field Centers, whose functions ought to emphasize matters of technical

feasibility. However, if each Research Center Director is to be responsible

for establishing budget priorities, questions concerning the comparative

"program relevance" of particular "ideas" which would be in competition with

on-going projects will have to be resolved. As it is, there is little

incentive, and much difficulty, in motivating individuals to propose new

projects from the Field level.

41
Perhaps this is the main beneficial result of the Project SOAP

Exploratory Survey: the gaining of insights by the people within "the

system" about precisely what it is they believe is wrong, and what ought

to be done by way of remedy. One of the facts generally recognized about

operational improvement is that the remedy cannot work unless most of the

people involved are convinced th^t it will work. There is, we have found,

much more to implementing a new concept of Program Management than systems

analysis alone can provide.

The SOAP team concluded from its preliminary interpretation of the results

of Exercise I that the mechanism sijijulated for determining priorities based

on separate assessments of "quality*' and "relevance" appears to provide

a reasonable basis for further work^ at least for MERC and BERC. That is,

the development of a management tool for determining the priorities of

individual projects at the Center level upon which to base program priorities

for Mid -Year Budget Review and for evaluating New Idea Proposals seems

feasible. How^ever, more work is needed before this evaluation mechanism

can be useful as an indicator of specific program performance* based upon

the two generalized research management criteria: teclmical excellence and

relevant results. ,

Certain changes in this mechanism are suggested. They include the

forma.tion of more sharply contrasting "peer group" juries; the translation

of Mission Statement "goals" into achievable Energy Program "objectives";

and improvement of the Project Proposal descriptions themselves. These

procedural items can be worked out in practice.

'^The former is judged typically by professional peers, and the latte.r by

Budget Administrators, Congressmen and various people within particular

industrial research organizations

.

42
A second simulation exercise is recommended incorporating the above

improvements but also involving all six USBM Energy Research Centers and

dealing with actual projects (both on-going and new proposals) to be'

evaluated at tlie forthcoming Mid-Year Budget Review in October 1973.

It is anticipated that decision-makers at various levels in the Bureau of

Mines' organization will then be able to obtain a more precise picture

of what is happening within their overall program of Energy Research.

Further development and use of this '"peer review" mechanism would provide

a new basis for making decisions about program priorities and strategies

effecting a more rational balance in research resource allocations.

43

Prajeet SOAP

Technical Anaiytit Divltian National Bareau af Standardt
. I-
APPENDIX

(Section A)

Letters and Memoranda
O 00 H ^

0) CD u sup; P <H O

J-i CD U

(U o 35

t/) o &

rH

?s

CD •H

CD o 5-1 ouo CD >.

CQ n

fH H CD q-i C O W

^3 if) 1

CD

0) CO c

•H o

fHPh m,3 o !h •H B

0) 0)

;3 CD M ^ M O

+-> r-H PC 4-;

c:: 0) r:^ CD O

O X!

•H

+-> S> -M 5

(/) •H o

r-)P.

+-> 03

CD tj o

03 CD

E-i FS CD U)

1^ O cd J—

!

•H p

nj

U) '^

p: m

0 o

i

1

1—

1

O

X

w

1—

1

w

r^ to

eg <i> o

Ph

-H

(D .H

<4-l O CD

4->

!=; +j

CD g

0.. ^

r-H

!-> 03

CD O

&a

o

E—I I—

I

r (J

(/) CD

93 o

1—1 ^

03 s ^

CO r

CD

u

03

f-i •P Ic

>^ o

M-l -H •H (/) -P (/) C -H

CD S

•H mo

o

-a

03

tfl -P

<U 03

"c^ P. P^ S

Ph

H

P 03

03 1—

I

Sh O

CD

U5 —

t

03 'B §D U 03 Q ^ •ri O

rH 5h • P.

•H (/i +-> 03 Mh 3 O

I—

(

o3 t/i > h

g 0) P

P Jh -P W O

CO ^ -H C-H -P

+-> 03 c3

(/) -en H y O

X © fH W 6 P.

03 x; • ^ u

p; c3

• H <i> M-H 1/5 CD CD nd fH O Mh H O

I 8

P

O fH y (u C 53 o u

p

t/j ccS I—

I

H N 0) U -H -H 0 C <4-t 03 ^

.-H fi S

o3 O

P.

•H ^ .p ^y 03

Cu to 03

t+H Cii

fH i-H •H Q

+-» c

03 W

•H (/) +-> o

•H <4-l P. °

^ S -P

CD 03

fH

03 0) .H

o

a*

a: 2

^'p

(D

p: cs +j •H -H Pi ^^•^

•H m

< :s o

x;1Cu

C/5

0) p

QW CO U J

Cl< CO

03

fH

0) -ap

T3

fHp.

O

CD

0

fH

fH

0

CO 0 :=) XI

•H p; (/) •p -H 03 -P CM

3 ^

crpL, g 0 o m y -H op:

o

0 o

o

to

ccS 0

fH0

fH0

t/i 0 0

0

0

03

!h0

P.O

so

CC0

fH <

0 i-i

PUil-J

p; o3 H 0

p: '.p 0 0 o <+-! -Ti 03 e u • H +J o 0 t3 0

4-> p: fH e rH ^ pi, 0 op 0 0 & o

•H 6 P

^

fH 0 -H u 0 +->

(/) fH to 0 bfi-H _ _ Q <; U M-I

to P o o

PU-H

•p

•H§

fH030

fH (/) O 0

A-1
March 9, 1975

Mr. V/illicun L. Crentz

Assistant Director - Energy

U.S. Ixireau o£ Mines

Wasliington, D. C. 20240

Dear Kr. Crentz:

As a result of the March 6 meeting at the National Bureau o£ Standards

between Harry Johnson,, Leo Schrider, and /\ridy Decora of your staff and

members of our Project SO.-\P Team, v;e are pleased to assist your Energy

Group staff in implementing the new mission and management concepts

associated with the National Energ)' Program.

We understand that your primary interest is improving communications

between Headquarters and field levels. This would be useful for developing

more effective management controls for directing individual research activities toward the new goals defined by the Energy Research Program. In

this regard you would like us to make' an objective examination of functional

operating activities and infonriation flows at various decision levels within

the organization. Such an examination will serve to clarify certain issues

concerning your broader mission and new evaluation requirements for midyeai- project reporting.

Of secondary concern, but possibly the simpler problem to address, is the

need for a practical mLechanism for evaluating both major program segments

^d individual research projects in terms of program goals. Because of

the large numiber of ongoing projects and' the many new and potentially

useful proposals wnich come to the Research Center Directors as contract

proposals (or from, the research interests of individual scientists and

engineers v/ithin their field research centers) a systematic m.eans for

determining "technical quality" and "program relevance" is an important

management tool for them to use when approving new research projects and

for establishing priorities. Tliese program relevance values can also be

used as criteria for integrating special emphasis research areas, for

making inter-program trade-offs, and for providing visible evidence of the

coherence of program management at all levels.

In order to be of real help to your Program's staff, we must first clarify

the principal issues and identify the actual probJems that currently exist.

One of these central issues, as Harry and Leo described it to us, is one

of communication of new ideas between Headquarters and the field- -m.ore specifically, between the individual researchers, who are the main strength
of the energy program, and top nianagcn.ent personnel in Washington. Further

we were informed, there is a need for improving vertical infonrntion flow,

and horizontal communication (cross -talk) between individual scientists at

one field center wirh those of other centers and bot^.veen individual project

leaders aiid research supervisors with their peers elsewhere. TliereforCj

before undertaking prescription of any particular methods for improving

the accuracy and precision of feedback infonriation, we believe, as a result of our experience, that a careful examination must be made to

assure that the desired benefits to the organization would result.

In short, we propose to make a three-month exploratory suivey of your

"Prograjn Management System." V/e hope to develop, for your staffs a basis

for objectively determining the nature and extent of the "commrunication

problem" at each level, Tliis initial phase should also provide you with

a basis for considering the feasibility of extending this approach,

• leading ultimately toward your organization's iniplementing a quantitative

method for improved management control.

We are ready to start this assignment on March 19, 1973,. and plan to

complete the work by June 30, 1973. Participation from your staff is,

of course, required as soon as possible, so that cur necessary orientation and survey planning can proceed on schedule.

Upon receipt of a formal work order in the amount of $37,400, for reimburse

Ficnt of the estimated salary/ and expenses indicated on the proposed budget j this project will be authorized. If :here are further questions or related

matters, please let me know.

We look forward to working closely with your staff on this interesting

and important assignment.

Sincerely,

Louis C. Santone, Acting Chief

Technical Analysis Division

Enclosure

cc: Cutler
!

I
2

Ur. R. S. Cutler, NBS, 6/1/73

Under New Idea Development, the idea referral system appears to be an effective

system for killing ideas. The Issue Paper route is better but ill-defined, and

normally to be used only on invitation from the Washington Office. There is little incentive to develop a new program if the reply is to kill some present

programs in order to reprogram funds. The development of strong programs re- quires a dialogue between the Washington Office who establish program relevance

and the field whose function is to determine technical feasibility.

The Clarification of the Headquarters Staff Function would perhaps help to focus

discussion where it would do the most good. We are rather uncertain as to what

mix of monitoring functions and purely staff operations are desired. Manuscripts

and convention travel requests go directly to staff while proposals and management review reports apparently are not staff functions. Some industry advisory

groups have fiaison through the Washington Office while others are directly with

the Research Centers.

The section labeled "Failure of Communication" was changed from "Communication

Gap" because we believe it is not a problem of speaking different languages but

rather of failing to speak at all. This is understandable with the urgency and confusion, but it does at times present problems. The positions presented by the Project

SOAP team as representing Washington thinking did not always match our understanding. Specifically we found some differences between the Mission Statement

and our understanding of the energy research policy. We were surprised to hear that

Washington regarded our acceptance of programs and priorities as less than complete.

We do feel that reporting requirements have been increased and wish we had more

feedback to indicate that they are being used. We are particularly uninformed as

to budgetary decisions and are unable to relate such decisions to the preparation of

further proposals. We feel that the present energy crisis is the first opportunity in many years to increase our usefulness and are dismayed that instead we face a reduction.

I believe that the visit of the SOAP team was beneficial in that it caused us to stop and assess our problems. In particular, the exercise of rating projects brought

A-S
3

Ur. R. S. Cutler, NBS, 6/1/73

home to some of our people the comoetition that they face. We wili look forword

with interest to your report of June 30,, If there is additional informotscri v/e can

furnish, call or write us.

Sincerely yours.

John S. Bali

Research Director

Enclosure

AProblem Areas Identified as a Result of

Project SOAP Initial Visifr to Bartlesville Energy Research Center

May 21-23, 1973

Improved Project Management Review Mechanism

• Determination of "Technica! Quality" and "Program Relevance" of Research

Activities • Continuing Assessment with Feedback of Center Priorities

0 Assessment of Program Results as a Basis for Program Evaluation

New Idea Development/Project Proposal System • Present Procedure Vague and Ineffective • Lack of Incentive (Funding) to Stimulate Innovation • Requires Meaningful Feedback

Clarification of Headquarters Energy Division Staff Function • Improved Program Planning/Evaluation by Staff

• Research Monitor Function not Clearly Defined • Liaison with Technical/Industry Advisory Groups

i

Recognition of Failure of Communication Between Washington Headquarters and

Field Stations and Between Fseld Stations

• Clarification of Research Direction —Correlation of Policy Positions and

Mission Statement • Acceptance of Specific Energy Research Program (and Priorities) from

Headquarters by Centers • Proliferation of Reports and Lack of Evidence of Their Usefulness

• Budgetary Decision Progress • Inadequate Funding Support to BERC Ccpabilities in Face of Current National

Energy Crisis

A7
ii-PPEl\T)IX

(Section Bj

Exercise I Materials


2

For this preliminary Exercise, six review groups, or "juries,"

consisting each of three to five members will rate a representative

san^le of research project proposals submitted to the Assistant

Director-Energy, U. S. Bureau of Mines, for FY-74 funding. The

juries selected are as follows:

Juries Research Review Groups

I . USBM Washington Staff

II. USEM Washington Staff

III. MERC Research Supervisors

IV. • WASH/MERC/BERC Staff

V. BERC Research Supervisors

VI. ' Industry Technical Advisory

Panel

Procedure

:

Following the orientation briefing by Project SOAP team describing

the background and purpose of this preliminary Exercise, each member of

each "Jury" will rate the sample set and indicate a "priority score"

based upon his own judgment of the facts and information presented, in

accordance with the appropriate "criteria" described below.

Juries I, II, IV, and V will rate each project for "program relevance" only, \A\ile Juries III and VI will rate "technical quality" only.

Each member will register his vote (priority score) on the tally

sheet si:5)plied. This will indicate the Juror's professional opinion of

the particular Research Project rated. Scoring will consist of selecting

a number between 1.0 to 5.0 (or N/R) where 1.0 signifies HIGHEST value

and 5.0 Lowest value, with N/R meaning "not qualified to render judgment."

(0.5 increments are allowed.)

After each reviewer has completed his rating, he is to send the

tally sheet only to NBS in the si:pplied pre-addressed envelope.

Criteria :

For this Exercise, "Technical Quality" refers to the probability of

achieving the stated research objective, talcing into account:

1. Qualifications of Principle Investigator.

2. Availability of Coirpetent Staff/Facilities.

B-2
3

3. Scope of Work/Research Cbjectives.

4. Technical Approach/Work Schedule.

5. Adequacy of Proposed Budget.

In addition to the above necessary technical considerations

,

"program relevance" is intended to be used as a variable measure for

estimating the utility (or "worth-'whileness") of the proposed research

results for which no generally accepted standards or criteria has been

established. In other words, assuming the particular research project

is successful, how important would it be toward meeting the goals of

the Bureau of Mines Energy Research Program (as described in the Energy

Research Mission Statement, May 25, 1972).

It has been possible for many people (Research Scientists, Administrators, Congressmen, Industrial Executives), to render judgments about

"program relevance," based upon various criteria. It has been difficult,

however, for the Research Scientists to explicate these 'TDenefits" of

proposed research output, it is not clear to Budget examiners within

the Department of Interior (and The Office of Budget and Management)

whether similar criteria are being used by the various people, particularly those with obviously different backgrounds, experience, and

interests. Moreover, the importance of evaluating Energy Research

"results" in terms of meeting the needs of the Energy Crisis (and

admittedly these "results" are difficult to define) is often lost in

this process

For consideration by participants in this Exercise, the following

perspective is offered:

"To establish a conceptual framework for our role, research

might be viewed as a continuous flow of infomation that begins

with basic discoveries and continues through commercial development.... The problem- oriented research we conduct is both useful

and essential for the development of new processes (of Energy

Production) needed by our society. ... In delineating major

Research activities to be undertaken by Energy Research, and

relative priorities, we have been guided by: (1) the national

energy policy, (2) technological needs, and (3) our concept of

mission. ..."

These statements should be regarded as a framework for establishing

a common base for evaluating the proposed results of Energy Researchconducted by the U. S. Bureau of Mines in advancing production technology

of coal, oil shale, petroleum and natural gas.

B-3
4

Results Expected :

Based on the analyses of the priority scores received from the six

Jury groups, a series of tabulations and charts will be developed to

indicate:

1. Any significant differences or agreement between the

various review groups in assessing "program relevance"

of the particular research projects rated.

2. Coirputed rank order listing of sample set of research

projects, based on mathematical combination of technical

: "quality" and "program relevance."

3. Degree of acceptability (in principle or practice, as indicated by abstentions) with this mechanism for

evaluating energy research project proposals within the

. 'O' U. S. Bureau of Mines.

4. List of explicit "program relevance" criteria.

Project SOAP

Teehnieti Analysis Division National Buroao of Standards

RSC/RMC

B-4 5-15-73
Project SOAP

Exercise I Tally Sheet

ponc UJuh

(

'DDTADTTV r CrnDP" KIUKl 1 1 oLUKJc :Scale: 1.0-5.0)

" , (Basis for Judgment)

A Stimulating Tar Sand

Production

B Formcoke

C Earth Fracture Systems -- Application to Hydrocarbon

Recovery

D Utilization of Solid

Wastes from Combustion

and MinincT of Coal

E Studies of Asphalt and

Asphaltic Materials

F Reduced S0-> in Coni)Ustion

Gases

G Demonstration of Improved

Secondary and Tertiary

Methods of Recovering

Petroleum

H Coal Disproportionation

by Catal\tic Hydro genati on

I Factors Affecting New Oil

Recovery

1

J Development of Marginal Gas

Resources Utilizing Natural

and Indiicpd FractiiTPS

K Corrosion of Fireside Surfaces in Coal- Fired Boilers

L Fuels and Engine Systems to Enhance Fuel Economy in Clean

Air Cars

M Waste Oil Recycling

N Pressurized Gas Producer and

Gas CleanuD and Purification

0 Development of a "Prebum"

Propellant for in situ

Underground Coal Gasification

P

v Demonstration of ^tethane - Methanol Coal Conversion Planit

Q Underground Gasification of

Coal

Jury No. Name (optional)

B-5
ATTACHMENT I

U. S. BUREAU OF MINES

ENERGY RESARCH PROGRAM

MISSION STATBIENT -

May 25, 1972

(Annotated Excerpts from Mission

Statement for reference by Exercise I Participants - 5/15/73)

B-6

POLICY IMPLEMENTATION

The specific actions by the Department of the Interior and the

Bureau of Mines to implement President Nixon's Clean Energy Policy are

guided by the President and by the directives of Congress. The policy

of Congress, embodied in various laws, has recently been supplemented

by enactment of the Mining and Minerals Policy Act of 1970 (84 Stat.

1876). That act states:

. . . it is the continuing policy of the Federal Government in the

national interest to foster and encourage private enterprise in

(1) the development of economically sound and stable domestic

mining, minerals, metal, and mineral reclamation industries,

(2) the orderly and economic development of domestic mineral

resources, reserves, and reclamation of metals and minerals to

help assure satisfaction of industrial, security, and environmental needs, (3) mining, mineral, and metallurgical research,

including the use and recycling of scrap to promote the wise and

efficient uso of our natural and reclaimable mineral resources,

and (4) the study and development of methods for the disposal,

control, and reclamation of mineral waste products, and the

reclamation of mined land, so as to lessen any adverse impact of

mineral extraction and processing upon the physical environment

that may result from mining or mineral activities.

The Secretary of the Interior has been assigned the responsibility to

carry out these policy directives in exercising his legal authority.

To implement this responsibility, the Interior Department has

established the following energy and minerals mission:

To assure a continuing and adequate flow of energy and minerals

from reliable sources at the lowest cost consistent with the

attainment of social and environmental objectives

and has charged the Bureau of Mines with the responsibility:

To conduct those programs of inquiry and regulation necessary to

keep the Government informed and to stimulate the private sector

to produce the minerals and fuels needed to supply an appropriate

and substantial share of the national needs in a manner acceptable

to the public interest.

To help discharge these responsibilities, 'the mission of Energy Research

within the Bureau of Mines has been formulated as follows:

To develop effective means that will increase the availability

and improve the utilization of the Nation's energy resources at

B-8
reasonable costs and in a manner that will help attain desirable

social and environmental improvements.

Specific areas of research are conducted at six energy research

facilities which employ more than 900 people. The location, field

manager, and mission for each of these facilities are given in Appendix

A. Priorities for the research are based upon anticipated patterns of

energy consumption and the technologic needs as subsequently outlined

in this statement.

B-9
TECHNOLOGIC NEEDS-'

The production and consumption of energy in the United States are

complex activities that are slow to respond to change. Thus, the

investments already made for installed electrical generating capacity

and the more than 100 million vehicles currently in use have committed

us, over the short-term, to established patterns of energy consumption.

Our only option in this time frame is to continue to supply adequate

amounts of those fuels that now constitute our energy mix.

Not all of the short-term potential problems lend themselves to

solution through advanced technology. For example, a shortage in gaseous fuels is expected to be in full force within the next few years

despite anything that might be done. This shortage is already having

an effect on applications for new customers; some industrial and other

users will be forced to use alternate energy sources- -petroleum and

coal. Adequate domestic supplies of these, commodities cannot be produced quickly and/or used freely under current environmental standards,

forcing increasing reliance on low-sulfur imports of foreign oil.

Although improved technology will not solve all short-term energy

problems, it can contribute to substantial reductions in potential

impacts for many of them. Over the longer term, new and revolutionary

technology may be developed and adopted that will help provide energy

in the forms and amounts needed by the Nation while at the same time

preventing environmental degradation. Some of the more important

\_l Needed technologic developments are actually more complex

than as indicated in this section. For a more thorough discussion see: U.S. Department of the Interior, United States

Energy; A S;;mnijry Review. January 1972, and Energy Rese a rch

Needs . a report to the- National Science Foundation by Resources

for the Future, Inc., in cooperation with MIT Environmental

Laboratory, October 1971,

B-10
possible approaches that are expected to yield results over the short,

intermediate, and long term are discussed in the sections that follow.

Short term needs (1972-1980) - An immediate problem is to increase

our available energy supply base and develop means by which these

resources can be used without environmental degradation. One way to

increase supplies is by a concentrated effort to stimulate production

from known oil and gas fields. Such fields contain nearly 60 billion

barrels of oil and 300 trillion cubic feet of gas that do not lend

themselves to economical recovery at current prices with existing

technology. Methods that may be employed in an effort to recover these

resources include

o fracturing methods: nuclear, chemical explosives, hydraulic,

and gas

;

o displacement techniques: improved water and gas flooding,

miscible, solvent, and thermal drives;

o production methods: fracture orientation and subsurface hydraulic

fracture mapping, and

o combinations of various methods that would improve total recovery.

Increased exploration and development for all of the Nation's

primary energy resources must be stimulated, including the search for

new, low-sulfur coal deposits and low-cost uranium. Developments are.

needed in

o exploration techniques: radiometric, seismic, magnetic, radiation halo, infrared, ultraviolet and laser photography, and

side blocking radar and side looking sonar;

o production and distribution techniques for new production areas:

Northern Alaska and offshore; and

o means to convert coal to gaseous utility fuels.

As supplies and consumption of these energy fuels increase, methods

B-11
must also be developed to use them in a manner designed to enhance the

environment. Of importance are:

- o methods to reduce sulfur discharges from stack gases; and

o fuel/engine changes to restrict undesirable vehicular emissions

with minimal impact on combustion efficiency.

• Intermediate term needs (1975-1985) - This period represents the

transitional time between, and thus overlaps^ the short and long-run

time frame. Of importance in this period is the development of indigenous energy resources in the form most needed, including

:. o conversion of coal to liquid fuels and pipeline quality gaseous

fuels;

o production of gaseous and liquid fuels from organic wastes; and

o production of liquid and gaseous products from the Nation's

. oil shale, heavy oil, and tar sand deposits, and methods of

processing them. * . ,

A distinguishing feature of each of the above is that the primary

resource is a solid or semi-solid. Except for organic wastes, each

must be processed in situ or mined, then it must be converted to a

usable liquid or gaseous form. The materials extraction and handling

problems will be on a scale never before undertaken and current technology is not adequate for the task. Significant developments are

needed in '

o materials handling systems to process bulk materials at an

unprecedented rate;

o developing new concepts that would optimize resource recovery,

provide for underground placement of waste products for local

support and subsidence control, and protect the health and

safety of personnel and the environment;

o in situ techniques to recover energy fuels; and

B-12
o training greater numbers of engineers and skilled workers in

the general minerals technology field.

Long term needs (1980-2000) - The Nation is rapidly moving toward

electricity as the predominate energy form. But on the average^ less

than a third of the energy going into che boiler of a thermal electric

powerplant emerges at the bus bar in the form. of available electrical

output. Nuclear plants now being built approximate the general nation

wide average of 32 percent power generating efficiency, while the very

best steam electric plants attain an efficiency level of only 40 percent. Moreover, from 5 to 10 percent of the generated electricity is

lost during transmission. Every effort must be made to increase the

conversion efficiency of our power generating and transmission methods

Chief options that need to be pursued include

o improving the methods of upgrading fossil fuels;

o developing advanced power cycles;

o improving fossil fuels conversion processes: fuel cells^ MHD,

thermionic, and thermoelectric;

o developing more efficient nuclear breeder reactors;

o developing new energy systems: nuclear fusion, tidal, wind,

and solar; and

o developing unique energy transmission systems: electricity at

cryogenic temperatures and natural gas and hydrogen moving in

liquefied form through pipelines.

B-13
'

. ENERGY RESEARCH PROJECT SELECTION

The preceding section outlined some of the research needed, if we

are to meet the Nation's future demands for clean energy. Energy

Research can continue to make many important contributions. However,

certain boundaries on our activities and establishment of priorities

are required, if our expertise is to be effectively utilized.

To establish a conceptual framework for our role, research might

be viewed as a continuous flow of information that begins with basic

discoveries and continues through corriniercial development. Such a flow

can be illustrated as

Within this framework, our research covers some of the basic area,

the applied area, and some of the engineering area, all of which fall

within the boundaries established by our mission concept. Full development of any discovery to commercial practice is rarely pursued, as is

basic research which seeks "knowledge for the sake of knowledge alone."

The problem-oriented research we conduct is both useful and essential

for the development of new processes and products needed by our society,

whether for economic, cultural, or military applications. Ordinarily,

industry supports only that research which has a practical aspect--

that which leads to increased profit margins and new corporate earnings. Moreover, this is usually on a short range payout basis, generally

less than 5 years. General guidelines to determine whether a particular

B-14
research effort is suitable for action by the Federal Government are

detailed in Appendix B.

If a particular activity is judged to be suitable for research

by the Federal Government^ the proposed research will be considered

appropriate for Energy Research, provided it also receives affirmative

judgments in three major regards: (1) The research represents progress

toward accomplishing our assigned mission, (2) the research is unique

or complementary in reference to prior, ongoing, or anticipated research

by others, and (3) adequate staff and other resources can be provided

to accomplish the research in a timely fashion. If these criteria

are met affirmatively, detailed proposals will be developed for consideration in our proposed budget. Priorities for the proposed work

will be established by the Assistant Direc tor- -Energy in consultation

with the appropriate Division Chief and Research Director.

Ordering of priorities requires and will receive continual review

as the results of ongoing programs are evaluated and as new ideas prove

worthy of increased funding support. Our best assessment of current

program emphasis is contained in the following section; this list will

be revised as needed to guide our overall efforts toward the attainment

of our mission.

B-15

FORM NBS-n4A (1-71) U.S. DEPT. OF COMM. 1 . PU BLIC AT ION OR RE PORT NO. 2. Gov't Accession

BIBLIOGRAPHIC DATA No. SHEET NBSIR 73-245

3. Recipient's Accession No.

4. TITLE AND SUBTITLE

Toward New Concepts of Management Applicable To The

II S Bureau of Mines' Eneersv Research Proeram

5. Publication Date July 1973

6. Performing Organization Code

7. AUTHOR(S)

Robert S. Cutler, Mary Ellen Colby

8. Performing Organization

NBSIR 73-245

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, D.C. 20234

10. Project/Task/ Work Unit No. 4314420

11. Contract/Grant No. WFA-S0133054

12. Sponsoring Organization Name and Address

U. S, Bureau of Mines

Department of the Interior

Washington, D. C. 20240

13. Type of Report & Period otaLUS Kepoi L April - June 1973

14. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

16. ABSTRACT (A 200-word or less factual summary of most significant information. If document includes a significant

bibliography or literature survey, mention it here.) This report describes the initial progress of an effort to develop new concepts

of management applicable to the U. S. Bureau of Mines' program of Energy Research. A

systems analysis task group was organized to identify principal problems associated

with inplementing a miss ion -oriented research program, and in particular to design

a mechanism for determining research projects priorities in terms of stated goals.

The Phase I Exploratory Survey, consisting of 14 group interviews at the

Washington Headquarters and at two operational Research Centers, is detailed. Results

thus far include listings of acknowledged "problem areas" where reasonable improvements

are considered possible. In addition, a methodology analogous to the "peer-review"

procedure used elsewhere in government for evaluating research projects in terms of

"technical quality" and "program relevance" was introduced and tested in a simulation

exercise. Preliminary results suggest need for additional work before this mechanism

can become a useful management tool for budget review purposes. A second exercise

is recommended involving all six USBM Energy Research Centers in conjunction with

a forthcoming Mid-Year Budget Review.

7. KEY WORDS (Alphabetical order, separated by semicolons)

Energy research; Program Management; R^D management; systems analysis; Organizational

Development; peer-review; program evaluation.

18. AVAILABILITY STATEMENT

1 1 UNLIMITED.

19. SECURITY CLASS

(THIS REPORT)

UNCL ASSIFIED

21. NO. OF PAGES

69

XXI FOR OFFICIAL DISTRIBUTION. DO NOT RELEASE

TO NTIS.

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

22. Price

USCOMM-DC 66244.P71