A UNITED STATES DEPARTMENT OF COMMERCE

PUBLICATION

\

NBS TECHNICAL NOTE 761

Project

A Systems Approach to

Biomedical Research

Program Management

U.S. EPARTMENT

OF

COMMERCE

National

of

is 100

U5753

A Case Study
NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards1 was established by an act of Congress March 3, 1901. The Bureau's overall goal is to strengthen and advance the Nation's science and

technology and facilitate their effective application for public benefit. To this end, the Bureau conducts research and provides: (1) a basis for the Nation's physical measure- ment system, (2) scientific and technological services for industry and government, (3) a technical basis for equity in trade, and (4) technical services to promote public safety. The Bureau consists of the Institute for Basic Standards, the Institute for Materials Research, the Institute for Applied Technology, the Center for Computer Sciences and

Technology, and the Office for Information Programs. THE INSTITUTE FOR BASIC STANDARDS provides the central basis within the United States of a complete and consistent system of physical measurement; coordinates

that system with measurement systems of other nations; and furnishes essential services leading to accurate and uniform physical measurements throughout the Nation's scientific community, industry, and commerce. The Institute consists of a Center for Radiation Research, an Office of Measurement Services and the following divisions: Applied Mathematics — Electricity — Mechanics — Heat — Optical Physics —

Linac Radiation - — Nuclear Radiation - — Applied Radiation - — Quantum

Electronics :i — Electromagnetics ' — Time and Frequency 3 — Laboratory

Astrophysics ' — Cryogenics 3

. THE INSTITUTE FOR MATERIALS RESEARCH conducts materials research lead- ing to improved methods of measurement, standards, and data on the properties of

well-characterized materials needed by industry, commerce, educational institutions, and

Government; provides advisory and research services to other Government agencies; and develops, produces, and distributes standard reference materials. The Institute consists of the Office of Standard Reference Materials and the following divisions: Analytical Chemistry—Polymers—Metallurgy—Inorganic Materials—Reactor

Radiation—Physical Chemistry.

THE INSTITUTE FOR APPLIED TECHNOLOGY provides technical services to pro- mote the use of available technology and to facilitate technological innovation in industry and Government; cooperates with public and private organizations leading to the development of technological standards (including mandatory safety standards), codes and methods of test; and provides technical advice and services to Government agencies upon request. The Institute also monitors NBS engineering standards activities and

provides liaison between NBS and national and international engineering standards

bodies. The Institute consists of a Center for Building Technology and the following

divisions and offices: Engineering and Product Standards—Weights and Measures—Invention and

Innovation—Product Evaluation Technology—Electronic Technology—Technical Analysis—Measurement Engineering—Building Standards and Code Services4—Housing Technology4—Federal Building Technology4—Structures, Materials and Life Safety4—Building Environment4—Technical Evaluation and

Application4—Fire Technology.

THE INSTITUTE FOR COMPUTER SCIENCES AND TECHNOLOGY conducts re- search and provides technical services designed to aid Government agencies in improving cost effectiveness in the conduct of their programs through the selection, acquisition, and effective utilization of automatic data processing equipment; and serves as the principal focus within the executive branch for the development of Federal standards for automatic data processing equipment, techniques, and computer languages. The Center

consists of the following offices and divisions: Information Processing Standards—Computer Information—Computer Services —Systems Development—Information Processing Technology.

THE OFFICE FOR INFORMATION PROGRAMS promotes optimum dissemination and accessibility of scientific information generated within NBS and other agencies of

the Federal Government; promotes the development of the National Standard Reference Data System and a system of information analysis centers dealing with the broader

aspects of the National Measurement System; provides appropriate services to ensure

that the NBS staff has optimum accessibility to the scientific information of the world, and directs the public information activities of the Bureau. The Office consists of the following organizational units: Office of Standard Reference Data—Office of Technical Information and

Publications—Library—Office of International Relations.

1 Headquarters and Laboratories at Gaithersburg, Maryland, unless otherwise noted ; mailing address Washington, D.C. 20234.

• Part of the Center for Radiation Research.

3 Located at Boulder. Colorado 80302.

• Part of the Center for Building Technology.
!H% u -

6 1973 To Project SOAP: A Systems

- Approach to Biomedical Research

Program Management • a case study

Robert S. Cutler

Technical Analysis Division

Institute for Applied Technology

/fJ.J< National Bureau of Standards '" Washington, U.C. 20234

NBS Technical Notes are designed to supplement the Bureau's regular publications program. They provide a means for making available scientific data that are of

transient for limited interest. Technical Notes may be

listed or referred to in the open literature.

/ w

Ml

*.

U.S. DEPARTMENT OF COMMERCE, Frederick B. Dent, Secretary

NATIONAL BUREAU OF STANDARDS, Richard W. Roberts, Director

Issued April 1973
National Bureau of Standards Technical Note 761

Nat. Bur. Stand. (U.S.), Tech. Note 761, 75 pages (Apr. 1973) CODEN: NBTNAE

For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C. 20402 (Order by SD Catalog No. 013.46:761). Price $0.95 domestic postpaid or $0.70 G.P.O. Bookstore
Project SOAP: A Systems Approach to

Biomedical Research Program Management

A Case Study

SYNOPSIS

This report documents a case study in the field of

management science in which a systems approach resulted in

the implementation of a discipline for managing biomedical

research programs within a Federal research support agency.

As such, it has value for practitioners of the systems

approach in itself, and also as a potential guide for

Science Administrators and Budget Analysts elsewhere in

Government. It describes a method for allocating funds to

basic/applied research activities, particularly the planning

and evaluation of publicly "funded project grant programs.

Project SOAP (an acronym for Systems/Operations Analysis

of Programs) was the name given a joint task group established

under an interagency agreement between the Technical Analysis

Division (TAD) of the National Bureau of Standards (NBS) and

the National Institute of Allergy and Infectious Diseases

(NIAID) of the National Institutes of Health (NIH)

This case study describes the methodology and substantive information developed during Project SOAP's twoyear assignment (from July 1969 to June 1971), which

resulted in the introduction of a discipline for program

management for improving the administration of basic/applied

research project grants supported by NIAID. Although application of the quantitative methods of operations research was

iii
not possible during this effort, the basic logic of the

classical resource allocation model was used as a framework

for guiding management decisions.

The task group was created to explore the applications

of systems analysis and operations research to management

issues within the National Institute of Allergy and

Infectious Diseases, particularly to the problem of resource

allocation in its Extramural grants program. Together,

the task group investigated the methods and criteria the

Institute used for selecting projects within its various

research programs.

Initially, it was found that the Institute's traditional

administrative practices for budget allocation did not

provide an adequate basis for managerial decision-making,

since attempts to establish program priorities were

ineffective. The agency was not able to justify to higher

authority its desired choices when arbitrary budget cuts

imposed by external circumstances seriously impaired

important programs. The task group concluded that the best

solution was the adoption of an explicit discipline for

management: a discipline which would permit more effective

control over what had become a major cause of friction and

confusion within the Federal-Science environment. This

"dissonance" stemmed from the presence of two value systems:

the "scientific merit" criteria (an emphasis of the Scientific Community) , on the one hand, and the notions of public
interest (here "health relevance" criteria, as conceived by

the National Advisory Council), on the other.

The benefits which accrue to this Institute's Science

Administrators from the full implementation of such an approach include not only greater assurance of "optimum"

resource allocation but also a basic pattern of reasoning

useful for guiding discussions with members of the

Scientific Community, representatives of other Federal

health research agencies, and the Office of Budget and

Management.

Moreover, it has become evident in recent years that

the single "peer review" procedure for selecting grant

payment priorities cannot be expected to be responsive to

changing National priorities unless stronger incentives are provided by Science Administrators for focusing research

investigators' attention on nationally important health

research problems. The DAPA (Dollar Allocation by Program

Area) research investment procedure recommended by the task

group is simply a more deliberate mechanism through which

the Science Administrator and his Advisory Council can use

the advice of experts from the Scientific Community jointly

with other considerations in the process of establishing

grant payment priorities for the Institute.

The author favors the establishment of more clear-cut

and explicit planning and evaluation procedures for the

management of research programs within the Federal Government.

v
Present efforts at decentralizing the management of NIH's

biomedical research activities, with their emphasis on Institute determination of priorities aimed at particular

National health problems, are a start in this direction.

The author also believes the approach described in

this report can be used to provide Science Administrators

elsewhere in Government with an approach for explicitly

planning for and evaluating the performance of their

basic/applied research programs, in order to balance the

dollar costs with the expected benefits of particular

research project proposals.

Within the National Institute of Allergy and Infectious

Diseases, the Extramural Program science administrators are currently in the process of adopting the DAPA procedure to

perform this function.
ACKNOWLEDGMENTS

A project of this nature depends upon the cooperation

and assistance of many people, both from within and outside

our respective organizations. We acknowledge our grateful

appreciation to many individuals for their helpful contribution to various activities of this joint task group.

Our special gratitude is extended to Dr. Alfred M. Webb,

whose guidance and forebearance were principal ingredients

in the conduct of this project, and to Dr. C. West Churchman,

Professor of Business Administration, University of

California, and member of the NIAID Advisory Council, who

originally suggested the Institute explore the use of

systems analysis as a means for improving its allocation

of resources.

Vll
TABLE OF CONTENTS

Page

SYNOPSIS iii

1 . INTRODUCTION 1

1 .

1

The Necessity for Change 5

1 .

2

Task Group ' s Approach

1.3 Project SOAP Orientation 9 1 .

4

Purpose and Scope 10

1.5 The Initial Task 12

1.6 Systems Description 15

1.7 The Desired System 16

2. THE EXPLORATORY PHASE 22

2.1 The Institute's Organization 23

2.1.1 Collaborative Research Program

Operations 23

2.1.2 Extramural Research Program

Operations 25

2 .

2

Grant Allocation Procedure 27

2 .

3

Program Relevance Concept 29

3. THE DIAGNOSTIC PHASE 31

3 .

1

The Preferential Payment Issue 31

3.1.1 Study Section Priority Scores 33

3.1.2 "Program Relevance" Criteria 35

3.1.3 Preferential Payment Procedure 36

3.2 Preliminary Study of "Health Relevance"

Assessment (Exercise I) 37

3.3 Biomedical Research Relevance Criteria

(Exercise II) 38

4

.

THE PRESCRIPTIVE PHASE 41

4 .

1

The Program Management Concept 41

4.2 Recommended Criteria for Extramural

Program Management 44

4.3 Research Investment Procedure: Dollar

Allocation by Program Area (DAPA) 45

4.3.1 The DAPA Procedure 48

5. CONCLUSION 54

REFERENCES 59

APPENDIX

viii
Project SOAP: A Systems Approach to

Biomedical Research Program Management

(A Case Study)

Robert S. Cutler

This case study describes the activities of an interagency task group that applied systems analysis to

improve management controls within a biomedical

research agency of the federal government. The

results were the formulation and implementation of

a discipline for program management which explicitly

makes use of multiple criteria in arriving at

resource allocation decisions.

The text details the necessary preliminary analysis

describing operational activities, information flows,

and key decision points within the organization. It

goes on to identify the techniques employed and the

difficulties encountered while attempting to improve

the decision-making process for selecting research

projects, under conditions of reduced funding. In

particular, a comparison is made between: (1) the

agency's traditional single-criterion "peer review"

judgment for determining budget priorities, and

(2) the multiple-criteria judgments required to

effect more positive management control. The systematic use of separate "scientific merit" and

"health relevance" ratings is compared with the

organization's actual experience. The criteria

used by various participants in the decision process are analyzed, and a dollar allocation "investment" procedure based on these findings is developed.

A procedure which organizes relevant information for

research program planning and evaluation is presented,

and extension of this recommended procedure to wider

use by science administrators elsewhere in government is discussed.

Keywords: Biomedical research; "health relevance";

program planning and budgeting; R§D management;

systems analysis; systems approach.

1. INTRODUCTION

The National Institute of Allergy and Infectious

Diseases (NIAID) is one of the ten National Institutes of

Health. It supports a broad range of biomedical research on the causes of allergy and the prevention and treatment of

1
diseases associated with such infectious agents as bacteria,

viruses, fungi, and parasites.

The budget for the National Institute of Allergy and

Infectious Diseases in recent years has been approximately

$100 million a year. Half of this--some $50 million--is

appropriated by the Congress for Extramural research and

training grants which support biomedical research investigators at academic institutions throughout the United States

and in some foreign countries. Of the approximately 1,700

grant applications received each year by NIAID, nearly one- half are non-competitive continuations; i.e., they are longterm commitments made by the Institute in previous years and

are based on prior review and National Advisory Council*

approvals

.

Of the remaining 800 to 900 applications, about

70 percent are judged by a peer review procedure to have

sufficient scientific merit. However, after the prior year's

commitments have been paid from the allotted $50M, only some

*The National Allergy and Infectious Diseases Council consists

of a group of 13 to 15 prominent biomedical scientists,

public health officials, and laymen appointed by the

Secretary of the Department of Health, Education and Welfare

for a term of four years. This Council was created to advise

the Director of NIH on policy matters. Moreover, it has the

statutory responsibility of approving recommendations for

research project grants (through the Director of NIH) to the

Surgeon General of the United States. Only with the

Council's express approval can awards for Extramural project

grants be paid by the Institute. Formal meetings of the

Advisory Council are scheduled three times during each fiscal

year, in November, March and June. Those Extramural grant

applications initially accepted by a peer review group and

favorably evaluated by the Institute Staff and the appropriate Council subcommittee are finally recommended for payment.
two-thirds of those remaining grant applications recommended

for award can be paid.

As early as 1963, when the previous large increases in

NIH appropriations first began to level off, there was much

discussion among the NIAID Staff and Advisory Council members

about the desirability of giving special priority to certain

areas of research for support by Extramural Grants. It was recognized that such areas of research needed more support

than other fields because they dealt with problems of particular importance, or held promise, or had been neglected in

the past.

In 1964, when the NIAID Extramural Grants budget was actually restricted, a new and serious problem confronted

the Institute: how to assure the means for support for

those research activities which the Institute considered

most relevant to its categorical mission of improving public

health through the support of basic science and the performance of targeted research. The problem was not so much

deciding how to determine the most scientifically meritorious

projects from among those screened by the initial review

groups (Study Sections) and referred to NIAID for financial

support but, rather, deciding how to determine the preferred

order of payment of those research project proposals

approved- -particularly as research budgets declined- -and how

to make this kind of decision on a more rational and systematic basis.
Finally, in 1967, faced with the practical problem of

preserving support for important biomedical research projects

while at the same time faced with an actual cut in budget for

Extramural Grants, NIAID responded by designating, from among

the many possible biomedical research areas, ten Special

Emphasis Research Programs* (SERPs) as the ones to receive

priority. Grants associated with these areas would be given

a "two-decile"** advantage in payment priority over grants

associated with other areas of research.

This procedure became the subject of much discussion

within the NIAID management. It was called "decile-diddling"

by some, and there were increasing demands for justifying

the allocations resulting from such informal procedures.

In essence, the rationality of the allocation process

itself became an important issue.

The result of all this was that, in 1968, NIAID

agreed to take a more sophisticated look at various

methods for determining grant payment priorities in the

*The initial NIAID Special Emphasis Research Programs were:

Drug Resistance and Microbial Diseases; Streptococcal

Infections and Sequelae; Congenital Defects caused by

Microbial Agents; Antiviral Substances; Chronic and

Degenerative Diseases; Infectious Hepatitis; Emphysema

and Chronic Lung Disease; Transplantation Immunology;

Clinical Allergy and Immunology; and Malaria. NIAID Staff

Paper (June 5, 1967) . **For explanation, see pp. 22, 27-29.
Extramural program, a decision largely the result of the

appointment of Dr. C. West Churchman* as a lay member to

the NIAID Advisory Council.

1 . 1 The Necessity for Change

When the change in Federal budget priority for supporting

NIH Extramural research resulted in the uniform cutbacks of

1969,** it became obvious to many within the Institute that

NIAID' s external environment was changing and that perhaps

their basic decision criteria ought to be examined in a different light.

At the same time, after several Advisory Council meetings

it became clear that different interpretations were being made

by different Council members of the Study Sections' (peer

review) ratings of grant requests. That this ambiguity

resulted in the difficulty when these priority scores were

considered by the Council was apparent, in particular, from

the way final payment priorities for certain grants differed

from those intended in ratings of the Study Sections. Some

members of the Advisory Council felt that the use of Study

Sections' priority scores together with the decile adjustments

*Dr. C. West Churchman, Professor of Business Administration,

University of California, Berkeley, and a leading exponent

of the "systems approach."

**In July 1969, NIH instituted an across-the-board 20 percent

cut in research grants up for renewal. Later, because of

vociferous reaction by the biomedical research community,

DHEW reduced the budget cuts to 5-10 percent.
recommended by the Institute Staff did not result in their

notion of a balanced research program, or at least did not

result in a program the rationality of which was evident.

In seeking a change for the better, the Institute began

to address itself to developing more explicit goals and

criteria for evaluating the performance of its programs.

The existing ones were found to be too vague and inappropriate for planning and guiding its decision-making process

in relation to its public responsibility of improving the

nation's health through research.

Dr. Dorland J. Davis, the Director of NIAID, best

summed up the problem when he asked: "What actions should

the Institute take to assure the most effective and

responsible use of these funds entrusted to its stewardship?" Further, he posed, "How can we be assured five

years from now--with present budget allocations- -that some

of our best [research] programs won't be starving, and

our worst programs flourishing? Our goal ought to be to

improve the management of our programs , not to perpetuate

the controls we now exercise."*

In an attempt to provide this desired improvement,

Project SOAP was thus established as a joint task group

under an interagency agreement between the Technical

Analysis Division (TAD) of the National Bureau of

*Initial meeting with Project SOAP, August 8, 1969.

6
Standards (NBS) and the National Institute of Allergy

and Infectious Diseases (NIAID) of the National Institutes

of Health (NIH)

The objective of SOAP (an acronym for Systems/Operations

Analysis of Programs) was to explore the application of

systems analysis and operations research to significant

management problems of NIAID, particularly to those of its

Extramural Research Grants Program.

The title of the report prepared for the sponsor

at the conclusion of the assignment, "Toward a Discipline

for Program Management," * epitomized the basic discipline

used by the Project SOAP task group. This discipline

introduced to research scientists and science administrators

at NIAID a more consistent and rationally-based framework

for resource allocation. The "Program Management Discipline"

relates operational decision-making information to higherlevel considerations of health research and public policy.

It is strategic in nature; it attempts continually to

reexamine goals as well as to determine the appropriate

means for achieving the desired degree of managerial control.

Today, in practically every agency of the Federal

Government there are operations researchers, management

scientists, system scientists- -all attempting to look at

*Cutler, R. S. and Martino, V. A., "Toward a Discipline for

Program Management" (A Summary of Project SOAP Activities,

FY 70-71), National Bureau of Standards Report No. 10-626

(January 31, 1972)
the problems of Government from the so-called "systems

viewpoint." As scientists by profession, they are interested in characterizing the nature of operational

systems in such a way that the decision-making function can

be performed in a logical and coherent fashion. Furthermore,

by applying scientific discipline and substantive knowledge

to a particular field of activity, they expect to be able

to develop measures which will give as adequate information

as possible about the performance of these systems.

This report documents one such systems analysis study.

No claim for a final successful solution is made. Rather,

the following text describes both some specific "subsystem"

analyses and the steps taken toward delineating a management

decision-making structure within which the Institute itself

later began to undertake more comprehensive and definitive

programs. An abstract of the Official Minutes of the

June 1971 Council meeting describing the "first step in

implementation of the Dollar Allocation Procedure" is shown

in the Conclusion on page 55.

1 . 2 Task Group's Approach

The Institute's interest in this systems analysis project was centered in the top echelon of the organization.

The Institute Director assigned two men* from his staff to

*Dr. Alfred M. Webb, Chief, Office of Program Planning and

Projections (NIAID); and Charles Myers, Management Analyst

(NIAID), later reassigned.

8
join on a half-time basis the two full-time systems analysts

from TAD.* This joint task group and their subsequent

activities became known as Project SOAP.

The SOAP group received support and encouragement

from NIAID's Director, Dr. Dorland J. Davis, who arranged

the necessary introductions and scheduled meetings with

professional staff operations personnel. Thus, the group's

entree to the operational levels of the organization was assured

1.3 Project SOAP Orientation

Initially, there were orientation problems to resolve.

Because the scope of the assignment and the objectives of

the study were not clearly definable in advance, NIAID's

management personnel were uncertain about what to expect in

terms of project results. A portion of the problem lay

with the Institute's unfamiliarity with systems analysis

and operations research terms and techniques. Similarly,

the systems analysis people from TAD were unfamiliar with

biomedical research. Thus, there were mutually recognized

difficulties in achieving useful results within the allotted

time. Fortunately, however, there was also a sincere resolve

on both sides to make a meaningful and productive start.

*Robert S. Cutler, Project Leader, and Vincent A. Martino,

Operations Research Analyst, Technical Analysis Division,

National Bureau of Standards.
In the beginning, the SOAP task group spent much of its

time interacting with various key members of the NIAID

organization. The discussions were quite candid, and one

charge, frequently reiterated, emerged: that the TAD people

were lacking in biomedical knowledge. Acknowledging this,

the TAD analysts pointed out in response that the real issues

of concern were in the area of management control and that

expertise and "discipline" in this field was what they

brought to the project.

1. 4 Purpose and Scope

The primary aim of this TAD/NIAID study was to explore

the application of systems analysis and operations research

to the principal management problems of the National

Institute of Allergy and Infectious Diseases, particularly

to its problem of allocation of resources.

The scope of the assignment included two operational

divisions within the Institute: the Collaborative Program

and the Extramural Program. (The third division, the

Intramural Program, was not included in the Project SOAP

study because it was undergoing a reorganization at the

time.) Major attention was to be given to the Extramural

Program, which is responsible for awarding research project

grants

.

10
It was expected that this assignment would also create

within the organization the broader perspective require& to

coordinate the management of some of the Institute's other

biomedical research activities.

Specifically, the joint task group was organized to:

1. Review the process for awarding Extramural

Project Grants and study the impact of the

existing allocation procedure;

2. Develop improved selection criteria and

management methods for use by the Institute's

Staff, its management, and its Advisory

Council

It was anticipated that systems analysis techniques-

such as PERT (Program Evaluation/Review Technique)* for

network planning of Collaborative research, and quantitative

methods for measuring research accomplishments- -would lead to more effective management procedures. These, in turn, would

provide a basis for integrating, within NIAID, closely asso- ciated research activities into Institute-wide programs--an

approach already under serious consideration by the NIAID

Director at that time. It was further expected that if the

tools provided by management science proved effective in

handling the NIAID Extramural grants program they might,

similarly, prove effective for other NIH Institutes, and

perhaps even for NIH as a whole.

^A basic tool of management science used for detailed plan- ning and control of complex program elements.

11
a The initial work assignment for the Technical Analysis

Division began with a one and one-half man-year effort. As

the project became operational, additional support was added.

The level of effort in the first year was $93,000. The project was continued in June 1970 for the second year at

$130,000, to which an additional $33,000 was added during the

last quarter of the fiscal period. In all, approximately

seven man-years and a total of $263,500 were expended during

this two-year assignment.

1.5 The Initial Task

The first task to be tackled by Project SOAP, before any

meaningful systems analysis could be performed, was the

development of a primary information base. They had to find

out what functions the various operational programs were

currently performing; what kinds of information key decisions

depended upon; and, in some sort of logical fashion, to

describe what this "present system" consisted of. Each part

of the "system" did not exist in isolation but, rather,

within a context, or environment, that had to be explicated.

Elements of "environment" were originally classified as "internal" (the NIAID context) , "external" (the NIH context)

,

or "Institutional" (the real world context) , a term which

ultimately describes the relationship of both NIAID and NIH

to the "outside world."

12
In setting up this systems framework, organizing the

work tasks, and isolating the kinds of problems amenable to

solution, the task group found that flowcharts, such as that

illustrated in figure 1 (following page) , were useful for

providing effective communication. They provided a visual

means for describing the process under investigation and for

identifying the loci of specific problems.

The management problems identified in the NIAID organization could be isolated thus: 1) "operating problems,"

i.e., problems dealing with routine internal operating

situations (such as the processing of approved grants)

2) "strategic problems," i.e., problems relating to the

anticipated changes in the external environment (such as program planning and evaluation); and 3) "policy problems,"

i.e., problems concerning long-term trends in the NIH

institutional (outside world) context (such as health

research priorities)

As previously stated, the primary aim of Project SOAP

was the application of the techniques of systems analysis

to the appropriate "problems," once the problems themselves

were identified. Where necessary information about the

operational effectiveness of particular programs was obscure

or even lacking, the introduction of a qualitative, organizing framework was often all that could be immediately

provided; but even such first steps proved far from

worthless

13
CO

CDO

OC

Q_

U_o

CO to <n to (O >= E

_i (A o <

z.< Q

COO o

0_c CO u

CD E z C a. O T3 o

a> 6 O >Q

LC o> o3 hJ CD Q. <f o O o \ o CD

CO CO <D a> ?

Lil to o

1- _c

T3 co "5 E

>- o o

CO

-z.

m

a.<o

CO

1-o

Ll! ~Do

(Z

Q_

COcoo > u o Q.

O o < 3

of c o

V, \(ft c en

in

0) \ o c c \ c < oc

COE

4)

LlJ o CO T(/> T \- > o X lil •3 UJ 1 O 1- m rr z o a. o rr a. o 0. <

UJ UJ ui X j. T H h- H

a

.2 o a <- =» UJ 1 5

_i to UJ

a.a.<

en

UJ3OzIo w to

co

m

UI or

F

d * 1

=»

CO c a > S3a. co

T3

CO "O CD i

<*

CO c CT> 2= Si •^ co to Q < 11a. to zo

1-<Z 10 »- 0) UJ

a:O

ment

g

ach res o

mane

co c »S»o

II• ;?

a: a. o cl

a t £

UJ

_i

CQ

O Q. CDm o < 5 a.

O

a.

"DllO <o

y) o>

id

to

0) «- c to a> c to E.E

<D

C

co o £ =

Objectiv Constrai Variable Criteria

a:

Q E

CO

co

CD _>»

CO o

>i c

CO < o en

as E

ui t» <D "55 Q >.

C7J Zo

1-z E UJ <D

— "7* a.

a: 1- <= Q.

(ftO 3 3 o >»:c o O < w .9-<£ K ^_-j

i- c o ^ a> to o) * Q z w ® o CO <

UJ

-Jo • OF

at "-" ps Z

QJ>i3

bo

UJ £ - Jo i * 3 ^ § 5 i J

14
The "cost/effectiveness" part of systems analysis was necessarily deferred in favor of laying a foundation for

determining more precise objectives and criteria, and for

obtaining better bases for more policy-relevant data. In

the case of the Extramural program, this kind of detailed

examination served to focus the Institute's attention on particular information gaps in its primary decision process.

1 . 6 Systems Description

The primary accomplishment of the initial three-month

exploratory phase was what might be termed an "inputoutput-effect" analysis of two organization divisions of

NIAID: the Collaborative Program and the Extramural Program,

The results obtained were functional descriptions of the

"existing systems," which formed the basis for identifying

"problems" and formulating requirements for the "desired

systems." Also prepared were descriptive outlines which

clarified the meanings of words used by the people making

the operational decisions in each program and which documented, in operational terms, the basic activities actually

performed. Included were carefully worded brief statements

of program goals, objectives, inputs/outputs, and associated

performance measures. The attempt here was not as yet to

seek to make improvements, only to gather the kinds of

information required for "Program Management," in order to

accurately describe the decision process.

15
For example, a decision-oriented information flowchart

(see figure 2, following page) was prepared to describe the

existing operations of the Extramural Program. The level of

detail reflected the understanding of the Institute's

Associate Director for that program of what kinds of operational functions were both necessary and sufficient to

define his principal decision processes and to relate the

"inputs" and "outputs" of the Extramural Program to other

organizational components. (A detailed description of these

operations, including flowcharts, is contained in Project

SOAP Notes No. 3 and 4.*)

1 . 7 The Desired System

As the current operations became more clearly understood by both the NIAID managers and the SOAP task group,

the next question to be answered was: What ought to be the

"desired system"? Not unexpectedly, the task group ran into

difficulties. By focusing attention on the problem of

"What should the system be accomplishing"? the group automatically raised the next question: "What are the criteria

for measurement of performance?" In other words, they

wanted to know WHAT the organization was attempting to

achieve and HOW it actually evaluated its degree of success.

*"Present System Description, Collaborative Research Program," Project SOAP Notes No. 3, December 15, 1969.

"Extramural Program: Research Grants Award System

Description," Project SOAP Notes No. 4, January 30, 1970

16
Hi!I!

17
And these questions raised additional questions about WHOM

the Institute serves and WHAT are the consequences of its

actions?- -all important concerns to a management responsible

for a $100 million-a-year enterprise.

In this way, the SOAP task group began to lay the

framework for an Institute-wide program management system

(the "desired system") . This approach, it was hoped,

would also eliminate many of the existing internal communications barriers between the Intramural, Extramural and

Collaborative divisions by integrating some of their

separate efforts toward a common goal: the development of

biomedical research programs designed to be more responsive

to national health research needs.

The attainment of this paramount goal, it was realized,

would require considerable effort. The Institute's basic

management functions- -planning , organizing, and evaluating--

would have to be strengthened, and formal methods and

techniques to assist decision-making at various levels would

have to be developed.

More specifically, the fundamental management problem

confronting NIAID was "allocation of resources." The

Institute was searching for a quantitative method for

assessing the relevance of both its major research programs

and its individual project proposals in terms of its

explicit public health mission. During the initial task

group orientation, for example, the following memorandum,

18
by the Assistant Scientific Director for Collaborative

Research to his staff, described the type of management

system desired by the Institute for evaluating its

Collaborative Research Programs in terms of NIAID

objectives

a) Collaborative programs should be extensions

of Intramural Research at NIH, to assure the

availability of competent Project Officers.

b) Collaborative programs should be coordinated

with the Extramural and Intramural Programs

of NIAID, with other NIH Institutes and

Government agencies, and with any similar

research activity in the private sector.

Where major technological advances are sought, the interests of prospective user organizations should be an important

consideration.

c) Full use of Government, industry, and academic

scientists should be made in obtaining advice

and assistance in defining the scientific

base, goals, and feasibility of new research

programs

d) Programs must have major Institute review and

evaluation at intervals not to exceed five

years

e) Contract Review Committees are charged with

assessment of technical proposals, contractor's

competence and facilities, and reasonableness

of budgetary cost and time estimates.*

Furthermore, because of the large number of research

proposals which come to NIAID in the form of Extramural

Grant applications, or Collaborative contract proposals, or through the research interests of senior scientists on its

Intramural Staff, NIAID desired a more systematic means for

*Memorandum, Dr. John R. Seal to Dr. Dorland J. Davis,

Director NIAID, "Role of Scientific Director, Advisory

Committees, Branch Chiefs in Collaborative Research,"

February 9, 1970.

19
judging what it termed "program relevance": the importance

to the Institute of supporting a particular project proposal.

It believed that a quantitative method for evaluating program performance in general would offer the potential for

improving its current process for selecting research projects and establishing priorities. The sheer volume and

complexity of this type of decision-making defied intuitive

attempts to arrive at an optimum solution. Nevertheless,

definite choices and hard trade-offs were being made--

albeit, under tacit assumptions about what was and was

not "relevant."

Therefore, the Project SOAP task group set forth to

create an awareness of what a systems analysis capability

within the NIAID Institute could do for facilitating the

development of improved management techniques while, at

the same time, promoting the wider acceptance of such new procedures throughout the Institute. While there were some

signs after the first year (June 1971) that significant

changes in NIAID' s management philosophy had taken place,

the ultimate effectiveness of the recommended DAPA procedure

developed by the SOAP task group has yet to be realized.

The Institute has recently (March 1972) begun to emphasize the

planning and evaluation of overall programs (rather than the mere review of the details of individual projects), and it

ft

*DAPA: Dollar Allocation by Program Area- -to be fully

explained in the following pages.

20
has apparently gained confidence in entering more actively

into the proposed Dollar Allocation method for guiding

investment in socially relevant areas of health research.

To come as far as it did, Project SOAP used an approach

which involved three major phases--an Exploratory Phase,

a Diagnostic Phase, and a Prescriptive Phase--which are described in the following sections of this report.

21
2. THE EXPLORATORY PHASE

The Project SOAP task group commenced its assignment with

a review of the Collaborative Program. This operational program consists chiefly of target-oriented research projects

funded by contracts; it also involves some selected Intramural laboratory activities. It was believed that the

techniques of systems analysis- -for instance, Program

Evaluation and Review Technique (PERT) and Critical Path

Method (CPM) Network Planning techniques for organizing

time-dependent and information-dependent activities, and the

development of briefing charts for progress display purposes--

would result in a more effective program management procedure.

The second undertaking involved the Extramural Program.

It included the development of an operational flowchart (see

previously mentioned figure 2) and studies of the implications of the "Decile System"* and of the current allocation

procedure for awarding research project grants, including an investigation of criteria for selecting Special Emphasis projects. The objective was more explicit criteria, leading in

turn to improved operating procedures

.

A brief management overview of the NIAID Institute and

its operating activities and procedures is given next, as background for the subsequent accounts of the two specific

tasks sketched above.

*A ranking method for equalizing the distribution of project

grant priorities recommended by the various Study Sections.

22
2. 1 The Institute's Organization

The operations of NIAID are divided into three major

divisions: the Intramural Program, which includes the

research conducted by the in-house laboratories; the

Extramural Program, which is concerned primarily with

research and training grants awarded to individual

outside investigators; and the Collaborative Program,

which directs, under contract, research with industry

and other institutions. To put it another way, the

operations are divided into three major organizational

activities: General Laboratories and Clinics

(Intramural) , Research and Training Grants (Extramural)

and Nationally Organized Research Programs (Collaborative)

Below are more fully expanded functional descriptions of

the Collaborative and Extramural programs as they

existed when the SOAP Task Group was first introduced

to them in October 1969.

2.1.1 Collaborative Research Program Operations

The Collaborative Research (and development)

Program is one of the three major operational units of

the Institute. It differs from the others in that its

operations are conducted primarily through contracts

with outside companies and research laboratories. Its

role is to direct, coordinate, and provide support for

developmental research on specific biomedical products

and related scientific information.

23
The Collaborative program's major purpose is the

translation of the findings of basic research (biomedical

knowledge) into methods or products for the treatment and

control of allergic and infectious diseases. Specific

objectives are: the development and testing of specific

vaccines against viral and bacterial agents; the production,

control and distribution of biomedical research

reference reagents; and the development of methods and

knowledge about various types of human tissues and their

compatibility with other tissues for use in transplantation

of human organs. In addition, the members of the staff

provide support and scientific management for two international programs: a cooperative medical program with

Japan, and a domestic research and training program

with an international component.

These collaborative efforts are conducted via

the mechanism of research contracts with industry,

universities, federal agencies, and other biomedical

organizations. They are directed by members of the

NIAID scientific and administrative staff, with advice

and counsel obtained from various ad hoc advisory

groups .

24
The Contract Review Group (CRG)*, composed of the

top management team, review each contract proposed for

the development of specific outputs, such as new experimental vaccines. They examine the basic data

and contractors' proposals for development and production,

and then approve those proposals which they find best meet

the Institute's needs in that area, such as the development

of a new vaccine.

2.1.2 Extramural Research Program Operations

The Extramural Research Program provides support,

through the mechanism of research grants, to public

and other non-profit institutions and investigators

to establish, expand, and improve research activities

in the health sciences and related fields.

The existing process for the award of Extramural

research grants begins when applications are received

by NIH from prospective research investigators. Each

application is assigned by the NIH Division of

Research Grants (DRG) to a categorical Institute

*Contract Review Group. Its members are the Institute

Director, the Scientific Director, the Assistant

Scientific Director, the Associate Director for

Collaborative Research, the Associate Director for

Extramural Programs, the Special Assistant to the

Director, the Executive Officer, and the Chief of

Program Planning.

25
(such as NIAID) whose mission most closely involves the proposed research activities, and to one of 55 Study Sections

(discipline-oriented Peer Review Groups) for review and

assessment. This process for selecting recipients for project grants from NIH is widely known and accepted by many

segments of the scientific community. It is similar in

principle to that used by other Government agencies

supporting basic research, like the Atomic Energy Commission

and the National Science Foundation.

(As previously stated, the Project SOAP task group

did not examine the activities of the Intramural Program

because it was undergoing a reorganization at the time.)

The number of programs within these three organizational divisions- - the Intramural, the Collaborative, and the

Extramural- -could total anywhere from ten to about fifty,

depending on the enumerator's choice among various meanings

of the word "program." Activities wholly within the

Collaborative Research Program, such as the Vaccine

Development Program, or the Pneumococcal Vaccine Program,

or the Research Reference Reagents Program, etc., were

frequently spoken of simply as "programs." In the

Extramural organization there were research activities

supported by grants in loosely defined areas called Special

Emphasis Research Programs, such as Transplantation Immunology, Infectious Hepatitis and Antiviral Substances. The

fact was that a kind of vagueness about many of the terms

26
used by members of the Institute- -terms such as "program"-

accounted for the serious semantic difficulties the SOAP

group encountered while attempting to formalize precise

"systems descriptions."

2 . 2 Grant Allocation Procedure

The allocation of resources to Extramural project

grants had been a major concern of the NIAID Institute for

a number of years. Since 1962, the "decile system" has

been used by the Institute as the basis for allocating

grants. It employs a decile-ranking method for equalizing

the distribution of priorities recommended by some ten

to twenty different Study Sections. Following is a more

detailed description of the system as employed by NIAID.

THE DECILE SYSTEM OF THE NIAID

Since 1962, the National Institute of Allergy and

Infectious Diseases has used a "decile" system to

equalize the differing levels of priorities on research grant applications recommended by different

Study Sections. The details of this system and

some of its effects are described below.

In accordance with the policy of the National

Institutes of Health, every Study Section provides

a priority score for each application recommended

for approval by the majority of its participating

members (for grants assigned to NIAID for payment)

This priority score is derived in the following

manner

1. Each participating Study Section member

rates every approved application on the

basis of order of payment from 1 (first

order) to 5 (last order) . Criteria used

in making this judgment include scientific

merit, study design, originality, competence

of investigator, and adequacy of working

environment

27
2. After the meeting, scores for each application are averaged and the result multiplied

by 100 to provide a 3-digit rating.

3. The resulting priority score is recorded on

a confidential "resume" of actions of the

Study Section and is distributed to the

awarding Institutes/Divisions.

When such a "resume" is received by the NIAID, all

priority scores for that Study Section are listed in

numerical order, regardless of the Institute to

which assigned. The list is then divided into

10 equal parts known as "deciles," with the first

decile representing the most favorable priority.

If an application ties with another or falls

between deciles, it receives the more favorable

rating. The decile number is then marked on each

summary sheet sent to the Council. Applications

falling in the 10th decile are not paid unless the

priority is raised by specific vote of Council.

Throughout a fiscal year, awards are made on grants

which fall in the most favorable deciles to the

financial level that can be maintained throughout the

year. Renewal grants in the least favorable deciles

are brought to a termination and those in the intermediate deciles are extended with the expectation that

they will be renewed later in the year if sufficient

funds become available. After the March NAAIDC

meeting, applications in less favorable deciles from

this and prior Council meetings are awarded in order

as long as funds are available. Unless an applicant

withdraws a pending application, it remains in

competition for available funds until the end of the

fiscal year.*

Various methods for managing the allocation of grants

had been proposed, and the one in use partially reordered

the Study Section- imposed priority ranks of approved applications in accordance with Institute preference criteria,

termed "program relevance." This method had not proven

entirely satisfactory to all members of the Advisory Council.

*See also footnote (*) , p. 34.

28
Selected grant proposals in Special Emphasis Research

categories were given an MHPR"* rating, resulting in an arbitrary two-decile escalation and often raising them to

payment status. This practice (facetiously described as "decile-diddling") effectively introduced the Institute

Staff's program preferences but did not provide the Advisory

Council with the kinds of information that several members

felt were needed "to make responsible recommendations."**

2 . 3 Program Relevance Concept

It was primarily this dilemma- -whether to use Study

Section priority scores "as is" or to modify them by the

Institute's program relevance ratings in order to determine

payment priorities- -that led the Institute Director to

appoint a special subcommittee of the Advisory Council to

look into the matter. Specifically, the Criteria Subcommittee was to develop an acceptable concept of "relevance"

*HPR: A rating given a particular project considered by the

Institute to be of "High Program Relevance." The use of

preferential payment designations "HPR," "LPR," and "NPR"

evolved over several Council meetings between the years

1965 and 1969. As the effects of such reordering of

Study Sections' priority recommendations (two decile

levels, up or down) were felt by the scientific community,

and reacted to by the Advisory Council, steps toward modification of this procedure were taken. Finally, at the

March 1970 Council meeting, a motion was carried to discontinue the LPR and NPR ratings until the basic questions

concerning the staff's preferential payment criteria could

be clarified and an improved operating procedure adopted.

**Dr. Churchman, November 1970 Council Meeting.

29
and establish explicit criteria for use in judging factors

associated with "relevance." In addition, they were to

consider an improved relevance rating procedure.

The plain fact of the matter was that the Study Section

members themselves realized that all their time and effort

spent on reviewing a particular grant in a research area might come to naught; regardless of its scientific merit, a particular aspect of research might not get funded if it

were not deemed "relevant" to some mission. Similarly, they

realized, nationally recognized scientists might not receive

support for work they considered extremely important to a particular area because their work was judged "non-relevant,"

while inferior scientists might receive support because their

work was judged "more relevant."

30
3. THE DIAGNOSTIC PHASE

The purpose of this phase was the diagnosis of key

issues requiring policy-level resolution before an analysis

of possible operational changes could be made.

The following is a brief outline of Project SOAP's

identification of three principal factors associated with

the management of the Extramural Program:

Study Section Priority Scores

Program Relevance Criteria

Preferential Payment Procedure

3. 1 The Preferential Payment Issue

In 1966, when the NIAID originally presented its plan

for stimulating and supporting, preferentially through

grants, more original investigations bearing on disease

problems deemed of special importance, its advisory

committees (particularly its Advisory Council) gave consideration to ways in which such a programmatic structure could

be achieved within the Extramural Program.

The following year, the Institute's staff, with the

help of its Advisory Council, identified several areas of

"high program relevance" which they began to call Special

Emphasis Research Programs. They felt these fields of

biomedical research were deserving of special emphasis

within the context of the Extramural Program. At that time,

these areas comprised about 35 percent of grant projects and

31
funds. They were judged to be of greater importance to the

health needs of the nation than other areas. The intention

clearly was not to limit or to direct original scientific

research supported by NIAID grants but, rather, to give

scope to individual investigators in pursuing their preferred

lines of research in a way which would foster advancement on disease problems of greatest national importance.*

The challenge was clear, but the mechanism was not,,

especially during the subsequent fiscal retrenchment which,

for the next several years, resulted in making payment

priority decisions difficult for the Institute.

In the opinion of several outspoken members of the

Advisory Council, the review of any publicly-funded program

requires, at the outset, clarification and agreement on a statement of program goals and priorities. These individuals

have pointed out that such a requirement is of particular

importance when budgets are limited and apparently worthy

efforts must go unfunded.

A substantial number of Study Section participants in

the NIAID Extramural grants award process felt, however, that

considerations other than intrinsic scientific quality ought

not to influence research grant payment. This was confirmed

*Dorland J. Davis, "Remarks on Special Emphasis Programs of

the National Institute of Allergy and Infectious Diseases,"

The Journal of Infectious Diseases, University of Chicago,

Vol. 121, No. 2, February 1970, p. 231.

32
by the study of rating criteria in Exercise II.* One

of the basic difficulties appeared to lie in an uncertainty

about desired "ends," a dilemma which is associated with the

current debate between applied research vs. basic

research. Questions raised by this situation can perhaps

be phrased thusly: Is the Advisory Council expected to

develop planning criteria, provide judgment on program

balance, and evaluate program implementation, or is it

merely to reassure the Institute's Director that the

Extramural program is moving along the right track?

3.1.1 Study Section Priority Scores

It became a matter of concern to members of NIAID's

National Advisory Council that different interpretations

were being given by different Study Sections to their

"scientific merit" judgments and resulting priority

*Exercise II, "Biomedical Research Relevance Criteria,"

Project SOAP, National Bureau of Standards Report No.

10-423, June 30, 1971. This Exercise will be explained

in the following pages

33
scores.* Study Sections were never required to provide

explicit advice on order of payment, but practical con- siderations in some NIH Institutes made such rating scores useful for payment priority purposes.

The priority scores were ostensibly based on con- siderations of "scientific merit," but, in reality, the scores reflected the Study Section members' personal

notions of the extent or degree to which the proposal

should be supported by extramural funds. (An earlier

study of NIH grant application procedures by Saunders and

Gordon** concluded that Study Section priority scores had

become "a haphazard mixture of two separable considerations"

*A priority score was traditionally derived in the following

manner:

1. Each participating Study Section member rated

every "approved" grant application on the basis

of "scientific merit" from 1.0 (highest) to

4.5 (lowest) in 0.5 increments.

2. These two "digit ratings for each application

were averaged and the result multiplied by 100

to provide a three-digit numerical score.

3. The resulting "priority score" was recorded

on a confidential resume of action by the

Study Section and distributed to the previously

designated NIH Institute for funding.

An example: Each of six Study Sections assigns

"scientific merit" scores 1.5, 2.5, 2.0, 1.5, 2.0, 2.5.

Priority Score = \ (1.5 + 2.5 + 2.0 + 1.5 + 2.0 + 2.5)

x 100 = 12 -° x 100 = 200.

6 **Saunders, J. Palmer and Gordon, Mordecai, H. , "NIH Study

Section Ratings: Scientific Merit or Order of Payment,"

National Cancer Institute (1965)

.

34
referring respectively to the evaluation of research grant

proposals for scientific "quality" and for "order of payment."

Their hypothesis of separability was inferred statistically

from the scores of four Study Sections who reviewed 225

individual Extramural research grant applications assigned

to the National Cancer Institute in 1965.)

The fact of the matter was that the components of

the concept of "scientific merit" as applied by the Study

Sections were not explicitly known. The resulting confusion

raised the question of the appropriateness of certain other

criteria in this decision process and thus the desirability

of using multiple criteria judgments as the basis for

determining grant payment priority.

3.1.2 "Program Relevance" Criteria

During the March 1970 Advisory Council Meeting,

several members voiced concern about how decisions were

made on matters pertaining to "program relevance." They

wanted to know, specifically, what "criteria" were used

for determining HIGH, LOW, or NO "program relevance."

A motion was made and carried by the majority that the

"Council not consider Low Program Relevance (LPR) or NO

Program Relevance (NPR) ratings in evaluating payment

of grants."* This meant that the Council would no longer consider these special cases until further informa-

*Minutes of March 1970 meeting, NIAID Council, March 18, 1970

35
tion was provided about the criteria used. "HPR" was allowed, however, in order to provide in the interim some expression of Institute payment preference.

3.1.3 Preferential Payment Procedure

Normally, the NIH Extramural grant review procedure

is explicitly concerned with the "scientific merit" of the

proposals, with other aspects represented only implicitly.

However, biomedical research supported by NIAID and the

other Institutes of NIH affects ultimately the health and

well-being of the entire American people. Realization of

this fact has underscored legislation concerning NIH

for over 25 years, with confirmation of benefits from several

viewpoints: physical, social and economic* Both fundamental

and applied research produce new knowledge which can make

immediate or long-range contributions toward the effective

control of certain diseases and associated health problems.

Grant proposals selected for the Special Emphasis

Research categories were given an "HPR" rating, resulting

in an arbitrary two-decile escalation, which often raised

them to payment status. This practice introduced the

Institute's preferences ("program relevance") but did not

provide the Advisory Council with the kind of information

it felt was needed before it could make responsible

*See Stephen P. Strickland, "Policy Making in Biomedical

Research: A Perspective on Relevance," Project SOAP

Notes No. 16, March 15, 1971.

36
recommendations. For example, the Council wanted to know

the effect of such re-ranking action on the other competing

grants, the relative "importance" to the Nation's health

of each Special Emphasis area, and the rationale or plan

behind each selection of such an area. The task group

set out to satisfy these needs.

3. 2 Preliminary Study of "Health Relevance" Assessment

(Exercise I)

In June 1970, Project SOAP conducted a limited

investigation among the three members of the Advisory

Council Criteria Subcommittee: Drs . C. West Churchman,

Irwin C. Gunsalus, and Maxwell I. Wintrobe. They

agreed to try to assess "health relevance" as a concept

distinct from "scientific merit." Twenty applications

were sent by mail to each of the three Criteria

Subcommittee members . Their review was made without

knowledge of the "scientific merit" priority scores, and

without any discussion or communication between the

three members

The results of this study, based on a cursory analysis

of the twenty "relevance" rating scores, indicated that it was possible for Advisory Council members to attach a "health relevance" category to proposals, and that there

appeared to be some interesting relationships between the scores of this Ad Hoc Group and the NIH Study Sections'

priority scores. For example, comparison of these

37
relevance scores with the priority scores for the same proposals showed that eight out of 11 judged "high" for

"scientific merit" scored "high" for "health relevance,"

and four out of six judged "low" for "scientific merit"

scored "low" for "health relevance." The consensus of

the group was that, in spite of the lack of specified

criteria, there appeared to be some commonly held basis

for assessing "relevance."

These findings were presented by the chairman of

the Criteria Subcommittee to the NIAID Advisory Council

at its June 1970 meeting. This led to the Council's

recommending an expansion of the study of relevance

criteria (Exercise II), with instructions to include

a more representative sample of research grant applications

and a wider participation by other qualified reviewers

.

3 . 3 Biomedical Research Relevance Criteria (Exercise II)

This more comprehensive exploratory investigation was made in September 1970 in an attempt to improve the general

understanding of the decision process for awarding Extramural

research project grants at NIAID. Exercise II involved a dual review, the rating of research proposals separately,

first, for "scientific merit" and, second, for "health

relevance;" and it attempted to identify the various

decision criteria used by different groups within the

NIH Extramural grants award system.

38
A total of 120 professionals, most of whom were directly involved in the system of awarding grants, were

grouped into six "juries" for this exercise. Each participant rated a sample of 24 research applications (or subsample of that 24) on dimensions of "scientific merit" and/or

"health relevance." These ratings were analyzed to determine

if there existed a meaningful and consistent concept underlying these dimensions and, if so, what the characteristics

of this concept were. Corollary evidence was obtained from

statements of criteria elicited from some of the participants.

Analyses of jury mean scores (using nonparametric

correlation techniques) indicated that there were statistically significant similarities among juries in scores associated with "health relevance" for the sample applications

rated. One jury, composed of six Study Sections, both

reported and exhibited its inability to distinguish clearly

between assessments for "health relevance" and those for

"scientific merit." The general consistency among the

"health relevance" ratings and their agreement with the

"scientific merit" ranking suggested that the Study Sections'

judgments about priorities were generally related to their

concept of "scientific merit," at least on these test applications. Characteristics of the criteria stated by individual jurors as their bases for judgment, moreover, documented

the existence of review considerations other than scientific

quality.

39
The results of this exercise were not definitive.

However, certain characteristics were indicated. There

was general although not unanimous agreement among reviewers

about their concepts of "health relevance." (The lack of

unanimity was not surprising, since the concept of "health

relevance" is a complex one requiring greater specificity

to assure a common understanding.) The concept underlying

"scientific merit" as interpreted by Study Sections included

more than simple scientific quality considerations. This

additional dimension was closely related to their concept

of "order of payment."

40
4. THE PRESCRIPTIVE PHASE

The accomplishment of this phase was the formulation of

specific guidelines and recommendations leading to an improved Program Management discipline for NIAID. Several

mutually related actions pertaining to resource allocation

and NIAID Program Management were recommended.

4. 1 The Program Management Concept

The system recommended for managing the various programs, projects, and activities within the Institute was shown in comparison to the then-present mode of operation

(see figure 3, following page). The principal attributes of

the existing system were characterized as being detailed,

retrospective, and custodial in nature. By comparison, the

proposed management system provided a broader perspective and

depended upon anticipatory planning and the use of explicit

measures of performance. This was considered to be a desirable improvement by many Council members and Institute Staff.

Under the existing system, the cost/effectiveness

relationships were highly subjective. Moreover, the payment

priority advice was received from many individual Study

Sections that might not have had appreciation of the

Institute's major health research programs. The basic

improvement in resource allocation desired by the Institute

required a means for maintaining some proper balance between

its two primary mission goals: the support of basic science,

41
3

CO J> M O

tocICOHU

pq

OSB

Q

oo E3 O H b< to 9 >^ Ci CO

B,O

J i—i m CO

1-H >UJ oMu£

o3o

a: Puua oI

so3P

ETM1 a \*r. j> >

CO E-h fr •H

4-) 3£ T3 O u

(1) B. CO 03 •H Ph £ o

ns a? fn

4-> a, 4-1 <u 3 !1) Q CO OS

po t/5 • H P

4-1 o

a3 +j 3^ (/) crt i—1 w (1) (/) u

03 T3 tj 0) •H ££ s g

-ap ^ E > hH ua w H M 5 n -1-1 !)

M-l

u •H 1/1 P cy P (1) F 4-> •h S a> m o (/) u. 4-4 c C3 i w» M-J o I ) 03 E <j-t u

r-1 -H o .P

w —.

B. t-H

U ^, a> P, t/> p i—i u 5 3

rt o P

03

(1) 03 •H

a

hU 3 inU ^H M^ tn (U O cd O crt OS t/) m

fH <U , % B. ^

T3 P

(1) 4->

M c 03 CD •H S 4-J ft

fe. P

CI)

4-J I £P 3

P. & a;

4-J o >

c3 B.

C

T3O

•i-i T> 01

0) 3

+->

O c p o

a> 0) •H

•H £ M

ft

•H

T-H 3 i P, •H

4->

H (- 3 O M

U4 t/1 O) p H •H

C ^ O bfl H p

4-1 •H

crt P e V) CJ •H CD V)

jjj •H

4-1

crt

C) •T-fK 4-1 u

aj o r— •H <D u -P

a> (X 03 4J U 3 tw •H

4-J o c

•H 3

4-> £

U)

K 1

ou 1ua

Ml

Sis3

42
and the achievement of targeted research. In addition,

there was need for a consistent method for communicating

the Institute's programs, plans, and performance to others

(inside and outside) , while also providing a more rational

basis for decision-making, particularly for the Extramural

Grants Program.

Furthermore, to remedy the Institute's problem in

determining a preferred order of payment for grants,

Project SOAP recommended a management concept for

coordinating the planning, budgeting, and resource

allocation decisions in a way that would provide the

Institute greater control over its program performance.

The resource allocation problem was reduced to three

sets of decisions within the Institute: 1) the program

planning decision, 2) the "scientific merit" rank

decision, and 3) the program category assignment decision.

The effect was to delegate to the operational level the

authority to judge the relevance or contribution of a particular research project proposal and to assign it

to a proper program category. These categories were

the ones associated with a prior Council commitment of

funds to a Special Emphasis Research area in which the

ultimate solution of a particular disease problem was being sought.

43
Progress toward such health-oriented goals could be

assessed by the estimated reduction in time and resources

remaining to reach a level of scientific knowledge which,

in the opinion of experts, is sufficient to transfer the

project to the Intramural/Collaborative Program (supported

by other funds). In the event such progress lagged, a decision could be made to abandon the particular approach

by phasing out investment in it. In either case, the

appropriate funding (transfer or abandonment) would call

for Council/Staff/Institute interaction.

While this summation is a simplification of the

Institute's program management role, it clearly indicates

a resolution of the major portion of the dissonance in

the current "dual-review" system which stemmed from the

failure to define and delegate clear functions to each

level in the decision-making hierarchy.

4. 2 Recommended Criteria for Extramural Program Management

As a result of analysis of the criteria statements

received during the research relevance exercises (Exercise I and II, mentioned above) and subsequent discussions with

members of the Institute's Staff and Advisory Council,

Project SOAP recommended the following four criteria* for

evaluating Extramural research projects for quality,

importance, and cost:

*For a more comprehensive explanation, see "Recommended

Criteria for Extramural Program Management," Project SOAP

Notes No. 17, March 15, 1971.

44
I. Quality of Research ("Scientific Merit")

II. Importance (Relevance) to Health Research Programs

III. Distribution of Funds Between Programs (ShortTerm Balance)

IV. Investment in Science (Long-Range Forecast)

It is expected that these "Guideline Criteria" will

develop for the Institute a more explicit pattern of

reasoning for guiding the decision-making process between

the Council, the Staff, the Study Sections, and the Division

of Research Grants of NIH. Later, these guidelines can

be used to establish a basis for assessing and revising

the Institute's Special Emphasis Research Programs. Thus,

they can become the medium for communicating Extramural

Program priorities to NIH, the Office of Management and

Budget, the Congress, and the scientific community

at large.

4.3 Research Investment Procedure: Dollar Allocation

by Program Area (DAPAT

After exploring several possible approaches to solving

NIAID's priority management problems, Project SOAP concluded

its study with a proposal for a new procedure. The "Dollar

Allocation by Program Area: An Investment Procedure"

was presented to and accepted by the NIAID Advisory

Council at its March 18, 1971, meeting.

45
DAPA is an alternative to decile manipulation as a device for allocating funds to Extramural research activities.* It is based on the concept of capital investment to areas of socially important health research, within the

context of projects supported by NIAID Extramural grants.

The procedure is designed to replace the present "program

relevance-decile adjustment" system, while also preserving

the existing practice of ordinal ranking for "scientific

merit" by the Study Sections . This procedure is analogous to a two-part securities

investment portfolio, such as that for common stocks and

bonds. Figure 4 (following page) shows how research funds

had previously been allocated and served to indicate to the

Advisory Council and NIAID staff certain funding imbalances

in their existing program structure. This illustrates a feature of using DAPA. The first part of the portfolio

reflects selected investments in specified Special Emphasis

Program areas (SERP's) of particular importance to the

mission of the Institute. The purpose of these SERP's

is to increase the probability of yielding "health relevant"

results from grant- supported research. The second part

concerns investments of Extramural funds in support of freeranging investigator-inspired research. Grant applications

are paid in order of peer-recommended "scientific merit"

priority scores.

*For a detailed description, see "DAPA Research Investment

Procedure," Project SOAP Notes No. 19, June 15, 1971.

46
NIAID Extramural Research Project Grants DAPA Research Investment Portfolio

(March 1971)

Program Category

Number of

Grants

$ Million

Amount

Percent

Extramural

Budget

I. Special Emphasis Research Programs

A. Clinical Allergy 58 3.2 6.3%

B. Transplantation Immunology 71 3.7 7.3%

C. Strep. Disease § Sequelae 25 0.9 1.8%

D. Drug Resistance 210 7.1 13.9%

E. Infectious Hepatitis 5 0.3 0.6%

F. Chronic § Degenerative Diseases 45 1.5 2.9%

G. Antiviral Substances (Interferon] 116 5.3 10.4%

H. Biological Regulation of Vectors 40 1.4 2.8%

I. Immunity to Animal Parasites 44 1.4 2.8%

SERP Total 614 24.8 48.8%

II. Investigator Inspired

Research Projects 565 23.7 46.5%

III. Other Research Projects 66 2.4 4.7%

TOTAL 1,245 $50.9 100.0%

Figure 4

Pra|itl SOA.P

Tttk.lcl AMlytb DhWM

NitlHilBma if State*

47
4.3.1 The PAPA Procedure

At least once a year, the Advisory Council meets to

advise the Institute on Extramural Program investment

policy. The Institute staff provides a review of present

SERP activities, with summaries of the distribution of funds

to grantees in each support area. Staff also submits a proposed investment plan for Council approval, which includes

necessary historical perspectives of specific programs.

.

Recommendations for new SERP's or terminations of present

ones would require support documentation. Incremental

adjustments within previously approved guideline ranges need

only be proposed to the Council for approval.

After Staff's presentation, the Council may deliberate

the merits and implications of the plan as well as call upon

outside experts for additional information. The Council then

resolves the issue by "approving" a modification of the

Institute's investment proposal. This includes the total

dollars and percentage range to be allocated for: 1) freeranging research projects, and 2) each Special Emphasis

Research Program category. Also to be approved (perhaps

with modifications) are statements describing each SERP's

area (Program Definition Statements) , including its specific

goals, scope, research characteristics and a brief state-ofthe-art summary.

Special Subcommittees of the Advisory Council ad hoc

advisors, and Institute Staff would be appointed and

48
specifically assigned Extramural program planning and

evaluation functions. These working groups would meet

during periods between scheduled Council meetings to determine the nature and magnitude of each approved SERP category,

including major revisions if warranted. Specialized information and competent advice from other advisory groups

(like Study Sections, consultant experts, and contract

studies) would be available to them.

Pertinent health statistics on disease/disease problems, populations, specific disease mortality and morbidity

rates, hospitalization costs, epidemiology, and related

health information will be included. The use of these data

suitably combined in an appropriate model would provide a quantitative basis for establishing the programmatic "health

relevance" for each SERP category.

The qualitative judgments of the Staff /Council Subcommittee would, of course, require such data as a basis for

proposing appropriate levels of investment and establishing

SERP payment priorities.

In brief, the DAPA procedure is part of a program

management discipline for distributing limited research

funds. It provides a reasonable method for Staff/Council

interaction in balancing budget allocations between desired

ends (health research "needs") and available means (biomedical research opportunities) , while at the same time

relieving the Advisory Council of the task of reviewing

49
each individual grant proposal recommended by the StudySections for payment approval. (A flow chart of the

DAPA Procedure is shown in figure 5, following page.)

Grants supporting free-ranging (FR) research

investigators would continue to be awarded on the basis

of Study Section priority scores. Those proposals

which further qualify for support under one of the

Institute's Special Emphasis Research Programs would

be paid, on the basis of similar order of priority

scores, to the limit of designated program funds

allocated to each prior approved category.

The application of the results of this systematic

planning and evaluating process would become the responsibility of each SERP Program Manager appointed by the

Director, subject to the advice and continued review of

his subcommittee. When a particular SERP category has

reached maturity (attained that "critical mass" of fundamental knowledge required for applied research- -as the

category of hepatitis did a few years ago) , the decision

to "transfer" support and scientific review to a more

appropriate Intramural or Collaborative research

activity within the Institute could be made. This

ultimately would lead to more direct health research

applications in one of the NIAID Institute's national

clinical centers.

50
3 s< o 9. m id I

(lEAOjddv) -\

All

Pi

f^~\ 4-> o ni

ma \

oj i/> 03 M <n o W tO < 4-> C/3

L "

<; o -h o 2 05 D-, c/5

Cu ^u

n) 3 a

<uu

u. >

a

u

u.

u. <

_Ji_

*

<k>

Study

Sections

Review Ratings

-pd x e H C

•Hm

in <

^

J k

Xq.Turuiiiiioo oijiausiDS ipxessaa iroxpauioTg I

51
Program areas and their budgets would be reviewed

regularly by Staff and modified by Council/Staff agreement,

as new and more current information is obtained. Progress

or stagnation could thereby be more clearly discerned, with

greater assurance of the Council's making timely assessments

as well as fully-based initial decisions.

Implicit in the DAPA approach is the necessity of

professional judgment and reliable information upon which

to base these decisions. Under existing conditions of

great uncertainty regarding the technological, economic,

and social feasibility of emphasizing a particular field

of research, this procedure provides increased flexibility

for guiding the Extramural program planning and evaluation

activities. Moreover, it offers an operational method

for potentially doing a better job with the skills

and resources available.

The Project SOAP task group believes that the use of

the dollar allocation method (DAPA) could become an important management tool for science administrators

elsewhere within NIH for negotiating the "health relevance"

of biomedical research programs with the Office of Management and Budget. Further testing and evaluation of the

DAPA Research Investment Procedure was, therefore,

recommended. As noted earlier, initial steps have been

indicated in this direction.

52
Because of its potential importance for long range

program planning, the Task Group also investigated the possibility of developing a Biomedical Research Utility Model to

aid in planning and evaluating the social/economic impacts

of major health research programs. This part of the study,

which is contained in Project SOAP Notes No. 14, was not

implemented due to initial reluctance on the part of NIAID

Staff to embrace so sophisticated a modeling method without

a greater understanding of and confidence in the validity

of its results. The model basically employed statistical

data from epidemiological sources to evaluate the National

importance of a particular disease.

It attempted to estimate the "economic cost" to the

Nation in terms of medical expenses, hospitalization costs,

lost work time, and a factor for human discomfort. The

approach is similar to that for estimating insurance risk,

but does not have the advantage of clear empirical data.

For this reason, it was felt the actual application of a Biomedical Research Utility Model was premature.

(Developing such models is, of course, a difficult

and demanding task, but an inescapable one in arriving

at an appropriate quantitative basis for the inherently

quantitative decisions of resource allocation.)

53
5. CONCLUSION

Summing up: this case study documents the methodology

and substantive information developed by the Project SOAP

task group during its two-year assignment to perform analytical studies for the National Institute of Allergy and Infectious Diseases of the National Institutes of Health between

1969 and 1971.

Project SOAP created for NIAID a clearer understanding

and a more focused awareness of the nature of its resource

allocation problems, particularly those of the Extramural

Grant Program. This first step was reflected in the action

taken by the NIAID Advisory Council at its meeting, June 21,

1971. (See following page, which was abstracted from the

official minutes of the June 1971 Council.)

It would be unreasonable to expect that any task group

could attain so profound an insight into the myriad problems

of a complex organization that it could actually identify any

one problem as the crux of matter and determine how it should

be solved. There are certain problems far too complicated

for systems analysis as yet to comprehend. Nevertheless,

given its limited scope of understanding in approaching the

management situation it faced, the Project SOAP task group

stimulated NIAID to perceive its new role in a changing

environment: that of managing its resources explicitly.

With this new awareness, NIAID was then in a far better position to accept and adopt the use of recommended management

methods for improving its decision-making processes.

54
Report of NIAID Advisory Council Meeting

(Abstracted from Official Minutes of the June 1971 Council

National Institute of Allergy and Infectious Diseases/NIH)

Allocation of Extramural Resources

(.Informational J The first step in implementation of the

"Dollar Allocation Program" is taken.

In taking the first step in the implementation of the

"Dollar Allocation Program" which has evolved from the

Institute-sponsored SOAP study, Council reviewed the Special

Emphasis programs of the Institute, including the definitions

of these programs, and the funds which might be allocated to

them. Rather than following the past procedure of raising

selected research grant applications to payable priority

scores they concluded that during FY 72, funds should be

allocated to specific programs to enable the payment of more

applications assigned to these programs. The Council would

review this allocation at each of its subsequent meetings and

would not, with the present limit of funds, approve the payment of research grants beyond the upper 60% of approved

applications in any of the programs receiving special

allocation.

In the future, Council will be emphasizing review of programs

rather than review of individual projects within these programs. The Council chose to continue the present disciplineoriented subcommittee system for the review of Institute

programs as well as for the grants assigned to them.

Background :

In 1969 an Interagency Agreement between the Technical

Analysis Division of the National Bureau of Standards and the

NIAID was entered into for the purpose of developing criteria

for the selection of grants and/or programs for preferential

treatment to better meet the needs of the health problems of

the country. The Criteria Subcommittee of Council was appointed to act as liaison between Project SOAP (Systems/

Operations Analysis of Programs) , the working group of this

agreement, and the Council in these efforts. After tentatively exploring a number of possible avenues, the Dollar

Allocation for Program Areas was presented to and accepted

by Council at the March 1971 meeting.

Council instructed staff to redefine, in a more concise

manner, the Special Emphasis areas of the Institute for con- sideration by Council at the June meeting. It was agreed

that the dollar allocation approach would be entered into

gradually. Program definitions were reviewed carefully,

revised by staff and were presented to this Council for con- sideration.

55
PARTICIPANTS

We acknowledge with gratitude the helpful contributions

of the following persons to the various activities

associated with this joint task group:

NIAID/NIH

Dr. Dorland J. Davis, Director

Dr. Francis R. Abinanti

Dr. Robert J. Byrne

Mrs. Grace Ellis

Dr. William I. Gay

Mr. Walter M. Magruder

Mr. Charles B. Myers

Mrs. Gwenndolyn Northcutt

Mrs. Rosemary Sachs

Dr. Robert T. Scholes

Dr. John R. Seal

Dr. Alfred M. Webb

NIAID Advisory Council

Criteria Subcommittee

Dr. C. West Churchman, Chairman

Dr. Irwin Gunsalus

Dr. James H. Matthews

Dr. Duard Walker

Dr. Maxwell M. Wintrobe

TAD/NBS

Dr. W. Edward Cushen, Chief

Mrs. Sadie L. Berry

Mrs. Elsie M. Clark

Mr. Ben E. Clayton

Dr. Dwight Erlick

Dr. Nancy R. Kingsbury

Dr. Julius Lieblein

Mr. Vincent A. Martino

Mr. Edward G. Neigut

Mrs. Penny Reeder

Mrs. Janette Taylor

Miss June L. Watkins

Consultants

Dr. Burton V. Dean

Dr. Thomas A. Goldman

Dr. David B. Hertz

Dr. Emmett Keeler

Dr. Stephen P. Strickland

Dr. Mary Ellen Golby

Deserving of special recognition are those individuals

who gave invaluable help to Project SOAP. They are:

Dr. Mary Ellen Golby, who edited the material that appears

in this document, Drs . Alexander R. Craw and Alan J. Goldman,

who critically reviewed the successive drafts of the manuscript, and Mr. Louis C. Santone, who provided encouragement

and financial support for its documentation. Special

acknowledgment is also due Mrs. Janette E. Taylor, without

whose assistance this report would not have been possible.

56
CDP

(/) cd

m •H o x

C P o LO P • >> CQ CD Q P <-H MS i-H I-H H o c \ p <

to •H

Q 3 1—

P 10 ^ < U 2

CDp

O

H > 1—

o • •> H 3 -4 CO p £ 10 o CD a c P (J P • H P H p P CD <+H £ CD p hiim O X to cti o o • H P X !S Cti P o to

05 to • .«.

H P p m < to CO P c o •H >s

1/) o p > rH •H 10 p £ 03 cd S CD X CD Q P

•H P bo

O < s a•H c • 13 P •H p P < •> > f= c

CD

p CD to cd . „ S

t/i P C CD <D * P i—

bC

C •H

p

x <; cd •H 6 O taoi—

C

C/l £ p •H 2 TO o o u p s

pq o CD •* £1 P o M •» m p •H p C to O CO ( •H P p c CD p rt t= PI >* o • H i—

CD cci S

CO P< .—i—

I/) CD l—

Ph to

CD

p 2 •< CD

m

H P.E lH o P •> 3 ctS P

inU COO P Cd P- H Pi MX

CD > M

O

U •» XI cd p P P Q ^ PL, T3

TO (/)

p e<p • cd » cd XI O l^) P cp u <U . «l p C t H co 3 cd C < X M X 6 re O U 2 U P i-H co •H P » a P cd C p X< to X Q U X H o U CD CD S • i—

S •» T3 P O p • Pi o Pi • CD u rt C JS-d o TO • cd p <p nd CD P H o CD J Q c CD P p CO to <P P •* o H p i—

U

CD "4-i

U Oj < CD > H P. XI i—i O rH CD 6 • O x cd X CD P P < u o EP&.

57
PROJECT SOAP ISSUES PRELIMINARY FINDINGS ON

'HEALTH RELEVANCE' RATING EXERCISE

A recent interagency agreement between IAT's Technical Analysis Division (TAD) and the National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health, resulted in the creation of Project SOAP. Not a detergent, Project SOAP (System/Operations Analysis Programs) is concerned with developing quantitative methods for determining priorities in allocating NIAID'

s

limited research funds. One issue considered was whether it would be feasible to give some weight to the "health relevance" of research projects in deciding whether to fund extramural research grant applications. Preliminary findings by TAD indicate

that it is indeed practical and possible to consider the health relevance of a pro- posed research project as well as its "scientific merit" implications. Implicit in these findings is the need for development of "criteria" for health relevance, and work is currently in progress along those lines. Above, far right, Dr. W. E. Cushen, Chief of TAD, presents the results of the health relevance analysis to (left) Dr. Dorland J. Davis, Director of NIAID, and Dr. C. West Churchman, Professor of Business Administration, University of California, and Chairman of the NIAID Advisory Council Subcommittee on Criteria.

Reprinted from the NBS Standard, Vol. XVI, No. 2 (February 1971)

5£
REFERENCES

1. Ackoff, Russell L., A Concept of Corporate Planning , New York, Wiley- - Interscience , John Wiley § Sons, Inc.,

(1970).

2. Bartee, Edwin M. , "Problem Solving with Ordinal

Measurement," Management Science Journal, Volume 17,

No. 10, (June 1971).

3. Citron, Marvin J., "QUEST:" Quantitative Utility Estimates

for Science and Technology," IEEE Transactions on Eng.

Mgmt., Volume EM- 14, No. 1, (March 1967).

4. Chacko, George K. (Editor), The Recognition of Systems

in Health Services , ORSA, (1969)

5. Churchman, C. West, The Systems Approach , Dell

Publishing Co., New York, (1968).

6. Churchman, C. West, Challenge to Reason , New York,

McGraw-Hill, (1968).

7. Churchman, C. West and Eisenberg, H. B., "Deliberation

and Judgment," Human Judgments and Optimality , (Shelly II,

Maynard W. , Editor), New York, John Wiley § Sons, Inc.,

(1964).

8. Danto, A. and Morgenbesser , S. (Editor), Philosophy of

Science , New York, World Book Company, (I960) .

9. Davis, Dorland J., "Special Emphasis Research Programs

at NIAID," The Journal of Infectious Diseases, Volume 121,

No. 2, (February 1970)

10. Dean, Burton V. et al, "Allocation of Resources for Basic

Research in a Mission Oriented Government Agency,"

Case Western Reserve University, Technical Memo No. 128,

(August 1968)

11. Hertz, David B., New Power for Management , McGraw-Hill,

Inc., (1969).

12. Human Judgments and Optimality , (Shelly II, Maynard W.

,

Editor), New York, John Wiley $ Sons, Inc., (1964).

13. Kahn, Herman and Wiener, Anthony J., The Year 2000 , New York, MacMillan, (1967).

14. Keeler, Emmett, "Models of Disease Costs and their Use

in Medical Research Resource Allocations," RAND,

(December 1970) . 5-9
15. Kistiakowski , G. B., Federal Support of Basic

Research in Institutions of Higher Learning,

National Academy of Science- -National Research

Council, (1964).

16. Rice, Dorothy, "Estimating the Cost of Illness,"

National Institutes of Health, (1964).

17. Taylor, V., "How Much is Good Health Worth?,"

Policy Sciences Journal 1, (1970).

60
APPENDIX

(Sections A and B)

SECTION A - LIST OF NOTES AND WORKING PAPERS

PROJECT SOAP NOTES*

Notes

No. Author Title Date

1. Robert S. Cutler "Definition of Terms" Aug. 29, 1969

2. Robert S. Cutler "Systems Concepts Re: Program Management"

Sept. 26, 1969

3. Vincent A. Martino "Systems Description of Dec. 15, 1969

Robert S. Cutler Collaborative Research Program"

4. Robert S. Cutler "Extramural Program: Research Jan. 30, 1970

Vincent A. Martino Grants Award System Description"

5. Project SOAP "Task Group Briefing to NIH" Feb. 5, 1970

6. Dr. Julius Lieblein "Preliminary Analysis of Data, Mar. 15, 1970

NIAID Decile System"

7. Edward G. Neigut "Prospective Assistance Areas Mar. 31, 1970

for NIAID Collaborative Programs"

8. Robert S. Cutler "Criteria Subcommittee Meeting," May 29, 1970

Berkeley, California,

April 25, 1970

9. Vincent A. Martino "Analysis of Issue: Relevance June 30, 1970

Criteria"

10. Vincent A. Martino "Preliminary Study of Group Aug. 30. 1970

Assessment of Health Relevance,

Exercise I"

11. Harold R. Millie "In-Basket Survey of Public Oct. 1, 1970

Inquiries Regarding NIAID

Health Information"

12. Dr. Julius Lieblein "Review of Report: Change in Oct. 29, 1970

Method of Calculation of Deciles"

13. Robert S. Cutler "Unbundling Criteria: Science Dec. 4, 1970

Merit/Health Relevance"

14. Dr. Emmett Keeler "Models of Disease Costs and December 1970

Their Use in Medical Research

Resource Allocations"

*Copies of Notes are available from: Project SOAP, Technical Analysis

Division, National Bureau of Standards, Washington, D.C. 20234

A-l
Notes

No. Author Title Date

15. Stephen P. Strickland

16. Robert S. Cutler

Thomas A. Goldman

17. Robert S. Cutler

Vincent A. Martino

18. Robert S. Cutler

19. Robert S. Cutler

Vincent A. Martino

20. Robert S. Cutler

"Policy Making in Biomedical Mar. 15, 1971

Research: A Perspective on

Relevance"

"Thoughts from Exercise II Pertaining to Social/Economic

Impact Criteria"

"Recommended Criteria for

Extramural Program Management"

Mar. 15, 1971

Mar. 15, 1971

"Recommendations for Council" Mar. 18, 1971

"DAPA Research Investment June 1, 1971

Procedure"

Exercise II Report: "Biomedical June 30, 1971

Research Relevance Criteria"

A2
WORKING PAPERS

Historical Perspective for Relevance Rating Oct. 13, 1969

(P. Reeder)

TAD/NIAID Progress Report Dec. 15, 1969

(R. S. Cutler)

Discussion of Issue: Program Relevance Apr. 17, 1970

(R. S. Cutler)

NIH Decision Making: Role of the Science Administrator May 12, 1970

(F. R. Abinanti)

Report on Criteria Subcommittee Exercise I July 24, 1970

Re: Health Relevance

(V. A. Martino)

NIAID Goals for the 1970 's: Preliminary Staff Paper Aug. 24, 1970

(A. M. Webb)

Exercise II Criteria Statements Dec. 23, 1970

(V. A. Martino)

Council Policy Guidelines Dec. 23, 1970

(R. S. Cutler)

Social/Economic Impacts --External Criteria Jan. 29, 1971

(Dr. T. A. Goldman)

Exercise II Summary: Research Relevance Rating Feb. 25, 1971

and Criteria

(R. S. Cutler and V. A. Martino)

Project SOAP Progress Report, FY 71 Mar. 4, 1971

(R. S. Cutler)

Systems/Operations Analysis of Programs Oct. 30, 1971

(An analysis of the studies performed by the

Project SOAP Task Group)

(R. S. Cutler)

ASECTION B - BACKGROUND DOCUMENTS

NIH Study Section Ratings: Scientific Merit or Order

of Payment?

(J. Palmer Saunders, and Mordecai H. Gordon, NCI Paper)

NIAID Special Emphasis Research Programs

(NIAID Staff)

Remarks on Special Emphasis Programs of NIAID

(Dr. Dorland J. Davis)

Transcript of November 1970 Council Meeting

(G. Northcutt)

Transcript of March 1971 Council Meeting

(J. Taylor)

Interagency Agreement File (Modifications 1-9)

NAAIDC Criteria Subcommittee Recommendation

Council Resolution (March 1971)

Report of June 1971 Council Meeting

Apr. 10, 1965

June 5, 1967

Oct. 25, 1969

Nov. 16, 1970

Mar. 18, 1971

Mar. 16, 1969

through

Mar. 21, 1971

June 19, 1970

Mar. 25, 1971

June 21, 1971

U. S. GOVERNMENT PRINTING OFFICE : 1973 —511-325/232

A4
FORM NBS-114A (1-71) U.S. DEPT. OF COMM. BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO. NBS TN-761

2. Gov c Accession No.

3. Recipient's Accession No.

4. TITLE AND SUBTITLE 5. Publication Date Project SOAP: A Systems Approach to Biomedical

Research Program Management . A Case StudyApril 1973

6. Performing Organization Code

7. AUTHOR(S)

Robert S. Cutler

8. Performing Organization

10. Project/Task/Work Unit No. 4310900

9. PERFORMING ORGANIZATION NAME AND ADDRESS NATIONAL BUREAU OF STANDARDS DEPARTMENT OF COMMERCE

WASHINGTON, D.C. 20234

11. Contract/Grant No.

12. Sponsoring Organization Name and Address Same as No. 9

13. Type of Report & Period Covered Final

14. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

16. ABSTRACT (A 200-word or less factual summary of most significant information. If document includes a significant bibliography or literature sutvey, mention it here.) This case study describes the activities of an interagency task group

that applied systems analysis to improve management controls within a biomedical research agency of the federal government. The results were

the formulation and implementation of a discipline for program management,

which explicitly makes use of multiple criteria in arriving at resource allocation decisions.

The text details the necessary preliminary analysis describing operational

activities, information flows, and key decision points within the organization. It goes on to identify the techniques employed and the difficulties encountered while attempting to improve the decision-making process for selecting research projects, under conditions of reduced funding

In particular, a comparison is made between: (1) the agency's traditional

single-criterion "peer review" judgment for determining budget priorities,

and (2) the multiple-criteria judgments required to effect more positive

management control. The systematic use of separate "scientific merit"

and "health relevance" ratings is compared with the organization's

actual experience. The criteria used by various participants in the

decision process are analyzed, and a dollar allocation "investment"

procedure based on these findings, is developed.

A procedure which organizes relevant information for research program

planning and evaluation is presented, and extension of this recommended

procedure to wider use by science administrators elsewhere in government

is discussed.

17. KEY WORDS (Alphabetical order, separated by semicolons) Biomedical research; "health relevance"; program planning and budgeting

R§D management; systems analysis; systems approach.

18. AVAILABILITY STATEMENT

jfXl UNLIMITED.

I I FOR OFFICIAL DISTRIBUTION. DO NOT RELEASE

TO NTIS.

19. SECURITY CLASS

(THIS REPORT)

UNCLASSIFIED

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

21. NO. OF PAGES

75

22. Price

.95 Domestic postpaid

.70 GPO Bookstore

USCOMM-DC 66244. P71

NBS TECHNICAL PUBLICATIONS

PERIODICALS JOURNAL OF RESEARCH reports National Bureau of Standards research and development in

physics, mathematics, and chemistry. Comprehensive

scientific papers give complete details of the work,

including laboratory data, experimental procedures, and theoretical and mathematical analyses. Illustrated with photographs, drawings, and charts. Includes

listings of other NBS papers as issued.

Published in two sections, available separately:

• Physics and Chemistry (Section A)

Papers of interest primarily to scientists working in these fields. This section covers a broad range of physical and chemical research, with major emphasis on standards of physical measurement, fundamental

constants, and properties of matter. Issued six times

a year. Annual subscription: Domestic, $17.00; Foreign, $21.25.

• Mathematical Sciences (Section B)

Studies and compilations designed mainly for the mathematician and theoretical physicist. Topics in mathematical statistics, theory of experiment design, numerical analysis, theoretical physics and chemistry,

logical design and programming of computers and

computer systems. Short numerical tables. Issued quarterly. Annual subscription: Domestic, $9.00; Foreign,

$11.25.

TECHNICAL NEWS BULLETIN

The best single source of information concerning the Bureau's measurement, research, developmental, co- operative, and publication activities, this monthly

publication is designed for the industry-oriented

individual whose daily work involves intimate contact with science and technology

for engineers, chemists,

physicists, research managers, product-development

managers, and company executives. Includes listing of

all NBS papers as issued. Annual subscription: Domestic, $6.50; Foreign, $8.25.

NONPERIODICALS Applied Mathematics Series. Mathematical tables, manuals, and studies. Building Science Series. Research results, test methods, and performance criteria of building materials, components, systems, and structures. Handbooks. Recommended codes of engineering and industrial practice (including safety codes) de- veloped in cooperation with interested industries, professional organizations, and regulatory bodies. Special Publications. Proceedings of NBS conferences, bibliographies, annual reports, wall charts, pamphlets, etc. Monographs. Major contributions to the technical

literature on various subjects related to the Bureau's

scientific and technical activities. National Standard Reference Data Series. NSRDS provides quantitative data on the physical and chemical properties of materials, compiled from

the world's literature and critically evaluated. Product Standards. Provide requirements for sizes, types, quality, and methods for testing various industrial products. r These standards are developed co- operatively with interested Government and industry groups and provide the basis for common understanding of product characteristics for both buyers and

sellers. Their use is voluntary.

Technical Notes. This series consists of communications and reports (covering both other-agency and

NBS-sponsored work) of limited or transitory interest. Federal Information Processing Standards

Publications. This series is the official publication within the Federal Government for information on

standards adopted and promulgated under the Public Law 89-306, and Bureau of the Budget Circular A—86

entitled, Standardization of Data Elements and Codes

in Data Systems. Consumer Information Series. Practical information, based on NBS research and experience, covering areas of interest to the consumer. Easily understandable language and illustrations provide useful background knowledge for shopping in today's tech- nological marketplace.

BIBLIOGRAPHIC SUBSCRIPTION SERVICES

The following current-awareness and literature-survey bibliographies are issued periodically by the Bureau

:

Cryogenic Data Center Current Awareness Service (Publications and Reports of Interest in Cryogenics). A literature survey issued weekly. Annual subscription: Domestic, $20.00; foreign, $25.00.

Liquefied Natural Gas. A literature survey issued quarterly. Annual subscription: $20.00. Superconducting Devices and Materials. A literature survey issued quarterly. Annual subscription: $20.00. Send subscription orders and remittances for the preceding bibliographic services to the U.S. Department

of Commerce, National Technical Information Service, Springfield, Va. 22151.

Electromagnetic Metrology Current Awareness Service (Abstracts of Selected Articles on Measurement

Techniques and Standards of Electromagnetic Quantities from D-C to Millimeter-Wave Frequencies). Issued monthly. Annual subscription: $100.00 (Special rates for multi-subscriptions). Send subscription order and

remittance to the Electromagnetic Metrology Information Center, Electromagnetics Division, National Bureau

of Standards, Boulder, Colo. 80302. Order NBS publications (except Bibliographic Subscription Services) from : Superintendent of Documents, Government Printing Office, Washington, D.C. 20402.
U.S. DEPARTMENT OF COMMERCE

National Bureau of Standards

Washington. D.C. 20234

OFFICIAL BUSINESS

Penalty for Private Use, $300

POSTAGE AND FEES PAID

U.S. DEPARTMENT OF COMMERCE

215