I
Operations Research

Proceedings of a Conference

For Washington Area Government Agencies

April 20, 1966

United States Department of Commerce

National Bureau of Standards

Miscellaneous Publication 294
THE NATIONAL BUREAU OF STANDARDS

The National Bureau of Standards1 provides measurement and technical information services

essential to the efficiency and effectiveness of the work of the Nation's scientists and engineers The

Bureau serves also as a focal point in the Federal Government for assuring maximum application of

the physical and engineering sciences to the advancement of technology in industry and commerce To accomplish this mission, the Bureau is organized into three institutes covering broad program areas of research and services: THE INSTITUTE FOR BASIC STANDARDS . . . provides the central basis within the United

States tor a complete and consistent system of physical measurements, coordinates that system with the measurement systems of other nations, and furnishes essential services leading to accurate and uniform

physical measurements throughout the Nation's scientific community, industry, and commerce This

Institute comprises a series of divisions, each serving a classical subject matter area- —Applied Mathematics—Electricity—Metrology—Mechanics—Heat—Atomic Physics—Physical Chemistry—Radiation Physics—Laboratory Astrophysics2—Radio Standards Laboratory,2 which

includes Radio Standards Physics and Radio Standards Engineering—Office of Standard Refer- ence Data. THE INSTITUTE FOR MATERIALS RESEARCH . . . conducts materials research and provides

associated materials services including mainly reference materials and data on the properties of materials. Beyond its direct interest to the Nation's scientists and engineers, this Institute yields services which are essential to the advancement of technology in industry and commerce. This Institute is or- ganized primarily by technical fields: —Analytical Chemistry—Metallurgy—Reactor Radiations—Polymers—Inorganic Materials—Cry- ogenics-—Office of Standard Reference Materials.

THE INSTITUTE FOR APPLIED TECHNOLOGY . . . provides technical services to promote the use of available technology and to facilitate technological innovation in industry and government The

principal elements of this Institute are: —Building Research—Electronic Instrumentation—Technical Analysis—Center for Computer Sci- ences and Technology—Textile and Apparel Technology Center—Office of Weights and Measures —Office of Engineering Standards Services—Office of Invention and Innovation—Office of Vehicle

Systems Research—Clearinghouse for Federal Scientific and Technical Information3—Materials

Evaluation Laboratory—NBS/GSA Testing Laboratory.

1 20234

qUarterS Lab0ratories at Gaithersburg, Maryland, unless otherwise noted; mailing address Washington, D. C,

- Located at Boulder, Colorado, 80302.

3 Located at 5285 Port Royal Road, Springfield, Virginia 22151.
UNITED STATES DEPARTMENT OF COMMERCE • Alexander B. Trowbridge, Secretary

NATIONAL BUREAU OF STANDARDS • A. V. Astin, Director

Operations Research

Proceedings of a Conference

for Washington Area Government Agencies

April 20, 1^66

Held at Museum of Natural History,

Smithsonian Institution, Washington, D. C.

Editor: J. A. Joseph

Technical Analysis Division

Institute for Applied Technology

National Bureau of Standards

Washington, D. C. 20234

National Bureau of Standards Miscellaneous Publication 294

Issued December 1967

For sale by the Superintendent of Documents, Government Printing Office

Washington, D.C., 20402 - Price 25 cents

Contents

Page

Preface

Program for the Meeting vi j Opening remarks by Dr. Allen V. Astin 1 O.R. in the Civilian Sector of the Government 3 Professor Merrill M. Flood

The Planning-Programming-Budgeting System 7 Dr. John Haldi

Operations Research and Government O.R 13 Dr. Alan J. Goldman

Reports of the Panel Meetings 19

List of Attendees 23 Acknowledgements 24

Library of Congress Catalog Card Number: 67-62625

III

Preface

was On April 20, 1966, the National Bureau of Standards sponsored a one-day symposium whose purpose

that

totheyconvene might

operations research practitioners and users in the agencies of^thVPedeS ESSSZSStlZ exchange information on topics of mutual concern

^eaerai government so

related activiSf

COnsiderations underlay the &TMmng of the conference, and our hopes for future

(1) There does not exist, but there should exist, a reference inventory of completed and on-eointr operations research/systems analysis/multidisciplinary research/scientific planning activities irf the oltett ctldTeTctlitatod^

&

°

f meth°dS

> be"efit -aS^Uh!

tiona

(

i aeffSi

eveeness

e meth°dS t0 ^ aPPHed that n0t UniqUe t0 °ne agency> e^> tests of organiza-

(b) There are suggestive analogies among problem types from agency to agency, e g a net- work of highways and a network of post offices. «-seiicy, e.g., a net

(c) There are interaction situations, e.g., the full program benefit to one agency may be in- completely calculated due to excluding benefits arising to other Federal missions

(2) The growth in opportunity to test the usefulness of systems analysis in new non-defense con- texts underscores the importance of making sure that any initial study has the fullest possible measure reasumng

of success at an early date, and this might exclude repeating exploratory work others have done No encyclopedic memory bank exists. (3) Each of us knows of a dozen or two other studies or groups, but the creation of new e-roups is making it more difficult and time-consuming to keep up to date. We need a new one-time updating

SvJTM <l^ll°ia

C*\0lff th6 n6W

r

U?

aU ot

- the B

. udget PIannin^ Programming, and Budgeting System (Circular 66-3) places an explicit requirement on agencies to provide systems analysis, binefit- cost studies, and supporting documentation. Some agency chiefs have asked OR groups, "Tell me what

fXr^n/ to, heiP ; th

,e °R ^oups need to meet with the PPB originators on a technical basTs for a fuller understanding of the substantive requirements and desires so that they can help their parent agency to respond adequately to this new system. P parent

(5) The problem of identifying measures of effectiveness for agencies is a common one; some have made good progress in this area, and others are searching for ways to make a good start. Furthermore many agencies have missions whose ultimate measures of effectiveness can be partly stated in common terms, e.g., stimulate the economy improve the standard of living, etc. Aside from the recent Brookings Institution publications Roland McKean's earlier book, the BOB bibliography of benefit-cost studies, and a few other attempts of a more nebulous nature, the best information on measures of benefit is scattered among our individual heads and in individual files. scduueiea

(6) The changing semantics and connotative nuances placed on the terms, "operations research " systems analysis, multidisciplinary problem-solving teams," and "planning systems," are further confused by actual practice. For example, OR is frequently defined to be identical with what the statis- tics group does, or with the problems that are sent to the computing lab, or with the management analysis program of an agency, or with just plain horse sense. Accordingly, there does not seem to be

a good common dictionary of what to include, or what to exclude from a conference such as this Each

agency has one or more groups with something significant to contribute or gain; these groups are all talking, with greater or lesser degree of specialization, to a common problem—that of providing a ra- tional scheme for identifying and comparing alternatives in terms of their expected payoff and resource requirements. Similar differences exist in the dimension that starts with pure methods research and ends somewhere with "the back of the envelope calculation" or "the horseback guess." We thought it better to invite all these audiences in the first round, and let the agency decide whether the conference was rele- vant enough to participate.

(7) With non-defense OR getting a stronger impetus, there is an obvious need to profit from the lessons learned m the Defense business. There is also the need to make sure that when defense readiness

is a relevant measure of effectiveness of civilian programs, it is included; the reverse is also true Furthermore, because of the scarcity of professional personnel at the present time, it is important that the growth of non-Defense OR not compromise Defense requirements.

(8) Procedural and institutional problems related to starting and effectively using OR groups probably have a large degree of similarity from agency to agency. The lone systems analyst who has been instructed to prepare a plan to start a viable activity in the agency should have access to plans that have a good chance of succeeding.

(9) There are agency officials who would like to get OR started, and would like some impartial

advice, but who aren't quite sure how to get it. Furthermore, they would like to see at first hand what

the nature of the projects suitable for such a group might be. (10) The agency OR groups need to have an inventory of external resources available to help him in his problem solving: universities that want graduate thesis topics and support, professors who want a sabbatical, colleges that are willing to carry out a class project on a systems problem for an

agency, etc.; contractors and their strong points and experience; expert personnel resources; data sources that could be relevant and could be used; etc. We recently compiled a list of contracting agencies with

a Washington office and which advertised OR or systems analysis as a specialty, and ended up with 40 on the first trial. RFP lists should be complete and relevant, and there is no good way of being reasonV
ably sure of this at present. It is not common knowledge how much or on what basis one agency can call on another (e.g., Bureau of the Budget or Bureau of the Census) for substantive help or data exchange.

(11) Currently employed methods of quality control on OR studies could be improved if experts

in other agencies were known and could be asked to participate in project design or review.

(12) There is a need for an experimentation laboratory in systems analysis in addition to the specific

agency opportunities. Field experimentation has always been part of a complete analysis, yet for most purposes, a truly experimental opportunity does not exist. (13) There will soon be a need for some analytic scheme to tackle the inter-agency program

balance problem, be it a system of rebuttal and debate with common reference terms, or a government

management game, or an inter-agency analysis group, or some other method.

(14) There is a need for an OR textbook suitable for Government agency use. (15) There may be a need for an explicit mechanism to rotate OR personnel among agencies, to universities and back, etc. (16) There is a need for a census of resources including hardware, software, programs, computers,

etc. The recent rapid growth in operations research/systems analysis/multi disciplinary problem-solving teams in the non-defense agencies of the Federal Government has opened a challenge of unusual proportions to the O.R. community.

At issue are questions concerning the development of analytic methods, test methods, data systems,

and means of drawing inferences in studies whose purpose is to improve the operations of the nondefense agencies of government. Questions related to the definition, the criteria, the measures, and the means of measuring the effectiveness of non-defense programs are but the beginnings of the work ahead. There is the problem of determining the best program mix within an agency whose missions are stated

in many different ways and which sometimes appear irreducible to a single common scalar purpose. There

is the question of how to determine the effectiveness of a Federal program when it is designed to assist

in improving individuals, communities, states, and regions, especially since the instruments of bringing

about that improvement are in the hands of many sources of power in the democratic system of rebuttal and debate that characterizes the American society.

Accordingly, operations research in the non-defense agencies is confronted with the unavoidable

task of discussing issues that are at the heart of our democratic society. It is important that analytical,

methodological, and substantive discoveries made in one agency that are suitable for use by another

agency be made a portion of the Government memory bank as easily and quickly as possible.

One of the aftermaths of the O.R. Conference was an expressed need, as articulated by many groups,

for the following two kinds of activity as a minimum.

(1) The convening of small informal ad hoc discussion sessions by agencies of the Government to which the O.R. people from other agencies are invited.

(2) The maintenance by some agency of an up-to-date mailing list of all those government agencies

O.R. groups, universities, and contractors, who should be informed of relevant items, including an up-to- date inventory of on-going O.R, projects and people finders in Government.

It appeared that each agency might want to take its own initiative in convening informal group

meetings of interest to itself. The Technical Analysis Division of the National Bureau of Standards

volunteered to perform the second function insofar as it was able to do so. In the course of preparing

for the symposium, we had already circulated questionnaires inquiring as to O.R. activities in various

agencies. The wide variations in respondents' interpretations of "operations research" precluded the

replies' being sufficiently consistent to warrant inclusion in these Proceedings, but the experience gained

should prove valuable to us in designing a more carefully phrased survey. Prior to the Conference, a number of agencies mailed us summary statements about the contents

of their operations research programs. These were posted at the entrance to the auditorium at the time

of the meeting. These forms appear to be a good starting point for the creation of an informal and

frequently updated notebook of agency O.R. work that might be circulated widely, both among the agencies and to external institutions such as consulting firms and universities which might assist in solving

agency problems. Our current plan is to issue that first set of reports as a separate document, and

then to undertake the administrative chore of issuing updated versions of it. We deeply appreciate your participation in this conference and sincerely hope that this volume will serve a continuing useful purpose for you. We hope that it will be but the first of a long series. W. E. Cushen, Chief

Technical Analysis Division

VI
PROGRAM

Wednesday, April 20, 1966

CHAIRMAN: Dr. Walter E. Cushen

Technical Analysis Division

National Bureau of Standards

Morning

ADDRESS OF WELCOME

Dr. Allen V. Astin

Director

National Bureau of Standards

GOVERNMENT SECT°R °

F THE

Professor Merrill M. Flood

Visiting Prof. Mgt. (MIT), and

Professor, University of Michigan

GENERAL DISCUSSION

The Improvement of Communications Among O.R. Government Agencies in the Civilian Sector ANALYS?S

ITS RELATI0N T0 SYSTEMS

Dr. John Haldi

Office of the Director

Bureau of the Budget

GUIDELINES FOR THE AFTERNOON PANELS

The Chairman

Afternoon

PANEL I: SPECIAL PROBLEMS OF O.R IN

CIVIL AGENCIES OF GOVERNMENT.

Chairman: Mr. Ezra Glaser

Health, Education and Welfare

PANEL II: MANAGEMENT VIEWS INOR/SA

TO FULFILL PPB REQUIRE- MENTS.

Chairman : Dr. Robert Grosse

Health, Education, and Welfare

PANEL III: BROADENING THE O.R. COMPE- TENCE OF MID-CAREERISTS.

Chairman: Mr. Joseph Lowell, Jr.

Civil Service Commission

GROUP SESSION

Chairman : Mr. John Moss

Social Security Administration

OPERATIONS RESEARCH RESEARCH

Dr. Alan J. Goldman

Operations Research Section

National Bureau of Standards

REPORTS OF THE PANEL CHAIRMEN

CONCLUDING REMARKS

Dr. Walter E. Cushen

Opening Remarks Dr. Allen V. Astin

Director

National Bureau of Standards

Washington, D. C. 20234

The National Bureau of Standards has a

statutory responsibility to provide scientific

and technical advice and assistance to other

Federal Government agencies. It is in this

capacity that the National Bureau of Standards has sought to apply operations research

to the solution of a variety of technical problems of the Federal Government and has served

as the sponsor for this conference. In seeking

to develop techniques and opportunities for the

wider application of operations research in the

Federal Government, we wish to pay tribute

to the Department of Defense for the leadership it has taken in this area for more than

two decades. Essentially we are seeking to achieve throughout the Government establishment the type of benefits whose possibility

has already been demonstrated by the Defense

Agencies.

The National Bureau of Standards has still another interest in the development and extension of operations research. The central con- cern of the NBS is in advancing and utilizing

the science of physical measurement. We believe, as did Lord Kelvin, that when you can measure what you are talking about and express it in numbers your knowledge is much

more useful than when you cannot measure it. In operations research we are seeking to ex- tend the measurement process into areas where

subjective judgment rather than measurement

has been traditionally used. Our interest in

operations research goes back many years. The

Operations Research Society of America grew

out of a symposium held at NBS under the

auspices of our Applied Mathematics Division.

Several years ago we established in that Division a section for operations research, and a year and a half ago we established a Technical

Analysis Division to serve as a center for the

application of operations analysis to the evaluation of problems within the Department of

Commerce and in other interested Government

agencies.

With this background information you will

readily understand my pleasure in welcoming

you to this symposium. I trust it will prove

fruitful in advancing both practical and methodological aspects of the application of operations research to the many pressing and diffi- cult problems confronting Government agencies today.

1

Operations Research in the Civilian Sector of the Government

Professor Merrill M. Flood

University of Michigan

Ann Arbor, Michigan 481004

It is a rare privilege for me to meet with

you on this occasion, and to have an opportunity to express some of my views regarding

the great opportunities for the future of operations research in the civilian sector of the government. Although I speak often, and freely,

about operations research opportunities with

respect to major national problems, and es- pecially in connection with those not directly

related to national defense, it is rare that I have an opportunity to discuss such matters

with those of you actually responsible for the

work in the federal government.

I believe, quite sincerely, that you have an

unparalleled opportunity and one unique during our time for any nation in the world, to demonstrate that the approach of the scientist,

the engineer, the mathematician, the economist, the behavioral scientist, and others

knowledgeable about the scientific method, and

adept in its use, can indeed contribute substantially in the future to the solution of some of our most important problems at the national

level.

I would not wish to take the time, nor is it at all necessary with this distinguished group of

operations researchers and systems analysts,

to attempt one more definition of "Operations

Research," or to offer one more listing of cur- rently relevant techniques, or to give one more

cataloging of the important tools used in our

trade. Suffice it to say that I share with all of

you the belief that our kit of techniques and

tools gets better each year, and that our shared

experiences in their use makes us increasingly

skillful and therefore more successful as the

years go by.

Nevertheless, I also feel sure that you will all agree that we have had very modest success

indeed, in comparison with our expectations

for the future ; or as judged objectively by ourselves and others as we assess past accomplishments, both inside the federal government and

elsewhere. I hasten to add that this is not in any

way intended to minimize the very substantial

progress already made with respect to national

defense—in this case even at the highest policy

levels. There has certainly also been progress

outside the Federal government, but as yet primarily at lower operating levels. I am sure that

we are all vastly impressed with the successes

of Secretary McNamara, Mr. Hitch, Dr. Enthoven, and their colleagues, in introducing systems analysis and other management and planning methods into the Defense Department so

effectively during recent years. I am sure that

we all share the feeling that President Johnson

has given this approach a tremendous boost by

his endorsement with respect to other departments of the Federal government; I look forward to hearing more about this from Mr.

Haldi, your next speaker.

You may or may not be disappointed to learn

that I do not intend here to suggest openings

for future effective application of OR skills,

primarily because my time is too limited for

this. Instead I shall concentrate upon one important national and world development that I feel is particularly significant for the immediate future, in the context of the discussions in

this conference.

. My remarks will concentrate upon opportunities for communication among all of you who

are concerned with the application of O.R.

skills, for I believe that the civilian sector of

the Federal government has a special characteristic that makes this opportunity exceptionally attractive in comparison with similar work

for national defense or in competitive industry.

I refer to the fact that the detailed results of

investigations can normally be passed freely

among all workers both during the progress

of the studies and when they are completed

and perhaps even more important, after there

is an opportunity to gauge the degree of success, or lack of success. You are all familiar with the considerable

difficulty encountered in communicating about

studies connected with defense problems. In

spite of the many efforts made to bring defense

operations researchers together, often under

the barrier of security classification, there has

been relatively little communication—at least

in the sense of free and open publication and

discussion of the type that has seemed to be so

vital for progress in the world community of

scientists. This conference is an example of the

kind of interchange that is possible, and should

be encouraged and promoted increasingly in

the future.

However, I am stressing quite another type

of communication, or of information exchange

as I should perhaps call it, that I believe will be

increasingly important and especially significant for O.R. work such as yours. We seem to be on the threshold, finally, of the next great

3
advance in our system for storing and communicating information of all kinds. I refer, of

course, to all those kinds of things that we

would bring into this conversation if we were

talking about computerized information nets,

automated libraries, electronic data centers,

and so forth. And, of course, I include in this

development ordinary communications as represented by publication, both in the traditional sense and, in the future, in new ways through

the information transmission net. These developments may even affect the nature of our con- ferences and meetings in the future, if and when

we can afford to interact frequently and effec- tively over a communications system and without need to gather together physically for such

interchanges.

I shall narrow my remarks even further, to a very special aspect of this newly developing

national and international information net- work. I refer to the storage and transmission

of raw data, or summaries of data, or formal

models that represent large bodies of data compactly, and all other "facts" that can be stored

and made accessible quickly and efficiently

through such information network facilities. Important among these many forms of stored

models are the increasingly powerful computer

programs that we can all share in use while

profiting from the systematic synthesis they

provide, often as a result of the work of a quite

small group of persons.

More specifically, I propose as well worth

your careful consideration, the possibility that

all operations researchers throughout the federal government can look toward the development of great information files that will be

accessible for all who have a real need. This

constitutes a very important new form of communication, since the information stored by one group need be made available to others' only

after they recognize a real need for it, and

then in a form where it has been precisely described and is easily used.

Some may say that my proposal is equally

applicable to many other kinds of persons and

activities. Although this is true to some extent,

and is certainly true very importantly in some

instances, I am arguing that the analysts

throughout the government are in the best position to develop these information stores so that

they are indeed reliable and can indeed be used

easily. Nor do I mean to limit this to relatively

simple tasks such as the standardization of

accounting data, or programming and budgeting data, important as these are; rather, the

intent is to include as data such things as a

highly perfected and validated simulation of a major agency or operation within the government. I hasten to remark for the benefit of any who may feel that I am far too optimistic about

such hopes, that I would expect this to be a

gradual development but will be very surprised

if it is not clearly significant within ten or

fifteen years from now.

Let us consider the proposal from a rather

different point of view, stressing the importance of working toward a systematic understanding through modelling, of larger and

larger portions of the Federal government. It is probably impractical currently to construct a

valid mathematical model, or useful computer

simulation, of the entire Department of State i or Department of Justice, or even of major

subordinate bureaus or agencies. But I would

not consider it naive, or undesirable, to begin

the attempt immediately.

I am reminded of the story about President

Jefferson when he was impatient with his gar- j dener about planting a tree and immediately

responded, when reminded that the tree would

not reach maturity for 20 or 30 years, that this

made it all the more imperative to plant it that same day. It is in this spirit that I urge

| the importance of keeping this larger goal in

1 mind over the coming years, as you all meet

with success in analyzing and understanding

larger and larger segments of Federal governmental activities.

For example, and from a purely practical

standpoint, I would think it not too soon now to begin the development of models that might

eventually enable us to evaluate tradeoffs between such major activities as education and

justice. More specifically, it should be very

instructive and helpful to make a study that

attempts to evaluate the tradeoffs between an- other million dollars for the FBI in contrast to another million dollars for educational research, or for care of the aged, or for foreign aid, or

for any one of the other major national programs that surely are all directed toward the same set of national goals. This is not at all to say that such work will be easy, or yield immediate results, but it will happen sooner and

better if an early attempt is made to develop

the study activities in a way that will increase

the likelihood of eventual success. How might some relative simple steps be

taken now to promote progress of the kind I have proposed? Obviously there are many present activities that are substantial and already

making contributions in this sense. In particular, the activity that Mr. Haldi represents,

will surely yield very important ingredients in- deed for just such tradeoff evaluations.

My primary suggestion is that an explicit

attempt be organized and made to accelerate

this advance by cooperation among all of you

in the development of a computerized information network that will store such information

and make it much more readily accessible

throughout the Federal government for use by
operations researchers, and for others having

need for such a resource. As J.C.R. Licklider often puts it, you can

create your own online intellectual community.

My own first association with an operations

research type of effort within the Federal government was a little over 30 years ago when

I worked with a distinguished panel of technical experts on an advisory task for Attorney

General Homer Cummings. Our panel was part

of a broad study under Dean Wayne Morse,

now Senator Wayne Morse, that was intended

to bring expert advice to the Attorney General

in connection with his broad responsibility for

parole and release procedures. This panel was

chaired by the late Professor Samuel Stouffer,

eminent sociologist, and included among others,

Samuel Wilks and Frederick Stephan, both distinguished statisticians. The panel also included

criminologists, economists, psychologists, mathematicians, and lawyers and was supported by

a strong staff group.

We could barely discuss basic questions such

as the negative social value of a released murderer, or other criminal, as compared with the

social value of keeping him in custody or under

parole supervision. It was my impression then,

at a time when I was engaged in studies of this

kind in New Jersey, that we knew essentially

nothing about major social questions of this

type, nor did we have the talents or the trained

personnel to tackle them. From what little I know at present about this field, I suspect that

we remain pretty much in that same position

today, except for the fact that we certainly have

new talents and much more expert personnel

who could tackle such problems—perhaps with

considerable success. The newly established National Commission

on Crime, and similar commissions that have

been established recently in some of our states,

reflect a growing public awareness of the importance of dealing with major social problems more systematically and more effectively.

My most recent operations research kind of

service for the federal government was a study

that I completed this past September for the

National Commission on Technology, Automation and Economic Progress. My assignment was a study of the national information net- work development, or the "computer utility" as it is sometimes called at MIT by Professor

Fano and others. My report is just now being

published by the National Commission, but

Dr. Cushen has asked me to comment regarding some of its conclusions — I am, of course, pleased to have this chance to do so. Briefly, I am impressed by the great opportunity afforded our nation in terms of the

rapidly developing information network, which

is a good and exciting prospect indeed, but I am

disappointed by the slow pace at which this

will all move unless steps are taken by the

Federal government and industry to anticipate

and correct some of the glaring deficiencies in our communications system and in our laws

and administrative practices. I suppose that none of the difficulties I foresee has already

been observed and discussed somewhere within

government or industry, but I concluded on the

basis of my study that a great deal could be

done to improve the situation.

For example, the common communications

carriers, such as the American Telephone &

Telegraph Company and the Western Union

Telegraph Company, are moving very vigorously to develop and construct a national communications network that will permit the speedy and

efficient transmission of data as well as voice

messages. However, it is generally agreed by

experts on communications systems, who are

also knowledgeable about data transmission

systems, that this must be a "store and forward" system, rather than a system which

merely provides an exclusive line connecting

two users for some period of use. In other

words, data will originate at many individual

nodes on the network, then be transmitted to intermediate nodes where it will in turn be

sorted and re-routed, perhaps after temporary

storage, and eventually reach "terminal"

nodes, where it will often be processed and

sent back through the nodes of the system,

often through quite complex paths.

All of this requires new kinds of electronic

switching centers, computerized to handle, store and forward activity of this type, and industry

is moving ahead to provide this type of service.

Unfortunately, data transmission demand will

surely exceed transmission capacity in the very

near future. The need is to take steps soon which will minimize this disparity between

capacity and demand in future years; my re- port suggests a few moves that might be made

toward this goal.

For example, there seems now to be no single

Federal agency that is clearly and obviously

charged with responsibility for insuring that

this important development does take place

within our nation at the proper pace.

Another example, and a lesson learned from

experience with some of our major time-shared

computing systems, such as one at MIT and

another at the System Development Corporation, is the importance of bringing the future

users into the system early to help ensure that

its developing direction is a good one. Toward these ends, the report proposes that

at least one major national information net- work be sponsored by the Federal government

in the near future and that several major sys- tems be made available in the immediate future

for use by individual citizens, as students, or housewives or merchants, and so on. The report

5
also stresses a number of social problems that

require changes in legislation, or in regulations, or in administration, if we are not to suffer

from too leisurely development of the national

information network facility. The report also

urges that the Federal government look very

carefully at its own internal operations to en- sure that this type of network capability is made available and used appropriately soon by Federal agencies.

It is in this spirit, within the Federal government, that I believe there is an unusual opportunity for analysts to cooperate in specifying

the nature of such a system in order that it meet their needs later. Meanwhile, there will be

many opportunities for all of you to exert your

influence through usual channels as this developing information network progresses within

the government, and for your agencies.

It may strike you as strange that I have

devoted most of my time to a proposal that

rests so heavily upon recent technological developments in information networks within the

Federal government and elsewhere. I have done

this partly because of my recent experiences

with such activities, but primarily because I have been deeply interested in these developments for I feel that they are so centrally important for the future of scientific and

analytical efforts to improve operations and

management within the Federal government

and elsewhere.

I close by repeating my opening remark, that I the special opportunity that seems to me to be

available to all of you working within the Federal government on problems primarily within

the civilian sector is the relative openness of

this system and the opportunity you have to communicate with each other regarding your

work both while it is in progress and after it is completed. Furthermore, I see the developing

information networks providing you with a

truly advanced method of communications, or information transfer, since you can in the

future share your experiences and your data by

storing them in the information network and

taking advantage of that store as you each ;

i proceed with your own work.

If the information network facility will be

with us in the near future, then I predict that I you will be sorely pressed to develop good meth- 9 ods in time for preparing your experiences for

storage in that system, so that they can in fact

be useful and be used by others working on

similar problems. Just as President Jefferson

remarked with respect to his tree, since it will

take so long to finish, you should begin immediately.

I

—
The Planning-Programming-Budgeting System

Dr. John Haldi

Program Evaluation Staff Bureau of the Budget

Washington, D. C. 20234

I welcome this opportunity to discuss the

Planning-Programming-Budgeting (PPB) sys- tem with you. Historically, suggestions for Federal program budgeting date from 1912, when

a Commission reporting to President Taft proposed that a "budget bureau" be established

and that Government agencies present their

budgets along program lines. More recently,

two Hoover Commissions have recommended

the improvement of Federal management

through the adoption of program budgeting.

Before Mr. McNamara came to the Department of Defense, many agencies already had

some form of this type of budgeting. Many

facets of PPB are in use in certain Government agencies.

It is probably the combination of Planning,

Programming, and Budgeting into an integrated system which is most new to all Government

agencies. The system, which most economists

view as a straightforward application of com- mon sense, has a few fundamental concepts.

All agencies are being required to: (1) define

their activities by objective within what is called a program structure, (2) prepare a multi-year program and financial plan, (3) translate program decisions into the conventional

appropriation format, and (4) provide for

periodic reappraisals of their major programs

and objectives.

The first step in the installation of a PPB

system is definition of outputs and drawing up

program structures or outlines grounded on

those outputs. For the last several months all major agencies have been developing program

structures focused on what the agency produces, whether for the benefit of the public or

of other Government agencies. The program

structures include categories, subcategories,

and program elements. Program elements are

the smallest units in the program structure of an agency.

The major purpose of the program structure

is to serve as a tool for top management review.

We hope that it will lead managers to ask the

"right questions"—as President Johnson has

said, to reexamine the programs to see if we

are really doing the right thing, or to see whether we are actually funding obsolete programs. The primary focus of this exercise is to

make sure that we are selecting the right alter- native among plausible policies, and directing resources to their best use. A program structure, intended primarily for consideration by

the Secretary, the Director of the Budget Bureau, and the President, must be aggregated

into a few meaningful program categories.

Here are illustrations of program categories.

Example 1: A program category in the Department of Defense is "Strategic Retaliatory

Forces." This program consumes billions of

dollars. Subcategories within it are "Offensive

Forces" and "Defensive Forces," and Offensive

Forces are broken down into sub-subcategories,

such as "Manned Bomber Systems" and "Missile Systems." Program elements in the Defense

structure are B-52's and Polar Missiles. In the

Defense Department structure, the element

"Polaris Missile System" is the lowest level of

detail shown. Yet this element has a budget

which is bigger than many total agency budgets. Included in the Polaris Missile System ele- ment are not only the missiles, but also the

submarines needed to keep these missiles on

station, the men that man the submarines, and

the tenders and other supply facilities neces- sary to keep these submarines operating. Thus

the program element is an aggregate of many

things. This illustrates the total system concept. It enables management to see at a glance

how much of its resources are devoted to this

specific element. The program element indicates

the purpose for which these resources are brought together, and it provides a focus for

the measure of performance. Management's at- tention is thus focused on the performance of

the entire weapons system and not on the performance of support items.

Example 2: In many instances, Government

agencies do not produce tangible products. The

Department of Justice, for example, produces no hardware. Yet here, again, program budgeting means deciding what the money should be

spent for. That is, the purposes for which the

program is engaged, and what the program ac- complishes. The Department of Justice program structure would include programs dealing

with anti-trust declaratory policy, investigations and prosecutions under O.R. relating to other laws which are the direct responsibility

7
of Justice, investigations for other Government

agencies, etc. Example 3: The program structure for the

Forest Service includes such major categories

as timber production, wildlife, grazing, and

recreation. Within the Department of Agriculture also, there are services directly providing

outdoor recreation facilities, and also two more

services indirectly providing recreation through

conversion of crop land. This year, for the first time in Federal budgeting, the Department of

Agriculture has pulled together all its efforts on recreation and reviewed and analyzed them

at the Secretary level. This review is one immediate benefit of program budgeting.

This Recreation example shows how program

budgeting can shed light on Government programs. For example, the Recreational Advisory Council produced a study which showed

that about 80 percent of all Government ex- penditures on recreational activities are made

west of the Mississippi whereas about 75 percent of the people live east of the Mississippi.

Because of PPB, we have now changed the

emphasis to providing more recreation facili- ties in the East.

Example W- Some agencies, like those just

discussed, use a mission-oriented program

structure. The Post Office Department, on the

other hand, has a program structure oriented

toward function, because that seems more appropriate for their needs. The four most important categories in the Post Office program

structure are: (1) Getting Mail into the System, which consists of two primary subcategories—window operations and mail collection

services; (2) Processing the Mail; (3) Transporting the Mail; and (4) Delivering the Mail. In addition, there is the catch-all "General Support" category for overhead.

"Processing the Mail" is an intermediate

operation which might appear to be merely part

of a broader system. Why was it made a major

category? The reasons are: —Over $2 billion is spent to process the mail

and —Mail processing involves many interesting "trade-off" possibilities—alternatives

which merit investigation and analysis.

The dollars involved raise technical discussions

all the way to the Postmaster General, and mail

processing decisions are often made at that

level. Even though mail processing is an intermediate output, there are sound, pragmatic

reasons for considering it a program.

Although some agencies have outputs which

cannot be measured in physical terms, almost

all programs should be output-oriented. This

method shows what is to be accomplished, not

merely the old accounting of object classes, or inputs (personnel, buildings, vehicles, etc.)

Setting up a structure requires rigorous con- o ceptualization, but not many man-months. Once

the structure is set up, an enormous amount of

work goes into the Programming task. This in- i volves taking the existing budget and converting input categories into new output format.

It produces what is called the "crosswalk," and I this is essential since the traditional categories are used for submissions to Congress. Since

much top management time is devoted to budget review, and since top management

wants to think in terms of programs, it is important that agency resources be structured in

this way. Programmers speak the planners'

language, and they are expert at checking and

translating costs. Through programming we

ensure that work of the budget people and the

planning people corresponds even though the

conventional budget structure is used with the

planning structure. Once the structure is set up, the agency begins developing a multi-year Program and,

Financial Plan (PFP), which specifies the out- puts and cost estimates for each program ele- ment. The PFP time span can be as short as

five years beyond the current Fiscal Year and

the next Fiscal Year, or as long as ten years

for the Corps of Engineers' water projects, or even 80-100 years for the Forest Service planting program. In the PFP the agencies are asked

to reflect a comprehensive plan, corresponding

to the program structure. The PFP is the integrated financial document for the agency. Three

sets of tables make up the PFP: Part I sets forth outputs for each year by program, ele- ment; Part II shows dollar costs associated

with each program element; Part III shows

whatever other information the agency determines is relevant and necessary, including al- ternative measures of outputs. Agencies that

have not been obtaining such data all along may

soon be making somewhat greater demands on

field personnel than in the past.

Analysis is the cornerstone of the PPB

system. We are giving this major emphasis— ? forming planning and analytic staffs at the

Secretary level where they do not exist, and f improving those which already exist. Without \ much better analysis than we have previously ] had, this whole effort would fail. This year the budget submission for the

g spring preview will consist solely of the Program and Financial Plan (PFP) and Program \ Memoranda (PM) in support of the PFP. The

Program Memoranda are to be hard analytic

documents which deal with the issues confront- ! ing the agency and which propose alternatives

for study during the year, before critical deci- i sions are made. I stress the word "alterna- v

8
lives." This is fundamental to the PPB system.

Without alternatives, the analyst is likely to be- come the decision maker. If a study is presented

with costs, details, and recommendations—but

no alternatives—then the agency head's choice

is either to accept the study, stick with the

status quo, or perhaps, suffer a budget cut. The reason a study is usually commissioned in

the first place is that the agency head (or perhaps the President) has already decided that

the status quo must be changed.

PPB does not envision all-powerful analytic

staffs. It is envisioned that these staffs will come up with alternatives, will study their pros

and cons, quantify their implications and present them to the agency head. Under PPB

i the agency head may become much more

involved in decision-making in the future than

previously.

The Budget Bureau is interested in seeing

how well alternatives have been studied. A good

review of alternatives is fundamental to justification of major expenditures. The Bureau of

the Budget does not and will not make the

agency's decisions. It may make recommendations regarding programs, and it may send proposals back for further study. The Bureau does

not anticipate doing studies for the agencies,

and it does not intend to develop a vested position in special analysis. The analysis will be

done in the agencies.

In the near-term, we hope to improve major

decision-making. In addition, we hope to see improvement in daily management. If we can

develop better measures of agency output, in

the long run we can use these measures as primary means of management control. This

will also affect the budget making process. In

the long run, perhaps in the 1980's or later,

I anticipate that PPB will not only give us

better program planning, but also help us achieve more rational control of the day-to-day

functions.

Appendix

In accordance with the instructions in the

now well-known Bureau of the Budget Bulletin

66-3 and its Supplement, Government agencies

are busily engaged in developing their Planning-Programming-Budgeting (PPB) systems.

The Program Evaluation Staff, which I represent, has been deeply involved in the Budget

Bureau's recent task of overseeing the establishment of the PPB system, in offering guidance, and in arranging for training opportunities for the agency staffs. In this paper, I will

emphasize the "analysis" and "planning" as- pects of the documentation and conclude with some implications of the advent of PPB.

PROGRAM STRUCTURES. Agencies have

been defining their activities in a systematic,

formal manner through categories, subcategories, and program elements which describe

what the agency does. For this year's spring

preview, that job is now finished. In some instances, however, the existing

structures still contain obvious shortcomings. A

good program structure should define agency

outputs in a way that will be of most use to agency top management, and some structures can be considerably improved in this regard.

Future refinements must, therefore, seek to ac- complish this. A second problem with some program struc- tures is the lack of uniformity among agencies

j engaged in similar programs. For example, at

least three major agencies are involved in providing outdoor recreational opportunities:

Agriculture, Interior, and the Corps of Engineers. All three agencies now show the recreational opportunity they provide by geographic

region, but the regions don't coincide. If each

of these agencies reported the recreational

opportunities they offer along similar geographic lines, it would then be very easy to construct a total Federal recreational budget

which would show exactly (by area) what the

total program amounts to. In many such areas interagency uniformity of this type is desirable. This year, though, there simply has not

been enough time for any of this to be

accomplished.

PROGRAMMING. This system formally es- tablishes the base five-year plan in detail, dic- tates how program change proposals will be

processed, and determines when and how the

base-year plan will be extended.

A better grasp of what programming is about

can easily be obtained by considering some of

the questions which programming staffs have

been grappling with recently. For example:

What information should be requested from

submitting bureaus or agencies? How much detail should the initial plan contain? How much

detail should the plan ultimately contain? What

forms should be used? To what extent do Department of Defense forms and procedures

apply to an agency, and to what extent is DOD

experience not applicable?

The programming system must provide for

a "crosswalk" which will convert the budget in

the new program structure into the present
appropriation structure. But how much accu- racy and detail are needed in this conversion?

To what extent is accounting precision necessary, and to what extent are statistical cost- finding procedures acceptable? Should we try

to allocate any joint costs? Which joint costs?

What allocation procedure should be used? Or

should all joint costs be put in a "general support" category? In passing, let me note that

there are no pat answers to these questions.

Good programming systems must be individually tailored to the particular needs and circumstances of each agency, and good programming

systems therefore demand a lot of hard work.

Definition of the "costs," to be shown in the

PFP, has been something of a problem. Some

financial people have interpreted "cost" with

much preciseness. However, the Supplement to Bulletin 66-3 was written chiefly by economists

who were thinking of costs in an analytic sense —that is, the cost of the resources that go into

the plan. Now it so happens that one of our BOB Circulars contains a precise, accruedaccounting definition of costs which includes

depreciation, changes in inventories, accrued

annual leave not taken, etc. In application,

many have interpreted costs as used in the

Supplement to Bulletin 66-3 in terms of this

formal definition, while it was our intention to interpret costs variously for different agencies

in whatever way makes the most sense from an

economic point of view. For this year, we must

live with whatever confusion we have generated. In the future, these multi-year program

and financial plans should be tailored to suit

the needs and the requirements of the individual programs.

One comment about costs which I feel can be

made with some degree of assurance is that in

future years the multi-year plan will show in- vestment and operating costs separately for

each program element. Many of the problems

that we are now trying to solve will then be a

lot easier to handle.

MULTI-YEAR PROGRAM AND FINANCIAL PLAN. Probably the two most important

parts of the whole PPB effort are output data

and analyses. These are covered in two basic

documents : the multi-year Program and Financial Plan and the Program Memoranda.

The Program and Financial Plan (PFP) will

consist of three parts—all tables. Part I will be

a table of outputs, showing measures for each

program element expressed as physical units

of what an agency is building, producing, or encouraging, year by year. Part II will show

the financial implications of Part I; e.g., how

much the units of the building plan are going to cost each year. Part III will contain relevant

supplementary tables.

Submission of a multi-year Program and

Financial Plan to the Bureau of the Budget will

be new to all agencies. The degree of additional

planning required will vary considerably among

agencies since some have been planning ahead

only on a year-to-year basis.

The PFP is not intended to be an analytic

document and it should not be interpreted as a

refined cost-benefit analysis. We are not going

to be able to take the cost in Part II, divide it by unit outputs in Part I, and automatically come up with something that says this is the

"cost per unit of output" or that the reverse shows "benefits per dollar."

The PFP is a summary planning document.

As a quantified expression of an agency's plans, j it becomes a valuable document for dialog

among the bureau chief, the head of the

agency, the Director of the Budget Bureau, and

the President.

PROGRAM MEMORANDA. This is the j analytic document. It should state the basis

and objectives of the analysis, describe the concepts and assumptions that were used, and contain all the narrative associated with the budget

submission. Just as the PFP states what is planned, the Program Memoranda explains I why.

This year, of course, we do not expect to J receive a thorough analysis of all programs.

But we do feel there is time to (1) formulate I reasonably clear and precise program objec- I tives, (2) define the major issues, and (3) at ; least mention some of the principal alternatives

j which were weighed against the selections reflected in the FY 1968 budget. Our plan at BOB

is to give each agency a critique on its PM's

and have them reworked during the summer ! and resubmitted in the fall. The final version ill of this year's PM's will be the springboard of

those to be written for the following spring

j preview. Through this process, we hope to in-

,jj ject a note of continuity to the budget review j process.

Let me elaborate for a moment on the subject I of alternatives and their role in the budget j review process. Past agency submittals to the

Budget Bureau rarely suggested program alter- \ natives (except for Defense, which now makes i this a regular habit) . The fact that the Budget i Bureau is now asking for alternatives to be j discussed explicitly has caused many people to i ask me whether BOB intends to start "running j the aerency" or "making major agency deci- i sions." The answer to such questions is decided-

| Iv NO ! If you will reread the instructions carefully, you will note that they do not ask the

agency to submit alternatives to us for a decision. Budget submissions should continue to

reflect decisions made by the agency head. However, when significant sums of money are involved, decisions should reflect the agency's

10
judgment of what constitutes the best choice

among the most imaginative alternatives possible. That is, major- decisions with large spending implications should be the consequence of a

systematic search process. (NOTE : The Airlift Sealift memo contained in the Supplement

to Bulletin 66-3 illustrates a proper procedure

for handling alternatives in budget submittals.)

A tenet of program budgeting is that until the

;, best and most efficient means of achieving a

stated goal has been thoroughly and systematically searched for and analyzed, major program

spending decisions are not justified. I repeat:

the Budget Bureau is not asking any agency to abdicate its decision-making responsibility.

The Budget Bureau is asking each agency to justify its decisions by showing that it has dili- gently searched for the best and most efficient means of achieving stated goals and objectives. In the past, justification of this nature has

been, by and large, noticeable by its absence.

The old form of "justification," which rarely

j contained the rational basis for any justification whatsoever, is no longer considered

acceptable.

ROLE OF THE ANALYST IN PPB. Let me

distinguish between decision-makers and analysts in the PPB system. They may, in rare

cases, be the same individuals, but there is a

conceptual difference between the two roles

which is worth noting.

PPB assumes that the decision-makers are agency heads and their principal assistants.

Analysts are people trained to dissect problems, to look at them different ways, to develop

alternatives, to employ quantitative analysis

where practicable, and to make suggestions for

the decision-maker's consideration.

The phrase "develop alternatives" is really

the key to the distinction between decisionmakers and analysts. All too frequently a proposal submitted to the decision-maker describes

but one course of action. The only choice the

decision-maker then has is to accept it or reject

it. This procedure is neither logical, necessary, nor desirable. It de facto turns the analyst into

the decision-maker. Program budgeting strives

to separate the two functions by having analysts develop and present choices between

meaningful and imaginative alternatives.

11

Operations Research Research and Government O.R.

Dr. Alan J. Goldman 2

National Bureau of Standards

1. Introduction

I'd like to lead up to my "message for today"

with a little analogy. It concerns, on the one

hand, the typical relationship between the manager and the operations researcher—and, on

the other hand, that between everyman and his

physician. What I have in mind is that, unfortunately, most of us don't visit a doctor unless

and until we're already feeling uncomfortably

ill—and when we do arrive, what we long for is that famous "fast, fast relief." Similarly, an

operations analyst's introduction to a new task

is all too often in terms like these: "We have

this problem. It's hurting us badly, very badly.

Please give us a guaranteed optimal solution by

yesterday."

Under these circumstances, it's pretty clear

what will happen. Perhaps, by rare good fortune, the problem will happen to fall neatly into one of those areas in which both theory and

practice are especially well developed. But most

of the time, the "solution" obtained will necessarily be of the "quick and dirty" variety.

Such a solution may (quite properly) be

accepted and acted upon as "best available."

However, it may in fact be so crude as to lead

to really unsatisfactory results—we all know

that even our best efforts can't ensure the

soundness of an O.R. study's conclusions. Or,

the quick and dirty solution may be technically

satisfactory, but considerably more expensive

to implement than some alternative that was

missed. The proposed solution may even be

fairly acceptable on all counts, but the manager

will never know how much better an answer

might have been forthcoming if only a little more time or effort could have been spared

or, and this is the possibility I want to stress

if only the state of the art had been a little more advanced.

Revised text of a presentation to the Operations Research Conference for Non-Defense Washington Area Government Agencies. 4/20/66. Includes sections omitted from the delivered address for

the sake of brevity. I am indebted to J. R. Rosenblatt and J. M. Cameron for helpful comments.

2 Chief, Operations Research Section, Applied Mathematics Division Institute for Basic Standards, National Bureau of Standards. Washington, D. C. 20234. The text is an expression of the

1 author s views on the subjects treated, rather than an "official pronouncement" of any sort.

This brings me close to my message. If most

government practitioners of operations re- search are feverishly racing the clock in at- tempts to solve pressing practical problems,

then just who is going to advance the state of

the art? There is a quite obvious need for the

analog of medical research—for groups whose

dominant concern is not the rapid resolution of

specific agency problems, but rather the sys- tematic improvement, extension and creation of

the methods of operations research. This is what I like to call "operations research re- search," i.e., research into the techniques and

tactics of operations research. Abbreviation:

O.R.R.

A natural reaction is that O.R.R. activities are properly the province of the universities.

And so they are largely, but certainly not exclusively. It's easy to see why:

(a) We're concerned here with research

which is applied rather than basic,3 i.e., takes its main stimulus from present and anticipated

real-world problems rather than the general

urge to advance human knowledge.

(b) More specifically, we're concerned with

methods of operations research needed in government O.R. studies. Appraisal of these needs

requires extensive exposure to such government

studies; bringing the new methods into effec- tive use quickly also requires close contact with

the broad stream of government O.R. activity.

I suggest that any university group which finds

itself in such a position needs to do some

earnest soul-searching about the primary functions of a university.

(c) Any university outfit will (and often

should) be deeply concerned with issues distinctly peripheral to the needs of governmental

O.R. A not infrequent occurrence is the tendency to slide the research toward some area which

is sufficiently fashionable academically as to permit the guiding professors to publish rather

than perish, and the guided graduate students

to achieve acceptable theses. 4

3 This rather fuzzy distinction is inappropriate in many of the

contexts in which it appears, but seems genuinely relevant here.

1 The phrase "not infrequent" was carefully chosen; "typical" would have been unjust.

13
2. Government O.R.R.: Demand and Supply

For the reasons just given, there's a clear

necessity for operations research research

groups within the government, located at focal

points for a number of Federal O.R. studies.

Let me spell out this necessity more clearly.

Among the effective resources of the operations

research practitioner, I do not count all that

has been discovered, or even all that has been

published, but rather only those items which

have been so clearly written up and so appropriately implemented (e.g., in a computer program) as to be truly "available" for application or adaptation within the time and resource

limits of a particular O.R. study. On this basis, as a direct or consultative participator in a number of government studies, I can say flatly

(and with no doubt that many of you have had

similar experience) that the state of the O.R.

art is all too often a recognizably critical limiting factor on the quality of what can be

achieved within an assigned deadline or budget

ceiling. And instances of this kind go unrecognized in many more cases, since we naturally

tend (without conscious decision) to formulate

problems only in terms which correspond to an

available "solution" method.

What makes this even more frustrating is the presence of "observed regularities," i.e., the recognition of essentially the same gap in

methodology or implementation as arising in

project after project, although no one sponsor

has the time or resources to permit attacking

the beast head-on. So we end up ad-hoccing our way through, in more or less makeshift fashion, on each individual occasion. The efficient approach to these recurrent problems often involves looking at a version general enough to refer not only to Agency X's problem in 1966,

but also to what will be the problems of umpteen other agencies (but perhaps not Agency

X) by 1976. As we know, to support such re- search out of Agency X's 1966 budget is typically unthinkable. Work of this sort naturally

constitutes the role of an independently supported operations research research group with

the specific mission of ascertaining and re- sponding to these general needs of governmental O.R.

With the "demand" side of the picture for

government operations research research thus

indicated, let's turn to the "supply" side. First,

the military O.R. sector. Here one thinks at once of the outstanding achievements of the

RAND Corporation, then of the many contributions of ORO/RAC, OEG/CNA, WSEG/IDA,

etc. (Note, though, that none of these are exactly within government.)

Now for the civilian side. To exclude the

other "giant" as well, I'll call it the non-military

terrestrial sector, comprising government activities concerned with such minor items as the

education and international dealings. Despite

nation's economy, natural resources, health, some notable individual efforts, if one compares the available resources for government

O.R.R. in this sector with:

(a) the difficulty and number of the problems involved, and

(b) the resources for analogous functions in

the military sector, then it seems very hard to arrive at a rating significantly different from

zero. The gist of my "message" is that this just

won't do. It is a platitude that yesterday's O.R.

methods will fall short of adequacy for today's

problems—they fell short for yesterday's, and

today's are certainly no easier. Moreover, I can | see a clear and present danger that techniques

developed for military O.R. studies will be j stretched well past the validity point to cover

the kinds of objective functions injected into

non-military analyses by public interest considerations.

The contention here is that lack of appreciation for the need for adequate civilian terrestrial government O.R.R. has hurt us in the

past, and is hurting now. Without remedial action, the pain will rise to the agony level as PPB and its offspring make ever-increasing

demands on the O.R. capabilities of all government agencies, and thus on the O.R. art itself. And these are needs which can't be met "on the j cheap," in dedication or time or dollars. Who should bell this cat ? Despite the government tradition that questions of organization

generally take priority over questions of substance, I have no considered views to present.

In the long run, it seems quite likely that each l Executive Department (and several of the ma- 11 jor independent agencies) will require at least one O.R.R. facility attuned to the peculiarities

of its own needs.5 In the short term, it seems

apparent from my prejudiced viewpoint that m one natural locus for such work will be the i National Bureau of Standards, with its tradi- ll tion of O.R.-type services to other agencies, i and its special assigned responsibilities regard- I ing both (i) cost-benefit analyses and (ii) help- • ing to achieve increasingly efficient government 1 use of that very valuable O.R. tool, the digital

computer. But my personal acquaintance with ' many talented government O.R. professionals J just doesn't amount to the kind of knowledge,

of agency structures and traditions, that would i permit me to identify the other natural loci or t nuclei which must surely exist.

5 Such facilities would still need access to specialized consultants expert in the more esoteric spells.

14
3. Data, Dissemination, Documentation

Before becoming a little more specific about

promising areas for operations research re- search, I want to remind you of some other

aspects of government O.R. (or operations re- search in general) which have their own unpleasant features to add to the difficulties arising from inadequate methods.

One such aspect is the typical quality of the

data with which we must deal. It is depressing

to recall that to the scientist, "data" are carefully defined numbers derived from precise

measurement.53 It is even more depressing to observe that according to the Latin, "data" means "things which are given," i.e., are lying

at hand and freely available. In contrast, the

numerical material of most O.R. studies seem to

fall into four categories:

(a) Predictions or extrapolations into the

future. My only comment here is the recom- mendation that anyone who applies the word

"data" to any glimpse into any crystal ball should be shot as soon as possible (within

constraints of time and budget, of course).

(b) "Data" which, though essential to your

purpose, were so far from "freely available"

that acquiring them has left you too exhausted

and broke to go on and use them. In the military O.R. field, it has not been uncommon for

inter-service rivalries to force data-gathering

to take forms just short of espionage. I hope

this won't recur in the civilian sector.

(c) "Data," so-called, which started out as reasonably honest numbers. But they've been

aggregated and re-aggregated and disaggre-

• gated and interpolated and scaled and adjusted —in short, subjected to so many operations,

each somewhat open to question, that the ratio

of fact to artifact is slim indeed. You don't

want any decision significant for our nation to lean on numbers like these, put together with

pipe cleaners and chewing gum. You wouldn't

even want the fate of Northwest Saratoga Falls

to depend on them. But—they're what you've

got.

(d) My final category consists of those numbers which are indeed "freely available," but

simply aren't to be believed—they have perhaps been prepared to bias the analysis you're

attempting, or else with no axe to grind but in

total ignorance of proper sampling technique,

or may reflect drastically different usage of cer- tain key words by you and by your informant.

This list of horrors reminds us that O.R.

studies often involve conceptually complex

processing of their input data; knowing how

far we can go (and in what directions) requires

a rather sophisticated understanding of just

G« The contrast being drawn is too sharp to he blunted by the

qualifications needed to make this over-simplification respectable.

what the numbers mean and how they were

arrived at. It requires recognizing that the

"data" as received are generally not raw strings

of digits, but rather have had a structure implicitly imposed on them by the methods (accounting rules, consistency criteria, formulas

based on economic or social or demographic

theories) used in checking and "adjusting"

them. Hopefully our models will be compatible

with this structure, perhaps even incorporating some of its elements. Achieving the kind of un- derstanding yearned for here, would certainly

be facilitated by close association with the planning and evolution of the coming generation of

Federal reporting systems.

So much for data. On dissemination, I simply

call to your attention how much high-quality

operations research work, which could provide

essential starting points for efforts by other

agencies, is either never written up properly, or written up only years and years after the

fact, or is written up well and on time only to be buried in the deepest dungeon of "administrative classification" by the agency involved.

This leads to a third striking aspect of the

government O.R. scene—the scandalously low

average quality of documentation of (in-house

and contracted) operations analyses. Let me describe the standards used in that appraisal. In

looking at a piece of O.R. documentation—from

the viewpoint of the analyst, not necessarily

that of the manager for whom the study was

made—what is one justified in expecting?

To permit concentration on the intellectual

content, the text must be written in clear and

reasonably smooth style, without side expeditions to drag in and display clusters of whatever terms (e.g., "system") are believed to be

in current fashion. Turning to substance, we

note that the motivation for reading the document is presumably a hope of employing its results as partial basis for some decision or as inputs to some further analysis, or perhaps of

adapting its methods for use in some new study.

None of these purposes is well-served unless

the serious reader can without undue effort

track down what assumptions and approximations were made (and with what justification),

what particular meanings were attached to significant, but. somewhat ambiguous terms, and so

on. Getting at this sort of information is hard

enough today, when the principal author is perhaps only as far away as the other side of the

Federal Triangle. And if the present quality of

documentation remains typical, and if Professor Flood's predictions are realized and one will

typically be applying X's model to Y's data

after picking both up "blind" through computer tie-in with points 1000 miles distant

L5
then the odds against producing anything but

garbage seem intolerably high.

But there is still another class of readers to consider, those anxious to find what the course

of progress of the documented study has to teach us in operations research. They would

need to know what alternatives to the approach

actually chosen were considered, and why they were rejected. What was the institutional

framework in which the work was done, and

the backgrounds of those with whom and for

whom the work was done, and how did these

factors influence the conduct of the study? If a

predictive model was developed, what provision was made for follow-up observation to permit

adaptive corrections, and (hopefully in a fol- low-up volume) how successful were the predictions and what are the implications for the

predictive technique used?

Some of the reasons for the present average

level of O.R. documentation lie fairly near the

surface. In the first place, the kinds of material

demanded above are really extremely hard to exposit, and can easily run into agency clear- ance problems. (Perhaps this explains why the

mathematicians—whose contributions are particularly easy to write up and to present abstractly without reference to the controversial

issues of a particular study—are now nearly

monopolizing the O.R. journals, thus reinforcing the widespread impression that operations

research is basically a collection of esoteric

mathematical magics. As an operations re- search mathematician, I certainly appreciate

having my colleague's mathematical contributions to O.R. readily available to me in print,

but I long for a professional literature which

would provide me with more useful insights on

the all too many facets of operations research

in which my technical training doesn't leave me

fairly capable of fending for myself.)

Second, project budgets almost invariably

underestimate, quite substantially, the drain on

time and dollars and manpower needed to do

such recording right. This in part reflects the

fact that much of the quality described above is not strictly necessary for the short-term inter- ests of the agency funding the study. It also reflects the lack of a tradition, like that in the

4. Communication, Or

In this and the following section, I want to mention a few potentially important areas for

operations research research. The list is of course quite an individual matter, reflecting my

own reactions to those difficulties encountered

during the particular O.R. studies in which

I've happened to participate. Some of you

would likely have selected a totally disjoint list for special mention, but there would probably

be substantial agreement that the subjects

sciences, of documentation of a sort directed at the professional community. This lack is not

surprising, since so much operations research

is carried out in areas where considerations of

military security or industrial "proprietary"

reticence are appropriate. But these constraints

should not apply to the civilian government

O.R. sector! For the agency sponsoring an O.R. study

which has been documented in the usual slovenly style, the consequence is that if the manager

for whom the study was done leaves—and he

will—or if the study team breaks up—and it will—then the future potential return from the

resources invested in the project becomes essen- » tially nil. The benefits thus jeopardized, to the

agency and to other agencies with related problems, can be quite substantial.

For the O.R. community, the prevalence of

inadequate documentation means systematic

deprivation of precious opportunities to learn

from each other's insights and aberrations.

Some of what isn't written down does ulti- mately get communicated to a few colleagues

by word of mouth, but this is the prehistoric

mode of information transfer, scarcely one to

rely on for efficiently promoting the advance of

the state of the art in operations research.

I suspect that studies exceeding a certain

(what?) level of effort should routinely include

a project historian—even though this may rule

out a type of doctoral thesis (The Story of

Project Q: "What Fools We Mortals Be")

otherwise likely to be quite popular in the next

decade. I hope to see an increasing stream of

solid case histories published in our journals; j: the forthcoming "applications" issue of the

O.R. Journal should either set the necessary \ precedent, or provide keener insight into the

difficulties involved, or do some of both. But

most strongly of all, I want to urge that in

allocating resources and energies among the i various phases of our work, we remember that j| in the documentation phase we are not writing ; exclusively for this decision-maker and this

year, but also for our present and future colj leagues, and have a heavy obligation to con- I tribute what we can to our profession.

ganization, and Strategy

sketched below do rank high in significance and

promise for O.R.R.6 The first topic is that of communication

with the manager or decision-maker for whom ' the O.R. study is carried out. It's often said that

the success of an O.R. project should be judged

c In restrospect, the questions relating to the proper selection of measures of effectiveness or benefits (i.e., 'objective functions"), and to the related foundational field of utility theory, seem so central as to make their absence from this Section and the next somewhat embarrassing.

16
by the extent to which its recommendations are implemented. I believe that's a gross exaggeration—I have to believe it, since otherwise I'd have become so discouraged as to leave the field long ago. What is essential is that the framework, the model underlying the study, has been

communicated to the manager clearly enough,

and is sufficiently convincing and illuminating

to him, as to substantially help him in reaching

a decision. If for example his ultimate selection

over-rides whatever recommendations the O.R.

report contains, then at least he will have a good idea of what has been sacrificed, and will

have been compelled to think through (and

will therefore be prepared to explain to others)

what additional considerations compel this

sacrifice. But how often do we achieve this degree of

communication? What can we do during the

study to increase its likelihood? At what

points, at which levels, and how frequently

should the manager and his staff be brought

into the study? A number of articles have

offered sage (and sometimes snide) counsel on

such topics, but I think the time is ripe for

carefully designed experiments, and for in- formed empirical observations, that could shed

substantial light on these important questions.

Besides communication with the sponsor,

there is also the matter of communications

within the O.R. team itself. How much in- formation should junior staff be given—with

the risk on the one hand of distracting them

from their assignments, and on the other hand

of missing their potentially valuable suggestions and failing to give them a full sense of

participation? What degree of consensus

should be required for significant strategy de5. Mathematical

The topic of this section is the one closest to my own heart. I am a bit reluctant to display

my enthusiasm, however, for fear of reinforcing a widely held image of the mathematician

in operations research as infatuated with his

favorite abstruse technique to the point of unconcern with the actual needs of the problem at hand. This image is in some cases all too well deserved, but I suspect that in many

instances it arises from a somewhat oversimplified view of quantitative models

:

To be at all manageable, a model must omit

the overwhelming preponderance of the

bloomin' buzzin' confusion that is reality. Since

complete realism is out of the question, it fol- lows that there is no one "right" model for a given situation, just more or less suitable

models. A certain fidelity to the main features

of the actual situation is obviously necessary if

cisions in the study—e.g., does one want to re- ceive unanimous consent by task leaders, or perhaps adopt a policy of "O.K. if no-one objects in writing?" What's to be done about the

fact that the participants in a 5-person meeting will go off with at least 5 different impressions of what's been decided?

I have found these questions to be sadly significant, to the point of wondering whether

each 10 or so staff members don't normally

need another man to provide mere coordination,

as distinguished from technical direction. Here

again the adage "know thyself" seems applicable to O.R.—simulation and experimental

studies on these questions of O.R. project or- ganization seem well worth pursuing.

There are also some more basic questions of

project strategy. It takes mighty steady nerves

to "keep you cool," to sit tight while systematically working out a program of activities, to do

preliminary explorations of several alternative

approaches, to wait for clarifying data or "customer" resolution of ambiguities—all this

while deadline time ticks closer and closer. There is the compulsion to do something—to get cracking—to get something going which

can be reported as progress to the customer,

and most of all to oneself. Yet my own experience, at least, indicates that a deliberate planning phase is essential for substantial projects,

that 15-20% of the allotted study period is probably even scanty for this phase, and that

shortchanging it leads to premature commitments which require agonizing and basically

unsatisfactory patching up toward the end of

the job. Do we know enough, in operations re- search, to be able to study this sort of decision

process? I would certainly like to think so. Are

they worth studying? I have no doubt they are.

Methods and Models

relevant answers are to be obtained. But to yield any answers at all, the model must be

tractable to whatever "solution" or "manipulation" operations are required. And so a "best

fit" to the desired degree of realism must be

chosen from among those models for which

solution procedures do exist, not because use

of these procedures is an end in itself, but

simply because there's nothing else to be done.

The role of mathematical O.R.R. is to enlarge

this area of choice, of "solvable" mathematical

models, so as to permit a better fit. In cases where no decent fit was previously available

and one is now provided, we find sudden recognition that instances of model M are really numerous and important—how could they have

been overlooked for so long? Solutions make

problems as well as vice versa, i.e., it is the

existence of an efficient solution method for a

17
certain model which creates the incentive to investigate whether a situation under study

can't be subsumed under that model without

unacceptable distortion.

There are two areas of mathematical operations research in which the last few years have

seen especially notable progress,7 with every

prospect of even greater successes in the years

to come. One is the field of combinatorial

optimization, largely but not exclusively con- cerned with network problems (routing, scheduling, partitioning, etc.). Here the "exponential

explosion"—the fact that the number of alternatives to be chosen among increases exponentially (or worse!) with problem size—is being

coped with successfully in more and more

cases. The second field is that of nonlinear programming, i.e., the maximization or minimization of a (not necessarily linear) function of

many variables, perhaps subject to a number of

(not necessarily linear) side conditions.

Mathematical O.R.R. ought to increase its efforts in these fields, so as to exploit promptly

the present and anticipated breakthroughs.

Furthermore the results have to be turned into

reliable tools, readily available for use during

O.R. studies which require them. This calls for

efficient well-documented computer codes, as

well as systematic computational experiments

to compare alternative methods and learn which

among them are best suited to particular types

of problems.

Of course many of the problems we will want

to solve will still turn out to be mathematically

intractable with the tools available. I would

hope to see more emphasis in O.R.R. on developing and disseminating rigorous sophisticated

procedures for approximating intractable models by simpler ones, in particular for "surrounding" the original situation with a number of

tractable ones, each deliberately biased in a known way.

It is natural to wonder whether civilian government O.R. will have some distinctive demands to make on mathematics. One speculative notion is that there will be an increasing

need to determine appropriate sorts of equilibria rather than optima, that it may be less fruitful to seek a solution which is "best" in some sense than to aim at striking a proper bal- ance between conflicting public interests.

I must confess to a general uneasiness about

the basic appropriateness, for much of operations research, of traditional mathematics.

Mathematics originated in connection with human activities—surveying, accounting, navigation—characterized by rather precisely quantifiable "data," and a requirement for rather

precise numerical results. The further development of mathematics occurred in connection

with the sciences and engineering, again fields where "hard numbers" prevail. It seems unlikely that techniques developed in such contexts

would be generically suitable for situations involving "data" scarcely deserving of the name,

and requiring not the precise evaluation of

various courses of action but rather only their

ranking according to quite fuzzy criteria, or even just the determination of the top one or two or three among them. There may well be a

need for some new kinds of mathematical concepts and operations, or at least a greater application in O.R. of the notions of "modern mathematics" with its stress on structures and rela- tionships rather than numbers.

6. Summary

Operations research has been described as an

art_ which "gives bad answers to problems

which otherwise would receive even worse an- swers." This art is being and is going to be

applied, by the non-military sector of our government, to problems so important, so difficult and so numerous as to require every effort to minimize the "badness" of our answers. We can make some progress toward this goal by raising the quality of our documentation

from its present abysmally low level, so that

we can make better use of each other's work

and thus free resources for resolving those diffi- culties unique to the problem at hand. We can

delve more deeply into the reliability of the

7 ^ wo.u]d be '"appropriate, in a non-technical paper such as this, to cite the specific publications involved.

"data" entering our work, and adjust our methods and conclusions accordingly. But even with all this done, we would be faced with substantial gaps between our needs and the O.R.

methods available to us. Our work will frequently call for knowledge and methods which

don't yet exist, or which exist in theory but not

in practically usable form—perhaps for material there was no real incentive to develop in

the climate of military or industrial O.R. A

substantial in-house capability to advance the

art of operations research for civilian terrestrial branches of government is needed, but

practically none is not present. In the present

paper I have tried to indicate the reasoning behind these assertions, and to identify some of

the areas on which a proper O.R.R. effort

should focus.

18
Reports of Panel Meetings

Editor's Note: As noted in the agenda, three panel sessions were held in the afternoon: Panel I, Special

Problems of OR in Several Agencies of Government; Panel II, Management Views in OR/SA to Fulfill PPB Requirements; Panel III, Broadening the OR Competence of Mid-Careerists. The panel chairmen were asked to prepare a set of notes on the activities of their panels. Notes on Panel I and Panel III were

received and are published herein.

Panel I

"Special Problems of Operations Research in Civil Agencies

of Government"

Chairman: Mr. Ezra Glaser

Health, Education, and Welfare

This panel was to review communication

problems among technical operations researchers and to attempt to specify those factors

which limit the quality and extent of operations

research practice in the civil agencies.

The conference proceeded by asking those in

attendance to express agreement with a number of propositions, each of which postulated an important limiting factor for the quality and

extent of operations research practice. Four

classes of problems and/or limiting factors were identified as "severe," for the quality or

extent of operations research practice:

1. Difficulties in dealing with the upper

echelon of the organization. The prevalence of preconceived problems and preconceived answers were symptomatic of these

difficulties. Lack of understanding by the

upper echelon of what operations research

is and how it might usefully be employed

in solving the problems of the organization.

2. The training and quality of operations

analysts.

3. The difficulty of developing measures of

effectiveness in civilian agencies. While

there has been some success in constructing measures of effectiveness in business

organizations and military agencies, there

is very often great difficulty in finding means of measuring the effectiveness of

civilian programs. The problem is how to be quantitative without by-passing the

principal reasons for the existence of the

program.

4. Communications among operations re- searchers. This applies both to the techniques employed and to results obtained.

There was little support for the proposition

that inadequacy of financial resources was a

serious limiting condition. The group indicated

a belief that sufficient funds were generally

forthcoming in situations where there was

some satisfactory way of handling the problems

noted above.

Relatively little weight was given to the inadequacies in the technical literature, which

might impede teaching or the upgrading of

quality of operations researchers.

Each of the four principal obstacles will be

discussed in turn.

1. Relation to Upper Echelon of the Organization

One principal and frequent problem is the

indirectness of the O.R. staff's organizational

relationship to the head of the agency. This means that the staff is often not in a position to watch the development and identification of

major problems from the viewpoint of the agency head, and therefore not in a position to prepare in advance for studies aimed at assisting in reaching important decisions. Experience shows that such operations research studies require substantial lead time for highquality results, but in practice this time is generally unavailable. The likely result is a

hasty and ill-considered effort which can take

but little advantage of the potentialities and

resources of O.R. techniques.

2. The Training and Quality of O.R.

Personnel

It was pointed out that technical courses in

O.R. that might prepare analysts for government applications leave much to be desired.

It was suggested that an interdepartmental

and consortium approach be utilized to offer

enough kinds of courses, for different clienteles

and describing different techniques and focussing on different problem applications to fill out

the field in some systematic manner. While no

single institution could collect enough students

or assign enough specialized faculty for such a program, it seems likely that a group of educational institutions could arrange just such a

spectrum of courses. Another problem area is the recruitment of

college personnel for operations research work.

Generally the Government is not competitive

with private enterprise. Competitiveness re- quires sending out technical recruiters who

are able to discuss the technical opportunities

in language meaningful to recent graduates of

19
technical courses. Moreover, recruiters for private enterprise can make a commitment on the

spot whereas the Federal recruiter can onlypoint to the long and tortuous path of examinations, ratings, etc., which might eventually lead

to some type of promotion. While these constraints are consistent with Civil Service philosophy and practice, they should be recognized

as placing the Federal establishment at a

serious disadvantage in competing with private

organizations for superior students.

The group recommended more emphasis on

sending rusty students back to school, renewing

their technical abilities and extending these into

operations research studies. There is also need

for a series of shorter courses, especially those

directed at technique in the modelling process,

with use of "live" exhibits of real problems

used to the maximum extent possible.

3. Measures of Effectiveness

This is one of the most difficult problems in

the useful application of operations research to the programs of civilian agencies. One cannot measure the benefits from various program

elements or administrative alternatives by their

effects on "profits." Nor are there well defined

military type objectives. The difficulty is serious

enough as regards provision of current services

(health, occupational retraining, vocational re- habilitation, etc.) but is an order of magnitude more severe in dealing with those programs

whose benefits lie in the improvement of future

services (improvements in elementary school

education, biomedical research)

.

There is almost a total absence of training

materials suitable for most of the civilian agencies of the Government, which communicates

the few hard-won currently known approaches

to the problems of constructing measures of

effectiveness. In practice this problem tends to be linked

with the relation of the operations research

group to the top man in the organization—in

the absence of opportunity for this group to formulate and inject any more objective measure of effectiveness, the evaluation necessarily

turns out to be an expression of his personal

preferences.

4. Communication Among Operations Researchers

The key notion here seems to be the existence

of "natural aggregates" in Government programs. For example, the interests of the Fed- ! eral Government in transportation are scat- tered over many agencies, many programs, and

many objectives. It would seem desirable for

operations researchers who are working on the

many aspects of transportation program to be

in communication with others working within

the same aggregate.

A yet broader combination can be obtained

by grouping the interests of transportation and

communication, since these two are interchangeable or substitutable in some situations.

The Planning-Programming-Budgeting System will probably provide an organizational en- try into these natural groupings by the requirements of PPB that common objectives be identified and put into some rational programming

framework. „ For operations researchers whose activities

fall into some natural aggregate (not quite as

strictly interpreted as Derek Price's Invisible

Colleges), it would seem desirable to program

a certain number of technical colloquia and I seminars. These are not envisaged as meetings j to compile lists of proposed projects or on- going research, but rather as solid discussions j in considerable depth of technical problems, at- tempts to use various techniques, obstacles met, I tentative results achieved, and so forth.

One question that arises in the reconstruction

of budget decision making categories is the way j in which the Bureau of the Budget itself would j adapt to such natural aggregates as are not i now expressed in its own organization, and

what part they might play in any colloquia

which are organized in behalf of some substantial common interests among the Federal agencies and programs.

Panel II

The Summary of Proceedings for Panel II is not available.

20
Panel III

Broadening the O.R. Competence of Mid-Careerists

Chairman: Mr. Joseph Lowell, Jr. Civil Service Commission

Panel III was attended by 13 participants,

almost all of whom are actively engaged in

Operations Research.

Perhaps the most significant result of Panel

Ill's discussion was the refinement of the panel

topic. Participants agreed that the topic was too

broad, including at least two sub-problems

worthy of quite separate treatment.

The first problem involves the identification,

recruiting and training of those persons presently in non-O.R. positions who have the potential to be made over into O.R. practitioners.

The second problem involves the assistance

and/or training which non-technical people

need in order to understand enough about O.R.

to know when and where to turn to O.R. practitioners for help.

(A third problem, raised but not extensively

discussed, involved the education of O.R. practitioners in the art of communicating their

profession and its techniques in lay terms.)

It was generally agreed that each of the

above mentioned problems merited serious and

separate discussion. Recommendations were made that they be treated individually at any

follow-up sessions. There was general agreement that participants in such discussions

should be pre-registered and pre-selected to avoid the situation of one group discussing the

problems of another, unrepresented, group;

e.g., O.R. people deciding independently what

non-O.R. people need to know, or vice versa.

21

List of Attendees

Name

If ALEXANDER, Mark A. ASTIN, Allen V. AZE, Albert Y. BERSHAD, Max A. K BLACK, Guy

BOULAND, Heber D. BREZINA, Dennis W. I CANNON, Edward W. I CASE. C. Marston , COFFEY, Matthew B. I CONNOR, Lawrence W. I CONWAY, Paul T. ! CRUMLISH, Joseph D. I CURTIS, Donald W. I DAVIS, Charles J. | DE RAAT, Jacob H. DONAHUE, Gilbert E. DORFMAN, William I DOUTY, Harry M.

FAZAR, Willard : FLORY, John A. I FLYNN, John C. I GILES, J. K. GLASER, Ezra I GLEASON, Jacqueline I GLEASON, Maynard G. I GOLDMAN' Alan J. | GOLDMAN, Morris R. I GOLUBIN, Nicholas M. I GORDON, Virginia B. E GREEN, John C. ? GREENHOUSE, Samuel

M § GREENWOOD, Joseph A.

[' GROSSE, Robert N. I HAHN, Walter A. 1 HALDI, John I HALL, Glenn 0. ' HANNA, C. R. I HARTEL, Robert J. I HATRY, Harry \ HEARN, Saul D. i HENIG, Seymour I HEYL, Arnold A. I HIPP, Fred C. I HOBBAH, Reginald V. HOOPER, William L. HOPKINS, Cleveland

HOWE, E. Eugene

JENKINSON, Ernest H. KAITZ, Hyman

KIT, Boris

KLINE, Robert C, Jr. LANDIS, Rex

LANKA, George

LAVELL, Robert J. LEARY, Thomas J. I LINER, C. D. LOMBARD, Edward E. LOWELL, Joseph W., Jr. McMANUS, Frank i MAHONEY, Edward J. & MEGLES, Richard C.

ft MENDENHALL, 0. H. MICHELSON, Aaron I MILILLO, Frank A. | MILLER, Robert F. I MILLIE, Harold R. i MORROW, James C. I MOSS, John | NASSETTA, F. C. ] O'BRIEN, James L. I PAIGE, Harvey C.

|! PANZER, Fred

:H PARKER, Walter W.

Agency

Atomic Energy Commission

National Bureau of Standards

Internal Revenue Service

Bureau of Census

Council of Economic Advisors

National Bureau of Standards

Science Policy Research Division National Bureau of Standards

Office of Education

The White House Office Post Office Department

Office of the Chief of Naval Operations

National Bureau of Standards

Treasury Department

Department of the Army

National Bureau of Standards

Department of Labor

Environmental Science Services Administration

Department of Labor

Bureau of the Budget

Economic Development Administration

Economic Development Administration

Post Office Department

National Institutes of Health

Office of Education

Maritime Administration

National Bureau of Standards

Office of Business Economics

Defense Supply Agency

Economic Development Administration

Office of Emergency Planning

Veterans Administration

Food and Drug Administration

Health, Education and Welfare

Environmental Science Services Administration

Bureau of the Budget

Economic Development Administration

Department of Agriculture

Interstate Commerce Commission

OSD

—Systems Analysis

Social Security Administration

National Bureau of Standards

Health, Education and Welfare

National Bureau of Standards

National Bureau of Standards

Health, Education and Welfare

National Bureau of Standards

HQ USA CDC

Department of Agriculture

Business and Defense Services Administration

National Bureau of Standards

Interstate Commerce Commission

Post Office Department

Federal Aviation Agency

Department of the Interior Internal Revenue Service

Office of the Secretary of Defense

Small Business Administration

Civil Service Commission

National Bureau of Standards

General Accounting Office Department of Labor

Federal Aviation Agency

Veterans Administration

Naval Command Systems Support ACT

Post Office Department

National Bureau of Standards

U.S. Coast Guard

Social Security Administration

Naval Command Systems Support ACT

Census Bureau

National Academy of Sciences

The White House

Patent Office
PICCIRELLI, Robert

PICKERING, Harlan

PINCUS, Saul

POCINKI, Leon

RAUCHSCHWALBE, Otto REIFMAN, Lucille RIEKERT, F. M.

ROOT, James G. ROSENBLATT, David

ROTHMAN, Abe

SAMORDIC, Michael

SAUNDERS, Edward R., Jr. SCHEIG, Sue

SCHWALB, Louis

SIMPSON, J. R. SINGER, Burton 0. SIROTA, Milton

SLAUGHTER, Lawrence A. SOLOMON, David

SPENCER, Richard A. STERNBERGER, Jesse L., Jr. STEVENS, John T. SUSZYNSKI, N. SUTHERLAND, Henry C. SUZUKI, George

THAYER, T. C. TREIRES, James J. TRIPLETT, Robert S. TYNER, Richard

URBAN, Bernard

VAN TRAIN, W. A. VERNIER, Claire M.

VINCENT, Alvin R. WAGNER, Lloyd

WALTERS, Stanley S. WEIL, Lisa

WEINGARTEN, Harry

WIGLER, Ken

WILKES, John D. WILLIAMS, Jacob A. WILLOUGHBY, John E. WILSON, Dale W.

YUROW, Jerome A. ZABROWSKI, Edward J. ZIMMERER, Robert W.

ZWINSCHER, Walter H.

National Bureau of Standards

Environmental Science Services Administration

Department of Defense

Environmental Science Services Administration

National Park Service

National Institutes of Health

Small Business Administration

National Bureau of Standards

National Bureau of Standards

Department of Labor

Office of Economic Analysis

Executive Office of the President

Civil Service Commission

Department of the Army

Department of the Navy

Civil Aeronautics Board

Federal Aviation Agency

Department of State Bureau of Public Roads

Patent Office Federal Aviation Agency

Navy Department

Smithsonian Institution Patent Office National Bureau of Standards

Office of Assistant Secretary of Defense

Economic Development Administration

National Civil Defense

State Department

Bureau of the Budget

Office of Naval Warfare Analyses Veterans Administration

Atomic Energy Commission

Veterans Administration

U.S. Arms Control and Disarmament Agency

National Bureau of Standards

Bureau of Public Roads

Department of Agriculture

Agency for International Development

Social Security Administration Veterans Administration

DSAH-LSO Cameron Station

National Bureau of Standards

Office of Education

National Bureau of Standards

Federal Aviation Agency

Acknowledgment

The assistance of many people contributed

to the success of the Conference. Among those

who helped were W. Hooper and V. McRae

(Executive Office of the President) ; W. Fazar

(Bureau of the Budget) ; A. J. Goldman and D.

Rosenblatt (National Bureau of Standards) ; J. Moss (Health, Education and Welfare) ; Speakers and Panel Chairmen.

We acknowledge with great gratitude the

secretarial services of Mrs. Alice Kelly in preparing for the conference, and the assistance

of Mrs. Margery King in the operation of the

registry and information desk while the con- ference was underway.

it U. S. GOVERNMENT PRINTING OFFICE: 1967 —301-022/67

24
NBS TECHNICAL PUBLICATIONS

PERIODICALS

JOURNAL OF RESEARCH reports National Bureau of Standards research and development in physics, mathematics, chemistry, and

engineering. Comprehensive scientific papers give

complete details of the work, including laboratory

data, experimental procedures, and theoretical and

mathematical analyses. Illustrated with photographs,

drawings, and charts.

Published in three sections, available separately: • Physics and Chemistry

Papers of interest primarily to scientists working in these fields. This section covers a broad range of

.physical and chemical research, with major emphasis on standards of physical measurement, fundamental

constants, and properties of matter. Issued six times

a year. Annual subscription: Domestic, 35.00;

foreign, 36.00*. • Mathematics and Mathematical Physics

Studies and compilations designed mainly for the

mathematician and theoretical physicist. Topics in mathematical statistics, theory of experiment design,

numerical analysis, theoretical physics and chemistry, logical design and programming of computers

and computer systems. Short numerical tables. Issued quarterly. Annual subscription: Domestic,

32.25; foreign, 32.75*.

• Engineering and Instrumentation

Reporting results of interest chiefly to the engineer

and the applied scientist. This section includes many

of the new developments in instrumentation resulting

from the Bureau's work in physical measurement,

data processing, and development of test methods.

It will also cover some of the work in acoustics,

applied mechanics, building research, and cryogenic

engineering. Issued quarterly. Annual subscription:

Domestic, 32.75; foreign, 33.50*. TECHNICAL NEWS BULLETIN

The best single source of information concerning

the Bureau's research, developmental, cooperative

and publication activities, this monthly publication

is designed for the industry-oriented individual

whose daily work involves intimate contact with

science and technology

for engineers, chemists,

physicists, research managers, product-development managers, and company executives. Annual subscription: Domestic, 31-50; foreign, 32.25*.

•Difference in price is due to extra cost of foreign mailing

N0NPERI0DICALS

Applied Mathematics Series. Mathematical

tables, manuals, and studies. Building Science Series. Research results, test methods, and performance criteria of building

materials, components, systems, and structures. Handbooks. Recommended codes of engineering

and industrial practice (including safety codes)

developed in cooperation with interested industries,

professional organizations, and regulatory bodies.

Miscellaneous Publications. Charts, administrative pamphlets, Annual reports of the Bureau,

conference reports, bibliographies, etc. Monographs. Major contributions to the technical literature on various subjects related to the Bureau's scientific and technical activities. National Standard Reference Data Series.

NSRDS provides quantitative data on the physical

and chemical properties of materials, compiled from

the world's literature and critically evaluated.

Product Standards. Provide requirements for

sizes, types, quality and methods for testing various

industrial products. These standards are developed

cooperatively with interested Government and in- dustry groups and provide the basis for common

understanding of product characteristics for both

buyers and sellers. Their use is voluntary.

Technical Notes. This series consists of communications and reports (covering both other agency

and NBS-sponsored work) of limited or transitory

interest. CLEARINGHOUSE

The Clearinghouse for Federal Scientific and

Technical Information, operated by NBS, supplies

unclassified information related to Governmentgenerated science and technology in defense, space, atomic energy, and other national programs. For

further information on Clearinghouse services, write:

Clearinghouse

U.S. Department of Commerce

Springfield, Virginia 22151

Order NBS publications from:

Superintendent of Documents

Government Printing Office Washington, D.C. 20402
U.S. DEPARTMENT OF COMMERCE

WASHINGTON, D.C. 20230

OFFICIAL BUSINESS

Postage and fees paid

u.s. department of commerce

1

I