procurement of Evaluation

Systems: A Case Study of the

Parametric Factor Evaluation

Approach to Source Selection

AlllOS 1S7M7Q

Ardwm 3. Librran

July 1980

100

U56

80-2092

Experimental Technology Incentives Program

National Bureau of Standards

Washington, DC 20234
CENTER FOR FIELD METHODS

THE EXPERIMENTAL TECHNOLOGY INCENTIVES PROGRAM

The Experimental Technology Incentives Program (ETIP) of the National Bureau of

Standards pursues an understanding of the relationships between government

policies and technology-based economic growth. The pursuit of this objective is

based on three premises:

o Technological change is a significant contributor to social and economic

development in the United States.

o Federal, State, and local government policies can influence the rate and

direction of technological change.

o Current understanding of this influence and its impact on social and

economic factors is incomplete.

ETIP seeks to improve public policy and the policy research process in order to

facilitate technological change in the private sector. The program does not

pursue technological change per se. Rather, its mission is to examine and

experiment with government policies and practices in order to identify and assist

in the removal of government-related barriers and to correct inherent market

imperfections that impede the innovation process.

ETIP assists other government agencies in the design and conduct of policy

experiments. Key agency decisionmakers are intimately involved in these experiments to ensure that the results are incorporated in the policymaking process.

ETIP provides its agency partners with both analytical assistance and funding for

the experiments while it oversees the evaluation function.

Because all government activities potentially can influence the rate and direction of technological change, ETIP works with a wide variety of agencies, including those that have regulatory, procurement, R&D, and subsidy responsibilities.

Programs are currently underway with the General Services Administration, Food

and Drug Administration, Veterans Administration, Securities and Exchange Commission, Department of Energy, Environmental Protection Agency, Occupational

Safety and Health Administration, and other Federal agencies as well as various

State and local agencies.

Director

Center for Field Methods

National Engineering Laboratory

National Bureau of Standards

U.S. Department of Commerce
NBSIR 80-2092

PROCUREMENT OF EVALUATION

SYSTEMS: A CASE STUDY OF THE

PARAMETRIC FACTOR EVALUATION

APPROACH TO SOURCE SELECTION

HAIiO.

1_. .i AUP 1 2 1980

1U5i ate."

QC^

\oO

dO'di

1^90

Ardwin S. Libman

Regulatory Programs

Experimental Technology Incentives Program

July 1 980

Prepared by

Experimental Technology Incentives Program

I

PROCUREMENT OF EVALUATION SYSTEMS:

A CASE STUDY OF THE PARAMETRIC FACTOR EVALUATION APPROACH

TO SOURCE SELECTION

By

ARDWIN S. LIBMAN

EXPERIMENTAL TECHNOLOGY INCENTIVES PROGRAM

NATIONAL BUREAU OF STANDARDS

June 1980
IIIii
Ill

ABSTRACT

This document is a report of a case study of the use

of a system contractor source 'selection process by a federal

program. It is intended to be of both research and administrative value. Its primary research value is as a bench

mark for future case studies on source selection techniques

in general and for future research on the specific source

selection process reported on here. Its primary administrative value is to those responsible for procuring systems

(and other complex and uncertain products, e.g., large evaluations) as an evaluation of the outcome of the use of the

specific technique in terms of the issues critical to effective source selection.

The specific problem for which the technique was used

was the selection of two contractors to develop an ongoing

capability to evaluate the results (agency and commercial

impacts) of experimental modifications in procurement pro-

# cedures by selected federal, state, and local government

agencies. These evaluation systems were to be implemented

by organizations outside of the developing organization

(The Experimental Technology Incentives Program of the National Bureau of Standards).
iv

The primary administrative finding is that the federal

program was able to adapt successfully the technique, which

was developed and used to procure "hardware” systems, to

meet its requirements to select contractors to develop these

"software" systems. The successes and failures in dealing

with numerous critical issues are detailed in Chapter V.

Section V.4 provides a summary of these results with crossreferences to the related discussions in the text.

The primary research contribution is the offering of

an evaluation structure and an example of its use for future

case studies of source selection techniques. This work

also provides a departure point for further research on

the specific technique evaluated here. The prior literature

is void of detailed case material. This document provides

a structure for comparative evaluation and much of the data

which was used for this one.

The method followed is to: 1) provide background in

terms of the problems of the specific procurement (section

1.2) and related procurements (section II. 2, II. 3) and background on the source selection process evaluated here (section II. 5); 2) review the related issues of source selection

raised in prior literature (section II. 4); 3) supply a detailed chronology of the use source selection process (Chapter IV); and '4) evaluate the results of using the process

with a structure built around the critical issues raised
V

by participants (drawn from the Chronology) and by the prior

literature (Chapter II).

Implications of this study for future use of the process and for future related research are presented in Chapter VI. A background document on the source selection process, documents from the RFP and the actual ratings from

the use of the process which is evaluated here are provided

in the Appendix.

The author is an employee of the federal program which

used this process. He has been involved in two subsequent

source selections using the technique. The report of this

research is the author's doctoral dissertation. His committee chairman is the developer of the technique being evaluated. A self-report of the author's biases appears in

section III. 5.
Vi

ACKNOWLEDGEMENTS

Various people have contributed to this work since

its inception. These contributions have been wide ranging

and varied. I am thankful to the many people who were in^

volved in the completion of this research. Several of these

people merit separate mention here.

I wish to acknowledge my dissertation chairman, Charles

Thompson, for stimulating my interest in this topic and

for his occasional guidance and sustained encouragement.

I thank my other committee members, Victor Berlin, Gustave

Rath, and Joseph Schofer for their various contributions

to the final form of this research. ‘

Several friends/colleagues provided reviews of parts

of this work. Thanks go to Warren Frederick, Stephen Garrity, and Roland Weiss for their comments on sections of

the text and for the discussions on the research which helped

me to remain focused on my objective.

Special thanks go to Darlene Carver who typed several

drafts of the manuscript and enabled me to meet deadlines.

The research reported on here was performed as a project of the Experimental Technology Incentives Program of

the National Bureau of Standards. I wish to acknowledge
vii

Daniel Fulmer for his support during the later stages of

this work.

Finally, I am thankful to Elise Blumenthal for both

understanding and encouragement.
vlii

TABLE OF CONTENTS

ABSTRACT iii

ACKNOWLEDGEMENTS iv

TABLE OF CONTENTS viii

LIST OF FIGURES xvi

I. INTRODUCTION 1

1.1 The General Problem of Acquiring Evaluation Systems 1

1.2 The Specific Problem Which ETIP Faced. . . 1 1.2.1 ETIP 2

1.2.2 ETIP’s System Procurement Problem 2

1.3 The Topic of this Research 4

1.4 Value of this Research 5

1.5 The Organization of this Document 5

II. BACKGROUND 8

II. 1 Introduction 8

11. 2 Evaluation Systems 8

11. 2.1 Concept of Evaluation Systems . . 8

11. 2. 2 The Level of Evaluation in

the Federal Government 10

11. 2. 3 The Evaluation Community 11

11. 2. 4 The Pains of Adolescence 12

11. 2. 5 Contract Evaluation 14

11. 2. 6 Relationship to Systems Procurement 15

11. 2. 7 Summary of Evaluation Systems

Background 16

11. 3 Systems Procurement 16

11. 3.1 The Concept of Systems 16

11.3.2 Size of System Procurement. ... 17

11. 3. 3 Research Related to Systems

Procurement 18

11.3.4 Characteristics of Systems

Procurement 20

11. 3. 5 The Role of Source Selection. . . 27

11.3.6 Issues Related to Source Selection 28
ix

II.3»6.1 Preparation and Release

of the RFP and briefing

potential bidders ... 30

11.3*6.2 Proposal preparation 35

11.3*6.3 Proposal evaluation

panel meets 37

11.3*6.4 Summary, review and

award 41

11.3*7 Summary of Systems Procurement

Background 44

11. 4 Parametric Factor Evaluation 45

11. 4.1 History 45

11. 4. 2 Claims 46

11. 4. 3 Characteristics 47

II. 4. 4 Summary 52

11. 5 Summary 52

III. OBJECTIVES OF THE RESEARCH 54

111.1 Introduction 54

111. 2 Purpose of this Research 54

111. 3 Format Followed in the Case Study 56

111. 4 Value of this Research 59

III. 4.1 Research Value 60

III. 4. 2 Administrative Value 62

111. 5 Biases of the Author 65

111. 6 Summary 67

IV. CHRONOLOGY 68

IV. 1 Introduction .... I 68

IV. 2 Decision to Contract for Evaluation

Systems 70

f IV. 3 Preparing the Request for Proposals. ... 75

IV. 4 Involving the Contracting Office 86

IV. 5 Involving NBS Management 88
IV. 6 Involving the Program Administrative

Agencies 89

IV. 7 Involving the Prospective Bidders 92

IV. 8 Selecting and Involving the Evaluation

Panel 99

IV. 9 The Proposals Arrive 104

IV. 10 The Factor Teams Meet 107

IV. 10.1 Introduction 107

IV. 10.2 The First Factor Team Meeting . . Ill

IV. 10.3 The Second Factor Team Meeting. . 121

IV. 10.4 The Third Factor Team Meeting . . 127

IV. 10.5 The Fourth Factor Team Meeting. . 128

IV. 10.6 The Fifth Factor Team Meeting . . 129

IV. 10.7 The Sixth Factor Team Meeting . . 130

IV. 10.8 The Seventh Factor Team Meeting . 131

IV. 10.9 The Eighth Factor Team Meeting. . 132

IV. 10. 10 The Ninth Factor Team Meeting . . 133

IV. 10. 11 The Tenth and -Eleventh Factor

Team Meetings 134

IV. 10. 12 The Twelfth Factor Team Meeting . 134

IV. 10. 13 The Thirteenth and Fourteenth

Factor Team Meetings 135

IV. 11 Summarizing the Results for the Contracting Office 135

IV. 12 Preparing for Orals 140

IV. 13 Orals and Written Confirmations 141

IV. 14 Summary of Results and Award 145

IV. 15 Debriefing 147

IV. 16 Summary and Transition to Chapter V. . . . 148

V. EVALUATION/CRITIQUE 150

V. 1 Introduction 150

V.2 The Issues, Additional Data, and Parameters .... 151

V. 2.1 Listing of Issues 151

V.2. 2 Categories of Issues 152

V.2. 3 Selection of Specific Issues

for Discussion 162
V. 2 .U Additional Data i 62

V. 2.5 General Parameters to the Evaluation 165

V . 2 . 5 .

Memory 166

V.2.5 .

Subsequent experience

with the systems development 167

V.2.5 .

Subsequent experience

with the PFE process. . 167

V.2.5.^ Other related source

selection experience. . 170

V.2.5 .

Outcome of the source

selection 171

V.2.5. 6 Ego issues related

to arguments about

what to do 171

V. 2 . 5.

Lying 173

Evaluation of the Parametric Factor

Evaluation Approach Against the Specific

Critical Issues 176 V.3.1 Introduction ’ 176 V.3.2 Effective Communication 177

V. 3 . 2.1 Effectiveness of

the structure 179

V.3.2. 2 Size of proposals . . . 191

V.3.2 .

Understanding basis

for evaluation 194

V.3.2.U Differences in uncertainty among bidders. . 199

V. 3.2.5 Communication of

nature of project

and required skills . . 20 ^ V. 3.2.6 Appropriateness of

specific sub-factors. . 208

V.3.2 .

Parroting 213

V.3.3 Sensitivity and Selectivity

in Obtaining Bids 213

V.3.3. 1 Sensitivity 214

V.3.3.2 Selectivity 217

V.3.4 Assessment of Bidders' Capability 218

V.3.4 .

Evaluation of management capability .... 220

V.3.4 .

Evaluation of technical capability 222

V.3.4 .

Anticipation of problems known to exist

in the project 223
V.3.5

V.3.6

Use of Experts in the Evaluation of Proposals 225

V. 3 . 5 . 1 Obtaining participation by experts .... 225

V. 3 . 5 . 2 Use of expert information 227

The Panel 's Role in Evaluation

of the Proposals 228

V.3.6.

1

Meeting attendance

and ability to determine initial ratings. . 229

V.3.6.2 Reaching agreement. . . 230

v.3.6.

3

Motivation of panel . . 236

v.3.6.

4

Dealing with extreme

scores or greater

variability 239

V. 3 . 6 .

5

Time required to

agree on sub-factor

weights 241

V. 3 . 6 .

6

Input by junior panel

members 246

V. 3.7

V.3.8

V. 3.9

Equity of

Bidders . Treatment Across

246

V. 3 . 7 .

1

Treatment of bias . . . 247

V. 3 . 7 .

2

Consistency of evaluation across bidders. . 278

v.3.7.

3

Admissible information 280

Risk Associated with the Statement of Work 283

V. 3 . 8 .

1

Flexible statement

of work 284

V. 3 . 8 .

2

Effect of statement

of work on bid decisions 285

V. 3 . 8 .

3

Ability to assess

risk 286

Basis for Award 287

V. 3 . 9 .

1

Overall technical

discrimination 288

V. 3 . 9 .

2

Technical discrimination on specific

issues 288

V. 3 . 9 .

3

Technical leveling

and technical transfusion 291
xlll

V. 3 . 9 .^ Weight given to bid

price 292

V. 3.10 Traceability of the Process . . . 293

V. 3 . 10.1 Bidders' understanding of the evaluation

process 29^

V. 3 . 10.2 Ability to trace

scores through the

process 297

y. 3*10.3 Degree to which the

rules were followed . . 299

V.3*10.^^ Weighting of the

sub-factors 302

V.3.11 Relationship with the Contracting Office 303

V. 3 . 11.1 PFE and the Federal

procurement Regulations 304

V. 3 . 11 . 2 Ability to defend

under investigation . . 305

V. 3 . 11.3 Revisions of panel

results 307

V. 3 . 11.4 Timeliness of award . . 308

V. 3.12 Resource Requirements 309

V. 3 . 12.1 Resources required

prior to issuing

the RFP 310

V. 3 . 12.2 Resources required

for bidders to respond. 210

V. 3 . 12.3 Resources required

for panel to evaluate

proposals 312

V. 3 . 12.4 Resources required

following the panel's

evaluation 313

V. 3.13 Education of Government Personnel 314

V. 3 . 13.1 Education provided

for evaluating proposals 315

V. 3 . 13.2 Education provided

for working with

winning .bidders .... 316

V. 3.14 Education of Bidders 317

V. 3.15 Additional Comments from the

Questionnaires 318
xlv

V. 4 Summary 319

V.4.1 Summary of Issues for Which

ETIP's Use of PFE Proved Successful 322

V.4. 2 Summary of Issues for Which

ETIP’s Use of PFE Proved Unsuccessful 325

V.4.^ Summary of Issues for Which

the Results of ETIP’s Use of

PFE are Equivocal 326

VI. BROADER IMPLICATIONS 328

VI. 1 Introduction 328

VI. 2 The Case Study Method for Exploratory

Research 328

VI.

3

Comparison of ETIP’s Results to Other Uses

of the PFE Process 331

VI. 3.

1

Other Experiences 331

VI. 3 .

2

Lessons Learned by Comparison to

the Wright Field Experiences. . . 332

VI. 3 . 2.1 Departures

VI. 3 . 2. 2 Improtance of early

332

involvement 332

VI. 3 . 2. 3 Selection of experts. . VI. 3 . 2. 4 Benefit of orals prior

333

to factor team meetings

VI. 3*2. 5 Problems from adding

an illustrative state334

ment of work

VI. 3 . 2. 6 Transferability of the

process to a ’’software”

335

problem 335

VI. 3. 3 Modifications Made in Later Uses

of PFE by ETIP 336

VI. 4 Adaptability of the PFE Process to Other

Types of Problems 337

VI. 5 Possible Additional Research 338

VI. 6 Summary 339

REFERENCE NOTES 340
BIBLIOGRAPHY 3^7

APPENDIX A Documents from the RFP 355

APPENDIX B Description of the PFE Process 443

APPENDIX C Instructions Given to Evaluation Panel

Members 464

APPENDIX D Ratings from the Agency Impact Contract

Evaluation 470

APPENDIX E Ratings from the Commercial Impact Contract Evaluation 511

APPENDIX F Cover letter and Questionnaire for Panel

Members 552

APPENDIX G Cover letter and Questionnaire for Bidder

Representatives 559

APPENDIX H Enlargement of Figure IV. 1 567
LIST OF FIGURES

Figure

Number Title Page

11. 1 Two Phases of the Procurement of Systems. . 20

11. 2 Four Phases of the Source Selection

Process 29

IV. 1 The Activities in ETIP’s Source Selection . 71

IV. 2 Initial Concept of the Structure of the

Evaluation Contracts 84

IV. 3 The Commerce Business Daily Announcement. . 94

IV. 4 The Question Form for Use in Obtaining

Clarifications 108

IV. 5 Schedule of Factor Team Meetings 109

IV. 6 A Completed Sub-Factor Rating Form 112

IV. 7 The Form of Posting Scores 113

IV. 8 Assignment of the First Score on Sub-Factor

VI . 1 116

IV. 9 Assignment of All Scores for Sub-Factor

VI. 1 117

IV. 10 Posting of Scores to Begin Rating on SubFactor VI .2 118

IV. 11 Assignment of Factor Scores for Factor VI . 122

IV. 12 A Completed Factor Rating Form 123

IV. 13 Time Required to Rate Sub-Factors

(estimated from tapes of meetings) 126

IV. 14 Summary Display of Scores for the Contract

Negotiator 139

V.1 Issues and Issue Categories 153

V.2 General Parameters to the Evaluation of the

Outcome of ETIP's Use of PFE 166

V.3 Panel Members* Self-Reports on Parameter of

Recall. 168

V.4 Bidder Representatives* Self-Reports on

Parameter of Recall 169

V.5 Panel Members* Source Selection Experience. 172

V.6 Bidder Representatives* Source Selection

Experience 173

V.7 Panel Members* Responses to Question la on

the Effectiveness of the Structure I8l

V.8 * Bidder Representatives* Responses to Question la on the Effectiveness of the

Structure 182

V.9 Panel Members* Responses to Question 1b on

the Effectiveness of the Structure 183

V.10 Bidder Representatives* Responses to Question 1b on the Effectiveness of the

Structure 184
V.11 Bidder Representatives' Responses to Question 1c on the Effectiveness of the

Structure 184

V.12 Panel Members' Responses to Question 1c on

the Effectiveness of the Structure l85

V .13 Responses to Open Ended Questions Which

Relate to Structure 185

V.IU Sizes of the Proposals ' 192

V .15 Relationship Between Length of Proposal and

Total Panel Score 195

V.I 6 Bidder Representatives' Responses to Question 2a on Confidence in Being Able to

Communicate Qualifications to Panel .... 196

V .17 Bidder Representatives' Responses to Question 2b on Confidence in Being Able to

Communicate Qualification to Panel 197

V .18 Responses to Open Ended Questions Which

Relate to Confidence in Being Able to Communicate Qualifications to Panel 197

V .19 Responses to Open Ended Questions Which

Relate to Differences in Uncertainty Among

Bidders 203

V.20 Bidder Representatives' Responses to Question 3a on Whether the RFP Communicated

What the Government Wanted to Buy 206

V.21 Bidder Representatives' Responses to Question 3b on Whether the RFP Communicated

What the Government Wanted to Buy 207

V.22 Responses to Open Ended Questions Which

Relate to Whether the RFP Communicated

What the Government Wanted to Buy 207

V .23 Panel Members' Responses to Question 2a on

Appropriateness of the Sub-Factors 209

V.24 Bidder Representatives' Responses to Question 4a on Appropriateness of the SubFactor 210

V.25 Panel Members' Responses to Question 2b on

Appropriateness of the Sub-Factor 210

V.26 Bidder Representatives' Responses to Question 4b on Appropriateness of the SubFactor 211

V.27 Responses to Open Ended Questions Which

Relate to Appropriateness of the Sub-Factor 212

V .28 Panel Members' Responses to Question 3

Which Relates to Sensitivity 215

V .29 Responses to Open Ended Questions Which

Relate to Selectivity 217

V .30 Responses to Open Ended Questions Which

Relate to Expert Participation 227

V .31 Example of Minority Opinions Prevailing . . 233

V .32 Time Required for the Factor Team Meetings. 234
V.33 Responses to Open Ended Questions Which

Relate to Reaching Agreement 235

V .34 Examples of Different Types of Variability

Used in Rating the Bidders 240

V .35 Example of Dealing with Differences in

Variability Among Bidders 242

V.37A Comparison Between the Numerical

Averages of the Sub-Factor Scores and the

Agreed Upon Factor Scores 244

V.37B Comparison Between the Numerical Averages

of the Sub-Factor Scores and the Agreed

Upon Factor Scores 245

V .38 Individual Difference Scores for Panel

Member 205 252

V .39 Difference Scores for Sub-Factors on

Which Panel Member 205 Participated .... 253

V.40 Bias and Influence Displays for Panel

Member 205 254

V.41 Individual Difference Scores for Panel

Member 208 255

V.42 Difference Scores for Sub-Factors on Which

Panel Member 208 Participated 256

V .43 Bias and Influence Displays for Panel

Member 208 257

V.44 Individual Difference Scores for Panel

Member 210 258

V .45 Difference Scores for Sub-Factors on Which

Panel Member 210 Participa^ted 259

V.46 Bias and Influence Displays for Panel

Member 210 260

V .47 Individual Difference Scores for Panel

Member 211 26

1

V.48 Difference Scores for Sub-Factors on Which

Panel Member 211 Participated 263

V .49 Bias and Influence Displays for Panel

Member 211 265

V.50 Individual Difference Scores for Panel

Member 102 266

V. 5 I Difference Scores for Sub-Factors on Which

Panel Member 102 Participated 267

V .52 Bias and Influence Displays for Panel

Member 102 268

V .53 Individual Difference Scores for Panel

Member 105 269

V.54 Difference Scores for Sub-Factors on Which

Panel Member 105 Participated 270

' V.55 Bias and Influence Displays for Panel

Member 105 271

V .56 Individual Difference Scores for Panel

Member IO 8 272
V.57 Difference Scores for Sub-Factors on Which

Panel Member 108 Participated 273

V.58 Bias and Influence Displays for Panel

Member 108 274

V.59 Ratio of Panel Movement Toward/Away From

Rankings of Each Panel Member 275

V.60 Responses to Open Ended Questions Which

Relate to the Treatment of Bias 278

V.6l Panel Members' Responses to Question 4a

Which Relates to Admissible Information . . 280

V.62 Panel Members* Responses to Question 4b

Which Relates to Admissible Information . . 28l

V.63 Distribution of Final Overall Scores. •. . . 289

V.64 Responses to Open Ended Questions Which

Relate to Technical Discrimination of

Specific Issues 291

V.65 Bidder Representatives' Responses to Question 5a on Bidders' Understanding of the

Evaluation Process 294

V.66 Bidder Representatives' Responses to

Question 5b on Bidders' Understanding

of the Evaluation Process 295

V.67 Bidder Representatives' Responses to Question 7 on Ability of the Panel to Corrupt

the Process 295

V.68 Responses to Open Ended Questions Which

Relate to Ability of the Panel to Corrupt

the Process * . 298

V.69 Bidder Representatives' Responses to Question 6 on Weighting of Sub-Factors 303

V.70 Responses to Open Ended Questions Which

Relate to Resources Required for Bidders

to Respond 311

V.71 Miscellaneous Responses to Open Ended Questions 320

1

CHAPTER I

INTRODUCTION

I • 1 The General Probler!! of Acouirlng Evaluation Systems

In recent years the level of resources devoted by government to program evaluation has grown to a significant

level. This activity is expected to continue to expand

with increasing congressional interest in program oversight.

Because of requirements for evaluation, some programs have

become interested in developing capabilities to evaluate

their programs on an on-going basis. Some of these desired

capabilities have been labeled as evaluation systems.

Much of the federal government's evaluation work is

done under contracts. The track record of such evaluation

has not been good. The products of this work have been

criticized as being of faulty design, poor timing, and irrelevance for use in decision making.

There has been a growing recognition of the importance

of the selection of the contractor as a determinant of satisfaction with the outcome of the evaluation. Professionals in the evaluation community have begun to focus on the

role of the source selection process in evaluation.

I • 2 The Specific Problem which ETIP Faced
2

1.2.1 ETIP

ETIP (the Experimental Technology Incentives Program),

at the National Bureau of Standards (NBS), was established

by Presidential decree in 1972 to find methods which the

government could use to stimulate technological innovation.

As the program developed it pursued this mandate by working

with government agencies to experiment with modifications

in policies and procedures which were hypothesized to have

an inhibiting effect on innovation. The idea was to work

with those people in government who have the responsibility

for various programs in order to design, implement, and

evaluate the results of these "policy experiments."

1.2.2 ETIP’s System Procurement Problem

In 1975 ETIP decided that it wanted to develop some

kind of ongoing capability to evaluate the results (agency

and commercial impacts) of the experiments in procurement

practice which it was conducting with various federal,

state, and local government agencies. This capability was

to be developed using outside contractors. The responsibility for this fell on the newly hired head of evaluation

for ETIP.

He faced several problems in trying to decide how to

contract for this evaluation capability. These included

that:

0 This project was only one of his responsibilities and at the time he had no staff working
for him. This created a resource problem if

he were to meet the deadline he was given of

awarding the contract by the end of the fiscal

year.

o He had no directly related prior experience

with either the source selection process or

with systems management.

o He was not yet trusted by other members of

the ETIP staff.

o There was a great deal of uncertainty over

what the requirements for the evaluation capability would be, including who would own it,

what specific types of experiments it would

be required to evaluate, and how many experiments would be evaluated during the development

of the systems.

o The evaluation industry had a poor track record

for getting its work used in decision-making.

It had no track record at developing evaluation

systems

.

o ETIP had little visibility in the evaluation

community. Its prior attempt at contracting

for evaluation services resulted in cancellation of the RFP after the proposals had been

evaluated

.

o ETIP had a high profile with the administration

of MBS and parts of the Department of Commerce.

The source selection would be subject to close

internal scrutiny.

The manager of evaluation decided to ask Charles W.

N. Thompson, a special employee of the program, for help.

Thompson chose to use a source selection process which he

had developed and used to procure "hardware” subsystems

for the Air Force. Use of the method, called Parametric

Factor Evaluation (PFE), represented a significant departure from the way evaluations were being purchased by the

government

.
4

There was a great deal of skepticism in ETIP over

whether the source selection would be effective. People

were concerned with whether:

o the process was legal;

o any firms would respond;

o the proposals which were received could be

evaluated; and

o the selection would stand.

I . 3 The Topic of this Research

This research is a case study of ETIP’s use of the

Parametric Factor Evaluation Process to select two contractors to develop the evaluation systems which it desired.

The case study is constructed by the author from:

o background on the PFE process from both historical documents and personal communications with

its primary developer;

o participation in the process at ETIP and access

to records of the source selection; and

o interview and questionnaire information from

other participants in the source selection.

The format followed in presenting the case is to provide

the reader with background on both the source selection

process and the PFE process, to present a specific detailed

chronology of ETIP’s use of the PFE process, and to evaluate

ETIP’s experience against the set of relevant issues. Fi-
5

nally, some broader implications suggested by this research

are presented.

1 . 4 Value of this Research

This research is intended to be of value both for researchers and for people responsible for acquiring evaluation systems (or other types of systems). Its value for

researchers is as a bench mark, in terms of method and results, for those interested in further research on PFE or

comparative research on other source selection techniques.

It is also offered as an example of a credible method for

doing exploratory research. Its value for people responsible for acquiring systems is as a description of the PFE

process which answers some of the questions which someone

considering using the process may have. It provides a detailed level of information on how the process works, what

problems it will and will not solve, and how those who are

involved may feel about it.

1 . 5 The Organization of this Document

This document is organized to progress from the general to the specific and back to the general. Chapter II

contains general background and information on relevant

prior states-of-the-art in both program evaluation and systems procurement. The size of the problems, the people

working in the topic areas, the prior research, and the
6

issues which are raised by other authors are identified.

Chapter II also contains background on the Parametric Factor

Evaluation process. History of the process, claims for

where it will be useful, and a description of its characteristics are provided. Chapter III describes the central

purposes of the research. It contains a description of

both what the author set out to accomplish and the approach

which was taken. A discussion of the proposed value of

the research and a disclosure of the biases of the author

are also given. Chapter IV is a chronology of ETIP’s use

of the PFE process for its source selection. It reports

what occurred, including what issues were raised by participants in the process. Chapter V presents the evaluation

of the outcome of ETIP’s use of PFE against a set of issues

either raised by participants or contained in related literature. Section V.4 presents a summary of the results in

terms of successes, failures, and remaining uncertainties.

It contains cross-references back to the discussion of each

issue in the preceding text. Chapter VI reviews the implications of this research which go beyond the specific case

study. In it the author comments on the methodology used

for the research, draws lessons from comparison of the results of this use of PFE with other prior and subsequent

uses, mentions other problems for which parts of the PFE

process may present a useful solution, and indicates possible topics for additional research on PFE.
7

Two terms which are used quite often in later chapters

are "bidder" and "panel." The term "bidder" is used to

refer to a firm responding to a request for proposals.

Technically, the term "offeror" refers to a firm responding

to a negotiated procurement. However, since the use of

the term "bidder" is common in the literature and in practice (including use in ETIP's RFP), the author chose to

use it throughout this document. The term "panel" is used

to refer to the group of people who meet to perform the

review and scoring of the technical proposals.
8

CHAPTER II

BACKGROUND

11 . 1 Introduction

This chapter reviews background to provide perspective

for the case study which follows. Information is presented

on the related states-of-the-art in terms of issues raised

by researchers and practitioners of evaluation and systems

procurement. A description of the Parametric Factor Evaluation approach to source selection is also presented. The

review of the literature is not intended to be comprehensive. It represents a modest selective sampling of much

larger literatures in both evaluation and systems procurement. The intention is to give the reader an understanding

of the larger contexts within which ETIP’s procurement problem resided. The description of PFE is included to provide

a context for the case study in terms of the history and

characteristics of the process which ETIP used.

11 . 2 Evaluation Systems

II. 2.1 Concept of Evaluation Systems

Evaluation and evaluation systems have no agreed upon

conceptual framework (Tien, 1979). There are, however,

many government programs doing evaluation work (Comptroller

General, 1976). This work varies in scope and method.
9

Some of it is classified as developing evaluation systems

(Comptroller General, 1978, 1979). Other works, which are

large program evaluations, share many of the same characteristics and problems as that which is directed toward developing evaluation systems. It is not the purpose here to analyze these similarities and differences but merely to indicate some of the problems which have been claimed to exist.

Evaluation systems may be useful to programs which

are expected to have a continuing requirement for information related to impacts, where at least some of it is of

a stable and predictable nature. They are primarily software systems rather than hardware. They may often be large

and complex and take a long time to acquire (Thompson, 1976a).

In development they can require a large number of people

0 of diverse skills and interests to work together over a

long period of time. As such they exhibit problems of cooperation and communication and turnover in people who support or oppose the systems (Patton, 1977; Thompson, 1976a).

For comparison, they are probably most closely related to

management information systems and logistics systems.

The definition of evaluation system used by ETIP was;

EVALUATION SYSTEM - Refers to the set of policies

and procedures which provides the basis for evaluating the (agency and/or commercial) impacts

of a set of specific procurement or related experiments. The form, detail and completeness of the

system will progressively change. In Phase One,

it may begin as a preliminary outline within which

detailed "single thread" designs are developed
10

for the early evaluation of specific procurement experiments; later, the results of the preliminary systems

analysis and the "pilot test" will provide the basis

for a preliminary evaluation systems design. In Phase

Two, progressive refinement should result in_a relatively complete evaluation system which can be tested

as a prototype, in part, through the evaluation of

specific experiments. In Phase Three, the evaluation

system should be in the form of a stable and complete

system (Thompson, 1976, p. 5).

II. 2. 2 The Level of Evaluation in the Federal Government

The level of resources devoted to government program

evaluation has increased significantly in recent years and

it appears that this trend will continue. It has been estimated that the federal government alone spent $243 million

on executive branch program evaluation in fiscal year 1977

(Granquist, 1977). This may be compared to an estimate

of $20 million spent in fiscal year 1969 (Knezo, 1974).^

In the last several years Congress has demonstrated

increased interest in expanding and exercising its oversight

authority (Comptroller General, 1977; Havens, 1977). On

October 11, 1978, the Senate passed an oversight reform

proposal, S. 2, The Sunset Act of 1978 . This legislation,

which has facetiously been referred to as the Evaluators

Full Employment Act, would require periodic review and re1 Though the methods used in counting probably differed,

these figures do give an indication of magnitude and trend.

It should also be noted that the $243 million does not include Congressional evaluation.
authorization of nearly all federal programs. Though the

House of Representatives did not pass a similar bill during

the 95th Congress, there is some probability that similar

legislation will exist in the near future. The result will

be an even greater demand for program evaluations.

II. 2. 3 The Evaluation Community

A community of professional evaluators has grown up

in response to the government's demand for their services.

These people reside both within and outside of the government. Within the government most executive branch agencies

have high-level organizational units specifically devoted

to evaluation and many have evaluation functions built into

the lower levels. The people who work in these units either

perform in-house evaluations or manage cor;tracts and grants

which support their evaluation programs. Congressional

arms of the government such as the General Accounting Office,

the Congressional Budget Office, and the Office of Technology Assessment also have sizable in-house evaluation functions. Outside of government, professional evaluators exist

in both universities and in contract research organizations.

These people do evaluations for government programs on contracts or grants and also do research on methods for improved

evaluation. In a sense they form a research industry (Biderman & Sharp, 1972).
Communication networks have evolved to serve the needs

of this professional community. They have their own professional organization (e.g., The Evaluation Research Society;

The Council for Applied Social Research) and journals (e.g.,

Evaluation Quarterly ; Evaluation ) , and contribute to discussions in the media of other related organizations (e.g.,

The Institute of Management Sciences; The American Psychological Association). University programs have developed

to research topics related to evaluation and to teach students how to do this work.

II. 2.The Pains of Adolescence

The proliferation of evaluation in the government has

brought with it a debate among practitioners and intended

users as to the quality of the resultant products. This

debate takes place in the media and at the conferences of

evaluators and program administrators and at Congressional

hearings. It often centers on the fact that the results

of many evaluations are not used in decision-making about

the programs which they review ( Agarwala-Rogers , 1977; Berli

1977; Chelimsky, 1977, Vol. II, pp. 33-44; Cohen & Caret,

1975; Weiss, 1972, pp. 10-11; Wholey, Scanlon, Duffy, Fukumato & Vogt, 1970; Williams ,& Evans, 1969).

Those who critique evaluations raise questions of the

relevance, methodological appropriateness, and political

astuteness of evaluators. Many of the claims of problems
13

with evaluators center on the irrelevance of their work

to the management of the program being evaluated. It is

claimed that evaluators often have little understanding

of the program processes or the systems in which the programs operate (Horst, Nay, Scanlon, Wholey, 1974). They

frequently put a great deal of emphasis on peripheral issues

which are of little interest to program managers (Chelimsky,

1977, Vol II, p. U3; Granquist, 1977; Guttentag, 1973; Havens,

1977). Aspects of programs being evaluated are not put

into a context in which they address specific policy issues

(Cook, 1978; Lynch, 1976). Evaluations are also criticized

for being submitted too late to be of use in decision-making

(Evans, 1977; Havens, 1977; Wholey et al., 1970).

In the area of methodology it is claimed that faulty

or simplistic designs are used which lead to false or misleading or suspect conclusions (Horst, et al., 1974).

The relevance of measurement schemes to the program being

evaluated (Horst, et aL, 1974), the adequacy of data (Havens,

1977), and the quality of analysis are all questioned (Havens,

1977). Attacks of methodology are frequently used to undermine confidence in the results of evaluations (Evans, 1977).

Evaluators are criticized for their lack of political astuteness (Patton et al, 1977). There has been a tendency on the part of some evaluators to work in isolation

from the managers of the programs they are evaluating and
to favor surprise findings (Brands, 1977; Patton, et al.,

1977). Related to this is the mis-direction of evaluation

reports to higher levels in the organization or to the press

rather than on-going presentation of results to those with

direct control of the programs (Patton, et al, 1977).

Evaluation reports have also been criticized for their large

volume, pretentiousness, and frequent use of jargon (Cook,

1978; Lynch, 1976).

Evaluators and program managers may work toward differing purposes, evaluators being concerned with technical

excellence of research designs and conclusiveness of attribution of effect and program people being concerned with

relevance, timeliness, breadth of analysis and responsiveness (Chelimsky, 1977, Vol. II, p. 31).

II. 2. 5 Contract Evaluation

In looking for causes of the disappointing outcomes

of evaluations, some authors have turned attention toward

the procurement process (Bernstein & Freeman, 1975; Biderman

& Sharp, 1972, 1972a, 1974, 1974a; Mitchell, 1973; 0MB,

1975). There seems to be a growing recognition of the importance of finding ways for basing the selection of contractors upon criteria which have a stronger relationship

to expected performance. Articles on how to write better

RFPs (Weidman, 1977) and critiques of issued RFPs (Evans

& Anderson, 1978; Hess, Floden, Sproull, & Conry, 1978)
have appeared in journals. A recent meeting of the Evaluation Research Society (November 1978) held a session on

the RFP Award Process. Another professional organization

began sponsorship of an annual award for the best evaluation

RFP (Council for Applied Social Research, Note 1).

Some of the problems which are raised with the procurement process include the high cost to firms of submitting

proposals, the lack of guidance for self selection, the

vagueness of RFPs, and the suspicion that the government

has a preferred source in mind in advance of receiving proposals (Biderman i Sharp, 1972a). Another problem is that

the evaluation which is undertaken often has little relation

ship to the evaluation which was described in the RFP (Bider

man & Sharp, 1972a). Some authors question the competence

of federal employees to develop RFPs and evaluate proposals.

They suggest more input into the process by the academic

community, (Bernstein & Freeman, 1975), including issuing

RFPs for contracts to write RFPs (Hess, et. al, 1978).

II. 2. 6 Relationship to Systems Procurement

Comparisons can be drawn between the problems of acquir

ing evaluation systems and the problems of acquiring weapons

systems (Thompson, 1978a; Yarmolinsky, 1976). However,

the literature on source selection for weapons systems is

rich by comparison to that of evaluation systems. It is

a more mature area of study to which the government and
16

research community have devoted considerably more attention.

Therefore, it is useful to examine the literature on systems

procurement to better understand the procurement process

and to identify the issues which may also be important for

evaluation systems procurement. That is the purpose of

section II. 3»

II. 2. 7 Summary of Evaluation Systems Background

This section has reviewed some of the literature related to evaluation systems. Emphasis was placed on identifying the scope of federal evaluation, some of the problems

identified with its use, and especially those identified

with the procurement process. The next section reviews

literature on systems procurement.

II . 3 Systems Procurement

II. 3.1 The Concept of Systems

The word ’’system" has been used to refer to many differ

ent things (Ackoff, 1971). For purposes of this research,

a brief indication of system characteristics is probably

more useful than using any one definition. Rudelius (1969),

indicates aspects of large systems which make them unique

by summarizing the first three chapters of Peck and Scherer

(1962, pp. 3-64). He lists these as being the presence

of:
o a single buyer,

o a single seller,

o high technological and political uncertainty,

o high cost,

o high complexity,

o high lead time,

o limited production, and

o contract awards made before development of

a precise system definition (Rudelius, 1969,

p. 23).

Thompson (1978) describes some dimensions which may be used

to differentiate among various systems. These are:

o Hardware <—•> Software

(the degree to which the system can be described in terms of specific physical items),

o Many —•> Few

(the degree to which the system will be produced in quantity),

o Incremental <—> Novel

(the degree to which the system represents

a change from the "system" it replaces),

and

o Inhouse <—•> Isolated

(the degree to which the developer of the

system is responsive to the direction and

control of the user).

II. 3.2 Size of System Procurement

Systems procurements represent a significant element

of federal government expenditures. The Office of Management and Budget Circular Number A-109 states that’

the acquisition of major systems by the Federal

Government constitutes one of the most crucial

and expensive activities performed to meet national needs. Its impact is on technology, on the
Nation’s economic and fiscal policies, and on

the accomplishment of Government agency missions

in such fields as defense, space, energy and transportation (Office of Management Budget, 1976,

p . 1 ) . Several years ago Hall (1975, p. 88) estimated that the

Department of Defense had over 150 major and minor systems

acquisition programs. He placed the cost of these programs

at over $200 billion for development plus an amount equal'

for follow-on.

Much systems work is done by private firms under contract to the government. Livingston (1959), in an early

article on the system concept, indicates that

The philosophy underlying the weapon system concept holds that authority for the design, development, and production of an entire weapon or support system. .. should be assigned to a single

[original in italics] management organization (p.

83).

The primary reason for this is the great complexity of devel

oping and integrating all of the system components. Since

the military services found that they did not possess sufficient in-house capability to manage the development of

their programs they began contracting for systems management

(Livingston, 1959).

II. 3. 3 Research Related to Systems Procurements

Research on procurement has been sponsored by both

government and industry. Spagnola (1978, p. 397) traces

the Department of Defense’s formal interest in procurement

research to the Defense Procurement Pricing Conference held

in Hershey, Pennsylvania in November, 1977. Since then
the various services have set up offices to conduct procurement research. The Army established its Procurement Research

Office (APRO) in 1959; the Air Force established the Business Management Research Center (AF3MRC) in 1973; and the

Navy began its research program in 1977 (Spagnola, 1978,

p. 397). There has been the addition of a graduate procurement curriculum to the Air Force Institute of Technology

and a Systems Acquisition Management curriculum to the Naval

Postgraduate School (Martin, Heuer, Kingston, i Williams,

1978, p U22). The Defense Systems Management College has

also initiated a program of acquisition research (Spagnola,

1978, p. 397). In 1976 the Office of Management and Budget

established the Federal Procurement Institute (now the Federal Acquisition Institute) to serve as a government-wide

focal point for procurement research. Industry research

has been sponsored by groups such as the Aerospace Industries Association (e.g., Aerospace Industries Association,

1969).

The research in this area is reported in professional

Journals (e.g., Esposito, 1973; Goldberg, 1977; Skolnick,

1969), conference proceedings (e.g., Cochran 4 Rowe, 1977;

Martin, et.al, 1977), theses and dissertations (e.g., Bailey,

1964; Logan, 1966), books (e.g.. Fox, 1974; Peck 4 Scherer,

1962; Scherer, 1964), and reports (e.g.. Commission on Govern

ment Procurement, 1972). It is conducted both by practi-
20

tioners (e.g., Carnes, 1976) and academics (e.g., Hynes,

1977; Roberts, 1964).

11.3.4 Characteristics of Systems Procurements

For purposes of discussion, the procurement of systems

is divided here into two phases. This division is illustrated in Figure II. 1. The first phase, labeled source selection, is that part of the procurement process which terminates with the signing of a contract between the government

and a contractor. The second phase, labeled the system

Source Selection N System Development'

Sign (Contract

* FIGURE II. 1 TWO PHASES OF THE PROCUREMENT OF SYSTEMS

development, is that part of the procurement process which

follows the signing of the contract and results in work

toward implementing an in-place system. Though this research

is primarily concerned with the source selection process,

it is important to look at the development process to the

extent that the characteristics of this process interact

with source selection. That is, it is useful to look at

some of the characteristics of the development process which

make the source selection process difficult and in turn

some of the characteristics of the source selection process

which may affect the development process. O’Connor (1972)
21

has pointed out that early misunderstandings may lead to

expensive mistakes downstream in the program. It is therefore desirable to use a source selection process which will

be able to predict and in some manner favorably affect the

development process.

The characteristics of complexity and uncertainty lead

to problems. System developments have both technical and

organizational components. Technical complexity results

from the large number of technical problems present, the

interrelationship among these problems, and reliability

requirements of the individual components (Peck & Scherer

1962, p. 42). Organizational complexity results from the

large number of objectives which different people have for

any system, the changes which occur in objectives, and the

need to resolve conflicts among them (Peck & Scherer, 1962,

p. 80). Included here are differences between the objectives of the government and those of the systems contractor.

Interface between technical problems and organizational

problems is important.

Uncertainty in systems development results from the

complexities of the problem and the required long development time (Ruth, 1978; Thompson, 1978). Complexity leads

to technical difficulties which were not forseen in the

early stages of the program (Fox, 1974, p. 372; Peck & Scherer, 1962, p. 24). Some of these problems may have no fea-
22

sible solution (Eyring, 1966, p. 170) and must result in

modifications in the system's performance characteristics.

The long development time may result in required changes

due to advances in the technical state-of-the-art (Peck

& Scherer, 1962, p. 46; Srull, 1972, p. 4), modifications

to strategic requirements (Esposito, 1973, P* 9; Fox, 1974,

p. 369; Peck & Scherer, 1962, p. 48), or new government

policies (Peck & Scherer 1962, p. 50). The demand for a system may also vary over time, particularly as competing

systems become available and with turnover in government

personnel who support or oppose them (Thompson, 1978).

There will also be turnover of personnel by the system contractor which may affect approaches used (Ruth, 1978).

The literature which discusses problems with the development of systems emphasizes the large numbers of changes

which occur. These changes are in cost, schedule, and performance characteristics. Cost is often discussed in terms

of overruns, schedule in terms of slippages, and performance

in terms of both increased and decreased capabilities.

Cost, schedule, and performance characteristics are all

closely interrelated and subject to similar influences (Anderson, 1969, p. 163; Rudelius, 1969). For example a technical

problem in meeting performance capability may lead to increased cost and longer time until delivery.. An attempt

to cut costs may lead to reduced capabilities and longer

time until delivery. An attempt to speed delivery may lead
to greater cost and reduced performance. Because of this

interrelatedness, the causes of schedule slippage and performance modifications will be covered here within the discussion of cost overruns which follows.

The literature on the causes of overruns can be grouped

into six categories. These are;

o having to lie about the cost--buying in,

o spending time on things which are unimportant,

o having to do something different from what

was originally planned,

0 doing a poor Job of estimating costs,

o making mistakes requiring re-work, and

o catastrophic external events (Cochran & Rowe,

1977; Thompson, Note 2).

Buylng-ln can be attributed to circumstances which

require that both the government and the bidder lie about

the initial estimated cost of development. The government

program manager may use an unrealistically optimistic cost

estimate in order to obtain approval for his system (Fox,

1974, p. 159; Peck 4 Scherer, 1962, pp. 412-413). Thus

the program is initially inadequately funded (Patterson,

1977, p. 61). Bidders are forced by the government to underestimate their own costs if they wish to obtain contracts

(Peck 4 Scherer, 1962, p. 412-413). Bidders often believe

that there is little chance of obtaining future contracts

without initial contracts (Babione, 1978; Patterson, 1977,

p. 60). They may also be more concerned with obtaining
revenue to cover fixed costs than with longer term cost

issues (Patterson, 1977, p. 60). Firms realize that price

may be the primary consideration in making an award (Packard,

1972 ), since the government must defend a selection of anyone other than the lowest bidder (Fox, 1974, p. 277), and

the technical evaluation may result in an inability to discriminate among bidders (Carnes, 1976, p. 2). When under

contract it becomes clear that the award price will not

result in the desired performance characteristics, the successful bidder requests contract modifications (Anderson,

1969 , p. 163 ) and the government program manager requests

a supplemental budget (Peck & Scherer, 1962, pp. 412-413).

In procurements which are characterized by high uncertainty,

the initial price may have little relationship to the cost

to perform (Thompson, 1978a).

Spending time on things which are unimportant may be

related to differences in objectives between the government

and its contractor, management inefficiencies, or the desire

by the government or its contractor to justify future costs.

Objectives of the contractor which are not held by the government may lead to the addition to systems of non-essential

performance characteristics ( "goldplating" ) (Fox, 1974, p.

369 ). Contractors may make their decisions on allocation

of effort (and on doing work in-house vs. contracting out)

based on factors such as desire to:
0 enhance their competitive position (e.g.,

maintain competence, public image, and market

share) (Fox, 197^, d. 441; Peck & Scherer,

1962, p. 458)

o cover overhead (e.g., increased direct cost

results in increased charges to overhead

and indirect cost)(Fox 1974, pp. 232, 299),

and

o maintain stability of employment (e.g., during

delays while waiting for approval (Fox, 1974,

pp. 165, 419; Peck & Scherer, 1962, p. 458K

Management inefficiencies may be the result of lack of responsibility for managing costs (Fox, 1974, p. 434), or

the tendency for work to be authorized within functional

organizations without coordination with upper level management (Fox, 1974, p. 419). The government project manager

may encourage ’’goldplating" in order to justify future year

budget requests (Fox, 1974, p. 482). By increasing costs

the contractor may also try to justify higher future estimates (Fox, 1974, p. 232).

Being required to do something different from what

was originally planned results from the need to respond

to the customer’s evolving concept of uses of the system

(Cochran & Rowe, 1977, p. 540). Changes in the state-ofthe-art can lead to required new features which were not

originally anticipated (Cochran & Rowe, 1977; Fox, 1974,

p. 165). In many instances the government requires bidders

to specify approaches in their proposals in areas of high

uncertainty (Drake, 1970, p. 124). This can lead to early
26

decisions which lock the program into a single solution

and require considerable additional expense to undertake

other options (Hall, 1975, p. 97; Keeling, 1975, p. 130).

Poor cost estimates are attributed to not dealing adequately with uncertainties. Bidders are required to cost

out systems with little or no historical data upon which

to base many of their estimates. This is particularly true

where the system is required to push the state-of-the-art

(Fox, 1974, p. 165; Keeling, 1974, p. 130; Martin, Glover

& Lenz, 1977; Srull, 1972, p. 3). The estimates which result

do not reflect differences in uncertainty over the costs

of various parts of the system (Anderson, 1969; Drake, 1970;

Thompson, 1978a)

Mistakes requiring re-work may be caused by setting

overly ambitious requirements or by poor workmanship. If

the original requirements call for significant increases

in the state-of-the-art for various parts of the system,

it is possible that one or more of these advances will prove

too ambitious and require re-design (Comptroller General,

1973, p. 26). Instances of poor workmanship may also require re-work (Fox, 1974, p. 165).

Catastrophic external events may lead to significant

^ uncontrollable overruns. Included here are events such

as natural disasters, civil disorder, major strikes, and

fire (Cochran & Rowe, 1977, p. 536). Severe unanticipated

inflation might also be included in this category.
Some of the factors which lead to contract changes

may be anticipated before or during source selection. In

some cases the source selection process may be a cause of

the problem. In others the ability to be able to better

evaluate the technical and management capabilities of a

bidder may prevent problems from occurring. Some feel that

the selection of the contractor may be the most important

aspect of an acquisition program (Helman & Taylor, 1976,

p . 88 )

11.3*5 The Role of Source Selection

The primary roles of the source selection process are

to select a contractor, allocate risk between the government

and the contractor, and to provide for review and management

by the government once a contract is signed (Lenk, 1977).

The procurement process has its roots in the purchase of

food, soap and other necessities for George Washington’s

Army (Report of the Commission on Government Procurement,

1972, Appendix G) . Various contractual mechanisms have

evolved to try to meet most effectively the government’s

requirements for obtaining specific types of goods and services. Waks (I960) traces the use of the system source

2Appendix G of the Report of the Commission on Government Procurement (1972) provides an informative history

of the evolution of the procurement process.

3 Lenk (1977) provides a good overview of the various

contractual mechanisms used by the government.
selection concept to the spring of 1955 when it was first

used by the Air Force. He explains that source selection

shifts the emphasis from making a technical analysis of

the relative merits of specific design proposals, to making

an overall analysis of the relative ability of a select

4

group of sources to accept a particular design assignment.

The literature on source selection suggests that there

is dissatisfaction with both the process of source selection

and the results of the process (Esposito, 1973? P* 9; Hynes,

1977). This dissatisfaction occurs both among government

and industry personnel (Keeling, 1975, pp. 129-130). The

significant resources required have not paid off in the

achievement of the outputs desired (Blue Ribbon Defense

Panel, 1970; Esposito, 1973, p. 9).

11.3.6 Issues Related to Source Selection

In reviewing the issues raised about the source selection process, it is useful to divide the process into four

stages. A somewhat simplistic division is represented in

Figure II. 2. The first stage incorporates selecting the

type of contractual mechanism to be used, writing and distributing the RFP and briefing the prospective bidders.

4 The term System Source Selection was adopted by the

Air Force for a set of guidelines for accomplishing this

type of solicitation and selection.
29

oHs0

•r-l >0

cc

>,u

Cd T3 B Sh S cd 3 3

CO <AI I CQ Cd -P 3 0

rH 0

Cd s: >

Cd rH0

rH

C

Cd Cd w cu oa c 0 o

Si *r-t 01A

«4-l O

hC

O c Cd

•tH SL, C -P

bC

C

•H

O

CO CJ

CO

CO

Cd

CJ0

05

01:so

I—I O

Cd

-JCd

CO

CdO

05

3)O

CO

CdKH

Cci o

CO

CdCO<

oc: Di

05

3)o

bj

C\J

Cd

(no

I—I

Cd
The second stage of proposal preparation includes the decisions by firms on whether and how to bid and the actual

writing of proposals. The third stage includes the recruiting and briefing of the proposal evaluation panel and conducting the technical evaluation of proposals which are

received. Summary, review, and award is that part of the . process which includes summarizing the results of the panel

meetings, determining the competitive range, reviewing cost

proposals, obtaining clarifications, negotiation, signing

the contract, and debriefing unsuccessful bidders. Though

these stages are, for the most part sequential, they do

overlap somewhat in time (e.g., selection of the panel generally occurs before proposal writing is completed).

It is generally desirable that each stage be conducted

both effectively and efficiently. It is also desirable

that each stage support what comes later (e.g., the format

required for proposals may affect the ability of the panel

to accomplish its evaluation) . The remainder of this subsection will use these four phases to highlight what the

literature raises as issues relating to the effectiveness

and efficiency of the source selection process.

11 . 3 . 6.1 Preparation and release of the RFP and briefing potential bidders . The RFP has been described as being

primarily a business action (Esposito, 1973, p. 9); its

purpose being to provide accurate information and guidance
in a competitive marketplace (Esposito, 1973, p. 11; Morris,

1973, p. 19). The information provided and the form for

the responses should allow prospective bidders to determine

effectively whether to bid and enable them to convey their

capabilities to the government (Fox, 1974, p 301; Keeling,

1975). Hynes (1977), in reporting on a survey of defense

contractors, indicates that there is a great deal of dissatisfaction with the RFP as a communication instrument.

One of the first tasks when preparing an RFP is to

select an appropriate contract type. Gordon (1980) argues

strongly for realistically matching the contract terms to

the expectations for the system and the inherent technical

and financial risks. Systems procurements currently tend

to make use of cost reimbursement contracts rather than

fixed price contracts (Lenk, 1977). It is intended that

this mechanism be used where there is uncertainty as to

what the final product is, its form, how much it will cost

and the quantity to be produced (Fox, 1974, pp. 230-231;

Skolnick, 1969). Contractors are reimbursed for allowable

direct and indirect costs, overhead and often a fee bas.ed

c Anderson (1969) describes how problems occurred when

Secretary McNamara required the Department of Defense to

procure systems using fixed price contracts. The most publicized case is the use of Total Package Procurement for

the C-5A program.
32

upon a percentage of the negotiated contract price. Systems

procurements also tend to make use of negotiated rather

than advertised procurements (Fox, 197^, p. 252; Schnitzer,

1973) • In negotiated procurements both price and other

factors (technical management) are to be evaluated in determining who wins.

A number of important issues relate to the writing

of the RFP. One of the first to arise is to determine who

should be involved in writing it. Esposito (1973, p. 10)

and Skolnick (1969, p. 74) argue for writing of the RFP

and retention of the source selection process by the staff

people who are most closely concerned with the specific

aspects of the program and its management.

The basis for competition among bidders must be determined. Peck and Scherer (1962, pp. 343-355) discuss design

competition and the management competition as two ends of

a scale for the technical evaluation. In design competition

bidders are evaluated on the basis of the specific designs

which they propose for the system. In management competition

the evaluation is based upon the approach the bidder intends

to use in dealing with various tasks to be accomplished

in developing, the system and the capability of the proposed

Lenk (1977) reviews some of the variations on these

two basic types of contracts, such as those utilizing various incentive schemes.
33

team (Peck & Scherer, 1962, pp. 3^3, 354, 362, 367). Helman

and Taylor (1976, p. 88) indicate that often bidder’s management capability is not adequately covered during proposal

evaluation

.

The level of detail of advance system design work provided in the RFP must also be determined. If the government

has the time and resources, it may wish to develop a detailed

design in advance of issuing an RFP and require bids on

developing the system based upon this design. If time and

resources do not allow this, the government may require

bids against a set of performance specifications (Hall,

1975, p. 98). Esposito (1973, p. 12) indicates that a key

problem in providing information on requirements is to reflect and deal effectively with the varying levels of uncertainty. He argues for separating the knowns from the unknowns by doing a good job of defining the knowns and by

identifying the unknowns and a management approach for dealing with them.

A format for the bidders’ responses must be determined.

Decisions are required on both the outline and the allowable^

length for proposals. Frequently the outline is arranged

in a manner which does not mesh with the other elements

of the RFP (i.e., the statement of work, the evaluation

criteria, the work breakdown). The result is wasted time

and poor proposals (Morris, 1973, p. 23). Morris (1973,
34

p. 24) advocates using an outline which would allow proposals to be so uniform that they could be evaluated almost

on a paragraph-by-paragraph basis. Many people also advocate limiting the allowable length of proposals. Claims

in favor of this are that proposals can be evaluated using

fewer government resources (Fox, 1974, p. 266; Skolnick,

1969 , p. 82 ), and that bidders are forced to concentrate

on the most important issues (Fox, 1974, p. 266). Logan

( 1962 , p. 32 ) mentions the problem of brochuremenship or

"gold plating" of proposals, where contractors hope to make

up for weakness through volume and fancy art work. He indicates that this tends to prejudice the evaluator who has

to "sift through the chaff in order to find the wheat" (Logan,

1966 , p. 32 ). An alternate view of this problem is that

bidders resort to brochuremenship or "gold plating" because

they cannot tell from the RFP what information the government will use in its evaluation. In order to be safe they

try to provide information on every issue they think might

arise during the evaluation panel’s deliberations (Thompson,

1978a).

Another issue which is closely related to this is to

decide what information to place in the RFP about the proposal evaluation process. An argument for complete information on the process is that bidders will submit more responsive proposals which will take less time to evaluate (Hostler,
35

1973, P 13; Keeling, 1975, p. 125; Morris, 1973, pp. 21-

22). A counter argument which the author has heard, but

has not noticed in the literature, is that telling the bidders the criteria for evaluation will enable them to tell

you only what they think you want to hear, thereby not giving

you a basis for discriminating among them.

A final issue on RFP preparation is to determine how

much information to furnish to prospective bidders. Steward

(1973) argues for limiting the size of RFPs. His claim

is that it forces better writing, creates fewer errors of

inconsistency, can be read and understood at the corporate

officer level, and that it limits the amount of information

submitted in response (Stewart, 1973, p. 1^). Another approach is to try to furnish anything which may help to minimize misunderstandings (Thompson, 1978). This includes

presentations and question and answer sessions at bidders

briefings

11.3.6.2 Proposal preparation . During the proposal

preparation phase a prospective bidder must determine whether

to bid and if they decide to bid, they must determine what

approach to take in writing their proposal (Goldberg, 1977,

p. 257). Since the government assumes the cost of evaluation of proposals and most of the cost of preparation of

proposals (through reimbursable overhead on other contracts),

it may be interested in a process which allows firms to
36

self select effectively on bidding (Bailey, 1964, p. 16;

Roberts, 1964, p. 76).

Several criteria are discussed as being important for

a prospective bidder to consider in determining whether

to bid. They relate to market strategy and whether the

firm thinks it has a good chance of winning. The firm's

decision on bidding may be influenced by desire to develop

or maintain a reputation in the line of work required by

the RFP and by factors of availability of company resources

(Peck & Scherer, 1962). If bidding is consistent with the

company's market strategy, they may next assess their chances

of winning the contract. They will want to assess their

market intelligence to determine whether the contract is

"wired" (i.e., whether the government has established a sole source position on the award) (Fox, 1974, p. 296),

whether they have a positive image with the program personnel (Fox, 1974, p. 296; Roberts, 1964, p. 75), and whether

the actual objectives of these personnel differ from what

is said in the RFP (Fox, 1974, p. 296). Bidders may also

consider the resources available to them for bidding (i.e.,

time of specific people and money) and whether they feel

that they can successfully demonstrate their capability

with the resources that they are able to apply (Goldberg,

1977, p. 257).
If a firm decides to bid, it may decide to choose one

of two major strategies. One is to try to submit the lowest

bid while being at least satisfactory on all other aspects

of the proposal. The other is to try to submit the superior

proposal and to have a cost in the competitive range (Fox,

1974, p. 296).

11 . 3 . 6.3 Proposal evaluation panel meets . The role

of the proposal evaluation panel is to assess the degree

to which each proposal is potentially technically acceptable

on each of the evaluation criteria. Some of the key issues

for this part of the source selection process are recruiting

and preparing the panel, dealing with the large amount of

information which must be processed in order to assess success, determing a scale for use in evaluating proposals,

deciding what information is admissible for evaluation purposes, establishing a process for obtaining a group evaluation, and having a method for dealing with missing information.

When recruiting an evaluation panel, it may be useful

to obtain at least one person who is qualified to represent

each important aspect of the system. Both technical experts

and program participants are important (Bureau of Naval

Weapons, 1965, pp. 8-9).'^ Carnes ( 1976) and Logan ( 1966)

Roberts (1964, p. 74) indicates that in some cases,

the technical initiator may select the panel with the objective of getting participants who will favor his preferred

bidder

.
indicate that it is frequently quite difficult to obtain

participation of experts on evaluation panels. The time

required for preparation and participation may prevent experts from taking part.

Panel members are often responsible for sorting through

vast amounts of information which is present in proposals

in order to arrive at an estimate of the probability that

each bidder will perform effectively. In working with large

amounts of information people are constrained by limits

on their ability to process it (Miller, 1956; Weaver, 19^9).

The proposal evaluation process will generally have some

method for chunking information and then integrating scores.

A form of rating scale must also be determined. Dycus

(1977, pp. 257-258) identifies five general categories of

rating scale as numeric, adjective-numeric (e.g., 1-10 scale

with qualitative adjective for each score), check off (e.g.,

acceptable/unacceptable), adjective (e.g., color codes),

and relative rank (i.e., ordering from best to poorest).

After comparing them for various scale effects, he suggests

that "the best approach might be a rating scale that asks

the evaluators to consciously make both absolute and relative evaluative judgments" (Dycus, 1977, p. 264).

The issue of admissible evidence is primarily one of

how to deal with bias where the members of the panel know
more about some bidders than others. Roberts (1964) points

out that the lack of bias in proposal evaluation is impossible and probably undesirable. He indicates that technical

"facts” are subject to dispute by experts and that appraisals are based upon judgments, technical prejudices, and

other subjective factors (Roberts, 1964, p. 74). The issue

of bias is how to recognize it and incorporate it into the

evaluation so that it is not used selectively for or against

only certain specific bidders. This is an especially difficult problem when it comes to incorporating information

about a bidder’s past performance into the evaluation.

Nassr (1978) indicates that DOD has tended to place insuffic

ient emphasis on doing this. Richardson (1977) quotes an

Army General as saying "the concept [use of past performance

in evaluation] is unbelievably difficult to implement and

there does not appear to be any ready formal solution" (Rich

ardson, 1977, p. 425). Keeling (1975, p. 130) reports that

the methods which have been tried for doing this have been

based on historical data and an assumption that contractors

are monolithic. He argues for analyzing past performance

based upon the track records and the proposed extent of

involvement of the specific personnel on the company’s proposed project team.

The evaluation panel meetings are a group decisionmaking process and as such require some type of process
for determining how scores are assigned. Voting, reaching

consensus, and leaving the decision to the chairman are

three possibilities. Since this is a group decision-making

process, issues of how the ground rules affect motivation

and effectiveness are relevant. Some important issues may

include whether:

o the group members feel confident that the

results of their evaluation will be used

in the final decisions (Bradford, Stock,

& Horwi tz , 1953);

0 there is a process for dealing with strong

individual disagreements (Bradford, Stock

& Horwitz, 1953);

o lower status individuals who feel that their

opinions are better than higher status individuals are able to express themselves and

prevail (Torrance, 1957, pp. 314-318);

0 the group is able to resist undue influence

by dominant personality types (Dalkey & Helmer,

1963; Roberts, 1964); and

o implied threat of sanctions from more knowledgeable members is controlled (Dalkey &

Helmer, 1963 ) •

A final aspect of the evaluation panel process is determining how to deal with missing information. In some instances the panel will not understand an approach being

proposed by a bidder. This may be due to lack of information provided in the proposal (Carnes, 1976, p. 7). If

the panel has no way of obtaining this missing information,

it may result in the bidder receiving a lower evaluation

than is warranted (Fox, 1974, p. 273). Carnes (1976, p.
41

7) describes how incorrect scoring of initial proposals

occurs when the only form of communication is through a

deficiency reporting process.

11.3*6.4 Summary, review and award . After the panel

has completed its evaluation, the results of the process

are summarized in written form and submitted to the contracting office for review, clarification and negotiation. The

summary is generally done- by the chairman of the evaluation

panel (often the Contracting Officer’s Technical Representative) who also recommends those bidders who may be technically qualified for the contract after clarification and

negotiation. Often the program will have a person or group

of people above the level of the panel chairman who review

his report and recommendations before they go to the contract

ing office. The technical evaluation is then reviewed together with an analysis of the cost proposals. Clarifications are obtained from bidders if necessary. The clarification process may result in some bidders being excluded from

the remaining competition. The contracting office negotiates with the remaining bidders on technical considerations,

price considerations, and provisions for future changes.

The contracting officer, after consulting with technical

people, makes a determination of the offer which is most

favorable and signs a contract. The signing is often fol-
42

lowed by requests for debriefing and occasionally protests

of the award by unsuccessful bidders.

Determination of the competitive range is largely within the discretion of the contracting officer. He is not

bound by the recommendations of the evaluation panel

(Schnitzer, 1973). The^ relevant statute (10 U.S.C. 2304(g))

requires that "an offeror is held to be in the competitive

range if his proposal can be improved reasonably to the

point where it becomes the most acceptable." In practice

this is often a difficult determination to make. Scores

of competing contractors are often very close. The large

number of items evaluated often levels out the overall rating

(Blue Ribbon Defense Panel, 1970). In other cases a bidder’s overriding superiority in a particular area may saturate the evaluation, even though the bidder is weak in other

areas of the proposal (Skolnick, 1969, p. 75). The numerical scores may obscure some of the more important issues

(Blue Ribbon Defense Panel, 1970). Schnitzer (1973? p.

17) stresses that competitive range should be decided on

the basis of the array of scores received rather than an

overall passing grade.

If there is a higher level review committee, it may

disagree with the recommendations of the panel. In this

case, the panel may be asked to revise their report until

the senior group agrees with it (Carnes, 1976, pp. 27-30;
Roberts, I960, p. 75). The discussions and wisdom of the

panel are ignored.

Because of the uncertainty associated with information

in the proposals, clarifications will often be required

before a confident technical assessment can be completed.

Representatives of firms in the competitive range are called

in for discussions with technical and contracting office

representatives. Technical leveling is a problem which

often arises here. The technical people may wish to get

all firms to incorporate approaches which they found favor

with in the proposals of other firms. When firms are willing to modify their proposals to comply, there is little

basis for discriminating among proposals on technical grounds

(Babione, 1978; Carnes, 1976, pp. 30-31; Hall, 1975, p.

89; Schnitzer, 1973, p. 27).

If there is no basis for technical discrimination,

the result is a price competition (Carnes, 1976, p. 2; Fox,

T974, pp. 251, 277, 470; Keeling, 1975, p. 127; Richardson,

1977, p. 423). In effect this turns a negotiated procurement into an advertised procurement. Award is generally

made to the low bidder, since there is a requirement to

justify the choice of any other bidder on technical grounds

(Fox, 1974, pp. 251, 277). This encourages "buy-ins," pricing the bid low in order to get the contract and then requesting price or performance modifications when it is clear
that the promised system cannot be produced at that price

(Keeling, 1975, p. 127; Logan, 1966). The contractor may

use various techniques to come up with a lower cost, including overstating his future business base to obtain a lower

overhead rate (Keeling, 1975, p. 127) and using less costly

personnel than required to do the work.

Often after the contract is signed unsuccessful bidders

will request debriefings to discover where they were considered weak. Buying agencies sometimes have trouble explain

ing the basis for their evaluation (Peck & Scherer, 1962,

p. 372). If bidders feel that they were evaluated unfairly,

they may file a protest of the award with the General Account

ing Office. A successful protest may result if the government did not follow its stated evaluation procedure, i.e.,

it based the award on criteria other than what were listed

in the RFP or if systematic bias entered into the evaluation

process. Even if a formal protest is not filed, a bidder

who feels that he was treated unfairly may be reluctant

to bid on future work by the government agency.

11.3*7 Summary of Systems Procurement Background

This section has reviewed the concept of system and

some of the literature related to the acquisition of systems. Emphasis has been placed on describing the issues

related to system source selection which are raised in the

literature. The next section describes the history and
characteristics of the Parametric Factor Evaluation source

selection process.

g II . 4 Parametric Factor Evaluation

II . 4 . 1 History

The Parametric Factor Evaluation (PFE) process for

source selection was developed in the mid~1950s during the

early days of the weapons system management concept (Livingston, 1959; Thompson, 1976a, pp. 169-170). It was used

by the Electronic Reconnaissance Section, Aerial Reconnaissance Laboratory, at Wright Field in the procurement of

two electronic reconnaissance subsystems and one weather

reconnaissance subsystem (Thompson, 1976a, p. 170). The

process was developed by Charles Thompson to meet several

specific problems in trying to procure these subsystems.

First, the requirements (or specifications) were

constantly changing because of the need to respond

to new developments in radar (and other emitters).

Second, few laboratory personnel had any significant prior experience in designing or procuring

systems. And, finally, equipment suppliers had

little relevant experience (except, in some cases,

as subcontractors to systems contractors) (Thompson,

1976a, p. 170).

g The reader who is interested in a more detailed description of the Parametric Factor Evaluation process will

wish to read Thompson’s (1976a) description which is reproduced in the appendix. The article was drawn upon for this

section.
The existing published information on the process is Gontained in three documents (Thompson, 1976, 1976a; U. S.

Air Force , 1964) .

II. 4. 2 Claims

Thompson has made claims for the circumstances under

which PFE will and will not be useful and for what it will

accomplish. He has indicated that the process is applicabl

to large and complex systems (hardware or software) where

the follov/ing criteria are met:

a) that there is a degree of uncertainty or

complexity in the item to be procured which

makes it difficult to describe the item

to bidders and to identify a suitable basis

for evaluating bidders’ proposals, using

available procurement methods;

b) that there are limits on time, money, or

personnel which preclude considering other

alternatives; and

c) that there is some basis in prior experience for believing that it is critical

to both the procuring agency and the (successful) bidder to develop a reasonably

sound initial mutual understanding in order

to ensure a progressive and successful

mutual accomplishment of the object of

the procurement (Thompson, 1976a, p. 179).

He has indicated that the process would probably be inappli

cable to

procurements in which there is a single (or

small number of) critical basis(es) for evaluation, e.g., procurements for well-defined products or services where the basis is price,

and research procurements where the basis is

a particular technical approach or capability

(p. 178).
He claims that in general

there would appear to be no need for this procurement process where the prior experience

of the procuring agency (and the set of prospective suppliers) has provided a basis for

successful procurement and especially where

there are comparable previous procurements

(p. 178).

In addition he lists other procurement situations in which

alternat ive methods are being used. These include:

a) where the procuring agency has the time

and resources to carry out the preliminary

design phase (or even further stages) and

prepare detailed specifications;

b) where the procuring agency has the money

(and time) to let parallel (competitive)

contracts for the preliminary design (or

concept) phase;

c

)

where the procuring agency has the time

(and money) to let a "sole source" contract

for the preliminary design phase, with

the flexibility to drop the program or

to start over if the first effort was unsuccessful ;

d) where one or more prospective suppliers

are willing and able to carry out the preliminary design phase (or even further

phases) prior to action by by procuring

agency;

e) where the item to be procured can be "decomposed" from technical and scheduling interfaces through the use of standards, and

the like, or where there are acceptable

alternatives available (Thompson, 1976a,

p. 178). •

II. 4. 3 Characteristics

This sub-section describes, in summary form, the activ

ities required to use PFE and the characteristics which
make it similar or dissimilar to other source selection

processes

The activities required to use PFE are listed below.

They are approximately in the order in which they occur

in time.

1) Assess whether PFE is appropriate for the

' proposed procurement. Is it appropriate

in terms of the characteristics of the

thing to be procured, the procuring agency,

and the set of potential bidders?

2) Review what is known about the procurement

to identify the conditions which help to

define what it is you are going to procure.

3) Refine this information into a series of

statements describing the characteristics

of a prospective bidder/contractor which

you believe critical to the program’s success. This should result in 20-70 individual statements, called sub-factors,

written to minimize gaps and overlaps.

Additional' specific illustrative questions

can be added for clarity.

4) Group the sub-factors into 5-15 major areas

of concern called factors. These are administrative groupings selected to reflect

related problems and related interests

and skillsof people who will serve on

the evaluation panel.

5) Prepare the "Statement of Work" and

"Schedule" specifying the output of the

project according to the format provided

by the factor structure.

6) Prepare the "Scope of Effort" document

following the same format to provide a

framework for the bidder to present his

estimate of how he will allocate manpower

and dollars to various parts of the program.
49

7) Gather copies of supporting documents to

provide bidders with any information which

might be helpful in preparing proposals.

This step can occur at other stages, as

long as this material is available to bidders when the RFP is released.

8) Prepare the "Proposal Preparation Procedure" to explain the reason for using the

PFE process, the format for response, and

how the proposals will be evaluated.

9) Combine all of the documents with the formal

parts of the RFP to assemble the entire

package

.

10) Brief the bidders by making presentations

on the purposes of the project and use

of the PFE technique. Encourage questions

from those present. If PFE is a departure

from past practice, it is desirable to

hold two briefings. One to allow bidders

to decide whether or not to bid. The second

primarily to answer questions.

11) Prepare the "Proposal Evaluation Procedure"

to provide detailed guidelines for members

of the evaluation panel.

12) Select members of the evaluation panel

from those concerned directly with the

success of the program and from individuals

whose background and skills are necessary

for the evaluation of particular factors.

It is desirable for as many panel members

as possible to participate in developing

the list of sub-factors. Early selection

is recommended.

13) Panel members prepare for the evaluation

by familiarizing themselves with the entire

RFP including the supporting documentation.

Panel members are invited to participate

in briefing the prospective bidders.

14) Proposals are received and reviewed by

panel members for preparation of initial

ratings of bidders on the specific subfactors to which they are assigned.
15) Bidders make a presentation on their proposals to the panel. Panel members are

encouraged to question bidders. 9

16) Factor teams meet to arrive at a single

aggregate rating of each bidder. Teams

arrive at ratings for each sub-factor based

upon criteria of bidder’s comprehension

or understanding of the problem presented,

approach described, and present or potential capability to perform. Panel members

turn in signed rating forms before the

discussion begins. The panel arrives at

agreement in any manner it chooses, excluding voting. The panel rating forms are

signed by all team members, with strong

individual disagreements and specific comments for the negotiation process recorded

on the form. All meetings are tape recorded.

17) Prepare a summary of all factor ratings

for use in final review and negotiation.

Thompson (1976a) indicates the set of general and specific characteristics which when combined make the PFE process unique. The overall characteristics are:

1) The process is not applicable everywhere.

It is designed to meet a particular kind

of procurement problem.

2) The process is pragmatic in proposing that

uncertainty be faced and not avoided.

Bidders are told the degree of uncertainty

in the program personnel’s knowledge.

3) The process is open, explicit, and cooperative in exchanging information and preserving records.

9 ETIP has substituted a written question and response

mechanism for the oral presentation. Panel members who

require clarifications prior to their individual ratings

submit questions to the bidder through the contracting office.
The specific characteristics are:

1 ) The process neither assumes nor requires

any formal structure for the object of

the procurement. The sub-factors are

treated as separate "probes,” obtaining

measures along a set of dimensions which

describe the program.

2) The process uses a single framework for

all of its documents, i.e., the factor

structure

3) No limitation is placed on length.

4) The evaluation team is formed from among

those who have a prior interest in the

subject of the procurement or who have

special competence to evaluate particular

sub-factors. Participation may be limited

to individual sub-factors.

5) There are not restrictions on information

which the evaluation panel may use.

6) Members of the evaluation team must rate

each bidder affirmatively, i.e., have a

basis for both low and high ratings. It

cannot assign a "zero" but must obtain

required information through a specific

question.

7) The process of aggregating ratings is sequential with a summary hard-copy record

retained at each stage. Ratings are aggregated from individual team member ratings

of sub-factor to team ratings of sub-factors

and from team ratings of sub-factors to

team ratings of factors. The panel may

use any process for doing this, excluding

voting. Factor scores are then aggregated

to an overall score using weighted sums.

The sub-factor rating approach is contrasted with other

approaches. One approach is that which is used in the case

where there may be a comparably high degree of uncertainty

on the part of the procuring agency, but it may be able
to limit the evaluation to a "single” or small set of techni

cal problems, together with an evaluation of the capability

of the individuals who work on the project. This may often

be the case in the procurement of research (Thompson, 1976a)

The other extreme is where uncertainty is low and the evalua

tion may be based upon a single question. An example of

this situation is production procurements to detailed specifications which are evaluated on the question , of price,

together with proof of the existence of the required production facilities (Thompson, 1976a).

II . 4 . 4 ' Summary

This section has reviewed the PFE process in terms

of its history, claims for where it is and is not useful,

and characteristics for using it. This is intended as a brief summary of more comprehensive descriptions which are

reproduced in the appendix.

II . 5 Summary

The purpose of this chapter was to provide background

on the general problem of procurement of evaluation systems

and on the specific method which ETIP chose to procure its

evaluation systems. The intention was to give the reader

a basis for understanding the utility of this research in

terms of problems that exist and in terms other research

which has come before. It was also intended to provide

sufficient information on the Parametric Factor Evaluation
process to enable the reader

presented in the case study.

As review, the starting

that

:

to understand the information

point for this research is

0 the evaluation community is

recognize the importance of

tion process;

beginning to

the source seleco the systems procurement community has done

much writing on problems with the source

selection process and of guidelines for solving it. A number of case studies exist

but they are of a summary nature.

o The writing on PFE is limited to articles

by Thompson. No evaluations of the use of

the process exist.

The next chapter describes the objectives of the author

in furthering this existing state of knowledge and provides

a description of the methods used.
54

CHAPTER III

OBJECTIVES OF THE RESEARCH

111.1 Introduction

This chapter is intended to be a guide to the central

purposes of the research. It describes what the author

set out to accomplish and the approach taken in meeting

these objectives. A discussion of the proposed value of

this research and a disclosure of the biases of the author

which might affect the outcome are also given.

111. 2 Purpose of this Research

The primary purpose of this research is to provide

a detailed case study of an experiment in the selection

of contractors to design and develop systems. Specifically,

it is an evaluation of the use of the Parametric Factor

Evaluation approach by the Experimental Technology Incentives Program to select contractors for the acquisition

of two evaluation systems. These source selections occurred

during the 1976 fiscal year.

The case study is based upon;

o background on the PFE process from both historical

documents and personal communications with its primary developer;
o participation in the process at ETIP by the author

and direct access to records of the source selection

of the two contractors; and

o interview and quest ionnaire information from other

participants in the source selection.

The author’s intention is to provide a full description,

with data from the process, of ETIP’s use of the PFE process

and to evaluate how successful the process was in solving

some of the important problems which existed. It is hoped

that this will help those with similar problems to assess

whether the technique is applicable to their own situation.

The selective sampling of the literature on source

selection and procurement, partially reviewed in the last

chapter, revealed an absence of detailed case studies.

The closest similar case material found is in the form of

summary recollections of the author without specific data

from the process (Carnes, 1976). Most of the literature

appears to be in the form of prescriptive guidelines following discussions of problems with the source selection process or problems in the work of the resulting contractors.

That literature which is centered on ”how to” often provides

only the most general guidance for the potential user.

There is little offered in the form of detailed procedures

to relate these guidelines to the complexity of an actual

procurement

.

This research is, in part, an attempt to begin to fill

in some of the gaps in the source selection literature.
It is intended to be a relatively full, self-conscious recon

struction of the PFE process as it was used by ETIP, presented together with the actual data from the evaluation.

This is follov/ed by an evaluation of how well the process

controlled for some of the key problems which ETIP faced

in the procurement.

III. 3 Format Followed in the Case Study

The case study is accomplished largely in the next

two chapters, with the final chapter used to discuss the

relationship of the process to issues of broader significance than the ETIP procurements.

Chapter IV is a relatively full description of ETIP’s

application of the PFE technique to the selection of contractors for the acquisition of the two evaluation systems.

Information is presented on background leading up to the

writing of the RFP, the roles, extent, and timing of involve

ment of various people in the process, and issues which

arose at various stages of the process. Both anecdotes

and samples of data from the proposal evaluation process

are given in the text. All of the initial and final ratings

from the evaluation are provided in the appendix.

The process followed in writing Chapter IV was to draw

upon various sources of information, including:
57

o the author’s recollections;

o the formal records of the process (including

tapes of the evaluation panel meetings and

transcripts of the bidders’ briefings);

o notes from the files of the Contracting Officer’s Technical Representative (COTR) and the

author; and

o interviews with the COTR and with the author

of the RFP;

to write a draft chronology of the process. This chronology

was then given to the COTR and to the author of the RFP

to validate the statements made, to obtain additional cross

.validation of events which occurred and issues which arose,

< and to obtain additional information to fill in holes in

the chronology. Later interviews with other participants

were also used to obtain additional information on undocumented events. The emphasis in the chronology is to describe

the process in a manner which would readily answer some

of the basic questions which a practitioner might have.

These would include:

o What was ETIP trying to do?

o What did ETIP actually do?

Chapter V is outcome-oriented. It presents an evaluation of how well the PFE process worked in solving many

of the critical issues faced by ETIP which were raised by

participants or are raised by the literature on source selection. Actual data from the process is used in assessing

PFE’s performance against each of the issues evaluated.
Where external standards exist in the literature or were

raised by participants, they are compared to ETIP’s experience. For issues where the author did not consider himself

an acceptable source of information, either through lack

of access to the required information or through his bias,

additional information was solicited from other participants

through questionnaires and interviews.

The chapter presents a list of issues which were considered for the evaluation. A sub-set of these issues was

selected for thorough consideration. The choice was based

on judgments of both the importance of the issue and the

availability of information for evaluation of the issue.

Next, the set of parameters, the factors which may affect

the interpretation of the results, is presented. The potential impact of each one is described. Each selected

issue is then evaluated. The issue is described, a conclusion about ETIP’s experience in using PFE to deal with the

issue is reached, and the data supporting this conclusion

are presented. After all of the issues are evaluated individually, the results of using the process are summarized

in terms of successes, failures, and issues where the results are uncertain.

The issues in Chapter V against which PFE is evaluated

are those which relate solely to the source- selection part

of the system acquisition process and not to the outcome
of the projects for which contractors were selected. An

evaluation of how accurately PFE predicted the performance

of the successful bidders would have been a useful addition

to this research. Unfortunately, time series evaluations,

by the government, of the contractor’s performance, were

not collected and the author decided not to attempt to do

a post hoc reconstruction of this. However, there are some

comments in Chapter VI related to subsequent performance.

Chapter VI is a presentation of implications of ETIP’s

use of PFE and the results of this case study in terms of

subsequent events, potential further applications of PFE,

and potential future research. Subsequent events include

a brief review of results of the two evaluation system acqu

sitions and a brief description of ETIP’s later uses of

PFE for other source selections. Potential further applica

tions of PFE for both source selection and other problems

are speculated upon. Finally, some suggestions are made

for possible related research on topics beyond the scope

of this case study.

III. 4 Value of this Research

This project was undertaken at the request of the former Director of ETIP. After the two contracts for the eval

nation systems had been signed, he asked the COTR to evaluate how well the process worked. At that time the author,

an employee of the COTR, was trying to find a suitable dis-
60

sertation topic. An evaluation of the PFE technique seemed

as though it might meet the requirements for a dissertation and at the same time satisfy ETIP's desire to evaluate

how well the process worked. ETIP has recently been going

through a reorientation and the acquisition of evaluation

systems will probably not be an objective of the new organization. It is therefore useful to explore the value of

this project for both the researcher and for those interested in acquiring evaluation systems. A claim and counterclaim will be used to try to illustrate the value for both

audiences

III. 4.1 Research Value

The claim for researchers was made by Donald Campbell

and Julian Stanley in their classic survey of designs for

research (Campbell & Stanley, 1966). In their discussion

of the one-shot case study, the method used in this research,

they state that:

such studies have such a total absence of control as

to be of almost no scientific value (Campbell & Stanley,

1966 , p . 6 )

They further state that:

It seems well-nigh unethical at the present time to

allow, as theses or dissertations in education,

case studies of this nature (i.e., involving a single group observed at one time only )( Campbell

& Stanley, 1966, p. 7).

Finally they propose that:
The many uncontrolled sources of difference between a present case study and potential future

ones which might be compared with it are so numerous as to make justification in terms of providing

a bench mark for future studies also hopeless

(Campbell & Stanley, 1966, p. 7).

The counter-claim for this research is that it will lead

to judgments about the effectiveness of the technique being

tested and that it will serve as a bench mark for those

interested in improving upon and evaluating PFE. It will

also set a bench mark for those interested in evaluation

of source selection techniques.

This research is the first evaluation of an application

of the PFE technique. Though PFE was developed and used

several times by the Air Force in the 1950’s, there is no

published evaluation of its effectiveness. Also this was

the first application of the technique in a very different

setting. In addition, to the author’s knowledge, no careful, well defined, evaluation of any system source selection

technique exists. The existing literature on source selection describes many of the issues which are important; it

examines problems which have occurred in past procurement.

The literature serves as a departure point to enable the

author to specify these issues and use them to structure

an evaluation for system source selection techniques. This

case study evaluation sets up a standard for comparative

evaluations of other techniques. .Both the issues selected

and the method for evaluation may be improved upon.
When little or no information exists on a topic as

broad as the one considered here, the traditional ’’controls"

applied to a priori proposition testing are inapplicable.

A good description and understanding of how the process

works are important first steps for later more controlled

research. The investigative method used in this research

is felt to provide^ a scientific basis and point of comparison for future research on source selection procedures.

III. 4. 2 Administrative Value

The claim for those interested in acquiring evaluation

systems was made by one of the bidders from ETIP’s source

selection during an interview for this evaluation. His

claim was that, as a taxpayer, this research is "far too

expensive" and the results "too insignificant" to justify

it. The reasons he gave for his claim were related to his

beliefs about how "policy change" gets made in Washington.

He indicated that he felt that in an emergency, such as

a war, people would move things rapidly in the direction

of improving source selection. He indicated that whatever

the results of this study, a thoughtful, reflective committee behind a powerful mover could have implemented it at

considerably less cost and in greatly reduced time.

The counter-claim comes from the apparent differences

between the author’s beliefs and the bidder’s beliefs about

how change is caused in government. The author believes
that high level directives alone will not lead to meaningful

change. It is primarily the bureaucrats, those people who

survive changes in administrations and who are responsible

for translating vague policy directives into specific procedures, who are the real movers in government. The author

has experienced and observed how directives from the top

get permanently delayed in red tape because those responsible for implementing the directives oppose them. This

is not to say that support from the top is unimportant;

however, the author believes that the key to change, at

least in civilian agencies, is the user, the person who

has to live with the change. Change occurs when a person

has a problem, pressure to solve it, and a solution in which

he has confidence.

The literatures on evaluation, systems and source selec

tion clearly indicate that there are problems. These problems become specific to users, i.e., government project

managers, contracting officers, and other government technical personnel when they are trying to acquire specific systems . The pressure to solve these problems is not quite as

obvious, particularly where evaluations are concerned.

There is unlikely to be an emergency which would impact

this type of work. The most direct pressure seems to have

come from Congress, the Office of Management and Budget,
and the intended users of the systems. In some cases it

comes from a high level administrator and in other instances

from the independent desire of the users of source selection

methods to do better.

The intent and thus value of this research is to increase awareness and, through a credible evaluation, modify

confidence in a potential solution to source selection problems. Increased awareness is provided by the description

of the process, its history, and the types of problems for

which it might be applicable (Section II. 4), and the full

description of how it was implemented at ETIP (Chapter IV).

The evaluation of how well ETIP was able to deal with the

key issues of source selection by using the PEE process

(Chapter V) is intended to enable a user to judge how confident he is that the process will be useful for solving his

own source selection problems.

The author agrees with the bidder that this research

alone will probably not convince anyone. The type of confidence required to use a technique such as this comes f<rom

either or both trust in the people advocating the approach

(ETIP’s reason) or the ability to verify independently for

oneself that it solves the problems one has. The goal here

is to provide the ability to verify whether the process

works. Clearly, it is not a totally independent verification since the reader is dependent on the author’s reconstruction and interpretation. However, a great deal of
raw data from the process is provided to enable the reader

to reach his own conclusions on where the process did and

did not work. This research is offered as a departure point

for those who may consider using or improving upon the PFE

process

.

III. 5 Biases of the Author

When examining the case study, the reader may find

it helpful to keep the biases of the author in mind. These

certainly have had some effect on the presentation of the

problems, the emphasds in the description of events, and

the evaluation of results. This section is presented as

an attempt to disclose the author's self-conscious biases.

The author is not an expert in procurement, though

he has done some examination of the literature and talked

with those involved in the process (i.e., contracting officers, project managers, and contractors). Procurement

is not an area in which he claims special expertise. He

has no background in contract law, and only a modest background in procurement policies and the practice of procurement . The author does claim special knowledge in what occurred in the specific procurements under examination here.

Working on these procurements was the author's first assignment when he arrived at ETIP in March 1976. He was involved
in recruiting members for the evaluation panel, scheduling

of meetings, evaluating proposals, and assisting in the

oral and written clarification process. He has participated

in the use of the PFE process twice subsequently and had

an active role in writing portions of the RFPs. He does

not have personal experience with any other procurement

techniques, though he has discussed with others their experiences.

Finally, and possibly most significantly, the PFE process was primarily developed by the author's dissertation

advisor. Aside from the obvious potential bias caused by

wanting to pass, there is the additional factor that the

author’s advisor has greatly shaped his values about what

the important problems of contracting are and how to solve

these problems. This has been a non-trivial influence.

This disclosure is intended as a caveat to the reader.

The author has attempted to compensate for these biases

by several methods. Raw data from the evaluation process

are provided in both Chapter IV and in the Appendix. The

reader is encouraged to examine it and to do his own indepen

dent analyses. Extensive use is made of reference notes

in Chapter IV to indicate sources other than the author’s

own notes and recollections. Where available, cross references are used. Additional data collected from panel

members and bidders is used in Chapter V in the evaluation

of some of the. issues. Where the author felt that he might
67

not be a credible source of evaluation information, he sought

additional data through questionnaires given to other participants in the process.

III. 6 Summary

This chapter has presented a guide to the objectives

of this dissertation. It has described what the objectives

are and the process being followed to achieve them. The

claimed value of this research has been presented and the

biases of the author, which are parameters to interpretation

of results, have been disclosed.

Chapter IV is intended to describe how ETIP used PFE

to try to solye its procurement problem. It presents much

of the factual context for the eyaluation portion of the

case study which follows in Chapter V. If the reader has

not done so already, it might be useful before proceeding,

to glance at the sections of the appendix which contains

copies of the RFP and of Thompson’s (1976a) paper describing

the PFE process. This will provide a useful context for

the Chronology.
68

CHAPTER IV

CHRONOLOGY

IV, 1 Introduction

This chapter describes how Parametric Factor Evaluation

(PFE) was actually used at ETIP in 1975 - 1976 to procure

the two evaluation systems contracts. The description is

based on:

o background on the PFE process available in

the form of historical documents and descriptions by the primary developer;

o the author’s observations during his participation in ETIP’s use of PFE and access to

both the formal records of the process and

the informal notes of some key ETIP participants; and

0 interviews with some key participants, including bidders and members of the evaluation

panel

The author’s involvement in the procurement began two

weeks prior to the second bidder’s briefing (March 8, 1976),

during the selection of the proposal evaluation panel.

Sources for information on events prior to this date are

the records and recollections of others. The author did

not decide to use this problem as the topic for a dissertation until about two months after the contracts were signed.

Therefore, no special effort had been made to set up a contemporaneous data collection scheme. Fortunately, the Con-
tracting Officer’s Technical Representative (COTR) kept

extensive files including his notes of meetings and phone

conversations, and copies of drafts of various documents.

These were used together with interviews and the author’s

own recollections in order to reconstruct the process.

An attempt is made to note each source used in this reconstruction. Where the information was furnished recently

through the recollection of a participant, as opposed to

from the records, the reader will note a 1979 or 1980 research note. Any statements which are made without citations are the recollections of the author.

The organization of this chapter is, for the most part,

chronological. It traces the procurements from the decision

to contract for evaluations through to the signing of contracts and the debriefing of one of the unsuccessful bidders. For each section of the chapter information is provided on the decisions which were being made, the people

involved, and the actual time frames in which events occurred. Issues which were raised during the process by partici

pants, actions which were taken, and problems which were

encountered are presented. The emphasis is necessarily

the author’s. Some details of the NBS project plan approval

process which did not seem central to t.he problem have been

omitted

.
Several figures have been included to illustrate how

the evaluation teams actually displayed the ratings of bidders during their meetings. A complete set of ratings from

the process are provided in the appendix.

Figure IV. 1 is provided as a reference point. It illus

trates the relationship in time among the various overlapping activities which occurred over the course of the procurement. It also identifies the people or groups of people

who were involved in these different activities. The figure

has been reduced to one page to provide the total time perspective at one glance. A reproduction on a more legible

scale is provided on two pages in the appendix.

This chapter does not contain a step by step guide

on how to accomplish source selection by using Parametric

Factor Evaluation. Nor does it contain explicit descriptions of the various documents which are included in an

RFP using PFE. Such descriptions have been presented by

Thompson (1976a) in a paper which is reproduced in the appen

dix. A summary of the process appears in section II. 4 of

this document.

IV . 2 Decision to Contract for Evaluation Systems

The decision to contract for and develop evaluation

systems was somewhat diffuse. It occurred over a period

of about a year and a half and involved the input and deci-
71

Omtu

V)

Vi 9 T U u ec e

Vi*i > X c > —

to »• *; t.

*^<0 <aa > s

au. a 3

*» T. n

O *i» O — U ^ sue U I S 3 — rs I

W — u W ^

O c sa V —

«r I c c

u * z —

= '.l Ci n a (Q n n It U U‘ u s

U W —

C C C (- o u o tu

*» 0*^0 0^0

s u c •

u — u O — o X O It s

3 z s — s

O — » X — X > o

to u: » u

> • V s

i. w

o s u: •ox — S tt - I C H> M e I t- — r fJ c

tu c s

; a « <•- < rFSn-

» O s «.> <

; £ z 9) • n o —I * Jf H

f} « UJ I s w s u*' c z — <r«— X — X C

9 eo

E Z

> O —

J ^ H H O

X • e

e L. o — V a* «r» 3 E u a

«»— E V U (» o

c a (- £ — « o H

o

V u bc •

&' a c s

E —

C *•> — — o u

(t bO It — «• fc. X .> U ( — CO;

c r o *>- A«

U ^

It c a o

V o

O 'O Ea X

u o O f>

f} z S U

mm OXJ

t- <5 «S

03 U

to X c > X o —

O H» c <

o ^ s

E U H

u r to O 3 IU. (Q z

e — — os X 4^ a

Z U <C

l: o s c S

u o c <

u 4^ s

< (. C (3

<c a s

S s ck k s u

s u X s X n sx —

<e U3 Cl u>

T3»- O

H» —

O W

3u: ^ CSC

— r

- 19 b' w ^ c o — —

tr c one u w a — k X

16 Z -> C>X — O

<*» C* < C 13 o z c z

tr M 1,0 b M

C ?• CO — X

e au

9) e Ti b wu: a c c s X o r -o

19 h» b « V

c»: —

« o <0 — X

to — o s o

CO > s s b s^u ss

I C S M4 ^ I > S <0 0} H

I — O b < o

mi R C X ^

. O C 7 ^

I C R S *0 < C

I X(T»«*—> —

I R 0> H X U Q.

•>•0 C 44 o

b C o C3 o V e c z —

cj a a<^ >

It c c R > e — 4IOO-J —

b T* b Q

C C S3- b vs c z c CCD X

C O — b

o *' o a a

4-> If Z (t

IT b — *0 b C C' t.^ •j — cj z s

c *a — tr— f— c — o c k- o

c A n c b' u

o z

— «

£ s X c a c

« • —

r 6

c o c

a X X CH R

44 S

— K

3 O

C 3 • — It b

> O

R (t C —

CO O O r ec — R C

H- C C

—

b o

S CJ

— US

«•c oo —

e rt s

c. xs

E <. C O

C X to

b

o & s c

; m < l:

r c*. u: s

- C' ts e o < — c It - s — b I

» o IT c e

> w c o

•’ — o

It <

C — S S I C IT h- C I SI > O S - SU b' s V

' C S Z «0 b o

O.— N ts k- *- I- s-< cs. u O

o s < u u

) c -a .

. b c a L

I a— — —

s — s

03 ss I 0} I

R 4' _

C b c u so coo Z I

^ R O X

(li V 44 Z

r s to

> s o: C C It s

C C *4 44 n c X R s

a«.- — O

b <1^ £ s

o — X — >* X b a

CO30

^ a — «» 44

I 9 39, 44 4 O Kmm O i

: c H- lb I

: b C03 b I O *i R • e

' o c e o I R O O U I

I C — R 4

> o 44 a< I

^ a R e

R Z O C 4 C CX C ' U V H —

6 44 < C E • CO : C O b —

44 V O 44

44 VX O — b^ V

b a c> Z ^ R Z

^ZO Om O (9 S

b 03 K

Xi-4 X

A

A I

JUL

SEP

OCT

HOV

DSC

JAN

FEB

HAH

APR

MAI

JUN

JUL

AUO

197 N

1975

’976

Figure

IV.

1

The

Activities

in

ETIP’s

Source

Selection
72

sions of several people. It originated in 1974 out of a

desire by the director of ETIP for more explicit integration

of evaluations into the programs' activities (ETIP Director,

Note 1 )

The major part of the program effort to develop an

evaluation capability began in early September of 1975 with

the arrival at ETIP of the person who became the Contracting

Officer's Technical Representative for the two evaluation

systems contracts. The COTR was brought into ETIP on an

Intergovernmental Personnel Act appointment. He was to

provide evaluations of the program's experiments which woul'd

produce information for use in an overall ETIP program review scheduled for 1980 (COTR, Note 2).

The Director of ETIP decided that money would be obligated to contract for evaluation services (COTR, Note 3).

Since the COTR had no staff, there was never any serious

consideration of doing the work in-house (COTR, Note 4).

It was evident to the COTR that if the program were

to continue to exist beyond its 1980 program review it would

need some type of on-going capability to evaluate its future

experiments, as well as its past and current experiments.

He wished to develop an evaluation capability based on general evaluation designs which would build upon experience

1 This person will be referred to as the COTR throughout

this document.
and not require having to start over by developing designs,

data collection plans, and data analysis plans for each

new experiment. He also wanted those capabilities to be

developed within the government as opposed to having to

depend largely on one or more contractors over an extended

period of time for evaluation expertise. In order to do

this he realized that he needed contractors who were capable

of working with ETIP over a period of time to design, develop and transfer such an evaluation capability (COTR,

Note 5 , Note 6 )

.

The use of the term system and the incorporation of

other system concepts did not become part of the projects

until work on preparing the RFP began (COTR, Note 7). In

this earlier period of the development of the project the

objectives were described in terms of developing designs,

capabilities, and doing specific experiments (COTR, Note

8). These were all incorporated into what were later described as the systems.

Records indicate that the COTR started writing a plan

for a project to evaluate the commercial impacts of ETIP’s

procurement policy experiments as soon as he arrived at

ETIP (COTR, Note 9). The administration of NBS required

ETIP to prepare and submit, for its review and approval,

a plan for each project which ’would result in the obligation

of more than $10,000. At this time in the history of the

program the approval process for ETIP project plans was
quite extensive. Four people, ending with the Acting Director of NBS, were required to sign off on the plans and sever

al others reviewed them before they were submitted to those

who were required to sign.

The COTR used the requirement to prepare the project

plan as a method for familiarizing himself with the problem

area with which he would be dealing in his new job. He

spent several months talking with other staff at both ETIP

2 and at the Program Administrative Agencies (PAAs) and reading existing program publications in order to determine

potential topics and methods for evaluation. When he completed the first draft of his plan he circulated it to the

ETIP staff and used their comments to revise it into a format which the ETIP reviewers thought would be more acceptable for the NBS review process. The completed commercial

impact evaluation plan was then used as the basis for writing a plan to evaluate the agency impacts of the same experi

ments (COTR, Note 10). This entire process of writing,

review, and approval lasted until March, 1976 (NBS Acting

Director, Note 11).

2 Program Administrative Agencies (PAAs) is the term

ETIP used to refer to the organizations having responsibility and authority for decisions to introduce experimental

interventions. In this case the Federal Supply Service

(FSS), the Veterans’ Administration, and state and local

procurement agencies (or their consortia).
An important uncertainty which existed was whether

the required capabilities existed in any contractor organiza

tions (Thompson, 1976a). No one in ETIP was aware of any

instances where the government had successfully procured

a similar evaluation capability. This was particularly

troublesome since contracts for single evaluations had such

a poor track record (see section II. 2) and few people in

the evaluation industry appeared to have the system design

and development experience which seemed necessary to provide

for the systems aspects of the project. In addition there

were problems of attracting bids by any firms which might

have the required capabilities and then evaluating the propo

sals to determine the degree to which the capability or

potential capability did, in fact, exist. The procurement

process would have to address these problems.

IV. 3 Preparing the Request for Proposals

The COTR wanted to develop a RFP in parallel with his

project plans. However, he realized that he would be unable

to do both effectively within a time frame he felt satisfied

with (COTR, Note 12). In early October he asked Charles

Thompson to write an RFP for a contract to evaluate the

commercial impacts of the ETIP experiments with state and

Thompson had been familiar with ETIP since the summer

of 197^. He had been associated with the program as a special employee reporting to the Director since September,

1975, while remaining at Northwestern University.
local procurement organizations (COTR, Note 13)* He wrote

to Thompson and sent him some background materials from

a prior unsuccessful attempt at awarding a similar contract

(COTR, Note 14). Thompson's background in both evaluation

and systems management was an indication to the COTR that

he might be able to provide the help which was needed (COTR

Note 15).

At this time the COTR had not thought through the number or structure of the contracts which would be awarded

when he received NBS authorization on the two project plans

he was writing. He knew that at least one RFP was needed

and that he could use that as a model for other RFPs (COTR,

Note 16 )

The COTR and Thompson first met to discuss the RFP

on October 20, 1975. They began talking about some of the

issues which would be important in the proposed procurement

Thompson felt it was important that the RFP be successful

in obtaining proposals and result in a contract award.

Specifically, he was concerned that go'od companies would

begin to turn down (not bid) or ignore ETIP work if the

procurement process ended up rejecting all bids (COTR, Note

17). This concern was particularly salient since the prior

RFP for evaluation of ETIP's procurement experiments had

been cancelled that same year after the proposals had been

evaluated.
other items discussed during the October 20 meeting

included

;

o how to provide for an information exchange

through the procurement process which would

be effective in satisfying the needs of both

prospective bidders and the proposal evaluation panel. That is, the needs of prospective bidders to determine how to match his

proposal with the requirements under the

contract and the needs of the evaluation

panel to be able to determine the bidder’s

understanding of the project and his capability to perform effectively;

o how to assess the importance of the project

to top management of a bidder’s firm in order

to be confident that the necessary resources

will be devoted throughout the contract;

o how to make ETIP’s priorities clear in the

RFP; and

o how to determine whether a bidder will be

able to meet the multiple objectives of various key people involved in the projects

being evaluated (COTR, Note 18).

Thompson decided to use the Parametric Factor Evaluation approach (PFE) when he recognized the apparent similarities between the procurement of an evaluation capability

which the COTR desired for ETIP and the procurement of the

military hardware sub-systems, for which the technique was

developed (Thompson, 1976a). He described, in a subsequent

paper, three similarities which occurred to him in making

this decision.

First, the evaluation requirements were not only

constantly changing because of the on-going progress in the initiation and carrying-out of procurement experiments but also included a wide
variety of objectives ranging from very specific

to very broad, and from intermediate to longrange. Second, ETIP had, with the exception of

the writer [Thompson], little or no experience

in systems procurement, and neither the time nor

personnel to accomplish a problem or concept definition design. Finally, it did not appear that

the "evaluation industry" had any significant

present capability to design or manage an "evaluation system" to meet ETIP requirements (Thompson,

1976a)

The situation was one in which high mutual uncertainty

existed between ETIP and its potential bidders. It was

unclear what the specific nature of the final evaluation

systems would be. Such basic issues as how many systems

there would be and who would own them were still unresolved.

It was also uncertain what specific steps would need to

be taken in the development process. The nature of the

situation seemed to suggest the need for a statement of

work which would allow for the uncertainty rather than one

which specified numerous details (COTR, Note 19).

For ETIP a flexible statement of work presented

risks that the contract would not result in the desired systems. Therefore, it was important to be

able to determine the degree to which it was within

each bidder's interest as well as their capability

to develop the systems. The RFP would have to provide the

structure for making these assessments.

Thompson’s role was to work with the COTR in developing

the concepts of what ETIP would be procuring and to write
the RFP based on this. It was Thompson who introduced the

concept of evaluation systems as the method for procuring

the on-going evaluation capability which the COTR desired

(Thompson, Note 20 ; & COTR, Note 21 ). Thompson’s role also

included participation in the evaluation and advising the

COTR on all other aspects of using the PFE process.

The COTR and Thompson met for a second time on November

3 and 4 to further discuss issues of concern related to

the RFP and the eventual contracts. These meetings served

a mutual education purpose. Thompson learned the COTRs’

and others’ desires and concerns which he used in writing

the sub-factors. He also worked to make the COTR aware

of some of the important aspects of system procurement (COTR,

Note 22 )

ft On November 5, Thompson sent a letter to the COTR together with some key documents from the prior Air Force

uses of the PFE process. The purpose of sending these documents was to begin to provide the COTR with perspective

and background on the process and to build his confidence

in the process by showing that it was not untested (Thompson, Note 23).

On November 17, Thompson met with the COTR and described a list of evaluation factors. There were two management

factors and five technical factors. The management factors

were
o management style of the contractor in relating to ETIP and

o internal management of the project.

The technical factors were:

o understanding and conceptualization of the

problem,

o collecting background on sites,

o research design,

o collecting data, and

o analyzing data (COTR, Note 24).

The management factors came from a scaling down of the larger number of management factors used by the Air Force (Thompson, Note 25). Thompson derived the technical factors based

on his knowledge of what ETIP was buying, his reading of

ETIP background documents, and his discussion with ETIP

staff (COTR, Note 26; Thompson, Note 27). This background

suggested issues for the sub-factors and, at a higher level

of aggregation, the factors. Determining the factors and

sub-factors was an iterative process rather than sequential.

Additional sub-factors were added after the factor groupings

were determined (Thompson, Note 28).

Following his discussions with the COTR, Thompson wrote

* a draft of the document entitled "Proposal Evaluation Factors" (Thompson, Note 29). This was the listing of all

of the factors and sub-factors which would be used as the

structure for both the preparation and evaluation of propo-
4 sals. The structure would also be used for organizing

the ’’Statement of Work,” ’’Schedule,” and ’’Scope of Effort”

documents which remained to be written. Thompson worked

on writing these and other documents which would be given

to the contract officer to be included in the solicitation

document. By February 11, he had completed drafts of the

following

o ’’Introductory Note,”

o ’’Definitions,”

o ’’Statement of Work,”

o ’’Statement of Work (with illustrative sub-items),”

o Schedule (overall),”

o ’’Schedule (Phase D,”

o ’’Proposal Preparation Procedure,”

0 ’’Proposal Evaluation Factors,”

0 ’’Proposal Evaluation Factors (with illustrative

sub-questions) ,

o ’’Scope of Effort,” and

o ”Factor/Sub-factor worksheet” (Thompson, Note 30).

4 There were seven factors and 33 sub-factors in the

final version.

5 The form and in some cases the content of these documents followed that of prior Air Force procurements using

PFE. Texts of the ’’Proposal Preparation Procedure” and

’’Scope of Effort” follow closely those used by the Air Force.

The final version of each of these documents may be found

in the appendix.
As drafts of documents and information about the PFE

process became available at ETIP, questions and concerns

about the procedure began to arise. An issue arose during

discussion among some of the ETIP staff which related to

the proposed ”non-blind" evaluation of proposals. Some

staff members felt strongly that the proposals should be

evaluated "in the blind." This was the technique which

ETIP had been using in its other procurements. That is,

proposals were submitted with no identification of the bidders. The evaluation panel based its ratings solely on

the proposed approach in the document without knowing who'

the bidders were. It was felt that this would produce a more objective evaluation and that the panel could not be

accused of being biased (COTR, Note 3 1 )

By contrast, PFE

requires explicit identification of the bidder and many

of the illustrative questions for the sub-factors inquired

into both prior and anticipated future work of the firm

and the staff which would be assigned to the proposed project. Proposal evaluation panel members having any informa

tion which would bias their ratings either for or against

a specific bidder are required to make this known to the

others during the factor team meetings. This issue was

resolved by the director of ETIP after Thompson argued that

it would be impossible to evaluate the factors without know

ing the identity of the bidders and that it would be impos-
83

sible to keep the identity of the bidders a secret even

if it was blind. The director agreed to a non-blind evaluation (Thompson, Note 32).

When strategy for the scope and number of evaluation

contracts was discussed in December, it was thought that

four contracts would be awarded (Figure IV. 2). There would

be one to develop a system to evaluate the commercial impacts of federal, state and local procurement experiments,

one each to develop systems to evaluate the agency impacts

of a) federal and b) state and local experiments and a fourth

to evaluate both the agency and commercial impacts of experiments with the Veterans Administration (COTR, Note 33; Note

34). The specific program decision to attempt to award

two contracts for the design and development of one or more

evaluation systems occured on January 16, 1976, when this

issue was discussed in a meeting among the Director of ETIP,

the COTR, and Thompson. The COTR was instructed by the

Director to award two contracts, one for each of the project

plans he was writing, and to do so by June 30 (the end of

the fiscal year). Thompson argued that ETIP would be unable

to handle the evaluation process for more than two contracts

(Thompson, Note 35). The Director felt that any more than

two contracts would be too much for the COTR to manage effectively.

A budget needed to be established for each contract

in order to indicate the size of the contracts to prospec-
84

AGENCY COHMERCIAL

IMPACT IMPACT

CONTRACT

A

CONTRACT

B

CONTRACT

C

CONTRACT

D

Federal Supply Service

n

m

State and Local

P

P

Veterans Administration

P

P

P

P

P

i

FIGURE IV.

INITIAL CONCEPT OF THE STRUCTURE OF THE

EVALUATION CONTRACTS
tive bidders. Due to uncertainties, anticipated and unanticipated, it was not feasible to determine what the eventual costs would be. The budgets needed to be attractive

to bidders, be responsive to the pressures to obligate program funds within the fiscal budget cycle, and allow the

government reasonable control over the development expenditures. If the budget were too small there was thought to

be a good chance that organizations would not invest in

responding. The program was under pressure from the Department to spend money so as not to have a budget carryover

at the end of the fiscal year (COTR, Note 36). Counterbalancing pressures existed from the Program Office at NBS

to control costs (COTR, Note 37). When the Director of

ETIP decided to have two contracts, he also decided that

they would be phased contracts and that the first two phases

of each would be funded at approximately one million dollars

(Thompson, Note 38). This presented projects of reasonable

size to attract bids, allowed the program to obligate its

budget, was thought to be an acceptable number of contracts

for the COTR to manage. The phasing of the contracts provided a formal checkpoint at the end of the first year to

determine whether the money was being spent effectively

(Thompson, Note 39).

With drafts of the technical aspects of the RFP nearing

completion and the assignment of a June 30 award deadline.
it was important to contact the Department of Commerce’s

contracting office in order to begin the solicitation process .

IV. 4 Involving the Contracting Office

The contracting office had not yet been involved in

any of the preparations for this procurement. It was now

important to determine whether the PFE approach appeared

to be acceptable in general and to begin working out the

specifics needed in the process (Thompson, Note 40).

Thompson had been having arguments with the director

of ETIP and another senior staff member over the statement

of work. The statement of work followed the structure of

the factors and was very broad. It indicated the categories

of work to be done but was not specific about interim deliverables. The senior staff member and the director wanted

it to be more detailed (Thompson, Note 41). The ETIP administrative officer set up a meeting with people in the contracting office to try to resolve whether the PFE process

and the flexible statement of work would be acceptable.

She arranged to meet with the contracting officer and contract negotiator who she knew wpuld be most receptive to

helping ETIP determine whether the process would be legal

and workable (ETIP Administrative Officer, Note 42).
The meeting was held on January 27 and was attended

by Thompson, the COTR, the ETIP administrative officer,

the contracting officer, and the contract negotiator. The

contracting officer was very receptive to the PFE approach.

He had worked with the Maritime Administration and saw the

approach as similar to those used to purchase ships (ETIP

Administrative Officer, Note 43). After it was agreed that

PFE process could be used, some of the specific steps were

discussed. Since the PFE process was a departure from most

other Department of Commerce procurement methods, it was

agreed to brief the bidders twice. This would give the

bidders an opportunity to familiarize themselves with the

requirements and to ask questions. It was felt that an

open exchange of information would enable bidders to see

that ETIP had no sole source position and would help bidders

to determine whether they wanted to respond (ETIP Administrative Officer, Note 44). Scheduling of the two briefings

and of the evaluation process were discussed and the contracting officer assigned the responsibility for the next

steps in the procurement to the contract negotiator (COTR,

Note 45). It was agreed that the final RFP would be issued

prior to the second bidder’s briefing.

Though the contracting officer agreed that the broad

statement of work was sufficient, the director of ETIP still

wanted additional detail specified. To meet this require-
88

merit, Thompson wrote a second statement of work which contained detailed illustrative sub-items (Thompson, Note 46).

This was included in the RFP , together with the actual statement of work.

The schedule which was worked out with the contracting

office indicated that it was necessary to begin briefing

prospective bidders in mid-February in order to be able

to meet the June 30, award deadline. The announcement of

the solicitation had to be made immediately.

IV. 5 Involving NBS Management

On January 29, the director of ETIP sent a memo to

the Acting Director of the National Bureau of Standards

indicating ETIP’s intention to award two parallel contracts

for approximately one million dollars each. With the memo

he sent a copy of the announcement of the procurement and

the schedule of briefings which were being sent to the Commerce Business Daily . He indicated that this announcement

did not obligate ETIP to make an award but that the importance of beginning to educate contractors about the procurement required publishing the notice before NBS approval

of the project plans. He promised that the plans would

be circulated for approval prior to the pre-bidders briefing

of February 26th (ETIP Director, Note 47).
Issuing an announcement of intention to let contracts

prior to NBS review of project plans was counter to normal

procedure. However, this was thought to be necessary in

order to achieve the June 30 award target. It is likely

that this action combined with the size of the proposed

contracts (large in comparison to most other NBS contracts)

created a great deal of anxiety among the NBS administration. The program office wanted assurance that the output

of the project would support commitments which NBS had made

for future evaluations of the ETIP program and that no implicit future funding commitments were being made (COTR,

Note 48, Note 49, Note 50). The COTR and others at ETIP

met with the program office staff throughout the project

plan review process to brief them and answer their questions

about the projects. On March 31, 1976 the Acting Director

of NBS approved the project plans (NBS Acting Director,

Note 51). This cleared the way for the award of contracts

following completion of the evaluation of proposals.

IV. 6 Involving the Program Administrative Agencies

It was thought to be important to use the proposal

evaluation process as a method for involving and educating

people who would be involved in the project. Participation

would engage people in a dialogue about what ETIP was trying
90

to do with the project and some of the problems involved

in doing it. This was particularly important to staff of

the PAAs.

By February 11, drafts of most of the technioal documents for the RFP were completed and ready for review by

the staffs of ETIP and the Program Administrative Agencies.

Since people in the PAAs would be working with the contractors, it was important that they be involved in the solicitation process. Normally, as part of the PFS process, their

aid would have been requested in determining the sub-factors

and factors. However the time pressures of this procurement

prevented this level of involvement (COTR, Mote 52). All

that could be accomplished now would be to obtain their

review of the various draft documents to insert anything

which may have been overlooked.

The only Program Administrative Agency to be involved

actively in the review process was the Federal Supply Service

(FSS). The National Institute for Government Procurement

and the National Association of State Purchasing Officials,

which coordinated ETIP’s local and state purchasing experiments, could not be involved in the review of the drafts

since people who were not federal government employees were

not permitted to see the RFP prior to its release (COTR,

Note 53 )• ETIP's work with the Veterans’ Administration

was in a state of flux. The primary contact point had recently changed and there was some disagreement and
confusion regarding the role that the contracts would play

in this work. The VA was represented in the initial review

of the RFP but did not participate in the eventual evaluation of proposals.

The primary ETIP contact at FSS had recently assumed

his position there as Director of their Experimental Technology Division after having been chief of ETIP's procurement

program. He was familiar with ETIP’s plans for evaluation

and was quite concerned about the large size of the effort

and prospect of the resulting contractors working in his

agency (COTR, Note 54). When the RFP documents were sent

to FSS for review he and his staff made extensive comments

on both the process and the specific sub-factors. Their

concerns included the following;

o that the amount of control and responsibility

given to the contractor seemed excessive.

They felt that many of the sub-factor issues

which the contractors were being asked to

respond to were ETIP’s responsibility, e.g.,

rationalizing multiple and conflicting objectives of ETIP, telling them what information

ETIP will need to review the project,

o the work to be performed overlapped with

existing projects,

o the amount of money being allocated was "absurd"

for the work they felt needed to be done,

o much of the information provided to bidders

was unnecessary, particularly that information which gave the history of the PFE process

and the details of how the proposals would

be evaluated,

that the organization of the RFP was bad

and some paragraphs were confusing.

o
0 that it was unclear why two statements of

work were included (one with illustrative

sub-items)

,

o that it would take the panel two to three

months to evaluate the proposals which would

be received in response to the RFP,

o many of the sub-factors were not useful,

especially those on management and reporting, and

o that the RFP would scare off good bidders

who would find it confusing,

o that successful contractors would provide

material to the PAAs which would be unusable

(FSS representatives. Note 55).

These issues continued to be a source of concern to some

FSS staff throughout the solicitation process (FSS represen

tative , Note 56 )

IV.

Involving the Prospective Bidders

One objective of the PFE process is to provide prospec

tive bidders with as much information, openly and explicitly, as is possible (Thompson, 1976a). One method used

for doing this was through the use of two bidders’ briefings. The first was to give bidders an overview of ETIP,

a presentation of the objectives of the procurement, and

to allow for extensive questions and answers. The second

was to allow for further questionning after bidders had

an opportunity to review both oral and written materials

(Thompson, 1976).
On February 17th, the same day that the COTRs ’ two

project plans were sent to the NBS program office for approv

al, notice of the solicitation appeared in the Commerce

Business Daily (CBD) . The announcement described the general subject nature, level of effort, and time frame of the

intended contracts and announced the schedule and procedure

for distribution of preliminary documents, briefings, and

responses (’’Design and develop evaluation systems for experi

ments...,” 1976 & Figure IV. 3). Individuals who wrote or

stopped by the contracting office and those who were on

the bidder’s list^ were furnished copies of the preliminary

7 RFP prior to the pre-bidders briefing. The CBD announcement was the first notification of the RFP to be issued

to the contracting community. The only other prior related

information was from conversations which the COTR (and probably others) had with contractor organizations to describe

ETIP, its work and the general direction being taken in

the evaluation of ETIP experiments. By doing this the COTR

hoped to create an awareness of the program’s activities

so that when CBD announcements were published people would

take the time to consider whether to bid (COTR, Note 57).

£

°The bidders list was composed of names of individuals

and firms known to ETIP and thought to have a potential

interest in learning about the project.

7 The final version of the RFP was to be issued prior to

the second briefing.
H—DESIGN AND DEVELOP EVALUATION SYSTEM FOR PROCUREMENT

EXPERIMENTS on behalf of ETIP (Experimental Technology Incen

fives Program). Obtain evaluation system to measure the

agency impact and commercial impact of a series of procurement experiments being conducted in cooperation with specifi

government purchasing agencies. Two contracts, one for

the agency impact and one for the commercial impact, each

with a level of effort of approx. 100 man-months are to

be awarded. Each contract will be for a three year period,

performance in three phases. Funding for the first two

phases of each contrac't is presently available. Because

of the complexity of the program, and the importance of

selecting contractors qualified to provide comprehensive

program management to design, develop and test an evaluation system and to carry out specific evaluations, information for prospective offerors will be extensive and will

be furnished in the following steps: a) o/a 12 Feb 76 a preliminary description of both solicitations will be made

available upon written request from prospective offerors;

b) a preliminary briefing will be held at 10 o’clock am,

local Washington, DC time, 26 Feb 76, in Rm 6802 at the

Dept, of Commerce, l4th and Constitution Ave. , NW. Washington, DC, to provide a full description of these requirements

of the solicitations, and to answer questions; a copy of

the solicitation will be furnished to each firm having a

r.epresentat ive at the preliminary briefing, provided the

firm has furnished the Government prior written notice of

attendahce; firms in attendance without prior notification

will be mailed a copy of the solicitation as soon as possible thereafter. Other firms will be provided a copy as

soon as possible upon receipt of a written request, accompanied by a description of the firm’s capability to perform

the type of work required; c) the formal pre-proposal conference will be held at 11:00 am, local Washington, DC time,

on 18 Mar 76 in Rm 4833 at the Dept, of Commerce (same address as preliminary briefing) to provide additional information and answer questions. Proposals in response to Sol.

6-35756 shall be due o/a 15 Apr 76. See Notes 24 and 80

(P042)

US Dept, of Commerce,

Procurement Div., Research and Tech. Asst.,

Washington, DC 20230

FIGURE IV.THE COMMERCE BUSINESS DAILY ANNOUNCEMENT
Following the issuing of the CBD announcement and completion of his project plans the COTR turned his attention

to planning the pre-bidder’s conference. A dry run for

the pre-bidders briefing was held on February 25th and was

attended by the contract negotiator, several ETIP staff

and staff from VA and NIGP. The COTR’s objective for the

first briefing was to communicate what it was ETIP was trying to buy, to generate enthusiasm for bidding, to give

prospective bidders an opportunity to ask questions, and

to obtain feedback on the preliminary RFP which would allow

for clarifications prior to the next briefing (COTR, Note

58 ). He wanted to convey that ETIP was interested in buying

both systems and specific evaluations and that methodologists were needed who could integrate the different type

of specialists required into an effective team. He also

felt it important to convey the message that ETIP was looking for contractors who would assume a lead position in

doing the work while being sensitive to the needs of ETIP

and its PAAs. Finally he wanted to convey to the bidders

that ETIP did not have a sole source position on the contract and to provide as much information as possible about

the program and the proposal evaluation process so that

firms could make informed decisions about whether to bid

(COTR, Note 59; Note 60 ).
The pre-bidders briefing was attended by representatives of 47 organizations ("Attendees at Briefing," Note

6l). It was opened with some remarks by the contract negotiator. He indicated that though the solicitation document

itself would not be available until a later date, the information which was already furnished to prospective bidders

constituted the technical aspects of the solicitation.

The document itself would include some additional instructions and conditions, special provisions applicable to the

contract, and other boiler-plate provisions (Transcript,

Note 62). He also indicated that the two contracts would

be awarded to separate firms and he corrected an error which

g had appeared in the CBD announcement.

With his announcements completed the contract negotiator introduced the COTR and turned the briefing over to

him. In his presentation the COTR briefly described the

ETIP program and the objectives of the specific procurements. He also indicated the interrelationship among the

various items to be included in the solicitation document

(Transcript, Note 63). After this, he introduced the chief

of the procurment program for a description of the substantive area for which the systems were to be developed.

The announcement had indicated that the proposed level

of effort of the contracts was 100 man-months each rather

than the actual forecast which was 200 man-months each.

A corrected copy of the announcement was made available

in the briefing room.
The chief of the procurement program gave a slide talk.

He described the history of the procurement program and

the philosophy used in selecting and carrying out specific

experiments. He also described products and incentives

which were being experimented with at that time (Transcript,

Note 64 )

After the second presentation was concluded the contract negotiator requested that questions be submitted.

Questions were accepted annonyraously in writing on index

cards or orally from participants. The questions pertained

to the following:

o the scope of the work and the projected role

of the successful bidder in the various activities related to ETIP procurement experiments,

o coordination between the two contractors

in developing the systems,

o potential for and advisability of coordination among firms in bidding,

o the availability of the names of contractors

who had previously participated in the design

of ETIP procurement experiments,

o the projected role of the various PAAs in

the projects, and

o requested clarifications of statements made

earlier in the briefing (Transcript, Note

65 )

, The meeting was adjourned after all questions were

answered. Participants were given three weeks prior to

the second briefing to reflect on the pre-bidders briefing

and to examine the technical elements of the solicitation.
The second briefing was held on March l8 and was attended by representatives of 29 firms ("Attendees at ETIP preproposal conference, Note 66).^ It was primarily a question

and answer session. The contract negotiator distributed

copies of the solicitation document and indicated that it

would be modified to require 20 copies of each proposal

rather than the stated 15 (Transcript, Note 67). He then

opened the meeting for questions. This time the questions

included questions on:

o whether the requirement for oral presentations

remained , '

o what outputs were expected from Phase I and

Phase II of the projects in light of the

small number of procurement experiments which

were in existence,

0 how the efforts of the two contractors would

be coordinated,

o requirements and format for responding to

factors, sub-factors, and illustrative questions,

o whether a list of the attendees at the briefing

would be available to aid people who were

looking for co-bidders,

o the expected involvement of state and local

procurement organizations in the experiments.

Q Eighteen of these had been at the February briefing

and 11 were new.

^^The process originally called for each bidder to make

an oral presentation on their proposal to the proposal eval

uation panel. This was modified to allow written questions

submitted to bidders by the panel through the contract negotiator .
99

o the anticipated transition process from Phase

I to Phase II, whether much down time was

expected, and about how the 200 man-months

should be divided between Phase I and Phase

II, and

o where the PFE approach had been used previously

(Transcript, Mote 68).

After the questions and answers were completed the

Director of ETIP spoke briefly. He thanked the representatives of firms for their time and interest and told them

that things were beginning to move quite rapidly in the

procurement program and that many more projects were expected to be in place soon (Transcript, Mote 69 ).

With the briefings concluded, organizations had four

weeks to determine whether to bid and to complete preparation of their proposals.

IV .

Selecting and Involving the Evaluation Panel

A substantial portion of the time prior to the second

briefing was spent preparing for proposal evaluation. It

was during this period that the author arrived (March 8,

1976) and was assigned tasks of both contacting potential

panel members to solicit their participation and planning

for the panel meetings.

An effort was made to get significant participation

on the panel by people outside of ETIP. Some of those contacted were from PAAs and others were people not directly

related to the program. Those not directly related included:
o people who were thought to be experts on

some of the specific sub-factors being evaluated ,

o people who were familiar with some of the

likely bidders,

o people who could lend credibility to the

project with the NBS program office, and

o people who were responsible for managing

other government evaluation programs who

might have an interest in the PFE technique

as a method for procuring contractor support

(COTR, Note 70).

Nineteen of the 30 people who were invited to join the panel

were from organizations other than ETIP (compiled from lists.

Note 71 )

When soliciting participation in the process, both

the COTR and the author stressed the ability of a person

to limit the extent of their participation. They were told

that they could participate by evaluating as few as one

sub-factor if their time was severely limited. Eleven of

the 19 people invited from outside agencies eventually participated or sent representatives to participate in the

evaluation.

Members of the panel who had been recruited by midMarch were called together for a meeting on March 17 to:

o review the process for proposal evaluation,

o determine any additional information which

should be related to bidders at the second

bidders briefing on March 18,
o distribute copies of the the technical portions of the solicitation document and a

list of supporting documents which had been

made available to bidders, and

o review the schedule for the April factor

team meetings (COTR, Note 12 ).

Prior to this meeting Thompson had written a document entitled "Proposal Evaluation Procedure" and had prepared

the forms which would be used by the panel during the evaluation process (Thompson, Note 73). One form was for rating

the bidders on sub-factors and factors and the other form

was for use in obtaining clarification from bidders between

the time proposals arrive and the time that factor ratings

are assigned.

At the meeting the documents from the solicitation

and the list of supporting documents which had been

furnished to the bidders were distributed to the

1 1 11 invited panel members who were present. Panel members

were encouraged to at least scan through all materials sometime in the month that remained prior to the arrival of

proposals. The evaluation process was then reviewed and

much of the remainder of the meeting was spent in an argument concerning the weighting of sub-factors within the

factors

1

Three of these later dropped off of the evaluation

panel
102

At least two of the people present felt strongly that

the relative weights of sub-factors within each factor should

be pre-determined by the COTR. Their reasons were that

they felt that:

0 waiting until individual sub-factor scores

are known with respect to the names of the

proposers presents an appearance of conflict

of interest, with the factor team being able

to manipulate the weights in favor of a particular proposal ("Memorandum,” Note 7^;

"Memorandum," Note 75), and

o a great amount of time would be taken trying

to reach agreement on the weights in the

team meetings ("Memorandum," Note 76).

The argument was quite heated and resulted in several of

the participants writing memos afterwards to clarify their

positions ("Memorandum," Note 77; "Memorandum," Note 78;

COTR, Note 79).''^

The COTR decided not to request a change in the process.

He felt that one substantial input which the panel members

could have would be assessing the relative importance of

sub-factors. He also felt that the time for decisions would

not be adversely affected by the lack of pre-assigned weights

(COTR, Note 80).

Another change in the process was proposed at the meeting by the representative of the NBS program office. He

suggested that an executive committee be established to

1

Two of these people were among the three at the meeting who eventually dropped off of the panel.
103

review the decisions made by the panel. The COTR rejected

this proposal after conferring with the contract negotiator.

The contract negotiator advised against such a committee.

He felt that the process would provide him with a clear

indication of the technical merits of the proposals and

did not desire a panel which would filter these recommendations (COTR, Note 8 l). In his follow-up memo to those who

attended the meeting, the COTR expressed his faith in the

panel and the process and said that he felt that such a

review committee would be unlikely to have sufficient familiarity with the individual proposals to be effective and

would give the appearance of second guessing the panel (COTR,

Note 82 )

.

This was the only time the panel met as a group prior

to the arrival of proposals. The remainder of the time

prior to the closing date was spent recruiting additional

panel members and making assignments of individuals to specific sub-factors. In general assignments were made so

that individuals would evaluate sub-factors where they had

a special expertise or interest. In several cases, where

individuals were expected to have a close relationship with

the projects, they were assigned to a wide range of subfactors. Each sub-factor had at least two people assigned

to it

.
IV. 9 The Proposals Arrive

The closing date for receipt of proposals was April

15. On this day the contracting office received eight proposals. Five were for consideration as the, agency impact

contractor and two were for consideration as the commercial

impact contractor. ^ One firm asked that its proposal be

considered for both. Another firm had submitted separate

proposals for each contract. That is, seven firms bid.

Four bid on only the agency impact contract, one bid on

only the commercial impact contract and two bid on both

contracts

The proposal evaluation panel was convened the morning

of April 16 . Nineteen of the 23 panel members were able

to participate. The first agenda item to be- covered was

to review the evaluation process. Members were asked to

look at the "Proposal Evaluation Process" document in their

packet as it was reviewed with them. They were reminded

that there were two minimum requirements on each member

of the evaluation team. These were:

1

The proposal submitted by one firm was not clear as

to which contract the firm wished it be evaluated for.

The contract negotiator was called and requested to obtain , a clarification from the firm. The firm indicated that

it was submitting for only the commercial impact contract.

The factor teams on agency impact included this firm in

their evaluation until the clarification was obtained.

These ratings are not included in the data analysis in Chapter V.
1 ) If a panel member finds ambiguity or missing

information which will substantially interfere with their ability to rate a bidder,

it needs to be uncovered right away so the

contract negotiator can obtain an answer

back from the bidders in time for evaluation

(question forms were provided to panel members for this purpose).

2) Panel members assigned to a sub-factor need

to fill oub one of the rating forms. The

names of the bidders are to be put in the

blank space. The rating can be based solely

upon reading that one section of the proposal. What they want to read beyond that

is up to each individual (Tape of meeting,

Note 83 ) . Panel members were cautioned that they were not to acquire

additional information from bidders except through the contract negotiator (Tape of meeting. Note 84).

Members were reminded that the factor team was responsible for coming up with a single rating across the factor

for every one of the bidders and that each team had control

of most of the process by which they chose to do this.

The ground rules were:

o each individual was required to submit a

signed sub-factor rating sheet at the start

of the session,

o the team had to reach an agreement on the

ratings without voting,

o all members of the team were to sign the

factor rating form (any comments, references,

and reservations were also to be included

as a part of the record), and

o the sessions were to be taped (Tape of meeting, Note 85 )

.

The method for reaching agreement, including the order of

consideration of sub-factors, the relative weighting of
the sub-factors in the final factor weightings, and whether

to consider the factor as a whole rather than rating the

sub-factors were up to each team (Tape of meeting, Note

86 ).

The panel was told that results of the individual factor teams would be combined using the factor weights listed

in the RFP. A narrative description of the results of the

panel process would be prepared for the contract negotiator.

This would be followed by meetings for clarification with

the contending bidders (Tape of meeting. Note 87).

Individuals' assignments to sub-factors were reviewed,

factor/sub-factor rating sheets were distributed, copies

of the proposals were signed out to panel members and the

meeting broke for lunch (Tape of meeting. Note 88). During

the break a tentative schedule of factor team meetings was

worked up by charting the schedule conflicts which panel

members had during the next week. The best schedule which

could be worked out resulted in several panel members being

unable to participate due to schedule conflicts.

The remainder of the day on April 16 , was allocated

to initial review of the proposals. Two question forms

were submitted which pointed out some sections which were

apparently missing from proposals submitted by two different

bidders (Question for bidder #101, Note 89; Question for

bidder #102, Note 90). These questions were called in to
the contract negotiator for transmittal to the bidders.

Figure IV. 4 is a copy of the form which was used.

Team members who were unable to attend the April 16,

meeting were briefed individually and provided with copies

of the proposals and the schedule.

IV. 10 The Factor Teams Meet

IV. 10.1 Introduction

For the two projects there were a total of fourteen

factors and thus fourteen teams. All fourteen factor team

meetings were held during the week of April 19th (see Figure

IV. 5). They were scheduled at a rate of three factors per

day with the exception of the first day, for which only

two meetings were scheduled. The nominal time allocated

for each meeting was two hours.

1

The meetings were held in ETIP’s conference room.

The room was equipped with a blackboard, a tape recorder

and extra rating and question forms.

Each meeting having new participants who had not been

in earlier meetings was begun with a brief discussion of

the evaluation procedures and a decision on how to proceed.

The meetings then continued until scores were determined

for all bidders on each sub-factor and on the factor as

a whole.

An exception was the first factor team meeting which

was held in a larger NBS lecture room.
108

RFP 6-35756

' C3UESTION FOPM

a Requested by:

Bidder: , Date:

Factor (or svib- factor)

Proposed

Specific

Question:

Qlto the bidder Note: Write the qxiestion as Q lb (specify) : specifically as possible in a form

that the bidder can respond to directly. Give this form to the CCTR.

Question (Revised, if necessary) : Date given to CO: Date answer received:

FIGURE IV.

THE QUESTION FORM FOR USE IN OBTAINING

CLARIFICATIONS
gency

Factor

Commercial

Factor

Commercial

Factor

Commercial oo

CO co z. o o

4J 4-5 4J o o o

0) to CO 1—

ta 4J Cl.C

rH

CO o f-H 0) rH *rH u CO s to -P

•H QJ •H CO

CO to . O bOO 3

4-) CO £, CO £- rH

C

CO a; <UC a> co bo

Q o E CO E > o E S E Cd to 1—

i.

OO CU > a. U M U > Q

to C.

•rH o O

CO 4J 4J o x> <U

r-C to c CO "O

CO Ci4 CO Cl4

C c 3 C

•a; rH to rH

O

O

CO 4J 3 £-. CO •r4 J_ •p b04J

to o o CO

O Ja: cO CO to s- a 3 t. O E c CD 0) CD QJ (U cO S- O E (X •rH E 03 O w O E > E «M

t—

t.

O t-t 0)O > G > cu a i-i cs CJ M HH M

1—

HH > M

to0) t. CO

G > O —

O

•iH 4.5 CO 4.5 T3 »

-P o >.

OG o CO .—1 cO G

0) Cz. CO Cl,

•’“5 c to CO CO >!<S CO >.44. 3 o o 0) O G (U C CO o G O -p

1—

0) 4J o (u

a >

1—

bO CO J- bO CU CU HH < Q a. < ec «

CO(U

G T3 > O 4J O G

-P c 4H 3 G

4J

O (U CU

OO

C CO £ cfl

G *P o Cl, OJ Cl. b04J

bO ^ CO J3 >: to >•- o s O O C O CO G c to G oa o

1—

CU S 0 Gi

!VH bC h0>

G

ci; I—I <c t-c «S l-t

M I o O tH to 4J .-I n o o o 2 CO o o

bu

O

CO L, >.4J Ou O CO c a c

(U o OhOi—I -H

4J o c

-P o o T-c cO -P

ta CO3

>.1-1

C

O cfl bO C > -r-C 0) Ci3 CO ObO <U Q

C\Jt OJ 1 o r— tv^ 1 ••

m o C\J 1 r- CO

FIGURE

IV. 5

SCHEDULE

OF

FACTOR

TEAM

MEETINGS
What follows in this section are examples of how the

team meetings worked. Anecdotes are presented from the

meetings to illustrate the types of issues which arose and

how they were resolved. They are presented to illustrate

the types of discussions which took place over the course

of the evaluation. They are not meant to represent the

total discussion during any of the meetings. The sources

of these are the tape recordings which were made of each

of the meetings. Displays of the initial scores for each

rater and the final scores for the teams are included in

the appendix for readers who wish to follow the actual numbers from the evaluation process. The names of individuals

involved in the discussion and firms being rated are not

included in these descriptions or in the appendix.

The first factor scheduled to be evaluated was Factor

VI, Data Collection Processes on the agency impact contract.

This factor was selected as the first because it was felt

to be less difficult to evaluate than some of the others,

e.g., management, and would be a good one for panel members

to get a feeling for how the process functioned. All team

members had been encouraged to come and observe the first

meeting
Ill

IV. 10. 2 The First Factor Team Meeting

The first factor team meeting, Agency Impact, Factor

VI, Data Collection Process, began Monday, April 19, 1976,

at approximately 10 A.M. The four assigned panel members

were all present. The tape recorder was turned on and the

meeting began with some discussion on how to proceed. Ground

rules were • quickly reviewed and the first set of signed

sub-factor rating forms were handed in by panel members.

(Figure IV. 6 is an example of what a completed subfactor

rating form looked like.) Some time was taken getting the

« scores posted on the blackboard in order to determine a

method which would make the distributions clear (Figure

IV. 7 illustrates the form that the postings took.) This

was followed by discussion on how to proceed in order to

k obtain scores for each bidder. One panel member suggested

averaging the scores. Another argued that this would not

be acceptable since there was a need to know why one person’s score for a bidder was a nine while another person

scored the same bidder at one.

Discussion on the individual proposals began with people giving the reasons behind their ratings. A panel member

inquired about the extent to which personal knowledge about

bidders was admissible and he was assured by another member

that it was a very proper thing to put onto the table.

Another question was whether the panel should be concerned
112

RFP 6-35756

FACTOR (AND SUB-FACTOR) WORKSHEET

Factor or sub-factor (number & title)

VI.] Instrument Design

When used by individual team menbers for

siib-factor ratings:

Signature : EVAL205

Date: April ] 9, 1976

When used by team to record final

factor ratings only:

Signatures:

Date:

Nothing can stop

him from being

a success

Should perform

well

ID

B205

Almost as good

Acceptable as is

B206

Acceptable ’.»ath negotiation

B203 B204

B202

Cll Qiedc if continuation sheet acoorpanies this form

Marginal at best

Unacceptable

, without significant

• revisions

i Possibly still

I xmacceptable with

' revisions

Probably still

, unacceptable with

revisiOTis

No hope (totally

unacceptable)

Note: Individual team members may use form as worksheet, making dianges, erasures,

etc., and adding, notes. Prior to the beam meeting, ratings should be entered

in ink . Subsequent notes and revisions may be made during the meeting by

cross-out, etc. , and the resultant form will become part of the record.

FIGURE IV.

A COMPLETED SUB-FACTOR RATING FORM

«
113

>

LD

HH>

>

..

cn>

‘cm >

.

>

CM

_Jcc>

UJ

B202B204 B205B206 B203 O

CM

CD

EVAL210

o

]go

C'J ca — <o

CD ffl O O

m^ o ffl Si ^ CD (O

CSJ ‘ ca

EVflL208

IDO

QD O

CSI O ^CD

CSi CO

B203B204 B206

•

EVflL205

IDoM

CD s § ^ ® to oS ® 6 flo

CSJ CD

>

01 6 8 L CD ID cn CXI

^

FIGURE

IV.

7

THE

FORM

OF

POSTING

SCORES
at this point about whether certain people in a winning

bidder's organization would be tied to the contract through

a key personnel clause. He was told that anything which

the factor team was concerned about which influenced its

rating should be noted in the record of the process so that

it could be discussed with potential winners.

It was pointed out during this and other panel meetings

that all bidders falling into the competitive range would

be called in for discussion and clarification prior to final

evaluation. It was therefore important to enter notes into

the record where the panel had uncertainty about their ratings. The related issues would then be raised with bidders

during the subsequent discussions in order to reduce the

range of uncertainty.

The key point of disagreement on sub-factor VI. 1 was

whether'a specific bidder did, in fact, have the in-house

capability to perform in the area of instrument design.

The issue was resolved by agreeing that the bidder was probably acceptable on the sub-factor but the remaining uncertainty would be recorded with a note in the record. The

note was to be sure that they have at least one person integrated into their team who is strong on instrument design

in the event that they are found to be strong enough overall

to be considered competitive.

As the panel arrived at a score for each of the bidders, that bidder's score was posted on the board under
115

the sub-factor column and removed from under the individual

rater columns (see Figure IV. 8). After all bidders had

been rated on a sub-factor the left side of the board was

clean and the column for the sub-factor score was complete

(Figure IV. 9). Panel members scores were then posted under

their column for the next sub-factor (Figure IV. 10) and

the same process was repeated.

There was a great deal of variability in the amount

of discussion required under each sub-factor to arrive at

scores for different bidders. As would be expected, it

seemed that in cases where there was substantial initial

agreement the time would be brief. Conversely, the time

required would generally be longer in cases where at least

one panel member’s initial rating varied from those of others

on the panel. Often the panel began discussion on bidders

for which there was greater initial agreement. This enabled

the panel to make progress in rating and at the same time

discuss some of the issues related to the sub-factor without

strong arguments over the bidders.

Several times during the first meeting there was discussion about the significance of the different numbers on

the rating form. This discussion seemed to establish the

interpretation of the numbers for the remaining factor team

meetings. The distinction between a score of six and a

score of seven was emphasized. Using the words next to

the numbers on the form as a guide, it was felt that any
116

>•

1

1

LOHH>

1

I

H-

>

^ !

1

m

t—

>

1

1

CM>

1

1

>

1

CSI o

Csj 1 IS

EVflL2ll

1 OOcvjcvj cn ^ coo o 1 o

CM C'J o CQ QQ

ega 1

EVRL210

1 o S eg — <o o o o S*“ ”O Si Sil QQ O

CM < GO

EVflL2D8 B205

B201 B203B204 B206

EVflL205

1 I

lO { CO S

eg T

flQ 1 oa g

eg

CO

>

^

CD CO l> CD LO

r—1 1 CO C\J -H

I

I

FIGURE

IV.

ASSIGNMENT

OF

THE

FIRST

SCORE

ON

SUB-FACTOR

VI.
117

>

LD>

• >

m

1 >

CM

I—. >

>

U3 CSI O O

(\J CM CQ CQ B201

B203

B206 B204

EVAL21

1

EVflL210

EVRL208

EVflL205

t—

>

cn CO

r-H CD LD

^cn c\j

^

FIGURE

IV.

ASSIGNMENT

OF

ALL

SCORES

FOR

SUB-FACTOR

VI.
118 !1!1

LO> 1 >

cn>

CM>> B205 B202 0201

0203

0206

B204

EVHL21 B202

B201

B205

B206

B203

B204

EVAL210

(Oo S CSJ c s “ s CQ I o D CM

03

si ^ B g 3 a B206

EVF1L208

LOO

CSI o

CMmoo— cn

CM CM CM O 03 CD

CM

03

B206

EVAL205

lOo

CM

CD

(D (1 O C CM C CD C"i ^ CM 5 0 O bpJ CM CM

CO

CM>

10 9 8 7 CD LO cn CM ^

OJ>

ccoHO

<C

CZ4I cn

COoo

hH

E-<

<a: cc

t—

o

DJ

CQO

COw

(Xoo

CO

Duoo

E-i COO

CuO>

I—

CdKo

I—Du IiIIIIIi
bidder who was rated at seven or above was thought to have

an acceptable probability of performing with respect to

the issues raised by the sub-factor. A bidder was rated

at a six or below if there was some doubt that the bidder’s

performance would be acceptable. This meant that in the

event that the bidder appeared to be acceptable overall

they would be called in for discussions. These discussions

would be used to resolve existing uncertainties over the

bidders’ scores, i.e., to clarify whether the bidder could

be expected to perform well on the sub-factor if they were

under contract. A six reflected a modest doubt as to the

bidder’s capability. The bidder was rated at a five if

the panel had more serious doubts that the bidder would

perform acceptably. Scores of four or below were used if

the panel had very serious doubts about the bidder’s present

or potential capability with respect to the sub-factor issues. A bidder with more than one or two fours on subfactors was thought to be unacceptable unless they were

rated significantly higher elsewhere and could demonstrate

that they could improve their score on these low subfactors .

In arriving at sub-factor scores the panel was

often reminded by its members that it was to assess

bidders using the criteria of the bidder’s understanding

of the sub-factor, the proposed approach for dealing

with the issues raised by the sub-factors, and the
bidder’s present or potential capability for dealing with

the issues. There were cases in the first factor team meeting, and in many of the others, where the panel felt that

the bidder’s response indicated an acceptable understanding

and approach but there was a reasonable doubt that they

had the capability to perform.

As an example there was one bidder who had a team that

was quite heavily loaded with consultants. It appeared

that the understanding of the issues was held by these consultants but it did not appear that these consultants were

integrated effectively into the decision making process

for the project. In this case the bidder was given a score

of five, with the understanding that this point would be

investigated during discussions if the bidder were in the

competitive range.

Ratings seemed to be made on both an absolute

and a relative basis. Bidders’ responses were discussed as they compared with the standards which each panel

member used in applying the three evaluation criteria to

their ratings. There was also discussion across bidders

of the relative degree to which they dealt with the subfactor issues. For example, a panel member would indicate

the evidence he used in his rating of a specific bidder

and then contrast it with what another bidder said as an

indication of why he rated that bidder either the same or

differently
After all of the sub-factors had been rated in factor

VI, the panel was prepared to determine the factor score

for each bidder. There was some discussion of how the various sub-factors should be weighted. The statement of the

factor was read and the panel members proceeded to assign

the factor score by visually averaging the sub-factor scores.

No one had a compelling desire to weight any sub-factor

greater than any other. The scores were posted on the board

(Figure IV. 11) and when the process was complete they were

copied onto a factor rating sheet. The sheet was labeled

with the factor name, dated, and passed to each member of

the panel for a signature. (Figure IV. 12 is an example

of what a completed factor rating form looked like.) Determining factor scores once the sub-factor scores were complete took about five minutes. The first panel meeting

was concluded two hours after it began.

IV. 10. 3 The Second Factor Team Meeting

The second meeting was Agency Impact, Factor V, Evaluation Design. It began Monday, April 19, at about 3 P.M.

with the seven panel members present. After a brief review

of ground rules, the ratings for the first sub-factor were

posted and discussion began.

During rating on the first sub-factor (V.l) there was

a case where information presented by a bidder as a strong
122

I

ASSIGNMENT

Or

FACTOR

SCORES

FOR

FACTOR

VI
RFP 6-35756

FfiCTOR (AND SUB-FA3TDR) WORKSHEET

Factor or sub-factor (number & title)

VI Data Collection Process

When used by team to record final

factor ratings only:

When used by individual team msnbers for EVAL208

sub-factor ratings: EVAL210

1 Signature:

EVAL211

Date: Date : April 19, 1976 —

1

! Nothing can stop

him from being 10

I a success

Should perform

1

* well 9

1

‘ B205 1 Almost as good 8

B203

1

Acceptable as is 7

1 - Acceptable with

B202 ; 6

1

negotiation

I B206

1 Marginal at best 5

' tfriacceptable

B201 B204 , without significant 4

revisions

1

1

Possibly still

unacceptable with 5

Note: Should B202 get the contract they'

must bring someone on board who is

revisions

competent in instrument design (VI. 1). • Probably still

B202's performance must be further unacceptable with 2

examined on sub-factor VI. 3. revisions

No hope (totally

Check if ocntinuation sheet accompanies this form i unacceptable) i

Note: Individual team ment)ers may use form as worksheet, making changes, erasures,

etc., and adding notes. Prior to the team meeting, ratings should be entered

in ink . Subsequent notes and revisions may be made during the meeting by

cross-out, etc. , and the resultant form will beccne part of the record.

FIGURE IV. 12 A COMPLETED FACTOR RATING FORM
example of their capability in the area of the sub-factor

was disputed by a panel member with outside information.

The bidder had claimed to be making a major contribution

in a substantive area in which the panel member claimed

to be knowledgeable. The panel member indicated that he

had just completed a major review of work in this area and

was unaware of this bidder's involvement in the field.

It is unclear how much this information affected the final

score for this bidder since others had argued in the firm’s

favor. Panel members had rated the firm between four and

seven and a half and after discussion the team, settled on

a f i ve

When rating another bidder two panel members disagreed

about prior performance of members of the bidder’s proposed

team in the area of evaluation design. This resulted in

a discussion of the characteristics of the prior work in

question and its relationship to the present work and the

standards of judgement that each of these two panel members

was using. The rating was resolved by the panel in favor

of the claim that the work which the bidder had done effectively in the past was not sufficiently comparable to what

was required here.

In discussion on a rating for another bidder, a panel

member from outside of ETIP was familiar with their prior

performance. There had been some argument for a low score
and this panel member was able to cite prior positive experience with the people from this bidder’s organization to

influence a higher rating for them than was desired initially by some of the ETIP raters.

The rating of sub-factor V.1 presented the panel with

its first difficult problem in achieving consensus on a

single rating for a bidder. After panel members had presented their views on the bidder there remained a strong

disagreement between two panel m.embers. One would not rate

them below a six and the other would not come above a four.

$ The panel agreed to leave this temporarily unresolved.

The bidder was put at both four and six on the board with

brackets around both scores indicating that the score would

be resolved after the other sub-factors had been discussed

and before assigning the factor score. Eventually they

were given a score of six on this sub-factor and six on

the factor. This technique was used several times during

the week and proved to be useful in keeping the evaluation

process moving and in resolving differences.

The time to accomplish the rating for the first subfactor was about 75 minutes. Lengthy discussion on the

first sub-factor tended to be a pattern throughout the factor meetings. The arguments which were made provided a

context for the discussion on other sub-factors which then

proceeded more rapidly (Figure IV. 13). The ratings on subfactor V.2 were accomplished in about five minutes.
126

I . 1 ,2 • ^ .4

Agency OO 6 18 9 Commercial 1 1 7 4 21

II . . 1 . 2 .3 ,4

Agency iiy 25 18 16

Commercial 18 22 14 4

III . 1 .2 ' .3 .4 .5

Agency 31 19 25 15 23

Commercial 38 18 15 24 7

IV . 1 .2 .3 . 4 .5 Agency 26 10 . 16 3 8

Commercial 14 10 10 4 2

V . 1 . ? .3 .4 . 5 Agency 74 3 20 8 30

Commerc ial 33 16 6 7 2

VI . 1 .2 .3 . 4 c

• J Agency 39 21 19 1 1 23

Commercial 31 6 12 6 2^

VII . 1 0 .3 . 4 Agency 39 38 20 25

Com.m.ercial 63 5 15 5

FIGURE IV. 13 TIME REQUIRED TO FATE SUS-FACTOPS

(estimated from taoes of meetir.ss)

O
This factor team meeting (factor V) lasted a

little over two and a half hours. Since it was held

at the end of the day, three of the team members

had to leave after the fourth sub-factor, when the

time got late.

In this case, and in several others during the week,

panel members were unable to be present for part or all

of the team meeting. If they had furnished their signed

rating forms to the panel, their ratings were posted on

the blackboard as a reference point for the team. If the

team felt it needed to inquire into the reason for a rating

of a person who was not present, that person could be called.

This option was used in one of the later meetings where

the absent team member had rated a bidder higher than other

team members who were present.

IV. 10. 4 The Third Factor Team Meeting

The third factor team met at approximately 9 A.M.

on Tuesday, April 20. This was factor III, Objectives for

Agency Impact. The team had six members, one who arrived

after discussion of the first sub-factor had been completed.

One issue which was pointed out during this meeting

was that at least two of the team members were tending to

take extreme positions when rating certain bidders. This

was because one member stressed the importance of bidders’
128

qualifications with respect to the substantive area of procurement, while the other member stressed the importance

of qualifications in methodology. They tended to rate bidders as extremely high or extremely low on many sub- factors

according to these biases. In effect they were evaluating

the proposal as a whole on each sub-factor. This led to

discussion on the importance of various experiences and

skills and how they relate to potential performance with

respect to the specific sub-factor issues. Discussion and

recommendation of consensus ratings by other team members

was important in forcing team members to examine biases

against the specific requirements of each sub-factor.

IV. 10. 5 The Fourth Factor Team Meeting

The fourth team to meet was Factor I, Management for

Agency Impact. This team met on Tuesday, April 20, at 12:30,

with four team members.

Much of the discussion during this meeting was about

the distinction between capability to do general project

management and the capability to manage this specific project to develop evaluation systems. At least one panel

member felt that the former was sufficient. Others argued

the merits of experience in the management of specific related types of work.

There was also discussion on the commitment of the

firms to provide a full time team for the necessary two
129

to three years of the project. Sub-factor II. 2 specifically

raised the issue of the internal commitment of the firm

to the project. On this sub-factor the team was unable

to differentiate among bidders from any readily available

information. It was decided that all bidders would be rated

at 6, and the necessary clarification on this sub-factor

would be obtained during any subsequent oral discussions

with the bidders.

IV. 10. 6 The Fifth Factor Team Meeting

Agency Impact, Factor IV met on Tuesday, April 20,

# at 3 p.m. Five panel members were present for the meeting.

In this meeting there was a situation in which one

panel member was arguing for a rating of four for a bidder

on a sub-factor. Two other panel members had rated the

bidder at seven and one had rated them at nine. The high

rating was from a subordinate of the person who gave the

low rating.

One of the people who rated the bidder at seven indicated that he had used information which the bidder provided

under another sub-factor (within the factor) to support

his relatively higher rating on this one. The person who

was at a nine indicated that he would be willing to come

down to a seven if his boss would come up. At this point,

the low rater indicated that he would not want to come above

a six

.
There was additional discussion in which the panel

member who had found information elsewhere in the proposal

pointed out the specific sentences which he felt were relevant. The low panel member eventually agreed to com.e up

to seven, indicating, however, that he was uncomfortable

in doing so.

IV. 10. 7 The Sixth Factor Team Meeting

Commercial Impact, Factor III, Objectives, met on Vvednesday, April 21, at 10 A.M. Five panel members were present.

In determining ratings under this factor there was a substantial amount of discussion concerning the appropriateness

of various sub -objectives as related to the overall objective of developing the systems. One panel member explained

that he had rated a bidder low on a sub-factor where the

bidder had indicated that ETIP should do some simplistic

evaluations early in the project in order to have some favorable outcomes. He felt that ETIP's objective was to produce

credible results. Another panel member argued for the bidder’s position and stressed the importance of ETIP having

some successes in the first year in order to build constituency support for the evaluation systems. The first panel

member was persuaded and raised his rating.

Another series of discussions on this factor surrounded

the match between the approach proposed by a bidder and

the objectives of the project. The bidder had been developing a specific model which they believed was applicable
131

to ETIP’s requirements. The proposal indicated that they

would further test and develop this model using the various

procurement experiments. One panel member stressed that

this bidder wished to pursue his own independent objective

of developing his model and was not considering other methods which might be far more appropriate under various circumstances which would occur during the project. This panel

member was successful in persuading others on the team to

reduce their ratings.

IV. 10. 8 The Seventh Factor Team Meeting

* The next meeting was Agency Impact, Factor VII, Data

Analysis Process. It met Wednesday, April 21, at 12:30

with four panel members. During this factor meeting a discus

sion occurred concerning the capability of a bidder who ^ admitted to a lack of capability in dealing with a priori

proposition testing. One team member used this as a justification for his low rating. He was reminded by another

team member that the standard for evaluation is present

or potential capability. Since the bidder seemed to have

a good understanding of where their specific weakness was

it was argued that they had a strong potential capability.

The panel settled on a score of seven.

This was one factor where the team made specific use

of a panel member from outside of ETIP who was an expert

in the substantive area of the factor. This panel member
presented discussion on what knowledge and skills he felt

contractors would need to have in order to match with what

he saw as the requirements of the project. He also discussed the appropriateness of the data analysis techniques

which various bidders were suggesting.

During this meeting one team member inquired as to

whether it was possible to determine which bidders could

no longer be considered acceptable and to save time by elim

inating these bidders from further rankings. That is, sine

this was the sixth factor team meeting on the agency impact

contract it would be possible to determine whether any bidder’s ratings were so low overall that they would be unacceptable no matter how well they did on this and the remain

ing factor ratings. Another panel member pointed out that

the rules of the procurement did not allow the panel to

do this. The procedure required evaluation of each bidder

on all sub-factors.

IV. 10.9 The Eighth Factor Team. Meeting

Agency Impact, Factor II, Reports and Reviews was the

final Agency Impact factor team to meet. The meeting was

convened at 3 P.M. on Wednesday, April 21, with five panel

members. During this factor team meeting a lengthy discussion took place when one panel member wanted to rate one

of the bidders as a 10 on the first sub-factor. The panel

member claimed that this was one of three sub-factors in

the procurement where he was an expert and that he felt
that this particular bidder really understood the issues

here. To support his argument he referred to his prior

experience with other contractors on other projects in dealing with these issues. In the end the panel could not be

convinced to come above an eight.

IV. 10. 10 The Ninth Factor Team Meeting

The Commercial Impact, Factor VII, Data Analysis meeting began at 10 A.M. on April 22. Four panel members were

present. The ground rules were reviewed since a panel member who had not previously participated was present. Follow

ing this brief review the team was asked to turn in their

first set of ratings. The new panel member indicated that

he was prepared to discuss the proposals as a whole but

that he had not attempted to follow the rating sheet format.

He was told that the rules of the procurement specified

that each panel member is required to hand in at least one

sub-factor rating form in order to participate as a team

member. He indicated that he was unwilling to do that since

he felt that rating sub-factors was evalua-ting the issues

out of context. Discussion with the other panel members

continued for several minutes concerning the bases for the

evaluation and what the new panel member felt they should

be. He indicated that he would summarize his views of the
three proposals in a memorandum and said that he would

not go through the PFE evaluation process. He said that

he thought it was "hogwash." At this point he left the

room and the panel proceeded with its evaluation.

IV. 10. 11 The Tenth and Eleventh Factor Team

Meetings

Commercial Impact, Reports and Reviews (Factor II)

and Background (Factor IV) met on Thursday, April 22, at

12:30 and 3 P.M. respectively. Factor II had five panel

members and Factor IV had seven members. These factors

were rated fairly rapidly. Some of the discussion referred

back to points brought out on the same factors under the

1

agency impact evaluation. There tended to be considerablv

less discussion on the commercial impact evaluation as compared to the agency impact evaluation.

IV. 10. 12 The Twelfth Factor Team Meeting

The final day of panel meetings was April 23. The

day began with Commercial Impact Factor VI, Data Collection

Processes. Five panel members were present including two

new panel members. After the evaluation process was reviewed

1

This same panel member had sent a memorandum to the

director of ETIP on April 16, indicating his concern over

ETIP's plans to commit up to $2 million for the projects

and stating that he found the procurement process "overly

complex and poorly focused."

1

Two of the three bidders on the commercial impact

contract had been bidders under agency impact.
one of the panel members expressed a preference for discussing each bidder on the entire factor before doing any ratings. Rating sheets for all of the sub-factors were turned

in and about 15 minutes was taken to discuss what the various bidders wrote under factor VI. The rating process

then proceeded as in the other team meetings.

IV. 10. 13 The Thirteenth and Fourteenth Factor

Team Meetings

The final two factor team meetings were Commercial

Impact Factors I, Management and V, Evaluation Design.

They met at 12:30 and 3 P.M. with four and five panel

members respectively. These meetings were again

relatively short, with discussion referring back

to prior team meetings. The last meeting concluded

at about 4:15 p . m.

IV. 11 Summarizing the Results for the Contracting

Office

Following the last factor team meeting the COTR and

the author began the summary of results for presentation

to the contract negotiator. The factor scores for each

bidder were multiplied by the preassigned factor weights

to obtain thp overall ratings. The scores were then summarized in tabular forms for each of the two contracts to

obtain a visual display of the factor and sub-factor distributions. A determination of a recommended competitive range
was made by looking at both the final overall scores and

the distribution of each bidder's scores across all factors

and sub-factors. It was felt that bidders with low overall

scores or bidders which had acceptable overall scores but

had an unacceptable number of factors rated at four or below

could be considered unacceptable. An examination of the

distribution of scores revealed that on the Commercial Im1 7

pact contract bidder #101 clearly dominated the other

two bidders. In fact there were no sub-factors where the

scores of either of the other two exceeded bidder #101.

On the agency impact contract the situation was quite different. Here three of the six bidders were rated significantly higher overall then the other three. They were also

quite close to each other in overall score. One of the

three more favorable bidders on the agency impact contract

was the same bidder which dominated on the commercial impact

contract. However, since ETIP had stipulated at the outset

of the solicitation that it would not award both contracts

to the same firm, this bidder could not ultimately be considered for both contracts.

The recommendation which the COTR was prepared to make

was that bidder #101 be given further consideration as the

1

The bidders will be referred to by number rather than

name. These numbers are also used with the data presented

in figures in the text and in the appendix.
commercial impact contractor and that bidders #202 and #203

be given further consideration as agency impact contractors

In this case further consideration would require clarification on sub-factors where various factor teams had raised

issues for clarification with each bidder, or where the

bidder was rated below a seven.

In a May 4, 1976 memorandum, the COTR conveyed these

'recommendations to the contract negotiator. He provided

a chart (Figure IV. 14) giving the scores and then provided

seven pages of narrative to summarize the issues on which

each bidder had problems. (COTR, Note 91). The narrative

followed the break points set out on the factor/sub-factor

rating sheet. A bidder was said to be acceptable or better

on factors where their score was seven of greater and unacceptable on factors where the score was less than seven.

In elaborating further, a bidder was said to have moderate

problems on issues related to sub-factors where the agreed

upon score was six, more serious problems where the score

was five, and very serious problems where the score was

four or below (COTR, Note 92). As an example, the section

from the memo on bidder #206 read as follows:

Bidder #206

The ETIP panel rated #206 as acceptable on only

two of the seven proposal evaluation factors.

#206 was considered unacceptable on:
13

o Management

o Reports and Reviews

o Background Information

0 Evaluation Design

o Data Collection Process

Among the issues with which #206 had moderate

problems were:

o Its evaluation management capabilitv

o Its understanding of and capability to

define the detailed evaluation objectives

o Its ability to anticipate the incorporation

of the results of Phase One in the restated

objectives for Phase Two

Among the issues with which #206 had more serious problems were:

o Its understanding of the soecific objectives

of the program(s) and/or exoer iment ( s

)

that are to be evaluated

o Its ability to design the overall data

collection process

o Its ability to design the overall data

analysis process

Among the issues with which #206 had very serious

problems were:

0 Its ability to coordinate its activities

with those of other related individuals

and organizations not specifically referred

to in section II

o Its understanding of the programs, organization policies and procedures of other

relevant organizations.

This gave the contract negotiator considerably more

information than was available in the final numerical score

CO
The final ratings are based on the following factor ratings

assigned by the ETIP panel, and by the weights established

at the outset of the solicitation:

WEIGHTS

FACTORS

o

•H CO

4-^ *1—

-o c c CO

4-> c CO T3 O c CD >5 c cd 0) C -H o t—

1—

<D > O •i-H 1—1 cd S •fH o cd O 00 C CO o -p s -p tn S cd c O CO <i; CO bO S- 0) o bO O bO CD 0

03 O .rS <D oc: o (—I ‘H Cd o Cd o

C D. > •<-) O <+H Cd CO -P o -P O

cd O 0) JO cd c > CD cd s- cd s CC K O PQ M Cd Q Q CU Q (X. 15 15 20 15 15 10 10

I II III IV V VI VII

FINAL

RATINGS

BIDDERS

Agency Impact

B205 7 7

B202 6 6

B203 7 8

B206 6 5

B204 6 6

B201 4 4

Commercial

Impact

B101 8 7

B102 5 4

BIOS 4 4

8 9 7

8 8.7

7 6 6

7 5 5

4 4 4

5 5 4

8 8 8

4 4 4

4 3 3

8 7 760

6 7 695

7 8 695

5 7 575

4 5 470

4 4 435

8 5 755

5 4 425

3 4 360

FIGURE IV. 14 SUMMARY DISPLAY OF SCORES FOR THE

CONTRACT NEGOTIATOR
1^0

and in a manner which was consistent across all bidders.

In this way the important issues for use in clarification,

negotiation and debriefing were not obscured by the numbers.

IV. 12 Preparing for Orals

Preparation for face-to-face discussions with the three

remaining bidders was accomplished by the COTR, the contract

negotiator and several ETIP staff through a review of the

notes on issues for clarification which viere raised by the

factor teams. That is, on any sub-factor where the bidder

was rated at six or below or where the panel had otherwise

requested a specific clarification, the COTR went back to

the notes of the evaluation process to prepare questions

to initiate the discussions. At least one question was

written for each sub -factor where the nan el required clarifications. During this time period, telephone calls were

placed to people referenced by the bidders in their proposals as being familiar with their past performance.

With the factor team meetings completed, the

bidders' cost proposals were provided for review by both

the COTR and a cost analyst in the contracting office.

There was a considerable difference in the costs proposed

by the two remaining bidders on the agency impact contract.

The cost difference was accounted for by a number of factors

including, primarily, different total man months of effort
proposed, different costs for direct and indirect labor

(benefits), overhead, and G&A (General and Administrative),

and different travel and secretarial costs.

The task which remained for the COTR was to

improve the estimates of each bidder’s actual subfactor score (i.e., decrease the range of uncertainty),,

obtain clarification of cost estimates and consider all

of the information together to make a determination of which

offer appeared to be in the best interest of the government

(cost and other factors considered). A meeting was held

with the contract negotiator on May 12 (COTR, Note 93) to

plan the process for this review. It was decided to invite

each bidder in for three hours. They would be allowed to

make a brief presentation on their' proposal, followed by

a question and answer period on both cost and technical

information. All meetings would be tape recorded.

The bidders were called by the contract negotiator

and told that ETIP wished to meet with them for face-toface clarifications of some issues in their proposals.

They were told which sub-factors these issues related to

so that they could prepare for the meeting.

IV . 13 Orals and Written Confirmations

Bidder #203 was invided to appear at the Department

of Commerce the morning of May 25, 1976. They arrived with
six members of their oroposed team, including the project

manager and various technical area leaders.

Following initial introductions the project manager

was given the ooportunity to make a brief presentation on

the organization's proposal. He responded by discussing

the characteristics of the organization including its size,

the technical skills of its staff, and some of its areas

of prior work. He said that in responding to the format

of the RFP they were unable to bring out a comprehensive

illustration of how they might deal with an experiment and

hoped to do so during the discussions. The meeting then

proceeded with a dialogue surrounding the specific questions

which ETIP had prepared (Tape of meeting. Note 9^).

Questions were asked about both management and technical aspects of the project. The management questions ranged

from asking which senior management people would be full

time on the project to asking about the internal process

for resolving conflict among people on their team. The

technical dialogue included questions on who wrote specific

portions of the proposal to discussion about what the appropriate research techniques would be for specific evaluations

under various circumstances (Tape of Meeting, Note 95).

Bidder #202 was invited to appear on the afternoon

of May 25. They brought four members of their team including the project manager, one of the technical group leaders

and two technical staff members.
Following introductions, the project manager described

the process which was used in preparing their proposal and

discussed in very general terms what their organization

would do in carrying out the work. He also indicated that

they had some questions concerning the extent and nature

of the involvement of the ETIP staff in the project. The

meeting proceeded with discussion related to the issues

on which the evaluation panel suggested clarifications.

(Tape of meeting, Note 96).

Questions covered both management and technical aspects

of the project. Management questions included asking about

costing of various aspects of the proposal relative to the

work to be done and asking about the process of integrating

various technical specialists into the work as they are

required. Technical discussion ranged from asking where

specific expertise was within their team to questions of

what specific technical methods would be appropriate for

various potential types of evaluations (Tape of meeting.

Note 97).

Bidder #101 was invited to appear the morning of May

26: They brought six people to the meeting including a

representative of company management, the project supervisor,

the project leader, technical group leaders, and technical

staff.
The project supervisor introduced those present and

described the corporate commitment to the project. He then

turned the presentation over to the project leader who discussed what bidder //lOl saw as some of the key issues of

the work and how they generally intended to deal with these

issues. This was followed by a dialogue on the issues which

required clarification. (Tape of meeting, Note 98).

Again, this covered both management and technical matters. Management questions included asking about the level

of commitment that specific people would make to the project

and asking about how resource allocation decisions would

be made. Technical issues tended to center around the process for developing and finding a home for the commercial

impact evaluation system (Tape of meeting. Mote 99).

Following the face-to-face meetings the COTR sent the

contract negotiator written lists of questions and statements for each of the three bidders. This was to be used

to obtain written confirmation of positions taken by the

bidders or observations made by the ETIP staff which participated in the oral discussions.. The approach taken, as indicated in the memo transmitting the questions to the contract

negotiator, was to present to the bidders with both:

o A standard concerning where ETIP felt the bidder should be with respect to understanding,

approach or capability along a single or particular set of factors or sub-factors, and
1^5

o ETIP’s observations as to where the bidder

actually was with respect to this standard

(COTR, Note 100).

In other words, the purpose was to indicate clearly where

ETIP felt that the bidder still required improvement his

position.

For example, one of the questions was:

The commercial impact contract requires knowledge and experience in the area of economic

analysis at the firm level of aggregation.

It is important for the contractor to be able

to obtain access to data which is not generally

publicly available. The contractor must also

be able to analyze this data from economic

perspective. Your proposal does not evidence

this capability. Please comment on this in

terms of past experience of the individuals

who are proposed for this project with respect

to subfactors VI. 2, VI. 3, VII, 1. Limit your

response to 1 page (COTR, Note 101).

The questions were sent out by the contracting office

on June 4, and responses were requested in by the close

of business June 11, 1976. Bidder #203 was sent three questions, bidder #202 was sent nine questions, and bidder 101

was sent one question.

IV. 14 Summary of Results and Award

The COTR and two other panel members reviewed the written responses from each of the thre*e bidders, together with

the records of the face-to-face meetings and the findings

of the technical evaluation team. For each sub-factor where

an issue had been re-examined a more precise rating was

assigned by this team based on the clarifications and ad-
ditional information which were provided by the bidders.

The overall results were that bidders #203 and #101 were

both considered acceptable on all seven prooosal evaluation

factors while bidder #202, considered unacceptable on three

factors after the panel evaluation, was now considered unac

ceptable on an additional three factors. The COTR ’ s decision was therefore to recommend that bidder #101 be negotiated with for the commercial impact evaluation contract

and that bidder #203 be negotiated with for the agency impact evaluation contract (COTR, Note 102).

These recommendations were conveyed to the contract

negotiator in a memorandum on June 17, 1976. The form of

this memorandum followed the narrative form of the COTR '

May 4 memorandum, i.e., discussion of acceptability based

on factor scores of seven or greater and unacceptability

based on factor scores below seven with elaboration of mode

ate, more serious and very serious problem.s based on the

specific sub-factor scores.

The COTR sent a second memorandum to the contract nego

tiator describing additional information which needed to

be considered during final negotiations. This included:

o advising that the total sum of both contracts

could not exceed $1.8 million and that it

would be technically appropriate to award

a $.9 million contract to bidder #101 and

a $.9 million dollar contract to bidder #203*
147

o suggesting a method for negotiating on the

sub-factor issue where bidder #203 continued

to have a moderate problem, and

0 suggesting obtaining full time commitments

from two individuals on bidder #203 ’s team

(COTR, Note 103).

Best and final offers were requested on June 23, and

received by telecopy on June 24. They were reviewed by

The COTR and on June 25 he sent memos to the contract negotiator indicating that he felt the quality and quantity

of effort proposed to be reasonable and necessary for execution of the projects (COTR, Note 104, Note 105).

The contracting office then requested a written indication of the shifts which resulted in bidder #202’ s sub-factor

ratings based on additional information which was provided

from the oral and written clarifications. This listing

was prepared by the COTR and the author and sent to the

contracting office and June 28 (COTR & Libman, Note 106).

Contracts dated June 30, 1976 were signed with

both bidder #101 and #203 and work on the projects

began during the second week of July.

IV. 15 Debriefing

On July 9, the COTR and the author sent a memorandum

to those who participated on the evaluation panel. The

process which followed the evaluation panel meetings and

its results were described and panel members were thanked

for their participation (COTR & Libman, Note 107).
The contracting office received one request for a debriefing. This was from bidder #202. The COTR, the author

and the contract negotiator met with three representatives

of bidder #202 on August 4, 1976. The initial discussion

centered around the factors on which bidder #202 was found

to be less than acceptable (COTR, Note 108). The dialogue

and tone of the meeting indicated that bidder #202 disagreed

considerably with the evaluation of their technical- capability and were angry about losing when they considered

themselves more qualified than the winner (COTR, Note 109).

However, they did’not file a formal protest.

IV. 16 Summary and Transition to Chapter

This chapter has provided a chrono

ted to the procurement of two evaluatio

by the Experimental Technology Incentiv

tion was presented on issues which part

to the process and on how they were res

the actual evaluation process were pres

and anecdotes were related to give the

of how the process resulted in ratings

review and summary process was describe

process progressed from the evaluations

the signing of contracts.

The next two chapters will evaluat

_V

loey of events relan systems contracts

es Program. Informaicipants brought

olved Data f rom. ented in figures

reade r a picture

for b idders

The

d to show how the

by the panel to

e the results of

using PFE against some of the issues, which participants
in the procurement raised. The results will also be evaluated against some of the issues which are raised by the

literature which was reviewed in Chapter II. The description presented in this chapter and other data from the process located in the appendix will be used in this evaluation.

This information is presented to enable the reader to perform his own independent evaluation.
150

CHAPTER V

EVALUATION/CRITIQUE

V . 1 Introduction

The purpose of this chapter is to evaluate the outcome

of ETIP’s use of the Parametric Factor Evaluation process.

It presents an assessment of how well the PFE process worked

in solving many of the critical issues which ETIP faced

during its source selection. The issues for this evaluation

are those which were raised by participants in the ETIP

project, together with a selection of issues from the literature which the author felt were applicable to ETIP*s problem.

The format for presenting the evaluation is to provide

some general discussion of the issues and procedures for

evaluation, to discuss the outcome for each selected issue

individually, and finally to summarize the results of ETIP's

use of PFE in terms of successes, failures, and uncertainties. The general discussion of the issues indicates how

an initial set of issues were accumulated and placed into

categories and how a specific set of issues were singled

out for in-depth discussion. The technique for gathering

additional data from participants in the source selection

is described and the parameters which may affect the inter-
pretation of the results are discussed. The evaluation

of each specific issue contains a conclusion about how successful PFE was in dealing with the issue and presents the

information which the author used in reaching this conclusion. Finally the summary of the evaluation is presented

in which the issues are placed into categories of those

for which PFE appears to present a successful solution,

those which PFE appears to be unsuccessful at solving, and

those for which a conclusion cannot be confidently made

using the data available. An indication of the points at

which ETIP departed from the process as it was originally

used at Wright Field and a description of changes which

ETIP made in its own subsequent applications of the process

are presented as part of Chapter VI.

V . 2 The Issues, Additional Data, and Parameters

V.2.1 Listing of Issues

The issues presented in this section provide the basis

for evaluating the effectiveness of the PFE process in solving ETIP’s source selection problem. The list of issues,

which is presented in Figure V.1, was drawn by two methods.

The first was by examining the records of ETIP’s use of

the PFE process for questions which were raised before,

during, and after the source selection by the people who

were involved. The second was by writing down issues the

author came upon, through his readings, discussions- with

others, or reflection on the problem, which seemed appli-
1

cable to ETIP’s situation. Comprehensiveness in covering

the major issues rather than independence among the issues

was the objective. In fact, it is partly the underlying

interrelatedness among these issues which makes system source

selection a difficult problem. The listing of issues is

not meant to imply that they are necessarily of equal importance. Also, issues of significance to some source selections but not directly relevant to ETIP’s problem are not

covered here.

V.2.2 Categories of Issues

When a preliminary listing of issues was accom.pl ished

,

the list turned out to be quite long. In order to provide

some structure to this evaluation, a grouping was sought.

Each issue was sorted into a group with the other issues

to which it seemed most closely related. As this was being

done, the common elements within groups emerged and headings

were assigned. A statement was written for each heading

to try to illustrate the common elements. Issues were then

moved among these groupings if, upon re-examination, they

seemed like they fit better elsewhere. As a few new issues

were later discovered, they were placed in one of the existing groups. In some cases an issue could have fit under

one or more other headings and was assigned somewhat arbitrarily. For evaluation purposes the issue is more important than the major category under which it is grouped.
153

1 .

2 .

3.

4.

5.

6 .

7.

8 .

9.

10 .

1

1

.

12 .

13.

ISSUE CATEGORIES

Effective Communications

Sensitivity and Selectivity in Obtaining Bids

Assessment of Bidders’ Capability

Use of Experts in the Evaluation of Proposals

The Panel’s Role in Evaluation of the Proposals

Equity of Treatment Across Bidders

Risk Associated with Statement of Work

Basis for Award

Traceability of the Process

Relationship with the Contracting Office

Resource Requirements

Education of Government Personnel

Education of Bidders

FIGURE V.1 ISSUES AND ISSUE CATEGORIES
1 . EFFECTIVE COMMUNICATIONS

Did the format and content requirements of the PFE process

provide for effective communications between the government

and potential bidders and between bidders and the STIP proposal evaluation panel?

Format

+ Did PFE provide too much structure for the bidders’

response?

Is the bidder able to display his competence?

Does it present problems for the bidder or the panel

when no formal structure is used for the object

of the procurement?

+ Did the panel have difficulty evaluating the proposals

on a sub-factor basis?

Was the panel able to find the information it needed?

Did they feel that the evaluation was being done

out of context from the project?

Was the RFP poorly organized?

Was the RFP confusing?

Was the existence of two statement s-of-work confusing?

Was the RFP too long? Should its size have been limited?

+ Should the size of proposals have been limited?

(Did length correlate with final scores?)

+ Did the bidders understand what the evaluation panel

looked at in making their evaluations? Did knowing

this help?

Did the briefings allow for adequate questioning by

the prospective bidders? (differences in information

among bidders)

+ Did the process enable the panel to deal effectively

with differences in uncertainty in information among

bidders during the evaluation?

Did the question form provide an effective mechanism

for obtaining missing information and clarificationafter the proposals were received?

+ indicates that the issue is explicitely discussed in

Section V.3

FIGURE V.1 Continued
Content

+ Did the process communicate to potential bidders what

it was the government wanted to buy?

(Did bidders know what they were bidding on?)

+ Did the process communicate to prospective bidders

the skills which would be required to do the work on

the project?

Did the process succeed in explaining to bidder the

reasons why it was being used?

Was information provided to bidders on the history

of the PFE process harmful or useful or unnecessary

+ Were many of the sub-factors not useful to the bidders

or the panel (e.g., Management or Reports & Reviews

sub-factors)?

Was too much detail required from the bidders?

(Information which was not useful for effective

communication.) Does the process encourage gold

plating of proposals?

Did the process make the priorities of the government

clear?

Did the process enable the government to describe aspects of the project for which few details were known

as well as those for which there was a great deal of

detail?

Were problems caused by the requirement for the bidder

to respond to issues which are traditionally thought

to be the responsibility of the government? (e.g.,

how to rationalize multiple and conflicting objectives

of the project )

.

+ Was parroting back of information from the RFP by bidders to the exclusion of other information a problem

for the panel?

2. SENSITIVITY AND SELECTIVITY IN OBTAINING BIDS

Was ETIP successful in obtaining an acceptable number of

responsive bids?

FIGURE V.1 Continued
156

+

+

Were enough responsive bids received to allow for a confident evaluation?

Did the process enable prospective bidders to selfselect effectively on bidding?

Were a large number of non-respons i ve bids received?

Did a lot of qualified firms not bid?

Did the process help the bidder to know which people

to use in responding to the RFP?

3. ASSESSMENT OF BIDDERS' CAPABILITY

Did PFE enable ETIP to assess whether bidders had the required capabilities when there was no track record in contracting organizations, i.e., of having a large number of

people with the diverse skills and interests which were

required here to work together over a long period of time

to develop the evaluation systems.

+ Did PFE provide for an effective evaluation of management capability?

+ Did PFE provide for an effective evaluation of technical capability?

+ Did PFE anticipate the problems which were known to

exist in doing evaluations and in developing systems?

Did the criteria provide for an assessment of the

relationship between what was proposed and later

performance?

Was the award based upon proposals which would hav

some relationship to the approach subsequently use

Was the material provided by the successful contractors useful or not to the PAAs?

4. USE OF EXPERTS IN THE EVALUATION OF PROPOSALS

Was ETIP able to obtain and make use of experts in the evaluation of proposals?

+ Was it difficult to get experts (specialist and those

with prior experience with prospective bidders) to

participate when there was a large RFP and the proposals were expected to be large?

Was expert information useful in the evaluation? Did

the panel always follow the advice of experts?

FIGURE V.1 Continued

CL

(D
157

How difficult was it to identify people to put on the

evaluation team? How difficult was it to train them

in the evaluation process? How difficult was it to

motivate them to do a good job?

5. THE PANEL'S ROLE IN EVALUATION OF THE PROPOSALS

Did the Factor Team process work effectively in obtaining

agreement on ratings on all sub-factors for all bidders?

+ Did panel members who agreed to participate actually

show up?

How difficult was it to schedule the meetings?

+ Were panel members able to -arrive at initial ratings

prior to factor team meetings?

+ Was the panel able to arrive at final scores without

voting?

Was the panel able to deal effectively with strong

individual disagreements?

+ Did the panel members realize how their individual

positions affect the final score, i.e., do they see

other panel members modifying their position in response

to the discussion.

Do panel members understand how the final score

i?s reached?

+ Do the final scores differ significantly from the means

of the Initial scores of the individual panel members?

+ Were the panels ratings and recommendations used after

the panel meetings were concluded?

+ What impact did those with extreme scores or greater

variability in their ratings have on the final scores?

What affected the time to reach agreement?

The distribution of initial scores?

The specific panel members present?

+ How long did it take to reach agreement on factor scores

after the sub-factor scores were agreed to?

+ Were ’’junior” panel members able to affect the score

if their bosses were in the room and their initial

scores differed?

FIGURE V.1 Continued
15

Did panel members read the correct parts of the proposals in arriving at their ratings?

6. EQUITY OF TREATMENT ACROSS BIDDERS

Did the process provide for a fair and equitable treatment

of all of the bids?

+ Did the panel deal effectively with the biases of its

members?

+ ^ Did the criteria used in evaluation provide for a consistent evaluation across proposals? Were the criteria

applied consistently across prooosals?

Were any problems caused due to the lack of a requirement on the bidders to respond to all of the illustrative questions?

+ Did the lack of restriction on admissible information

lead to any inequity in the process?

+ Did evaluation of the proposals while knowing the identity of the bidders create any problems - biases in

the final ratings, protests?

7. RISK ASSOCIATED WITH THE STATEMENT OF WORK

Did PFE enable ETIP to deal effectively with the "added

risk" associated with a flexible statement of work.

+ Was ETIP able to write a statement of work which would

spell out deliverables and yet avoid numerous contract

changes which generally occur in situations where requirements, policies, and the state of the art are

changing?

Did ETIP management accept it?

Did the contracting officer accept it?

+ Did the flexible statement of work scare off bidders?

+ V/as ETIP able to assess the commitment of the bidders'

top management to devote the necessary resources to

the project?

Was ETIP able to determine whether it was within the

longer term interest of management to develop the desired systems?

FIGURE V.1 Continued
159

Did PFE deal with the lack of incentive for cost control in CPFF contracts?

Would PFE allow ETIP to alter the timing and level

of effort of the project after the contract was signed?

8. BASIS FOR AWARD

9 Did use of PFE provide a basis for discriminating among

bidders on criteria other than price in determining who

to negotiate with?

+ Did PFE provide for overall technical discrimination

among bidders? Was there a sufficient distribution

in the final scores to determine a competitive range?

+ Did PFE provide a basis for discriminating among bidders

on specific technical issues? i.e., Did the specific

issues get washed out by the overall technical score?

How much of a factor was schedule in the final decision?

Did PFE provide a capability for meeting requirements

to allow a bidder to improve his proposal?

+ Was technical leveling a problem?

+ Was technical transfusion a problem?

Did the process provide the contract negotiator with

the information he needed to negotiate a contract award?

+ How much of a factor was bid price in the final decision? Did the procurement turn out to be a price competition? Did the process provide a capability for

dealing with potential buy-ins?

9. TRACEABILITY OF THE PROCESS

Are the results of the evaluation traceable so that an independent examination of the record would reconstruct the

bases for the final selections?

+ Did the process communicate that there was no sole

source position? Can the process be fixed?

+ Did bidders understand how the decisions on who wins

actually get made?

FIGURE V.1 Continued
+ Can changes in scores be traced between the initial

ratings of panel members and final panel ratings and

between final panel ratings and the award decisions?

-I- Did the panel follow the written procedure?

+ Did leaving the relative weightings of the sub-factors

up to the teams lead to any problems (e.g., appearance

of inequities)?

10. RELATIONSHIP WITH THE CONTRACTING OFFICE

Did the contracting office feel vulnerable to award protests

by using PFE?

+ Is PFE legal under the Federal Procurement Regulations?

+ Did the contracting officer feel that the selections

made by using the process would be defensible if there

were a GAO investigation?

+ Did the contract negotiator accept the evaluation of

the panel? Did he want the panel to revise their report? Did he want another level of technical review

of the panel’s evaluation?

+ Did using PFE result in timely awards?

+ Were there any protests of the award?

+ Were any debriefings requested?

11. RESOURCE REQUIREMENTS

What level of resources over what interval of time were

required to effect the procurements?

+ How much time was required to do system or evaluation

design work prior to issuing the RFP?

o elapsed

o total time of various people

How much time did it take to write the RFP?

o elapsed

o total time of various people

+ How much time did it take for bidders to respond to

the RFP? What was the volume of material submitted

(as a surrogate for cost of responding)?

FIGURE V.1 Continued
I6l

+ How much time did it take for the panel to evaluate

the proposals? How much time did it take to read the

proposals? How much time did the panel meetings (and

preliminary meetings) take? How much time would oral

presentations have taken?

+ How long did it take to make an award once the panel's

evaluations were complete?

How much time did it take to write up the results

of the panel meetings?

How long did it take to prepare for oral clarifications? How long did it take to request written

confirmations of oral clarifications? How long

did it take to reach a decision once the written

clarifications were received?

+ How long did it take to prepare for the debriefing?

Would the same decision have been made by using another

method? Does another method deal as effectively with

the same issues for less effort?

12. EDUCATION OF GOVERNMENT PERSONNEL

Did use of PFE result in education of government personnel

who would be involved in the evaluation system development

projects?

+ Did use of PFE result in education of government personnel so that they would be able to effectively evaluate

the bidders’ proposals?

+ Did use of PFE result in education of government personnel so that they would be better able to work with

or manage the resulting contractors?

Were program participants willing to be involved

in the procurements?

13. EDUCATION OF BIDDERS

Did use of PFE result in education of bidders useful to

their future interactions with ETIP?

+ Did use of PFE result in training, learning, and education of the successful bidders which was useful for their

performance on the contracts?

o early performance

o later performance

Did use of PFE result in education of unsuccessful

bidders which was useful in bidding on other ETIP RFPs?

FIGURE V.1 Continued
The categories serve mostly as a structure for the discussion which follows in Section V.3*

V.2.3 Selection of Specific Issues for Discussion

Equal treatment of all of the issues was thought to

be neither desirable nor possible within the scope of this

case study. It was therefore necessary to make some decisions concerning depth of evaluation for each issue. There

were two criteria for these decisions. They /were: 1) the

relative importance of the issue to the problem.; and 2)

the availability of data to support an evaluation of the

issue. The' issues treated most extensively are those which

were considered most important and for which data was avail

able or obtainable. The issues treated least extensively,

or ignored, are those which were considered less i.mportant

and for which no data was available or reasonably obtainable . V.2.4 Additional Data

At the time when the specific issues were selected

for evaluation, the data which were available to support

the evaluation included the history of the PEE process,

Thompson's written description of the process, and the writ

ten records and the author's recollections of ETIP's use

of the process which were used to construct the chronology.

The process of selecting issues helped to highlight those

for which this data was insufficient to enable presentation
of conclusions which represented anything more than the

author’s own beliefs. Questionnaire items were written

for those issues where it seemed that additional information

could be obtained from other participants in the process

which would enable a more confident evaluation of specific

issues

.

The items were divided into a questionnaire for panel

members and a questionnaire for representatives of bidders’

firms. Each questionnaire was then pilot tested. The panel

questionnaire was given to a person who was familiar with

PEE through his participation on later panels where ETIP

used the process. The bidder questionnaire was given to

two people in a firm which had bid on subsequent source

selections where ETIP used the process. Some minor modifications were made in the wording of several items and additional background material was included with the questionnaire as a result of the pilot test. The test also indicated that response to the bidder questionnaire is sensitive

to the person in the firm who answers the items. The two

individuals in the same firm who were both closely involved

in preparing their firm’s proposal had divergent responses

1 to some of the items.

1 This served to highlight the importance of knowing

the values and beliefs of individuals and what role they

will have in management of the system development. Firms

are not monolithic in the way they view a given problem

and thereby in the way they manage an approach to solving

it

.
The method for administering the items combined a ques2

tionnaire and interview format. Questionnaires were mailed

to participants together with some background information

on the source selection (a list of the factors and subfactors and a list of other material which had been included

in the RFP ) to help remind them of the process. The bidder

questionnaire was mailed to the person in the firm who appeared to have managed the firm's response to the RF?.

The cover letter indicated that the author would phone to

•3 provide clarifications and to obtain their responses.*'

Participants were then interviewed after having at least

a week to review the items.

During the interviews each item was reviewed with the

respondents. For the yes/no answers, people frequently

provided additional information to justify or qualify their

responses. After all of the questionnaire items were answered, the author asked two additional questions. One

was whether it was difficult to recall the events and their

reactions related to the items they were being asked about.

The other was whether there was anything which they had

not already been asked about which stood out in their mind

2 Several were hand delivered.

3

One participant, who was called in advance to obtain

his new address, asked that additional background information be sent with the questionnaire. In this case, the

set of illustrative questions was also included with the

sub-factors

.
concerning the source selection process. Copies of the

questionnaires and cover letters are presented in the Appendix.

Response to the questionnaires was very good. All

panel members were interviewed and all but one bidder representative were interviewed. The one bidder was traveling

quite extensively during the three weeks the author tried

to speak with him and could not be reached.

Of the seventeen panel members, fifteen provided full

responses and two provided partial responses. The two who

provided partial responses indicated that:

o I didn’t remember participating and would be giving

reactions to your current description of the process ;

o I remember that it was an interesting day and that

the proposers that came to the top we all agreed

and they were indeed the best.

The first had participated on four sub-factors and did not

answer any of the questionnaire items. The second had participated on two sub-factors and only answered several of

the questionnaire items. All six out of the seven representatives of bidders firms who were interviewed provided

full responses.

V.2.5 General Parameters to the Evaluation

When interpreting the data to determine the effectiveness of PFE in dealing with the specific issues, it is impor4 One panel member mailed his questionnaire back and

was therefore not asked these questions.
tant to consider parameters which might lead to plausible

rival interpretations. A summary list of general parameters

is presented in Figure V.2 and discussed in the next several

paragraphs

.

o Memory

0 Subsequent Experience with the Systems Development

o Subsequent Experience with the PFE Process

o Other Related Source Selection Experience

o Outcome of the Source Selection

o Ego Issues Related to Arguments About What To Do

o Lying by Respondents or the Author

FIGURE V.2 GENERAL PARAMETERS TO THE EVALUATION OF THE

OUTCOME OF ETIP’s USE OF PFE

V.2. 5.1 Memory . One of the main parameters is memory.

Much of the data used in this case study is it is based

on the recollections of various participants in the source

selections and was collected three to four years after the

events occurred. Present recollections of facts and prior

opinions may contain errors due to difficulty of recall.

Respondents were asked how much of a problem recall

was, in order to obtain their self-report on this parameter.
167

Of the fourteen panel members who responded to this item,

three said that memory was a problem, seven said that it

was not, and four said that it presented some problems.

Of the six bidders, one said that memory was a problem,

four said that it was not, and one said that it presented

some problems. Figures V.3 and V.4 list additional responses

under each of these categories.

It should be noted that a response of ”no" does not

necessarily mean that a respondents* recall was always accurate. The author found factual errors in the recall of

at least one panel member who reported that memory was not

a problem.

V.2.5.2 Subsequent experience with the systems development . Many of the participants in the source selection

subsequently had a role in working on the development of

the evaluation systems. This is true for some of the panel

members, the contract negotiator, the author and, of course,

the successful bidders. Responses by these people concerning the effectiveness of the PFE process in dealing with

some of the issues of source selection may be mediated by

these later experiences. No data was collected to try to

measure participants on this , parameter

.

V. 2 . 5.3 Subsequent experience with the PFE process . ETIP has used the PFE process on two subsequent procurements.

One each in 1977 and 1978. Some of the participants in
1 /:

Was recall a problem in responding to the questionnaire

items?

Yes - 3 No - 7 Somewhat - 4

Remarks by those answering Yes

o Remembering was a problem, particularly memoriesof specific exchanges, circumstances, and my own reactions (PM5).

o Memory was quite a problem. The reprint of the factors

and sub-factors helped (PM9).

o Without seeing a proposal it was very hard to remember

(PM4)

.

Remarks by those answering No

0 No, but these were reflex answers. I might have answered

differently if you asked closer to the events (PM11).

o No, since I had some notes from the experience to help.

Otherwise it would have been more difficult (PM1).

o No, it stuck in my mind because it was unusual (PM15).

o Not a major one. It made quite an impression on me.

It was the most elaborate, largest exercise of this type

I'd ever been through. There may have been a lot of specific points I would have remembered if it weren't four

years ago. What stands out are the elaborateness of the

process, the fact that I didn't find that to be a problem,

the interaction among the people who reviewed each section

was very constructive, and the overwhelming sense that

it was an inappropriately written RFP (PM3).

Remarks by those answering Somewhat

0 My answers were more on impressions than on substance.

I do not remember any of the substance of the proposals

(PM10).

o My overall impressions of strengths and weaknesses are

accurate. Some things were a little hard to remember

(PM6)

.

FIGURE V.3 PANEL MEMBERS' SELF-REPORTS ON PARAMETER OF RECALL
o I remembered fairly well, even specific deliberation.

It wasn’t easy though. I have a moderate amount of memory

of it (PM7).

0 It wasn’t as bad as I thought it was going to be, but

it was still somewhat of a problem. Since I haven’t been

on other panels it made it easier (PM16).

FIGURE V.3 Continued

Was recall a problem in responding to the questionnaire

items?

Yes - 1 No - 4 Somewhat - 1 Remarks by person answering Yes

0 As I thought about it more items of memory seemed to be

coming back to me (BR1).

Remarks by those answering No

0 If you had not sent me the material I would not have felt

as comfortable with my answers (BR4).

0 Not in relation to the questions you posed, given the

appendix you attached (BR6).

o Not the least bit. It was a big job to propose on it.

I didn’t even have to refer to the extracts which you

included (BR3).

o I’ll never forget that RFP. It was quite different (BR2).

Remarks by person answering Somewhat

o Not my feelings about the process. What we wrote, I don’t

remember (BR5).

FIGURE V.4 BIDDER REPRESENTATIVES’ SELF-REPORTS ON

PARAMETER OF RECALL
the 1976 source selection participated in one or both of

these later procurements. The author, for example, was

closely involved in writing project plans, writing portions

of the RFPs, and coordinating the evaluation panel meetings

for both of the later procurements. The same contract nego

tiator was responsible for the 1977 procurement. One of

the bidders on the 1976 source selection also bid in 1977.

Some of the same panel members took part in one or both

of the later procurements. The responses by these people

concerning the 1976 source selection may be influenced by

their experiences in these later procurements. No special

data was collected to try to measure the effect of this

parameter on responses.

V.2.5.4 Other related source selection experience . Experiences of participants in system acquisition, and specifically source selection, probably effects their reaction

to the PEE process and therefore their answers to the questionnaire items. If a respondent has no system.s or related

experience, they may not understand some of the problems

which the process was trying to address. Two items were

included in the questionnaires in order to obtain an indica

tion of experience in source selection. One asked how many

RFPs/evaluat ion panels the respondent had responded to/serv

on. The other asked how many of these were for projects

of similar size, complexity and uncertainty to this one.

Definitions of complexity and uncertainty were given in
171

the background note at the beginning of the questionnaire.

These terms were used to describe characteristics of systems

related experience. It is possible that respondents interpreted them differently than the author intended. The responses to these items are presented in Figures V.5 and

V.6. As might be expected there is quite a bit of variation

in experience among respondents, both panel and bidder.

V.2.5.5 Outcome of the Source Selection .

* Another

parameter to consider is the effects of the outcome of the

source selection. Responses to subsequent questions about

the effectiveness of PFE in solving specific problems may

be partially a function of who won. A bidder who lost may

find the process less effective than one who won. A panel

member who disagreed with the selection of the successful

bidder may be biased against the process and visa versa.

No specific measurement was attempted to try determine the

extent to which this affected responses.

V.2.5.6 Ego issues related to arguments about what

to do . Chapter IV indicated that there was heated debate

among some of the panel members concerning many of the events

related to the procurement. Some involved the selection

of PFE as the method which would be used for the source

selection. There were also many clashes, after the signing

of the contracts, concerning the development of the systems.

It is possible that the responses of some of the panel mem-
172

Approximately how many other

evaluation panels have you

served on?

Of the total number of

other proposal evaluation

panels which you have served

on, about how many were

for projects of the m.agnitude (i.e., 200 man-months),

complexity, and uncertainty

of this one?

1 /O (PM10

)

3-4/3-^ (PM1 1

)

10-11/3-4 (PM12)

20/8 (PM1 )

10/7 (PM13)

35/5 (PM2)

30/3-4 (PM50)

10/0 (PM14)

6/3 (PM9 ) 0/0 (PM6)

25/5 (PM7)

2/2 (PM8)

6-7/1 (PM4)

3/0 (PM15)

6/0 (PM3)

0/0 (PM16

)

FIGURE V.5 PANEL MEMBERS' SOURCE SELECTION EXPERIENCE
Approximately

RFPs have you

how many other

responded to?

Of the total number of

RFPs which you have responded to, about how many

were for projects of similar size (i.e., 200 manmonths), complexity, and

uncertainty as this one?

150/20 (BR5)

600-1000/175 (BR4)

100/25-50 (BR6)

100/less than 12 (BR3)

30/2-3 (BR2)

30/5 (BR1 )

FIGURE V.6 BIDDER REPRESENTATIVES’ SOURCE SELECTION

EXPERIENCE

bers may have been influenced by ego related issues such

as the degree to which they felt involved in the development

of the RFP and the degree to which they perceived they had

influence in various parts of the systems developments.

Though no specific measurement was attempted here, it is

unlikely that this parameter affected the responses of more

than two or three panel members.

V.2.5.7 Lying . A final parameter to consider here

is whether the respondents are telling the truth. For responses from bidder representatives this may be mostly a

function of whether the person who knows the truth was the
In a large RFP many difone who answered the questions,

ferent people are involved in preparation of the proposal.

The answer to a question is partially a function of whether

the author located the person who was responsible for the

aspects of the proposal related to the issue under question.

For the bidders, telling the truth may also be a function

of their expected future with ETIP. If they feel that their

responses may affect this, they mav portray themselves as

more favorable than they actually are. For the panel members (including the author) and for the contract negotiator,

there is the possibility that the truth is being altered

by the desire to not admit that anything might have been

done incorrectly. This is part of a well known phenomenon

in government circles which is known as "covering your ass."

The responses and interpretations of the author may also

be suspect on the grounds that he would like to complete

his degree requirements. Since his committee chairman is

the primary developer of PFE, his ability to be candid may

be questioned by some readers. Responses of at least two

of the panel members, the COTR, and Thompson might also

be questioned on the basis that they have a stake in the

success of PFE and are favorably biased toward it.

The author’s impression was that all of the bidder

representatives who were interviewed were closely involved

in the preparation of their firm’s proposal in response
175

to the RFP. It also seemed that the respondents were quite

I candid in their responses to the questionnaire items. However, as one respondent was careful to point out, the answers represented his personal views and not necessarily

j those of the firm. Others who worked on the proposal may

have had quite different responses.

[ The author believes that the panel members were equally

candid in their responses. In part this may be due to the

fact that most of these people no longer have any relationship with the resulting projects or with ETIP. Some who

were outside evaluators never had a close relationship with

the project and others who were ETIP or PAA employees have

since taken new jobs. No special steps have been taken

to control for the responses of the author, the COTR, or

Thompson. However, steps have been taken to try to compensate for biases which may affect the author’s interpretations of data. An attempt has been made to provide the

reader with as much unfiltered information describing the

process and data from the process as possible. This has

I been done through including:

o the history of the process (Section II. 4)

1 o the chronology of ETIP’s use of the process

(Chapter IV),

) o prior documents from and about the RFP (Appendix),

1

I
o respondents' elaborations on their responses

to the questionnaire items (Figures in this

chapter ) , and

o the actual ratings from the evaluation process

(Appendix )

.

It is hoped that this information will enable the readers to form their own evaluations on both the issues treated

extensively by the author and on those issues which readers

feel are important but which the author has selected to

treat in less depth.

V .

3

Evaluation of the Parametric Factor Evaluation Approach

Against the Specific Critical Issues

V.3.1 Introduction

This section is divided into sub-sections according

to the category groupings of the issues. For each subsection the category is described and the issues which are

evaluated within it are listed.

The paragraphs in each sub-section which follow the

category description are used for the evaluation of the

issues within that category. Each issue is identified and

its significance is noted. A statement is then made about

whether the information available indicates that ETIP was

successful or unsuccessful at dealing with the issue by

using PFE or whether the data are equivocal. The information used in reaching this conclusion is then described.

In some cases this information will be from the analysis
of questionnaires sent to participants in the process or

analysis of the factor teams’ ratings. In other cases the

author merely reports the results as he was able to observe

them through his participation in the process. An identify

ing number was assigned to each of the respondents so that

readers may compare their remarks across items when reviewing the Figures in this section.

V.3.2 Effective Communication

The category of effective communication encompasses

issues dealing with both format and content requirements

of the RFP and the proposals. The issues are concerned

with whether PFE provided for effective two way communications between the government and potential bidders during

the source selection. That is, whether the information

provided by the government and the form in which it was

provided allowed bidders to assess what needed to be done

and whether the responses and the form in which they were

required enabled the evaluation panel to accurately determine the capabilities of the bidders.

Eight issue discussions are included under this category. Four are primarily related to format of communications and three are primarily related to content of communi

cations. The issues are:
Format

i?e

1.

a) Did PFE provide too much structure for the

bidders' response?

Is the bidder able to display his competence?

Does it present problems for the bidder

or panel when no formal structure is used

for the object of the procurement?

b) Did the panel have difficulty evaluating

the proposals on a sub-factor basis?

Was the panel able to find the information

it needed?

Did they feel that the evaluation was being

done out of context from the project?

2. Should the size of the proposals have been

limited?

Did length correlate with final score?

3. Did the bidders understand what the evaluation panel looked at in making their evaluations?

Did knowing this help?

4. Did the process enable the panel to deal

effectively with differences in information

among bidders during the evaluation?

Did the question form provide an effective

mechanism for obtaining missing information

and clarification after the proposals were

received?

Content

l.a) Did the process communicate to bidders what

it was the government wanted to buy?

Did bidders know what they were bidding

on?

b) Did the process communicate to prospective

bidders the skills which would be required

to do the work on the project?

Were many of the sub-factors not useful to

the bidders or the panel (e.g., the Management or Reports & Reviews sub-factors)?

2 .
3. Was parroting back of information from the

RFP by bidders to the exclusion of other

information a problem for the panel?

V. 3 . 2.1 Effectiveness of the structure . The first

format issue is whether PFE provided too much structure

for the bidder’s response. The primary concern here was

that constraining bidders to follow the sub-factor structure

would lessen their ability to effectively demonstrate their

qualifications. The corollary of this concern was that

the sub-factor structure would interfere with the panel's

ability to evaluate effectively the capability of the bidders. The aspect of PFE which is salient to this issue

is that it did not provide for information exchange in terms

of the object or program to be procured. Bidders were not

asked to propose a design for the evaluation systems. They

were asked instead to respond to the problems which the

government thought would arise in developing such a system.

At least one panel member voiced concern during a team meeting that this reduced the bidders’ ability to demonstrate

their capabilities.

The structure was generally well received by both the

panel and the bidders. A question with three sub-items

was included in both the panel and the bidder questionnaires

to obtain reactions to the issues related to the structure.

The items and responses to them are presented in Figures

V.7 to V.12. Figure V.13 lists responses to the open ended
questionnaire items which pertain to structure. The panel

members generally felt that:

o the structure enabled bidders to demonstrate

their ability to deal with the critical problems of the project (Yes - 13; No - 1 ; Yes

& No - 1 ) ;

0 the structure did not create any critical gaps

in evaluating bidders' capabilities (Yes it

created gaps - 4; No, it did not create gaps

- 10; Don't know - 1 K’ and

0 information relevant to the issues being evaluated was readily located in the proposals (Yes

-14 ; No - 1 )

.

One aspect of structure which came out in the response

of some panel members is the tradeoff between consistency

of evaluation across proposals which occurs when using the

PFE structure, and the possibility of missing some insights

which might have been presented by bidders under a less

structured format. In fact one panel member felt that the

structure was both the strongest and the weakest aspect

of the process (see Figure V.13). The author believes that

this tradeoff needs to be examined as a function of what

the government is trying to procure. If the object of the

procurement is uncertain and complex, a structure which

tends to be free of gaps and consistent across proposals

is probably more appropriate than one which evaluates proposals for solutions which may be inappropriate under a changed environment.
Did you feel that the factor/sub-factor structure enabled

bidders to demonstrate their ability to deal with critical

problems which might arise during the project?

Yes - 13 No - 1 Yes & No - 1 Remarks by those answering Yes

None

Remark by person answering No

o It confined bidders to a tight recipe for how to answer

and did not provide them with the ability to innovate.

It didn't demonstrate their ability to deal with the

critical problems. You only found out which contractor

could write best. They could demonstrate that they

could identify problems but I don't think they could

show whether they could deal with the problem (PM4)

Remark by person answering Yes and No

o Yes, the structure covered it and allowed bidders to demon

strate their ability.

No. it got too refined. Too many sub-factors. The kind

of interconnection between judgments gets lost. There

were too many factors and sub-factors by a factor of about

two. The result of the judgment was not necessarily the

sum of the parts (PM9)

FIGURE V.7 PANEL MEMBERS' RESPONSES TO QUESTION la

ON THE EFFECTIVENESS OF THE STRUCTURE

Did you feel that the factor/sub-factor structure enabled

bidders to demonstrate their ability to deal with the problems which might arise during the project?

Yes - 4 No - 2

Remarks by those answering Yes

0,1 thought it was too structured (BR2).

0 It was fairly well thought out. It covered all of the

points quite well, to the point of exhaustion. There

seemed to be some repetition (BR1).

Remarks by those answering No

o I think the structure stunk. The approach is suitable

for hardware not program evaluation and/or experimental
design. When dealing with program evaluation you are

dealing with nominal data at best and therefore the summation of nominal data might not hold. It also forced peopl

to prepare proposals in a non-research way. One that

would be used for an RFQ, not an RFP. The project was

poorly defined and so were the parts of the proposal.

The documents loosely pointed to the evaluation but there

wasn’t a clear set of program or project objectives (BR5).

o We were trying to create objectivity where the problems

arising and the sensitivities that had to be appealed

to were things which were subjective. The desire to be

objective is to be applauded. However, we tried to create

objectivity where it didn’t exist. That’s not saying

it wasn’t a good RFP (BR4).

FIGURE V.8 BIDDER REPRESENTATIVES' RESPONSES TO QUESTION la

ON THE EFFECTIVENESS OF THE STRUCTURE

Did you feel that there were critical gaps in evaluating

the bidders’ capabilities to perform effectively on the

project?

Yes - 4 No - 10 Don’t know - 1

Remarks by those answering Yes

o It’s strictly a feeling only. I don’t remember what they

were. I was somewhat uneasy about how well I understood

it ( PM1 6 )

.

o Some of the sub-factor statements seemed too theoretical.

I’m not sure it’s a result of the structure. For example,

sub-factor 1.2. You can’t measure that sub-factor as

it is stated. It needs to be broken down further to make

it operational (PM1).

o The structure of the factors and sub-factors was too complicated. Some elements of the required skills were included in more than one factor, or at least the possibility of this seemed to exist. The large number of subfactors meant the distribution of points was too great

to enable sufficient weighting of the most important.

Although this was not necessarily the case, it had a tendency to work out that way (PM2).

0 The evaluation criteria and the RFP were written from

an academic, formal, research design perspective that

rewarded firms who could respond as if this were an exam-
183

ination in graduate school, rather than an opportunity

to develop a process for program evaluation in the real

world. Thus, a bidder’s recognition of the constraints

of reality of evaluation was not given much, if any, weight,

even if such considerations were present in the proposal.

The RFP discouraged bidders from including such discussions (PM3).

Remark by a person answering No

o I don’t remember how the proposals were written so I don’t

know if there were any gaps. I guess there probably weren’t

any critical gaps (PM4).

Remark by person answering Don’t know

o Any structure presents problems of addressing questions

at interfaces between sets of factors/sub-factors. I

don’t remember whether I sensed that with this evaluation.

What most of us do in evaluating complicated proposals

is form an overall impression and then segment it. This

process assured that we all segmented it in the same way.

It had the advantage of consistency and the disadvantage

that if we had done it differently we might have received

some added insight (PM5).

FIGURE V.9 PANEL MEMBERS’ 'RESPONSES TO QUESTION 1b ON THE

EFFECTIVENESS OF THE STRUCTURE

Did following the factor/sub-factor structure create any

problems for your team, in trying to demonstrate its capability, which would not have been present if a different

structure had been used?

Yes - 4 No - 2

Remarks by those answering Yes

o It was a problem of trying to prepare a proposal to conform to an evaluation structure rather than the design

of the project. It was designed to facilitate the giving

of scores to be added up and although I can understand

the desire to do that, from the reviewers’ point of view,

I don’t see that as the way to present an approach to

a problem. An alternative would be to post a sample task

and evaluate everyone on their response to it. People

would be bidding on the same job or at least the same

task (BR5).

0 It was a new structure. The biggest problem was figuring

out what you meant. You came out with a glossary. The
definitions were not clear enough. They were generally

too short. On sub-factor 1.2 (Importance of Program to

Bidder), you can’t do this with a large corporation.^

If it’s a large corporation, it’s going to do it. If^

it’s a new corporation, it’s going to go beyond what it^

said in the proposal. The point is that you really can't

measure that. It is subjective, not objective (3RU).

o The cost to develop it was rather exorbitant for a small

firm. Following the structure was not difficult. It

didn’t create problems^to understand it. The relationship

of the factors and sub-factors was excellent (ER3).

o There was repetition which made it very tedious and discouraged people. It was too highly structured with apparent repetition of things (BR2).

Remarks by a person answering No

o It forced us to go through each of the points you wanted

us to go through. The redundancy may have inhibited our

style of writing. On some of the points we felt that

we didn’t have much to say or that they didn’t deserve

to have much said about them, but that may have been an

indication of some areas in which we were lacking (3R1).

FIGURE V.10 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

lb ON THE EFFECTIVENESS OF THE STRUCTURE

Was your team in any manner placed at a competitive disadvantage because you had to follow the factor /sub-factor structure?

Yes - 2 No - 3 Don’t know - 1 Remarks by those answering Yes

o It inhibited market entry for smaller firms to bid on

a larger project, because of the manpower required (3R5).

o Because we are small and we try to put our best foot forward. Small companies are not as able to bear the cost.

It took a hell of a lot of work, research, and coordinat ion (BR3 )

Remarks by those answering No

o Because everyone had to go through it. It created an

element of confusion and unnecessary concern because of

its newness (BR4 )
Remark by person answering Don’t know

0 Those people who evaluated the submittals would be able

to tell better. At the time I don’t think we had any

feelings of being placed at a disadvantage (BR1).

FIGURE V.11 BIDDER REPRESENTATIVES' RESPONSES TO QUESTION

1c ON THE EFFECTIVENESS OF THE STRUCTURE

Were you able to readily locate the information in the proposals which was relevant to the issues you were evaluating?

Yes - 14 No - 1 Remarks by those ansv/ering Yes

o In most cases. Some bidders followed the directions better than others (PM6).

o There may have been one or' two exceptions. It was a matter of the bidder not following directions (PM7).

o However, sometimes it took quite a bit of reading when

there was extensive cross-referencing (PM8).

o If the bidders did it right (PM4).

Remark by person answering No

o Some of the bidders didn’t follow directions. In crossreferencing if you follow it to where they referenced

and read that section, they still haven’t dealt with what

they should have. I don’t think the problem was in the

instructions in the RFP (PM1).

FIGURE V.12 PANEL MEMBERS’ RESPONSES TO QUESTION 1c ON

THE EFFECTIVENESS OF THE STRUCTURE

Responses From Panel Question 5 on Strongest Aspect of the

Process

o The common structure for the evaluation. It was a good

structure. Without more thought or experimentation I

can’t say whether it was the best possible structure (PM5).

o The structure provided by the factors simplified the process for evaluating and comparing proposals (PM6).
0 The factors. They covered many of the essential aspects

and having them specifically laid out and paid attention

to was the great strength (PM9).

Responses From Panel Question 6 on Weakest Aspect of the

Process

0 The common structure perhaps inhibited generation of insights which might have contributed something of value

(PM5)

o It was a little like a college exam. You lead them too

far along so you're not sure how much is their writing

what they think would be a successful response vs. their

actual capability (PM10).

Responses From Panel Question 10 on Any Other Aspects of

the Process Which You Wish to Get Out

o At the time I had an instinctive reaction that the process

was overstructured. I still don't know whether or not

it was. Defining the dimensions and defining the factors

seems hard but worthwhile (PM5).

o For a million dollar or more proposal you would need something like this to be able to find things. If it were

$500,000 or less I wouldn't bother with this technique.

It would be overkill. The structured approach seemed

better for guiding the reviewer than for letting the proposer show his ingenuity. Another question is what the

process did to keep people from wanting to bid (PM4).

Responses From Bidder Question 10 on the Strongest Aspect

of the Process

o Concern for the structural process. The pre-bidders conference. Giving a lot of thought to weights (3R2).

o The need to specify ideas and expertise in all of the

sub-groups which were specified. It caused us to test

within ourselves whether that detail met the objectives

of the project as we understood it (BR1).

Responses From Bidder Question 1 1 on the Weakest Aspect

of the Process

o The weakest element at the time the proposal was written

was the confusion of the new system which we had to respond to (BR4).

FIGURE V.13 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO STRUCTURE
187

The bidders were more divided in their reactions to

the structure. They tended to feel that the structure enabled them to demonstrate their ability to deal with the

critical problems of the project (Yes - 4; No - 2). They

tended to feel that structure created some problems in trying to demonstrate their capability which would not have

been present under a different structure (Yes - 4; No -2).

They were divided on whether they felt that they were placed

at a competitive disadvantage by the structure,

A remark which one bidder representative (BR4) made

in his answers to the questions on structure, and in answers

to some of the other questions, is that the evaluation tried

to measure subjective things in an objective manner. He

cites sub-factor 1.2 (Importance of Program to Bidder) as

an example, saying that it cannot be measured. The respondent seems to treat objective and subjective as dichotomous

variables. The author believes they are ends of a continuum.

Nowhere does PFE claim that the measurements of bidders

on sub-factors are at the objective extreme of the continuum.

The method of measurement is for the panel to agree on subjective probability estimates through sharing all available

informationj arguing about the confidence in this information, and using the same criteria (understanding, proposed

approach, and present or potential capability) for all bidders. The degree of subjectivity may vary from sub-factor
188

to sub-factor and the confidence intervals around the probability estimates on each sub-factor may vary from bidder

to bidder. This is true for all proposal evaluation. ?FE

makes confidence in the ratings explicit. Where the confidence interval is wide enough so that a factor team feels

that a bidder may actually be unacceptable on a sub-factor

where his score is acceptable (and vice versa) the team

notes this for use in clarification sessions.

Under PFE, if an issue is considered critical to the

success or failure of the project, it is written down as

a sub-factor and an attempt is made to measure it. If the

issue is not measureable, then it cannot be used as a basis

for competitive selection (with PFE or any other method).

It is a risk which applied across all bidders and the project (Thompson Note 1). The program must then determine

whether it is willing to assume this risk and award a contract.

Sub-factor 1.2, on Importance of Program to Bidder,

was considered to be one of the most critical issues of

the evaluation. Because of the uncertainties invo.lved in

the program, including changes which would certainly occur

in ETIP and PAA management and staff, it was important that

the contractor have a continuing long-term interest in the

implementation of the systems. This' was considered to require evidence of an interest beyond the norm of the bidder
assuring that he would do a good job because the firm does

a good job on all of its contracts. Such a commitment may

affect issues such as contractor staff stability, access

to the best staff in the firm, and the willingness of the

contractor to work toward the long term requirements of

system implementation, rather than responding to short term

variations in government objectives.

This issue proved troublesome but possible to measure.

The Management Factor Team for Agency Impact was the fourth

team to meet. After deliberation on the information available the team decided that it was unable to differentiate

among the bidders and that clarifications would be required

on this issue with any bidder who appeared to be possibly

acceptable overall. All bidders were rated at six. As

the team meetings continued, information was occasionally

brought out which related to sub-factor 1.2 (e.g., see Section IV. 10.7, paragraph two). This information was available for use in the oral clarifications and for use in rating firms who also bid on the commercial impact contract.

Attention was paid during the oral clarifications to remarks

which related to the firms’ commitments to developing the

systems. When all available information was considered,

ETIP was able to differentiate among bidders on this issue.

5 When the Management factor team met for Commercial

Impact (the 6th factor team on that contract) it was able

to find a basis for differentiating among the bidders on

that contract.
190

The author feels, from recalling the discussion at the time,

that the confidence intervals around these estimates were

greater than for some of the other sub-factors, but that

there was definitely evidence for differentiation.

Bidder representatives' responses to the question on

whether the structure created any special problems (Figure

V.10) did not show any patterns. Four respondents indicated

that it did create problems. One said that trying to understand what ETIP wanted was a problem. Another felt that

there was repetition which caused problems. A third felt

the "relationship of the factors and sub-factors was excellent 'but' the cost to develop it was exorbitant." The

fourth said it was difficult to prepare a proposal to conform

to an evaluation structure but didn't specify why.

Two bidders felt that their team was placed at a competitive disadvantage by the structure (Figure V.11). They

felt that the process was too costly for small firms to

respond to. This is clearly an issue the government must

consider in determining whether or not to use PFE. The

author has heard estimates of cost to respond placed between

$10,000 and $20,000. This must be weighed against the type

of project and the total estimated cost of the contract.

If a less costly process will enable sufficient confidence

in the selection of a contractor, given the nature of the

contract, then PFE should not be used. If the size of the
contract is too small to enable bidders to invest the amount

required to respond, then the government must decide whether

other available procedures will provide sufficient confidence to allow the government to assume the added risks.

A final point about the structure of PFE is its strong

interrelatedness. Morris (1973) indicates that RFPs are

frequently difficult to respond to because the various parts

do not mesh. The evaluation requirements do not correspond

to the statement of work, scope of effort, etc. In PFE

this is not a problem. The statement of work, scope of

effort, schedule, and evaluation criteria are all tied to

the factor/sub-factor structure. This enables a comparable

basis for evaluating capability, division of effort, and

schedule

.

V. 3 . 2.2 Size of proposals . The second format issue

is whether a limitation should have been placed on the size

of the bidders’ proposals. PFE allows the proposals to

be as long as the bidder feels is necessary. Some members

of the ETIP and PAA staff were concerned that the proposals

which would be received would be excessively lengthy and

would take an unusually long time to evaluate (see section

IV, 6). This is related to one component of what the literature identifies as ’’goldplating” of the proposal. If the

bidder feels that the length of the proposal is going to

be one of the major factors in determining their rating,

they will present as much information as possible.
192

The results of ETIP’s source selection would indicate

that there was no need to place a length limitation on bidders' proposals. The proposals did not seem long for the

size and nature of the procurement. The time to evaluate

them did not seem excessive.

Proposals ranged from 90 pages for the shortest to

186 for the longest (see Figure V.14). The average size

for agency impact proposals was 141 pages and for commercial

Pages in proposals, excluding appendicies, resumes, supporting

documents, and scope of effort information.

AGENCY COMMERCIAL

IMPACT IMPACT

Pages Pages

167 90

104 186

112 119

186

163

113

X = 14 X r 132

.D. = 35 S.D. = 49

FIGURE V.14 SIZES OF THE PROPOSALS
193

impact proposals was 132 pages. This is only an approximation of the volume of information since the type faces,

spacing, and margins varied.

Goldplating of the proposals did not seem to be a problem. Most of the bidders did not seem to be concerned with

providing the evaluation panel with information .in excess

of what was specifically requested in the sub-factors.

The primary exception to this being one bidder who submitted

a 23 page introductory section. Bidders who wished to supply evidence to support claims made in their sub-factor

responses did so in the proposal appendix in the form of

resumes and past project summaries or they sent along reports from prior contracts.

The panel members were given the proposals on a Friday

and the first factor team was able to meet on Monday. Though

some panel members had to read over the weekend and at night

during the next week, there were no requests for delays

in any of the factor team meetings. At most there were

comments about the pace of the team meetings by some of

the panel members who were participating on a large number

of the factor teams. Since there were fourteen team meetings in a one week period, people who participated heavily

could spend four to six hours a day in meetings.

As one rough check on the significance of length of

response as a factor in determining ratings, the relation-
19 ^

ships between final score and overall length of the body

of the proposals are displayed in Figure V.15. Inspection

of the graphs indicates that page length does not seem to

be directly related to score. Bidders were not rewarded

for volume.

The issue of length of response is also considered

to be closely related to the next issue on whether the bidders understood what the evaluation panel looked at in making their evaluations.

V. 3 . 2.3 Understanding basis for evaluation . The

next format issue is whether the bidders understood what

the evaluation panel looked at in making their evaluations.

Of concern here is whether the bidders understood what information to include and where in their proposals to place

it so that they would feel confident that the panel would

use it in evaluating specific capabilities. This is related

to the issue of size of proposals since if the bidder is

unsure what the panel will base their evaluation on that

there is incentive to put everything possible into the proposal. To avoid receiving overly long proposals the government sometimes uses page limitations when it cannot tell

the contractors exactly what it wants to know (Thompson,

Note 2). This adds additional uncertainty to the proposal

evaluation process. If the panel does not find the information it is looking for in a bidder’s proposal, it will not
AGENCY

IMPACT

»

700 -

600 -

Total

Score

500 «

•

400 —1—1—

1—1—1—1—1-

COMMERCIAL

IMPACT

700 -

110 120 130 140 150 160 170 l80 190

^ Pages

600 -

Total

500 -

400 —

•

•

300 4-1—1—1—1—1—1—h-1—

FIGURE V.15

90 105 120 135 150 165 180 195

Pages

RELATIONSHIP BETWEEN LENGTH OF PROPOSAL AND

TOTAL PANEL SCORE
know whether the information was excluded because the bidder

was weak on the issue they are evaluating or whether the

bidder merely guessed wrong on the government’s priorities

and excluded the material in order to meet the page limitat ion

The bidders were generally confident about being able

to communicate their qualifications to the evaluation panel.

A question with two sub-items was included in the bidder

representatives’ questionnaires to determine this. The

items and responses are presented in Figures V.l6 and V.17.

Figure V.l8 lists responses to the open ended questionnaire

items which pertain to this issue.

Were you confident that you had written something to demonstrate yoiir qualifications on all of the capabilities which

the panel would be evaluating?

Yes-5 Mo-1

Remarks by those answering Yes

o As we had understood things. We did not have confidence

that we know what the evaluators would be thinking (3R4).

o We went through point by point (BR1),

Remark by person answering No

0 It wasn't clear what they were trying to evaluate. There

were no a priori objectives defined, therefore, the respondents weren't responding to the same project. It

wasn't clear whether the notion of cross-referencing mentioned in the RFP could be used. It was questionable

whether any one reviewer would have a whole package to

look at to cross-reference. It wasn't clear how any panel
member would be able to see how any particular part fit

the overall project (BR5).

FIGURE V.16 BIDDER REPRESENTATIVES ’ RESPONSES TO QUESTION

2a ON CONFIDENCE IN BEING ABLE TO COMMUNICATE

QUALIFICATIONS TO PANEL

Were you confident that the panel would find the information

in your proposal which was relevant to each specific capability at the time when that capability was being evaluated?

Yes - 4 No - 2

Remarks by those answering Yes

o But we were trying to create objectivity where the problems arising and the sensitivities that had to be appealed

to were things which were subjective (BR4).

o I think we had a cross-reference sheet, therefore, I would

feel confident that the information would be found (BR1).

Remarks by those answering No

o It wasn’t clear that cross-referencing would work. It

wasn't clear that the reviewers would have the whole package so that they could cross-reference (BR5).

o With the mixed panel I didn't feel that they were capable.

I didn't think that they knew enough about ETIP. I didn't

know all of the people on the panel but I remember at

the time I felt uneasy. The nuances of the program, I

had problems with whether the average government guy sitting on the panel would have the discipline to go through

the structure to make it really carry (BR2).

FIGURE V.17 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

2b ON CONFIDENCE IN BEING ABLE TO COMMUNICATE

QUALIFICATION TO PANEL

Response fr6m Panel Question 6 on Weakest Aspect of the

Process

o Trying to get bidders to understand what you want them

to put into the procurement package so that they could

be evaluated on their ability to do the job, since it

was new and different. Wanted to be sure that we got
enough information so we could make a judgment and so

they know what it was they needed to provide in order

for us to make the judgment (PM15).

FIGURE V.18 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO CONFIDENCE IN BEING ABLE TO COMMUNICATE

QUALIFICATIONS TO PANEL

The bidder representatives generally felt confident

that

o they had written something to demonstrate their

qualifications on all of the capabilities which

the panel would be evaluating (Yes - 5; No -

1 ) ; and

o the panel would find the information in their

propsal which was relevant to each specific

capability at the time when that capability

was being evaluated (Yes - 4; No - 2).

The one bidder representative who answered No to both items

either did not read or did not believe the statem.ent in

the RFP that explained that all members of the evaluationteam would have access to the complete proposal. The other

bidder representative who answered No to the second item

did not feel that the people on the panel would be disciplined enough to follow the process which was described

in the RFP.

Information provided in item 1c of the panel members’

questionnaire relates to this issue. The item asked panel

members whether they were readily able to locate information

in the proposals which was relevant to the issues being

evaluated. The response was Yes - 14, No - 1 (see Figure

V.12). The five respondents who provided additional rem.arks
indicated that the only times they had problems were when

the bidders did not cross-reference properly or otherwise

follow the directions.

V.3.2.4 Differences in uncertainty among bidders .

The final format issue concerns the panel’s ability to deal

effectively with differences in uncertainties among bidders

t during the evaluation. There were two ways in which this

became an issue. One was that bidders who had worked with

ETIP before had a greater prior knowledge of the program,

its projects, and the interests of its staff. They would

therefore be able to write and speak more knowledgeably

about program content than bidders who had no prior experience. The other way was a corollary of this. The panel

would have more information about bidders who had worked

for ETIP before than those who had not. Its ability to

rate capabilities on some of the sub-factors for bidders

who had been past contractors would be greater than for

those who had not. This relates closely to the issue in

section V. 3.7 on whether there were problems in dealing

with the lack of restrictions on admissible information.

The PFE process seemed to be successful in dealing

with both of these types of uncertainties. It provided

for an evaluation based on present or potential capabilities

rather than on present knowledge. It also provided mechan-
isms for using all useful information about bidders where

it was available and for obtaining clarifications where

it was not. One outcome measure here is that one of the

two contracts was won by a bidder who had not previously

worked for ETIP.

There are two structural mechanisms of PFS which dealt

with the problem of enabling bidders without prior program

experience to compete effectively. One is that the process

provided as much information about ETIP’s plans as was possible. The other is that the evaluation was based upon

the bidder’s understanding, approach, and present or potential capabilities to deal with specific management and tech

nical problems rather than upon a proposed design requiring

integrating detailed knowledge about the program into the

proposal

Information about ETIP and the proposed evaluation

systems was provided in the form of the two bidders' briefings and the supporting written materials and bibliography

available with the solicitation document. The goal was

to provide information about both areas of high and low

certainty concerning the proposed systems and to allow prospective bidders to obtain clarifications through questions

The one problem which arose in providing information was

ETIP's inability to distribute the COTR ' s project plans.
Since the plans had not been approved by NBS administration

until after the second briefing, they were unavailable for

distribution to bidders. If they had been available, they

would have provided some additional written detail on the

projects to support the information presented during the

first bidder’s briefing.

By evaluating proposals based on responses to problems

which the government felt were going to be critical, rather

than on proposed systems designs, ETIP minimized the degree

to which prior knowledge of the program would be critical.

Also, the criteria for assessing capability was whether

the bidder had a present or potential capability. If a bidder did not have a specific in-house capability but recognized the need for such a capability and had an approach

for obtaining the capability, he was not penalized. Chapter

IV presented an example of a situation where a rating was

resolved in favor of a bidder based on a potential capability (section IV. 10. 8).

There were two primary mechanisms for dealing with

the panel’s differences in uncertainty about the various

bidders. The first was the question form for obtaining

specific information to resolve uncertainties. The second

was the process of making notes in the record on specific

uncertainties and obtaining subsequent oral and written

clarifications following the panel meetings.
In measurement terms the amount of uncertainty can

be thought of as the confidence interval around the panel’s

estim.ate of the bidder’s score. On bidders for which there

is less uncertainty the confidence limits are tighter.

The clarification process can be thought of as a process

for tightening the confidence intervals in the areas where

there was the greatest amount of uncertainty.

The question form proved to be of minimum usefulness.

Only two forms were used and they were submitted to obtain

sections which were missing from two of the proposals.

The process of submitting question forms to obtain clarifications was used in place of having each bidder make an oral

presentation on their proposals to the panel. This was

done since it was uncertain how many bids would be received

and it was thought that there might be scheduling difficulties in having the panel hear the presentations. Since there

were no oral presentations to the panel and since the question form was not used for clarification, the burden for

clearing up uncertainties was placed on oral and written

clarifications after the panel meetings.

The oral and written clarification process worked quite

well. If the team had difficulty reaching agreement on

a single score because of uncertainty about the bidder,

a note was included in the written record indicating that

the specific area of uncertainty needed to be investigated
during the oral clarifications. This usually happened when

there was strong disagreement between team members concerning how to interpret something which a bidder stated in

his proposal. Someone on the team would usually propose

a score which the majority of the panel seemed to be favoring and suggest that the strong disagreement be resolved

by adding a note which would require clarification prior

to negotiations. An example of this was given in chapter

IV in the description of the first factor team meeting.

The example tells of a disagreement over whether a bidder

did, in fact, have the in-house capability in instrument

design. The issue was resolved by giving the bidder an

acceptable score (seven or above) and adding a note to the

record to verify this score (section IV. 10.2). All of the

notes from the record were compiled and used to prepare

questions for the oral clarifications.

One panel member, in responding to item 10 on the panel

questionnaire (see Figure V.19), indicated that he would

like to have had information from reference checks available

Responses From Panel Question 10 on Any Other Aspects of

the Process which You Wish to Get Out

o Checking of bidders* references should be organized

more thoroughly in advance of the panel meetings (PM12).

o The orals were very important. Since I didn’t participate I don’t know how well they worked. They shouldn’t

be considered pro forma. It’s your last chance to

ask tough questions and see the answers (PM10).

FIGURE V.19 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO DIFFERENCES IN UNCERTAINTY AMONG BIDDERS
20U

during the factor team meetings. The way the process worked

was that references (people listed under Factor I as having

knowledge of the bidder's past performance) were checked

after the team meetings and before the oral clarifications.

The author believes that this information and information

which would have been available from orals prior to the

panel meeting would have enabled the panel to tighten the

confidence intervals around its scores and take some of

the burden off of the subsequent oral clarification sessions.

V. 3.2.5 Communication of nature of project and required

skills . The first content issue under the Effective Communications category is whether PFE communicated to bidders

what it was the government wanted to buy? Of importance

here is whether the bidders understood the general nature

of the projects well enough so that they could effectively

respond to the sub-factors. A level of understanding which

conveyed both the skills required and the context for responding to the sub-factors in terms of the project objective was important. Related to this is that it was important that the requirement for the projects which were conveyed

in the RFP were the requirements which the government actually undertook after the contracts were signed, i.e.,

the actual projects be the projects on which the source

selection was based. Often the criteria for competitive

selection turn out to have little relationship to the pro-
ject which the successful bidder ends up undertaking (see

Sect ion II . 3 • 5 )

.

The extent to which the process was able to communicate

t;he general nature of the projects and the required skills

is somewhat uncertain. A question with two sub-items was

included in the bidder questionnaire to obtain information

on this. The items and responses are presented in Figures

V.20 and V.21. Figure V.22 presents a response to the open

ended questionnaire items which pertains to this issue.

The bidders were:

o split on whether they felt that the information

provided by ETIP allowed them to assess the

skills which would be required to do the work

( Yes - 3 ; No - 3 ) ; and

o almost unanimous in feeling that ample information was provided in the briefings and supporting documents so that they were confident that

they had sufficiently related the sub-factor

issues to the objectives of the projects (Yes - 5; No - 1).

Remarks under the first item (question 3a, Figure V.20)

indicate that some of the bidders did not feel that the

objectives of the project and ETIP were clearly defined.

Another bidder felt that he did not know enough about the

shortcomings of the people managing ETIP.

The latter eoncern is one for which it would be dif- •

ficult to fault the structure of content of the source selection process. It is doubtful that any process could effectively provide information on the management styles of pro-
Did the information provided by ETIP allow you to assess

the skills which would be required to do the work?

Yes - 3 Mo - 3

Remarks by those answering; Yes

o Generally. I didn’t think ETIP exactly knew v;hat it wante

The REP was as good as it could be under the circumstances

It was the nature of the program. ETIP was very into

process and could have discussed more of the substance

of the program. Especially the politics and communications skills to handle a job like that (BR2).

o We felt the information was there to assess the skill

requirements but we had a feeling that more skills were

asked for than were really required. That may have been

due to a misinterpretation of the RFP which could be construed as poor wording in the RFP or maybe not (3R1).

Remarks by those answering No

o There weren't any well defined project objectives. Since

there was no general task, one couldn't show what selection of skills was necessary (BR5).

o We needed a clearer picture of what ETIP hoped to accomplish. It didn't have anything to do with the factor/subfactor issues. It was that the mission of ETIP was not

clear ( BR4 )

0 We needed a better way to anticipate the ineptitude on

the part of ETIP management (BR6).

FIGURE V.20 BIDDER REPRESENTATIVES ' RESPONSES TO QUESTION

3a ON WHETHER THE RFP COMMUNICATED WHAT THE

GOVERNMENT WANTED TO BUY
Was ample information provided in the briefings and supporting documents so that you were confident that you had sufficiently related the sub-factor issues to the objectives

of the projects?

Yes - 5 No - 1 Remarks by those answering Yes

o Qualified by that I didn’t feel the true feelings were

reflected. You were trying to take an objective approach

where in the end subjective feelings had a lot to do with

it (BR4).

o Thompson was too much into process. I felt he didn’t

really know government and it showed. More substance

and less process (BR2).

Remark by person answering No

o There were no good objectives or purpose other than that

ETIP was politically in trouble at the time (BR5).

FIGURE V.21 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

3b ON WHETHER THE RFP COMMUNICATED WHAT THE

GOVERNMENT WANTED TO BUY

Response to Bidder Question 13 on Any Other Aspects of the

Process Which You Wish to Get Out

o There seemed to be a great deal of jargon in the RFP.

We spent quite a bit of time trying to figure out what

you were really saying (BR1).

FIGURE V.22 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO WHETHER THE RFP COMMUNICATED WHAT THE GOVERNMENT WANTED TO BUY
gram personnel. This seems to be the type of information

which is somewhat available to bidders through contact with

others in the professional community.

The former concern of lack of clarity of objectives

is difficult to reconcile with the near unanimous response

to the second item (question 3b, Figure V.21) which asked

whether the information presented was sufficient to relate

the sub-factor issues to the project objectives. In part

one bidder (a successful bidder) may be responding to problems which occurred after the source selection was complete

and work began on the projects.

The problem of the project actually undertaken bearing

little relationship to the project described in the RF?

was eliminated by using PFE. Since the source selection

material specified both the areas of certainty (e.g., evaluation systems for commercial and agency impacts of procure

ment experiments were required) and uncertainty (e.g., the

number of evaluation systems, their eventual owners, and

the specific projects to be evaluated during the development) the bidders were not responding with specific solutions to objectives which would change.

V. 3 . 2.6 Appropriateness of the Specific Sub-factors .

The next content issue deals with the appropriateness of

the sub-factors which were selected. One of the PAA staff

members who reviewed a draft of the RFP indicated that he

felt that many of the sub-factors under Management and Re-
ports & Reviews would not be useful in the evaluation (see

Section IV. 6). Another related concern is whether there

were any significant issues which were not included as subfactors. The first concern, if true, would result in some

waste of resources by bidders in responding to the RFP and

by the government in evaluating the RFP. The second concern

would be more serious, since it could result in selecting

an unqualified bidder by not evaluating some critical issue.

Both panel members and bidder representatives largely

felt that the selection of sub-factors was appropriate.

A question with two sub-items was included in both the panel

and bidder questionnaires to obtain reaction to the specific

set of sub-factors which was used. The items and responses

to them are presented in Figures V.23 to V.26. Figure V.27

presents a response by a bidder to the open ended questionnaire items which pertains to this issue.

Did you feel the sub-factors encompassed most of the issues

which would be critical to designing the evaluation systems

which ETIP desired?

Yes - 12 No - 2 Don't know - 1 Remark by a person answering Yes

o If you cover everything you don't miss much. The subfactors covered just about everything, probably too much

(PM4).

Remarks by those answering No

o Economic issues were given insufficient weight in the

commercial impact solicitation (PM2).
210

0 A bidder's recognition of the constraints of reality of

evaluation was not adequately covered (PM3).

Remark by person answering Don't know

0 I assumed ETIP knew best what it wanted (PM6).

FIGURE V.23 PANEL MEMBERS' RESPONSES TO QUESTION 2a ON

APPROPRIATENESS OF THE SUB-FACTORS

Did you feel the sub-factors encompassed most of the issues

which would be critical to designing the evaluation systems

which ETIP desired?

Yes - 4 No - 2

Remark by a person answering Yes

o Most of the sub-factors were O.K. but they weren't really

important. More emphasis should have been placed on the

problems of working with other agencies. You had a generally inexperienced staff (BR2).

Remarks by those answering No

o There weren't any objectives, therefore it's hard to say

what issues derive from them (BR5).

o Failure on the part of ETIP management to com.prehend the

process by which innovations take place (BR6).

FIGURE V.24 BIDDER REPRESENTATIVES ' RESPONSES TO QUESTION

4a ON APPROPRIATENESS OF THE SUB-FACTORS

Did you feel that many of the sub-factors dealt with issues

unimportant for designing the evaluation systems which ETIP

desired?

Yes - 1 No - 13 Don't know - 1 Remark by person answering Yes

I 0 Many of the agency and system-related factors are either

of marginal importance or can only be developed through

an inductive process over time, not through a deductive

process which was the strategy behind the evaluation system

(PM2).
Remarks by those answering No

o Comments from people who saw it or bid on it felt it was

complex to respond to. But it was simple to respond to

if you read it and understood the format. It was not

complicated. It was a question of devoting time to it

(PM10)

.

o A few seemed superfluous (PM1).

o A lot of things are important but not important enough

to be set out as separate sub-factors. You could subsume

the sub-factors under the factors. You’re cutting the

pie too fine (PM4).

o But "design," in the formal sense, received too much emphasis (PM3 )

.

Remark by person answering Don’t know

0 Didn’t feel I had enough information about the total project to judge the importance of the issues covered (PM6).

FIGURE V.25 PANEL MEMBERS’ RESPONSES TO QUESITON 2b ON

APPROPRIATENESS OF THE SUB-FACTORS

Did you feel that many of the sub-factors dealt with issues

unimportant for designing the evaluation systems which ETIP

desired?

Yes - 2 No - 4

Remarks by those answering Yes

o I felt yes at the time but I really don’t remember why

(BR5).

o I don’t remember which ones (BR1).

Remark by a person answering No

o There was a lack of balance and some of them could have

been handled under more general topics. A more substantive discussion could have given better balance (BR2).

FIGURE V.26 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

4b ON APPROPRIATENESS OF THE SUB-FACTORS
Response to Bidder Question 10 on the Strongest Aspect of

the Process

2

o Its balance and its all inclusiveness. It was an awesome

job to take it initially. After analysis, it was still

awesome but you could see where it was going. It was

clear and the requirements were in balance. Nothing was

missing ( BR3 )

FIGURE V.27 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO APPROPRIATENESS OF THE SUB-FACTORS

The panel members generally felt that the subfactors:

o encompassed most of the critical issues for

designing the evaluation system.s (Yes -12;

No - 2; Don’t know - 1); and

o did not deal with issues unimportant for designing the evaluation systems (Yes they did deal

with unimportant issues - 1 ; No they did not -

1 3 ; Don ’ t know - 1 )

The bidders were more evenly divided. However, they also

felt that the sub-factors:

o encompassed the critical issues (Yes - 4 ; No -

2 ) ; and

o did not deal with unimportant issues (Yes they

did - 2; No they did not - 4).

No trends appear in the remarks of the respondents. However, it is interesting to note the not unexpected contrast

in perspective between two people responding to the same

RFP. Bidder representative 2 (under question 4b, Figure

V.26) indicates that there was a lack of balance among the

sub-factors, while bidder representative 3 (under question

10, Figure V.27) indicates that the balance and all inclusiveness was the strongest aspect of the process.
V.3.2.7 Parroting . A final content of communication

issue is concerned with whether the bidders supply information in their proposals which goes beyond problem statements or potential solutions which are contained in documents supplied as part of the solicitation. Parroting of

information from the solicitation is a problem when it is

t done to the exclusion of information which would enable

the panel to evaluate the bidders. Some ETIP staff had

raised this issue prior to the release' of the RFP (Thompson,

Note 3 ) • There did not seem to be any significant problems with

parroting. Since the bidders were not given a specific

system design to respond to, there was no proposed solution

to echo back. The bidders’ responses addressed the issues

which were raised in the sub-factor statements or the illustrative questions. In several cases the team found that

the information which was provided under a specific subfactor section of the proposal was sparse and the bidder

had not cross referenced other sections of his proposal.

If a panel member had read information in other parts of

the proposal which seemed relevant to the sub-factor being

evaluated, this information was presented to the panel for

consideration (see Section IV. 10. 6).

V. 3.3 Sensitivity and Selectivity in Obtaining Bids

The category of sensitivity and selectivity covers

issues relating to whether ETIP was able to obtain an accept-
able number of responsive bids. The two issues under this

category are:

1 . Were enough responsive bids received to allow

for a confident evaluation?

2. Did the process enable prospective bidders

to self-select effectively on bidding?

Were a large number of non-respons i ve bids

received?

Did a lot of qualified firms not bid?

V. 3 . 3.1 Sensitivity . The issue of sensitivity was

one of how to attract bidders who might have the required

capabilities. The program's budget was small compared to

other federal programs offering evaluation or systems contracts. It was therefore not attractive as a continuing

source of funds. Also, it did not have the visibility in

the professional evaluation community that a bidder might

desire. The program had begun to establish a poor track

record in contracting by cancelling its last RF? for an

evaluation contract after the proposals were evaluated.

One of the concerns of people at ETIP both prior to and

after the writing of the RFP was whether enough responsive

bids would be received to allow for a confident evaluation

and selection of qualified bidders (see Section IV. 3).

Also, when the RFP was circulated for review, a PAA staff

person felt that it was so confusing that it would scars

off good bidders (see Section IV. 6).

The responses of the panel members suggest that enough

bids were received to allow confidence in the accuracv of
the evaluation. Whether or not any firms with greater qualifications did not bid is uncertain.

An item in the panel questionnaire asked respondents

whether being able to rate more proposals would have increased their confidence in the accuracy of the scores which

were assigned. For the agency impact contract the answers

were; Yes - 1 ; No - 10. For the commercial impact contract

they were: Yes - 3 j No - 7 (see Figure V.2S).

Did you feel that being able to rate more proposals would

have increased your confidence in the accuracy of the scores

which were assigned?

a) Agency Impact

Yes - 1 No - 10

Remark by person answering Yes

o About a dozen in all. If you were evaluating, you need

to see a number of selections. It was more of an issue

for those participating in only one factor team (PM1).

b) Commercial Impact

Yes - 3 No - 7

Remarks by those answering Yes

o About two more. You want to weigh them to be sure

you’ve seen the best that’s around (PM12).

o About two more (PM7).

0 At least three more and maybe ten total (PM15).

Remark by a person answering No

o I could have picked the winners without any scoring

(PM4).

FIGURE V.28 PANEL MEMBERS’ RESPONSES TO QUESTION 3 WHICH

RELATES TO SENSITIVITY
The responses for the commercial impact contract, for

which three proposals were received, seem to indicate that

it is possible to scale bidders with the process when only

a small number of proposals are re.ceived. However, the

confidence that some of the panel members had in their evaluation on the commercial impact contract may have been increased by having been able to scale two of the bidders

during the agency impact evaluation (i.e., two out of the

three had bid on agency impact as well and had already been

evaluated once on most of the factors).

Even though enough responsive bids were received to

allow for confidence in the evaluation, there remains the

question of whether any better qualified firms did not bid

and if so whether this was a function of the ?FE proposal.

This question had been raised by several people following

the source selection.

The problem with this question is that there is probably no way to answer it. Since there was not a well established group of firms with a track record in implementing

evaluation systems (one of the conditions for using ?FE),

it was not possible to assess a priori, or without evaluation, who the more qualified contractors were. People observed that some of the firms that they expected to bid

did not. This, however, may be a function of many variables

including the novelty of the source selection method and

effective self-selec t ion
V.3.3.2 Selectivity . The goal of self-selection is

to allow those prospective bidders who would not be qualified to do the work to determine this themselves and not

invest in bidding. In this way resources of both the bidders and the government are saved. The literature has discussed the waste which occurs when many unqualified bidders

submit proposals (see Section II. 3. 5). This issue relates

to the issues on resource requirements which are described

in section V . 3 . 1 2

.

The RFP seemed to be relatively effective on the issue

of self-selection. Though bidders were evaluated by the

teams as being unacceptable on specific sub-factors and

some bidders were unacceptable on a large number of subfactors, the solicitation did not result in a large number

of obviously unacceptable proposals. Those which ended

up as unacceptable required consideration by the teams on

each sub-factor to determine this (see related responses

to open-ended questions displayed in Figure V.29).

Response to Panel Question 5 on the Strongest Aspect of

the Process

o It weeded out the trivial bidders. You don't want the

casual people taking up your time. Those who responded

clearly were sincere (PM10).

Response to Panel Question 10 on Any Other Aspects of the

Process Which You Wish to Get Out

o It provides information to potential bidders for selfselection (PM13).

FIGURE V.29 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO SELECTIVITY
218

Self-selection may have occurred by each of two mechanisms. One was by prospective bidders balancing the objectives of the project against the objectives of their firm,

and the resources available to their firm. The other was

by determining whether they would be able to respond effectively to the RFP . The author has heard, third hand, about

both types of selections by firms which some ETIP staff

had expected would bid. The first being firms where management decided that the project did not match their current

or long term objectives. The second being fir.ms which were

upset about the form of the RFP and did not wish to invest

in responding. One possibility is that some of the firms

which were upset about the form of the RFP would have been

equally upset about the complexity of the requirements of

the projects if they were under contract.

V.3.4 Assessment of Bidders’ Capability

The category of Assessment of Bidders’ Capability is

concerned with issues of whether PFE enabled an effective

evaluation of both management and technical capabilities

° 0 n a later procurement using PFE, one firm which declined to bid was quite open about their opposition to the

form of the RFP. They listed numerous problems which they

found with it. A staff member of this firm later indicated

that in his opinion the firm was not .set up to handle the

type of unstructured system development work which was required and that the management was upset because they wanted

a contract to work in the substantive area of the project

in the manner which they felt was required.
and whether this evaluation could be related to subsequent

performance. This was an important problem since there

was no known track record of developing evaluation systems

in the contracting community. The meaning here of effective

evaluation is that the process be successful in enabling

an ordering of bidders from best to worst, with demarcations

for acceptability and unacceptability, and that this ordering

be based on the actual capability of the bidders proposed

team rather than on the capability of the proposal writers.

The question of subsequent performance is concerned with

whether an evaluation in terms of capability for dealing

with problems known to exist is effective in predicting

a bidder’s actual performance.

Three issues are included in this section. They

are

1 . Did PFE provide for an effective evaluation

of management capability?

2. Did PFE provide for an effective evaluation

of technical capability?

3. Did PFE anticipate the problems which were

known to exist in doing evaluations and in

developing systems?

Did the criteria provide for an assessment

of the relationship between what was proposed

and later performance?

Was the award based upon proposals which

would have some relationship to the approach

subsequently used?

Was the material provided by the successful

contractors useful or not to the PAAs?
220

V.3.4.1 Evaluation of management capability . The

need for source selection procedures to be able to deal

more effectively with management capabilities is evident

in both the evaluation and the systems literature. The

evaluation literature indicates an insensitivity of evaluators to problems of liaison and scheduling (e.g., Patton,

et al., 1977) and some of the systems literature indicates

the need to develop better capability for evaluating management during source selection (e.g., Helman 4 Taylor, 1975).

This issue primarily examines whether ETIP was able to differentiate among bidders on the various management issues

which were included as sub-factors. The question of the

relationship between these ratings and future performance

is discussed under section V.3.4.3.

The author believes that PFE provided a strong basis

for evaluating management capability. The discussion by

the factor teams of the management sub-factors was extensive. Much of the time was taken to draw out the differences between the management requirements on these projects

and general project management requirements. The teams

used the written sections in the proposal and all supporting

evidence (resumes, project summaries and information from

panel members who were familiar with the bidders) to assess

understanding, approach and capability. For the most part

the teams were able to come to agreement on unique scores

for each bidder. However, there was an important management
221

sub-factor which proved difficult for one of the teams to

measure. On sub-factor 1.2, Importance of Program to the

Bidder, the agency impact contract factor team assigned

a 6 to all of the bidders. The team felt that from the

information available to them there was no basis for distinguishing among the bidders. The score of six was assigned

because the team felt that this issue needed to be explored

during clarifications with any bidders considered competitive . Sub-factor 1.2 was considered by some to be one of

the most important in predicting the success of the projects.

Therefore, a considerable amount of attention was devoted

to it during clarifications. The bidders were asked directly about what assurances they could give of top management

support of the project. Their responses to other questions

were, in part, assessed to determine what they felt the

primary objectives of the project were and what they saw

as their role in the systems developments. It was important

for ETIP to know if a bidder had strong ideas about what

the primary objectives were and if so whether the bidder

would subordinate ETIP’s objectives to its own. The oral

clarification session proved very useful in enabling ETIP

to more confidently assess these issues and assign scores.

However, there was recognition that the confidence intervals
on some of the bidders was still relatively large, especially in trying to predict commitment into the future.

The clarifications on management also allowed ETIP

to determine which people on the bidders’ teams held the

understandings which the panel had rated favorably. For

example, in one instance it became evident which member

$ of a bidder’s proposal team wrote the responses to certain

sub-factors which received high ratings. ETIP was then

able to negotiate for this person’s involvement in the project.

V.3.4.2 Evaluation of technical capability . The second issue is whether PFE provided for an effective evaluation of technical capability. As with the preceding issue,

this primarily examines whether ETIP was able to differentiate among bidders on the various issues which were included as sub-factors. The relationship between these ratings

and anticipated future performance is also discussed in

section V. 3 • ^ • 3

All of the technical factor teams were successful at

agreeing upon unique orderings for the bidders on each subfactor. The teams were able to discuss the information

available on each bidder under each sub-factor and compare

it against issues in question and the criteria for evaluation.
223

Discussion on the categories of the Panel's Role in

Evaluation of the Proposals (section V. 3 . 6 ), Equity of Treatment Across Bidders (section V.3.7), and Effective Communication (section V.3.2) indicate some of the elements which

affected the panel’s ability to assign scores. Such things

as the range of panel members’ initial ratings, the biases

of various panel members, and the amount of information

available under each sub-factor heading seemed important.

As with the management sub-factors, the clarifications

sessions proved useful for gathering additional information

where the panel had greater uncertainty in the scores whic*h

they assigned.

V.3.^.3 Anticipation of problems known to exist in

the project . The final issue under this category asks whether PFE anticipated the problems which were known to exist

in doing evaluations and in developing systems. The last

two issues indicated that the teams were able to differentiate among bidders and assign scores on both management

and technical sub-factors. However, these scores have very

little meaning unless they have a positive relationship

to subsequent performance. Before the RFP was issued, a

PAA staff member indicated that he felt that the relationship would be inverse. He thought that any bidder who would

be successful in responding to the RFP would supply the

PAAs with useless materials once under contract (section

IV. 6 )

.
In order to do an adequate job of responding to this

issue, the subsequent performance of the successful

bidders would have to be assessed and compared to the problems which arose during the contracts and the sub-factors

which were included in the RFP. Such a detailed analysis

of the post award time period is not within the scope of

this dissertation. However, some of this information is

contained in a report by Garrity (1980). Without doing

a detailed analysis, the most that can be said about the

relationship between the evaluation and subsequent performance is that it seemed mixed.

In analyzing how well PFE anticipated future performance, there are two variables which seem important. One

is whether ETIP knew about all of the critical issues in

advance and the second is whether there was a way to m.easure

accurately the bidders acceptability on each issue.

On the first variable of whether the critical issues

were known, ETIP seemed fairly strong. One measure of this

is that on subsequent procurements for similar contracts

in support of ETIP's regulatory work very few new sub-factors

were added.

There were more substantial problems on the second

$ variable of measurability. In large part this was a problem

of trying -to predict bidder’s ability on sub-factors into

the future. The author believes that ETIP’s measurements

of the bidders against the criteria were fairly accurate
225

at the time that they were made. However, circumstances

which occurred over time seemed to change the bidders position on some of the sub-factors. This was particularly

problematic on sub-factor 1.2, Importance of Program to

the Bidder. The bidders’ acceptability on this sub-factor

seemed to change over time. Since this sub-factor affects

the resources which are devoted to capabilities on other

sub-factors, the acceptability on other sub-factors also

varied over time.

The inability to make measurements which predict capability into the future seemed to be one of the failings

of the source selection. However, it is difficult to attribute this to the PFE process since it is doubtful that any

other source selection techniques are effective at this

either. It is one of the risks of awarding a contract.

V. 3.5 Use of Experts in the Evaluation of Proposals

The category of the Use of Experts in the Evaluation

of Proposals is concerned with the issues of both how to

obtain the participation of experts and how to make use

of the information provided by experts.

V. 3 . 5 . 1 Obtaining participation by experts . The issue

of obtaining the participation by experts includes identifyt ing them, convincing them to participate, and having a schedule which allows them to participate. This is identified

as a problem in the literature on systems procurement (e.g.,

Carnes, 1976). ETIP was quite successful on this issue.
Experts were identified by the COTR and the author

throu gh discussion 'with other ETIP and PAA staff and f rom

their own knowledge of relevant people who might par t ic ipate

The nineteen people outside of ETIP who we r 6 ident

f ied included those who were

o either part of or closely related to the procurement experiments in the PAAs,

o thought to be experts on some of the specific

sub-factors being evaluated, or

o familiar with some of the likely bidders.

Some of these people were considered because in addition to their expert information they:

0 could lend credibility to the project with

the NBS program office, or

0 were responsible for managing other government

evaluation programs and might have had an interest in learning of the PFE technique as a method

for procuring contractor support.

In convincing experts to participate, the project and

evaluation process were explained to them and the ability

to limit participation to specific sub-factors was stressed

Of the nineteen outside people who were invited to join

the team, nine agreed to participate and two agreed to send

representatives. Those who declined generally indicated

prior commitments during the week for which the evaluation

was scheduled. It was easiest to get people in other programs in NBS and in related programs in the National Scienc

Foundation. People who were in evaluation programs which
were not substantively related to ETIP’s work most frequently indicated that they were busy.

Scheduling was, for the most part, quite successful

but did present some problems. The schedule for the times

when specific factor teams would meet was not drawn up until

the panel met to receive the proposals. The best schedule

which could be drawn up resulted in several people being

unable to participate due to conflicts at different points

during the evaluation week. In all, eight people from outside of ETIP participated.

V.3.5.2 Use of expert information . The panel members

were mixed in their reaction to the use of outsiders in

the process. In response to the question on the strongest

aspect of the process, one of the ETIP panel members felt

that being able to allow experts to limit their contribution

was the greatest strength of the process (see Figure V.30).

However, three outside participants had doubts about the

usefulness their contributions to the evaluation. Two of

their responses suggest that experts should at least have

some knowledge of, and relationship to, the projects.

Response to Panel Question 5 on the Strongest Aspect of

the Process

o It enabled people with particular expertise to limit

their contribution to the relevant areas (PM2 ).
228

Responses to Panel Question 6 on the Weakest Aspect of the

Process

o Being present at only some of the factor team meetings

limited the contribution a person could make and stimulation they could receive which might prompt further

useful responses. There was clearly a trade-off with

the burdens of having everyone participate in everything. These tradeoffs are difficult to make (PM5).

o You have to be very careful of the people that you

pick for it. I may not have had enough knowledge of

what the actual program was and a good enough understanding of just what was wanted in the contract (PMl6).

0 There were too many people with too little understanding of the total project. If you had fewer people

and limited it to ETIP and GSA staff, you would have

gotten the same or better results. It took us a long

time to get an idea of what you wanted (PM6).

FIGURE V.30 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO EXPERT PARTICIPATION

V.3*6 The Panel’s Role in Evaluation of the Proposals

The category of the Panel’s Role in Evaluation of the

Proposals is concerned with the effectiveness of the factor

teams in reviewing the information available about the bidders and reaching agreements on ratings for all of the subf actors

Six issue discussions are included under this category.

They are:

1.

a) Did panel members who agreed to participate

actually show up?

How difficult was it to schedule meetings?

b) Were panel members able to arrive at initial

ratings prior to factor team meetings?

2. Was the panel able to arrive at final scores

without voting?

Was the panel able to deal effectively

with strong individual disagreements?
3.

a) Did the panel members realize how their individual positions affect the final score,

i.e., do they see other panel members modifying their positions in response to the discussion?

Do panel members understand how the final

score is reached?

b) Do the final scores differ significantly

from the means of the initial scores of the

individual panel members?

c) Were the panels^ ratings and recommendations

used after the panel meetings were concluded?

4. What impact did those with extreme scores

or greater variability in their ratings have

on the final scores?

5. How long did it take to reach agreement on

factor scores after the sub-factor scores

were agreed to?

6. Were "junior” panel members able to affect

the scores if their bosses were in the room

and their initial scores differed?

V.3.6.1 Meeting attendance and ability to determine

initial ratings . The first issues related to the panel

process is whether people who agreed to participate actually

showed up at the factor team meetings and whether they were

able to arrive at initial ratings prior to the start of

these meetings.

Participation in the evaluation by those who agreed

to be panel members was good. Each of the .two contracts

had 12 people on the panel, with some overlap in membership.

Of the people who received copies of the proposals at the

April 16 proposal distribution, two were unable to attend

due to other activities which had higher priority for them.
With one exception, the panel members were able to

arrive at initial ratings of bidders prior to factor team

meetings. The one exception was a person who felt that

rating bidders on the sub-factors was evaluating the issues

out of context. He came to his factor team’s meeting to

indicate his willingness to discuss the proposals as a whole

but he declined to participate in the evaluation process

(see Section IV. 10. 10).

The initial ratings for each sub-factor can be found

in the appendix. In several cases one of the panel members

had a range of uncertainty with respect to one or more bidders. This panel member placed the bidders initials at

two different scores and indicated to the panel that he

felt that the bidder was somewhere in the range between

the scores. In the cases where this occurred, ohe average

value was used for purposes of plotting and statistical

computations

V. 3 . 6.2 Reaching agreement . One of the major issues

in controversy prior to the beginning of the evaluation

was whether the panel would be able to arrive at final scores

without voting. This was particularly important because

of the large number of places where agreement was needed,

i.e., on 33 sub-factors for each of two contracts and on

combining the sub-factor scores into factor scores. It

was known that some of the panel members held divergent
231

views about the objectives of the projects and were likely

to have strong disagreements during the evaluation.

In all cases the panel was able to arrive at final

scores without voting. In cases where strong disagreements

persisted after team members presented the information on

which they based their ratings, the teams used two mechanisms to facilitate agreement. The method most commonly

used was 'to assign the score which most of the team was

tending toward and indicate that the issue needed further

investigation during any subsequent clarifications. The

other method, which was used twice, was to post both scores

on the board and proceed with discussion on other sub-factors

within the factor, leaving the decision on the sub-factor

in question until more discussion had occurred on related

issues. In no case did the panel have to resort to the

mechanism available in PFE of having a dissenting team member

enter a signed exception into the record.

The way the rating process generally worked was that

following the posting of initial sub-factor ratings a bidder

was selected for discussion and a panel member with either

the highest or lowest score was asked to present the reasons

for his rating. After he finished, someone with the opposing opinion would be asked to talk. Other panel members

entered into the discussion indicating what evidence they

were using in either agreeing or disagreeing with the posi-
tions presented or adding an additional perspective, As

the discussion continued, team members would either reinforce or modify their initial perspective. If most of the

team was tending toward one score but one or two members

still felt that a different score was appropriate, the panel

would focus on the minority views. People would look for

information either opposed to or in support of the minority

opinion. In some cases related information on the bidder's

understanding, approach or capability for dealing with the

sub-factor would be discovered under another sub-factor

which had not been cross referenced. In other cases information from a panel member who was familiar with the bidder's past performance on the issues would be presented.

Discussion would continue until the panel agreed on a score.

In some cases the score would be closer to that held

by the minority position than that of the m.ajority. For

example Figure V.31 shows the initial and final ratings

for sub-factor VI. 2 on the agency impact contract evaluation. There are two examples in this sub-factor of a minority opinion convincing the majority. Evaluator 205 was

able to convince the other team members that Bidder 204 's

proposal was unacceptable on this sub-factor without significant revisions (a score of four). Likewise Evaluator 211

was able to convince the team that Bidder 202 was acceptable

as is (a score of seven).
233

>

0

LD>

4 >

ro

h—

t

>

CM

t—

H

> B205 B202 B203

0206

;B201

B204

> B205

0202

B201

0203

0206 6204

CM

cr>

UJ

B202

B201

B205

B206 6203

B204

EVAL210

(Oo £O

Csj e (D fl ® o q

CSi o

sJ ^

B S

3 03

9029

EVHL208

6205 O

cvj — m CO o o

CSJ CSJ CNJ Q CQ £

CS4 CD

0206

EVflL205

UDO

CsJ CD

(O ( o c

Csl C CD a

3 ^ CM

} o o

J CM CM

b CD CO

CM>

^

CD CO ri 1 CD LD

^CO C\J

I

FIGURE

V.31

EXAMPLE

OF

MINORITY

OPINIONS

PREVAILING
The time taken to get through the factor meetings varied. The shortest team meeting was forty minutes and the

longest was two hours and thirty-five minutes. The average

time was about an hour and a half (see Figure '1 . 32 ).

Agency Impact Commercial Impact

Factor 0

I 74 minutes 46 minutes

II 104 58

III 1 13 105

IV 74 • 40

V 155 76

VI 120 91

VII 127 115

X = 1 10 X = '^6 SD = 29 SD = 29

R = 81

X = 93

SD = 33

R = 75

R = 1 15

These numbers are estimates of the meeting times which we

computed from running the tapes of the meetings. They do

not include times for breaks which were taken during seve

of the meetings.

FIGURE V .32 TIME REQUIRED FOR THE FACTOR TEAM MEETINGS

Figure V.33 presents responses to the open ended questions which to some degree relate to reaching agreement

on sub-factor scores. Several panel members felt that the

deliberation and reaching agreement on the sub-factors was

either the strongest aspect of the process or the one which

they remember best. Two others felt that the requirement

to make judgments or reach consensus on each sub-factor
Response to Panel Question 5 on the Strongest Aspect of

the Process

o The detail of the sub-factors. More thought was given

to what you really wanted and there was quite a bit of

deliberation on the sub-factors (PM7).

t o The group discussion. Everyone brought relevant information and people had different impressions of what the

bidders meant and you were able to agree on what they

meant (PM4 )

.

o The combination of formal disaggregated review and discussion periods presented and, in some cases altered, in

light of other reviewers’ arguments, information, and

perspective (PM3).

0 The group meeting and discussion in coming up with final

factor scores was by far the strongest part. The team

effort.

Response to Panel Question 6 on the Weakest Aspect of the

Process

o The sub-factors. Making an individual judgment on each

one got too complicated. You could have let them address

the factors and use the sub-factors as things to think

about (PM9).

o The insistence on a consensus and the unnecessary complexity (PM2).

Response to Panel Question 10 on Any Other Aspects of the

Process Which You Wish to Get Out

o The one thing I remember quite well is that every single

person initially had different rankings and they were

able to arrive at consensus. It was either that, or peopl

were very tired of the lengthy process (PM15).

FIGURE V.33 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO REACHING AGREEMENT
was the weakest aspect of the process. Other issues in

this category which are closely related to this one are

those on: 1 ) whether panel members realize their impact

on the team's scores (section V. 3*6.3)) 2 ) the impact of

those with greater variability on the final scores (section

V.3.6.4), and 3) the time required to reach agreement on

the weighting of sub-factors within a factor (section

V.3.6.5). The issue dealing with the bias of panel members

is also closely related to this one (section V. 3*7.1).

V.3 . 6.3 Motivation of panel . An issue which is important to the motivation of panel members is whether they

understand how their individual contributions affect the

results of the process. As part of this, they must also

feel that these results will be used and that they are not

just going through the motions. This is an issue which

appears, to a small degree, in the group process literature

(e.g., Bradford, et al., 1953).

The mechanics of PFE ensured that panel members were

able to realize the impact of their contributions to the

scores. This occurred through the requirements for all

team members to put their initial ratings of bidders into

the formal records of the process, to reach agreement on

the sub-factor scores, and to sign the final factor rating

sheets. Through the discussion of each bidder the panel

was able to see others modify their initial positions as

information was put onto the table and evaluated. By not
allowing voting, the panel had to confront and resolve their

differences. A dissenting opinion could not be ignored.

This was true for both the individual sub-factor scores

and the final factor scores. The contrast between this

and the methods used by some other proposal evaluation techniques was highlighted to the author by a colleague who

had returned from participating on another proposal evaluation panel. He indicated that in contrast with his experience on a PFE panel, he had a very fuzzy understanding of

how his discussion affected the panel's scores. He also

had no idea how these scores would be combined to reach

a decision on which bidder to negotiate with.

Another way the panel members realized their impacts

is that each team owned the process for coming to agreement.

They could decide upon the order of discussion of the subfactors and the methods for reaching agreement. In most

cases the team chose to discuss each sub-factor sequentially

and reach agreement prior to moving on. Section IV. 10. 12

described the one meeting where the team chose to discuss

the factor as a whole prior to rating the individual subfactors. Both methods worked well.

Several of the responses listed in Figure V.32 pertain

to the issue of satisfaction with the group process. Five

of the panel respondents chose to mention their satisfaction

with the group decision process under the open ended questions. Two mentioned dissatisfaction.
The other part of this issue is whether the panel's

inputs were used following the final team meeting and whether the panel members had any indication of how they would

be used. The PFE process was very effective at incorporating the panel's recommendations into clarification sessions

with the bidders who were in the competitive range. The

COTR's report to the contract negotiator was written by

using the actual sub-factor scores to describe the areas

where bidders were found to have problems requiring clarification, negotiation, or revision (see section IV. 11). The

competitive range was determined by examining both the overall scores and the distribution of individual sub-factor

scores

During the panel meetings, areas of uncertainty were

noted in the formal record of the process when the panel

indicated that clarifications were required. Since the

process for noting issues for clarification was mentioned

quite frequently, the panel knew that these notes would

go to the contracting office for follow-up. The questions

for the oral clarifications were based upon these notes

and the sub-factors on which the

below a seven.

teams had rated the bidders

Some additional discussion related to this issue is

included in section V.3.11 which described the reaction

of the contracting office to a suggestion for a committee

to review the panels' recommendations.
V.3.6.4 Dealing with extreme scores or great variability . One of the difficult problems to deal with in proposal

evaluation is how to agree upon a final score when individuals are using different standards for their evaluation.

This includes the question of what impact those team members

with extreme initial scores or greater variability in their

ratings had on the final scores. Figure V.34 illustrates

the different types of distributions in scores which are

possible. Graph A is an example of the range of scores

of a team member with a great deal of variability in his

ratings. He finds that some bidders are excellent and that

some bidders are awful on the sub-factor issues. Graph

B is an example of the range of a team member with a moderate amount of variability in his ratings. He seldom assigns

extremely high or extremely low scores. Graph C is an example of someone with a small amount of variability in his

ratings. He finds that most or all bidders are acceptable

on the sub-factor issues. Graph D is an example of someone

with a small amount of variability in his ratings. To him

most or all bidders are unacceptable on the sub-factor issues. All of these situations arose during the team meetings . Through discussion the team members were able to agree

on scores. Those who took extreme positions were required
10

9

8

7

6

5

4

3

2

1

240

A B C D

v.34 examples of different types of variability

USED IN RATING THE BIDDERS
to justify their scores against the scale. For example,

if someone rated a bidder as a four, in contrast to higher

ratings of others, the other members pressed him to indicate

why the bidder was unacceptable without significant revisions and to describe what specific types of revisions he

felt were necessary. Referring to the words on the scale

and making relative comparisons to ratings assigned to other

bidders seemed to be very helpful in achieving agreement.

Figure V.35 presents a good example of differences

in variability among evaluators and shows the factor team’s

ability to deal with it. Note that Evaluator 205 and 208

made considerable changes in their ratings after the team’s

discussions comparing standards and other information related to the proposals.

Despite the teams’ ability to reach agreement, one

panel member still felt that insufficient attention was

given to comparing standards on the meaning of sub-factors

during the evaluation (see Figure V. 36 ).

V. 3 . 6.5 Time required to agree on sub-factor weights . One of the major, concerns prior to the evaluation was that

the process of arriving at factor scores once the sub-factor

scores were known would be extremely ^time consuming (see

section IV. 8). It was feared that it would be difficult

to 'determine the relative weightings of the sub-factors
2^2 - 1

111.5 (i ' 11 m o CSi CM O CD s g ' CM

B203 o

CM

CDO

CMD

(Nl

B205 B202 B203

(0 ^ — o o o

CM CM CM

(D CD CQ E B205 B206 B202 B203 O

CMCD °O 11

EVAL21 1

CM '<9' o o

CM CM CQ CD

B2030205

U206 o

CMa

EVflLZlO

B203B205 B206 ^ o CM CSI O Q CM® o

CM2 -

EVHL209

IDO

CM

to

— o o

CM CM

OQ CO

mo

CM’^r DO 1 CM CM OA

CMm

COO

csi

(X>

UJ

B202 B205

0203 !

to

— o o o

CM CM CM to 20 a

EVRL205 B206 B205

CM o ; o ' CM CM a a

EVflL204

COo LO CM CM O O ^ oo 03 g

CM

OS o

CM

CD i»' II

.3

Q (D CO rT—

CD LO I ^

CO (XJ ;1

FIGlJHb;

V.35

FXAMPLPJ

OF

DFAI.l

N(]

WITH

1)1

FFFi^FNOES

IN

VAHl

ABILITY

AMONG

BIDDERS
243

Response to Panel Question 6 on Weakest Aspect of the Process

o Needs to have more spelled out during the evaluation about

what the sub-factors mean to assure that everyone is using

the same standards (PM1).

FIGURE V.36 RESPONSE TO OPEN ENDED QUESTIONS WHICH RELATES

TO EXTREME SCORES OR GREATER VARIABILITY

within each factor if this decision was left up to each 0 team. It was also thought that doing this could create

impropriety if the weighting were not agreed to until after

the sub-factor scores were known, or that the appearance

of impropriety would be present even if the sub-factor ratings were agreed to at the start of the factor team meeting.

This issue did not present any time delay problems

during the evaluation. In all cases, the weightings were

agreed to after the sub-factors were rated and in no case

did the scoring take more, than six minutes. The process

for arriving at the factor scores was for the most part

one of taking an eyeball average of the sub-factor scores.

If the bidders fell between two factor scores, there was

discussion of whether some of the factors should be weighted

more heavily than others. For example, one panel member

felt that sub-factors dealing with Phases II and III of

the project should be given less weight. Where the panel

felt there was justification for weighting a sub-factor

more heavily, that was taken into account in the final

scores

.
If not, a visual average of the scores would be assigned.

Figures V.37A and V.37B display the differences between

the numerical averages of the sub-factor scores and the

agreed upon factor scores. The differences are generally

fairly small.

Commercial Impact Contract Evaluation

Factor I

Factor II

Factor score

Average subfactor score

Factor score

Average subfactor score

Factor III Factor score

Average subfactor score

Factor IV

Factor V

Factor VI

Factor score

Average subfactor score

Factor score

Average subfactor score

Factor score

Average subfactor score

Factor

*

VII Factor score

Average subfactor score

Bidder

101

8

7.62

7

7.0

3

7.6

8

8.0

8

7.1

8

8.0

5

5.75

Bidder

201

5

U.62

4

4.37

4

4.2

4

4.0

4

3.8

5

5.0

4

4.5

Bidder

301

4

4.0

7

4.0

4

3

,

3

3.0

3

3.3

3

3.0

4

3.75

FIGURE V.37A COMPARISON BETWEEN THE NUMERICAL AVERAGES

OF THE SUB-FACTOR SCORES AND THE AGREED UPON

FACTOR SCORES
2^5

Agency Impact Contract Evaluation

Bidder Bidder Bidder Bidder Bidder Bidder

201 202 203 204 205 206

Factor I Factor scores Average sub6 6 7 4 7 6 factor score 5.75 5.75 6.75 4.5 6.75 6.12

Factor II Factor score Average sub6 6 8 4 7 5 factor score 6.5 6.0 7.75 3.75 7.0 5.0

Factor III Factor score Average sub4 8 7

6.8

5 8 7

factor score 4.4 7.9 4.7 7.8 6.6

Factor IV Factor score Average sub4 8

8.0

6 5 9 5 factor score 4.7 6.2 5.2 8.4 4.8

Factor V Factor score Average sub4 7 6 4 7 5 factor score 4.17 7.0 6.67 4.17 7.42 5.58

Factor VI Factor score Average sub4 , 6 7 4 8 5 factor score 4.4 6.8 6.8 5.3 7.7 5.2

Factor VII. Factor score Average sub5 7 8 4 7 7 factor score 5.5 7.0 7.25 4.5 7.0 6.75

FIGURE V.37B COMPARISON BETWEEN THE NUMERICAL AVERAGES OF THE SUB-FACTOR

SCORES AND THE AGREED UPON FACTOR SCORES
The question of whether the process of agreeing on

the factor scores presented the appearance of impropriety

is investigated in section

V.3.6.6 Input by .junior panel members . The final

issue to be considered in this category is whether "junior"

panel members were able to affect the panel’s score if their

bosses were in the room and their initial scores differed

(e.g., see Torrance, 1957).

It is unclear how effective PFE was in dealing with

this issue. There were several cases where junior panel

members and their bosses differed and the panel took the

position of the junior panel member. However, frequently

the junior member was already in agreement with the majority

of the panel.

» PFE has a mechanism for dealing with the tendency of

junior panel members to agree with their bosses. 3y having

the initial ratings posted and put into the formal record,

an incentive is created for each panel member to make both

the reasons for his initial score and his changes in the

score known to the others. This may cause the junior panel

member to defend his position more actively. However, the

author feels that the evidence available from the data is

inconclusive on this issue.

V. 3.7 Equity of Treatment Across Bidders

The category of Equity of Treatment Across Bidders

is concerned with whether the panel applied any undue differ-
ential evaluation for or against any of the bidders. Of

Goncern are issues of how bias was dealt with, whether the

criteria were applied consistently, and whether the lack

of restriction on admissible information (including knowledge of the identity of the bidders) created any inequity.

Three issue discussions are included under this category. They are:

'1. Did the panel deal effectively with the biases

of its members?

2. Did the criteria used in evaluation provide

for a consistent evaluation across proposals?

Were the criteria applied consistently

across proposals?

3.

a) Did the lack of restriction on admissible

information lead to any inequity in the process?

b) Did evaluation of the proposals while knowing

the identity of the bidders create any problems - biases in the final ratings, protests?

V. 3 . 7.1 Treatment of bias . The issue of bias is not

whether or not there was bias among the panel members.

There was. It is a matter of whether the panel incorporated

this bias into the evaluation process in a manner which

did not improperly advantage or disadvantage any of the

bidders

.

PFE provided a process for revealing and exploring

the biases of its panel members. First, factor team members

were required to sign and turn in their initial independent

ratings of the bidders prior to discussion on the subfactors .
2H

Those ratings became part of the record of the process.

Next, panel members were required to discuss the basis for

their ratings. This was true for low as well as high ratings. Those team members rating each sub-factor were required to continue the discussion until they reached agreement on a score. They could not resolve their di sagreements

by voting or merely averaging their scores. They were required to explore the bases for their differences and discuss the evidence they were using. A taped record was made

of this discussion. As others on the factor team would

participate in the arguments, the team members would generally be drawn toward a position. If one or more members

of the team persisted in having a strong disagreement with

the score which mobt of the team wanted, there was a mechanism for assigning this score and recording the signed dissenting opinion. This mechanism was never used. The closest

the teams came to this was assigning a score and noting

in the record that clarification was required on the points

of uncertainty. The signed rating sheets and the tape recordings created a record of all biases. The process of requiring team members to reach agreement required an evaluation

of the merits of these biases.

One special situation of dealing with bias is the case

of being biased for or against specific bidders because

of prior experience with them as contractors. For example.

CO
with one of the bidders who had been an ETIP contractor,

one panel member claimed that there was no reason to rate

this bidder since he had worked with him and knew that he

was totally unacceptable. The panel understood how this

person felt about this bidder but realized it was still

necessary to rate this bidder on his merits on each separate

sub-factor

.

Some calculations were run on the data in order to

examine the biases and the extent of influence of the various members of the evaluation panel. As a first step, each

panel member’s ratings of the bidders were changed to rankings in order to eliminate the problems of differential

n variability among the raters (see section V.3.6.4).' Patterns of orderings of the bidders seemed more important

in computing bias that the actual numbers assigned. To

obtain a measurement of each members bias, the difference

between that member’s rating of each bidder on a given subfactor and the average rating of other panel members evaluating the same bidders on the sub-factor was computed. A

summation of these differences was then computed across

all sub-factors which the panel member participated on.

I.e., bias of panel member I for bidder J ,is computed as:

7 A correction factor was included for panel members

who failed to rate all of the bidders on a sub-factor.

Where there were six bidders and the panel member only rated

five of them the ordering was shifted from 1, 2, 3> 4, 5

to 1.5, 2.5, 3«5, 4.5, 5.5, where 1 is the lowest rating

and six is the highest.
250

BIAS

IJ

RANK IJk

k=sub-f actors

on which I participated

E RANK. j^/Cn-l )1

i=all other

panel members

on sub-factor k

M

where

n = total number of panel members rating bidder J on subfactor K, and

N = total number of sub-factors on which panel member 1

rated bidder J.

The assumption underlying this computation is that if a

panel member is unbiased toward bidder J, relative to the

other panel members, BIAS^j should be close to zero, i.e.,

his ranking of bidder J should be higher than the other

panel members’ rankings about as often as it is lower.

A large positive or large negative value for BlASxj would

indicate a relative bias for or against bidder J.

The next step was to compute a measurement of each

member's influence over the final rankings of the panel.

This was done by determining the degree to which the direC'

tion of the difference between the final rankings and the

average initial rankings tended to follow each panel member's rankings. The differences were computed for each

sub-factor as follows:
251

PANEL DIFFERENCE jk FINAL RANK jk RANK

1 i=all panel

members on

sub-factor k

/(n-1)

ijk

where

,

n = total number of panel members rating bidder J on subfactor K.

These numbers were then compared to the values of individual

differences of each panel member, I, where:

INDIVIDUAL DIFFERENCEj^j^ = RANK^^j^ - 2 RANK . ( n- 1 ) i=all other

panel members

on sub-factor

k

A table was constructed for each panel member where, if

the sign of PANEL DIFFERENCE .. was the same as the sign

J ^ of INDIVIDUAL DIFFERENCE., a " + " was entered. If the signs

were different a was entered. If both were equal to

zero a "0” was entered. Figures V.38 through V.58 display

the values and tables for the four most active participants

on the agency impact contract evaluation and the three most

active participants on the commercial impact contract.

Figure V.59 displays the sum of ratios of +/- for each of

the evaluators, a measure of overall influence.

Evaluator 205 *s difference scores show both a large

number of extreme rankings and overall bias for bidders

201 and 206 and against bidder 202. Examination of the
EVAL

5 . 1 . 2 . 3 . SUB- 4 . FACTOR

5 . 6 . 7 . a. 9.

BIDDER 20 1 . 3.00 2.75 3.50 3.50 3.00 2 .75 3.33 .67 . 00

202 . - 1.63 - . 75 - 2.50 - 1 .75 - 1.70 .63 - 2.50 - 1.50 - .63

20 ?. - 2.00 - 2.75 -.67 .25 - 1.20 -.33 -. 33 -.17 .50

204 . -.67 - 2.50 .50 1.00 - 1 .60 - 1.12 -.50 -.17 -.25

205 . -.67 - .50 - 1.17 - 2.50 - 1.30 -2 .25 - 1.67 .00 - 1.00

2 C 6 . 2.17 3.75 . 33 -.50 2.50 .38 1 . 67 2.50 2.25

1 0 . 1 1 . 12 . 1 3 . 19 . 20 . 2 1 . 22 . 23 . 24 .

• o O .00 . 00 . 00 .50 -.13 1 . 20 - . S 3 2.33 1 . 50

on 1 • -2 .60 .13 . 13 - 2.00 - 2.75 - 3. 50 - 2.17 - 2.83 - 4.17

.83 1.00 - 1.12 -1 .50 -.25 1 .88 - 1.50 - . 17 - 1.50 1 .50

- 1.00 . 1 0 - 1.37 - 1.00 — .63 .25 1.00 1 . 30 .67 2 . 1

'’

- 3.33 -1 .50 - 2.62 -1 .62 .38 . 1 3 . 30 1 . 00 . 83 1.33

3.67 1 .90 3.25 2.87 2.00 .63 2.50 .67 . 50 - .67

25 . 2 e . 27 . 30 . 31 . 32 . 33 . Z N

• o o .00 .00 2.83 3.17 3 .33 3.33 2.21 18.00

- 2.83 - 3.00 -.67 - 2.50 - 3.50 - 1.00 — 2.33 - 1.91 26.00

. 1 7 .63 - 1.00 - 1.50 . 00 - 1.67 - 1.83 - . 48 26.00

-.67 -.17 - 2.00 .83 -1 .67 -1 .33 -.83 -.36 26.00

. 1 7 . 1 7 .63 -1 .17 - 1.67 - 1.67 .00 -.78 25.00

^ • 0 c 2.50 1.17 1 . 50 3.67 2.33 2.17 1 . 85 26.00

FIGURE V.38 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

205
EVAL SUB-FACTOR

s . 1 . 2. 3. 4. 5. 6. 7. a . 9.

BIDDER 201. 4 .00 2.00 3.00 3.00 3.50 2.00 2.50 .00 .00

202. -2.00 .50 -1.50 -1.00 -1 .50 1 .00 -3.50 .50 -1.00

202. -2.50 -1 .50 -2.50 -1.00 -3.00 .00 -1 . 00 -1.50 .00

204. .00 -2.50 .50 1.00 .00 .00 .50 .50 .50

205. -1 .50 -.50 -1.50 1 CM • 0 o - 1 . 00 -3.50 . 00 .CO -1 .50

2 oe . 2 . CO 2.00 2.00 .00 2.00 .50 1.50 1 .00 .50

10 . 1 1 . 12. 13. 19. 20 . 21. to to . 23. 24.

• o O .00 .00 . 00 1 . 50 1 .00 2.50 .50 3.50 1 . 50

1 •

CJ 0 -4 .50 -1.00 -1

.

00 -3.00 -3.50 t • o o 1m • 0 o -4.00 -5.00

. oc 1.50 -2. CO -1 . 00 -1.00 2 .00 -2.50 -.50 -2.00 1 .50 — .5 0 1.00 -.50 -.50 .50 .50 • oo 2.50 1 . 50 2.00

-3.5 C -1.00 -1.50 1CM • 0 O .00 .00 .50 .50 1 .00 2.00

2.50 1.00 2.50 2.00 2.00 .00 2.50 . 00 . CO -2.00

25. 25. 27. 30. 31 . 32. 33. I N

.00 .00 .00 2.50 3.00 2.00 3.00 2.28 18.00

-3.50 -3 . SC .00 -1.50 -3.50 -1.50 -2.00 -2. 12 26.00

.OC -.50 -1 .50 -2.00 -1.50 -1 .00 -2.00 -.98 26.00

2.00 1.00 -1.00 1.00 .00 .00 .00 • 42 26. CO

-.5 C -.50 .00 -1 . 00 -1.00 .00 -1.00 -.78 25.00

1.5 C 1.50 .50 1.00 3.00 .50 3.00 1 . 27 26.00

FIGURE V.39 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 205 PARTICIPATED
2

EVAL BIAS 5.

BIDDER 20 1 . + + • + + + + + + - +

202 . - -

203. - - 444.--Q--

204. — - + -- -44 + 44---4.---

205. ------- __«__4.44.4.4444. + ___0

206. + + + + 4 + +

EVAL INFLUENCE

5 . BIDDER 201. t--i- + + + + +- +- + - +

2C2.

203. + — —

204. - + + T- -- -- -*-«-+4'-* + 4-*-*--'f>---

205. + »+• + - 4444-.f--4444.--44.__

206. 444- 4444444444-4--44444444

FIGURE V.40 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

205
255

EVAL SUB- 8 . 1 . 3 * 5 . 6 . BIDDER 201 . - 2.23 -.50 .60 - 2.25

202 . .82 .83 . 10 1.38

202 . .67 2.00 -.60 .25

204 . o o -.17 1.40 .13

205 . 2.00 1.50 1.10 .25

208 . - 3.17 - 3.67 -2.60 -.25

14 . 15 . 16 . 1 7 . 18 . -.25 .50 -3.00 -.50 - 1.00

-.82 -.50 -.33 .25 -.25

.00 1 .32 1 .33 -.75 .25

2.17 -.32 .83 1.50 1.25

00 I'J . 1 .50 .67 . 25 .75

- 2.17 -1 .82 .33 - . 25 -.75

z N

-.14 1 9 . 0 C

.76 19.00

-.0 1 19.00

.44 1 9.00

1.09 19.00

- 1.67 19.00

FACTORS

9 . O 0 1 1 . 12 . 13 . -.25 2.00 - 1.17 .50 2.25

1 .25 .50 2.60 .75 .75

-.75 -1 .83 . 40 - 1.12 -.25

1 .62 1 .00 -. 50 1 .12 - 1.00

.88 1.33 . 30 1.75 1 .50

• 3.37 - 2.33 - 1.10 - 1.75 - 2.12

25 . 26 . 27 . 28 . 29 . 2.50 - 1.25 .00 -.25 .75

2.50 1.00 1 . 33 1.00 .75

- 1 .17 - 1.17 1 .67 -.50 .00

-2.00 2.50 .00 - 1 .75 -1 .50

.17 .83 . 17 2.00 2.00

-1 .33 - 2 . 17 -1 . SO -.25 - 1.50

FIGURE V.41 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

208
256

EVAL

a. 1 . 3. 5.

SUB- 6.

FACTORS

9. 10. 1 1 . 12. 13 . BIDDER 201 . . 00 .00 1.50 -2.00 2.50 3.00 -.50 .50 2 .50

202. .CO 1 . 00 .00 2.00 .50 .00 . oc -.50 - .50

203. - .so - .50 -2.50 .50 -1.00 -2.00 1 .00 -2.00 .00

204. 2.00 . 00 2.50 1 .00 2.00 1 . 00 . 50 1 .5C - .50

205. .50 .50 1.00 - 1 .50 . 00 .00 . 5C 2.00 .50

206. -2 . 00 -1 . CO -2.50 . CO -4 . 00 -2.00 -1.53 -1.50 -2 . 00

1 4 . 15. 16. 17. IS. 25. 26. 27. 26. 29.

• O o l.OO - 3.00 .50 .00 1 .00 1.50 - . 50 .50 1 . 00

.30 -.50 .00 . 00 - 1.00 .50 -.50 1 . 50 .00 -.50

• o o . 5 C 1 .00 - 1 .00 .00 - 1 .00 - 2.00 . 50 - 1.00 .00

2 . 3 C -.50 1 . 30 1 . 00 1 .00 1 .00 3.00 . 50 -1 .CO .00

• o o .50 .00 . 00 1 . 00 -.50 . 00 - • 50 1 .CO .50

- 2 . oc -1.00 1.00 -.50 - 1.00 - 1.00 -2.00 - 1.50 . 5C - 1.00

N

.50 19.00

. 1 1 19. OC

-.52 IS. 00

.55 19.00

.25 19.00

1.22 19. CC

FIGURE V.42 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 208 PARTICIPATED
257

EVAL BIAS a.

BIDDER 201 . 202 . + + —

202 . +-+--+-- 0 ++

204 . +— + + - +

205 . + + + • + • + ••

206 . ------------

EVAL INFLUENCE

e.

BIDDER 201 . --++-+++-++

202 . - -

202 . --++++++- 0 ++

204 . + — + 4 + + — < + • + + 205 . ++---+++--

206 . + 4+-+++++++4

- - - 0 - 4

4 - 4 4 4 4 4

- 4 - - 4 - 0

4 4 - 4 0 --

4 4 4 4 4 4 4

- - 4 - - - 4 - 44 - 4 --

4 - 4 4 4 4 0

4 4 - 4 - 4 -

- 4 - - - 4 4

4 4 4 4 4 — 4

FIGURE V.43 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

208
25

EVAL

10. 5 . 1 1 . 12.

SUB- FACTORS

13. 1<5. 20. 2 1 . 22. 23.

BIDDER 20 1

-2 .4 0 -.50 -.25 -.75 -.13 - . 13 1 . 20 3.17 2 .23

202. -1.70 -1.00 -2.37 -1.75 - . 13 - .88 - 1 . 1 0 -.17 - .17

202. .60 1 .60 2.00 2.25 1 .62 1 .66 . 9 C -.17 1 . 1 7

204. 2.60 -1.10 1 .75 2.12 -1.25 -.38 -2 .00 -2.50 - 1 .33

205. - .70 .20 .50 .66 .26 .12 . 30 -.22 - .50

206. 1 .C 0 1 . 20 -.50 -2.12 -.50 - .6 3 .70 .00 -1 .50

24 . 2 ^ • 2 o . 27 . 26 . 29 . 20 . 3 1 . 32 . 32 .

.00 -.50 1 .00 . 00 1 . 25 . 00 - 1 . 63 -1 . 50 - 2.00 - 2.00

1.17 -.82 - 1.00 - .67 - 2.-^5 - 2.25 .17 . 50 - 1.67 2.50

. 17 2.17 2.17 - 2.22 .25 -.75 1.17 - 1.22 . 22 -.50

1 • o 1 .22 — . 6 o .67 -.25 .00 . 17 3.67 2.00 1.17

-.67 .62 .63 .83 . 00 1 .25 - 1.17 1 . 00 - 1 . CO -2 . CO

- 1.22 - 2.67 - 2.17 - . 17 2 . 00 .CO 1 . 50 - 2.32 2.22 .82

y N

-.16 17.00

-.74 19.00

.69 19.00

.26 19.00

.05 1 a.oc -.20 18.00

FIGURE V.44 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

210
259

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

I

EVAL SUB-FACTOR

1 0.

BIDDER 5. 11. 12. 13. 19. 20. 21. 22. 23.

201. -1 .00 .00 .00 .50 1 . 00 1 .00 2. 50 3.50 3.50

202. -1 .50 -3.00 -3.00 -2.50 -1.50 -2.00 -2. 00 -1.50 -2.00

203. -1 .50 2.00 . 50 2.00 .50 2.00 -. 50 -.50 .00

204. 3.50 .00 2.00 2.00 .00 .00 -1.50 -.50 .00

205. -.SO .50 1.00 .00 .00 .00 .50 -.50 .00

206. 1 .00 .50 -.50 -2.00 .00 -1 .oo 1 .00 -.50 -1.50

24, .

in 26. 27. CD • 29. 30. 31 . 32. 33.

.00 -1.00 3.00 . 00 1.50 .50 -l.CO -.50 -2.00 -1 .00

1 • O o -2.00 -2.00 . 00 -2.50 -2.50 .50 -.50 -2.00 1 .00

.30 1 .50 .50 -2.50 -. 50 -.50 . 00 -2.50 .50 -1.00

• o o 3.50 .50 1 . 00 . 00 1 .00 .50 4.00 2.30 1 .50

.50 .00 .00 . 00 .00 .00 -1.00 1 . 00 .50 -2.50

-2.50 -2.00 -2 .00 -.50 2.00 .00 1.00 -1 . 50 .50 2.00

2 N

.62 17,00

i.se 19. CO

.03 19.00

1.05 19.00

-.03 18.00

-.33 1 £.0C

FIGURE V.45 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 210 PARTICIPATED
260

EVAL BIAS 10 . BIDDER 201 . -+ + 0 ----

202 .

203 . + + - +

204 . + + +

205 . - + •«• + + + -- - + >+ 4- + --

206 . + 4 ----+ 0 -----+ 4-4+

EVAL INFLUENCE

10 . BIDDER 201 . + -- -- -444 4+ 4- + + 4t

202 . 4++*+4++4-44-444-44

» 203 . -4++44-4-4+44-4-+44

204 . 4-+4--44--4-4--4444

205 . 4+4---++----- -44.4

206 . ++++-++-++4+4+ 44+4

»

FIGURE V.46 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

210
261

EVAL SUB- FACTOR

11 . BIDDER1 . 3 . 4 . 5 . 6 . 7. a. 9 .

20 1 . 1 .67 1 .25 - 1.83 - 2.50 1 • 0 1 .25 -.67 .00 .50

202 . -1 . 17 .00 .17 2.00 .70 .63 . 83 -.17 1 .88

203 . .67 1 .00 .00 -.50 2.40 1 .50 . 33 1.17 -.13

204. -.67 2.75 -. 17 -.50 - 1.60 -1 .12 - 1.17 - 1.50 -.88

205 . .67 - 2.00 .17 1.25 -.10 1 .50 1.00 .25 -.38

2oe . - 1.17 - 3.00 1.67 .25 -.80 1 .00 -.33 -.17 - 1.50

10 . 1

1

. 12 . 1 14 . 15 . 16 . 17 . 18 . 19 . -3.0 0 - 1.17 — .25 - 1 .50 -.25 -1 .00 1.50 . 50 1 .00 - 2.62

.50 2.60 1 .37 .75 -.83 . 1 7 .33 .25 1.25 -.75

.83 -.20 1.37 -.25 2.67 -1 .33 -.67 .75 .25 2.87

.

(ii Id 3.1 C -.75 — . 38 -.50 .33 . 17 -.75 -1 .00 .63

1.33 - 1 . 5 C -.75 — 1 .62 -.83 -1 . S 3 .00 .25 -.75 -.88

-.33 - 2.30 .13 3.50 .50 4.17 - 1.00 -.25 .00 .75

20 . 21 . 22 . 23 . 24 . 25 . 26 . 27 . 28 . 29 .

1 ro • o o - 1.80 - 2.17 -3.00 -1 .50 - 2.00 .25 . 00 - 1.00 -.75

-.25 1 .30 .50 1 .83 2.50 1.17 3.00 .00 1.75 1.50

• o o • 90 .50 .50 -1 . 17 - 1.17 - 1.83 1.67 • 25 .75

.25 -.80 3.50 2.67 -.50 1 .33 - 1.50 1.33 2.00 1.50

. 13 -.30 - 1 .67 - 1.17 - 1.33 -1 .17 - 1.83 - 1 . 83 - 2.00 - 3.25

1 .88 .70 -. 67 -.83 2.67 2.00 1.83 . 50 - 1.75 1.50

FIGURE V.47 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

211
26

30 . 31 . 32. 33. N

X . 1 7 -2.17 -2.00 -.67 -.95 33.00

3.50 1.83 1 .00 -.17 .87 33.00

-.53 1 .33 .33 2. 17 .49 33.00

-.50 -.33 .00 . 50 . 18 33.00

.83 -1.00 1 .67 1 .33 -.48 33.60

1.83 .33 -1.00 -3.17 .10 23.00

FIGURE V.47 Continued
263

EVAL SUB-FACTOR

1

1

. 1 . 2 . 3 . 4 . 5 . 6 * 7 . a. 9 . BIDDER 201 . 3.00 1 . 00 - 1 . 00 -1 .00 .50 -2 .00 -.50 -.50 3.00

202 . -1 .50 1 .00 .50 1.50 .50 .00 - 1.00 1 .50 1 .00

203 . - .50 1.00 - 2 . 00 - 1.50 .00 1 .50 -.50 -.50 -.50

204 . .00 1.00 .00 .00 .00 .00 . 00 -.50 .00

205 . -.50 - 1.50 -.50 .50 .00 .50 2 . 00 1.00 - 1 .00

206 . -.50 - 2.50 3.0 0 .50 - 1.00 1 .00 .00 - 1.00 0

tn • 1 CM

1 0 . 1 1 . 12 . 13 . 14 . 15 . 16 . 17 . 16 . 19 .

.00 -.50 .00 • O O • o o .00 .00 1 . 00 1.00 - 1 .00

.00 .00 .00 -.50 . 00 • O O .50 .00 . 00 1 CM • 0 O

.00 .50 .00 .00 2.00 -1 .50 -.50 .00 .00 l .50

.50 ' 3. 50 .00 .00 • O o .00 .50 -.50 -.50 1.50

• o O -1 .00 .00 1 (M • 0 o - 2.00 - 2.00 -.50 .00 • o O 1 • 0 o

0 1 in - 2.50 .00 2.50 .00 3.50 .00 -.50 -.50 1 .00

20 . 21 . 22 . 23 . 24 . .

CM in 26 . 27 . 28 . 29 .

0

tn 1 • • 00 -.50 -.50 . 00 -2.00 2.50 -.50 .00 .00

0

to 1 • • 00 - 1.00 -.50 .00 -.50 1.00 .50 .50 .00

.50 -•50 . 00 -.50 -.50 -1 .00 - 2.50 .50 -.50 • 50

.50 -.50 4.00 3.00 .00 ^.50 .00 1 . 50 1 . 50 2.00

. 00 • CO - 1.50 -.50 . 00 -1 .50 - 2.00 - 2.00 - 1.00 - 3.00

1.00 1.00 - 1 .00 - l.OO .50 1 .50 1.00 • 00 -.50 .50

FIGURE V.48 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 211 PARTICIPATED
2Sk

30 . 31 . 22 . 33 . T N

-.50 - 1.00 - 2.00 .00 -.06 33.00

3 . CO .5 0 . 00 - 1.00 . 08 33.00

1 . 5 C - .50 . 50 1 .00 -.17 33.00

.00 1.00 1 .00 1 .00 .73 33.00

.50 -.50 2.50 .00 -.55 32.00

1 .50 .50 - 2.00 - 1 .00 -.03 33.00

FIGURE V.48 Continued
265

EVAL

11 . BIDDER 201 . ++-----0

202 . - 0 +++-+-

2 C 3 . • + C -+ + + +

204 . 205 . -++-+++

206 .

EVAL

11 . BIDDER 201. + + + + -»• + -

202 . +-+

0 202 .

204 . 205 . -+-+--

206 . + + + -•- +

BIAS

+

-+- + --4--- + + + 0

- + + — •f — - --- + + 4 + -- 0 44

INFLUENCE

+ - 4 ----- + + + +

+ -- ”- -- + ~- + 4 -- -- + •»• + -- -

- + + — - ++ 44 -f

— -4 + -- - + -

+ ++ - + - + -- +

• + 44 + + 044 '*- 44 ~

+ + 4 - -- 4 + + - + + + ---•- + 44--04

+ - — + + +

-+ + - + + -- - + + + -

---•• + + 4 - + 4 - + -4- 44 + - 4 - + + 4 — - -4

- 44 - + + 44 + + 44 - -f 4 + 4 + - + + 4 + + +

FIGURE V.49 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

21 1
2^6

EVAL 2. 1 . 2. 3. 4 BIDDER

1 0 L . -1.50 -1.17 -.50 -.30

1 02. .67 ,67 -.17 - .33

103. .00 • o o .00 .00

16. 17. 18. 19. 20.

1 .50 -.50 -.50 -.50 -.50

.50 - .50 -.50 -.25 -.38

. oc .00 .00 .00 .00

SUB-FACTOR

9 . 10. 1 1

14. 1 5

50 - .67 -.30 0

U) 1 • 1 • o

00 • o O -.50 -.13 -.38

00 .00 o o > C 0 • o o

26. 30. 3 1 . 32. 33.

.50 . 25 . 00 .25 -.75

.40 -.75 -1 . 00 - 1 . 00 .CO

.00 .00 .00 .00 .00

2 , N

-.55 15. CO

-.26 17. CO

.00 .00

FIGURE V.50 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

102
267

EVAL SUB-FACTOR

2 . BIDDER1 • 2 . 3 . 4 . 9 . 10 . 1

1

. 14 . 15 .

10 1 . .38 .63 . 13 .13 . 10 .50 . 25 . 1 0 .10

102 . .00 - 1.00 .38 .25 .00 .00 -.42 1 • 0 .20

1 C 3 . .CO • o o .00 .00 . 00 .00 • o o • o o .00

16 . 17 . 18 . 19 . 20 . 26 . 30 . 31 . 32 . 33 .

.50 .25 .25 .10 .10 .08 .67 .50 .67 .50

-.17 .25 .25 -.20 -.80 .17 - 1.00 -. 17 - 1.17 .00

.00 .00 .00 .00 .00 .00 .00 .00 .00 .00

2 N

• 3 X 19 . CC

-.21 17 . OC

.30 .00

FIGURE V.51 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 102 PARTICIPATED
26 ?

EVAL bias

2 . BIDDER 101. ----------------O-*--

102 . ++--

102 .

eval influence

2 . BIDDER 101 . -

IC2. - -- - + M - 0

1 03.

FIGURE V.52 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

102
269

EVAL SUB-FACTOR

5 . BIDDER5 . 1 1 • 12 . 13 . 19 . 20 . 21 . 22. 23 .

10 1 . .83 .30 • 33 .33 .13 .13 • c o -.50 -.50

1 02 . -1 .50 .10 .00 .00 -.88 -.38 .17 .83 .67

103 . .67 -.25 -.33 -.33 1 . 00 .50 -. 17 -.33 -.17

24 . 25 . 26 . 27 . 28 . 29 . 30 . 31 . 32 . 33 .

. OC .00 .10 .00 .00 .00 1.00 .75 1 .00 1 .50

.17 .25 .20 .38 .17 .17 -.75 -.25 -.25 -.75

-. 17 -.25 -.13 -.38 -. 17 -.17 .00 . 00 -.50 -.50

z . N

• 2 e 19.00

-.05 19 . CO

- -. C 5 19.00

FIGURE V.53 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

105
EVAL

5. 5 . 1 1 BIDDER 10 1. .C3 .25

102. - . 13 -.42

103 . -.50 .30

24 . 25 . 26.

. OC .00 .08

.3t .20 .17

.36 — • 2 C -.10

SUB-FACTOR

12. 13. 19.

.25 .25 .10

.00 - 1.00 -.20

.75 .75 .25

N #

CM 28. 29.

• o O • o o o o

.30 . 13 . 1 3 - 1

-.30 -.13 -.13

20. 21. 22. 23

1 0 o o .13 . 13

eo . 1 2 . 12 • o o

as -.12 -.25 -.12

30. 3 1 . 32. 32 .

.67 • in o .67 .50

c> • o - . 17 -1.17 • o o

•

in o • o o .75 -.25

: N

.22 1 9 . 0 C -.27 19.00

.12 19 . C C

FIGURE V.54 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 105 PARTICIPATED
271

EVAL

5*

BIDDER

BIAS

XOX. •>«• + + + + 0 --00 + 000 + + -»- +

102. - + 0 0 - - + + + -

103. » 1 1 1 1 1 1 1 0 o 1 1

EVAL

5.

BIDDER

INFLUE^JCE

101. + + + 0 --00+C00 + + + +

102. +---+++ + -- + + + + + + -

103. ----+++ + +- + + + + + -0- +

FIGURE V.55 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

105
272

EVAL 6. 1 . 2. 3. 4.

SUB- FACTOR

5. 6. 7. e. 9.

BIDDER 10 1. .50 .83 . 1 7 .17 .83 .50 1.00 .75 -1.12

102. - .67 - .67 -. 17 -.33 - .83 1 .33 -1.75 -1.75 .63

103. .50 .00 >25 .50 . 00 .63 •y c; • w 1.00 .50

10 . 1 1 . 12. 13. 14. 15. 19. 20. 2 1 . 22.

-.67 -.90 -1.00 -1.00 . 1 3 . 1 3 . 1 3 . 13 .00 . 17

1 .25 1 .30 1.33 1 .33 -.13 .25 . 38 .25 . 17 . 1 7

-.25 -.25 -.33 -.33 . 1 7 - . 17 -.33 - . 17 - . 17 ~ • 3

^

23 . . 24 . 25. 26- 27. 2b . 29. 30. 3 1 . 32 .

. 17 .00 . 00 .10 . 00 .00 .00 -1.25 - .75 -1.25

-.67 . 1 7 “ 1 • C 0 -.40 -.25 - .50 -.50 1 . 50 1 .25 1.25

.50 -.17 • o o .50 .25 .50 .50 . 00 .00 .50

33. T N

•.75 -.10 30.00

.75 .04 30.00

.50 .2 1 30.00

FIGURE V.56 INDIVIDUAL DIFFERENCE SCORES FOR PANEL MEMBER

108
273

EVAL SUB-FACTOR

8 . BIDDER1 . 2 . 3 . 4 . 5 . 6 . 7 . 8 . 9 .

tox. ..38 .63 .13 .13 . €3 •

CJ CD .67 .50 .10

102 . .00 1 • o o .38 .25 -. 13 -.50 -.67 -.17 .13

IC 2 . -.17 •

in o -.33 -.17 1 •

<n o .13 .00 -.33 -.13

10 . It. 12 . 13 . 14 . 15 . 19 . 20 . 21 . 22 .

.50 .25 .25 .25 .10 .10 .10 . 10 .00 .13

-.17 -.42 1 • 0 o - 1 .00 -. 10 .20 -.20 -.30 .13 .13

-. 17 .30 .75 .75 . 13 -.13 . 25 . se -.13 -.25

23 . 24 . 25 . 26 . 27 . 28 . 29 . 30 . 3 1 . 32 .

. 1 3 .00 .00 . 08 .00 .00 .00 .67 .50 .67

.30 -.38 .20 ,17 .30 .13 .13 -1 .00 -.17 - 1.17

-.13 .38 -.20 -.10 -.30 -.13 -.13 .50 .00 .75

33 . 2 N

.50 .26 30.00

.00 -.22 30.00

•.25 .06 30.00

I

I

I

FIGURE V.57 DIFFERENCE SCORES FOR SUB-FACTORS ON WHICH

PANEL MEMBER 108 PARTICIPATED

I

I
21h

EVAL BIAS

a.

101 . + + + + + + + + -- -- - + + + + 0 -*- * 00 ^ 000 ----

102 . - + + + + *

102 . 4 0 i *0*-***----*~--~-*-**-*-*-*’00*

EVAL INTLUENCF

a. BIDDER 101 . + + + + ^ + + + -- -- -•f •^ + •0 + 00 000 ----

102 . -+--++>++----++-- ------------

102 . + - +

FIGURE V.58 BIAS AND INFLUENCE DISPLAYS FOR PANEL MEMBER

108
Evaluator +/- Ratio Evaluator

201 20/30 .67

202 27/48 .56

203 4/11 .36

204 24/50 .48

205 66/147 .45

206 12/24 .50

207 6/24 .25

208 55/112 .49

209 8/12 .67

210 48/110 .44

211 112/198 .56

212 22/35 .63

+/- Ratio

101 18/26 .69

102 9/35 .26

103 26/32 .81

104 18/26 .69

105 35/50 .70

106 3/4 .75

107 14/26 .54

108 33/83 .40

109 2/11 .18

110 14/15 .93

111 2/6 .34

112 14/23 .61

FIGURE V.59 RATIO OF PANEL MOVEMENT TOWARD/AWAY FROM

RANKINGS OF EACH PANEL MEMBER
influence indicators show that he was unable to sway the

other panel members. For bidders 201 and 202 the teams'

final rankings tended to move away from their average rankings in a direction opposite evaluator 205 's initial scores

For bidder 206 they moved slightly toward 205’s initial

scores. On the average, across bidders, the panel rankings

tended to move toward bidder 205 's rankings 44^ of the time

For bidders 201, 202, and 206, these percentages were 23!o,

31^ and 62^ respectively.

Evaluator 208 ' s scores show bias toward bidder 205

and against bidder 206. For bidder 205 the teams' rankings

moved somewhat toward evaluator 208 's while for bidder 206

they moved slightly away from 208 's. Across sub-factors

the panel moved toward bidder 208 ' s rankings ^ 9 % of the

time. For bidders 205 and 206 the percentages were 7

'^%

and 26^.

The scores of evaluators 210 and 211 do no show any

strong bias for or against any of the bidders. For evaluator 210 the percentage of +/- across, bidders is 44“5. For

evaluator 211 it is 56 . 6 %.

The percentage scores from the agency impact contract

data indicate that panel members had varying degrees of

influence over the final rankings. Comparison of the difference scores shows that panel members who were biased

toward specific bidders were not able to sway the results.
In fact, the panel often tended to move away from those

who were biased.

The data for commercial impact is more difficult to

interpret. The computations for bias and influence are

sensitive to missing data, the number of bidders, and the

g degree of initial agreement in the rankings. Since there

were only three bidders, since evaluator 102 failed to rate

bidder 103 on any of the nineteen sub-factors on which he

was a participant, and since there was a high degree of

initial agreement in the scores, it is difficult to say

much about this data, other than that with the exception

of bidder 102, there does not seem to be any bias in it.

Figure V.60 presents two responses to open ended questions which relate to the treatment of bias.

Other issues related to this one include: how the

panel dealt with differences in uncertainty among bidders

during the evaluation (discussed in section V.3.2.4); and

whether evaluation while knowing the identity of bidders

created any problems (discussed in section V.3.7.3).

Where a panel member rates only bidder 101 and 102

the rankings come out 1.5 for the lowest and 2.5 for the

highest. If there is a great deal of initial agreement

that 101 is highest (i.e., those rating all three bidders

rank him 3), the panel member will always appear biased

against bidder 101. If the team agrees that bidder 101

is highest, the final team score will always appear to move

away the panel members with missing data and toward the

other panel members.
278

Response From Panel Question 5 on Strongest Aspect of the

Process

o The system was superb. I wouldn’t want to do it any

other way. Its strength is that you don’t have to

pretend that you have no biases. It brings a very

professional way to channel personal opinion and judgment and feeling in an honest, constructive fashion.

It makes subjective objective (PM12).

Response From Bidder Question 11 on Weakest Aspect of the

Process

o It gave people at ETIP an opportunity to try to take

sub jective ‘ things and make them objective in the sense

that they were using the same rigor as a serious qualified proposal team would use in laying out how they

were going to respond. I’m not saying it worked (3R4).

FIGURE V.60 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO THE TREATMENT OF BIAS

V. 3 . 7.2 Consistency of evaluation across bidders .

The next issue concerns the criteria for rating sub-factors

and whether they provided a consistent evaluation across

all proposals. When broad categories of evaluation criteria

are used and the evaluation is made against separate designs

proposed by each bidder, there may be considerable difficulty in achieving a comparable basis for evaluation. Each

proposal will surface different issues and there may be

little basis for comparability. For example, in a prior

ETIP RFP for evaluation services, the broad categories of

Methodology and Qualifications of Offeror were used. There

were five sub-categories within the first category and two

within the second. The RFP indicated that the panel would

complete its evaluation on the Methodology category before
279

beginning its evaluation on the Qualifications category.

In an evaluation like this, the panel must read through

the methodology section and decide on criteria within each

of the sub-categories. Since issues which the bidders choose

to discuss will vary, it may be difficult to get a strong

basis for comparison. Also since qualifications are evaluated later, the scores on methodology are awarded with no

consideration given to the capability for carrying out .the

approaches proposed. When qualifications are considered,

the structure provides no basis for relating them back to

the specific approach' proposed by each bidder. The result

may be that the qualifications points are awarded merely

on the reputations of the various people whose resumes are

included in the proposal. Consideration to whether they

» have a capability to work together as a team to manage and

carry out the approach proposed may be neglected.

The PFE process was successful at dealing with this

issue. The sub-factors provided the categories for consistent response by bidders and the three criteria of understanding, approach and present or potential capability provided for consistent evaluation. All bidders were required

to not only demonstrate methods or approaches but they were

required to link the evidence for capability to each method

or approach. Every bidder was evaluated against the same

issues

.
2

Sometimes during a team meeting a panel member would

argue for a high score for a bidder based upon his proposed

approach for dealing with a sub-factor. Another panel member might agree that the approach looked good but point

out that he could find no evidence that the bidder was capable of carrying it out. The evidence on all three criteria

would be examined until agreement was reached.

V.3.7.3 Admissible information . The final issue in

this category of equity of treatment is the impact of the

lack of restriction on admissible information. There was

a strong concern over both potential inequities and appearances of inequities which might be caused by a "non-blind”

evaluation of proposals (see section IV. 3 ).

The ability to use all available information was a strength of the evaluation. There is some inconclusiveness

about whether any inequities were created. A question with

two sub-items was included in the panel questionnaire to

obtain reaction to this issue the items and responses to

them are presented in Figures V.6I and V. 62 . The panel

Did being able to use all available information about bidders in the evaluation improve the accuracy of the scores?

Yes - 15 No - 1

• Remark by a person answering Yes

o I don’t really remember. My feeling is that this is generally more useful (PM6 ).

CO
Remark by person answering No

o I think it's better not to know who the bidders are (PM1).

FIGURE V.61 PANEL MEMBER’S RESPONSES TO QUESTION 4a WHICH

RELATES TO ADMISSIBLE INFORMATION

Did being able to use all available information about bidders in the evaluation introduce unfair bias into the scores

Yes - 3 No - 13

Remarks by those answering Yes

o For those that you know, you start reading into the proposals things that they don’t mean or haven’t said because

you are biased (PM1).

o The additional information improves accuracy, as would

any relevant increment of information. However, the person supplying the information often exhibits a bias (i.e.,

it is easier and more comfortable to consider working

with people you know) (PM2).

o If one individual knew the firm and had problems with

it, that convinced the rest of the group. Whether they

really did poor work, or he just thought it was, I don’t

know. It did carry a lot of weight. If only one of the

team knew of the company the team tended to accept it.

If more than one knew it there would be discussion and

I think it would come out fair (PM16).

Remarks by those answering No

o Obviously there is a value judgment as to what is unfair.

Information on past performance is relevant and not at

all unfair (PM5).

o If a panelist has had any unusual or direct experience

you ought to be aware of that so you can interpret any

bias (PM9).

If 'a bidder never completes their work on time and therefore you don’t select him, I don’t think it’s unfair bias.

I think it’s a smart way of doing business (PM4).

o
2

0 One proposal had been submitted by a contractor who had

not done a good job for ETIP in the past. The parts of

the proposal that I read would lead you to believe that

he had done a superlative job. The fact that we could

discuss his past performance and bring that to bear, I think, was a good idea (PM15).

FIGURE V.62 PANEL MEMBERS’ RESPONSES TO QUESTION 4b WHICH

RELATES TO ADMISSIBLE INFORMATION

members generally felt that being able to use all available

information about bidders in the evaluation:

o improved the accuracy of the scores (Yes -15;

No - 1 ) ; and

o introduced unfair bias into the scores (Yes

it did - 3; No it did not - 13).

The remarks by the three people who answered Yes to

question 4b (see Figure V.62) indicate how they feel unfair

bias may have entered into the scores. PM1 and PM2 indicate

why they feel that people evaluating proposals may be biased

but do not indicate how this bias- may have affected the

final sub-factor/factor scores (as compared to the panel

members’ initial scores). PM3 indicates that if only one

person knew of a bidder’s prior performance, this tended

to carry a lot of weight. Possibly too much weight.

Remarks by some of the people who answered No to question 4b indicate why they feel that the use of outside information led to proper use of justifiable biases. They indicate that outside information allows the panel to evaluate

the information in the proposals better and to interpret

CO
283

the biases of panel members. The panel does not have to

rely only on what the proposal writers say (e.g., see response of PM15, Figure V.62).

"Blind” proposal evaluation may frequently not be "blind"

at all. Panel members will recognize bidders they are familiar with through the approaches proposed, writing style

or the way the proposals are bound. In this ca^e panel

members do not have to justify their biases (since they

are not able to admit that they know whose proposal, they

are evaluating). Also the panel is not able to differentially score a bidder who has not worked for the program

before and makes an error of interpretation and a bidder

who has worked for the program before and makes an error

because he is unable to learn. This relates to the discussion under section V.3.2.4 on differences in uncertainty

among bidders. This issue also relates to the issue of

treatment of bias (section V.3-7.1) and those on traceability of the process (section V. 3 .IO).

V. 3.8 Risk Associated with the Statement of Work

The next category of issues concerns whether ETIP was

able to accomplish the writing of a flexible statement of

work and deal with the added risk associated with this.

It covers issues of writing and getting the Statement of

Work accepted, having bidders understand it, and being able
to make the assessment of bidders’ objectives necessary/

to assuming the added risk associated with it.

The three specific issues covered in this section are:

1. Was ETIP able to write a statement of work

which would spell out deliverables and yet

avoid numerous contract changes which generally

occur in situations where requirements, policies, and the state of the art are changing?

Did ETIP management accept it?

Did the contracting officer accept it?

2. Did the flexible statement of work scare off

bidders?

3. Was ETIP able to assess the com.mitment of the

bidders' top management to devote the necessary

resources to the project?

Was ETIP able to determine whether it was

within the longer term interest of management to develop the desired systems?

V.3.8.1 Flexible statement of work . The first issue

under this category is whether ETIP was able to write an

acceptable statement of work which would spell out deliverables and yet avoid numerous contract changes which generally occur in situations where requirements, policies, and

the state-of-the-art are changing. Acceptable is taken

in the proximal sense of whether a flexible statement of

work would be approved by both the program and the contract

ing office.

STIP's results on this issue were mixed. Thompson

$ was able to write a short flexible statement of work which

followed the format of the factors and would not require

modifications as changes occurred. It indicated that both

progress and final reports would be expected on both techni
cal and management aspects of the project. However, it

left the dates and specific subject matter of these reports

undefined. ETIP management felt that the statement of work

needed to be more specific. Therefore, Thompson also included another version of the same statement of work to which

he added illustrative sub-items which included specific

activities and reports and dates for the reports. These

items were intended only as examples to the prospective

bidders of the types of things which could be furnished

under the statement of work. This second illustrative statement of work caused problems for ETIP and the successful

bidders during the early parts of Phase I (see section

V.3. 14)

The contracting officer had no problems with the statement of work. He had previously contracted for ships for

the Maritime Administration and recognized the usefulness

of having flexibility in a complex, uncertain contract.

V.3.8.2 Effect of statement of work on bid decisions .

An issue which was raised by some of the ETIP staff and

PAA staff was that the flexible statement of work would

scare off bidders. The point was that they would not understand what was expected of them and would therefore not

bid.

The answer to whether this happened is unclear. It

is quite possible that some of the bidders were confused
by the statement of work . However, whethe r this was good

or bad is related to the issue on selectiv ity of the proc

( se c t io n V. 3.3.2). It may be that bi dders who were scare

off by the statement of work and the uncer ta inty of what

was exp ected of them wou Id have failed rap idly under the

unc erta inty which was present in the pro je ct env ironment

Whe then some potentially acceptable b idder s were als 0 sea

off is not known.

V. 3 . 8.3 Ability to assess risk. The final iss ue he

i s whether ETIP was able to make the asses sment of t he bi

der s’ long term objectiv es which was felt to be impo rtant

to acce pting the added r isk of using a fie xible stat emen t

of work . This included assessing the comm i tment of the

bid der ’ s top management to devote the nece ssary reso urces

to the project and asses sing whether i t wa s with in t he lo

term interest of the management of th e firms to deve lop

the desired systems.

ETIP’s results here were reported under the first and

third issues of section V.3.^. In sum, the management factor team for the agency impact evaluation system contract

found it was not able to differentiate among bidders. The

clarifications sessions enabled ETIP to assess differences

among bidders in terms of which objectives they would devote

resources to and to assign unique scores to each bidder.

However, it was noted that the level of commitment of a
successful bidder could change over time and that this issue

would have to be worked at under the contract. This subfactor was probably the most difficult of all to assess.

V. 3.9 Basis for Award

This category discusses the outcome of the process

of moving from the results of the deliberations of

the evaluation panel to the selection of specific bidders. ' It covers such issues as whether the level of

aggregation of ratings enabled differentiation among

the bidders on technical bases and whether both the

technical and cost issues were considered in making

the final award decisions.

The four issues reported in this category are:

1 . Did PFE provide for overall technical discrimination among bidders?

Was there a sufficient distribution

in the final scores to determine a competitive range?

2. Did PFE provide a basis for discriminating

among bidders on specific technical issues?

Did the specific issues get washed out

by the overall technical score?

3 . Was technical leveling a problem?

Was technical transfusion a problem?

4. How much of a factor was bid price in the

final decision?

Did the procurement turn out to be a

price competition?

Did the process provide a capability

for dealing with potential buy-ins?
V.3.9.1 Overall technical discrimination. One issue

of aggregation is whether PFE provided for overall technical

discrimination among bidders. That is, whether there was

a sufficient distribution in the final scores to determine

a competitive range. The literature on source selection

indicates the overall scores are frequently close (e.g.,

Fox, 1974, p. 269).

The distribution in the final scores was fairly wide,

though there were no bidders at the extremes of the scale.

Figure V.63 illustrates the distributions which resulted

after multiplying the factor scores by the preassigned

weights. On the Commercial Impact contract, there is a marked difference between the overall score of bidder 101

and the other two bidders. Bidder 101 was the only one

considered competitive. The results of agency impact contract were less strikingly demarcated but still provided

a strong basis for overall discrimination. Bidders 205,

202, and 203 were all considered to be in the competitive

range, with 65 points separating bidder 205 from the other

two. Bidder 575 was 120 points below 202 and 203 and was

considered unacceptable overall, as were 201 and 204 which

were further down the scale.

V. 3 . 9.2 Technical discrimination on specific issues .

The other issue of aggregation is whether PFE provided a

basis for discriminating among the competitive bidders on
Agency Impact Commercial Impact

1000 1000

900 900

800 B205 (760)

800

700 - . B202, B203 (695) 700

600 - B206 (595) 600

500 . B201 (470)

500

400

" B204 (435) 400

300 300

200 200

100 100

B101 (755)

B102 (425)

B103 (360)

FIGURE V.63 DISTRIBUTION OF FINAL OVERALL SCORES
specific technical issues, or whether these got washed out

in the overall score. This is identified as a problem in

the literature (e.g., Blue Ribbon Defense Panel, 1970).

It is important for clarifications, negotiation and final

selection that the relative strengths of the various bidders

are evident. It would be undesirable to select a bidder

who had a high overall score due to great strengths in some

areas yet had critical weaknesses in other areas.

PFE was very strong at dealing with this issue. The

individual sub-factor scores made it easy to prepare for

clarifications with the competitive bidders. The notes

in the record indicated the specific questionable aspects

of the sub-factors which the panel felt clarifications were

needed on. ETIP was able to probe the bidders for more

information on these specific issues and develop a more

confident evaluation of the bidders' ratings. A bidder

with an unresolved extremely low score on one sub-factor

could have been removed from the competitive range even

though his overall score was high. However, this case did

not occur.

Two of the panel members felt that the scoring on the

specific sub-factors was the weakest aspect of the process

(see Figure V.64). One person felt that the factor/subfactor rating scale did not provide for enough differentia-
Responses From Panel Question 6 on Weakest Aspect of the

Process

o The overall scoring aspect (the scale). I don’t think

the sheets allowed for enough division if the proposals

were close ( PM? )

.

o It went on too long and there were too many numbers

when they were really unnecessary (PM4).

FIGURE V.64 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO TECHNICAL DISCRIMINATION OF SPECIFIC ISSUES

tion among bidders who were close. The other felt that

there were too many ratings (i.e., sub-factors) involved

in the process.

V. 3 . 9.3 Technical leveling and technical transfusion .

An important issue for the clarification and negotiations

is how to handle the requirement to allow bidders to improve

their proposals without creating problems of technical transfusion and technical leveling. Technical transfusion arises

when, during clarifications or negotiation, an approach

proposed by one bidder is suggested to another as a method

for dealing with their technical problem. Though this is

illegal the literature indicates that it occurs frequently

(e.g., Babione, 1978). If used across all competitive bidders, it results in technical leveling. That is, all bidders eventually- propose the same technical design in order

to give the government what they feel it wants to hear.

PFE enabled ETIP to deal with clarification and negotiation without problems of technical transfusion. Since
clarifications were in terms of the sub-factor issues and

not in terms of a system design, there was no opportunity

for transfusion to occur.

During a discussion with the ETIP's contract negotiator

the author raised the issue of technical transfusion. The

contract negotiator indicated that it something which a contracting person always watches for but that he did not

remember it being a concern in this source selection (Contract negotiator, Note 4 ).

V.3.9.4 Weight given to bid price . An issue which

relates closely to technical leveling is whether the source

selection turns into a price competition. If there is no

basis for discriminating among competitive bidders on technical acceptability, the sole remaining criteria is price.

That means the lowest competitive bidder always wins. This

encourages the practice of "buying-in." "Buying-in" occurs

when a bidder submits a proposal which is competitive at

the lowest cost and requests modifications once under contract when it becomes clear to the government that the performance characteristics or schedule cannot be met with

the available resources. If all that is available for the

technical evaluation is a final score, and the final scores

are close, the contracting office is at risk by awarding

to anyone other than the low bidder.
This issue received a strong test in the ETIP case

and was dealt with quite successfully. On the agency impact

contract, there were two bidders who were called in for

clarifications (see section IV.11). After oral and written

clarifications, it was quite easy to differentiate the relative strengths and weaknesses of these bidders on the various sub-factors and overall. Had it not been able to obtain

clarifications on the issues which the panel was concerned

with, the contract would probably have been awarded to the

other bidder, since they came in with a lower cost estimate.

V. 3.10 Traceability of the Process

This category is concerned with whether the evaluation

can be reconstructed from the records. As part of this,

it is also concerned with whether the bidders perceive that

the process could be easily traced and therefore difficult

to fix. This category is quite closely related to the issues in section V. 3 .II on relationship with the contracting

office.

The four issues covered here are;

1 . Did the process communicate that there

was no sole source position?

Can the process be fixed?

Did bidders understand how the decisions

on who wins actually get made?

2. Can changes in scores be traced between

the initial ratings of panel members and

final panel ratings and between final panel

ratings and the award decision?
3 Did the panel follow the written procedure?

4. Did leaving the relative weightings of

the sub-factors up to the teams lead to

any problems (e.g., appearance of inequities)?

V . 3 . 1 0 . 1 Bidder's understanding of the evaluation

process . The issue of whether the RF? communicated that

the process could not be successfully fixed and that there

was no sole source position was important to ETIP. It was

thought to be critical to obtaining bids (section 17.7).

Since ETIP did not have an established record with evaluation contractors and was a fairly small program. , it was

thought that there might be problems attracting bids on

these projects with their associated complexity and uncertainty . ETIP generally did not communicate that PFE provided

any more or less integrity to the evaluation than any other

process. Three items were included on the bidder questionnaire to obtain perceptions about both understanding of

the evaluation process and whether it was capable of corrup

t ion

Did the information provided enable you to understand the

process which ETIP would use to assign scores on each subfactor?

Yes - 5 No - 1
Remark by a person answering Yes

o The amount of study and cross referencing was very costly

for a small firm (BR3).

Remark by person answering Mo

0 There was not sufficient detail (3R1).

FIGURE V.65 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

5a ON BIDDERS’ UNDERSTANDING OF THE EVALUATION

PROCESS

Did the information provided enable you to understand how

the ratings on the various sub-factors would be combined

to arrive at final scores?

Yes - 5 No - I FIGURE V.66 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

5b ON BIDDERS’ UNDERSTANDING OF THE EVALUATION

PROCESS

Did you feel that the process would be a) difficult

or b) easy to corrupt? i.e., For one person, or a group

of people, who favored a specific bidder to exercise sufficient control over the process to sway the award decision.

Easy - 3 Difficult - 2 Other - 1 Remarks by those answering Easy

0 There are some people who are domineering and since the

subjective judgments are altered in discussion, it’s very

simple to corrupt, but no more so than anything else (BR4).

o By who you select for you panel. A sophisticated proposal

team is going to try to meet each of the factors. If

your teams have had good experience with a company, that

enters in. I think there should be more emphasis on the

selection of the panel than on the structure of the process. No matter what quantitative system you have, 'it

can be corrupted by the people who are in there (BR2).

o Any subjective review is corruptible. That did not bother

us. Maybe corrupt is too strong a word. Influence may

be better (BR 1 )

.
296

Remarks by those answering Difficult

o I really wanted a choice of c) no different than any others. People who are determined to corrupt are going to

corrupt anything. If someone has made up his mind about

a particular company, there is always a way to corrupt

it (BR4).

o It would be difficult unless the entire review team favored a specific bidder (BR6).

Other

o It worked out that way. We were low bidder and had a solid proposal, but the award was made to a non-profit.

I believe the award was obviously slanted. I don't know

whether they didn't want this company. I believe personally that they wanted an academic non-profit (BR3).

FIGURE V.67 BIDDER REPRESENTATIVES' RESPONSES TO QUESTION

7 ON ABILITY OF THE PANEL TO CORRUPT THE PROCESS

The bidders generally felt that:

o the information provided by ETIP enabled them,

to understand the process for assigning scores

to each sub-factor (Yes - 5 ; No - 1 ) ;

o the information provided enabled them to understand how the sub-factor ratings would be combined into a final score (Yes - 5; No - 1);

and

o the process could be corrupted (Easy - 3; Difficult - 2; Other - 1).

The remarks of four of the six respondents to question

7 indicate that they believe the process was subject to

corruption to the same degree as any other evaluation process (see Figure V.67). One of these people felt that the

rigor applied to the evaluation did not match that which

the bidder was required to apply in responding (see Figure

V.68). One of the remaining two respondents felt that the
process, as applied, actually was corrupt. He felt the

oral clarification session was prejudical (see Figure V.68).

The final respondent felt the process would be difficult

to corrupt. V/hether some firms not bidding felt that there

was a sole source position is not known. As to whether

the process could in fact be easily fixed, the reader is

referred to the next issue (V. 3. 10.2) and the discussion

on how the process actually worked in sections IV. 10, V.3.6

and V . 3 . 7

.

V . 3 . 1 0 .

2

Ability to trace scores through the process .

This issue addresses whether changes can be traced from

the initial individual scores through to the award decision.

This was thought to be important for keeping the process

honest and for conveying to bidders that it was honest.

A panel member would probably be less likely to try to influence others improperly if a record is kept of whether this

happened.

The records which are kept by the PFE process make

it quite easy to trace the changes in scores between initial

individual ratings and the final panel ratings. The changes

in scores between the final panel ratings and the award

decision are traceable but less easily so.

The records of the panel process include the signed

rating sheets submitted by each panel member for each subfactor which they rated, tapes of the discussions by the
Response From Bidder Question 1 1 on the Weakest Aspect of

the Process

o The evaluation method. There was sufficient strength,

detail, and specificity concerning the content that a contractor was to put down. That the method of evaluation, as we understood it, didn’t seem to be as rigorous.

Some contractors may not have bid because they felt the

COTR would make the choice in the end (BR1).

Response From Bidder Question 13 on Any Other Aspects of

the Process Which You Wish to Get Out

o The oral session [oral clarifications] was highly unstructured and prejudicial. I felt they were rigged against

us (BR3).

FIGURE V.68 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO THE ABILITY OF THE PANEL TO CORRUPT THE

PROCESS
299

factor teams, and the signed final factor rating form.

To reconstruct the process, one only needs to listen to

the tapes while having copies of the rating forms. The

tapes contain a record of both the biases of the panel members and the team discussions which explored these biases

in orde^r to work toward agreement upon final scores.

The documentation which is available for the later

portions of the process includes the memorandums submitted

to the contract negotiator, the tapes of the oral clarification sessions, and the written responses of the bidders

to the questions for clarifications. From the memos and

tapes it is possible to trace how the issues which the panel

raised were followed up in the clarification process. The

one part of the process on which there is not a strong record

is the deliberations involved in re-evaluating sub-factor

ratings based on the new information obtained during clarifications. The actual new ratings are available but there

is not a taped record of the discussions as there is with

earlier parts of the process. In order to recreate this

part of the process, it would be necessary to find the

changes between the final sub-factor scores and the scores

assigned by the panels and compare these with the new information which was made available during the clarifications.

V. 3. 10.3 Degree to which the rules were followed . An issue related to traceability is whether the panel mem-
300

bers actually followed the evaluation procedure which the

RFP said would be followed or whether they changed the process or criteria in mid-evaluation. There are two ways

in which the panel might alter the rules. One is that during the evaluation process the panel might find that one

or more of the bidders have raised issues and proposed approaches which the * government did not include in the RFP.

The panel might then proceed to include these issues in

their evaluation of all of the other proposals. The other

problem arises if the panel finds that it cannot work with

the evaluation criteria which were included in the RFP.

They may be either too specific or not specific enough.

The panel might then make up new criteria as it goes along.

ETIP had no problems with this issue. The sub-factors

provided the issues for the bidders to respond to and were

the units where measurements of acceptability were taken.

The criteria of understanding, approach, and capability

were used throughout the evaluation. Had the panel discovered that a sub-factor was missing it could have been sent

to all bidders as an amendment to the RFP. However, this

did not arise.

One point which should be noted is that the sub-factor

rating process as described in RFP sounds more sequential

than it worked in practice. The RFP stated:

Members of the proposal evaluation team concerned

with each factor will meet and perform a joint
301

evaluation. They will follow an established procedure entitled, "Factor Team Guidelines," to

assure an orderly and efficient process. Briefly,

they will first compare notes informally to assure

common standards and information; second, they

will arrange the bidders in descending order;

third, they will establish the relative spread

among the bidders; and finally, their result will

be permanently recorded and signed by all team

members present... (Thompson, 1976).

As can be seen from the anectodes in section IV. 10, all

of these "steps" occurred during more or less in the same

time frame and were also somewhat iterative. The team meetings would begin with a review of the evaluation procedure,

for the benefit of new team members. Next the individual

scores for the first set of sub-factors would be posted.

In general, the panel would then discuss the bidders, one

at a time, and work to find an appropriate score. The order

* and spread would both be discussed at the same time. In

some cases the panel would discuss whether they should revise the ranking of a bidder covered early in the discussion, to reflect the panel’s improved level of certainty

with respect to the spread.

Another illustration of the panel holding to the procedure was given in section IV. 10.8. At one of the later

factor team meetings of the agency impact evaluation system

contract, a relatively senior panel member (from outside

of ETIP) suggested that time could be saved by' determining

which bidders were already outside of the competitive range

and excluding them from further discussion. Another panel
member pointed out that the rules of the procedure did not

allow for this. The panel continued rating all of the bidders. This has some significance for evaluating this issue

since it points to how people who are experienced in proposal evaluation may quite casually suggest alterations in

the procedures once the evaluation is under way.

V. 3. 10.4 Weighting of the sub-factors . The last issue

covered under this category concerns whether leaving the

relative weightings of the sub-factors up to the teams led

to any inequities or appearances of inequities. This was

a major issue raised by some ETIP and PAA staff prior to

the evaluation (see section IV. 8). It was feared that leaving the decision on the relative weightings up to the individual teams would present the ability to improperly bias

the evaluation for or against specific bidders.

The process of weighting sub-factors did not seem to

create any inequities. This is discussed in V.3.6.5, under

the issue on time required to agree on sub-factor weights.

Also, the process of weighting generally did not present

the appearance of inequities. An item was included in the

bidder questionnaire which asked whether the respondent

was concerned that not specifying relative weightings of

f the sub-factors might lead to inequities. The response

was: Yes - 1 ; No - 5 (see Figure V.69). The one person

answering yes thought that the entire process would lead
to inequities since there were no predetermined criteria

for each scale score.

Were you concerned that not specifying the relative v/eightings of the sub-factors might lead to inequities in the

process?

Yes - 1 Mo - 5

Remark by person answering Yes

0 The whole process would lead to inequities. You’re

taking nonparametr ic data and trying to make it parametric. There were no boundary conditions for scoring

so that someone could say that an A contains the following five items or, here is an example of an A vs.

an F (BR5).

Remarks by those answering No

o Inequities was not the issue. We were concerned about

accuracy (BR4).

0 The teams probably had too much instruction as it was.

I didn’t see that as a problem (BR2).

FIGURE V.69 BIDDER REPRESENTATIVES’ RESPONSES TO QUESTION

6 ON WEIGHTING OF SUB-FACTORS

V.3.11 Relationship with the Contracting Office

A category of issues which is perhaps most important

to achieving timely award of contracts is the relationship

between the technical program staff and the contracting

office. Since the contracting officer is the one whose

name goes on the contract and is legally responsible, it

is in the interest of contracting office personnel to proceed cautiously and not allow themselves to be rushed by

program people who are anxious to meet their deadlines.
In ETIP’s case, the use of PFE was a departure from more

conventional source selection techniques. As such, it had

the potential of leaving the contracting office open to

protests from losing bidders. People at ETIP were concerned

that the process, with its flexible statement of work, was

not legal or that the contracting officer would not allow

ETIP to use it for fear of protests.

The issues covered under this category are:

1 . Is PFE legal under the Federal Procurement

Regulations?

2. Did the contracting officer feel that the

selections mads b'y using the process would

be defensible if there were a GAO investigat ion?

3. Did the contract negotiator accept the evaluation of the panel?

Did he want the panel to revise their report? » Did he want another level of technical

review of the panel’s evaluation?

4.

a) Did using PFE result in timely award?

b) Were there any protests of the award?

c) Were any debriefings requested?

V. 3. 11.1 PFE and the Federal Procurement Regulations .

A key issue to resolve was whether PFE was legal under the

Federal Procurement Regulations (FPR). This was established

early in a January 27 meeting at the contracting office.

The contracting officer had a background in systems procurement and recognized immediately that there would be no legal

problems
305

The FPR allows for a great deal of flexibility in the

source selection process as long as its minimal requirements

are met. PFE did not go against the grain of the FPR (Contract negotiator, Note 5).

V.3.11.2 Ability to defend under investigation . The

next issue was whether the contracting officer felt that

the selections made by using the PFE process would both

appear honest and be defensible under a GAO investigation.

It was important that the RFP convey sufficient detail about

the evaluation process and the documentation produced that

bidders would realize that it was not only a fair process

but that ETIP would have no difficulty proving this under

a protest investigation.

During a discussion about this source selection with

the contract negotiator he indicated that the major concern

of contracting people is whether there are any protests.

To have a protest means that you have done something so

that people feel that they have an opening to justify a

protest. One measure of how well you’ve done is whether

you get a protest (Contract negotiator, Note 6). The contract negotiator indicated that he was very pleased about

the results of this source selection since there were no

protests. This was especially true since one bidder, who

was thoroughly versed in procurement law and practice, could

not find a basis for protest, even though he seemed very

angry about losing (Contract negotiator. Note 7).

i
In discussion, the contract negotiator indicated that

he was initially skeptical about whether or not the process

would produce protests. It was novel and of greater magnitude than other processes (Contract negotiator, Note 8).

A key reason he was willing to use the method was because

he had faith in the technical people at ETIP whom he had

previously worked with. He knew that they had a sensitivity

to the areas where the contracting office might be vulnerable (Contract negotiator. Note 9). Also, he realized that

the RFP let the company know what they had to respond to

and that the bidder would know that the inform.ation would

reach the evaluators. If there were an oversight it would

be the bidder's and not ETIP's (Contract negotiator. Mote

10 ) . Another thing which helped to establish his confidence

in the process was the response from firms which received

the RFP. He indicated that:

A contracting officer can't anticipate all of

the problems, especially when they are of a technical nature. So, you rely a lot on the private

sector to come back and ask questions and raise

concerns. You get a feeling at that point in

time on how well you've done your hom.ework and

how well the technical people you've relied on

have done their homework (Contract negotiator.

Note 11).

0 Also, he felt that it was clear that it would be difficult

for one person to influence

well expressed in the RFP.

collusion for that to work

the panel and that this was

Everyone would have to be in

Contract negotiator. Note 12).
The contract negotiator's final skepticism about whether the process would work disappeared after the panel report

came in. At that point it was clear to him that it would

stand up under any investigation if it went to the GAO (Contract negotiator, Mote 13). He felt that the strongest

aspect of the process was that, from a technical evaluation

standpoint, "when you get down to the bottom line, there

is no doubt that it is going to stick" (Contract negotiator.

Mote 14).

V. 3 . 11.3 Revisions of panel results . Another issue

which came up prior to the proposals being received was

the MBS administration's suggestion that a committee be

set up to review the findings of the panel. This might

have negated some of the work of the panel.

ETIP had no problems in having the report of the panel's findings accepted. When the COTR told the contract

negotiator of the MBS request for a review panel, the contract negotiator indicated that he did not wish to have

this intermediate step put into the process. When the COTR

submitted his memorandum to the contract negotiator, there

were no requests for revisions.

During subsequent discussions, the contract negotiator

indicated his reasons for not wanting a review panel. He

said that he felt that the evaluation process which was

established was clear and thorough. To have another level
of review, which could negate the panel's recommendations,

would open the process to criticism and probably couldn't

be supported (Contract negotiator, Note 15).

He felt that the report from the COTR was very thorough

and documented with respect to specific points. He added

that bidders normally aren't told what they are going to

be evaluated on beyond the specific factors. This provided

the capability to get down to sub-factors (Contract negotiator , Note 1 6 )

In elaborating on his reaction to the COTR ' s report,

the contract negotiator said:

Sometimes [programs] don't support their recommendations sufficiently. You have to ask them why

they want to negotiate with these two but not

this one who is only five points lower. They

don't have specific reasons documented. 'Vhen

you came back and said that these guy^ are acceptable or unacceptable on this sub-factor, I believed

it. A lot of times when you're told that a guy

is technically acceptable you are not sure they

can support it or whether they have [just worked

up the score] (Contract negotiator. Note 17).

V.3.11.4 Timeliness of award . The final issue in

this category concerns the timeliness of the award and wheth'

er there were any protests which delayed the start of the

work

ETIP was successful in meeting its deadline of the

end of the fiscal year and there were no protests. There

was a request for one debriefing. The bidder making the

request was clearly upset at having lost, since he consi-
dered himself most qualified. He let it be known that he

felt that ETIP made the wrong choice. However, he did not

choose to file a formal protest. The contract negotiator

felt good about the debriefing process since the COTR was

able to tell the bidder what his strong and weak points

were (Contract negotiator, Note 18).

V. 3.12 Resource Requirements

The category of resource requirements deals with issues

of the costs of the source selection to the government and

the bidders. These costs are considered in terms of the

total amounts of time required by various personnel, the

dollar costs of responding to the RFP, and the length of

time over which these resources were spent.

There are four issues covered in this section. They

are

1 . How much time was required to do system or

evaluation design work prior to issuing the

RFP?

o elapsed

o total

2. How much time did it take for bidders to

respond to the RFP?

3 . How much time did it take for the panel to

evaluate proposals?

How much time did it take to read proposals?

How much time did the panel meetings (and

preliminary meetings) take?

How much time would oral presentations

have taken?
4. a) How long did it take to make an award once

the panel's evaluations were complete?

How much time did it take to write up the

results of the panel meetings?

How long did it take to prepare for oral

clarif ications?

How long did it take to request written confirmations or oral clarifications?

How long did it take to reach a decision

once the written clarifications were received?

b) How long did it take to prepare for the debriefing?

V. 3.12.1 Resource required prior to issuing the RFP .

The first issue under this category concerns the resources

required by ETIP to write and issue the RFP. The total

elapsed time between the time when the COTR asked Thompson

to write the RFP until the time when the draft RFP was ready

for distribution was four months. Most of the actual time

required in writing the RFP was Thompson's, though he held

some discussions with other ETIP staff to help him define

the issues. The time to write the RFP probably shortened

somewhat since Thompson used the Wright Field RFP as a base

for the management sub-factors, the Proposal PreparationProcedure, Introductory Notes, and the Scope of Effort Documents. Had the process of getting program participants

involved in formulating the sub-factors been used, the araount of time contributed by others, and probably the overall elapsed time, would have been greater.

V. 3 . 12.2 Resources required for bidders to respond . The issue of the cost to bidders of responding is important

in two respects. First, some qualified bidders may decide
that the expected value of bidding (cost of bidding weighed

against subjective probability of winning and value of winning) is not great enough, causing the government to not

get the bids it desires. Second, much of the cost of responding is eventually paid for by the government through

reimbursable overhead on other contracts. Though this does

not show up as a direct cost on the source selection, it

does increase the costs of government procurements. This

problem is related to the issue of having bidders be able

to self-select effectively on whether or not to bid (section

V.3.3.2) so that large numbers of unqualified firms do not

invest in responding.

Some people felt that the process was quite costly

for bidders to respond to. The author has heard estimates

of cost to respond ranging from $10,000 to $20,000. This

can be quite a burden for a small firm (see Figure V.11

and V.70). However, with clear criteria for the evaluation,

the firms should be better able to assess the risk.

Response From Bidder Question 11 on the Weakest Aspect of

the Process

o For small profit making firms it was just too much

of a good thing. The same thing could have been done

without all of the tasks and sub-tasks. It could have

been made simpler and less costly for small firms.

FIGURE V.70 RESPONSES TO OPEN ENDED QUESTIONS WHICH RELATE

TO RESOURCES REQUIRED FOR BIDDERS TO RESPOND
312

The cost required must be weighed against dollar value

of the contract and the risk of making a selection by using

a less comprehensive (and less costly) procedure. ETIP

felt that the risk justified using the procedures.

V . 3 . 1 2 . 3 Resources required for panel to evaluate

proposals . A major portion of the government resources

devoted to source selection is in the evaluation of the

proposals by the panel. This includes the tim.e to learn

the evaluation process, the time to read the proposals and

assign initial ratings, and the time for the evaluation

panel meetings. One of the major concerns of some of the

ETIP and PAA staff was that the time required for this would

be on the order of months (section IV . 6 )

This was one of the areas where there was a clear advantage by using PFE. The tim.e to learn the evaluation

process was minimal (see section V.3.13). No extra reading

period was required in the schedule. The proposals were

distributed on a Friday and the first factor team was prepared to meet on Monday. The panel members were able to

accomplish all of their reading over the weekend, at night,

or during the meetings of team.s on which they were not members., All fourteen factor team meetings (seven, for each

contract) were completed over a one week period. The meeting times ranged from one hour to two hours and 35 minutes,

with the average meeting lasting one and a half hours (see
Figure V.31). In all, approximately 100 hours of total

time of various people was taken for meetings to evaluate

14

the proposals for both contracts ( Yj where is the

number of people on the i^ team and 1^ is the approximate

length of the i^^ team meeting). Specialists who participated on only one team were able to limit their time in

factor team meetings to one or two hours.

The time of people to read appropriate parts of the

proposals (which is not known) must be added this to actually determine the total amount of time required.

The amount of time required by the government to evaluate the proposals seemed minimal given the significance

and magnitude of the contracts.

If the original plan for having bidders make oral presentations to the panels had been followed, the total amount

of meeting time required by the panel would probably have

at least tripled.

V. 3. 12.4 Resources required following the panel*s

evaluation . One of the issues in source selection is that

it frequently takes a considerable amount of time to reach

an award decision and sign a contract once the panel’s evaluation is complete. This can be. taken up reviewing cost

proposals, obtaining clarifications, preparing and revising

written justifications for decisions reached, and negotiating with bidders considered competitive. In some instances

this can go on for many months.
In ETIP's case the award was made a little over two

months following the completion of the last panel meeting.

The memorandum summarizing the panel's ratings was sent

to the contracting office 11 days after the last meeting

(May 4, 1976). The oral clarification sessions were held

three weeks later (May 25-26, 1976). Questions for written

clarification were sent out on June 4 with responses due

on June 11. The COTR's recommendations were sent to the

contracting office on June 17. The contracting office's

negotiations took two additional weeks and the contracts

were signed.

A debriefing was held with one of the unsuccessful

bidders a month after the signing of the contracts. It

was given by the contract negotiator with the assistance

of the COTR and the author. The preparation by the COTR

and the author consisted of reviewing their notes and the

records from the evaluation on the afternoon before the

meeting

V. 3.13 Education of Government Personnel

The category of the Education of Government Personnel

is concerned with whether the PEE process provided training

and education to government personnel which would help them,

in both the evaluation of proposals and in the management

of the subsequent contracts.

CO
The two issues covered here are:

1. Did use of PFE result in education of government personnel so that they would be able to

effectively evaluate the bidders' proposals?

2. Did use of PFE result in education of government personnel so that they would be better

able to work with or manage the resulting contractors?

Were program participant^ willing to be

involved in the procurements?

V. 3. 13.1 Education provided for evaluating proposals .

The issue of providing education in source selection to

government personnel prior to proposal evaluation is one

of enabling them to have a sufficient knowledge of the project, the RFP, and the evaluation process so that they will

be able to perform effectively when the proposals arrive.

They must understand which sections of the proposals to

read and how to apply the criteria for evaluation to the

sections for which they are responsible.

The panel seemed to be well prepared for the evaluation

given small amount of prior preparation which occurred.

Most of the information concerning the project and the evaluation process was conveyed to panel members through the

copies of the RFP documents which were distributed during

the month before the proposals were due and through the

short briefing the day after the proposals were received.

Some of the panel members also attended a meeting a month

before the proposals were due during which the evaluation

process was reviewed (see section IV. 8).
316

At each factor team meeting where new panel members

were pres ent , a c oupl e of

grou nd ru les for the evalu

were able to hand in their

pr io r to dis cussi on 0 f the

were organiz ed by f ac tors

sect ions of the propo sals

was not a pr oblem •

V. 3.13*2 Education provided for working with winning

bidders . The issue of whether training, learning, and education was provided for the governm.ent personnel who would

work on the contract was thought to be critical to ensure

a progressive and successful accomplishment of the objectives of the procurement (Thompson, 1976a). It was equally

important for the governm.ent people to understand the issues

involved in achieving the objectives as it was for the successful bidder.

The author believes that ETIP was only minimally successful in dealing with this issue. One reason was that

only one PAA staff member participated in the evaluation

and two key ETIP staff did not join the program until immediately after the signing of the contracts. Perhaps of even

greater significance was that the government personnel who

would work on the project were not involved in the process
of selecting and writing the sub-factors. This is a part

of the PFE process which was foregone due to time constraints. It is likely that significantly more education

would have occurred had this process been followed.

V. 3.14 Education of Bidders

The category of education of bidders is concerned with

whether the process of responding to the RFP provided infor

mation to bidders which was useful in their future, interactions with ETIP. In the case of the successful bidders

the issue is whether education was provided which would

enable them to perform more effectively on the contract.

In the case of the unsuccessful bidders, the issue is wheth

er education was provided which would help them in responding or deciding not to respond on future ETIP RFPs. Only

the former issue is covered here.

The issue of whether training, learning, and education

was provided for the successful bidders during the source

selection process was thought to be critical to the accomplishment of the objectives of the procurement (Thompson,

1976a). It was important for the successful bidders to

be aware of what was known about the procurement, to understand what the government felt the important issues were,

and to have some approaches for dealing with these issues

(including how to improve their capabilities in areas where

they realized that they were weak). It was also important
for the successful bidders to have a good understanding

of the Statement of Work and how it would be satisfied

through the successive approvals and acceptances process.

The two successful bidders were asked, in the bidder

questionnaire, whether responding to the RFP gave an early

understanding of how to deal with the issues v/hich were

important to successful performance on the contract. One

answered yes and the other no. The bidder answering no

remarked that:

We were not prepared for the preoccupation of

ETIP management with the evaluation of exper im.ents

where the initial problem of design, initiation

and execution of experiments were not adequately

addressed by ETIP management.

One of the failures of this application of PFE was

the understanding of the statement of work and how it would

be satisfied. The RFP contained both the brief statement

of work which specified only that periodic and final reports

would be delivered (with no schedule) and a longer statement

of work containing lists of illustrative reports and an

accompanying proposed schedule. From the early performance

of the contractors and the ETIP staff, it was clear that

everyone was trying to comply with the illustrative reporting schedule. This created unintended em.phasis on the reports as opposed to developing the systems.

V.3.15 Additional Comments from the Questionnaires

Some of the remarks which were made by respondents

to the open ended questions do not fit properly under any
of the specific issues covered in the preceeding sections.

They are of a more general nature. These remarks are listed

in Figure V . 7 1

.

Some of these comments provide additional perspective

for understanding answers given by the respondents to the

other questions. For example, in BR5’s answer to the question about the strongest aspect of the process he said that,

"I didn’t think there were any." In response to the question which asked whether there was any other aspect of the

process which he wished to mention he said, "I think it

was theoretically unsound and without basis." The author

followed up and asked why he bid if he felt that way. His

answer was: "Because its money and my boss said to do it.

If I had free choice I wouldn’t have bid."

V.4 Summary

This section presents a summary of the results of ETIP’

use of the Parametric Factor Evaluation process for its

source selection. The section is divided into a summary

for those issues for which the evaluation indicates that

ETIP’s use of PFE:

o proved successful,

» o proved unsuccessful, and

o are equivocal.
Responses to Panel Question 5 on the Strongest Aspect of

the Process

o The confidence

choice from the

interchange of

der s ( PMl 1 )

it gave in knowing we had mad

proposals we received due to

information between the panel

e the best

the maximum

and the bid0 The evaluation process itself. The thorough way

i t was laid out . I thought it was excellent (PMl

0 The confidence that we were do ing a good job. and

knew i t ( PMl 3 )

in which

).

others

o The fact that we got a good

able to get an experimental

cracy ( PMl 5 )

contractor and that we were

procurement through the bureauResponses to Panel Question 6 on the Weakest Aspect of the

Process

o The panel was not involved up front in develooine the

RFP (PM11).

o The relative lack of early input (and commitment from

stakeholders, e.g., PAA ’ s and MBS) (PM13).

Responses to Panel Question 10 on Any Other Aspects of the

Process Which You Wish to Get Out

o Mot having a really solid conception of what we were loo

ing for and where we were going and why Phases II and II

were important and other things influenced the process.

It probably didn't affect the outcome in the sense of

which offer we selected. It may have caused us to probe

on certain issues and get certain commitments up front

(PMl 1 )

0 I felt the evaluation procedures were theoretically excellent. It was a little difficult operationally (PMl).

o There might have been a little too much detail, but I didn’t know the issues. That was for you to say (?M6).

Responses to Bidder Question 10 on the Strongest Aspect

of the Process

o I didn’t think there were any (BR5).

FIGURE V.71 MISCELLAMEOUS RESPOMSES TO OPEN ENDED QUESTIONS
o The evaluation procedure which ETIP used was the strongest

aspect. No particular aspect (BR6).

Responses to Bidder Question 11 on the Weakest Aspect of

the Process

o I didn’t see the basis for doing it. I think it failed

to look at the theory they were espousing at the time.

I fail to see where the notions that they espoused in

terms of Campbell and Stanley were used to design this

process (BR5).

o I have 'no problem with your evaluation process. The problem is that the ETIP program had flaws and there was nothing coming up to evaluate. The RFP did not reflect the

shortcomings of ETIP management. The information was

voluminous- and provided us with a good array of information up to that point. I don't think we would have felt

we were treated unfairly if we had lost. Under the work

you were too preoccupied with evaluation (BR6).

0 Absence of a substantive and policy understanding of the

problem they were approaching in relationships with multiple agencies and the use of softer designs. There was

a greater emphasis on quasi-experimental designs where

the issue was, how were you going to do some good case

studies and descriptive materials (BR2).

Responses to Bidder Question 13 on Any Other Aspects of

the Process Which You Wish to Get Out

0 I think it was theoretically unsound and without basis

(BR5).

o There was a kind of euphoria and a kind of pride in the

RFP, by the people who prepared it, that I don’t think

would have been equally shared by old crusty government

contractors (BR4).

o ETIP didn't seem to be as much in control as Charlie Thompson. I think most of the people who were bidding got

the feeling that he was handling his graduate students

and he reflected a lack of knowledge of government (BR2)..

FIGURE V.71 Continued
Chapter VI presents summary findings from the evaluation

which go beyond the scope of evaluating the results of this

source selection against the specific issues.

V.4.1 Summary of Issues for Which ETIP's Use of PFE

Proved Successful

Overall ETIP's use of Parametric Factor Evaluation

for its source selection for the two contracts was successful. It provided for a timely award without protests and

it was able to deal effectively with many of the issues

which were raised by participants or described in the literature on source selection. ' ETIP was successful in taking

a process which had been developed and used in a different

organization, for the procurement of a different type of

system, and adapting it for its own use.

The following is a listing of the specific issues for

which ETIP's use of PFE proved successful (with the sectionnumbers where these issues were discussed);

o The structure enabled bidders to demonstrate

their ability to deal with the critical aspects

of the projects (V.3.2.1).

o The structure did not create critical gaps

in evaluating bidders' capabilities (V.3.2.1).

o Information in the proposals was readily located by the panel when it was needed (V.3.2.1).

o The interrelated documents eliminated contradictions in the RFP (V.3.2.1).

o Goldplating of proposals was not a problem,

the proposals were not too large (V.3.2.2).
323

o Bidders were confident that they had communicated their qualifications to the evaluation

panel ( V . 3 . 2 . 3 )

o ETIP was able to deal with differences in uncertainty among bidders (V.3.2.4).

0 Bidders were confident that they had related

sub-factor issues to the objectives of the

projects ( V . 3 . 2 . 5 )

.

o The basis for competitive evaluation matched

with the subsequent work requirements of the

projects (V.3.2.5 and V.3.4.3).

o The sub-factors encompassed most of the issues

which would be critical for developing the

systems ( V . 3 • 2 . 6 )

.

o The sub-factors did not deal with unimportant

issues ( V . 3 . 2 . 6 )

.

o Parroting back of information from the RFP

to the exclusion of other information required

for evaluation was not a problem (V.3.2.7).

o Enough proposals were received so that the

panel was confident in the scores it assigned

(V.3.3.1).

o Trivial bids were not received (V.3.3.2).

0 The process enabled a strong capability for

evaluating management issues (V.3.4.1).

0 The process enabled a strong basis for evaluating technical information (V.3.4.1).

o ETIP was able to obtain the participation of

experts on the evaluation panel (V.3.5.1).

o Members of factor teams were able to arrive

at initial ratings of bidders prior to the

team metings (V.3.6.1).

0 Factor teams were able to reach agreement on

sub-factor scores for each bidder without voting ( V . 3 . 6 . 2 )

.
32U

o Members of the factor teams were able to see

how their participation affected the ratings

(V.3.6.3)

o Members of the factor teams were informed of

how the ratings and other inform.ation would

be used to select contractors (V.3*6.3).

o The factor teams were able to deal effectively

with members who used different standards for

their initial ratings (V.3-6.4).

o The time to agree on weightings for sub-factors

within the factors was minimal (V.3.6.5).

o The panel was able to evaluate the biases of

its members and use them in making its ratings

(V.3.7. 1 )

o The process kept records of the biases of its

members ( V . 3 • 7 • 1 )

0 Panel members who were biased for or aeainst

specific bidders were not able to sway the

rest of the team (V.3.7.1).

o All proposals were evaluated on the same issues

using the same criteria (V.3.7. 2).

o The ability to use all outside information

increased the accuracy of the scores (V.3.7. 3).

0 The flexible statement of work which ETI? wrote

was acceptable to the contracting office

(V.3.8. 1 )

0 The final scores provided for a range of overall technical discrimination among bidders

(V.3.9.1).

o The sub-factor scores provided for technical

discrimination among bidders on specific issues

(V.3.9.2).

o Technical leveling and technical transfusion

were not problems (V.3.9.3).

o Undue weight was not given to bid price in

the final evaluation (V.3.9.4).
o Bidders were able to understand the process

which was used to evaluate their proposals

(V. 3. 10.1).

o It was possible to trace the results from the

initial individual ratings of the team mem.bers

to the final award decision (V. 3. 10.2).

o The panel followed the rules which were described to the bidders (V.3.10.3).

o The weightings of sub-factors after the scores

were known did not produce inequities (V. 3. 10.4).

0 The process is legal under the Federal Procurement Regulations (V. 3.1 1.1).

o The contract negotiator was confident in being

able to defend the results (V.3.11.2).

0 The contract negotiator accepted the contracting officer’s technical representative's report

on the panel’s recommendations without requesting revisions (V.3.11.3).

o The contracts were awarded in a timely manner

which met ETIP’s fiscal year deadline and brought

no protests (V.3.11.4).

o The resources required by ETIP to prepare the

RFP and evaluate the proposals were not excessive ( V . 3 . 1 2 . 1 )

.

o The resources required by ETIP to follow-up

after the panel meetings were not excessive

(V. 3. 12.4).

o The process provided for sufficient education

of the panel members to enable them to understand their role (V. 3. 13.1).

V.4.2 Summary of Issues for Which ETIP’s Use of PFE

Proved Unsuccessful

There were several issues which ETIP’s use of PFE did

not deal with successfully. These were:
326

o ETIP was not able to make measurements which

would predict the bidder’s future capabilities

on some issues. This was particularly true

for trying to assess future commitment of management to the projects (V. 3 .^. 3 ).

o The RFP was not successful in communicating

that the process would be more difficult to

corrupt than most other processes (V. 3 .T 0 . 1 ).

o ETIP's use of PEE did not provide sufficient

education about the projects to government

personnel who would be working with the contractors (V. 3 . 13.2).

V.4.3 Summary of Issues for Which the Results of ETIP’s

Use of PEE are Equivocal

The results of ETIP's use of PEE for some of the issues

are uncertain, given the information used in this evaluation. These issues are:

0 Whether the structure was too expensiv e to

respond to given the size and requirement s of the project (V.3.2.1 and V .3.12.2);

0 Whether the objectives of the projects and

of ETIP were communicated effectively (V .3

0 Whether any better qualif ied firms did not

bid because of the use of the PEE orocess

{v.3.3. 1 );

o Whether the source selection process did, in

fact, anticipate the problems which actually

occurred during the projects (V. 3 . 4 . 3 );

o Whether experts made a significant contribution

to the evaluation (V. 3 . 5 . 2 );

0 Whether junior panel members made a significant

contribution to the evaluation (V.3.6.6);

o Whether the use of the

illustrative sub-items

tual statement of work

statement of work with

in addition to the acwas confusing (V.3.8.I);
327

o Whether the use of a flexible statement of

work caused any better qualified firms to not

bid (V.3.8.2); and

o Whether responding to the RFP provided education for the bidders which was useful in their

future interactions with ETIP (V. 3. 1 ^. 2 ).
328

CHAPTER VI

BROADER IMPLICATIONS

VI . 1 Introduction

This chapter presents a discussion of the implications

of this research which go beyond the specific case studied.

Four topics which are reviewed are

o the methodology used for this research;

o comparison of ETIP's results to prior and subsequent uses of the PFE process;

o adaptability of the PFE process to other tvpes

of problems; and

o Dossible topics for additional research on

PFE.

VI . 2 The Case Study Method for Exploratory Research

The form of this research was exploratory. It w

the initial evaluation of the use of a process for wh

no prior research had been published. As such, it wa

portant that a thorough description of the process be

veloped and that a relatively comprehensive set of cr

for the evaluation be generated. In addition, the au

wished to represent the criteria in a structure which

enable a point of departure for future research on PF

for future comparisons of PFE to other source select!

as

ich

s imdei t e r i a thor

would

E and

on
techniques. The case study method which was used was felt

to provide a credible means for doing this.

When little prior information is available about a program or process, a complete description of how it really

works (as contrasted with descriptions of how it should

work) is an important antecedent to later evaluatioji. It

provides a context (model) for exploring the nature of the

relationships among important variables. In this research

the description was based upon:

0 background on the PFE process from both historical documents and personal communications

with its primary developer;

o participation in the process at ETIP by the

author and direct access to records of the

source selection of the two contractors; and

0 interview and questionnaire information from

other participants in the source selection.

Background on PFE was studied to insure an accurate context

for describing the process. Interviews and questionnaires

were used to obtain information on undocumented events.

Extensive documentation of sources was provided with the

chronology.

The criteria for the evaluation were generated by examining the critical issues which ETIP faced during its source

selection. A list of issues was drawn by two methods.

One was by examining the records of ETIP’s use of the PFE

process for questions which were raised before, during,

and after the source selection by the people who were in-
volved. The other was by writing down issues which seemed

applicable to ETIP's situation, which were identified from

the literature on source selection, discussions v;ith others,

or reflection on the problem. Comprehensiveness in covering

the major issues, rather than indeoendence among them, was

the objective. The issues were then sorted into groupings

of related issues to provide a structure for discussion.

A subset of these issues were selected for examination

based upon the two criteria of: 1) the relative importance

of the issue to the problem of source selection; and 2)

the availability of data to support an evaluation of the

issue. The issues treated most extensively were those which

were considered most important and for which data were available or obtainable. Where additional data required to evaluate an important issue could be obtained from other participants in the process they were requested through questionnaires.

For each selected issue, a conclusion was reached about

whether PFE did or did not present an effective solution

or whether the results appeared inconclusive given the data

available. In each case the data used to reach the conclusion were described.

The unique structure of issues developed for this research provides a basis for departure and comparison for

further research. For several issues which were identified.
the data available here were not sufficient to enable con

fident conclusions about the effectiveness of PFE in dealing

with the problems. During future uses of PFE, data collection schemes which are instrumented in advance may enable

such conclusions.

Comparisons between PFE and other system source selection methods may be made by using the issue structure.

Effectiveness of various source selection methods may be

compared if evaluations are accomplished by using largely

the same set of issues. Case study designs which facilitate

departures and comparisons add to both the research and

administrative value of the results. This case study is

felt to be an important contribution to research methods

on source selection and good example of a credible approach

to exploratory research in general.

VI . 3 Comparison of ETIP*s Results to Other Uses of the PFE

Process

VI. 3.1 Other Experiences

There have been both prior and later experiences with

using PFE. The prior experiences were the original uses

of PFE by the Air Force to procure hardware subsystems.

The later experiences were the subsequent uses of PFE by
ETIP (in 1977 and 1978) to develop an evaluation capability

for its regulatory program.

VI. 3. 2 Lessons Learned by Comparison to the Wright

Field Experiences

VI. 3. 2.1 Departures . ETIP made several departures

from the original Wright Field applications of the PtE process. These were:

o The ETIP RFP was largely prepared by one person. Others with a stake in the outcom.e of

the procurement were not involved in selecting

the sub-factors.

o Some of the exoerts who participated on STIP’s

factor teams had no expected direct relationship to the future projects.

o The use of a written question form. (Figure

IV. 4) was substituted for holding oral clarifications prior to the factor tea.m meetings.

o A statement of work with illustrative sub-item.s

for the first phase of the contract and a corresponding illustrative schedule were added to

the ETIP RFP.

VI. 3 .2.

Importance of early involvement . A .m.ajor

lesson which is demonstrated by this experience is the importance of getting key people on the project involved early

in the preparation for the source selection. This participation is critical to the development of support and understanding which will be required during the future project.

Time considerations precluded the active involvement

of others in the preparation of the ETIP RFP. This did

not effect ETIP’s ability to select contractors. However,

O'!
it probably did affect the course of the projects after

the successful bidders were under contract. Commitment

of many key people toward the development of the systems

seemed low. The level of understanding or agreement on

the importance of many of the sub-factor issues was also

low

.

Early involvement of key people (those who have a stake

in the process or outcome of the project) in selection of

the sub-factors would have provided an important two-way

education process between those most directly responsible

for manageing the contracts and others who were less directly involved. This early involvement would also have helped

demonstrate to people that their concerns regarding the

contracts would be dealt with in a direct manner, by including them as sub-factors in the source selection. Commitment

would probably have been greatly enhanced if people had

realized that they could be full participants in the decisionmaking process. The author believes that the background

provided and added commitment obtained through early involvement by people who will later be involved in the project

is critical to future development. This step should not

be compromised in future uses of the process.

* VI. 3. 2. 3 Selection of experts . Some of the experts

who participated on the factor teams were people who were

not expected to have any involvement in the development
of the systems. The discussion under section V.3.5 indicates that, though ETIP was quite successful in obtaining

the participation of experts in the evaluation, reactions

were mixed over the usefulness of ^their contribution. The

key question was whether the people had enough knowledge

about the projects to make their inputs worthwhile. Also,

because they would not be involved later, there was no incentive for them to treat the evaluation as anything m.ore than

an academic exercise. The author believes that the evaluation was not harmed by the involvement of these people,

and often they did provide additional useful perspective.

However, he also believes that, where available, experts

should be drawn from the set of people who will have to

work with the resulting contractors.

VI. 3.2. 4 Benefit of orals prior to factor team m.eetings . In section V. 3.2.4 the author indicated that the

question form was only of minimum usefulness. The burden

of the clarification process fell upon the meetings with

the bidders who remained competitive following the factor

team meetings. Though this process was effective for obtaining clarifications, it seem.s as though it also caused

the addition of the appearance of some potential impropriety

or "sub ject iveness" in the process. The author believes

that this problem would have been reduced had oral clarifications been held before the factor team meetings.
335

VI. 3.2.5 Problem from adding an illustrative statement

of work . The addition of the statement of work with illustrative sub-items for the first phase of the contract seemed

to cause confusion after the contract was signed. Both

ETIP and contractor personnel began the projects by treating

the illustrative sub-items as though they were contractually

t required. This was a source of problems since it diverted

effort from the major objective of developing the evaluation

systems.

VI. 3. 2. 6 Transferability of the process to a ”software” problem . A major lesson, which was documented by

this case study, is that the PFE process may be adapted

for use from the procurement of "hardware” systems to the

procurement of "software" systems. Use of the process by

ETIP was initiated on the premise that the underlying problems were sufficiently similar (Thompson, 1976a, p. 170).

The adaptation was clearly successful.

Two points which stand out from the experience of using

PFE for both "hardware" and "software" applications are

that: 1) the technical evaluators are able to retain responsibility for the technical recommendations; and 2) the

resultant selection is not subject to successful attack.

Those in the chain of command are unable to exercise political control over the selection. The thoroughness and trace1
ability of the process prevent this. These same attributes

of the process make it clear that the selection will withstand any protest investigation.

VI. 3*3 Modifications Made in Later Uses of PFE by

ETIP

The factors and sub-factors which were used for the

later ETIP source selections were essentially the same as

those used i'n RFP for the procurement evaluation systems.

They were considered to be both the essential issues and

relatively complete. However, there were some additions

to the illustrative questions under the sub-factors.

Some changes were made in the way the process was implemented. There was earlier and more extensive involvem.ent

by those who would be working on the projects (both within

and outside of ETIP). The schedules of factor team m.eetings

were distributed a week or two in advance of the closing

date for proposals, enabling panel members to adjust their

appointments to the factor team meeting tim.es rather than

vice versa. Finally, references which bidders listed in

their proposals were called in advance of the m.anagement

factor team meetings, so that this information could be

considered in assigning ratings. All of these m.odif ications

from ETIP’s initial use of PFE seemed. to be improvements.

Others contemplating use of the PFE process in the procurement of evaluation systems may wish to consider additional

modifications

O'!
VI . 4 Adaptability of the PFE Process to Other Types of Problems

Parts of the process of PFE may be useful for problems

other than goyernment source selection. Though

tion required for portability to other problems

been proyided here, it is worth mentioning what

possible applications may be. Thompson (1976a)

lated about types of problems for which the PFE

the informahas not

some other

has specutechnique

might proye useful. He has written that:

A number of possible applications might be suggested, and the categories which follow, and the

illustrations, represent only one way in which

they might be organized. First, the case of multiple candidates (bidders or projects) which must

be compared against an uncertain or complex set

of requirements, including:

a) competitiye procurem.ent based on a complex

set of design requirements or of a system,

e.g., the present case;

b) rank ordering of a series of competing

projects (which may not be directly comparable) against the twin standards of likelihood of success and probable payoff (future

yalue), i.e., the "project selection" process ;

c) comparison of alternatiye techniques, solutions, or (assumed) resources and constraints against a set of "independent"

and/or differentially affected requirements,

e.g., cost-benefit analysis.

Second, the case of a single Candidate (bidder

or project) which must be compared against a complex or uncertain set of requirements, including:

a) a project or proposal submitted for "acceptance" under a complex statutory requirement, e.g., application for qualification
as a health maintenance organization (HMO),

submission of an ’’impact statement" for

a power plant or other facility;

b) comparison of a prooosed oroject against

a set of "independent" requirements, e.g.,

cost-benefit analysis.

Third, the case where one (or more) candidates

are to be compared against one (or m.ore) uncertain

or complex requirements by a set of raters who

may differ not only in the kind of infor.mation

they will accept but also in their value structure; in at least some cases where group decision

processes are used (e.g., arbitration, negotiation, the Delphi Method, and perhaps, some political processes) it may be desirable to initially

separate out those areas which are "value-free,"

i.e., facts or agreed statements of facts, to

allow a more narrow concentration on those rem.aining areas in which there are value-based differences (Thompson, 1976a, pp. 179-lSO).

These types of problems occur regularlv in both government and industry and are often resolved in a less than

successful manner. Where current practice is not expected

to produce satisfactory results, experimentation with alternative techniques, such as PFE, seems desirable.

VI .Possible Additional Research

Several areas of further research may produce additional useful information about the effectiveness of the PFE

process or may suggest modifications for improving the process.

This case study focused primarily on evaluating the

effectiveness of PFE in solving problems during source selection. It did not follow the projects past the signing of
33

the contracts to try to determine the match between ratings

on sub-factors and subsequent performance. In the future,

where it can be instrumented in advance, it may be possible

to try to obtain time series measurements on the performance

of successful bidders on each sub-factor. It may also be

possible to keep track of problems which occur during the

projects and check to see whether or not th'ey were anticipated by the source selection process.

Research directed toward improving PFS may focus on

determining the sensitivity of the group decision-making

process to changes in various independent variables. Such

things as influence structures, order of discussion of bidders, use of oral clarifications prior to factor team meetings, and variations in group size all seem to be fruitful

areas for further investigation.

VI . 6 Summary

This chapter reviewed implications of this case study

which go beyond the evaluation of the specific use of PFE

for the selection of contractors for ETIP’s procurement

evaluation systems. It reviewed the methodology which was

used for the study, the increases in knowledge about the

PFE technique gained from comparison to prior and subsequent uses of the process, and the potential portability

of the PFE process to other problems. Finally, suggestions

were made for areas for future related research on the process .
3^0

REFERENCE NOTES

Chapter II

1. Council for Applied Social Research, Newsletter, Vol.

1, No. 1, 1977.

2. Thomoson, C.W.N., Personal communication, Seotember

3, 1976.

Chapter IV

1. ETIP Director, Memorandum to Deputy Director, National

Bureau of Standards, July 15, 197^.

2. COTR Personal communication, July 16, 1979.

3 . Ibid.

4 . Ibid.

5 . COTR Commercial and Technolos;ical Impact of Procurement

Experiments: An Evaluation Project Plan (Draft), November 7, 1975

6. COTR Personal communication, July 16, 1929.

7 . Ibid.

8. COTR op. cit. November 7, 1975.

9. COTR Draft of several pages of com.mercial impact project plan, from ETIP files, dated September 5, 1975.

10. COTR Personal communication, July 16, 1979.

11. NBS Acting Director, memorandum to ETIP Director, March

31, 1976.

12. COTR Personal communication, July 16, 1975.

1 3 . Ib id

14. COTR Letter to C. W. N. Thompson, October 8, 1975.

15. COTR Personal communication, July 16, 1979.

1 6

Ibid.
341

17. COTR Notes from his files of meeting with C. W. N.

Thompson, October 20, 1975.

18. Ibid.

19. Ibid.

20. Thompson, C. W. N. Personal communication, April

1, 1979.

21. COTR Personal communication, July 16, 1979.

22. Ibid.

23. Thompson, C. W. N. Letter to COTR, November

5, 1975

24. COTR Notes from his files of meeting with C.

W. N. Thompson, November 17, 1975.

25. Thompson, C.W.N. Personal communication,

December 12, 1979.

26. COTR Personal communication, July 16, 1979.

27. Thompson, C. W. N. Personal communication, December 12, 1979.

28. Ibid.

29. Thompson, C. W. N. Proposal evaluation factors

(draft), December 1976.

30. Thompson, C. W. N. Note in file of status of

documents for RFP , February 12, 1976.

31. COTR Personal communication, July 16, 1975.

32. Thompson, C. W. N. Personal communication, December 12, 1979.

33. COTR Master Evaluation Plan for .Procurement , note from files, December 9 , 1978.

34. COTR Personal communication, July 16, 1979.

35. Thompson, C. W.. N. Personal communication,

December 12, 1979.

36. COTR Personal communication, July 16, 1979.
37. Ibid.

38 . Thompson, C. W. N. Personal communication, December 12, 1979.

39 . Ibid.

40. Thompson, C. W. N. Thin.gjs to Do , notes in file, January

15, 1976 .

41. Thompson, C. W. N. Personal communication, December

12, 1979

42. ETIP Administrative Officer personal communication,

February 28, 1980'.

43 . Ibid.

44. Ibid.

45 . COTR Personal communication, July I 6 , 1979

46. Thompson Personal communication, December 12, 19'^9.

47 . ETIP Director, Memorandum, to Acting Director, National

Bureau of Standards, January 29, 1976.

* 48. COTR Memorandum to MBS Program Analyst responding to

questions on Project Plan 120.

49 . COTR Memorandum to MBS Program Analyst responding to

questions on Project Plan 121.

50 . COTR Personal communication, July I 6 , 1979.

51 . MBS Acting Director, memorandum to ETIP Director, March

31 , 1976.

52. COTR Personal communications, July I 8 , 1979.

53 . COTR Personal communication, July I 6 , 1979.

54 . Ibid.

55. FSS representatives, Com.ments on RFP written on returned

copy and accompanying handwritten notes, February 24,

1976.

56 . FSS representative. Letter to COTR, April 15, 1976.

57 . COTR Personal communication, July I 8 , 1979.
58. Ibid.

59. Ibid.

60. COTR Notes in file on conversation with C. W.

N. Thompson, February 2, 1976.

61. Attendees at Briefing, Typed list of attendees

at February 26 briefing, March 12, 1976.

62. Transcript of tape of February 26 briefing.

63 . Ibid.

6'4 . Ibid.

65 . Ibid.

66. Attendees at ETIP Pre-Proposal Conference, Typed

list of attendees at March I 8 briefing, March

18 , 1976.

67 . Transcript of tape of March I 8 briefing.

68. Ibid.

69 . Ibid.

70 . COTR Personal communication, July I 8 , 1976.

71 . Compiled from notes listing names of invited

panel members.

72 . COTR Memorandum to members of the evaluation

panel, March 14, 1976.

73 . Thompson, C. W. N. Proposal Evaluation procedure

(draft), March 8, 1976

.

74 . Memorandum from an invited panel member to COTR,

April 5, 1976 .

75 . Memorandum from an invited panel member to COTR,

April 15, 1976.

76 . Ibid.

77. Ibid.

78 . Memorandum from an invited panel member to COTR,

April 5, 1976 .
79. COTR Memorandum to participants in March 17,

meeting, March 29, 1979.

80. Ibid.

8 1 . Ibid.

82. Ibid.

83 . Tape of meeting, April I 6 , 1976.

84 . Ibid

85. Ibid.

86. Ibid.

87 . Ibid.

88. Ibid.

89 . Question for bidder 101, April I 6 , 1976.

90 . Question for bidder 102, April I 6 , 1976.

91 . COTR Memorandum to contract negotiator, Mav

4, 1976 .

92 . Ibid.

93 . COTR Notes in file on meeting with contract

negotiator. May 12, 1976.

94 . Tape of meeting with bidder #203, May 25, 1976.

95. Ibid.

96 . Tape of meeting with bidder #202, May 25, 1976.

97. Ibid.

98 . Tape of meeting with bidder #101, May 26, 1976.

99 . Ibid.

100. COTR Memorandum to contract negotiator, -June

3, 1976.

101. Ibid.

102. COTR Memorandum to contract negotiator, June

17 , 1976. i

fi
345

103. COTR Memorandum to contract negotiator, June

17, 1976a.

104. COTR Memorandum to contract negotiator, June

25, 1976.

105. COTR Memorandum to contract negotiator, June

25, 1976a.

106. COTR & Libman, A. S. Memorandum to contract

negotator, June 28, 1976.

107. COTR & Libman, A. S. Memorandum for participants

on evalution panel, July 9, 1976.

108. COTR Notes on debriefing, August 1976.

109. Ibid.

Chapter V

1

.

Thompson, C. W. N. Personal

cember 17, 1979.

communication.

2. Ibid.

3. Thompson, C. W. N. Personal

cember 12, 1979.

communication.

4. Contract negotiator Personal

February 26,, 1980.

communication.

5. Ibid

.

6

.

Ibid.

7. Ibid.

8. Ibid.

9. Ibid.

10. Ibid.

»

11 . Ibid.

12. Ibid.

13. Ibid.
ohf:

14. Ibid

15. Ibid.

16. Ibid.

17. Ibid. 1 8 . Ibid.
347

BIBLIOGRAPHY

Ackoff, R. L. Towards a system of system concepts. Management Science , 1971, J_7, 661-671.

Aerospace Industries Association. An AIA study of United

States Air Force requests for proposals, critique and recom.-

mendat ions . Washington , DC : December 1969.

Agarwala-Rogers , R. Why is evaluation research not utilized.

In M. Guttentag, & S. Saar (Eds.), Evaluation studies review

annual (Vol. 2). Beverly Hills: Sage Publications, 1977.

Anderson, R. M. Handling risk in defense contracting. Harvard business review , 1969 , ^(4), 90-98.

Anderson, R. M. Anguish in the defense industry. Harvard

business review , 1969 , ^(6), 162-180.

Babione, D. R. (Chairman). A study to test and evaluate

new source selection procedures . Office of the Deputy Under

Secretary of Defense, Research and Engineering. Washington,

DC: April 1 , 1978.

Bailey, B. S. The evaluation process in United States Air

Force source selection . Unpublished thesis, Air University,

Maxwell Air Force Base, Alabama, April 20, 1964.

Berlin, V. N. Statement before the Senate Committee on Human

Resources. Hearings on cost, management and utilization

of human resources program evaluation . Washington, DC:

October 6, 1977

.

Bernstein, I. N. , & Freeman, H. E. Academic and entrepreneurial research: the conseuences of diversity in federal

evaluation studies . New York: Russell Sage Foundation,

1975.

Biderman, A. D., & Sharp, L. M. The competitive evaluation

research industry . Washington, DC: Bureau of Social Science Research, Inc., May 1972.

Biderman, A. D., & Sharp, L. M. Evaluation research: procurement and method. Social science information, 1972a,

11(3/4), 141-170.

Biderman, A. D., & Sharp, L. M. The evaluation research

community: RFP readers, bidders and winners. Evaluation,

1974, 12(1), 36-40.
Biderman, A. D., & Sharp, L. M. An analysis of 36 competitive procurements of social program evaluation studies . Washington, DC: Bureau of Social Science Research, Inc.,

October 1974.

Blue Ribbon Defense Panel. Report to the President and

the Secretary of Defense on the Department of Defense, Appendix E. Staff report on major weapon systems accuisition

process . Washington, DC: July 1970.

Bradford, L. P., Stock, D., 4 Horwitz, M. Hov/ to diagnose

group problems. Adult leadership , 1953> 2(7), 12-19.

Brands, P. Remarks In E. Chelimsky (Ed.), Proceedings of

a symposium on the use of evaluation by federal agencies

( Voi . 1 ) . McLean , VA : MITRE Coro. ( Report M7?-39 y , March

1977.

Bureau of Naval Weapons, Department of the Navy. Concerning

principles underlying the construction of evaluation criteria and scoring systems for use in the source selection process . Washington , DC : November To"i 1965.

Campbell, D. T., & Stanley, J. C. Experimental and QuasiExperimental Designs for Research. Chicago: Rand McNallv,

19b6 . Carnes, C. P. Participation in source selection: a case

study . Fort Belvoir, VA: Defense Svstem.s Management School,

May 1976. (DLSIE No. LD 36 3874)

Chelimsky, E. Proceedings of a symposium, on the use of evaluation by federal agencies (Vol. 2). McLean, VA: MITRE

Corp . (Report M77-39), March 1977.

Cochran, E. B., & Rowe, A. J. The sources of disruption

to project cost and delivery performance. Proceedings of

the sixth annual Department of Defense procurement research

symposium . Fort Lee, VA: June 1977. (DLSIE No. LD J'’059A)

Cohen, D. K., & Caret, M. S. Reforming educational policy

with applied social research. Harvard education review,

45(1), 17-41.

t Commission on Government Procurement. Report of the Co-mniission on Government Procurement . Washington, DC: December

1972.
Comptroller General of the United States. Cost growth in

maior weapon systems (B-I 63058 ). Washington, DC: March

26, 1973 . Comptroller General of the United States. Federal program

evaluation; a directory for the Congress . Washington, DC:

WiV.

Comptroller General of the United States. Finding out how

programs are working: suggestions for Congressional oversight ( PAD- 78 - 3 ) « Washington , DC : November 22, 1977

.

Comptroller General of the United States. HUD*s evaluation

system—an assessment (PAD-78-44). Washington, DC: July

20 , 1978.

Comptroller General of the United States. Evaluation of

programs in the Department of Transportat ion--an assessment

( PAD-79- 1 3 ) . Washington, DC : April 3, 1979.

Cook, T. D. Utilization, knowledge-building, and institutionalization: three criteria by which evaluation research

can be evaluated. In T. D. Cook (Ed.), Evaluation studies

review annual (Vol. 3). Beverly Hills: Sage Publications,

1978.

Cormany, T. P., & Donnellan, J. P. Criteria for evaluating

contractor management potential during the source selection

process for the acquisition of major weapon systems . Unpublished thesis. Air University, Maxwell Air Force Base, Alabama, January 1975. (DLSIE No. LD 33224A)

Dalkey, N. C., & Helmer, 0. An experimental application

of the Delphi method to the use of experts. Management

science , 1963, £, 458-467.

Drake, H. B. Major DOD procurements at war with reality.

Harvard business review , 1970, 4_8 ( 1 ) , 119-140.

Dycus, B. Improving the source selection process by measuring the human response of proposal evaluators. Proceedings

of the sixth annual Department of Defense procurement research symposium . Fort Lee, VA : June 1977 . (DLSIE No.

LD 41059A)

Esposito, A. L. An analysis of frustration: the RFP-proposal cycle. Defense management journal , 1973, 9(1), 8-12.
Evans, J. W. , & Anderson, J. K. Comments on "The RFP as

an instrument for the design of federal evaluations. Evaluation , 1977, 157-160.

Evans, J. W. , Remarks. In E. Chelim.sky ''Ed.), Proceedings

of a symposium on the use of evaluation by federal agencies

( V o 1 . 1 ) . McLean , VA : MITRE Coro. ''Reccrt M77-39 ; , March

1977.

Eyring, H. B. Some sources of uncertainty and their consequences in engineering design projects. IEEE transact ions

on engineering management , 1966, EM- 1

, 169-130.

Fox, J. R. Arming America; how the U. S. buys weapons . Boston; Harvard University Press, 1974.

Garrity, S. D. Monitoring system, development: a framework

and a case study. Doctoral dissertation, Northwestern University, Evanston, IL, for thcom.ing

Goldberg, V. P. Competitive bidding and the production of

precontract information. The Bell journal of econom.ics , 8 ( 1 ), 250 - 261 . Gordon, H. J. The role of the contract in syste.ms acquisition. Defense systems managem.ent review , 1980, 30-

42.

Granquist, W. Statement before the Senate Com.mittee on Huma

Resources. Hearings on cost, management and utilization

of human resources program program evaluation . Washington,

DC; October 6, 1977.

Guttentag, M. Evaluation of social intervention programs.

Annals of the New York Academy of Sciences , 1973, 2J_3_, 3-

13.

Hall, R. B. Reinstating competition in svstem.s acquisition;

a four-dimension framework. National contract management

journal , 1975, 9(1). 87-104.

Havens, H. Statement before the Senate Committee on Human

Resources. Hearings on cost, management and utilization

of human resources program evaluation . Washington, DC:

October 6, 1 977
351

Helman, T., & Taylor, R. L. A conceptual model for evaluating contractor management during source selection. National

contract management .journal , 1976-77, J[^(2), 88-108.

Hess, R. D., Floden, R. E., Sproull, L. S., & Conry. An

RFP for evaluation of Title I programs: will OE learn what

it wants to know. Evaluation , 1977, 4, 152-156.

Hill, L. S., Systems engineering in perspective. IEEE transactions on engineering management , 1970, EM- 1

7

, 124-131.

Horst, P., Nay, J. N., Scanlon, J. W. , & Wholey, J. J. Program management and the federal evaluator. Public administration review , 1974, 34., 300-307.

Hostler, H. K. Burdensome procedures involved in awarding

research and development contracts . Carlisle Barracks,

PA: U.S. Army War College, March 7, 1973. (DLSIE No. LD

29468)

Hynes, C. V. Taking a look at the request for proposal.

Defense management .journal , 1977, 13.(4), 26-31.

Keeling, G. F. The games people play in source selection

competitions. National contract management .iournal , 1975,

9(1), 121-131.

Knezo, G. J. Program evaluation: emerging issues of possible legislative concern relating to the conduct and use

of evaluation in Congress and the executive branch . Washington, DC: Congressional Research Service, Library of

Congress, November 1974.

Lenk, B. R. Government procurement policy: a survey of

strategies and techniques . Washington, DC: Program in

Logistics, George Washington University, (Serial T-354),

May 12, 1977.

Livingston, J. S. Weapon system contracting. Harvard business review , 1959, ^(4), 83-92.

Logan, K. W. An analysis of source selection board procedures for Air Force procurement unpublished thesis. Air

University, Maxwell Air Force Base, Alabama, June *1966.

Lynch, T. D. Public policy forum, introduction. The bureaucrat . 1976, 5(1).

Martin, M. D., Glover, W. L., & Lenz, J. 0. Uncertainty

assessment for the development contract. Proceedings of

the sixth annual Department of Defense Procurement Research

Symposium . Fort Lee, VA : June 1 977 . (DLSIE No. LD 41059A)
Martin, M. D., Heuer, G. R. J., Kingston, J . C., 4 Williams

E. L. An evaluation of the definition classification and

structure of procurement research in the Department of Defense. Proceedings of the seventh annual acquisition research symposium . Hershey, PA; May Bl-Gune 2, 197d.

Miller, G. A. The magical number seven, plus or minus two:

some limits on our capacity for processing information.

Psychological review , 1956, ^(2), 81-97.

Mitchell, T. R. An interview study points out the oroblem

with ’contract’ evaluation. Evaluation , 1973, 1(2), 21-

23.

Morris, S. ’’Communications effectiveness” needed in RFPproposal-contract award cycle. Defense management journal , 1973, 9(1), 17-25.

Nassr, M. A. Past performance: an essential element in

source selection. Defense systems management review , 1978,

1(7/8), 7-14.

O’Connor, E. F. AFSC procurement evaluation panel: new

check in defense systems acquisition. Defense management

review , 1972, 8(1).

Office of Management and Budget, Evaluation and Program

Implementation Division, Evaluation management: a background paper . Washington, DC; May 1975.

Office of Management and Budget, Major systems acquisition

(circular no. A-109). Washington"! DC , 1976.

Packard, D. Improving R&D management through prototyping.

Defense management journal , 1972, 8(2), 3-6.

Patterson, M. B. Government-contractor adversarial relationships. Defense management journal , 1977, 13.(3), 57-62.

Patton, M. Q., Grimes, P. S., Guthrie, K. M. , Brennan, B.

D. F., & Blyth, D. A. In search of impact; an analysis

of the utilization of federal health evaluation research.

In C. H. Weiss (Ed.) Using Social Research in Public Policy

Making . Lexington, MA; DC Heath & Co., 1977.

Peck, M. J., & Scherer, F. M. The weapons acquisition process; an economic analysis . Boston: Graduate School of

Business Administration, Harvard University, 1962.
353

Richardson, J. H. Defense systems acquisition: working

toward improving the process. Proceedings of the sixth

annual Department of Defense Procurement Research Symposium , Fort Lee, VA: June 1977. (DLSIE No. LD 41059A)

Roberts, E. B. How the U.S. buys .research . International

science and technology , 1964, 33,, 70-105.

Rudelius, W. Methods of analyzing the impact of program

stretchouts. IEEE transactions on engineering management,

1969, EM-16, 23-35.

0 Ruth, S. R. The advanced logistics system: an idea whose

time had not quite arrived. Defense management .journal,

1978, 14(2), 35-39.

Scherer, F. M. The weapons acquisition process: economic

incentives . Boston: Graduate School of Business Admini-

’ stration. Harvard University, 1964.

Schnitzer, P. Discretion is the better part of valor in

negotiated procurement. Defense management .journal , 1973,

9(1), 26-28.

Skolnick, A. A structure and scoring method for judging

alternatives. IEEE transactions on engineering management , 1969 , EM-16 , 72^^^3’i

Spagnola, J. C., Jr. Procurement/acquisition research and

the civil agencies. Proceedings of the seventh annual acquisition research symposium. Hershey, PA: May 31-June 2,

1978.

Srull, D. W. Parametric cost estimating aids DOD in system

acquisition decisions. Defense management .journal , 1972,

8 ( 1 ), 2 - 5 . Stewart, J. T. Source selection process faces winds of change.

Defense management .journal , 1973, 9( 1 ), 13-16 . Thompson, C. W. N. System proposal preparation and evaluation procedure (Final report , NBSIR 76-1 1 24 ) . Washington

,

DC: Experimental Technology Incentives Program, National

Bureau of Standards, February 1976.

Thompson, C. W. N. Design and procurement of evaluation

systems. Proceedings of the 1976 international conference

on procurement and grants management . Charlottesville,

VA: University of Virginia, April 1976a.
Thompson, C. W. N. Procuring systems: experience from civilian and defense agencies. Proceedings of the seventh

annual acquisition research symposium . Hershey, PA: May

31-June 2, 1978.

Thompson, C. W. N., The functional value of uncertainty

in the procurement process. Proceedings of the seventh

annual acquisition research symposium . Kershey, PA: May

31-June 2, 1978a!

Tien, J. M. Toward a systematic approach to program evaluation design. IEEE transactions on systems, man and cybernetics , 1979, SMC-9 , 49^-515.

Torrance, E. P. Group decision making and disagreement.

Social forces , 1957, 35, 314-313.

U. S. Air Force. "Management-Proposal," Section M-‘'-l4.0,

and "Technical Proposal," section S-‘'5-l4, AFSCM 310-1/AFLCM

310-1, March 15, 1964.

Waks, N. System source selection , Bedford, MA : MITRE Corp.

( SR -2 1 - 1 ) , December i960.

Weaver, W. Recent contributions to the mathematical theorgy

of communication. In C. E. Shannon & W. Weaver, The m.athem.atical theory of communication . Urbana, IL: The University

of Illinois Press, 1949.

Weidman, D. R. Writing a better RFP: ten hints for obtaining more successful evaluation studies. Public administration review , 1977 37.(6), 714-717.

Weiss, C. H. Evaluation research: methods of assessing

program effectiveness. Englewood Cliffs, NJ : Prentice

Hall, 1972.

Wholey, J. S., Scanlon, J. W. , Duffy, H. G. , Fukumoto, J.

S., & Vogt, L. M. Federal evaluation policy . Washington,

DC: The Urban Institute, 1970.

Williams, W. & Evans, J. W. The politics of evaluation:

the case of Headstart. Annals of the American Academy of

Political and Social Science , 1969, 385 , 118-132.

Yarmolinsky, A. Discussion: evaluation of social experiments. In C. C. Abt (Ed.) The' evaluation of social programs . Beverly Hills: Sage Publications, 1976.
355

APPENDIX A

Documents from the RFP
NBSIR 76-1124

EVALUATION SYSTEM PROPOSAL

PREPARATION AND

EVALUATION PROCEDURE

Charles VI. N. Thompson

Office of Experimental Technology Incentives Program

National Bureau of Standards

Washington, D. C. 20234

February 1976

Final Report

Prepared by

EXPERIMENTAL TECHNOLOGY INCENTIVES PROGRAM

O' '

U.S. DEPARTMENT OF COMMERCE, Elliot L Richardson, Sscratsr/

James A. BaUer. 111. Undar Secretary

Dr. Betsy Anckor-Johnson, Assistant Secretary for Science and Technology

NATIONAL BUREAU OF STANDARDS. Ernest Ambler. Acting Director
357

CONTENTS

Introduction i

Introductory Notes. 1

Some Definitions 4

Proposal Preparation Procedure 7

Revisions to Proposal Preparation Procedure 17

Evaluation System Design Process 19

Statement of Work 21

Statement of Work (With Illustrative sub-items for.

Phase One) 23

Proposed Schedule - Overall 33

Proposed Schedule - Phase One 34

Proposal Evaluation Factors 35

Proposal Evaluation Factors (With illustrative questions) 42

Numerical Weights for Proposal Evaluation Factors 75

Scope of Effort 77
I. INTRODUCTION

The objective of the research v/hich culminated in this set of

documents i-^as to develop a process for the procurement of two

evaluation systems. IVithin the context of applicable procurement

regulations and program plans , the set provides a complete and

integrated framework for both the preparation of proposals by

prospective bidders and the evaluation of proposals by the evaluation team.

The document entitled Proposal Preparation Procedure provides

both an overall description of the process and specific guidance

to prospective bidders. All of the salient or significant

characteristics of the system to be procured, the program under

which it will be developed, and supporting elements are included

in a set of elements called sub-factors which are grouped, for

convenience, into a set of factors. Each of the key documents --

the Statement of V/ork, Schedule, Scope of Effort, and Evaluation

Factors - -follows a parallel organization of the set of sub-factors

It is this integration of all of the central documents which

provides both proposer and evaluator with a single, common structure .

1
359

The initiation of the development of this process arose out

of the recognition that the ''evaluation systems" to be procured

required the services of a systems contractor and that it appeared that neither ETIP nor the "evaluation industry" had any

significant present capability to design or m.anage such a system.

A review of available proposal preparation and evaluation procedures suggested that some further development was required.

The process described herein was used to procure the tv;o

systems, and, as of the initial getting started period under the

two contracts, appears to have worked out reasonably well in terms

of our confidence that we have chosen not only the best but also

reasonably competent contractors.

While this set of documents was specifically designed for the

procurement of these two evaluation systems, the process, with

modifications in the set of sub-factors, is applicable to comparable systems problems involving complex or uncertain requirements.

One of the purposes of this report is to make these documents

available on the basis that others may be interested in further

applications.

11
^ ^ Q

RFP 6-35756

Introductory Notes

The purpose of these two proposed procureraents is to obtain

contractor assistance in planning, designing, and carrying

out the evaluation of a series of procurenent experin'.ents

which are being undertaken by ETIP in cooperation v/ith several

governmental procurement agencies.

The set of objectives of the evaluation program includes the

following

a. an overall description of the experiments as well as

selected detailed descriptions

b. an overall assessm.ent of both the immediate and the

subsequent effects (impacts) of the experiments, as

well as selected detailed assessments

c. An evaluation system or process v/hich can be used by

the appropriate governm.ent agency (ies) to obtain

evaluations on a continuing basis of these as v;ell

as future, similar experiments.

There are a number of choices v/hich could be made in obtaining

contractor assistance, ranging from a single overall contractor

to individual contractors for each experiment, or co.mbinations

in between. The choice v;hich has been made is to divide the

overall evaluation program into tv/o procuremLents , as follows:

a. an evaluation of "agency impact" v/hich will direct

its attention to a description of the experim.ents

an assessment of the effects on the several governm.ent

agencies, and the design of the related evaluaoicn

system. "Agency impact" includes both immediate and

subsequent effects on the program administrative agency

but also other affected or related governm.ent agencies

b. an evaluation of "commercial impact" v/hich v/ill direct

its attention to the immediate and subsequent effecrs

upon the commercial (or industrial) sector, and, as

appropriate, consumers or users of the products, and

the design of the related evaluation system.

1
RFP 6-35756 361

The set of "experiments” includes the following:

a. specific procurement experiments - including

"completed," on-going, and planned

b. specific related or supportive experiments - including procedures for selecting experim.ents , training

of agency personnel, processes for obtaining from

or providing to the industrial sector information

required for carrying out procurement experiments,

and other administrative changes

c. general "experimental" changes - related to or in

support of the above.

The set of experiments may be conveniently grouped into broad

categories according to these dimensions:

a. by program administrative agency - CD Federal

Supply Services (FS3) ; (2) state and local procurement agencies (S&L) ; and (3) Veterans Administration

(VA)

b. by type of procurement experim.ent (1) life cycle

costing (LCC) ; (2) value incentive contracting (VIC)

(3) performance specifications; (4) multi-year awards;

etc

.

c. by the kind of effect intended - (1) economic or performance advantage to the using government agency;

(2) related or derivative improvement in procurement

capabilities of the program administrative agency;

(3) increased or improved technological innovation

in the industrial sector; and (4) economic or performance advantage to civilian sector users.

To obtain the evaluation desired we propose to let two contracts,

each for a three-year period, divided into three phases, as

follows:

Phase One - to include, generally, three activities

(a) preliminary systems analysis

(b) evaluation of specific, selected procurement or

other experiments

(c) design and pilot test of the related evaluation

system

2
RFP 6-35756

362

Phase Tv/o - to include, generally:

(a) refinement of (a) above

(b) evaluation of additional specific procurement

or other experiments

(c) refinement and prototype test of the related

evaluation system

Phase Three - to include, generally, "turn key" implementation of the related evaluation system by

the appropriate governm.ent agency

The complete procurement package has been designed to assure the

fullest possible interchange of information betv;een prospecrive

bidders and those responsible for the evaluation of proposals.

Because of this it v/ill be necessary to examine the complere

set; particular attention is directed to the fact than the

Statement of Work and the Schedule must be read in conjunction

with the accompanying documents.
363

RFP 6-35756

Some Definitions

Note: In most cases, the context, or reference to other parts

of the bid set and associated documents, should resolve

ambiguities or otherwise clarify definitions of terms. These

"definitions" are provided as a convenience in reference.

ETIP - Refers to the Experimental Technology Incentives Program. In context, the reference may be to the objectives or

content of the program, to the office or organization itself

(e.g., as contracting agency), or to individual staff members.

PAA - Refers to the Program Administrative Agency (or Agencies)

.

There are, for these procurements (agency impact and comjnercial

impact), three PAA, as follows: 1) Federal Supply Service

(FSS) ; 2) state and local procurement agencies (SSL )

3) Veterans' Administration (VA)

.

.In context, the reference

v/ill usually be to the specific procurem^ent office (s) and/or

responsible personnel in the agency which have the direct

authority and/or. responsibility for the decision to introduce the specific experimental intervention and, usually,

for the conduct of the program and/or function to which the

experimental condition is applied. For FSS, this may include

both central and regional procurement functions or offices.

For SSL, this may include NASPO, NIGP, and/or specific state

and local procurement agencies. For VA this may include both

central and regional or local procurement agencies.

AGENCY IMPACT - Refers to effects or impacts on or in the PAA

and, where applicable, closely related governmental agencies,

and, particularly, agencies which are the users of the products

procured. These impacts or effects include not only those

which are the specific objectives of a specific or general

experiment, but also significant other effects, such as

changes in administrative policies and procedures

.

COMMERCIAL IMPACT - Refers to effects upon the commercial

sector and/or civilian sector users. These impacts or effects

include not only those which are the specific objectives of a

specific or general experiment, but also significant other

effects, such as increased or earlier use of new technology.

EXPERI14ENT - Refers, generically, to any combination of

intervention and desired (or hypothesized) effect. .In a

complex of interventions and effects, there may be a number

of "experiments," including experiments which are combinations of other experiments, and experiments where the effect

of one is the intervention of another.

4
RFP G-35 7^^.^

SPECIFIC EXPERIMENT - Refers, usually, to a specific, identified

intervention and desired (or hypothesized) effect. The major,

and most significant, cases are the specific procurement experiments (e.g., use of LCC to procure air conditioners). T.here

may be other (related) specific experim.ents (e.g., the FSS training program)

VARIABLE - Refers, generically, to any object, event, state,

value, function, etc. In the context of a specific experiir.e.nt

(or study), the independent variable is, usually, the i.ntervention, and the dependent variable the desired (or hypothesized)

effect

PAPJi^lETER - Refers to those variables v;hich, whether controllable

(directly or indirectly) or not, affect (or are hypothesized

to affect) plausibly and/or significantly uhe inrerprerability

and/or credibility of the observed (or hypothesized) relationship betv;een the variables in a specific experimeno

A PRIORI PROPOSITION TESTING - Refers to those specific experiments for v/hich one of, if not the prim.ary, purposes is to

establish, with a relatively high degree of credibility, the

relationship betv/een the interve.ntion and the effeco observed.

In practice, this will be largely limited to relatively v;ell

defined interventions and proximal effects, and vrhere

reasonable control of paramieters can be achieved.

EXPLORATORY (AND/OR DESCRIPTIVE) - Refers to experim.ents (and/

or studies) where it is either not feasible or not: required

to meet the requirem.ents of a priori proposition testing. In

practice, this will apply to requirements to describe and/or

measure sets of variables v.dnere either the description is

sufficient itself or provides a basis for identifying hypotheses,

variables, and/or parameters.

EVALUATION SYSTEM - Refers to the set of policies and procedures

which provides the basis for evaluating the (agency and/or

commercial) impacts of a set of specific procurem.ent or related

experiments. The form, detail and completeness of the system,

will progressively change. In Phase One, it may begin as a preliminary outline within which detailed "single thread"

designs are developed for the early evaluation of specific

procurement experiments; later, the results of the preliminary

systems analysis and the "pilot test" will provide the basis

for a preliminary evaluation sysrems design. In Phase Two,

progressive refinement should result in a relatively com.plete

evaluation system v/hich can be tested as a prototype, in part,

through the evaluation of specific expt=rim.ents . In Phase

Three, the evaluation system, should be in the form, of a stable

and complete system.

5
365

RFP 6-35756

BASIC (OR BASE-LINE) EVALUATION PROCESS - Refers to that part

of the evaluation system which includes those evaluation

activities which deal with key, predictable, identifiable,

continuing, common objectives for v/hich the process can be

institutionalized

.

SPECIAL EVALUATION PROCESS - Refers to those evaluation activities which do not meet the above requirements because of

specialized, one-time or changing objectives.

FRAJ4EW0RK OR MODEL - Refers to graphical or other conceptual

representation of the set of interrelations among objectives,

organizations, experiments, and/or programs, and the evaluation system.
366

RFP G-35756

PROPOSAL PREPAPJ\.TIOM PROCEDURE

A. Purpose

The purpose of this docuraent is to provide a guide for

bidders in the preparation of their proposals.

B. Basis for Preparation of This Document

B.l Historical Note

A v/ide variety of methods have been used for the procurement

of the services of evaluation contractors, including uhe

following: 1) adding evaluation as an express or im.plied

requirement to the basic contract for the program or exueriment; 2) contracting separately with a contractor or consuluan

for the evaluation; 3) obtaining an evaluation, usually post

hoc / as part of the study phase of a ccnrract for a new or

different program. The description of the services desired

may take many forms, including the following: 1) a brief,

general requiremient "to perform an evaluation"; 2) stareraents or descriptions of various lengths and v;ith various

degrees of detail and comiuleteness v;hich outline the objectives of the program and/or the specific questions to be

answered; 3) relatively detailed and complete " specif icaticns

of the services required, including instrum^ents to be used,

sampling plans, and the form of the analysis required.

In some cases, and for some purposes, these m.ethods provide

a satisfactory base for defining the services required,

which, in turn, may provide a satisfactory basis for deoermining the qualifications of prospective contractors

None of the above appear appropriate as a basis for determining the qualifications of prospective ccntracoors on this

procurement. For this reason this docu.ment was prepared,

using a "parametric factor analytic approach" which v;as

developed for and used on a number of subsystem, procurem.ents

by another agency of the governm.ent.

7
RFP 6-3575

B.2 Basis for Evaluation (of Bidders)

Generally/ evaluation (including proposals) requires three

things: first, a standard or base for comparing; second,

information to be compared against the standard; and, third,

an effective method for doing the comparing. These are

discussed in the three paragraphs below.

B.2.1 The first requirement of the evaluation of competitive

proposals is a standard or base for comparing, and this is,

obviously "v;ho can do the job best." but this is a matter of

predicting because it is a future matter, and there is

uncertainty in specifying the job. The choice of standard

for selection includes:

a. The v/hole program, either as it is or as it should

be stated to obtain the selection base.

b. The "significant independent variable" - if one

part will determine, either by its absence or

. presence, the one bidder who can do it, then

this is the basis.

c. Some sample of parameters (variables or "factors")

which on a presence or absence basis, or qualitative

scale, is the determinant.

The choice is between b) and c) because of the inability to

precisely define the skills and content of the future

completely; and because agreement on b) presumes considerable

confidence in past experience, the basis must be c) . The

development of c) is summarized in the paragraph below in

terms of a brief statement of the program.

The nature of the item to be procured determines the program.

The evaluation here is not limited to fully designed evaluations of specific experiments, nor is the alternative of a

broad, overall descriptive case study sufficient. It is

proposed to obtain not only preliminary systems study which

will include both of the above, but also both kinds of

evaluations; in subsequent phases, it is proposed to

develop and test an ongoing capability to carry out such

evaluations in such form and detail as will allov; the

government to implement subsequent phases. Direct

description of such a program must necessarily be accomplished

by selecting and tabulating the significant parts of the

program; and these parts may then be further defined and

amplified by supporting data. These parts, identified as

"factors," are the bases for the establishment of the

standard; and, in turn, the information to be furnished

and the evaluation procedures.

8
368

KFP G-35756

i

B.2.2 The information to be compared against the standard

is that furnished by, or about, the bidders. This requires

cooperation, varying from a little to a lot. Too little

occurs v/hen the bidder's data is absent or cannot be translated into a comm.on dimension or is irreleva.nt. Too much

occurs v/hen the bidder essentially repeats back the guidelines furnished. The problem of "too m.uch" has not been

experie.nced in prior evaluations, and those fev/ cases of

"parroting" were obvious. The concern is to assure that

the bidder has thorough guidance in preparing the necessary

information, and by this means the bidder v/ill be aided

in directing his efforts more efficiently.

B.2.3 Because of the large numl^er of factors to be evaluated, the large volume of data to be considered, a.nd the

size of the evaluation team, the process needs to be v;ell

organized and supported with efficient a.nd clear procedures.

Forms and instructions must be prepared, a.nd arra.ngements

for scheduling m^eeti.ngs and for carrying out the evaluation have to be thought out in advance. A corollary benefit

is an increase in the assurance of impartiality throughavoidance of the confusion of setting rules and judging

at the same time.

B. 3 Preparation

This document, and the associated statement of v/ork, v/as

prepared upon the above basis. As source m.aterial,

procedures similar to this, as well as a number of

other evaluation procedures, were reviewed.

!

I

1

I

I

I

I

9
369

RFP 6-35756

C. General Coinments on Proposal Evaluation Procedure

To provide bidders with a general background and perspective,

a brief outline of the overall procedure is furnished belov;.

(It should be noted that these comments are intended only as

a general description, and some changes may be made.)

C.l Preparation for Proposal Evaluation

C.1.1 To assure continuity, personnel responsible for setting

up and administering the program for which proposals are being

solicited, will conduct the proposal evaluation, v/ith the

advice and assistance of others who are less directly concerned. Direct management responsibility is centered in ETIP

in the Contracting Officer’s Technical Representative (COTR)

.

He, together with other members of the ETIP staff, v;ill be

working closely with key members of the staff of the program

administrative agency in the type of close relationship

which is characteristic of "administrative experimentation."

As a minimum, it is intended that at least two members of

the proposal evaluation team will be assigned prim.ary responsibility for each subfactor, with backup assistance in

review by other team members. These evaluation team mem±)ers

are responsible for preparing and/or reviewing the bases

for the evaluation and for briefing the bidders.

C.1.2 The results of the preparation are incorporated,

primarily, in the RFP. In addition to the Form 33 and

accompanying provisions, the PRP includes a series of

related documents prepared for the primary purpose of

furnishing a base for the preparation of proposals. The

necessary clarification and revision to provide a contractual basis will be accomplished by negotiation with the

selected contractor. Briefly, these documents are

grouped as follows:

Statement of Work - tabulates the significant parts

of the program with clarification limited primarily

to a brief summary. This v;ill include those separately identifiable "deliverables" and a schedule.

Proposal Preparation Procedure (this document) - provides

an explanation of the basis for proposal preparation.

Proposal Evaluation Factors - provides the basis for

proposal preparation.

Scope of Effort - cost proposal requirements

Other Documents - development plans, reports, references

etc., to assist the bidders in preparing their proposals.

10
370

RFP 6-35706

C. 1.2.1 To the extent possible, all of these documents follov;

the same order. The Statement of V.’ork includes ite.ms of

required v;ork divided into three phases. The Proposal

Preparation Procedure is designed to match the organization

of the Statement of VJork. It is intended to provide, v;here

necessary, cross-reference to the other documents.

C.1.2.2 To assure that all bidders have complete sets of the

related documents, and especially those included under

"Others" which may be furnished at the briefing, a complete

index of all documents v;ill also be furnished during the

briefings, as required.

C.1.3 It is planned to brief the bidders twice. At the

initial briefing, a summary presentation on the progra.m

will be given, including a presentation by the program

administrative agency on the administrative objectives

for the program. (To assure the continuity specified above,

representatives of the program adm.inistrative agency v/ill

participate in the evaluation.) Following the presentation,

an extensive question period v;ill be scheduled. The second

briefing will follov; a fev; v;eeks later after the bidders

have had an opportunity to study the program; this is prim.arily

to allow for further questioning by the bidders to correct

oversights and resolve ambiguities. No inf or.matio.n v;ill

be available during the proposal preparation period on an

individual basis.

C.2 Proposed Evaluation Procedure

C.2.1 After receipt of the v;ritten proposals, the proposal

evaluation team will have a period of a few days to

familiarize themselves generally with the proposals; and,

specifically, the parts within their responsibility. This

is primarily to furnish a basis for the next stage.

C.2.According to a schedule to be established at the second

of the two briefings noted above, each bidder will be invited

to make an oral presentation, approxim.ately one hour in

length, following which the evaluation team will have two to

four hours for questioning. Purpose of the questioning is

to assure that the evaluation team, understands the proposals

and has an adequate basis for evaluation on each sub-factor.

Both weak and strong points v;ill be explored, and check lists

will be recorded to assure that team memJDers may have available

to them both the written and oral presentations

II
371

RFP 6-35756

C.2.3 Following the above will come a further period of

review, during which individual team members will complete

their individual evaluations of sub-factors for which they

are responsible or concerned. These individual evaluations

v;ill be noted with the assistance of a check list based on

a value rating scale to avoid the confusion caused by

numerical rating.

C.2.4 Members of the proposal evaluation team concerned with

each factor will meet and perform a joint evaluation. They

will follow an established procedure entitled, "Factor Team

Guidelines," to assure an orderly and efficient process.

Briefly, they will first compare notes informally to assure

common standards and information; second, they v/ill arrange

the bidders in descending order; third, they vzill establish

the relative spread among the bidders; and, finally, their

result v;ill be permanently recorded and signed by all team

members present. This record becomes part of the permanent

evaluation files. Generally, they will rate on each subfactor, as applicable, three central points:

a. Comprehension or understanding of the problem

presented

b. Proposed approach (or, in the alternative, the

means for establishing the approach)

c. Present or potential capability to accomplish this

Evaluation team members may use information obtained from the

RFP and associated documents, the bidders' v/ritten and oral

proposals and independent or prior information. However, in

the latter case, any significant or critical information must

be considered by all members of the factor team. Emphasis

will be placed on the sub-factor being rated, but the impact

of other factors will not be ignored. Comparative weighting

of individual sub-factors and the weight of each factor team

members ' s views will be within the responsibility of the

factor team.

C.2.5 The results of the factor team evaluations will be

summarized and reviewed.

C.2.5.1 The summary process will include the necessary combining

of the individual factor ratings to achieve an overall rating,

with consideration of the interaction of factors, their individual significance (acceptability) , and the "summation"

according to previously established weights.

12
'72

RFP 6-357:^6

C.2.5.2 In general, the range .of v;eights among the factors

will be no more than 2:1.

C.2.5.3 The reviev; process will necessarily provide for

latitude in reflecting the results of the reviev;, such as

reassessi.ng of v/eights or reexamination of specific areas

of the evaluation; and, such changes will be incorporated,

v;ith the supporting basis in the record of the evaluation.

C.3 Comment on Factor and Sub-factor Structure

All of the factors in the evaluation are set out in the

Proposal Evaluation Factors documents, and are largely

self-explanatory. The rationale of this particular

organization of the sam.ple is based, primarily, on the

sub-factors, the factors being ad.ministrative groupings

for convenience in organization of the various documents

and the evaluation. Generally, t.he factor/sub-f actor organiza

tion is based on significant identifiable effort/capability

areas. The intent is to select comprehensive exclusive/

inclusive samples of the total program. In total, the

factor/sub-factors are inte.nded to include samples of all

significant parts of the program. An effort has been miade

to minimize duplication; and, v;ith a few intentional

exceptions, no sub-factor sam.ples an area covered by another

factor. Similarly, with certain intentional exceptions,

each sub-factor samples on the basis of understanding,

approach, and capability; as a result, those aspects, v;hich

include "effective experience," are incorporated into specific

substantive areas rather than amorphous generalized categories

It is important to understand that the distinction am.cng the

factors is primarily an administrative distinction to reflect

the differences in perspective required in presentation and

evaluation

13
373

RFP 6-35756

D. Use of this Document

D.l General Considerations

D.1.1 As noted above, the Proposal Evaluation Factors document covers all of the factors to be considered in the

evaluation and are intended to provide a substantially

complete framework for the preparation of the proposal.

Because the background and perspective against v/hich they

were prepared is set out in this part, the other related

documents, and the briefings for the bidders, it is

extremely important that this interpolation be understood

and preserved during the preparation period by assuring that

substantive contributors have sufficient access to the

overall framework. As a general guide, cross reference

should be made to the applicable points included in the

document on Scope of Effort.

0 D.2 Content Considerations

D.2.1 Because the evaluation is organized on a factor/subfactor basis, it is important that the treatment of each

factor be substantially complete as possible. The several

factors are necessarily interrelated and interacting, but

repetition is not required; hov/ever, where significant

material considered necessary to assure understanding of

a factor is contained in another factor, specific (and

annotated) cross-references should be furnished. All members

of the evaluation team will have access. to the complete

proposal ,' and it is expected that certain factors will be

reviewed jointly.

D.-2.2 Each factor and sub-factor is defined by a specific

descriptive sentence setting out the standard against

which each bidder will be measured. Because the factor

statements reflect the fact that the factor is primarily

an administrative convenience, no specific presentation

with regard to the overall factor is required. Each subfactor, however, will be the subject of a specific rating,

and the proposal content should reflect the necessity for

furnishing the evaluation team with sufficient information

to establish a comparative position on each sub-factor

THE BASIC ORGANIZATION OF THE PROPOSAL MUST BE ON A SUBFACTOR BASIS.

14
374

RFP 6-35756

D.2.2.1 It should be specifically noted that the sub-paragraphs

under each sub-factor are provided for the purpose of amplifying

and defining the standard, that is, to present the types of

questions v/hich need to be answered to establish the rating on

the sub-factor. It is not required 'to ansv/er each and all of

these subsidiary questions. Nor is it required that the

presentation be limited to these particular subsidiary questions.

In some cases, a subsidiary question v;ill be a critical determinant; in other cases, an effective presentation may be

accomplished by partial treatm.ent. It was not intended tc

specify the proposal content through these subsidiary questions.

D.2.2.2 These subsidiary questions are presented in three for.ms

One form is that of a question, such as, "Do you consider the

proposed schedule realistic? too tight? If nor, what do ycu

propose and v;hy?" The second general form is that of a statement calling for comment or the furnishing of certain information, such as, "Testing facilities (floor space, equipment,

personnel)." The third general form is that of a statem.ent

commenting on a problem area or outlining a proposed solution

which furnishes a basis for conment, such as, "The Advisory

Committee may raise a technical question which requires

investigation in order to allov; the committee to m.ahe a decision

at the next meeting. The contractor v;ill furnish the necessary

personnel to develop each of the conflicting points of view

independently. "

D.2.2.3 As noted elsewhere, the relative v;eighting, or significance, of each sub-factor v/ithin a factor is v/ithin discretion

of the factor team performing that part of the evaluation. In

most cases, the sub-factors represent approxim.ately equivalent

areas of significance. The amount of subsidiary questions

furnished does not necessarily reflect _ an unusually high or

low v;eight.

D.2.3 Latitude is encouraged in the treatm.ent of the content

presented v/ith respect to the sub-factors. VThere alternate

assumptions or conclusions to clarify ambiguities or fully

establish the position is considered advisable, no arbitrary

limitation is im^posed. The substantive objective is to provide

information as a basis for evaluation, and this procedure

is intended only as a minim.um framework to increase the

efficiency and effectiveness of the proposing and evaluating

process

D.3 Physical Considerations

D.3.1 Primary concern v;ith regard to the physical format and

organization should be to facilitate the work of the evaluation team, recognizing the circumstances and procedures v;hich

will be used.

15
RFP 6-35756 375

D.3.2 It is suggested that the proposal use the same major

numerical and subject headings of the Proposal Evaluation

Factors document to avoid the confusion of an additional

numbering system. This should be extended only to the

factors and sub-factors, and no requirement to follow the

numerical order of the subsidiary questions within a subfactor is intended. Where it is considered desirable to

include additional material v/ithin the framework of Sections

I through VII, an expanded numbering sequence should be

used, i.e., to include material after I.l, but before 1.1.1,

use a new number, 1.1.0, with subsidiary numbers, I. 1.0.1,

I . 1. 0 . 2 , etc.

D.3.3 To facilitate the physical handling of the proposal,

certain physical divisions are requested.

D. 3.3.1 Physically bulky material, such as extensive

personnel or facility brochures, or supporting descriptive

documents furnished in amplification, should be treated as

physically separate appendices, v/ith suitable identification.

In this regard, where material which has been prepared in

another connection is furnished, a suitable cover sheet

noting limitations of application may be used to avoid

extensive revision and republication.

D.3.4 Provision should be made for sufficient indexing and

cross-reference summ.aries to facilitate reference to

particular parts of the proposal. Unless considered

desirable, pagination need not be accomplished. It is

recommended that covers of separate documents contain

clear reference to the numerical and subject heading.

*

16
376

P.rP 6-35756

REVISIONS TO PROPOSAL PREPARATION PROCEDURES

A. Page 5. Delete C.2.2, and substitute the following:

C.2.2 V/here factor team members require specific

additional information for their evaluation, specific

written questions will be directed to the specific

proposer through the contracting officer.

B. Page 6, C.2.3, lines 6 and 7. Place period (.) after

"scales,” and delete the remaining material.

C. Page 6, C.2.4, lines 24 to 27. Delete "Comparative

weighting" and add "Comparisons," and delete "weight"

and add "efforts."

> ft

17
377

lU

CO

CS

a,X

Li. ^ o C-3

kU

Llsi

®o

LLJ- ^2

«S^ Jil o

t—

r®

Q S caw •* b <

til

e=»aad ^ m-~<^ -~m ^ 4* S5J o

19
statement of V/ork

RFP 6-3S756 378

Note: The items listed belov; provide a catagorized summary of the

services and materials required on this program.

Phase One: Planning and Design, and preliminary evaluation.

Item 1. Plan, organize, staff, direct, and control, including

. both schedule and costs.

Item 2. Provide reports, liaison, and services for purposes

of review, coordination, and approvals and acceptances, including the following:

a. Periodic and special formal and informal

management progress reports, final report.

b. Periodic and final scheduling reports,

including projected schedule for the next phase.

c. Periodic and final cost reports, including

projected costs for the next phase.

d. Periodic and special formal and inform.al

technical progress reports and final report,

including proposed program for the next phase.

(Note: the substantive v;ork to be reported

here is to be performed under items 3 through

7) . Item 3. Review and structure the proposed objectives in

terms of the experimental setting and related

background, and establish an overall plan and

set of priorities.

Item 4. Review and structure the proposed set of experiments,

and related background in terms of the proposed

objectives, and provide a reference base.

Item 5. Develop the detailed evaluation requirem^ents (problem

definition and experimental or study design) , in

relation to collection and analysis.

Item 6. Develop the detailed data collection process,

including instruments, identification of, and

access to, sources, and m.anaging the process;

and collect the data.

21
RFP 6-35756 379

Item 7: Develop the detailed data analysis process;

carry out the analyses; present findings;

present recommendations.

i

i

I

«

fl

i

w

i

n

Phase Tv;o : Prototype Evaluation Process Testing

Item 8. Same as Item 1.

Item 9. Same as Item 2 , but change reference to Items

10 through 14.

Item 10. Refine and revise the work accomplished under

Item 3, as related to the further objective of

designing and testing a prototype of the complete

evaluation process.

Item 11. Refine and revise the v;ork accomplished under

Item 4, as related to the further objective of

preparing a formal framework within which the

objectives, organizations, and programs may be

related to the evaluation process.

Item 12. Refine and revise the work accomplished under

Item 5, as related to the further objective

to achieve a comprehensive, detailed design

to meet both basic data (base line data) and

special data requirements.

Item 13. Refine and revise the v;ork accomplished under

Item 6, as related to the further objective

of collecting both basic and -special data.

Item 14. Refine and revise the work accomplished under

Item 7, as related to the analysis of both

basic and special data; and provide findings

and recommendations

.

Phase Three; On-line/Turnkey Evaluation

Items 15

thru 21. Revision and refinement of respective prior

items in relation to the additional objective

of an -on-line turnkey evaluation process.

22
Item

380

RFP G-35756

stateme:it of v;ork

(with illustrative sub-items for Phase One)

Note: The items listed below provide a categorized

sumjnary of the services and materials required

on this program.

Phase One Planning and Design, and Preliminary Evaluation

1. Plan, organize, staff, direct, and control, including

both schedule and costs.

a. Provide the necessary services and materials for

management of the overall program, including a

management plan.

b. Prepare a preliminary detailed schedule, and

provide progressive analysis and rescheduling,

including proposed schedule for Phase Tv;o

c. Prepare periodic cost analyses and projections,

including proposed costs for Phase Tv/o

23
Item 2

.

381

RFP 6-35756

Provide reports, liaison, and services for purposes

of review, coordination, and approvals and acceptances, including the following:

Provide services and materials necessary for liaison

with ETIP, PAA, and relevant other individuals and

organizations

.

a. Periodic and special formal and informal management progress reports, final report

a(l) . Submit program management plan at or before

completion of 3d month, and report modifications as required.

a(2) . Submit final report at or before completion

of 15th month.

b. Periodic and final scheduling reports, including

projected schedule for the next phase'

b(l) . Submit projected Phase One schedule at or

before completion of 3rd month, and report

modifications as required.

b(2). Submit projected Phase Two schedule at or

before completion of 9th month.

b(3) . Submit final report at or before completion

of 15th month.

c. Periodic and final ‘cost reports, including projected

costs for the' next phase.

c(l) . Submit projected Phase One costs at or before

completion of 3rd month, and report modifications as required.

c(2) . Submit final report at or before completion

of 15th month.

d. Periodic and special formal and informal technical

progress reports, final report, including proposed

program for the next phase. (Note: The substantive

work to be reported here is to be performed under

Items 3 through 7.)

24
382

RFP 6-3575G

"

Submit reports, including formal reports for the items in the

follov;ing table on or before completion of the month (s)

specified

3a 3rd and 6th 6a 6th

b 3rd b -

c 6th c 6 th

d 9th d -

4a 3rd and 6th 7a 6 th

b 6th and 12th b 12th

5 a 3rd and 6th c 12th

b 6th d 15 th

c 6 th e 15 th

d 12 th

25
383

RFP 6-35756

Item 3. Review and structure the proposed objectives in

terms of the experimental setting and related

background, and establish an overall plan and

set of priorities.

a. Review, through docum.ents and interviev;s , the

objectives of ETIP, PAA, and other relevant

individuals and organizations, including

professional standards of evaluation, and prepare

a description and supporting analysis suitable

for providing a basis for planning and setting

of priorities for Phase One.

b. Based on the above, identify, and prepare preliminary recommendations and supporting detail

requirements for, those specific evaluations

which should be accomplished during Phase One,

including both exploratory (and/or descriptive)

evaluations and a priori proposition testing

evaluations

.

c. Based on the above, prepare preliminary statement

of requiremients for the pilot evaluation system

in Phase One.

d. Based upon the above and the other activities

in Phase One, prepare a preliminary statem.ent

of the requirements for the prototype evaluation

system in Phase Two, together with supporting

analysis.

26
384

RFP G-3575G

Item 4. Reviev; and structure the proposed set of experiments,

and related background in terms of the proposed

objectives, and provide a reference base.

a. Reviev;, through documents and interviews, the

specific procurement experiments (past, present,

and proposed) , and the related programs and

organizations*, and prepare a description and

supporting analysis suitable for planning

the activities of Phase One in conjunction v/ith

the work in Item 3a.

* (AGENCY IMPACT ONLY) V/i.th prim^ary emphasis

on PAA, and not on industry sector.

(COlblERCIAL IMPACT ONLY) V7ith primary emphasis

on industry sector, and not on PAA.

b. Prepare a preliminary framiework m.odel v/hich

provides a base for synthesizing objectives,

organization, program.s , and the evaluation

process (system), v;ith supporting analysis.

27
Item 5.

385

RFP 6-35756

Develop the detailed evaluation requirements (problem

definition and experimental or study design) , in

relation to collection and analysis.

a. Prepare a preliminary design of both exploratory

(and/or descriptive) and a priori evaluations to

be conducted during Phase One, including 1) statement of problem or hypothesis, 2) supporting

theories, and 3) definitions of variables and

parameters

.

b. Based on the above, prepare a detailed design of

each evaluation, with special emphasis on specific

procurement experiments, including 1) choice of

experimental design, 2) (where appropriate)

sam.pling strategies, and 3) data collection and

analysis methods.

c. Prepare a preliminary design of the pilot evaluation system in Phase One, including data collection

and analysis.

Based on the above, prepare a preliminary design of

the prototype evaluation system in Phase Two,

including both basic and special processes

.

d.
Item 6

V.

386

RFP 6-35756

Develop the detailed data collection process, including instruments, identification of, and access

to, sources, and managing the process; and collect

the data.

a. Design and develop the data collection process

required for the specific Phase One evaluations.

b. Based upon the v;ork in Items 3, 4, and 5, collect

additional data required for specific exploratory

and/or descriptive) evaluations.

c. Design and pilot test the instruments necessary for

specific a priori proposition testing evaluations.

d. collect the data reauired for specific a priori

proposition testing evaluations.

§

29
387

RFP 6-35756

Item 7. Develop the detailed data analysis process; carry out

the analyses; present findings; present recommendations.

a. Design and develop the data analysis process required for the specific Phase One evaluations.

b. Carry out the specific Phase One exploratory (and/or

descriptive) evaluations, and present findings.

c. Carry out the specific Phase One a priori propositioi

testing evaluations, and present findings.

d. Analyse the activities of Phase One which affect the

design of the evaluation system, and prepare

recommendations

.

e. Analyse the activities of Phase One, and prepare

recommendations for Phase Tvzo

.
Phase Tv;o

333

RFP 6-35756

Prototype Evaluation Process Testing

Item 8. Same as Item 1.

Item 9. Same as Item 2, but change reference to Items 10

through 14

Item 10. Refine and revise the

Item 3, as related to

designing and testing

evaluation process.

work accomplished under

the further objective of

a prototype of the complete

Item 11. Refine and revise the v;ork accomplished under Item 4,

related to the further objective of preparing a

formal framework v;ithin v;hich the objectives, organizations, and programs may be related to the evaluation

process

Item 12. Refine and revise the v/ork accomplished under Item 5,

as related to the further objective to achieve a

comprehensive detailed design to meet both basic

data (base line data) and special data requirements.

Item 13. Refine and revise the v/ork accom.plished under Item 6,

as related to the further objective of collecting both

basic and special data.

Refine and revise the v;ork accomplished under Item 7,

as related to the analysis of both basic and special

data; and provide findings and recommendations.

Item 14
389

RFP 6-35756

Phase Three: On-Line/Turn-Key Evaluation

Items 15 through 21. Revision and refinement of respective

prior items, in relation to the additional objective of an on-line/turnkey evaluation process.

$

32
390

PROPOSED SCHEDULE - OVERALL

{Agency Impact &. Cofnmercial Impact)

ITEM DESCRIPTION

PHASE ONE

1. PROGRAM MANAGEMENT

2. REP ORTS & REVIEWS

3. OBJECTIVES

4. BACKGROUND

5. EVALUATION DESIGN

6. DATA COLLECTION

7. DATA ANALYSIS

PHASE TWO

8. PROGRAM MANAGEMENT

9. REPORTS & REVIEWS

10. OBJECTIVES

11. BACKGROUND

12. EVALUATION DESIGN

13. DATA COLLECTION

14. DATA ANALYSIS

PHASE THREE

15. PROGRAM MANAGEMENT

16. REPORTS & REVIEWS

17. OBJECTIVES

18. BACKGROUND

19. EVALUATION DESIGN

20. DATA COLLECTION

21. DATA ANALYSIS

NOTE: (FORMAL) PROGRESS REPORT

FINAL REPORT

ONLY
391

V *

ii

G 0

PROPOSED SCHEDULE - PHASE ONE

(Agency impact & Commercial impact)

Tasks, plus

selected Months

sub-tasks

1. PROGRAr'/l MANAGEMENT

Phase Two cost proposal

2. REPORTS & REVIEWS

Design of reporting system

3. OBJECTIVES

Preliminary analysis

Identification of specific Phase One evaluations

Preliminary evaluation system objectives

Evaluation system objectives

1 I 2 I 3 I 4

T

7 I 8 I 9 Ho

I-V

V

V

V

1 1M2 131 i4

7

4. BACKGROUND

Preliminary analysis

Analysis of relevant organizations

Preliminary framework

5. EVALUATION DESIGN

Analysis of Phase One evaluations

Design of Phase One evaluations

(including sampling strategies)

Preliminary design of evaluation system

(including collection & analysis)

Design of evaluation system

(including basic & special)

6. DATA COLLECTION

General data collection

Instrument design & pilot test

Data on specific experiments

7. DATA ANALYSIS

Exploratory analysis & findings

A priori analysis & findings

Analysis of evaluation system

Recommendations

54
392

RFP 6-35756

I

I.l

1.2

1.3

1.4

PROPOSAL EVALUATION FACTORS

Management

This factor is a measure of the background

and overall management capability of the bidder.

Evaluation Management Experience

^his sub-factor measures the bidder's experience

and/or capability of the personnel he v/ill

use on thi.s program.

Importance of Program to the Bidder

This sub-factor is a measore of the degree to

which the proposal represents the approval and

direction of the company rather than the efforts

of professional proposal v/riters , and also is a

measure of the warranty offered by the company

to back the program with the necessary resources.

Program Management

This sub-factor is the m^easure of the bidder's own

proposed method for planning, organizing, and

controlling the program.

Schedule and Cost Control

This factor measures the bidder's ability to

predict the time and cost requirements of his

efforts, to plan the interrelation of the several

parts, and to exercise the necessary control.

I

35
393

m

«

*

RFP 6-35756

II. Reports and Reviev/s

This factor is the measure of the bidder's

ability to assure that the government is

able to effectively and progressively review

a very large scale effort v/ith a minimum of

people, and of the bidder's ability to

coordinate his activities v/ith those of

other related individuals and organizations.

11. 1 Reports and Liaison with the Experimental

Technology Incentives Program (ETIP)

.

This sub-factor measures that part of the

overall factor v/hich relates to the

Experimental Technology Incentives Program

(ETIP)

.

11. 2 Reports and Liaison with the Program Administrative

Agency (PAA)

This sub-factor measures that part of the overall facto

which relates to the program administrative agency

(and, where appropriate, its subelemients or related

organizations)

.

II. 3 Liaison v/ith Others

This sub-factor measures that part of the

overall factor which relates to individuals

and organizations other than those described

above

.

II. 4 Approvals and Acceptances

This sub-factor measures the bidder's understanding

and acceptance of the necessity for progressive

and substantial approval and acceptance during

each phase, as well as at the end of each phase.

36
III. objectives

3Q/|

RFP

\ III. 1

III. 2

III . 3

III. 4

III.

This factor is a measure of the

undcrstandinc; of the objectives

(and related programs)

bidder's overall

of this procurement

ETIP Evaluation Objectives

This sub-factor is a measure of the bidder's understanding of the specific objectives of this procurement.

Program Administrative Agency (PAA.) Objectives

This sub-factor is a measure of the bidder's

understanding of the specific objectives of the

program (s) and/or experiment (s) that are to be

evaluated.

Other Relevant Objectives

This sub-factor is a measure of the bidder's understanding of relevant and related objectives, goals,

criteria, standards, and the like, including the

state of the art in administrative experimentation

(and/or evaluation)

I Phase Tv;o Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the incorporation of the results of

Phase One in the restated objectives for Phase Tv/o

Phase Three Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the reformulation of objectives in

Phase Three.

37
395

RFP 6-35756

IV.

IV. 1

IV. 2

IV. 3

IV. 4

IV. 5

Background Information

This factor is a measure of the bidder's overall

understanding of the background and characteristics

of this procurement.

ETIP

This sub-factor is a measure of the bidder's understanding of ETIP and the specific program (s) and/or

experiment (s) , in terms of past history, present

status and plans for the future.

Program Administrative Agency (PAA)

This sub-factor is a measure of the bidder's

understanding of the program administrative

agency (and, where appropriate, its subelem.ents

or related organizations), its organization,

policies and procedures.

Other Relevant Background

This sub-factor is a measure of the bidder's understanding of the programs, organization, policies

and procedures of other relevant organizations.

Phase Two Background

This sub-factor is a measure of the bidder's

capability to prepare a comprehensive framework

or model of the program(s) and/or experiment (s)

and the evaluation process.

Phase Three Background

This sub-factor is a measure of the bidder's awareness of possible further refinements in the model

necessary for the continuation of the evaluation

process

,

38
396

RFP 6-35756

V. Evaluation Design

This factor is a measure of the bidder’s capabilities

to carry out the evaluation design requirements.

V.l Exploratory (Descriptive) and A Priori Proposition

Testing Hypotheses, Variables and Parameters

This sub-factor is a measure of the bidder's ability

to define the detailed evaluation objectives.

V.2 Experimental and/or Study Designs

This sub-factor is a measure of the bidder's

ability to define the overall experimental

and/or study design.

V.3 Data Collection Design

This sub-factor is a measure of the bidder ' s ability

to design the overall data collection process , including

sources of information, sam.pling strategies, timing,

etc

V.4 Data Analysis Design

This sub-factor is a measure of the bidder's

ability to design the overall data analysis process,

including processing and analysis, and presenting

results and recommendations

V.5 Phase Two Evaluation Design

This sub-factor is a measure of the bidder's ability

to redefine the detailed experimental and/or study

design to reflect the results of Phase One, the

several objectives, and model of the process in a

comprehensive detailed design to m.eet both basic

data (base line data) and special data requiremients

V.6 Phase Three Evaluation Design

This sub-factor is a measure of the bidder's ability to

further redefine the above, to provide for a detailed

design, and necessary supporting materials, v/hich v7ould

allow an on-going evaluation process.

59
m

«

n

0

' M

0

397

RFP 6-35756

I

VI. Data Collection Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

acquisition of data on the variables and parameters

of interest.

VI. 1 Instrument Design

This sub-factor is a measure of the bidder's ability

to develop (and test) the various questionnaires

,

protocols, schedules, and the like required for data

collection.

VI. 2 Information Sources

This sub-factor is a measure of the bidder's

ability to identify the various inform.ation

sources, and to gain access.

VI. 3 Data Collection

This sub-factor is measure of the bidder's capability

to organize, staff, train, direct and control the

personnel who obtain the data by survey, interview,

observation, or use of records.

VI. 4 Phase Two Data Collection

This sub-factor is a measure of the bidder's ability

to anticipate the requirements of the Phase Tv;o

collection of both basic (base line) and special

data.

VI. 5 Phase Three Data Collection

This sub-factor is a measure of the bidder's ability

to plan against the requirement to design an

ongoing collection process.
VII.

VII. 1

VII. 2

VII. 3

VII. 4

RFP GData Analysis Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

analysis of the dfita and to present the results and

recommendations

Analysis

This sub-factor is the measure of the bidder's ability

to identify and apply the appropriate analytical

(including statistical) techniques to the data

collected

Exploratory (and/or Descriptive) Findings

This sub-factor is a measure of the bidder's

capability to identify and present exploratory

findings

A Priori Proposition Testing Findings.

This sub-factor is a measure of the bidder's

capability to identify and present a priori

findings

Recomm.endations

This sub-factor is a measure of the bidder's

understanding of the relationship betv/een the

results of Phase One arid the work in Phase Two,

in terms of additional, revised, or deleted

hypotheses, changes in objectives or background

information, and the relative feasibility and/or

usefulness of alternative designs for evaluation.

41
1

i'

a

a

a

a

a

a

a

a

a

a

a

a

399

RFP 6-35756

2/1/76

PROPOSAL EVALUATIOn FACTORS

(V7ith Illustrative Questions)

Note: These factors and sub-factors apply to both agency

and commercial impact proposals. Where specific

illustrative questions apply primarily to either agency

or commercial impact only, they are so identified.

I . Management

This factor is a measure of the background

and overall management capability of the bidder.

I.l Evaluation Management Experience

This sub-factor measures the bidder's experience

and/or capability of the personnel he will

use on this program.

1.1.1 Discuss, with specific examples, recent experience in the

management of field experiments, projects, and/or programs.

Indicate size and complexity of program, extent of ov;n responsibility for design of the program (or experiment) and the evaluation. Indicate by tabulation of cross-references other subsections of this proposal v/here you have referred to any program

discussed here. Outline the original contracted-for schedules

and costs and requirements, and explain changes. If you have

had any "bad" experience here, discuss the reasons and, if

appropriate, indicate how you would prevent them from occurring

on this program. If appropriate, you should provide the

names, addresses, and telephone numbers of cognizant COTR's.

1.1.2 If you do not have the kind of experience discussed above,

discuss your basis for believing that you can m.eet the requirements of this program. Discuss in terms of the personnel you

intend to use.

1.1.3 How V7ill you combine and coordinate the efforts of

staff members who are familiar wU-h the procurement process,

the program administrative ager»cy/ or the commercial sector

with those staff members v;ho hn^'c the technical skills in

evaluation?

I J
1 .

Importance of Program to the Bidder

UOO

RFP 6-3575^,

This sub-factor is a measure of the degree to

which the proposal represents the approval and

direction of the company rather than the efforts

of professional proposal writers, and also is a

measure of the v/arranty offered by the company

to back the program with the necessary resources.

1.2.1 What is the relative size of this program with respect

to the overall sales of the company? If your com.pany has

significant decentralization or there are other factors v/hich

should be presented for clarity, please furnish this background. If the proportionate size is relatively large, why

do you think that the problems associated with the "single

product" will not arise? If, on the other hand, the proportionate part is relatively sm.all, v;hat assurance is there

that this program v/ill not be downrated by others after the

contract is signed?

1.2.2 Discuss, if appropriate, hov/ this program fits in with

the long range planning of your com.pany. What is its significance to your future position? If you envisage an increase in

programs of this type, what assurance is there that key personnel

on this program, will not be v/ithdrawn for nev;er programs?

1.2.3 It should be noted that the extent and nature of your

effort in proposing v/ill be considered in the evaluation of

this sub-factor.

43
401

«

n

«

a

a

a

RFP 6-35756

1.3 Program Management

This sub-factor is the measure of the bidder's ovm

proposed method for planning, organizing, and

controlling the program.

1.3.1 It is proposed that the planning and carrying out

of the preparation of this proposal provides an opportunity

for the bidder to demonstrate his capability in planning a

complex effort. Please describe briefly your process and,

particularly, the extent to v/hich it is a "pilot" of the

process you will use in managing this program.

1.3.2 Detail your proposed organization by time phases,

including buildup, division of responsibility; generally

describe its operation, how it is organized, and special

features not self-explanatory. Can the effectiveness of

your proposed organization be projected from its performance

in putting together this proposal? If the organization you

propose is essentially "new," either in form or in the

persons assigned, comment on the extent to which you have

been able to "test" it out in preparing this proposal.

\

44
RFP 6-3 5^?;^

1.4 Schedule and Cost Control

This factor measures the bidder's ability to

predict the time and cost requirements of his

efforts, to plan the interrelation of the several

parts, and to exercise the necessary control.

1.4.1 A detailed, thorough schedule provides not only a comprehensive plan but also a pov;erful method of controlling

and reviev/ing the progress of the program. The proposed

schedule does not include a detailed breakdown of all of

the interrelated areas but does furnish a broad guide for

planning purposes. In general, do you think the schedule

is realistic? Is it too tight? V7hat additional assumptions

are required? What conditions or events v/ill affect the

schedule?

1.4.2 Discuss the method of preparation of the schedule you

V7ill use, how you v;ill update it, how you v;ill use it.

1.4.3 Prepare a supporting analysis and sumimary of any areas

which you believe reflect a significant relative cost advantage

or disadvantage on your part. Indicate to v;hau extent your

in-house or subcontracting plans will take advantage of

in-being capabilities and at v;hat potential savings. To

what extent have provisions for contingencies been m.ade?

1.4.4 Discuss the form and timing of your detailed cost

proposal for the major field data gathering activity in

the second and third phases.

1.4.5 While the overall schedule reflects the requirem.ent

tq design an on-going evaluation system, there is an

important requirement to provide during Phase One (and each

subsequent phase) specific outputs in the form of evaluations

of selected specific experiments. How will you assure this?

1.4.6 The timing of the experiments varies across the several

PAA’s. What advantages or disadvantages do you see in this?

1.4.7 It is anticipated that another ("agency i.mpact")

contractor will be primarily responsible for evaluating the

intervention and proximal imipacts of the several experim.ents ; it is also anticipated that some, if not all, ccmiaercial

impacts will substantially lag the experiments. What advantages

or disadvantages do you see in this? (COMMERCIAL IMPACT OMLY)

45
i

i

i

1

e a

a

a

a

a

a

a

a

'a

a

a

U03

RFP 6-35756

II. Reports and Reviews

This factor is the measure of the bidder's

ability to assure that the government is

able to effectively and progressively review

a very large scale effort v/ith a minimum of

people, and of the bidder's ability to

coordinate his activities with those of

other related individuals and organizations.

II. 1 Reports and Liaison with the Experimental

Technology Incentives Program (ETIP)

This sub-factor measures that part of the

overall factor which relates to the

Experimental Technology Incentives Program

(ETIP) .

11. 1.1 Discuss generally the extent and nature of liaison

with both the Contracting Officer (CO) and the Contracting

Officer's Technical Representative (COTR) that you envision

in this program. Discuss generally the information needs of

both government and contractor personnel; who will establish

what the needs are and how they will be satisfied?

11. 1.2 It would be useful if the reporting system were

designed in a way that facilitated an ongoing process

evaluation of a) the experiments conducted by the program

administrative agency, and b) the evaluation effort of the

contractor himself. The data provided by such a system could

serve as the basis for a descriptive case history of the

experiments and their evaluation. Discuss generally how

feasible this proposal is, and how it might be accomplished.

You may wish to relate it to your discussion in IV. 1.2, and

comparable discussions.

46
RFP 6-35^56

II . 2

11. 2.1

both management personnel and technical or specialized staff.

How will you reconcile the desirability of direct:, informal communication with the necessity, in at least some cases, for

observing formial clearance and approval "through channels"?

11. 2.

The evaluation effort may sometimes be perceived as

threatening by various personnel in the program administrative agency. Discuss hov/ the reporting and liaison arrangements can be developed to reduce the likelihood of this.

11. 2.

Discuss the differences, if any, in how you will relate

to the several PAA. To what extent v;ill these be separately

identifiable efforts?

II. 2.It is anticipated that another ("agency im.pact")

contractor will be prim.arily responsible for coordination

with the several PAA. Vlhat do you think of this, a.nd how

will you deal with it? (C02-D1ERCIAL IMPACT ONLY)

Reports and Liaison with the Program Administrative

Agency (PAA)

This sub-factor measures that part of the overall facto

which relates to the prcgra::. adm.inistrative agency

(and, v/here appropriate, its subelem.ents or re laced

organizations)

Discuss generally the extent and nature of liaison v/ith

47
U05

RFP 6-35756

m

N

ii

«

«

II. 3 Liaison with Others

This sub-factor measures that part of the

overall factor which relates to individuals

and organizations other than those described

above

.

11. 3.1 It is anticipated that there will be related programs

and program evaluations being carried out by the governmient

agencies either in-house or through the use of an outside

consultant or contractor. Discuss the nature and form of

the relationship you see with them. How will you ensure

that the necessary coordination, avoidance of duplication,

overlap, or gaps, etc., is accomplished?

11. 3.2 To some extent, it is expected that individual companies

(e.g., manufacturers of products which are the subject of program experiments), industry associations, and others miay desire

or require a liaison relationship. Comment.

11. 3. 3 Reporting and liaison relationships may arise with

individuals and organizations with professional interests

(e.g., professional societies, conferences, professional

journals, consumer or public interest groups) in the program or program evaluation. Discuss.

II. 3. 4 It is anticipated that another ("commercial impact")

contractor will be primarily responsible for coordination with

individual companies, etc. Comment. (AGENCY IMPACT ONLY)

II. 3.5 It is anticipated that the bidder will be primarily

responsible for coordination with individual com.panies , etc.

Comment. (COMMERCIAL IMPACT ONLY)

48
^06

RFP 6-3575C

II. 4 Approvals and Acceptances

This sub-factor measures the bidder's understanding

and acceptance of the necessity for progressive

and substantial approval and acceptance during

each phase, as v/ell as at the end of each phase.

II. 4.1 It is intended that the formal approval and acceptance

of the v;ork of the contractor for each phase v;ill be based,

primarily, upon the several reports submitted during and

at the end of the phase, v/ith final approval at the end of

the phase. It is intended, however, that this final approval

and acceptance will be largely a formal confirmation of

already established acceptable performiance and/or results.

This is required not only to overcome the substantial delays

of tandem or sequential review, but also to avoid re’.;ork,

redesign and argument under pressure of time and after the

fact. You should discuss your understanding of this, the

advantages and disadvantages, and your willingness to accept

the contractual and schedule implications, that is, that this

means that the delivery dates are dates of delivery of

approved items, not merely dates for submission of items

with an open time term.

49
RFP 6-35756 ^07

«

N

«

m

m

a

m

a

a

a

a

a

a

a

a

a

a

III

.

Objectives

This factor is a measure of the bidder's overall

understanding of the objectives of this procurement

(and related programs).

111.1 ETIP Evaluation Objectives

This sub-factor is a measure of the bidder's understanding of the specific objectives of this procurem.ent

.

111.1. 1 The specific objectives for this procurement, and

related objectives, including the overall objectives of

ETIP, appear in various forms in the several parts of this

bid set, and in other sources, and reflect a variety of

authors, circumstances and purposes. It is proposed that

it is neither necessary nor desirable for ETIP to prepare

a single, integrated, fully rationalized, and "authoritative"

statement of the objectives for the guidance of the bidders

(or the successful bidder) . Comment on why you believe this

is (or is not) justified?

III.

1.2

It is expected that the various stated objectives

may not appear (or be) consistent; how will you handle this?

For example, a high credibility (a priori proposition testing)

evaluation of a specific experiment is desired but it: is not

possible to achieve this (v;ith reasonable time and money) . For example, a complete and candid description of how a

particular experiment was carried out may reflect gross

incompetence or dereliction on the part of specific ETIP

(or program administrative agency) personnel. For example,

the results of a particular evaluation may disclose a

systematic defect in the past performance of the agency

(or a contractor)

.

111.1. 3 It is anticipated that, for a particular experiment,

there may be as many as four parallel objectives: a) to

describe what happened, what went well, what went wrong;

b) to determine whether the result was "caused" by the

experimental manipulation; c) to demonstrate the ability

to "evaluate" scientifically with credible results; d) to

claim credit for the experiment and/or the results. How

does one determine the relative importance of these several

objectives for a specific experiment over the set of all

comparable experiments?

111. 1.4 It is anticipated that ETIP evaluation objectives or

priorities may shift during the course of the evaluation

effort. How does one ensure that he will be sensitive to

such shifts and how does one deal with such shifts? How would

you go about improving the clarity of ETIP's evaluation

objectives?

50
RFP

III. 2 Program Administrative Agency (PAA) Objectives

This sub-factor is a measure of the bidder's

understanding of the specific objectives of the

program (s) and/or experiment (s ) that are to be

evaluated

III. 2.1 Questions comparable to those suggested under III.l

may appear here. Comment. What special proble.ms, if any,

do you anticipate because of the multiple PAA?

III. 2. 2 How will you reconcile conflicts between ETI? a.nd

the program administrative agency objectives for a particular experiment? PAA objectives for the experiment and

other PAA objectives? What kinds of problems will com.e

up? How do you plan to deal v;ith them?

III. 2. 3 It is anticipated that, during the period that the

work on this contract is accomplished, there may be

changes in policy or procedures or of key personnel of the

program administrative agency. How v/ill you deal v:ich this?

For exam.ple, the "results" of a particular experim:enu may be

sufficiently "obvious" to the agency to no longer m.ake

necessary any further evaluation. Will a continuation of

the evaluation be undertaken under any circum.stances? And,

if so, how v/ould you justify it and/or obtain agency cooperation?

III. 2. 4 Suppose the agency adds new objectives or objectives

not presently contemplated are uncovered v/hich represent

potentially significant changes in the direction or scope

of the present evaluation program? Hov; will you react to

this?

III. 2. 5 It is anticipated that another ("agency im.pact")

contractor will be primarily responsible for exam.ining the

objectives of the PAA. Comment.' (C0>'E-1ERCIAL IMPACT OMLY)

51
409

RFP 6-35756

m

n

\

m

«

a

a

a

a

a

a

a

a

III. 3 Other Relevant Objectives

This sub-factor is a measure of the bidder's understanding of relevant and related objectives, goals,

criteria, standards, and the like, including the

state of the art in administrative experimentation

(and/or evaluation)

.

111. 3.1 It is proposed that the objectives of ETIP and PAA

do not exist in isolation, that both specific and general

objectives of a wide variety of other individuals, organisations, and larger cominunities will necessarily interact with

the present program. For example, personal and career objectives of key individuals. For example, political, social', and

legal objectives of the federal governm.ent . For example,

objectives expressed or implied in the state-of-the-art and

professional standards in experimental and evaluation research.

Of what significance are these, and how do you expect to deal

with them?

111. 3.2 Your understanding of the state-of-the-art,

particularly in administrative experimentation and/or

evaluation, v;ill be measured by your discussion of relevant

questions raised elsewhere. If you consider it appropriate,

you may discuss this here, and you may wish to cross-reference

significant material which appear^ elsewhere.

111. 3. 3 Are there relevant professional standards in

experimentation that you must observe that may conflict

with ETIP or PAA objectives? How will you handle this?

You may wish to relate your answer to II. 3.

111. 3. 4 It is anticipated that another ("commercial impact")

contractor will be primarily responsible for exam.ining the

objectives of individual companies and others in the

commercial sector. Comment. (AGENCY IMPACT ONLY)

111. 3.5 It is anticipated that the objectives of individual

companies, associations, etc., may significantly affect not

only the experiments and their evaluation but also the

activities of the bidder. Comment. (COMMERCIAL IMPACT ONLY)

52
RFP 6-35756 H

III. 4 Phase Two Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the incorporation of the results of

Phase One in the restated objectives for Phase Two.

III. 4.1 It is proposed that, as a result of the work in

Phase One, a reasonable description and integration of the

several objectives will have been realized, and that this

will be used as a basis for the several other tasks but

also as a basis for refinement, revision, and additions

during Phase Tv/o . In addition to this continuing interest

in the objectives introduced in Phase One, it is proposed

that a separate objective of Phase Two is to design and

test a prototype of the complete evaluation process

necessary to meet the several objectives. It is expected

that this may include at least tv/o separately identifiable

parts, as follows: a basic evaluation process to include

those evaluation activities ^vhich deal with key, predictable,

identifiable, continuing, common objectives for which the

process can be "institutionalized"; and a special evaluation process to include those evaluation activities which

do not rrieet the above requirem.ents because of specialized,

one-time, or changing objectives. If you believe your

discussion elsewhere does not adequately reflect your understanding of this, you may comment here.

III. 4. 2 What activities during Phase One v/ill be expected

to contribute to the refinement of Phase Tv;o objectives?

What other sources may be expected to provide input? You

may wish to cross-reference your comments under IV. 4, or

elsewhere

55
^11

RFP 6-35756

i

i'

i

1

a

a

a

a

a

a

a

a

a

a

III. 5 Phase Three Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the reformulation of objectives in

Phase Three.

III. 5.1 In addition to a continuation of the concern with the

objectives of the previous two phases, it is proposed that

the objective of Phase Three will be to provide a firm basis

for any further related evaluation efforts, including any of

the following: a) a standby or "now to" capability to design

and carry out evaluations for comparable future experiments;

b) an on-line, turnkey data gathering and analysis process

which might be implemented by the PAA; c) an assessment of

the need (or lack of need) for other specific or general

future evaluation efforts. How realistic do you think this

is, and to v/hat extent will the prior phases prepare you to

successfully define and then meet these objectives?

$

54
RFP 6-35756 1;_2

Background Information

This factor is a measure of the bidder's overall

understanding of the background and characteristics

of this procurement.

ETIP

This sub-factor is a measure of the bidder ' s understanding of ETIP and the specific progra.m(s) and/or

experiment (s ) , in terms of past history, present

status and plans for the future.

IV. 1.1 Analogous to the comments in III. 1.1, it is not

proposed to provide a complete and comprehensive descriptio.n

of ETIP and the specific program(s) and/or experimient ( s ) for

the guidance of the bidders (or the successful bidder) . Discuss the adequacy of your present understanding of ETIP

and the specific programs; if you believe it is not adequate,

how do you propose to correct this consistent v.'ith the

requirements of Phase One?

IV. 1.2 It is assumed that a reasonably com.plete and current

description of at least the specific program.(s) a.nd/or

experiment (s ) will be desirable, if not necessary, during

Phase One (and, perhaps, on a continuing basis) for use

not only by the bidder but also by ETIP. How can this be

accomplished? What forms and procedures do you visualize

will be used to provide for an up-to-date record? How

will inputs (revisions) be made? How will outputs be

available?

IV. 1.3 The availability of such an information base (and

system) would necessarily increase the opporounities to

make corrections in ongoing activities (experim.ents

and may suggest other improvements and chances in the

program. To what extent do you see it as your responsibility

to facilitate this process? What effect v/culd these additicr.a

changes have on your evaluation work?

IV. 1.4 Discuss your understanding of the overall set of

programs and/or experiments to be eyaluated, including at

least the following: the set of individual product procurements under life cycle costing (LCC) both planned and

implemented; the FSS decision to adopt LCC as a procurement

practice; the several workshops and the training program as

planned and implemented in all FSS offices around the country,

other planned or implem.ented procurement experiments, such

as value incentive contracting. Can these be described or

modeled in such a form as to disclose their interactions,

or, conversely, can specific experim^ents or programs be

defined in such a way as to allov; separate treatment?

IV.

IV. 1

55
1

i'

1

«

i

i

i

i

IV. 2 Program Administrative Agency (PAA)

This sub-factor is a measure of the bidder's

understanding of the program administrative

agency (and, where appropriate, its subelements

or related organizations), its organization,

policies and procedures.

IV. 2.1 Questions comparable to those suggested under IV. 1.1

may appear here. Comment. What special problems, if any,

do you anticipate because of the multiple PAA?

IV. 2. 2 As a further restatement of the above, if you do not

have a present acquaintance v;ith the program administrative

agency (and its constituent elements) , hov; v;ill you acquire

the necessary background?

IV. 2. 3 At least some parts of the organization, policies,

and procedures of the program administrative agency will

be critically related to the present program. For example,

those elements directly involved in the experiments, and

procurement policies and procedure. Discuss your understanding of these areas. If you do not have prior experience,

v;hat effect will this have on your capability?

• « IV. 2. 4 It is anticipated that another ("agency impact")

contractor will be primarily responsible for obtaining

information about and access to the PAA. How will you deal

with this? (COMI-IERCIAL IMPACT ONLY)
RFP 6-357S6

IV. 3 Other Relevant Background

This sub-factor is a measure of the bidder's unde

standing of the programs, organization , policies

and procedures of other relevant organize tions

.

IV. 3.1 It is anticipated that information about and access

to organizations other than ETIP and P.'Ji may be importa.nt

to this program. For example: other governm.ent age.ncies

which are the end users of the products procured; other

government agencies v/ith legal, financial, policy, or

other responsibilities which interact v/ith the program

area; specific com.panies, industry segments, a.nd industry

associations; consum>er organizations, safety, advertising,

news media, and the like; other ETIP or program, or

evaluation contractors. Discuss your understanding of the

potential effects of these interactions and hov/ you v/ill

deal with them.

IV. 3.2 It is anticipated that another ("commercial imicact")

contractor will be primarily responsible for obtaining info

about and access to individual com.panies and others in the

commercial sector. Comm.ent. (AGSb'CY III? ACT OITLY)

IV. 3. 3 It is anticipated that the bidder v/ill be primarily

responsible for obtaining inform.ation about and access to

individual companies a.nd others in the comm.ercial sector.

Comment. (COmERCIAL IMPACT ONLY)

57
RFP 6-35756 415

s

t

'

IV. 4 Phase Two Background

This sub-factor is a measure of the bidder's

capability to prepare a comprehensive framework

or model of the prograra(s) and/or experiment (s

)

and the evaluation process.

IV. 4.1 It is anticipated that, after Phase One, the bidder

v/ill be capable of preparing a formal framework v;ithin

which the objectives, organizations, and programs may be

related to the evaluation process. Such a framework would

not only integrate the background from Phase One but provide

a convenient and efficient capability for revision, additions

and modifications without having to start over again. Ccrrmen

on the feasibility of this; if you can, suggest hcv; this

might be done or how one v/ould determine whether it was

successful or not? You may wish to cross-reference your

comments on III. 4.

IV. 4. 2 Will the distinction between agency impact and

commercial impact require significantly different

frameworks? If tv;o separate frameworks are developed,

to what extent will it be necessary and/or feasible and/or

desirable to relate the frameworks?

IV. 4. 3 Discuss the same question with respect to the

distinction among the several PAA.
416

RFP 6-3575G

IV. 5 Phase Three Background

This sub-factor is a ir.easure of the bidder's av;areness of possible further refinements in the model

necessary for the continuation of the evaluation

process

.

IV. 5.1 It is proposed that the model, and its supporting

descriptive raaterial would provide a sufficient basis for

a turnkey operation so that PAA personnel could, with

modest specialized assistance, carry out on-going

evaluations for comparable programinatic extensions.

Comment on this.

IV. 5. 2 It is anticipated that the specific P.AA v;culd be

primarily interested in, and, be-, logically, the

appropriate organization to m^anage , the on-going "agency

impact" evaluation system (or that part concerned with

the specific agency). Comment. (AGENCY IMPA.CT OELY)

IV. 5. 3 It is anticipated that significant parts of the

"commercial impact" evaluation system may already be withi.n

the present capability (or future plans) of related o

tions (e.g.. Department of Commerce, industry associa

etc.). VThat advantages or disadvantages do you see in this

possibility, and how will you react to it? (CC2-D1ZRCIAL

IMPACT ONLY)

59

fti-t
417

RFP 6-35756

ia

izas,

is

»

I

I

I

i

V. Evaluation Design

This factor is a measure of the bidder's capabilities

to carry out the evaluation design requirements.

V. l Exploratory (Descriptive) and A Priori Proposition

Testing Hypotheses, Variables and Parameters

This sub-factor is a measure of the bidder's ability

to define the detailed evaluation objectives.

VI. 1.1 It is proposed that the various evaluation design requirements v;ill include problems ranging over several dimensions,

and combinations of dimensions, including the following:

a) from exploratory and/or descriptive evaluations (e.g.,

how did a specific experiment get started; what happened;

what are the more significant secondary effects) to a priori

proposition testing (e.g., did specific intervention X "cause"

the observed result Y) ; b) from very specific, identifiable

experiments (manipulations and/or effects) to general, diffuse,

overall changes; c) from post hoc evaluations of completed

experiments through evaluations of ongoing experiments, to

planning and design of future experiments. Commient on your

understanding of this.

V.l.

2

At least initially, the bidder will be called upon to

accomplish a preliminary systems design or study. If your

evaluation experience has been primarily in response to

detailed evaluation requirements provided in the RFP or

RFQ, discuss the relevance of your experience or capability

to the requirements of this program.

V.’1.3 Discuss your approach to the detailed definition of

a research (or experimental or evaluation) problem; comment

on theory building, definition of variables and param.eters

,

etc. If considered necessary, distinguish the approach to

a priori proposition testing from that used- in exploratory

and/or descriptive evaluation.

60
V. 2 Experimental and/or Study Designs

RFP 6-357SG

This sub-factor is a measure of the bidder's

ability to define the overall experimental

and/or study design.

V.2.1 Discuss the problem (s) of and approaches to experir.ental

and/or study design. Alternatively, ycu nay v;ish to dem.onstra

your capability by reference to the releva.nt education and

experience of your staff. It is suggested that you may v;ish

to distinguish among the several kinds of desig.ns v;hich may

be appropriate for meeting the several kinds of objectives

v/hich are the outcome of the process discussed in V.l.

V.2.2 Discuss the problem (s) of and approaches to ide.ntifying

and controlling for the potential effects of parameters.

V.2.3 it is proposed that the bidder can design an overall

evaluation system and, in parallel, carry cut evaluations of

specific selected experiments during Phase One. Discuss

the advantages and disadvantages of this. Ca.n you identify

specific experiments which can be evaluated duri.ng Phase

One? Can you identify base-line dara v;hich can, or should,

be collected during Phase One? v;hich can, or should, be

collected during later phases?

V.2.4 To what extent do you see differences among the several

PAA which V7ill affect the design of uhe evaluation system.?

Will differences in the progress in initiating experim.entis

among the several PAA. be an advantage or disadvantage?

V.2.5 It is anticipated that some part or all of the bidder's

activity in this area will be dependent upon the activities

of another ("agency impact") contractor. Hov; critical is

this from a time point of view? from an interface point of

view? Will segmentation (by industry, technology, market)

provide advantages or disadvantages? to what extent will

longitudinal designs be important, and v;ill this be an

advantage or disadvantage (COilMERCIAL IMPACT ONLY)

61
419

RFP G-3575G

I

Dtal

'Strata

sli

y

of

I

ral

I

I

I'

I

I

I

I

I

I

V.3 Data Collection Design

This sub-factor is a measure of the bidder's ability

to design the overall data collection process, including

sources of information, sampling strategies, timing,

etc

.

V.3.1 Discuss the various kinds of sources or m^ethods of

collecting information (e.g., observations, questionnaires,

interviews, records) in terms of your expectations for

their use here, and their advantages and disadvantages.

V.3.2 The nature and extent of the information required will,

at least in some cases, require the development of sampling

strategies. Describe, by reference to relevant experience,

if appropriate, your capability to develop sampling strategies

for a priori proposition testing evaluations. You may wish

to discuss the approaches used for exploratory evaluation.

You may wish to comment on some of the sampling strategies

provided as illustrations in the several reference documents.

V.3.

3

Discuss any special considerations v;hich may be introduced by the differences among the several PAA.

V.3.4 It is anticipated that the evaluation of agency impact

and commercial impact v/ill require the obtaining of, insome cases, not only the same data but also separate

additional data from coimnon sources. How can this be

coordinated to minimize duplication and, particularly,

the burdening of sources?

62
420

RrP 6-35756

V.4 Data Analysis Design

This sub-factor is a measure of the bidder's

ability to design the overall data analysis process,

including processing and analysis, and presenting

results and recommendations.

V.4.1 It is proposed that this part of the process should be

developed concurrently v/ith those discussed in V.2 and V.3

instead of waiting until the data is collected. Comment on

the desirability, necessity, and/or feasibility of this.

V.4.

2

Specific individual experiments and specific exploratory

studies may have not only different, but also severable

analytical require.ments (and discussion of these may more

appropriately appear in VII). It is expected, however, than

there may be some degree of overlap or interrelation among

the several experiments and studies. To v:hat extent v;ould

an overview of the overall process improve the ability to

perform the individual analyses?

V.4.3 It is proposed that the quantity and variety of the

various data collected (and to be collected in the future)

present an opportunity for careful planning with respect

to formatting identification, quality control, standardization

,

etc. To what extent do you think this is necessary and/or

feasible? To v;hat extent can com.patability be achieved wizr.

related existing data banks?
RFP 6-3 5%6

V.5 Phase Two Evaluation Design

This sub-factor is a measure of the bidder's ability

to redefine the detailed experimental and/or study

design to reflect the results of Phase One, the

several objectives, and model of the process in a

comprehensive detailed design to meet both basic

data (base line data) and special data requirements

.

V.5.1 Discuss, or, if appropriate, cross-reference discussions

elsewhere of, your understanding of the problems outlined in V,1

through V.4 as applicable to Phase Two, and with reference

to the requirements of Phase Two.

V.5.

2

It is anticipated that the bidder v;ill provide a

comprehensive and detailed design of an evaluation system

which will provide a capability for both basic and special

evaluation. Discuss your present understanding of the

form and/or extent of the evaluation system. Discuss the

advantages and disadvantages of prototype testing the system

by evaluating specific procurement or other experiments?

64
RFP 6-3575G

U22

V.6 Phase Three Evaluation Design

This sub-factor is a measure of the bidder's ability to

further redefine the above, to provide for a detailed

design, and necessary supporting materials, v;!;ich v;ouj.d

allov; an on-going evaluation process.

V.6.1 Discuss or, if appropriate, cross-reference discussions

elsewhere v/hich deraonstrate your ability.

V.6.

2

Discuss your understanding of what v/ill be necessary to

meet turnkey requirements. V/hat consideratio.n of agency policies

and procedures v;ill be required? Hov; v/ill the necessary skills

and experience be transferred? Hov; will previously collected

data be transferred? What coxuversion of instruments anc. sof~v/are

will be required? What v/ill be the effect of on-going evaluauicns

of specific experiments on the transition process? and vice versa?
RFP 6-35756 423

VI. Data. Collection Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

acquisition of data on the variables and parameters

of interest.

VI . 1 Instrument Design

This sub-factor is a measure of the bidder's ability

to develop (and test) the various questionnaires,

protocols, schedules, and the like required for data

collection

,

VI . 1 . 1 Discuss (you may use an example and/or reference to

prior experience) the problems and solutions which you consider critical (and/or characteristic) in instrumentation

for an a priori proposition testing type of evaluation.

VI . 2 Discuss the differences, if any, in the approach to

descriptive or exploratory evaluation.

VI. 1.3 Under what circumstances, and for what purposes,

will you conduct pilot tests? Validation tests?

VI. 1.4 It is anticipated that some variables of interest

will be of ongoing importance (as key variables or parameters

common to a number of experiments, or as key exploratory

or descriptive variables) in a sense comparable to "social

indicators" and will become part of what will be known as

"basic data" in future phases. To V7hat extent can this be

anticipated, and what difference will this make during this

phase?

VI. 1.5 Discuss your understanding of the special requirements,

if any, which a government agency (or its contractor) must

meet before use of instruments for data collection.

VI. 1.6 What advantages or disadvantages are introduced by

relevant or related existing data banks in terms of the

instruments they use (e.g., various definitions of industry

classifications, kinds of technology, etc.)?

66
42U

RFP C-3575C

\

VI. 2 Information Sources'

This sub-factor is a measure of the bidder's

ability to identify the various information

sources, and to gain access.

VI. 2.1 Discuss the important or critical problems you anticipate

in this area and your capability to deal v/ith them.

VI. 2. 2 Discuss the various strategies for identifying sources

of information. Distinguish, if appropriate, betv/een sou

of information needed for evaluating a specific experimen

with those to be used for a broad descriptive (case) study.

VI. 2. 3 Discuss the issues of confidentiality and privacy v/ith

respect to information obtained from individuals. Discuss

the issues related to information, in general, with restrictions

on dissemination, e.g., proprietary inform.at io.n , certain

types of personnel and financial information.

VI. 2. 4 Discuss the issues involved in access to i.ndividuals

and to records. A special issue of concern is the demands

on the time of individuals in the program ad.ministrat ive

agency, and others who may have a considerably less direcn

interest in the evaluation.

VI . 2 . 5 Much background and parametric inform.ation may be

available in some form, in existing records v;hich have been

collected for some other purpose; how will you identify these

and what advantages or disadvantages do they present?

67

it
425

RFP 6-35756

VI . 3 Data Collection

This sub-factor is measure of the bidder's capability

to organize, staff, train, direct and control the

personnel v/ho obtain the data by survey, interview,

observation, or use of records.

VI. 3,1 It is anticipated that the several requirements for

data collection may require different skills, different

organization, and different methods of supervision and

control. Discuss by example or reference to prior experience

your capability in this regard.

VI . 3 . 2 With reference to the question asked in VI. 1.4,

v/hat difference in approach V70uld be v/arranted with respect

to "basic data"?

VI . 3 .

3

Discuss or describe the process for handling data

collection (e.g., flowcharting). Discuss problems of

monitoring and evaluating data quality, of privacy and

confidentiality, of machine acceptable versus non-machine

acceptable data (e.g. , unstructured interviev/s) .

68
RFP 6-3575G

H26

VI. 4 Phase Tv/o Data Collection

This sub-factor is a measure of the bidder's ability

to anticipate the requirements of the Phase Tv/o

collection of both basic (base line) and special

data

.

VI . 4 . 1 It is proposed that the bidder v/ill be able to

institutionalize the collection of basic data (i.e., hey

variables and parameters common to a n'umber of present

or expected experim.ents , or key exploratory or descriptive

variables) during this phase; from this, base line data

can be obtained and a "test" of the process can be

accomplished. Comment on this.

VI. 4. 2 It is proposed that there v/ill remain other requirements not included within the above (e.g., specific one-tim.e

requirements, unusual or specialized requirem.ents ; added or

revised requirem.ents) v/hich cannot or should not be commingled with the above. Discuss this.

69
RFP G-35756 427

VI . 5 Phase Three Data Collection

This sub-factor is a measure of the bidder's ability

to plan against the requirement to design an

ongoing collection process.

VI. 5.1 At least the basic data collection process should

be sufficiently stabilized and described to allov/ an

ongoing, turnkey implementation. Discuss the problems

and prospects of this.

VI . 5 . 2 To v/hat extent can new or other specialized requirements be anticipated?

VI . 5 . 3 Discuss any special problems with respect to access,

privacy, confidentiality, etc., which may be introduced when

an agency takes over management and operation of the evaluation system.

70
RFP C-35756 ^28

VII. Data Analysis Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

analysis of the data and to present the results and

recommendations

.

VII. 1 Analysis

This sub-factor is the measure of the bidder ' s ability

to identify and apply the appropriate analytical

(including statistical) techniques to the data

collected

.

VII. 1.1 Discuss, by exam.ple or reference to prior experience,

your ability to accomplish the various analytical tasks

involved

.

VII. 1.2 For at least some experiments (and/or policy changes),

a significant if not prim.ary objective is econcm;ic. Discuss

your capability in analysing "cost/benefits," including

problems introduced by estimating, allocating, and forecasting costs and benefits, both direct and indirect.

VII. 1.3 For at least some experiments (and/or policy changes),

a significant if not primary objective is to bring about

some change in the policies and procedures of either the

program administrative agency or of segm.ents of the industry.

Discuss your capability in analysing administrative

(behavioral) changes.

VII. 1.4 For at least some experim.ents (and/or policy changes),

a significant if not primary objective is to bring about some

change in the use of technology - change in the rate, application, etc. Discuss your capability an analysing technological change.

71
VII . 2

429

RFP 6-35756

Exploratory (and/or Descriptive) 'Findings

This sub-factor is a measure of the bidder's

capability to identify and present exploratory

findings

.

VII. 2.1 It is anticipated that in many areas of the evaluation, it will be neither required nor feasible to develop

findings which meet the more stringent formal requirements

of a priori proposition testing. To what extent do you

consider it your responsibility to reexamine cases where

there is no requirement, especially v;here it appears that

a priori proposition testing ^ feasible?

VII. 2. 2 You may wish to comment on some of the requirements

as presently described in the several reference documents.

VII. 2. 3 You may v/ish to discuss the various forms of

presentation (i.e., case studies, informal reports,

status reports, systems studies, etc.)

VII. 2. 4 In addition to exploratory (and/or descriptive) findings

with respect to experiments (and their related parameters and

background) , it is proposed that the evaluation system design

is, itself, an exploratory or descriptive "finding." What

differences, if any, will be required in the presentation and

related supporting documentation or justification of the

evaluation system design?

72
^30

RFP C-35756

VII. 3 A Priori Proposition Testing Findings.

This sub-factor is a measure of the bidder's

capability to identify and present a priori

findings.

VII. 3.1 It is anticipated that not only specific exper i.m.ents

(at least in terras of their pror.iinal effects or impacts) but

also other specific questions v/hich m.ay be subject

to study or experiment will require (and it v/ill be feasi

to accomplish) a priori proposition testing evaluation,

what extent do you consider it your responsibility to reexamine cases v/here there is no requiremen t? Cases v/here

there is a requirement, but it is not feasible?

VII. 3. 2 You may wish to comme.nt on so.me of the requirements

as presently described in the several reference docu.me.nts.

VII. 3. 3 You may wish to discuss the various fo 2rms of

presentation

.

VII. 3. 4 Do you think it is desirable and/or feasible to develop

a procedure or process v;hich can be used to make prelim.ina ry

assessments of the likelihood that a particular proposed

experiment v/ill be capable of a priori proposition testing?

75

*-9

ir
431

RFP 6-35756

VII. 4 Recommendations

This sub-factor is a measure of the bidder's

understanding of the relationship betv/een the

results of Phase One and the v;ork in Phase Tv;o

,

in terms of additional, revised, or deleted

hypotheses, changes in objectives or background

information, and the relative feasibility and/or

usefulness of alternative designs for evaluation.

# VII. 4.1 In addition to the specific findings discussed in

VII. 2 and VII. 3, it is anticipated that the v;ork in each

phase will provide a basis for the work in subsequent

phases. It is anticipated that this input to the next'

phase may become available before the completion of all

of the work in the current phase. Comment on the feasibility

of an overlap, the advantages and disadvantages.

VII. 4. 2 It is anticipated that a "final report" will be

prepared at, or after, the completion of each phase v/hich

V7ill describe v;hat the bidder sets out to do, what he did,

including relevant activities of others, and the results

(VII. 2, VII. 3, VII. 4). Comment on the relation of the

final report to the several other forms (and timings) of

outputs

.

7 4
I

.

II.

Ill

IV.

V.

VI.

VII

RFP 6-35756 U32

NUMERICAL V/EIGHTS FOR PROPOSAL EVALUATION FACTORS

Factors

Management

This factor is a measure of the background

and overall management capability of the

bidder

.

Reports and Reviews

This factor is the measure of the bidder's

ability to assure that the Government is

able to effectively and progressively reviev;

a very large scale effort with a minirr.um of

people, and of the bidder ' s ability to

coordinate his activities v;ith ti'icse of

other related individuals and organizations.

Weights

15

Ob j ectives

This factor is a m.easure of the bidder's ove:

all understanding of the objectives of this

procurement (and related programs)

.

20

Background Information

This factor is a m.easure of the bidder's ove:

all understanding of the background and

characteristics of this procurement. 15

Evaluation Design

This factor is a m.easure of the bidder's

capabilities to carry out the evaluarion

design requirements. 15

Data Collection Process

This factor is a m.easure of the bidder's capability to carry out ail of the functions

necessary to the acquisition of data on the

variables and param.eters of interest. 10

Data Analysis Process

This factor is a m.easure of the bidder's capability to carry out all of the functions

necessary to the analysis of the data and to

present the results and recomm.endaticns .

‘ 10

Seven Evaluation Factors Total 100

75
RFP 6-35756

433

SCOPE OF EFFORT

Introduction

A. Purpose

The purpose of this document is to provide a guide to the

bidder in the preparation of supporting data on the scope

of effort ha proposes.

B. Basis for Preoaration of This Docum.ent

This document is intended to be used in conjunction vjith the

associated Proposal Preparation Procedure, Proposal Evaluation

Factors, Schedule, and Statement of Work. The objective is to

assist the bidder in presenting his supporting data on scope

of effort in an orderly and usable form so that it may be

evaluated concurrently v;ith his written (and oral) proposal.

B.l In order to achieve an orderly and usable form, the supporting data on scope of effort must be, at least substantially,

related to the factor (and sub-factor) categories which are the

basis for the evaluation. The detailed outline furnished in

the next . section is for this purpose.

B.2 In addition to the above, it is necessary to assure that

the data furnished by the several bidders be on a reasonably

comparable basis. To accomplish this the detailed guidance

included in this Introductory Section is furnished. It should

be understood that the guidance in this respect is necessarily

arbitrary in the interests of meeting the specific objective.

B.3 The format requirements reflect the fact that evaluation

team members need, and can use, only selected data, primarily

in terras of man/months of effort in certain areas and certain

materials costs. A summary analysis of the scope of effort

against cost data will be accomplished.

77
c. General Comments

43U

RFP 6-35756

To assist the bidder in understanding mtore fully the part

which these data will have in the evaluation./ these general

comments are offered.

C.l This type of program which establishes a broad objective

but provides for planning and design to work out v;ith more

detail and precision the mteans for acccmplish.ment cannot be

costed with the closeness normally experienced on procure.r.ents

of items with a stabilized design. For this reason the

bidders’ respective total cost esti.mates will not necessarily

reflect the actual realized costs to the govern.ment, nor do

they necessarily reflect the relative costs of the bidders

in providing a ccm.parable capability. Therefore, relative

total cost, as such, cannot and v/ill not be given any direct

weight in the evaluation. Cost, in terms of scope of effort,

however, will aid in establishing the kind and extent of efforc,

and this information will serve to further defi.ne the "v/ordpictures" included in the written (and oral) proposals. The

formal evaluation of relative cost and associaced areas is

included as part of o.ne of the major factors in the evaluation

(1.4), and further reference should be directed there.

C.2 T,vhile the major impact of these data in the evaluation v;ili

be in providing additional perspective to the v/ritten (a.nd

oral) proposals, certain corollary information will be used.

C.2,1 A measure of how realistically and thoroughly the costing

has been done will be established. A too high estimate i.ndicates the obvious disadvantages of "gold plating" or a

potentially loose operation. In contrast, however, a too lev;

estimate presents, in the long run, even m.ore serious disadvantages. First, it may indicate a serious misundersta.ndir.g

of the scope and objective of the program. Second, it may

raise a serious question as to the bidder's ability to plan

and estimate costs, thus im>pairing his ability, and uhau of

the government, to review, predict and control costs. Finally,

it may indicate something less than com^plete candor in presentin

the cost part of the proposal.

78
435

RFP 6-35756

C.2.2 The data presented here will be used to supplement the

data furnished v;ith respect to cost as a factor in the overall evaluation.

C.2.3 The data presented here will be used in negotiation with

the successful bidder both as a basis for further refining the

contractual statement of work and in establishing the contract

price

.

79
D. Use of Format

^36

KFP 6-35/5G

D.l It is of critical importance that all persons directly or

indirectly preparing data on scope of effort be thoroughly

familiar with the rules on use of the format. This is

especially true because the figures to be included do not,

in all cases, follow the accounting distinction between direct

and indirect costs. Further, certain types of effort are to

be tabulated as separate ite.ms. An effort has been made to

use format and terminology as close to the general practice

as possible. tNlhiere this has not been donfe , the basis has

been to assume that useful data for the evaluation can be

obtained. Certain distinctions, such as direct and indirect

costs, have little significance for this purpose. Si.milarly,

certain areas of effort can only be evaluated if separated

from proportionately large associated areas of effort.

D.2 The next section specifies the scope of effort data on

a factor by factor basis. To facilitate the evaluation, chese

data should be collated v;ith the respective factors. Because

the procedure for evaluation provides for each faccor to be

evaluated, substantially, without mandatory recourse to other

parts of the proposal, it is critically i.mportanc that the

data on scope of effort and the v/ritten proposal cn a specific

factor, taken together, is ccmpleue and se If -expla.natory . Care

must be exercised to assure that the evaluation team does not

inadvertently penalize the v/ritten proposal because of an

inadequate’ scope of effort.

D.2.1 Under each factor, the scope of effort should be set

out separately for the first phase of the program. V.'here

the bidder proposes to begin or com.plete items in another

phase, note should be made to avoid misunderstanding

.

D.3 It is expected that in m.any instances the scope of effort

proposed will be, literally, the bidder’s present estimace

of the approximate level (or scope) of effort he expects

will be required to accomplish what he proposes to do in

his written (and oral) proposal. A certain lauiuude is

expected, and these data on scope of effort: v;ill be used,

as stated before, primarily to assist the evaluation team

in understanding what the bidder is actually proposing.

There are, however, certain areas which may prevent, a clear

understanding, and some of these are tabulated below.

80
437

RFP 6-35756

D.3.1 The breakdown in the next section is a minimum guide

in tabulating the scope of effort. VThere a particular figure

is considered unusually small or large, or for some other

reason may be misleading, it should be footnoted with an

explanation or further breakdown.

D.3.2 In some cases identification of particular types of

effort with a proposed "associate" or subcontractor v/ill be

of assistance. This will be particularly true where part or

all of an effort is proposed to be accomplished by some

specialist group.

D.3.3 The analysis of comparative cost advantage will not be

accomplished, as a separate factor, by the overall evaluation

team. It is necessary, however, where a bidder is estimating

costs which are, on a relative basis, low that the basis for

this be included in the scope of effort data to assure that

the evaluation team understands that the bidder does intend

to accomplish what he has said in his written proposal.

D.3.4 Conversely, where a possible misunderstanding may occur

due to a lack of appreciation by the evaluating team of the

cost associated with accomplishing a particular part of the

proposal, the bidder should make this clear.

D.3.5 Man/month estimates should be divided according to

distinct classes to avoid confusion or ambiguity in two

respects. First, the bidder's data should distinguish between

direct and indirect. Second, the data should distinguish

between senior professional personnel and supporting technicians,

draftsmen, etc., and between various . management personnel and

clerical assistance, etc.

D.3.6 Any further analysis considered necessary to assure that

the evaluation team properly assesses the relative scope of

effort as related to the written proposal is encouraged. This

may be particularly true in terms of level of training of

personnel, availability of facilities, ability to control design,

and the like.

81
438

RFP 6-35756

E . Summary Data

E.l The relationship of the individual scope of effort on

the several factors to the bidder's overall cost estimate v/ill

be examined. For this purpose, certain sum.mary data v/ill

be required. These summary data should be furnished as a

separate document v/hich should also include a complete reference

set of the scope of effort data sheets.

E.2 Basic summary data v/ill include presentation of sub-total

and total cost estimates in standard format - direct costs,

burden, materials, G&A, fee, etc.

E.2.1 These summ.ary sheets should be accompanied v/ith notes

reconciling the scope of effort data sheets. Any unusual

data should be explained.

E.3 If the proposal includes a substantial proposed subcontracting (or "associate") cost figures, a summary analysis

should be prepared and related to the basic summary data.

E.4 Supporting schedules should be prepared detailing the

hourly wage rate of personnel and this related to the basic

summary data.

E.5 Supporting schedules should be prepared for travel costs

and related to the basic summary data. Schedules should

include breakdown based on distance (or destination) , duration,

number of travelers, and numJDer of trips. Similarly, supporoing

schedules on telephone (and teletype) by nummer of calls, and

average cost per call.

82
RFP 6-35756

Phase One Scope of Effort

Note: The detailed organization parallels the organization

of the Statement of Work, Schedule (s), and Proposal

Evaluation Factors, reference to which should be made

for clarification.

I. Management

(Note that the scope of work here is limited to administrative or program management, and should be distinguished

from effort properly reported under other factors.)

Program Director M/M

Senior Management Advisors M/M

(Include general officers, or other senior

corporate personnel, concerned v/ith administrative management policy, if appropriate)

Other (specify) • M/M

(Include comparable personnel of associates,

subcontractors, consultants, if appropriate;

include program director's personal staff,

if any)

Scheduling M/M

(Include costs of analysis, preparation,

monitoring , etc .

)

Costing M/M

(Include costs of analysis, preparation,

monitoring, etc.)

Telephone $

(Include telephone, teletype costs, etc., and

breakdown by number of calls, etc., if appropriate)

Travel $

(Include breakdown by destination, duration,

breakdown by number of trips, if appropriate)

II. Reports and Reviews

(Note that you should distinguish efforts properly

attributable to other factors.)

Liaison with ETIP M/M

(Include time spent in conferences or visits both

at ETIP and contractor's plant, or elsewhere breakdown by number of trips, duration, number of

travelers .

)

Liaison with P7VA

(Include on same basis.)

Liaison with Others

(Include on same basis.)

M/M

M/M

83
III.

IV.

V.

VI.

UiiQ

Reports. and Services

(Include time spent in direct preparation,

technical v;riting, drafting, etc., and

costs of reproductio.n . Ereakdov/n by type

of report, etc., if appropriate.)

RFP G-3‘j756

Objectives

Review of objectives M/M

(Breakdov;n between docume.nts and interviews,

and among ETIP, PAA, industry, and others.)

t Preparation of description and analysis M/.M

(Breakdown among ETIP, PA_A, i.ndustry, and

others .

)

Preparation of preliminary recomme.ndatio.ns M./M

Preliminary requirements for pilot evaluation ;*/:*

system

Preliminary requirements for prototyce M/M

evaluation system

Travel $

Background

Review of background M/M

(Breakdovm between documents

and interviews, and among experiments,

programs and organizations.)

Preparation of framework m.odel 2-1/M

Travel $

Evaluation Design

Preliminary Design M/M

Detailed Design M/M

Preliminary design of pilot evaluation system^ M/M

Preliminary design of prototype evaluation system M/M

Data Collection Process

(Note that you should distinguish data collections

which may have been included under prior factors)

Design and development of data collection process M/M

Data collection for exploratory evaluations M/M

84
4^1

RFP 6-35756

Design and pilot test of instruments M/M

Data collection for a priori proposition M/M

testing evaluations

Travel $

(Breakdown, as appropriate, between exploratory

and a priori proposition testing evaluations;

provide, as appropriate, basis for estimating

in terms of trips, etc.)

Telephone, postage, etc. $

(Breakdown, as above.)

VII. Data Analysis Process

(Mote that you should distinguish from effort

included with Factor II.)

Design aitd developm.ent of data analysis process M/M

Exploratory analysis and findings M/M

A priori proposition testing analysis and findings M/M

Analysis of evaluation system, and recommendations M/M

Analysis of Phase One, and recommendations M/M

85
4^2

RFP G- 35 756

a Phase Two and Three Scope of Effort

Phase Two

It is not expected that the bidder v/ill be able to provide a

comparable level of detail for Phase T^.-/o . Hcv/ever, a rough

estimate V7ill be useful in assessing those references to

Phase 11//0 in the proposal, and in planning.

The minimum required breakdov/n is to distinguish betv;een the

following

1. Those costs (labor, travel, etc.) directly a function of

data collection a.nd analysis for both basic and special

data processes.

2. All of the remainder (management, design, etc.)

Additional inform.ation , consistent with these purposes,

V70uld be appreciated, but it is not required.

Phase Three

A similar distinction should be observed in providing estimates fo

Phase Three. It is expected, that these may be m.ore speculacive

,

but v/here there are identifiable assumptions v.’hich v;ill largely

determine the scope of effort, discussion v;ould be helpful.

I

1

86

e
443

APPENDIX B

Description of the PEE process

*

1

I
DESIGN AND FRCCUR

OF EVALUATION SY

by

Charles W.N. Thcrpsor.

Presented at the 1976 International Conference on

Grants Management, Charlottesville, VA

W

u)
DESIGN AND PROCUREMENT OF EVALUATION SYSTEMS

4 445

Charles W. N. Thompson, Associate Professor

Technological Institute, Northwestern University

INTRODUCTION

The subject of this paper is the description of a procurement process developed

to meet a specific procurement problem, and which is currently in the process

of being used to procure two evaluation systems by the Experimental Technology

Incentives Program (ETIP) , National Bureau of Standards, Department of Commerce, The two systems (see Figure I which outlines the design process each

will go through in parallel) are to provide an on-going capability to evaluate

the agency impact and commercial impact, respectively, of a series of procurement experiments initiated in the Procurement Policy Area of ETIP and being

carried out in cooperation with, and under the direction of, several governmental procuring agencies. These experiments, which are not the direct subject

of this paper, are designed to test specific improvements in the methods used

by governmental agencies to procure products, with specific emphasis on methods

which will increase the utilization of new and improved technology and achieve

economic savings and better performance. The procurement process to be described here was not part of the set of procurement experiments but has become

an "unintended experiment" itself.

This paper is the first published report (known to this writer) which describes

the process, although it was first developed and used nearly twenty years ago.

In the sections which follow, a brief historical background is furnished, followed by two sections describing the process, a discussion of the salient

characteristics of the process, applications of the process, and related

theory.

The generic reference to the "procurement process" reflects one of the unsolved

problems, that of giving it a descriptive name. The earliest references, when

a name was required, were a choice among "Project Level Systems Management,"

"Proposal Preparation Procedure," or the "Golden Fleece," a reference to the

system to be procured. More recently the name "Parametric Factor Analysis"

was used, but, while technically accurate, it uses terms which have a different, and specialized, technical meaning in a major discipline. Similarly, a number of other potentially descriptive terms (multi-attribute, discrete, independent, ranking, etc.) present problems. Recently a colleague of the writer

referred to the process as "Thompson's Method" for want of a more convenient

handle, but the problem remains.

BACKGROUND

One of the major periods in the development and application of methods for the

procurement of large-scale weapons systems (and, later, space systems) began at

Wright Field during the early 1950's with the introduction of the "weapons systems management concept" as an alternative to the parallel development under

the direction of functional laboratories of the basic aircraft and its component "black boxes." The preference for "project management" over "functional

management" was centered on the need to solve the problems of coordinating the

169
Uks

scheduling of development and achieving technical interface. During this same time there was a growing recognition that the aircraft was, itself, a "black

box" part of a "system" designed to carry out a specific function or mission.

From this point of view, any specific part ("black box") could be considered

part of one or more systems, often overlapping (e.g., weapon system, logistics

system, reconnaissance system, tactical or strategic warfare system, etc.;, and

should be designed (and procured) with consideration of its relation to the

several systems of which it would be part. In the Electronic Reconnaissance

Section, Aerial Reconnaissance Laboratory, the project engineers responsible

for development of airborne equipment to intercept and analyse electronic signals from potentifl enemy radars and other emitters proposed to design and procure systems (called "subsystems") Instead of "black boxes" but were faced with

several problems. First, Che requirements (or specifications) were constantly

changing because of the need to respond Co new developments in radar (and other

emitters). Second, few laboratory personnel had any significant prior ex- perience in designing or procuring systems. And, finally, equipment suppliers

had little relevant experience (except, in some cases, as subcontractors to

systems contractors). In this context, the procurement process to be described

was developed and applied, initially, to two electronic recorj.aissance and one weather reconnaissance subsystems. The original procurement exhibits were subsequently extended and incorporated into a manual for procurement. [Iw]

The application of this procurement process, developed for military hardware

subsystems, to the procurement of an evaluation (software) capability was

initiated about six months ago when the apparent similarity of the situations

(and the problems) was first recognized. First, the evaluation requirements

t?ere not only constantly changing because of the on-going progress in the

initiation and carrying out of procuretcent experiments but also included a vide

variety of objectives ranging from very specific to very broad, and from immediate to long-range. Second, ETI? had, with the exception of the writer,

little or no experience in systems procurement, and neither the time nor personnel to accomplish a problem or concept definition design. Finally, it did

not appear that the "evaluation industry" had any significant present capability' to design or manage an "evaluation system" to meet ETIP requirements.

Based upon this opportune coincidence, the development of the procurement package and the organization of the procurement was accotcpl ished with the close,

and progressive cooperation of personnel of ETIP, the contracting office, the

program administrative agencies, and others. In the next two sections, the

procurement process is described in outline form in two parts. First, the detail development of the entire procurement process, which is incorporated into

the "Proposal Preparation Procedure" [15] and associated documents. Second,

a description of the proposal evaluation process itself, which is incorporated

into the "Proposal Evaluation Procedure." [15]

PROPOSAL PREPARATION PROCEDURE

The initial step, upon recognition that this procurement process is appropriate,

is to review what you "know*' about the procurement. This is a kind of preliminary design or concept or problem analysis, and is in the form of the identification or listing of parametric factors (those conditions or limits which

help to define wliat it is you are going to procure). These should include, for

170
t

«

example

:

a) an overall, if rough, schedule of the time available to let the contract, any early, specific outputs desired, beginning and ending of

phases (if, for example, the program can be broken down into phases),

key milestones, and the expected completion date;

b) a comparable cost estimate, broken down, if possible, into allocations by phase, by major types of activity (e.g., study and design

versus field data collection), or other rough divisions of effort;

c) a preliminary outline description of what you want, preferably in

terms of the kinds of things you believe need to be ‘accomplished

throughout the life of the program (process specification) rather

than specific end products (or services)

;

d) a preliminary identification of reference documents which may be

useful to you (and, eventually, to the bidders) in further defining the program; and, finally, and perhaps one of the most

important areas,

e) Identification of the key personnel who are presently concerned

with the success of the program and who will be actively concerned

in the future (and who will provide the basis for the evaluation

team) , ,

The next step is to refine the above set of parametric factors in the form of

a series of statements describing the characteristics of a prospective bidder/

contractor (in terms of understanding of the problem and/or capability to perform) which you believe critical to the successful carrying out of the program.

This can begin by identifying major areas of concern or problem areas (e.g.,

management, program objectives, major technical problems or skill areas) or by

listing specific critical problem areas (e.g., schedule and cost control, ETIP

objectives, data collection design). In this process reference can be made to

the results of the previous stage, or the progressive drawing upon contributions from individuals who will be members of the evaluation team, from program documents (Including organizational descriptions of the objectives of the

agency or of the specific program), and from other sources, such as prior proposal evaluation procedures. The specific concern is to obtain statements in

the form of critical specific characteristics. It is desirable that these

statements minimize overlap and gap, but this is not a critical requirement.

For convenience of both the bidders and the evaluation team, the number of individual statements (of specific critical problem areas) should range from no less than twenty to no more than seventy. These individual statements (referred to hereafter as sub-factors) should then be grouped into (approximately

five to fifteen) major areas of concern (hereafter referred to as factors)

which reflect related problems (and related interests and skills on the part

of evaluation team members). Because these factors (and sub-factors) will

provide the central framework for the preparation of proposals and for the

evaluation team, it is important that the focus be clear; for this purpose,

additional, specific, Illustrative questions can be added to each sub-factor.

171
UUP^

An illustration of the results of this process is provided in the three tables

at the end of this paper.

Using this framework of factors (and sub- factors) , the "Statement of Work" [15]

and "Schedule" [15] are prepared; this serves as a double-check on wiiether all

the significant parts of the program are covered. The final key document is

called "Scope of Effort" [15], and, in parallel with the above, it provides a framework for the bidder to present his estimate of how he will allocate man- power and dollars to the various parts of the program. If the program is to

be divided into time phases, this may be reflected, as required, in the organization of the above documents. To these may be added copies of program lescriptions, descriptive or amplifying notes, and any other documents which

might be helpful to the bidders.

A summary description of the above, and detailed guidelines for the preparation

of proposals, is incorporated in the "Proposal Preparation Procedure" [15],

which, together with a description of the sub-factors and factors incorporated

in a separate document called the "Proposal Evaluation Factors" [15], and the

other documents noted above, makes up the procurement package which is appended to the formal part of the Request for Proposal. Where, as in the present

case, the procurement process represents a significant departure from prior

practice, it is considered desirable to hold two bidders' briefings, the first

to allow prospective bidders to decide whether or not to bid and to prepare

questions which can be answered at the second. For the same reason, and for

the additional purpose of increasing openness, communication, and confidence,

the "Proposal Preparation Procedure" [15] also describes briefly how and why

the process was developed (where we are coming from) and how the proposal

evaluation will be carried out (where we are going).

PROPOSAL EVALUATION PROCEDURE

The broad outlines of the procedure to be used for proposal evaluation are included in the "Proposal Preparation Procedure" [15] described above and which

is provided to the bidders. In addition, members of the evaluation team re- ceive a more detailed set of guidelines incorporated in the "Proposal Evaluation Procedure" [15], (Note: At the time of this writing, the second bidders'

briefing ha? been completed but proposals have not been received; the description which follows is, therefore, based upon the preparation for evaluation on this procurement, and in the perspective of prior experience.) The members of

the evaluation team are drawn, as much as possible, from the set of individuals

who will be concerned directly with^ the successful carrying out of the program

which results from the award; where particular background or skills are necessary to the evaluation of a particular factor, others may be included. In all

cases, membership is limited to those who accept responsibility for evaluating

at least one sub-factor. The steps of the process include the following:

a) individual preparation, in which members of the team become familiar

with the entire RFP , receive copies of all of the key documents (and

have access to all of the reference documents) , and are invited to

participate in the briefings of bidders;

172
449

b) prelirainary review, by individual members, of the proposals after

receipt, and preparation of preliminary ratings (on at least specific, assigned sub-factors) from "Nothing can stop him..." to

"No hope (totally unacceptable)" which will become part of the

record

;

c) a aeries of sequential (factor team) meetings at wiiich individuals

responsible for the set of sub-factors within each factor agree on

a single, aggregate rating of each bidder for each factor;

d) the preparation of a sunnnary of all of the factor ratings, together with a summary description, for final review, negotiation

with the successful bidder, and award.

There are few restrictions on the judgment process of the individual team member; he may use prior information (but must disclose it to other team members)

;

he must read the parts of the proposals directed to the sub-factors to which

he is assigned, but he may read (and use) any or all other parts; he will know

the identity of the bidders and, for that matter, will have access to any information any other member of the team has; he is required to put the results

of his individual review in the record, but he may change his mind during the

review by the factor team; and he may attend meetings of factor teams of which

he is not a part.

While individuals are encouraged to do so, the factor team must rate each subfactor, and the overall factor, by considering the bidder's comprehension or understanding of the problem presented, approach described or outlined, and

present or potential capability to perform. They may cross-reference other

parts of the proposal, and may draw upon the "Scope of Effort" [15] information as a check on what the bidder expects to do. There is no formal voting

procedure, and no prescribed comparative weighting of the sub-factors; they

may inquire into the basis for the position of a specialist in his area, they

may record strong individual disagreements, and they are encouraged to add

specific comments which would be useful in negotiation. Each factor team must

continue until it has reached an agreement, which is then recorded on the same

form, and with the same scale, that the members used in their individual re- view, Because these guidelines are critical to the factor team process, they

appear not only in written form but are also reviewed (or even read aloud) at

the beginning of each factor team meeting.

Ope of the few significant changes from the procedure as used previously is

the substitution of written questions for an oral presentation. In the original procedure, each bidder was invited to make a brief oral presentation and

to answer specific questions (both of which were recorded) from the evaluation

team in order to assure that the team correctly understood what the bidder proposed and had sufficient information upon which to base a confident rating

(whether in favor of or against the bidder). Because of the uncertainty of the

number of bidders in the present procurement and insufficient time (if there

were a large number) , the procedure was changed to provide that members of the

team could direct specific, written questions (through the Contracting Officer)

to an individual bidder if necessary to clarify an ambiguity or to otherwise

173
^50

make a confident rating possible.

COMMENTS ON TliE PROCESS

The process described in outline form above has a number of features wfiich,

taken separately, probably are not novel or unique. The process as a whole,

however, is considered a significant departure from present methods of procuring program evaluations (and evaluation systems or capabilities), and, at the

time of its initial use for military hardware subsystems procurement, was a significant departure from even the then developing weapons systems procurement methods. In the paragraphs which follow some of the salient characteristics of the process will be identified or described, noting whether they are required or desired characteristics, w’hat the alternatives are, and wh.y this

approach was used, as appropriate. Overall characteristics will be discussed

first, followed by characteristics of specific parts of the process.

The first overall characteristic is that the process is designed to meet a particular kind (or class) of procurement problem(s) , in terms of the procuring

agency, the set of prospective suppliers (bidders/contractors), and/cr the kind

of thing to be procured. No claim is made that the process is applicable

everywhere (e.g., advertised procurements), and this characteristic will be

discussed further in the next section of this paper on "Applications."

The second overall characteristic is that the process is essentially pragmatic.

It proposes that uncertainty be faced and not avoided, that where we know a lot

about V7hat we want that we use that knowledge, that where ve are dot sure or do

not know that we disclose the state of our knowledge and try to develop the

"best" description (which is likely to be a process rather than end product

description) we can. And it proposes that we disclose this to prospective bidders, so that they can concentrate on providing us the "best" information

available for the next step in the process--the selection of the contractor who

will work with us.

The third general characteristic is that it is an open, explicit, cooperative

process, recognizing the crucial importance of improving communication (both

the content of the information shared and confidence in that information) among

the several parties. Prospective bidders are provided, as much as possible,

with all of the information available (with the narrow exception of that barred

by law or regulation, and, of course, the evaluation team deliberations while

they are going on) , and both the bid set and the selection of reference documents are designed to facilitate their ability to prepare proposals (and to

have confidence in the process). Similarly, the evaluation team (which, where

possible, includes representatives of other agencies, such as "using agencies,"

with a significant interest in the success of the procurement, and others who

may be interested in the process) has access to all of the information available, and each member is assured that his part in the process is preserved in

the record and in the selection of the winning bidder.

The most salient specific characteristic of the process is the method of identifying and describing the factors and sub-factors wiiich provide the basis for

preparing proposals and for evaluating proposals. The process neither assumes

174
« 451

nor requires that any formal structure or framework exists for the object of

the procurement; to the extent there is structure, it can be used, but the

process is not dependent upon a complete and balanced description, or a set of

Inclusive and exclusive parts. Instead, the process treats each sub-factor as

a separate "probe,” obtaining a measure along a dimension (or viewpoint) of the

whole program (or whole capability of the bidder), a measure which may, to some degree, overlap or intersect some other measure (sub-factor) , and, when combined with all of the other measures, is intended to provide a set of measures

which describe the program (or the capability of the bidder). The several advantages of this include the following:

a) it is not required that all parts of the system be described or specified to the same degree of detail, each part can be described

to the extent its description is known, parts which depend upon the

prior definition of some other part can be defined as such, and

even inconsistent parts (e.g., statements of objectives prepared at

different times for different purposes by different people) can be

included

;

b) it is not required that there be a major preoccupation with assuring

that each factor (or sub-factor) be completely independent (and nonoverlapping) with respect to others, allowing a concentration on

••what it measures” rather than trying to avoid duplication;

c) it is possible to start any%^ere, to jump around, to repeat and re- work individual sub-factors (and factors), a process which "builds

up" to the whole, and recognizes that the set may not fit neatly

into any single, explicit, formal, external framework.

With considerable oversimplification, the above approach taay be contrasted with

several other approaches. In the procurement of research (and, in some cases,

of professional services) , there may be a comparably high degree of uncertainty

on the part of the procuring agency (and even the prospective supplier), but it

may be reasonable to limit the evaluation to a "single" or small set of technical problems, together with an evaluation of the capability of the individual (a) who will work on the project; similarly, but at the other end of the

uncertainty spectrum, a production procurement to detailed specifications may

reasonably be evaluated on the "single" question of price, together with an evaluation of the facilities needed to perform. Some, if not many, of the

proposal evaluation boilerplates appear to follow this binary breakdown between

a single (or small set of) selected major characteristic (s) and a "lumped"

assessment of capability (of personnel and/or the organization).. The forced

selection of a (usually) small number of technical problems may result in an

overemphasis on some problems (and ^^ich may Include problems which cannot be

adequately described or "solved" within the bidding period) and the exclusion

of others, perhaps equally critical, which the bidder may feel compelled to

respond to because of prior knowledge or from "hints" he receives from other

descriptive materials, e.g., program plans provided as part of the bid set.

Coupled with this is the difficulty of assigning a single, overall measure to

the "qualifications" of the bidder, either as an Independent assessment or one which parallels the technical part. For at least some of these "middle-ground"

175
452

procurements, the additional flexibility and freedom to define the evaluation

in terms of whatever factors can be identified would appear to be desirable.

The next specific characteristic, and related to the above, is the use of a single framework, the set of "Proposal Evaluation Factors" [15], for the preparation of proposals and for their evaluation. 3y organizing the "Statement of

Work," the "Schedule," and the "Scope of Effort" [15] on the same basis, and

making clear that all other descriptive materials are relevant only to the ex- tent they contribute to the central framework, the bidders and the evaluation

team have a common frame of reference. The bidder can concentrate his attention on providing information which will describe his capability (understanding of the problem, present or proposed approach, and present or potential

capability to perform) with respect to a specific set of sub-factors; the

evaluation team can, likewise, concentrate its attention on the same set, and

with a minimum risk of overlooking the relevant part of the proposal. They

share, to a large degree, a conmon set of information and know the rules.

A related specific characteristic is the elimination of the necessity for a limitation on length. The evaluation team member can quickly find t.ne information relevant to a specific sub-factor, and the bidder can be sure that what

he is presenting will be found; each can, if necessary, develop or quickly

identify necessary cross-references to other parts of the proposal. The ne- cessity (or desirability) for over-long, "shotgun" approaches to cover any

conceivable area which may be of interest to some individual member of the

evaluation team, or to cover ambiguities in the amount -of detail required, is

diminished; in contrast, there is an obvious premium on clarity and conciseness

in zeroing in on the specific sub-factor.

The next specific characteristic is the makeup of the evaluation team. Instead

of centering the responsibility (and burden) for the entire evaluation on the

project officer (together with anyone he can trap into helping him), the team

is formed by calling upon a wider set from among those who have a prior (and

continuing) interest in the subject of the procurement or who have special

competence to evaluate particular sub-factors (and factors). Because of the

framework used, participation in the evaluation process can be limited, by the

individual, to reading parts of the proposal, evaluating those parts, and participating in only those factor team meetings concerned with those parts. This

functional breakdown substantially limits the time commitment, and it allows

concentration on parts which are of specific interest (and/or within the specific competence) of the individual, A larger evaluation team provides an opportunity for those interested in parts of the procurement (and who may be

called upon to work with the successful contractor in the future) to provide

inputs and to contribute their expertise and perspective; and, similarly, specialists in particular areas can increase the likelihood of an effective evaluation. In a systems sense, these team members provide an interface with the

several systems they represent.

The next specific character

information the evaluation

a "blind" evaluation, i.e.,

nical part." While the rat

istic is the remova

team members can us not knowing the bi

ing of a "brilliant

1 of artificial

e. There is no dder's identity,

technical idea"

restrictions on requirement for

of the "techin the abstract

176
$ H53

«

may be useful In a research procurement, the object here is to rate the specific sub-factor in the light of all the relevant evidence, and this requires

access to the whole proposal, if desired. Similarly, it is considered only

realistic to recognize that individual members of the evaluation team bring a wide variety of prior experience (and information) not only on areas relevant

to the several sub-factors but also, in some cases, with respect to the capabilities of specific bidders. While this information may either add to or aubtract from the rating of the specific bidder, there is no advantage to pretending it will be not considered, and its weight is more properly assessed,

and accounted for, by the evaluation team through a requirement that it be

openly disclosed.

The next characteristic is a corollary of the above. The members of the evaluation team are instructed that they must rate each bidder affirmatively,

t.e, , they must have a basis for low ratings as well as high ratings. Where

there is ambiguity or an unclear or missing response, the evaluation team cannot assign a "zero" but must obtain the required information by asking a specific question. It is recognized that in some cases of an extremely (and uniformly) weak bidder, the team may properly decide that the proposal is a clear

and unambiguous representation of the capability of the bidder.

The final specific characteristic to be noted is the process for "aggregating"

the individual ratings of each team member with the ratings of the other team

members, and. In turn, "aggregating" sub-factor ratings into factor ratings,

and, in turn, into an -overall rating. The process is sequential, with a sum- mary hard-copy record retained of each stage to provide a traceable record of

the process and as additional assurance that the contributions of the individual team members are reflected in the end result. The basis for aggregation of

Individual team member ratings and of the sub-factor ratings is the factor

team; using any process acceptable to them, the members of the factor team determine the relative rating of all of the bidders against a fixed scale (the

same one each used for individual ratings on individual oub-factors) . Formal

voting is precluded; somewhat analogous to a petit Jury or an arbitration

team, the factor team is required to interchange information and explore their

several positions until they arrive at a mutually acceptable result. Provisions are made to append clarifying or qualifying notes, and, particularly, to

identify areas which will require negotiation (or further clarification) in the

event a particular bidder is chosen. In prior experience with nearly forty

factor teams there were no instances in which this was not accomplished. The

final step of aggregation, the combining of the ratings across all of the separate factors, is the first one in which formal weights are introduced, and

these, characteristically, are limited to a range of no more than two to one

(and this is disclosed to the bidders). This step is largely mechanical, and

the summary report will not only Include a description of the strong and weak

points of the several bidders but also be accompanied with the complete set of

factor ratings. While the summary weighted score nay serve to identify the

"winner" and other acceptable bidders, the individual factor ratings direct attention to any relative weak points. Similarly, the summary score and factor

ratings will identify "in-between" and weak bidders, and specific areas of

unacceptabtllty

.

177
If the process can be captured In terms of simple underlying concepts, there

are at least three which would appear to be central. First, where uncertainty

(of various kinds) makes unlikely (or difficult) the identification of wriat,

in research terras, might be referred to as the dominant independent variable

(or ’’summing independent variable"), the set of parameters (plausible rival

hypotheses) are used as a surrogate independent variable. In contrast to the

more familiar (research) approach in a priori proposition testing, wnere parameters are, at best, "unwanted" variables, to be controlled or otherwise

isolated from the central problem of interest, in this process we "use" the

parameters tc increase the likelihood that we shall achieve our (administrative) purpose of obtaining some desired value of the dependent variable.

Second, the process is structured sequentially to provide for a series of

decompositions and compositions in several dimensions, including a degree of

"value free" partitioning. Third, the process is explicit, open, relatively

fully defined and self-checking, including the sense of "rule of law."

APPLICATION’S OF THE PROCESS

As has been noted, this procurement process v;as developed to meet a specific

procurement problem and may be neither needed for nor applicable to otner

specific procurement problems. It would appear to be inapplicable to procurements in which there is a single (or small number of) critical basis (es) for

evaluation, e.g., procurements for vrell-def ined products or seir.’ices where the

basis is price, and research procurements where the basis is a particular

technical approach or capability. More generally, there would appear to be no need for this procurement process where the prior experience of the procuring

agency (and the set of prospective suppliers) has provided a basis for suc- cessful procurement and especially where there are comparable previous procurements. There are probably a number of other procurement situations in

which alternative methods are being used, including the following;

a) where the procuring agency has the time and resources to carry out

the preliminary’ design phase (or even further stages) and prepare

detailed specifications;

b) where the procuring agency has the money (and time) to let parallel

(competitive) contracts for the preliminary design (or concept)

phase

c) where the procuring agency has the time (and money) to let a "sole

source" contract for the preliminary design phase, with the flexibility to drop the program or to start over if the first effort

was unsuccessful;

d) where one or more prospective suppliers are willing and able to

carry out the preliminary design phase (or even further phases)

prior to action by the procuring agency;

e) where the item to be procured can be "decomposed" from technical

and scheduling interfaces through the use of standards, and the

like, or where there are acceptable alternates available.

178
1)55

#

With these exceptions noted, it should not be assumed that this procurement

process is limited to large and complex systems, whether hardware or software.

The basic criteria for use of this process include the following:

a) that there is a degree of uncertainty or complexity in the item to

be procured which makes it difficult to describe the item to bidders

and to identify a suitable basis for evaluating bidders* proposals,

using available procurement methods;

b) that there are limits on tine, money, or personnel which preclude

considering other alternatives; and

c) that there is some basis in prior experience for believing that

it is critical to both the procuring agency and the (successful)

bidder to develop a reasonably sound initial mutual understanding

in order to ensure a progressive and successful mutual accorrplishment of the object of the procurement.

A maaber of possible applications might be suggested, and the categories which

follow, and the illustrations, represent only one way in which they might be

organized. First, the case of multiple candidates (bidders or projects) which

must be compared against an uncertain or ccxnplex set of requirements, including:

a) con?»etitlve procurement based on a complex set of design requirements

or of a system, e.g., the present case;

b) rank ordering of a series of competing projects (which may not be

directly comparable) against the twin standards of likelihood of

success and probable payoff (future value)', i.e., the ’’project se- lection" process;

c) comparison of alternative techniques, solutions, or (assumed) resources

end constraints against a set of "independent" and/or differentially

affected requirements, e.g., cost-benefit analysis.

Second, the case of a single candidate (bidder or project) which must be compared against a complex or uncertain set of requirements, including:

a) a project or proposal submitted for "acceptance" under a complex

statutory requirement, e.g., application for qualification as a health maintenance organization (HMO) , submission of an "impact

statement" for a power plant or other facility;

b) comparison of a proposed project against a set of "independent" re- quirements, e.g., cost-benefit analysis.

Third, Che case where one (or more) candidates are to be compared against one

(or more) uncertain or complex requirements by a set of raters v»ho may differ

not only in the kind of information they will accept but also in their value

structures; in at least some cases where group decision processes are used

179
\

V 456

(e.g., arbitration, negotiat

cal processes) it may be des

are "value- free , i.e., fact

narrow concentration on thos

differences

.

ion, the Delphi

irable to initi

s or agreed sta

e remaining are

Method, and, perhaps,

ally separate out those

tcments of facts, to al

as in which there are v

some are

low

alue

pol it i- as wr ich

a more

-based

It is important Co make clear that these comments on other applications are,

at this point, to a degree speculative, and are offered to suggest possible

areas for further consideration.

THEORETICAL CONSIDERATIONS

The main purpose of this paper has been to present

procurement process, and the discussion here will b

dication of related theoretical areas. The origins

of the process was largely etapirical, and the exten

entered into the process can be traced through refe

written just prior to that event and which was used

the process vras based. A more generally available,

presented, subsequently, at a technical conference.

a description of a specific

e limited to a brief in1 development and testing

c to which prior theor-/

rence to a paper [5] as the model upon which

and briefer, version, was

[ 10 ]

do this for seme of the

Any reasonably thorough examination of the academic and philistine literature

would uncover a virtually unlimited variety of articles related to an area as complex and pervasive as this one. No attempt has been made to

purposes of this paper, but an attempt will be uiade to identify

major areas and to include some representative references which may provide a useful frame of reference. First, and perhaps most obvious, is the extensive

area of procurement, including federal and armed services procurement regulations, their statutory' bases, amplifying and clarifying administrative and

court decisions, and a wide variety of related agency policies and procedures.

Complementing this government documentation are articles in trade magazines,

books, conference proceedings, scientific journals, and law reviews commenting

on present practice and/or suggesting alternatives and improvements . The

importance of procurement to industry has generated a market-oriented "how to"

literature which may be the most accurate assessment of present procurement

practice, and there is a related literature on project and systems management.

Less directly related to procurement but concerned more directly with the

technical and related problems in the design and management of systems, and

comparably complex problems, is the relatively diffuse literature on systems

engineering, systems management, systems analysis, cost-benefit analysis,

urban planning, and specific kinds of systems, e.g., transportation, health

care, environmental.

In the scientific literature there is an early and extensive record of work in

small group theory which is, perhaps, as good a place to start as any other

comparably specific area in the wide variety of research in the management,

management science, organization theory, behavioral science, etc., literature.

Simon's early work [5] is representative of a small and select group of

authors who described the administrative decision process in terms which were

clear and understandable. Several different "schools" have developed to pursue the problems of decision making, some concentrating on empirically-based

180
^57

behavioral models and others on more formal analytical models, with or without

empirical testing. In this latter group, and of central importance to the

present subject, are a number of researchers variously described as decision

or utility theorists. With some risk of simplification, reference will be

made to a useful, and brief collection of readings [3] which includes an article more or less directly applicable, entitled "On Subjectively Optimum Selections among Hulti-attribute Altemat ives , " [4 ] It should be made clear that

the exploration of the theoretical implications of the proposed process, and

the examination and further testing of the specific mechanisms used, remains

to be done.

One other, and extensive, literature is relevant to the present paper, and

that is the subject of program evaluation. Two basic references are

Sucteian [6] and Caro [2]; and two useful and relevant current discussions

appear in Federal Evaluation Policy [16] and Academic and Entrepreneurial Re - search . [1*]

The subject area has been a toatter of interest to the present writer, starting

with the previously mentioned papers. [8] [10] The general area of systems is

represented in two papers [12] [13]; the problems of cooperative decision

making are discussed in two papers [7] [9]; and the subject of evaluation is

described in another. [11] These do not provide the comprehensive basis that

a more thorough review of the literature would achieve, but they do provide an

additional perspective on the considerations which were instrumental in the

present application.

SUMMARY

A brief description has been presented of a procurement process developed and

applied to meet a specific procurement problem. To provide a modest perspective, additional comments have been made on background, the salient characteristics and applications of the process, and relevant theory.

While the process has been previously "tested," the current application is in

mid'process, and the "test" remains to be completed. From past experience,

the process should be successful in rating bidders from strong to weak, and

in choosing a "winner" whose performance after award will reasonably meet our

expectations.

181
REFERENCES

^53

[1] Bernstein, Ilene N. , and Freetnan, Howard E. , Acad err.ic and Entreoreneurlal Research . New York: Russell Sage Foundation, 1^75.

[2] Caro, F. G. (ed.) , Read ings in Evaluation Research . New York: Russell

Sage Foundation, 1971.

[3] Edwards, Ward, and Tversky, Amos (eds.). Decision Making . Baltimore,

Md. : Penguin Books, 1967.

[4] Shepard, R. N. , "On Subjectively Optixuro Selections among Multi-attribute Alternatives." In Decision Making , Edwards, Ward and Tversky, Amos

(eds.) Baltimore, Md. : Penguin Books, 1967, pp. 257-233.

[5] Simon, Herbert A., Administrat ive Behavior . New York: The Free Press,

second edition, 1965.

[6] Suchman, E. A., Evaluation Research . New York: Russell Sage Foundation,

1967.

[7] Thompson, Charles W, N. , "Administrative Experiments: The Experience of

Fifty-eight Engineers and Engineering Managers," IEEE Transactions on Engineering Management , Vol, EM-21, No. 2, M-ay 1974, pp . 42-50.

[8] Thampson, Charles W. N. , "Design for the Decision-Point: A decisioncentered conceptual approach to the procedural problems of structure in

the theory of organization." Master's Thesis, Ohio State University,

June 1956.

[9] Thompson, Charles W. N., "Implementation," Defense Transportation

Journal, Vol. 30, No. 1, January-February 1974, pp. 30-31.

[10] Thompson, Charles W. N. , "Model of Human Performance Reliability in

Health Care Systems," Annals of Assurance Sc iences . Vol. 7, No. 2, 1974,

pp. 335-339.

[11] Thompson, Charles W. N. and Rath, Gustave J., "The Administrative Experiment: A Special Case of Field Testing or Evaluation," Human Factors . Vol. 16, No. 3, 1974, pp. 238-252.

[12] Thompson, Charles W. N. and Rath, Gustave J., ’'Making Your Health Systems Work: A System Analysis Approach." Proceedings of the Annual Meet - ing of the American .Academy of Pediatrics , Chicago, Illinois, 20-24

October 1973.

[13] Thompson, Charles W. N. and Rath, Gustave J., "Planning Solutions Aided

by Management and Systems Technology," The Annals of the .American

Academy of Political and Soc ial Science . Vol. 405, January 1973,

pp, 151-162.

182
A

^59

[14] U. S. Air Force, ''Management - Proposal," section M-1-14.0, and "Technical Proposal," section S-15-14, AFSCM 310-1/AFLCM 310-1, 15 March 1964.

[15] U, S. Department of Conmerce, Experimental Technology Incentives Program, "Proposal Preparation Procedure," February 1976. (Note: This is

part of RFP 6-35756, as are other related documents identified in the

text; copies of these parts of the bid set may be obtained from ETIP.)

[16] Wholey, Joseph S., Scanlon, John W., Duffy, Hugh G. , Fukumoto, James S,

and Vogt, Leona M. , Federal Evaluation Policy . Washington, D.C.: The

Urban Institute, 1975.

183
460

TABLE I PROPOSAL EVALUATION FACTORS

I. Management

This factor is a measure of the background and overall management capability of Che bidder.

II. Reports and Reviews

This factor is the measure of the bidder's ability to assure chat the

Government is able to effectively and progressively review a very large

scale effort with a minimum of people, and of the bidder's ability to

coordinate his activities with those of other related individuals and

organizations.

III. Objectives

This factor is a measure of the bidder's overall understanding of the

objectives of this procurem.ent (and related programs).

IV. Background Information

This factor is a measure of the bidder's overall understanding of the

background and characteristics of this procurement.

V. Evaluation Design

This factor is a measure of the bidder's capabilities to carry out the

evaluation design requirements.

VI. Data Collection Process

This factor is a measure of the bidder's capability to carry out all o the funct ions -necessary to the acquisition of data on the variables and

parameters of interest.

VII, Data Analysis Process

This factor is a measure of the bidder's capability to carry out all of

the functions necessary to the analysis of the data and to present the

results and recommendations.

184
4

461

TABLE II FACTOR WITH SUB-FACTORS

V. Evaluation Design

This factor is a measure of the bidder's capabilities to carry out the

evaluation design requirements.

V.l Exploratory (Descriptive) and A Priori Proposition Testing Hypotheses,

Variables and Parameters

This sub-factor is a measure of the bidder's ability to define the detailed evaluation objectives.

V.2 Experimental and/or Study Designs

This sub-factor is a measure of the bidder's ability to define the

overall experimental and/or study design.

V.3 Data Collection Design

This sub-factor is a measure of the bidder's ability to design the

overall data collection process, including sources of information,

sampling strategies, timing, etc.

V.4 Data Analysis Design

This sub-factor is a measure of the bidder's ability to design the

overall data analysis process, Including processing and analysis, and

presenting results and recomnendat ions

.

V,5 Phase Two Evaluation Design

This sub-factor is a measure of the bidder's ability to redefine the

detailed experimental and/or study design to reflect the results of

Phase One, the several objectives, and model of the process in a comprehensive detailed design to meet both basic data (base line data) and

special data requirements,

V.6 Phase Three Evaluation Design

This sub-factor is a measure of the bidder's ability to further redefine

the above, to provide for a detailed design, and necessary supporting

materials, which would allow an on-going evaluation process.

185
462

TABLE III SUB-FACTOR WITH ILLUSTRATIVE QUESTIONS

V.2 Experimental and/or Study Designs

This sub-factor is a measure of the bidder's ability to define the overall experimental and/or study design.

V.2.1 Discuss the problera(s) of and approaches to experimental and/or study

design. Alternatively, you coay wish to demonstrate your capability by reference to the relevant education and experience of your staff. It is sug 2 ested

that you may wish to distinguish among the several kinds' of designs which may

be appropriate for meeting the several kinds of objectives which are the outcome of the process discussed in V.l.

V.2. 2 Discuss the problem(s) of and approaches to identifying and controlling

for the potential effects of parameters.

V.2. 3 It is proposed that the bidder can design an overall evaluation system

and,»in parallel, carry out evaluations of specific selected experiments during

Phase One. Discuss the advantages and disadvantages of this. Can you identify

specific experiments which can be evaluated during Phase One? Can you identify

base-line data which can, or should, be collected during Phase One? which can, or should, be collected during later phases?

V.2.4 To what extent do you see differences among the several PAA which v/ill

affect the design of the evaluation system? Will differences in the progress

in initiating experiments among the several PAA be an advantage or disadvantage ?

V.2, 5 It is anticipated that some part or all of the bidder's activity in

this area will be dependent upon the activities, of another ("agency impact")

contractor. How critical is this from a time point of view? from an interface

point of view? Will segmentation (by industry, technology, market) provide

advantages or disadvantages? to what extent will longitudinal designs be important, and will this be an advantage or disadvantage (COMMERCIAL IMPACT ONLY)

186
H63 IIIIIII%

i|iii u

es

a.

C/) -m

cn

a

yj e o s:Xu

1^o

hW

CL

i/i zO

laU <

C/J <

>• 2l CO «so ^ ccZ

187
APPENDIX C

Instructions Given to Evaluation Panel ?>!err.be
465

RFP 6-35756

PROPOSAL EVALUATION PROCEDURE

GENERAL

Information and guidance on proposal evaluation is contained in

the Proposal Preparation Procedure and Scope of Effort documents.

Covered below are additional suggestions to assure a uniform,

thorough and efficient evaluation. Comments on how to improve

the procedures are welcomed and should be furnished to the

COTR for early consid-eration and dissemination.

INDIVIDUAL PREPARATION

All proposal evaluation team members should become familiar

with the entire RFP, at least sufficiently to know what

documents are contained in it. All team members should have

copies of the Statement of Work, Schedule (s). Proposal

Preparation Procedure, Proposal Evaluation Factors, and Scope

of Effort. In addition, team members should have blank copies

of the Factor (and Sub-factor) V7orksheets . During the entire

evaluation period, you should make notes on important points

or questions you have, and be prepared to progressively rate

each bidder on each sub-factor you have comments on. These

notes will aid you in the factor team discussions when the

relative positions of the bidders are established.

If you have any question at any time concerning the process,

you should contact the COTR.

• PROPOSALS

All copies of the bidders' proposals will be received by the

Contracting Officer on April 15. The Contracting Officer

will retain the contractual provisions, including price data,

and one or more copies of the rest of the proposal. Fifteen

copies of each bidder's proposal, which includes the information responsive to the Proposal Evaluation Factors and Scope

of Effort documents, will be provided to the COTR. The COTR

will provide copies to the members of the evaluation team,

together with any required additional guidance or forms. A

sign-out procedure will be used to keep track of the proposals.

Access will be limited to members of the evaluation team, and

to personnel of DOC whose official responsibilities require

access

.

PRELIMINARY REVIEW

It is proposed that all team members will spend Friday,

April 16, at ETIP to accomplish a preliminary review. Proposals

will be distributed in the morning, and members will spend

most of the day reviewing the proposals both overall and with

Page No . 1
^66

RFP 6-35756

respect to specific factors and sub-factors. The purpose of

this is the follov;ing: a) to estimate the overall magnitude

of time required for reviev; as a basis for planning and

scheduling the reviev; process; b) to identify any critical

problems faced by evaluation team members; c) to identify

any critical specific questions v/hich must be directed to a

specific bidder (through the C.O.); d) to determine if one

or more factor teams will be ready to meet by .'-'onday .morning;

e) to establish which of the tv;o procurements will be evaluat

first; and f) to review the assignments of team me.mbers to

specific sub-factors.

By the end of the day (Friday) , it is proposed to schedule on

or more factor team meetings on ‘lo.nday; a.nd , possibly te.ntati

schedule subsequent factor team meetings.

QUESTIONS r Because it is not anticipated t.hat any cf the individual bidd

will be invited for oral presentations and questioning (prior

to the completion of the factor tea.m m.eemings; , a.ny additiona

information from the bidders can be obtained only by directi.n

specific questions to the specific bidder through the Contrac

Officer. These questions should be identified by sub-factor,

and limited to the clarification of ambiguities or to

clearing up omissions (on apparent strong or v;eak points, v;hi

would be expected to substantially affect the team's ability

to rate the sub-factor. ^ im.por tant that these be ident

i

f ied as early as possible to allcw time to obtain a.nswers ; waiting fo r an answer m.ay delay the scheduling of the af f ecte

factor team m.eetina. - - - I , , , Specific questions should be entered on the Question Form, and

given to the COTR. Because of the time delay involved, each

question should be reviewed to be sure that the additional

information is absolutely necessary.

If additional information is required from- someone other than

the bidder (e.g., a COTR on another program), put that quest!

on a Question Form, and give to the COTR.

Note: No questions should be directed to anyone outside of

the evaluation team, except by the COTR through the CO.

INDIVIDUAL REVIEW

The process of individual review is largely within the discre

of the evaluation team member. There are only two specific

checkpoints: 1) that you identify specific questions (see

previous heading) as early as possible, and 2) that you have

Page No . 2
467

RFP 6-35756

completed your review of the relevant (and/or assigned) part

of the bidders' proposals prior to each factor team meeting

(which will be scheduled with this requirement in mind)

.

All evaluation team members are encouraged to review all

parts of all proposals, and may attend all factor team

meetings

.

It is recommended that all members of a specific factor team

review all of the sub-factors which make up the factor.

Each member of a factor team must review the sub-factors to

which he is assigned. For this purpose (and, optionally,

for the others), a Factor (and Sub-factor) Worksheet must be

completed for each assigned sub-factor. You may use duplicate

copies of the Worksheet, or any convenient form. The purpose

is to be sure that you have established clearly what the

bidder proposes, and that you have a quick reference to

your rating and supporting reasons for them for use during

the factor team meeting.

FACTOR TEAxM MEETINGS

Members of the proposal evaluation team who have been assigned

to one or more sub-factors within a specific factor make up

the "factor team" for that factor.

When the members of the team are ready, a factor team meeting

will be scheduled. These factor team meetings will be

scheduled separately for each factor and will last about

an hour.

The procedure of the factor team meetings will include the

following steps:

First, an informal discussion and comparing of notes to

establish that we are using the same standards and information

(to uncover major discrepancies or omissions)

.

Second, comparing and/or discussing individual reviews, and

rough ranking. •

Third, establishing relative differences or distribution.

The following ground rules should be observed in the conduct

of the factor team meetings:

Information to be considered:

1st - RFP, and associated documents

2nd - Written proposals, including "Scope of Work"

Page No. 3
463

RFP 6-3576

3rd - Responses to specific questions, if any

4th - Individual experience of the factor team members - if there is critical personal information, it should

be presented to the team for consideration

.

Rating (factor and sub-factor) shall be by considering these

points, all together:

1st - Comprehension or understanding of the problem

presented

.

2nd - Approach described or outlined

3rd - Present or potential capability to accomplish this.

(Rating of potential capability is a very importan*:

point-the teams must try to project, formally, the

‘^capability of the bidder to correct def icie.ncies

which are due to misunders ta.nding or inadver fence . ) Major eiT'phasis should be on the factor or sub-factor bei.nc

rated; however, no factor exists i.n total isolation, a.nd the

impact of other factors should not be excluded. The overall

standard is " the extent to v.'hich you believe the factor will be

accomplished, keeping in mind his unders tandi.ng , approach a.nd

capability

.

"

Comparative "weighting" of sub-factors and i.ndividual evaluations: The factor team is expected to hammer out the comparative

evaluation of the several sub-f actors ; there is no preco.nceived

arbitrary division. Similarly, the factor team mem.bers are

required to establish a mutually satisfactory evaluation. Other

members of the factor team may ’properly i.nquire i.nto the basis

$nd reasoning of a specialist in his area; a.nd, conversely,

should give appropriate weight to his evaluation. No arbitrary

voting procedure is established.

The process to be used in arranging the order and distribution o

the contractors is to be established by each factor team.

Various summary steps, such as proceedi.ngs by sub-factors, by

individual evaluators, or the overall factor, m^ay be used.

Distribution: After a rough ran.king of the bidders, the tea.m

should separate the bidders to indicate the distribution, that i

the relative gaps reflecting the differences observed. This m^ay

be accomplished by having a team miember adjust themi on a blackboard until the team agrees on the distribution. Once a.n order and a distribution have been mutually agreed upon, the

distribution shall be transcribed to record form, (the Fa

Worksheet) for file purposes and signed by each mie.mber o

factor team. Any strong individual disagreemients wioh t

outcome of the team should be noted on the record form

Page No. 4

lJ‘

1

11 o
RFP 6-35756

469

together with a concise supporting statement. Also, any

points which will require negotiating in the event a certain

bidder is ultimately chosen, should be noted on the same form.

When using the blackboard as a graphical scale, these suggestions

are made. The top of the board should be reserved for the

bidder, if any, who makes you "sing," you believe that nothing

can stop him from being a success on this factor (or sub-factor)

.

The bottom should be reserved for the bidder who is hopeless

within reasonable money and time limits. Care should be taken

to avoid bunching everybody in the middle. Reflect the comparative value of the bidders over the total scale.

REVIEW

The COTR will incorporate the factors and prepare a summary

of the evaluation for presentation and/or any procurement

review. Procurement practice requires that this stage and

up to signing of contract be limited to minimum personnel

required to prepare reviev/ documents.

Page No. 5
^70

APPENDIX D

Ratings from the Agency Imapct Contract Evaluation

Displays of the way the ratings appeared on t*^e blackboar

during the factor team meetings. The initial individual

ratings and the agreed upon sub-factor ratings appear

on one sheet per sub-factor.

The displays of agreement on factor ratings are provided

after the last sub-factor within each factor.
1 ~

1

1

!

1

1

•

t ^ 1

1

1

oo

1

!

1

! ^

i •

!

i

1

UDO

CSJ C2m

o

CM

Xi

CM CO o o o

CM CM CM

CD 03 X)

1 i

O

CM

CD

1

EVflL

21

1

<no

CM CO CDo

«*CM

CM

cn

U3a

CM -T- cn o

CM CM o CO

CM

CO

EVflL208

uo cn o o

CM CM

, CD ffi

CMo o

CM CM

oa GQ

— (O o a

CM CM CO GO

EVflL205 — (jO O O

CM CM GQ GO

lOa

CM

CD

CO CM ^ O O O

CM CM CM

CO 03 (O

CSJ o

(M

_l

CL

>

LU

COo

CvJco OQO

CSJ CM OCD

CMCD

U3 — O O

Cs* , CSJ CD CO o

CM

GO

CD rr^ ro CD LD

L

'tI- PO Pv 1 1 H-H LJ J ULJ L ^ k * ) LXl

1

1
€77 a?2 i ! 1 ; 1 1 ! 1 1 ^1 • 1

oo ; 1

^CSJCD CCG?

CSjCSiCsj GCG

m—u2 3CO

GOCOD 1i1i NJ

22 ^ X

— ^ ^ G G G i!1 CM>

UJ % C.2

0302 3 —m ^ GC *32 (NJCSl GGQ

G

EVRL208

liT) O

(M

_1

CC>

LU

CDO

CSJ CDO

(Ni CO

CsJ

G

OG

Cvi (Nl G G ;

cn ^

cG

G

G

CMa

CM

CE>

UJ

03O

cviio 03O

<NjC^ i^OD

Cvi CO 2

<N3

—

D o ^ csi O wQ

!Mca 1

CM

V—

°

CO CO (O' OD LO 1 1 ^

cn CM 1i1
473 «««« I J o

CM

CsJ o

CM CO c CCc

a

cn ^CM CM o o m

CM 1 m ' 'Ca

CSJ ^CSiCD oco CSJCsiCSJ 0DC2CD

fn

—lo ooo CSJCsJOJ 03 cam

uoa

CM CM 03 m 1 ^ O o 11 O

CO CM CsJ CM CM o 1 23 m 11 22 III

ooo

(N

—I(X

LU

Lno

CSJ _1

LU

CsJ O

CSJ

LU

on

O

U3 cn CM 1 CO O o O o o a

CM CM CM CM CM CM CO CO CO ca m CD o

CMO

COO

CMm

COa

CMca

U3 cn O o CM CM

OQ OQ CMO

CM o

CM

OO 03

CD CO r- CD o U3 cn CM a o 0=

CM

GQ

CMm CMO

CM

oo

a

CM

oa

LO CO CXJ

475 IIIIIIIIIIII«
U -^6

cno

CM

23

O

03

CM

CM

CsJ I ^ I > !cno

CM

CO

•J3 a

CMOQ

rsj a

CS4 23

O ^co 30

CVi CM 3CC

rcLO ^ ^

3C3 33 3

CMCvj 30

CQa

CSJ _jcr>

'lU

CM"^ OC

U3 CMCM O 30

CM

—cn o ooCMCM OO

O3

CM

EVAL207 [ -VCM

in \ CMCNJO o 1 ciCc:

CM cn—CM

CO 300

CMCm OO

|eVRL20S 

CD 1 O

CM u3 T) ® o o — i CM CM O 1 o o

CMO '

Cm -c CM Cm 3 3

EVRL202

CvJ LD . O o

(NJ (NJ OO GO

23 cn CM C 23 ^ — ® 3

CMO 3

CM

33

10 9 8 7 (jD lo ^

CO (N.

—

I
11 1 i1

i!,1 ^ ^ 1 ' 1 i !

:n ! — 1 * 1 i1!

CSJ '

Od3 3 C'Jtn ' 3 3 ‘2^0 3

<M

—.Cvl 1 CM 3 CCC ' 3

(M3 i ^ t ^ i ! -Z

en 3 33 3

COO

CM ^

CO 3

CM CM O 03

CMCO i1 CM 1

CM i 11

03O

CM cvj 3 o

cn CM o 33

CMCO 1

CO

— -V o

CM loo ; ^J OQ , O OQ

CM

_Ic:>

LU 1i11i COa

CM

cr>

LU 33 CM CM O CO

5rJ CQ ^

^sJ CD o

CM CO CDO

^CM

CMCO

EVAL207

8205

B201

8204

8203

8202

8206

uoa

CM

_i

cr>

LU

CM o o

CM CM

GO CD

6203

8205

8206 , ^o

CM ! CO 1 CMa

CM

_Jcr>

UJ

cno CM CsJ O CO

20

O

CM3 3 3 O O

CM CM 3 3 ii O 1 s

CM CD CO CD LO ^

on (XI

—^ ^


i

4P2
i

M83

CM

Csl CQ

CSJ o

(N*

cna

CMCD

CDo

CM

CO ^ —o

CM CM

CO

LOa

(NCD

CDO

CM

CO

CMc

CMCD

cno

CM

CD

O

CM

CD

O

CM

CD

CM I CMO

CM

CD

O

CM

lDO

CD

cno

CM

23

cco

CM

23

CM23

LOO

CM CO CDo

cnCM OCQ

CMCD

CMO

CM

03

O

CMCDO

CM

CD

IDO

CM

OD

CDO

CMCD

O

CM

03

cno

CDo

CM CM qCD

CM

CD

CM ID cn CO a o o a o o

CM CM CM CM CM CM 03 CD ca 03 CO 03

CDO

CM

CD s

IDa

CM

OD

O

CM

CD

CMO

CMCD

cn

ID CM

COo

CM o o 03 CM CM cn OQ CD o

CM

GO o

CMCD ^

CD CO r- CD LO CO (XI
li^U

I

I

I
485 IIIIII1
436
487 IIIIIIIIIIIIIII1 >1 1 1

LC

i" i 11 1

CD> 1 1i1 11i ^

i>i111 >

CD

U3 CSICD a oQo

(Nl £\i<NJ 33 OCD

CSJ CQ

a o

CSJ CM

CQ CD !i 1

>•

LU S

cn cvico o

CsJ CM^

aa coa

CM

CD

tt

—

OO

CM CSJ CD CD

I*!1

COa

CsJ _1

cr>

LU

LOo

CSJ CQ B202B203 B204

CO

— o a

CM CM

OO CD

EVAL206

'Ta

CSJ _jcx>

LU

B202B205 B206

B203B204

cna

CSJ. _j

CE>

LU ^ o <N CsJ a 03O OD

CSJ cnoj „ IS . os Q

Si S S <a

w ' 1 > 10 9 8 7 CD LD ^

CO C\J ^ I
U88

CSJ NJ cao

•o —

^2

onCSJO 23c

CsjCsi

CS4

2Ni C3 20
489
490 > I
491 IIIIIIIII1IIII
U92
493 IIIIIIIIIIIIIIIIII
U9U
H95 IIIIIIIIIIIII1iI I1 i I i I rn I • I > I GO

CM on G o

CM Cm G G

CM

CD !i CMG 1 CD.

CDO

CSJ o

CSJ cX

CM o

CM

CO

CDG

OJG

cno

CMG

O

CSJ G

O

CMG , . I > ! GO

CMG

cnG

CSi G

CMO

CMG

(DO

CMG

O

(MG

G

CMG 1 — G CM on i- i CM G G o CD 1 —'

CM CM CM CM G G G G >

LU

GO

CMGO

CMG

G

CMG

CMO

CMG

GG

CMG

cno

CMG

O

CMG

O

CMG

GO

CMG

CMG

ooCM CM GG o

—on CM oo o

CM CM GG

GG

CMG

CMO

CMG

O

CMG 0 GOCMG

COo

CMG

GO

CMG

GO

CMG

O

CMGO

CMG

<r> O

CMG

CMO

CMG

on>

U3a

CM M" 1 . cn CO G O o o a

CM CM CM CM CM G G G G G

CM

CD o 01 CO [> CD ID CO CM i
496 > i1

CD ' >iL.>; 111 >

too

CnJ CDmO

OJ

CD

CM 3 3 3

CM OJ 3 3

•rCM3

5m

'j^ si i

CD 5§

-r3 111

lOO 3

Nl CSi OD 2 3 'n

—

Cm ^ ^ Cm3

>*

LDa

CVi CD 5 - a 3 o

IM CM a a a a

az>

Ui

iTj cn ^ o C3 3 . 3 JM CM rsi _ ^ 03 a n ^ ^

CM33

EVAL21 1 o

<M U3 (Ni CD o O

cn csi csj 3 CO C

(M3 3

CM3Cw

EVflL2lO

cn04UD 30

—CM 03

CM3

CM ^ 3 o

CM CM

Jms

03a

04

a:>

UJ

EVRL205 1 3 cn o O 3

CM CVJ CM 3 3 3

CM S S 3 3 3 o

(N

_lcx>

LU• > 10 9 8 7 6 5 ^

CD Cm ^ III
it97 IIIIIIIIII h IIIIIII >

cnCMUD cao

CsjCM CCD

CSJ CD oo

CM

CD 3

C4

CDO

CM

CD

•-0 o O CM CM CO CM

CO o o CO • cn CM CM > o CO CD o CM CM CO CD • JZ O o

CM o ca CM cn CD

O

CM CMO !

LO CM CO cn 1 ^ i o o o o o

CM CSJ CM CM CM CM 1 > CD CO o CO CD a 1

cnD I LO CM (O ^ ! O OO o O o CM CM CM CM > O 1 M i a D D D 1 CMo 1 CM cno 1 iO CSJ to 1 DO c o

CM CSJ CSJ CM CVl j ; >

OO o CO CDcnuo ooOsJCsJ CDCO CSJ^ OOCSJCsi OCQ Oo

CM

OO o

CM

OO

<no

CMUO CDO

^CM

CM

<n

CMO

CMCD

Oo

CM

CDO

CM

QD

U3a

Csl <x>

LU •

<r

U3 i" ^ o o °Qo a

CSJ cvi

mcsl CSJ GO CO oGQ OO

CM

OO

a

CSJ cr>

UJ :

UD •> 10 9 8 7 iO UD CO C\J ^ i
493

I

I

i

P

I
499
IA

LO->>! ro 1 • i >>

VI. 1

CM o r; CM CJ

TJ

CM tC *T ZcM

CM •

CM

c:

>•LU

9OZ0SOZ9 ^nzozoza ?M3 a

CM

_J

cc>

LU .

CM CM

— 03 S u a ci o ^ CO CM CM CM

CM

OQ COa

CM5>

LU

LOO

CM

CO

O

CsJ CM o CQ CMCO

CM to cnCM c®

23 LDO

CM

_Jcz>

LU

U3a

CMOQ 1

<=

® CM ^J O CD , —

= 3 i

CM20 • > o (D CO r- (ID LO ^

CO (Xl ^
501 IIIiI%I1iIIIIIIIII i j 11 ; !i ^' “ ' 1 • 1 i1 1 ^. >i1 i i1,1, >

uD :\j o a

CQ CD ' 1

»r '

on 'JD 1 ^ ' — = 03 1 _ Q3 3D ' —

CM33 '1 "1 * >

j: '.nj ^ o

?SJ OJ n CQ '

cna

•>J CD ^

230 c: ^C\J CSJ

CSJ CD ti

CSJ —1d>

LU ^ O INI <NJtD (M O CDo 03 CM —CM _

CQ oCQ

g

CD

a s

CsJ _Jd>

Lu

cno ^ CSJ c o CD a “ 5 Q

CSJ CD

sj ^ 0 O

kJ CsJ 3 CD 1 33oCM f CQ 1

EVHL208

1/3 O

CSJ CQ

O

CSI fO

QD

O

O

CSJ CSI CSI o ^ ^

(NJ CD i11 CD I S i G3

LOO

C\J _l

(X>

LU

U3O

CsJ 03

CO 0 O C

CSI c CD a j1 bo1 CSJ G ‘ U CsJ CSJ CD 03 ’

CM>

cn CO r- CD LD ^

CD OJ

—-
I 1 '-O 1

302

m

J3cC

I >•

<M

OvJ LO csi 1 cn O . O 1 O 3

CM 1 CM

CQ 22 * z: Xi

Ilk'll 1

IK^l

VI

. 1

1

!

1 n

J3 CM >iCO 3 a 30 o

CM CM —CM ?si 22 32 O

Cvj

EVRl.21

1

O

o cn ^ ^ O O a

^ ^ 03 aa 5

^sJ o

CM —

O 3

CM >U 03 03 o

CM>

UJ

LOO

CMoa o

CMo

o

,no 3

COa

CMcc>

LU

cn ld o o o

CM CM CO o o

L_._

^2 —

^J Csi CM 2 22 2

LDo

CM

o:>

LU

CDo ID CM O ^

“ s

CMGO

CM C^ 2 2

cn

t— 00 LiO,

>

^

^ CO

^CO Cm ^—
503
50 ^

lO

O

NJ

O

CSJ 23

rn

>4

m eg JO w o ii o

eg eg 1 eg

OQ CQ 11 a o

C\i 33

as

'JO eg

23

•?c

03 23

CN

CSi «:d eg

m

eg CSI x: i - 

! ^ ; • ^ •JO eg 1 ' >3 3 —a 3

i>!

og eg ^cg a a T

a

egc
505
z ^ ^

cn^ :

CsJcD 130

oncsj 003 o

^J3
507 I1IIIIIIIIII

VII 1 1 1 I ^! >

viI

.3 » 1i1j1

(XI >

CM

=3

a

OvJ CM o cc

CM

23

cn

— 3 CD CM CM

CD CD 1 § >

LCO

CSJ(^ CQO

roCM qCQ

CM

CD

(MO

CSJ CD o — Csi o n i11

C\J

rr~ >

LiJ § CM cn cvj 3 a aa ^ lO CD CD g

CM

CO

T

c:

CSJ CSJ CD ZC ii!1 , 1

EVAL210 o

CNJlO ODO

0^0

CsJ CO

(T5 O

CSJ CD •

•’O o

CM

CDO

LDO

(XI _1

CL>

LU

CD -• cn o o o

CM CM CM

03 CD OO

ID CM o o

CM CM CD CD

G

CMG

O

(N

_l

(X>

LU

IDO

CM 0 CD c CM C o c CM

CD 0 ^ CO 3 O O

•J CSJ CSJ 2 CO GO

VO

CSJ G

VII

.2 cn CO o CD UD ^

CO OJ

—

H 1
508

iTi

CO

C

^d£

^sJ o

cno

CSJUD ac

cca

Csj

a

CSi

cno

Cnj 23

CSJ

03

CM

03 £

?4
509 IIi1iIIIIIIII

cn 1 o

csjua CD 030 o o

csj 1 !NJ ii ^ 23 CD 1 1111 V1I

.3 CDO

CSJ s

CvJ c

CSi 02

ono

Cslio C3C

CCQ

-r3

CSJ 3 .

CSJ i . CDO 1 ; CSJ ID CO rr uQ o 3 o O 1 CSl CSJ CSJ CSJ CSJ :> o CD GO cn cn

CM

CQ • 1 ' 1 1 LO CSJ a 1 CM(D

23 C3 22 ! CD CM CSJ > p03 "D 3 CsJ CSJ cn I i
510

—

H

-C

-r; -r*

'VjOsJ ' . , —
511

APPENDIX E

Ratings from the Commercial Imapct Contract Evaluation

Displays of the way the ratings appeared on the blackboard

during the factor team meetings. The initial individual

ratings and the agreed upon sub-factor ratings appear

on one sheet per sub-factor.

The displays of agreement on factor ratings are provided

after the last sub-factor within each factor.
r

^ '

•

,

cn

i

^J rs

i “*

j

1 - -

i ^

1 3

} zr 3

j ^

LU

. 3

o

o

_J - T) g: o

> C3 a . 3

LU

cn

CD

t—

1

u X) CL o 2 2 > CO 3

LU

CvJ O * *—

1 _J Ni —

CE a 3

> 3 a

UJ

^

CD CD OD LO^rD cnv. ^


515

517
> — I II • _

z:

— .

i_111 z - =

ZZ ^ I c: ; > , i !
519
I
521

IIINIIIIIIN

523
52H
^ 1i 1-! ! ' 1 ^i < 1 • 1 1 m CM a C3 CQ X X

cn 1 m a 1 o 03 1 l-M 03 ooo 1 Csj 1 • —.

CM s-n o 3 ;

XI CO 11 • CM 1 o ' • — -n 1 N-, J3 3 I —

“ o 3

23 CM 1 - — CM cn ^ -J a o — : 5 CO CD ii := cn CM 1 ^ o w

(X>

03 03 03 LU

03 1 a

CM

— cn ( _i o O Z) \ > cn C33 oa ! UD .. . : iOa

—1 i CL> i LU

LOO

CSJ (T) —I o o a

a:>

03 OQ 3

lU

CMa

_1 ' d>

UJ• CD CO !>- CD LO ^

CO (XI --

T—1

527 IIII1I\IIII

529

I
530

I

t

I

i
1

I

I

I

I

I

I

I

I

I

s

I

I

I

I

I

1

1

EVflL

102

EVflLlOa

EVRL

107

EVflL

108

EVFIL

1

CVm

I

12

IV.

1

IV.

2

IV.

3

IV.

4

|

IV.

531

I

1

i

I

3

02 :3

o

CO

TJ

CO

1

I

t

c

CO

I

J.

I

X

o

GO

CM3

CO

o

OQ o cncor^cDLO^coc\j

B102

13103
c -J
I

!

I

\

I

I

1

I

I

!

I

1

1

I

I

1

1

I

EVflLlQ2

EVflLlOa

EVni

107

|E

VOL

108

533

i

f

O CD CO CD ID CO (XJ

535 IIIt

II

IIB

IIB

II

II1III I1I1

CD Ii 'Si I • i >

>• !

537

I

I

>

>

os

CSJ 3

Ti

I cn f :=;i

C

CD

rjo

i

;

i > I c

GQ

m

03

?MO

539

> i

1

1

1 —

1

! LO ! CM 3

> '

1 03 03 = i

1

1 - 1

1 CM

1 cn !

• 1 3 O

> : = 1

1

CO —

!

.•ri 1

i 3 ^ 1 23

i

1

1 -

! <N 1

1 CM O i => a

> 1

1

03 — CO

!-!

I

5^1

543

5“5

I

I

5D7 IIIIIII i iTi ! ' ; i> 1 V

11. 

11 ! >1I

1:‘ t—, — r) ' ! > C3 C3 -

ccc

r—' cn 1 _J CSJ o 1 (X o aa > cn o 1 LU ' CD LG 1 CD t m 1 —J —

2 1 X o :

D> 03 CSJ o

LU 1 d3 CMO

_JX

O

O > * CD m

LU

1—^ CD cn CO C' CD LD 1 ^

CD C\J

— r—

> 111

on

CM

ca

551

rII I -

i:> i:

<NJ “ ! ^

(TJ

•z> - ! a 11 '' . ^ m -Nj 1 ^ o — > r: C2 I 1» " ] — Csj Xi 1 = = 3 >I[ 1 £3 CQ CD a ! _ 1 — CM
552

APPENDIX F

Cover letter and Questionnaire for Panel Members
February 15, 1980

Dear ;

I am writing to request your assistance in a case study of a proposal evaluation

process in which you participated in April of 1976. The procurement was for the

development of evaluation systems for experiments in government procurement

on behalf of the Experimental Technology Incentives Program (ETIP). This case

study is designed to assess the effectiveness of the method which ETIP used to

solicit and evaluate proposals. Your answers to the enclosed questions will provide

ETIP with a basis for reaching conclusions on several parts of this evaluation.

The total time for your participation will probably be between a half hour to an

hour.

As you may recall, one of the key characteristics of the RFP was the use of a factor/sub-factor structure which required responses to issues which were thought

to be critical for the procurement. To help you in answering the questions I have

enclosed a list which contains the factor and sub- factor names and a summary

statement of each one. You need not mail the questionnaire back. I will be calling you in about a w^eek

to try to provide any clarifications and to obtain your answers. Prior to this, olease

go through the questions and write down your initial answers. Your name or the

name of the firm which you represented will not be identified in the reoort on

this research.

This case study will be published in NTIS and be disseminated to people interested

in the procurement of evaluation systems.

Sincerely,

Bud Libman

Policy Analyst

Experimental Technology Incentives Program

Center for Field Methods

Enclosures
Background Note

The RFP to which these questions refer was for the procurement of two systems

to evaluate the impacts of experiments with government procurement methods.

One system was to evaluate impacts on the commercial sector and the other

was to evaluate impacts on governmient agencies. They were each estimated to

require 200 man-months of effort for the first two phases of development.

It appeared to ETIP that there was considerable complexity and uncertainty Involved

in developing these systems. Complexity came in part from requirements to combine the technical concerns of doing evaluations and designing the systems with

the political concerns of meeting the information needs of people at different

levels in several organizations. Uncertainty was present since objectives were

expected to shift over the three year period of the contracts.

Three proposals were received for the commercial impact contract. Six proposals

were received for the agency impact contract.

The evaluation panel met in factor teams during the week after the proposals

were received. You participated on factors on the commercial im pact contract evaluation and factors on the agency impact contract evaluation.

Questions

1. This question addresses the effectiveness of the RFP's factor sub-factor structure for enabling the panel to evaluate the bidders' qualifications.

a) Did you feel that the factor/sub-factor structure enabled bidders to dem.onstrate their ability to deal with critical problems which might arise during

the project?

Yes No

If no, what modifications in the structure would have enabled bidders to

better demonstrate their abilities?

b) Did you feel that there were critical gaps in evaluating the bidders' capabili ties to perform effectively on the project?

Yes No

vn
2

555

If yes, what were these gaps?

c) Were you able to readily locate the information in the proposals which

was relevant to the issues you were evaluating?

Yes No

If no, what changes in the structure would have helped you to more readily

locate the information?

2. This question addresses the appropriateness of the sub-factors which were

included in the RFP.

a) Did you feel that the sub-factors encompased most of the issues which

would be critical to designing the evaluation systems which ETIP desired?

Yes No

If no, which important issues were not adequately covered?

b) Did you feel that many of the sub-factors dealt with issues unimportant

for designing the evaluation systems which ETIP desired?

Yes No

If yes, which sub-factors or issues did you feel were unimportant?
3

3. This question addresses whether the number of proposals evaluated was sufficient to enable panel members to be confident in their ratings.

Do you feel that being able to rate more proposals would have increased your

confidence in the accuracy of the scores which were assigned'’

a) For the agency impact evaluation system contract?

Yes No

If yes, about how many more?

b) For the commercial impact evaluation system contract?

Yes No

If yes, about how many more?

4. This question addresses the effectiveness and the equity of the lack of restrictions on admissible information. Of concern is the effect of knowing the

identity of the bidders (i.e., non-blind evaluation) and being able to use reports

of the experiences of other panel members who had previously workec w-itn

the bidders.

Did being able to use all available information about bidders in the evaluation:

a) improve the accuracy of the scores?

Yes No

b) introduce unfair bias into the scores?

Yes No

If yes, how did this occur?
4 55

5. What did you feel the strongest aspect of the evaluation process was?

6.

What did you feel the weakest aspect of the evaluation process was?

7.

Approximately how many other evaluation panels have you served on?

8.

Of the total number of other proposal evaluation panels which you have served

on, about how many were for projects of the magnitude (i.e., 200 man-months),

complexity, and uncertainty of this one?
Bidders were furnished the following information either in the Preliminary Description

or at the first Bidders Conference on February 26, 1976:

Information for Prospective Bidders

Corrected Version of the Commerce Business Daily Announcement

Introductory Notes

Definitions

Statement of Work

Statement of Work (with illustrative sub-items for Phase One)

Proposed Schedule - Overall

Schedule Phase One

Scope of Effort

Proposal Preparation Procedure (with revisions)

Proposal Evaluation Factors

Proposal Evaluation Factors with Illustrations

Numerical Weights for Proposal Evaluation Factors

ETIP Program Plan, dated February 5, 1974

ETIP Progress Report, dated April 4, 1975

ETIP Progress Report, dated August 1975

Selected Bibliography of Published Documents Related to This Procurement

Message from the President of the United States Concerning Science and Technology

Evaluation System Design Process

Description of Governmental Procurement Policy Program - ETIP

The Council of State Governments and Associated Organizations

NIGP Dictionary of Purchasing Terms

NIGP 1974 Annual Conference and Products Exposition

Memorandum to Members Regarding NIGP/ETIP Project, dated October 9, 1975

Memorandum to Members Regarding NIGP/ ETIP Project on Window Air Conditioners,

dated November 26, 1975

NIGP Membership Application

Prerequisites for Certification of Public Purchasing Agents

NIGP Letter Service
559

APPENDIX G

Cover letter and Questionnaire for Bidder Representatives
February 15, 1980

Dear :

I am writing to request your assistance in a case study of a Deoartment of Commerce procurement which you responded to in Aorii of 1976 ^RFP 6-35736). The

procurement was for the development of evaluation systems for exoerimients m

government procurem.ent on behalf of the Experimental Technology Incentives

Program (ETIP). This case study is designed to assess the effectiveness of tne

method which ETIP used to solicit and evaluate orooosals. Your answers to the

enclosed questions will provide ETIP with a basis for reaching conclusions on several

parts of this evaluation. The total time for your participation will probably be

between a half hour to an hour.

As you may recall, one of the key characteristics of the RFP was the use of a factor/sub-factor structure which required responses to issues which were thought

to be critical for the procurement. To help you in answering the auestions I have

enclosed a list which contains the factor and sub-factor names and a summary

statement of each one. I have also enclosed a list of the written materials which

were included with the RFP.

You need not mail the questionnaire back. I will be calling you in about a week

to try to provide any clarifications and to obtain your answers. Prior to this, please

go through the questions and write down your initial answers. Your name or the

name of the firm which you represented will not be identified in the report of

this research.

This case study will be published in NTIS and be disseminated to people interested

in the procurement of evaluation systems.

Sincerely,

Bud Libman

Policy Analyst

Experimental- Technology Incentives Program

Center for Field Methods

Enclosures
561

Background Note

The RFP to which these questions refer was for the procurement of two systems

to evaluate the impacts of experiments with government procurement methods.

One system was to evaluate impacts on the commercial sector and the other was

to evaluate impacts on government agencies. They were each estimated to require

200 man-months of effort for the first two phases of 'development.

It appeared to ETIP that there was considerable complexity and uncertainty involved

in developing these systems. Complexity came in part from requirements to combine the technical concerns of doing evaluations and designing the systems with

the political concerns of meeting the information needs of people at different

levels in several organizations. Uncertainty was present since objectives were

expected to shift over the three year period of the contracts.

Questions

1. This question addresses the effectiveness of the RFP's factor/sub-factor structure

for enabling bidders to demonstrate their qualifications.

a) Did you feel that the factor/sub-factor structure enabled bidders to demonstrate their ability to deal with the problems which might arise during

the project?

Yes No

If no, what modifications in the structure would have enabled bidders to

better demonstrate their abilities?

b) Did following the factor/sub-factor structure create any problems for

your team, in trying to demonstrate its capability, which would not have

been present if a different structure had been used?

Yes No

If yes, what special problems were created by the sub-factor structure?
2

562

c) Was your team in any manner placed at a competitive disadvantage tiecause

you had to follow the factor/sub-factor structure'’

Yes No

If yes, please describe how you were disadvantaged.

2. This question addresses whether the proposal format enaoled bidcers to be

confident that:

(1) they included information in their proposals on all of the capabilit.es

which the evaluation panel would be looking for, and

(2) the panel would be able to find this information when it was reeded.

a) Were you confident that you had written something to demonstrate your

qualifications on all of the capabilities which the panel a'ouIc oe evahjat.r.g''

Yes No

If no, why not?

b) Were you confident that the panel would find the information in your proposal which was relevant to each specific capability at the time when

that capability was being evaluated?

Yes No

If no, what information did you feel might be overlooked and why did you

feel that this might happen?
3

563

3. This question addresses whether the information included in the solicitation

document, the supporting documents and the two briefings gave bidders a

sufficient general understanding of what ETIP wanted to buy. Since many

of the specific objectives of the projects were not known when the RFP was

issued the concern here is whether sufficient information was provided on

the general nature of the projects to enable bidders to respond effectively

to the sub-factor issues.

a) Did the information provided by ETIP allow you to assess the skills which

would be required to do the work?

Yes No

If no, what additional information would have helped?

b) Was ample information provided in the briefings and supporting documents

so that you were confident that you had sufficiently related the sub-factor

issues to the objectives of the projects?

Yes No

If no, what type of additional information would have been useful?

4. This question addresses the appropriateness of the sub-factors which were

included in the RFP.

a) Did you feel that the sub-factors encompased most of the issues which

would be critical to designing the evaluation systems which ETIP desired?
Yes No

If no, which important issues were not adequately covered*’

b) Did you feel that many of the sub-factors dealt with ;ss'.es jr.irr.portar.t

for designing the evaluation system’s which ETIP cesirec*’

Yes No

If yes, which sub-factors or issues did you feel *.ere jram portar.t*’

5. This question adcresses the degree to which bidders jnderstood tr.e proposal

evaluation process which ETIP used.

a) Did the information provided enable you to uncerstanc tre process wncn

ETIP would use to assign scores on each sub-factor?

Yes No

If no, please describe which aspects of the evaluation process were not

clear.

b) Did the information provided enable you understanc how tr.e ratings on

the various sub-factors would be combined to arrive at fl.nai scores?

Yes No

If no, what aspects of this were not clear?
5

6.

This question addresses whether the process of combining scores appeared

equitable. The RFP provided weights for combining the factor scores into a final score. However, the relative weightings of the sub-factors within

the factors was left to the discretion of each factor team. Were you concerned

that not specifying the relative weightings of the sub-factors might lead to

inequities in the process?

Yes No

If yes, how so?

7.

Did you feel that the process would be a) difficult or b) easy to cor- rupt? i.e., For one person, or a group of people, who favored a specific bidder

to exercise sufficient control over the process to sway the award decision.

Please explain your answer.

8.

Approximately how many other RFPs have you responded to?

9.

Of the total number of other RFPs which you have responded to, about how

many were for projects of similar size (i.e., 200 man-months), complexity,

and uncertainty as this one?
10. What do you feel was the strongest aspect of the process which ETIP used?

11. What do you feel was the weakest aspect of the process which ETIP used ?
567

APPENDIX H

Enlargement of Figure IV. 1

The Activities in ETIP's Source Selection
X

X

Hu c X —

o Cd 0, (D 1 V « 4-) 4-3 1 '.1 M

CO 03 4-3 03 0 0 — ft M W

s: S- O C- X o 1 CJ 0 ;2

Ctl tiO CO 0) 4J o 1 0 0 2 ^ — 2

C-i (U CD •'n CO 0 X C- X Cd w 23 CO 4J > O •• i- X X X oa X — n

>-t c CO •tH i- c c X o O 0 0 0 < j2 CO 4-3 SJ cx o c Cd 0 ", V- M

O j- s CO o bO 0 c 0 u X X CU 03 z o 0) 03 <M T3 CO X X o c • 2 X > 3 > > 73 o 4-> •<—) O (0 -J X E •^4 1 E 0 M C — 73 MM Q 2

1—

1

o CO < X o o X X •3 X o 0 X TT 73 03 £- c 4J rH CO CO X 3 X o 0 O 0 0 0 <3: 2= V C 03

«a: 03 a a 03 CD O 0 H X CJ 0 0 0 5 X cc r.i — 73 2.

r—

1

03 > 2 Om 4J X S- 0 Im X X C- ' Im c: 73 5— C. r3 MM MM a OT Cm O O •M be T3 X X O X X hm 0 CO MM 73 o; — 4-3 J- ~ X s- c S, C X c O 3 X 0 O 3 M, 1 c ^ 21 > o s c acM CU 3 — 0 o o o I- O Cd — o;; 0 2 1 M 23 1 V

Cd 4J 03 o a. Cm > 3 0 o C.CJ 0 c —

Cm Ola Im 1 > o 03 03 X o 0 ao - X X 0 s: CU MM CK CO to 4.3 o 4-> O £ 2 X X « C c X X X Cm > C ^ ®<3 M O c 0 c X3 CO Cxu > O O X X c t. o 0 "C 2 < 7Z 73 a M c; Cbi Cm o c c c X X £- -4 0 0 0 s 0 uJ o X C 03 55 03 &c a •iH cd Ou 6- o S- CJ 3 E 0 X 3 S X 3 S 0 CJ 03 C 3

£-* <v M CO 0 1 X c «S <U .M c o 0 0 0 < 3 Cm CU M 2^ 73 CJ o 03 3 e- X a 1 X o •‘X Cd S- X o o 0

^ M 0 X cc 0 Cm m c M. CU 2 0

<c 4J 3 0 Cd X S X 0 X X =M C 3 X X < c. 3 X 00 0 o M-i 1. 2

cc t. c (—

(

Of O X X 0 H 2 O O £-• 0 2 —M O > o s

^ « w C 03 Ola e- o •rH 03 > ^ X X X 0 O O X O su c Cm < 22 0 22 o; tl£'>M M

s 4-3 > <v cc X £-• C CJ O 0 SU Cd « <13 o 0 S X X Cm ^ _ c 73 o; c •> o O c (1) SX E-t X X 0 0 a c 0= o

0

c_ Cd C 2. c c mM Im o <0 o O «M Cd 0 O X •o «Cd E •M X o CO c Cd Cmm 3 <u 03 03 c

M-H Cm •*0 Cm X C, 2 C X O Cm 0 Cdx X •M 0 X X X o “C • u o •fH 4^ O Ou do 1 cd 6^ CO bCX S O £-• fH X a CJ X X c 0 2 X O 0 =-< 3 03 C 73 03 H Q 03 Cm (—

1

C 1 4J Cd C 0 X 0 Cd 0 O •M 0 X X 3 73 23 0- MM 3 E- -4 CO CO o ««. Cm O CJ 0 c. •> 0 X CJ < 0 CJ 0 X 0 j2 M 03 c 2 CU (—1 (S Cd 4J C 2 XXX bO 0 2 X X X S- 0 0 o 2 0 2 2 1 -p-^ "C -M c O 1—

1

03 03 < I—

1

6-. -M H o S- X X c o > •M Cd X 00 C, X X X 03 17 22 — oc M e- > o 4J) £m 1—1 ^ CC o t- o Jm X > c o o X X o Cm 0 > X •*- > CO > X Ii4 L, 03 CO Cd 0) CJ 03 3: C.CU <3: CJ 3 CJ X £3 X X X CJ 0 X CO X c 2

•• Im X CJ 22 2m C 20 73 1—

1

X O o C w

CJ 1 1 1 X t 1 1 > 1 1 > 1 > 1 > 1 1 1 Cd X 2 2 2 2

Q X X X Ml X
5f9

3

cn c

Cil

•» 2: CT3 > X

< cn <13

' X 0 *3 E^ £- C s- 4J 4-> cn 03 b 1 03 0

c >1 c. CC 0 cn •* —

9-1 C X 3 d C C X

JC 0 1—

z ?

C C 3 0 0 E 03 0 X

0 0 ca h

c« 0 0 CJ .-4 0 >1 0 3

H-t

-*-} t: CL C-I 4i> cn X X X > X 1 Cm 03

B-4 0) 4.) 0 c 4J XX r-4 •» 03 03 1 0

<< 0 c 0 cn E X 03 £1 C3 £. 03 z »d

Z3 <U 0 c X 0 03 0 03 X 0 Z 03 d

•*-( 1-4 E- 3 X X X X X •d 03 X c c £.

<3: (U su c z 0 CTE-* <n cn 03 -H e- E X (jO 0

> <-> n 0 t- 0 4J c 1-4 £. 0 Z 03 *14 X X

Cx3 CL —• 0 0 od «X CO 0 X 03 CJ 03 Z X cn 0 03 0 CO cn <*-• 0 £- X Z 0 X 1 > £- 03 03 —4 Cd OJ cn Ci3 E z 0 £-• 0 m d C3 1 -4 CT3 cn £. X

X Q. (V 1^ cn z 03 03 X d H-i cn 03 cn a 03 3X X X 0

&-• X3 £ £-• E 3d &- 03 c £. £. X 0 03X c c d

(1) s JC (13 03 3 cn X 0 03 <e 03 E- 03 0 0

0 -k-> 53 3 03 X 03 03 £- HE X X X "O 3: £. Cd £1 0 c

Z s cn e 0 4.) > 0 X C C3 03 Z c 0 X

H-( > £-• Cx4 —4 03 «X H-4 0 03 X cn cn c •» X

> c Cx3 C- rH (d £1 C. X 03 Id £1 cn c X •d 03 C C 0 c 0

-J 0) c > 0 03 Cx3 CO 3 CO &-I Z X X 0 on 0 0 C3 0 0

0 c 0 M <4-1 C r H d <n 0 X 0 03 C X (Z CO c on X £. > OJ cn az 03 d £. X 0 0 CJ 3 0 K-4 X 0 X Xad 03 X

z 0 Q. 0. 0: 03 <n CO z 0 X cd d a £- cn 0 d X 03 E C

H-t ^ E < 4^ ^ 2: CO X 0 (n >, 03 z X 1 d 03 X X cn z 0 c lU 0

4_> C 0 (0 c (0 < d 4-> c. 0 X C d 1 3 X CO 03 cx 0 0 C3 Q 3 0 Z *3 0 cn (d X 3 0 X a. e-> c. n « 0*z X d £- 03E- 1-4 d . Z 03 E-* 0 H

• 03 4-> 0 0 Z X E- 03 £1 C 03 03 X s X 0 m

<; I.) <; b0 4-3 a 0 (T3 Cx4 £- 03 C3 H-t X 03 OCX X C E « 03 c X

od (0 «d cn C 03 0 X z •d --4 X C. 03 X C3 z n 03 X cn Cb 03 0 £. 0 E-* 0 r-( 3 0 3 t. 0 H-t 4J 0 03 t, 3 z X X —4 0 X C3 0 X z 0

z ts 0) 1—

CC CO —

CC.E-* SI X 0 Z XXX X E X E X X 03 X 0 0 >-t CJ

H-t £-• c ca E- 0 0 03 0 >-t bOH-4 cn 03 c -d 03 X 0 —4 X L-* >4 -4 t-4-> d (£3 Cx-

£-« 0 (ti > 0 X r-( > C X Ci3 0 03 X 0 £. 0 £- X £- 3 E- a: t. 3 03 £. d >4 bo O.Ci3 0 a. 0 Cx3 0 Cti <c 0 0 C < 0 X 0 CO 0 (ZE- !3 on d c !3 «d 03 Z X »—

X

Cx3 2: M X d 2: X

-J 1 1 Cz3 1 1 Cz3 X Id 1 Cx3 1 <C 1 1 2: 1 1 X 1

CxJ X X Z Id X X z d

C/1 C-i E- CO 0 X 0 CO Q

hah

Al’H

HAy

JUN

J(JL

AUtl

SEP

19/6
JSC
mh