o NBSIR 76-1124

Evaluation System Proposal

Preparation and Evaluation

Procedure

Charles W. N. Thompson

Office of Experimental Technology Incentives Program

National Bureau of Standards

Washington, D. C. 20234

Final Report

February 1,976

Prepared by

Experimental Technology Incentives Program

National Bureau of Standards

Washington, D. C. 20234

NBSIR 76-1124

EVALUATION SYSTEM PROPOSAL

PREPARATION AND EVALUATION

PROCEDURE

Charles W. N. Thompson

Office of Experimental Technology Incentives Program

National Bureau of Standards

Washington, D. C. 20234

Final Report

February 1 976

Prepared by

Experimental Technology Incentives Program

U S. DEPARTMENT OF COMMERCE, Elliot L. Richardson, Secretary

Edward O. Vetter, Under Secretary

Dr. Betsy Ancker-Johnson, Assistant Secretary for Science and Technology

NATIONAL BUREAU OF STANDARDS, Ernest Ambler. Acting Director
1-i ....

•• ... . .
CONTENTS

Introduction i

Introductory Notes 1

Some Definitions 4

Proposal Preparation Procedure 7

Revisions to Proposal Preparation Procedure 17

Evaluation System Design Process 19

Statement of Work 21

Statement of Work (With Illustrative sub-items for

Phase One) 23

Proposed Schedule - Overall 33

Proposed Schedule - Phase One. 34

Proposal Evaluation Factors 35

Proposal Evaluation Factors (With illustrative questions)42

Numerical Weights for Proposal Evaluation Factors 75

Scope of Effort 77
.
I. INTRODUCTION

The objective of the research which culminated in this set of

documents was to develop a process for the procurement of two

evaluation systems. Within the context of applicable procurement

regulations and program plans, the set provides a complete and

integrated framework for both the preparation of proposals by

prospective bidders and the evaluation of proposals by the evaluation team.

The document entitled Proposal Preparation Procedure provides

both an overall description of the process and specific guidance

to prospective bidders. All of the salient or significant

characteristics of the system to be procured, the program under

which it will be developed, and supporting elements are included

in a set of elements called sub-factors which are grouped, for

convenience, into a set of factors. Each of the key documents--

the Statement of Work, Schedule, Scope of Effort, and Evaluation

Factor s - -follows a parallel organization of the set of sub-factors.

It is this integration of all of the central documents which

provides both proposer and evaluator with a single, common structure .

1
The initiation of the development of this process arose out

of the recognition that the "evaluation systems" to be procured

required the services of a systems contractor and that it appeared that neither ETIP nor the "evaluation industry" had any

significant present capability to design or manage such a system.

A review of available proposal preparation and evaluation procedures suggested that some further development was required.

The process described herein was used to procure the two

systems, and, as of the initial getting started period under the

two contracts, appears to have worked out reasonably well in terms

of our confidence that we have chosen not only the best but also

reasonably competent contractors.

While this set of documents was specifically designed for the

procurement of these two evaluation systems, the process, with

modifications in the set of sub-factors, is applicable to comparable systems problems involving complex or uncertain requirements.

One of the purposes of this report is to make these documents

available on the basis that others may be interested in further

applications
RFP 6-35756

Introductory Notes

The purpose of these two proposed procurements is to obtain

contractor assistance in planning, designing, and carrying

out the evaluation of a series of procurement experiments

which are being undertaken by ETIP in cooperation with several

governmental procurement agencies.

The set of objectives of the evaluation program includes the

following:

a. an overall description of the experiments as well as

selected detailed descriptions

b. an overall assessment of both the immediate and the

subsequent effects (impacts) of the experiments, as

well as selected detailed assessments

c. An evaluation system or process which can be used by

the appropriate government agency (ies) to obtain

evaluations on a continuing basis of these as well

as future, similar experiments.

There are a number of choices which could be made in obtaining

contractor assistance, ranging from a single overall contractor

to individual contractors for each experiment, or combinations

in between. The choice which has been made is to divide the

overall evaluation program into two procurements, as follows:

a. an evaluation of "agency impact" which will direct

its attention to a description of the experiments,

an assessment of the effects on the several government

agencies, and the design of the related evaluation

system. "Agency impact" includes both immediate and

subsequent effects on the program administrative agency

but also other affected or related government agencies

b. an evaluation of "commercial impact" which will direct

its attention to the immediate and subsequent effects

upon the commercial (or industrial) sector, and, as

appropriate, consumers or users of the products, and

the design of the related evaluation system.

1
RFP 6-35756

The set of "experiments" includes the following:

a. specific procurement experiments - including

"completed," on-going, and planned

b. specific related or supportive experiments - including procedures for selecting experiments, training

of agency personnel, processes for obtaining from

or providing to the industrial sector information

required for carrying out procurement experiments,

and other administrative changes

c. general "experimental" changes - related to or in

support of the above.

The set of experiments may be conveniently grouped into broad

categories according to these dimensions:

a. by program administrative agency - (1) Federal

Supply Services (FSS) ; (2) state and local procurement agencies (S&L) ; and (3) Veterans Administration

(VA)

b. by type of procurement experiment (1) life cycle

costing (LCC) ; (2) value incentive contracting (VIC)

(3) performance specifications; (4) multi-year awards;

etc

c. by the kind of effect intended - (1) economic or performance advantage to the using government agency;

(2) related or derivative improvement in procurement

capabilities of the program administrative agency;

(3) increased or improved technological innovation

in the industrial sector; and (4) economic or performance advantage to civilian sector users.

To obtain the evaluation desired we propose to let two contracts,

each for a three-year period, divided into three phases, as

follows

Phase One - to include, generally, three activities

(a) preliminary systems analysis

(b) evaluation of specific, selected procurement or

other experiments

(c) design and pilot test of the related evaluation

system

2
RFP 6-35756

Phase Two - to include , generally:

(a) refinement of (a) above

(b) evaluation of additional specific procurement

or other experiments

(c) refinement and prototype test of the related

evaluation system

Phase Three - to include, generally, "turn key" implementation of the related evaluation system by

the appropriate government agency

The complete procurement package has been designed to assure the

fullest possible interchange of information between prospective

bidders and those responsible for the evaluation of proposals.

Because of this it will be necessary to examine the complete

set; particular attention is directed to the fact that the

Statement of Work and the Schedule must be read in conjunction

with the accompanying documents.

)

3

RFP 6-35756

Some Definitions

Note: In most cases, the context, or reference to other parts

of the bid set and associated documents, should resolve

ambiguities or otherwise clarify definitions of terms. These

"definitions" are provided as a convenience in reference.

ETIP - Refers to the Experimental Technology Incentives Program. In context, the reference may be to the objectives or

content of the program, to the office or organization itself

(e.g., as contracting agency), or to individual staff members.

PAA - Refers to the Program Administrative Agency (or Agencies) . There are, for these procurements (agency impact and commercial

impact), three PAA, as follows: 1) Federal Supply Service

(FSS) ; 2) state and local procurement agencies (S£L) ; 3) Veterans' Administration (VA) . In context, the reference

will usually be to the specific procurement office (s) and/or

responsible personnel in the agency which have the direct

authority and/or responsibility for the decision to introduce the specific experimental intervention and, usually,

for the conduct of the program and/or function to which the

experimental condition is applied. For FSS, this may include

both central and regional procurement functions or offices.

For SSL , this may include NASPO, NIGP, and/or specific state

and local procurement agencies. For VA this may include both

central and regional or local procurement agencies.

AGENCY IMPACT - Refers to effects or impacts on or in the PAA

and, where applicable, closely related governmental agencies,

and, particularly, agencies which are the users of the products

procured. These impacts or effects include not only those

which are the specific objectives of a specific or general

experiment, but also significant other effects, such as

changes in administrative policies and procedures.

COMMERCIAL IMPACT - Refers to effects upon the commercial

sector and/or civilian sector users. These impacts or effects

include not only those which are the specific objectives of a specific or general experiment, but also significant other

effects, such as increased or earlier use of new technology.

EXPERIMENT - Refers, generically, to any combination of

intervention and desired (or hypothesized) effect. In a complex of interventions and effects, there may be a number

of "experiments," including experiments which are combinations of other experiments, and experiments where the effect

of one is the intervention of another.

4
RFP 6-35756

SPECIFIC EXPERIMENT - Refers, usually, to a specific, identified

intervention and desired (or hypothesized) effect. The major,

and most significant, cases are the specific procurement experiments (e.g., use of LCC to procure air conditioners). There

may be other (related) specific experiments (e.g., the FSS training program)

VARIABLE - Refers, generically, to any object, event, state,

value, function, etc. In the context of a specific experiment

(or study), the independent variable is, usually, the intervention, and the dependent variable the desired (or hypothesized)

effect

PARAMETER - Refers to those variables which, whether controllable

(directly or indirectly) or not, affect (or are hypothesized

to affect) plausibly and/or significantly the interpretability

and/or credibility of the observed (or hypothesized) relationship between the variables in a specific experiment.

A PRIORI PROPOSITION TESTING - Refers to those specific experiments for which one of, if not the primary, purposes is to

establish, with a relatively high degree of credibility, the

relationship between the intervention and the effect observed.

In practice, this will be largely limited to relatively well

defined interventions and proximal effects, and where

reasonable control of parameters can be achieved.

EXPLORATORY (AND/OR DESCRIPTIVE) - Refers to experiments (and/

or studies) where it is either not feasible or not required

to meet the requirements of a priori proposition testing. In

practice, this will apply to requirements to describe and/or

measure sets of variables where either the description is

sufficient itself or provides a basis for identifying hypotheses,

variables, and/or parameters.

EVALUATION SYSTEM - Refers to the set of policies and procedures

which provides the basis for evaluating the (agency and/or

commercial) impacts of a set of specific procurement or related

experiments. The form, detail and completeness of the system

will progressively change. In Phase One, it may begin as a preliminary outline within which detailed "single thread"

designs are developed for the early evaluation of specific

procurement experiments; later, the results of the preliminary

systems analysis and the "pilot test" will provide the basis

for a preliminary evaluation systems design. In Phase Two,

progressive refinement should result in a relatively complete

evaluation system which can be tested as a prototype, in part,

through the evaluation of specific experiments. In Phase

Three, the evaluation system should be in the form of a stable

and complete system.

5
RFP 6-35756

BASIC (OR BASE-LINE) EVALUATION PROCESS - Refers to that part

of the evaluation system which includes those evaluation

activities which deal with key, predictable, identifiable,

continuing, common objectives for which the process can be

institutionalized

SPECIAL EVALUATION PROCESS - Refers to those evaluation activities which do not meet the above requirements because of

specialized, one-time or changing objectives.

FRAMEWORK OR MODEL - Refers to graphical or other conceptual

representation of the set of interrelations among objectives,

organizations, experiments, and/or programs, and the evaluation system.

6

REP 6-35756

PROPOSAL PREPARATION PROCEDURE

A. Purpose

The purpose of this document is to provide a guide for

bidders in the preparation of their proposals.

B . Basis for Preparation of This Document

B.l Historical Note

A wide variety of methods have been used for the procurement

of the services of evaluation contractors, including the

following: 1) adding evaluation as an express or implied

requirement to the basic contract for the program or experiment; 2) contracting separately with a contractor or consultant

for the evaluation; 3) obtaining an evaluation, usually post

hoc , as part of the study phase of a contract for a new or

different program. The description of the services desired

may take many forms, including the following: 1) a brief,

general requirement "to perform an evaluation"; 2) statements or descriptions of various lengths and with various

degrees of detail and completeness which outline the objectives of the program and/or the specific questions to be

answered; 3) relatively detailed and complete "specifications"

of the services required, including instruments to be used,

sampling plans, and the form of the analysis required.

In some cases, and for some purposes, these methods provide

a satisfactory base for defining the services required,

which, in turn, may provide a satisfactory basis for determining the qualifications of prospective contractors.

None of the above appear appropriate as a basis for determining the qualifications of prospective contractors on this

procurement. For this reason this document was prepared,

using a "parametric factor analytic approach" which was

developed for and used on a number of subsystem procurements

by another agency of the government.

7

i
RFP 6-35756

B.2 Basis for Evaluation (of Bidders)

Generally, evaluation (including proposals) requires three

things: first, a standard or base for comparing; second,

information to be compared against the standard; and, third,

an effective method for doing the comparing. These are

discussed in the three paragraphs below.

B.2.1 The first requirement of the evaluation of competitive

proposals is a standard or base for comparing, and this is,

obviously "who can do the job best." but this is a matter of

predicting because it is a future matter, and there is

uncertainty in specifying the job. The choice of standard

for selection includes:

a. The whole program, either as it is or as it should

be stated to obtain the selection base.

b. The "significant independent variable" - if one

part will determine, either by its absence or

presence, the one bidder who can do it, then

this is the basis.

c. Some sample of parameters (variables or "factors")

which on a presence or absence basis, or qualitative

scale, is the determinant.

The choice is between b) and c) because of the inability to

precisely define the skills and content of the future

completely; and because agreement on b) presumes considerable

confidence in past experience, the basis must be c) . The

development of c) is summarized in the paragraph below in

terms of a brief statement of the program.

The nature of the item to be procured determines the program.

The evaluation here is not limited to fully designed evaluations of specific experiments, nor is the alternative of a broad, overall descriptive case study sufficient. It is

proposed to obtain not only preliminary systems study which

will include both of the above, but also both kinds of

evaluations; in subsequent phases, it is proposed to

develop and test an ongoing capability to carry out such

evaluations in such form and detail as will allow the

government to implement subsequent phases. Direct

description of such a program must necessarily be accomplished

by selecting and tabulating the significant parts of the

program; and these parts may then be further defined and

amplified by supporting data. These parts, identified as

"factors," are the bases for the establishment of the

standard; and, in turn, the information to be furnished

and the evaluation procedures.

8
RFP 6-35756

B.2.2 The information to be compared against the standard

is that furnished by, or about, the bidders. This requires

cooperation, varying from a little to a lot. Too little

occurs when the bidder's data is absent or cannot be translated into a common dimension or is irrelevant. Too much

occurs when the bidder essentially repeats back the guidelines furnished. The problem of "too much" has not been

experienced in prior evaluations, and those few cases of

"parroting" were obvious. The concern is to assure that

the bidder has thorough guidance in preparing the necessary

information, and by this means the bidder will be aided

in directing his efforts more efficiently.

B.2.3 Because of the large number of factors to be evaluated, the large volume of data to be considered, and the

size of the evaluation team, the process needs to be well

organized and supported with efficient and clear procedures.

Forms and instructions must be prepared, and arrangements

for scheduling meetings and for carrying out the evaluation have to be thought out in advance. A corollary benefit

is an increase in the assurance of impartiality through

avoidance of the confusion of setting rules and judging

at the same time.

B.3 Preparation

This document, and the associated statement of work, was

prepared upon the above basis. As source material,

procedures similar to this, as well as a number of

other evaluation procedures, were reviewed.

9
RFP 6-35756

C . General Comments on Proposal Evaluation Procedure

To provide bidders with a general background and perspective,

a brief outline of the overall procedure is furnished below.

(It should be noted that these comments are intended only as

a general description, and some changes may be made.)

C.l Preparation for Proposal Evaluation

C.1.1 To assure continuity, personnel responsible for setting

up and administering the program for which proposals are being

solicited, will conduct the proposal evaluation, with the

advice and assistance of others who are less directly concerned. Direct management responsibility is centered in ETIP

in the Contracting Officer's Technical Representative (COTR)

He, together with other members of the ETIP staff, will be

working closely with key members of the staff of the program

administrative agency in the type of close relationship

which is characteristic of "administrative experimentation."

As a minimum, it is intended that at least two members of

the proposal evaluation team will be assigned primary responsibility for each subfactor, with backup assistance in

review by other team members. These evaluation team members

are responsible for preparing and/or reviewing the bases

for the evaluation and for briefing the bidders.

C.1.2 The results of the preparation are incorporated,

primarily, in the RFP. In addition to the Form 33 and

accompanying provisions, the RFP includes a series of

related documents prepared for the primary purpose of

furnishing a base for the preparation of proposals. T’he

necessary clarification and revision to provide a contractual basis will be accomplished by negotiation with the

selected contractor. Briefly, these documents are

grouped as follows:

Statement of Work - tabulates the significant parts

of the program with clarification limited primarily

to a brief summary. This will include those separately identifiable "deliverables" and a schedule.

Proposal Preparation Procedure (this document) - provides

an explanation of the basis for proposal preparation.

Proposal Evaluation Factors - provides the basis for

proposal preparation.

Scope of Effort - cost proposal requirements

Other Documents - development plans, reports, references

etc., to assist the bidders in preparing their proposals.

10
RFP 6-35756

C. 1.2.1 To the extent possible, all of these documents follow

the same order. The Statement of Work includes items of

required work divided into three phases. The Proposal

Preparation Procedure is designed to match the organization

of the Statement of Work. It is intended to provide, where

necessary, cross-reference to the other documents.

C.l.2.2 To assure that all bidders have complete sets of the

related documents, and especially those included under

"Others" which may be furnished at the briefing, a complete

index of all documents will also be furnished during the

briefings, as required.

C.1.3 It is planned to brief the bidders twice. At the

initial briefing, a summary presentation on the program

will be given, including a presentation by the program

administrative agency on the administrative objectives

for the program. (To assure the continuity specified above,

representatives of the program administrative agency will

participate in the evaluation.) Following the presentation,

an extensive question period will be scheduled. The second

briefing will follow a few weeks later after the bidders

have had an opportunity to study the program; this is primarily

to allow for further questioning by the bidders to correct

oversights and resolve ambiguities. No information will

be available during the proposal preparation period on an

individual basis.

C.2 Proposed Evaluation Procedure

C.2.1 After receipt of the written proposals, the proposal

evaluation team will have a period of a few days to

familiarize themselves generally with the proposals; and,

specifically, the parts within their responsibility. This

is primarily to furnish a basis for the next stage.

C.2.According to a schedule to be established at the second

of the two briefings noted above, each bidder will be invited

to make an oral presentation, approximately one hour in

length, following which the evaluation team will have two to

four hours for questioning. Purpose of the questioning is

to assure that the evaluation team understands the proposals,

and has an adequate basis for evaluation on each sub-factor.

Both weak and strong points will be explored, and check lists

will be recorded to assure that team members may have available

to them both the written and oral presentations.

11
RFP 6-35756

C.2.3 Following the above will come a further period of

review, during which individual team members will complete

their individual evaluations of sub-factors for which they

are responsible or concerned. These individual evaluations

will be noted with the assistance of a check list based on

a value rating scale to avoid the confusion caused by

numerical rating.

C.2.4 Members of the proposal evaluation team concerned with

each factor will meet and perform a joint evaluation. They

will follow an established procedure entitled, "Factor Team

Guidelines," to assure an orderly and efficient process.

Briefly, they will first compare notes informally to assure

common standards and information; second, they will arrange

the bidders in descending order; third, they will establish

the relative spread among the bidders; and, finally, their

result will be permanently recorded and signed by all team

members present. This record becomes part of the permanent

evaluation files. Generally, they will rate on each subfactor, as applicable, three central points:

a. Comprehension or understanding of the problem

presented

b. Proposed approach (or, in the alternative, the

means for establishing the approach)

c. Present or potential capability to accomplish this

Evaluation team members may use information obtained from the

RFP and associated documents, the bidders' written and oral

proposals and independent or prior information. However, in

the latter case, any significant or critical information must

be considered by all members of the factor team. Emphasis

will be placed on the sub-factor being rated, but the impact

of other factors will not be ignored. Comparative weighting

of individual sub-factors and the weight of each factor team

members ' s views will be within the responsibility of the

factor team.

C.2.5 The results of the factor team evaluations will be

summarized and reviewed.

C.2.5.1 The summary process will include the necessary combining

of the individual factor ratings to achieve an overall rating,

with consideration of the interaction of factors, their individual significance (acceptability) , and the "summation"

according to previously established weights.

12
RFP 6-35756

C.2.5.2 In general, the range of weights among the factors

will be no more than 2:1.

C.2.5.3 The review process will necessarily provide for

latitude in reflecting the results of the review, such as

reassessing of weights or reexamination of specific areas

of the evaluation; and, such changes will be incorporated,

with the supporting basis in the record of the evaluation.

C.3 Comment on Factor and Sub-factor Structure

All of the factors in the evaluation are set out in the

Proposal Evaluation Factors documents, and are largely

self-explanatory. The rationale of this particular

organization of the sample is based, primarily, on the

sub-factors, the factors being administrative groupings

for convenience in organization of the various documents

and the evaluation. Generally, the factor/sub-factor organization is based on significant identifiable effort/capability

areas. The intent is to select comprehensive exclusive/

inclusive samples of the total program. In total, the

factor/sub-factors are intended to include samples of all

significant parts of the program. An effort has been made

to minimize duplication; and, with a few intentional

exceptions, no sub-factor samples an area covered by another

factor. Similarly, with certain intentional exceptions,

each sub-factor samples on the basis of understanding,

approach, and capability; as a result, those aspects, which

include "effective experience," are incorporated into specific

substantive areas rather than amorphous generalized categories.

It is important to understand that the distinction among the

factors is primarily an administrative distinction to reflect

the differences in perspective required in presentation and

evaluation

13
RFP 6-35756

D. Use of this Document

D.l General Considerations

D.1.1 As noted above, the Proposal Evaluation Factors document covers all of the factors to be considered in the

evaluation and are intended to provide a substantially

complete framework for the preparation of the proposal.

Because the background and perspective against which they

were prepared is set out in this part, the other related

documents, and the briefings for the bidders, it is

extremely important that this interpolation be understood

and preserved during the preparation period by assuring that

substantive contributors have sufficient access to the

overall framework. As a general guide, cross reference

should be made to the applicable points included in the

document on Scope of Effort.

D.2 Content Considerations

D.2.1 Because the evaluation is organized on a factor/subfactor basis, it is important that the treatment of each

factor be substantially complete as possible. The several

factors are necessarily interrelated and interacting, but

repetition is not required; however, where significant

material considered necessary to assure understanding of

a factor is contained in another factor, specific (and

annotated) cross-references should be furnished. All members

of the evaluation team will have access to the complete

proposal, and it is expected that certain factors will be

reviewed jointly.

D.2. 2 Each factor and sub-factor is defined by a specific

descriptive sentence setting out the standard against

which each bidder will be measured. Because the factor

statements reflect the fact that the factor is primarily

an administrative convenience, no specific presentation

with regard to the overall factor is required. Each subfactor, however, will be the subject of a specific rating,

and the proposal content should reflect the necessity for

furnishing the evaluation team with sufficient information

to establish a comparative position on each sub-factor

THE BASIC ORGANIZATION OF THE PROPOSAL MUST BE ON A SUBFACTOR BASIS.

14
RFP 6-35756

D.2.2.1 It should be specifically noted that the sub-paragraphs

under each sub-factor are provided for the purpose of amplifying

and defining the standard, that is, to present the types of

questions which need to be answered to establish the rating on

the sub-factor. It is not required to answer each and all of

these subsidiary questions. Nor is it required that the

presentation be limited to these particular subsidiary questions.

In some cases, a subsidiary question will be a critical determinant; in other cases, an effective presentation may be

accomplished by partial treatment. It was not intended to

specify the proposal content through these subsidiary questions.

D.2.2.2 These subsidiary questions are presented in three forms.

One form is that of a question, such as, "Do you consider the

proposed schedule realistic? too tight? If not, what do you

propose and why?" The second general form is that of a statement calling for comment or the furnishing of certain information, such as, "Testing facilities (floor space, equipment,

personnel)." The third general form is that of a statement

commenting on a problem area or outlining a proposed solution

which furnishes a basis for comment, such as, "The Advisory

Committee may raise a technical question which requires

investigation in order to allow the committee to make a decision

at the next meeting. The contractor will furnish the necessary

personnel to develop each of the conflicting points of view

independently.

D.2.2.3 As noted elsewhere, the relative weighting, or significance, of each sub-factor within a factor is within discretion

of the factor ream performing that part of the evaluation. In

most cases, the sub-factors represent approximately equivalent

areas of significance. The amount of subsidiary questions

furnished does not necessarily reflect an unusually high or

low weight.

D.2.3 Latitude is encouraged in the treatment of the content

presented with respect to the sub-factors. Where alternate

assumptions or conclusions to clarify ambiguities or fully

establish the position is considered advisable, no arbitrary

limitation is imposed. The substantive objective is to provide

information as a basis for evaluation, and this procedure

is intended only as a minimum framework to increase the

efficiency and effectiveness of the proposing and evaluating

process

D.3 Physical Considerations

D.3.1 Primary concern with regard to the physical format and

organization should be to facilitate the work of the evaluation team, recognizing the circumstances and procedures which

will be used.

15
RFP 6-35756

D.3.2 It is suggested that the proposal use the same major

numerical and subject headings of the Proposal Evaluation

Factors document to avoid the confusion of an additional

numbering system. This should be extended only to the

factors and sub-factors, and no requirement to follow the

numerical order of the subsidiary questions within a subfactor is intended. Where it is considered desirable to

include additional material within the framework of Sections

I through VII, an expanded numbering sequence should be

used, i.e., to include material after 1.1, but before 1.1.1,

use a new number, 1.1.0, with subsidiary numbers, 1. 1.0.1,

I . 1 . 0 . 2 , etc

D.3.3 To facilitate the physical handling of the proposal,

certain physical divisions are requested.

D.3.3.1 Physically bulky material, such as extensive

personnel or facility brochures, or supporting descriptive

documents furnished in amplification, should be treated as

physically separate appendices, with suitable identification.

In this regard, where material which has been prepared in

another connection is furnished, a suitable cover sheet

noting limitations of application may be used to avoid

extensive revision and republication.

D.3.4 Provision should be made for sufficient indexing and

cross-reference summaries to facilitate reference to

particular parts of the proposal. Unless considered

desirable, pagination need not be accomplished. It is

recommended that covers of separate documents contain

clear reference to the numerical and subject heading.

16
RFP 6-35756

REVISIONS TO PROPOSAL PREPARATION PROCEDURES

A. Page 5. -Delete C.2.2, and substitute the following:

C.2.2 Where factor team members require specific

additional information for their evaluation, specific

written questions will be directed to the specific

proposer through the contracting officer.

B. Page 6, C.2.3, lines 6 and 7. Place period (.) after

"scales," and delete the remaining material.

C. Page 6, C.2.4, lines 24 to 27. Delete "Comparative

weighting" and add "Comparisons," and delete "weight"

and add "efforts."

17

EVALUATION

SYSTEM

DESIGN

PROCESS

(INCLUDING

RELATION

TO

EVALUATIONS

OF

SPECIFIC

EXPERIMENTS)

19

RFP 6-35756

Statement of Work

Note: The items listed below provide a catagorized summary of the

services and materials required on this program.

Phase One: Planning and Design, and preliminary evaluation.

Item 1. Plan, organize, staff, direct, and control, including

both schedule and costs.

Item 2. Provide reports, liaison, and services for purposes

of review, coordination, and approvals and acceptances, including the following:

a. Periodic and special formal and informal

management progress reports, final report.

b. Periodic and final scheduling reports,

including projected schedule for the next phase.

c. Periodic and final cost reports, including

projected costs for the next phase.

d. Periodic and special formal and informal

technical progress reports and final report,

including proposed program for the next phase.

(Note: the substantive work to be reported

here is to be performed under items 3 through

7) . Item 3. Review and structure the proposed objectives in

terms of the experimental setting and related

background, and establish an overall plan and

set of priorities.

Item 4. Review and structure the proposed set of experiments,

and related background in terms of the proposed

objectives, and provide a reference base.

Item 5. Develop the detailed evaluation requirements (problem

definition and experimental or study design) , in

relation to collection and analysis.

Item 6. Develop the detailed data collection process,

including instruments, identification of, and

access to, sources, and managing the process;

and collect the data.

21
RFP 6-35756

Item 7: Develop the detailed data analysis process;

carry out the analyses; present findings;

present recommendations.

Phase Two: Prototype Evaluation Process Testing

Item 8. Same as Item 1.

Item 9. Same as Item 2, but change reference to Items

10 through 14.

Item 10. Refine and revise the work accomplished under

Item 3, as related to the further objective of

designing and testing a prototype of the complete

evaluation process.

Item 11. Refine and revise the work accomplished under

Item 4, as related to the further objective of

preparing a formal framework within which the

objectives, organizations, and programs may be

related to the evaluation process.

Item 12. Refine and revise the work accomplished under

Item 5, as related to the further objective

to achieve a comprehensive, detailed design

to meet both basic data (base line data) and

special data requirements.

Item 13. Refine and revise the work accomplished under

Item 6, as related to the further objective

of collecting both basic and special data.

Item 14. Refine and revise the work accomplished under

Item 7, as related to the analysis of both

basic and special data; and provide findings

and recommendations.

Phase Three: On-line/Turnkey Evaluation

Items 15

thru 21. Revision and refinement of respective prior

items in relation to the additional objective

of an on-line turnkey evaluation process.

22
RFP 6-35756

STATEMENT OF WORK

(with illustrative sub-items for Phase One)

Note: The items listed below provide a categorized

summary of the services and materials required

on this program.

Phase One Planning and Design , and Preliminary Evaluation

Item 1. Plan, organize, staff, direct, and control, including

both schedule and costs.

a. Provide the necessary services and materials for

management of the overall program, including a management plan.

b. Prepare a preliminary detailed schedule, and

provide progressive analysis and rescheduling,

including proposed schedule for Phase Two.

c. Prepare periodic cost analyses and projections,

including proposed costs for Phase Two.
RFP 6-35756

Item 2. Provide reports, liaison, and services for purposes

of review, coordination, and approvals and acceptances, including the following:

Provide services and materials necessary for liaison

with ETIP, PAA, and relevant other individuals and

organizations

a. Periodic and special formal and informal management progress reports, final report

a ( 1 ) . Submit program management plan at or before

completion of 3d month, and report modifications as required.

a (2) . Submit final report at or before completion

of 15th month.

b. Periodic and final scheduling reports, including

projected schedule for the next phase

b ( 1 ) . Submit projected Phase One schedule at or

before completion of 3rd month, and report

modifications as required.

b (2 ) . Submit projected Phase Two schedule at or

before completion of 9th month.

b ( 3 ) . Submit final report at or before completion

of 15th month.

c. Periodic and final cost reports, including projected

costs for the next phase.

c(l) . Submit projected Phase One costs at or before

completion of 3rd month, and report modifications as required.

c (2) . Submit final report at or before completion

of 15th month.

d. Periodic and special formal and informal technical

progress reports, final report, including proposed

program for the next phase. (Note: The substantive

work to be reported here is to be performed under

Items 3 through 7.)

24
RFP 6-35756

Submit reports, including formal reports for the items in the

following table on or before completion of the month (s)

specified

3a 3rd and 6th 6a 6th

b 3rd b -

c 6th c 6th

d 9th d -

4a 3rd and 6th 7a 6 th

b 6th and 12th b 12th

5a 3rd and 6th c 12th

b 6th d 15th

c 6th e 15th

d 12th

25
RFP 6-35756

Item 3. Review and structure the proposed objectives in

terms of the experimental setting and related

background, and establish an overall plan and

set of priorities.

a. Review, through documents and interviews, the

objectives of ETIP, PAA, and other relevant

individuals and organizations, including

professional standards of evaluation, and prepare

a description and supporting analysis suitable

for providing a basis for planning and setting

of priorities for Phase One.

b. Based on the above, identify, and prepare preliminary recommendations and supporting detail

requirements for, those specific evaluations

which should be accomplished during Phase One,

including both exploratory (and/or descriptive)

evaluations and a priori proposition testing

evaluations

c. Based on the above, prepare preliminary statement

of requirements for the pilot evaluation system

in Phase One.

d. Based upon the above and the other activities

in Phase One, prepare a preliminary statement

of the requirements for the prototype evaluation

system in Phase Two, together with supporting

analysis

26
RFP 6-35756

Item 4. Review and structure the proposed set of experiments,

and related background in terms of the proposed

objectives, and provide a reference base.

a. Review, through documents and interviews, the

specific procurement experiments (past, present,

and proposed) , and the related programs and

organizations*, and prepare a description and

supporting analysis suitable for planning

the activities of Phase One in conjunction with

the work in Item 3a.

* (AGENCY IMPACT ONLY) With primary emphasis

on PAA, and not on industry sector.

(COMMERCIAL IMPACT ONLY) With primary emphasis

on industry sector, and not on PAA.

b. Prepare a preliminary framework model which

provides a base for synthesizing objectives,

organization, programs, and the evaluation

process (system), with supporting analysis.

27
RFP 6-35756

Item 5. Develop the detailed evaluation requirements (problem

definition and experimental or study design) , in

relation to collection and analysis.

a. Prepare a preliminary design of both exploratory

(and/or descriptive) and a priori evaluations to

be conducted during Phase One, including 1) statement of problem or hypothesis, 2) supporting

theories, and 3) definitions of variables and

parameters

b. Based on the above, prepare a detailed design of

each evaluation, with special emphasis on specific

procurement experiments, including 1) choice of

experimental design, 2) (where appropriate)

sampling strategies, and 3) data collection and

analysis methods.

c. Prepare a preliminary design of the pilot evaluation system in Phase One, including data collection

and analysis.

d. Based on the above, prepare a preliminary design of

the prototype evaluation system in Phase Two,

including both basic and special processes.

28
RFP 6-35756

Item 6. Develop the detailed data collection process, including instruments, identification of, and access

to, sources, and managing the process; and collect

the data.

a. Design and develop the data collection process

required for the specific Phase One evaluations.

b. Based upon the work in Items 3, 4, and 5, collect

additional data required for specific exploratory

and/or descriptive) evaluations.

c. Design and pilot test the instruments necessary for

specific a priori proposition testing evaluations.

d. collect the data reauired for specific a priori

proposition testing evaluations.

29
RFP 6-35756

Item 7. Develop the detailed data analysis process; carry out

the analyses? present findings; present recommendations.

a. Design and develop the data analysis process required for the specific Phase One evaluations.

b. Carry out the specific Phase One exploratory (and/or

descriptive) evaluations, and present findings.

c. Carry out the specific Phase One a priori proposition

testing evaluations, and present findings.

d. Analyse the activities of Phase One which affect the

design of the evaluation system, and prepare

recommendations

e. Analyse the activities of Phase One, and prepare

recommendations for Phase Two.

30
RFP 6-35756

Item

Item

Item

Item

Item

Item

Item

Phase Two: Prototype Evaluation Process Testing

8. Same as Item 1.

9. Same as Item 2, but change reference to Items 10

through 14.

10. Refine and revise the work accomplished under

Item 3, as related to the further objective of

designing and testing a prototype of the complete

evaluation process.

11. Refine and revise the work accomplished under Item 4,

related to the further objective of preparing a formal framework within which the objectives, organizations, and programs may be related to the evaluation

process

12. Refine and revise the work accomplished under Item 5,

as related to the further objective to achieve a comprehensive detailed design to meet both basic

data (base line data) and special data requirements.

13. Refine and revise the work accomplished under Item 6,

as related to the further objective of collecting both

basic and special data.

14. Refine and revise the work accomplished under Item 7,

as related to the analysis of both basic and special

data; and provide findings and recommendations.

31
RFP 6-35756

Phase Three; On-Line/Turn-Key Evaluation

Items 15 through 21. Revision and refinement of respective

prior items, in relation to the additional objective of an on-line/turnkey evaluation process.

32
PROPOSED SCHEDULE - OVERALL

(Agency Impact & Commercial Impact)

ITEM DESCRIPTION

PHASE ONE

1. PROGRAM MANAGEMENT

2. REPORTS & REVIEWS

3. OBJECTIVES

4. BACKGROUND

5. EVALUATION DESIGN

6. DATA COLLECTION

7. DATA ANALYSIS

PHASE TWO

8. PROGRAM MANAGEMENT

9. REPORTS & REVIEWS

10. OBJECTIVES

11. BACKGROUND

12. EVALUATION DESIGN

13. DATA COLLECTION

14. DATA ANALYSIS

PHASE THREE

15. PROGRAM MANAGEMENT

16. REPORTS & REVIEWS

17. OBJECTIVES

18. BACKGROUND

19. EVALUATION DESIGN

20. DATA COLLECTION

21. DATA ANALYSIS

NOTE: f (FORMAL) PROGRESS REPORT

FINAL REPORT . MINOR EFFORT

COORDINATION OR REPORT WRITING

ONLY

33
Tasks, plus

selected

sub-tasks

PROPOSED SCHEDULE - PHASE ONE

(Agency impact & Commercial impact)

Months

1. PROGRAM MANAGEMENT

Phase Two cost proposal

2. REPORTS & REVIEWS

Design of reporting system

3. OBJECTIVES

Preliminary analysis

Identification of specific Phase One evaluations

Preliminary evaluation system objectives

Evaluation system objectives

4. BACKGROUND

Preliminary analysis

Analysis of relevant organizations

Preliminary framework

5. EVALUATION DESIGN

Analysis of Phase One evaluations

Design of Phase One evaluations

(including sampling strategies)

Preliminary design of evaluation system

(including collection & analysis)

Design of evaluation system

(including basic &. special)

6. DATA COLLECTION

General data collection

Instrument design & pilot test

Data on specific experiments

7. DATA ANALYSIS

Exploratory analysis & findings

A priori analysis & findings

Analysis of evaluation system

Recommendations

1011

12 1 31 1 41 15 j

1

1

'1

7 A
RFP 6-35756

PROPOSAL EVALUATION FACTORS

I . Management

This factor is a measure of the background

and overall management capability of the bidder.

1.1 Evaluation Management Experience

This sub-factor measures the bidder's experience

and/or capability of the personnel he will

use on this program.

1.2 Importance of Program to the Bidder

This sub-factor is a measure of the degree to

which the proposal represents the approval and

direction of the company rather than the efforts

of professional proposal writers, and also is a measure of the warranty offered by the company

to back the program with the necessary resources.

1.3 Program Management

This sub-factor is the measure of the bidder's own

proposed method for planning, organizing, and

controlling the program.

1.4 Schedule and Cost Control

This factor measures the bidder's ability to

predict the time and cost requirements of his

efforts, to plan the interrelation of the several

parts, and to exercise the necessary control.

35
RFP 6-35756

II. 1

II . 2

II . 3

II. 4

Reports and Reviews

This factor is the measure of the bidder's

ability to assure that the government is

able to effectively and progressively review

a very large scale effort with a minimum of

people, and of the bidder's ability to

coordinate his activities with those of

other related individuals and organizations.

Reports and Liaison with the Experimental

Technology Incentives Program (ETIP)

This sub-factor measures that part of the

overall factor which relates to the

Experimental Technology Incentives Program

(ETIP)

Reports and Liaison with the Program Administrative

Agency (PAA)

This sub-factor measures that part of the overall factor

which relates to the program administrative agency

(and, where appropriate, its subelements or related

organizations)

Liaison with Others

This sub-factor measures that part of the

overall factor which relates to individuals

and organizations other than those described

above

Approvals and Acceptances

This sub-factor measures the bidder's understanding

and acceptance of the necessity for progressive

and substantial approval and acceptance during

each phase, as well as at the end of each phase.

36
RFP 6-35756

III

III. 1

III. 2

III. 3

III. 4

III. 5

Objectives

/ This factor is a measure of the bidder's overall

understanding of the objectives of this procurement

(and related programs)

ETIP Evaluation Objectives

This sub-factor is a measure of the bidder's understanding of the specific objectives of this procurement.

Program Administrative Agency (PAA) Objectives

This sub-factor is a measure of the bidder's

understanding of the specific objectives of the

program (s) and/or experiment (s) that are to be

evaluated

Other Relevant Objectives

This sub-factor is a measure of the bidder's understanding of relevant and related objectives, goals,

criteria, standards, and the like, including the

state of the art in administrative experimentation

(and/or evaluation)

Phase Two Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the incorporation of the results of

Phase One in the restated objectives for Phase Two.

Phase Three Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the reformulation of objectives in

Phase Three.

37
RFP 6-35756

IV.

IV. 1

IV. 2

IV. 3

IV. 4

IV. 5

Background Information

This factor is a measure of the bidder's overall

understanding of the background and characteristics

of this procurement.

ETIP

This sub-factor is a measure of the bidder's understanding of ETIP and the specific program (s) and/or

experiment (s) , in terms of past history, present

status and plans for the future.

Program Administrative Agency (PAA)

This sub-factor is a measure of the bidder's

understanding of the program administrative

agency (and, where appropriate, its subelements

or related organizations), its organization,

policies and procedures.

Other Relevant Background

This sub-factor is a measure of the bidder's understanding of the programs, organization, policies

and procedures of other relevant organizations.

Phase Two Background

This sub-factor is a measure of the bidder's

capability to prepare a comprehensive framework

or model of the program(s) and/or experiment (s)

and the evaluation process.

Phase Three Background

This sub-factor is a measure of the bidder's awareness of possible further refinements in the model

necessary for the continuation of the evaluation

process

38
RFP 6-35756

V. Evaluation Design

This factor is a measure of the bidder's capabilities

to carry out the evaluation design requirements.

V.l Exploratory (Descriptive) and A Priori Proposition

Testing Hypotheses, Variables and Parameters

This sub-factor is a measure of the bidder's ability

to define the detailed evaluation objectives.

V.2 Experimental and/or Study Designs

This sub-factor is a measure of the bidder's

ability to define the overall experimental

and/or study design.

V.3 Data Collection Design

This sub-factor is a measure of the bidder's ability

to design the overall data collection process, including

sources of information, sampling strategies, timing,

etc

V.4 Data Analysis Design

This sub-factor is a measure of the bidder's

ability to design the overall data analysis process,

including processing and analysis, and presenting

results and recommendations.

V.5 Phase Two Evaluation Design

This sub-factor is a measure of the bidder's ability

to redefine the detailed experimental and/or study

design to reflect the results of Phase One, the

several objectives, and model of the process in a comprehensive detailed design to meet both basic

data (base line data) and special data requirements.

V.6 Phase Three Evaluation Design

This sub-factor is a measure of the bidder's ability to

further redefine the above, to provide for a detailed

design, and necessary supporting materials, which would

allow an on-going evaluation process.

39
RFP 6-35756

VI.

VI. 1

VI. 2

VI. 3

VI. 4

VI. 5

Data Collection Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

acquisition of data on the variables and parameters

of interest.

Instrument Design

This sub-factor is a measure of the bidder's ability

to develop (and test) the various questionnaires,

protocols, schedules, and the like required for data

collection.

Information Sources

This sub-factor is a measure of the bidder's

ability to identify the various information

sources, and to gain access.

Data Collection

This sub-factor is measure of the bidder's capability

to organize, staff, train, direct and control the

personnel who obtain the data by survey, interview,

observation, or use of records.

Phase Two Data Collection

This sub-factor is a measure of the bidder's ability

to anticipate the requirements of the Phase Two

collection of both basic (base line) and special

data

Phase Three Data Collection

This sub-factor is a measure of the bidder's ability

to plan against the requirement to design an

ongoing collection process.

40
RFP 6-35756

VII

VII . 1

VII . 2

VII . 3

VII . 4

Data Analysis Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

analysis of the data and to present the results and

recommendations

Analysis

This sub-factor is the measure of the bidder's ability

to identify and apply the appropriate analytical

(including statistical) techniques to the data

collected

Exploratory (and/or Descriptive) Findings

This sub-factor is a measure of the bidder's

capability to identify and present exploratory

findings

A Priori Proposition Testing Findings.

This sub-factor is a measure of the bidder's

capability to identify and present a priori

f indings

Recommendations

This sub-factor is a measure of the bidder's

understanding of the relationship between the

results of Phase One and the work in Phase Two,

in terms of additional, revised, or deleted

hypotheses, changes in objectives or background

information, and the relative feasibility and/or

usefulness of alternative designs for evaluation.

41

RFP 6-35756

2/1/76

PROPOSAL EVALUATION FACTORS

(With Illustrative Questions)

Note: These factors and sub-factors apply to both agency

and commercial impact proposals. Where specific

illustrative questions apply primarily to either agency

or commercial impact only, they are so identified.

I. Management

This factor is a measure of the background

and overall management capability of the bidder.

1.1 Evaluation Management Experience

This sub-factor measures the bidder's experience

and/or capability of the personnel he will

use on this program.

1.1.1 Discuss, with specific examples, recent experience in the

management of field experiments, projects, and/or programs.

Indicate size and complexity of program, extent of own responsibility for design of the program (or experiment) and the evaluation. Indicate by tabulation of cross-references other subsections of this proposal where you have referred to any program

discussed here. Outline the original contracted-for schedules

and costs and requirements, and explain changes. If you have

had any "bad" experience here, discuss the reasons and, if

appropriate, indicate how you would prevent them from occurring

on this program. If appropriate, you should provide the

names, addresses, and telephone numbers of cognizant COTR's.

1.1.2 If you do not have the kind of experience discussed above,

discuss your basis for believing that you can meet the requirements of this program. Discuss in terms of the personnel you

intend to use.

1.1.3 How will you combine and coordinate the efforts of

staff members who are familiar with the procurement process,

the program administrative agency, or the commercial sector

with those staff members who have the technical skills in

evaluation?

42
RFP 6-35756

1.2 Importance of Program to the Bidder

This sub-factor is a measure of the degree to

which the proposal represents the approval and

direction of the company rather than the efforts

of professional proposal writers, and also is a measure of the warranty offered by the company

to back the program with the necessary resources.

1.2.1 What is the relative size of this program with respect

to the overall sales of the company? If your company has

significant decentralization or there are other factors which

should be presented for clarity, please furnish this background. If the proportionate size is relatively large, why

do you think that the problems associated with the "single

product" will not arise? If, on the other hand, the proportionate part is relatively small, what assurance is there

that this program will not be downrated by others after the

contract is signed?

1.2.2 Discuss, if appropriate, how this program fits in with

the long range planning of your company. What is its significance to your future position? If you envisage an increase in

programs of this type, what assurance is there that key personnel

on this program will not be withdrawn for newer programs?

1.2.3 It should be noted that the extent and nature of your

effort in proposing will be considered in the evaluation of

this sub-factor.

43
RFP 6-35756

1.3 Program Management

This sub-factor is the measure of the bidder's own

proposed method for planning, organizing, and

controlling the program.

1.3.1 It is proposed that the planning and carrying out

of the preparation of this proposal provides an opportunity

for the bidder to demonstrate his capability in planning a complex effort. Please describe briefly your process and,

particularly, the extent to which it is a "pilot" of the

process you will use in managing this program.

1.3.2 Detail your proposed organization by time phases,

including buildup, division of responsibility; generally

describe its operation, how it is organized, and special

features not self-explanatory. Can the effectiveness of

your proposed organization be projected from its performance

in putting together this proposal? If the organization you

propose is essentially "new," either in form or in the

persons assigned, comment on the extent to which you have

been able to "test" it out in preparing this proposal.

44
RFP 6-35756

1.4 Schedule and Cost Control

This factor measures the bidder's ability to

predict the time and cost requirements of his

efforts, to plan the interrelation of the several

parts, and to exercise the necessary control.

1.4.1 A detailed, thorough schedule provides not only a comprehensive plan but also a powerful method of controlling

and reviewing the progress of the program. The proposed

schedule does not include a detailed breakdown of all of

the interrelated areas but does furnish a broad guide for

planning purposes. In general, do you think the schedule

is realistic? Is it too tight? What additional assumptions

are required? What conditions or events will affect the

schedule?

1.4.2 Discuss the method of preparation of the schedule you

will use, how you will update it, how you will use it.

1.4.3 Prepare a supporting analysis and summary of any areas

which you believe reflect a significant relative cost advantage

or disadvantage on your part. Indicate to what extent your

in-house or subcontracting plans will take advantage of

in-being capabilities and at what potential savings. To

what extent have provisions for contingencies been made?

1.4.4 Discuss the form and timing of your detailed cost

proposal for the major field data gathering activity in

the second and third phases.

1.4.5 While the overall schedule reflects the requirement

to design an on-going evaluation system, there is an

important requirement to provide during Phase One (and each

subsequent phase) specific outputs in the form of evaluations

of selected specific experiments. How will you assure this?

1.4.6 The timing of the experiments varies across the several

PAA's. What advantages or disadvantages do you see in this?

1.4.7 It is anticipated that another ("agency impact")

contractor will be primarily responsible for evaluating the

intervention and proximal impacts of the several experiments;

it is also anticipated that some, if not all, commercial

impacts will substantially lag the experiments. What advantages

or disadvantages do you see in this? (COMMERCIAL IMPACT ONLY)

45
RFP 6-35756

II. Reports and Reviews

This factor is the measure of the bidder's

ability to assure that the government is

able to effectively and progressively review

a very large scale effort with a minimum of

people, and of the bidder's ability to

coordinate his activities with those of

other related individuals and organizations.

II. 1 Reports and Liaison with the Experimental

Technology Incentives Program (ETIP)

This sub-factor measures that part of the

overall factor which relates to the

Experimental Technology Incentives Program

(ETIP)

II. 1.1 Discuss generally the extent and nature of liaison

with both the Contracting Officer (CO) and the Contracting

Officer's Technical Representative (COTR) that you envision

in this program. Discuss generally the information needs of

both government and contractor personnel; who will establish

what the needs are and how they will be satisfied?

II. 1.2 It would be useful if the reporting system were

designed in a way that facilitated an ongoing process

evaluation of a) the experiments conducted by the program

administrative agency, and b) the evaluation effort of the

contractor himself. The data provided by such a system could

serve as the basis for a descriptive case history of the

experiments and their evaluation. Discuss generally how

feasible this proposal is, and how it might be accomplished.

You may wish to relate it to your discussion in IV. 1.2, and

comparable discussions.

46
RFP 6-35756

II. 2 Reports and Liaison with the Program Administrative

Agency (PAA)

This sub-factor measures that part of the overall factor

which relates to the program administrative agency

(and, where appropriate, its subelements or related

organizations)

11. 2.1 Discuss generally the extent and nature of liaison with

both management personnel and technical or specialized staff.

How will you reconcile the desirability of direct, informal communication with the necessity, in at least some cases, for

observing formal clearance and approval "through channels"?

11. 2. 2 The evaluation effort may sometimes be perceived as

threatening by various personnel in the program administrative agency. Discuss how the reporting and liaison arrangements can be developed to reduce the likelihood of this.

11. 2. 3 Discuss the differences, if any, in how you will relate

to the several PAA. To what extent will these be separately

identifiable efforts?

11. 2. 4 It is anticipated that another ("agency impact")

contractor will be primarily responsible for coordination

with the several PAA. What do you think of this, and how

will you deal with it? (COMMERCIAL IMPACT ONLY)

47
RFP 6-35756

II. 3 Liaison with Others

This sub-factor measures that part of the

overall factor which relates to individuals

and organizations other than those described

above

.

11. 3.1 It is anticipated that there will be related programs

and program evaluations being carried out by the government

agencies either in-house or through the use of an outside

consultant or contractor. Discuss the nature and form of

the relationship you see with them. How will you ensure

that the necessary coordination, avoidance of duplication,

overlap, or gaps, etc., is accomplished?

11. 3.2 To some extent, it is expected that individual companies

(e.g., manufacturers of products which are the subject of program experiments), industry associations, and others may desire

or require a liaison relationship. Comment.

11. 3. 3 Reporting and liaison relationships may arise with

individuals and organizations with professional interests

(e.g., professional societies, conferences, professional

journals, consumer or public interest groups) in the program or program evaluation. Discuss.

11. 3.4 It is anticipated that another ("commercial impact")

contractor will be primarily responsible for coordination with

individual companies, etc. Comment. (AGENCY IMPACT ONLY)

11. 3.5 It is anticipated that the bidder will be primarily

responsible for coordination with individual companies, etc.

Comment. (COMMERCIAL IMPACT ONLY)

48
RFP 6-35756

II. 4 Approvals and Acceptances

This sub-factor measures the bidder's understanding

and acceptance of the necessity for progressive

and substantial approval and acceptance during

each phase, as well as at the end of each phase.

II. 4.1 It is intended that the formal approval and acceptance

of the work of the contractor for each phase will be based,

primarily, upon the several reports submitted during and

at the end of the phase, with final approval at the end of

the phase. It is intended, however, that this final approval

and acceptance will be largely a formal confirmation of

already established acceptable performance and/or results.

This is required not only to overcome the substantial delays

of tandem or sequential review, but also to avoid rework,

redesign and argument under pressure of time and after the

fact. You should discuss your understanding of this, the

advantages and disadvantages, and your willingness to accept

the contractual and schedule implications, that is, that this

means that the delivery dates are dates of delivery of

approved items, not merely dates for submission of items

with an open time term.

49
RFP 6-35756

III. Objectives

This factor is a measure of the bidder's overall

understanding of the objectives of this procurement

(and related programs).

111.1 ETIP Evaluation Objectives

This sub-factor is a measure of the bidder's understanding of the specific objectives of this procurement.

111. 1.1 The specific objectives for this procurement, and

related objectives, including the overall objectives of

ETIP, appear in various forms in the several parts of this

bid set, and in other sources, and reflect a variety of

authors, circumstances and purposes. It is proposed that

it is neither necessary nor desirable for ETIP to prepare

a single, integrated, fully rationalized, and "authoritative"

statement of the objectives for the guidance of the bidders

(or the successful bidder) . Comment on why you believe this

is (or is not) justified?

III.

1.2

It is expected that the various stated objectives

may not appear (or be) consistent; how will you handle this?

For example, a high credibility (a priori proposition testing)

evaluation of a specific experiment is desired but it is not

possible to achieve this (with reasonable time and money)

For example, a complete and candid description of how a particular experiment was carried out may reflect gross

incompetence or dereliction on the part of specific ETIP

(or program administrative agency) personnel. For example,

the results of a particular evaluation may disclose a systematic defect in the past performance of the agency

(or a contractor)

111. 1.3 It is anticipated that, for a particular experiment,

there may be as many as four parallel objectives: a) to

describe what happened, what went well, what went wrong;

b) to determine whether the result was "caused" by the

experimental manipulation; c) to demonstrate the ability

to "evaluate" scientifically with credible results; d) to

claim credit for the experiment and/or the results. How

does one determine the relative importance of these several

objectives for a specific experiment over the set of all

comparable experiments?

111. 1.4 It is anticipated that ETIP evaluation objectives or

priorities may shift during the course of the evaluation

effort. How does one ensure that he will be sensitive to

such shifts and how does one deal with such shifts? How would

you go about improving the clarity of ETIP's evaluation

objectives?

50
RFP 6-35756

III. 2 Program Administrative Agency (PAA) Objectives

This sub-factor is a measure of the bidder's

understanding of the specific objectives of the

program(s) and/or experiment (s ) that are to be

evaluated

III. 2.1 Questions comparable to those suggested under III.l

may appear here. Comment. What special problems, if any,

do you anticipate because of the multiple PAA?

III. 2. 2 How will you reconcile conflicts between ETIP and

the program administrative agency objectives for a particular experiment? PAA objectives for the experiment and

other PAA objectives? What kinds of problems will come

up? How do you plan to deal with them?

III. 2. 3 It is anticipated that, during the period that the

work on this contract is accomplished, there may be

changes in policy or procedures or of key personnel of the

program administrative agency. How will you deal with this?

For example, the "results" of a particular experiment may be

sufficiently "obvious" to the agency to no longer make

necessary any further evaluation. Will a continuation of

the evaluation be undertaken under any circumstances? And,

if so, how would you justify it and/or obtain agency cooperation?

III. 2. 4 Suppose the agency adds new objectives or objectives

not presently contemplated are uncovered which represent

potentially significant changes in the direction or scope

of the present evaluation program? How will you react to

this?

III. 2. 5 It is anticipated that another ("agency impact")

contractor will be primarily responsible for examining the

objectives of the PAA. Comment. (COMMERCIAL IMPACT ONLY)

51
RFP 6-35756

III.

3

Other Relevant Objectives'

This sub-factor is a measure of the bidder's understanding of relevant and related objectives, goals,

criteria, standards, and the like, including the

state of the art in administrative experimentation

(and/or evaluation)

111. 3.1 It is proposed that the objectives of ETIP and PAA

do not exist in isolation, that both specific and general

objectives of a wide variety of other individuals, organizations, and larger communities will necessarily interact with

the present program. For example, personal and career objectives of key individuals. For example, political, social, and

legal objectives of the federal government. For example,

objectives expressed or implied in the state-of-the-art and

professional standards in experimental and evaluation research.

Of what significance are these, and how do you expect to deal

with them?

111. 3.2 Your understanding of the state-of-the-art,

particularly in administrative experimentation and/or

evaluation, will be measured by your discussion of relevant

questions raised elsewhere. If you consider it appropriate,

you may discuss this here, and you may wish to cross-reference

significant material which appears elsewhere.

111. 3. 3 Are there relevant professional standards in

experimentation that you must observe that may conflict

with ETIP or PAA objectives? How will you handle this?

You may wish to relate your answer to II. 3.

111. 3. 4 It is anticipated that another ("commercial impact")

contractor will be primarily responsible for examining the

objectives of individual companies and others in the

commercial sector. Comment. (AGENCY IMPACT ONLY)

111. 3.5 It is anticipated that the objectives of individual

companies, associations, etc., may significantly affect not

only the experiments and their evaluation but also the

activities of the bidder. Comment. (COMMERCIAL IMPACT ONLY)

52
RFP 6-35756

III. 4 Phase Two Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the incorporation of the results of

Phase One in the restated objectives for Phase Two.

111. 4.1 It is proposed that, as a result of the work in

Phase One, a reasonable description and integration of the

several objectives will have been realized, and that this

will be used as a basis for the several other tasks but

also as a basis for refinement, revision, and additions

during Phase Two. In addition to this continuing interest

in the objectives introduced in Phase One, it is proposed

that a separate objective of Phase Two is to design and

test a prototype of the complete evaluation process

necessary to meet the several objectives. It is expected

that this may include at least two separately identifiable

parts, as follows: a basic evaluation process to include

those evaluation activities which deal with key, predictable,

identifiable, continuing, common objectives for which the

process can be "institutionalized"; and a special evaluation process to include those evaluation activities which

do not meet the above requirements because of specialized,

one-time, or changing objectives. If you believe your

discussion elsewhere does not adequately reflect your understanding of this, you may comment here.

111. 4. 2 What activities during Phase One will be expected

to contribute to the refinement of Phase Two objectives?

What other sources may be expected to provide input? You

may wish to cross-reference your comments under IV. 4, or

elsewhere

53
RFP 6-35756

III. 5 Phase Three Objectives

This sub-factor is a measure of the bidder's ability

to anticipate the reformulation of objectives in

Phase Three.

III. 5.1 In addition to a continuation of the concern with the

objectives of the previous two phases, it is proposed that

the objective of Phase Three will be to provide a firm basis

for any further related evaluation efforts, including any of

the following: a) a standby or "how to" capability to design

and carry out evaluations for comparable future experiments?

b) an on-line, turnkey data gathering and analysis process

which might be implemented by the PAA; c) an assessment of

the need (or lack of need) for other specific or general

future evaluation efforts. How realistic do you think this

is, and to what extent will the prior phases prepare you to

successfully define and then meet these objectives?

54
RFP 6-35756

Background Information

This factor is a measure of the bidder's overall

understanding of the background and characteristics

of this procurement.

ETIP

This sub-factor is a measure of the bidder's understanding of ETIP and the specific program (s) and/or

experiment (s ) , in terms of past history, present

status and plans for the future.

IV. 1.1 Analogous to the comments in III. 1.1, it is not

proposed to provide a complete and comprehensive description

of ETIP and the specific program(s) and/or experiment (s ) for

the guidance of the bidders (or the successful bidder)

Discuss the adequacy of your present understanding of ETIP

and the specific programs; if you believe it is not adequate,

how do you propose to correct this consistent with the

requirements of Phase One?

IV. 1.2 It is assumed that a reasonably complete and current

description of at least the specific program (s) and/or

experiment (s ) will be desirable, if not necessary, during

Phase One (and, perhaps, on a continuing basis) for use

not only by the bidder but also by ETIP. How can this be

accomplished? What forms and procedures do you visualize

will be used to provide for an up-to-date record? How

will inputs (revisions) be made? How will outputs be

available?

IV. 1.3 The availability of such an information base (and

system) would necessarily increase the opportunities to

make corrections in ongoing activities (experiments)

and may suggest other improvements and changes in the

program. To what extent do you see it as your responsibility

to facilitate this process? What effect would these additional

changes have on your evaluation work?

IV. 1.4 Discuss your understanding of the overall set of

programs and/or experiments to be evaluated, including at

least the following; the set of individual product procurements under life cycle costing (LCC) both planned and

implemented; the FSS decision to adopt LCC as a procurement

practice; the several workshops and the training program as

planned and implemented in all FSS offices around the country,

other planned or implemented procurement experiments, such

as value incentive contracting. Can these be described or

modeled in such a form as to disclose their interactions,

or, conversely, can specific experiments or programs be

defined in such a way as to allow separate treatment?

IV.

IV. 1

55
RFP 6-35756

IV. 2 Program Administrative Agency (PAA)

/ This sub-factor is a measure of the bidder's

understanding of the program administrative

agency (and, where appropriate, its subelements

or related organizations), its organization,

policies and procedures.

IV. 2.1 Questions comparable to those suggested under IV. 1.1

may appear here. Comment. What special problems, if any,

do you anticipate because of the multiple PAA?

IV. 2. 2 As a further restatement of the above, if you do not

have a present acquaintance with the program administrative

agency (and its constituent elements) , how will you acquire

the necessary background?

IV. 2. 3 At least some parts of the organization, policies,

and procedures of the program administrative agency will

be critically related to the present program. For example,

those elements directly involved in the experiments, and

procurement policies and procedure. Discuss your understanding of these areas. If you do not have prior experience,

what effect will this have on your capability?

IV. 2. 4 It is anticipated that another ("agency impact")

contractor will be primarily responsible for obtaining

information about and access to the PAA. How will you deal

with this? (COMMERCIAL IMPACT ONLY)

56
RFP 6-35756

IV . 3 Other Relevant Background

This sub-factor is a measure of the bidder's understanding of the programs, organization, policies

and procedures of other relevant organizations.

IV. 3.1 It is anticipated that information about and access

to organizations other than ETIP and PAA may be important

to this program. For example: other government agencies

which are the end users of the products procured; other

government agencies with legal, financial, policy, or

other responsibilities which interact with the program

area; specific companies, industry segments, and industry

associations; consumer organizations, safety, advertising,

news media, and the like; other ETIP or PAA program or

evaluation contractors. Discuss your understanding of the

potential effects of these interactions and how you will

deal with them.

IV. 3.2 It is anticipated that another ("commercial impact")

contractor will be primarily responsible for obtaining information

about and access to individual companies and others in the

commercial sector. Comment. (AGENCY IMPACT ONLY)

IV. 3. 3 It is anticipated that the bidder will be primarily

responsible for obtaining information about and access to

individual companies and others in the commercial sector.

Comment. (COMMERCIAL IMPACT ONLY)

57
RFP 6-35756

IV. 4 Phase Two Background

This sub-factor is a measure of the bidder's

capability to prepare a comprehensive framework

or model of the program (s) and/or experiment (s)

and the evaluation process.

IV. 4.1 It is anticipated that, after Phase One, the bidder

will be capable of preparing a formal framework within

which the objectives, organizations, and programs may be

related to the evaluation process. Such a framework would

not only integrate the background from Phase One but provide

a convenient and efficient capability for revision, additions,

and modifications without having to start over again. Comment

on the feasibility of this; if you can, suggest how this

might be done or how one would determine whether it was

successful or not? You may wish to cross-reference your

comments on III. 4.

IV. 4. 2 Will the distinction between agency impact and

commercial impact require significantly different

frameworks? If two separate frameworks are developed,

to what extent will it be necessary and/or feasible and/or

desirable to relate the frameworks?

IV. 4. 3 Discuss the same question with respect to the

distinction among the several PAA.

58
RFP 6-35756

IV. 5 Phase Three Background

This sub-factor is a measure of the bidder's awareness of possible further refinements in the model

necessary for the continuation of the evaluation

process

IV. 5.1 It is proposed that the model, and its supporting

descriptive material would provide a sufficient basis for

a turnkey operation so that PAA personnel could, with

modest specialized assistance, carry out on-going

evaluations for comparable programmatic extensions.

Comment on this.

IV. 5. 2 It is anticipated that the specific PAA would be

primarily interested in, and, be, logically, the

appropriate organization to manage, the on-going "agency

impact" evaluation system (or that part concerned with

the specific agency). Comment. (AGENCY IMPACT ONLY)

IV. 5. 3 It is anticipated that significant parts of the

"commercial impact" evaluation system may already be within

the present capability (or future plans) of related organizations (e.g.. Department of Commerce, industry associations,

etc.). What advantages or disadvantages do you see in this

possibility, and how will you react to it? (COMMERCIAL

IMPACT ONLY)

59
RFP 6-35756

V. Evaluation Design

This factor is a measure of the bidder's capabilities

to carry out the evaluation design requirements.

V . 1 Exploratory (Descriptive) and A Priori Proposition

Testing Hypotheses, Variables and Parameters

This sub-factor is a measure of the bidder's ability

to define the detailed evaluation objectives.

VI. 1.1 It is proposed that the various evaluation design requirements will include problems ranging over several dimensions,

and combinations of dimensions, including the following:

a) from exploratory and/or descriptive evaluations (e.g.,

how did a specific experiment get started; what happened;

what are the more significant secondary effects) to a priori

proposition testing (e.g., did specific intervention X "cause"

the observed result Y) ; b) from very specific, identifiable

experiments (manipulations and/or effects) to general, diffuse,

overall changes; c) from post hoc evaluations of completed

experiments through evaluations of ongoing experiments, to

planning and design of future experiments. Comment on your

understanding of this.

V.1.2 At least initially, the bidder will be called upon to

accomplish a preliminary systems design or study. If your

evaluation experience has been primarily in response to

detailed evaluation requirements provided in the RFP or

RFQ , discuss the relevance of your experience or capability

to the requirements of this program.

V.1.3 Discuss your approach to the detailed definition of

a research (or experimental or evaluation) problem; comment

on theory building, definition of variables and parameters,

etc. If considered necessary, distinguish the approach to

a priori proposition testing from that used in exploratory

and/or descriptive evaluation.

60
RFP 6-35756

V.2 Experimental and/or Study Designs

This sub-factor is a measure of the bidder's

ability to define the overall experimental

and/or study design.

V.2.1 Discuss the problem (s) of and approaches to experimental

and/or study design. Alternatively, you may wish to demonstrate

your capability by reference to the relevant education and

experience of your staff. It is suggested that you may wish

to distinguish among the several kinds of designs which may

be appropriate for meeting the several kinds of objectives

which are the outcome of the process discussed in V.l.

V.2.

Discuss the problem (s) of and approaches to identifying

and controlling for the potential effects of parameters.

V.2.

3

it is proposed that the bidder can design an overall

evaluation system and, in parallel, carry out evaluations of

specific selected experiments during Phase One. Discuss

the advantages and disadvantages of this. Can you identify

specific experiments which can be evaluated during Phase

One? Can you identify base-line data which can, or should,

be collected during Phase One? which can, or should, be

collected during later phases?

V.2.

To what extent do you see differences among the several

PAA which will affect the design of the evaluation system?

Will differences in the progress in initiating experiments

among the several PAA be an advantage or disadvantage?

V.2.

5

It is anticipated that some part or all of the bidder's

activity in this area will be dependent upon the activities

of another ("agency impact") contractor. How critical is

this from a time point of view? from an interface point of

view? Will segmentation (by industry, technology, market)

provide advantages or disadvantages? to what extent will

longitudinal designs be important, and will this be an

advantage or disadvantage (COMMERCIAL IMPACT ONLY)

61
RFP 6-35756

V.3 Data Collection Design

This sub-factor is a measure of the bidder's ability

to design the overall data collection process, including

sources of information, sampling strategies, timing,

etc

V.3.1 Discuss the various kinds of sources or methods of

collecting information (e.g., observations, questionnaires,

interviews, records) in terms of your expectations for

their use here, and their advantages and disadvantages.

V.3. 2 The nature and extent of the information required will,

at least in some cases, require the development of sampling

strategies. Describe, by reference to relevant experience,

if appropriate, your capability to develop sampling strategies

for a priori proposition testing evaluations. You may wish

to discuss the approaches used for exploratory evaluation.

You may wish to comment on some of the sampling strategies

provided as illustrations in the several reference documents.

V.3.

3

Discuss any special considerations which may be introduced by the differences among the several PAA.

V.3.4 It is anticipated that the evaluation of agency impact

and commercial impact will require the obtaining of, in

some cases, not only the same data but also separate

additional data from common sources. How can this be

coordinated to minimize duplication and, particularly,

the burdening of sources?

62
RFP 6-35756

V.4 Data Analysis Design

This sub-factor is a measure of the bidder's

ability to design the overall data analysis process,

including processing and analysis, and presenting

results and recommendations.

V.4.1 It is proposed that this part of the process should be

developed concurrently with those discussed in V.2 and V.3

instead of waiting until the data is collected. Comment on

the desirability, necessity, and/or feasibility of this.

V.4.

Specific individual experiments and specific exploratory

studies may have not only different, but also severable

analytical requirements (and discussion of these may more

appropriately appear in VII) . It is expected, however, that

there may be some degree of overlap or interrelation among

the several experiments and studies. To what extent would

an overview of the overall process improve the ability to

perform the individual analyses?

V.4.

3

It is proposed that the quantity and variety of the

various data collected (and to be collected in the future)

present an opportunity for careful planning with respect

to formatting identification, quality control, standardization,

etc. To what extent do you think this is necessary and/or

feasible? To what extent can compatability be achieved with

related existing data banks?

63
RFP 6-35756

V.5 Phase Two Evaluation Design

This sub-factor is a measure of the bidder's ability

to redefine the detailed experimental and/or study

design to reflect the results of Phase One, the

several objectives, and model of the process in a comprehensive detailed design to meet both basic

data (base line data) and special data requirements.

V.5.1 Discuss, or, if appropriate, cross-reference discussions

elsewhere of, your understanding of the problems outlined in V.l

through V.4 as applicable to Phase Two, and with reference

to the requirements of Phase Two.

V.5.

It is anticipated that the bidder will provide a comprehensive and detailed design of an evaluation system

which will provide a capability for both basic and special

evaluation. Discuss your present understanding of the

form and/or extent of the evaluation system. Discuss the

advantages and disadvantages of prototype testing the system

by evaluating specific procurement or other experiments?

64
RFP 6-35756

V.6 Phase Three Evaluation Design

This sub-factor is a measure of the bidder's ability to

further redefine the above/ to provide for a detailed

design, and necessary supporting materials, which would

allow an on-going evaluation process.

V.6.1 Discuss or, if appropriate, cross-reference discussions

elsewhere which demonstrate your ability.

V.6.

Discuss your understanding of what will be necessary to

meet turnkey requirements. What consideration of agency policies

and procedures will be required? How will the necessary skills

and experience be transferred? How will previously collected

data be transferred? What conversion of instruments and software

will be required? What will be the effect of on-going evaluations

of specific experiments on the transition process? and vice versa?

65
RFP 6-35756

Data Collection Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

acquisition of data on the variables and parameters

of interest.

Instrument Design

This sub-factor is a measure of the bidder's ability

to develop (and test) the various questionnaires,

protocols, schedules, and the like required for data

collection

VI . 1 . 1 Discuss (you may use an example and/or reference to

prior experience) the problems and solutions which you consider critical (and/or characteristic) in instrumentation

for an a priori proposition testing type of evaluation.

VI. 2 Discuss the differences, if any, in the approach to

descriptive or exploratory evaluation.

VI. 1.3 Under what circumstances, and for what purposes,

will you conduct pilot tests? Validation tests?

VI. 1.4 It is anticipated that some variables of interest

will be of ongoing importance (as key variables or parameters

common to a number of experiments, or as key exploratory

or descriptive variables) in a sense comparable to "social

indicators" and will become part of what will be known as

"basic data" in future phases. To what extent can this be

anticipated, and what difference will this make during this

phase?

VI. 1.5 Discuss your understanding of the special requirements,

if any, which a government agency (or its contractor) must

meet before use of instruments for data collection.

VI. 1.6 What advantages or disadvantages are introduced by

relevant or related existing data banks in terms of the

instruments they use (e.g., various definitions of industry

classifications, kinds of technology, etc.)?

VI

VI . 1

66
RFP 6-35756

VI. 2 Information Sources

This sub-factor is a measure of the bidder's

ability to identify the various information

sources, and to gain access.

VI. 2.1 Discuss the important or critical problems you anticipate

in this area and your capability to deal with them.

VI. 2. 2 Discuss the various strategies for identifying sources

of information. Distinguish, if appropriate, between sources

of information needed for evaluating a specific experiment

with these to be used for a broad descriptive (case) study.

VI. 2. 3 Discuss the issues of confidentiality and privacy with

respect to information obtained from individuals. Discuss

the issues related to information, in general, with restrictions

on dissemination, e.g., proprietary information, certain

types of personnel and financial information.

VI. 2. 4 Discuss the issues involved in access to individuals

and to records. A special issue of concern is the demands

on the time of individuals in the program administrative

agency, and others who may have a considerably less direct

interest in the evaluation.

VI. 2. 5 Much background and parametric information may be

available in some form, in existing records which have been

collected for some other purpose; how will you identify these

and what advantages or disadvantages do they present?

67
RFP 6-35756

VI . 3 Data Collection

/ This sub-factor is measure of the bidder's capability

to organize, staff, train, direct and control the

personnel who obtain the data by survey, interview,

observation, or use of records.

VI. 3.1 It is anticipated that the several requirements for

data collection may require different skills, different

organization, and different methods of supervision and

control. Discuss by example or reference to prior experience

your capability in this regard.

VI. 3. 2 With reference to the question asked in VI. 1.4,

what difference in approach would be warranted with respect

to "basic data"?

VI. 3. 3 Discuss or describe the process for handling data

collection (e.g., flowcharting). Discuss problems of

monitoring and evaluating data quality, of privacy and

confidentiality, of machine acceptable versus non-machine

acceptable data (e.g., unstructured interviews).

68
RFP 6-35756

VI . 4 Phase Two Data Collection

This sub-factor is a measure of the bidder's ability

to anticipate the requirements of the Phase Two

collection of both basic (base line) and special

data

VI. 4.1 It is proposed that the bidder will be able to

institutionalize the collection of basic data (i.e., key

variables and parameters common to a number of present

or expected experiments, or key exploratory or descriptive

variables) during this phase; from this, base line data

can be obtained and a "test" of the process can be

accomplished. Comment on this.

VI. 4. 2 It is proposed that there will remain other requirements not included within the above (e.g., specific one-time

requirements, unusual or specialized requirements; added or

revised requirements) which cannot or should not be commingled with the above. Discuss this.

69
RFP 6-35756

VI . 5 Phase Three Data Collection

i This sub-factor is a measure of the bidder's ability

to plan against the requirement to design an

ongoing collection process.

VI . 5 . 1 At least the basic data collection process should

be sufficiently stabilized and described to allow an

ongoing, turnkey implementation. Discuss the problems

and prospects of this.

VI . 5 . 2 To what extent can new or other specialized requirements be anticipated?

VI. 5. 3 Discuss any special problems with respect to access,

privacy, confidentiality, etc., which may be introduced when

an agency takes over management and operation of the evaluation system.

70
RFP 6-35756

VII. Data Analysis Process

This factor is a measure of the bidder's capability

to carry out all of the functions necessary to the

analysis of the data and to present the results and

recommendations

VII. 1 Analysis

This sub-factor is the measure of the bidder's ability

to identify and apply the appropriate analytical

(including statistical) techniques to the data

collected

VII. 1.1 Discuss, by example or reference to prior experience,

your ability to accomplish the various analytical tasks

involved

VII. 1.2 For at least some experiments (and/or policy changes),

a significant if not primary objective is economic. Discuss

your capability in analysing "cost/benefits," including

problems introduced by estimating, allocating, and forecasting costs and benefits, both direct and indirect.

VII. 1.3 For at least some experiments (and/or policy changes),

a significant if not primary objective is to bring about

some change in the policies and procedures of either the

program administrative agency or of segments of the industry.

Discuss your capability in analysing administrative

(behavioral) changes.

VII. 1.4 For at least some experiments (and/or policy changes),

a significant if not primary objective is to bring about some

change in the use of technology - change in the rate, application, etc. Discuss your capability an analysing technological change.

71
RFP 6-35756

VII. 2 Exploratory (and/or Descriptive) Findings

This sub-factor is a measure of the bidder's

capability to identify and present exploratory

findings

VII. 2.1 It is anticipated that in many areas of the evaluation, it will be neither required nor feasible to develop

findings which meet the more stringent formal requirements

of a priori proposition testing. To what extent do you

consider it your responsibility to reexamine cases where

there is no requirement, especially where it appears that

a priori proposition testing i_s feasible?

VII. 2. 2 You may wish to comment on some of the requirements

as presently described in the several reference documents.

VII. 2. 3 You may wish to discuss the various forms of

presentation (i.e., case studies, informal reports,

status reports, systems studies, etc.)

VII. 2. 4 In addition to exploratory (and/or descriptive) findings

with respect to experiments (and their related parameters and

background) , it is proposed that the evaluation system design

is, itself, an exploratory or descriptive "finding." What

differences, if any, will be required in the presentation and

related supporting documentation or justification of the

evaluation system design?

72
RFP 6-35756

VII. 3 A Priori Proposition Testing Findings.

This sub-factor is a measure of the bidder's

capability to identify and present a priori

findings

VII. 3.1 It is anticipated that not only specific experiments

(at least in terms of their proximal effects or impacts) but

also other specific questions which may be subject

to study or experiment will require (and it will be feasible

to accomplish) a priori proposition testing evaluation. To

what extent do you consider it your responsibility to reexamine cases where there is no requirement? Cases where

there is a requirement, but it is not feasible?

VII. 3. 2 You may wish to comment on some of the requirements

as presently described in the several reference documents.

VII. 3. 3 You may wish to discuss the various forms of

presentation.

VII. 3. 4 Do you think it is desirable and/or feasible to develop

a procedure or process which can be used to make preliminary

assessments of the likelihood that a particular proposed

experiment will be capable of a priori proposition testing?

73
RFP 6-35756

VII. 4 Recommendations

This sub-factor is a measure of the bidder's

understanding of the relationship between the

results of Phase One and the work in Phase Two,

in terms of additional, revised, or deleted

hypotheses, changes in objectives or background

information, and the relative feasibility and/or

usefulness of alternative designs for evaluation.

VII. 4.1 In addition to the specific findings discussed in

VII. 2 and VII. 3, it is anticipated that the work in each

phase will provide a basis for the work in subsequent

phases. It is anticipated that this input to the next

phase may become available before the completion of all

of the work in the current phase. Comment on the feasibility

of an overlap, the advantages and disadvantages.

VII. 4. 2 It is anticipated that a "final report" will be

prepared at, or after, the completion of each phase which

will describe what the bidder sets out to do, what he did,

including relevant activities of others, and the results

(VII. 2, VII. 3, VII. 4). Comment on the relation of the

final report to the several other forms (and timings) of

outputs

74

RFP 6-35756

I.

II.

Ill

IV.

V.

VI.

VII

NUMERICAL WEIGHTS FOR PROPOSAL EVALUATION FACTORS

Factors Weights

Management

This factor is a measure of the background

and overall management capability of the

bidder. 15

Reports and Reviews

This factor is the measure of the bidder's

ability to assure that the Government is

able to effectively and progressively review

a very large scale effort with a minimum of

people, and of the bidder's ability to

coordinate his activities with those of

other related individuals and organizations. 15

Obj ectives

This factor is a measure of the bidder's overall understanding of the objectives of this

procurement (and related programs) . 20

Background Information

This factor is a measure of the bidder's overall understanding of the background and

characteristics of this procurement. 15

Evaluation Design

This factor is a measure of the bidder's

capabilities to carry out the evaluation

design requirements. 15

Data Collection Process

This factor is a measure of the bidder's capability to carry out all of the functions

necessary to the acquisition of data on the

variables and parameters of interest. 10

Data Analysis Process

This factor is a measure of the bidder's capability to carry out all of the functions

necessary to the analysis of the data and to

present the results and recommendations. 10

Seven Evaluation Factors Total 100

75

RFP 6-35756

SCOPE OF EFFORT

Introduction

A. Purpose

The purpose of this document is to provide a guide to the

bidder in the preparation of supporting data on the scope

of effort he proposes.

B. Basis for Preparation of This Document

This document is intended to be used in conjunction with the

associated Proposal Preparation Procedure, Proposal Evaluation

Factors, Schedule, and Statement of Work. The objective is to

assist the bidder in presenting his supporting data on scope

of effort in an orderly and usable form so that it may be

evaluated concurrently with his written (and oral) proposal.

B.l In order to achieve an orderly and usable form, the supporting data on scope of effort must be, at least substantially,

related to the factor (and sub-factor) categories which are the

basis for the evaluation. The detailed outline furnished in

the next section is for this purpose.

B.2 In addition to the above, it is necessary to assure that

the data furnished by the several bidders be on a reasonably

comparable basis. To accomplish this the detailed guidance

included in this Introductory Section is furnished. It should

be understood that the guidance in this respect is necessarily

arbitrary in the interests of meeting the specific objective.

B.3 The format requirements reflect the fact that evaluation

team members need, and can use, only selected data, primarily

in terms of man/months of effort in certain areas and certain

materials costs. A summary analysis of the scope of effort

against cost data will be accomplished.

77
RFP 6-35756

C . General Comments

To assist the bidder in understanding more fully the part

which these data will have in the evaluation, these general

comments are offered.

C.l This type of program which establishes a broad objective

but provides for planning and design to work out with more

detail and precision the means for accomplishment cannot be

costed with the closeness normally experienced on procurements

of items with a stabilized design. For this reason the

bidders' respective total cost estimates will not necessarily

reflect the actual realized costs to the government, nor do

they necessarily reflect the relative costs of the bidders

in providing a comparable capability. Therefore, relative

total cost, as such, cannot and will not be given any direct

weight in the evaluation. Cost, in terms of scope of effort,

however, will aid in establishing the kind and extent of effort,

and this information will serve to further define the "wordpictures" included in the written (and oral) proposals. The

formal evaluation of relative cost and associated areas is

included as part of one of the major factors in the evaluation

(1.4), and further reference should be directed there.

C.2 While the major impact of these data in the evaluation will

be in providing additional perspective to the written (and

oral) proposals, certain corollary information will be used.

C.2.1 A measure of how realistically and thoroughly the costing

has been done will be established. A too high estimate indicates the obvious disadvantages of "gold plating" or a potentially loose operation. In contrast, however, a too low

estimate presents, in the long run, even more serious disadvantages. First, it may indicate a serious misunderstanding

of the scope and objective of the program. Second, it may

raise a serious question as to the bidder's ability to plan

and estimate costs, thus impairing his ability, and that of

the government, to review, predict and control costs. Finally,

it may indicate something less than complete candor in presenting

the cost part of the proposal.

78
RFP 6-35756

C.2.2 The data presented here will be used to supplement the

data furnished with respect to cost as a factor in the overall evaluation.

C.2.3 The data presented here will be used in negotiation with

the successful bidder both as a basis for further refining the

contractual statement of work and in establishing the contract

price

79
RFP 6-35756

D. Use of Format

D.l It is of critical importance that all persons directly or

indirectly preparing data on scope of effort be thoroughly

familiar with the rules on use of the format. This is

especially true because the figures to be included do not,

in all cases, follow the accounting distinction between direct

and indirect costs. Further, certain types of effort are to

be tabulated as separate items. An effort has been made to

use format and terminology as close to the general practice

as possible. Where this has not been done, the basis has

been to assume that useful data for the evaluation can be

obtained. Certain distinctions, such as direct and indirect

costs, have little significance for this purpose. Similarly,

certain areas of effort can only be evaluated if separated

from proportionately large associated areas of effort.

D.2 The next section specifies the scope of effort data on

a factor by factor basis. To facilitate the evaluation, these

data should be collated with the respective factors. Because

the procedure for evaluation provides for each factor to be

evaluated, substantially, without mandatory recourse to other

parts of the proposal, it is critically important that the

data on scope of effort and the written proposal on a specific

factor, taken together, is complete and self-explanatory. Care

must be exercised to assure that the evaluation team does not

inadvertently penalize the written proposal because of an

inadequate scope of effort.

D.2.1 Under each factor, the scope of effort should be set

out separately for the first phase of the program. Where

the bidder proposes to begin or complete items in another

phase, note should be made to avoid misunderstanding.

D.3 It is expected that in many instances the scope of effort

proposed will be, literally, the bidder's present estimate

of the approximate level (or scope) of effort he expects

will be required to accomplish what he proposes to do in

his written (and oral) proposal. A certain latitude is

expected, and these data on scope of effort will be used,

as stated before, primarily to assist the evaluation team

in understanding what the bidder is actually proposing.

There are, however, certain areas which may prevent a clear

understanding, and some of these are tabulated below.

80
RFP 6-35756

D.3.1 The breakdown in the next section is a minimum guide

in tabulating the scope of effort. Where a particular figure

is considered unusually small or large, or for some other

reason may be misleading, it should be footnoted with an

explanation or further breakdown.

D.3.2 In some cases identification of particular types of

effort with a proposed "associate" or subcontractor will be

of assistance. This will be particularly true where part or

all of an effort is proposed to be accomplished by some

specialist group.

D.3.3 The analysis of comparative cost advantage will not be

accomplished, as a separate factor, by the overall evaluation

team. It is necessary, however, where a bidder is estimating

costs which are, on a relative basis, low that the basis for

this be included in the scope of effort data to assure that

the evaluation team understands that the bidder does intend

to accomplish what he has said in his written proposal.

D.3.4 Conversely, where a possible misunderstanding may occur

due to a lack of appreciation by the evaluating team of the

cost associated with accomplishing a particular part of the

proposal, the bidder should make this clear.

D.3.5 Man/month estimates should be divided according to

distinct classes to avoid confusion or ambiguity in two

respects. First, the bidder's data should distinguish between

direct and indirect. Second, the data should distinguish

between senior professional personnel and supporting technicians,

draftsmen, etc., and between various management personnel and

clerical assistance, etc.

D.3.6 Any further analysis considered necessary to assure that

the evaluation team properly assesses the relative scope of

effort as related to the written proposal is encouraged. This

may be particularly true in terms of level of training of

personnel, availability of facilities, ability to control design,

and the like.

81
RFP 6-35756

E . Summary Data

E.l The relationship of the individual scope of effort on

the several factors to the bidder's overall cost estimate will

be examined. For this purpose, certain summary data will

be required. These summary data should be furnished as a

separate document which should also include a complete reference

set of the scope of effort data sheets.

E.2 Basic summary data will include presentation of sub-total

and total cost estimates in standard format - direct costs,

burden, materials, G&A, fee, etc.

E.2.1 These summary sheets should be accompanied with notes

reconciling the scope of effort data sheets. Any unusual

data should be explained.

E.3 If the proposal includes a substantial proposed subcontracting (or "associate") cost figures, a summary analysis

should be prepared and related to the basic summary data.

E . 4 Supporting schedules should be prepared detailing the

hourly wage rate of personnel and this related to the basic

summary data.

E.5 Supporting schedules should be prepared for travel costs

and related to the basic summary data. Schedules should

include breakdown based on distance (or destination) , duration,

number of travelers, and number of trips. Similarly, supporting

schedules on telephone (and teletype) by number of calls, and

average cost per call.

82
RFP 6-35756

Phase One Scope of Effort

Note: The detailed organization parallels the organization

of the Statement of Work, Schedule (s), and Proposal

Evaluation Factors, reference to which should be made

for clarif ication .

'

I. Management

(Note that the scope of work here is limited to administrative or program management, and should be distinguished

from effort properly reported under other factors.)

Program Director

Senior Management Advisors

(Include general officers, or other senior

corporate personnel, concerned with administrative management policy, if appropriate)

Other (specify)

(Include comparable personnel of associates,

subcontractors, consultants, if appropriate;

include program director's personal staff,

if any)

Scheduling

(Include costs of analysis, preparation,

monitoring, etc.)

Costing

(Include costs of analysis, preparation,

monitoring, etc.)

Telephone

(Include telephone, teletype costs, etc., and

breakdown by number of calls, etc., if appropriate)

Travel $

(Include breakdown by destination, duration,

breakdown by number of trips, if appropriate)

II. Reports and Reviews

(Note that you should distinguish efforts properly

attributable to other factors.)

Liaison with ETIP M/M

(Include time spent in conferences or visits both

at ETIP and contractor's plant, or elsewhere breakdown by number of trips, duration, number of

travelers .

Liaison with PAA M/M

(Include on same basis.)

Liaison with Others M/M

(Include on same basis.)

M/M

M/M

M/M

M/M

M/M

$

83
RFP 6-35756

Reports and Services M/M & $

(Include time spent in direct preparation,

technical writing, drafting, etc., and

costs of reproduction. Breakdown by type

of report, etc., if appropriate.)

III. Objectives

Review of objectives M/M

(Breakdown between documents and interviews,

and among ETIP, PAA, industry, and others.)

Preparation of description and analysis M/M

(Breakdown among ETIP, PAA, industry, and

others .

)

Preparation of preliminary recommendations M/M

Preliminary requirements for pilot evaluation M/M

system

Preliminary requirements for prototype M/M

evaluation system

Travel $

IV. Background

Review of background M/M

(Breakdown between documents

and interviews, and among experiments,

programs and organizations.)

Preparation of framework model M/M

Travel $

V. Evaluation Design

Preliminary Design M/M

Detailed Design M/M

Preliminary design of pilot evaluation system M/M

Preliminary design of prototype evaluation system M/M

VI. Data Collection Process

(Note that you should distinguish data collections

which may have been included under prior factors)

Design and development of data collection process M/M

Data collection for exploratory evaluations M/M

84
RFP 6-35756

Design and pilot test of instruments M/M

Data collection for a priori proposition M/M

testing evaluations

Travel $

(Breakdown, as appropriate, between exploratory

and a priori proposition testing evaluations;

provide, as appropriate, basis for estimating

in terms of trips, etc.)

Telephone, postage, etc. $

(Breakdown, as above.)

VII. Data Analysis Process

(Note that you should distinguish from effort

included with Factor II.)

Design and development of data analysis process M/M

Exploratory analysis and findings M/M

A priori proposition testing analysis and findings M/M

Analysis of evaluation system, and recommendations M/M

Analysis of Phase One, and recommendations M/M

85
RFP 6-35756

Phase Two and Three Scope of Effort

Phase Two

It is not expected that the bidder will be able to provide a comparable level of detail for Phase Two. However, a rough

estimate will be useful in assessing those references to

Phase Two in the proposal, and in planning.

The minimum required breakdown is to distinguish between the

following:

1. Those costs (labor, travel, etc.) directly a function of

data collection and analysis for both basic and special

data processes.

2. All of the remainder (management, design, etc.)

Additional information, consistent with these purposes,

would be appreciated, but it is not required.

Phase Three

A similar distinction should be observed in providing estimates for

Phase Three. It is expected that these may be more speculative,

but where there are identifiable assumptions which will largely

determine the scope of effort, discussion would be helpful.

86
NBS-1 14A (REV. 7-73)

U.S. DEPT. OF COMM. BIBLIOGRAPHIC DATA

SHEET

1. PUBLICATION OR REPORT NO. NBSIR 76-1124

4. TITLE AND SUBTITLE

2. Gov’t Accession

No.

EVALUATION SYSTEM PROPOSAL PREPARATION

AND EVALUATION PROCEDURE

7. AUTHOR(S)

Charles W. N. Thompson

3. Recipient’s Accession No.

5. Publication Date February 1976

6. Performing Organization Code

8. Performing Organ. Report No.

9. PERFORMING ORGANIZATION NAME AND ADDRESS

NATIONAL BUREAU OF STANDARDS

DEPARTMENT OF COMMERCE

WASHINGTON, D.C. 20234

10. Project/Task/Work Unit No. 1150066

11. Contract/Grant No.

12. Sponsoring Organization Name and Complete Address (Street, City, State, ZIP)

National Bureau of Standards

Department of Commerce

Washington, D.C. 20234

13. Type of Report & Period Covered

Final

14. Sponsoring Agency Code

15. SUPPLEMENTARY NOTES

This set of documents was prepared for, and was included in, RFP6-35756;

the complete RFP contains, in addition, solicitation information and a number of reference document s.

16. ABSTRACT (A 200-word or less (actual summary of most significant information. If document includes a significant

bibliography or literature survey, mention it here.) As part of its program with respect to the conduct of procurement exper

iments , the Experimental Technology Incentives Program of the National

Bureau of Standards developed an integrated set of documents to provide

information for use by both bidders and the proposal evaluation team in

the procurement of two evaluation systems. The set of documents includes

the following: (1) Introductory Notes; (2) Some Definitions; (3) Proposal Preparation Procedure (with revisions); (4) Evaluation System Design Process; (5) Statement of Work; (6) Statement of Work (with illustrative sub-items for Phase One); (7) Proposed Schedule - Overall; (8)

Proposed Schedule - Phase One; (9) Proposal Evaluation Factors; (10) Proposal Evaluation Factors (with Illustrative Questions) ; (11) Numerical

Weights for Proposal Evaluation Factors; and (12) Scope of Effort. Except

for introductory and descriptive material, all of the documents are

organized on the basis of seven factors (and thirty-three sub-factors) to

facilitate both preparation and evaluation of proposals. Each sub-factor

measures the bidder's understanding, present or proposed approach, and

present or potential capability to perform with respect to an important

problem or characteristic of the program. While the procedure is considered limited to certain systems and related procurements, it is considered a significant departure from prior procedures.

17. KEY WORDS (six to twelve entries; alphabetical order; capitalize only the first letter of the first key word unless a proper name; separated by semicolons

)

Evaluation systems; Experimental Technology Incentives Program;

procurement experiments; procurement procedures; proposal evaluation

procedures; proposal preparation procedures

18. AVAILABILITY Uni imited

For Official Distribution. Do Not Release to NTIS

Order From Sup. of Doc., U.S. Government Printing Office Washington, D.C. 20402, SD Cat. No. 0 3

lX ! Order F rom National Technical Information Service (NTIS)

Springfield, Virginia 22151

19. SECURITY CLASS

(THIS REPORT)

UNCL ASSIFIED

20. SECURITY CLASS

(THIS PAGE)

UNCLASSIFIED

21. NO. OF PAGES

92

22. Price

$5.00

USCOMM-DC 29042-P74