[GRAPHIC] \SW9CVR.GIF





                MEMBERS OF THE FEDERAL COMMITTEE ON
                      STATISTICAL METHODOLOGY

                            (June 1983)


Maria Elena Gonzalez (Chair)
Office of Information and Regulatory Affairs (OMB)

Barbara A. Bailer
Bureau of the Census (Commerce)

Norman D. Beller
National Center for Education Statistics (Education)

Yvonne M. Bishop
Energy Information Administration (Energy)

Edwin J. Coleman
Bureau of Economic Analysis (Commerce)

John E. Cremeans
Bureau of Industrial Economics (Commerce)

Zahava D. Doering
Defense Manpower Data Center (Defense)

Marie D. Eldridge
National Center for Education Statistics (Education)

Daniel H. Garnick
Bureau of Economic Analysis (Commerce)

Charles, D. Jones
Bureau of the Census (Commerce)

Daniel Kasprzyk
Bureau of the Census (Commerce)

William E. Kibler
Statistical Reporting Service (Agriculture)

Thomas Plewes
Bureau of Labor Statistics

Raymond C. Sansing
Internal Revenue Service (Treasury)

Fritz J. Scheuren
Internal Revenue Service (Treasury)

Monroe G. Sirken
National Center for Health Statistics (Health and Human Services)

Wray Smith
Energy Information Administration Energy

Thomas G. Staples
Social Security Administration (Health and Human Services)





           OFFICE OF INFORMATION AND REGULATORY AFFAIRS

                 Christopher DeMuth, Administrator

            Thomas D. Hopkins, Deputy Administrator for
                Regulatory and Statistical Analysis

                  Maria E. Gonzalez, Chairperson
           Federal Committee on Statistical Methodology

                              PREFACE

The Subcommittee on Contracting for Statistical Surveys was formed
to review current practices by agencies in contracting for surveys
and to make recommendations for their improvement.  This working
paper, the report of that Subcommittee, addresses the following:
technical expertise needed in statistical contracting, preparation
of the Request for Proposal, contractor selection, and post-award
activities.

While the report is intended to be useful to agencies that may
utilize contracting for surveys, a broader audience may find the
report of interest.  Some" of the material should be useful as a
supplement to the existing training provided, agency project
officers.  Seminars and meetings will be organized to discuss the
findings of this subcommittee with Federal agency personnel
involved with contracting for surveys.

The working paper was prepared by the members of the Subcommittee
on Contracting for Statistical Surveys, Federal Committee on
Statistical Methodology.  The Subcommittee was chaired by Thomas G.
Staples, Social Security Administration, Department of Health and
Human Services.  As a subcommittee report, this document does not
necessarily represent the views of the Office of Management and
Budget.





                  MEMBERS OF THE SUBCOMMITTEE ON
                CONTRACTING FOR STATISTICAL SURVEYS


Thomas G. Staples, Chairperson
Social Security Administration
Department of Health and Human Services

Henry P. Brehm
Social Security Administration
Department of Health and Human Services

David W. Chapman
Bureau of the Census
Department of Commerce

Maria E. Gonazalez
Office of Information and Regulatory Affairs
Office of Management and Budget

Nancy D. Pearce
National Center for Health Statistics
Department of Health and Human Services

Richard J. Pratt
Bureau of Labor Statistics
Department of Labor

Margaret Weidenhamer
Statistical Reporting Service
Department of Agriculture

                               (ii)





                         ACKNOWLEDGEMENTS


This report represents the collective effort of the Subcommittee on
Contracting for Statistical Surveys.  Although the subcommittee
members reviewed and commented on all parts of the report, the
following specific individuals were responsible for preparing the
chapters:

     Chapter   Authors
       I       Henry P. Brehm
      II       David W. Chapman
     III       Margaret Weidenhamer/Richard J. Pratt
      IV       Henry P. Brehm
       V       Nancy D. Pearce/Thomas G. Staples

     Appendix

      I   J. Timothy Sprehe
     II   Nancy D. Pearce


In addition to the subcommittee members, special recognition goes
to the Federal Committee on Statistical Methodology, whose
membership identified the need for this report.  Portions of
several committee meetings were devoted to feedback to the
subcommittee as the report was developing; individual members of
the committee provided detailed oral and written comments.  In
addition, special recognition goes (1) to the contractor and
Federal agency personnel who reviewed draft copies of the report
and (2) to the contractor and Federal agency representatives who
participated in the surveys of sponsoring agencies and contractors. 
Also, several individuals who made significant contributions to the
report participated in the early stages of the subcommittee's work. 
These include Thomas B. Jabine, Leon Gilford, Barbara A. Bailar,
Bette S. Mahoney, Henrietta Hyatt, David B. Orr, Irene C. Montie,
J. Timothy Sprehe, and Corrinne B. Lennox.  Jacob Deutch and David
Arnaudo participated in the surveys of agency and contractor
representatives.

Among the subcommittee members special recognition goes to Nancy D.
Pearce for organizing the draft and final versions of the report
and to Nancy D. Pearce and Henry P. Brehm, for pulling together a
compact version of the report for the 1982 proceedings of the
American Statistical Association.

Maria E. Gonzalez, on numerous occasions, chaired meetings and
performed other responsibilities when the chairperson was absent.

                         Thomas G. Staples


                               (iii)








                         TABLE OF CONTENTS

Chapter 1.      Introduction. . . . . . . . . . . . . . . . . . . 1

     Background and Purpose . . . . . . . . . . . . . . . . . . . 1
     The Chapters to Follow . . . . . . . . . . . . . . . . . . . 4

Chapter 2.     Technical Skills Required in Contracting for
               Surveys. . . . . . . . . . . . . . . . . . . . . . 7

     Introduction . . . . . . . . . . . . . . . . . . . . . . . . 7
     Basic Types of Skills Required . . . . . . . . . . . . . . . 8
     Technical Input at Various Stages of Contracting for 
         a Survey. . . . . . . . . . . . . . . . . . . . . . . . 12
     Some Technical Aspects Associated with Current Practice. . .14

Chapter 3.     Considerations in the Preparation of Requests for
               Proposal . . . . . . . . . . . . . . . . . . . . .17

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .17
     Initial Considerations . . . . . . . . . . . . . . . . . . .17
     Possible Contents of Requests for Proposal . . . . . . . . .20
     Role of the Project Officer. . . . . . . . . . . . . . . . .28
     Evaluation Criteria. . . . . . . . . . . . . . . . . . . . .29

Chapter 4.     Contractor Selection Process . . . . . . . . . . .31

     Introduction . . . . . . . . . . . . . . . . . . . . . . . .31
     Components of the Selection Process. . . . . . . . . . . . .31
     Technical Evaluation of Content and Capability . . . . . . .33
     Discussions with Offerors in the Competitive Range . . . . .34
     The Final Award Process. . . . . . . . . . . . . . . . . . .35

Chapter 5.     Post-Award Activities. . . . . . . . . . . . . . .37
     Introduction . . . . . . . . . . . . . . . . . . . . . . . .37
     General Techniques for Monitoring Contractor Performance . .38
     Administrative Responsibilities of the Project Officer . . .38
     Quality Assurance Responsibilities of the Project Officer. .40

Appendix I.    Informal Survey of Agencies and Contractors
               Concerning Contracting for Surveys . . . . . . . .47
     Introduction . . . . . . . . . . . . . . . . . . . . . . . .47
     Federal Agency Responses . . . . . . . . . . . . . . . . . .48
     Contractor Responses . . . . . . . . . . . . . . . . . . . .50

Appendix II.   Office of Management and Budget Requirements:
               Information Collection Budget and Clearance. . . .55
     Introduction . . . . . . . . . . . . . . . . . . . . . . . .55
     Information Collection Budget. . . . . . . . . . . . . . . .56
     Clearance. . . . . . . . . . . . . . . . . . . . . . . . . .56
     After Receipt of Approval. . . . . . . . . . . . . . . . . .58

                                (v)








                      Chapter 1. INTRODUCTION


BACKGROUND AND PURPOSE

Surveys are a major source of data for many agencies of the Federal
Government.  Surveys deal with a multitude of subjects, for
example, demographic, economic, psychological, sociological, and
medical topics, spanning a multiplicity of diverse governmental
concerns for the general welfare.  Included are surveys on the
effectiveness of government programs, the impact of regulated and
unregulated industrial activities, and problems for which
population groups are currently or may in the future be at risk or
in need of government assistance in some form.  Although these
topics are highly diverse. the issues in the use and conduct of
surveys to obtain data on them are far more amenable to systematic
treatment with a unified focus.

For the Federal Government to conduct a survey, there are the three
following primary options:

-    do the work in-house;

-    enter into an interagency agreement; or

-    contract the work out.

These options can be used in combination with each other, and in
actual practice this is often the case.  An agency may have the
data collection, reduction, and tabulation phases of a survey
conducted under a contract or an interagency agreement.1  It may
choose to design the survey and conduct the analysis in-house.

The Annual Survey of Manufactures is an example of a statistical
survey carried out by in-house staff.  Planning and preparation,
data collection and tabulation, and presentation of the data are
done by or under the immediate supervision of Bureau of the Census
employees.  Additional staff needed to satisfy the demands of high
volume data collection and handling are obtained on a limited
individual employment basis.

Because of its capacity to conduct statistical surveys, the Bureau
of the Census is one of the agencies often used by Government
agencies to conduct surveys.  For example, the Current Population
Survey conducted by the Bureau of the Census for the Bureau of
Labor Statistics is a statistical survey carried out through an
interagency agreement.  However, agencies that do reimbursable work
have limited subject matter expertise beyond their own missions. 
Additionally, their facilities are in demand and are by no means
unlimited.  For many survey efforts, then, it becomes necessary
that agencies of government utilize the services of outside
organizations.  A range of such services is available under
contract from non-federal government
___________________________

1Statistical Policy Working Paper 8, "Statistical Interagency
Agreements" (October 1982; Regulatory and Statistical Analysis
Division, Office of Information and Regulatory Affairs, Office of
Management and Budget), prepared by the Subcommittee on Statistical
Interagency Agreements, presents data on the prevalence of
interagency agreements and summarizes the prevailing practices and
policies of Federal agencies with respect to their use.

                                 1





sources.  Depending on the needs of the individual agency or
survey, these may include carefully delineated and defined services
related to data collection and processing or more general
involvement in all phases of a survey from planning to final
report.  In some cases the suggestion of the subject matter area in
which a survey is needed may originate outside the government and
result in a proposal for the entire effort, including the
definition of the area for study.

The overall functions of the Office of Management and Budget (OMB)
as related to review and approval of agency budgets, program plans,
and data collection plans set limits and provide guidance for the
use of statistical surveys.  For the most part, the issues of
concern center on the need for the data for policy and program
activities, the reasonableness of the data collection plan and
questionnaires, the coordination among agencies with overlapping
interests to avoid duplication of effort, and the concern for the
burden on potential respondents.

From another perspective, the Federal Procurement Regulations and
the unit within each agency responsible for the administrative
aspects of formal procurement actions, in accordance with these
general regulations, provide standards and requirements for a
contract as a legal document independent of what is being
purchased.  OMB Circular A-76 sets restrictions and requirements on
developing in-house capacity versus using nonfederal contracting
sources to obtain various services.  This affects the justification
for moving work in-house that has been done previously by
contractors and contracting work out for which an in-house capacity
exists.  The implications of Circular A-76 for contracting for a
survey should be reviewed with the procurement office.

However, beyond the general standards there is no specific guidance
on the use by Federal agencies of contractors for the conduct of
surveys.  They do not define when and how the procurement mechanism
should be used for such a purpose nor do they indicate what the
relationships should be among an agency's need for survey data and
analysis, its staff resources, and the specification of contract
requirements and monitoring of contractor performance.

The Subcommittee on Contracting for Statistical Surveys of the
Federal Committee on Statistical Methodology was organized to
suggest ways to improve the survey contracting process in the U.S.
Federal Government in the future.  The Subcommittee's work was done
under the general mandate of OMB.  Because of its unique
responsibilities, OMB was able to call upon the experience of a
variety of individuals with backgrounds in diverse aspects of the
conduct of surveys and the use of contracts.  This report
represents the collective effort of the Subcommittee.  The opinions
expressed here reflect the collective judgment of the Subcommittee
and do not necessarily reflect those of OMB.

It is not possible within any given presentation of this nature to
cover all the issues related to the use and conduct of surveys. 
The following questions are some that need to be answered: What is
the problem on which the study should focus? Should a survey be
conducted as opposed to the use of some alternative research
strategy or data collection approach? If a survey is to be used,
should it be done totally in-house, totally under an interagency
agreement, totally under a contract, or using some combination of
these three approaches? The Subcommittee focused on what should be
done after the decision is made to contract out the survey, and not
on the factors involved in selecting this option as against an
interagency agreement or in-house conduct of the survey.

                                 2





As related to survey design and conduct, issues are: What should be
the universe for study? What is the most appropriate sample design?
Should the data be collected by mail, by telephone, or by personal
interview? What specific questions should be asked and how should
the data be analyzed? These and many other issues must be addressed
at some point in the course of deciding to use, planning and
preparing for, and actually conducting a survey.  While the need to
address such issues is discussed, there is no effort to provide
input into the decision making process on such matters.  There are
many references and resources available on survey design and
content considerations so that it would be inappropriate to try to
duplicate these here.

This report is intended for circulation to Federal agencies and
offices which may utilize contracting for surveys.  The report
tries to draw attention to the issues that need to be considered,
how to use or obtain the necessary expertise in preparing for and
using the contract mechanism, how to prepare specifications and
carry out contractor selection, and how to maximize contractor
performance.

It should be pointed out that this report does not deal with
surveys conducted or supported under grants or interagency
agreements.  The report focuses on contracts awarded through an
open competitive negotiated procurement mechanism based on
evaluation of the technical and business proposals submitted in
response to generally available Requests for Proposal (RFPs).  An
RFP is a formally issued notice of a government agency's need to
have work done by outside organizations where contractor selection
cannot be based on price alone.  The RFP is the publicly released
notice which contains the internally approved program Request for
Contract.  The primary reason for focusing on this mechanism is
that it offers the agency the maximum opportunity to set the
requirements for the survey and the standards for contractor
performance and to select the organization that offers the best
combination of anticipated product quality and cost factors,
without regard to issues unrelated to these concerns.  Other
contract approaches, while serving other purposes, do not
necessarily promote selection of the best choice of contractor for
conducting a given survey at a given point in time.

Among the alternative contract approaches are sole source selection
because of a unique capacity to perform or based on an unsolicited
proposal, and set-aside programs for competition only among small
businesses in general or minority-owned small businesses.  The
small business set-aside and the minority-owned business programs
permit competition for prime contract work among firms which are
qualified for these programs but totally exclude the larger
organizations with greater resources and all nonprofit research
units, including academically-based units.  This is part of the
Federal Government's efforts to promote the economic viability of
small, and particularly minority-owned, businesses.  However, on
any given procurement, these restrictions limit the range of
competition.

Although excluded from the small business and minority-owned
business programs, nonprofit organizations are eligible for sole
source contracting.  The mechanism totally restricts competition,
either because only one organization is determined to have the
capacity to perform or because the organization submitted an
unsolicited proposal that represents a new and unique approach and
opportunity to obtain a valuable product.  In this latter
situation, the organization submitting the proposal controls the
product offered, that is, the particular survey to be performed,
and is the only one to which the contract can be awarded.

In line with the intention to provide information designed to
improve the ability to use the contracting mechanism for
procurement of survey services it was important to learn more about
what the agencies were doing at that time.  Since no useful data

                                 3





existed on this process, Subcommittee members interviewed officials
of several selected Federal agencies involved in contracting for
surveys and some of the contractors with which the agencies dealt. 
Appendix I summarizes the data collected.

The agencies selected for the survey of their contracting
operations were asked to have representatives of both the program
and the administrative aspects of contracting operations available. 
The da were collected by personal interview.  The questions were
directed toward contracting for surveys only.  The questions traced
the use of this mechanism from the factors in the decision to use
different types of contracts (cost or fixed-price) and how often
they were used, through developing RFPS, soliciting bids and
evaluating proposals, to monitoring contracts.  Similar information
was collected from a limited number of contractors.  This
information has been analyzed and its implications assessed for
developing suggestions designed to improve the use of the contract
mechanism for conducting surveys.  The various reviews of current
practice that have been made have been interwoven into this
presentation.


THE CHAPTERS TO FOLLOW

In structuring the material to be presented, an effort was made to
follow a logical order of stages in the development and management
of contracts for surveys.  The sequence of events in this process
was a guiding principle within the general format, but not an
absolute basis for organizing the chapters.  The different chapters
deal with the development of requirements for RFPs and with issues
in the selection and monitoring of a contractor after an open
competitive RFP has been issued.  It must be understood that no
part of this process is independent but is closely linked to and an
integral part of the entire process of defining a requirement to be
done under contract and seeing the contract work through to
completion.  In this report, "technical skills" refers to survey
and programmatic content skills as distinct from management and
administrative aspects of the procurement process.

Chapter 2 deals extensively with general issues related to the
technical skills involved in survey planning and conduct and in the
use of contractors for the performance of various tasks in the
process.  The remaining chapters follow an order of activities in
the use of contracts.  Chapter 3 contains a review technical survey
design issues and other considerations in preparing RFPS.  Chapters
4 and 5 deal with issues in the selection of contractors and the
management and evaluation of contracts after award.

Chapter 2 is a detailed discussion of the skills required in
contracting for surveys and the types of individuals who may serve
as technical advisors.  This chapter also includes a discussion of
the activities and concerns in the planning and preparation for a
survey to be done under a contract, the competitive selection of a
contractor, and the monitoring of performance as these relate to
the skills required at various stages in this process.  A variety
of technical survey research skills, as well as content area
competencies, is needed to deal with issues such as determining the
overall research design to be followed, the design of the data
collection instrument or instruments, the universe to be studied,
the design of the sample, the conduct of the data collection effort
itself, data editing, coding, and quality control.

At each stage of a survey performed under a contract, there are
various technical skills required to maintain a smooth operation. 
Differences of in-house technical capabilities of agencies and the
possible need to supplement those skills are discussed.  Both the
use and misuse of technical advisors by agencies are considered.

                                 4





Chapter 3, on considerations in the preparation of RFPS, contains a
review of the tasks that may need to be performed in preparing a
statement of work for competition among prospective offerors rather
than the skills that need to be available to the agency in
undertaking this effort.  The chapter includes a discussion of
issues to be considered in developing an RFP.  The interaction
between technical requirements and degree of specification and the
contract reimbursement method (that is, fixed-price or cost-type
contract) is also reviewed.  Different types of contracts imply
different assumptions about the ability to predict and specify
activities, results, and costs and, therefore, impose different
limitations on the ability to trade off between cost and
performance.

In Chapter 4, on the contractor selection process, the selection
process for open market competitive RFPs is discussed.  The various
stages of the contractor selection and contract negotiation process
and the roles played by the contracting officer, project officer
and technical evaluation panel are described.  This is the
framework within which the technical merits of the work proposal
are presented and evaluated, and in which a contract is awarded. 
The process of debriefing unsuccessful offerors is also discussed.

The last of the substantive chapters deals with contract management
and evaluation after award.  This is the period of actual
contractor performance in line with the terms and requirements of
the contract.  The project officer and the contracting officer are
jointly involved in monitoring and controlling the contract effort. 
The discussion in the chapter centers on the roles of these
individuals and contact that should be maintained between the
project officer and the contractor in monitoring these activities.

Depending on the specifics of the contract and the background and
competence of the project officer, this person's input into the
technical aspects of the survey will vary.  There may be specific
activities which require approval; data collection instruments
usually require formal clearance.  The project officer must certify
invoices for payment and must, therefore, be aware of contractor
progress and current and planned activities under the contract as
these relate to the remaining time and funds available.

In addition to these four substantive chapters, there are two
appendixes.  These are a review of the informal survey of agencies
and contractors conducted by the Subcommittee, and a discussion of
the OMB clearance process for surveys.

                                 5








  Chapter 2. TECHNICAL SKILLS REQUIRED IN CONTRACTING FOR SURVEYS

INTRODUCTION

A survey is generally a rather complex activity.  Furthermore, the
results to be derived from a survey are usually difficult to
specify, for example, types of data and analyses desired. 
Consequently, the task of contracting for a survey is not
straightforward.  Specifically, a considerable amount of effort is
required to provide complete and clearly written survey
requirements in a Request for Proposal (RFP), select the best
offeror to receive the contract, and adequately monitor and review
the progress of the survey.

Technical expertise is essential to the success of contracting for
and monitoring the execution of a survey.  In this chapter,
technical skills that should be available when contracting for
surveys are presented.

Following are areas in which technical skills are required:

-    Developing project objectives and specifications, Subject
     matter knowledge,
-    Project cost and scheduling,
-    Questionnaire design and testing,
-    Sample design and selection,
-    Data collection, including follow-up procedures,
-    Data processing, including coding, editing and file creation,
-    Data analysis,
-    Quality control, and
-    Survey documentation and report preparation.

The subject matter knowledge required for a particular survey will,
of course, vary depending on the nature of the survey.  As
mentioned previously, surveys deal with many subjects, for example,
demographic, economic, psychological, sociological, and medical
topics.  The other types of technical skills listed above are more
generally applicable to all surveys.

It is very unusual to find all the skills required for a particular
survey in one individual, although it is often true that some
people will possess more than one of these skills.  Consequently,
it is usually necessary that there be a Government project team
consisting of the project officer and one or more technical
advisors to supply the required technical skills.  The number and
types of advisors needed will vary according to the size and
complexity of the survey being planned and the experience of the
project officer.  Smaller and less complex surveys will usually
require fewer, and less specialized advisors than large, complex
surveys.

                                 7





It is important that the need for a project team and the types of
members needed be recognized early in the development of a project. 
At least some of the advisors should participate with the project
officer in the following major phases of the survey contracting
process: (1) project planning and preparation of the RFP, (2)
evaluating proposals and awarding the contract, (3) and monitoring
the work done under the contract.

In many cases the agency contracting for a survey will not have on
its staff all the technical advisors needed.  In such a case, the
agency should use technical advisors as part of the project team. 
These advisors could be persons employed by other Federal agencies
or nonfederal consultants.  Nonfederal consultants who assist in
development of the RFP are not eligible to respond to the RFP.  If
advisors are needed, they should be brought in at the beginning of
the contracting process so that they can participate in planning
the project, in preparing the RFP, and in evaluating the proposals.
if advisors are brought into the project early, they can provide
the guidance and assistance needed in designing and executing a
survey.

It is generally a mistake to rely solely on the contractor to
provide the technical expertise required during the execution of
the survey plan and the processing and analysis of the data.  If
this is done, the technical quality of the survey may be diminished
because the contractor's technical expertise may not be fully
adequate, or the contractor may decide that it is not in his or her
best interest to take steps to ensure the highest technical
quality.  Even with the highest quality contractor, the Government
should still provide technical review and guidance as part of the
monitoring of the survey process; otherwise, the value of the
project may be reduced.

The technical aspects of some phases of a survey are often not
fully appreciated.  For example, in the design of the data
collection activities of a survey, a reasonable time schedule
cannot be formulated without regard to technical input on follow-up
activities, such as the number and type of follow-up attempts
required.

The next section of this chapter contains a discussion of the
required technical skills listed earlier.  The following section
covers the type of technical input required at the three major
stages of the contracting process.  The final section includes a
discussion of some of the technical aspects of current practice in
contracting for surveys.


BASIC TYPES OF SKILLS REQUIRED

Developing Project objectives and Specifications


The formulation and definition of project goals and objectives is
the most fundamental aspect of a project.  Often goals and
objectives are not defined adequately.  In such cases there will be
confusion regarding the type and amount of survey information to be
collected, and hence the strong possibility that the project
objectives will not be met.  Sometimes a project may have multiple
objectives that may be somewhat conflicting.  For example, if both
subgroup and total population estimates are important, a decision
must be made on how to allocate the sample to these subgroups.  The
sample allocation that would provide equal precision, or some,
other specified precision, for subgroup estimates will generally
differ from the sample allocation that would maximize the precision
for total population estimates.  Priorities must be set for each
established goal and objective at the beginning of the project in
order to resolve any conflicts that may subsequently evolve.

                                 8





In addition to stating the objectives clearly, an effort should be
made to define the objectives to be as narrow or specific as
possible.  If the objectives are too broad or general, the survey
might become too large in scope to manage adequately for the time
and funding available.

If the goals and objectives are defined adequately and if the study
population has been defined, then appropriate technical
specifications can be developed for instrument design, sample
design, and data analysis. consequently, these technical aspects of
the project should be kept in mind as the goals and objectives are
developed.


Subject Matter

As mentioned previously, sample surveys cover many different
subjects.  In order to formulate and develop a project adequately,
there should be a subject matter expert on the project team at the
start of the project.  Such subject matter expertise is required in
order to insure that all data requirements. are delineated.  The
subject matter expert may not know the appropriate manner in which
to phrase a question in order to elicit the desired response, but
will know the fundamental issues that must be explored in order to
satisfy the data requirements.  For example, to develop a survey to
investigate the accuracy of the amounts paid to railroad retirement
beneficiaries, there should be someone on the project team with
knowledge of the railroad retirement system.  This subject matter
expertise should allow for the formulation of the most appropriate
survey questions and should enable the project to collect adequate
information from respondents.


Project Scheduling and Costing

Scheduling and costing various phases of a survey are generally
difficult.  Typically, a gross cost for the project is set very
early, particularly for one-time surveys.  Agency personnel are
generally limited by the initial gross cost allocated to the
project.

The amount of time required for planning and writing the RFP is
often underestimated.  This can delay the scheduled start of the
survey which, in turn, can jeopardize the quality of the work. 
Sufficient time for execution of the survey by the contractor must
be allowed.

Since there are several aspects of a survey, such as printing, data
collection, and data processing, that may require special
clearances, applicable requirements should be determined as early
as possible in development of a survey to be conducted under
contract.  The procurement office can provide a list of required
clearances so that their applicability to a particular survey can
be determined and provision for compliance can be made in the
project schedule.

In addition, there are several survey tasks which involve aspects
for which the completion time and cost are typically hard to
predict.  For example, a survey may involve a large frame
development phase in which a number of frame sources have to be
investigated.  This could require working with a number of computer
files involving problems of incompatibility between various
computer systems, different record layouts and file definitions,
and matching elements in different files.  These types of
activities are typically difficult to assess in terms of time and
cost.

Other survey activities that are typically difficult to assess in
terms of cost and time requirements are questionnaire design and
testing, data collection, and data processing.  Due to the
potential for cost and scheduling problems as indicated above, it
is important during the project development phase to include a
person on the

                                 9





project team who has experience in developing project schedules and
cost estimates for surveys.


Questionnaire Development and Testing

Proper development of a survey questionnaire and other data
collection instruments is a deceptively difficult phase of a
project.  The knowledge, time, and effort required to complete this
phase adequately is often underrated.  In addition to preparing the
instruments, time must be allowed to adequately test them.  Testing
is important since it is generally impossible to anticipate all the
problems that will arise when an instrument is used.  It is
important to include someone on the project team who has had
considerable experience in questionnaire development and testing so
that the common pitfalls of instrument design can be avoided and so
that the instrument may be designed to facilitate data processing.

If a questionnaire is used to collect the survey data and the
questionnaire is not developed by someone with appropriate
experience, it is likely that some questions will be unclear to the
respondents, will be misinterpreted by the respondents, will
confuse or upset some respondents, or will tend to influence or
"lead" the respondents.  The position of the question on the
questionnaire may influence the response rate to the question.  For
example, sensitive questions, like income, are usually answered at
a higher rate if they are asked after the respondent has become
more comfortable with the interviewer.  Also, the nature of the
questions surrounding a question of interest may influence response
proportions by several percentage points.  Furthermore, the
questionnaire may be difficult to administer and may not provide
adequate survey information.  In addition to operational
difficulties, these problems can cause both response and
nonresponse biases in survey estimates.


Sample Design and Selection

Even for a survey that seems straightforward in terms of the
information to be gathered and the definition of the target
population, there are usually some difficult questions that arise
in designing the sample selection method, particularly if the
survey has multiple objectives.  Questions about sample size
determination, frame development, stratification procedures, sample
allocation to strata, or whether or not to use differential
probabilities of selection arise in all surveys.  These questions
become particularly difficult to answer when there are conflicting
objectives.  For example, if the project team has not decided
whether subgroup estimates are more important than overall
estimates or vice versa, satisfactory decisions on stratification
and allocation cannot be made.

,Many of the questions that arise are not easily answered by
referring to standard textbooks.  Consequently, the project team
should include a person who not only has. the theoretical knowledge
of sampling methods, but who has also had substantial experience in
the practical application of sampling principles to the design of
the type of sample survey involved in the contract.  This type of
experience would include, for example, knowledge of available frame
sources and of data available to use for strata formation or for
assigning differential probabilities of selection.


Data Collection

The data collection phase of a project contains several aspects. 
First, the mode of interviewing - personal visit, self-administered
questionnaire (usually mail), or telephone - should be selected
based on the type of questions to be asked, the population to be
studied, and the time and resources available.  Next, interviewers
must be recruited and trained for the in-person or telephone
administration of a

                                10





questionnaire.  Even if a staff of experienced interviewers is
available, some training for each new questionnaire is necessary.

A budget and schedule for interviewing, including callback
procedures, must be developed.  For personal interviews, the
schedule often includes a mailing or telephone call to the selected
sample units prior to the personal interview.  The number and
timing of callback visits allowed in the schedule will have an
important impact on the survey response rate, which, in turn, is a
key factor in controlling nonresponse bias.  Acceptable response
rates need to be determined and follow-up procedures must be
devised and implemented.

Another aspect of data collection is questionnaire check-in.  In
addition to recording the receipt of the questionnaires, this
involves the development of editing procedures to check for faulty
or missing data.  Procedures also have to be developed to follow-up
on questionnaires that fail this edit.  These procedures are
important in order to improve data quality and reduce the response
and nonresponse biases.


Data Processing

The data processing phase usually involves entering the survey data
into a computer file so that it can be accessed and analyzed.  The
first step is to provide any coding and editing of questionnaire
responses that are needed.  The coding is needed to convert all the
responses into numerical codes for entry into a computer file.  For
example, if occupation was a survey item, a numerical code would be
assigned to each occupation to use in recording the responses on
the file.  Proficiency in this activity requires considerable
practical experience in coding and editing of survey data and
knowledge of computer systems.

Editing is necessary to detect responses that are unreasonable or
that are inconsistent with other responses.  The editing may either
be performed manually or by use of the computer.  Developing
editing rules is difficult and requires input from subject matter
specialists.  Responses that fail edit checks can either be
followed up, replaced (imputed for) or discarded.  All imputed data
should be flagged on the file.

The survey data are placed into a computer file by having the coded
questionnaire responses keyed to a specified record format.  It is
important that the computer record layout be documented completely
and accurately so that the data will be correctly accessed.


Data Analysis

An analytic plan and strategy should be developed in the earliest
stages of the survey development process.  Often included in the
plan are specific table shells to be used in survey tabulations. 
This plan is the foundation for sample and questionnaire design and
is essential for the survey to provide the results intended.

Often the survey analysis phase includes the estimation of
population and subpopulation characteristics (for example, means,
totals, and proportions).  In addition, there are many types of
more complex statistical analyses that may be applied to survey
data, depending on the objectives of the survey.  Some examples of
analyses performed are the following: hypothesis testing on
population means or proportions, regression and correlation,
analysis of variance, and log-linear analysis.

Before statistical analyses are applied to survey data, imputed (or
pseudo) responses are often inserted in the records of respondents
that have missing or faulty responses.  Setting up an imputation
procedure for missing data is a complex process that

                                11





requires experience.  Also, differential probabilities of selection
should be taken into account in applying statistical analyses. 
This is often accomplished by assigning weights to survey
respondents to reflect varying probabilities of selection.  These
weights are usually adjusted to account for eligible sample units
that do not respond.

Once the survey estimates and statistical analyses have been made,
the results must be examined.  Study conclusions, and perhaps
recommendations, are then made.  The data analysis phase is a very
important part of the project and requires the assistance of
appropriate subject matter experts and of a statistician who has
had experience in analyzing data from surveys.


Quality Control

During virtually any survey operation there are many points when
data collection or processing errors can be made that diminish the
accuracy of survey data.  These instances include sample selection,
interviewing,and/or completing the survey instrument, check-in and
field editing, coding and office editing, and keying the survey
data.

It is strongly recommended that quality control procedures be set
up at each step to detect and correct as many of these survey
operations errors as is feasible.  At a minimum, the quality
control program should cover interviewing, field editing, coding,
and keying.  These procedures should be set up by a person who has
had experience with survey quality control procedures.


Survey Documentation and Report Preparation

It is important throughout the project to prepare enough project
reports to adequately document the survey procedures, results, and
conclusions.  Poor documentation is a very common problem with
survey projects.  One reason for inadequate documentation is that
project funds often run low and time becomes short near the end of
a project.  A conscientious effort should be made to allow adequate
time in the project schedule for ongoing documentation of the
survey procedures used.

These procedures include definition of the target population and
frame construction; interviewing procedures, including callback and
follow-up procedures; sample design and selection; editing and
coding procedures; weighting and nonresponse adjustment methods;
estimation and variance estimation equations and statistical
analyses used; and quality control methods applied.

The final project report should include survey results, analyses,
conclusions, recommendations based on the results, and suggestions
for additional research.  Project specialists, such as subject
matter specialists and statisticians, should be involved in
preparation of the final report.


TECHNICAL INPUT AT VARIOUS STAGES OF CONTRACTING FOR A SURVEY

As mentioned earlier, substantial technical input is needed at each
of the following three major phases of contracting for a survey:
project planning and preparation of the RFP; evaluating proposals
and awarding the contract; and monitoring the contractor's
performance.

                                12





Unfortunately, the importance and/or extent of technical input
needed at these stages is often underestimated, thus diminishing
the quality of the survey.  The required inputs are discussed below
for each of these major phases.


Project Planning and Preparation of the RFP

Right from the start of the process of contracting for a survey, it
is important to have technical input and program and subject matter
expertise.  Subject matter experts and statisticians can help
define specific project goals and objectives that will satisfy the
purpose of the study and yet will be achievable, within the time
and budget available for the project, by applying appropriate
survey and statistical methods.

A statistician should participate in the development of the RFP so
that it will be complete and accurate in terms of the statistical
requirements of the survey.  In cooperation with the project staff,
the statistician should prepare or assist with the preparation of
the following portions of the RFP;1

     -    Sample size specifications, if any, for the target
          population and/or for population subgroups,
     -    Precision requirements for one or more key survey
          estimates, including any requirements for population
          subgroups,
     -    Response rate required and/or number of callbacks
          specified,
     -    Requirements for nonresponse imputation procedures,
     -    Quality control procedures,
     -    Variance estimation requirements, and
     -    Specification of particular population or subpopulation
          estimates required.

If a statistician does not participate in the preparation of those
portions of the RFP that address the above topics, it is virtually
certain that some of these specifications will be omitted from the
RFP, and some of those included will not be properly stated.  In
such a case, offerors would be confused and may prepare proposals
that miss the real objectives or which are somewhat vague,
incomplete, or misdirected in terms of important technical aspects
of the proposed survey.  This, in turn, would make it more
difficult to compare the proposals and to accurately evaluate them
in terms of technical content.


Judging and Selecting Among Offerors

Technical input is essential in the evaluation of the proposals. 
In addition to subject matter specialists, at least two persons
with varying experience in the different aspects of a survey should
be included on the proposal review panel.  At least one of these
persons should be a statistician.

A major focus of the technical review of proposals is the offerors'
responsiveness to the survey specifications, such as frame
construction, sample design and selection, quality control plans,
and estimation and analysis plans.  Just as important as evaluating
these technical responses is the assessment of the offerors'
abilities to fulfill the statements and claims made in their
proposals.
___________________________

1For a general type of RFP the technical input discussed above is
still needed.  However, some of these items would not usually be
specified.

                                13





For example, if they claim that they would achieve an 85 percent
response rate, is there sufficient evidence to support this? The
basis for this type of evaluation would primarily be the previous
experience and perforMance of each offeror and the technical skill
and experience of the proposed project staff.  Of course, the
survey plans in the proposal would also be used to help assess the
offerors' potential to achieve the proposed statistical standards.


Monitoring the Contractor's Performance

After a contract is awarded and the project begins, technical input
is still very important to the success of the project.  The
contractor will presumably be describing details of their survey
plans at meetings and in written documents.  Technical personnel
representing the agency should attend all project meetings which
involve technical issues and should review all project documents to
ensure that specified technical standards are met.  Even if the
contractor's project staff appears to be technically sound, it is
necessary for the agency to continue technical input.  This should
ensure that the standards of the contract are met and may avoid
technical problems that might have appeared near the end of the
project if technical input had been interrupted.

Any problems on technical issues between the agency and the
contractor should be discussed and resolved.  If resolution of
these problems requires changes in the specifications, the changes
should be proposed to the contracting officer for decision and
formal action.  To facilitate resolution of problems, there should
be an open line of communication between the contractor's
statisticians and other technical staff personnel and a liaison for
the agency's project staff.  Good communication between contractor
and agency staffs can prevent substantial misunderstandings on
technical issues that may arise.  Ideally, there should be enough
discussion of technical issues between the agency staff and
contractor staff so that when the contractor submits a technical
document, such as a sampling or analysis plan, agency personnel
will be aware of the major technical aspects of the document.  If
communication has not been good and major problems with technical
issues arise, there may be substantial time delays in the project
work.

In addition to participating in meetings and reviewing documents,
in some surveys it is useful for agency technical personnel to make
site visits.  The purpose of a site visit might be to observe and
review the implementation of sampling procedures, interview
activities, coding and editing, or quality control activities.


SOME TECHNICAL ASPECTS ASSOCIATED WITH CURRENT PRACTICE

Use and Misuse of Technical Advisors

As expressed previously, the use of technical advisors in
contracting for a survey is vital to the success of the project. 
Although project managers generally recognize this need, technical
advisors are not always used properly.

In the previous section it was pointed out that it is important to
have technical input and program-specific expertise from the
beginning of the process.  Often the technical advisors are added
to the project team too late to assist with the preparation of the
RFP.  In such a case, the specifications of the survey may be
inadequately defined.  For example, the precision requirements for
the survey may be unclear, causing confusion and apprehension among
offerors.  As another example, the minimum acceptable response rate
may have been omitted, which would probably cause problems in
comparing proposals.

                                14





Technical advisors are generally called upon to review and judge
the technical proposals submitted.  However, in addition to rating
proposals on technical aspects with which they have knowledge and
or experience, members of the review panel are also asked to rate
proposals with respect to technical aspects with which they are
less "familiar, or even unfamiliar.  For example, a statistician
who specializes in sample design may be asked to rate the proposals
with respect to questionnaire design and quality control
procedures, in addition to sampling aspects.  The reviewer's
experience may be rather limited in areas other than sampling.  As
a result, ratings of these aspects of the proposals might be
inappropriate.  To try to minimize this problem, individual
reviewers, during the discussion and review of ratings, may revise
their original score on an item to reflect the opinion of another
reviewer with particular expertise in the area.  To allow for this
type of rating adjustment, it is advisable for an agency to attempt
to balance the members of the review panel with respect to the
important technical aspects of the proposals that need to be rated.


Use of Outside Help to Supplement in-house Technical skills

For many surveys being contracted, the in-house technical skills
needed to design the survey, to judge offerors, and to monitor
survey progress, are not fully adequate.  In such cases technical
advisors from other Federal agencies or nonfederal consultants
should be added to the agency's project team.  For example,
advisors specializing in sample" design, questionnaire design, or
field operations may be needed. it is, of course, important that
the advisors required are added to the project team early in the
development of the survey.

As was indicated earlier in this chapter, the agency should have
its own technical advisors to provide the technical expertise in a
particular area, rather than relying solely on the contractor's
expertise.  Even if outside consultation is needed, it is important
to have the appropriate technical advisors on the agency's project
staff to protect the agency's interests.

There are a number of Federal agencies that have technical staff
who should be able to provide assistance directly or to suggest
others with appropriate skills.  These agencies include the
following: Bureau of Economic Analysis, Department of Commerce;
Bureau of Labor Statistics, Department of Labor; Bureau of the
Census, Department of Commerce; Bureau of Justice Statistics,
Department of Justice; National Center for Education Statistics,
Department of Education; National Center for Health Statistics,
Department of Health and Human Services; Office of Research and
Statistics, Social Security Administration, Department of Health
and Human Services; Statistical Reporting Service, Department of
Agriculture; Statistics Division, Internal Revenue Service,
Department of the Treasury.

                                15





       Chapter 3. CONSIDERATIONS IN THE PREPARATION OF RFPS


INTRODUCTION

This chapter covers some considerations in the preparation of a
Request for Proposal (RFP) for a survey.  These comments and
suggestions are not all-inclusive; additional considerations will
be warranted for some RFPS.  For example, certain types of
surveys - such as repetitive or longitudinal surveys - pose special
problems that are not addressed here.  All of the possibilities
cannot be covered because of the wide range of purposes and survey
designs encountered in surveys sponsored by the Federal Government. 
Emphasis is placed on elements which are included in many of the
RFPs prepared in Federal Government agencies and on those portions
of the RFP to which the project officer or other technical
personnel are most likely to contribute.1  On the other hand, the
comments and suggestions which are presented are not likely to be
applicable to every RFP, either because of the nature of the
proposed surveys or because of variations in regulations, policies
or procedures among government agencies.


INITIAL CONSIDERATIONS

Because of the variations noted above, it is essential to contact
the procurement off ice that services the sponsoring agency at the
conceptual stage when a survey contract is contemplated.  Early
discussions with the contracting office will enable determination
of the contracting options, if any, which are available, the input
expected from the program staff, the steps which should be followed
from development of the RFP and a list of potential offerors
through award of the contract, and the amount of time each step is
likely to take.


Payment Provisions

One option that might be available concerns the payment provisions
of the proposed contract.  A contract may stipulate a fixed-price
which is agreed upon in advance by the sponsor and the contractor,
or it may provide for reimbursement of allowable costs (up to a
preset maximum) incurred by the contractor in performing the
survey.  The decision about which approach to use is usually made
by the procurement office based on technical contracting
considerations, not by the program staff. Some agencies use only
one of these financial arrangements for all of their contracts;
some agencies use one or the other, depending" on the nature of the
survey to be undertaken.  The RFP should indicate whether the
proposed contract will stipulate a fixed price or will provide for
reimbursement of costs incurred by the contractor.

There are advantages and disadvantages to each of these payment
mechanisms.  For example, fixed-price contracts assure, in advance,
that sufficient funds are available for completion of the work and
keep the sponsor's administrative audit expenses to a
___________________________

1A complete RFP contains a number of standard provisions or
sections originating in the contracting office to which the program
staff have little or no input.

                                17





minimum.  However, some qualified potential offerors will not
submit a proposal for a fixed-price contract and competition is
therefore reduced. it is particularly important that fixed-price
contracts contain comprehensive, detailed statements of the
requirements for the work to be performed by the contractor to
avoid misunderstandings about what the agreed-upon price actually
covers.  These requirements should be based on a combination or
merging of the specifications in the RFP and the contents of the
successful offeror's final proposal.

If there are many unknown or undecided factors in the project plan
that affect the predictability of costs, a cost-type contract may
be preferable, provided adequate funding will be available.  A
"dollar cap" or maximum cost can be included in a cost-type
contract.  If a cap is specified, flexibility should be built into
the survey design so that if avoiding cost over-runs, which
jeopardize completion of the survey, is not possible, a limited but
still useful product can be obtained.


Phased Surveys

If there are any serious doubts about the feasibility of key
aspects of the survey plans because of unknown factors such as the
ability or willingness of designated respondents to provide the
information sought, it may be prudent to limit the scope Of the
project to which the sponsor will be committed.2

One approach which might be appropriate in this situation is to
issue an RFP solely for an evaluation of the proposed methodology,
with the stated intention of issuing another RFP for a full-scale
survey if the initial effort indicates the methodology is feasible. 
Obviously, this approach would consume more time because of the two
RFPs involved.

Another possible approach is to specify work segments which may be
done during the course of the contract if the concerns are resolved
in the earlier stages of the contract.  If a phased survey is
deemed advisable, early discussions with the Procurement office
will indicate what contracting alternatives are available for
consideration and their advantages and disadvantages.


Survey Design Specification

RFPs may state all, some, or none of the details of the survey
design.  The level of survey design specification found in RFPS, in
practice, ranges from "This is exactly what we want, including the
questionnaire to be useful" to "We have a problem and need
information to help solve it; what do you suggest?" The choice of
the level of specification is dictated by a number of
considerations such as the nature of the survey to be conducted,
the applicable regulations and policies, and the technical skills
available among the sponsor's staff or advisors.

Detailed survey design specifications may be appropriate if the
sponsoring agency has highly qualified and experienced statistical
and survey staff available to develop the RFP who are aware of
potential contractors' capabilities and resources, and who know
exactly how the agency wants the survey to be conducted.
___________________________

2Although the general provisions incorporated in Federal survey
contracts usually provide for termination of the contract for the
convenience of the Government, exercising provision can be
difficult, complex, and time consuming.

                                18





Such an RFP may be more difficult and time consuming to develop
than one that calls for offerors to suggest survey designs. 
Maximal specification of the survey design will indicate the
quality standards of the sponsor to potential offerors, and will
also permit them to repeat the requirements in their proposals
without demonstrating capability or understanding of the problem. 
It may be easier and quicker to evaluate the responses to a
detailed RFP, but more difficult to determine the technical
capabilities of the offerors.  Selection is therefore more likely
to be made primarily on price.  Any unnecessary constraints imposed
on the survey design would preclude desirable flexibility and
innovation by offerors which could improve the quality of the
survey or decrease its cost.  Furthermore, some of the more
competent survey research organizations dislike this approach and
are reluctant to respond to this type of RFP.

Less detailed RFPs which, invite offerors to make suggestions or
develop survey designs, allow them to apply their knowledge and
demonstrate their ingenuity or creativity.  However, minimum design
criteria or quality standards should be stated, when known, even if
the offerors are asked to develop the survey design.  Evaluating
the proposals received in response to a less detailed RFP is likely
to be more difficult and time consuming.  When an RFP does not
specify the details of the survey design, the successful offeror's
explicit statement of the survey design specifications is commonly
incorporated in the contract awarded.

An alternative that incorporates many of the advantages of both the
detailed and less detailed survey design specification approaches
has been used effectively by some agencies.  In this procedure,
potential offerors are invited to submit proposals based on
detailed specifications but are also encouraged to suggest
modifications to the detailed plans which they believe will be
advantageous to the sponsoring agency.


Bidders' Conferences and Due Dates

Some RFPs - usually only those for large or complex surveys -
include provision for a Bidders' Conference on a specified date to
which prospective offerors are invited.  The conference should be
held after prospective offerors have had the opportunity to study
the RFP, but well before the due date for proposals.  They may be
asked to submit their questions beforehand so that answers can be
prepared in advance for the conference.  At the meeting, technical
and procurement personnel from the sponsoring agency answer
questions raised by potential offerors about the RFP or arrange to
provide answers later for any issues which cannot be resolved
during the conference.  They may also review contractual
administrative requirements.  Discussions at a Bidders' Conference
will benefit the sponsoring agency if they identify RFP
specifications which are misunderstood or which appear to potential
contractors to be inconsistent, inadvisable or unacceptable.  Any
additional information or material developed as a result of the
Bidders' Conference should be mailed to all requesters of the RFP.

It is very important in any RFP to allow a reasonable amount of
time for preparation and submission of proposals - at least 30
calendar days after the RFPs are mailed and more than 30 days if
the proposed survey is complex or offerors are asked to develop
substantial portions of the survey design.  The availability of an
RFP is announced through the Commerce Business Daily. it is also
advisable that the names and addresses on mailing lists of those to
receive the RFP are current.

                                19





POSSIBLE CONTENTS OF RFP's

When developing an RFP, it may be helpful to refer to RFPs for
similar surveys conducted by the sponsor or by another Federal
agency as sources of ideas for content and wording.  However, these
RFPs should be used together with comments or suggestions from
persons familiar with those surveys about problems or
misunderstandings which arose during the contractor selection
process or the conduct of the study - particularly difficulties
which might have been forestalled if the RFP had been written
differently.

The goals of the procurement process should be kept in mind when an
RFP is written.  Ideally, an RFP does the following:

     -    Requests the development of or specifies a technically
          sound survey design which will meet the sponsor's
          information needs and is realistic, considering the time
          and funds available for the project and the capabilities
          and resources of potential contractors.

     -    Conveys all the information needed by potential
          contractors to prepare a responsive proposal, including
          all the terms, conditions, and provisions that the
          sponsor intends to incorporate in the contract.

     -    Asks for all the information that procurement and
          technical personnel will need to conduct an accurate and
          equitable evaluation of the proposals received.

     -    Attracts responsive proposals from enough of the
          qualified potential contractors to provide adequate
          competition and to permit a suitable selection.

Some agencies include in the RFP a table of contents for responses
to be followed by all offerors and stipulate that staffing
requirements should be specified in a common measure (for example,
hours).  This simplifies reading and comparison of proposals.

Specific suggestions for developing an RFP that meets these
criteria are presented in the remainder of this chapter.


Statement of Survey Objectives

Every RFP should contain a clear statement of the specific
objectives of the survey to be conducted under the ensuing
contract.  Discussing the proposed research in the larger context
of the mission of the agency may also be advisable, particularly if
the project is part of an ongoing integrated program.  The
statement of the survey objectives should also indicate what groups
(such as government agencies, congressional committees, state or
local governments, trade associations, and business firms) will be
the key users of the findings.  If data from the survey or an
interpretative report of the findings must be provided in time to
meet a deadline, the date of and reason for the deadline should
also be stated.

Any germane reference material that would be helpful to potential
offerors in planning the broad outline of the proposed survey or in
estimating time, skill, or cost requirements should be cited and
also included in the RFP package.  If the material cannot be
provided with the RFP, information on where and when it can be
accessed or obtained should be supplied.  Appropriate reference
material might include such information as estimated numbers of
eligible respondents, geographic distribution of eligible
respondents, and the methodology and results of any related
research that has been conducted by or is known to the sponsoring
agency.  However, materials which

                                20





would be of interest only to the successful offeror should not be
included with the RFP.

Although RFPs sometimes state anticipated level of effort in terms
of person-years or similar proxies for dollar estimates, the
Subcommittee's interviews with contractors indicated that these
proxies are less helpful to them than dollar estimates would be
because of wide variations from agency to agency in the dollar
amount one person-year signifies.  If agency procurement policies
permit, it may be desirable to inform potential offerors of the
approximate amount of funds available for the contract.  Providing
offerors with this information would help avert some of the
difficulties that might otherwise be encountered in the contractor
selection stage, such as the problems that arise when the cost
estimates for all of the technically sound proposals received by
the sponsoring agency substantially exceed the funds available for
the contract.  If some but not all of the technically sound
proposals fall outside the anticipated price range, this reduction
in competition may result in a real but less obvious disadvantage
to the sponsoring agency.


Request for Information About the Offeror

The RFP should request the relevant information about the offeror. 
It is not sufficient for an RFP to simply define the requirements
for the technical proposals.  Before a contract is awarded, the
reliability of the contractor should be established.  When
appropriate, the following specific requests should be responded to
by the offerors before a contract is awarded.

Description of experiences.  The following list contains areas
where it could be important to obtain specific descriptions of the
offerors' experiences:

-    frame refinement processes,

-    probability sample designs, especially for large-scale
     surveys,

-    data collection methodology (including callback procedures),

-    questionnaire development and testing,

-    estimation procedures, including nonresponse adjustments,

-    variance computation procedures, especially for complex
     samples,

-    training and supervision of interviewers, coders, and others,

-    analysis of data from large-scale surveys,

-    data processing,

-    quality control, and

-    publications.


Examples of previous relevant work.  The examples of offerors' work
should include questionnaires, sampling documentation, instruction
manuals, and reports containing the results of the projects.  The
agency should request information on all previous work pertaining
to the subject matter areas for a specified period of time.  The
time period should be such that enough material is received but
that the offeror does not inundate the agency with paper.  In most
cases, five years should be a reasonable period for both the
offerors and the agency.  Included with this information should be
the name and current telephone number of the sponsor's
representative (or government project officer).  At least a sample
of these references should be verified by the agency before the
award of any contract.

                                21





Staff experience.  This should include the experience of the
current staff.  This could differ substantially from the company's
experience.  Their background in statistical methodology, previous
contracts, and subject matter areas, especially the ones related to
the RFP, should be stated.

Key project staff.  The names of key project staff members should
be required along with the percent of time each one will work on
the project.  Their resumes should be included with the proposal. 
It may be desirable to have a key personnel clause stating which
staff members cannot be replaced on the project without government
approval.  Letters of intent should be provided by key persons who
are not members of the offeror's firm.

Field staff.  The interviewers' field operations background in the
primary sampling units (PSUs) in the frame from which the sample
will be drawn should be requested.  Also their experience in
handling nonrespondents should be stated.  The background of
supervisory and clerical field staff should also be requested.

Conflicts of interest.  The RFP should contain a section requesting
information that will allow the agency to determine if any possible
conflicts of interest exist.  This section could include questions
about the offeror regarding other clients, activities of staff
members, subsidiaries, or parent companies.

Facilities.  In the RFP, the agency should request information
about the potential offerors! facilities in order to make certain
they are adequate to complete the contract.  This is especially
true of computer facilities.  The offeror should have the computer
capabilities to handle all data processing requirements.  If
necessary, this should include a requirement of compatibility with
the agency's data processing facilities.

Subcontracting.  If an offeror plans to use a subcontractor in any
aspect of performance under the contract, detailed information
should be provided on their experience, resources, and facilities. 
If subcontracting is anticipated but there is no identified
subcontractor, standards for subcontractor selection should be
specified.


Survey Design Checklist

The purpose of this section is to present a descriptive checklist
which contains topics that should be addressed or considered when
writing the survey design portion of an RFP.  As was stated
previously, the amount of specification is dependent on many
factors.  Ideally the amount of detail that each item in the
checklist requires will depend on the type of study to be
conducted.  For this reason, the following list should be used only
as a guide and the appropriateness of each item should be judged
separately for each RFP.  However, if a design topic is not
specified in detail, it should be mentioned as an item to be
discussed by the offeror.  If an item is not mentioned in the RFP,
the agency might not be able to use it as a criterion for
evaluating the proposals.  In addition, the importance of
consulting a qualified sampling statistician before proceeding with
this section cannot be overemphasized.


Design and selection

     Definition of the target population and key subgroups.  The
     objectives of the survey should define the population that is
     intended to be covered.  However, it may be necessary to
     eliminate certain segments of the population and thereby
     redefine the scope of the survey because of practical
     problems.  Examples would be excluding Alaska and Hawaii
     because of the high cost of collecting information by personal
     visit, excluding nonrepriceable items in the production of a
     price

                                22





     index, and excluding nontelephone households when telephone
     interviewing is the primary method of data collection.

     The survey results will apply only to the population sampled. 
     However, it might be possible to obtain estimates for the
     excluded populations by using alternative collection methods
     or by using auxiliary data which are correlated to the
     information gathered in the survey.  The RFP could require
     that proposals include a method of estimating for the excluded
     population.  Any conclusions for these groups would have to be
     stated separately and their limitations would have to be
     noted.

     A description of any subpopulations for which estimates are
     desired should be included.  Examples could be geographic and
     demographic detail.

     Sampling frame.  Ideally the sampling frame should fully cover
     the target population.  Every effort should be made to.ensure
     that the source is as current and as complete as possible. 
     Consideration should be given to requiring procedures for
     updating the available source, including merging information
     from different records and removing duplicates.  If more than
     one frame is available, a decision must be made on which one
     to use or whether to use multiple frames in the survey.  If
     multiple frames are used, a method to identify and handle
     overlap in the frame must be developed.

     Sample selection.  The RFP should specify that a probability
     sample be selected for the survey and that the sample design
     be adequately defined.  This includes a description of the
     proposed sampling plan for each stage of sampling.  The
     details of     this description should include the following
     topics for each stage of selection:

     -    Definition of the sampling unit.  A sampling unit is a
          cluster of one or more elementary (or observation) units. 
          Each elementary unit should belong to exactly one
          sampling unit.  In some RFPs the sampling units for some
          of the stages of selection (for example, the primary
          sampling units or PSUS) are specified.  Any sampling
          units that are not specified in the RFP should be clearly
          defined in each proposal.  Examples of sampling units are
          counties, enumeration districts, households, persons,
          companies, and retail outlets.

     -    Number (or expected number) of sampling units to be
          chosen.  In many cases the RFP gives the target final
          stage sample size (for example, 5,000 completed
          interviews) and sometimes the first stage target sample
          size (for example, 100 PSUS).  A proposal should indicate
          all selection stage sample sizes not specified in the
          RFP.

     -    Specific method of selecting units from the frame.  The
          method of selecting sampling units at a given stage (for
          example, simple random sampling, systematic sampling, or
          stratified sampling) should be clearly specified so that
          the probabilities of selection of the sampling units can
          be derived.  In addition, any variation from strict
          probability sampling that an offeror feels is required
          should be clearly described and justified in the
          proposal.  Examples of cases for which variations from
          probability sampling may be appropriate are unavoidable
          coverage deficiencies, or the selection of a major brand
          to represent a product line in a price survey.

          Offerors should base their choices of sampling units,
          sample sizes, and selection methods for the various
          stages of a sample design on optimization. considerations
          (that is, maximum precision per unit of cost).  The RFP
          should require that proposals include a justification for
          all the sample design choices made.

                                23





          It is essential that at least one sampling statistician
          with experience in the particular area of interest assist
          in the preparation of the sampling portions of the RFP.

          It should be noted that unless probability sampling is
          required in the RFP, a proposal cannot be considered
          technically unacceptable just because it allows for some
          form of nonprobability sampling.  This could cause
          problems during the contractor selection procedure.  In
          addition, the Office of Management and Budget (OMB)
          requires justification if probability sampling is not
          used.

     Sampling error.  Two different approaches are commonly used in
     RFPs for sample surveys to specify the level of effort desired
     by sponsors to provide estimates that are sufficiently
     reliable for their purposes.  One method is to specify the
     sample size; the other is to specify the precision (that is,
     desired minimum level of sampling error) for estimates of one
     or more key variables.  It is not desirable or practical to
     specify both sample size and precision in the RFP, since the
     desired precision may be impossible to achieve with the
     specified sample size.  Whichever is specified, the other
     should be an important factor in evaluating the proposal.  The
     justification for using one method over the other is dependent
     on the study to be done and the resources available.

     The RFP often specifies the target sample size, usually in
     terms of the number of completed questionnaires or interviews. 
     The derivation of a target sample size may be based on cost
     considerations or on the approximate precision levels desired. 
     If a sample size is specified in the RFP, it is important to
     clearly define the final stage unit and whether or not the
     sample size is in terms of original selections, inscope
     selection, or completed interviews.  Any required sample sizes
     for population subgroups should be clearly stated.  Because of
     the different components of the sampling error, it may be
     necessary to require a minimum number of sampling units at
     each stage of selection.  Otherwise the resulting sample,
     although meeting all the specified criteria, might not yield
     acceptable results.  Finally, if the sample size requirements
     are in terms of completed interviews, the RFP should include a
     precise definition of what data must be collected before an
     interview is complete.

     If the sampling error is to be specified then a decision must
     be made whether it will apply to one key variable or more than
     one key variable.  If it applies to more than one, will there
     be a controlling variable, that is, the design which meets
     requirements for that variable will exceed requirements for
     all other variables.  In addition, it must be made clear
     whether the precision requirements are in terms of the
     standard error, the coefficient of variation (relative
     standard error), or 95 percent confidence levels (+ 2 standard
     errors).  Any misinterpretation of these requirements can lead
     to gross differences in the proposed sample sizes.  Also it
     must be made clear whether the required minimum sampling error
     will apply to specific subpopulations "or just the overall
     population under consideration. A qualified sampling
     statistician should be consulted to assist with the decisions
     made in these areas.


Data collection, questionnaire development and testing

This section of the RFP, more than any other, will require input
from the.people who handle the agency's OMB clearance process.  In
addition to knowing the agency's procedures regarding data
collection they are familiar with OMB's regulations. (See appendix
11 for a discussion of OMB's clearance and, Information Collection
Budget

                                24





requirements.) Their assistance can help in making the clearance
process go as smoothly as possible.

When preparing an RFP, the following issues concerning data
collection, questionnaire development, and testing, should be
considered.  While they may not apply in all cases, when they do
they will have an effect on the contract price.  Therefore, in an
attempt to avoid any misinterpretation by potential offerors, these
issues should be addressed when applicable.  Depending on the
situation, they can be either explicitly defined or stated in
general terms and then used in the evaluation of the proposals.

     Method of data collection.  The three most common types are
     personal (face-to-face) interview, telephone interview, and
     self-administered (primarily mail) questionnaires.  Each has
     specific advantages and disadvantages which must be
     considered.  They may also be used together in a survey.  An
     example would be a self-administered questionnaire and a
     personal interview follow-up.  It is important to make certain
     that the desired information can be collected successfully or
     more accurately and that the target response rates can be
     achieved by the proposed method or combination of methods. 
     Mail questionnaires seldom produce adequate response rates
     without some form of telephone or field follow-up.

     Data specifications or information to be obtained.  They
     should be listed in sufficient detail to indicate what the
     questionnaire content should be and what the amount of effort
     required in developing and testing the questionnaire will be. 
     They should be complete and realistic.  When the survey is
     covering a new topic, the sponsor should identify the
     variables for which data are needed, but should allow the
     contractor, once selected, to play a major role in developing
     the specific questions needed to get data on these variables.

     Pretest.  The need for a pretest is paramount in any survey. 
     Without it the chances of achieving good results are minimal. 
     The three following primary issues arise in a pretest which
     must be considered in any survey: sample size, probability
     sampling, and the survey time schedule.

     Sample sizes should be sufficiently large to learn about
     significant problems that may be encountered.  For this
     reason, specifying a pretest of nine respondents so that the
     pretest does not have to be cleared through OMB should be
     avoided.  Furthermore, clearance at an early stage of
     development may identify possible overlap of the planned
     survey with work of other agencies or may bring to light
     experience in other surveys that will help to avoid repetition
     of past mistakes.  It is better to determine the sample sizes
     needed without reference to clearance requirements and to
     ensure that the survey development schedule allows enough time
     for necessary clearances.

     While probability sampling at all levels of selection is
     desirable for nearly all surveys, it may not always be
     indicated for pretests.  Purposive sample designs are
     frequently used in pretests to ensure inclusion of important
     subpopulations.  Probability selection methods at the final
     stages (that is, households within block) are still desirable
     to avoid selecting the "easy to get" units.

     The survey time schedule should allow for adequate time and
     professional staff to both conduct and analyze the findings
     from the pretest and to use the findings effectively in
     subsequent stages of development.

                                25





     The following is a minimum list of topics which should be
     addressed in the planning of a pretest:

     -    Should exploratory interviews be conducted? The purposes
          might include identifying items that are difficult or
          sensitive to answer, determining if the data
          specifications are complete, or ascertaining what
          records, if any, are needed and if they are available.

     -    Will the pretest be conducted in waves? That is, will the
          pretest sample be sent out in parts, each of which
          contains questionnaires modified based on the findings in
          the previous part?

     -    Will the effect of the sequence of the questions be
          considered?

     -    How long is the average time for pretest interviews
          likely to be?

     -    Will the pretest determine the required experience level
          of the interviewers or is this one of the requirements in
          the RFP?

     -    What records, if any, will the respondent need to refer
          to and will they be reasonably accessible?

     -    What results are necessary before advancing to the next
          stage of the study?


     Respondents' obligation to reply.  Any information about the
     respondents' obligation to reply that would help the offerors
     should be included in the RFP.  Most surveys are voluntary. 
     However, if the survey is mandatory, if a response is required
     in order to obtain or retain benefits, or if informed consent
     is required, this could greatly affect the proposals and
     should be included.

     Length of interviews.  If the contractors are to develop the
     questionnaire, a ceiling on the interview time should be set. 
     A respondent may agree to cooperate only if this time is kept
     to a minimum.  Care should be taken to obtain the required
     information in this limited time.  To do this, a determination
     of the minimum number of questions needed to accomplish the
     survey's purpose should be made.

     Confidentiality.  Any questions regarding confidentiality or
     the Privacy Act system of records should be addressed in the
     RFP.  If there is a promise of confidentiality in the survey,
     the contractor may have to take special provisions to allow
     for it.  These could include locking up data and only allowing
     access by authorized personnel.  Special provisions regarding
     the security of the questionnaires might also be needed.

     Payments to respondents.  If there is to be financial
     compensation to respondents for completing a questionnaire,
     the offerors should not only be made aware of this fact, but
     also the reason for it.  Following are several reasons (each
     of which could have different effects on the bids) are:
     studies have shown that the success of the project depends on
     the agency providing compensation, or the data collection
     requires a significant amount of work on the part of the
     respondents and they should be compensated for their time.

     In addition, OMB and many Government agencies require a
     description or justification of any remuneration to persons
     supplying information.  If the success of the survey depends
     on these payments, approval should be obtained from both the
     Department and OMB before the contract is signed.

     Advance notice to respondents.  Many surveys require advance
     notice to the respondents.  This is so they can consult their
     records and documents, thereby

                                26





     making the interview as useful as possible.  If this time is
     needed, the offerors should be made aware of it so they can
     plan their resources accordingly.

     Dates for data collection.  There may be reasons that specific
     beginning and ending dates for the data collection must be
     set.  These reasons could be that the particular event being
     measured will only occur during those times or that the
     collection is not allowed while the decennial census is being
     conducted.  Make certain the reasons are justified.  Moving up
     the time schedule for a survey is usually not a sufficient
     justification.

     Training.  It must be determined who will be responsible for
     training interviewers and preparing the necessary materials. 
     In addition, what specific types of training requirements
     exist and will the sponsor participate in the training.,

     Response rate.  The required minimum response rate should be
     specified along with the method which will be used to
     calculate it.  Included with this information should be the
     number of completed questionnaires required and the criteria
     for defining a completed questionnaire.  The procedures for
     field follow-ups should be stated.  These should include any
     time-of-day and day-of-week requirements along with the number
     of follow-ups and a requirement that the contractor complete
     all specified follow-ups, even if the required minimum
     response rate is met.  In addition, there should be a
     provision that the contractor must exceed the minimum number
     of callbacks if they are required to meet the minimum response
     rate.  Of course, any additional field follow-ups needed to
     meet the response rate will have a cost associated with them. 
     This tradeoff should be considered while decisions on the
     survey requirements are being made.  No attempt is made in
     this report to determine what the minimum response rate should
     be for every survey; each survey will have factors that will
     have an effect on this number.  The best advice is to check
     current policy and consult a survey statistician before
     writing this section of the RFP.

     Quality control.  The quality controls that will be required
     in the data collection process and the validation requirements
     must be addressed in the RFP.  Who will be conducting these
     tests must also be specified.  It is usually best to indicate
     that quality control is a necessary part of the survey and let
     the offerors indicate how they propose to control quality at
     each stage.

     Interviewer qualifications.  If the interviewers must possess
     certain qualifications because the survey has special
     circumstances that require them, they must be specified in the
     RFP if they are to be used in the evaluation process.  These
     could include expertise in the specific field of study,
     knowledge of a foreign language, previous data collection
     experience, or knowledge of survey sampling.


     Additional questionnaires.  If there are to be different
     versions of the questionnaire , these requirements must be
     specified along with the reasons for needing them.  They could
     be necessary due to the questionnaire must be tailored to the
     various regions of the country because of content or other
     considerations.  Examples are asking about citrus fruit
     production only in the appropriate areas or having a foreign
     language version of the questionnaire in areas where English
     is not the predominant language.


Tabulations and reports

The final products and quality of the completed work should be
monitored very closely.  To facilitate the monitoring process,
certain deliverables should be

                                27





specifically requested in the RFP.  A schedule should be set up and
rigidly followed.  The following list contains several of the more
important deliverables.

     Progress reports.  Progress reports should be required
     according to an agreed upon plan.  The p!an should specify
     content as well as time of delivery.

     Draft reports for agency review.  The RFP should indicate that
     a draft of each required task report will have to be submitted
     for approval before the final report is written and the task
     is considered complete.

     Tabulations required.  When it is feasible, the format and
     content of all required tables should be included in the
     proposals.  The RFP should also state whether the contractor
     will be expected to perform analyses of the data and in what
     form they must be delivered.

     Data descriptions.  All data supplied should be accompanied by
     sufficient documentation.  For example, the formats of any
     data tapes must be required and the formula used to calculate
     any weights must be supplied with the weights.

     Survey documentation.  The RFP should require a complete
     detailing of the methodology used.  This report should include
     the contractor's impressions of the effectiveness of the
     survey procedures, suggestions for improved methods in future
     surveys, and all appropriate technical materials.  Included
     would be the following: copies of all forms; documentation of
     all (especially sampling and data collection) procedures in
     all phases; all adjustments for nonresponse and missing data;
     survey estimates and estimates of sampling errors; full
     accounting of all data collection results, including response
     rates and interviewer validation results; coding error rates,
     along with all quality control procedures used; methodological
     findings; and a description of the procedures used in handling
     confidential data.

     Analytic reports.  If the agency wants the contractor to
     produce one or more reports analyzing the findings of the
     survey, such a report should be requested in the RFP. 
     Frequently this analysis and the resulting report are done in-
     house.  If this is to be the case, it should be stated in the
     RFP.  Analytic reports should contain a complete analysis of
     the survey data along with any appropriate conclusions and
     recommendations.


ROLE OF THE PROJECT OFFICER

Monitoring functions of the project officer that directly affect
the performance of the contractor should be clearly discussed in
the RFP.  This allows the offerors to know in advance how the
project officer intends to monitor the contract.

Following are several important points which should be defined. 
The RFP should list all areas where reviews and approvals will be
required.  These could include sampling plans, questionnaires,
training materials, and data processing requirements.  If site
visits are planned or meetings are to be scheduled, they should be
mentioned in the RFP.  The RFP should clearly specify the role of
the Project Officer regarding attendance at any training that is
planned.  Detailing the role of the project officer in the RFP
allows the offeror to plan for additional resource requirements and
permits a smoother relationship in the performance of the contract.

                                28





EVALUATION CRITERIA

Every RFP must include the criteria to be used in evaluating the
technical merits of the offeror's proposal.  These will be listed
in descending order of points to be allocated to the item and may
show the actual number of points.

The technical evaluation criteria need to cover the critical
elements to be used in determining the acceptability of the offeror
and the proposal submitted for the contract survey. 
Differentiation between acceptable and unacceptable proposals and
offerors is made on the basis of these criteria.  If a vital
element of approach or capacity is not reflected in the evaluation
criteria, it may not be possible to distinguish among offerors or
their ratings based on the issue.  All too often program staff
members concern themselves with technical aspects of the survey
(that is, sampling, interviewer training, field work, editing and
coding) and ignore the need to translate these specifications into
the criteria that will be used to evaluate the proposals received. 
The criteria will be used in the selection process.  The survey
specifications are the basis on which the offeror provides the
proposal that will be the subject for evaluation.  Proposals can
only be evaluated based on the published criteria.

                                29








              Chapter 4. CONTRACTOR SELECTION PROCESS


INTRODUCTION

The work to be done under the contract will have been described in
the Request for Proposal (RFP) as well as the material and
presentation to be submitted by the offeror as a basis for
evaluating the response.  The criteria and scoring to be used in
this evaluation are also included in the RFP, thus giving each
offeror the same knowledge about evaluation of the technical
proposal.  It is in this sense that contractor selection starts
when the decision is first made to issue a competitive RFP.  By
that time, there should have been consideration of the possibility
of doing the work in-house or through another Federal agency, of
contracting with an 8(a) firm, or of limiting competition to small
business firms under the set aside program.

The decision by the contracting officer to issue an open
competitive RFP instead of taking one of these other options
involves decisions and justifications within the agency as to the
conduct of the work or some basic characteristics that potential
contractors must have.  These may relate to performance or capacity
requirements or experience or reputation requirements of potential
contractors.  These characteristics should be part of the
specifications included in the RFP.  Since the agency has made the
decision to go competitive based on specific requirements of the
proposed contract as these relate to capacities of potential
offerors, it is incumbent upon the writers of the RFP to make this
information available to potential offeror organizations.  It will
be useful to them in making their decisions as to whether to
respond to the RFP.  The mechanism for doing this is the RFP
itself.  An adjunct to this that is sometimes used is a Bidders'
Conference as discussed in Chapter 3.

The thinking that resulted in a competitive RFP should provide the
framework of offeror capacities and expected activities performance
that will be used to evaluate responses.  The remainder of this
chapter is based on the premise that the RFP discussed what was
being looked for in the contractor selection process.


COMPONENTS OF THE SELECTION PROCESS

The response to an RFP is submitted in two separate parts, the
Technical Proposal and the Business and Cost Proposal.  These are
evaluated separately.  Evaluation of the Business and Cost Proposal
is the responsibility of the contracting officer and is done in
terms of the appropriateness and documentation of costs relative to
resources to be devoted to the effort.

The technical proposal is evaluated in terms of the material on
project conduct and management, the staff and other resources to be
devoted to the project, and the qualifications of the staff.  If
the RFP was specific in the description of the work to be done, the
capacity and experience of the organization and staff will probably
be weighted more heavily in the evaluation criteria.  If the RFP
was more general in the description of the work to be done, the
approach to conducting the project proposed by the offeror will
probably be weighted more heavily.  In any case, the RFP should

                                31





have specified what the offeror was to submit as a basis for
technical evaluation and the criteria to be used.

The contracting officer has the responsibility to ensure that the
technical evaluation is conducted in a way that satisfies the
Federal Procurement Regulations and agency requirements.  The
program office bears responsibility for the conduct of this review. 
The guidelines in the regulations governing technical review
require establishing a selection process that will insure fair and
impartial treatment of all offerors, the selection of sources whose
performance is expected to meet the contract requirements at a
reasonable cost or price, and avoidance of arbitrary or capricious
behavior, inequitable treatment, or undue influence.  It is
essential that persons involved in the evaluation process have no
conflict of interest with this function and that no information
concerning the evaluations be released to another individual except
one who is participating in the process.

The project officer for the contract is responsible for proposing
the names of individuals to be on the technical evaluation panel to
his/her superiors within the program office.  The project officer
is expected to be a member of the panel and may serve as its
chairperson.  The panel is usually composed only of government
personnel.  Outside evaluators are used only under unusual
circumstances, such as when required expertise for evaluation is
not available within the government.

The project officer should consult with the contracting officer,
who bears official responsibility for the overall contracting
process, on aspects of the technical review process and current
standards and requirements of the Federal Procurement Regulations
and agency regulations on technical evaluation.  In general,
project officers are given a significant amount of flexibility and
latitude in establishing panels and review plans and in conducting
technical evaluations.  However, a list of persons to serve on the
technical evaluation panel and a plan for the technical evaluation
must be submitted to the contracting officer before the technical
proposals are released to the project officer for panel evaluation. 
To the extent possible, members of the project team should be
included on the panel.  The contracting officer should attend the
initial panel meeting to clarify the basic rules and considerations
under which the technical review will take place.  Beyond that, the
prime requirements are that the process be equitable, that the
basis for the evaluation of each proposal be documented, and that
the evaluation be conducted in terms of the criteria published in
the RFP for contractor selection and the evaluation plan.  The
purpose is to make the contractor selection process as fair and
open to competition as possible.  The process is designed so that
all potential offerors have the same base of information on which
to assess their qualifications for competition for award of the
contract, to know the basis on which a selection will be made and
to have the selection based on preset criteria.  The goal of this
approach is to eliminate the actuality and, to the extent possible,
the appearance of bias and preselection.

In assembling the technical review panel, the project officer
should try to obtain a balance of areas of survey expertise and
subject matter interest as related to the purpose of the contract
and the needs of the agency.  For example, this might include
experts on survey methodology and sample design and experts
representing interrelated content and program areas.  Expertise and
experience available in other agencies should be used as a
resource.  Representation of other agencies is also advantageous
when there is overlapping program interest.  A panel that is biased
in composition toward a given expertise may distort the contractor
selection in that direction independent of the balance built into
the technical evaluation criteria and" plan.  A minimum of three
persons should be on the panel to allow for more effective

                                32





discussion and resolution of disagreements.  The project officer
should arrange for a meeting place where panel members can discuss
the merits of each of the proposals in private.  It is important
that there be continuity over the full process of proposal review
and contractor selection.  Individuals selected for the panel
should be available to review later modifications and additions as
well as the initial responses to the RFP.


TECHNICAL EVALUATION OF CONTENT AND CAPABILITY

The technical evaluation process is designed to carry out the
objective consideration of the technical merits of proposals
submitted in response to an RFP in terms of the published
evaluation criteria contained in the RFP.  By the end of the
technical review process each proposal will be identified as being
either acceptable or unacceptable.

A proposal can be determined at the outset to be technically
unacceptable without detailed review if it fails to meet a specific
expressed requirement of the RFP.  As an example, if the RFP
requires probability sampling for the planned statistical survey
and an offeror proposes only an alternate sampling approach, the
proposal can be considered an inappropriate response to the RFP and
technically unacceptable.

A technical evaluation plan should be prepared by the program
office.  It should include information for the panel members based
on the specific criteria, as included in the RFP, to be used in
rating proposals.  The evaluation plan should provide more specific
guidance for the panel on details to be looked for within an
offeror's proposal in scoring it under various elements of the
evaluation criteria.  The plan will specify the number of points to
be allocated to each part of the evaluation criteria.  In general,
these criteria should deal with the offeror's approach to the
contract tasks and the organizational ability to perform those
tasks.  The evaluation criteria for ability or capacity to perform,
which should have been built into the RFP, should include some
forms of the following considerations: demonstrated organizational
expertise in the subject area, staff availability and their
expertise in the subject area, past experience in surveys under
contract, ability to meet special requirements, and, availability
of needed resources.  The references for past surveys under
contract should be checked.

Before discussing the proposal in a group, each member of the
evaluation panel should individually read each proposal and prepare
a tentative indication of its strengths and weaknesses and a
preliminary score on each evaluation criterion using the rating
sheets in the technical evaluation plan.  Adequate time should be
allowed for this key aspect of the process.  After this has been
done, the panel members should meet to discuss the relative merits
of each proposal and try to arrive at a consensus opinion on the
strengths and weaknesses of each proposal and the possibilities for
correcting the weaknesses.  The evaluators should then individually
rate each proposal.  The individual scores of the panel members
have to be combined into a single ranking of all of the proposals. 
While a numerical average of the panel members ratings for each
proposal is generally used in developing a ranking, this is not a
requirement and depends on the rating plan used.  The panel then
should identify each proposal as acceptable or unacceptable. 
However, this can not be done by the use of predetermined cutoff
scores.

The definition of a proposal as acceptable or unacceptable is based
on technical consideration of the offeror's proposal in terms of
the requirements of the RFP "as assessed, using the evaluation
criteria published in the RFP.  A determination of

                                33





unacceptability must be supported by concrete factual statements
consistent with the evaluation criteria and indicate that the
proposal's deficiencies are so major it could not be brought up to
an acceptable level without the equivalent of a new proposal being
submitted.

The technical evaluation panel members will prepare and sign a
report to the contracting officer showing the ranking of the
proposals and identifying each as acceptable or unacceptable.  The
individual rating sheets should be included.  A narrative
assessment of each proposals strengths and weaknesses must be a
part of the report.  The report should also include any
considerations related to the selection of sources for negotiation
and award and any specific issues and questions for discussions.


DISCUSSIONS WITH OFFERORS IN THE COMPETITIVE RANGE

After the technical evaluation and analysis of the business
proposals, the contracting officer will determine which proposals
are in the competitive range as a basis for conducting written or
oral discussions.  Discussions are communications essential for
determining a proposals acceptability or to provide the offeror
with the opportunity to revise or modify its proposal based on
deficiencies or weaknesses noted in the evaluation process.  More
than one round of discussions may be held by the contracting
officer with offerors depending on the situation.  The
determination for inclusion in the competitive range will be based
on price or cost and technical and other important factors as
stated in the RFP.  The competitive range will include all
proposals that have a reasonable chance of being selected for
award.  In making this determination, the contracting officer may
request the technical evaluation panel to review the cost or price
data, or request comments from the project officer on labor and
other resources needed to conduct the project.  Discussions will be
held with all offerors in the competitive range.

The competitive range may be narrowed after initial discussions by
dropping an offeror if a revised proposal is determined to no
longer have a reasonable chance of being awarded.

All discussions with offerors are under the control of the
contracting officer.  Offerors are given the opportunity to correct
deficiencies, resolve uncertainties and errors, and to submit
revised material based on these discussions.  However, no
information is to be given to offerors in these discussions on
other offerors' proposals or the evaluations, or on competitive or
estimated prices other than to indicate that the Government
considers its price too high.  No specific help should be given an
offeror in rewriting the proposal.

At the end of the discussions with all offerors in the competitive
range, each offeror will be notified that the discussions have
ended and that they have an opportunity to submit a "best and
final" offer by a common cutoff date, including a revision of an
earlier offer or a confirmation that a prior offer is a "best and
final" offer.  These "best and final" technical and business
proposals are subject to evaluation on all significant factors by
the contracting officer and project officer and, if necessary, a"
full technical rescoring and reranking by the technical evaluation
panel.  At the end of this process, the contracting officer will
select for award the proposal that offers the Government the
greatest advantage, price, and other factors considered.

While the final selection decision is the responsibility of the
contracting officer, this' decision is usually made after
consultation with the project officer.  Since it will be

                                34





the project officer's responsibility to monitor technical
performance, he or she should be part of the assessment of the
relative importance of technical quality and price differences. 
The analysis of business management and price factors is the
contracting officer's domain.  Although price or cost is one factor
in the final decision, whether certain items are allowable and the
assessment of the basis for estimating cost is an administrative
issue.  Usually, whether a contract is to be a fixed-price or a
cost-type is specified in the RFP.  However, the final decision to
use a fixed-price or a cost-type contract is a contracting officer
decision and the ability to estimate certain costs is a factor in
this decision.


THE FINAL AWARD PROCESS

.After the successful offeror has been selected, the contracting
officer may hold limited negotiations with this offeror.  However,
these negotiations cannot relate to any factor which could have an
effect on the Selection process.  It is the Contracting Officer's
responsibility to ensure that negotiations do not change the RFP,s
requirements nor make any other changes that would impact on the
selection process.  If there is a material change in requirements
because of negotiations, the competition must be reopened to all
offerors who were in the competitive range.  A significant increase
in the offeror's cost proposal may also require reopening of the
competition.

After any negotiations are held, the final contract document is
prepared.  In preparing the contract document, including all the
agreed upon terms and conditions, the option should be considered
of specifically incorporating the offeror's final proposal to avoid
any question as to what was offered and accepted.  This can be of
particular value when the RFP specified only an area of survey
concern and not the specific approach to be used.

After the contract has been awarded, the unsuccessful offerors will
be notified.  Unsuccessful offerors can, and often do, request
debriefing.  The project officer and possibly other members of the
technical evaluation panel may be. called upon in a requested
debriefing of an unsuccessful offeror.  It is at this point that
the documentation of the basis for technical evaluation can be of
critical importance in explaining why the offeror's proposal was
not selected for award and in establishing that the competitive
process was appropriately conducted.  Unsuccessful offerors may
formally appeal a contract award.  However, a well documented and
unbiased basis for selection will reduce such appeals of the
selection process.

                                35









                 Chapter 5. POST-AWARD ACTIVITIES


INTRODUCTION

Following award of a contract, both the contractor and the
Government have responsibilities that must be fulfilled.  Post-
award activities are all activities that occur or should occur once
the negotiated contract has been signed by both parties. it is the
joint responsibility of the contractor and of the Government to see
that work under the contract is accomplished in a timely and proper
manner.  Government policy and procedures designed to ensure that
the contractor and the Government staff understand their mutual
responsibilities after the contract is awarded are set forth in the
Federal Procurement Regulations.

The major elements of post-award contract administration include
monitoring of performance (including review and approval of
products, services, or reports), contract modifications, approval
of subcontracts, property administration, and contract closeout. 
Aspects of these elements may be performed by various Government
personnel, but the two principal persons are the project officer
and the contracting officer.  It should be made clear at the outset
of the contract exactly what the respective roles of these two
individuals are with respect to contract administration and
monitoring.  Typically, the contracting officer, the person who
actually signed the contract on behalf of the Government, delegates
to the project officer responsibility for monitoring technical
aspects of work under the contract.

In most instances, a contract identifies tasks, subtasks,
schedules, review processes, criteria identifying skills,
techniques and methodologies to be used, and a schedule of
deliverables.  These requirements are usually included in the
Request for Proposal (RFP) and are addressed specifically in the
contractor's technical proposal, which is often incorporated into
the final negotiated contract.  The project officer must monitor
closely all aspects of work to ensure that it is carried out as
specified.

In addition to technical and procedural requirements for conduct of
the work associated with the survey, the contract should also
include performance requirements and a description of measures to
be used in determining if certain requirements, such as the
specified response rate, have been met.  The contract is the basis
for monitoring and assessing contractor performance.  A project
officer should have a complete copy of the final negotiated
contract for a survey, should read it completely to ensure that it
contains all that it should contain, and should reread it
periodically to insure that all of its terms and specifications are
being followed as specified.

Most post-award responsibilities of the project officer relate
directly or indirectly to technical aspects of the work.  The role
of the project officer is discussed in the following material,
which emphasizes activities related to insuring that the product
resulting from the contract is of the highest quality possible.

                                37





GENERAL TECHNIQUES FOR MONITORING CONTRACTOR PERFORMANCE

Meetings with the Contractor

To the extent possible, there should be regular meetings between
the project officer and the contractor.  For a large, complex
contract such meetings may have been included in the statement of
work and travel funds provided for contractor participation.  If
such meetings are not possible or are only possible on an
infrequent basis, conference telephone calls may serve as a
reasonable, though less satisfactory, substitute.  These meetings
provide a forum for structured, though somewhat informal, review of
activities and can offer an early alert to technical problems or to
slippage of the project schedule.


Maintenance of Files

Ideally, the same project officer will serve from the inception of
the project through its completion.  Experience shows, however,
that this is often not the case and that project officers may
change several times during the course of a project.  Thus, it is
essential that the project officer maintain complete, organized
files pertaining to the project.  Included in the files should be
the following: all formal contract documents, including the
negotiated, signed contract and any modifications to it; copies of
all correspondence relating to the project between the contractor
and the project officer; notes or minutes of any meetings or
substantive telephone conversations with key contractor staff;
copies of all progress reports and other materials received from
the contractor; copies of or notes on all communications with the
contracting officer; and documentation of any other significant
materials, including various approvals such as required
departmental clearances and Office of Management and Budget
approval.  Project files are also important as a source of
documentation if there is a disagreement between the contractor and
the project officer about whether information was communicated, and
if so how and when.


ADMINISTRATIVE RESPONSIBILITIES OF THE PROJECT OFFICER

Review of Contractor Progress Reports and Submissions

The project officer should, as soon as possible upon receipt, read
and provide comment, preferably in writing, to the contractor on
all interim progress reports and other documents received.  No
matter how often there are conversations between project officer
and contractor, they do not replace the formal progress reports
that should be required by that contract.  These reports are the
appropriate forum for the contractor to officially inform the
project officer of project accomplishments and problems.  Reports
should contain sufficient information to determine that work is
being accomplished on or ahead of schedule and that it is being
completed in a satisfactory manner.

It will facilitate the project officer's review, and perhaps even
the contractor's preparation, of progress reports if a mutually
agreed upon outline is followed for each periodic progress report. 
Such an outline may even have been specified in the contract.  In
addition to a description and summary of activities since the last
report, topics to be addressed might also include a statement of
any problems encountered and their resolution, personnel changes, a
summary of expenditures to date, and planned activities for the
coming reporting period.

When data collection is involved, the contract should also specify
provision to the project officer, for review and approval, of all
data collection plans; instruments and

                                38





associated materials, including sampling procedures; letters to
respondents; interviewer or respondent instructions for completion
of the questionnaire; training materials for data collection;
coding manuals; and data processing specifications.  If the project
officer does not have the requisite expertise to review and approve
these materials; one or more technical advisors on the project team
should have such expertise and should be asked to conduct the
review, giving advice on appropriate action.

Should the contractor fail to submit required reports or other
deliverables in a timely or technically acceptable manner as
specified in the contract, the project officer should notify the
contracting officer so that determination of appropriate action can
be made.  Unless the delay is minor or affects only a small,
inconsequential portion of the contract, it will probably be
necessary for the contracting Officer to extend officially the
contract schedule or terminate the contract completely.


Review of Financial Reports

The contracting officer will, particularly in the case of cost-
reimbursement contracts, often ask the project officer to review
the periodic vouchers submitted by the contractor for payment of
costs incurred.  For a large or complex project, it may be helpful
to have the information for the financial reports provided in
specified categories.  This will facilitate the assessment of
whether the project is likely to be completed within the budgeted
resources and will be useful if the survey is ever repeated.  In
addition to a report of the current status of expenditures, it may
also be desirable to require projections of expenditures for the
coming project period and perhaps for the remainder of the project.

Review of expenditure information in conjunction with technical
progress reports will enable the project officer to assess
likelihood of cost overruns and to take early steps to avoid or
minimize them.  The project officer must determine that
expenditures are in keeping with technical progress before
recommending to the contracting officer that payment be made.  If
any problems are identified, the project officer should inform the
contracting officer and effort can be initiated to take remedial
action.


Modifications to the Contract

When the contractor or project officer identifies the apparent need
for a change in the contract specifications, they may discuss it
informally between themselves, but no deviations from the
specifications of the written contract may occur until the contract
has been formally modified in writing by the contracting officer. 
If there is a need to change aspects of the work requirements,
deliverables, or schedule for completion of activities, formal,
written modification to the contract will be necessary.


Approval of Subcontracts

It is not unusual for a contractor to engage a subcontractor to
complete one or more tasks under a contract.  It is important to
the Government that the capabilities of the subcontractor be such
that a high quality job will result. if it is known that a
subcontract will be necessary, it and the proposed subcontractor
should be identified before the contract is signed.  However, the
need for a subcontract is sometimes not identified until after the
contract has been awarded.  Therefore, the negotiated contract
should contain a clause requiring approval by the contracting
officer of any subcontract.

                                39





Before entering into a subcontract, the prime contractor should
provide information about the work to be done under the
subcontract, the anticipated costs and dates for completion of
work, along with identification of the proposed subcontractor and
information about why and how the subcontractor was selected.  Upon
receipt of notification of plans to subcontract, the contracting
officer will provide the information to the project officer for
concurrence that the proposed subcontractor has the requisite
capabilities to perform in a technically acceptable manner.  The
project officer should monitor the prime contractor's management of
work under the subcontract.

However, if the work of the subcontractor or the management of the
subcontractor by the prime contractor is not satisfactory, the
project officer should not take direct action.  Instead, the
contracting officer should be notified and asked to take
appropriate action through the prime contractor.  Only under rare
circumstances would the project officer engage directly in contact
with the subcontractor.


Administration of Government Property

The contractor will usually have all equipment and other resources
necessary for conduct of the work under a contract.  However, a
contract may specify that the Government will provide certain
materials necessary to complete the work.  If data collection is
involved, this might be a sampling frame based on administrative or
other records that are in the possession of the sponsoring agency. 
It might be access to Government computer facilities, or it might
be acquisition of specialized equipment specifically for the work
of the contract.  The project officer is responsible for seeing
that the appropriate arrangements are made for contractor access
to, acquisition of, or use of such equipment or materials.  The
contracting officer is also likely to seek the advice of the
project officer on disposition of any equipment purchased under the
contract.


Closing Out the Contract

Upon completion of all work under the contract and provision to the
Government of all required reports, materials and information, and
review and approval of all specified deliverables by the project
officer, the final payment will be made to the contractor. 
Although contract closeout is the responsibility of the contracting
officer, the project officer may be called upon to assist.


QUALITY ASSURANCE RESPONSIBILITIES OF THE PROJECT OFFICER

It is implicit throughout the following discussion that the project
officer will probably not have the full range of technical skills
needed to monitor all aspects of the work.  As appropriate, other
members of the project team should be called upon to participate in
post-award monitoring activities.  These persons should have the
necessary technical knowledge, and familiarity with and
understanding of the objectives and specifications of the contract
to provide advice to the project officer.

Every survey should include a program of built-in quality assurance
activities.  It is the responsibility of the project officer to
ensure that appropriate steps are taken to quantify and document
the quality of survey data.  Provision should have been made in the
contract specifications for inclusion of various quality assurance
components, and the contractor's technical proposal should have
addressed them.

This section of the chapter presents an overview of quality
assurances that should, as appropriate, be built into any survey. 
The discussion is intended to apprise a project

                                40





officer of the variety of ways in which data quality can be
maximized.  Not all of these measures will be appropriate or
necessary for every survey.  In general, large studies require all
or most of these quality assurance procedures; smaller, less
complex studies may require only some of them.

The project officer should be thoroughly familiar with the required
and proposed steps to be taken by the contractor to measure data
quality and achieve a satisfactory level of quality at all steps
from instrument development through data collection, data
reduction, and final data tape preparation.  If no quality
assurance program is specified in the contract the project officer
should discuss plans for one with the contractor's project director
and agreement should be reached and documented on a step-by-step
basis so that there is no room for misunderstanding; a formal
contract modification may be made to make the quality assurance
program a contract requirement.

A key element in assurance of data quality is the active
involvement of the project officer at all steps in the data
collection and data reduction processes.  Active participation by
the project officer and other members of the project team will
often involve costs for travel and other activities.  There should
be provision for these costs in the agency's operating budget over
the period of the project.

The role of the project officer is sometimes specified in the
contract itself; the project officer should make clear to the
contractor's project director as early as possible the intended
nature and extent of this involvement.  Following is an enumeration
and discussion of quality assurance activities in which the project
officer should take an active role.


Sample Selection, Questionnaire Design and Data Collection

Data collection activities conducted under contract usually require
approval by the Office of Management and Budget.  It is the
responsibility of the project officer to learn about these
requirements, determine how they pertain to a particular project,
oversee the development of necessary descriptive materials, and
shepherd the submission through the review and approval process. 
Appendix 11 gives an overview of

these requirements but does not replace early consultation with the
agency reports clearance officer.

In addition to reviewing and approving sample selection
specifications, interviewer and coder manuals, and training
materials, the project officer should attend and, as appropriate,
participate in interviewer and coder training. project officer
involvement in activities relating to sampling, questionnaire
design, interviewer training, and field procedures during the
pretest and pilot surveys is particularly important.  Sample design
and selection are critical survey underpinnings.  Questions for a
survey must be carefully worded to ensure that what is conveyed to
a respondent is what is intended, and responses must be recorded
(either verbatim or in predetermined categories) so that the
planned analysis can be adequately accomplished.  Similarly, the
interviewer training and actual data collection activities
constitute survey activities in which errors or oversights that
occur can negate the success of the entire project.

Sample selection.  The process of sample selection is critical and
should be carefully monitor d by both the project officer and the
contractor.  Depending on the type of survey and whether the
sponsor provides a sampling frame, it may be necessary for the
contractor to obtain, create, or refine the frame from which sample
units for the survey are to be selected.  It is important to
assess, prior to selection of the sample,

                                41





the completeness of coverage of the sampling frame, the currency of
the frame and its contents, and the associated problems arising
from limitations of coverage and currency.  These steps are
essential even if the survey is a census of the universe of
eligible individuals or other units.

Once the frame has been determined to be satisfactory, it is
necessary to ensure that the specified procedures are followed in
selection of the sample.  This should involve a monitoring and
review of the selection that takes place at each stage of sampling. 
If the sample procedures are being carried out incorrectly at one
or more stages of selection, biases in survey estimates will
probably be introduced.  If independent population and
subpopulation counts (or good estimates of them) are available from
a census or large sample, these should be used to help evaluate the
quality of the sample.  For example, suppose that a national
probability sample of persons age 18 years and older. is selected. 
Estimates of population counts by age, race, and sex cross-
classifications from the sample should be compared to corresponding
census counts; any substantial variations could indicate that the
sample is not being selected correctly.

Questionnaire development.  On occasion an agency provides the
contractor with a completely developed questionnaire or set of data
collection instruments.  But even when this occurs, there is
usually the need for a pretest of the questionnaire because seldom
has it been used exactly as intended in the present effort.

More typically, the contractor is responsible for development of
the data collection instruments.  The advice and experience of
subject matter experts are usually the basis for instrument
development, and to the extent possible, previously developed
questions whose efficacy has been demonstrated should be used. 
Questionnaires should receive several levels of pretesting, the
least formal of which is testing on office staff or friends.  This
type of testing offers an opportunity to identify difficulty in
understanding the intent of the question and question flow. 
Subsequently, there should be more formal tests of the
questionnaires.  The project officer should not only review the
questionnaires and any accompanying instructions for their
administration or completion, but should also test the instructions
and questionnaires by administering or completing them.  Only in
this way can first-hand experience be gained with the instruments
and the problems associated with their administration.

Interviewer and coder training.  The project officer is responsible
to monitor training for data collectors in order to be certain that
instruction, given is accurate and thorough.  If possible, the
project officer should attend the training sessions; at a minimum,
the project officer can explain the objectives and importance of
the study to provide project staff with an understanding of why the
data are needed and how they will be used.  This is also an
opportunity for direct acquaintance with the data collectors and
the level of project understanding with which they embark upon
their work.

Interviewer observation.  Throughout the data collection process,
as circumstances permit or dictate, the project officer should
participate in an interviewer field observation program.  Survey
firms usually have a program for observation of each interviewer
early in their field assignment or for monitoring telephone
interviewers throughout the survey.  This observation or monitoring
is typically done by a field supervisor and is intended to
determine how the interviewer presents himself or herself and the
survey to respondents, whether procedures are followed as specified
(including asking the questions exactly as they are worded on the
questionnaire), and what problems are encountered in the field
(including efficient use of time in the field and correct field
editing of completed data collection instruments).

                                42





The project officer should try to observe or monitor several
interviewers, possibly identifying at the training session those to
accompany to the field or otherwise monitor.  Interviewers are
seldom perfect in their work, but by observing several the project
officer will be able to assess strengths and weaknesses in the data
collection process.  Observers, however, must remember that their
role is a passive one and that no active role should be taken
during the interview to either converse with or correct the
interviewer during the interview itself.

Reinterview program.  Many surveys have as a component of their
quality assurance program a reinterview program in which a
subsample of the respondents and nonrespondents are recontacted to
validate selected information reported by the interviewers. 
Reinterview programs are usually developed to check on the work of
individual interviewers in order to identify interviewers who are
reporting imaginary interviews.  For some surveys, reinterviews are
used to try to measure response bias and variation.  Some survey
organizations routinely send respondents a postcard to verify that
the interviewer contacted them and the length of the interview. 
This augments but does not replace a personal recontact with a
sample of the original respondents for each interviewer in which a
brief reinterview is conducted to verify that the original
interview was, in fact, conducted.  Effort is made to conduct the
reinterview with the initial respondent; typically included are a
few basic questions about household or respondent characteristics
needed to verify the completeness and accuracy of information about
their eligibility for the survey.

Response rate.  It is important that the project officer monitor
closely the response rate being achieved by the contractor. 
Several aspects of the response rate should be calculated by the
contractor on an ongoing basis and should be communicated to the
project officer on a regular basis, perhaps weekly.  Overall
response rates, interviewer-specific response rates, and response
rates for key subgroups or characteristics of the study population
should be monitored.  Problems of failure to locate sample units
and to complete interviews or obtain completed questionnaires for
sample units must be identified as quickly as possible in the data
collection effort so that corrective action can be taken.  It is
important to achieve acceptable response rates for specific
subgroups as well as for the entire population.  Poor response
rates can lead to substantial nonresponse biases that cannot be
estimated.  Furthermore, if there are large differences in the
nonresponse rates among population subgroups, there may be
differential nonresponse biases in subgroup estimates. 
Consequently, there may be some misleading subgroup comparisons due
to differential biases.

If there is more than one stage to data collection, such as a
screener questionnaire to identify those eligible for a more
detailed questionnaire, the response rate for the screener is
particularly critical and must be given special attention.  The
nonresponse rate at the screening stage is part of the survey
nonresponse rate.  Furthermore, a high screener nonresponse rate
could create a substantial bias in the estimate of the size of the
target subpopulation.


Data Reduction

Tolerable error levels for coding and data entry must be
established and may have been specified in the contract, but there
must be a monitoring system to assure that they are met.  These
levels are often specified in the contract.

As for interviewers, training is very important for coders.  The
project officer should consider observing at least some of the
coder training to ensure that it is comprehensive and clear.

                                43





Coding.  Questionnaires are usually processed in batches through
the coding and data entry phases of the survey with a single
individual having responsibility for them at each stage.  For
example, early in a project all of a coder's work is usually
checked or verified on a 100-percent basis.  Then, if the person is
operating within the limit of tolerance for error for the project,
this recheck is cut back to a fraction of the completed work. 
Should it occur that the acceptable error level is exceeded for a
batch, the entire batch may then be checked, the coder may be given
special retraining if appropriate, and the coder's work is again
verified on a 100-percent basis until tolerable accuracy is
achieved or the person is removed from the project.  A similar
approach is used for data entry, but data entry is often verified
on a 100-percent basis for the duration of a project.

Surveys often include open-ended items that allow the respondent to
describe or explain something in his or her own words.  Such
information is then coded into predetermined categories by
specially trained persons.  It is common practice to have two
persons code the responses to some or all of the open-ended
questions on a survey.  This is known as independent coding when
the second coder or verifier works without knowing how the first
person coded the information.  When the two coders disagree, there
may be adjudication.  In highly complex coding, there may be three-
way independent verification in which two verifiers independently
code a sample of a production coder's work with a majority rule
applying when two or more coders agree on a code.  Independent
coding may be continued throughout a project.

For large scale surveys it is not unusual for the contractor to
turn over to the sponsoring agency, for totally independent coding,
a small percentage of completed records.  These should be selected
in such a way that they can be considered representative of the
completed work done by the contractor's staff.  Independent coding
is done by agency staff or another contractor using the detailed
coding manual (including documentation on coding decisions made on
difficult and unusual cases).  Any discrepancies are then reviewed
to determine their nature, reasons, and impact on the data base and
analyses.

Data cleaning.  Vigilance to ensure high quality survey results
must Continue through the data processing phase of the project.  To
ensure that computer programs are actually doing what they are
specified to do, it is prudent to put through the system test data
for which the desired values are known through hand or other
independent calculation.  This set of test data should represent
the full range of operations expected of the complete program. 
Then, if the expected results are not obtained, it is clear that
there is at least one problem with the purportedly debugged
program.  Even though the contractor may think the programs are
functioning, it is wise for the project officer to make certain
that it is indeed the case that only valid codes are read and that
data manipulation is occurring properly.

It is important to be certain that only the admissible codes are
contained for each item on a survey record.  However, it is
possible for errors to be present on the final data tape.  In
addition to checking that only reasonable, allowable codes are
present for each item, there are usually between item checks made
to test for consistency within a subject's data record.  For
example, a medical procedure code for hysterectomy can be a valid
code in studies of hospitalizations, but such a code should not be
valid if the person involved is a male.  Such a combination of
codes on a record requires checking back to determine the source
and nature of the error.

                                44





Data Documentation

The project officer should be certain that complete documentation
of all sample selection, data collection, reduction, and processing
activities is created, reviewed, and kept up to date so that it can
be provided with the data tape or tabulations that are received
from the contractor.  This documentation should include information
about sample design specifications, sample frame construction,
sample selection, calculation of respondent weights, field
procedures for initial respondent contact, follow-up of
nonrespondents, and out-of-scope cases.  Any and all decisions
about coding, resolution of apparent inconsistencies in data items
for a given respondent, imputation for missing data, and other
nonresponse adjustments, as well as full documentation of all codes
used in the processing of data and preparation of the final data
tape, must be documented and made available to anyone who will be
involved in analysis, reanalysis or review of data from the survey. 
The data tape received from the contractor should include "flags"
to indicate any imputed values so that analysts will know which
data were actually reported by respondents and which were
statistically derived.  Without this information, along with
documentation of how sampling weights and estimation procedures are
derived and used, the value and utility of the survey data will be
greatly diminished.


Data Analysis

If the contractor is responsible for data analysis, care should be
taken by the project officer that there is a good understanding of
the specific tabulations to be provided by the contractor.  The RFP
for a large, complex survey often includes either actual or
representative table shells for tabulations to be provided by the
contractor.  Manipulation of large and complex data bases is a
difficult and costly task and provision should be made well in
advance for it.

In addition to the tabulations to be produced, attention should be
given to the analytic procedures and tests to be used in data
analysis.  If certain assumptions (for example, a random sample)
must or can be made about the selection of the sample in order to
use existing software packages, it is important to be sure that the
requisite assumptions are reasonably well met by the sample.  It is
recognized, however, that for some purposes a systematic random
sample may be equivalent to a simple random sample.

Appropriate statistical tests to be applied in determining whether
differences between various estimates are significant should be
agreed upon in advance if the contractor's staff is to perform
analysis of the data.  If the contractor performs data analysis,
information on testing procedures used should be included in the
final documentation for the project provided by the contractor.

                                45








Appendix 1.    INFORMAL SURVEY OF AGENCIES AND CONTRACTORS
               CONCERNING CONTRACTING FOR SURVEYS


INTRODUCTION

When the Subcommittee on Contracting for Statistical Surveys set
out to do its work, the members believed it would be important to
solicit facts and judgments from persons who were directly involved
both from the point of view of Federal agencies and from the point
of view of contracting firms.  The Subcommittee realized that,
within its frame of reference, it could not field a formal
statistical survey of Federal agencies and contractors concerning
the broad spectrum of problems encountered in contracting for
statistical surveys.  On the other hand, the Subcommittee could and
did adopt some procedures which would assure then that its report
reflected more than just the views of its members and their
immediate contacts.  It should be emphasized that the results are
not the careful findings one would encounter in a scientific sample
survey.  They are the considered judgments of a selection of
individuals who are personally involved in the contracting process
both from the Federal agency and private contractor point of view.

The Subcommittee devised two interview schedules - one for
representatives of contracting firms and one for representatives of
Federal agencies which use contracts for statistical surveys. 
Respondents were usually groups of individuals who were interviewed
simultaneously; in a typical agency interview, for example, a
subject matter specialist, contracts expert and administrative
official would be present.

The agencies were selected first and then a list of their
contractors was obtained.  The sample of contractors was selected
from that list.  This procedure offered some assurance that the two
groups of interviewees were both concerned with similar Federal
contracts, although from different points of view.

All the contractors were engaged in the competitive procurement
process with the exception of one contractor selected from those
largely involved in noncompetitive procurements.  For the
competitive contractors, the first step in the sample selection
process was to review the contract awards of the selected agencies
in the statistical contracting area.  All contracts that seemed to
be within the area of review were identified.  Next, the sample was
drawn and the contracts were reviewed more carefully to ensure that
they were consistent with Subcommittee judgments regarding
statistical contracts.

For the non-competitive contractor group, a list of such
contractors was obtained.  The list was reviewed by the
Subcommittee.  Most were determined not to engage in statistical
contracting.  One contractor was selected from the remaining few
based on Subcommittee members' knowledge of the contractors and the
type of work they typically performed.

Interviews were conducted by Subcommittee members. The interview
schedule was comprised mostly of open-ended questions which would
elicit principally qualitative information.  Results were compiled
at the Office of Management and Budget (OMB),

                                47





and Subcommittee members used the results as background information
in preparing this report.


FEDERAL AGENCY RESPONSES

Eleven interviews were ultimately conducted with Federal agencies,
two in the nature of pretests.  The results described below do not
always pertain to the same number of agencies and contractors
because the pretest schedule differed in some respects from the
schedule finally adopted and because the interviews were
unstructured.  Seven of these agencies permit contractors to use
subcontractors, although some restrictions may be placed on the
extent of subcontractor involvement.  The number of statistical
survey contracts awarded by these agencies during fiscal year 1977
and fiscal year 1978 ranged from a low of 2 in one agency to a high
of "25 or more in another."  Four agencies used cost-type
contracts; one used fixed-price contracts; and three used both. 
Contract amounts varied between $40,000 for 1 year to $20 million
over a 3-year period.


Developing RFPs and Soliciting Bids

Typically, the first step for agencies is to develop a Request for
Proposal (RFP) which provides prospective contractors with
information concerning the specifications of the proposed survey. 
Usually, no formal guidelines or checklists are used in preparing
the RFP.  On the other hand, the respondents in the agencies agreed
as to features which should appear in all RFPS.  More than half of
the interviewees stated that the following items should appear in
RFPS: proposal evaluation weights, population to be studied, level
of effort, key content variables, data collection method, sampling
frame, required response rate, analysis of data, and sampling
design.  Items such as data processing procedures, sampling
reliability, level of response, questionnaire development, trade-
offs between cost and quality, and quality control procedures were
believed to be less important.  Sometimes agencies consult OMB
concerning the contents of a survey RFP before issuance; sometimes
they ask other agencies for review and comment.

The availability of the RFP is usually announced in the Commerce
Business Daily as a formal notification to potential contractors. 
Advertisements may also be put in trade papers or announcements
placed in the Federal Register.  Mailings are also made to lists
derived from past agency experience or from the expertise of
persons in the industry.  A few agencies utilize sources-sought
announcements as a way to stimulate offeror interest.

Of ten agencies make use of Bidders' Conferences to explain the RFP
and answer technical questions.  Sometimes these conferences result
in changes, or clarifications in the RFP.

The number of proposals received after solicitation of bids varies
widely among the agencies and by the specific subjects of the RFPS. 
Some receive only three or four proposals for each RFP, others may
receive as many as 30 proposals.  The size and cost of the
contemplated survey are often major factors in determining the
number of proposals offered: generally, the bigger the size and
cost, the greater the number of proposals.

                                48





Evaluating Proposals

Agencies are governed by Federal procurement laws and regulations
in evaluating proposals.  The following results are not a
description of the legal procedures but rather an encapsulation of
the processes agencies generally go through.

Agencies evaluate first the technical aspects of proposals and then
the business aspects.  Selection committees are formed to evaluate
and choose the best proposal.  The technical review committee
usually consists of three to eight people and includes persons with
backgrounds in statistical surveys and contracting.  Actual costs
are usually not known by the committee.  Some agencies allow
personnel from outside the agency to participate in these
committees, particularly when special expertise is needed.  The
business evaluation is usually conducted by contract and
procurement personnel.  This process consists of a review for
appropriateness and consistency of financial resources and project
organization.

Following the initial evaluation of proposals, offerors may be
asked to revise their proposals based on results of the initial
evaluation.  A final evaluation is then performed.  In the final
evaluation, rating sheets are completed for each technically
acceptable proposal.  The contracting officer determines the
official competitive range.  The finally selected proposal must
always be acceptable on technical criteria but need not be the
least expensive.

From issuance of the RFP to awarding of the contract requires at
least 60 to 90 days, often longer.  The final selection process can
take at least 30 days.  All agencies suggested that the length of
time for the entire process is too long.  They suggested
eliminating certain procedures to expedite selection of the
contractor.

Offerors not selected may ask for a debriefing, at which time
unsuccessful offerors can learn about the deficiencies in their
proposals.  Few of the agency respondents had experienced any
protests.


Monitoring and Other Considerations

A project officer, who is usually a person with academic training
and job experience in surveys, manages the survey for the life of
the contract.  Some agencies report problems with turnover in
project officers.

Agencies generally cited some problems in the execution of survey
contracts.  Some contractors have not performed well; others have
needed more time to complete the job; some difficulties have been
encountered with response levels and deciding when to stop seeking
survey responses.  A few agencies require progress reports and some
send staff for site visits.  For the most part, however, feedback
concerning contractor performance is obtained informally.

Most of the agencies stated that they perceived no ethical problems
in contracting.  Of those reporting ethical problems, one had a
staff member who had previously worked for a company specializing
in the survey subject.  Another expressed the fear that technical
personnel may sometimes tend to reveal too much information arising
from the confidential evaluation process.

                                49





CONTRACTOR RESPONSES

Eleven interviews were conducted with contractors, two of which
were pretests.  The interviewees were eight commercial statistical
firms and three nonprofit institutions.  Most were engaged in a
broad range of statistical work.

Of the eleven contractors that were interviewed, five specified the
professional staff available at their organizations.  Most of their
staffs included statisticians, computer systems analysts,
programmers, sociologists, and support staff.  Psychologists,
economists, and market researchers were also available full time on
some staffs.  One of the contractors stated that key personnel are
maintained in-house but extra staff are hired for contracts.


Sources of Information on RFPs

Almost all contractors consulted the Commerce Business for RFPS but
all have other sources of information as well such as personal and
professional contacts.  Most considered sources other than CBD to
be their most productive, particularly the personal and
professional ones.  Several contractors expressed reservations
about relying solely on announcements in CBD, saying that they
rarely win contracts they first heard about in CBD.


Level of Specificity in RFPs

Contractors had much to say about the level of specificity in RFPS. 
They noted that "the level varies widely among agencies.  Some
believed that contracting agencies make unrealistic specifications;
others said ambiguity or overspecification in RFPs reflect a lack
of sophistication on the part of the RFP writer.  A few stated they
were very comfortable with either very general or very specific
RFPS but the ones in between proved most troublesome.

Different levels of specificity were believed appropriate for
different agencies.  Agencies with a good technical staff could
write a detailed RFP (for example, including specifications of
sampling reliability, specific questions to be asked, and required
response rate); those without such staff could write only very
general RFPS.  Levels of specificity were also related to the kinds
of studies.  Some thought evaluation studies needed very general
requirements; studies where a specific set of data is sought should
have stricter requirements.

A frequent complaint was that the RFPs did not specify the level of
effort desired by the agency.  As to specific items to be included,
there was a general desire to have the RFP specify the population,
the key variables, and to a lesser extent the response rate.  Other
factors should be left to the offeror, particularly sampling
design, data collection method, data processing procedures, quality
control, and analysis of data.


Improvements to RFPs

Many contractors suggested that RFPs should be written more
clearly.  Several wanted a clear statement of goals and the amount
of money available.  Others suggested that the level of effort
desired should be specified.  A few suggested that outside experts
be hired to write RFPS.  A few mentioned that time constraints were
a problem; the time limitation on preparing proposals often hurts
small contractors especially.

Other suggestions made for improving RFPs included making RFPs more
consistent among agencies and giving the agency and the contractor
more flexibility to make

                                50





changes during the course of the contract.  A few contractors
stated they were reluctant to bid on fixed-price contracts.

A number of suggestions and comments were made regarding whether
the sponsoring agency or the offerors should provide various
technical aspects of a survey.  The majority of the contractors
preferred that the RFP state the target population and the key
variables to be studied.  Several contractors also suggested that
the level of effort (anticipated budget) be included in the RFP. 
Most contractors preferred that the offerors specify the particular
questions to be asked, sampling frame, sampling reliability, data
collection method, data processing and quality control procedures,
and analysis of the data.


Problems with Inadequate RFPs

Most contractors stated they have had problems with inadequate
RFPS.  One major problem was unrealistic expectations, learning
that the agency actually wants much more than was implied in the
RFP.  Problems were thought to arise from hasty writing or
authorship of the RFP by someone who did not understand the
technical issues.  Instances were pointed out where precision
requirements were incorrectly specified.  This can lead to
substantial problems in comparing proposals.  Other problems have
been lack of clarity and lack of understanding of confidentiality
issues.  In one instance, a contractor was forced to hire a lawyer
to educate agencies on issues of confidentiality.


Bases for Decisions to Submit Proposals

The two most important criteria determining whether contractors
submitted proposals were expertise in the subject field and
perceived chance of winning the contract.  Other factors were
expected profitability, size of contract, availability of staff,
and the amount of competition, including whether the firm had a
head start or inside track.  The nonprofit institutions gave
"interest in the topic" as a major reason to bid on a contract. 
More than half of the contractors did not respond to a question
concerning the types of professional staff they had available;
others who did respond said they had a full staff or could easily
get the services of any staff they needed.


Reactions to Bidders, Conferences

Almost all of the contractors stated that they always attended
Bidders' Conferences.  They did so mainly to assess the competition
and to make sure the competition did not get any advantages over
them.  A few felt the conferences were occasionally useful in
clearing up ambiguities, but one noted that offerors are often
reluctant to ask the real questions lest they help competitors. 
Others stated that Bidders' Conferences were most useful for large
technical studies.


Preparing Proposals

No strategy was consistently applied by contractors in preparing
proposals.  Replies on this topic tended to be vague, emphasizing
case-by-case basis or outdoing the competition.  Almost half said
they tried to submit a technically superior proposal.

Cost estimates for preparing proposals ranged from a low of $1,000
to high estimates of $10,000 and even one of $30,000.  Several
mentioned that their dollar figures did not take into account time
spent by staff.  A few gave costs in terms of percentage of the
final contract, such as 5 percent of expected dollar amount.

Costs of preparing proposals could be reduced by improving RFPs by
making them more explicit about capabilities and qualifications. 
Some contractors suggested

                                51





limiting the length of proposals or eliminating unnecessary and
redundant demands from RFPS.  One suggestion was for a two-stage
bidding process.  The first stage would be open to all offers, who
could submit a general description of their proposal and a list of
qualifications.  The agency would then select the most promising of
these for a second stage and invite detailed proposals.  This would
save money and reduce the number of detailed proposals to be
reviewed.  Other suggestions were for more qualified project
officers and for greater use of sole-source contracts when there is
clearly a front running firm.


Agency Selection Procedures

Most contractors believed that agency selection procedures did not
generally result in selection of the best proposal.  The primary
reason given was that, the winning proposal is not selected
primarily on technical grounds; too much emphasis is put on cost. 
Contractors attributed this to inexperienced or incompetent agency
personnel.  Contractors felt that lowest offerors were often not
well qualified but that taking the lowest offeror was a safe course
for agencies to follow.

The selection of the winning proposal was not believed by the
contractors to revolve around technical issues because agencies
judge proposals on the reputation of the firm and its staff.  They
felt insufficient time is allowed for adequate technical judgment
and that agency staff have poor memories, repeatedly selecting
contractors who have done poorly in the past.

There was some difference of opinion among the contractors between
those who believe a proposal should be judged on technical grounds
solely and those who believe more emphasis should be placed on the
firm's ability to perform.  A few of the contractors felt the best
proposal was not generally selected because of the alleged practice
of "wiring" contracts, where the agency knows beforehand which firm
will get the contract.


Improvement in Selection Procedures

Many suggestions focused on getting more technically qualified
people to evaluate the proposals within the agencies.  Some stated
that the selection procedures will be improved when the agencies
have competent people to write and evaluate the proposals. 
Suggestions were made for more peer review or for specification of
the methodology of selection.


Debriefings

Most contractors indicated that they had asked for a debriefing on
a contract not awarded to them.  Most said the debriefing was
useful, especially if they could compare their proposal to the
winning bid.  One contractor said the debriefings were good for
indicating what areas of the proposal were weak, what the agency
was really looking for and how the procurement process works. 
Another contractor noted that agencies are occasionally defensive
at debriefings, especially when minority firms are involved.


Ethical Considerations

Several contractors stated that, while the selection system is
generally clean and fair, some contract rigging or "wiring" exists. 
A number mentioned problems of confidentiality.  A few were
concerned with whether their proposals were kept confidential
during the evaluation process; they believed there were occasions
when they had not been awarded the contract, but that their ideas
had found their way into the winning proposal.

                                52





Another confidentiality problem concerns survey respondents' rights
to privacy.  Contractors expressed concern about being able to
maintain confidentiality of data collected, some contractors noted
that agencies lack experience in knowing how to use such data and
require education by contractors.

Another issue was sole-source contracts.  Some contractors believed
government regulations should be changed so that an agency can
award follow-on sole-source contracts to firms with which it is
well satisfied.

Other ethical issues raised were the assumption that small firms
cannot finance a large job and a bias against g(a) firms.  An
unethical practice on the part of the agencies was asking the
offeror to propose alternatives to the RFP specifications then
rewriting the RFP to use these alternatives.


Other Considerations

Among the other comments made by contractors were complaints
concerning the lengthy OMB clearance process as well as other long
delays.  Some of them felt the length of time it takes to get
questionnaires cleared was an ethical problem.  Others complained
of too much irrelevant information being required on proposals.
some contractors felt the government should consolidate the
procurement regulations of the various agencies.  Others complained
that year-end contracts were not really competitive because time
constraints precluded careful evaluation, that references on
proposals were not checked, and that many contractors defined
person-years differently which led to problems in comparing costs.

                                53








     Appendix II.   OFFICE OF MANAGEMENT AND BUDGET REQUIREMENTs:
                    Information Collection Budget and Clearance

INTRODUCTION

Almost all Federal agencies are required under the Paperwork
Reduction Act of 1980 (P.L. 96-511)1 to obtain approval from the
Office of Management and Budget (OMB) each time they propose to
collect or sponsor, under a contract or other agreement, the
collection of identical information from more than nine
respondents.  In addition to main data collection, activities
related to development or testing of data collection plans are also
subject to OMB review and approval if identical information is
sought from more than nine respondents in either a formal or
informal manner.  The document to be submitted foe OMB approval and
the time of submission will depend in part upon the agency
practice.  The development and shepherding through the applicable
channels of the request for approval is one of the most important
and demanding responsibilities of the project officer on a
contract.  Due to variations in agency requirements and policies
pertaining to OMB clearance, it is especially important to
determine them at the earliest possible stage in development of a
project.

Documents that require approval under the Paperwork Reduction Act
include the following:

-    report forms, application forms, questionnaires, or interview
     guides for in person or telephone surveys.

-    orders, regulations or other directives that include
     requirements for respondents to provide information or
     maintain records to be used or made available for use in the
     collection of information.

-    requests for answers to identical questions which are
     addressed to respondents by telephone, telegraph, form
     letters, information circulars and other devices.

-    any supplementary documents involved in these reporting or
     recordkeeping requirements, such as instructions or covering
     letters or introductory statements.

A number of standards have been developed and promulgated relating
to surveys, including standards for statistical surveys,
publication of statistics, and definitions for certain items; these
must be adhered to as they apply to a particular survey.  Early in
the development of a survey, the project officer should discuss the
nature of and plans for the survey with the agency reports
clearance officer.  This will provide an opportunity to become
familiar with the most recent directives pertaining to OMB
clearance and to acquire an understanding of the process as it will
apply to the particular survey.  A list of agency clearance
officers is available from the Office of
___________________________

1This legislation supersedes and extends the Federal Reports Act
of 1942.

                                55





Information and Regulatory Affairs at OMB if there is any
difficulty in identifying an appropriate agency official with whom
to consult.


INFORMATION COLLECTION BUDGET

A project officer should also be certain that provision for a
particular project is made in the agency's Information Collection
Budget (ICB).  The ICB is an annual, comprehensive submission from
each agency to the Director of the Office of Management and Budget
describing each existing and new information collection project
proposed to be implemented or continued in the succeeding fiscal
year.2 In the case of a project that has not been included in the
agency ICB, it will be necessary to determine the implications for
project timing and scope.


CLEARANCE

Types of Clearance

When the Request for Proposal (RFP) prescribes the data collection
plan (that is, the information to be collected and the method of
collection), OMB permits submission of the plan for OMB approval
prior to issuance of the RFP or signing the contract.  Agency
contracting policies vary with respect to insistence on receipt of
OMB clearance prior to actual signing of a contract.  But for large
data collection projects, it is wise to obtain OMB approval before
the contract is signed.  Contracts that include data collection
activities should contain a special provision in the RFP and in the
final negotiated contract that no data may be collected without
receipt of OMB approval.

If data collection is required by or implicit in the statement of
work for the contract, the project might be submitted for
preliminary (or concept) approval from OMB prior to signing of the
contract.  This can obviate some problems later, after work on the
project has begun, and it should facilitate approval of subsequent
OMB clearances for the Project.

A survey that has a pretest designed to help make decisions about
final questionnaire content or survey methodology will usually
require separate approvals for the pretest and the final data
collection plan.  Thus, up to three separate clearances may be
required for a survey and it is not unusual for a survey to require
two separate clearances.


Timetable for Clearance

Clearance is a lengthy process and ample time should be allowed for
it.  Several agency and contractor representatives interviewed by
the Subcommittee expressed
___________________________

2Upon completion of its review of each department's ICB request,
OMB transmits a "Passback" which gives a total information
collection allowance of respondent burden-hours for the department
for the fiscal year.  This passback may disallow specific projects. 
It may also contain specific suggestions for elimination or
reduction of respondent burden.  In providing this passback detail,
OMB is not committing itself to providing Paperwork Reduction Act
approval for any of the items contained in the ICB request;
Paperwork Reduction Act decisions will be based upon the merits of
each individual request for approval.

                                56





their frustrations about the time required by the OMB approval
process and mentioned additional expenses resulting from delays in
the OMB clearance process.  The agency clearance officer can
provide information about how long it can be expected to take for a
particular survey.

In general, at least 2 weeks should be allowed for each office that
must handle and review the clearance request within the agency. 
Under the Paperwork Reduction Act, the Director of OMB is given 60
days after receipt of a proposed information request to notify the
agency involved of the decision to approve or disapprove the
request and is to make such decisions publicly available.  However,
the Director may determine that a request submitted for review
cannot be reviewed within 60 days, in which case after notice to
the agency involved the Director may extend the review period for
an additional 30 days.  If the Director does not notify the agency
of an extension, denial or approval within 60 days (or if the
review period has been extended for an additional 30 days and
notice of action is not given within the time of the extension) a
control number is to be assigned without further delay, approval
may be inferred, and the agency may collect the information for a
period of not more than 1 year.

A prudent project officer will allow for the possibility of lengthy
delays in obtaining OMB approval for a project.  Delays do occur
from time to time and when a contractor is involved there can be
significant problems that arise at such a time if the project
officer has not had a contingency plan.  Such a contingency plan
should be built into the contract.  One way to handle this problem
is to write a phased contract, in which certain activities are
clearly identified to be done prior to receipt of OMB clearance and
others are to be done only after receipt of OMB clearance.  Another
method, which can be used with fixed-price contracts, is to include
provisions that: (1) no additional sums are authorized because of
delays in clearance, and (2) the duration of the contract is day-
for-day extended for delays encountered in clearance.  The program
contracts office can provide information on how each agency handles
this type of provision.


Clearance Submissions

There are formal specifications for the information to be provided
in support of a request for approval of data collection plans.  The
format for the supporting statement, as specified by OMB, must be
followed in writing a clearance request.  Clarity and brevity
should be relied upon in preparation of the clearance supporting
statement, with lengthy detailed description and discussion
reserved for inclusion in attachments.

It is the responsibility of the project officer to prepare the OMB
clearance request, submit it for approval, and answer any questions
that are raised by reviewers.  Once a contract has been let, the
project officer may wish to have the contractor assist in
preparation of the description of survey procedures to be used; in
any case, the project officer bears full responsibility for
reviewing any material prepared by the contractor for completeness,
correctness, and adequacy.  Often it is helpful to have the reports
clearance officer review the clearance package in draft prior to
formal submission.

At an early-stage in the project, there should be contact with
relevant statistical or other programmatic offices with subject
matter related programs and experience.  This,will usually ensure
proper coordination with other projects in the subject area and
will usually facilitate clearance.  Such contacts. should be
documented in the clearance submission.

                                57





In developing a timetable for the project, a realistic period
should be allowed for each stage in preparation for and conduct of
data collection activities (including follow-up of nonrespondents
and recontact of respondents for any reason).  Some slippage of
schedules inevitably occurs and should be allowed for in
determination of a requested expiration date.  If it becomes clear
that data collection cannot be completed by the expiration date for
OMB approval, a request for an extension should be submitted to OMB
at least 60 days before the existing approval is due to expire.

Once the project officer has submitted the clearance request
through channels to the appropriate clearance officer, he or she
should make every effort to remain informed about its progress
through the review process.  This will enable the project officer
to give the contractor current and correct information about the
status of the review and will hopefully keep the contractor from
trying independently to make contacts in the clearance chain -
something the contractor should never do.  Project officers, too,
should make all inquiries about projects in the review process
through proper channels.


AFTER RECEIPT OF APPROVAL

Once a survey has received OMB approval, the approval number and
the expiration date for the approval must appear in specified
locations on the approved documents.  Finally, a number of copies
of the final printed form(s), transmittal letter, instructions, and
any other document sent to each respondent must be provided for the
official clearance files.

Before a material revision or change is made in an approved
information collection, a formal request for revision must be
submitted to OMB for approval Changes in an information collection
requiring approval include the following: any increase in the kind
or amount of information sought; any increase in coverage; any
increase in the timing or frequency of reporting; any change in the
sample design or collection method; or a change in the purpose for
which the information is collected or required to be maintained.

                                58





                     Reports Available in the
              Statistical Policy Working Paper Series

1.   Report on Statistics for Allocation of Funds GPO Stock Number
     003-005-00178-6, price $2.40

2.   Report on Statistical Disclosure and Disclosure-Avoidance
     Techniques GPO Stock Number 003-005-00177-8, price $2.50

3.   An Error Profile: Employment as Measured by the Current
     Population Survey GPO Stock Number 003-005-00182-4, price
     $2.75

4.   Glossary of Nonsampling Error Terms: An Illustration of a
     Semantic Problem in Statistics (A limited number of copies are
     available from OMB.)

5.   Report on Exact and Statistical Matching Techniques GPO Stock
     Number 003-005-00186-7, price $3.50

6.   Report on Statistical Uses of Administrative Records GPO Stock
     Number 003-005-00185-9, price $5.00

7.   An Interagency Review of Time-Series Revision Policies (A
     limited number of copies are available from OMB.)

8.   Statistical Interagency Agreements (A limited number of copies
     are available from OMB.)

9.   Contracting for Surveys (A limited number of copies are
     available from OMB.)


Copies of, these working papers, as indicated, may be ordered from
the Superintendent of Documents, U.S. Government Printing Office,
Washington, D.C. 20402