Technology Division

December 1992 Information

Technology: An Aud

Guide for Assessing

Acquisition Risks

GAO/IMTEC-8.1.4

The federal government depends heavily o

of information technology products and se

serve the public. Each year the governmen

billions of dollars on computer equipment

software, and telecommunications. The su

failure of information system acquisitions 

executive agencies’ credibility with the Co

the public as well as their abilities to carry 

missions effectively and efficiently.

The General Accounting Office (GAO) and 

inspectors general have consistently ident

problems with information technology acq

Problems identified in numerous evaluatio

information systems that do not meet user

exceed cost estimates, or take significantly 

than expected to complete.

This guide provides a logical framework fo

evaluating information technology acquisi

incorporates a risk assessment methodolo

to reduce audit planning time and ensure t

significant issues are included. It is based 

of the acquisition process developed by GA

cooperation with a wide range of federal a

sector officials. 1

 The model outlines the p

to acquire information technologies and id

elements of the process that are essential 

and carrying out acquisitions.

This guide is intended for use in planning a

conducting risk assessments of computer 

and software, telecommunications, and sy

development acquisitions. A risk assessme

process of identifying potential risks in a s

under development and then determining 

significance of each risk in terms of its like

impact on the acquisition’s cost, schedule, 

1

Information Technology: A Model to Help Managers 

Acquisition Risks (GAO/IMTEC-8.1.6, August 1990).

Page 1 GAO/IMTEC-8.1.4 Assessing Acq
to meet the agency’s needs. Such assessme

have their greatest impact if carried out ea

an agency can more easily alter its acquisi

and strategy to manage and control the ide

risks.

The audit guide consists of 10 chapters, wi

appendixes. Chapter 1 introduces the purp

guide, explains its essential concepts and t

and provides direction in tailoring the guid

on specific assignments. Chapters 2 throug

address specific activities in the acquisitio

The nine chapters present audit guidance 

• management and user support,

• project staffing,

• needs/requirements/ specifications,

• alternatives,

• acquisition planning,

• solicitation document,

• source selection,

• contract management, and

• test and acceptance.

Each chapter lists audit objectives, commo

expected documentation, detailed audit qu

and references to federal regulations and g

The chapters are intended to assist in the

identification of specific risk areas and to 

to an overall assessment of how well an ag

meeting its acquisition objectives.

This audit guide is also available in a softw

accompanied by reference materials such 

model of the acquisition process, relevant 

acquisition regulations, General Services

Administration (GSA) guidance, and Office 

Management and Budget (OMB) circulars. 

software version utilizes a hypertext softw

package to help auditors quickly and flexib

Page 2 GAO/IMTEC-8.1.4 Assessing Acq
all the included documents. This software 

requested from GAO by returning the card 

with this guide.

This guide supplements and does not repla

GAO policies or procedures. It was prepar

the direction of Jack L. Brock, Jr., Directo

Government Information and Financial Ma

who can be reached at (202) 512-6406. Oth

contributors to the guide are listed in appe

Ralph V. Carlone

Assistant Comptroller General

Information Management and

 Technology Division

Werner Grosshans

Assistant Comptroller General

 for Policy

Page 3 GAO/IMTEC-8.1.4 Assessing Acq
Preface 

Chapter 1 

Introduction

Objectives 

Approach 

Assignment Planning 

Organization and Use of the Audit Guide 

Application of Audit Guide to Prototyping

Methodology

Presenting Conclusions and

Recommendations

Chapter 2 

Management and

User Support

Audit Objectives 

Documentation Required 

Audit Steps: Top Management Support 

Audit Steps: User Involvement 

References 

Chapter 3 

Project Staffing

Audit Objective 

Documentation Required 

Audit Steps: Project Management 

Audit Steps: Project Staff 

References 

Chapter 4 

Needs / 

Requirements / 

Specifications

Audit Objectives 

Documentation Required 

Audit Steps: Needs Determination 

Audit Steps: Requirements Analysis 

Audit Steps: Specifications 

Audit Steps: Test Plans 

References 

Page 4 GAO/IMTEC-8.1.4 Assessing Acq
Chapter 5 

Alternatives

Audit Objectives 

Documentation Required 

Audit Steps 

References 

Chapter 6 

Acquisition

Planning

Audit Objective 

Documentation Required 

Audit Steps 

References 

Chapter 7 

Solicitation

Document

Audit Objectives 

Documentation Required 

Audit Steps 

References 

Chapter 8 

Source Selection

Audit Objective 

Documentation Required 

Audit Steps 

References 

Chapter 9 

Contract

Management

Audit Objectives 

Documentation Required 

Audit Steps 

References 

Chapter 10 

Test and

Acceptance

Audit Objectives 

Documentation Required 

Audit Steps 

References 

Appendixes Appendix I: Acquisition Profile 

Page 5 GAO/IMTEC-8.1.4 Assessing Acq
Appendix II: Management Metrics 

Appendix III: Reference Materials 

Appendix IV: Major Contributors to This

Audit Guide

Glossary 

Table Table I.1: Map to the Audit Guide 

Figures Figure II.1: Problem Reports in Software

Project

Figure II.2: Problems By Priority Levels an

Number of Months Unresolved

Figure II.3: Software Requirements Chang

Figure II.4: Development Progress 

Figure II.5: Software Size Estimates 

Page 6 GAO/IMTEC-8.1.4 Assessing Acq
Abbreviations

ANSI American National Standards 

APR agency procurement request

CO Contracting Officer

COCOMO Constructive Cost Model

COTR Contracting Officer Technical

Representative

DPA delegation of procurement au

FAR Federal Acquisition Regulatio

FIPS Federal Information Processi

Standards

FIRMR Federal Information Resource

Management Regulation

GAO General Accounting Office

GSA General Services Administrat

IEEE Institute for Electrical and Ele

Engineers

IFB invitation for bid

IMTEC Information Management and

Technology Division

NIST National Institute of Standard

Technology

OMB Office of Management and Bu

PM program manager

RFP request for proposals

SSA source selection authority

SSAC Source Selection Advisory Co

SSEB Source Selection Evaluation B

Page 7 GAO/IMTEC-8.1.4 Assessing Acq
This audit guide is based on and incorpora

information technology acquisition model

describes three phases in the acquisition p

presolicitation, solicitation and award, and

postaward. It sets out essential activities in 

phase along with critical factors related to 

activities. The model is intended to give m

overview of the acquisition process and to 

decrease acquisition risks.

The model’s critical factors were drawn fr

judgment and expertise of a wide range of

knowledgeable individuals from governme

private industry. Compliance with these cr

factors can increase the likelihood that an 

will meet an agency’s needs at a reasonabl

in a timely manner.

Objectives This guide is intended to help auditors con

focused reviews of information technology

acquisitions by enabling them to quickly id

significant areas of risk. Using this guide w

auditors identify critical factors not addres

management, make a general assessment o

procurement risks, and provide rapid feed

agency officials so they can take correctiv

timely and efficient manner. Use of the gui

be selectively tailored to the requirements 

particular reviews and adapted to the statu

acquisition.

Auditors will need to exercise professiona

in assessing the significance of audit resul

findings. For example, the guide assists au

determining how an agency identified and 

requirements. Professional judgment is ne

evaluate this information and determine if 

conducted an adequate requirements analy

Page 8 GAO/IMTEC-8.1.4 Assessing Acq
Some areas of assessment will require the 

of auditors with specific technical skills an

experience. These specific areas include k

of solicitation procedures, benchmarking a

performance or capability validation techn

knowledge of technical areas such as data

management and telecommunications netw

These areas are identified within the guide

team should include experienced member

enough knowledge of information technol

satisfy the Government Auditing Standard

requirement that auditors have appropriat

knowledge.

Approach The audit approach described in this guide 

to result in a risk assessment of an acquisi

at any point in its development. The audito

trying to determine whether a project will 

specified product or level of performance 

delivered at a specified time and cost. An a

should use this guide to identify areas that 

likely to result in technical failures, unmet 

cost overruns, or schedule delays. Those r

then be brought to the attention of approp

agency officials. The audit steps and quest

provided are directed toward assessing wh

agency has sufficiently addressed critical f

including support from managers and user

project staff, and controls over the acquisi

before and after a contract is awarded.

Assignment

Planning

When planning a risk assessment, the audi

first review the agency’s acquisition polici

directives to identify the organizations and 

responsible for approving procurements. A

thresholds, for example, should show whic

have the authority to review and approve a

Page 9 GAO/IMTEC-8.1.4 Assessing Acq
The agency’s directives should also show t

milestones and documentation required fo

procurement.

The auditor should also review previous st

audits of the acquisition project and of the 

information resources management functi

Reports by GAO or other auditing instituti

provide valuable background information. 

auditor should also determine whether pre

recommendations have been carried out.

Organization and

Use of the Audit

Guide

The chapters of this audit guide focus on l

distinct steps of the acquisition process, as 

on page 10 of GAO’s acquisition model. 1 T

following table identifies the appropriate c

reviewing the various steps of an acquisiti

general, the auditor will want to concentra

steps that are relevant to the phase of the 

being reviewed. However, regardless of ho

acquisition has advanced, at a minimum th

should always ascertain whether senior m

users were involved in the project’s initiati

auditor should also verify that the agency 

its needs and requirements to support its m

that those requirements continue to be val

acquisition progresses through contract aw

contract management.

1

GAO/IMTEC-8.1.6, August 1990.

Page 10 GAO/IMTEC-8.1.4 Assessing Acq
Table I.1: Map to the

Audit Guide Acquisition Phase Steps in Each Phas

I: Presolicitation Initiate Project 

Analyze

Requirements 

Identify Alternatives 

Prepare Acquisition

Plan 

Prepare

Specifications 

II. Solicitation and

Award

Maintain Project

Structure 

Prepare Solicitation 

Release Solicitation 

Evaluate Proposals 

Negotiate with

Vendors 

Select Contractor 

III. Postaward Establish Contract

Management 

Monitor Contract

Performance 

Test and Accept

System 

Each of the following chapters provides re

regulations and other guidance relevant to 

the chapter. In addition, each chapter iden

specific audit objectives and documentatio

for the major activities at that point in the 

process. The documents listed may exist w

different names than those used here. The 

should refer to agency-specific requiremen

information. Finally, each chapter sets out 

to help plan and conduct the assessment o

acquisition. The questions may need to be 

Page 11 GAO/IMTEC-8.1.4 Assessing Acq
the particular circumstances of an audit, u

subject agency’s requirements.

The appendixes provide further informatio

planning and conducting an acquisition au

Appendix I contains a worksheet, called an

acquisition profile, to summarize essential

information about the acquisition project. 

worksheet should contain such informatio

names and locations of units responsible f

acquisition, the project purpose, and the e

cost and time frames. The completed profi

be kept available for later reference. If a p

from an earlier assignment, the auditor sho

and update it.

Appendix II of this guide describes techniq

in identifying software development risks 

and cost overruns, known collectively as m

metrics or indicators. These techniques re

information about the size of the system b

developed and about progress after develo

begun. Thus, the techniques are only appro

use after the agency has completed its des

progressed into developing the system. An 

using this guide to review an acquisition th

significant software development and has 

in place should review appendix II for tech

help identify variances from the agency’s c

schedule estimates. In some cases, the pro

may be unable to complete a project on tim

an unrealistic schedule. The cost models d

appendix II can be used to make a rough e

how long a system development may requi

estimate can be compared to agency plans 

the agency has committed itself to unreali

short time frames. In other cases, delays in

completing scheduled activities, such as sy

design, coding, and testing, can lead to pro

Page 12 GAO/IMTEC-8.1.4 Assessing Acq
slippage later on. Auditors will need to tail

of the tools described in appendix II to the

circumstances of the audit.

Appendix III provides a comprehensive lis

references cited throughout the guide, as w

publications that may be useful for technic

information. Finally, a glossary is attached 

defines a wide range of technical and proc

terminology.

Application of

Audit Guide to

Prototyping

Methodology

This acquisition guide is intended for use i

any information technology acquisition, re

the system development methodology bein

guide is structured around the federal acqu

process, and is independent of developme

for information systems. An auditor should 

that there are different system developmen

that may be used when a system developm

acquired. Such models may include the “w

model, rapid prototyping, or evolutionary

development. Some of the documents requ

different development approaches may dif

Prototyping, for example, may act as part 

requirements definition process, helping th

identify and control areas of high uncertai

technical risk. In this situation the auditor 

(1) focus on determining how one or more 

or incremental versions function to define 

agency’s requirements and (2) determine h

system development methodology used by 

controls the prototyping process.

One approach to using prototyping as part 

system development process has been des

“spiral” model of system development. 2

 Th

2

For a description of prototyping and the spiral mode

Pressman, Software Engineering: A Practitioner’s Ap

(New York: McGraw-Hill, Inc., 1992), pp. 26-34.

Page 13 GAO/IMTEC-8.1.4 Assessing Acq
model portrays a process in which an agen

iteratively (1) determines its objectives, al

and constraints; (2) evaluates its alternativ

identifies and resolves risk issues; (3) deve

verifies its next-level products; and (4) pla

phases. Prototypes are developed or modi

of the second phase. As part of this model

prototype is developed or revised wheneve

analysis shows that significant areas of un

remain that pose substantial risks to proje

When the system has been defined well en

manage risks effectively, the agency devel

tests a full-scale system.

In order to review an acquisition that is us

prototypes, the auditor should determine w

regulations or guidance the agency has to 

prototyping methodology. This methodolo

establish the documentation and approval 

agency officials should meet. The auditor c

measure the progress of the prototypes ag

agency’s criteria.

Presenting

Conclusions and

Recommendations

The auditor should conclude the review by 

outstanding areas of risk and the agency’s 

address those risks. Note should be made 

the agency has addressed the critical facto

1990 model of information technology acq

well as agency compliance with acquisitio

regulations, standards, and other federal g

Results of the review should be communic

agency officials for their comment, in acco

with government auditing standards.

The auditor should also recommend chang

ensure that the agency is properly address

critical factors in the GAO model. For exam

3

GAO/IMTEC-8.1.6, August 1990.

Page 14 GAO/IMTEC-8.1.4 Assessing Acq
auditor may recommend a greater role for 

they are not involved in approving alternat

validating system requirements. Similarly, 

should recommend that the agency take ap

actions where senior management involve

lacking, or where the project organization 

and subject to high turnover. The recomm

should be reviewed with agency officials a

appropriate to the probability and significa

risks identified.

Page 15 GAO/IMTEC-8.1.4 Assessing Acq
This chapter focuses on levels of commitm

support for a planned acquisition by senio

and users, two key stakeholders in an acqu

Senior managers include those who have o

agency responsibility for strategic, includi

information-related, objectives. Users are 

operate or rely on agency information reso

include the managers and staff responsible 

policies and programs supported by the ac

The involvement and support of senior ma

throughout an acquisition is essential for s

Senior management should envision the ag

acquisition goals, define strategic objectiv

oversee the projects that implement the ov

vision. One or more senior managers shou

system sponsors, with sufficient authority 

that applicable resources are available for 

Users should also be involved and provide 

throughout the acquisition to ensure that t

requirements are understood and that the 

system is both accepted and used. User inv

should be sustained from the needs determ

phase through final acceptance and implem

User involvement in the acquisition proces

avoid the development of products that ult

not meet agency requirements.

The audit steps in this section should be u

assess the potential risks posed by the lack 

management or user support. An acquisitio

lacks either or both of these elements is at 

incurring unnecessary cost overruns, not m

planned delivery schedule, and not satisfyi

needs.

Page 16 GAO/IMTEC-8.1.4 Assessing Acq
Audit Objectives 1. To ensure that senior managers support 

actively involved throughout the developm

implementation of an acquisition.

2. To ensure that users support the acquisi

actively participating in defining procurem

requirements, developing the solicitation d

and verifying that the equipment and/or se

contracted for meet the agency’s needs.

Documentation

Required

• Decision papers, memoranda, or other rec

senior management oversight and approva

acquisition objectives and plans.

• Program management directives or other w

directives from senior managers stating go

objectives of the acquisition and delegatin

to carry out the acquisition.

• Budget exhibits and plans showing that su

funding is committed to the acquisition.

• Project plans showing the role of users in 

and overseeing the acquisition. Any docum

the users’ role in validating acquisition req

alternatives, and the solicitation documen

roles and responsibilities may be detailed 

memorandum of understanding between u

organizations and the program office.

• Agency policies or guidelines on the struct

steering committees or other oversight bo

responsibilities of project members and se

managers delineated.

Audit Steps: Top

Management

Support

1. Identify senior management officials res

for the acquisition. Include senior program 

heading the user organizations, informatio

management officials, and members of sen

oversight or steering committees. Determi

Page 17 GAO/IMTEC-8.1.4 Assessing Acq
and responsibilities of any groups or comm

senior managers, and the relationships bet

2. Review the documentation related to th

acquisition and determine if the senior ma

identified in the acquisition profile:

a. Approve the goals and objectives of the 

b. Designate a program sponsor who is res

and accountable for the acquisition.

c. Establish a formal process to keep conc

parties appropriately informed.

d. Participate in the specified reviews and 

• Determine how promptly management rev

conducted and approvals given and if the a

are given at the appropriate levels.

• Review documentation of such reviews, su

decision memoranda and review-committe

• Determine the frequency of reviews and th

direction senior managers give to project p

e. Provide initial funding for the project an

near- and long-term funding commitmen

periodically informing Congress of acqu

objectives and status.

f. Secure any necessary support from key 

organizations, such as OMB, GSA, and re

congressional committees.

g. Assign independent officials to ensure th

and internal controls needs are met.

3. On the basis of the above audit steps an

contacts with other project officials, deter

Page 18 GAO/IMTEC-8.1.4 Assessing Acq
whether senior managers fostered good w

relations among the sponsor, acquisition m

(program manager), other top managers, t

offices, and the contracting community. Sp

determine whether the managers:

a. Coordinate agreement for developing th

evaluation and source selection plans an

acceptance of the evaluation process an

b. Coordinate an agreement between the p

staff and technical and contracting offic

managing the contract.

c. Obtain the support of important acquisit

officials in the department or agency, su

official responsible for source selection

4. Obtain users’ and project staff’s evaluati

management’s support in the above steps. 

how long it took to get approvals from ma

and the direction management gave to pro

personnel.

Audit Steps: User

Involvement

1. Identify the population of users from pr

documents and the agency’s organization c

Review agency criteria (regulations and pr

to determine the roles the agency assigns t

Determine from the program manager and 

users which user organizations are activel

in the acquisition. Identify significant user 

who are not involved in the acquisition.

2. Determine if users:

a. Are involved in periodic reviews, and if 

frequently they are involved.

Page 19 GAO/IMTEC-8.1.4 Assessing Acq
b. Sign off on needs and requirements stat

otherwise validate the requirements and

corresponding solutions when develope

more information see ch. 4, step 3 under 

Determination, step 4 under Requiremen

Analysis, and step 1 under Specification

c. Validate alternatives against original req

(See step 1 of ch. 5 for related audit step

d. Approve the alternative selected (such a

choice between off-the-shelf technologi

custom development, centralized or dist

processing, etc). (See step 1 of ch. 5 for 

audit steps.)

e. Validate the specifications against the

requirements. (For more information se

under Specifications, in ch. 4.)

f. Provide acceptance criteria.

g. Assist in preparing the solicitation docu

awarding the contract (for example, use

representatives may assist in developing 

criteria and participate in the source sel

evaluating alternative proposals). (See c

and ch. 8, step 1d, for related informatio

h. Participate in postaward activities that m

installation, test, and acceptance of equ

i. Participate in reviews of contractor-prep

deliverable documents such as design

specifications, system analyses, and user 

training manuals.

j. Participate in government/contractor wo

groups.

Page 20 GAO/IMTEC-8.1.4 Assessing Acq
k. Participate in the postaward audit for as

degree of success of the acquisition.

3. Determine if users allocate staff time an

resources to the project.

4. Determine if the users participating in th

have a high, moderate, or low turnover rat

5. Assess the adequacy of users’ funding su

the project.

a. Obtain funding commitment from users

b. Identify appropriations to be used and t

availability to support contract award.

References • Federal Information Resource Managemen

Regulation (FIRMR), Part 201-2: Designate

Officials.

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase I, 

3; Phase II, step 1; Phase III, step 1.

Page 21 GAO/IMTEC-8.1.4 Assessing Acq
Project management for an acquisition is

accomplished primarily by a program man

staff responsible for carrying out project a

The program manager should have sufficie

authority and an appropriate mix of skills 

experience to successfully manage the pro

The acquisition project staff should be ass

roles and responsibilities. The team should 

members who are skilled in the informatio

technology procurement process, understa

technology, and have experience in manag

contracts. The team should also have mem

knowledgeable about the programs that th

acquisition is to support.

Audit Objective To determine if the acquisition team has th

necessary skills and authority to effectivel

execute the acquisition.

Documentation

Required

• A list of key project team members showin

responsibilities, job titles, and experience. 

auditor may have to generate this list on th

interviews if one is not available.

• The agency procurement request for a dele

procurement authority from GSA, showing 

experience of senior project officials (as r

GSA guidance detailed in its FIRMR Bullet

Audit Steps:

Project

Management

1. Review the agency’s criteria for acquisit

determine the responsibility and accounta

program manager and his/her required qua

2. Review the documentation related to th

acquisition to determine how clearly the p

Page 22 GAO/IMTEC-8.1.4 Assessing Acq
manager’s responsibility and accountabilit

defined. Determine if the program manage

a. A charter to establish authority, respons

 accountability.

b. A clearly defined relationship with the p

 technical, and procurement offices.

c. A clearly defined relationship with the s

 users.

d. Access to senior agency officials.

e. The authority to manage acquisition fun

f. Input to the budgetary process.

3. Review the qualifications of the program 

to determine if the program manager has t

appropriate mix of skills and experience. H

program manager directed other projects o

size and complexity? Is he or she trained t

complex acquisitions (GSA’s Trail Boss pr

be an example of such training).

4. Review management continuity on the a

as shown by the turnover rate of program 

Audit Steps:

Project Staff

1. Review the agency’s acquisition criteria 

determine both the responsibility and acco

of project personnel as well as their requir

qualifications.

2. Review the make-up of the project team 

that a mix of appropriate acquisition skills 

represented. (See ch. 9, steps 2 and 3, for r

steps.)

Page 23 GAO/IMTEC-8.1.4 Assessing Acq
a. Determine the authority and experience 

Contracting Officer’s Technical Represe

b. Identify key project staff and review the

experience and qualifications. Determin

the Contracting Officer is trained and ex

in information technology acquisitions.

c. Determine if the project staff is experien

managing contractors.

d. Determine the extent of turnover within 

staff.

3. Determine if the agency trains project st

maintain their skills and qualifications.

4. Review the reasonableness of project m

and schedule with project team members.

References • OMB Circular A-109: Major System Acquis

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase I, 

• GSA, Overview Guide, pp. 2-3 to 2-7.

• GSA, Guide for Contracting Officers’ Tech

Representatives, Chapter 2.

• FIRMR Bulletin C-5: Delegation of Procure

Authority for a Specific Acquisition.

Page 24 GAO/IMTEC-8.1.4 Assessing Acq
Requirements / 

Specifications

The purpose of this chapter is to guide the 

determining whether the agency has devel

accurate description of its information tec

needs. The acquisition should be clearly lin

program needs, to the agency’s overall stra

to governmentwide policies and standards

agency should expand on its basic descrip

needs to define specific requirements so p

information technology can respond with 

solutions. In some cases, an agency may u

prototypes to help define or validate its re

The requirements then form the basis for e

detailed specifications.

In identifying its requirements, an agency s

for testing the information resources it nee

plans should cover acceptance, security, a

certification requirements. The test plans d

at this point form the basis for later evalua

contract performance.

Failure to clearly and accurately define inf

technology requirements poses high risks 

agency. For instance, improperly defined

requirements may preclude alternatives, re

competition, increase the risk of cost and 

overruns, and lead to systems that are inco

with an agency’s overall architecture and

incompatible with other agency systems. T

hardware or software purchased may also 

inconsistent with government standards. D

and implementing a system is also more di

input, output, and processing specification

incomplete or inaccurate. In addition, if se

internal control requirements are not well 

control over sensitive information or other 

be lost.

Page 25 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

Audit Objectives 1. To ensure that the acquisition is based o

understood needs or opportunities and tha

consistent with the overall strategy and ar

used by the agency.

2. To ensure that the agency defines its req

based on the needs identified earlier and v

functional users, well enough to support th

acquisition of hardware, software,

telecommunications, and system developm

services. These requirements should prima

expressed in functional terms in accordan

FIRMR policy.

3. To ensure that system specifications cle

accurately summarize the agency’s require

Documentation

Required

• Needs statement.

• Requirements analysis or functional requir

 document.

• System specifications, if prepared. Also, dr

specifications with industry comments if d

specifications were released.

• Test plan and requirements prepared befo

award. Test requirements may be summar

test and evaluation master plan.

Audit Steps:

Needs

Determination

1. Review the agency’s stated needs, which 

documented in a Mission Element Needs S

Statement of Operational Need, or System

Operational Concept. Determine whether 

statement clearly and accurately reflects t

needs as indicated in the mission statemen

strategic objectives of the users’ organizat

strategic information plan, or the compute

plan.

Page 26 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

2. Check the needs statement for:

a. Existing system architecture and functio

supported (i.e., description, cost, volum

projected growth).

b. Justification for changes, such as correc

deficiencies in existing capabilities, com

new or changed program requirements, 

advantage of opportunities for increased 

and efficiency.

3. Contact several users as well as project 

are not users to determine if they generally 

the needs analysis presented in the needs 

adequately addresses actual problems. (Se

related questions.)

Audit Steps:

Requirements

Analysis

1. Review the requirements analysis to det

describes the current system. This descrip

include all the functions of the existing sys

any new system will have to perform. The 

functions, work load, operating costs, and

components of the current system should 

identified.

2. Confirm that the agency has defined its 

requirements for the new information reso

These requirements include:

a. Information now being received or infor

is needed but that is not being received.

b. Information to be provided to or obtaine

other agencies or the public.

c. Sources available from which to obtain 

information.

Page 27 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

d. Information relationships.

e. The degree of information validation, in

 accuracy, completeness, and reliability.

f. The quantity of information to be proces

 types of output expected.

g. The timeliness and format of the inform

h. The security, accessibility, and privacy

 requirements.

3. Determine if the agency has defined its f

and support requirements, including:

a. Present and projected work loads and c

 analysis, including peak load requiremen

 requirements for future capacity manag

b. Privacy and security requirements.

c. Contingency requirements for resources 

would either prevent or significantly imp

agency from performing its mission or w

an adverse impact on the nation.

d. Records management factors relating to 

of electronic records with other agency 

records retention and disposition, and s

against unauthorized use or destruction 

e. Space and environment factors, such as 

loading, heat dissipation, and power sup

f. Federal standards with which the new te

must comply.

g. Organizational training needs.

Page 28 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

h. Interfaces with other systems.

i. User interface requirements.

j. Compatibility limitation requirements.

k. Capability or performance validation re

4. Determine whether the requirements an

provides for:

a. A methodology for having users validate 

requirements analysis for both capabilit

performance. (See ch. 2 for related ques

b. Measurable requirements that may be u

verify system effectiveness.

5. Determine if the requirements are prese

functional or performance terms in consid

full and open competition. Functional requ

promote full and open competition, while

performance requirements may not.

6. With regard to restrictive requirements 

not lead to full and open competition), det

a. If brand-name-or-equal or specific make 

restrictions are appropriately justified.

b. Whether all required justifications are c

 and approved for other compatibility-lim

 requirements.

(See ch. 6, step 3, for more information on

procurements that do not promote full and 

competition).

Page 29 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

7. With regard to the process of revising re

during the acquisition, determine:

a. Whether a core of basic requirements ha

identified in order to maintain project sc

b. If a formal change control process has b

established to manage changes when ne

by a changing environment.

c. Who is responsible for reviewing and ap

changes to requirements.

d. How often requirements have been chan

e. Whether proposed new requirements ar

against mission needs.

f. What process is used to analyze the impa

changes on the other elements of the req

Audit Steps:

Specifications

1. Determine whether functional users and

program manager confirmed that the spec

accurately reflect the requirements and co

the approved acquisition strategy discusse

acquisition strategy module. Did users or t

manager sign off on the system specificati

ch. 2 for related questions.)

2. Examine the specifications document fo

a. A summary of the functional requiremen

satisfied by the technology.

b. Performance evaluation requirements.

c. Performance requirements that address

information accuracy; data integrity requir

Page 30 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

timing for response, update processing, in

transfer, transmission, and throughput; an

to changes in the requirements.

d. An identification of new types of equipm

required (e.g., processors, input/output de

information transmission devices).

e. An identification of support and test sof

f. A description of interfaces.

g. A description of overall security and pri

requirements.

h. A description of the operational control

i. A description of the operating characteri

user and computer centers where the softw

used.

j. A description of the logic flow of the ent

k. A specification of the functions to be sa

the software.

3. Determine how restrictions to full and o

competition in the specifications (e.g., equ

characteristics and performance elements

handled. Ensure that all required justificat

completed and properly approved for such

restrictions. (See ch. 6, step 3, for more inf

on procurements that do not promote full 

competition.)

4. Determine how changes to the original

specifications are handled.

Page 31 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

a. Establish responsibility for reviewing an

 approving changes to specifications. Int

 users to determine whether or not they 

 reviewed and approved changes to spec

 (See ch. 2, step 2e, under User Involvem

b. Review documentation on change reque

Determine how often specifications are ch

whether new specifications are validated a

requirements, and what process is used to 

impacts of changes on other elements of

specifications.

5. Determine if feedback (comments and q

from users and industry is accounted for, c

and incorporated as appropriate on a cont

(See ch. 7, step 6, for related information.)

Audit Steps: Test

Plans

1. Determine if the agency has developed t

based on acceptance criteria furnished or 

by the users.

2. Verify that test plans incorporate securi

certification requirements.

3. Determine if test plans adequately meas

performance requirements to be specified 

request for proposals (RFP).

(Note: Refer to ch. 10 for more information 

plans.)

References • FIRMR 201-20.1: Requirements Analysis.

• FIRMR 201-20.303: Standards.

• Federal Acquisition Regulation (FAR) Part 

Competition Requirements.

Page 32 GAO/IMTEC-8.1.4 Assessing Acq
Requeets / 

Specifications

• FAR Part 10: Specifications, Standards, an

Purchase Descriptions.

• GSA, Guide for Requirements Analysis and 

of Alternatives, Chapter 2: Requirements A

• American National Standards Institute/Ins

Electrical and Electronic Engineers (ANSI

Standard 830: IEEE Guide to Software Req

Specifications.

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase I, 

11, 12, 13, 14.

• Federal Information Processing Standards 

Publication 64: Guidelines for Documentat

Computer Programs and Automated Data 

the Initiation Phase.

• FIPS Publication 101: Guideline for Lifecy

Validation, Verification, and Testing of Com

Software.

Page 33 GAO/IMTEC-8.1.4 Assessing Acq
After identifying its requirements, the agen

assess alternatives for cost-effectively mee

requirements. The approach selected shou

an understanding of what is available in th

commercial market as well as what is avai

the government. Approaching the acquisiti

will lessen, but not eliminate, the risk that 

may select an alternative that does not full

requirements or that is unnecessarily com

expensive.

Audit Objectives 1. To determine if the agency has consider

reasonable alternatives for meeting its nee

2. To determine if the agency identified the 

costs, and benefits of each alternative.

3. To verify that the agency selected an alt

balancing expected benefits against costs, 

risks of failure.

Documentation

Required

• Record of alternatives analysis, such as a s

decision paper. Economic and risk analyse

accompany or be a part of the decision pap

• Market survey research conducted to iden

alternatives for meeting user needs and to 

cost estimates.

• Findings and approvals statements to supp

restrictions on specifications, such as

compatibility-limited requirements.

• Cost/benefit analysis to justify the selectio

alternative selected over other alternatives

terms or in terms of some other criteria, su

effectiveness.

Page 34 GAO/IMTEC-8.1.4 Assessing Acq
Audit Steps 1. Assess the involvement of responsible p

verify whether:

a. Users agreed with the range of alternativ

considered and were involved in validat

alternatives against the original requirem

related information on this and the next 

ch. 2, steps 2c and 2d, under User Involv

b. Users agreed with the alternative finally 

c. Appropriate senior management approv

alternative selected. (See ch. 2, step 2, u

Top-management Support, for more info

d. The Contracting Officer or other contra

 personnel participated in the alternative

 to ensure that a feasible acquisition app

 selected.

e. Project staff conducted market surveys 

 determine how industry can best meet t

 requirements.

2. Ensure that the agency considered, as a

the alternatives included in GAO’s acquisit

and FIRMR 201-20.203-1.

3. Assess how the agency evaluated altern

determining whether:

a. The agency consistently analyzed altern

the same criteria for each alternative.

b. The alternatives are described in suffici

 support time and cost estimates and cos

 analyses.

Page 35 GAO/IMTEC-8.1.4 Assessing Acq
c. The alternatives considered fit within th

 information architecture.

d. The range of alternatives considered wa

 by resource assumptions (staff or fundin

 limitations).

4. Determine whether the agency consisten

analyzed the costs and benefits of each alt

Ensure that the economic analysis include

values for costs and benefits and is update

periodically. Identify the system life used a

for evaluating alternatives and determine w

appears realistic in light of user needs, exp

changes in the technology, expected availa

maintenance and other support, and the ti

to prepare subsequent acquisitions. Verify 

economic analysis includes a sensitivity an

identify factors that affect the choice of on

alternative over another. The costs and be

considered for each alternative should inc

a. Conversion costs.

b. Personnel costs.

c. Operation and maintenance costs.

d. Nonrecurring but quantifiable benefits i

 information processing, administration, 

 support (these may include cost reducti

 resulting from improved system operati

 value enhancement through improved u

 resources).

e. Recurring and quantifiable benefits on a 

 and/or quarterly basis over the system li

 reductions in such items as salaries, frin

 supplies, utilities, and space occupancy.

Page 36 GAO/IMTEC-8.1.4 Assessing Acq
f. Any nonquantifiable benefits, such as im

 service and enhanced organizational im

5. Determine whether the agency analyzed 

following noncost factors for each feasible

alternative:

a. Obsolescence: strategies for avoiding ou

resources over the system life. A techno

upgrade clause is one way to avoid obso

allowing an agency to buy advanced ver

equipment or software when they becom

b. Availability: to what extent the system w

available to users.

c. Reliability: how frequently the system re

corrective maintenance.

d. Maintainability: the ease with which fail

components can be repaired, taking into 

the level of service, personnel support, a

supplies needed.

e. Expandability: the ease with which the s

be enhanced to meet anticipated growth

f. Flexibility: the extent to which the altern

 accommodate changes in the nature of t

 load.

g. Security: the ability to prevent unauthor

 and tampering and consideration of nati

 security and emergency preparations.

h. Privacy: the extent to which the privacy 

personnel-related data can be maintaine

Page 37 GAO/IMTEC-8.1.4 Assessing Acq
i. Affect on personnel: the impact on the le

 support personnel needed, including the 

 required.

j. User acceptance: the overall impact on t

 community, including the amount of cha

 procedures.

k. Accountability: the ability of the alterna

allow system activity to be tracked and 

6. Review the risk analysis to see if it ident

sensitive data and vulnerabilities. Verify th

magnitude of each vulnerability has been s

Determine whether or not the risk analysis 

with the standards identified in FIPS Publi

and 73 and OMB Circular A-130.

7. Verify that each alternative is evaluated 

financial, technical, and schedule risks. Fin

refers to the extent to which each alternat

subject to unexpected additional costs. Te

indicates the probability that each alternat

technical objectives will prove difficult to 

whole or in part. Schedule risk is the exten

each alternative is subject to unexpected s

delays and slippage in meeting the system

objectives, regardless of cost.

8. Ensure that the agency has selected the 

advantageous and realistic alternative with 

benefits, costs, and risks (based on steps 4 

9. Verify that users and senior managers ap

changes to the planned scope of the projec

Page 38 GAO/IMTEC-8.1.4 Assessing Acq
References • FIRMR Part 201-20.2: Analysis of Alternati

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase I, 

9.

• GSA, Guide for Requirements Analysis and 

of Alternatives, Chapter 3: Analysis of Alte

• OMB Circular A-130: Management of Fede

Information Resources.

• OMB Circular A-109: Major System Acquis

• OMB Circular A-76: Performance of Comm

Activities.

• FIPS Publication 64: Guidelines for Docum

Computer Programs and Automated Data 

the Initiation Phase.

• FIPS Publication 65: Guidelines for Autom

Processing Risk Analysis.

• FIPS Publication 73: Guidelines for Securi

Computer Applications.

Page 39 GAO/IMTEC-8.1.4 Assessing Acq
Acquisition planning is the process of coor

and integrating the efforts of personnel res

for acquisitions. A major objective of acqu

planning is to promote and provide for full 

competition. To ensure that the planning i

accomplished in an effective, economical, 

manner, the agency should prepare an acq

plan containing an overall strategy for man

preaward, acquisition, and postaward pha

An effective acquisition plan is critical to p

success. The plan sets out what the agency 

complete a procurement and how it will do 

plan also specifies the type of contract tha

awarded, how the agency will select a con

cost and schedule goals, milestones, signif

areas, and contract management controls.

Auditors should assess the extent to which 

agency’s acquisition planning is realistic an

comprehensive. One part of this assessme

be the review of the Agency Procurement 

(APR) to ascertain whether it is complete 

accurately reflects the objectives and scop

project.

Audit Objective To verify that the agency has defined an ef

strategy and plan for selecting a contracto

managing contract performance.

Documentation

Required

• Acquisition plan and related documents as

appropriate, such as a plan of action and m

• Agency procurement request and other

correspondence with GSA.

Page 40 GAO/IMTEC-8.1.4 Assessing Acq
Audit Steps 1. Determine whether the acquisition plan 

reviewed and approved in a timely manner 

officials designated in the agency’s acquisi

regulations. Ensure that the program mana

periodically reviews the acquisition plan a

it when necessary.

2. Review the acquisition plan and determi

contains the elements required by the FAR 

and GAO’s Information Technology: A Mo

Managers Decrease Acquisition Risks. The

include acquisition objectives; cost goals; 

decision-makers; capability or performanc

characteristics; risks associated with techn

matters, scheduling, and costs; plan of act

competition; source selection procedures; 

type and special contract provisions; contr

management procedures or organization; b

funding; information needed to monitor co

performance; test and evaluation; security 

privacy; and acquisition milestones. The p

also identify the acquisition method, key “

points, a formal training plan, and a contin

to minimize losses.

3. Determine if the acquisition plan calls fo

open competition. If the plan calls for limi

acquisition to resources compatible with e

equipment, verify that conversion cost stu

been completed to justify compatibility res

the plan calls for other than full and open

competition, verify that restrictions on com

such as make and model restrictions or so

requirements, have been justified and appr

designated authority. (See ch. 4 step 6, und

Requirements and step 3, under Specificat

4. Determine whether the agency has plan

“grand design” project or organized the ac

Page 41 GAO/IMTEC-8.1.4 Assessing Acq
into modules. Incremental purchasing may 

by identifying problems earlier, which allo

easier change or correction.

5. Evaluate the project management tools 

techniques to satisfy management informa

requirements for monitoring contractor pe

tracking progress against the acquisition p

taking action on cost or schedule slippage

6. Review the agency’s schedule for develo

solicitation document and for source selec

activities. Assess the reasonableness of th

through discussions with procurement off

reviews of project progress.

7. Identify the dollar limit of the agency’s g

delegation of procurement authority from 

Confirm that the agency receives a specifi

of authority from GSA if the value of the a

exceeds the agency’s authority level. Conf

new delegation of procurement authority w

received before a solicitation document is 

contract awarded.

8. Review the APR if the acquisition excee

agency’s delegated procurement authority 

Determine if the APR identifies the official

responsible for managing the effort, in acc

with the GSA guidance provided in FIRMR 

C-5. The APR should include:

Page 42 GAO/IMTEC-8.1.4 Assessing Acq
a. Names and titles of senior project offici

description of their roles in the organiza

acquisition exceeds $25 million, the APR 

also describe the project manager’s exp

previous acquisitions, responsibilities, a

authority, and the reporting structure fo

official as well as whether each official i

full- or part-time to the acquisition.

b. The project title and a brief description 

 acquisition.

c. Information resources currently in use.

d. Resources to be acquired.

e. The contracting approach. The approach 

include any limitations on competition, 

dates for release of the solicitation and f

award, and a strategy for a follow-on

implementation if a prototype is to be us

f. Estimated contract life and contract cos

g. Completion dates for key project docum

References • FAR Part 7: Acquisition Planning.

• FAR Part 34: Major System Acquisition.

• GSA, Overview Guide, p. 4-3.

• OMB Circular A-109: Major System Acquis

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase I, 

• FIRMR Bulletin C-5: Delegation of Procure

Authority for a Specific Acquisition.

Page 43 GAO/IMTEC-8.1.4 Assessing Acq
A solicitation document provides informat

necessary for vendors to propose equipme

software, and services to meet the agency’

requirements. In most cases, information r

will be purchased by issuing an RFP, whic

basis for the resulting contract. Less comm

agency may acquire information resources 

invitation for bids. An RFP may be preced

request for information or request for quot

An RFP should be clear and comprehensiv

include the elements described in GSA’s gu

standard solicitation documents. 1 The are

interest to auditors include section C:

Description/Specifications/Work Statemen

E: Inspection and Acceptance, and section 

Evaluation Factors for Award. Section C d

the tasks to be performed by the contracto

products to be delivered. Section E sets ou

government and contractor responsibilitie

ensuring that contract deliverables are acc

the agency. Section M explains how the ag

intends to select a winning contractor by d

the importance of all factors to be conside

evaluating proposals.

In developing the RFP an agency may hold

presolicitation or preproposal conferences 

seek industry views on the planned acquis

encourage companies to offer proposals. O

RFP is developed, it may be released in dr

order to obtain industry questions and rea

Auditors should determine what steps the 

taken to get feedback on its requirements, 

agency has handled comments or question

proposed RFP, and whether the agency ha

ensure that contractor proposals are comp

1

See, for example, U.S. General Services Administrat

Information Resources Management Service, Overvie

Acquisition of Information Resources, (Jan. 1990).

Page 44 GAO/IMTEC-8.1.4 Assessing Acq
A proper RFP is a critical element of a suc

acquisition because it becomes part of the 

contract once a proposal is made and acce

RFP does not accurately and clearly descr

agency’s requirements, or if the evaluation 

not accurately reflect the agency’s prioritie

resulting acquisition may not meet user ne

RFP is unjustifiably restrictive, favoring on

contractor over others, the agency may be 

benefit from full and open competition.

Auditors should become familiar with the 

format for an RFP. The audit team should 

persons sufficiently knowledgeable about 

information technology being purchased t

well the agency has defined its requiremen

RFP. Team members should include or hav

people who can identify areas where the R

not define the agency’s needs well enough 

the government’s interests. These persons 

know enough about performance or capab

validation techniques to determine whethe

agency’s requirements are reasonable and 

Audit Objectives To determine whether or not the solicitati

document is complete, clear, and consiste

that requirements continue to reflect user 

determine if the proposed evaluation proc

result in an effective and economical acqu

Documentation

Required

• Record of presolicitation or preproposal c

• Solicitation document: RFP or invitation fo

RFPs and Requests for Information, if any 

issued.

• Report of a solicitation review panel or co

appropriate.

• Source selection plan.

Page 45 GAO/IMTEC-8.1.4 Assessing Acq
• Benchmark materials or other capability a

performance validation requirements.

• Vendor comments or questions on solicita

document. If a vendor protests the solicita

determine the basis of the protest and how 

resolved.

• Proposal evaluation guide.

Audit Steps 1. Determine whether the solicitation docu

contains:

a. A statement of work or specifications st

that clearly and accurately describes the

government’s requirements, including a 

definition of all deliverables and the con

their acceptability.

b. A clear definition of government and co

 responsibilities.

c. The relative importance of evaluation fa

d. A proposal format requiring that cost an

elements be separated.

e. Reasonable provisions that protect the a

(such as liquidated damages provisions) 

incentives to the contractor (such as bo

good performance). Identify option clau

create uncertainty in work-load projecti

2. Examine the evaluation criteria to ensur

a. They are consistent with the requiremen

specifications, and proposal preparation

instructions.

Page 46 GAO/IMTEC-8.1.4 Assessing Acq
b. They provide all the factors and signific

subfactors to be considered in evaluatin

and the relative importance of different 

or cost factors, in accordance with FAR 

3. Evaluate the performance evaluation pa

agency will use. Determine whether the ag

reimburses contractors for participating in

benchmarking or other testing, and wheth

of such performance evaluation efforts co

barrier to competition.

a. If there is a benchmark, has the benchm

 independently examined by some outsid

 organization? Review the benchmark pl

 confirm that demonstration criteria are 

 stated. Determine how the agency selec

 representative mix of programs for the b

 Determine if the complexity of program

 benchmark is representative of the proje

 load. Determine how the agency validat

 benchmark as representative of the agen

 load.

b. If there is simulation or modeling, deter

 the agency selected parameters for the m

 Review any concerns or complaints rais

 vendors.

c. If a compute-off (demonstration of proto

 be used, confirm that the agency has est

 plans for prototype and follow-on contra

4. Interview users and managers, if necess

determine whether they concur with the s

Determine whether users or managers ide

changed requirements not included in the 

out if there are any factors considered imp

selecting an offer that are not included in t

Page 47 GAO/IMTEC-8.1.4 Assessing Acq
evaluation criteria or if other included fact

restrict the competition. (For related infor

ch. 2, step 2g, under User Involvement.)

5. Review the agency’s source selection pl

that it clearly describes the source selectio

organization and activities. Ensure that the 

selection plan has been approved by the so

selection authority before presolicitation c

are held or the solicitation document is iss

accordance with FAR 15.612.

6. Review the industry feedback process a

determine if:

a. Comments received on the draft solicita

identify any need for clarification, restri

specifications, or alternative ways of sat

needs. (See ch. 4, step 5 under Specifica

related question.) 

b. The agency responded promptly and tho

the comments received. (Judgmental sa

techniques may be required in this secti

number of vendors and vendor commen

substantial.)

c. The feedback provided adequate comme

product availability in the market.

d. The use of an ombudsman facilitated th

 addressing vendor concerns, disputes, a

 grievances.

7. Determine who has performed legal rev

solicitation document.

8. Determine if any vendor submitted a pro

whom (the agency, GAO, or GSA’s Board o

Page 48 GAO/IMTEC-8.1.4 Assessing Acq
Appeals), and how it was resolved. Determ

basis of the protest and its resolution.

References • GSA, Overview Guide, Chapter 6.

• FAR, Part 15: Contracting by Negotiation, 

15.4: Solicitation and Receipt of Proposals 

Quotations; Subpart 15.612: Formal Source 

Subpart 15.605: Evaluation Factors.

• FAR Part 14: Sealed Bidding.

• FAR Part 5: Publicizing Contract Actions.

• FAR Part 34: Major System Acquisition.

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase II

through 7.

• FIPS Publications 75 and 42-1.

Page 49 GAO/IMTEC-8.1.4 Assessing Acq
The source selection process is critical to 

the best value for the government. All prop

be evaluated in accordance with the criter

published in the RFP. If the evaluation pro

not conform to the RFP the agency runs a 

of a successful bid protest by losing vendo

agency should receive proposals, evaluate 

technical and cost merits of different prop

negotiate with contractors, and award a co

accordance with a source selection plan d

before release of the RFP.

The auditor should be aware of the agency

organization and procedures for making a 

award. Agencies may use some or all of th

positions:

• Contracting Officer (CO). The Contracting 

publicizes the procurement, amends the R

necessary, and conducts all negotiations w

offerors.

• Source Selection Authority (SSA). The SSA 

final decision on contract award. The Con

Officer may be the SSA for some procurem

in other cases a more senior manager may 

SSA.

• Source Selection Advisory Council (SSAC)

reviews the evaluations of different propo

makes a recommendation to the SSA on co

award.

• Source Selection Evaluation Board (SSEB

conducts technical and cost evaluations of 

proposals.

Audit Objective To ensure that the source selection proces

planned and carried out in order to succes

a contract that gives the best value to the g

Page 50 GAO/IMTEC-8.1.4 Assessing Acq
Documentation

Required

• Source selection plan, including source se

organization.

• Lessons learned report or other report by 

Contracting Officer describing negotiation

selection activities.

• Contracting Officer’s contract file.

• Records of debriefings, if any.

• Results of benchmarks or other performan

capability validation techniques used.

• Correspondence between offerors and the 

regarding questions or clarifications and a

amendments to the RFP.

• Proposal evaluation guide.

• Preaward survey reports.

Audit Steps 1. Examine the evaluation process by revie

records of the source selection procedures 

determine:

a. If evaluation personnel strictly adhered 

applied the publicized evaluation proces

criteria in the solicitation.

b. If evaluation factors were applied that w

listed in the solicitation.

c. Whether cost and technical evaluations 

 separately.

d. The role users played in the evaluation p

 (See ch. 2, step 2g, for related question.)

e. If the process resulted in (1) the establis

 competitive range and (2) removal of of

 further consideration, in accordance wit

 15.609.

Page 51 GAO/IMTEC-8.1.4 Assessing Acq
2. Determine how many vendors received 

solicitation and how many submitted prop

3. Determine if any vendor submitted a pro

so, whether the protest was handled by th

GAO, or GSA’s Board of Contract Appeals

the basis of the protest? How was the prot

resolved?

4. Determine whether the Contracting Offi

established prenegotiation objectives for c

and fee, and other issues in accordance wi

15.807. These objectives should help deter

overall reasonableness of proposed prices

be based on an independent government c

estimate and other information, such as a 

report on each contractor’s proposal.

5. Determine if the agency obtained field p

support in accordance with FAR 15.805-5. 

preaward audit of the cost proposal obtain

used during negotiations? Were the offero

proposed rates compared with the direct, 

overhead, and general and administrative r

recommended by the appropriate contract 

activity?

6. Examine the negotiation process.

a. Confirm that discussions with all vendo

 competitive range were held and the pro

 documented.

b. Determine from a review of documentat

 controls that existed to protect the secu

 sensitive information.

c. Contact responsible officials for their ev

of security.

Page 52 GAO/IMTEC-8.1.4 Assessing Acq
d. Determine if technical leveling or techn

transfusions occurred that would chang

proposals. FAR 15.610 describes these a

prohibited actions, such as indicating a 

price that offerors must meet to be cons

contract award or informing a vendor of 

standing relative to others seeking the c

e. Determine if estimated life cycle costs w

reconciled with each vendor’s proposal 

that cost estimates appear realistic.

7. Examine how the agency handled best a

offers.

a. Determine if the agency made multiple c

best and final offers, justified in accorda

FAR 15.611.

b. Determine if the best and final offers we

and evaluated in accordance with the so

selection plan.

8. Determine if all debriefings:

a. Were scheduled as soon as possible whe

requested by the vendor.

b. Were based on a debriefing plan that ad

resolved issues likely to cause concern 

complaints among the losing vendors.

c. Were documented.

Page 53 GAO/IMTEC-8.1.4 Assessing Acq
d. Adequately explained why the losing ve

the contract. (Note: In the explanation, 

cannot make point-by-point comparison

other proposals, but can point out the

government’s evaluation of significantly 

deficient elements in the proposal of the 

being debriefed. Refer to FAR 15.1003 fo

information.)

9. Examine the effort made to identify less

learned.

a. Determine what policies or procedures 

users have to evaluate acquisition result

communicate “lessons learned” to staff 

future assignments and to other agencie

include a comparison of the preaward a

the acquisition plan?

b. Review the lessons learned report, if on

completed, to determine how agency of

assessed the contracting process.

References • FAR Part 15: Contracting by Negotiation:

Subpart 15.4: Proposals and Quotations.

Subpart 15.6: Source Selection.

Subpart 15.8: Price Negotiations.

Subpart 15.10: Notifications, Protests, and 

• FIRMR Part 201-39: Acquisition of Federal

Information Processing Resources by Con

• GSA, Overview Guide, Chapter 7: Source S

• GSA, Guide for Acquiring Commercial Sof

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase II

through 13.

Page 54 GAO/IMTEC-8.1.4 Assessing Acq
Contract management includes the steps r

ensure that the agency receives products a

within established costs and time frames. A

is required to monitor contractor performa

ensuring that work done conforms to the a

requirements. The agency must also contr

to the contract and accept or reject contra

deliverables. Finally, an agency should con

postimplementation reviews to determine 

acquisition goals were met and whether th

information resources acquired should be 

replaced.

The agency’s Contracting Officer and Cont

Officer’s Representative/Contracting Offic

Technical Representative hold primary res

for administering the contract. The Contra

Officer monitors costs as required by the c

type (fixed-price or cost-reimbursable) an

contract modifications as needed. The pro

manager helps monitor contractor perform

ensure that user requirements are met by t

products or services delivered and that sen

officials provide support and oversight.

A contract consists of the agency’s RFP, a

and the successful vendor’s proposal. The 

should specify all deliverables required fro

vendor. The Contracting Officer and Contr

Officer’s Representative/Contracting Offic

Technical Representative should ensure th

deliverables are received as required. Any 

cost reports required from the contractor 

reviewed and action taken to correct prob

necessary. Training, documentation, and m

requirements should be fulfilled.

Page 55 GAO/IMTEC-8.1.4 Assessing Acq
Audit Objectives To ensure that the agency:

1. Oversees contractor performance.

2. Ensures that contract requirements con

accurately reflect user needs.

3. Verifies that products and services deliv

user needs.

4. Implements configuration management.

5. Modifies the contract only as needed.

6. Enforces contract provisions intended t

the agency, such as warranties or liquidate

clauses.

Documentation

Required

• Agency regulations or directives specifyin

requirements for periodic reviews, manage

oversight, and configuration management.

• The contract as awarded and with modific

• The agency’s contract management organi

structure.

• Current status reports and cost or schedul

projections.

• Current budget reports.

• The configuration management plan for th

Audit Steps 1. Review agency directives to identify the 

requirements for contract oversight. Depa

Defense Standard 2167A, for example, req

periodic reviews of contract deliverables, 

and status reports from the contractor. De

has directives governing configuration ma

activities to ensure that the contractor del

equipment or services called for and that n

Page 56 GAO/IMTEC-8.1.4 Assessing Acq
to the contract are made without consider

their overall impact.

2. Identify the roles of users and senior ma

monitoring the contract, verifying that bot

senior managers are involved in managing 

contract and approving any changes.

3. Determine the authority and experience 

Contracting Officer’s Representative or Co

Officer’s Technical Representative. (See ch

for a related step.)

4. Evaluate the project staff.

a. Identify key project staff and review the

experience and qualifications. Is the Con

Officer trained and experienced in infor

technology procurement? Does the proj

staff experienced in managing contracto

b. Determine how much turnover there ha

within the project staff, including the pr

manager. (See ch. 3, step 2, for a related 

5. Assess changes to the agency requireme

ensure that the contract continues to refle

user needs. Review configuration managem

activities to verify that changes to requirem

recorded and controlled and that impacts 

to contract requirements are identified.

6. Assess changes to the agency’s cost and 

estimates. Are variances in cost and sched

projections tracked by the project manage

Contracting Officer’s Technical Represent

cost and schedule estimates changed appr

Page 57 GAO/IMTEC-8.1.4 Assessing Acq
7. Determine if agency monitoring of the c

performance includes:

a. Periodically reviewing both scheduled a

 completed deliverables and effectively r

 any delays. Determine how the agency c

 contractor progress with the contract w

 schedule.

b. Periodically reviewing contract reports. 

 sample of status and cost reports to veri

 are regularly submitted by the contracto

 required by the contract. Discuss their u

 with the project staff.

c. Assessing the adequacy of the contracto

assurance process.

8. Assess the effectiveness of the agency’s 

relationship with the contractor.

a. Verify that the agency has controlled ch

contract and integrated the change proc

acquisition management structure. Dete

impact of changes on contract cost and 

b. Determine if corrections are made, awa

 implemented, and damages assessed, as

 appropriate.

9. Determine if the agency controls contra

modifications by:

a. Requiring the contracting office to appro

contract modifications.

b. Establishing a review process to ensure 

proposed engineering changes are withi

of the contract.

Page 58 GAO/IMTEC-8.1.4 Assessing Acq
c. Regularly comparing contract expenditu

the delegation of procurement authority 

that the agency does not exceed its auth

of total expenditures.

References • GSA, Overview Guide, Chapter 8: Contract

Administration.

• GSA, Guide for Contracting Officers’ Tech

Representatives, Chapter 2.

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase II

Page 59 GAO/IMTEC-8.1.4 Assessing Acq
Testing provides the basis for making deci

whether to accept contract deliverables. F

to be effective, it must be addressed relativ

the acquisition so it can be properly includ

planning. Test plans provide testing proce

the evaluation criteria to assess results of 

An agency should establish its initial test p

presolicitation phase. These plans should 

the agency will verify that the acquired equ

software, or services meet user needs and 

security requirements. After a contract is a

agency will need to carry out test and acce

procedures. The auditor should ensure tha

planning is conducted early enough so tha

requirements are included in the contract.

In assessing the postaward phase, the aud

ensure that the agency has not accepted eq

software that does not meet its requiremen

contract should specify conditions for acc

performance. For example, the contract m

that a computer operate successfully for 3

consecutive days out of a 90-day test perio

personnel should ensure that the contracto

meets the conditions for acceptable perfor

Internal auditors may be required to verify 

equipment or software pass the specified t

Assessing the test and acceptance phase m

a high level of technical skill on the part of 

such as when an agency has contracted fo

development services and must test the qu

delivered software. The auditor should be 

understand the system requirements, deve

methodologies, and test tools being used.

Page 60 GAO/IMTEC-8.1.4 Assessing Acq
Audit Objectives To confirm that the agency has:

1. Fully defined its requirements for testin

technology to be purchased.

2. Effectively carried out test and acceptan

procedures to verify that the resources pu

meet the agency’s needs.

Documentation

Required

• Agency directives giving requirements for 

status reporting, configuration managemen

management oversight.

• Records of configuration reviews or other 

reports.

• Trouble reports or other records of deficie

by agency personnel.

• Records of product acceptances.

• Test plans for inspection and acceptance.

Audit Steps 1. Determine if test plans were developed 

determine whether the mandatory:

a. Functional requirements were satisfied.

b. Security requirements arising from gove

policy, agency mission needs, and speci

needs were satisfied.

2. Determine if the test plans include:

a. Types of testing.

• Unit testing—e.g., in software, individual c

modules are tested by the programmer wh

them.

Page 61 GAO/IMTEC-8.1.4 Assessing Acq
• Integrated testing—e.g., in software, aggre

functions formed by groups of modules an

intermodule communication links are test

• System testing—examines the operation o

system as an entity in an actual or simulat

operating environment.

b. The locations for testing.

c. A realistic testing schedule.

d. The resource requirements.

• Test equipment needed, including the spec

of use, types, and quantities needed.

• Software needed to support the testing.

• Personnel from both user and acquisition 

their needed numbers and skills specified.

e. Testing materials to be used.

• Documentation needed, such as source co

manuals.

• Software to be tested and its medium.

• Test inputs and sample outputs.

• Test control software and worksheets.

f. Training in testing to be given, personne

trained, and the training staff.

3. Determine whether criteria have been e

for certifying that security requirements ar

4. Determine whether the appropriate user

representative has formally acknowledged 

completion of testing and acceptance of th

not, determine the reasons and the potenti

Page 62 GAO/IMTEC-8.1.4 Assessing Acq
a. Determine if deficiencies discovered in 

deliverables are expeditiously resolved.

b. Determine if any requirements that were 

by the delivered hardware, software, an

telecommunications are still pending an

5. Interview system operators and users to 

if the system has been successfully integra

existing environment.

References • GSA, Overview Guide, chapter 9: Installati

Operation.

• GAO, Information Technology: A Model to 

Managers Decrease Acquisition Risks

(GAO/IMTEC-8.1.6, August 1990), Phase I, 

Phase III, steps 4 to 6.

• FIPS Publication 101: Guideline for Lifecy

Validation, Verification, and Testing of Com

Software.

Page 63 GAO/IMTEC-8.1.4 Assessing Acq
The acquisition profile, which is a mechan

documenting key information about an acq

under review, is used to help auditors plan 

conduct assessments of an acquisition. Th

which is kept available for future referenc

the

• overall characteristics of the acquisition in

objectives shown within the context of the 

or function(s) to be supported,

• management organization and staffing of t

acquisition project, and

• acquisition schedule and cost estimates.

Overall

Characteristics

1. What is the project’s name?

2. What is the purpose of the information t

acquisition? What missions or functions is 

acquisition to support?

3. What type of acquisition is it?

• system integration,

• commercial off-the-shelf applications,

• software conversion,

• software development,

• hardware, or

• other (specify).

4. Are there any related in-house efforts?

5. With what systems will this acquisition i

6. Is the acquisition based on

• full and open competition,

• competition restricted by compatibility lim

Page 64 GAO/IMTEC-8.1.4 Assessing Acq
• limited competition (such as make or mod

or

• noncompetitive, sole source.

7. Is the contract based on

• fixed prices (specify type of fixed-price co

• cost reimbursement (specify type of

cost-reimbursement contract).

8. What project management tools or techn

in use to oversee the project (such as Gant

critical path method/Program Evaluation a

Technique).

9. What, if any, development tools and tech

(such as Computer-Aided Software

Engineering-CASE) are in use?

10. If a primary contractor has been select

acquisition, list the contractor name, addre

telephone number, and point of contact.

11. Identify key subcontractors, with comp

addresses, telephone numbers, and points 

Management

Organization

and Staffing

12. How is the management of the acquisit

organized and structured?

13. What are the title, name, and phone nu

the

• acquisition sponsor,

• acquisition manager,

• contracting officer,

• user representatives, and

• senior officials who approve acquisitions.

Page 65 GAO/IMTEC-8.1.4 Assessing Acq
14. What are the responsibilities and dutie

acquisition manager? What is his/her autho

example, planning, budget, staffing, or pro

reporting)?

15. Has an acquisition steering committee 

established? What are the committee’s res

and duties?

16. Who staffs the acquisition team and wh

team members’ qualifications?

Acquisition

Schedule and

Cost Estimates

17. What was the most recent milestone or 

decision point approved?

18. What are the actual or estimated dates 

following:

• original start/completion dates,

• current start/completion dates,

• requirements analysis completed and appr

• solicitation document completed for relea

• contract awarded,

• initial operation,

• test and acceptance, and

• full operation.

19. Is the project on schedule or has there 

slippage?

20. Has the agency completed an estimate 

cost? If so, what are the original and curre

estimates?

21. Identify the budget authority and outla

for the project.

Page 66 GAO/IMTEC-8.1.4 Assessing Acq
22. Identify the net benefits or cost savings 

by the cost/benefit analysis used to justify 

alternative, if the acquisition has progresse

point.

23. Has the project received scheduled fun

resources or have there been shortfalls? (E

variances and their effects.)

Page 67 GAO/IMTEC-8.1.4 Assessing Acq
Purpose This appendix describes tools and techniq

measuring the status of acquisitions involv

significant software development. It descr

techniques, known as software metrics, fo

quantitatively measuring how closely a pro

conforms to development plans and assess

whether an acquisition is at risk of delay o

increases. These techniques require consid

information about the system being develo

can only be used after system design. The 

described here will generally be used after 

award, in order to assess how well the age

managing the system development proces

Software metrics, which use mathematica

measure elements of the development pro

intended to help organizations better unde

manage the relationships between resourc

development schedules, and the cost of so

projects. By using software metric tools an 

can independently evaluate software deve

projects by analyzing project budgets, requ

schedules, and resources, and then confirm 

question cost, schedule, and resource estim

Different metrics may be useful for an aud

depending on the objectives and status of 

acquisition. Using cost models to estimate 

and length of time necessary to develop a 

software system, for example, is appropria

after requirements have been defined, a sy

has been developed, and the size of the ne

has been estimated. The models described 

require estimates of either the lines of cod

number of function points that the new sy

include. Cost models project the time and 

develop a system on the basis of estimates 

system’s size and other pertinent factors. T

used before a solicitation for software dev

Page 68 GAO/IMTEC-8.1.4 Assessing Acq
issued in order to assess the reasonablene

agency’s schedule (if a system design has b

prepared), or after a contract has been aw

software development has begun. The oth

described in this appendix require that sys

development work be underway. A compa

actual number of functions successfully te

number planned in the acquisition schedul

example, requires that system testing be u

Cost Models Cost models are tools that estimate the eff

to develop software, based on assumed re

between the size of a system and the effor

design, code, and test the software. These 

help the auditor assess whether the acquis

estimated cost and schedule are reasonabl

model uses cost drivers, which are parame

as the level of experience of the programm

reliability requirements of the programs, a

complexity of the project, along with the e

project size, to derive overall cost and sch

estimates for the acquisition. When a syste

software is being developed, one or more 

may be useful. A model will be more reliab

takes into account the agency’s historical 

in developing systems.

Cost models are available both commercia

from a few agencies within the Departmen

Defense. Most of the tools were developed 

use with Defense department projects, but 

used with non-Defense systems. Many are 

industry-recognized models such as the Co

Cost Model (COCOMO), PRICE, Putnam, a

The commercial tools range in cost from a

to well over $20,000. Government-develop

government-modified tools are available fr

Page 69 GAO/IMTEC-8.1.4 Assessing Acq
charge with a nominal charge for upgraded 

the software.

Because many different packages are avai

because more than one can be used, audit

determine what models, if any, are used in 

agencies. Typically cost models are derive

gathered from years of experience with a w

of software projects. Therefore, accuracy 

predictions may improve as further experi

accumulated in an agency.

Auditors should use cost models to look fo

discrepancies with the cost and schedule e

established for an acquisition. By varying t

estimated values for cost drivers, the audit

be able to perform a sensitivity analysis ill

where project estimates are most suscepti

change. However, cost models have signifi

limitations in accuracy unless their underl

assumptions of system size and cost driver

carefully chosen and reflect the agency’s p

experience in system development. It is a m

auditor judgment to decide how discrepan

estimates and estimates provided by cost m

should be to raise concerns about risks of 

schedule overruns. In making this judgmen

auditor should take into account the uncer

estimates and assumptions made in using 

model.

Auditors should use a cost model to provid

estimates and not precise figures. Therefo

applying software metrics to audit work, c

taken to follow generally accepted govern

auditing standards when drawing conclusi

on the results of software metrics. Specific

findings should be qualified by the recogni

these tools are limited by the accuracy of t

Page 70 GAO/IMTEC-8.1.4 Assessing Acq
estimated system size and other project da

for the model, the historical data from whi

model was developed, and the fact that all 

are projections of an inherently uncertain 

Procedures for minimizing the effect of th

limitations require, for example, proper te

advice, qualifying language in audit report

review of data and assumptions by agency 

Auditors should also ensure that the mode

consistent with the software development

methodology of the project under conside

system developed through rapid prototypi

example, should be evaluated with a mode

prototyping into account.

Other Indicators The following indicators are intended to h

assess how effectively an agency is manag

system development contract. These indic

measure differences between what an age

for its contractor to accomplish by certain 

the systems development life cycle and the 

results. In most cases this comparison is m

done graphically. Using more than one ind

give a broader picture of a project’s status

should use indicators appropriate to the pr

review and for which data are available.

Problem Reports This indicator involves tracking the numbe

and closed problems reported by a contrac

system is developed. A problem could be a

discovered during design, coding, testing, 

implementation. Problems are distinguishe

failures of code, which represent defects d

during operation. Contracts should specify 

problems are to be identified and reported

reviewing these reports and noting how qu

Page 71 GAO/IMTEC-8.1.4 Assessing Acq
problems are resolved, auditors can obtain 

understanding of how well the contractor 

performing.

The project manager or Contracting Office

Technical Representative may be the best 

problem reports. Because an agency may c

prioritize problems in order of their impac

reports should distinguish between priorit

problems.

Figure II.1 shows one way to present this k

analysis. It shows the number of new prob

reported and the number of problems clos

to show a trend over time and to show any 

that may be developing. The auditor may a

to report total problems reported and reso

break them out by level of priority. Figure 

the length of time that problems have rem

in order to demonstrate how quickly softw

problems are resolved once found. In this 

three levels of priority are distinguished fo

problems.

Page 72 GAO/IMTEC-8.1.4 Assessing Acq
Figure II.1: Problem Reports in Software Project

Page 73 GAO/IMTEC-8.1.4 Assessing Acq
Figure II.2: Problems by

Priority Levels and

Number of Months

Unresolved

Software Volatility Software volatility is also measured graph

Figure II.3 shows changes in the total num

approved system requirements. Cumulativ

are also tracked, including additions, delet

modifications to requirements. Steady incr

number of requirements and changes to re

may indicate that the project is at risk for 

cost overruns.

Page 74 GAO/IMTEC-8.1.4 Assessing Acq
Figure II.3: Software Requirements Changes

Development

Progress

This indicator involves the comparison of 

expected progress in the design, coding, an

integrating of system units. Units can be m

terms of computer software units or comp

software configuration items. If the projec

not complete its design or programming an

activities as planned, this indicator can sho

schedule delays before major milestones a

The progress indicator can show all eleme

design, coding, testing, and integrating, or 

them as separate indicators. Figure II.4 sh

Page 75 GAO/IMTEC-8.1.4 Assessing Acq
planned and actual progress in the design 

of software units can be displayed.

Figure II.4: Development Progress

Software Size This indicator records changes in the expe

magnitude of the software development ef

size of the system, measured in source line

as established by the system design, may c

the system is coded. Changes in size can b

Page 76 GAO/IMTEC-8.1.4 Assessing Acq
as total lines of code or can be broken out 

modified, and reused lines of code. Chang

estimated system size may indicate that th

was overestimated or underestimated in si

complexity. These changes may also indic

requirements are changing and may be rel

software volatility issue described earlier. 

estimates of software size should alert the 

that the project’s schedule and expected c

underestimated. Changes in the expected 

may necessitate reestimates of the develop

using the cost models described earlier in 

appendix. Figure II.5 shows how the estim

software size can be tracked over time.

Page 77 GAO/IMTEC-8.1.4 Assessing Acq
Figure II.5: Software Size Estimates

Personnel Stability Tracking the total number of people assign

system development effort compared to pl

staffing levels provides another indicator o

problems. The auditor can examine projec

actual levels of total personnel or of key, e

personnel. Understaffing may result in sch

slippage. Adding personnel late in a projec

actually cause further slippage.

Page 78 GAO/IMTEC-8.1.4 Assessing Acq
Other metrics may be chosen as needed. A

could, for example, compare expected to a

usages of hardware resources in order to i

emerging computer capacity problems. An

approach would involve tracking changes 

estimated completion date for a system. T

described here offer suggestions to be tail

as appropriate to particular projects.

Page 79 GAO/IMTEC-8.1.4 Assessing Acq
Each chapter of this guide lists applicable 

materials including federal regulations and 

published by GSA, OMB, and other agenci

using these references, summarized below

should ensure that the most current versio

is used.

Federal

Regulations

Applicable regulations include both genera

procurement, the FAR, and the FIRMR reg

written by GSA specifically for acquiring f

information processing resources. GSA iss

regulations under its Brooks Act authority

to the FIRMR, GSA issues bulletins that pr

guidance on a wide range of federal inform

processing acquisition issues:

GSA • Federal Acquisition Regulation (FAR)

• Federal Information Resource Managemen

Regulation (FIRMR)

OMB Circulars • A-109: Major System Acquisitions

• A-130: Management of Federal Information

Resources—under proposed revision

• A-76: Performance of Commercial Activitie

• A-11: Preparation and Submission of Budg

Estimates

GSA Acquisition

Guide Series

• Overview Guide

• Guide for Requirements Analysis and Anal

Alternatives

• Guide for Acquiring Maintenance Services

• Guide for Acquiring Commercial Software

• Guide for Contracting Officer’s Technical

Representatives

• Guide for Acquiring Systems Integration S

Page 80 GAO/IMTEC-8.1.4 Assessing Acq
National Institute of

Science and

Technology

(NIST)—formerly

the National Bureau

of Standards

• Federal Information Processing Standard 

Publications. FIPS Publications describe f

standards for hardware, software, and the

management of information technologies. 

Publications relevant to the acquisition of 

technology include the following:

FIPS PUB 11-3—Guideline: American Nati

Dictionary for Information Processing Sys

Feb. 1, 1991.

FIPS PUB 38—Guidelines for Documentat

Computer Programs and Automated Data 

Feb. 15, 1976.

FIPS PUB 42-1—Guidelines for Benchmar

Systems in the Competitive Procurement

Environment, May 15, 1977.

FIPS PUB 46-1—Data Encryption Standard

Jan. 22, 1988.

FIPS PUB 64—Guidelines for Documentat

Computer Programs and Automated Data 

the Initiation Phase, Aug. 1, 1979.

FIPS PUB 65—Guideline for Automatic Da

Processing Risk Analysis, Aug. 1, 1979.

FIPS PUB 73—Guidelines for Security of C

Applications, June 30, 1980.

FIPS PUB 75—Guideline on Constructing

Benchmarks for ADP System Acquisitions

Sept. 18, 1980.

FIPS PUB 76—Guideline for Planning and 

Data Dictionary System, Aug. 20, 1980.

Page 81 GAO/IMTEC-8.1.4 Assessing Acq
FIPS PUB 87—Guidelines for ADP Conting

Planning, Mar. 27, 1981.

FIPS PUB 88—Guideline on Integrity Assu

Control in Database Administration, Aug. 

FIPS PUB 96—Guideline for Developing a

Implementing A Charging System for Data 

Services, Dec. 6, 1982.

FIPS PUB 99—Guideline: A Framework fo

Evaluation and Comparison of Software D

Tools, Mar. 31, 1983.

FIPS PUB 101—Guideline for Lifecycle Va

Verification, and Testing of Computer Soft

June 6, 1983.

FIPS PUB 102—Guidelines for Computer S

Certification and Accreditation, Sept. 27, 1

FIPS PUB 106—Guideline on Software Ma

June 15, 1984.

FIPS PUB 124—Guideline on Functional

Specifications for Database Management S

Sept. 30, 1986.

FIPS PUB 127-1—Database Language SQL

Feb. 2, 1990.

FIPS PUB 132—Guideline for Software Ve

and Validation Plans, Nov. 19, 1987.

FIPS PUB 146-1—Government Open Syste

Interconnection Profile, Apr. 3, 1991.

FIPS PUB 151-1—POSIX: Portable Operati

Interface for Computer Environments, Ma

Page 82 GAO/IMTEC-8.1.4 Assessing Acq
FIPS PUB 158—The User Interface Compo

Applications Portability Profile, May 29, 19

• Special publications. Publications relevant 

information technology acquisition include 

following:

500-087—Management Guide for Software

Documentation, January 1982.

500-090—Guide to Contracting for Softwar

Conversion Services, May 1982.

500-105—Guide to Software Conversion M

500-106—Guide on Software Maintenance

500-109—Overview of Computer Security

Certification and Accreditation.

500-120—Security of Personal Computer S

Management Guide.

500-133—Technology Assessment: Method

Measuring the Level of Computer Security

500-134—Guide on Selecting ADP Backup 

Alternatives.

500-147—Guidance on Requirements Anal

Office Automation Systems (Update).

500-148—Application Software Prototypin

Fourth Generation Languages.

500-153—Guide to Auditing for Controls a

A System Development Life Cycle Approac

500-154—Guide to Distributed Database M

Page 83 GAO/IMTEC-8.1.4 Assessing Acq
500-155—Management Guide to Software 

500-161—Software Configuration Managem

Overview.

500-165—Software Verification and Valida

Role in Computer Assurance and Its Relati

with Software Product Management Stand

500-172—Computer Security Training Guid

500-173—Guide to Data Administration.

500-174—Guide for Selecting Automated R

Analysis Tools.

500-175—Management of Networks Based 

Systems Interconnection (OSI) Standards: 

Requirements and Analysis.

500-180—Guide to Software Acceptance.

500-183—Stable Implementation Agreeme

Open System Interconnection Protocols.

500-184—Functional Benchmarks for Four

Generation Languages.

500-187—Application Portability Profile, T

Government’s Open System Environment 

OSE/1 Version 1.0.

500-192—Government Open Systems Inter

Profile Users’ Guide, Version 2.

500-193—Software Reengineering: A Case 

Lessons Learned.

Page 84 GAO/IMTEC-8.1.4 Assessing Acq
800-4—Computer Security Considerations 

Procurements: A Guide for Procurement In

Contracting Officers, and Computer Secur

March 1992.

GAO Audit Guidance • Information Technology: A Model to Help 

Decrease Acquisition Risks (GAO/IMTEC-

August 1990).

• Government Auditing Standards, 1988 Rev

Page 85 GAO/IMTEC-8.1.4 Assessing Acq
Information

Management and

Technology

Division,

Washington, D.C.

Mark E. Heatwole, Assistant Director

David R. Turner, Evaluator-in-Charge

Bernard R. Anderson, Senior Evaluator

Peter C. Wade, Senior Evaluator

Trinh N. Hoang, Computer Programmer

David Chao, Technical Reviewer

Shane D. Hartzler, Editor

Page 86 GAO/IMTEC-8.1.4 Assessing Acq
Acquisition The obtaining, by contract with appropriat

of supplies or services (including construc

and for the use of the federal government 

purchase or lease, whether the supplies or 

are already in existence or must be created

developed, demonstrated, and evaluated. A

begins at the point when agency needs are 

and includes the description of requiremen

satisfy agency needs, solicitation and selec

sources, award of contracts, contract finan

contract performance, contract administra

those technical and management function

related to the process of fulfilling agency n

contract.

Acquisition

Planning

The process by which the efforts of all per

responsible for an acquisition are coordina

integrated through a comprehensive plan f

the agency need in a timely manner and at 

reasonable cost. It includes developing the 

strategy for managing the acquisition.

Agency

Procurement

Request (APR)

A request by a federal agency for GSA to a

information processing resources or for G

delegate the authority to acquire these res

Architecture The overall structure of a computer system 

hardware and software.

Availability The degree to which a system or compone

operational and accessible when required 

often expressed as a probability.

Page 87 GAO/IMTEC-8.1.4 Assessing Acq
Baseline (1) A specification or product that has bee

reviewed and agreed upon that thereafter 

the basis for further development and that 

changed only through formal change contr

procedures. (2) A document or set of such 

formally designated and fixed at a specific 

during the life cycle of a configuration item

Baselines, plus approved changes from tho

baselines, constitute the current configura

identification. (3) Any agreement or result 

and fixed at a given time from which chan

justification and approval.

Benchmark Test A test that uses a representative set of pro

data designed to evaluate the performance 

computer hardware and software in a give

configuration.

Best and Final

Offer

A final opportunity for offerors in the com

range to revise proposals.

Capability

Validation

The technical verification of the ability of 

system configuration, replacement compo

features or functions of its software, to sat

functional requirements. The intent is to e

the proposed resources can provide the re

functions. Performance requirements are n

or measured in the validation. Examples o

validation include:

a. operational capability demonstrations o

functions of the hardware, operating syste

support software;

b. verification of conformance with inform

processing standards;

Page 88 GAO/IMTEC-8.1.4 Assessing Acq
c. expert examination of the technical liter

supplied with the offer;

d. contacts with other users of the propose

information processing resource; and

e. vendor certification of conformance wit

functional requirements.

Certification (1) A written guarantee that a system or co

complies with its specified requirements a

acceptable for operational use. For examp

authorization that a computer system is se

permitted to operate in a defined environm

formal demonstration that a system or com

complies with its specified requirements a

acceptable for operational use. (3) The pro

confirming that a system or component co

its specified requirements and is acceptab

operational use.

Change Control See configuration control.

Commerce

Business Daily

A daily publication that lists the governme

procurement invitations, contract awards,

subcontracting leads, sales, surplus proper

foreign business opportunities. (GSA/IRMS

for Acquiring Commercial Software, Jan. 1

Compatibility (1) The ability of two or more systems or c

to perform their required functions while s

same hardware or software environment. 

ability of two or more systems or compone

exchange information.

Page 89 GAO/IMTEC-8.1.4 Assessing Acq
CompatibilityLimited

Requirement

A statement of requirements expressed in 

require items to be compatible with existin

information processing resources.

Competitive

Range

The group of offerors selected, after techn

cost evaluation, to whom award of a contr

reasonable possibility.

Configuration (1) The arrangement of a computer system 

network as defined by the nature, number

characteristics of its functional units. (2) T

and logical elements of an information pro

system, the manner in which they are orga

connected, or both. The term may refer to 

configuration or a software configuration.

Configuration

Control

An element of configuration management, 

of the evaluation, coordination, approval o

disapproval, and implementation of chang

configuration items after formal establishm

their configuration identification.

Configuration

Item

An aggregation of hardware and/or softwa

designated for configuration management 

as a single entity in the configuration mana

process.

Configuration

Management

The continuous control of changes made t

hardware, software, and documentation th

the development and operational life of th

Page 90 GAO/IMTEC-8.1.4 Assessing Acq
Contracting

Officer (CO)

A person with the authority to enter into, a

and/or terminate contracts and make relat

determinations and findings.

Contracting

Officer’s

Technical

Representative

(COTR)

An individual to whom the CO delegates c

contract responsibilities, usually related to 

acceptance issues.

Conversion Modification of existing software to enable 

operate with similar functional capability i

different environment; for example, conve

program from FORTRAN to Ada or conver

program that runs on one computer to run 

another.

CostReimbursement

Contract

A contract in which the government reimb

contractor for expenses so long as the con

provides its “best effort” to complete the w

for.

Delegation of

Procurement

Authority (DPA)

Authority to acquire information processin

up to a specified limit, issued by GSA in re

an agency procurement request.

Federal

Acquisition

Regulation (FAR)

The regulation that codifies uniform acqui

policies and procedures for executive agen

governmentwide.

Page 91 GAO/IMTEC-8.1.4 Assessing Acq
Federal

Information

Resources

Management

Regulation

(FIRMR)

The regulation that sets forth uniform poli

procedures for acquiring information proc

resources; used in conjunction with the FA

Fixed-Price

Contract

A contract that provides for a firm price or 

appropriate cases, a firm price with fees o

adjustments.

Functional

Requirement

A requirement that specifies a function tha

or system component must be able to perf

Independent

Verification and

Validation (IV&V)

Verification and validation performed by a

organization that is technically, manageria

financially independent of the developmen

organization. (See verification and validati

separately as follows.)

Invitation for Bid

(IFB)

The solicitation document used when cont

sealed bidding.

Interoperability The ability of information technology reso

provide services to and accept services fro

resources and to use the services so excha

enable them to operate effectively togethe

Liquidated

Damages

Compensation to the government for a con

failure to perform in a timely manner.

Page 92 GAO/IMTEC-8.1.4 Assessing Acq
Maintainability The ease with which maintenance of a fun

can be performed in accordance with pres

requirements.

Market Survey Attempts to ascertain whether other qualif

capable of satisfying the government’s req

exist. This testing of the marketplace may 

written or telephone contacts with knowle

federal and non-federal experts regarding 

duplicate requirements and the results of a

test recently undertaken, to the more form

sources-sought announcements in pertinen

publications (e.g., technical/scientific jour

Commerce Business Daily), or solicitation

information or planning purposes.

Performance

Validation

The technical verification of the ability of 

system configuration or replacement comp

handle agency-specific work-load volumes 

and expected) within agency-determined

performance time constraints.

Program Manager The key management official who represe

program office in formulating resource req

and managing presolicitation activities. In 

organizations the program manager or ano

management official is designated as the a

manager for a specific acquisition.

Protest A written objection by an interested party 

solicitation for a proposed contract, (2) a p

award, or (3) the award of a contract.

Page 93 GAO/IMTEC-8.1.4 Assessing Acq
Prototype A preliminary type, form, or instance of a s

serves as a model for later stages or for th

complete version of the system.

Prototyping A hardware and software development tec

which a preliminary version of part or all o

hardware or software is developed to perm

feedback, determine feasibility, or investig

or other issues in support of the developm

process.

Rapid

Prototyping

A type of prototyping in which emphasis is 

developing prototypes early in the develop

process to permit early feedback and analy

support of the development process.

Reliability The ability of a system or component to pe

required functions under stated conditions 

stated period of time.

Request for

Comment

An announcement in the Commerce Busin

other publication requesting industry com

draft specifications for resources.

Request for

Information

An announcement in the Commerce Busin

other publication requesting information f

industry about a planned acquisition, and, 

cases, corporate capability information.

Request for

Proposals (RFP)

The solicitation document used in negotia

procurements to communicate governmen

requirements and to solicit proposals.

Page 94 GAO/IMTEC-8.1.4 Assessing Acq
Solicitation An official government request for bids/pr

generally publicized in the Commerce Bus

in accordance with federal regulations.

Source Selection

Authority (SSA)

The government official in charge of selec

source for an acquisition. Most often the ti

when the selection process is formal and t

is other than the Contracting Officer.

Source Selection

Evaluation Board

(SSEB)

A board composed of technical, contract, i

resources managers, and other governmen

whose primary function is to evaluate prop

received in response to an RFP.

Source Selection

Plan

A document that describes the entire proc

awarding a contract—proposal evaluation 

evaluation methodology, evaluator’s respo

and final selection procedures.

Specific Make

and Model

Specification

A description of the government’s requirem

resources which is so restrictive that only 

manufacturer’s products will satisfy the go

needs.

Specification A written description of the technical requ

for resources stated in an IFB or RFP.

Statement of

Work

A technical description of resources, prep

inclusion in a solicitation document.

System Life A projection of the time period that begins 

installation of the resource and ends when 

agency’s need for that resource has termin

Page 95 GAO/IMTEC-8.1.4 Assessing Acq
Technical

Leveling

Helping an offeror to bring its proposal up 

of other proposals through successive rou

discussion, such as by pointing out weakn

resulting from the offeror’s lack of diligenc

competence, or inventiveness in preparing 

proposal.

Technical

Transfusions

Government disclosure of technical inform

pertaining to a proposal that results in imp

of a competing proposal.

Test Plan A plan prepared by the government that de

specific tests and procedures to be followe

Uniform Contract

Format

The format required by the FAR for prepar

solicitation.

User (1) Any person, organization, or functiona

uses the services of an information proces

system. (2) Any person or any thing that m

receive commands and messages to or fro

information system.

Validation The process of evaluating a system or com

during or at the end of the development pr

determine whether it satisfies specified re

Verification (1) The process of evaluating a system or c

to determine whether the products of a giv

development phase satisfy the conditions 

the start of that phase. (2) Formal proof of 

correctness.

Page 96 GAO/IMTEC-8.1.4 Assessing Acq
Work Load The mix of tasks typically run on a given c

system. Major characteristics include inpu

requirements, amount and kinds of compu

computer resources required.

(510757) Page 97 GAO/IMTEC-8.1.4 Assessing Acq
Ordering Information

The first copy of each GAO report and 

is free. Additional copies are $2 each. 

should be sent to the following addres

accompanied by a check or money orde

out to the Superintendent of Documen

necessary. Orders for 100 or more cop

mailed to a single address are discoun

25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax numbe

(301) 258-4066, or TDD (301) 413-000

Each day, GAO issues a list of newly a

reports and testimony. To receive fac

copies of the daily list or any list from 

30 days, please call (301) 258-4097 us

touchtone phone. A recorded menu w

information on how to obtain these lis
Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Postage & F

GAO

Permit No