a

GAO

United States Government Accountability Office

Report to Congressional Requesters

July 2004 INFORMATION 

TECHNOLOGY

DOD’s Acquisition 

Policies and Guidance 

Need to Incorporate 

Additional Best 

Practices and Controls

GAO-04-722
 www.gao.gov/cgi-bin/getrpt?GAO-04-722.

 To view the full product, including the scope 

 and methodology, click on the link above. 

 For more information, contact Randolph C. 

 Hite at (202) 512-3439 or hiter@gao.gov. 

Highlights of GAO-04-722, a report to 

congressional requesters 

July 2004

INFORMATION TECHNOLOGY 

DOD’s Acquisition Policies and Guidance 

Need to Incorporate Additional Best 

Practices and Controls 

DOD’s revised policies and guidance largely incorporate 10 best practices for 

acquiring any type of information technology (IT) business system. For 

example, the revisions include the requirement that acquisitions be 

economically justified on the basis of costs, benefits, and risks. However, the 

revisions generally do not incorporate 8 best practices relating to the 

acquisition of commercial component-based systems. For example, they do 

not address basing any decision to modify commercial components on a 

thorough analysis of the impact of doing so or on preparing system users for 

the business process and job roles and responsibilities changes that are 

embedded in the functionality of commercial IT products. In total, GAO 

found that DOD’s acquisition policies and guidance fully incorporate 8 of the 

18 best practices that they were evaluated against, partially incorporate 5 

practices, and do not incorporate the remaining 5 practices (see figure). 

DOD intends to expand its acquisition guidance to incorporate additional 

best practices by September 30, 2004, but department officials cite other 

priorities as a reason why they have not been able to complete this effort 

and could not provide a plan specifying how this will be accomplished. Until 

DOD’s revised policies and guidance incorporate key systems acquisition 

best practices, the risk that system investments will not consistently deliver 

promised capabilities and benefits on time and within budget is increased. 

DOD’s revised policies also do not contain sufficient controls to ensure that 

DOD components appropriately follow the best practices that are 

incorporated in its policies and guidance. According to acquisition best 

practices experts, as well as GAO’s internal control guidance, controls are 

effective if they are backed by measurements that are verified. Although the 

revised policies and guidance require acquisition managers to examine and, 

as appropriate, adopt best practices, they do not cite what that examination 

entails. DOD believes existing controls are sufficient, even though these 

controls do not provide for measuring and validating the practices’ use. 

Without specific requirements to measure and validate the use of best 

practices, the risk that they will not be followed increases, which, in turn, 

increases the risk that system investments will not meet expectations. 

DOD Incorporation of Best Practices 

Relevant to any IT business systems

acquisition

0

1

2

3

4

5

6

7

8

9

10

Fully

incorporated

Partially

incorporated

Not

incorporated

Relevant to commercial component-based 

IT business systems acquisitions 

0

1

2

3

4

5

6

7

8

9

10

Fully

incorporated

Partially

incorporated

Not

incorporated

Number of best practices Number of best practices

Source: GAO, based on analysis of DOD data.

The way in which the Department 

of Defense (DOD) has historically 

acquired its business systems has 

been cited as a root cause for its 

limited success in delivering 

promised system capabilities and 

benefits on time and within budget. 

In response, DOD recently revised 

its systems acquisition policies and 

guidance to incorporate best 

practices, including those 

pertaining to business systems. 

GAO was asked to determine 

whether DOD’s revised systems 

acquisition policies and guidance 

(1) are consistent with industry 

best practices, including those 

pertaining to commercial 

component-based systems, and (2) 

provide the necessary controls to 

ensure that DOD component 

organizations adhere to the 

practices. 

To improve DOD’s ability to 

acquire IT business systems, GAO 

recommends that the Secretary of 

Defense incorporate additional 

best practices in DOD’s acquisition 

policies and guidance, and that the 

department strengthen controls for 

ensuring that best practices are 

appropriately followed. 

In commenting on a draft of this 

report, DOD agreed or partially 

agreed with our recommendations 

for incorporating additional best 

practices, but did not agree that it 

needed (1) a plan to govern its 

incorporation of these practices or 

(2) stronger controls for ensuring 

that best practices are followed. 
Page i GAO-04-722 Defense Acquisition Best Practices

Contents

Letter 1

Results in Brief 1

Background 3

DOD’s Acquisition Policy and Guidance Are Consistent with Some, 

but Not All, Key Acquisition Best Practices 10

DOD’s Acquisition Policies Do Not Contain Sufficient Controls to 

Ensure That the Requirement Is Met for Appropriately Applying 

Best Practices 22

Conclusions 23

Recommendations for Executive Action 23

Agency Comments and Our Evaluation 25

Appendixes

Appendix I: Objectives, Scope, and Methodology 28

Appendix II: Best Practices 31

Best Practices Relevant to Any IT Business Systems Acquisition 31

Complementary Best Practices Relevant to Commercial 

Component-Based IT Business Systems Acquisitions 37

Appendix III: Comments from the Department of Defense 42

GAO Comments 47

Appendix IV: GAO Contact and Staff Acknowledgments 50

GAO contact 50

Staff acknowledgments 50

Tables Table 1: Organizational Responsibilities for the DOD 5000 Series 

Documents 6

Table 2: Summary of Business Systems Acquisition Best 

Practices 11

Table 3: Activity-by-activity Comparison of the 5000 Series to Best 

Practices Relevant to Any IT Business Systems 

Acquisition 17

Table 4: Activity-by-Activity Comparison of the 5000 Series to Best 

Practices Relevant to Commercial Component-based 

Business Systems Acquisitions 20

Figures Figure 1: Revisions to the DOD 5000 Series Documents 7

Figure 2: Simplified Diagram of DOD’s Acquisition Management 

Framework 9
Contents

Page ii GAO-04-722 Defense Acquisition Best Practices

Figure 3: DOD Acquisition Policies and Guidance Incorporation of 

Best Practices 15

Abbreviations

AT&L Acquisition, Technology, and Logistics

DOD Department of Defense

IT information technology

NII Networks and Information Integration

OT&E Operational Test and Evaluation

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately.
Page 1 GAO-04-722 Defense Acquisition Best Practices

United States Government Accountability Office

Washington, D.C. 20548

A

July 30, 2004 Letr

The Honorable John Ensign

Chairman

The Honorable Daniel K. Akaka

Ranking Minority Member

Subcommittee on Readiness and Management Support

Committee on Armed Services

United States Senate

This report responds to your request that we assess the Department of 

Defense’s (DOD) recently revised policies and guidance for acquiring 

business systems for incorporation of acquisition best practices. As you 

know, the way in which DOD has historically acquired information 

technology (IT) systems has been cited as a root cause of these systems 

failing to deliver promised capabilities and benefits on time and within 

budget. The use of these best practices—which includes those practices 

pertaining to any business system, whether custom developed or based on 

commercially available product components, as well as those unique to 

commercial component-based systems—is intended to improve on this 

performance.

As agreed with your offices, our objectives were to determine whether 

DOD’s recently revised policies and guidance for acquiring business 

systems (1) are consistent with industry best practices, including those 

pertaining to commercial component-based systems, and (2) provide the 

necessary controls to ensure that DOD component organizations adhere to 

best practices. We conducted our work between December 2003 and May 

2004 in accordance with generally accepted government auditing 

standards. Details of our objectives, scope, and methodology are included 

in appendix I. 

Results in Brief DOD’s revisions to its systems acquisition policies and guidance 

incorporate many best practices for acquiring business systems. For 

example, the revisions recognize such practices as (1) economically 

justifying system investments on the basis of costs, benefits, and risks and 

(2) continually measuring an acquisition’s performance, cost, and schedule 

against approved baselines. However, the revised policies and guidance do 

not incorporate a number of other best practices, particularly those 

associated with acquiring commercial component-based business systems. 

For example, they do not address basing a decision to modify a commercial 
Page 2 GAO-04-722 Defense Acquisition Best Practices

component on a thorough analysis of the impact of doing so; evaluating 

contractors on their ability to implement commercial components and 

using the results in source selection decisions; and preparing system users 

for the business process and job roles and responsibilities changes that are 

embedded in the functionality of commercial software products. According 

to officials responsible for revising DOD acquisition policies and guidance, 

additional best practices and lessons learned will be incorporated into the 

acquisition guidance by September 30, 2004. However, documented plans 

for this task do not exist, and the associated resources needed to complete 

this task have not been assigned due to higher priority needs. Until these 

missing best practices are included in DOD’s acquisition policies and 

guidance, the risk is increased that systems acquisitions will not deliver 

planned capabilities and benefits on time and within budget. 

DOD’s revised acquisition policies also do not contain sufficient controls to 

ensure that military services and defense agencies appropriately follow the 

practices. Controls are considered effective if they are measured and 

verified. Although DOD’s revised policies require both the project manager 

and the investment decision authority to examine the adoption of best 

practices, neither the policies nor the associated guidance provide for 

measuring and verifying the use of the practices. As a result, DOD is 

increasing the risk that best practice adoption and use will not occur, 

which, in turn, increases the risk that systems acquisitions will not deliver 

what is planned—on time and within budget.

This report makes 14 recommendations to the Secretary of Defense that 

are aimed at strengthening DOD’s acquisition policy and guidance by 

including additional business systems acquisition best practices and 

controls for ensuring that best practices are followed. 

In its written comments (reprinted in app. III) on a draft of this report 

signed by the Principal Director, Deputy Assistant Secretary of Defense 

(Command, Control, Communications, Space and Information Technology 

Programs), DOD agreed with the importance and relevance of the best 

practices that we cite in the report. DOD also agreed or partially agreed 

with most of our recommendations, stating that it would either incorporate 

those practices that we reported as missing from the department’s 

acquisition policies and guidance, or consider augmenting its coverage of 

those practices that deserve greater emphasis in its policies and guidance. 

Further, while DOD acknowledged that incorporation of additional best 

practices in its acquisition policies and guidance should be undertaken or 

considered, it did not agree that it needed an explicit plan to govern its 
Page 3 GAO-04-722 Defense Acquisition Best Practices

ongoing and future policy and guidance revision activities, stating that our 

recommendation to this effect was inappropriate. We do not agree with 

DOD that a plan governing incorporation of the practices is not needed. 

Given the importance of DOD’s acquisition policies and guidance as well as 

best practices, we believe that having an explicit plan that defines how and 

when incorporation of best practices will be added is essential. Among 

other things, a plan would highlight the resource constraints that this 

revision effort has been subject to, would allow measurement against 

defined milestones, and would allow disclosure of progress and 

impediments. In its comments, DOD also did not agree that stronger 

controls are needed for ensuring adherence to the best practices contained 

in its acquisition policies and guidance. We do not agree with the 

department’s position on this matter because its existing controls do not 

provide for either the measurement or verification of whether the practices 

are employed—both recognized elements of effective process controls. 

Background Best practices are tried and proven methods, processes, techniques, and 

activities that organizations define and use to minimize risks and maximize 

chances for success. As we have previously reported, using best practices 

can result in better outcomes—including cost savings; improved service 

and product quality; and, ultimately, a better return on investment. For 

example, two software engineering analyses of nearly 200 systems 

acquisitions projects indicated that teams using systems acquisition best 

practices produced cost savings of at least 11 percent over similar projects 

conducted by teams that did not employ the kind of rigor and discipline 

embedded in these practices.1

 In addition, our research2 shows that best 

practices are a significant factor in successful acquisition outcomes, 

including increasing the likelihood that programs and projects will be 

executed within cost and schedule estimates.

DOD, GAO, and the Congress have all advocated the use of best practices. 

For example, in September 2000, DOD established a steering group to 

1

Donald E. Harter, Mayuram S. Krishnan, and Sandra A. Slaughter, “Effects of Process 

Maturity on Quality, Cycle Time, and Effort in Software Product Development,” 

Management Science, vol. 46, no. 4, 2000; and Bradford K. Clark, “Quantifying the Effects of 

Process Improvement on Effort,” IEEE Software (November/December 2000).

2

U.S. General Accounting Office, Defense Acquisitions: Stronger Management Practices 

Are Needed to Improve DOD's Software-Intensive Weapon Acquisitions, GAO-04-393

(Washington, D.C.: Mar. 1, 2004).
Page 4 GAO-04-722 Defense Acquisition Best Practices

promote the use of systems acquisition best practices and lessons learned. 

Further, our 2001 report3

 cited the benefits of DOD adoption of best 

practices and provided recommendations for establishing a mechanism for 

sharing best practices throughout DOD. In the fiscal year 2003 Defense 

Authorization Act,4

 the Congress used our recommendations in directing 

DOD to expand its use of best practices. Specifically, it required the Under 

Secretary of Defense (Acquisition, Technology, and Logistics (AT&L)) and 

the Assistant Secretary of Defense (Command, Control, Communications, 

and Intelligence—now called Networks and Information Integration 

(NII))—to identify and serve as a clearinghouse for information regarding 

software acquisition best practices in the public and private sectors. In 

response, DOD assigned AT&L responsibility for serving as that 

clearinghouse. Further, the Defense Information Systems Agency created a 

Web site to provide information about acquisition best practices. 

DOD Relies Extensively on 

IT Systems to Perform a 

Variety of Business 

Functions

DOD is one of the largest and most complex organizations in the world. In 

fiscal year 2003, DOD reported that its operations involved over $1 trillion 

in assets, nearly $1.6 trillion in liabilities, disbursements of more than $416 

billion, and approximately 3.3 million military and civilian personnel. 

Execution of these operations spans a wide range of defense organizations, 

including the military services, defense agencies and field activities, and 

various combatant and joint operation commands. To execute these 

military operations, the department performs an assortment of business 

functions, including logistics management, procurement, healthcare 

management, and financial management. 

To support its business functions, DOD reports that it currently relies on 

about 2,300 IT systems, including accounting, acquisition, logistics, and 

personnel systems. Moreover, its future investment in business systems is 

expected to be sizable. For fiscal year 2004, DOD requested approximately 

$28 billion in IT funding to support a wide range of military and business 

operations. Approximately $9 billion of this amount is to support primarily 

command and control systems, and the remaining $19 billion is to support 

the operation, maintenance, and modernization of business systems. 

3

U.S. General Accounting Office, DOD Information Technology: Software and Systems 

Process Improvement Programs Vary in Use of Best Practices, GAO-01-116 (Washington,

D.C.: Mar. 30, 2001).

4

Bob Stump National Defense Authorization Act for 2003 (Pub.L. No. 107-314).
Page 5 GAO-04-722 Defense Acquisition Best Practices

Overview of DOD’s 

Acquisition Management 

Framework

Since the 1980s, DOD’s oversight of its systems acquisitions had been 

defined by a series of three documents—commonly called the 5000 

series—that provided the policies and guidance for departmental efforts to 

acquire service capabilities and systems: 

• DOD Directive 5000.1, The Defense Acquisition System—describes the 

management principles for DOD’s acquisition programs.

• DOD Instruction 5000.2, Operation of the Defense Acquisition 

System—outlines the framework for managing acquisition programs. 

• DOD 5000.2-R, Mandatory Procedures for Major Defense Acquisition 

Programs (MDAPS) and Major Automated Information System 

(MAIS) Acquisition Programs—provides the mandatory procedures 

for acquiring major defense programs. 

These documents have been revised several times. Most recently, in 

October 2002, the Deputy Secretary of Defense determined that the 

existing versions of these three documents required further revisions to 

improve acquisition efficiency, flexibility, creativity, and innovation. As a 

result, the Deputy Secretary canceled the existing versions of each 

document and instructed the Under Secretary for AT&L; the Assistant 

Secretary for NII; and the Director, Operational Test and Evaluation 

(OT&E) to jointly revise the documents. (Table 1 describes selected 

responsibilities of these three entities.) The revised directive and 

instruction were issued in May 2003. Both were shortened and modified to 

focus on required outcomes and legal requirements and to eliminate the 

“how-to” details in the previous versions. In doing so, DOD intended the 

revisions to provide program managers with more flexibility in executing 

their respective programs.
Page 6 GAO-04-722 Defense Acquisition Best Practices

Table 1: Organizational Responsibilities for the DOD 5000 Series Documents

Source: GAO, based on DOD documentation.

DOD 5000.2-R was renamed the Interim Defense Acquisition Guidebook

and was made optional guidance on best practices and lessons learned. 

(See fig. 1.) According to DOD officials, improvements to the guidebook 

are currently under way. Until they are completed, DOD 5000.2-R serves as 

the guidebook.

Organization Responsibility

Office of the Under Secretary of 

Defense (AT&L)/Defense Acquisition 

Executive

Advises the Secretary of Defense on all matters 

pertaining to DOD’s acquisition framework as well 

as research and development; advanced 

technology; developmental test and evaluation; 

production; logistics; installation management; 

military construction; procurement; environmental 

security; and nuclear, chemical, and biological 

matters.

Office of the Assistant Secretary of 

Defense (NII)/DOD Chief Information 

Officer

Advises the Secretary of Defense on achieving 

and maintaining information superiority through 

the collection, processing, and dissemination of 

an uninterrupted flow of information in support of 

DOD missions while exploiting or denying an 

adversary’s ability to do the same.

Serves as the principal assistant to the secretary 

for electronic business, information management, 

information operations and assurance, and IT.

Office of the Director (OT&E) Advises the Secretary of Defense on OT&E.

Issues DOD OT&E policy and procedures, and 

reviews and analyzes OT&E conducted for each 

major acquisition program and provides reports 

on adequacy and results to the Secretary, the 

Under Secretary for AT&L, and the Congress.
Page 7 GAO-04-722 Defense Acquisition Best Practices

Figure 1: Revisions to the DOD 5000 Series Documents 

According to the revised policy, an acquisition is a directed, funded effort 

that provides a new, improved, or continuing materiel, weapon or 

information system, or service capability. The revised 5000 series applies to 

acquisitions conducted by all of the department’s organizational 

components.5

 These components include the military services and the 

defense agencies, such as the Defense Information Systems Agency. 

5

Collectively, these oversight policy and guidance documents cover most—but not 

all—major acquisitions. The Secretary of Defense has delegated authority to the Missile 

Defense Agency and to the National Security Space Team to develop separate guidance for 

missile defense and space systems, respectively.

5000.2

Revised

Revised

Renamed

Interim Defense

Acquisition Guidebook

October 30, 2002

DOD 5000.2-R

April 5, 2002

(Mandatory)

DOD Instruction 5000.2 

April 5, 2002

(Mandatory)

DOD Directive 5000.1

October 23, 2000

(Mandatory)

DOD acquisition policy as of 

October 2002 

DOD acquisition policy/

guidance as of May 2003 

DOD Directive 5000.1

May 12, 2003

(Mandatory)

DOD Instruction 5000.2

May 12, 2003

(Mandatory)

Source: GAO, based on DOD documentation.

(Not mandatory)
Page 8 GAO-04-722 Defense Acquisition Best Practices

The 5000 series describes a management framework that is intended to 

translate mission needs and requirements into systems acquisition 

programs. To accomplish this, the framework specifies five phases: 

• Concept refinement: Intended to refine the initial system concept and 

produce a strategy for acquiring a system capability. A decision is made 

at the end of this phase (milestone A decision) whether to move to 

technology development.

• Technology development: Intended to determine the appropriate set of 

technologies to be integrated into the system by iteratively assessing the 

viability of various technologies while simultaneously refining user 

requirements. Once the technology has been demonstrated in a relevant 

environment, a decision is made at the end of this phase (milestone B 

decision) whether to move to system development and demonstration.

• System development and demonstration: Intended to develop a system 

or a system increment and demonstrate through developer testing that 

the system/system increment can function in its target environment. A 

decision is made at the end of this phase (milestone C decision) whether 

to move to production and deployment.

• Production and deployment: Intended to achieve an operational 

capability that satisfies the mission needs, as verified through 

independent operational test and evaluation, and ensures that the 

system is implemented at all applicable locations.

• Operations and support: Intended to provide a support program to 

meet operational support requirements and sustain the system in the 

most cost-effective manner over its total life cycle.

According to the framework, an acquisition program may begin at 

milestone A, B, or C, and its progress depends on obtaining sufficient 

knowledge to make an informed decision about whether to continue to the 

next acquisition phase. Although the framework permits programs to be 

managed as a single project, DOD Instruction 5000.2 states that the 

department prefers an evolutionary acquisition strategy that delivers a 

mature product in increments. Under such a strategy, the instruction states 

that each increment is to begin with a milestone B decision, and the 

production and deployment phase of each increment is to begin with a 

milestone C decision. Figure 2 provides a simplified diagram of the 

department’s acquisition management framework.
Page 9 GAO-04-722 Defense Acquisition Best Practices

Figure 2: Simplified Diagram of DOD’s Acquisition Management Framework

Typically, the Under Secretary for AT&L or a designee serves as the 

investment decision authority6

 for DOD acquisitions, but the Assistant 

Secretary for NII/Chief Information Officer or a designee serves as the 

decision authority for IT systems acquisitions. 

6

DOD policy establishes a decision authority, called a milestone decision authority, as the 

designated individual who has overall responsibility for an IT investment. This person has 

the authority to approve an investment’s move from one phase to the next phase of the 

acquisition process and is responsible for reporting cost, schedule, and performance results 

to higher authorities, including the Congress. 

User needs and

technology 

opportunities

Concept

refinement

Technology

development

System 

development

and 

demonstration

Production and

deployment

Operations and

support

A B C

Increment 1

Increment 2

Increment 3

B C

B C

B C

Evolutionary

acquisition

Single-step

acquisition

Source: GAO, based on DOD Instruction 5000.2.

A

B

C

Milestone decision points
Page 10 GAO-04-722 Defense Acquisition Best Practices

Past Evaluations of DOD 

Business Systems Have 

Revealed Acquisition 

Management Weaknesses

Due in part to its long-standing and pervasive IT acquisition management 

weaknesses, DOD has had limited success in acquiring IT resources to 

replace its outdated business systems. Both inspector general and 

departmental studies have cited these weaknesses on a variety of 

acquisition projects. We have also reported on business systems acquisition 

weaknesses.7

 For example, in 2002 we reported that the Defense Logistics 

Agency did not have effective corporate processes for consistently 

acquiring software (the most costly and complex component of systems), 

and that the agency did not have a software process improvement program 

in place to effectively strengthen its software acquisition processes. In 

2002, we also reported acquisition management problems with the Military 

Health System’s acquisition of DOD’s primary medical information system, 

including weaknesses in incremental economic justification, risk 

management, and contract management. In 2003, we reported that DOD 

had not economically justified four finance and accounting systems that 

have an estimated cost of more than $1 billion. In each of our reports, we 

have made recommendations for strengthening acquisition management 

through the adoption of best practices. DOD has largely agreed with our 

recommendations, but its progress to date in implementing them across the 

department has been uneven. 

DOD’s Acquisition 

Policy and Guidance 

Are Consistent with 

Some, but Not All, Key 

Acquisition Best 

Practices

We and others, such as Carnegie Mellon University’s Software Engineering 

Institute, have identified and promoted the use of a number of best 

practices associated with acquiring IT systems. For the purposes of this 

report, we have identified 18 relevant best practices and grouped them into 

two categories: (1) 10 best practices for acquiring any type of business 

system and (2) 8 complementary best practices that relate specifically to 

acquiring commercial component-based business systems. Examples of 

best practices relevant to any business systems acquisition include 

ensuring that (1) reasonable planning for all parts of the acquisition occur, 

(2) a clear understanding of system requirements exists, and (3) risks are 

proactively identified and systematically mitigated. Examples of best 

7

U.S. General Accounting Office, Information Technology: Inconsistent Software 

Acquisition Processes at the Defense Logistics Agency Increase Project Risks, GAO-02-9

(Washington, D.C.: Jan. 10, 2002); Information Technology: Greater Use of Best Practices 

Can Reduce Risks in Acquiring Defense Health Care System, GAO-02-345 (Washington,

D.C.: Sept. 26, 2002); and DOD Business Systems Modernization: Continued Investment in 

Key Accounting Systems Needs to be Justified, GAO-03-465 (Washington, D.C.: Mar. 28, 

2003). 
Page 11 GAO-04-722 Defense Acquisition Best Practices

practices relevant to commercial component-based systems acquisitions 

include ensuring that (1) commercial product modification is effectively 

controlled, (2) relationships among commercial products are understood 

before acquisition decisions are made, and (3) the organizational impact of 

using new system functionality is proactively managed. Each of these 

practices is composed of from one to eight activities and is described in 

table 2. DOD officials responsible for revising the 5000 series told us that 

each of these 18 practices are relevant to DOD business systems 

acquisitions. Appendix II provides additional details on each of these 

practices. 

Table 2: Summary of Business Systems Acquisition Best Practices

Best practices Activity

Best practices relevant to any business systems acquisition

Acquisition planning

To ensure that reasonable planning for all 

parts of the acquisition is conducted.

• Plans are prepared during acquisition planning and maintained throughout the 

acquisition.

• Planning addresses the entire acquisition process, as well as life cycle support of the 

products being acquired.

• The acquisition organization has a written policy for planning the acquisition.

• Responsibility for acquisition planning activities is designated.

Architectural alignment

To ensure that the acquisition is consistent 

with the organization’s enterprise architecture.

• The system being acquired is assessed for alignment with the enterprise architecture 

at key life cycle decision points, and any deviations from the architecture are explicitly 

understood and justified by an explicit waiver to the architecture.

• Product line requirements–—rather than just the requirements for the system being 

acquired–—are an explicit consideration in each acquisition.

Contract tracking and oversight

To ensure that contract activities are 

performed in accordance with contractual 

requirements.

• The acquiring organization has sufficient insight into the contractor’s activities to 

manage and control the contractor and ensure that contract requirements are met.

• The acquiring organization and contractor maintain ongoing communication; 

commitments are agreed to and implemented by both parties.

• All contract changes are managed throughout the life of the contract.

• The acquisition organization has a written policy for contract tracking and oversight.

• Responsibility for contract tracking and oversight activities is designated.

• The acquiring organization involves contracting specialists in the execution of the 

contract.

• A quantitative set of software and system metrics are used to define and measure 

product quality and contractor performance.

• In addition to incentives for meeting cost and schedule estimates, measurable, 

metrics-based product quality incentives are explicitly cited in the contract.

Economic justification

To ensure that system investments have an 

adequate economic justification.

• System investment decisions are made on the basis of reliable analyses of estimated 

costs, expected benefits, and anticipated risks.

• Large systems acquisitions are (to the maximum extent practical) divided into a series 

of smaller, incremental acquisition efforts, and investment decisions on these smaller 

efforts are made on the basis of reliable analyses of estimated costs, expected 

benefits, and anticipated risks.
Page 12 GAO-04-722 Defense Acquisition Best Practices

Best practices Activity

Evaluation

To ensure that evidence showing that the 

contract products satisfy the defined 

requirements are provided prior to accepting 

contractor products.

• Evaluation requirements are developed in conjunction with the contractual 

requirements and are maintained over the life of the acquisition.

• Evaluations are planned and conducted throughout the total acquisition period to 

provide an integrated approach that satisfies evaluation requirements and takes 

advantage of all evaluation results.

• Evaluations provide an objective basis to support the product acceptance decision.

• The acquiring organization has a written policy for managing the evaluation of the 

acquired products.

• Responsibility for evaluation activities is designated.

Project management

To ensure that the project office and its 

supporting organizations function efficiently 

and effectively.

• Project management activities are planned, organized, controlled, and communicated.

• The performance, cost, and schedule of the acquisition are continually measured, 

compared with planned objectives, and controlled.

• Problems discovered during the acquisition are managed and controlled.

• The acquisition organization has a written policy for project management.

• Responsibility for project management is designated.

Requirements development and 

management

To ensure that contractual requirements are 

clearly defined and understood by the 

acquisition stakeholders.

• Contractual requirements are developed, managed, and maintained.

• The end user and other affected groups have input to the contractual requirements 

over the life of the acquisition.

• Contractual requirements are traceable and verifiable.

• The contractual requirements baseline is established prior to release of the solicitation 

package.

• The acquisition organization has a written policy for establishing and managing the 

contractual requirements.

• Responsibility for requirements development and management is designated.

• Requirements that are mandatory versus optional are clearly delineated and used in 

deciding what requirements can be eliminated or postponed to meet other project 

goals, such as cost and schedule constraints.

Risk management

To ensure that risks are proactively identified 

and systematically mitigated.

• Projectwide participation in the identification and mitigation of risks is encouraged.

• The defined acquisition process provides for the identification, analysis, and mitigation 

of risks.

• Milestone reviews include the status of identified risks.

• The acquisition organization has a written policy for managing acquisition risk.

• Responsibility for acquisition risk management activities is designated.

Solicitation

To ensure that a quality solicitation is produced 

and a best-qualified contractor is selected.

• The solicitation package includes the contractual requirements and the proposal 

evaluation criteria.

• The technical and management elements of proposals are evaluated to ensure that the 

requirements of the contract will be satisfied.

• The selection official selects a supplier who is qualified to satisfy the contract’s 

requirements.

• The acquiring organization has a written policy for conducting the solicitation.

• Responsibility for the solicitation is designated.

• A selection official has been designated to be responsible for the selection process and 

decision.

• The acquiring team includes contracting specialists to support contract administration.

(Continued From Previous Page)
Page 13 GAO-04-722 Defense Acquisition Best Practices

Sources: See sources listed in appendix I of this report.

Best practices Activity

Transition to support

To ensure proper transfer of the system from 

the acquiring organization to the support 

organization.

• The acquiring organization ensures that the support organization has the capacity and 

capability to provide the required support.

• There is no loss in continuity of support to the products during transition from the 

supplier to the support organization.

• Configuration management of the products is maintained throughout the transition.

• The acquiring organization has a written policy for transitioning the products to the 

support organization.

• The acquiring organization ensures that the support organization is involved in 

planning for transition to support.

• Responsibility for transition to support activities is designated.

Complementary best practices relevant to commercial component-based business systems acquisitions

Component modification

To ensure that commercial product 

modification is effectively controlled.

• Modification of commercial components is discouraged and allowed only if justified by 

a thorough analysis of life-cycle costs and benefits.

Configuration management 

To ensure the integrity and consistency of 

system commercial components.

• Project plans explicitly provide for evaluation, acquisition, and implementation of new, 

often frequent, product releases.

• Modification or upgrades to deployed versions of system components are centrally 

controlled and unilateral user release changes are precluded.

Dependency analysis

To ensure that relationships between 

commercial products are understood before 

acquisition decisions are made.

• Decisions about acquisition of commercial components are based on deliberate and 

thorough research, analysis, and evaluation of the components’ interdependencies.

Legacy systems integration planning

To ensure reasonable planning for integration 

of commercial products with existing systems.

• Project plans explicitly provide for the necessary time and resources for integrating 

commercial components with legacy systems.

Organization change management

To ensure that the organizational impact of 

using new system functionality is proactively 

managed.

• Project plans explicitly provide for preparing users on the impact that the business 

processes embedded in the commercial components will have on the users’ respective 

roles and responsibilities.

• The introduction and adoption of changes to how users will be expected to execute 

their jobs are actively managed.

Solicitation

To ensure that a quality solicitation is produced 

and a best-qualified contractor is selected.

• Systems integration contractors are explicitly evaluated on their ability to implement 

commercial components.

Tradeoff analysis

To ensure that system requirements alone do 

not drive the system’s solution.

• Investment decisions throughout a system’s life cycle are based on tradeoffs among 

the availability of commercial products (current and future), the architectural 

environment in which the system is to operate (current and future), defined system 

requirements, and acquisition cost/schedule constraints.

Vendor and product research and 

evaluation

To ensure that vendor and product 

characteristics are understood before 

acquisition decisions are made.

• Commercial component and vendor options are researched, evaluated/tested, and 

understood, both early and continuously.

• A set of evaluation criteria for selecting among commercial component options is 

established that includes both defined system requirements and vendor/commercial 

product characteristics (e.g., customer satisfaction with company and product line).

(Continued From Previous Page)
Page 14 GAO-04-722 Defense Acquisition Best Practices

DOD’s acquisition policies and guidance largely incorporate the 10 best 

practices that are relevant to any business systems acquisition.8

 More 

specifically, they fully incorporate 7 of the 10 practices and partially 

incorporate the other 3 practices. However, they generally do not 

incorporate the 8 best practices that relate to acquiring commercial 

component-based business systems. In particular, they fully incorporate 1 

best practice, partially incorporate 2, and do not incorporate the remaining 

5. (See fig. 3 for a summary of our analysis.) At our request, DOD officials 

responsible for the 5000 series also assessed it against those 18 practices, 

and we incorporated information from their assessment into ours. These 

officials also told us that the acquisition guidebook is currently being 

expanded to incorporate additional best practices, but they did not provide 

us with a plan for accomplishing this. Until this is accomplished, DOD is 

increasing the risk that important and beneficial best practices will not be 

followed and that DOD business systems investments will not deliver 

promised capabilities and benefits on time and within budget.

8

We defined “fully incorporate” to mean the 5000 series addressed all of the practice’s 

activities; “partially incorporate” to mean it addressed some, but not all, of the activities; and 

“not incorporate” to mean it did not address any of the activities. 
Page 15 GAO-04-722 Defense Acquisition Best Practices

Figure 3: DOD Acquisition Policies and Guidance Incorporation of Best Practices

Acquisition planning

Architectural alignment

Contract tracking and oversight

Evaluation

Economic justification

Project management

Requirements development and management

Risk management

Solicitation

Transition to support

Component modification

Configuration management

Dependency analysis

Legacy systems integration planning

Organization change management

Solicitation

Tradeoff analysis

Vendor and product research and evaluation

Best practices relevant to any IT business systems acquisition

Best practices relevant to commercial component-based IT business systems acquisitions

Best practice category and name Best practice incorporated?

Fully incorporated

Partially incorporated

Not incorporated

Source: GAO, based on analysis of DOD data.
Page 16 GAO-04-722 Defense Acquisition Best Practices

The 5000 Series Largely 

Incorporates Best Practices 

Relevant to Any Business 

Systems Acquisition

Of the 10 best practices that we categorized as relevant to the acquisition of 

any business system, whether custom-developed or developed using 

commercial packages and products, essentially all have been incorporated 

into DOD’s acquisition policies and guidance. (See table 3 for our detailed 

comparative analysis of the 5000 series against the 10 best practices.) For 

example, those practices aimed at ensuring that the acquisition is well 

planned, that the system is adequately tested and evaluated against 

contractual requirements, and that the requirements are clearly defined 

and understood by all stakeholders are all addressed in the 5000 series. 

Similarly, for the 3 practices that are not fully addressed, this is the case 

because one activity associated with the practice is not addressed. 

According to DOD officials responsible for revising the 5000 series, the 

policies contain those best practices mandated by either law or DOD 

regulation, and other, optional best practices are contained in the interim 

guidance represented by the former DOD 5000.2-R.

Nevertheless, the activities that are missing from the 3 practices are 

important, and their absence increases the risk that the activities, and thus 

the practice, will not be adequately performed. In turn, this increases the 

risk that acquisition projects will fall short of expectations. The best 

practice aimed at ensuring that risks are proactively identified and 

systematically mitigated is a case in point. This practice has five activities 

associated with it, one of which the 5000 series does not address—project 

reviews include the status of identified risks. As with all the activities under 

this practice, this activity plays an important role in ensuring that the 

appropriate level of attention and visibility is regularly given to risk 

identification and mitigation to ensure that it is effectively executed. 

Conversely, if the activities are not provided for in policy and guidance, it is 

unlikely that they will be performed, and it is likely that acquisition risks 

will become cost, schedule, and performance problems. 
Page 17 GAO-04-722 Defense Acquisition Best Practices

Table 3: Activity-by-activity Comparison of the 5000 Series to Best Practices Relevant to Any IT Business Systems Acquisition

Best practice/Activity

5000 series 

addresses this 

activity?

5000 series 

incorporates this 

best practice?

Acquisition planning Fully

Plans are prepared during acquisition planning and maintained throughout the 

acquisition.

Yes

Planning addresses the entire acquisition process, as well as life cycle support of the 

products being acquired.

Yes

The acquisition organization has a written policy for planning the acquisition. Yes

Responsibility for acquisition planning activities is designated. Yes

Architectural alignment Partially

The system being acquired is assessed for alignment with the enterprise architecture at 

key life cycle decision points, and any deviations from the architecture are understood and 

justified by an explicit waiver to the architecture.

Yes

Product line requirements—rather than just the requirements for the system being 

acquired—are an explicit consideration in each acquisition.

No

Contract tracking and oversight Fully

The acquiring organization has sufficient insight into the contractor’s activities to manage 

and control the contractor and ensure that contract requirements are met.

Yes

The acquiring organization and contractor maintain ongoing communication; 

commitments are agreed to and implemented by both parties.

Yes

All contract changes are managed throughout the life of the contract. Yes

The acquisition organization has a written policy for contract tracking and oversight. Yes

Responsibility for contract tracking and oversight activities is designated. Yes

The acquiring organization involves contracting specialists in the execution of the contract. Yes

A quantitative set of software and system metrics are used to define and measure product 

quality and contractor performance. 

Yes

In addition to incentives for meeting cost and schedule estimates, measurable, 

metrics-based product quality incentives are explicitly cited in the contract. 

Yes

Economic justification Fully

System investment decisions are made on the basis of reliable analyses of estimated 

costs, expected benefits, and anticipated risks.

Yes

Large system projects are (to the maximum extent practical) divided into a series of 

smaller, incremental acquisition efforts, and investment decisions on these smaller efforts 

are made on the basis of reliable analyses of estimated costs, expected benefits, and 

anticipated risks.

Yes
Page 18 GAO-04-722 Defense Acquisition Best Practices

Best practice/Activity

5000 series 

addresses this 

activity?

5000 series 

incorporates this 

best practice?

Evaluation Fully

Evaluation requirements are developed in conjunction with the contractual requirements 

and are maintained over the life of the acquisition.

Yes

Evaluations are planned and conducted throughout the total acquisition period to provide 

an integrated approach that satisfies evaluation requirements and takes advantage of all 

evaluation results.

Yes

Evaluations provide an objective basis to support the product acceptance decision. Yes

The acquiring organization has a written policy for managing the evaluation of the 

acquired products.

Yes

Responsibility for evaluation activities is designated. Yes

Project management Partially

Project management activities are planned, organized, controlled, and communicated. Partly—

communication 

not cited. 

The performance, cost, and schedule of the acquisition are continually measured, 

compared with planned objectives, and controlled.

Yes

Problems discovered during the acquisition are managed and controlled. Yes

The acquisition organization has a written policy for project management. Yes

Responsibility for project management is designated. Yes

Requirements development and management Fully

Contractual requirements are developed, managed, and maintained. Yes

The end user and other affected groups have input into the contractual requirements over 

the life of the acquisition.

Yes

Contractual requirements are traceable and verifiable. Yes

The contractual requirements baseline is established prior to release of the solicitation 

package.

Yes

The acquisition organization has a written policy for establishing and managing the 

contractual requirements.

Yes

Responsibility for requirements development and management is designated. Yes

Requirements that are mandatory versus optional are clearly delineated and used in 

deciding what requirements can be eliminated or postponed to meet other project goals, 

such as cost and schedule constraints.

Yes

Risk management Partially

Projectwide participation in the identification and mitigation of risks is encouraged. Yes

The defined acquisition process provides for the identification, analysis, and mitigation of 

risks.

Yes

Milestone reviews include the status of identified risks. No

(Continued From Previous Page)
Page 19 GAO-04-722 Defense Acquisition Best Practices

Source: GAO, based on analysis of DOD data.

The 5000 Series Generally 

Does Not Incorporate Best 

Practices Relevant to 

Commercial 

Component-Based Business 

Systems Acquisitions

Of the 8 best practices relevant to acquiring commercial component-based 

business systems, few have been incorporated into DOD systems 

acquisition policies and guidance. (See table 4 for our detailed comparative 

analysis of the 5000 series against the 8 best practices.) For example, while 

the practice aimed at ensuring that adequate planning takes place for 

integrating commercial products with legacy systems is incorporated into 

the 5000 series, practices associated with closely controlling any 

modification to the software of these packages and products, thoroughly 

analyzing and understanding the dependencies among commercial 

Best practice/Activity

5000 series 

addresses this 

activity?

5000 series 

incorporates this 

best practice?

The acquisition organization has a written policy for managing acquisition risk. Yes

Responsibility for acquisition risk management activities is designated. Yes

Solicitation Fully

The solicitation package includes the contractual requirements and the proposal 

evaluation criteria.

Yes

The technical and management elements of proposals are evaluated to ensure that the 

requirements of the contract will be satisfied.

Yes

The selection official selects a supplier who is qualified to satisfy the contract’s 

requirements.

Yes

The acquiring organization has a written policy for conducting the solicitation Yes

Responsibility for the solicitation is designated. Yes

A selection official has been designated to be responsible for the selection process and 

decision.

Yes

The acquiring team includes contracting specialists to support contract administration. Yes

Transition to support Fully

The acquiring organization ensures that the support organization has the capacity and 

capability to provide the required support.

Yes

There is no loss in continuity of support to the products during transition from the supplier 

to the support organization.

Yes

Configuration management of the products is maintained throughout the transition. Yes

The acquiring organization has a written policy for transitioning the products to the 

support organization.

Yes

The acquiring organization ensures that the support organization is involved in planning 

for transition to support.

Yes

Responsibility for transition to support activities is designated. Yes

(Continued From Previous Page)
Page 20 GAO-04-722 Defense Acquisition Best Practices

products before acquiring them, and proactively managing the institutional 

change that results from implementing the functionality in commercial 

packages and products are not incorporated. According to DOD officials 

responsible for revising the 5000 series, these practices were not included 

in the recently revised version of DOD’s acquisition policies because they 

included only those in existing law or regulation.

Nevertheless, the absence of these practices from the 5000 series increases 

the risk that the practices will not be performed, which, in turn, increases 

the risk that acquisition projects will fall short of expectations. The 

practice intended to ensure development of a quality solicitation and 

selection of a best-qualified contractor illustrates this. Specifically, this 

practice calls for contract bidders to be evaluated on their ability to 

implement commercial components. This evaluation is important because 

integrating and implementing these component products is sufficiently 

different from developing customized system solutions; it requires different 

core competencies and experiences to be successful. By explicitly taking 

this into consideration in evaluating and selecting a contractor, the risk of 

contract award to a less-than-best-qualified contractor is reduced.

Table 4: Activity-by-Activity Comparison of the 5000 Series to Best Practices Relevant to Commercial Component-based 

Business Systems Acquisitions

Best practice/Activity

Does the 5000 

series address this 

activity?

Does the 5000 series 

incorporate this best 

practice?

Component modification No 

Modification of commercial components is discouraged and allowed only if justified by a 

thorough analysis of life-cycle costs and benefits.

No

Configuration management Partially

Project plans provide for evaluation, acquisition, and implementation of new, often 

frequent, product releases.

Yes

Modification or upgrades to deployed versions of system components are centrally 

controlled and unilateral user release changes are precluded.

No

Legacy systems integration planning Fully

Project plans explicitly provide for the necessary time and resources for integrating 

commercial components with legacy systems.

Yes

Dependency analysis No 

Decisions about acquisition of commercial components are based on deliberate and 

thorough research, analysis, and evaluation of the components’ interdependencies.

No
Page 21 GAO-04-722 Defense Acquisition Best Practices

Source: GAO, based on analysis of DOD data.

DOD Officials Report That 

They Are in the Process of 

Revising the Interim 

Defense Acquisition 

Guidebook

The DOD officials responsible for revising the 5000 series told us they 

recognize the need for the 5000 series to incorporate additional best 

practices. To this end, they reported that efforts are under way to expand 

the Interim Defense Acquisition Guidebook to include additional best 

practices and lessons learned across the department. However, the officials 

could not provide us with a documented plan and associated 

documentation showing how this task will be accomplished, what 

resources are needed and assigned to accomplish it, when it will be 

accomplished, and where the department stands in accomplishing it. 

Instead, the officials told us that progress on it has been slowed by other 

priorities, such as the need to first revise DOD Directive 5000.1 and DOD 

Instruction 5000.2. They also said that there is only a small number of 

staff available to work on what they described as being an extensive 

revision of the guidebook. According to these officials, 80 to 90 percent of 

the revision has been completed and reviewed and their goal is to publish 

the initial version of the revised guidebook by September 30, 2004. In our 

Best practice/Activity

Does the 5000 

series address this 

activity?

Does the 5000 series 

incorporate this best 

practice?

Organization change management No 

Project plans provide for preparing users for the impact that the business processes 

embedded in the commercial components will have on the users’ respective roles and 

responsibilities.

No

The introduction and adoption of changes to how users will be expected to use the 

system to execute their jobs are actively managed.

No

Solicitation No 

Systems integration contractors are explicitly evaluated on their ability to implement 

commercial components.

No

Tradeoff analysis No 

Investment decisions throughout a system’s life cycle are based on tradeoffs among the 

availability of commercial products (current and future), the architectural environment in 

which the system is to operate (current and future), defined system requirements, and 

acquisition cost/schedule constraints.

No

Vendor and product research and evaluation Partially

Commercial component and vendor options are researched, evaluated/tested, and 

understood, both early and continuously.

Yes

A set of evaluation criteria for selecting among commercial component options is 

established that includes both defined system requirements and vendor/commercial 

product characteristics (e.g., customer satisfaction with company and product line).

Partly—vendor/ 

commercial product 

characteristics not 

cited. 

(Continued From Previous Page)
Page 22 GAO-04-722 Defense Acquisition Best Practices

view, until the missing best practices that we cite in this report are included 

in DOD’s acquisition policies and guidance, the chance that business 

systems acquisitions will follow the policies and guidance and consistently 

produce a successful outcome is diminished.

DOD’s Acquisition 

Policies Do Not 

Contain Sufficient 

Controls to Ensure 

That the Requirement 

Is Met for 

Appropriately Applying 

Best Practices 

Federal laws and regulations define the need for effective controls over 

agency programs, and controls are a key factor in achieving program 

results, minimizing operational problems, and managing evolving demands 

and priorities.9

 Controls over defined processes, procedures, and support 

activities are considered effective if they entail measuring and verifying 

whether a given practice is followed. Without sufficient controls, it is 

unlikely that practices will be consistently employed, which, in turn, 

increases the probability that the positive program and project outcomes 

these practices are designed to produce will not occur.

DOD’s acquisition policy requires program managers and investment 

decision authorities to examine and, as appropriate, adopt best practices. 

However, neither the policies nor the accompanying guidance explain what 

“examine” means, including whether practice use is to be measured and 

verified. Instead, the policies state that any issues regarding the intent of 

the 5000 series, which would include whether practice adoption is to be 

measured and validated, shall be resolved by the investment decision 

authority, meaning that it is entirely up to this individual what information 

relative to the use of best practices is relevant and necessary to ensure that 

best practices are appropriately followed. According to the Chairman of 

the Defense Acquisition Policy Working Group,10 this control is sufficient, 

and explicit requirements for measuring and validating the use of best 

practices are not necessary.

In our view, not requiring that decision authorities examine the 

measurement and validation of best practices’ use increases the chance 

that important best practices will not be appropriately followed, as

9

The Federal Managers’ Financial Integrity Act of 1982 (Pub.L. No. 97-255); Government 

Performance and Results Act of 1993 (Pub.L. No. 103-62); Office of Management and Budget 

Circular A-123, June 21, 1995. 

10The Defense Acquisition Policy Working Group is the standing DOD acquisition policy 

working group that revised the 5000 series. 
Page 23 GAO-04-722 Defense Acquisition Best Practices

required by DOD policy. As we have previously reported,11 a lack of explicit 

controls that require review of relevant information at key decision points 

raises the risk of making uninformed project decisions, as well as the risk 

that investments will not meet cost, schedule, capability, and benefit 

commitments.

Conclusions DOD recognizes the importance of business systems acquisition best 

practices by including best practices in their revised policy and guidance. 

However, other practices that, if followed, could increase the odds of 

acquisitions delivering promised system capabilities and benefits on time 

and within budget have yet to be similarly included. In particular, those 

practices associated with the successful acquisition of commercial 

component-based business systems have not been sufficiently 

incorporated into either the policies or the guidance. Moreover, effective 

controls for ensuring that best practices are appropriately followed are not 

adequately provided for in the policies. Although DOD officials intend to 

expand the coverage of best practices in future versions of DOD’s 

acquisition guidance, it is unclear what the scope, nature, and status of 

these intentions are because explicit plans for revising the guidance and 

associated progress reports were not available. Until DOD incorporates the 

best practices we found missing in the 5000 series, and until it strengthens 

the means by which the appropriate use of these practices will be ensured, 

its business systems acquisitions will be exposed to unnecessary risk. 

Therefore, it is important for DOD to treat further revisions of its 

acquisition policy and guidance relative to business systems as a priority 

and move quickly to incorporate missing best practices and associated 

controls for ensuring that the practices are followed.

Recommendations for 

Executive Action

To improve DOD’s ability to acquire business systems, we recommend that 

the Secretary of Defense direct the Under Secretary for AT&L, in 

collaboration with the Assistant Secretary for NII and the Director, OT&E, 

to take the following actions:

1. Develop and implement an explicit plan for incorporating into the 5000 

series the best practices and associated activities currently missing 

11U.S. General Accounting Office, Defense Acquisitions: DOD's Revised Policy Emphasizes 

Best Practices, but More Controls Are Needed, GAO-04-53 (Washington, D.C.: Nov. 10, 2003).
Page 24 GAO-04-722 Defense Acquisition Best Practices

from the series. We recommend that the plan specify tasks to be 

performed, resources needed and assigned, and milestones for 

completing tasks. 

2. We further recommend that progress against this plan be tracked and 

reported as appropriate, and that the plan, at a minimum, incorporate 

each of the following best practice activities:

• Product line requirements—rather than just the requirements for the 

system being acquired—are an explicit consideration in each 

acquisition.

• Acquisition project management activities are communicated to all 

stakeholders.

• Acquisition reviews include the status of identified risks.

• Modification of commercial components is discouraged and allowed 

only if justified by a thorough analysis of life-cycle costs and benefits.

• Modification or upgrades to deployed versions of system components 

are centrally controlled, and unilateral user release changes are 

precluded.

• Acquisition decisions about commercial components are based on 

deliberate and thorough research, analysis, and evaluation of the 

components’ interdependencies.

• Acquisition plans provide for preparing users for the impact that the 

business processes embedded in the commercial components will have 

on their respective roles and responsibilities.

• Changes affecting how users will be expected to use the system to 

execute their jobs are actively managed.

• Systems integration contractors are explicitly evaluated on their ability 

to implement commercial components.

• Investment decisions throughout a system’s life cycle are based on a 

continuous set of tradeoffs among capabilities available in commercial 

components (current and future), the architectural environment in 
Page 25 GAO-04-722 Defense Acquisition Best Practices

which the system is to operate, defined system requirements, and 

existing cost/schedule constraints.

• Evaluation criteria are established for selecting among commercial 

component options that include both defined system requirements and 

vendor/commercial product characteristics. 

3. To ensure that the best practices provided for in DOD acquisition 

policies and guidance are appropriately followed, we also recommend 

that the above recommended plan incorporate steps to include in 

DOD’s acquisition policies a provision for measurement and 

verification of best practices. 

Agency Comments and 

Our Evaluation

In written comments on a draft of this report, signed by the Principal 

Director for Command, Control, Communications, Space, and Information 

Technology Programs in the office of the DOD Assistant Secretary for 

Networks and Information Integration, DOD agreed with the importance 

and relevance of the best practices that we cite in the report. Additionally, 

DOD agreed with 2 of our 13 recommendations for incorporating additional 

best practices, stating that the department would incorporate the 2 

practices in its policies and guidance.

DOD also partially agreed with 9 of our recommendations for incorporating 

additional practices, stating that it would consider augmenting its coverage 

of 5 of the practices and that it believed that 4 practices already existed in 

its policies and guidance. With regard to the 5 practices, DOD stated that it 

needed to review each practice further and determine the need for its 

emphasis or endorsement in the 5000 series. We understand DOD’s desire 

to carefully consider changes to its acquisition policies and guidance, and 

believe that such careful deliberation is consistent with the spirit of our 

recommendations.

With regard to the remaining 4 practices that DOD partially agreed with, we 

do not agree with the department’s comment that these best practices 

adequately exist in the 5000 series. For example, DOD commented that 

because its existing policies and guidance provide for the use of integrated 

product teams, which, according to DOD, are a means for promoting 

collaboration and facilitating communication among stakeholders, its 

policies and guidance therefore already provide for communicating 

information about management of a given acquisition project to all relevant 

project stakeholders. While we do not question the use of integrated 
Page 26 GAO-04-722 Defense Acquisition Best Practices

product teams as a way to communicate information, the point of our 

recommendation is that there needs to be an explicit recognition in policy 

or guidance of the type of information to be communicated and with whom 

it is to be communicated. Restated, our recommendation for incorporating 

the best practice of communicating acquisition management activities to all 

stakeholders is intended to permit communication vehicles, such as 

integrated product teams, to be more effective by explicitly providing for 

this best practice in relevant policies and guidance. As another example, 

we do not agree with DOD’s comment that 2 of the best practices that we 

recommended for incorporation in its policies and guidance—preparing 

users for the impact that business process changes embedded in 

commercial components will have on their roles and responsibilities, and 

actively managing changes in how users will use new systems—are already 

sufficiently contained in the 5000 series. In particular, while we agree that 

the series references an acquisition management toolkit that addresses 

these 2 best practices, this reference is provided only once in the 5000 

series, and this reference is only in relation to one phase of the acquisition 

cycle (the technology development phase). Given the importance and 

relevance of these practices to successful implementation of commercial 

component-based systems, our position, and thus the basis for our 

recommendations, is that the practices’ implementation would be more 

likely to occur if the practices were visible and better recognized in all 

relevant stages of DOD’s acquisition cycle. 

Also in its comments, DOD did not agree with our recommendations to 

develop and implement an explicit plan to govern its ongoing and future 

policy and guidance revision activities, specifically stating that the 

recommendation was inappropriate and offering updated information on 

the status of and associated milestone for completing its activities. While 

we have updated our report to include the revised status and milestone 

information, we do not agree with DOD that a plan governing these efforts 

is not needed. Given the importance of DOD's acquisition policies and 

guidance, and the need for their continuous review and update to reflect 

new acquisition best practices, we believe that having an explicit plan that 

defines how and when these policies and guidance will be incorporated is 

essential. Among other things, a plan would highlight the resource 

constraints that this revision effort has been subject to, would allow 

measurement against defined milestones, and would allow disclosure of 

progress and impediments.

DOD also did not agree with our recommendation to add stronger controls 

for ensuring adherence to the best practices that are contained in its 
Page 27 GAO-04-722 Defense Acquisition Best Practices

acquisition policies and guidance, stating that its existing oversight process 

includes the necessary compliance activities. We disagree. As we state in 

the report, DOD’s existing policy leaves these compliance activities to the 

discretion of the program manager and the investment decision authority, 

and it does not provide for measurement and verification of the use of best 

practices, both of which are recognized components of effective control 

processes.

A copy of DOD’s comments is reprinted in appendix III, along with our 

response.

We are sending copies of this report to the Chairmen and Ranking Minority 

Members of the Senate and House Committees on Armed Services; 

Subcommittees on Defense, Senate and House Committees on 

Appropriations; and the Subcommittee on Military Readiness, House 

Committee on Armed Services. We are also sending copies to the Director, 

Office of Management and Budget; the Secretary of Defense; the Under 

Secretary of Defense (AT&L); the Assistant Secretary of Defense 

(NII)/Chief Information Officer; and the Director, OT&E. We will make 

copies available to others on request. This report will also be available at 

no charge on our Web site at http://www.gao.gov.

If you or your staff has any questions concerning this report, please contact 

me at (202) 512-3439. I can also be reached by e-mail at hiter@gao.gov.

Other contacts and key contributors to this report are listed in appendix IV.

Randolph C. Hite

Director, Information Technology Architecture

 and Systems Issues 
Page 28 GAO-04-722 Defense Acquisition Best Practices

Appendix I

Objectives, Scope, and Methodology ApApenednxidexiIs

Our objectives were to determine whether the Department of Defense’s 

(DOD) revised systems acquisition policies for acquiring information 

technology (IT) business systems (1) are consistent with industry best 

practices, including those pertaining to commercial component-based 

systems, and (2) provide the necessary controls to ensure that the 

department’s component organizations adhere to the practices. 

To accomplish the first objective, we identified the DOD policies and 

guidance relevant to business systems. These policies and guidance are 

contained in three documents—DOD Directive 5000.1, DOD Instruction 

5000.2, and the Interim Defense Acquisition Guidebook—and are 

generally referred to as the 5000 series. We then reviewed each of these 

documents and discussed with DOD officials responsible for developing 

and revising the documents what steps were taken to incorporate best 

practices into each document. The DOD officials that we interviewed were 

from the offices of the Under Secretary of Defense (Acquisition, 

Technology, and Logistics (AT&L)); the Assistant Secretary of Defense 

(Networks and Information Integration (NII)); and the Director, 

Operational Test and Evaluation (OT&E). Next, we researched prior GAO 

reports; the work of federally funded research and development 

organizations, such as the Software Engineering Institute and The 

Aerospace Corporation;1

 and other authoritative sources to identify 

business systems acquisition best practices. Our research produced 18 best 

practices, including associated activities, that we placed into two 

categories—one category for the practices that are relevant to any business 

systems acquisition and one category for the practices that are relevant to 

commercial component-based business systems acquisitions. In particular, 

we drew extensively from the Software Engineering Institute’s Software 

Acquisition Capability Maturity Model.2

 In doing so, we selected practices 

from the model’s repeatable level of process maturity, which is level two on 

the model’s five-level scale. We used the repeatable level of process 

maturity because it is intended to provide the necessary process discipline 

1

For example, the Software Engineering Institute is a federally funded research and 

development center operated by Carnegie Mellon University and sponsored by DOD. The 

Software Engineering Institute’s objective is to provide leadership in software engineering 

and in the transition of new software engineering technology into practice. The Aerospace 

Corporation is a private, nonprofit organization that operates a federally funded research 

and development center for DOD that focuses on the government's need to develop spacerelated hardware and software. 

2

Software Engineering Institute, Software Acquisition Capability Maturity Model® 

version 1.03, CMU/SEI-2002-TR-010 (Pittsburgh, PA: March 2002).
Appendix I

Objectives, Scope, and Methodology

Page 29 GAO-04-722 Defense Acquisition Best Practices

to allow an organization to repeat earlier successes on similar projects. In 

addition, we included one Software Acquisition Capability Maturity Model 

level-three process area—risk management—because many experts 

consider it to be one of the most important process areas. We did not 

attempt to develop an exhaustive list of best practices and, in fact, fully 

recognize that additional best practices exist, such as ensuring that the 

appropriate level of human capital knowledge, skills, and abilities are 

employed, as well as that additional activities for the practices that we have 

identified exist, such as those configuration management activities 

associated with identifying, controlling, reporting on, and auditing 

configuration items and components. For the purposes of this report, we 

identified those practices that are embodied, recognized, and accepted 

acquisition models or frameworks, as well as those practices that are now 

being recognized as being unique to commercial component-based systems 

and for which there appears to be general agreement, including agreement 

with DOD officials responsible for revising the 5000 series, that the 

practices are relevant and important. Last, we analyzed each of the DOD 

5000 series documents to determine whether the documents addressed, 

either directly or indirectly by reference to another authoritative 

document, the 18 best practices that we identified. Based on this analysis, 

we judged whether the 5000 series documents fully, partially, or did not 

incorporate each best practice. In making these judgments, we used the 

following criteria:

• To fully incorporate the practice, the 5000 series addressed all of the 

practice’s activities.

• To partially incorporate the practice, the 5000 series addressed some, 

but not all, of the practice’s activities.

• To not incorporate the practice, the 5000 series did not address any of 

the practice’s activities. 

Additionally, we provided the DOD officials responsible for revising the 

5000 series with the 18 practices that we identified to obtain their views on 

whether the practices were relevant to DOD business systems acquisitions. 

The officials agreed that they were. We also requested that these officials 

perform their own assessment of the 5000 series against these practices, 

and we used these officials’ assessment in making our judgments as to 

whether the practices were fully, partially, or not incorporated into DOD’s 

acquisition policies and guidance. For a number of the activities, DOD 

identified the Federal Acquisition Regulation and the Defense Federal 
Appendix I

Objectives, Scope, and Methodology

Page 30 GAO-04-722 Defense Acquisition Best Practices

Acquisition Regulation Supplement as evidence that the activity was 

being performed within a particular practice. We accepted that as proof 

that the activity was being covered within DOD’s business systems 

acquisition policy. 

To address our second objective, we researched federal internal control 

standards and controls inherent in the business systems acquisition best 

practices that we identified. In particular, we reviewed the Software 

Engineering Institute’s Software Acquisition Capability Maturity Model 

framework and GAO’s internal control standards.3

 We then analyzed DOD’s 

revised acquisition policies and guidance to identify whether these controls 

were cited and to provide assurance that relevant best practices were being 

followed. We also interviewed DOD officials responsible for revising the 

5000 series to determine reasons why controls were addressed or not 

addressed in the policies and guidance.

We conducted our work at DOD offices in Arlington, Virginia, between 

December 2003 and May 2004 in accordance with generally accepted 

government auditing standards

3

U.S. General Accounting Office, Standards for Internal Control in the Federal 

Government, AIMD-00-21.3.1 (Washington, D.C.: November 1999).
Page 31 GAO-04-722 Defense Acquisition Best Practices

Appendix II

Best Practices ApendxiI

Additional information on each of the 18 best practices that we identified is 

provided in this appendix. 

Best Practices 

Relevant to Any IT 

Business Systems 

Acquisition

1. Acquisition Planning

Purpose: To ensure that reasonable planning for all parts of the acquisition 

is conducted.

Description: Acquisition planning is the process for conducting and 

documenting acquisition planning activities beginning early and covering 

all parts of the project. It extends to all acquisition areas, such as 

budgeting, scheduling, resource estimating, risk identification, and 

requirements definition, as well as the overall acquisition strategy. 

Acquisition planning begins with the earliest identification of a requirement 

that is to be satisfied through an acquisition.

Activities: (1) Plans are prepared during acquisition planning and 

maintained throughout the acquisition. (2) Planning addresses the entire 

acquisition process, including life cycle support of the products being 

acquired. (3) The acquisition organization has a written policy for planning 

the acquisition. (4) Responsibility for acquisition planning activities is 

designated.

2. Architectural Alignment

Purpose: To ensure that the acquisition is consistent with the organization’s 

enterprise architecture. 

Description: Architectural alignment is the process for analyzing and 

verifying that the proposed architecture of the system being acquired is 

consistent with the enterprise architecture for the organization acquiring 

the system. Such alignment is needed to ensure that acquired systems can 

interoperate and are not unnecessarily duplicative of one another. 

Exceptions to this alignment requirement are permitted, but only when 

justified and only when granted an explicit waiver from the architecture. A 

particular architectural consideration is whether requirements that extend 

beyond the specific system being acquired should be considered when 

selecting system components. Such product line (i.e., systems that are 

developed from a common set of assets and share a common and managed 

set of features) considerations can provide substantial production 

economies over acquiring systems from scratch. 
Appendix II

Best Practices

Page 32 GAO-04-722 Defense Acquisition Best Practices

Activities: (1) The system being acquired is assessed for alignment with 

the enterprise architecture at key life cycle decision points, and any 

deviations from the architecture are explicitly understood and justified by 

an explicit waiver to the architecture. (2) Product line requirements—

rather than just the requirements for the system being acquired—are an 

explicit consideration in each acquisition.

3. Contract Tracking and Oversight

Purpose: To ensure that contract activities are performed in accordance 

with contractual requirements.

Description: Contract tracking and oversight is the process by which 

contractual agreements are established and contractor efforts to satisfy 

those agreements are supervised. It involves information sharing between 

the acquirer and contractor to ensure that contractual requirements are 

understood, that there are regular measurements to disclose overall project 

status and whether problems exist, and that there are appropriate 

incentives for ensuring that cost and schedule commitments are met and 

that quality products are delivered. Contract tracking and oversight begins 

with the award of the contract and ends at the conclusion of the contract’s 

period of performance.

Activities: (1) The acquiring organization has sufficient insight into the 

contractor’s activities to manage and control the contractor and ensure that 

contract requirements are met. (2) The acquiring organization and 

contractor maintain ongoing communication; commitments are agreed to 

and implemented by both parties. (3) All contract changes are managed 

throughout the life of the contract. (4) The acquiring organization has a 

written policy for contract tracking and oversight. (5) Responsibility for 

contract tracking and oversight activities is designated. (6) The acquiring 

organization involves contracting specialists in the execution of the 

contract. (7) A quantitative set of software and system metrics is used to 

define and measure product quality and contractor performance.1

 (8) In

1

Richard J. Adams, Suellen Eslinger, Karen L. Owens, and Mary A. Rich, Reducing Risk in 

the Acquisition of Software-Intensive Systems: Best Practices from the Space System 

Domain (Los Angeles, CA: 2003). 
Appendix II

Best Practices

Page 33 GAO-04-722 Defense Acquisition Best Practices

addition to incentives for meeting cost and schedule estimates, 

measurable, metrics-based product quality incentives are explicitly cited in 

the contract.2

4. Economic Justification

Purpose: To ensure that system investments have an adequate economic 

justification.

Description: Economic justification is the process for ensuring that 

acquisition decisions are based on reliable analyses of the proposed 

investment’s likely costs versus benefits over its useful life, as well as an 

analysis of the risks associated with actually realizing the acquisition’s 

forecasted benefits for its estimated costs. Moreover, it entails minimizing 

the risk and uncertainty of large acquisitions that require spending large 

sums of money over many years by breaking the acquisition into smaller, 

incremental acquisitions. Economic justification is not a one-time event, 

but rather is performed throughout an acquisition’s life cycle in order to 

permit informed investment decision making. 

Activities: (1) System investment decisions are made on the basis of 

reliable analyses of estimated system life cycle costs, expected benefits, 

and anticipated risks. (2) Large systems acquisitions are (to the maximum 

extent practical) divided into a series of smaller, incremental acquisition 

efforts, and investment decisions on these smaller efforts are made on the 

basis of reliable analyses of estimated costs, expected benefits, and 

anticipated risks.

5. Evaluation

Purpose: To ensure that evidence showing that the contract products 

satisfy the defined requirements are provided prior to accepting contractor 

products.

Description: Evaluation is the process by which contract deliverables are 

analyzed to determine whether they meet contract requirements. It 

includes developing criteria such as product acceptance criteria to be 

2

Adams and Eslinger, “COTS-Based Systems: Lessons Learned from Experiences with COTS 

Software Use on Space Systems.” (Paper presented to the Southern California SPIN: Oct. 6, 

2003.)
Appendix II

Best Practices

Page 34 GAO-04-722 Defense Acquisition Best Practices

included into both the solicitation package and the contract. It should be 

conducted continuously throughout the contract period as products are 

delivered. It begins with development of the products’ requirements and 

ends when the acquisition is completed.

Activities: (1) Evaluation requirements are developed in conjunction with 

the contractual requirements and are maintained over the life of the 

acquisition. (2) Evaluations are planned and conducted throughout the 

total acquisition period to provide an integrated approach that satisfies 

evaluation requirements and takes advantage of all evaluation results. (3) 

Evaluations provide an objective basis to support the product acceptance 

decision. (4) The acquisition organization has a written policy for managing 

the evaluation of the acquired products. (5) Responsibility for evaluation 

activities is designated.

6. Project Management

Purpose: To ensure that the project office and its supporting organizations 

function efficiently and effectively.

Description: Project management is the process for planning, organizing, 

staffing, directing, and managing all project-office-related activities, such 

as defining project tasks, estimating and securing resources, scheduling 

activities and tasks, training, and accepting products. Project management 

begins when the project office is formed and ends when the acquisition is 

completed.

Activities: (1) Project management activities are planned, organized, 

controlled, and communicated. (2) The performance, cost, and schedule of 

the acquisition are continually measured, compared with planned 

objectives, and controlled. (3) Problems discovered during the acquisition 

are managed and controlled. (4) The acquisition organization has a written 

policy for project management. (5) Responsibility for project management 

is designated.

7. Requirements Development and Management

Purpose: To ensure that contractual requirements are clearly defined and 

understood by the acquisition stakeholders.

Description: Requirements development is the process for developing and 

documenting contractual requirements, including evaluating opportunities 
Appendix II

Best Practices

Page 35 GAO-04-722 Defense Acquisition Best Practices

for reusing existing assets. It involves participation from end users to 

ensure that product requirements are well understood, and that optional 

versus mandatory requirements are clearly delineated. Requirements 

management is the process for establishing and maintaining agreement on 

the contractual requirements among the various stakeholders and for 

ensuring that the requirements are traceable, verifiable, and controlled. 

This involves baselining the requirements and controlling subsequent 

requirements changes. Requirements development and management begins 

when the solicitation’s requirements are documented and ends when 

system responsibility is transferred to the support organization.

Activities: (1) Contractual requirements are developed, managed, and 

maintained. (2) The end user and other affected groups have input into the 

contractual requirements over the life of the acquisition. (3) Contractual 

requirements are traceable and verifiable. (4) The contractual requirements 

baseline is established prior to release of the solicitation package. (5) The 

acquisition organization has a written policy for establishing and managing 

the contractual requirements. (6) Responsibility for requirements 

development and management is designated. (7) Requirements that are 

mandatory versus optional are clearly delineated and used in deciding what 

requirements can be eliminated or postponed to meet other project goals, 

such as cost and schedule constraints.3

8. Risk Management

Purpose: To ensure that risks are identified and systematically mitigated.

Description: Risk management is the process for identifying potential 

acquisition problems and taking appropriate steps to avoid their becoming 

actual problems. It includes risk identification and categorization based on 

estimated impact, development of risk mitigation strategies, and execution 

of and reporting on the strategies. Risk management occurs early and 

continuously in the acquisition life cycle.

Activities: (1) Projectwide participation in the identification and mitigation 

of risks is encouraged. (2) The defined acquisition process provides for the 

3

Software Engineering Institute, Real-Time Systems Engineering: Lessons Learned from 

Independent Technical Assessments, CMU/SEI-2001-TN-004 (Pittsburgh, PA: June 2001); 

and Adams, Eslinger, Owens, and Rich, Reducing Risk in the Acquisition of Software 

Intensive-Systems: Best Practices from the Space System Domain.
Appendix II

Best Practices

Page 36 GAO-04-722 Defense Acquisition Best Practices

identification, analysis, and mitigation of risks. (3) Milestone reviews 

include the status of identified risks. (4) The acquisition organization has a 

written policy for managing acquisition risk. (5) Responsibility for 

acquisition risk management activities is designated.

9. Solicitation

Purpose: To ensure that a quality solicitation is produced and a bestqualified contractor selected.

Description: Solicitation is the process for developing, documenting, and 

issuing the solicitation package; developing and implementing a plan to 

evaluate responses; conducting contract negotiations; and awarding the 

contract. Solicitation ends with contract award.

Activities: (1) The solicitation package includes the contractual 

requirements and the proposal evaluation criteria. (2) The technical and 

management elements of proposals are evaluated to ensure that the 

requirements of the contract will be satisfied. (3) The selection official 

selects a supplier who is qualified to satisfy the contract’s requirements. (4) 

The acquiring organization has a written policy for conducting the 

solicitation. (5) Responsibility for the solicitation is designated. (6) A 

selection official has been designated to be responsible for the selection 

process and decision. (7) The acquiring team includes contracting 

specialists to support contract administration.

10. Transition to Support

Purpose: To ensure proper transfer of the system from the acquisition 

organization to the eventual support organization. 

Description: Transition to support is the process for developing and 

implementing the plans for transitioning products to the support 

organization. This includes engaging relevant stakeholders in the 

acquisition and sharing information about the system’s supporting 

infrastructure. Transition to support begins with requirements 

development and ends when the responsibility for the products is turned 

over to the support organization.

Activities: (1) The acquiring organization ensures that the support 

organization has the capacity and capability to provide the required 

support. (2) There is no loss in continuity of support to the products during 
Appendix II

Best Practices

Page 37 GAO-04-722 Defense Acquisition Best Practices

transition from the supplier to the support organization. (3) Configuration 

management of the products is maintained throughout the transition. (4) 

The acquiring organization has a written policy for transitioning products 

to the support organization. (5) The acquiring organization ensures that the 

support organization is involved in planning for transition to support. (6) 

Responsibility for transition to support activities is designated.

Complementary Best 

Practices Relevant to 

Commercial 

Component-Based IT 

Business Systems 

Acquisitions 

1. Component Modification

Purpose: To ensure that commercial product modification is effectively 

controlled. 

Description: Component modification is the process for limiting the 

chances of a commercial product being modified to the point that it 

becomes a one-of-a-kind solution because doing so can result in extensive 

life cycle costs. Such modifications, if not incorporated into the 

commercially available version of the product by the supplier, mean that 

every product release has to be modified in accordance with the custom 

changes, thus precluding realization of some of the benefit of using a 

commercial product. 

Activity: (1) Modification of commercial components is discouraged and 

allowed only if justified by a thorough analysis of life cycle costs and 

benefits.4

2. Configuration Management

Purpose: To ensure the integrity and consistency of system commercial 

components.

Description: Configuration management relative to commercial 

component-based systems is the process for ensuring that changes to the 

commercial components of a system are strictly controlled. It recognizes 

that when using commercial components, it is the vendor, not the 

acquisition or support organization, that controls the release of new 

component versions and that new versions are released frequently. Thus, 

4

Adams and Eslinger, “COTS-Based Systems: Lessons Learned from Experiences with COTS 

Software Use on Space Systems.” (Paper presented to the Southern California SPIN: Oct. 6, 

2003.)
Appendix II

Best Practices

Page 38 GAO-04-722 Defense Acquisition Best Practices

acquisition management needs to provide for both receiving new product 

releases and controlling the implementation of these releases. 

Activities: (1) Project plans explicitly provide for evaluation, acquisition, 

and implementation of new, often frequent, product releases.5

 (2) 

Modification or upgrades to deployed versions of system components are 

centrally controlled, and unilateral user release changes are precluded.

3. Dependency Analysis

Purpose: To ensure that relationships between commercial products are 

understood before acquisition decisions are made. 

Description: Dependency analysis relative to commercial componentbased systems is the process for determining and understanding the 

characteristics of these products so that inherent dependencies among 

them can be considered before they are acquired. It involves recognizing 

that the logical and physical relationships among products impact one 

another. This is necessary because commercial products are built around 

each vendor’s functional and architectural assumptions and paradigms, 

such as approaches to error handling and data access, and these 

assumptions and paradigms are likely to be different among products from 

different sources. Such differences complicate product integration. 

Further, some commercial products have built-in dependencies with other 

products that, if not known, can further complicate integration. 

Activity: (1) Decisions about the acquisition of commercial components 

are based on deliberate and thorough research, analysis, and evaluation of 

the components’ interdependencies.6

4. Legacy Systems Integration Planning

Purpose: To ensure reasonable planning for integration of commercial 

products with existing systems. 

5

Donald J. Reifer, Victor R. Basili, Barry W. Boehm, Betsy Clark, “COTS-Based Systems—

Twelve Lessons Learned about Maintenance.” (Presentation, 3rd International Conference 

on COTS-Based Software Systems, Redondo Beach, CA, Feb. 4, 2004.)

6

Tricia Oberndorf, Lisa Brownsword, and Carol A. Sledge, Ph.D., An Activity Framework 

for COTS-Based Systems, Technical Report CMU/SEI-2000-TR-010 (Pittsburgh, Pa.: 

Software Engineering Institute, Carnegie Mellon University, October 2000).
Appendix II

Best Practices

Page 39 GAO-04-722 Defense Acquisition Best Practices

Description: Legacy systems integration planning is the process for 

ensuring that the time and resources needed to integrate existing systems 

with the system being acquired are identified and provided for. It involves 

identifying which legacy systems will interact with the system being 

acquired and what kinds and levels of testing are required. Integration 

planning recognizes that, although some commercial products may provide 

mechanisms and information that is helpful in integration with legacy 

systems, the unavailability of the source code for commercial products and 

the different organizations that are responsible for the two will likely 

require additional time and effort. 

Activity: (1) Project plans explicitly provide for the necessary time and 

resources for integrating commercial components with legacy systems.

5. Organization Change Management

Purpose: To ensure that the organizational impact of using new system 

functionality is proactively managed. 

Description: Organization change management relative to commercial 

component-based systems is the process for preparing system users for the 

business process changes that will accompany implementation of the 

system. It involves engaging users and communicating the nature of 

anticipated changes to system users through training on how jobs will 

change. This is necessary because commercial products are created with 

the developers’ expectations of how they will be used, and the products’ 

functionality may require the organization implementing the system to 

change existing business processes. 

Activities: (1) Project plans explicitly provide for preparing users on the 

impact that the business processes embedded in the commercial 

components will have on the user’s respective roles and responsibilities. 

(2) The introduction and adoption of changes to how users will be 

expected to execute their jobs are actively managed.7

7

Suzanne Garcia, John Roberts, and Len Estrin, “Managed Technology Adoption Risk.” 

(Presentation, 3rd International Conference on COTS-Based Software Systems, Redondo 

Beach, CA, Feb. 2, 2004).
Appendix II

Best Practices

Page 40 GAO-04-722 Defense Acquisition Best Practices

6. Solicitation

Purpose: To ensure that a quality solicitation is produced and a bestqualified contractor is selected.

Description: Solicitation relative to commercial component-based systems 

is the process for ensuring that a capable contractor is selected. It involves 

ensuring that the selected contractor has experience with integrating 

commercial component products. This is important because expertise in 

developing custom system solutions is different from expertise in 

implementing commercial components; it requires different core 

competencies and experiences to be successful.

Activity: (1) Systems integration contractors are explicitly evaluated on 

their ability to implement commercial components.8

7. Tradeoff Analysis 

Purpose: To ensure that system requirements alone do not drive the system 

solution. 

Description: Tradeoff analysis relative to commercial product-based 

systems is the process for analyzing and understanding the tradeoffs 

among competing acquisition variables so as to produce informed 

acquisition decision making. It involves planning and executing 

acquisitions in a manner that recognizes four competing interests: defined 

system requirements, the architectural environment (current and future) in 

which the system needs to operate, acquisition cost and schedule 

constraints, and the availability of products in the commercial marketplace 

(current and future). This analysis should be performed early and 

continuously throughout an acquisition’s life cycle. 

Activity: (1) Investment decisions throughout a system’s life cycle are 

based on tradeoffs among the availability of commercial products (current 

and future), the architectural environment in which the system is to

8

Adams and Eslinger, “COTS-Based Systems: Lessons Learned from Experiences with COTS 

Software Use on Space Systems.” (Paper presented to the Southern California SPIN: Oct. 6, 

2003.)
Appendix II

Best Practices

Page 41 GAO-04-722 Defense Acquisition Best Practices

operate (current and future), defined system requirements, and acquisition 

cost/schedule constraints.9

8. Vendor and Product Research and Evaluation

Purpose: To ensure that vendor and product characteristics are understood 

before acquisition decisions are made. 

Description: Vendor and product research and evaluation relative to 

commercial component-based systems is the process for obtaining reliable 

information about both the product being considered and the vendor 

offering the product. It involves taking additional steps beyond vendor 

representations, such as obtaining information about the vendor’s history, 

obtaining information on the vendor’s business strategy relative to 

evolution and support of the product, and evaluating copies of the product 

in a test environment. 

Activities: (1) Commercial component and vendor options are researched, 

evaluated/tested, and understood, both early and continuously. (2) A set of 

evaluation criteria for selecting among commercial component options is 

established that includes both defined system requirements and 

vendor/commercial product characteristics (e.g., customer satisfaction 

with company and product line).

9

Software Engineering Institute, Evolutionary Process for Integrating COTS-Based Systems 

(EPIC): An Overview, CMU/SEI-2002-TR-009 (Pittsburgh, PA: July 2002).
Page 42 GAO-04-722 Defense Acquisition Best Practices

Appendix III

Comments from the Department of Defense ApendxiI

Note: GAO comments 

supplementing those in 

the report text appear 

at the end of this 

appendix.
Appendix III

Comments from the Department of Defense

Page 43 GAO-04-722 Defense Acquisition Best Practices

See comment 1.

See comment 2.

See comment 3.
Appendix III

Comments from the Department of Defense

Page 44 GAO-04-722 Defense Acquisition Best Practices

See comment 4.

See comment 5.
Appendix III

Comments from the Department of Defense

Page 45 GAO-04-722 Defense Acquisition Best Practices

See comment 6.

See comment 7.

See comment 8.
Appendix III

Comments from the Department of Defense

Page 46 GAO-04-722 Defense Acquisition Best Practices

See comment 9.

See comment 10.

See comment 11.
Appendix III

Comments from the Department of Defense

Page 47 GAO-04-722 Defense Acquisition Best Practices

The following are GAO’s comments on the Department of Defense’s letter 

dated July 1, 2004.

GAO Comments 1. We disagree. While we have updated our report to reflect the additional 

information provided in DOD’s comments on the status of its efforts 

and the associated milestone, the importance of revising and 

maintaining DOD's acquisition policies and guidance, and their 

incorporation of acquisition best practices, makes it essential to have 

an explicit plan. Among other things, a plan would highlight the 

resource constraints that this revision effort has been subject to, would 

allow measurement against defined milestones, and would allow 

disclosure of progress and impediments. 

2. See comment 1.

3. We do not question DOD’s statement that it has an initial repository of 

custom software components that can help adapt commercial 

components to defense use and reuse. However, this repository does 

not satisfy the product line requirements in this best practice and our 

recommendation. According to the Software Engineering Institute, the 

product line requirements best practice involves more than just reuse. 

Under the approach described in DOD’s comments, reuse generally 

involves items, such as software modules or components, that 

developers are encouraged to use. However, the product line 

requirements best practice is not simply encouraging reuse of items in a 

repository. Rather, it is planned, enabled, and enforced reuse of such 

assets as requirements, models, and architectures that have been 

designed and optimized for use in multiple systems. In short, it is 

proactive rather than reactive reuse. 

4. We have modified our recommendation to use the terminology 

“acquisition project management activities” instead of “acquisition 

management activities” to eliminate any confusion. Further, we do not 

question DOD’s use of integrated product teams as a way to 

communicate information. Further, the point of our recommendation is 

that there needs to be an explicit recognition in policy and guidance of 

the type of information to be communicated and to whom. 

Incorporating this best practice is based on the need to ensure that 

communication vehicles, such as integrated product teams, are 

effective. 
Appendix III

Comments from the Department of Defense

Page 48 GAO-04-722 Defense Acquisition Best Practices

5. We agree that the 5000 series contains information on acquisition risk 

management, as we state in our report, and that one of the purposes of 

DOD’s acquisition framework is to reduce risks. However, there is no 

provision in DOD’s acquisition policy or guidelines to ensure that the 

status of identified risks are discussed at key decision points. For 

example, DOD policy states that a criterion for passing milestone A is 

“management and mitigation of technology risk.” However, it does not 

provide for what is to be done to manage and mitigate risks, and it does 

not provide for reviewing risk status at milestones B or C. 

6. While we agree that the toolkit provides relevant change management 

information, the toolkit is referenced only once in the 5000 series; and 

the reference is only in relation to one phase of the acquisition cycle 

(the technology development phase). Given the importance and 

relevance of this practice to the successful implementation of 

commercial component-based systems, our position, and thus the basis 

for our recommendation, is that the best practice’s implementation 

would more likely occur if it was visible and recognized in all relevant 

stages of DOD’s acquisition cycle. 

7. See comment 6.

8. While the regulations and guidance that DOD cited and referenced in its 

acquisition policies discuss the use of information on contractors’ past 

performance, they do not discuss evaluating systems integration 

contractors on their ability to implement commercial components, 

which is the point of the best practice. Further, DOD’s objection to 

incorporating this best practice is not consistent with its own 

comments. Specifically, DOD commented that it has already taken 

steps, through its enterprise systems initiative, to establish blanket 

agreements with five contractors who were evaluated on, among other 

things, “explicitly proven abilities to implement commercial systems 

solutions.” Additionally, while we appreciate DOD’s concern that 

incorporation of this best practice can have unintended consequences, 

it is important to also recognize that our recommendation is not 

intended to restrict a contracting officer’s options. Rather, it is intended 

to be one of the many factors considered in the source selection 

process when it is relevant. 

9. We support the Defense Acquisition Executive’s decision to add more 

specifics on systems engineering to the 5000 series, including 

provisions that address this best practice and recommendation. 
Appendix III

Comments from the Department of Defense

Page 49 GAO-04-722 Defense Acquisition Best Practices

10. See comment 8. 

11. We disagree. While we do not question that statutory and regulatory 

compliance are referenced in DOD’s integrated process team and 

milestone decision point processes, we do not believe that these 

reviews are adequately defined with respect to implementation of best 

practices because DOD’s policy does not require that the practices’ use 

be measured and verified. Rather, it leaves these reviews to the 

discretion of the program manager and investment decision authority. 

As we state in our report, not requiring that the use of best practices be 

measured and verified increases the chance that the practices will not 

be followed. Therefore, our position remains that DOD’s policies do not 

provide effective controls for ensuring that best practices are 

appropriately followed. 
Page 50 GAO-04-722 Defense Acquisition Best Practices

Appendix IV

GAO Contact and Staff Acknowledgments ApendxiIV

GAO contact Carl L. Higginbotham, (404) 679-1824

Staff acknowledgments In addition to the individual named above, key contributors to this report 

included Nabajyoti Barkakati, Nancy Glover, Madhav Panwar, Morgan 

Walts, and Thomas Wright.

(310274)
GAO’s Mission The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go to 

www.gao.gov and select “Subscribe to Updates.”

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to:

U.S. Government Accountability Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Congressional 

Relations

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

Government Accountability Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00