a

GAO

United States General Accounting Office

Report to the Subcommittee on 

Readiness and Management Support, 

Committee on Armed Services,

U.S. Senate

April 2003 INFORMATION 

TECHNOLOGY

DOD Needs to 

Leverage Lessons 

Learned from Its 

Outsourcing Projects

GAO-03-371
The projects in GAO’s review substantially used leading commercial 

practices as specified in GAO’s framework for outsourcing IT services. 

Specifically, the agencies fully implemented 88 percent of the practices (not 

including practices not applicable to a particular project). This framework 

consists of practices organized into seven phases: (I) determine sourcing 

strategy, (II) define operational model, (III) develop the contract, (IV) select 

the provider(s), (V) transition to provider(s), (VI) manage the performance 

of the provider(s), and (VII) ensure services are provided. The figure below 

shows the percentage of practices that were implemented in each phase. 

Although DOD has acted on gathering and disseminating lessons learned and 

commercial leading practices related to general acquisition issues, its 

actions have generally not been focused on outsourcing or on sharing the 

lessons learned from IT services outsourcing across the department. By not 

systematically capturing and disseminating such information across the 

department, DOD is losing the opportunity to leverage the knowledge gained 

on IT services projects like those in GAO’s review. Lessons learned that are 

pragmatic and easily accessible would give DOD managers a more informed 

understanding of important issues to be addressed when making outsourcing 

decisions, as well as the factors to be considered to help ensure the success 

of these endeavors. 

Percentage of Practices Implemented, by Phase 

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but 

not all aspects of the practice and did not take alternative actions that fully satisfied the practice. 

Yes—The agency fully implemented the practice or took an alternative action that fully satisfied its 

intent. 

INFORMATION TECHNOLOGY 

DOD Needs to Leverage Lessons Learned 

from Its Outsourcing Projects 

www.gao.gov/cgi-bin/getrpt?GAO-03-371.

To view the full report, including the scope 

and methodology, click on the link above. 

For more information, contact Randolph C. Hite 

at (202) 512-3439 or hiter@gao.gov. 

Highlights of GAO-03-371, a report to the 

Subcommittee on Readiness and 

Management Support, Committee on 

Armed Services, U.S. Senate 

April 2003 

Given the magnitude of its reported 

spending on information 

technology (IT) services—more 

than $6.2 billion in fiscal year 

2001—it is critical that the 

Department of Defense (DOD) 

adopt effective practices for 

acquiring IT services. 

GAO researched leading 

commercial practices for the 

outsourcing of IT services, and, in 

November 2001, published a 

framework consisting of seven 

phases that span the full range of 

activities that are performed during 

the outsourcing of those services 

(this is an acquisition in which a 

client organization transfers 

responsibility for performing 

services to an external provider). 

GAO was asked to determine 

(1) the extent to which selected 

DOD projects for outsourcing IT 

services use leading commercial 

practices as specified in GAO’s 

framework and (2) whether DOD is 

sharing lessons learned from its IT 

outsourcing projects across the 

department. 

GAO is making recommendations to 

the Secretary of Defense aimed at 

leveraging lessons learned across the 

department from its components’ IT 

outsourcing experiences. 

DOD agreed that capturing lessons 

learned related to IT outsourcing 

initiatives is important and stated that 

it intends to explore a variety of 

mechanisms to do so. DOD’s plans are 

consistent with our recommendations.
Page i GAO-03-371 DOD IT Services Outsourcing

Contents

Letter 1

Results in Brief 2

Background 5

Projects Substantially Used Leading Commercial Practices 14

Leveraging Lessons Learned DOD-wide

Could Assist Other DOD Projects 23

Conclusions 27

Recommendations 27

Agency Comments and Our Evaluation 28

Appendixes

Appendix I: Objectives, Scope, and Methodology 30

Appendix II: Projects’ Implementation of Phase I: Determine Sourcing 

Strategy 32

Appendix III: Project’s Implementation of Phase II: Define Operational 

Model 38

Appendix IV: Project’s Implementation of Phase III: Develop the 

Contract 48

Appendix V: Projects’ Implementation of Phase IV: Select the 

Provider(s) 59

Appendix VI: Projects’ Implementation of Phase V: Transition to 

Provider(s) 66

Appendix VII: Projects’ Implementation of Phase VI: Manage Provider(s) 

Performance 77

Appendix VIII: Projects’ Implementation of Phase VII: Ensure Services are 

Provided 86

Appendix IX: Comments from the Department of Defense 90

GAO Comments 93

Appendix X: GAO Contact and Staff Acknowledgments 94

GAO Contact 94

Staff Acknowledgments 94

Tables Table 1: Definition of Phases for IT Outsourcing 8

Table 2: Profile of Outsourcing Projects Reviewed 13

Table 3: Percentage of Practices Implemented, by Project 17

Table 4: Summary of Projects’ Use of Phase I Practices 34
Contents

Page ii GAO-03-371 DOD IT Services Outsourcing

Table 5: Summary of Projects’ Use of Phase II Practices 41

Table 6: Summary of Projects’ Use of Phase III Practices 51

Table 7: Summary of Projects’ Use of Phase IV Practices 61

Table 8: Summary of Projects’ Use of Phase V Practices 69

Table 9: Summary of Projects’ Use of Phase VI Practices 80

Table 10: Summary of Projects’ Use of Phase VII Practices 88

Figures Figure 1: Percentage of Practices Implemented in Each Phase 3

Figure 2: Roles of the Client and Provider in an Outsourcing 

Relationship 6

Figure 3: GAO’s Framework for Outsourcing IT Services 7

Figure 4: Use of Leading Commercial Practices for Outsourcing of 

IT Services, by Project 15

Figure 5: Percentage of Practices Implemented in Each Phase 16

Figure 6: Project Implementation of Phase I: Determine Sourcing 

Strategy 33

Figure 7: Project Implementation of Phase II: Define Operational 

Model 40

Figure 8: Project Implementation of Phase III: Develop the 

Contract 50

Figure 9: Project Implementation of Phase IV: Select the 

Provider(s) 60

Figure 10: Project Implementation of Phase V: Transition to 

Provider(s) 68

Figure 11: Project Implementation of Phase VI: Managing Provider 

Performance 79

Figure 12: Project Implementation of Phase VII: Ensure Services Are 

Provided 87
Contents

Page iii GAO-03-371 DOD IT Services Outsourcing

Abbreviations

C3I command, control, communications, and intelligence

C4 command, control, communications, and computer 

CIO chief information officer

DOD Department of Defense

IT information technology

IT/IS Information Technology/Information Services

MHS/ITO Military Health System/Information Technology Organization

NETCOM Network Enterprise Technology Command

NIMA National Imagery and Mapping Agency

NMCI Navy and Marine Corps Intranet

OMB Office of Management and Budget

RFP request for proposals

SLA service-level agreement

TAC-SWA Total Army Communications—Southwest Asia

This is a work of the U.S. Government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. It may contain copyrighted graphics, images or other materials. 

Permission from the copyright holder may be necessary should you wish to reproduce 

copyrighted materials separately from GAO’s product.
Page 1 GAO-03-371 DOD IT Services Outsourcing

United States General Accounting Office

Washington, D.C. 20548

A

April 25, 2003 Letr

The Honorable John Ensign

Chairman

The Honorable Daniel K. Akaka

Ranking Minority Member

Subcommittee on Readiness and Management Support

Committee on Armed Services

United States Senate

The Department of Defense (DOD) is the government’s largest purchaser of 

information technology (IT) services, such as desktop support, network 

operations, and software development services. In fiscal year 2001, DOD 

reportedly obligated more than $6.2 billion on IT services,1

 and this amount 

is expected to grow substantially. Given the magnitude of DOD’s spending 

on such services, it is critical that the department adopt effective practices 

for acquiring IT services.

Since 1996, we have conducted a series of studies for the Senate 

Committee on Armed Services concerning how DOD can improve its 

acquisition processes by adopting proven practices of leading commercial 

organizations. In this vein, in November 2001, we issued a guide that 

organized leading commercial practices for the outsourcing2

 of IT services 

into a framework of seven phases that span the full range of activities that 

are performed during IT services outsourcing.3

This report responds to your request that we determine (1) the extent to 

which selected DOD IT services outsourcing projects use leading 

commercial practices as specified in our framework and 

(2) whether DOD is sharing lessons learned from its IT outsourcing 

projects across the department. To address the first objective, we selected 

five projects from a group of projects to outsource IT services that were

1

This figure is from the Federal Procurement Data System, which contains detailed 

information on contract actions over $25,000. 

2

IT services outsourcing is a type of acquisition in which a client organization transfers 

responsibility for performance of one or more IT services to one or more external providers.

3

U.S. General Accounting Office, Information Technology: Leading Commercial Practices 

for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov. 30, 2001). 
Page 2 GAO-03-371 DOD IT Services Outsourcing

identified by the military services and other DOD components.4 We then 

asked the component responsible for each project to perform a selfassessment against selected practices in our framework for outsourcing IT 

services.5

 Next, we obtained and reviewed agencies’ supporting 

documentation and interviewed the appropriate agency and provider 

project officials to independently determine whether a practice was met. 

To address the second objective, we reviewed applicable DOD approaches 

for capturing and disseminating lessons learned from IT services 

outsourcing projects and interviewed the applicable acquisition and IT 

officials. Details of our objectives, scope, and methodology are discussed 

in appendix I.

Results in Brief The projects in our review substantially used leading commercial practices 

as specified in our framework on outsourcing IT services. Specifically, the 

agencies fully implemented 88 percent of the practices.6

 This framework 

consists of practices organized into seven phases that span the full range of 

activities that are performed during IT outsourcing: (I) determine sourcing 

strategy, (II) define operational model, (III) develop the contract, (IV) 

select the provider(s), (V) transition to provider(s), (VI) manage 

provider(s) performance, and (VII) ensure services are provided. Figure 1 

illustrates the percentage of practices that were followed in each phase. 

Collectively, the projects fully implemented from 70 to 97 percent of the 

practices in each phase.

4

We asked the military services and other DOD components to identify candidate projects 

because DOD does not maintain a central list of IT services outsourcing projects. We chose 

each project on the basis of the following criteria: (1) no more than one project from each 

military service and two agencies, (2) illustrative example of DOD IT outsourcing, (3) dollar 

value greater than $10 million, and (4) enough time elapsed for services to have been 

delivered and performance measured.

5

We identified 70 practices in our November 2001 report on leading commercial practices 

that (1) are typically applied at the project level and (2) were verifiable through 

documentation and interviews. 

6

This calculation does not include practices that were not applicable to a particular project. 
Page 3 GAO-03-371 DOD IT Services Outsourcing

Figure 1: Percentage of Practices Implemented in Each Phase

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent. 
Page 4 GAO-03-371 DOD IT Services Outsourcing

Although implementing the leading commercial practices in our framework 

does not guarantee the success of an outsourcing project, the consensus 

view of the leading commercial activities that we studied is that these 

practices are the most critical to success when acquiring IT services.7

 In 

addition, not implementing or only partially implementing particular 

practices can produce negative consequences or add risk to a project. For 

example, the Department of the Navy project’s baseline of its existing 

environment (a phase I practice) was limited because it did not include an 

assessment of its legacy applications. Instead, project officials decided to 

rely on a preexisting inventory developed to address the Year 2000 

challenge. The Navy subsequently found that it had substantially 

underestimated the number of legacy applications, which, according to 

program officials, later contributed to the transition period slipping from

2-1⁄2 years to 3-1⁄2 years. 

As DOD gathers more experience in implementing projects for outsourcing 

IT services, it can benefit from leveraging the lessons derived from these 

initiatives. For example, the projects in our review have identified lessons 

learned in such areas as transitioning to the provider and partnering with 

the provider. Although DOD has taken action to gather and disseminate 

lessons learned and best practices on general acquisition issues, these 

efforts generally do not focus on outsourcing or include sharing the lessons 

learned from IT outsourcing projects across the department. By not 

capturing and disseminating such information in a systematic manner 

across the department, DOD is losing the opportunity to leverage the 

knowledge gained on IT services projects like those in our review.

We are making recommendations to the Secretary of Defense aimed at 

leveraging lessons learned across the department from its components’ IT 

services outsourcing experiences. 

In written comments on a draft of this report signed by the DOD Principal 

Director, Deputy Assistant Secretary of Defense (Deputy Chief Information 

Officer), DOD agreed that capturing lessons learned in the development 

and implementation of its IT outsourcing initiatives is important to 

continually improving its outsourcing methods and results achieved. The 

7

This consensus view was based on interviews with managers in leading commercial 

organizations, discussions with academic and professional authorities, and extensive 

research on IT acquisition practices. 
Page 5 GAO-03-371 DOD IT Services Outsourcing

department also stated that it intends to explore a variety of mechanisms 

for best exploiting lessons learned from its IT outsourcing initiatives. We 

agree that it is prudent to consider alternative means to leveraging these 

lessons learned, and we believe that this is consistent with our 

recommendations.

Background To protect the security of the United States, DOD relies on a complex array 

of computer-dependent and mutually supportive organizational 

components, including the military services, Commanders in Chief, and 

Defense agencies. As such, it invests tens of billions of dollars each year in 

a broad array of computer systems, which include weapon systems, 

command and control systems, satellite systems, inventory management 

systems, transportation management systems, health systems, financial 

systems, personnel systems, and payment systems. In addition, DOD 

spends billions of dollars annually on IT services, which include database 

management, help-desk operations, software maintenance, and network 

services. In fiscal year 2001, DOD reportedly obligated more than $6.2 

billion on IT services alone.8

Decisions regarding the purchasing of services are critical to ensuring the 

effectiveness of DOD’s operations as well as those of the government as a 

whole. Our November 2001 report recognizes the importance of such 

sourcing decisions and provides a framework that spans the full range of 

activities that are performed during IT services outsourcing.9

 At the same 

time, governmentwide policies, initiatives, and challenges exist that 

significantly influence the government’s sourcing decisions. 

8

This figure is from the Federal Procurement Data System, which contains detailed 

information on contract actions over $25,000. 

9

GAO-02-214. 
Page 6 GAO-03-371 DOD IT Services Outsourcing

GAO’s Framework for 

Outsourcing IT Services

Outsourcing of IT services has become increasingly popular in both the 

public and private sector. For example, according to the Giga Information 

Group, Inc., a leading research firm, such outsourcing is expected to grow 

an average of 5 to 6 percent in 2003.10 The federal sector’s outsourcing is 

predicted to rise at an even greater rate. For example, INPUT, an IT market 

research firm, forecasts that defense IT outsourcing will increase about 143 

percent between fiscal years 2002 and 2007.11

IT outsourcing involves the activities associated with acquiring services 

from one or more external providers. During outsourcing, a client 

organization transfers responsibilities for performing one or more IT 

services to one or more external providers. This responsibility is executed 

through control and management of the processes, people, and technology 

associated with these services. 

Figure 2 depicts the roles of the client and provider organizations in an 

outsourcing relationship.

Figure 2: Roles of the Client and Provider in an Outsourcing Relationship

10Giga Information Group, Inc., IT Trends 2003: IT Services (Dec. 19, 2002). We did not 

independently verify these data.

11INPUT, The Federal IT Outsourcing Market View (December 2002). We did not 

independently verify these data.
Page 7 GAO-03-371 DOD IT Services Outsourcing

Our November 2001 guide on leading commercial practices for outsourcing 

IT services provides a generic framework of practices from leading 

commercial organizations that can improve purchasing decisions and 

manage the resulting government/provider relationship.12 The framework is 

represented in figure 3 as a hierarchy of phases, practices, and critical 

success factors. 

Figure 3: GAO’s Framework for Outsourcing IT Services 

Note: The arrow from phase VII to phase I represents the need to reflect on lessons learned from 

previous phases. The arrows between phase III and IV represent the iterative nature of developing the 

contract and selecting the provider. Although there is a logical order to the sequence of the common 

phases, the order of the practices within each phase does not imply any priority or sequence. 

Table 1 provides a definition of each phase of the framework. Each of the 

phases has specific practices associated with it. Implementing these 

practices does not guarantee the success of an outsourcing project. 

However, our November 2001 study reflected a consensus view that these 

practices were the most critical to success when IT services are being 

acquired.13 Restated, application of these practices increases the 

probability of a successful outsourcing project.

12GAO-02-214. 

13This consensus view was based on interviews with managers in leading commercial 

organizations, discussions with academic and professional authorities, and extensive 

research on IT acquisition practices. 
Page 8 GAO-03-371 DOD IT Services Outsourcing

Table 1: Definition of Phases for IT Outsourcing

Source: GAO.

The organizations that we studied also identified certain capabilities 

(identified as critical success factors) that were essential for implementing 

the practices identified in our framework. First, executive leadership 

strengthens the interaction between executive management and the 

employees of the client organization. Second, partner alignment 

strengthens the interaction between the client and provider organization at 

the executive level, which ensures that the goals and objectives of these 

organizations support each other. Third, relationship management 

strengthens the interaction between the client and provider organization at 

the operational level. 

Influences on Government 

Sourcing Decisions

The federal government is one of the world’s largest users of services. 

Because of the large dollar value and the number of private- and publicsector jobs involved, deciding whether the public or private sector would 

be the most appropriate provider of the services the government needs (IT 

or otherwise) is an important, and often highly charged, question. Among 

the factors that agencies must consider as they determine how best to meet 

their missions is whether the public or private sector would be the most 

appropriate provider of the services the government needs. Phase I of our 

framework, determine the sourcing strategy, addresses the client’s 

Phase 

number Title Definition

I Determine sourcing strategy Client organizations determine whether internal capability or external expertise 

can more effectively meet IT needs.

II Define operational model Client organizations formalize executive leadership, team composition, client 

responsibilities, and operating relationships between client and provider 

organizations.

III Develop the contract Client organizations establish the legal terms for the IT outsourcing relationship.

IV Select the provider(s) Client organizations find one or more providers who can help them reach their 

IT outsourcing goals.

V Transition to provider(s) Client organizations transfer responsibility of IT functions to one or more 

providers.

VI Manage provider(s) performance Client organizations make sure each provider is meeting performance 

requirements.

VII Ensure services are provided Client organizations make sure that services are provided and end-user needs 

are met.
Page 9 GAO-03-371 DOD IT Services Outsourcing

assessment of whether expertise from within or outside of the organization 

can more effectively meet the client’s needs.

The National Defense Authorization Act for Fiscal Year 2001 required the 

Comptroller General of the United States to convene a panel of experts to 

study the current process used by the government to make sourcing 

decisions. The resulting Commercial Activities Panel conducted a year-long 

study and heard repeatedly about the importance of competition and its 

central role in fostering economy, efficiency, and continuous performance 

improvement. In particular, the panel reviewed the government’s 

implementation of the Office of Management and Budget’s (OMB) Circular 

A-76, which sets forth federal policy for determining whether federal 

employees or private contractors will perform commercial activities for the 

government.14 Circular A-76 (1) outlines conditions under which agencies 

are permitted to perform a commercial activity with government 

employees or by contract and (2) provides guidance for whether, and if so, 

how, agencies should conduct a cost comparison when they are 

considering transferring the performance of commercial activities from the 

public to the private sector (or vice versa). The panel reported that there 

were positive elements to Circular A-76 but that both federal employees 

and private firms complained that it does not meet the standard of a clear, 

transparent, and consistently applied process. For example, both federal 

employees and private firms criticized the Circular A-76 process as unequal 

and therefore unfair.

The Commercial Activities Panel strongly supported continued emphasis 

on competition and concluded that whenever the government is 

considering converting work from one sector to another, public/private 

competitions should be the norm. In addition, the panel made four 

recommendations, including that all sourcing decisions be consistent with 

the principles adopted unanimously by the panel, such as the principle that 

federal policy provide for accountability in connection with all sourcing 

decisions.15

14Office of Management and Budget, Circular A-76, Performance of Commercial Activities

(Washington, D.C.: June 14, 1999). In November 2002, OMB issued proposed revisions that 

would substantially change this circular. As of April 8, 2003, these revisions have not yet 

been made final.

15Final report of the Commercial Activities Panel, Improving the Sourcing Decisions of the 

Government (Washington, D.C.: Apr. 30, 2002).
Page 10 GAO-03-371 DOD IT Services Outsourcing

As part of the administration’s efforts to implement the recommendations 

of the Commercial Activities Panel, OMB has published proposed changes 

to Circular A-76. Key highlights of the proposed changes include presuming 

that all functions are commercial in nature unless they are justified as 

inherently governmental;16 limiting the length of time for competitions; and 

emphasizing awarding contracts on the basis of best value, not just lowest 

cost. Best value allows the contracting official to consider technical 

superiority, quality, innovation, and past performance as well as price.

However, we reported that there are several areas in which the proposed 

revisions to the circular are not consistent with the principles or 

recommendations of the Commercial Activities Panel.17 Specifically, the 

proposed revision does not include a link between sourcing policy and 

agency missions, has unnecessarily complicated source selection 

procedures, contains certain unrealistic time frames, and includes 

insufficient guidance on calculating savings.

Beyond the Commercial Activities Panel, other bodies have identified 

challenges that the federal government faces in reaching and executing 

effective sourcing decisions. For example, members of the Coalition for 

Government Procurement, the Professional Services Council, and the 

Information Technology Association of America told us that organizational 

culture is one of the biggest differences between the commercial sector 

and the federal government and one of the greatest barriers to the 

government’s use of commercial practices. Also, as we have previously 

reported, moving to outsourcing solutions can involve a cultural change for 

government organizations because it may require a change to an agency’s 

operating model, such as using a contractor to provide IT services 

previously performed by government staff or using a performance-based 

contract.18 This view was echoed by a 2001 study of DOD competitive

16Section 5 of P.L. 105-270, codified at 31 U.S.C. 501 note (1998), defines an inherently 

governmental function as a “function that is so intimately related to the public interest as to 

require performance by Federal Government employees.” 

17U.S. General Accounting Office, Proposed Revisions to OMB Circular A-76, GAO-03-391R

(Washington, D.C.: Jan. 16, 2003). 

18U.S. General Accounting Office, Desktop Outsourcing: Positive Results Reported, but 

Analyses Could Be Strengthened, GAO-02-329 (Washington, D.C.: Mar. 29, 2002). 
Page 11 GAO-03-371 DOD IT Services Outsourcing

sourcing that found cultural, process, execution, and training barriers.19

The study stated that these barriers need to be understood and mitigated 

before the benefits of outsourcing can be fully realized. Barriers such as 

these can be overcome by strong executive leadership, which is a critical 

success factor in our framework.

Another challenge is creating a productive agency/provider relationship—

another critical success factor in our framework. According to a report 

sponsored by the PricewaterhouseCoopers Endowment for the Business of 

Government, such public/private partnerships are based on trust, 

commitment to problem or conflict resolution, and the recognition that 

flexibility is necessary and that the relationship will evolve and change over 

time.20 If deadlines are not met, or public agency goals change with 

differing political climates, the partners need to discuss the basis of the 

partnership and construct a different relationship. Our prior report on 

desktop outsourcing found that developing a productive agency/contractor 

relationship is not always easy.21 Both sides must recognize and understand 

each other’s underlying motives and strive to achieve established 

expectations.

Finally, human capital issues are another challenge facing federal agencies 

that affect their ability to implement outsourcing. Our framework 

recognizes the importance of having the right skills in place to support the 

outsourcing relationship. However, as we have previously reported, 

procurement reforms and technological changes have placed 

unprecedented demands on the acquisition workforce.22 Contracting 

personnel are now expected to have a much greater knowledge of market 

conditions, industry trends, and the technical details of the commodities 

and services they procure. The Commercial Activities Panel report stated 

that developing and maintaining a skilled acquisition workforce is the 

critical first step in managing this more complex procurement 

environment. The panel also reported that DOD bore the brunt of a 22 

19The Industrial College of the Armed Forces, Case Study: Complex Business Management 

for Competitive Sourcing (2001). 

20PricewaterhouseCoopers Endowment for the Business of Government, Contracting for 

the 21st Century: A Partnership Model (January 2002). 

21GAO-02-329. 

22U.S. General Accounting Office, Contract Management: Taking a Strategic Approach to 

Improving Service Acquisitions, GAO-02-499T (Washington, D.C.: Mar. 7, 2002).
Page 12 GAO-03-371 DOD IT Services Outsourcing

percent downsizing of the federal acquisition workforce in the last decade, 

going from 96,000 staff in 1991 to about 68,000 in fiscal year 2001. 

Addressing human capital issues is not just a matter of the size of the 

workforce; it is also a knowledge and skills issue. According to the 

Commercial Activities Panel, it is critically important that federal agencies 

adequately address human capital needs in meeting the current and 

emerging needs of government and its citizens in the most effective, 

efficient, and economical manner possible. This will require increased 

emphasis on training and development, particularly in the area of 

technology. 

Description of Five Projects 

Reviewed

The five projects in our study varied in how they approached outsourcing 

IT services, such as in using various solicitation methods, including holding 

a public/private competition under the policies outlined in OMB Circular A76 or carrying out a negotiated competitive procurement. In addition, the 

types of services being outsourced differed: services ranged from the 

narrowly focused (e.g., help-desk services) to the very broad (e.g., 

enterprisewide end-to-end information services); contract terms ranged 

from 5 to 15 years (assuming all option years are exercised); and estimated 

contract values ranged from $23 million to $8.8 billion. Table 2 provides 

information on the variety of IT services and outsourcing approaches taken 

by the projects. 
Page 13 GAO-03-371 DOD IT Services Outsourcing

Table 2: Profile of Outsourcing Projects Reviewed

Source: DOD.

a

Estimated value if all option years are exercised.

b

NIMA performed an OMB Circular A-76 analysis and, on the basis of this analysis, implemented a 

direct conversion to a preferential provider rather than holding a public/private competition or obtaining 

an agency cost-comparison waiver.

c

As of February 24, 2003, NIMA had transitioned four of these functions to the provider: printing, digital 

replication, video and voice communications, and help-desk operations.

Agency/Project

Solicitation 

method

Date of 

contract 

award

Contract 

term

Contract 

type

Estimated total 

contract valuea Project description

Air Force/Kirtland Air 

Force Base’s 

Command, Control, 

Communications, and 

Computer (C4) 

Services

Competitive, 

under OMB 

Circular A-76 

and small 

business setaside rules

April 2000 1 year, with 

4 option 

years

Firm, fixedprice$23 million Management, operations, and 

maintenance of command, 

control, communications, and 

computer systems, multimedia 

services, and information 

management for the Kirtland Air 

Force Base.

Army/Network 

Enterprise Technology 

Command’s 

(NETCOM) Total Army 

Communications—

Southwest Asia (TACSWA)

Negotiated 

competitive 

solicitation 

March 2001 1 year, with 

4 option 

years

Firm, fixedprice$204 million Operation and maintenance 

services, including repair, 

installation, and supply, for 

communications equipment in 

Southwest Asia.

Military Health 

System/Information 

Technology 

Organization 

(MHS/ITO) Help Desk 

Negotiated 

competitive 

solicitation 

June 2001 1 year, with 

7 option 

years

Firm, fixedprice with 

incentive 

awards

$71 million Call and help-desk services for 

all MHS software applications.

Department of the 

Navy/Navy and Marine 

Corps Intranet (NMCI)

Negotiated 

competitive 

solicitation 

October 

2000

7 years, 

with an 

option for 3 

additional 

years

Firm, fixedprice with 

incentive 

awards

$8.8 billion Department of the Navy-wide 

end-to-end information services 

through a common computing 

and communication 

environment.

National Imagery and 

Mapping Agency’s 

(NIMA) Information 

Technology/ 

Information Services 

(IT/IS)

Sole-source, 

using a 

statutory 

preferential 

provider 

(Alaska native 

corporation)b

December 

2001

1 year, with 

14 option 

years

Cost plus 

award fees

$2.1 billion NIMA-wide IT/IS support 

services for printing, digital 

replication, networks, distributed 

and centralized systems and 

services, video and voice 

communications, information 

research, and help-desk 

operations.c
Page 14 GAO-03-371 DOD IT Services Outsourcing

Projects Substantially 

Used Leading 

Commercial Practices

As illustrated in figure 4, the five IT services projects substantially used 

leading commercial practices. Specifically, each project used at least 76 

percent of the practices.23 Reasons for projects implementing different 

percentages of the practices include differences in their individual 

circumstances and objectives. For example, the Army’s Total Army 

Communications—Southwest Asia (TAC-SWA) project, which used the 

fewest number of practices and had the largest number of practices that 

were not applicable, was largely a continuation of an existing approach 

that already relied on the private sector but with fewer providers. In 

contrast, the National Imagery and Mapping Agency’s (NIMA) Information 

Technology/ Information Services (IT/IS) project, which fully or partially 

implemented all of the applicable practices, involved a significant 

operational shift (e.g., functions previously performed by NIMA staff are 

now performed by a contractor) and was intended to result in substantial 

process improvements. In addition, the three projects that implemented the 

largest percentage of practices also used third-party assistance—including 

employing a contractor with sourcing expertise—to help formulate their 

sourcing strategy, which could account for the extent of their compliance.

23This calculation does not include practices that were not applicable to a particular project.
Page 15 GAO-03-371 DOD IT Services Outsourcing

Figure 4: Use of Leading Commercial Practices for Outsourcing of IT Services, by 

Project

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent. 

In addition, figure 5 illustrates that project compliance extended to each of 

the phases of our framework. Collectively, the projects fully implemented 

from 70 to 97 percent of the practices in each phase. Phase I, determine 

sourcing strategy, had the lowest percentage of practices implemented by 

the projects (70 percent). This result is not inconsistent with a recent Giga 

Information Group, Inc., survey, which found that only half of the
Page 16 GAO-03-371 DOD IT Services Outsourcing

respondent organizations had documented an IT sourcing strategy.24 This 

approach carries risk since phase I sets the tone for the outsourcing 

initiative within the client organization.

Figure 5: Percentage of Practices Implemented in Each Phase

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent. 

24Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key Stages and Phases 

of the IT Sourcing Process (Jan. 31, 2003). 
Page 17 GAO-03-371 DOD IT Services Outsourcing

In addition, collectively the projects fully implemented 88 percent of the 

practices (see table 3).25

Table 3: Percentage of Practices Implemented, by Project

Source: GAO.

a

These calculations do not include practices that were not applicable to a particular project.

The following provides additional information on the projects’ 

implementation of each phase of our framework.

25This calculation does not include practices that were not applicable to a particular project.

Phase

Percentage of practices implementeda

Air Force

C4 Services

project

Army

TAC-SWA

project

MHS/ITO Help

Desk project

Navy NMCI

project

NIMA IT/IS

project Overall

Phase I: Determine sourcing 

strategy (6 practices). 50 33 83 83 100 70

Phase II: Define operational 

model (13 practices). 92 83 100 77 100 90

Phase III: Develop the contract 

(16 practices). 64 83 100 94 100 89

Phase IV: Select provider(s) (7 

practices). 86 86 100 100 71 89

Phase V: Transition to provider(s) 

(11 practices). 82 88 100 82 100 90

Phase VI: Manage provider(s) 

performance (11 practices). 73 73 100 91 82 84

Phase VII: Ensure services are 

provided (6 practices). 83 100 100 100 100 97

Overall 76 79 99 89 94 88
Page 18 GAO-03-371 DOD IT Services Outsourcing

• Phase I: Determine sourcing strategy. In the first phase of our 

outsourcing framework, the client organization determines whether 

internal capability or external expertise can more effectively meet its IT 

needs. The purpose of a sourcing strategy is to achieve the optimal 

balance between internal and external capabilities, activities, processes, 

and services to ensure the achievement of strategic business objectives 

at the lowest risk.26 The five projects’ implementation of this phase was 

uneven. In particular, two of the six practices in this phase were fully 

implemented by all five projects, but the other four practices were not. 

Among the practices that were implemented by all of the projects was 

determining the business reasons for outsourcing. In addition, the three 

projects that implemented the largest percentage of practices in our 

framework—the Military Health System/Information Technology 

Organization (MHS/ITO) Help Desk, the Department of the Navy’s Navy 

and Marine Corps Intranet (NMCI), and NIMA IT/IS projects—used the 

third-party assistance practice in this phase to help formulate their 

sourcing strategy, which could account for the extent of their 

compliance. For example, the MHS/ITO Help Desk project, which 

implemented the largest percentage of practices, worked with the 

Department of the Interior’s GovWorks Program,27 the Defense 

Acquisition University staff, and a private-sector contractor to obtain 

expertise on sourcing strategies.

The practice in this phase that was the most unevenly implemented was 

the benchmarking28 and baselining of existing internal services. Of the 

five projects in our review, (1) one fully benchmarked and baselined the 

productivity of the activity being outsourced before making the final 

sourcing decision, (2) two partially baselined their existing activities, 

and (3) two did not perform benchmark and baseline analyses at all. 

The agencies’ reasons for not fully implementing this practice included 

that an executive decision had been made to conduct a public/private 

competition following the OMB Circular A-76 policy, so such an 

analysis would not have affected the sourcing decision, or that 

26Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing Strategy, Research 

Note R-18-1099 (Sept. 23, 2002). 

27The Department of the Interior’s GovWorks Program is a federal fee-for-service acquisition 

center that helps other federal agencies acquire supplies and services for their programs on 

a project-by-project basis.

28Gartner, Inc., defines benchmarking as a method to compare the cost or price of an IT 

environment to peer groups with the same workload characteristics. 
Page 19 GAO-03-371 DOD IT Services Outsourcing

available documentation to perform such an analysis was limited. 

Leading research firms suggest benchmarking and baselining the 

entity’s current processes before outsourcing because only then would 

it be able to determine whether the arrangement has been successful.29

In addition, the risk of not fully baselining the existing environment is 

illustrated by the NMCI project. Specifically, the NMCI project’s 

baseline of its existing environment was limited because it did not 

include an assessment of its legacy applications since project officials 

decided to rely on a preexisting inventory developed to address the 

Year 2000 challenge. The Navy subsequently found that it had 

substantially underestimated its number of legacy applications, that, 

according to program officials, later contributed to the transition 

period slipping from 2-1⁄2 years to 3-1⁄2 years. Appendix II provides 

additional information on projects’ implementation of the practices 

comprising this phase.

• Phase II: Define operational model. The operational model is an 

important mechanism for an organization to compare its plans with the 

expectations that were set when the decision to outsource was made 

and to ascertain whether these plans will enable the organization to 

meet expectations. The five projects had largely implemented the 13 

practices contained in this phase. Specifically, about 90 percent30 of the 

practices were implemented. For example, all projects implemented the 

practice that executive leadership be established to facilitate the 

outsourcing effort. NIMA, for instance, formed a strategic sourcing 

office to oversee the IT/IS project. Another practice—training the 

provider on the organization’s business environment and goals—was 

fully implemented by one project (in two cases, the practice was not 

applicable). One project that did not fully implement this practice was 

the Air Force’s Kirtland Air Force Base’s Command, Control, 

Communications, and Computer (C4) Services project. Although some 

training was provided (e.g., Kirtland held an orientation session for 

potential bidders), provider officials stated that they did not receive 

adequate training, which made the transition period more difficult.

Appendix III provides additional information on projects’ 

implementation of the practices comprising this phase.

29Giga Information Group, Inc., Payment and Incentives for Outsourcing Management

(July 27, 2000) and Gartner, Inc., Benchmarking Helps Outsourcing Deals Stay 

Competitive, Research Note COM-16-8055 (June 14, 2002).

30This calculation does not include practices that were not applicable to a particular project.
Page 20 GAO-03-371 DOD IT Services Outsourcing

• Phase III: Develop the contract. A well-written contract is necessary for 

the outsourcing organization to meet its requirements while allowing 

the service provider to make a fair profit. It sets the expectations for 

service levels, delivery of essential services, and continuous 

improvement and should protect the interests of all parties. The five 

projects largely implemented the practices in this phase. Specifically, 

about 89 percent31 of the practices were implemented in this phase, and 

two projects (the MHS/ITO Help Desk and NIMA IT/IS projects) 

implemented all of the practices. Several practices in this phase address 

performance requirements. For example, all five projects implemented 

the practices that called for basing performance requirements on 

business requirements and reviewing and updating them periodically. 

One practice that was not fully implemented by two projects was 

including performance measures that address both technical and enduser satisfaction aspects of performance. For example, the Army TACSWA project included technical performance measures in its contract 

but not measures related to end-user satisfaction, even though the 

contract included help-desk services. According to the project official 

that developed the performance work statement in the contract, the 

command did not include customer satisfaction measures because it did 

not think that it was necessary to have a performance standard for the 

help-desk service. However, without such measures, the agency does 

not have a contractual standard with which to judge the provider’s 

performance. Appendix IV provides additional information on projects’ 

implementation of the practices comprising this phase.

• Phase IV: Select the provider(s). Critical to the success of any 

outsourcing project for IT services is identifying potential providers and 

ultimately selecting a provider(s) that will best meet the needs of the 

agency. The five projects had largely implemented the seven practices 

contained in this phase. About 89 percent of the practices were 

implemented in this phase and two projects (the MHS/ITO Help Desk 

and Navy NMCI projects) used each applicable practice. For example, 

the five projects implemented the practice related to conducting due 

diligence activities to verify provider capabilities before signing the 

contract. In the case of the Army TAC-SWA project, the Network 

Enterprise Technology Command (NETCOM) evaluated the provider’s 

financial and past performance information. The Department of the 

Navy’s NMCI project also evaluated bidders’ past performance and 

31This calculation does not include practices that were not applicable to a particular project.
Page 21 GAO-03-371 DOD IT Services Outsourcing

performed reference checks. Moreover, as part of its due diligence 

activities, the Navy required all bidders to demonstrate that they had 

experience in implementing large seat management contracts.32 A 

practice that was not implemented by two projects was using third-party 

assistance when selecting the provider. Projects that did not implement 

this practice believed that they had adequate in-house expertise with 

outsourcing, making third-party assistance unnecessary. However, 

because third-party assistance provides an independent resource that 

can suggest options or processes that the client organization may not be 

aware of, these projects may have missed an opportunity to implement 

their outsourcing projects more effectively. Appendix V provides 

additional information on projects’ implementation of the practices 

comprising this phase.

• Phase V: Transition to provider(s). This phase focuses on the client 

organization’s transfer of the IT function to one or more providers. As 

part of this transition, the clear definition of responsibilities and the 

careful consideration of employees’ needs matched against the client 

organization’s needs enable both the client and provider to focus on 

execution and give staff confidence in their future employment. The five 

projects largely implemented the 11 practices associated with this 

phase. Specifically, about 90 percent33 of the practices were 

implemented in this phase, and two projects implemented all of the 

practices (the MHS/ITO Help Desk and NIMA IT/IS projects). Several 

practices in this phase address dealing with employees affected by the 

outsourcing projects. For example, in the four projects in which federal 

employees were affected, the projects provided assistance to those who 

did not want to transfer to the provider, including helping to place them 

in other positions and helping with résumé writing. A related practice is 

to clearly communicate to all employees what is going to happen and 

when it is going to happen. Two projects did not fully implement this 

practice. For example, the Navy used its normal chain of command to 

communicate transition information, but found that implementation of 

this practice was uneven. As a result, some staff did not know current 

information about how NMCI would affect them until the provider was 

ready to contact them regarding their possible transition to the 

32Seat management generally refers to service provision arrangements in which contractorowned desktop and other computing hardware, software, and related services are bundled 

and provided to a client organization at a fixed price per unit (or seat). 

33This calculation does not include practices that were not applicable to a particular project.
Page 22 GAO-03-371 DOD IT Services Outsourcing

contractor. However, according to the NMCI’s Director’s office, this 

problem was somewhat mitigated by the provider’s Web site that 

provides transition information to all NMCI customers/users. Appendix 

VI provides additional information on projects’ implementation of the 

practices comprising this phase.

• Phase VI: Manage provider(s) performance. The effectiveness with 

which the provider(s) performance is managed is critical to the 

successful implementation of an outsourcing project. Indeed, according 

to Gartner, Inc., a leading research firm, an outsourcing project can be 

thwarted by poorly designed, funded, and delivered processes for 

managing the delivery of services.34 The five projects generally 

implemented the 11 practices contained in this phase, with about 84 

percent of the practices being implemented in this phase. For example, 

the practices related to obtaining feedback on provider performance 

were largely implemented. This is important because different levels of 

an agency can have different perceptions about the value of the 

outsourcing project. For example, an outsourcing project may be 

considered successful by the agency’s executive management if it is 

focused on controlling costs, but be considered inadequate by business 

managers and users who may be expecting higher levels of service. Each 

of these viewpoints is valid and should be taken into account when the 

provider’s performance is evaluated. Two other practices—including 

incentives and penalties in contracts—were fully implemented by two 

projects. Incentives and penalties are important because they can help 

motivate the provider to exceed or meet performance requirements. 

Nonetheless, two projects did not include monetary incentives and two 

projects did not include monetary penalties in their contracts. For 

example, the Army TAC-SWA project did not include monetary 

incentives, although the contracting officer stated that incentives might 

have been useful to motivate the provider to exceed performance 

requirements. Incentives can also help control risks. According to a 

guide on performance-based services acquisition, if the incentives in the 

contract are right and if the provider and agency share the same goals, 

risk is largely controlled and effective performance is “almost the 

34Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals, Research Note 

COM-16-8425 (June 17, 2002).
Page 23 GAO-03-371 DOD IT Services Outsourcing

inevitable outcome.”35 Appendix VII provides additional information on 

projects’ implementation of the practices comprising this phase.

• Phase VII: Ensure services are provided. Although outsourcing 

transfers responsibility for performing the service to the provider(s), the 

client organization is ultimately responsible for ensuring that services 

are provided and that end-user needs are met. Accordingly, it is critical 

that the agency ensure that services are provided. The projects had 

implemented 97 percent of these practices, and four projects 

implemented all of them. For example, every project monitored the 

providers’ work. In the case of the Air Force’s C4 Services project, 

quality assurance evaluators monitored the provider’s work to identify 

problems or trends in accordance with the project’s quality assurance 

surveillance plan. The results were reported to the contracting officer 

and to the functional area chief for resolution. Another practice, using 

customer satisfaction surveys, was fully implemented by four of the five 

projects. However, the Air Force project did not conduct, or require its 

contractor to conduct, customer satisfaction surveys. Although the 

provider surveys staff annually, the Air Force is nevertheless relying on 

the provider to voluntarily implement an important practice for 

determining how customers view the services being delivered and 

whether changes need to be made. Appendix VIII provides additional 

information on projects’ implementation of the practices comprising 

this phase.

Leveraging Lessons 

Learned DOD-wide

Could Assist Other 

DOD Projects

We have previously reported on the importance of collecting and 

disseminating lessons learned.36 For example, a critical activity in IT 

investment management is establishing a process for developing and 

capturing lessons learned in a written product or knowledge base and 

disseminating them to decision-makers.37 In addition, one of the practices 

in our framework for outsourcing IT services addresses incorporating 

35An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based 

Services Acquisition, Benchmark Version (January 2002).

36U.S. General Accounting Office, NASA: Better Mechanisms Needed for Sharing Lessons 

Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002). 

37U.S. General Accounting Office, Information Technology Investment Management: A 

Framework for Assessing and Improving Process Maturity, GAO/AIMD-10-1.23, Exposure 

Draft (Washington, D.C.: May 2000).
Page 24 GAO-03-371 DOD IT Services Outsourcing

lessons learned from peers who have engaged in similar sourcing 

decisions. Use of lessons learned is a principal component of an 

organizational culture committed to continuous improvement. Sharing 

such information serves to communicate acquired knowledge more 

effectively and to ensure that beneficial information is factored into 

planning, work processes, and activities. Lessons learned can be based on 

positive experiences or on negative experiences that result in undesirable 

outcomes.

Although DOD has taken action to gather and disseminate lessons learned 

and best practices on general acquisition issues, these efforts generally do 

not focus on outsourcing or include sharing the lessons learned from IT 

outsourcing projects across the department. Specifically, a number of DOD 

Web sites provide guidance, lessons learned, and best practices related to 

general acquisition issues. However, using these sites to locate specific 

information on IT outsourcing best practices and lessons learned can be 

time-consuming and difficult because so many topics and information 

sources are provided. Specifically, MHS/ITO Help Desk project officials 

said that searching numerous Web sites to get relevant information to 

address questions and concerns about outsourcing IT services can 

consume hours. For example, when we entered the keywords “IT 

outsourcing” and “best practices” into the search feature on the Office of 

the Undersecretary for Defense of Acquisition, Technology, and Logistics 

site, ACQWeb (www.acq.osd.mil) in early March, it provided us with links 

to 1,251 documents. Likewise, lessons learned covers so many topics that it 

is difficult to search for an applicable IT lesson. For example, when we 

used the phrase “lessons learned in IT outsourcing” no documents were 

identified on the ACQWeb, but when we inserted “lessons learned” and “IT 

outsourcing” links to more than 1,700 documents were produced. 

One DOD Web site, Share A-76!, was established to address one of our 

previous recommendations,38 that is, to establish a framework for 

identifying and analyzing best practices and lessons learned from 

competitive sourcing studies and disseminating them DOD-wide. Share A76! promotes the sharing of best practices and lessons learned related to 

one form of outsourcing that was conducted under the OMB Circular A-76 

competitive sourcing process. Among other things, the site contains 

guidance, links to other relevant sites, sample documents, and a best 

38U.S. General Accounting Office, DOD Competitive Sourcing: Lessons Learned System 

Could Enhance A-76 Study Process, GAO/NSIAD-99-152 (Washington, D.C.: July 21, 1999). 
Page 25 GAO-03-371 DOD IT Services Outsourcing

practices library that communicates field staff experiences and advice 

about the Circular A-76 process. A NIMA project official said that NIMA’s 

staff routinely accesses the Web site because it contains a wealth of 

information on policies, procedures, lessons learned, and links to other 

outsourcing sites. The DOD analyst responsible for Share A-76! estimated 

that the site receives about 12,000 visits per month and said that on the 

basis of E-mails and anecdotes, user satisfaction is favorable. However, this 

Web site is specific to the OMB Circular A-76 process, which may not apply 

to other types of outsourcing. For example, Circular A-76’s policy pertains 

to public/private competitions and requires that the final evaluation 

between the government and the private sector be based exclusively on 

cost. 

DOD acquisition and IT officials acknowledged that there is no mechanism 

in DOD to easily share and leverage lessons learned relating to outsourcing 

IT services. However, these officials agreed that a departmentwide effort to 

identify, capture, and disseminate lessons learned and leading practices of 

projects with experience in carrying out IT outsourcing could offer 

valuable insights and new ideas that would benefit others. Moreover, 

officials from three of the projects in our review told us that there is value 

in collecting and disseminating the knowledge acquired from IT 

outsourcing projects in a systematic manner across the department. 

Each of the projects in our review identified knowledge and experience 

gained from their approaches to outsourcing IT services that could offer 

insights and practices for other ongoing and future projects to consider. 

For example:

• MHS developed specific guidance and lessons learned for implementing 

a performance-based incentive contract for help-desk operations. 

• The Department of the Navy’s NMCI project has developed a series of 

lessons learned related to transitioning to the provider that is being 

shared within the NMCI community; one example was that all personnel 

should be available during scheduled testing and deployment.

• NIMA has experience in contracting techniques emphasizing a 

partnering approach with providers to refine requirements and establish 

a common understanding of costs.

In addition, a departmental IT outsourcing knowledge-sharing approach 

could include links to information about other government agencies’ IT 
Page 26 GAO-03-371 DOD IT Services Outsourcing

outsourcing projects. For example, our 2002 report on desktop outsourcing 

includes an extensive discussion of lessons learned by agencies that have 

implemented this type of IT services outsourcing.39

Developing an effective lessons learned activity is not easy. For example, 

NMCI officials said that for a lessons learned initiative to be effective, a 

process must exist that is clearly understood by everyone and allows 

capturing and sharing of knowledge to occur with minimum effort. Other 

challenges in developing an effective lessons learned process were outlined 

by the Share A-76! analyst. The analyst stated that only a small number of 

site users have contributed lessons learned to the Share A-76! Web site, 

which she attributed, in part, to the amount of time and effort needed to 

document and obtain agreement by all levels of the organization on the 

lessons learned. In addition, the analyst stated that there is reluctance to 

share negative lessons, and often the review and approval process sanitizes 

best practice information so that it becomes too general to be most helpful 

to users. Such challenges can be overcome by executive-level support. 

Indeed, DOD acquisition and IT officials stated that for lessons learning 

activities to be effective, senior management must devote support and 

resources to the effort. This is consistent with our prior work, which 

showed that knowledge can be effectively shared only when employees are 

given adequate time as well as established places where they can actually 

transfer knowledge.40

Last year, we outlined a generic lessons learned process that could be used 

to guide the development of such a process for outsourcing IT services.41

Although the mechanism or processes used to collect, share, and 

disseminate lessons learned may vary, in general such a process comprises 

four main elements: collection, verification, storage, and dissemination. 

The collection process involves the capture of information through 

structured and unstructured processes. Verification serves to verify the 

correctness and applicability of lessons submitted. The storage aspect of 

lessons learned usually involves incorporating the lessons into an 

electronic database for the dissemination and sharing of information, 

including the ability to conduct information searches. The final element, 

and the most important, is the dissemination of lessons learned, since 

39GAO-02-329.

40GAO-02-195.

41GAO-02-195. 
Page 27 GAO-03-371 DOD IT Services Outsourcing

lessons are of little benefit unless they are distributed and used by people 

who will benefit from them. Lessons can be “pushed,” or automatically 

delivered to a user, or “pulled” in situations where a user must manually 

search for them. Lessons can also be disseminated with an assigned 

priority descriptor, which denotes the risk, immediacy, and urgency of the 

lessons learned content.

Conclusions The projects in our review substantially implemented leading commercial 

practices for outsourcing IT services, which has increased each project’s 

probability of success. Capturing how these projects operationalized 

leading commercial practices could help other IT services outsourcing 

projects succeed. Although currently there is no such DOD-wide 

mechanism, such as an electronic tool, to easily share and leverage lessons 

learned, DOD IT and acquisition officials agreed that a departmentwide 

effort to identify, capture, and disseminate lessons learned could offer 

valuable insights and new ideas that would benefit others. Lessons learned 

that are pragmatic and easily accessible could give DOD managers a more 

informed understanding of the important issues to be addressed when 

making outsourcing decisions, as well as the factors to be considered to 

help ensure the success of these endeavors. DOD managers can also 

benefit from lessons learned on the basis of negative experiences. The 

projects in our review were well into implementation, and therefore, at this 

late stage, we see little advantage for them to revisit practices that were not 

implemented. Nevertheless, an electronic tool for capturing and 

disseminating lessons learned would allow the rest of DOD to benefit from 

the negative consequences and increased risks associated with those 

practices that the projects did not implement. 

Developing a lessons learned mechanism is not easy; thus, senior 

management support and resources are keys to success. Without such 

support driving the capture and dissemination of lessons learned, DOD is 

losing an opportunity for wider application of leading practices and thus 

better ensuring that its IT outsourcing efforts are successful.

Recommendations To assist DOD organizations in planning and implementing outsourcing 

projects for IT services, we recommend that the Secretary of Defense 

direct the Undersecretary of Defense for Acquisition, Technology, and 

Logistics, working in conjunction with the Assistant Secretary of Defense 

for Command, Control, Communications, and Intelligence (C3I), to provide 
Page 28 GAO-03-371 DOD IT Services Outsourcing

senior management support and adequate resources to develop and 

implement an electronic tool to capture and disseminate examples and 

lessons learned from actual IT outsourcing projects. These examples and 

lessons learned, at a minimum, should include the results of our review of 

the five projects discussed in this report.

We also recommend that the Secretary of Defense direct the 

Undersecretary of Defense for Acquisition, Technology, and Logistics, 

working in conjunction with the Assistant Secretary of Defense for 

Command, Control, Communications, and Intelligence (C3I), to ensure that 

the method used to gather information for this electronic tool incorporate 

the main elements of a lessons learned process—namely, collection, 

verification, storage, and dissemination.

Agency Comments and 

Our Evaluation

In written comments on a draft of our report, signed by DOD’s Principal 

Director, Deputy Assistant Secretary of Defense (Deputy Chief Information 

Officer), the department partially concurred with our recommendations. 

Specifically, DOD agreed that capturing lessons learned in the development 

and implementation of its IT outsourcing initiatives is important to 

continually improve its outsourcing methods and results achieved. The 

department also stated that before deciding on a specific method to 

achieve this aim, it intends to explore a variety of mechanisms that could 

be used. In particular, the department stated that it currently has several 

processes and communities of interest that collect and disseminate lessons 

learned in other areas, which are logical starting points for determining the 

best path forward. DOD’s written comments are reproduced in appendix 

IX.

We agree that it is prudent to explore various alternatives to leveraging 

lessons learned from DOD’s IT services outsourcing experiences. Our 

recommendations are not prescriptive as to the electronic method to be 

used to capture and disseminate lessons learned. Therefore, the 

department’s plan to explore various alternatives is consistent with our 

recommendations.

We are sending copies of this report to the appropriate congressional 

committees, the Secretary of Defense, and the Director of the Office of 

Management and Budget. We will also provide copies to others upon 
Page 29 GAO-03-371 DOD IT Services Outsourcing

request. In addition, the report will be available at no charge on the GAO 

Web site at http://www.gao.gov. 

If you have any questions on matters discussed in this report, please 

contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director, at 

(202) 512-9556. We can also be reached by E-mail at hiter@gao.gov and 

lambertl@gao.gov, respectively. Other contacts and key contributors to this 

report are listed in appendix X.

Randolph C. Hite

Director, Information Technology Architecture

and Systems Issues
Page 30 GAO-03-371 DOD IT Services Outsourcing

Appendix I

Objectives, Scope, and Methodology AAppenednxidexiIs

Our objectives were to determine (1) the extent to which selected 

Department of Defense (DOD) information technology (IT) services 

outsourcing projects use leading commercial practices as specified in our 

framework and (2) whether DOD is sharing lessons learned from its IT 

outsourcing projects across the department.

To determine the extent to which selected DOD outsourcing projects for IT 

services use leading commercial practices, we identified the practices in 

our November 2001 report on leading commercial practices1

 that (1) are 

typically applied at the project level and (2) were verifiable through 

documentation and interviews. Because DOD did not centrally maintain a 

list of outsourcing projects for IT services, we asked the department to 

identify candidate projects for our evaluation. From this list, we selected 

the following five projects for our review: (1) Air Force Kirtland Air Force 

Base’s Command, Control, Communications, and Computer (C4) Services 

project; (2) Army Network Enterprise Technology Command’s (NETCOM) 

Total Army Communications – Southwest Asia (TAC-SWA) project; (3) 

Military Health System/Information Technology Organization (MHS/ITO) 

Help Desk project; (4) Department of the Navy’s Navy and Marine Corps 

Intranet (NMCI) project; and (5) National Imagery and Mapping Agency 

(NIMA) Information Technology/Information Services (IT/IS) project. We 

chose each project on the basis of the following criteria: (1) no more than 

one project from each military service and two agencies, (2) illustrative 

example of DOD IT outsourcing, (3) dollar value greater than $10 million, 

and (4) enough time elapsed for services to have been delivered and 

performance measured.

At our request, each project completed a self-assessment on whether and 

how it implemented leading commercial practices. We reviewed the agency 

self-assessments and accompanying documentation and interviewed the 

appropriate agency project officials to verify whether the practices were 

followed. In addition, we interviewed representatives from each of the 

providers associated with these projects.

We also researched additional information on commercial practices in our 

November 2001 guide2

 and the challenges the federal government faces in 

1

U.S. General Accounting Office, Information Technology: Leading Commercial Practices 

for Outsourcing of Services, GAO-02-214 (Washington, D.C.: Nov. 30, 2001).

2

GAO-02-214. 
Appendix I

Objectives, Scope, and Methodology

Page 31 GAO-03-371 DOD IT Services Outsourcing

implementing them. Specifically, we performed a literature search, which 

included reviewing reports issued by leading research firms, such as 

Gartner, Inc., and Giga Information Group, Inc. In addition, we interviewed 

representatives from industry organizations that have an interest in 

outsourcing IT services, including the Coalition for Government 

Procurement; the Information Technology Association of America; the 

Professional Services Council; and Acquisition Solutions, Inc. 

To determine whether DOD is sharing lessons learned from its IT 

outsourcing projects across the department, we identified and reviewed 

various approaches that DOD currently uses to capture and disseminate 

such information. This included identifying and reviewing various Web 

sites and performing key word searches on these sites to identify lessons 

learned for outsourcing IT services. We also interviewed applicable DOD 

acquisition and IT officials. Finally, we identified and reviewed a generic 

lessons learned process contained in our January 2002 report on the 

National Aeronautics and Space Administration’s lessons learned 

mechanisms.3

We performed our work at the Army’s NETCOM in Ft. Huachuca, Arizona; 

the MHS/ITO Help Desk project office in Falls Church, Virginia; the NMCI 

Director’s office in Crystal City, Virginia; and NIMA’s headquarters in 

Bethesda, Maryland. We conducted our review between May 2002 and early 

March 2003 in accordance with generally accepted government auditing 

standards.

3

U.S. General Accounting Office, NASA: Better Mechanisms Needed for Sharing Lessons 

Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002). 
Page 32 GAO-03-371 DOD IT Services Outsourcing

Appendix II

Projects’ Implementation of Phase I: 

Determine Sourcing Strategy ApendxiI

In the first phase of our outsourcing framework, the client organization 

determines whether internal capability or external expertise can more 

effectively meet its IT needs. The purpose of this sourcing strategy is to 

achieve the optimal balance between internal and external capabilities, 

activities, processes, and services to ensure that strategic business 

objectives are achieved at the lowest risk. Among the factors that an 

organization should evaluate in crafting this strategy are technology, 

business, financial, and personnel requirements and whether it has skilled 

business and IT managers. In addition, according to Gartner, Inc., sound 

sourcing decisions depend on whether IT organizations (1) know and 

understand their business priorities, (2) are prepared to invest in some 

skills and divest others, and (3) identify and assess trade-offs.1

The six practices in this phase that we used to evaluate the five projects are 

as follows:

• Use third-party assistance with experience in a variety of sourcing 

arrangements when formulating a sourcing strategy.

• Incorporate lessons learned from peers who have engaged in similar 

sourcing decisions.

• Estimate impact of sourcing decision on internal organization.

• Benchmark and baseline productivity of internal services before making 

the final sourcing decision.

1

Gartner, Inc., Five Tough Questions About Skill Sourcing, Research Note SPA-13-2537 

(Mar. 28, 2001). 
Appendix II

Projects’ Implementation of Phase I: 

Determine Sourcing Strategy

Page 33 GAO-03-371 DOD IT Services Outsourcing

• Determine the business reasons for outsourcing IT.

• Determine reasons for outsourcing IT that can improve the 

organization’s ability to use and manage technology.

Figure 6 shows that the implementation of the practices by the five projects 

in our review was uneven. 

Figure 6: Project Implementation of Phase I: Determine Sourcing Strategy

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent. 
Appendix II

Projects’ Implementation of Phase I: 

Determine Sourcing Strategy

Page 34 GAO-03-371 DOD IT Services Outsourcing

Table 4 provides detailed information on whether and how each project 

implemented each of the six practices in this phase. 

Table 4: Summary of Projects’ Use of Phase I Practices

Did the project use the practice?

Practice

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Use third-party 

assistance with 

experience in a 

variety of sourcing 

arrangements when 

formulating a 

sourcing strategy.

No—According to Air 

Force project 

officials, they 

decided that they 

had sufficient inhouse expertise, and 

the project had no 

funding available to 

employ a contractor.

However, the former 

functional area chief 

told us that using 

third-party 

assistance would 

have been beneficial 

because the 

requirements would 

have been better 

written. According to 

this official, the 

requirements had to 

be substantially 

rewritten 2 years 

after contract award.

No—According to a 

TAC-SWA project 

official, the Army 

decided that it had 

sufficient expertise 

in-house since the 

TAC-SWA contract 

was a consolidation 

of three existing 

contracts.

Yes—For example, 

MHS worked with the 

Department of the 

Interior’s GovWorks 

Program and the 

Defense Acquisition 

University staff to 

obtain expertise on 

sourcing strategies. 

Also, MHS employed 

a contractor with 

sourcing expertise. 

Yes—The Navy 

employed a 

contractor with 

sourcing expertise 

and contacted other 

government entities 

about their 

experiences.

Yes—NIMA 

employed a 

contractor with 

sourcing expertise.
Appendix II

Projects’ Implementation of Phase I: 

Determine Sourcing Strategy

Page 35 GAO-03-371 DOD IT Services Outsourcing

Incorporate lessons 

learned from peers 

who have engaged in 

similar sourcing 

decisions.

Yes—Primarily from 

other Department of 

Defense (DOD) 

projects that 

implemented Office 

of Management and 

Budget (OMB) 

Circular A-76 

policies.

Yes—According to a 

TAC-SWA project 

official, it used 

lessons learned from 

its prior contracts for 

these services and 

another similar Army 

contract.

Yes—Primarily from 

industry peers and 

MHS’s prior helpdesk function.

Yes—Navy officials 

stated that there 

were no peers that 

had engaged in 

similar sourcing 

decisions because 

no other outsourcing 

project was of as 

large a scale as 

NMCI. However, the 

Navy did talk to 

members of private 

industry and the 

National Aeronautics 

and Space 

Administration on 

their more limited 

efforts.

Yes—A NIMA 

contractor provided 

the agency with a 

report on industry 

best practices. In 

addition, NIMA held 

discussions with the 

National Security 

Agency on its 

outsourcing effort.

Estimate impact of 

sourcing decision on 

internal organization.

Yes—The Air Force 

estimated that there 

would be substantial 

internal impact, such 

as to its staff, due to 

its decision to 

outsource.

No—The Army did 

not analyze the 

impact on its internal 

organization 

because, according 

to the TAC-SWA 

program manager, 

NETCOM was 

outsourcing a 

function that was 

already contracted 

out.a

Yes—MHS assessed 

the staff and financial 

impact of its sourcing 

decision.

Yes—The Navy 

performed an 

assessment of 

staffing and other 

impacts, such as 

cost, related to its 

sourcing decision.

Yes—NIMA 

performed an 

assessment of 

staffing and other 

impacts, such as 

cost, related to its 

sourcing decision.

(Continued From Previous Page)

Did the project use the practice?

Practice

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix II

Projects’ Implementation of Phase I: 

Determine Sourcing Strategy

Page 36 GAO-03-371 DOD IT Services Outsourcing

Benchmark and 

baseline productivity 

of internal services 

before making the 

final sourcing 

decision.

No—The Air Force 

did not implement 

this practice 

because, according 

to project officials, an 

executive decision 

was made to 

outsource the C4 

services following the 

policies contained in 

OMB Circular A-76. 

Accordingly, Air 

Force officials stated 

that such analyses 

would not have 

affected the final 

sourcing decision.

No—The Army did 

not perform a 

benchmark or 

baseline analysis 

because, according 

to the TAC-SWA 

program manager, 

NETCOM was 

outsourcing a 

function that was 

already contracted 

out. 

Limited—MHS 

attempted to 

establish a baseline 

of its prior 

environment but, 

according to project 

officials, available 

documentation was 

limited to historical 

trouble ticket 

workload data; 

therefore, this 

baseline was a best 

estimate. In addition, 

MHS project officials 

stated that a 

contractor performed 

a benchmark 

analysis, but they did 

not provide 

supporting 

documentation.

Limited—The Navy 

had a private-sector 

firm benchmark its 

environment against 

seven large public 

and private 

organizations. In 

addition, the Navy 

and the Marine 

Corps performed an 

analysis at a sample 

of representative 

locations to obtain a 

baseline. However, 

this baseline did not 

include an 

assessment of the 

Department of the 

Navy’s legacy 

applications since 

project officials 

decided to rely upon 

an inventory 

developed in 

addressing the Year 

2000 challenge. The 

Navy subsequently 

found that it had 

substantially 

underestimated its 

number of legacy 

applications. 

According to NMCI 

program officials, this 

underestimation 

contributed to the 

transition period 

slipping from 2-1⁄2 

years to 3-1⁄2 years.

Yes—A NIMA 

contractor baselined 

the existing NIMA 

environment and 

benchmarked it to 

peers.

(Continued From Previous Page)

Did the project use the practice?

Practice

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix II

Projects’ Implementation of Phase I: 

Determine Sourcing Strategy

Page 37 GAO-03-371 DOD IT Services Outsourcing

Source: GAO.

a

Having had the activity previously performed by a contractor does not obviate the need to estimate the 

impact of a sourcing decision on the internal organization—there are still risks involved, such as the 

potential disruption of services during the transfer to the new contractor. In fact, the staff from the 

incumbent contractor did not transition to the TAC-SWA provider, and the provider had difficulty filling 

these slots within the schedule outlined in the contract.

b

The Federal Activities Inventory Reform Act of 1998 requires federal agencies to prepare and submit 

to OMB, by June 30 of each year, inventories of the commercial activities performed by federal 

employees.

Determine the 

business reasons for 

outsourcing IT. 

Yes—To achieve cost 

savings and to shift 

military personnel to 

other work.

Yes—To address 

shortages in military 

personnel to perform 

its mission.

Yes—To achieve 

increased 

productivity and 

customer satisfaction 

and to decrease 

costs.

Yes—To have private 

industry capitalize 

infrastructure 

improvements that 

were needed to 

quickly and securely 

share knowledge 

around the globe.

Yes—To obtain 

improved customer 

services and 

decreased costs. 

Also, this function 

was determined to 

be a commercial 

function under the 

Federal Activities 

Inventory Reform 

Act.b

Determine reasons 

for outsourcing IT 

that can improve the 

organization’s ability 

to use and manage 

technology.

No—According to 

Kirtland Air Force 

Base officials, the Air 

Force decided to 

hold a public/private 

competition following 

OMB Circular A-76 

policies; therefore, 

improving its ability 

to use and manage 

technology was not a 

factor in determining 

its outsourcing 

strategy.

No—According to 

TAC-SWA project 

officials, improving its 

ability to use and 

manage technology 

was not a factor in 

determining its 

outsourcing strategy.

Yes—To improve IT 

expertise and 

knowledge.

Yes—To improve, for 

example, system 

security, 

interoperability, 

reliability, and 

network response. 

Yes—To achieve 

better IT 

management 

performance.

(Continued From Previous Page)

Did the project use the practice?

Practice

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 38 GAO-03-371 DOD IT Services Outsourcing

Appendix III

Project’s Implementation of Phase II: Define 

Operational Model ApendxiI

Critical to the successful outsourcing relationship is an operational model 

for guiding the structure of the contract and the plans for transition. In 

defining the operational model, client organizations formalize executive 

leadership, team composition, client responsibilities, and operating 

relationships between the client and provider. The operational model helps 

the organization to compare its plans with the expectations that were set as 

the initial decision to outsource was made and to ascertain whether these 

plans will enable the organization to meet those objectives. An important 

aspect of the operational model is an explicit understanding of how the 

client organization plans to communicate its needs and provide feedback to 

the provider. In addition, communication between the business and IT 

offices within the client organization is always critical. This is particularly 

true in the case of outsourcing because the IT service provider is outside 

the client organization and disconnects are more likely to occur. Therefore, 

organizational processes to facilitate good communication are critical.

The 13 practices in this phase that we used to evaluate the five projects are 

as follows:

• Establish executive leadership for IT to facilitate the outsourcing 

initiative.

• Continually communicate/clarify outsourcing objectives, while 

correcting misinformation that affects the organization.

• Establish a core group of people who will be involved in all phases of 

outsourcing.

• Select a person involved in the negotiation of the contract to manage the 

outsourcing relationship. 
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 39 GAO-03-371 DOD IT Services Outsourcing

• Create and define a contract management structure with operational 

points of contact and managers.

• Define the role of internal IT managers and business leaders.

• Ensure that the right skills are in place to support the outsourcing 

relationship.

• Establish a point of contact high in the provider management structure 

for elevating provider performance concerns.

• Have provider establish an on-site support team to serve as liaison 

between client and provider.

• Train provider on client business environment and goals.

• Select or develop standard tools for managing the relationship.

• Use third-party assistance to take advantage of expertise from a variety 

of outsourcing arrangements in defining the operational model.

• Ensure that the provider management team has prior experience in the 

client’s field of business.

Figure 7 illustrates that the five projects in our review largely implemented 

the practices. 
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 40 GAO-03-371 DOD IT Services Outsourcing

Figure 7: Project Implementation of Phase II: Define Operational Model

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent.

Table 5 provides detailed information on whether and how each project 

implemented each of the 13 practices in this phase.
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 41 GAO-03-371 DOD IT Services Outsourcing

Table 5: Summary of Projects’ Use of Phase II Practices

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Establish executive 

leadership for IT to 

facilitate the 

outsourcing initiative.

Yes—The initiative 

was planned by a 

steering group made 

up of representatives 

from various major 

offices at Kirtland Air 

Force Base. 

Subsequent to 

contract award, the 

Air Force established 

a functional area 

chief to manage the 

initiative, and a 

Lieutenant Colonel 

was appointed to be 

responsible for this 

project. 

Yes—NETCOM used 

the existing 

leadership in its 

offices of operations 

and logistics to 

provide executive 

leadership. 

Yes—MHS used its 

existing leadership 

structure and 

processes. This 

structure includes a 

program executive 

office, steering 

committee, and 

program review 

board.

Yes—In 1999, the 

Navy established a 

program executive 

office for IT primarily 

to support the NMCI 

outsourcing effort. 

The Congress later 

directed the Navy to 

identify a single 

individual whose sole 

responsibility would 

be to oversee and 

direct the NMCI 

program. As a result, 

in February 2002, the 

Navy established the 

NMCI Director’s 

Office to take over 

responsibility for 

NMCI.

Yes—The NIMA 

enterprise 

transformation 

directorate formed a 

strategic sourcing 

office to provide 

executive oversight 

of the effort. 
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 42 GAO-03-371 DOD IT Services Outsourcing

Continually 

communicate/clarify 

outsourcing 

objectives, while 

correcting 

misinformation that 

affects organization.

Yes—The Air Force 

provided 

communication 

through, for example, 

briefings and 

meetings with 

employees.

Yes—According to 

TAC-SWA project 

officials, the 

applicable NETCOM 

officials were 

informed about the 

initiative and 

provided comments 

on the draft 

performance work 

statement. TAC-SWA 

project officials also 

brief new 

commanders on the 

contract before they 

are transferred to the 

military theater. 

Yes—MHS provided 

updates to the Chief 

Information Officer 

(CIO) and program 

review boards, 

provided briefings to 

the deputy surgeon 

general, and 

published questions 

and answers.

Yes—The Navy 

established an action 

collaboration team 

structure to involve 

the Navy community 

in the NMCI 

communications 

process. Also, the 

NMCI Information 

Bureau initiated 

press conferences, 

briefings, site visits, 

informational 

pamphlets, and 

promotional material. 

In addition, the NMCI 

Director and other 

staff provide 

numerous briefings 

and presentations to 

commands, industry 

gatherings, and 

government officials. 

Finally, Web sites 

and Web-based 

collaboration sites 

were established to 

facilitate 

communications.

Yes—NIMA created 

an internal Web site 

to post information 

on the outsourcing 

project, distributed 

periodic global 

electronic mails, and 

held town hall 

meetings.

Establish a core 

group of people who 

will be involved in all 

phases of 

outsourcing.

Yes—The core group 

included the 

contracting officer, 

functional area chief, 

and manpower 

specialist. 

Yes—According to 

TAC-SWA project 

officials, the core 

group included 

representatives from 

logistics, operations, 

and resource 

management as well 

as the contracting 

officer.

Yes—According to 

program officials, the 

core group included 

representatives from 

the MHS program 

executive office, 

MHS IT program 

offices, military 

department chief 

information officers’ 

offices, and Interior’s 

GovWorks 

organization, which 

provided contracting 

services. 

Yes—The program 

executive office for IT 

and the NMCI 

Director’s 

organization 

comprise the core 

group responsible for 

managing the 

outsourcing initiative.

Yes—NIMA 

established the 

Strategic Sourcing 

Office, which was 

dedicated to 

managing and 

facilitating the 

outsourcing initiative.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 43 GAO-03-371 DOD IT Services Outsourcing

Select a person 

involved in the 

negotiation of the 

contract to manage 

the outsourcing 

relationship. 

Yes—The contracting 

officer.a

Yes—The contracting 

officer. 

Yes—According to 

MHS program 

officials, the chair of 

the contract 

evaluation committee 

and the individual in 

charge of transition 

are responsible for 

managing the 

outsourcing 

relationship.

Yes—The primary 

contracting officer. 

Yes—A lead 

contracting officer.

Create and define 

contract 

management 

structure with 

operational points of 

contact and 

managers.

Yes—The structure 

was defined by the 

roles and 

responsibilities of the 

contracting officer 

and her staff. 

Yes—The overall 

responsibility for the 

contract rests with 

the contracting 

officer. The 

contracting officer’s 

representative acts 

as a liaison between 

the government and 

the contractor. 

Yes—Responsibility 

for contract 

management is 

jointly held by the 

contracting officer, 

who is part of 

Interior’s GovWorks 

organization, and the 

MHS program office, 

which provides dayto-day oversight of 

the provider.

Yes—The primary 

contracting officer 

establishes 

procedures and 

controls necessary 

for effective 

contractual oversight 

of the NMCI initiative 

and has a matrix 

relationship with the 

NMCI Director. The 

primary contracting 

officer certifies 

contracting officer 

representatives for 

the NMCI contract to 

provide technical 

coordination efforts. 

Yes—NIMA has 

appointed an 

operational point of 

contact for each of 

the seven functional 

areas being 

outsourced, which 

are documented in a 

“rules of 

engagement” 

agreement between 

the government and 

the contractor.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 44 GAO-03-371 DOD IT Services Outsourcing

Define the role of 

internal IT managers 

and business 

leaders.

Yes—Kirtland Air 

Force Base did not 

initially define the 

roles of its internal IT 

and business 

managers. However, 

subsequent to 

contract award, 

Kirtland Air Force 

Base established a 

CIO committee and 

configuration control 

board comprising the 

functional area chief 

(Kirtland’s CIO) and 

representatives from 

various business 

areas. Although not 

exclusively devoted 

to the C4 Services 

project, these groups 

help provide direction 

to the project. 

Limited—TAC-SWA 

project officials 

explained the roles of 

the principal staff 

involved with this 

project but generally 

did not have 

supporting 

documentation 

defining these roles.

Yes—MHS’s helpdesk performance 

assessment plan 

defines the roles of 

various groups and 

individuals 

associated with the 

program.

Yes—The NMCI 

execution plan 

describes the roles of 

internal and external 

organizations that 

directly or indirectly 

affect the 

management of 

NMCI and explains in 

detail the duties and 

responsibilities of the 

program executive 

office for IT and the 

NMCI program 

management offices.

Yes—NIMA 

transformation teams 

for the activities 

being outsourced 

define the roles of 

their managers and 

leaders. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 45 GAO-03-371 DOD IT Services Outsourcing

Ensure that the right 

skills are in place to 

support the 

outsourcing 

relationship, 

including those 

dealing with

• contract 

management, 

• financial 

management,

• IT management,

• negotiation 

strategies,

• teaming and 

interpersonal 

relationships, 

• project 

management, and 

• relationship 

management.

Yes—According to 

Air Force officials, 

the right skills are in 

place. 

Yes—According to 

the TAC-SWA project 

officials, the Army 

has created several 

positions to ensure 

that the right skills 

are in place to 

manage the 

outsourcing 

relationship, 

including a 

contracting officer, 

legal advisor, 

operations and 

logistics personnel 

as well as a 

NETCOM unit 

commander. 

According to Army 

TAC-SWA officials,

each position 

employs an 

individual with the 

necessary skills to 

support the 

outsourcing 

relationship. 

Yes—According to 

MHS program 

officials, the TriService Management 

Program Office, 

GovWorks (the 

Interior organization 

that provides 

contracting 

assistance to MHS), 

and a contractor 

collectively employ 

the skill sets needed 

to support the 

outsourcing 

relationship.

Limited—The 

Director, NMCI, has 

staff that report 

directly or are 

matrixed with him 

that are responsible 

for performing all but 

two of these 

functions. 

Specifically, at this 

time, NMCI does not 

have staff assigned 

to support teaming 

and interpersonal 

relationships and 

relationship 

management. The 

Navy recognizes the 

need for these skills 

and is taking, or 

plans to take, various 

actions to obtain 

these skills. 

Yes—According to 

NIMA, its 

transformation teams 

provide the skills 

necessary to support 

the outsourcing 

relationship.

Establish a point of 

contact high in the 

provider 

management 

structure for 

elevating provider 

performance 

concerns.

Yes—According to 

the contracting 

officer, concerns can 

be addressed to the 

senior vice president 

at the provider 

headquarters. 

Yes—The provider 

established a 

program manager as 

the point of contact 

for elevating 

concerns.

Yes—The provider 

established an 

executive program 

manager as the point 

of contact for 

elevating concerns.

Yes—The NMCI 

Director interacts 

directly with the 

provider’s program 

executive on a 

regularly scheduled 

basis; such meetings 

would include any 

performance issues.

Yes—The 

transformation team 

charters define 

procedures for 

addressing issues, 

including possibly 

elevating concerns to 

the provider’s 

general manager.

Have provider 

establish an on-site 

support team to 

serve as a liaison 

between client and 

provider.

Yes—The provider 

has established an 

on-site support team, 

which is led by the 

on-site manager.

Yes—Site managers 

have been 

established in 

accordance with the 

contract. 

Yes—The provider’s 

transition plan 

identifies the on-site 

support team. 

Yes—The provider 

establishes site leads 

that remain on-site 

as the location goes 

through the cutover 

to NMCI.

Yes—The rules of 

engagement define 

the on-site support 

team for each major 

NIMA location.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 46 GAO-03-371 DOD IT Services Outsourcing

Train provider on 

client business 

environment and 

goals.

Limited—Prior to 

contract award, 

Kirtland Air Force 

Base provided an 

orientation session in 

which they discussed 

their operations, 

business 

environment, and 

goals. Subsequent to 

contract award, 

specific Kirtland Air 

Force Base 

government 

personnel were 

responsible for aiding 

the provider during 

the transition period. 

However, provider 

officials stated that 

they did not receive 

adequate training, 

which made the 

transition period 

more difficult. 

Not applicable—The 

Army contract was a 

consolidation of prior 

contracts for this 

activity, and the 

winning bidder was 

one of the incumbent 

contractors.

Yes—MHS officials 

stated that they 

provided some 

training to the 

provider on their 

business 

environment. In 

particular, MHS 

trained and certified 

the provider’s staff on 

the agency’s 

application systems. 

In addition, the 

provider had 

previous MHS 

experience. 

Limited—The Navy 

did not provide 

training on its 

business 

environment and 

goals; instead it 

relied on the NMCI 

design reference 

mission document, 

which was included 

with the request for 

proposals (RFP). 

This document 

defined the NMCI 

operational 

environment. 

However, a provider 

official stated that 

although the 

document was 

useful, it did not 

identify Navy 

enterprisewide 

operations. 

According to NMCI 

program officials, the 

provider somewhat 

mitigated this 

problem by hiring 

several highly 

knowledgeable staff 

from the Department 

of the Navy shortly 

after contract award.

Not applicable—

According to the 

project’s contracting 

officer, such training 

was not necessary 

because the senior 

officers of the 

provider are all 

recent NIMA 

management 

employees and were 

already 

knowledgeable of 

these areas. For 

example, among 

those who 

transitioned to the 

provider was a 

former director of the 

hydrographic 

topographic center of 

the Defense Mapping 

Agency, a 

predecessor agency 

to NIMA.

Select or develop 

standard tools for 

managing the 

relationship (e.g., 

performance 

scorecards, 

enterprise resource 

management 

system).

Yes—For example, 

the revised contract 

outlines the use of 

specific software to 

help manage the 

provider’s 

performance.

Yes—NETCOM uses 

a monthly report 

from the contracting 

officer’s 

representative to 

evaluate the provider. 

Yes—According to 

MHS officials, the 

program office and 

the provider have 

selected, for 

example, an 

enterprise resource 

management 

system.

Yes—The Navy is 

using a balanced 

scorecard process to 

provide Navy and 

Marine Corps 

leadership with 

information to judge 

how well NMCI is 

supporting the 

missions and 

strategies of the 

department.

Yes—NIMA used 

various tools, 

including twice-a-day 

performance reports 

for the operational 

help-desk function 

and monthly 

performance reports.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix III

Project’s Implementation of Phase II: Define 

Operational Model

Page 47 GAO-03-371 DOD IT Services Outsourcing

Source: GAO.

a

The Air Force’s initial contract was not negotiated because Kirtland Air Force Base used an OMB 

Circular A-76 2-step, sealed bid process. However, the contracting officer was involved in the bid 

process as well as in managing the winning contractor.

b

This threshold could be met through multiple efforts, but at least one had to include at least 20,000 

seats.

Use third-party 

assistance to take 

advantage of 

expertise from a 

variety of outsourcing 

arrangements in 

defining the 

operational model 

(i.e., defining roles 

and responsibilities).

Yes—Although thirdparty assistance was 

not used at the onset 

of the C4 Services 

project, a person with 

prior experience in 

managing IT 

services contracts for 

the Air Force was 

brought in to provide 

advice. He was 

subsequently put in 

charge of the project.

No—The Army 

believed that it had 

sufficient expertise 

in-house. 

Yes—For example, 

MHS contacted the 

Interior’s GovWorks 

Program, Defense 

Acquisition 

University, and 

others. Also, MHS 

employed a 

contractor with 

sourcing expertise.

No—The Navy 

believed it had 

sufficient internal 

experience and 

expertise. 

Yes—NIMA 

employed 

contractors to 

develop contractual 

roles and assist in 

evaluating the 

contract.

Ensure that the 

provider 

management team 

has prior experience 

in the client’s field of 

business.

Yes—Provider 

management 

capabilities and 

experience in 

providing IT services 

were evaluation 

factors in the 

technical evaluation 

process. 

Yes—Vendor past 

performance was 

part of the evaluation 

criteria during source 

selection and the 

provider was an 

incumbent contractor 

with NETCOM.

Yes—The request for 

quote laid out MHS’s 

evaluation criteria for 

selecting a provider, 

which included prior 

experience in health 

care.

Yes—According to 

NMCI program 

officials, the Navy 

required bidders to 

provide information 

on their prior 

experience in related 

business fields. For 

example, the 

solicitation required 

bidders to 

demonstrate 

experience 

managing a similar 

effort of 100,000 or 

more seats at one 

time.b

Yes—It was expected 

that most of the 

provider’s workforce 

would be composed 

of former NIMA 

employees. Among 

those who 

transitioned to the 

provider was a 

former director of the 

hydrographic 

topographic center of 

the Defense Mapping 

Agency.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 48 GAO-03-371 DOD IT Services Outsourcing

Appendix IV

Project’s Implementation of Phase III: Develop 

the Contract ApendxiIV

Phase III focuses on the development of the contract, which defines the 

legal terms of the relationship between client and provider. While other 

phases in the outsourcing process describe the need for mutual trust and a 

close, flexible working relationship, this phase focuses on the development 

of the contract, which is the foundation on which a working relationship 

will be built. A well-written contract helps the outsourcing organization 

meet its requirements while allowing the service provider to make a fair 

profit. It sets the expectations for service levels, delivery of essential 

services, and continuous improvement and is intended to protect the 

interests of all parties. 

The 16 practices in this phase that we used to evaluate the five projects are 

as follows:

• Base performance requirements on business outcomes.

• Include measures that reflect end-user satisfaction as well as technical 

IT performance.

• Review and update performance requirements periodically.

• Require the provider to meet the minimum performance in each 

category of service.

• Require the provider to achieve escalating performance standards at 

agreed-upon intervals.

• Incorporate sufficient flexibility so that minimum acceptable 

performance can be adjusted as conditions change, as the provider 
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 49 GAO-03-371 DOD IT Services Outsourcing

becomes more adept at satisfying customer demands, and as 

improvement goals are achieved.

• Use service-level agreements (SLA)1

 to clearly articulate all aspects of 

performance, including management, processes, and requirements.

• Client and provider work together to define the appropriate number of 

SLAs and appropriate structure for each. 

• Specify circumstances under which the provider is excused from 

performance levels mandated by master service agreements.

• Client and provider work together to identify SLAs for which 

compensation is based, while additional SLAs may be defined to manage 

performance.

• The contract should include clauses for (1) determining pricing 

structures; (2) performing customer satisfaction surveys and using the 

results to redefine performance levels; (3) terminating the contract; (4) 

resolving disputes in a timely manner; (5) taking work away, without 

penalty, from provider for nonperformance; (6) declaring a significant 

event that can lead to a change in the contract; (7) defining performance 

requirements; and (8) conducting regularly scheduled meetings.

• Consider setting up a master services agreement under which all 

arrangements between client and provider operate.

• Include the appropriate representation from each major organizational 

unit on the contract negotiation team.

• Specify the use of volume purchases to obtain optimal discounts.

• Use third-party assistance in negotiating and developing the contract.

• Sign the contract after contract negotiations and final vendor selections.

As figure 8 shows, the five projects largely implemented the practices. 

1

SLAs define the agency’s expectations and are used to track and measure a contractor’s 

performance. 
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 50 GAO-03-371 DOD IT Services Outsourcing

Figure 8: Project Implementation of Phase III: Develop the Contract

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent.

Table 6 provides detailed information on whether and how each project 

implemented each of the 16 practices in this phase.
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 51 GAO-03-371 DOD IT Services Outsourcing

Table 6: Summary of Projects’ Use of Phase III Practices

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Base performance 

requirements on 

business outcomes.

Yes—The 

performance 

requirements are 

contained in the 

contract and are 

based on business 

outcomes. For 

example, the 

requirements define 

the percentage of 

system availability 

needed to support 

users.

Yes—The 

performance 

requirements are 

outcome based and 

stated in terms of the 

level of operation and 

maintenance 

services required. 

According to TACSWA officials, 

multiple trips to 

overseas locations 

were made to identify 

the performance 

requirements.

Yes—Industry 

standards and 

metrics were used to 

base performance 

requirements on 

business outcomes.

Yes—The 

performance 

requirements are 

included in the 

contract and are 

based on business 

outcomes. SLAs 

establish the 

performance 

standards and 

service quality for all 

types of NMCI seats.

Yes—Performancebased requirements 

are in the contract, 

including those 

related to customer 

satisfaction, process 

improvement, quality 

control, and timely 

and accurate 

completion of 

requirements.

Include measures 

that reflect end-user 

satisfaction as well 

as technical IT 

performance.

Limited—The Air 

Force measures 

include technical IT 

performance. 

However, end-user 

satisfaction 

measures are limited 

to the percentage of 

complaints received 

and do not measure 

overall customer 

satisfaction. Project 

officials did not know 

why overall end-user 

satisfaction 

measures were not 

established.

Limited—The Army’s 

contract includes 

technical IT 

performance 

measures but not 

end-user satisfaction 

performance 

measures. For 

example, the contract 

includes a 

requirement for helpdesk services, but 

does not include 

measures related to 

customer satisfaction 

associated with such 

services. According 

to the project official 

that developed the 

performance work 

statement in the 

contract, NETCOM 

did not include such 

customer satisfaction 

measures because it 

did not think that it 

was necessary to 

have a performance 

standard for that 

service.

Yes—MHS measures 

the service provider’s 

technical IT 

performance and 

uses an electronic 

customer satisfaction 

survey to assess the 

quality of help-desk 

services. Satisfaction 

rates and number of 

survey responses 

are tracked and used 

for both evaluation 

and incentive 

payments (positive 

and negative).

Yes—The SLAs 

include measures for 

technical IT 

performance and 

customer 

satisfaction. In 

addition, the contract 

requires the service 

provider to measure 

and report on overall 

customer satisfaction 

with NMCI services. 

Yes—The SLAs 

contain technical IT 

performance 

measures. In 

addition, after a 

function is 

transitioned, the 

contractor must 

provide a plan that 

includes measures 

for customer 

satisfaction, which 

the provider is 

contractually 

obligated to meet. 
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 52 GAO-03-371 DOD IT Services Outsourcing

Review and update 

performance 

requirements 

periodically.

Yes—The 

performance 

requirements were 

revised about 2 years 

after contract award.

Yes—The contract is 

reviewed and 

modified as needed 

on the basis of 

additions or changes 

to such 

requirements. 

According to TACSWA and provider 

officials, changes are 

coordinated between 

the government and 

the contractor before 

they are finalized. 

Yes—According to 

MHS program 

officials, performance 

requirements are 

updated as required 

and reviewed on an 

annual basis before 

the decision is made 

to execute the option 

year on the contract. 

Yes—Performance 

reviews and 

adjustments are 

ongoing. For 

example, the Navy is 

in the process of 

refining its SLAs to 

ensure that they 

adequately reflect 

technical 

performance and 

customer satisfaction 

needs.

Yes—According to 

NIMA, requirements 

are updated as 

needed or once a 

year when the 

government is 

assessing whether to 

exercise its annual 

option.

Require the provider 

to meet the minimum 

performance in each 

category of service.

Yes—The minimum 

performance is 

defined in the 

contract.

Yes—The contract 

defines minimum 

performance 

requirements.

Yes—The minimal 

acceptable 

performance criteria 

the contractor must 

meet are based upon 

commercial industry 

standards and are 

defined in the 

contract.

Yes—The SLAs 

provide the basic 

level of service the 

contractor must 

deliver for every 

NMCI seat. 

Yes—The provider 

must meet the 

performance 

standards as 

specified in the 

contract, SLAs, and 

NIMA’s performance 

assessment plan.

Require the provider 

to achieve escalating 

performance 

standards at agreedupon intervals.

No—According to 

Kirtland Air Force 

Base officials, they 

did not include 

escalating 

performance 

standards because 

the focus of this 

contract was on 

meeting 

requirements at the 

least cost.a

Not applicable—

According to a TACSWA project official, 

this practice was not 

applicable because 

NETCOM believed 

that the performance 

standards in the 

contract were 

already high. 

Provider officials also 

stated that the 

performance 

requirements in the 

contract were high. 

The Army expected, 

and the contractor 

agreed, to meet 

these standards 

immediately. 

Yes—The contract 

defines negative, 

acceptable, and 

positive incentive 

ranges associated 

with escalating 

performance 

standards.

No—According to 

NMCI program 

officials, the Navy 

expected vendors to 

develop their pricing 

proposal assuming 8 

yearsb of providing 

service in 

accordance with the 

SLAs included in the 

RFP.

Yes—The contract 

allows for the 

redefinition of service 

levels. In addition, 

according to NIMA 

program officials, the 

need for escalating 

performance 

standards will be 

evaluated on an asneeded basis as well 

as during the 

semiannual award 

fee analysis period 

and during the 

annual review on 

whether to exercise 

the contract option. 

However, such 

escalating standards 

have not yet been 

identified.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 53 GAO-03-371 DOD IT Services Outsourcing

Incorporate sufficient 

flexibility so that 

minimum acceptable 

performance can be 

adjusted as 

conditions change, 

as the provider 

becomes more adept 

at satisfying 

customer demands, 

and as improvement 

goals are achieved.

No—The contracting 

officer said that the 

requirements should 

be built into the 

contract, which can 

be modified if 

necessary. 

Not applicable—

According to a TACSWA project official, 

NETCOM believed 

that the performance 

standards in the 

contract were 

already high, and 

provider officials 

agreed. According to 

the contracting 

officer, the military 

theater where 

performance is 

delivered is not an 

area where anything 

less than meeting the 

defined standards is 

acceptable. 

Yes—According to 

MHS program 

officials, the contract 

incorporates 

sufficient flexibility to 

adjust minimal 

acceptable 

performance as 

conditions change. 

For example, the 

contract includes 

incentives for the 

service provider to 

introduce new 

capabilities and new 

services within the 

scope of the 

performance work 

statement without 

further competition.

Yes—The contract 

includes a provision 

requiring the provider 

to submit an annual 

plan for technology 

refreshment and 

deployment. This 

plan is to include 

proposed revisions to 

the contract and an 

estimate of changes 

in performance that 

would result. 

Yes—According to 

NIMA, the 

performance 

measures are 

designed to first 

stabilize NIMA’s IT 

environment and 

then to be adjusted 

to bring minimum 

performance into 

alignment with 

industry best 

practices. Also, the 

contract indicates 

that the SLAs may be 

further defined 

throughout the 

contract life.

Use SLAs to clearly 

articulate all aspects 

of performance, 

including 

management, 

processes, and 

requirements.

Yes—The SLA 

defines the 

requirements, 

processes, and who 

is responsible for 

meeting the 

requirements.

Yes—The TAC-SWA 

project did not use 

SLAs, but the 

contract addresses 

the management, 

processes, and 

requirements 

associated with the 

project. 

Yes—The contract 

defines the 

requirements, 

service performance 

standards, 

thresholds, 

objectives, and 

metrics as well as the 

requirements for 

quarterly 

management reviews 

and quality 

assurance plans.

Yes—According to 

NMCI officials, SLAs 

cover all aspects of 

provider 

performance. For 

example, the E-mail 

services SLA 

includes location and 

frequency of the 

service, performance 

categories, the 

performance 

measures, and 

methods of 

measurement. 

Yes—The contract 

incorporates SLAs, 

which articulate 

performance 

requirements and 

take effect when the 

function is 

transitioned to the 

provider.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 54 GAO-03-371 DOD IT Services Outsourcing

Client and provider 

work together to 

define the 

appropriate number 

of SLAs and 

appropriate structure 

for each. 

Yes—Although the 

original SLA was 

prepared by the Air 

Force, the revised 

SLA included 

contractor input and 

concurrence. 

Yes—SLAs are not 

used but, according 

to TAC-SWA and 

provider officials, 

they work together 

on any contract 

modifications dealing 

with requirements 

and performance 

levels. In addition, 

the partnering clause 

in the contract 

emphasizes a mutual 

commitment 

between government 

and industry to work 

as a team.

Yes—MHS did not 

work with the service 

provider to determine 

the number and 

structure of these 

requirements. 

However, according 

to MHS and provider 

officials, they have 

worked together on 

revisions to these 

requirements.

Yes—The SLAs were 

developed by the 

Navy with the help of 

a third-party 

contractor. According 

to the NMCI Deputy 

Director for 

Enterprise 

Operations, it would 

have been 

inappropriate to work 

with individual 

competing 

contractors before 

the contract was 

awarded. However, 

NMCI staff and the 

service provider are 

now working together 

on SLA revisions. 

Yes—According to 

NIMA and service 

provider officials, 

they used a 

partnering 

contracting approach 

to jointly define, 

develop, and 

structure the SLAs.

Specify 

circumstances under 

which the provider is 

excused from 

performance levels 

mandated by master 

service agreements.

Yes—The contract 

includes a clause 

that the contractor is 

excused in the event 

of government delay 

of work.

Yes—The contract 

includes clauses that 

the provider is 

excused from 

performance levels if 

there are 

government delays 

or factors beyond its 

control.

Yes—For example, 

the provider’s 

proposal, which is 

incorporated as part 

of the contract, 

states that during 

transition periods, 

metrics will not be 

reported for incentive 

and penalty 

purposes. As a 

result, according to 

program officials, the 

provider would be 

excused from 

required 

performance levels 

when baseline 

requirements are 

being established for 

new applications.

Yes—Contract terms 

and conditions 

specify the situations 

where the service 

provider does not 

have to meet the 

SLAs. For example, 

during transition, the 

service provider 

does not have to 

meet the 

performance levels 

set by the SLAs.

Yes—The contractor 

can be excused from 

meeting performance 

levels with the 

permission of the 

contracting officer if 

circumstances occur 

beyond the service 

provider’s control. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 55 GAO-03-371 DOD IT Services Outsourcing

Client and provider 

work together to 

identify SLAs for 

which compensation 

is based, while 

additional SLAs may 

be defined to 

manage 

performance.

Yes—According to 

Kirtland Air Force 

Base officials, the 

service provider had 

input on the revised 

SLAs, including 

those for 

compensation.

Not applicable—

SLAs are not used. 

Yes—Before the 

contract was 

awarded, MHS did 

not work with the 

contractor to identify 

SLAs for which 

compensation is 

based. However, 

according to MHS 

and provider officials, 

after award, there 

have been instances 

in which they have 

worked together on 

refining the SLAs, 

including those 

affecting 

compensation.

Yes—Meetings were 

held with potential 

bidders to obtain 

their input in defining 

the SLAs, including 

discussions of 

compensation. In 

addition, Navy and 

the provider are now 

refining the SLAs, 

including those 

affecting 

compensation.

Yes—NIMA and the 

service provider work 

together in 

transformation teams 

to develop SLAs, 

including those 

affecting 

compensation.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 56 GAO-03-371 DOD IT Services Outsourcing

The contract should 

include clauses for

• determining pricing 

structures; 

• performing 

customer 

satisfaction surveys 

and using the 

results to redefine 

performance levels;

• terminating the 

contract, including 

early terminations; 

• resolving contract 

disputes in a timely 

manner; 

• taking work away, 

without penalty, 

from the provider 

for 

nonperformance; 

• declaring a 

significant event 

that can lead to a 

change in the 

contract; 

• defining 

performance 

requirements; and 

• conducting 

regularly scheduled 

meetings.

Limited—All clauses 

are included in the 

contract, except a 

clause pertaining to 

customer satisfaction 

surveys.

Yes—All contract 

clauses are included.

Yes—All contract 

clauses are included.

Yes—All contract 

clauses are included.

Yes—All contract 

clauses are included.

Consider setting up 

master services 

agreement under 

which all 

arrangements 

between client and 

provider operate.

Yes—According to 

Kirtland Air Force 

Base officials, the 

contract, which 

incorporates the 

contractor proposal, 

governs all 

arrangements 

between the 

government and the 

contractor and is 

considered the 

master services 

agreement.

Yes—According to 

TAC-SWA project 

officials, the contract 

is considered the 

master services 

agreement.

Yes—According to 

MHS program 

officials, the contract 

is considered the 

master services 

agreement.

Yes—According to 

NMCI program 

officials, the contract 

is considered to be a 

master services 

agreement.

Yes—According to 

NIMA, the contract is 

considered the 

master services 

agreement.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 57 GAO-03-371 DOD IT Services Outsourcing

Include appropriate 

representation from 

each major 

organizational unit on 

contract negotiation 

team.

Not applicable—The 

C4 Services contract 

was awarded under a 

sealed bid process 

and was not 

negotiated. 

Yes—

Representatives from 

the field and 

headquarters 

commands and the 

contracting office 

participated in 

contract negotiation. 

Yes—Staff from 

affected program 

management offices 

and the chief 

information offices 

were on the 

negotiation team.

Yes—The contracting 

team was built with 

experts from each of 

the major systems 

commands. Also, the 

source selection 

evaluation board 

consisted of more 

than 50 people from 

various commands.

Yes—The NIMA 

contract negotiating 

team included 

representatives from 

each major 

organizational unit.

Specify the use of 

volume purchases to 

obtain optimal 

discounts.

Not applicable—

According to Kirtland 

Air Force Base 

officials, such volume 

purchases were not 

relevant to this 

contract.

Not applicable—

According to TACSWA project officials, 

such volume 

purchases were not 

relevant to this 

contract.

Yes—Call bandsc

 are 

used to obtain 

optimal discounts on 

the number of calls 

being fielded to the 

help desk.

Yes—The contract 

includes volume 

discounts. For 

example, seat 

moves, adds, and 

changes are 

purchased in bulk 

only, because prices 

for these actions are 

lower when 

purchased in groups 

of 250.

Yes—NIMA has 

authorized the 

provider to use 

government sources, 

such as the Federal 

Supply Schedule, in 

procuring products, 

services, and 

supplies related to 

this contract. 

According to the 

contracting officer, 

this authorization 

was made so that the 

provider could take 

advantage of 

discounts available to 

the government. 

Use third-party 

assistance in 

negotiating and 

developing the 

contract.

No—This was not 

done because the Air 

Force staff believed 

that it had adequate 

expertise available 

in-house. 

No—NETCOM 

officials believed that 

the Army’s 

designated 

contracting agency 

had adequate 

expertise and 

experience.

Yes—Private-sector 

and other 

government 

organizations 

provided assistance.

Yes—The Navy used 

private-sector firms 

to assist in 

developing the 

overall NMCI concept 

and negotiation 

strategies as well as 

in drafting the 

contract documents. 

Yes—NIMA obtained 

help from (1) private 

contractors to help 

develop the 

performance work 

statement, SLAs, 

and award fee plan; 

(2) another agency 

on a particular 

contract technique; 

and (3) a privatesector firm to 

compare the 

vendor’s proposal 

with industry best 

practices and trends 

and to attend some 

negotiation meetings 

to answer questions. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix IV

Project’s Implementation of Phase III: 

Develop the Contract

Page 58 GAO-03-371 DOD IT Services Outsourcing

Source: GAO.

a

Requiring a provider to meet escalating requirements and focusing on achieving results at the least 

cost are not mutually exclusive goals. As the provider becomes more familiar with the client 

organization, it may be able to exceed the original performance requirements at the same, or possibly 

lower, cost. 

b

The NMCI contract was subsequently extended to 7 years, with an option for an additional 3 years.

c

Call bands are call volume ranges used to determine contractor pricing.

Sign contract after 

contract negotiations 

and final vendor 

selections. 

Yes—The contract 

was signed after 

negotiations and final 

vendor selection.

Yes—The contract 

was signed after 

negotiations and final 

vendor selection. 

Yes—The contract 

was signed after 

negotiations and final 

vendor selection.

Yes—The contract 

was signed after 

negotiations and final 

vendor selection.

Yes—The contract 

was signed after 

negotiations with the 

selected vendor.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 59 GAO-03-371 DOD IT Services Outsourcing

Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s) ApendxiV

Critical to the success of any project to outsource IT services is the 

identification of potential providers and the ultimate selection of a 

provider(s) that will best meet the needs of the organization. Developing a 

strategy that will lead to the selection of the “right contractor” is especially 

important in a performance-based acquisition.1

 The overall success of the 

outsourcing project requires the contractor to understand the 

performance-based approach, know or develop an understanding of the 

organization’s requirement, have a history of performing exceptionally in 

the field, and have the processes and resources in place to support the 

mission. 

The seven practices in this phase that we used to evaluate the five projects 

are as follows:

• Conduct research on the state of the market, vendors, and technology 

before defining vendor selection criteria.

• Identify and evaluate various sourcing solutions (e.g., single vendor, 

multivendor, and alliance).

• Define a process for selecting vendors to be providers.

• Define vendor selection and evaluation (acceptance) criteria at the 

outset.

1

An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based 

Services Acquisition, Benchmark Version (January 2002).
Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s)

Page 60 GAO-03-371 DOD IT Services Outsourcing

• When issuing an RFP, identify services with expected performance 

levels and define client and provider roles and responsibilities.

• Use third-party assistance with expertise in a variety of outsourcing 

arrangements when selecting provider(s), including developing the RFP.

• Conduct due diligence activities to verify vendor capabilities before 

signing the contract.

As figure 9 illustrates, all five projects largely implemented the practices.

Figure 9: Project Implementation of Phase IV: Select the Provider(s)

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 
Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s)

Page 61 GAO-03-371 DOD IT Services Outsourcing

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent.

Table 7 provides detailed information on whether and how the projects in 

our review implemented each of the seven practices in this phase. 

Table 7: Summary of Projects’ Use of Phase IV Practices

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Conduct research on 

state of the market, 

vendors, and 

technology before 

defining vendor 

selection criteria.

Yes—The Air Force 

advertised in the 

Commerce Business 

Daily and interested 

parties submitted 

statements of 

capabilities. This was 

done before the Air 

Force identified the 

vendor selection 

criteria.

Yes—A Commerce 

Business Daily notice 

was published 

identifying TAC-SWA 

requirements to 

interested parties. In 

addition, a draft RFP 

was issued before 

the final solicitation to 

solicit comments 

from industry that 

might affect the 

requirements. 

According to the 

contracting officer, 

comments received 

from industry were 

incorporated into the 

final solicitation, 

where applicable. 

Additionally, a 

preproposal 

conference was 

conducted in the 

overseas operations 

location to inform 

industry 

representatives 

about issues and the 

procurement 

strategy. 

Yes—MHS had a 

third-party contractor 

perform market 

research.

Yes—The Navy and a 

private-sector firm 

performed market 

research to, for 

example, help define 

the market conditions 

and vendor selection 

criteria. The Navy 

also held meetings 

with leading 

companies that had 

outsourced IT 

services on an 

enterprisewide level. 

Finally, more than 

200 companies 

participated in the 

July 1999 NMCI 

Industry Day 

Conference, which 

informed companies 

about the NMCI 

vision, requirements, 

and procurement 

strategy.

Limited—NIMA staff 

and a contractor 

researched 

preferential providers 

before defining 

vendor selection 

criteria. NIMA 

program officials 

noted that they 

limited their analysis 

to such providers 

because they had 

previously decided 

on a strategy to 

directly convert their 

IT/IS activities to a 

preferential provider. 
Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s)

Page 62 GAO-03-371 DOD IT Services Outsourcing

Identify and evaluate 

various sourcing 

solutions (e.g., 

single-vendor, 

multivendor, and 

alliance).

Yes—As outlined in 

OMB Circular A-76, 

the Air Force 

evaluated privatesector and internal 

government staff 

proposals to satisfy 

the C4 services 

requirements.

Yes—A multivendor 

approach has been 

used in the past, 

which the Army 

opted not to 

continue. Instead, 

NETCOM chose to 

consolidate its 

requirements and 

contractor oversight 

by choosing a singlevendor solution.

Yes—According to an 

MHS program 

official, MHS 

evaluated continuing 

with the status quo, 

using another federal 

agency’s help-desk 

services as part of a 

cross-services 

agreement, and 

contracting with a 

commercial firm. 

According to the 

MHS program office, 

MHS chose a singlevendor solution to 

consolidate 

requirements and 

performance 

accountability. 

Yes—The Navy 

evaluated singlevendor and 

multivendor 

approaches. A joint 

Navy and Marine 

Corps team 

determined that 

single-vendor point of 

contact for 

accountability and 

responsibility was 

critical to mission 

success.

Yes—NIMA 

considered the 

sourcing solutions 

allowed under OMB 

Circular A-76, 

including the direct 

conversion to a 

preferential provider, 

holding a 

public/private 

competition, or 

obtaining an agency 

cost comparison 

waiver. NIMA 

decided that directly 

converting the 

selected IT functions 

to a preferential 

procurement sourcea

was the lowest risk to 

its mission and NIMA 

employees.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s)

Page 63 GAO-03-371 DOD IT Services Outsourcing

Define a process for 

selecting vendors to 

be providers (e.g., 

issuing an RFP and 

prequalifying 

vendors).

Yes—As explained in 

the invitation to bid, 

the Air Force 

followed the two-step 

process outlined in 

OMB Circular A-76 

for public/private 

competitions. 

Yes—The contract 

was awarded on 

“best value” 

considerations of 

technical and 

management 

capabilities, past 

performance, and 

price, as defined in 

the solicitation. 

Yes—MHS used a 

selection process 

that included the 

prequalification of 13 

industry leaders in 

both medical and 

commercial helpdesk operations. 

MHS also issued a 

request for comment 

and a request for 

quote, which defined 

vendor evaluation 

and selection criteria 

on the basis of 

technical approach, 

past performance, 

key personnel 

qualifications, 

organizational 

experience, and 

price.

Yes—The NMCI RFP 

used a negotiated 

commercial items 

evaluation process. 

The vendor 

evaluation criteria 

included technical 

approach, 

management plan, 

small business 

utilization, past 

performance, and 

price.

Limited—NIMA’s 

decision to use a 

phased direct 

conversion of its 

IT/IS functions to a 

statutory, preferential 

procurement vendor 

limited its vendor 

selection process to 

the identification of 

potential Alaska 

Native Corporation 

companies capable 

of performing the 

requirements. NIMA 

decided to take this 

approach because it 

believed that it would 

avoid schedule 

delays and mission 

risk that it thought 

would result from 

taking other 

approaches. In 

addition, NIMA 

program officials 

believed that the 

preferential provider 

approach would 

maximize the 

retention of 

institutional 

knowledge whether 

employees stayed at 

NIMA or transitioned 

to the provider.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s)

Page 64 GAO-03-371 DOD IT Services Outsourcing

Define vendor 

selection and 

evaluation 

(acceptance) criteria 

at the outset.

Yes—The criteria 

were included in the 

invitation to bid.

Yes—The solicitation 

included vendor 

evaluation and 

selection criteria. The 

source selection 

evaluation plan 

describes the 

evaluation process.

Yes—Vendor 

evaluation and 

selection were based 

upon industrydefined help-desk 

criteria. MHS used a 

multistep process 

that included an 

assessment of 

minimum 

qualifications, 

evaluation of written 

technical and cost 

proposals, and oral 

presentations for 

qualified vendors.

Yes—The final RFP 

required bidders to 

have implemented 

and provided service 

to at least 100,000 

seats, of which 

20,000 were to be on 

the same contract. In 

addition, bidders 

were evaluated on 

their technical 

approach, 

management plan, 

small business 

utilization, past 

performance, and 

price. Finally, vendors 

in the competitive 

range had to provide 

demonstrations of 

technical network 

operations centers to 

verify their ability to 

achieve required 

service levels set 

forth in technical 

proposals.

Yes—NIMA 

evaluated the past 

performance and 

financial capabilities 

of Alaska Native 

Corporation 

companies. 

According to NIMA, 

only two companies 

had demonstrated 

successful 

performance in 

service environments 

similar to NIMA’s 

requirements. 

However, to be able 

to fully meet NIMA’s 

requirements, these 

vendors formed a 

joint venture, which 

was awarded the 

contract.

When issuing an 

RFP, identify services 

with expected 

performance levels 

and define client and 

provider roles and 

responsibilities.

Yes—The services 

and performance 

levels were included 

in the original 

performance work 

statement included 

in the invitation to 

bid.

Yes—The RFP 

identified the 

government and 

contractor roles and 

responsibilities, and 

system operational 

and availability 

requirements. 

Yes—The request for 

quote identified 

services and 

expected 

performance levels 

and specified a 

performance-based, 

incentivized, sharedrisk relationship with 

the service provider. 

It also defined client 

and provider roles 

and responsibilities. 

Yes—The NMCI 

solicitation 

documents identified 

the required services 

and expected 

performance levels. 

These documents, 

along with the 

contract, define the 

client and provider 

roles and 

responsibilities.

Yes—Because this 

was a sole source 

contract, an RFP was 

not issued. However, 

according to NIMA 

and provider officials, 

they worked jointly to 

refine the 

requirements and 

expected 

performance levels 

and to define client 

and provider roles 

and responsibilities 

in the contract.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix V

Projects’ Implementation of Phase IV: Select 

the Provider(s)

Page 65 GAO-03-371 DOD IT Services Outsourcing

Source: GAO.

a

Preferential procurement programs are special commercial source programs, such as Federal Prison 

Industries and the workshops administered by the Committee for the Purchase from the Blind and 

Other Severely Handicapped under the Javis-Wagner O’Day Act.

Use third-party 

assistance with 

expertise in a variety 

of outsourcing 

arrangements when 

selecting provider(s), 

including developing 

the RFP.

No—This was not 

done because the Air 

Force believed that 

adequate in-house 

expertise was 

available.

No—The Army 

believed that it had 

adequate 

government 

experience and 

expertise covering 

technical, resource 

management, and 

contracting areas.

Yes—A privatesector contractor, the 

Defense Acquisition 

University, and 

GovWorks (an 

Interior organization 

offering procurement 

services to 

government 

agencies) provided 

assistance in this 

area. 

Yes—A private-sector 

firm assisted with 

market research and 

developing vendor 

pass/fail criteria. 

Another contractor 

assisted the Navy 

with developing SLAs 

and the technical 

evaluation of 

vendors. 

Yes—NIMA used a 

private-sector firm 

and the NIMA 

Acquisition Center to 

support the 

evaluation of the 

vendor proposal.

Conduct due 

diligence activities to 

verify vendor 

capabilities before 

signing the contract.

Yes—The Air Force 

performed a 

technical evaluation 

of the vendor’s 

capabilities. Also, as 

part of its due 

diligence activities, 

the Air Force 

reviewed the 

debarred list and 

DOD’s Central 

Contractor Registry.

Yes—NETCOM 

evaluated the 

contractor’s financial 

and past 

performance 

information before 

signing the contract. 

Yes—Minimum 

vendor qualifications 

were established and 

evaluated early in the 

process for selecting 

the provider.

Yes—As part of due 

diligence, the Navy 

ensured that all 

bidders had relevant 

experience 

implementing large 

seat management 

contracts. In addition, 

past performance, 

including reference 

checks, was a 

source-selection 

evaluation factor.

Yes—NIMA 

researched both 

parent companies of 

the new joint venture 

corporation to ensure 

that they had 

financial and 

technical backing. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 66 GAO-03-371 DOD IT Services Outsourcing

Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s) ApendxiVI

In this phase, client organizations transfer responsibility of IT functions to 

one or more providers. A clear definition of responsibilities among the 

various parties and the careful consideration of employees’ needs matched 

against the organization’s needs enable both the client and provider to 

focus on execution and give staff confidence in their future employment. If 

the contractor is assuming responsibility for functions previously 

performed by federal employees, it is especially important that the 

organization communicate a clear transition process. Without such 

communication, an outsourcing project can be negatively affected if 

misinformation and mistrust ensues. 

The 11 practices in this phase that we used to evaluate the five projects are 

as follows:

• Communicate a clear transition process to all key players from both 

client and provider organizations.

• Clearly communicate to employees what is going to happen and when it 

is going to happen.

• Establish a client transition team with representatives from across the 

organization to facilitate the transition.

• Place the transition under a single program manager.

• Create client/provider transition teams to address short-term transition 

tasks as required.

• Recognize that it takes time to effect transition and plan accordingly.
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 67 GAO-03-371 DOD IT Services Outsourcing

• Encourage the transition of staff to the provider, where appropriate, 

using bonuses, stock options, and other appropriate methods.

• Develop employee-retention programs and offer bonuses to keep key 

people, where appropriate.

• When consistent with organizational objectives, assist employees who 

do not want to transfer in finding other jobs, either within an 

organization or at another organization.

• Document key information to preserve organizational knowledge in the 

event that one or more providers change.

• Use change management strategies to help client employees deal with 

the transition.

Figure 10 demonstrates that the five projects in our review were largely 

using the practices.
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 68 GAO-03-371 DOD IT Services Outsourcing

Figure 10: Project Implementation of Phase V: Transition to Provider(s)

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent.

Table 8 provides information on whether and how the projects 

implemented each of the 11 practices in this phase. 
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 69 GAO-03-371 DOD IT Services Outsourcing

Table 8: Summary of Projects’ Use of Phase V Practices

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Communicate a clear 

transition process to 

all key players from 

both client and 

provider 

organizations.

No—There were two 

transition plans 

prepared, one by the 

government and one 

by the provider. 

However, the 

government’s plan 

contained conflicting 

direction. In addition, 

provider officials said 

that the two separate 

transition plans were 

not well coordinated 

and that the Air 

Force did not have a 

strong advocate to 

ensure that the 

transition process 

was well planned and 

executed. Both 

provider and Kirtland 

Air Force Base 

officials 

acknowledged that 

there were problems 

with the transition 

process, including 

incorrect information 

about upgrades that 

would be made 

before the transition 

and staff morale 

problems that 

hampered 

knowledge transfer to 

the provider. 

Yes—The provider 

developed a phase-in 

plan that addressed 

personnel issues, the 

integration of added 

locations, updating 

plans, and finalizing 

new subcontract/

sponsorship 

arrangements. This 

phase-in plan was 

discussed at a 

postaward 

conference, attended 

by representatives 

from both the client 

and provider.

Yes—The contract 

delivery order and 

the provider’s 

transition plan laid 

out a transition 

process. In addition, 

a kickoff meeting was 

held between the 

client and provider to 

communicate the 

transition process to 

all key players.

Yes—The Navy 

developed numerous 

guides and Web sites 

to communicate with 

the NMCI community 

about site, technical, 

and personnel 

transition processes, 

procedures, and 

tasks.

Yes—Several NIMA 

transformation teams 

were formed, 

consisting of both 

client and provider 

personnel, to develop 

a single transition 

plan. The plan was 

developed to 

formalize 

communications with 

all key players.
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 70 GAO-03-371 DOD IT Services Outsourcing

Clearly communicate 

to employees what is 

going to happen and 

when it is going to 

happen.

Yes—The Air Force 

kept its employees 

informed of the 

outsourcing project 

through town hall 

meetings and 

internal briefings. As 

part of this process, 

employees were 

informed about the 

reduction-in-force 

process and the 

procedure for those 

interested in being 

transitioned or 

reassigned.

Limited—According 

to TAC-SWA officials, 

the Army 

communicated with 

its employees, but 

they did not provide 

documentation 

supporting any 

communication 

efforts. However, the 

provider was 

required to submit a 

phase-in plan, which 

included a section on 

communicating with 

the incumbent 

contractors’ staff. 

Yes—The 

government task 

manager for the TriService 

Infrastructure 

Management Office 

held regular 

meetings with MHS 

program officials to 

discuss transition 

issues. The MHS 

program office also 

periodically sent 

global E-mails to 

MHS staff on 

transition activities 

and MHS leadership 

made on-site visits to 

San Antonio, the 

location of the helpdesk function.

Limited—The Navy 

has used its normal 

chain of command to 

communicate 

transition 

information, but 

found that the 

implementation was 

uneven. As a result, 

some staff did not 

know current 

information about 

how NMCI would 

affect them until the 

provider was ready to 

contact them 

regarding their 

possible transition to 

the contractor. 

However, according 

to the NMCI 

Director’s office, this 

problem was 

somewhat mitigated 

by the provider’s 

Web site that 

provides transition 

information to all 

NMCI 

customers/users.

Yes—With help from 

a contractor, NIMA 

developed a 

communications 

plan. The plan 

included town hall 

meetings, global 

E-mail messages, 

and a Web site 

posting that provided 

employees with 

access to transition 

information.

Establish client 

transition team with 

representatives from 

across the 

organization to 

facilitate the 

transition.

Yes—The 

government’s 

transition team 

included personnel 

from the functional 

areas being 

outsourced (i.e., 

communications and 

IT services) as well 

as personnel from 

other offices.

Yes—According to 

TAC-SWA officials, 

the contracting 

officer’s 

representative 

assisted the provider 

with the transition. 

The officials also 

stated that Army 

technical points of 

contact at each site 

also helped facilitate 

the transition to the 

new contract.

Yes—MHS program 

officials for the TriService 

Infrastructure 

Management Office 

established transition 

teams with 

representatives from 

the former TriService Medical 

System Support 

Center contract.

Yes—The 

headquarters-level 

transition team 

consists of 

representatives from 

the NMCI Director’s 

Office and the 

program 

management office. 

Also, every site has a 

transition team made 

up of customer and 

provider personnel.

Yes—NIMA’s 

transition team 

included 

representatives from 

various offices within 

NIMA to help 

facilitate the 

transition.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 71 GAO-03-371 DOD IT Services Outsourcing

Place transition 

under single program 

manager.

Yes—The Air Force 

established a single 

program manager, 

who is referred to as 

the functional area 

chief.

Yes—The Army 

designated the 

contracting officer’s 

representative to 

oversee the 

transition.

Yes—The transition 

was assigned to the 

government task 

manager in the TriService 

Infrastructure 

Management Office.

Yes—NMCI’s 

Director has overall 

responsibility for the 

transition.

Yes—The transition 

process is under a 

single program 

manager.

Create 

client/provider 

transition teams to 

address short-term 

transition tasks as 

required.

Yes—During the 

transition phase, 

management from 

both the government 

and the provider met 

weekly to track the 

progress of the 

transition.

Yes—Personnel from 

both the Army and 

provider formed 

three transition 

teams, one for each 

site location. These 

teams worked with 

the contracting 

officer’s 

representative and 

the provider’s project 

manager to deal with 

transition issues.

Yes—Client/provider 

transition teams were 

established to 

address short-term 

transition tasks. The 

teams met weekly to 

address any 

transition issues.

Yes—NMCI and the 

provider designated 

staff to work together 

on short-term 

transition issues.

Yes—An integrated 

NIMA transformation 

team comprising 

NIMA and provider 

staff, was 

responsible for 

addressing shortterm tasks, including 

(1) transition 

planning, (2) 

resources and 

recruitment, (3) 

program budget, and 

(4) contract 

development and 

costing.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 72 GAO-03-371 DOD IT Services Outsourcing

Recognize that it 

takes time to effect 

transition and plan 

accordingly.

Yes—The Air Force 

allowed 60 days for 

the transition.

Yes—The contract 

allowed a 60-day 

phase-in period.

Yes—The contract 

provided for a 

transition period of 

30 to 90 days. 

According to MHS 

program officials, the 

transition period 

lasted 90 days.

Limited—The Navy 

said that it had 

initially 

underestimated the 

scope of the project 

and the magnitude of 

the problems brought 

on by legacy 

applications and 

associated 

information 

assurance and 

cyber-security 

issues. According to 

NMCI program 

officials, addressing 

these problems and 

completing the 

operational testing 

that was mandated 

by the Congress 

subsequent to 

contract award led to 

the transition period 

being lengthened 

from 2-1/2 to 

3-1/2 years. NMCI 

officials also said the 

extension will allow 

the Department of 

the Navy to have 

time to operate NMCI 

as a fully transitioned 

enterprise before 

having to decide 

whether to exercise 

the contract option.

Yes—According to 

NIMA, it has 

scheduled its seven 

functional areas to 

be transitioned to the 

provider over a 3- to 

5-year period. The 

provider will not 

assume 

responsibility for a 

given functional area 

until the NIMA 

transition review 

board has given its 

approval to the 

provider’s turnover 

plan. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 73 GAO-03-371 DOD IT Services Outsourcing

Encourage transition 

of staff to provider, 

where appropriate, 

using bonuses, stock 

options, and other 

appropriate methods.

Yes—The Air Force 

staff were given the 

right to transfer to the 

contractor. It was left 

up to the contractor 

to decide whether to 

offer incentives or 

not, which it chose 

not to do.

Yes—The Army had 

no personnel 

expected to transition 

to the provider since 

the activity being 

outsourced had 

previously been 

contracted out. 

However, the 

provider was 

interested in 

retaining existing 

contractor staff and 

had a program to 

recruit them. 

Yes—According to 

provider officials, 

they extended offers 

to employees from 

the prior contractor 

and some MHS staff. 

Yes—The NMCI 

contract provides 

that displaced federal 

employees that 

transition to the 

provider under 

employment 

openings as a result 

of NMCI will be given 

guaranteed 3 years 

of employment with 

the provider, a 15 

percent salary 

increase, and a signon bonus.

Yes—The contract 

requires the provider 

to actively recruit, 

hire, and make 

reasonable efforts to 

retain NIMA staff. If 

the provider meets 

certain target 

thresholds for 

recruiting NIMA staff, 

it receives a 

monetary incentive. 

According to the 

provider’s general 

manager, the 

company offered 

signing bonuses to 

NIMA employees that 

wanted to transition.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 74 GAO-03-371 DOD IT Services Outsourcing

Develop employeeretention programs 

and offer bonuses to 

keep key people, 

where appropriate.

No—According to 

Kirtland Air Force 

Base officials, they 

did not develop an 

employee-retention 

program because 

they did not believe 

that they could offer 

incentives. 

Not applicable—

Since this activity 

was previously 

contracted out, 

employee retention 

of Army civilian 

personnel was not 

relevant.

Yes—The 

government task 

manager persuaded 

key government 

employees from the 

Tri-Service Medical 

System Support 

Center to temporarily 

work at the TriService 

Infrastructure 

Management Office 

until positions 

became available 

with a contractor that 

provides support to 

this office. In 

addition, according to 

a program official, to 

keep expertise in 

particular 

applications, key 

employees were 

transferred to the 

MHS program 

executive office and 

Tri-Service 

Infrastructure 

Management Office.

Yes—NMCI program 

officials noted that 

the Navy has human 

resources programs 

that would allow its 

organizations a 

range of options, 

including bonuses, to 

help keep key staff. 

Additionally, the Navy 

directed all Navy and 

Marine Corps 

commands affected 

by NMCI to develop 

civilian personnel 

transition plans to 

mitigate the impact of 

NMCI 

implementation on 

employees.

Yes—According to 

NIMA program 

officials, NIMA 

categorized affected 

employees into three 

tiers, on the basis of 

whether they would 

be allowed to 

transition to the 

provider or whether 

there would be 

restrictions on their 

activities if they did 

transition. This 

distinction was made 

to comply with 

government conflict 

of interest rules. 

According to the 

program manager, 

NIMA also sought 

personnel who 

wanted to remain at 

the agency to help 

monitor the 

provider’s 

performance and 

considered the 

qualifications of 

those who wished to 

serve in this role 

prior to deciding who 

to retain.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 75 GAO-03-371 DOD IT Services Outsourcing

When consistent with 

organizational 

objectives, assist 

employees who do 

not want to transfer in 

finding other jobs, 

either within an 

organization or at 

another organization.

Yes—Kirtland Air 

Force Base officials 

stated that they have 

reduction-in-force 

procedures to help 

place civilian 

employees in other 

Air Force jobs. 

Employees also 

receive priority status 

for other DOD job 

openings. Further, 

the base education 

office provided 

assistance with 

outplacement; 

résumé writing; and, 

for those nearing 

retirement, planning 

advice.

Not applicable—

According to the 

TAC-SWA 

contracting officer, no 

Army personnel were 

displaced by the 

contract.

Yes—According to 

MHS program 

officials, the 

government task 

manager assisted 

government 

employees with the 

Tri-Service Medical 

System Support 

Center staff in finding 

new positions with 

other MHS 

organizations. For 

example, he located 

open positions and 

contacted the 

organizations.

Yes—NMCI program 

officials said that 

existing Navy civilian 

personnel programs 

are in place to assist 

employees. They 

noted that two Navy 

organizations found 

other jobs within their 

command for their 

small number of 

employees affected 

by NMCI. Other Navy 

organizations are 

also reassigning 

affected personnel to 

other government 

jobs, where 

applicable.

Yes—For staff that 

did not want to 

transition, NIMA 

provided various 

types of assistance, 

such as retirement 

planning, résumé 

writing, and 

interviewing skills. In 

addition, NIMA 

reported that if an 

employee did not 

want to transition to 

the provider, that it 

would consider readjusting the 

employee’s work 

assignment or 

provide training to 

support the 

individual’s 

placement within 

other areas.

Document key 

information to 

preserve 

organizational 

knowledge in the 

event that one or 

more providers 

change.

Yes—Kirtland Air 

Force Base’s work 

functions and 

workload size are 

documented in the 

contract.

Yes—Documentation 

produced by the 

provider, including 

maintenance logs, 

become the property 

of the government.

Yes—The Tri-Service 

Medical System 

Support Center 

processes and 

procedures were 

captured during 

transition. The 

provider’s processes 

and procedures are 

currently being 

captured. The 

contract also 

includes 

requirements on 

documenting key 

information.

Yes—NMCI

maintains a Web 

portal with 

organizational 

knowledge 

information. In 

addition, a new 

Intranet site is being 

developed with up-todate interactive 

organizational 

knowledge relevant 

to the many varied 

NMCI communities.

Yes—The executing 

NIMA transformation 

team was charged 

with documenting the 

activities for each 

functional area that is 

being outsourced. 

This includes 

developing and 

capturing operational 

processes and 

procedures for each 

of the functional 

areas.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VI

Projects’ Implementation of Phase V: 

Transition to Provider(s)

Page 76 GAO-03-371 DOD IT Services Outsourcing

Source: GAO.

Use change 

management 

strategies to help 

client employees 

deal with the 

transition.

Yes—Kirtland Air 

Force Base held 

town hall meetings 

and internal briefings 

to communicate the 

objectives of the 

outsourcing project 

and the changes that 

would take place 

internally. It also had 

one-on-one meetings 

with each employee 

that would be 

displaced.

Not applicable—The 

activity had been 

previously contracted 

out.

Yes—According to 

MHS program 

officials, the 

government task 

manager for the TriService 

Infrastructure 

Management Office 

assisted government 

employees with 

finding new positions 

and kept them 

informed of transition 

issues. Weekly 

E-mails were also 

sent out to all 

affected employees.

Yes—The Navy has 

provided employees 

affected by NMCI 

with a range of 

employment options 

and incentives.

Yes—NIMA’s change 

management 

strategies included 

training seminars 

offered by its human 

resources office on 

résumé writing, 

interviewing skills, 

career transition 

workshops, and 

employment trends.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 77 GAO-03-371 DOD IT Services Outsourcing

Appendix VII

Projects’ Implementation of Phase VI: Manage 

Provider(s) Performance ApendxiVI

The effectiveness with which the performance of the provider(s) is 

managed—the focus of this phase—is critical to the successful 

implementation of an outsourcing project. Indeed, according to Gartner, 

Inc., an outsourcing project can be thwarted by poorly designed, funded, 

and delivered processes for managing the delivery of services.1

 This firm 

also points out that an enterprise needs to retain the resources to oversee 

the planning and implementation of the IT services being delivered by the 

provider to ensure that the contractor meets the client’s business needs 

throughout the life of the agreement.2 Moreover, frequent and clear 

communication between the client and provider ensures that potential 

problems are resolved before they cause disruptions. In addition, 

performance reviews should take place regularly to keep the project on 

course, measure performance levels, and make adjustments as necessary.3

The 11 practices in this phase that we used to evaluate the five projects are 

as follows:

• Consider incentives to motivate provider(s) to exceed performance 

requirements.

1

Gartner, Inc., Retain Enough Resources to Manage Outsourcing Deals, Research Note 

COM-16-8425 (June 17, 2002).

2

Gartner, Inc., Successful Outsourcing Means Retaining Some Staff, Research Note COM18-9692 (Dec. 18, 2002). 

3

An Interagency-Industry Partnership in Performance, Seven Steps to Performance-Based 

Services Acquisition, Benchmark Version (January 2002).
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 78 GAO-03-371 DOD IT Services Outsourcing

• Use penalties to motivate provider(s) to meet performance 

requirements.

• Periodically undertake studies to assess (1) how the provider’s 

performance compares with the value being delivered to similar clients 

and (2) the extent to which the provider’s performance is improving 

over time.

• Schedule periodic working-level meetings with both the end-user groups 

and the provider to review the provider’s performance.

• Conduct executive-level oversight meetings with the provider’s senior 

management to review provider’s performance.

• Distribute performance data to stakeholders.

• Reserve audit rights on performance data supplied by the provider.

• Ensure that the provider measures and reports on performance.

• Work with the provider to redefine service levels, as appropriate.

• Sample performance data frequently enough to perform trend analysis 

and to permit extrapolation based on historical data.

• Allow employees and possibly stakeholders to rate the provider on a 

regular basis.

As shown by figure 11, the five projects in our review generally 

implemented the practices. 
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 79 GAO-03-371 DOD IT Services Outsourcing

Figure 11: Project Implementation of Phase VI: Managing Provider Performance

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent. 

Table 9 depicts whether and how the five projects in our review 

implemented each of the 11 practices in this phase. 
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 80 GAO-03-371 DOD IT Services Outsourcing

Table 9: Summary of Projects’ Use of Phase VI Practices

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Consider incentives 

to motivate 

provider(s) to exceed 

performance 

requirements.

No—The Air Force 

did not include 

incentives in the 

contract because the 

agency expects the 

contractor to meet 

the contract’s 

requirements without 

incentives. 

Specifically, 

according to the 

former functional 

area chief, because 

the contract was 

issued under the 

rules established by 

OMB Circular A-76, 

the focus of the 

project was on 

achieving cost 

savings, and 

incentives were not 

included in the 

contract. As a result, 

he noted that the 

provider did not have 

any incentive to be 

innovative or 

creative.

No—There are no 

monetary incentives, 

but, according to 

TAC-SWA officials, 

the fact that the 

agency will prepare a 

performance 

evaluation report can 

help motivate a 

provider to meet 

requirements. 

However, the 

contracting officer 

said that incentives 

might have been 

useful to motivate the 

provider to exceed 

performance 

requirements. 

Yes—The contract 

includes incentives 

linked to each 

performance 

requirement. Each 

requirement has a 

positive, negative, 

and acceptable 

performance range 

that provides a basis 

for monetary 

incentives (as well as 

penalties).

Yes—The NMCI 

contract includes a 

one-time payment of 

$10 million if the 

provider successfully 

completes full 

operational 

capability. The 

contract also has 

incentives for 

customer 

satisfaction, 

information 

assurance, and small 

business and small 

disadvantaged 

business 

participation.

Yes—The contract 

includes monetary 

incentives in 

accordance with an 

award fee plan. It 

also includes sharein-savings provisions 

to encourage 

process 

improvements.
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 81 GAO-03-371 DOD IT Services Outsourcing

Use penalties to 

motivate provider(s) 

to meet performance 

requirements, such 

as

• assess penalties for 

failure to perform at 

required individual 

service as well as 

aggregate service 

levels;

• apply penalties in 

the form of credit to 

the client;

• increase penalty for 

recurring deficient 

performance;

• hold back a 

percentage of 

provider’s pay for a 

particular service 

until performance 

requirements are 

met;

• refund a penalty if 

the provider returns 

to agreed-upon 

performance levels 

within a designated 

period of time; and

• ensure that the 

provider will cover 

costs, but not profit, 

when a particular 

performance 

requirement is not 

met.

Yes—The contract 

contains monetary 

penalties that are 

linked to each of the 

performance 

requirements. If 

imposed, they would 

reduce the amount of 

the payment owed to 

the provider. 

Limited—The 

contract does not 

have specific 

monetary penalties 

linked to 

performance 

requirements. 

However, the 

contract states that 

unacceptable work 

must be redone at 

the provider’s 

expense, and, if the 

defects and services 

cannot be corrected, 

the government may 

reduce the contract’s 

price to reflect the 

reduced value of the 

services performed. 

TAC-SWA project 

officials stated that 

monetary penalties 

were not included in 

the contract because 

the Army was 

concerned that 

contractors might not 

bid on the contract, 

and administrating 

this type of contract 

is more difficult.

Yes—The contract 

includes monetary 

penalties linked to 

each performance 

requirement. Each 

requirement has 

positive, negative, 

and acceptable 

performance ranges 

that provide a basis 

for monetary 

penalties as well as 

incentives. 

Yes—The contract 

includes monetary 

penalties in the form 

of credits to the 

agency if the 

provider fails to 

perform to the levels 

specified in the 

SLAs.

Limited—NIMA did 

not include monetary 

penalties in its 

contract. However, if 

the provider does not 

meet certain 

minimum 

performance 

standards, only its 

costs would be 

covered. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 82 GAO-03-371 DOD IT Services Outsourcing

Periodically 

undertake studies to 

assess: (1) how the 

provider’s 

performance 

compares with the 

value being delivered 

to similar clients and 

(2) the extent to 

which the provider’s 

performance is 

improving over time.

Limited—(1) This has 

not been done 

because, according 

to a project official, 

the Air Force was not 

aware of similar 

clients. 

(2) This is done 

through periodic 

monitoring of 

performance by 

quality assurance 

evaluators and 

through monthly 

review meetings held 

by the functional area 

chief to identify and 

address any 

problems that are 

starting to occur.

No—TAC-SWA 

officials stated that 

they address 

performance issues 

at the time that they 

occur and that they 

have not performed 

any studies.

Yes—The provider’s 

performance is 

reviewed monthly by 

the government task 

manager and in 

quarterly 

management reviews 

of the provider’s 

performance. 

According to MHS 

program officials, the 

results are compared 

with peers and 

reviewed for how the 

provider has 

improved over time.

Limited—(1) NMCI 

only has 

performance data 

since October 2002. 

The Navy intends to 

do such a study but 

has not established a 

schedule for it. 

(2) The provider’s 

improvements over 

time are being 

monitored monthly.

Limited—(1) At this 

time, NIMA has not 

undertaken such 

studies because it 

only recently 

transitioned functions 

to the provider. 

However, NIMA 

program officials 

stated that they 

intend to use a 

private-sector firm to 

periodically compare 

the provider’s 

performance with 

those of similar 

organizations. 

(2) The provider’s 

improvements over 

time are being 

monitored monthly.

Schedule periodic 

working-level 

meetings with both 

the end-user groups 

and the provider to 

review the provider’s 

performance.

Yes—Monthly 

meetings are held at 

which the contractor 

briefs the functional 

area chief and other 

Kirtland Air Force 

Base 

representatives.

Yes—According to 

the provider and 

TAC-SWA officials, 

they hold daily 

meetings to discuss 

any issues.

Yes—Periodic 

working-level 

meetings are held 

with the government 

task manager and 

the program 

management office 

representatives to 

obtain help-desk 

feedback. 

Yes—At each 

implemented site, the 

provider and 

government 

managers meet on a 

frequent, as-needed 

basis to review 

performance and 

resolve any issues. 

Yes—Quarterly 

performance 

management review 

meetings are held 

and are attended by 

NIMA and the 

provider. In addition, 

the provider hired an 

ombudsman to 

obtain anonymous 

comments from 

NIMA staff and 

management on its 

performance.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 83 GAO-03-371 DOD IT Services Outsourcing

Conduct executivelevel oversight 

meetings with the 

provider’s senior 

management to 

review provider’s 

performance.

Yes—Air Force 

schedules executivelevel oversight 

meetings whenever 

they are necessary. 

A recent meeting 

included the 

president of the 

provider.

Yes—According to 

provider and TACSWA officials, 

quarterly in- progress 

reviews are held with 

the provider’s 

corporate 

management and the 

battalion 

commander. 

Yes—Executive 

oversight meetings 

are held quarterly 

with the provider’s 

senior management 

to review 

performance.

Yes—The NMCI 

Director and the 

provider executive 

meet weekly to 

review performance 

and discuss other 

NMCI 

implementation 

issues. In addition, 

the Department of 

the Navy recently 

established an 

operations advisory 

board consisting of 

Department of the 

Navy leadership and 

the provider. The 

goal of this board is 

to focus senior 

leadership on issues 

affecting NMCI in 

order to establish 

priorities and make 

decisions. 

Yes—This is done 

through the quarterly 

review meetings. 

Distribute 

performance data to 

stakeholders.

Yes—Performance 

data were not 

distributed to 

stakeholders at the 

beginning of the 

project, but began 

after complaints from 

stakeholders about 

the need for such 

information. Metrics, 

such as system 

reliability or “uptime,” 

are now provided to 

stakeholders on line.

Yes—The contracting 

officer’s 

representative 

distributes monthly 

performance reports 

to Army stakeholders 

for review. 

Performance issues 

or problems 

identified are 

discussed with the 

contracting officer’s 

representative and 

the contracting 

officer, and corrective 

actions are planned 

to prevent 

reoccurrence.

Yes—The help-desk 

monthly reports, 

including results 

against performance 

metrics, are e-mailed 

to stakeholders 

monthly. 

Yes—The NMCI 

Navy and Marine 

Corps program 

managers are 

responsible for 

providing SLA 

performance data to 

their commands.

Yes—Performance 

data are distributed 

at quarterly review 

meetings, which 

according to the 

contracting officer, 

are attended by 

stakeholders. 

Reserve audit rights 

on performance data 

supplied by the 

provider.

Yes—Addressed in 

the contract.

Yes—Addressed in 

the contract.

Yes—Addressed in 

the contract.

Yes—Addressed in 

the contract.

Yes—Addressed in 

the contract.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 84 GAO-03-371 DOD IT Services Outsourcing

Ensure that provider 

measures and 

reports on 

performance.

Yes—As required by 

the quality control 

plan in the contract, 

the provider makes 

information available 

on product or service 

quality and any 

actions needed to 

correct decreases in 

quality. In addition, 

according to the 

former functional 

area chief, the 

provider provides 

performance 

information during 

monthly meetings.

Yes—The contracting 

officer’s 

representative 

receives 

performance 

information in daily 

meetings with the 

provider’s project 

manager. 

Yes—The contract 

requires the provider 

to submit monthly 

and quarterly 

performance reports 

that are then 

reviewed and 

verified.

Yes—The contract 

specifies that the 

provider must 

measure and report 

on whether its SLA 

performance goals 

are being met. 

Yes—The contract 

requires the provider 

to submit data, 

including 

measurements of 

service, quarterly.

Work with provider to 

redefine service 

levels, as 

appropriate.

Yes—This is done on 

an as-needed basis. 

For example, the 

service levels were 

redefined in April 

2002.

Yes—According to 

TAC-SWA and 

provider officials, 

they have worked 

together to redefine 

performance 

requirements in the 

contract. 

Yes—MHS and the 

provider have worked 

together to redefine 

SLAs. For example, 

the first call closure 

performance metric 

was redefined to be 

more realistic.

Yes—The Navy and 

service provider have 

and continue to 

refine NMCI SLAs to 

ensure more precise 

performance 

measures and to 

more accurately 

capture user 

satisfaction with the 

system. For 

example, a contract 

modification 

standardized many 

of the performance 

categories that dealt 

with service 

availability. 

Yes—The redefinition 

of service levels is 

provided for in the 

contract. According 

to NIMA, the 

redefinition of service 

levels is expected to 

occur in the future, 

generally at the time 

that the annual 

contract option is 

exercised.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Appendix VII

Projects’ Implementation of Phase VI: 

Manage Provider(s) Performance

Page 85 GAO-03-371 DOD IT Services Outsourcing

Source: GAO.

Sample performance 

data frequently 

enough to perform 

trend analysis and to 

permit extrapolation 

based on historical 

data.

Yes—Air Force 

quality assurance 

evaluators monitor 

performance in 

accordance with the 

quality assurance 

surveillance plan and 

the performance 

requirements 

summary. Any trends 

identified are 

addressed in monthly 

status meetings.

Yes—The provider’s 

proposal and quality 

control plan state 

that it will provide 

various trend 

analyses to the 

government. 

According to TACSWA project officials, 

these data are 

submitted to the 

contracting officer’s 

representative for 

analysis. 

Yes—Performance 

data are sampled 

and reported monthly 

and quarterly. They 

are analyzed and 

verified, including 

any supporting data. 

In addition, the 

provider’s knowledge 

management system 

provides analysis 

and trend data to 

MHS.

Yes—The NMCI 

Director monitors 

provider performance 

to identify trends by 

assessing providersupplied information 

as well as 

information from 

Navy independent 

verification and 

validation testing and 

customer satisfaction 

survey results. In 

addition, NMCI 

program officials said 

that they have 

requested funding to 

develop a 

performance 

measurement data 

repository to support 

trend analyses.

Yes—The contract 

requires data to be 

provided periodically 

for NIMA to use for 

trend and other types 

of analyses. 

Allow employees and 

possibly 

stakeholders to rate 

the provider on a 

regular basis (e.g., 

scorecards and 

quarterly report 

cards).

Limited—The 

provider conducts 

annual customer 

satisfaction surveys, 

but it is not required 

to submit the results 

to the Air Force. 

Kirtland Air Force 

Base also obtains ad 

hoc feedback from 

employees who, after 

they report a 

problem, are asked 

to provide 

information about 

how well the problem 

was addressed and 

their overall 

satisfaction level. 

Yes—The contracting 

officer’s 

representative and 

on-site points of 

contact obtain 

feedback from 

employees on a 

regular basis. 

Yes—The customer 

satisfaction survey is 

one method used to 

rate the provider. In 

addition, according to 

MHS program 

officials, stakeholder 

input is also obtained 

from program 

management reviews 

and meetings held by 

the government task 

manager.

Yes—User surveys 

are used to measure 

satisfaction with 

specific services 

such as E-mail 

access, help desk, 

etc., and overall user 

satisfaction with the 

service provider’s 

performance. 

Yes—The provider 

surveys NIMA 

employees on their 

satisfaction with the 

help-desk function. 

The results of these 

surveys are reviewed 

by NIMA during 

quarterly meetings. 

In addition, 

stakeholders are also 

responsible for 

providing monthly 

assessments of 

contractor 

performance. 

Moreover, customer 

satisfaction is a 

factor in determining 

the amount of the 

semiannual award 

fee earned by the 

provider. 

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 86 GAO-03-371 DOD IT Services Outsourcing

Appendix VIII

Projects’ Implementation of Phase VII: Ensure 

Services are Provided ApendxiVI

Although outsourcing focuses on the provider’s ability to perform, the 

client organization is ultimately responsible for ensuring that services are 

provided and that end-user needs are met. The previous phases addressed 

the extensive preparation that must precede the provider’s assuming 

responsibility for the client organization’s services. This phase 

encompasses practices designed to ensure that an effective oversight 

approach is established. In addition, over the course of the outsourcing 

project, the client learns more about the capabilities of the provider, and 

market conditions may change. As a result, it is important to monitor 

service levels internally as well as maintain an external view of the 

performance of other providers in order to identify opportunities to 

improve and ensure that the outsourcing arrangement maintains its value 

to the client. 

The six practices in this phase that we used to evaluate the five projects are 

as follows:

• Monitor the provider’s work to anticipate issues for resolution.

• Make sure that the provider uses the standard tools and processes 

defined as part of the operational model.

• Use provider performance data to continuously improve processes.

• Pursue improvement based on customer satisfaction surveys.

• Ensure that an appropriately empowered individual from the client 

organization oversees the work.

• Set realistic time frames that are agreed to by the provider.
Appendix VIII

Projects’ Implementation of Phase VII: 

Ensure Services are Provided

Page 87 GAO-03-371 DOD IT Services Outsourcing

As illustrated by figure 12, the five projects in our review largely 

implemented the practices.

Figure 12: Project Implementation of Phase VII: Ensure Services Are Provided

Note: Not applicable—The practice was not relevant to the project’s particular circumstances. 

No—The agency did not implement the practice. Limited—The agency fully implemented some but not 

all aspects of the practice and did not take alternative actions that fully satisfied the practice. Yes—The 

agency fully implemented the practice or took an alternative action that fully satisfied its intent.

Table 10 provides details on whether and how each project implemented 

each of the six practices associated with this phase. 
Appendix VIII

Projects’ Implementation of Phase VII: 

Ensure Services are Provided

Page 88 GAO-03-371 DOD IT Services Outsourcing

Table 10: Summary of Projects’ Use of Phase VII Practices

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS

Monitor the 

provider’s work to 

anticipate issues for 

resolution.

Yes—Quality 

assurance evaluators 

monitor the quality of 

the provider’s work 

and identify problems 

or trends. The results 

and any problems 

are reported to the 

contracting officer 

and the functional 

area chief.

Yes—The contracting 

officer’s 

representative 

reports monthly on 

contractor 

performance. In 

addition, he meets 

daily with provider 

officials to discuss 

performance results.

Yes—The provider’s 

work is monitored 

through monthly 

reports and meetings 

as well as through 

on-site meetings and 

readiness 

assessments.

Yes—Navy 

representatives at 

each implemented 

site (including 

headquarters) 

monitor the 

provider’s work and 

identify issues. 

According to NMCI 

program officials, the 

NMCI Enterprise 

Management System 

enables the 

government to 

monitor the 

performance of the 

network and 

associated service 

delivery.

Yes—NIMA reviews 

the provider’s work in 

quarterly program 

management reviews 

and as needed in 

response to issues 

raised by NIMA’s 

performance 

monitoring officials. 

Make sure the 

provider uses the 

standard tools and 

processes defined as 

part of the 

operational model.

Yes—The provider 

uses software tools 

to help monitor 

system performance 

against performance 

standards to monitor 

and manage the 

help-desk function.

Yes—According to 

TAC-SWA project 

officials, the provider 

uses various 

standard tools 

outlined in its 

proposal.

Yes—According to 

MHS program 

officials, the provider 

uses the standard 

tools outlined in the 

contract.

Yes—The Navy has 

monitored the 

service provider’s 

use of standard tools 

and processes. Tools 

are being used to 

support legacy 

migration, client 

installation, and helpdesk procedures. 

Yes—The tools are 

documented in the 

quality management 

plan. For example, 

the provider is using 

a specific quality and 

process 

improvement 

methodology. 

Use provider 

performance data to 

continuously improve 

processes.

Yes—Although 

Kirtland Air Force 

Base does not rely 

on data provided by 

its provider, its 

quality assurance 

evaluators 

continually monitor 

provider 

performance. 

Yes—The contracting 

officer’s 

representative 

monitors provider 

performance data, 

and any potential 

areas for 

improvement are 

incorporated in the 

monthly report.

Yes—For example, 

MHS used provider 

performance data to 

improve the 

customer satisfaction 

survey process. 

Yes—The Navy uses 

the provider’s data to 

help determine ways 

to improve 

processes, such as 

to improve NMCI 

implementation 

procedures and the 

timeliness of helpdesk problem 

resolution.

Yes—NIMA’s 

performance 

monitors use 

provider data to 

assess progress and 

ways to improve 

performance.
Appendix VIII

Projects’ Implementation of Phase VII: 

Ensure Services are Provided

Page 89 GAO-03-371 DOD IT Services Outsourcing

Source: GAO

Pursue improvement 

based on customer 

satisfaction surveys.

Limited—The Air 

Force does not 

perform, or require 

the provider to 

perform, customer 

surveys. However, 

the provider has 

initiated such 

surveys but is not 

required to distribute 

the results to the Air 

Force. 

Yes—Surveys are 

received by the 

contracting officer’s 

technical 

representatives and 

provided as input for 

the contracting 

officer’s 

representative 

report.

Yes—Negative 

feedback that is 

received based on 

the results of the 

customer satisfaction 

surveys is reviewed 

by the MHS helpdesk manager.

Yes—Surveys of 

user satisfaction with 

specific services and 

overall satisfaction 

with provider 

performance are 

used to identify areas 

needing 

improvement.

Yes—To pursue 

improvement, NIMA 

uses both customer 

satisfaction surveys 

and interviews with 

senior-level 

customers.

Ensure that an 

appropriately 

empowered 

individual from the 

client organization 

oversees the work.

Yes—The functional 

area chief oversees 

the work, and the 

quality assurance 

evaluators support 

this oversight effort. 

Yes—The contracting 

officer’s 

representative is the 

empowered 

individual from 

NETCOM.

Yes—The 

government task 

manager is the 

individual 

empowered to 

oversee the work of 

the provider.

Yes—The NMCI 

Director oversees the 

work and discusses 

performance with the 

NMCI provider 

executive during 

weekly meetings.

Yes—The contracting 

officer’s 

representative and a 

staff of technical 

monitors oversee the 

work. 

Set realistic time 

frames that are 

agreed to by the 

provider.

Yes—Kirtland Air 

Force Base and the 

provider agree that 

reasonable time 

frames for 

performance have 

now been set. Some 

revisions were made 

to them in April 2002.

Yes—Time frames 

for new tasks are 

established by 

contract 

modifications that are 

agreed to by the 

provider. 

Yes—The monthly 

performance reports 

and quarterly 

program reviews set 

time frames that 

have been agreed to 

by the provider.

Yes—According to 

NMCI program 

officials, Navy’s 

practice is to work 

with the provider in 

establishing 

schedules.

Yes—NIMA and the 

provider have agreed 

on dates associated 

with specific actions.

(Continued From Previous Page)

Practice

Did the project use the practice?

Air Force C4 

Services Army TAC-SWA MHS/ITO Help Desk Navy NMCI NIMA IT/IS
Page 90 GAO-03-371 DOD IT Services Outsourcing

Appendix IX

Comments from the Department of Defense ApendxiIX

Note: GAO comments 

supplementing those in 

the report text appear 

at the end of this 

appendix.

See comment 1.

See comment 2.
Appendix IX

Comments from the Department of Defense

Page 91 GAO-03-371 DOD IT Services Outsourcing
Appendix IX

Comments from the Department of Defense

Page 92 GAO-03-371 DOD IT Services Outsourcing

See comment 3.
Appendix IX

Comments from the Department of Defense

Page 93 GAO-03-371 DOD IT Services Outsourcing

The following are GAO’s comments on the Department of Defense’s letter 

dated April 8, 2003. 

GAO Comments 1. Addressed in the Agency Comments and Our Evaluation section of this 

report.

2. On January 29, 2003, NIMA granted the provider authorization to use 

government sources in performing the IT/IS contract that was limited 

to “products, services, and supplies that support the performance of the 

specific functional areas and miscellaneous items required under this 

contract.” Accordingly, we did not modify this report. 

3. Addressed in the Agency Comments and Our Evaluation section of this 

report.
Page 94 GAO-03-371 DOD IT Services Outsourcing

Appendix X

GAO Contact and Staff Acknowledgments ApendxiX

GAO Contact Linda J. Lambert, (202) 512-9556

Staff 

Acknowledgments

Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and 

Thomas Wright made key contributions to this report.

(310239)
GAO’s Mission The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal government 

for the American people. GAO examines the use of public funds; evaluates federal 

programs and policies; and provides analyses, recommendations, and other 

assistance to help Congress make informed oversight, policy, and funding 

decisions. GAO’s commitment to good government is reflected in its core values of 

accountability, integrity, and reliability.

Obtaining Copies of 

GAO Reports and 

Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail this 

list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO 

Mailing Lists” under “Order GAO Products” heading.

Order by Mail or Phone The first copy of each printed report is free. Additional copies are $2 each. A check 

or money order should be made out to the Superintendent of Documents. GAO 

also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a single 

address are discounted 25 percent. Orders should be sent to:

U.S. General Accounting Office

441 G Street NW, Room LM

Washington, D.C. 20548

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061

To Report Fraud, 

Waste, and Abuse in 

Federal Programs

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470

Public Affairs Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Service Requested

Presorted Standard

Postage & Fees Paid

GAO

Permit No. GI00