This is the accessible text file for GAO report number GAO-03-371 
entitled 'Information Technology: DOD Needs to Leverage Lessons Learned 
from Its Outsourcing Projects' which was released on April 25, 2003.

This text file was formatted by the U.S. General Accounting Office 
(GAO) to be accessible to users with visual impairments, as part of a 
longer term project to improve GAO products' accessibility. Every 
attempt has been made to maintain the structural and data integrity of 
the original printed product. Accessibility features, such as text 
descriptions of tables, consecutively numbered footnotes placed at the 
end of the file, and the text of agency comment letters, are provided 
but may not exactly duplicate the presentation or format of the printed 
version. The portable document format (PDF) file is an exact electronic 
replica of the printed version. We welcome your feedback. Please E-mail 
your comments regarding the contents or accessibility features of this 
document to Webmaster@gao.gov.

Report to the Subcommittee on Readiness and Management Support, 
Committee on Armed Services,
U.S. Senate:

April 2003:

Information Technology:

DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects:

GAO-03-371:

GAO Highlights:

Highlights of GAO-03-371, a report to the Subcommittee on Readiness and Management Support, Committee on Armed Services, U.S. Senate

Why GAO Did This Study:

Given the magnitude of its reported spending on information technology 
(IT) servicesmore than $6.2 billion in fiscal year 2001it is critical 
that the Department of Defense (DOD) adopt effective practices for 
acquiring IT services.

GAO researched leading commercial practices for the outsourcing of IT 
services, and, in November 2001, published a framework consisting of 
seven phases that span the full range of activities that are performed 
during the outsourcing of those services (this is an acquisition in 
which a client organization transfers responsibility for performing 
services to an external provider).

GAO was asked to determine (1) the extent to which selected DOD 
projects for outsourcing IT services use leading commercial practices 
as specified in GAOs framework and (2) whether DOD is sharing lessons 
learned from its IT outsourcing projects across the department.

What GAO Found:

The projects in GAOs review substantially used leading commercial 
practices as specified in GAOs framework for outsourcing IT services. 
Specifically, the agencies fully implemented 88 percent of the practices
 (not including practices not applicable to a particular project). This 
framework consists of practices organized into seven phases: (I) 
determine sourcing strategy, (II) define operational model, (III) 
develop the contract, (IV) select the provider(s), (V) transition to 
provider(s), (VI) manage the performance of the provider(s), and (VII) 
ensure services are provided. The figure below shows the percentage of 
practices that were implemented in each phase.

Although DOD has acted on gathering and disseminating lessons learned 
and commercial leading practices related to general acquisition issues, 
its actions have generally not been focused on outsourcing or on 
sharing the lessons learned from IT services outsourcing across the 
department. By not  systematically capturing and disseminating such 
information across the department, DOD is losing the opportunity to 
leverage the knowledge gained on IT services projects like those in 
GAOs review. Lessons learned that are pragmatic and easily accessible 
would give DOD managers a more informed understanding of important 
issues to be addressed when making outsourcing decisions, as well as 
the factors to be considered to help ensure the success of these 
endeavors. 

What GAO Recommends:

GAO is making recommendations to the Secretary of Defense aimed at 
leveraging lessons learned across the department from its components 
IT outsourcing experiences.

DOD agreed that capturing lessons learned related to IT outsourcing 
initiatives is important and stated that it intends to explore a 
variety of mechanisms to do so. DODs plans are consistent with our 
recommendations.

www.gao.gov/cgi-bin/getrpt?GAO-03-371.
To view the full report, including the scope
and methodology, click on the link above.
For more information, contact Randolph C. Hite at (202) 512-3439 or 
hiter@gao.gov.

[End of section]:

Letter:

Results in Brief:

Background:

Projects Substantially Used Leading Commercial Practices:

Leveraging Lessons Learned DOD-wide
Could Assist Other DOD Projects:

Conclusions:

Recommendations:

Agency Comments and Our Evaluation:

Appendixes:

Appendix I: Objectives, Scope, and Methodology:

Appendix II: Projects Implementation of Phase I: Determine Sourcing
Strategy:

Appendix III: Projects Implementation of Phase II: Define Operational
Model:
 
Appendix IV: Projects Implementation of Phase III: Develop the
Contract:

Appendix V: Projects Implementation of Phase IV: Select the
Provider(s):

Appendix VI: Projects Implementation of Phase V: Transition to
Provider(s):

Appendix VII: Projects Implementation of Phase VI: Manage Provider(s)
Performance: 

Appendix VIII: Projects Implementation of Phase VII: Ensure Services 
are Provided: 

Appendix IX: Comments from the Department of Defense:

GAO Comments:

Appendix X: GAO Contact and Staff Acknowledgments:

GAO Contact:

Staff Acknowledgments: 

Tables:

Table 1: Definition of Phases for IT Outsourcing:

Table 2: Profile of Outsourcing Projects Reviewed:

Table 3: Percentage of Practices Implemented, by Project:

Table 4: Summary of Projects' Use of Phase I Practices:

Table 5: Summary of Projects' Use of Phase II Practices:

Table 6: Summary of Projects' Use of Phase III Practices:

Table 7: Summary of Projects' Use of Phase IV Practices:

Table 8: Summary of Projects' Use of Phase V Practices:

Table 9: Summary of Projects' Use of Phase VI Practices:

Table 10: Summary of Projects' Use of Phase VII Practices:

Figures:

Figure 1: Percentage of Practices Implemented in Each Phase:

Figure 2: Roles of the Client and Provider in an Outsourcing 
Relationship:

Figure 3: GAO's Framework for Outsourcing IT Services:

Figure 4: Use of Leading Commercial Practices for Outsourcing of IT 
Services, by Project:

Figure 5: Percentage of Practices Implemented in Each Phase:

Figure 6: Project Implementation of Phase I: Determine Sourcing 
Strategy:

Figure 7: Project Implementation of Phase II: Define Operational Model:

Figure 8: Project Implementation of Phase III: Develop the Contract:

Figure 9: Project Implementation of Phase IV: Select the Provider(s):

Figure 10: Project Implementation of Phase V: Transition to Provider(s):

Figure 11: Project Implementation of Phase VI: Managing Provider 
Performance:

Figure 12: Project Implementation of Phase VII: Ensure Services Are 
Provided:

Abbreviations:

C3I: command, control, communications, and intelligence:

C4: command, control, communications, and computer :

CIO: chief information officer:

DOD: Department of Defense:

IT: information technology:

IT/IS: Information Technology/Information Services:

MHS/ITO: Military Health System/Information Technology Organization:

NETCOM: Network Enterprise Technology Command:

NIMA: National Imagery and Mapping Agency:

NMCI: Navy and Marine Corps Intranet: :

OMB: Office of Management and Budget:

RFP: request for proposals:

SLA: service-level agreement:

TAC-SWA: Total Army Communications--Southwest Asia:

This is a work of the U.S. Government and is not subject to copyright 
protection in the United States. It may be reproduced and distributed 
in its entirety without further permission from GAO. It may contain 
copyrighted graphics, images or other materials. Permission from the 
copyright holder may be necessary should you wish to reproduce 
copyrighted materials separately from GAOs product.

Letter April 25, 2003:

The Honorable John Ensign
Chairman
The Honorable Daniel K. Akaka
Ranking Minority Member
Subcommittee on Readiness and Management Support
Committee on Armed Services
United States Senate:

The Department of Defense (DOD) is the government's largest purchaser 
of information technology (IT) services, such as desktop support, 
network operations, and software development services. In fiscal year 
2001, DOD reportedly obligated more than $6.2 billion on IT 
services,[Footnote 1] and this amount is expected to grow 
substantially. Given the magnitude of DOD's spending on such services, 
it is critical that the department adopt effective practices for 
acquiring IT services.

Since 1996, we have conducted a series of studies for the Senate 
Committee on Armed Services concerning how DOD can improve its 
acquisition processes by adopting proven practices of leading 
commercial organizations. In this vein, in November 2001, we issued a 
guide that organized leading commercial practices for the 
outsourcing[Footnote 2] of IT services into a framework of seven phases 
that span the full range of activities that are performed during IT 
services outsourcing.[Footnote 3]

This report responds to your request that we determine (1) the extent 
to which selected DOD IT services outsourcing projects use leading 
commercial practices as specified in our framework and 
(2) whether DOD is sharing lessons learned from its IT outsourcing 
projects across the department. To address the first objective, we 
selected five projects from a group of projects to outsource IT 
services that were
identified by the military services and other DOD components.[Footnote 
4] We then asked the component responsible for each project to perform 
a self-assessment against selected practices in our framework for 
outsourcing IT services.[Footnote 5] Next, we obtained and reviewed 
agencies' supporting documentation and interviewed the appropriate 
agency and provider project officials to independently determine 
whether a practice was met. To address the second objective, we 
reviewed applicable DOD approaches for capturing and disseminating 
lessons learned from IT services outsourcing projects and interviewed 
the applicable acquisition and IT officials. Details of our objectives, 
scope, and methodology are discussed in appendix I.

Results in Brief:

The projects in our review substantially used leading commercial 
practices as specified in our framework on outsourcing IT services. 
Specifically, the agencies fully implemented 88 percent of the 
practices.[Footnote 6] This framework consists of practices organized 
into seven phases that span the full range of activities that are 
performed during IT outsourcing: (I) determine sourcing strategy, (II) 
define operational model, (III) develop the contract, (IV) select the 
provider(s), (V) transition to provider(s), (VI) manage provider(s) 
performance, and (VII) ensure services are provided. Figure 1 
illustrates the percentage of practices that were followed in each 
phase. Collectively, the projects fully implemented from 70 to 97 
percent of the practices in each phase.

Figure 1: Percentage of Practices Implemented in Each Phase:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Although implementing the leading commercial practices in our framework 
does not guarantee the success of an outsourcing project, the consensus 
view of the leading commercial activities that we studied is that these 
practices are the most critical to success when acquiring IT 
services.[Footnote 7] In addition, not implementing or only partially 
implementing particular practices can produce negative consequences or 
add risk to a project. For example, the Department of the Navy 
project's baseline of its existing environment (a phase I practice) was 
limited because it did not include an assessment of its legacy 
applications. Instead, project officials decided to rely on a 
preexisting inventory developed to address the Year 2000 challenge. The 
Navy subsequently found that it had substantially underestimated the 
number of legacy applications, which, according to program officials, 
later contributed to the transition period slipping from
2- years to 3- years.

As DOD gathers more experience in implementing projects for outsourcing 
IT services, it can benefit from leveraging the lessons derived from 
these initiatives. For example, the projects in our review have 
identified lessons learned in such areas as transitioning to the 
provider and partnering with the provider. Although DOD has taken 
action to gather and disseminate lessons learned and best practices on 
general acquisition issues, these efforts generally do not focus on 
outsourcing or include sharing the lessons learned from IT outsourcing 
projects across the department. By not capturing and disseminating such 
information in a systematic manner across the department, DOD is losing 
the opportunity to leverage the knowledge gained on IT services 
projects like those in our review.

We are making recommendations to the Secretary of Defense aimed at 
leveraging lessons learned across the department from its components' 
IT services outsourcing experiences.

In written comments on a draft of this report signed by the DOD 
Principal Director, Deputy Assistant Secretary of Defense (Deputy Chief 
Information Officer), DOD agreed that capturing lessons learned in the 
development and implementation of its IT outsourcing initiatives is 
important to continually improving its outsourcing methods and results 
achieved. The department also stated that it intends to explore a 
variety of mechanisms for best exploiting lessons learned from its IT 
outsourcing initiatives. We agree that it is prudent to consider 
alternative means to leveraging these lessons learned, and we believe 
that this is consistent with our recommendations.

Background:

To protect the security of the United States, DOD relies on a complex 
array of computer-dependent and mutually supportive organizational 
components, including the military services, Commanders in Chief, and 
Defense agencies. As such, it invests tens of billions of dollars each 
year in a broad array of computer systems, which include weapon 
systems, command and control systems, satellite systems, inventory 
management systems, transportation management systems, health systems, 
financial systems, personnel systems, and payment systems. In addition, 
DOD spends billions of dollars annually on IT services, which include 
database management, help-desk operations, software maintenance, and 
network services. In fiscal year 2001, DOD reportedly obligated more 
than $6.2 billion on IT services alone.[Footnote 8]

Decisions regarding the purchasing of services are critical to ensuring 
the effectiveness of DOD's operations as well as those of the 
government as a whole. Our November 2001 report recognizes the 
importance of such sourcing decisions and provides a framework that 
spans the full range of activities that are performed during IT 
services outsourcing.[Footnote 9] At the same time, governmentwide 
policies, initiatives, and challenges exist that significantly 
influence the government's sourcing decisions.

GAO's Framework for Outsourcing IT Services:

Outsourcing of IT services has become increasingly popular in both the 
public and private sector. For example, according to the Giga 
Information Group, Inc., a leading research firm, such outsourcing is 
expected to grow an average of 5 to 6 percent in 2003.[Footnote 10] The 
federal sector's outsourcing is predicted to rise at an even greater 
rate. For example, INPUT, an IT market research firm, forecasts that 
defense IT outsourcing will increase about 143 percent between fiscal 
years 2002 and 2007.[Footnote 11]

IT outsourcing involves the activities associated with acquiring 
services from one or more external providers. During outsourcing, a 
client organization transfers responsibilities for performing one or 
more IT services to one or more external providers. This responsibility 
is executed through control and management of the processes, people, 
and technology associated with these services.

Figure 2 depicts the roles of the client and provider organizations in 
an outsourcing relationship.

Figure 2: Roles of the Client and Provider in an Outsourcing 
Relationship:

[See PDF for image]

[End of figure]

Our November 2001 guide on leading commercial practices for outsourcing 
IT services provides a generic framework of practices from leading 
commercial organizations that can improve purchasing decisions and 
manage the resulting government/provider relationship.[Footnote 12] 
The framework is represented in figure 3 as a hierarchy of phases, 
practices, and critical success factors.

Figure 3: GAO's Framework for Outsourcing IT Services:

[See PDF for image]

Note: The arrow from phase VII to phase I represents the need to 
reflect on lessons learned from previous phases. The arrows between 
phase III and IV represent the iterative nature of developing the 
contract and selecting the provider. Although there is a logical order 
to the sequence of the common phases, the order of the practices within 
each phase does not imply any priority or sequence.

[End of figure]

Table 1 provides a definition of each phase of the framework. Each of 
the phases has specific practices associated with it. Implementing 
these practices does not guarantee the success of an outsourcing 
project. However, our November 2001 study reflected a consensus view 
that these practices were the most critical to success when IT services 
are being acquired.[Footnote 13] Restated, application of these 
practices increases the probability of a successful outsourcing 
project.

Table 1: Definition of Phases for IT Outsourcing:

Phase number: I; Title: Determine sourcing strategy; Definition: Client 
organizations determine whether internal capability or external 
expertise can more effectively meet IT needs.

Phase number: II; Title: Define operational model; Definition: Client 
organizations formalize executive leadership, team composition, client 
responsibilities, and operating relationships between client and 
provider organizations.

Phase number: III; Title: Develop the contract; Definition: Client 
organizations establish the legal terms for the IT outsourcing 
relationship.

Phase number: IV; Title: Select the provider(s); Definition: Client 
organizations find one or more providers who can help them reach their 
IT outsourcing goals.

Phase number: V; Title: Transition to provider(s); Definition: Client 
organizations transfer responsibility of IT functions to one or more 
providers.

Phase number: VI; Title: Manage provider(s) performance; Definition: 
Client organizations make sure each provider is meeting performance 
requirements.

Phase number: VII; Title: Ensure services are provided; Definition: 
Client organizations make sure that services are provided and end-user 
needs are met.

Source: GAO.

[End of table]

The organizations that we studied also identified certain capabilities 
(identified as critical success factors) that were essential for 
implementing the practices identified in our framework. First, 
executive leadership strengthens the interaction between executive 
management and the employees of the client organization. Second, 
partner alignment strengthens the interaction between the client and 
provider organization at the executive level, which ensures that the 
goals and objectives of these organizations support each other. Third, 
relationship management strengthens the interaction between the client 
and provider organization at the operational level.

Influences on Government Sourcing Decisions:

The federal government is one of the world's largest users of services. 
Because of the large dollar value and the number of private-and public-
sector jobs involved, deciding whether the public or private sector 
would be the most appropriate provider of the services the government 
needs (IT or otherwise) is an important, and often highly charged, 
question. Among the factors that agencies must consider as they 
determine how best to meet their missions is whether the public or 
private sector would be the most appropriate provider of the services 
the government needs. Phase I of our framework, determine the sourcing 
strategy, addresses the client's assessment of whether expertise from 
within or outside of the organization can more effectively meet the 
client's needs.

The National Defense Authorization Act for Fiscal Year 2001 required 
the Comptroller General of the United States to convene a panel of 
experts to study the current process used by the government to make 
sourcing decisions. The resulting Commercial Activities Panel conducted 
a year-long study and heard repeatedly about the importance of 
competition and its central role in fostering economy, efficiency, and 
continuous performance improvement. In particular, the panel reviewed 
the government's implementation of the Office of Management and 
Budget's (OMB) Circular A-76, which sets forth federal policy for 
determining whether federal employees or private contractors will 
perform commercial activities for the government.[Footnote 14] Circular 
A-76 (1) outlines conditions under which agencies are permitted to 
perform a commercial activity with government employees or by contract 
and (2) provides guidance for whether, and if so, how, agencies should 
conduct a cost comparison when they are considering transferring the 
performance of commercial activities from the public to the private 
sector (or vice versa). The panel reported that there were positive 
elements to Circular A-76 but that both federal employees and private 
firms complained that it does not meet the standard of a clear, 
transparent, and consistently applied process. For example, both 
federal employees and private firms criticized the Circular A-76 
process as unequal and therefore unfair.

The Commercial Activities Panel strongly supported continued emphasis 
on competition and concluded that whenever the government is 
considering converting work from one sector to another, public/private 
competitions should be the norm. In addition, the panel made four 
recommendations, including that all sourcing decisions be consistent 
with the principles adopted unanimously by the panel, such as the 
principle that federal policy provide for accountability in connection 
with all sourcing decisions.[Footnote 15]

As part of the administration's efforts to implement the 
recommendations of the Commercial Activities Panel, OMB has published 
proposed changes to Circular A-76. Key highlights of the proposed 
changes include presuming that all functions are commercial in nature 
unless they are justified as inherently governmental;[Footnote 16] 
limiting the length of time for competitions; and emphasizing awarding 
contracts on the basis of best value, not just lowest cost. Best value 
allows the contracting official to consider technical superiority, 
quality, innovation, and past performance as well as price.

However, we reported that there are several areas in which the proposed 
revisions to the circular are not consistent with the principles or 
recommendations of the Commercial Activities Panel.[Footnote 17] 
Specifically, the proposed revision does not include a link between 
sourcing policy and agency missions, has unnecessarily complicated 
source selection procedures, contains certain unrealistic time frames, 
and includes insufficient guidance on calculating savings.

Beyond the Commercial Activities Panel, other bodies have identified 
challenges that the federal government faces in reaching and executing 
effective sourcing decisions. For example, members of the Coalition for 
Government Procurement, the Professional Services Council, and the 
Information Technology Association of America told us that 
organizational culture is one of the biggest differences between the 
commercial sector and the federal government and one of the greatest 
barriers to the government's use of commercial practices. Also, as we 
have previously reported, moving to outsourcing solutions can involve a 
cultural change for government organizations because it may require a 
change to an agency's operating model, such as using a contractor to 
provide IT services previously performed by government staff or using a 
performance-based contract.[Footnote 18] This view was echoed by a 2001 
study of DOD competitive 
sourcing that found cultural, process, execution, and training 
barriers.[Footnote 19] The study stated that these barriers need to be 
understood and mitigated before the benefits of outsourcing can be 
fully realized. Barriers such as these can be overcome by strong 
executive leadership, which is a critical success factor in our 
framework.

Another challenge is creating a productive agency/provider 
relationship--another critical success factor in our framework. 
According to a report sponsored by the PricewaterhouseCoopers Endowment 
for the Business of Government, such public/private partnerships are 
based on trust, commitment to problem or conflict resolution, and the 
recognition that flexibility is necessary and that the relationship 
will evolve and change over time.[Footnote 20] If deadlines are not 
met, or public agency goals change with differing political climates, 
the partners need to discuss the basis of the partnership and construct 
a different relationship. Our prior report on desktop outsourcing found 
that developing a productive agency/contractor relationship is not 
always easy.[Footnote 21] Both sides must recognize and understand each 
other's underlying motives and strive to achieve established 
expectations.

Finally, human capital issues are another challenge facing federal 
agencies that affect their ability to implement outsourcing. Our 
framework recognizes the importance of having the right skills in place 
to support the outsourcing relationship. However, as we have previously 
reported, procurement reforms and technological changes have placed 
unprecedented demands on the acquisition workforce.[Footnote 22] 
Contracting personnel are now expected to have a much greater knowledge 
of market conditions, industry trends, and the technical details of the 
commodities and services they procure. The Commercial Activities Panel 
report stated that developing and maintaining a skilled acquisition 
workforce is the critical first step in managing this more complex 
procurement environment. The panel also reported that DOD bore the 
brunt of a 22 percent downsizing of the federal acquisition workforce 
in the last decade, going from 96,000 staff in 1991 to about 68,000 in 
fiscal year 2001.

Addressing human capital issues is not just a matter of the size of the 
workforce; it is also a knowledge and skills issue. According to the 
Commercial Activities Panel, it is critically important that federal 
agencies adequately address human capital needs in meeting the current 
and emerging needs of government and its citizens in the most 
effective, efficient, and economical manner possible. This will require 
increased emphasis on training and development, particularly in the 
area of technology.

Description of Five Projects Reviewed:

The five projects in our study varied in how they approached 
outsourcing IT services, such as in using various solicitation methods, 
including holding a public/private competition under the policies 
outlined in OMB Circular A-76 or carrying out a negotiated competitive 
procurement. In addition, the types of services being outsourced 
differed: services ranged from the narrowly focused (e.g., help-desk 
services) to the very broad (e.g., enterprisewide end-to-end 
information services); contract terms ranged from 5 to 15 years 
(assuming all option years are exercised); and estimated contract 
values ranged from $23 million to $8.8 billion. Table 2 provides 
information on the variety of IT services and outsourcing approaches 
taken by the projects.

:

Table 2: Profile of Outsourcing Projects Reviewed:

Agency/Project: Air Force/Kirtland Air Force Base's Command, Control, 
Communications, and Computer (C4) Services; Solicitation method: 
Competitive, under OMB Circular A-76 and small business set-aside 
rules; Date of contract award: April 2000; Contract term: 1 year, with 
4 option years; Contract type: Firm, fixed-price; Estimated total 
contract value[A]: $23 million; Project description: Management, 
operations, and maintenance of command, control, communications, and 
computer systems, multimedia services, and information management for 
the Kirtland Air Force Base.

Agency/Project: Army/Network Enterprise Technology Command's (NETCOM) 
Total Army Communications--Southwest Asia (TAC-SWA); Solicitation 
method: Negotiated competitive solicitation; Date of contract award: 
March 2001; Contract term: 1 year, with 4 option years; Contract type: 
Firm, fixed-price; Estimated total contract value[A]: $204 million; 
Project description: Operation and maintenance services, including 
repair, installation, and supply, for communications equipment in 
Southwest Asia.

Agency/Project: Military Health System/Information Technology 
Organization (MHS/ITO) Help Desk; Solicitation method: Negotiated 
competitive solicitation; Date of contract award: June 2001; Contract 
term: 1 year, with 7 option years; Contract type: Firm, fixed-price 
with incentive awards; Estimated total contract value[A]: $71 million; 
Project description: Call and help-desk services for all MHS software 
applications.

Agency/Project: Department of the Navy/Navy and Marine Corps Intranet 
(NMCI); Solicitation method: Negotiated competitive solicitation; Date 
of contract award: October 2000; Contract term: 7 years, with an option 
for 3 additional years; Contract type: Firm, fixed-price with incentive 
awards; Estimated total contract value[A]: $8.8 billion; Project 
description: Department of the Navy-wide end-to-end information 
services through a common computing and communication environment.

Agency/Project: National Imagery and Mapping Agency's (NIMA) 
Information Technology/ Information Services (IT/IS); Solicitation 
method: Sole-source, using a statutory preferential provider (Alaska 
native corporation)[B]; Date of contract award: December 2001; Contract 
term: 1 year, with 14 option years; Contract type: Cost plus award 
fees; Estimated total contract value[A]: $2.1 billion; Project 
description: NIMA-wide IT/IS support services for printing, digital 
replication, networks, distributed and centralized systems and 
services, video and voice communications, information research, and 
help-desk operations.[C].

Source: DOD.

[A] Estimated value if all option years are exercised.

[B] NIMA performed an OMB Circular A-76 analysis and, on the basis of 
this analysis, implemented a direct conversion to a preferential 
provider rather than holding a public/private competition or obtaining 
an agency cost-comparison waiver.

[C] As of February 24, 2003, NIMA had transitioned four of these 
functions to the provider: printing, digital replication, video and 
voice communications, and help-desk operations.

[End of table]

Projects Substantially Used Leading Commercial Practices:

As illustrated in figure 4, the five IT services projects substantially 
used leading commercial practices. Specifically, each project used at 
least 76 percent of the practices.[Footnote 23] Reasons for projects 
implementing different percentages of the practices include differences 
in their individual circumstances and objectives. For example, the 
Army's Total Army Communications--Southwest Asia (TAC-SWA) project, 
which used the fewest number of practices and had the largest number of 
practices that were not applicable, was largely a continuation of an 
existing approach that already relied on the private sector but with 
fewer providers. In contrast, the National Imagery and Mapping Agency's 
(NIMA) Information Technology/ Information Services (IT/IS) project, 
which fully or partially implemented all of the applicable practices, 
involved a significant operational shift (e.g., functions previously 
performed by NIMA staff are now performed by a contractor) and was 
intended to result in substantial process improvements. In addition, 
the three projects that implemented the largest percentage of practices 
also used third-party assistance--including employing a contractor with 
sourcing expertise--to help formulate their sourcing strategy, which 
could account for the extent of their compliance.

Figure 4: Use of Leading Commercial Practices for Outsourcing of IT 
Services, by Project:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

In addition, figure 5 illustrates that project compliance extended to 
each of the phases of our framework. Collectively, the projects fully 
implemented from 70 to 97 percent of the practices in each phase. Phase 
I, determine sourcing strategy, had the lowest percentage of practices 
implemented by the projects (70 percent). This result is not 
inconsistent with a recent Giga Information Group, Inc., survey, which 
found that only half of the 
respondent organizations had documented an IT sourcing 
strategy.[Footnote 24] This approach carries risk since phase I sets 
the tone for the outsourcing initiative within the client organization.

Figure 5: Percentage of Practices Implemented in Each Phase:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

In addition, collectively the projects fully implemented 88 percent of 
the practices (see table 3).[Footnote 25]

Table 3: Percentage of Practices Implemented, by Project:

Phase: Phase I: Determine sourcing strategy (6 practices).; Percentage 
of practices implemented[A]: Air Force
C4 Services project: 50; Percentage of practices implemented[A]: Army 
TAC-SWA project: 33; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 83; Percentage of practices implemented[A]: Navy 
NMCI project: 83; Percentage of practices implemented[A]: NIMA IT/IS 
project: 100; Percentage of practices implemented[A]: Overall: 70.

Phase: Phase II: Define operational model (13 practices).; Percentage 
of practices implemented[A]: Air Force
C4 Services project: 92; Percentage of practices implemented[A]: Army 
TAC-SWA project: 83; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 100; Percentage of practices implemented[A]: Navy 
NMCI project: 77; Percentage of practices implemented[A]: NIMA IT/IS 
project: 100; Percentage of practices implemented[A]: Overall: 90.

Phase: Phase III: Develop the contract (16 practices).; Percentage of 
practices implemented[A]: Air Force
C4 Services project: 64; Percentage of practices implemented[A]: Army 
TAC-SWA project: 83; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 100; Percentage of practices implemented[A]: Navy 
NMCI project: 94; Percentage of practices implemented[A]: NIMA IT/IS 
project: 100; Percentage of practices implemented[A]: Overall: 89.

Phase: Phase IV: Select provider(s) (7 practices).; Percentage of 
practices implemented[A]: Air Force
C4 Services project: 86; Percentage of practices implemented[A]: Army 
TAC-SWA project: 86; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 100; Percentage of practices implemented[A]: Navy 
NMCI project: 100; Percentage of practices implemented[A]: NIMA IT/IS 
project: 71; Percentage of practices implemented[A]: Overall: 89.

Phase: Phase V: Transition to provider(s) (11 practices).; Percentage 
of practices implemented[A]: Air Force
C4 Services project: 82; Percentage of practices implemented[A]: Army 
TAC-SWA project: 88; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 100; Percentage of practices implemented[A]: Navy 
NMCI project: 82; Percentage of practices implemented[A]: NIMA IT/IS 
project: 100; Percentage of practices implemented[A]: Overall: 90.

Phase: Phase VI: Manage provider(s) performance (11 practices).; 
Percentage of practices implemented[A]: Air Force
C4 Services project: 73; Percentage of practices implemented[A]: Army 
TAC-SWA project: 73; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 100; Percentage of practices implemented[A]: Navy 
NMCI project: 91; Percentage of practices implemented[A]: NIMA IT/IS 
project: 82; Percentage of practices implemented[A]: Overall: 84.

Phase: Phase VII: Ensure services are provided (6 practices).; 
Percentage of practices implemented[A]: Air Force
C4 Services project: 83; Percentage of practices implemented[A]: Army 
TAC-SWA project: 100; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 100; Percentage of practices implemented[A]: Navy 
NMCI project: 100; Percentage of practices implemented[A]: NIMA IT/IS 
project: 100; Percentage of practices implemented[A]: Overall: 97.

Phase: Overall; Percentage of practices implemented[A]: Air Force
C4 Services project: 76; Percentage of practices implemented[A]: Army 
TAC-SWA project: 79; Percentage of practices implemented[A]: MHS/ITO 
Help Desk project: 99; Percentage of practices implemented[A]: Navy 
NMCI project: 89; Percentage of practices implemented[A]: NIMA IT/IS 
project: 94; Percentage of practices implemented[A]: Overall: 88.

Source: GAO.

[A] These calculations do not include practices that were not 
applicable to a particular project.

[End of table]

The following provides additional information on the projects' 
implementation of each phase of our framework.

* Phase I: Determine sourcing strategy. In the first phase of our 
outsourcing framework, the client organization determines whether 
internal capability or external expertise can more effectively meet its 
IT needs. The purpose of a sourcing strategy is to achieve the optimal 
balance between internal and external capabilities, activities, 
processes, and services to ensure the achievement of strategic business 
objectives at the lowest risk.[Footnote 26] The five projects' 
implementation of this phase was uneven. In particular, two of the six 
practices in this phase were fully implemented by all five projects, 
but the other four practices were not. Among the practices that were 
implemented by all of the projects was determining the business reasons 
for outsourcing. In addition, the three projects that implemented the 
largest percentage of practices in our framework--the Military Health 
System/Information Technology Organization (MHS/ITO) Help Desk, the 
Department of the Navy's Navy and Marine Corps Intranet (NMCI), and 
NIMA IT/IS projects--used the third-party assistance practice in this 
phase to help formulate their sourcing strategy, which could account 
for the extent of their compliance. For example, the MHS/ITO Help Desk 
project, which implemented the largest percentage of practices, worked 
with the Department of the Interior's GovWorks Program,[Footnote 27] 
the Defense Acquisition University staff, and a private-sector 
contractor to obtain expertise on sourcing strategies.

The practice in this phase that was the most unevenly implemented was 
the benchmarking[Footnote 28] and baselining of existing internal 
services. Of the five projects in our review, (1) one fully benchmarked 
and baselined the productivity of the activity being outsourced before 
making the final sourcing decision, (2) two partially baselined their 
existing activities, and (3) two did not perform benchmark and baseline 
analyses at all. The agencies' reasons for not fully implementing this 
practice included that an executive decision had been made to conduct a 
public/private competition following the OMB Circular A-76 policy, so 
such an analysis would not have affected the sourcing decision, or that 
available documentation to perform such an analysis was limited. 
Leading research firms suggest benchmarking and baselining the entity's 
current processes before outsourcing because only then would it be able 
to determine whether the arrangement has been successful.[Footnote 29] 
In addition, the risk of not fully baselining the existing environment 
is illustrated by the NMCI project. Specifically, the NMCI project's 
baseline of its existing environment was limited because it did not 
include an assessment of its legacy applications since project 
officials decided to rely on a preexisting inventory developed to 
address the Year 2000 challenge. The Navy subsequently found that it 
had substantially underestimated its number of legacy applications, 
that, according to program officials, later contributed to the 
transition period slipping from 2- years to 3- years. Appendix II 
provides additional information on projects' implementation of the 
practices comprising this phase.

* Phase II: Define operational model. The operational model is an 
important mechanism for an organization to compare its plans with the 
expectations that were set when the decision to outsource was made and 
to ascertain whether these plans will enable the organization to meet 
expectations. The five projects had largely implemented the 13 
practices contained in this phase. Specifically, about 90 
percent[Footnote 30] of the practices were implemented. For example, 
all projects implemented the practice that executive leadership be 
established to facilitate the outsourcing effort. NIMA, for instance, 
formed a strategic sourcing office to oversee the IT/IS project. 
Another practice--training the provider on the organization's business 
environment and goals--was fully implemented by one project (in two 
cases, the practice was not applicable). One project that did not fully 
implement this practice was the Air Force's Kirtland Air Force Base's 
Command, Control, Communications, and Computer (C4) Services project. 
Although some training was provided (e.g., Kirtland held an orientation 
session for potential bidders), provider officials stated that they did 
not receive adequate training, which made the transition period more 
difficult. Appendix III provides additional information on projects' 
implementation of the practices comprising this phase.

* Phase III: Develop the contract. A well-written contract is necessary 
for the outsourcing organization to meet its requirements while 
allowing the service provider to make a fair profit. It sets the 
expectations for service levels, delivery of essential services, and 
continuous improvement and should protect the interests of all parties. 
The five projects largely implemented the practices in this phase. 
Specifically, about 89 percent[Footnote 31] of the practices were 
implemented in this phase, and two projects (the MHS/ITO Help Desk and 
NIMA IT/IS projects) implemented all of the practices. Several 
practices in this phase address performance requirements. For example, 
all five projects implemented the practices that called for basing 
performance requirements on business requirements and reviewing and 
updating them periodically. One practice that was not fully implemented 
by two projects was including performance measures that address both 
technical and end-user satisfaction aspects of performance. For 
example, the Army TAC-SWA project included technical performance 
measures in its contract but not measures related to end-user 
satisfaction, even though the contract included help-desk services. 
According to the project official that developed the performance work 
statement in the contract, the command did not include customer 
satisfaction measures because it did not think that it was necessary to 
have a performance standard for the help-desk service. However, without 
such measures, the agency does not have a contractual standard with 
which to judge the provider's performance. Appendix IV provides 
additional information on projects' implementation of the practices 
comprising this phase.

* Phase IV: Select the provider(s). Critical to the success of any 
outsourcing project for IT services is identifying potential providers 
and ultimately selecting a provider(s) that will best meet the needs of 
the agency. The five projects had largely implemented the seven 
practices contained in this phase. About 89 percent of the practices 
were implemented in this phase and two projects (the MHS/ITO Help Desk 
and Navy NMCI projects) used each applicable practice. For example, the 
five projects implemented the practice related to conducting due 
diligence activities to verify provider capabilities before signing the 
contract. In the case of the Army TAC-SWA project, the Network 
Enterprise Technology Command (NETCOM) evaluated the provider's 
financial and past performance information. The Department of the 
Navy's NMCI project also evaluated bidders' past performance and 
performed reference checks. Moreover, as part of its due diligence 
activities, the Navy required all bidders to demonstrate that they had 
experience in implementing large seat management contracts.[Footnote 
32] A practice that was not implemented by two projects was using 
third-party assistance when selecting the provider. Projects that did 
not implement this practice believed that they had adequate in-house 
expertise with outsourcing, making third-party assistance unnecessary. 
However, because third-party assistance provides an independent 
resource that can suggest options or processes that the client 
organization may not be aware of, these projects may have missed an 
opportunity to implement their outsourcing projects more effectively. 
Appendix V provides additional information on projects' implementation 
of the practices comprising this phase.

* Phase V: Transition to provider(s). This phase focuses on the client 
organization's transfer of the IT function to one or more providers. As 
part of this transition, the clear definition of responsibilities and 
the careful consideration of employees' needs matched against the 
client organization's needs enable both the client and provider to 
focus on execution and give staff confidence in their future 
employment. The five projects largely implemented the 11 practices 
associated with this phase. Specifically, about 90 percent[Footnote 33] 
of the practices were implemented in this phase, and two projects 
implemented all of the practices (the MHS/ITO Help Desk and NIMA IT/IS 
projects). Several practices in this phase address dealing with 
employees affected by the outsourcing projects. For example, in the 
four projects in which federal employees were affected, the projects 
provided assistance to those who did not want to transfer to the 
provider, including helping to place them in other positions and 
helping with rsum writing. A related practice is to clearly 
communicate to all employees what is going to happen and when it is 
going to happen. Two projects did not fully implement this practice. 
For example, the Navy used its normal chain of command to communicate 
transition information, but found that implementation of this practice 
was uneven. As a result, some staff did not know current information 
about how NMCI would affect them until the provider was ready to 
contact them regarding their possible transition to the contractor. 
However, according to the NMCI's Director's office, this problem was 
somewhat mitigated by the provider's Web site that provides transition 
information to all NMCI customers/users. Appendix VI provides 
additional information on projects' implementation of the practices 
comprising this phase.

* Phase VI: Manage provider(s) performance. The effectiveness with 
which the provider(s) performance is managed is critical to the 
successful implementation of an outsourcing project. Indeed, according 
to Gartner, Inc., a leading research firm, an outsourcing project can 
be thwarted by poorly designed, funded, and delivered processes for 
managing the delivery of services.[Footnote 34] The five projects 
generally implemented the 11 practices contained in this phase, with 
about 84 percent of the practices being implemented in this phase. For 
example, the practices related to obtaining feedback on provider 
performance were largely implemented. This is important because 
different levels of an agency can have different perceptions about the 
value of the outsourcing project. For example, an outsourcing project 
may be considered successful by the agency's executive management if it 
is focused on controlling costs, but be considered inadequate by 
business managers and users who may be expecting higher levels of 
service. Each of these viewpoints is valid and should be taken into 
account when the provider's performance is evaluated. Two other 
practices--including incentives and penalties in contracts--were fully 
implemented by two projects. Incentives and penalties are important 
because they can help motivate the provider to exceed or meet 
performance requirements. Nonetheless, two projects did not include 
monetary incentives and two projects did not include monetary penalties 
in their contracts. For example, the Army TAC-SWA project did not 
include monetary incentives, although the contracting officer stated 
that incentives might have been useful to motivate the provider to 
exceed performance requirements. Incentives can also help control 
risks. According to a guide on performance-based services acquisition, 
if the incentives in the contract are right and if the provider and 
agency share the same goals, risk is largely controlled and effective 
performance is "almost the
inevitable outcome."[Footnote 35] Appendix VII provides additional 
information on projects' implementation of the practices comprising 
this phase.

* Phase VII: Ensure services are provided. Although outsourcing 
transfers responsibility for performing the service to the provider(s), 
the client organization is ultimately responsible for ensuring that 
services are provided and that end-user needs are met. Accordingly, it 
is critical that the agency ensure that services are provided. The 
projects had implemented 97 percent of these practices, and four 
projects implemented all of them. For example, every project monitored 
the providers' work. In the case of the Air Force's C4 Services 
project, quality assurance evaluators monitored the provider's work to 
identify problems or trends in accordance with the project's quality 
assurance surveillance plan. The results were reported to the 
contracting officer and to the functional area chief for resolution. 
Another practice, using customer satisfaction surveys, was fully 
implemented by four of the five projects. However, the Air Force 
project did not conduct, or require its contractor to conduct, customer 
satisfaction surveys. Although the provider surveys staff annually, the 
Air Force is nevertheless relying on the provider to voluntarily 
implement an important practice for determining how customers view the 
services being delivered and whether changes need to be made. Appendix 
VIII provides additional information on projects' implementation of the 
practices comprising this phase.

Leveraging Lessons Learned DOD-wide Could Assist Other DOD Projects:

We have previously reported on the importance of collecting and 
disseminating lessons learned.[Footnote 36] For example, a critical 
activity in IT investment management is establishing a process for 
developing and capturing lessons learned in a written product or 
knowledge base and disseminating them to decision-makers.[Footnote 37] 
In addition, one of the practices in our framework for outsourcing IT 
services addresses incorporating lessons learned from peers who have 
engaged in similar sourcing decisions. Use of lessons learned is a 
principal component of an organizational culture committed to 
continuous improvement. Sharing such information serves to communicate 
acquired knowledge more effectively and to ensure that beneficial 
information is factored into planning, work processes, and activities. 
Lessons learned can be based on positive experiences or on negative 
experiences that result in undesirable outcomes.

Although DOD has taken action to gather and disseminate lessons learned 
and best practices on general acquisition issues, these efforts 
generally do not focus on outsourcing or include sharing the lessons 
learned from IT outsourcing projects across the department. 
Specifically, a number of DOD Web sites provide guidance, lessons 
learned, and best practices related to general acquisition issues. 
However, using these sites to locate specific information on IT 
outsourcing best practices and lessons learned can be time-consuming 
and difficult because so many topics and information sources are 
provided. Specifically, MHS/ITO Help Desk project officials said that 
searching numerous Web sites to get relevant information to address 
questions and concerns about outsourcing IT services can consume hours. 
For example, when we entered the keywords "IT outsourcing" and "best 
practices" into the search feature on the Office of the Undersecretary 
for Defense of Acquisition, Technology, and Logistics site, ACQWeb 
(www.acq.osd.mil) in early March, it provided us with links to 1,251 
documents. Likewise, lessons learned covers so many topics that it is 
difficult to search for an applicable IT lesson. For example, when we 
used the phrase "lessons learned in IT outsourcing" no documents were 
identified on the ACQWeb, but when we inserted "lessons learned" and 
"IT outsourcing" links to more than 1,700 documents were produced.

One DOD Web site, Share A-76!, was established to address one of our 
previous recommendations,[Footnote 38] that is, to establish a 
framework for identifying and analyzing best practices and lessons 
learned from competitive sourcing studies and disseminating them DOD-
wide. Share A-76! promotes the sharing of best practices and lessons 
learned related to one form of outsourcing that was conducted under the 
OMB Circular A-76 competitive sourcing process. Among other things, the 
site contains guidance, links to other relevant sites, sample 
documents, and a best practices library that communicates field staff 
experiences and advice about the Circular A-76 process. A NIMA project 
official said that NIMA's staff routinely accesses the Web site because 
it contains a wealth of information on policies, procedures, lessons 
learned, and links to other outsourcing sites. The DOD analyst 
responsible for Share A-76! estimated that the site receives about 
12,000 visits per month and said that on the basis of E-mails and 
anecdotes, user satisfaction is favorable. However, this Web site is 
specific to the OMB Circular A-76 process, which may not apply to other 
types of outsourcing. For example, Circular A-76's policy pertains to 
public/private competitions and requires that the final evaluation 
between the government and the private sector be based exclusively on 
cost.

DOD acquisition and IT officials acknowledged that there is no 
mechanism in DOD to easily share and leverage lessons learned relating 
to outsourcing IT services. However, these officials agreed that a 
departmentwide effort to identify, capture, and disseminate lessons 
learned and leading practices of projects with experience in carrying 
out IT outsourcing could offer valuable insights and new ideas that 
would benefit others. Moreover, officials from three of the projects in 
our review told us that there is value in collecting and disseminating 
the knowledge acquired from IT outsourcing projects in a systematic 
manner across the department.

Each of the projects in our review identified knowledge and experience 
gained from their approaches to outsourcing IT services that could 
offer insights and practices for other ongoing and future projects to 
consider. For example:

* MHS developed specific guidance and lessons learned for implementing 
a performance-based incentive contract for help-desk operations.

* The Department of the Navy's NMCI project has developed a series of 
lessons learned related to transitioning to the provider that is being 
shared within the NMCI community; one example was that all personnel 
should be available during scheduled testing and deployment.

* NIMA has experience in contracting techniques emphasizing a 
partnering approach with providers to refine requirements and establish 
a common understanding of costs.

In addition, a departmental IT outsourcing knowledge-sharing approach 
could include links to information about other government agencies' IT 
outsourcing projects. For example, our 2002 report on desktop 
outsourcing includes an extensive discussion of lessons learned by 
agencies that have implemented this type of IT services 
outsourcing.[Footnote 39]

Developing an effective lessons learned activity is not easy. For 
example, NMCI officials said that for a lessons learned initiative to 
be effective, a process must exist that is clearly understood by 
everyone and allows capturing and sharing of knowledge to occur with 
minimum effort. Other challenges in developing an effective lessons 
learned process were outlined by the Share A-76! analyst. The analyst 
stated that only a small number of site users have contributed lessons 
learned to the Share A-76! Web site, which she attributed, in part, to 
the amount of time and effort needed to document and obtain agreement 
by all levels of the organization on the lessons learned. In addition, 
the analyst stated that there is reluctance to share negative lessons, 
and often the review and approval process sanitizes best practice 
information so that it becomes too general to be most helpful to users. 
Such challenges can be overcome by executive-level support. Indeed, DOD 
acquisition and IT officials stated that for lessons learning 
activities to be effective, senior management must devote support and 
resources to the effort. This is consistent with our prior work, which 
showed that knowledge can be effectively shared only when employees are 
given adequate time as well as established places where they can 
actually transfer knowledge.[Footnote 40]

Last year, we outlined a generic lessons learned process that could be 
used to guide the development of such a process for outsourcing IT 
services.[Footnote 41] Although the mechanism or processes used to 
collect, share, and disseminate lessons learned may vary, in general 
such a process comprises four main elements: collection, verification, 
storage, and dissemination. The collection process involves the capture 
of information through structured and unstructured processes. 
Verification serves to verify the correctness and applicability of 
lessons submitted. The storage aspect of lessons learned usually 
involves incorporating the lessons into an electronic database for the 
dissemination and sharing of information, including the ability to 
conduct information searches. The final element, and the most 
important, is the dissemination of lessons learned, since lessons are 
of little benefit unless they are distributed and used by people who 
will benefit from them. Lessons can be "pushed," or automatically 
delivered to a user, or "pulled" in situations where a user must 
manually search for them. Lessons can also be disseminated with an 
assigned priority descriptor, which denotes the risk, immediacy, and 
urgency of the lessons learned content.

Conclusions:

The projects in our review substantially implemented leading commercial 
practices for outsourcing IT services, which has increased each 
project's probability of success. Capturing how these projects 
operationalized leading commercial practices could help other IT 
services outsourcing projects succeed. Although currently there is no 
such DOD-wide mechanism, such as an electronic tool, to easily share 
and leverage lessons learned, DOD IT and acquisition officials agreed 
that a departmentwide effort to identify, capture, and disseminate 
lessons learned could offer valuable insights and new ideas that would 
benefit others. Lessons learned that are pragmatic and easily 
accessible could give DOD managers a more informed understanding of the 
important issues to be addressed when making outsourcing decisions, as 
well as the factors to be considered to help ensure the success of 
these endeavors. DOD managers can also benefit from lessons learned on 
the basis of negative experiences. The projects in our review were well 
into implementation, and therefore, at this late stage, we see little 
advantage for them to revisit practices that were not implemented. 
Nevertheless, an electronic tool for capturing and disseminating 
lessons learned would allow the rest of DOD to benefit from the 
negative consequences and increased risks associated with those 
practices that the projects did not implement.

Developing a lessons learned mechanism is not easy; thus, senior 
management support and resources are keys to success. Without such 
support driving the capture and dissemination of lessons learned, DOD 
is losing an opportunity for wider application of leading practices and 
thus better ensuring that its IT outsourcing efforts are successful.

Recommendations:

To assist DOD organizations in planning and implementing outsourcing 
projects for IT services, we recommend that the Secretary of Defense 
direct the Undersecretary of Defense for Acquisition, Technology, and 
Logistics, working in conjunction with the Assistant Secretary of 
Defense for Command, Control, Communications, and Intelligence (C3I), 
to provide senior management support and adequate resources to develop 
and implement an electronic tool to capture and disseminate examples 
and lessons learned from actual IT outsourcing projects. These examples 
and lessons learned, at a minimum, should include the results of our 
review of the five projects discussed in this report.

We also recommend that the Secretary of Defense direct the 
Undersecretary of Defense for Acquisition, Technology, and Logistics, 
working in conjunction with the Assistant Secretary of Defense for 
Command, Control, Communications, and Intelligence (C3I), to ensure 
that the method used to gather information for this electronic tool 
incorporate the main elements of a lessons learned process--namely, 
collection, verification, storage, and dissemination.

Agency Comments and Our Evaluation:

In written comments on a draft of our report, signed by DOD's Principal 
Director, Deputy Assistant Secretary of Defense (Deputy Chief 
Information Officer), the department partially concurred with our 
recommendations. Specifically, DOD agreed that capturing lessons 
learned in the development and implementation of its IT outsourcing 
initiatives is important to continually improve its outsourcing methods 
and results achieved. The department also stated that before deciding 
on a specific method to achieve this aim, it intends to explore a 
variety of mechanisms that could be used. In particular, the department 
stated that it currently has several processes and communities of 
interest that collect and disseminate lessons learned in other areas, 
which are logical starting points for determining the best path 
forward. DOD's written comments are reproduced in appendix IX.

We agree that it is prudent to explore various alternatives to 
leveraging lessons learned from DOD's IT services outsourcing 
experiences. Our recommendations are not prescriptive as to the 
electronic method to be used to capture and disseminate lessons 
learned. Therefore, the department's plan to explore various 
alternatives is consistent with our recommendations.

:

We are sending copies of this report to the appropriate congressional 
committees, the Secretary of Defense, and the Director of the Office of 
Management and Budget. We will also provide copies to others upon 
request. In addition, the report will be available at no charge on the 
GAO Web site at http://www.gao.gov.

If you have any questions on matters discussed in this report, please 
contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director, 
at (202) 512-9556. We can also be reached by E-mail at hiter@gao.gov 
and lambertl@gao.gov, respectively. Other contacts and key contributors 
to this report are listed in appendix X.

Randolph C. Hite
Director, Information Technology Architecture and Systems Issues:

Signed by Randolph C. Hite:

[End of section]

Appendixes :

Appendix I: Objectives, Scope, and Methodology:

Our objectives were to determine (1) the extent to which selected 
Department of Defense (DOD) information technology (IT) services 
outsourcing projects use leading commercial practices as specified in 
our framework and (2) whether DOD is sharing lessons learned from its 
IT outsourcing projects across the department.

To determine the extent to which selected DOD outsourcing projects for 
IT services use leading commercial practices, we identified the 
practices in our November 2001 report on leading commercial 
practices[Footnote 42] that (1) are typically applied at the project 
level and (2) were verifiable through documentation and interviews. 
Because DOD did not centrally maintain a list of outsourcing projects 
for IT services, we asked the department to identify candidate projects 
for our evaluation. From this list, we selected the following five 
projects for our review: (1) Air Force Kirtland Air Force Base's 
Command, Control, Communications, and Computer (C4) Services project; 
(2) Army Network Enterprise Technology Command's (NETCOM) Total Army 
Communications - Southwest Asia (TAC-SWA) project; (3) Military Health 
System/Information Technology Organization (MHS/ITO) Help Desk 
project; (4) Department of the Navy's Navy and Marine Corps Intranet 
(NMCI) project; and (5) National Imagery and Mapping Agency (NIMA) 
Information Technology/Information Services (IT/IS) project. We chose 
each project on the basis of the following criteria: (1) no more than 
one project from each military service and two agencies, (2) 
illustrative example of DOD IT outsourcing, (3) dollar value greater 
than $10 million, and (4) enough time elapsed for services to have been 
delivered and performance measured.

At our request, each project completed a self-assessment on whether and 
how it implemented leading commercial practices. We reviewed the agency 
self-assessments and accompanying documentation and interviewed the 
appropriate agency project officials to verify whether the practices 
were followed. In addition, we interviewed representatives from each of 
the providers associated with these projects.

We also researched additional information on commercial practices in 
our November 2001 guide[Footnote 43] and the challenges the federal 
government faces in implementing them. Specifically, we performed a 
literature search, which included reviewing reports issued by leading 
research firms, such as Gartner, Inc., and Giga Information Group, Inc. 
In addition, we interviewed representatives from industry organizations 
that have an interest in outsourcing IT services, including the 
Coalition for Government Procurement; the Information Technology 
Association of America; the Professional Services Council; and 
Acquisition Solutions, Inc.

To determine whether DOD is sharing lessons learned from its IT 
outsourcing projects across the department, we identified and reviewed 
various approaches that DOD currently uses to capture and disseminate 
such information. This included identifying and reviewing various Web 
sites and performing key word searches on these sites to identify 
lessons learned for outsourcing IT services. We also interviewed 
applicable DOD acquisition and IT officials. Finally, we identified and 
reviewed a generic lessons learned process contained in our January 
2002 report on the National Aeronautics and Space Administration's 
lessons learned mechanisms.[Footnote 44]

We performed our work at the Army's NETCOM in Ft. Huachuca, Arizona; 
the MHS/ITO Help Desk project office in Falls Church, Virginia; the 
NMCI Director's office in Crystal City, Virginia; and NIMA's 
headquarters in Bethesda, Maryland. We conducted our review between May 
2002 and early March 2003 in accordance with generally accepted 
government auditing standards.

[End of section]

Appendix II: Projects' Implementation of Phase I: Determine Sourcing 
Strategy:

[See PDF for image]

[End of figure]

In the first phase of our outsourcing framework, the client 
organization determines whether internal capability or external 
expertise can more effectively meet its IT needs. The purpose of this 
sourcing strategy is to achieve the optimal balance between internal 
and external capabilities, activities, processes, and services to 
ensure that strategic business objectives are achieved at the lowest 
risk. Among the factors that an organization should evaluate in 
crafting this strategy are technology, business, financial, and 
personnel requirements and whether it has skilled business and IT 
managers. In addition, according to Gartner, Inc., sound sourcing 
decisions depend on whether IT organizations (1) know and understand 
their business priorities, (2) are prepared to invest in some skills 
and divest others, and (3) identify and assess trade-offs.[Footnote 45]

The six practices in this phase that we used to evaluate the five 
projects are as follows:

* Use third-party assistance with experience in a variety of sourcing 
arrangements when formulating a sourcing strategy.

* Incorporate lessons learned from peers who have engaged in similar 
sourcing decisions.

* Estimate impact of sourcing decision on internal organization.

* Benchmark and baseline productivity of internal services before 
making the final sourcing decision.

* Determine the business reasons for outsourcing IT.

* Determine reasons for outsourcing IT that can improve the 
organization's ability to use and manage technology.

Figure 6 shows that the implementation of the practices by the five 
projects in our review was uneven.

Figure 6: Project Implementation of Phase I: Determine Sourcing 
Strategy:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 4 provides detailed information on whether and how each project 
implemented each of the six practices in this phase.

Table 4: Summary of Projects' Use of Phase I Practices: Did The Project
Use the Practice?:

Practice: Use third-party assistance with experience in a variety of 
sourcing arrangements when formulating a sourcing strategy.; Did the 
project use the practice?: Air Force C4 Services: No--According to Air 
Force project officials, they decided that they had sufficient in-house 
expertise, and the project had no funding available to employ a 
contractor. However, the former functional area chief told us that 
using third-party assistance would have been beneficial because the 
requirements would have been better written. According to this 
official, the requirements had to be substantially rewritten 2 years 
after contract award.; Did the project use the practice?: Army TAC-SWA: 
No--According to a TAC-SWA project official, the Army decided that it 
had sufficient expertise in-house since the TAC-SWA contract was a 
consolidation of three existing contracts.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--For example, MHS worked with the 
Department of the Interior's GovWorks Program and the Defense 
Acquisition University staff to obtain expertise on sourcing 
strategies. Also, MHS employed a contractor with sourcing expertise.; 
Did the project use the practice?: Navy NMCI: Yes--The Navy employed a 
contractor with sourcing expertise and contacted other government 
entities about their experiences.; Did the project use the practice?: 
NIMA IT/IS: Yes--NIMA employed a contractor with sourcing expertise.

Practice: Incorporate lessons learned from peers who have engaged in 
similar sourcing decisions.; Did the project use the practice?: Air 
Force C4 Services: Yes--Primarily from other Department of Defense 
(DOD) projects that implemented Office of Management and Budget (OMB) 
Circular A-76 policies.; Did the project use the practice?: Army TAC-
SWA: Yes--According to a TAC-SWA project official, it used lessons 
learned from its prior contracts for these services and another similar 
Army contract.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--Primarily from industry peers and MHS's prior help-desk function.; 
Did the project use the practice?: Navy NMCI: Yes--Navy officials 
stated that there were no peers that had engaged in similar sourcing 
decisions because no other outsourcing project was of as large a scale 
as NMCI. However, the Navy did talk to members of private industry and 
the National Aeronautics and Space Administration on their more limited 
efforts.; Did the project use the practice?: NIMA IT/IS: Yes--A NIMA 
contractor provided the agency with a report on industry best 
practices. In addition, NIMA held discussions with the National 
Security Agency on its outsourcing effort.

Practice: Estimate impact of sourcing decision on internal 
organization.; Did the project use the practice?: Air Force C4 
Services: Yes--The Air Force estimated that there would be substantial 
internal impact, such as to its staff, due to its decision to 
outsource.; Did the project use the practice?: Army TAC-SWA: No--The 
Army did not analyze the impact on its internal organization because, 
according to the TAC-SWA program manager, NETCOM was outsourcing a 
function that was already contracted out.[A]; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--MHS assessed the staff and financial 
impact of its sourcing decision.; Did the project use the practice?: 
Navy NMCI: Yes--The Navy performed an assessment of staffing and other 
impacts, such as cost, related to its sourcing decision.; Did the 
project use the practice?: NIMA IT/IS: Yes--NIMA performed an 
assessment of staffing and other impacts, such as cost, related to its 
sourcing decision.

Practice: Benchmark and baseline productivity of internal services 
before making the final sourcing decision.; Did the project use the 
practice?: Air Force C4 Services: No--The Air Force did not implement 
this practice because, according to project officials, an executive 
decision was made to outsource the C4 services following the policies 
contained in OMB Circular A-76. Accordingly, Air Force officials stated 
that such analyses would not have affected the final sourcing 
decision.; Did the project use the practice?: Army TAC-SWA: No--The 
Army did not perform a benchmark or baseline analysis because, 
according to the TAC-SWA program manager, NETCOM was outsourcing a 
function that was already contracted out.; Did the project use the 
practice?: MHS/ITO Help Desk: Limited--MHS attempted to establish a 
baseline of its prior environment but, according to project officials, 
available documentation was limited to historical trouble ticket 
workload data; therefore, this baseline was a best estimate. In 
addition, MHS project officials stated that a contractor performed a 
benchmark analysis, but they did not provide supporting documentation.; 
Did the project use the practice?: Navy NMCI: Limited--The Navy had a 
private-sector firm benchmark its environment against seven large 
public and private organizations. In addition, the Navy and the Marine 
Corps performed an analysis at a sample of representative locations to 
obtain a baseline. However, this baseline did not include an assessment 
of the Department of the Navy's legacy applications since project 
officials decided to rely upon an inventory developed in addressing the 
Year 2000 challenge. The Navy subsequently found that it had 
substantially underestimated its number of legacy applications. 
According to NMCI program officials, this underestimation contributed 
to the transition period slipping from 2- years to 3- years.; Did the 
project use the practice?: NIMA IT/IS: Yes--A NIMA contractor baselined 
the existing NIMA environment and benchmarked it to peers.

Practice: Determine the business reasons for outsourcing IT.; Did the 
project use the practice?: Air Force C4 Services: Yes--To achieve cost 
savings and to shift military personnel to other work.; Did the project 
use the practice?: Army TAC-SWA: Yes--To address shortages in military 
personnel to perform its mission.; Did the project use the practice?: 
MHS/ITO Help Desk: Yes--To achieve increased productivity and customer 
satisfaction and to decrease costs.; Did the project use the practice?: 
Navy NMCI: Yes--To have private industry capitalize infrastructure 
improvements that were needed to quickly and securely share knowledge 
around the globe.; Did the project use the practice?: NIMA IT/IS: Yes-
-To obtain improved customer services and decreased costs. Also, this 
function was determined to be a commercial function under the Federal 
Activities Inventory Reform Act.[B].

Practice: Determine reasons for outsourcing IT that can improve the 
organization's ability to use and manage technology.; Did the project 
use the practice?: Air Force C4 Services: No--According to Kirtland Air 
Force Base officials, the Air Force decided to hold a public/private 
competition following OMB Circular A-76 policies; therefore, improving 
its ability to use and manage technology was not a factor in 
determining its outsourcing strategy.; Did the project use the 
practice?: Army TAC-SWA: No--According to TAC-SWA project officials, 
improving its ability to use and manage technology was not a factor in 
determining its outsourcing strategy.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--To improve IT expertise and 
knowledge.; Did the project use the practice?: Navy NMCI: Yes--To 
improve, for example, system security, interoperability, reliability, 
and network response.; Did the project use the practice?: NIMA IT/IS: 
Yes--To achieve better IT management performance.

Source: GAO.

[A] Having had the activity previously performed by a contractor does 
not obviate the need to estimate the impact of a sourcing decision on 
the internal organization--there are still risks involved, such as the 
potential disruption of services during the transfer to the new 
contractor. In fact, the staff from the incumbent contractor did not 
transition to the TAC-SWA provider, and the provider had difficulty 
filling these slots within the schedule outlined in the contract.

[B] The Federal Activities Inventory Reform Act of 1998 requires 
federal agencies to prepare and submit to OMB, by June 30 of each year, 
inventories of the commercial activities performed by federal 
employees.

[End of table]:

[End of section]

Appendix III: Project's Implementation of Phase II: Define Operational 
Model:

[See PDF for image]

[End of figure]

Critical to the successful outsourcing relationship is an operational 
model for guiding the structure of the contract and the plans for 
transition. In defining the operational model, client organizations 
formalize executive leadership, team composition, client 
responsibilities, and operating relationships between the client and 
provider. The operational model helps the organization to compare its 
plans with the expectations that were set as the initial decision to 
outsource was made and to ascertain whether these plans will enable the 
organization to meet those objectives. An important aspect of the 
operational model is an explicit understanding of how the client 
organization plans to communicate its needs and provide feedback to the 
provider. In addition, communication between the business and IT 
offices within the client organization is always critical. This is 
particularly true in the case of outsourcing because the IT service 
provider is outside the client organization and disconnects are more 
likely to occur. Therefore, organizational processes to facilitate good 
communication are critical.

The 13 practices in this phase that we used to evaluate the five 
projects are as follows:

* Establish executive leadership for IT to facilitate the outsourcing 
initiative.

* Continually communicate/clarify outsourcing objectives, while 
correcting misinformation that affects the organization.

* Establish a core group of people who will be involved in all phases 
of outsourcing.

* Select a person involved in the negotiation of the contract to manage 
the outsourcing relationship.

* Create and define a contract management structure with operational 
points of contact and managers.

* Define the role of internal IT managers and business leaders.

* Ensure that the right skills are in place to support the outsourcing 
relationship.

* Establish a point of contact high in the provider management 
structure for elevating provider performance concerns.

* Have provider establish an on-site support team to serve as liaison 
between client and provider.

* Train provider on client business environment and goals.

* Select or develop standard tools for managing the relationship.

* Use third-party assistance to take advantage of expertise from a 
variety of outsourcing arrangements in defining the operational model.

* Ensure that the provider management team has prior experience in the 
client's field of business.

Figure 7 illustrates that the five projects in our review largely 
implemented the practices.

Figure 7: Project Implementation of Phase II: Define Operational Model:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 5 provides detailed information on whether and how each project 
implemented each of the 13 practices in this phase.

Table 5: Summary of Projects' Use of Phase II Practices:

Practice: Establish executive leadership for IT to facilitate the 
outsourcing initiative.; Did the project use the practice?: Air Force 
C4 Services: Yes--The initiative was planned by a steering group made 
up of representatives from various major offices at Kirtland Air Force 
Base. Subsequent to contract award, the Air Force established a 
functional area chief to manage the initiative, and a Lieutenant 
Colonel was appointed to be responsible for this project.; Did the 
project use the practice?: Army TAC-SWA: Yes--NETCOM used the existing 
leadership in its offices of operations and logistics to provide 
executive leadership.; Did the project use the practice?: MHS/ITO Help 
Desk: Yes--MHS used its existing leadership structure and processes. 
This structure includes a program executive office, steering committee, 
and program review board.; Did the project use the practice?: Navy 
NMCI: Yes--In 1999, the Navy established a program executive office for 
IT primarily to support the NMCI outsourcing effort. The Congress later 
directed the Navy to identify a single individual whose sole 
responsibility would be to oversee and direct the NMCI program. As a 
result, in February 2002, the Navy established the NMCI Director's 
Office to take over responsibility for NMCI.; Did the project use the 
practice?: NIMA IT/IS: Yes--The NIMA enterprise transformation 
directorate formed a strategic sourcing office to provide executive 
oversight of the effort.

Practice: Continually communicate/clarify outsourcing objectives, 
while correcting misinformation that affects organization.; Did the 
project use the practice?: Air Force C4 Services: Yes--The Air Force 
provided communication through, for example, briefings and meetings 
with employees.; Did the project use the practice?: Army TAC-SWA: Yes-
-According to TAC-SWA project officials, the applicable NETCOM 
officials were informed about the initiative and provided comments on 
the draft performance work statement. TAC-SWA project officials also 
brief new commanders on the contract before they are transferred to the 
military theater.; Did the project use the practice?: MHS/ITO Help 
Desk: Yes--MHS provided updates to the Chief Information Officer (CIO) 
and program review boards, provided briefings to the deputy surgeon 
general, and published questions and answers.; Did the project use the 
practice?: Navy NMCI: Yes--The Navy established an action collaboration 
team structure to involve the Navy community in the NMCI communications 
process. Also, the NMCI Information Bureau initiated press conferences, 
briefings, site visits, informational pamphlets, and promotional 
material. In addition, the NMCI Director and other staff provide 
numerous briefings and presentations to commands, industry gatherings, 
and government officials. Finally, Web sites and Web-based 
collaboration sites were established to facilitate communications.; Did 
the project use the practice?: NIMA IT/IS: Yes--NIMA created an 
internal Web site to post information on the outsourcing project, 
distributed periodic global electronic mails, and held town hall 
meetings.

Practice: Establish a core group of people who will be involved in all 
phases of outsourcing.; Did the project use the practice?: Air Force C4 
Services: Yes--The core group included the contracting officer, 
functional area chief, and manpower specialist.; Did the project use 
the practice?: Army TAC-SWA: Yes--According to TAC-SWA project 
officials, the core group included representatives from logistics, 
operations, and resource management as well as the contracting 
officer.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to program officials, the core group included representatives 
from the MHS program executive office, MHS IT program offices, military 
department chief information officers' offices, and Interior's GovWorks 
organization, which provided contracting services.; Did the project use 
the practice?: Navy NMCI: Yes--The program executive office for IT and 
the NMCI Director's organization comprise the core group responsible 
for managing the outsourcing initiative.; Did the project use the 
practice?: NIMA IT/IS: Yes--NIMA established the Strategic Sourcing 
Office, which was dedicated to managing and facilitating the 
outsourcing initiative.

Practice: Select a person involved in the negotiation of the contract 
to manage the outsourcing relationship.; Did the project use the 
practice?: Air Force C4 Services: Yes--The contracting officer.[A]; Did 
the project use the practice?: Army TAC-SWA: Yes--The contracting 
officer.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to MHS program officials, the chair of the contract 
evaluation committee and the individual in charge of transition are 
responsible for managing the outsourcing relationship.; Did the project 
use the practice?: Navy NMCI: Yes--The primary contracting officer.; 
Did the project use the practice?: NIMA IT/IS: Yes--A lead contracting 
officer.

Practice: Create and define contract management structure with 
operational points of contact and managers.; Did the project use the 
practice?: Air Force C4 Services: Yes--The structure was defined by the 
roles and responsibilities of the contracting officer and her staff.; 
Did the project use the practice?: Army TAC-SWA: Yes--The overall 
responsibility for the contract rests with the contracting officer. The 
contracting officer's representative acts as a liaison between the 
government and the contractor.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--Responsibility for contract management is jointly 
held by the contracting officer, who is part of Interior's GovWorks 
organization, and the MHS program office, which provides day-to-day 
oversight of the provider.; Did the project use the practice?: Navy 
NMCI: Yes--The primary contracting officer establishes procedures and 
controls necessary for effective contractual oversight of the NMCI 
initiative and has a matrix relationship with the NMCI Director. The 
primary contracting officer certifies contracting officer 
representatives for the NMCI contract to provide technical coordination 
efforts.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA has 
appointed an operational point of contact for each of the seven 
functional areas being outsourced, which are documented in a "rules of 
engagement" agreement between the government and the contractor.

Practice: Define the role of internal IT managers and business 
leaders.; Did the project use the practice?: Air Force C4 Services: 
Yes--Kirtland Air Force Base did not initially define the roles of its 
internal IT and business managers. However, subsequent to contract 
award, Kirtland Air Force Base established a CIO committee and 
configuration control board comprising the functional area chief 
(Kirtland's CIO) and representatives from various business areas. 
Although not exclusively devoted to the C4 Services project, these 
groups help provide direction to the project.; Did the project use the 
practice?: Army TAC-SWA: Limited--TAC-SWA project officials explained 
the roles of the principal staff involved with this project but 
generally did not have supporting documentation defining these roles.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--MHS's help-
desk performance assessment plan defines the roles of various groups 
and individuals associated with the program.; Did the project use the 
practice?: Navy NMCI: Yes--The NMCI execution plan describes the roles 
of internal and external organizations that directly or indirectly 
affect the management of NMCI and explains in detail the duties and 
responsibilities of the program executive office for IT and the NMCI 
program management offices.; Did the project use the practice?: NIMA 
IT/IS: Yes--NIMA transformation teams for the activities being 
outsourced define the roles of their managers and leaders.

Practice: Ensure that the right skills are in place to support the 
outsourcing relationship, including those dealing with; ; * contract 
management,; * financial management,; * IT management,; * negotiation 
strategies,; * teaming and interpersonal relationships,; * project 
management, and; * relationship management.; Did the project use the 
practice?: Air Force C4 Services: Yes--According to Air Force 
officials, the right skills are in place.; Did the project use the 
practice?: Army TAC-SWA: Yes--According to the TAC-SWA project 
officials, the Army has created several positions to ensure that the 
right skills are in place to manage the outsourcing relationship, 
including a contracting officer, legal advisor, operations and 
logistics personnel as well as a NETCOM unit commander. According to 
Army TAC-SWA officials, each position employs an individual with the 
necessary skills to support the outsourcing relationship.; Did the 
project use the practice?: MHS/ITO Help Desk: Yes--According to MHS 
program officials, the Tri-Service Management Program Office, GovWorks 
(the Interior organization that provides contracting assistance to 
MHS), and a contractor collectively employ the skill sets needed to 
support the outsourcing relationship.; Did the project use the 
practice?: Navy NMCI: Limited--The Director, NMCI, has staff that 
report directly or are matrixed with him that are responsible for 
performing all but two of these functions. Specifically, at this time, 
NMCI does not have staff assigned to support teaming and interpersonal 
relationships and relationship management. The Navy recognizes the need 
for these skills and is taking, or plans to take, various actions to 
obtain these skills.; Did the project use the practice?: NIMA IT/IS: 
Yes--According to NIMA, its transformation teams provide the skills 
necessary to support the outsourcing relationship.

Practice: Establish a point of contact high in the provider management 
structure for elevating provider performance concerns.; Did the project 
use the practice?: Air Force C4 Services: Yes--According to the 
contracting officer, concerns can be addressed to the senior vice 
president at the provider headquarters.; Did the project use the 
practice?: Army TAC-SWA: Yes--The provider established a program 
manager as the point of contact for elevating concerns.; Did the 
project use the practice?: MHS/ITO Help Desk: Yes--The provider 
established an executive program manager as the point of contact for 
elevating concerns.; Did the project use the practice?: Navy NMCI: Yes-
-The NMCI Director interacts directly with the provider's program 
executive on a regularly scheduled basis; such meetings would include 
any performance issues.; Did the project use the practice?: NIMA IT/IS: 
Yes--The transformation team charters define procedures for addressing 
issues, including possibly elevating concerns to the provider's general 
manager.

Practice: Have provider establish an on-site support team to serve as a 
liaison between client and provider.; Did the project use the 
practice?: Air Force C4 Services: Yes--The provider has established an 
on-site support team, which is led by the on-site manager.; Did the 
project use the practice?: Army TAC-SWA: Yes--Site managers have been 
established in accordance with the contract.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--The provider's transition plan 
identifies the on-site support team.; Did the project use the 
practice?: Navy NMCI: Yes--The provider establishes site leads that 
remain on-site as the location goes through the cutover to NMCI.; Did 
the project use the practice?: NIMA IT/IS: Yes--The rules of engagement 
define the on-site support team for each major NIMA location.

Practice: Train provider on client business environment and goals.; Did 
the project use the practice?: Air Force C4 Services: Limited--Prior to 
contract award, Kirtland Air Force Base provided an orientation session 
in which they discussed their operations, business environment, and 
goals. Subsequent to contract award, specific Kirtland Air Force Base 
government personnel were responsible for aiding the provider during 
the transition period. However, provider officials stated that they did 
not receive adequate training, which made the transition period more 
difficult.; Did the project use the practice?: Army TAC-SWA: Not 
applicable--The Army contract was a consolidation of prior contracts 
for this activity, and the winning bidder was one of the incumbent 
contractors.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--MHS officials stated that they provided some training to the 
provider on their business environment. In particular, MHS trained and 
certified the provider's staff on the agency's application systems. In 
addition, the provider had previous MHS experience.; Did the project 
use the practice?: Navy NMCI: Limited--The Navy did not provide 
training on its business environment and goals; instead it relied on 
the NMCI design reference mission document, which was included with the 
request for proposals (RFP). This document defined the NMCI operational 
environment. However, a provider official stated that although the 
document was useful, it did not identify Navy enterprisewide 
operations. According to NMCI program officials, the provider somewhat 
mitigated this problem by hiring several highly knowledgeable staff 
from the Department of the Navy shortly after contract award.; Did the 
project use the practice?: NIMA IT/IS: Not applicable--According to the 
project's contracting officer, such training was not necessary because 
the senior officers of the provider are all recent NIMA management 
employees and were already knowledgeable of these areas. For example, 
among those who transitioned to the provider was a former director of 
the hydrographic topographic center of the Defense Mapping Agency, a 
predecessor agency to NIMA.

Practice: Select or develop standard tools for managing the 
relationship (e.g., performance scorecards, enterprise resource 
management system).; Did the project use the practice?: Air Force C4 
Services: Yes--For example, the revised contract outlines the use of 
specific software to help manage the provider's performance.; Did the 
project use the practice?: Army TAC-SWA: Yes--NETCOM uses a monthly 
report from the contracting officer's representative to evaluate the 
provider.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to MHS officials, the program office and the provider have 
selected, for example, an enterprise resource management system.; Did 
the project use the practice?: Navy NMCI: Yes--The Navy is using a 
balanced scorecard process to provide Navy and Marine Corps leadership 
with information to judge how well NMCI is supporting the missions and 
strategies of the department.; Did the project use the practice?: NIMA 
IT/IS: Yes--NIMA used various tools, including twice-a-day performance 
reports for the operational help-desk function and monthly performance 
reports.

Practice: Use third-party assistance to take advantage of expertise 
from a variety of outsourcing arrangements in defining the operational 
model (i.e., defining roles and responsibilities).; Did the project use 
the practice?: Air Force C4 Services: Yes--Although third-party 
assistance was not used at the onset of the C4 Services project, a 
person with prior experience in managing IT services contracts for the 
Air Force was brought in to provide advice. He was subsequently put in 
charge of the project.; Did the project use the practice?: Army TAC-
SWA: No--The Army believed that it had sufficient expertise in-house.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--For example, 
MHS contacted the Interior's GovWorks Program, Defense Acquisition 
University, and others. Also, MHS employed a contractor with sourcing 
expertise.; Did the project use the practice?: Navy NMCI: No--The Navy 
believed it had sufficient internal experience and expertise.; Did the 
project use the practice?: NIMA IT/IS: Yes--NIMA employed contractors 
to develop contractual roles and assist in evaluating the contract.

Practice: Ensure that the provider management team has prior experience 
in the client's field of business.; Did the project use the practice?: 
Air Force C4 Services: Yes--Provider management capabilities and 
experience in providing IT services were evaluation factors in the 
technical evaluation process.; Did the project use the practice?: Army 
TAC-SWA: Yes--Vendor past performance was part of the evaluation 
criteria during source selection and the provider was an incumbent 
contractor with NETCOM.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--The request for quote laid out MHS's evaluation 
criteria for selecting a provider, which included prior experience in 
health care.; Did the project use the practice?: Navy NMCI: Yes--
According to NMCI program officials, the Navy required bidders to 
provide information on their prior experience in related business 
fields. For example, the solicitation required bidders to demonstrate 
experience managing a similar effort of 100,000 or more seats at one 
time.[B]; Did the project use the practice?: NIMA IT/IS: Yes--It was 
expected that most of the provider's workforce would be composed of 
former NIMA employees. Among those who transitioned to the provider was 
a former director of the hydrographic topographic center of the Defense 
Mapping Agency.

Source: GAO.

[A] The Air Force's initial contract was not negotiated because 
Kirtland Air Force Base used an OMB Circular A-76 2-step, sealed bid 
process. However, the contracting officer was involved in the bid 
process as well as in managing the winning contractor.

[B] This threshold could be met through multiple efforts, but at least 
one had to include at least 20,000 seats.

[End of table]:

[End of section]

Appendix IV: Project's Implementation of Phase III: Develop the 
Contract:

[See PDF for image]

[End of figure]

Phase III focuses on the development of the contract, which defines the 
legal terms of the relationship between client and provider. While 
other phases in the outsourcing process describe the need for mutual 
trust and a close, flexible working relationship, this phase focuses on 
the development of the contract, which is the foundation on which a 
working relationship will be built. A well-written contract helps the 
outsourcing organization meet its requirements while allowing the 
service provider to make a fair profit. It sets the expectations for 
service levels, delivery of essential services, and continuous 
improvement and is intended to protect the interests of all parties.

The 16 practices in this phase that we used to evaluate the five 
projects are as follows:

* Base performance requirements on business outcomes.

* Include measures that reflect end-user satisfaction as well as 
technical IT performance.

* Review and update performance requirements periodically.

* Require the provider to meet the minimum performance in each category 
of service.

* Require the provider to achieve escalating performance standards at 
agreed-upon intervals.

* Incorporate sufficient flexibility so that minimum acceptable 
performance can be adjusted as conditions change, as the provider 
becomes more adept at satisfying customer demands, and as improvement 
goals are achieved.

* Use service-level agreements (SLA)[Footnote 46] to clearly articulate 
all aspects of performance, including management, processes, and 
requirements.

* Client and provider work together to define the appropriate number of 
SLAs and appropriate structure for each.

* Specify circumstances under which the provider is excused from 
performance levels mandated by master service agreements.

* Client and provider work together to identify SLAs for which 
compensation is based, while additional SLAs may be defined to manage 
performance.

* The contract should include clauses for (1) determining pricing 
structures; (2) performing customer satisfaction surveys and using the 
results to redefine performance levels; (3) terminating the contract; 
(4) resolving disputes in a timely manner; (5) taking work away, 
without penalty, from provider for nonperformance; (6) declaring a 
significant event that can lead to a change in the contract; (7) 
defining performance requirements; and (8) conducting regularly 
scheduled meetings.

* Consider setting up a master services agreement under which all 
arrangements between client and provider operate.

* Include the appropriate representation from each major organizational 
unit on the contract negotiation team.

* Specify the use of volume purchases to obtain optimal discounts.

* Use third-party assistance in negotiating and developing the 
contract.

* Sign the contract after contract negotiations and final vendor 
selections.

As figure 8 shows, the five projects largely implemented the practices.

Figure 8: Project Implementation of Phase III: Develop the Contract:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 6 provides detailed information on whether and how each project 
implemented each of the 16 practices in this phase.

Table 6: Summary of Projects' Use of Phase III Practices:

Practice: Base performance requirements on business outcomes.; Did 
the project use the practice?: Air Force C4 Services: Yes--The 
performance requirements are contained in the contract and are based on 
business outcomes. For example, the requirements define the percentage 
of system availability needed to support users.; Did the project use 
the practice?: Army TAC-SWA: Yes--The performance requirements are 
outcome based and stated in terms of the level of operation and 
maintenance services required. According to TAC-SWA officials, multiple 
trips to overseas locations were made to identify the performance 
requirements.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--Industry standards and metrics were used to base performance 
requirements on business outcomes.; Did the project use the practice?: 
Navy NMCI: Yes--The performance requirements are included in the 
contract and are based on business outcomes. SLAs establish the 
performance standards and service quality for all types of NMCI seats.; 
Did the project use the practice?: NIMA IT/IS: Yes--Performance-based 
requirements are in the contract, including those related to customer 
satisfaction, process improvement, quality control, and timely and 
accurate completion of requirements.

Practice: Include measures that reflect end-user satisfaction as well 
as technical IT performance.; Did the project use the practice?: Air 
Force C4 Services: Limited--The Air Force measures include technical IT 
performance. However, end-user satisfaction measures are limited to the 
percentage of complaints received and do not measure overall customer 
satisfaction. Project officials did not know why overall end-user 
satisfaction measures were not established.; Did the project use the 
practice?: Army TAC-SWA: Limited--The Army's contract includes 
technical IT performance measures but not end-user satisfaction 
performance measures. For example, the contract includes a requirement 
for help-desk services, but does not include measures related to 
customer satisfaction associated with such services. According to the 
project official that developed the performance work statement in the 
contract, NETCOM did not include such customer satisfaction measures 
because it did not think that it was necessary to have a performance 
standard for that service.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--MHS measures the service provider's technical IT 
performance and uses an electronic customer satisfaction survey to 
assess the quality of help-desk services. Satisfaction rates and number 
of survey responses are tracked and used for both evaluation and 
incentive payments (positive and negative).; Did the project use the 
practice?: Navy NMCI: Yes--The SLAs include measures for technical IT 
performance and customer satisfaction. In addition, the contract 
requires the service provider to measure and report on overall customer 
satisfaction with NMCI services.; Did the project use the practice?: 
NIMA IT/IS: Yes--The SLAs contain technical IT performance measures. In 
addition, after a function is transitioned, the contractor must provide 
a plan that includes measures for customer satisfaction, which the 
provider is contractually obligated to meet.

Practice: Review and update performance requirements periodically.; 
Did the project use the practice?: Air Force C4 Services: Yes--The 
performance requirements were revised about 2 years after contract 
award.; Did the project use the practice?: Army TAC-SWA: Yes--The 
contract is reviewed and modified as needed on the basis of additions 
or changes to such requirements. According to TAC-SWA and provider 
officials, changes are coordinated between the government and the 
contractor before they are finalized.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, 
performance requirements are updated as required and reviewed on an 
annual basis before the decision is made to execute the option year on 
the contract.; Did the project use the practice?: Navy NMCI: Yes--
Performance reviews and adjustments are ongoing. For example, the Navy 
is in the process of refining its SLAs to ensure that they adequately 
reflect technical performance and customer satisfaction needs.; Did the 
project use the practice?: NIMA IT/IS: Yes--According to NIMA, 
requirements are updated as needed or once a year when the government 
is assessing whether to exercise its annual option.

Practice: Require the provider to meet the minimum performance in 
each category of service.; Did the project use the practice?: Air Force 
C4 Services: Yes--The minimum performance is defined in the contract.; 
Did the project use the practice?: Army TAC-SWA: Yes--The contract 
defines minimum performance requirements.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--The minimal acceptable performance 
criteria the contractor must meet are based upon commercial industry 
standards and are defined in the contract.; Did the project use the 
practice?: Navy NMCI: Yes--The SLAs provide the basic level of service 
the contractor must deliver for every NMCI seat.; Did the project use 
the practice?: NIMA IT/IS: Yes--The provider must meet the performance 
standards as specified in the contract, SLAs, and NIMA's performance 
assessment plan.

Practice: Require the provider to achieve escalating performance 
standards at agreed-upon intervals.; Did the project use the practice?: 
Air Force C4 Services: No--According to Kirtland Air Force Base 
officials, they did not include escalating performance standards 
because the focus of this contract was on meeting requirements at the 
least cost.[A]; Did the project use the practice?: Army TAC-SWA: Not 
applicable--According to a TAC-SWA project official, this practice was 
not applicable because NETCOM believed that the performance standards 
in the contract were already high. Provider officials also stated that 
the performance requirements in the contract were high. The Army 
expected, and the contractor agreed, to meet these standards 
immediately.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--The contract defines negative, acceptable, and positive incentive 
ranges associated with escalating performance standards.; Did the 
project use the practice?: Navy NMCI: No--According to NMCI program 
officials, the Navy expected vendors to develop their pricing proposal 
assuming 8 years[B] of providing service in accordance with the SLAs 
included in the RFP.; Did the project use the practice?: NIMA IT/IS: 
Yes--The contract allows for the redefinition of service levels. In 
addition, according to NIMA program officials, the need for escalating 
performance standards will be evaluated on an as-needed basis as well 
as during the semiannual award fee analysis period and during the 
annual review on whether to exercise the contract option. However, such 
escalating standards have not yet been identified.

Practice: Incorporate sufficient flexibility so that minimum 
acceptable performance can be adjusted as conditions change, as the 
provider becomes more adept at satisfying customer demands, and as 
improvement goals are achieved.; Did the project use the practice?: Air 
Force C4 Services: No--The contracting officer said that the 
requirements should be built into the contract, which can be modified 
if necessary.; Did the project use the practice?: Army TAC-SWA: Not 
applicable--According to a TAC-SWA project official, NETCOM believed 
that the performance standards in the contract were already high, and 
provider officials agreed. According to the contracting officer, the 
military theater where performance is delivered is not an area where 
anything less than meeting the defined standards is acceptable.; Did 
the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS 
program officials, the contract incorporates sufficient flexibility to 
adjust minimal acceptable performance as conditions change. For 
example, the contract includes incentives for the service provider to 
introduce new capabilities and new services within the scope of the 
performance work statement without further competition.; Did the 
project use the practice?: Navy NMCI: Yes--The contract includes a 
provision requiring the provider to submit an annual plan for 
technology refreshment and deployment. This plan is to include proposed 
revisions to the contract and an estimate of changes in performance 
that would result.; Did the project use the practice?: NIMA IT/IS: Yes-
-According to NIMA, the performance measures are designed to first 
stabilize NIMA's IT environment and then to be adjusted to bring 
minimum performance into alignment with industry best practices. Also, 
the contract indicates that the SLAs may be further defined throughout 
the contract life.

Practice: Use SLAs to clearly articulate all aspects of performance, 
including management, processes, and requirements.; Did the project use 
the practice?: Air Force C4 Services: Yes--The SLA defines the 
requirements, processes, and who is responsible for meeting the 
requirements.; Did the project use the practice?: Army TAC-SWA: Yes--
The TAC-SWA project did not use SLAs, but the contract addresses the 
management, processes, and requirements associated with the project.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--The contract 
defines the requirements, service performance standards, thresholds, 
objectives, and metrics as well as the requirements for quarterly 
management reviews and quality assurance plans.; Did the project use 
the practice?: Navy NMCI: Yes--According to NMCI officials, SLAs cover 
all aspects of provider performance. For example, the E-mail services 
SLA includes location and frequency of the service, performance 
categories, the performance measures, and methods of measurement.; Did 
the project use the practice?: NIMA IT/IS: Yes--The contract 
incorporates SLAs, which articulate performance requirements and take 
effect when the function is transitioned to the provider.

Practice: Client and provider work together to define the appropriate 
number of SLAs and appropriate structure for each.; Did the project use 
the practice?: Air Force C4 Services: Yes--Although the original SLA 
was prepared by the Air Force, the revised SLA included contractor 
input and concurrence.; Did the project use the practice?: Army TAC-
SWA: Yes--SLAs are not used but, according to TAC-SWA and provider 
officials, they work together on any contract modifications dealing 
with requirements and performance levels. In addition, the partnering 
clause in the contract emphasizes a mutual commitment between 
government and industry to work as a team.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--MHS did not work with the service 
provider to determine the number and structure of these requirements. 
However, according to MHS and provider officials, they have worked 
together on revisions to these requirements.; Did the project use the 
practice?: Navy NMCI: Yes--The SLAs were developed by the Navy with the 
help of a third-party contractor. According to the NMCI Deputy Director 
for Enterprise Operations, it would have been inappropriate to work 
with individual competing contractors before the contract was awarded. 
However, NMCI staff and the service provider are now working together 
on SLA revisions.; Did the project use the practice?: NIMA IT/IS: Yes-
-According to NIMA and service provider officials, they used a 
partnering contracting approach to jointly define, develop, and 
structure the SLAs.

Practice: Specify circumstances under which the provider is excused 
from performance levels mandated by master service agreements.; Did the 
project use the practice?: Air Force C4 Services: Yes--The contract 
includes a clause that the contractor is excused in the event of 
government delay of work.; Did the project use the practice?: Army TAC-
SWA: Yes--The contract includes clauses that the provider is excused 
from performance levels if there are government delays or factors 
beyond its control.; Did the project use the practice?: MHS/ITO Help 
Desk: Yes--For example, the provider's proposal, which is incorporated 
as part of the contract, states that during transition periods, metrics 
will not be reported for incentive and penalty purposes. As a result, 
according to program officials, the provider would be excused from 
required performance levels when baseline requirements are being 
established for new applications.; Did the project use the practice?: 
Navy NMCI: Yes--Contract terms and conditions specify the situations 
where the service provider does not have to meet the SLAs. For example, 
during transition, the service provider does not have to meet the 
performance levels set by the SLAs.; Did the project use the practice?: 
NIMA IT/IS: Yes--The contractor can be excused from meeting performance 
levels with the permission of the contracting officer if circumstances 
occur beyond the service provider's control.

Practice: Client and provider work together to identify SLAs for 
which compensation is based, while additional SLAs may be defined to 
manage performance.; Did the project use the practice?: Air Force C4 
Services: Yes--According to Kirtland Air Force Base officials, the 
service provider had input on the revised SLAs, including those for 
compensation.; Did the project use the practice?: Army TAC-SWA: Not 
applicable--SLAs are not used.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--Before the contract was awarded, MHS did not work 
with the contractor to identify SLAs for which compensation is based. 
However, according to MHS and provider officials, after award, there 
have been instances in which they have worked together on refining the 
SLAs, including those affecting compensation.; Did the project use the 
practice?: Navy NMCI: Yes--Meetings were held with potential bidders to 
obtain their input in defining the SLAs, including discussions of 
compensation. In addition, Navy and the provider are now refining the 
SLAs, including those affecting compensation.; Did the project use the 
practice?: NIMA IT/IS: Yes--NIMA and the service provider work together 
in transformation teams to develop SLAs, including those affecting 
compensation.

Practice: The contract should include clauses for; ; * determining 
pricing structures;; * performing customer satisfaction surveys and 
using the results to redefine performance levels;; * terminating the 
contract, including early terminations;; * resolving contract disputes 
in a timely manner;; * taking work away, without penalty, from the 
provider for nonperformance;; * declaring a significant event that can 
lead to a change in the contract;; * defining performance requirements; 
and; * conducting regularly scheduled meetings.; Did the project use 
the practice?: Air Force C4 Services: Limited--All clauses are included 
in the contract, except a clause pertaining to customer satisfaction 
surveys.; Did the project use the practice?: Army TAC-SWA: Yes--All 
contract clauses are included.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--All contract clauses are included.; Did the project 
use the practice?: Navy NMCI: Yes--All contract clauses are included.; 
Did the project use the practice?: NIMA IT/IS: Yes--All contract 
clauses are included.

Practice: Consider setting up master services agreement under which 
all arrangements between client and provider operate.; Did the project 
use the practice?: Air Force C4 Services: Yes--According to Kirtland 
Air Force Base officials, the contract, which incorporates the 
contractor proposal, governs all arrangements between the government 
and the contractor and is considered the master services agreement.; 
Did the project use the practice?: Army TAC-SWA: Yes--According to TAC-
SWA project officials, the contract is considered the master services 
agreement.; Did the project use the practice?: MHS/ITO Help Desk: Yes-
-According to MHS program officials, the contract is considered the 
master services agreement.; Did the project use the practice?: Navy 
NMCI: Yes--According to NMCI program officials, the contract is 
considered to be a master services agreement.; Did the project use the 
practice?: NIMA IT/IS: Yes--According to NIMA, the contract is 
considered the master services agreement.

Practice: Include appropriate representation from each major 
organizational unit on contract negotiation team.; Did the project use 
the practice?: Air Force C4 Services: Not applicable--The C4 Services 
contract was awarded under a sealed bid process and was not 
negotiated.; Did the project use the practice?: Army TAC-SWA: Yes--
Representatives from the field and headquarters commands and the 
contracting office participated in contract negotiation.; Did the 
project use the practice?: MHS/ITO Help Desk: Yes--Staff from affected 
program management offices and the chief information offices were on 
the negotiation team.; Did the project use the practice?: Navy NMCI: 
Yes--The contracting team was built with experts from each of the major 
systems commands. Also, the source selection evaluation board consisted 
of more than 50 people from various commands.; Did the project use the 
practice?: NIMA IT/IS: Yes--The NIMA contract negotiating team included 
representatives from each major organizational unit.

Practice: Specify the use of volume purchases to obtain optimal 
discounts.; Did the project use the practice?: Air Force C4 Services: 
Not applicable--According to Kirtland Air Force Base officials, such 
volume purchases were not relevant to this contract.; Did the project 
use the practice?: Army TAC-SWA: Not applicable--According to TAC-SWA 
project officials, such volume purchases were not relevant to this 
contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
Call bands[C] are used to obtain optimal discounts on the number of 
calls being fielded to the help desk.; Did the project use the 
practice?: Navy NMCI: Yes--The contract includes volume discounts. For 
example, seat moves, adds, and changes are purchased in bulk only, 
because prices for these actions are lower when purchased in groups of 
250.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA has 
authorized the provider to use government sources, such as the Federal 
Supply Schedule, in procuring products, services, and supplies related 
to this contract. According to the contracting officer, this 
authorization was made so that the provider could take advantage of 
discounts available to the government.

Practice: Use third-party assistance in negotiating and developing 
the contract.; Did the project use the practice?: Air Force C4 
Services: No--This was not done because the Air Force staff believed 
that it had adequate expertise available in-house.; Did the project use 
the practice?: Army TAC-SWA: No--NETCOM officials believed that the 
Army's designated contracting agency had adequate expertise and 
experience.; Did the project use the practice?: MHS/ITO Help Desk: Yes-
-Private-sector and other government organizations provided 
assistance.; Did the project use the practice?: Navy NMCI: Yes--The 
Navy used private-sector firms to assist in developing the overall NMCI 
concept and negotiation strategies as well as in drafting the contract 
documents.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA 
obtained help from (1) private contractors to help develop the 
performance work statement, SLAs, and award fee plan; 
(2) another agency on a particular contract technique; and (3) a 
private-sector firm to compare the vendor's proposal with industry best 
practices and trends and to attend some negotiation meetings to answer 
questions.

Practice: Sign contract after contract negotiations and final vendor 
selections.; Did the project use the practice?: Air Force C4 Services: 
Yes--The contract was signed after negotiations and final vendor 
selection.; Did the project use the practice?: Army TAC-SWA: Yes--The 
contract was signed after negotiations and final vendor selection.; Did 
the project use the practice?: MHS/ITO Help Desk: Yes--The contract was 
signed after negotiations and final vendor selection.; Did the project 
use the practice?: Navy NMCI: Yes--The contract was signed after 
negotiations and final vendor selection.; Did the project use the 
practice?: NIMA IT/IS: Yes--The contract was signed after negotiations 
with the selected vendor.

Source: GAO.

[A] Requiring a provider to meet escalating requirements and focusing 
on achieving results at the least cost are not mutually exclusive 
goals. As the provider becomes more familiar with the client 
organization, it may be able to exceed the original performance 
requirements at the same, or possibly lower, cost.

[B] The NMCI contract was subsequently extended to 7 years, with an 
option for an additional 3 years.

[C] Call bands are call volume ranges used to determine contractor 
pricing.

[End of table]:

[End of section]

Appendix V: Projects' Implementation of Phase IV: Select the 
Provider(s):

[See PDF for image]

[End of figure]

Critical to the success of any project to outsource IT services is the 
identification of potential providers and the ultimate selection of a 
provider(s) that will best meet the needs of the organization. 
Developing a strategy that will lead to the selection of the "right 
contractor" is especially important in a performance-based 
acquisition.[Footnote 47] The overall success of the outsourcing 
project requires the contractor to understand the performance-based 
approach, know or develop an understanding of the organization's 
requirement, have a history of performing exceptionally in the field, 
and have the processes and resources in place to support the mission.

The seven practices in this phase that we used to evaluate the five 
projects are as follows:

* Conduct research on the state of the market, vendors, and technology 
before defining vendor selection criteria.

* Identify and evaluate various sourcing solutions (e.g., single 
vendor, multivendor, and alliance).

* Define a process for selecting vendors to be providers.

* Define vendor selection and evaluation (acceptance) criteria at the 
outset.

* When issuing an RFP, identify services with expected performance 
levels and define client and provider roles and responsibilities.

* Use third-party assistance with expertise in a variety of outsourcing 
arrangements when selecting provider(s), including developing the RFP.

* Conduct due diligence activities to verify vendor capabilities before 
signing the contract.

As figure 9 illustrates, all five projects largely implemented the 
practices.

Figure 9: Project Implementation of Phase IV: Select the Provider(s):

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 7 provides detailed information on whether and how the projects 
in our review implemented each of the seven practices in this phase.

Table 7: Summary of Projects' Use of Phase IV Practices:

Practice: Conduct research on state of the market, vendors, and 
technology before defining vendor selection criteria.; Did the project 
use the practice?: Air Force C4 Services: Yes--The Air Force advertised 
in the Commerce Business Daily and interested parties submitted 
statements of capabilities. This was done before the Air Force 
identified the vendor selection criteria.; Did the project use the 
practice?: Air Force C4 Services: Yes--A Commerce Business Daily notice 
was published identifying TAC-SWA requirements to interested parties. 
In addition, a draft RFP was issued before the final solicitation to 
solicit comments from industry that might affect the requirements. 
According to the contracting officer, comments received from industry 
were incorporated into the final solicitation, where applicable. 
Additionally, a preproposal conference was conducted in the overseas 
operations location to inform industry representatives about issues and 
the procurement strategy.; Did the project use the practice?: Army TAC-
SWA: Yes--MHS had a third-party contractor perform market research.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--The Navy and 
a private-sector firm performed market research to, for example, help 
define the market conditions and vendor selection criteria. The Navy 
also held meetings with leading companies that had outsourced IT 
services on an enterprisewide level. Finally, more than 200 companies 
participated in the July 1999 NMCI Industry Day Conference, which 
informed companies about the NMCI vision, requirements, and procurement 
strategy.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA 
staff and a contractor researched preferential providers before 
defining vendor selection criteria. NIMA program officials noted that 
they limited their analysis to such providers because they had 
previously decided on a strategy to directly convert their IT/IS 
activities to a preferential provider.

Practice: Identify and evaluate various sourcing solutions (e.g., 
single-vendor, multivendor, and alliance).; Did the project use the 
practice?: Air Force C4 Services: Yes--As outlined in OMB Circular A-
76, the Air Force evaluated private-sector and internal government 
staff proposals to satisfy the C4 services requirements.; Did the 
project use the practice?: Air Force C4 Services: Yes--A multivendor 
approach has been used in the past, which the Army opted not to 
continue. Instead, NETCOM chose to consolidate its requirements and 
contractor oversight by choosing a single-vendor solution.; Did the 
project use the practice?: Army TAC-SWA: Yes--According to an MHS 
program official, MHS evaluated continuing with the status quo, using 
another federal agency's help-desk services as part of a cross-services 
agreement, and contracting with a commercial firm. According to the MHS 
program office, MHS chose a single-vendor solution to consolidate 
requirements and performance accountability.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--The Navy evaluated single-vendor and 
multivendor approaches. A joint Navy and Marine Corps team determined 
that single-vendor point of contact for accountability and 
responsibility was critical to mission success.; Did the project use 
the practice?: NIMA IT/IS: Yes--NIMA considered the sourcing solutions 
allowed under OMB Circular A-76, including the direct conversion to a 
preferential provider, holding a public/private competition, or 
obtaining an agency cost comparison waiver. NIMA decided that directly 
converting the selected IT functions to a preferential procurement 
source[A] was the lowest risk to its mission and NIMA employees.

Practice: Define a process for selecting vendors to be providers (e.g., 
issuing an RFP and prequalifying vendors).; Did the project use the 
practice?: Air Force C4 Services: Yes--As explained in the invitation 
to bid, the Air Force followed the two-step process outlined in OMB 
Circular A-76 for public/private competitions.; Did the project use the 
practice?: Air Force C4 Services: Yes--The contract was awarded on 
"best value" considerations of technical and management capabilities, 
past performance, and price, as defined in the solicitation.; Did the 
project use the practice?: Army TAC-SWA: Yes--MHS used a selection 
process that included the prequalification of 13 industry leaders in 
both medical and commercial help-desk operations. MHS also issued a 
request for comment and a request for quote, which defined vendor 
evaluation and selection criteria on the basis of technical approach, 
past performance, key personnel qualifications, organizational 
experience, and price.; Did the project use the practice?: MHS/ITO Help 
Desk: Yes--The NMCI RFP used a negotiated commercial items evaluation 
process. The vendor evaluation criteria included technical approach, 
management plan, small business utilization, past performance, and 
price.; Did the project use the practice?: NIMA IT/IS: Limited--NIMA's 
decision to use a phased direct conversion of its IT/IS functions to a 
statutory, preferential procurement vendor limited its vendor selection 
process to the identification of potential Alaska Native Corporation 
companies capable of performing the requirements. NIMA decided to take 
this approach because it believed that it would avoid schedule delays 
and mission risk that it thought would result from taking other 
approaches. In addition, NIMA program officials believed that the 
preferential provider approach would maximize the retention of 
institutional knowledge whether employees stayed at NIMA or 
transitioned to the provider.

Practice: Define vendor selection and evaluation (acceptance) criteria 
at the outset.; Did the project use the practice?: Air Force C4 
Services: Yes--The criteria were included in the invitation to bid.; 
Did the project use the practice?: Air Force C4 Services: Yes--The 
solicitation included vendor evaluation and selection criteria. The 
source selection evaluation plan describes the evaluation process.; Did 
the project use the practice?: Army TAC-SWA: Yes--Vendor evaluation and 
selection were based upon industry-defined help-desk criteria. MHS used 
a multistep process that included an assessment of minimum 
qualifications, evaluation of written technical and cost proposals, and 
oral presentations for qualified vendors.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--The final RFP required bidders to 
have implemented and provided service to at least 100,000 seats, of 
which 20,000 were to be on the same contract. In addition, bidders were 
evaluated on their technical approach, management plan, small business 
utilization, past performance, and price. Finally, vendors in the 
competitive range had to provide demonstrations of technical network 
operations centers to verify their ability to achieve required service 
levels set forth in technical proposals.; Did the project use the 
practice?: NIMA IT/IS: Yes--NIMA evaluated the past performance and 
financial capabilities of Alaska Native Corporation companies. 
According to NIMA, only two companies had demonstrated successful 
performance in service environments similar to NIMA's requirements. 
However, to be able to fully meet NIMA's requirements, these vendors 
formed a joint venture, which was awarded the contract.

Practice: When issuing an RFP, identify services with expected 
performance levels and define client and provider roles and 
responsibilities.; Did the project use the practice?: Air Force C4 
Services: Yes--The services and performance levels were included in the 
original performance work statement included in the invitation to bid.; 
Did the project use the practice?: Air Force C4 Services: Yes--The RFP 
identified the government and contractor roles and responsibilities, 
and system operational and availability requirements.; Did the project 
use the practice?: Army TAC-SWA: Yes--The request for quote identified 
services and expected performance levels and specified a performance-
based, incentivized, shared-risk relationship with the service 
provider. It also defined client and provider roles and 
responsibilities.; Did the project use the practice?: MHS/ITO Help 
Desk: Yes--The NMCI solicitation documents identified the required 
services and expected performance levels. These documents, along with 
the contract, define the client and provider roles and 
responsibilities.; Did the project use the practice?: NIMA IT/IS: Yes-
-Because this was a sole source contract, an RFP was not issued. 
However, according to NIMA and provider officials, they worked jointly 
to refine the requirements and expected performance levels and to 
define client and provider roles and responsibilities in the contract.

Practice: Use third-party assistance with expertise in a variety of 
outsourcing arrangements when selecting provider(s), including 
developing the RFP.; Did the project use the practice?: Air Force C4 
Services: No--This was not done because the Air Force believed that 
adequate in-house expertise was available.; Did the project use the 
practice?: Air Force C4 Services: No--The Army believed that it had 
adequate government experience and expertise covering technical, 
resource management, and contracting areas.; Did the project use the 
practice?: Army TAC-SWA: Yes--A private-sector contractor, the Defense 
Acquisition University, and GovWorks (an Interior organization offering 
procurement services to government agencies) provided assistance in 
this area.; Did the project use the practice?: MHS/ITO Help Desk: Yes-
-A private-sector firm assisted with market research and developing 
vendor pass/fail criteria. Another contractor assisted the Navy with 
developing SLAs and the technical evaluation of vendors.; Did the 
project use the practice?: NIMA IT/IS: Yes--NIMA used a private-sector 
firm and the NIMA Acquisition Center to support the evaluation of the 
vendor proposal.

Practice: Conduct due diligence activities to verify vendor 
capabilities before signing the contract.; Did the project use the 
practice?: Air Force C4 Services: Yes--The Air Force performed a 
technical evaluation of the vendor's capabilities. Also, as part of its 
due diligence activities, the Air Force reviewed the debarred list and 
DOD's Central Contractor Registry.; Did the project use the practice?: 
Air Force C4 Services: Yes--NETCOM evaluated the contractor's financial 
and past performance information before signing the contract.; Did the 
project use the practice?: Army TAC-SWA: Yes--Minimum vendor 
qualifications were established and evaluated early in the process for 
selecting the provider.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--As part of due diligence, the Navy ensured that all 
bidders had relevant experience implementing large seat management 
contracts. In addition, past performance, including reference checks, 
was a source-selection evaluation factor.; Did the project use the 
practice?: NIMA IT/IS: Yes--NIMA researched both parent companies of 
the new joint venture corporation to ensure that they had financial and 
technical backing.

Source: GAO.

[A] Preferential procurement programs are special commercial source 
programs, such as Federal Prison Industries and the workshops 
administered by the Committee for the Purchase from the Blind and Other 
Severely Handicapped under the Javis-Wagner O'Day Act.

[End of table]:

[End of section]

Appendix VI: Projects' Implementation of Phase V: Transition to 
Provider(s):

[See PDF for image]

[End of figure]

In this phase, client organizations transfer responsibility of IT 
functions to one or more providers. A clear definition of 
responsibilities among the various parties and the careful 
consideration of employees' needs matched against the organization's 
needs enable both the client and provider to focus on execution and 
give staff confidence in their future employment. If the contractor is 
assuming responsibility for functions previously performed by federal 
employees, it is especially important that the organization communicate 
a clear transition process. Without such communication, an outsourcing 
project can be negatively affected if misinformation and mistrust 
ensues.

The 11 practices in this phase that we used to evaluate the five 
projects are as follows:

* Communicate a clear transition process to all key players from both 
client and provider organizations.

* Clearly communicate to employees what is going to happen and when it 
is going to happen.

* Establish a client transition team with representatives from across 
the organization to facilitate the transition.

* Place the transition under a single program manager.

* Create client/provider transition teams to address short-term 
transition tasks as required.

* Recognize that it takes time to effect transition and plan 
accordingly.

* Encourage the transition of staff to the provider, where appropriate, 
using bonuses, stock options, and other appropriate methods.

* Develop employee-retention programs and offer bonuses to keep key 
people, where appropriate.

* When consistent with organizational objectives, assist employees who 
do not want to transfer in finding other jobs, either within an 
organization or at another organization.

* Document key information to preserve organizational knowledge in the 
event that one or more providers change.

* Use change management strategies to help client employees deal with 
the transition.

Figure 10 demonstrates that the five projects in our review were 
largely using the practices.

Figure 10: Project Implementation of Phase V: Transition to 
Provider(s):

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 8 provides information on whether and how the projects 
implemented each of the 11 practices in this phase.

Table 8: Summary of Projects' Use of Phase V Practices:

Practice: Communicate a clear transition process to all key players 
from both client and provider organizations.; Did the project use the 
practice?: Air Force C4 Services: No--There were two transition plans 
prepared, one by the government and one by the provider. However, the 
government's plan contained conflicting direction. In addition, 
provider officials said that the two separate transition plans were not 
well coordinated and that the Air Force did not have a strong advocate 
to ensure that the transition process was well planned and executed. 
Both provider and Kirtland Air Force Base officials acknowledged that 
there were problems with the transition process, including incorrect 
information about upgrades that would be made before the transition and 
staff morale problems that hampered knowledge transfer to the 
provider.; Did the project use the practice?: Army TAC-SWA: Yes--The 
provider developed a phase-in plan that addressed personnel issues, the 
integration of added locations, updating plans, and finalizing new 
subcontract/
sponsorship arrangements. This phase-in plan was discussed at a 
postaward conference, attended by representatives from both the client 
and provider.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--The contract delivery order and the provider's transition plan 
laid out a transition process. In addition, a kickoff meeting was held 
between the client and provider to communicate the transition process 
to all key players.; Did the project use the practice?: Navy NMCI: Yes-
-The Navy developed numerous guides and Web sites to communicate with 
the NMCI community about site, technical, and personnel transition 
processes, procedures, and tasks.; Did the project use the practice?: 
NIMA IT/IS: Yes--Several NIMA transformation teams were formed, 
consisting of both client and provider personnel, to develop a single 
transition plan. The plan was developed to formalize communications 
with all key players.

Practice: Clearly communicate to employees what is going to happen and 
when it is going to happen.; Did the project use the practice?: Air 
Force C4 Services: Yes--The Air Force kept its employees informed of 
the outsourcing project through town hall meetings and internal 
briefings. As part of this process, employees were informed about the 
reduction-in-force process and the procedure for those interested in 
being transitioned or reassigned.; Did the project use the practice?: 
Army TAC-SWA: Limited--According to TAC-SWA officials, the Army 
communicated with its employees, but they did not provide documentation 
supporting any communication efforts. However, the provider was 
required to submit a phase-in plan, which included a section on 
communicating with the incumbent contractors' staff.; Did the project 
use the practice?: MHS/ITO Help Desk: Yes--The government task manager 
for the Tri-Service Infrastructure Management Office held regular 
meetings with MHS program officials to discuss transition issues. The 
MHS program office also periodically sent global E-mails to MHS staff 
on transition activities and MHS leadership made on-site visits to San 
Antonio, the location of the help-desk function.; Did the project use 
the practice?: Navy NMCI: Limited--The Navy has used its normal chain 
of command to communicate transition information, but found that the 
implementation was uneven. As a result, some staff did not know current 
information about how NMCI would affect them until the provider was 
ready to contact them regarding their possible transition to the 
contractor. However, according to the NMCI Director's office, this 
problem was somewhat mitigated by the provider's Web site that provides 
transition information to all NMCI customers/users.; Did the project 
use the practice?: NIMA IT/IS: Yes--With help from a contractor, NIMA 
developed a communications plan. The plan included town hall meetings, 
global 
E-mail messages, and a Web site posting that provided employees with 
access to transition information.

Practice: Establish client transition team with representatives from 
across the organization to facilitate the transition.; Did the project 
use the practice?: Air Force C4 Services: Yes--The government's 
transition team included personnel from the functional areas being 
outsourced (i.e., communications and IT services) as well as personnel 
from other offices.; Did the project use the practice?: Army TAC-SWA: 
Yes--According to TAC-SWA officials, the contracting officer's 
representative assisted the provider with the transition. The officials 
also stated that Army technical points of contact at each site also 
helped facilitate the transition to the new contract.; Did the project 
use the practice?: MHS/ITO Help Desk: Yes--MHS program officials for 
the Tri-Service Infrastructure Management Office established 
transition teams with representatives from the former Tri-Service 
Medical System Support Center contract.; Did the project use the 
practice?: Navy NMCI: Yes--The headquarters-level transition team 
consists of representatives from the NMCI Director's Office and the 
program management office. Also, every site has a transition team made 
up of customer and provider personnel.; Did the project use the 
practice?: NIMA IT/IS: Yes--NIMA's transition team included 
representatives from various offices within NIMA to help facilitate the 
transition.

Practice: Place transition under single program manager.; Did the 
project use the practice?: Air Force C4 Services: Yes--The Air Force 
established a single program manager, who is referred to as the 
functional area chief.; Did the project use the practice?: Army TAC-
SWA: Yes--The Army designated the contracting officer's representative 
to oversee the transition.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--The transition was assigned to the government task 
manager in the Tri-Service Infrastructure Management Office.; Did the 
project use the practice?: Navy NMCI: Yes--NMCI's Director has overall 
responsibility for the transition.; Did the project use the practice?: 
NIMA IT/IS: Yes--The transition process is under a single program 
manager.

Practice: Create client/provider transition teams to address short-term 
transition tasks as required.; Did the project use the practice?: Air 
Force C4 Services: Yes--During the transition phase, management from 
both the government and the provider met weekly to track the progress 
of the transition.; Did the project use the practice?: Army TAC-SWA: 
Yes--Personnel from both the Army and provider formed three transition 
teams, one for each site location. These teams worked with the 
contracting officer's representative and the provider's project manager 
to deal with transition issues.; Did the project use the practice?: 
MHS/ITO Help Desk: Yes--Client/provider transition teams were 
established to address short-term transition tasks. The teams met 
weekly to address any transition issues.; Did the project use the 
practice?: Navy NMCI: Yes--NMCI and the provider designated staff to 
work together on short-term transition issues.; Did the project use the 
practice?: NIMA IT/IS: Yes--An integrated NIMA transformation team 
comprising NIMA and provider staff, was responsible for addressing 
short-term tasks, including (1) transition planning, (2) resources and 
recruitment, (3) program budget, and (4) contract development and 
costing.

Practice: Recognize that it takes time to effect transition and plan 
accordingly.; Did the project use the practice?: Air Force C4 Services: 
Yes--The Air Force allowed 60 days for the transition.; Did the project 
use the practice?: Army TAC-SWA: Yes--The contract allowed a 60-day 
phase-in period.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--The contract provided for a transition period of 30 to 90 days. 
According to MHS program officials, the transition period lasted 90 
days.; Did the project use the practice?: Navy NMCI: Limited--The Navy 
said that it had initially underestimated the scope of the project and 
the magnitude of the problems brought on by legacy applications and 
associated information assurance and cyber-security issues. According 
to NMCI program officials, addressing these problems and completing the 
operational testing that was mandated by the Congress subsequent to 
contract award led to the transition period being lengthened from 2-1/
2 to 
3-1/2 years. NMCI officials also said the extension will allow the 
Department of the Navy to have time to operate NMCI as a fully 
transitioned enterprise before having to decide whether to exercise the 
contract option.; Did the project use the practice?: NIMA IT/IS: Yes--
According to NIMA, it has scheduled its seven functional areas to be 
transitioned to the provider over a 3-to 5-year period. The provider 
will not assume responsibility for a given functional area until the 
NIMA transition review board has given its approval to the provider's 
turnover plan.

Practice: Encourage transition of staff to provider, where appropriate, 
using bonuses, stock options, and other appropriate methods.; Did the 
project use the practice?: Air Force C4 Services: Yes--The Air Force 
staff were given the right to transfer to the contractor. It was left 
up to the contractor to decide whether to offer incentives or not, 
which it chose not to do.; Did the project use the practice?: Army TAC-
SWA: Yes--The Army had no personnel expected to transition to the 
provider since the activity being outsourced had previously been 
contracted out. However, the provider was interested in retaining 
existing contractor staff and had a program to recruit them.; Did the 
project use the practice?: MHS/ITO Help Desk: Yes--According to 
provider officials, they extended offers to employees from the prior 
contractor and some MHS staff.; Did the project use the practice?: Navy 
NMCI: Yes--The NMCI contract provides that displaced federal employees 
that transition to the provider under employment openings as a result 
of NMCI will be given guaranteed 3 years of employment with the 
provider, a 15 percent salary increase, and a sign-on bonus.; Did the 
project use the practice?: NIMA IT/IS: Yes--The contract requires the 
provider to actively recruit, hire, and make reasonable efforts to 
retain NIMA staff. If the provider meets certain target thresholds for 
recruiting NIMA staff, it receives a monetary incentive. According to 
the provider's general manager, the company offered signing bonuses to 
NIMA employees that wanted to transition.

Practice: Develop employee-retention programs and offer bonuses to keep 
key people, where appropriate.; Did the project use the practice?: Air 
Force C4 Services: No--According to Kirtland Air Force Base officials, 
they did not develop an employee-retention program because they did not 
believe that they could offer incentives.; Did the project use the 
practice?: Army TAC-SWA: Not applicable--Since this activity was 
previously contracted out, employee retention of Army civilian 
personnel was not relevant.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--The government task manager persuaded key government 
employees from the Tri-Service Medical System Support Center to 
temporarily work at the Tri-Service Infrastructure Management Office 
until positions became available with a contractor that provides 
support to this office. In addition, according to a program official, 
to keep expertise in particular applications, key employees were 
transferred to the MHS program executive office and Tri-Service 
Infrastructure Management Office.; Did the project use the practice?: 
Navy NMCI: Yes--NMCI program officials noted that the Navy has human 
resources programs that would allow its organizations a range of 
options, including bonuses, to help keep key staff. Additionally, the 
Navy directed all Navy and Marine Corps commands affected by NMCI to 
develop civilian personnel transition plans to mitigate the impact of 
NMCI implementation on employees.; Did the project use the practice?: 
NIMA IT/IS: Yes--According to NIMA program officials, NIMA categorized 
affected employees into three tiers, on the basis of whether they would 
be allowed to transition to the provider or whether there would be 
restrictions on their activities if they did transition. This 
distinction was made to comply with government conflict of interest 
rules. According to the program manager, NIMA also sought personnel who 
wanted to remain at the agency to help monitor the provider's 
performance and considered the qualifications of those who wished to 
serve in this role prior to deciding who to retain.

Practice: When consistent with organizational objectives, assist 
employees who do not want to transfer in finding other jobs, either 
within an organization or at another organization.; Did the project use 
the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base 
officials stated that they have reduction-in-force procedures to help 
place civilian employees in other Air Force jobs. Employees also 
receive priority status for other DOD job openings. Further, the base 
education office provided assistance with outplacement; rsum writing; 
and, for those nearing retirement, planning advice.; Did the project 
use the practice?: Army TAC-SWA: Not applicable--According to the TAC-
SWA contracting officer, no Army personnel were displaced by the 
contract.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
According to MHS program officials, the government task manager 
assisted government employees with the Tri-Service Medical System 
Support Center staff in finding new positions with other MHS 
organizations. For example, he located open positions and contacted the 
organizations.; Did the project use the practice?: Navy NMCI: Yes--NMCI 
program officials said that existing Navy civilian personnel programs 
are in place to assist employees. They noted that two Navy 
organizations found other jobs within their command for their small 
number of employees affected by NMCI. Other Navy organizations are also 
reassigning affected personnel to other government jobs, where 
applicable.; Did the project use the practice?: NIMA IT/IS: Yes--For 
staff that did not want to transition, NIMA provided various types of 
assistance, such as retirement planning, rsum writing, and 
interviewing skills. In addition, NIMA reported that if an employee did 
not want to transition to the provider, that it would consider re-
adjusting the employee's work assignment or provide training to support 
the individual's placement within other areas.

Practice: Document key information to preserve organizational knowledge 
in the event that one or more providers change.; Did the project use 
the practice?: Air Force C4 Services: Yes--Kirtland Air Force Base's 
work functions and workload size are documented in the contract.; Did 
the project use the practice?: Army TAC-SWA: Yes--Documentation 
produced by the provider, including maintenance logs, become the 
property of the government.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--The Tri-Service Medical System Support Center processes 
and procedures were captured during transition. The provider's 
processes and procedures are currently being captured. The contract 
also includes requirements on documenting key information.; Did the 
project use the practice?: Navy NMCI: Yes--NMCI maintains a Web portal 
with organizational knowledge information. In addition, a new Intranet 
site is being developed with up-to-date interactive organizational 
knowledge relevant to the many varied NMCI communities.; Did the 
project use the practice?: NIMA IT/IS: Yes--The executing NIMA 
transformation team was charged with documenting the activities for 
each functional area that is being outsourced. This includes developing 
and capturing operational processes and procedures for each of the 
functional areas.

Practice: Use change management strategies to help client employees 
deal with the transition.; Did the project use the practice?: Air Force 
C4 Services: Yes--Kirtland Air Force Base held town hall meetings and 
internal briefings to communicate the objectives of the outsourcing 
project and the changes that would take place internally. It also had 
one-on-one meetings with each employee that would be displaced.; Did 
the project use the practice?: Army TAC-SWA: Not applicable--The 
activity had been previously contracted out.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--According to MHS program officials, 
the government task manager for the Tri-Service Infrastructure 
Management Office assisted government employees with finding new 
positions and kept them informed of transition issues. Weekly 
E-mails were also sent out to all affected employees.; Did the project 
use the practice?: Navy NMCI: Yes--The Navy has provided employees 
affected by NMCI with a range of employment options and incentives.; 
Did the project use the practice?: NIMA IT/IS: Yes--NIMA's change 
management strategies included training seminars offered by its human 
resources office on rsum writing, interviewing skills, career 
transition workshops, and employment trends.

Source: GAO.

[End of table]:

[End of section]

Appendix VII: Projects' Implementation of Phase VI: Manage Provider(s) 
Performance:

[See PDF for image]

[End of figure]

The effectiveness with which the performance of the provider(s) is 
managed--the focus of this phase--is critical to the successful 
implementation of an outsourcing project. Indeed, according to Gartner, 
Inc., an outsourcing project can be thwarted by poorly designed, 
funded, and delivered processes for managing the delivery of 
services.[Footnote 48] This firm also points out that an enterprise 
needs to retain the resources to oversee the planning and 
implementation of the IT services being delivered by the provider to 
ensure that the contractor meets the client's business needs throughout 
the life of the agreement.[Footnote 49] Moreover, frequent and clear 
communication between the client and provider ensures that potential 
problems are resolved before they cause disruptions. In addition, 
performance reviews should take place regularly to keep the project on 
course, measure performance levels, and make adjustments as 
necessary.[Footnote 50]

The 11 practices in this phase that we used to evaluate the five 
projects are as follows:

* Consider incentives to motivate provider(s) to exceed performance 
requirements.

* Use penalties to motivate provider(s) to meet performance 
requirements.

* Periodically undertake studies to assess (1) how the provider's 
performance compares with the value being delivered to similar clients 
and (2) the extent to which the provider's performance is improving 
over time.

* Schedule periodic working-level meetings with both the end-user 
groups and the provider to review the provider's performance.

* Conduct executive-level oversight meetings with the provider's senior 
management to review provider's performance.

* Distribute performance data to stakeholders.

* Reserve audit rights on performance data supplied by the provider.

* Ensure that the provider measures and reports on performance.

* Work with the provider to redefine service levels, as appropriate.

* Sample performance data frequently enough to perform trend analysis 
and to permit extrapolation based on historical data.

* Allow employees and possibly stakeholders to rate the provider on a 
regular basis.

As shown by figure 11, the five projects in our review generally 
implemented the practices.

Figure 11: Project Implementation of Phase VI: Managing Provider 
Performance:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 9 depicts whether and how the five projects in our review 
implemented each of the 11 practices in this phase.

Table 9: Summary of Projects' Use of Phase VI Practices:

Practice: Consider incentives to motivate provider(s) to exceed 
performance requirements.; Did the project use the practice?: Air Force 
C4 Services: No--The Air Force did not include incentives in the 
contract because the agency expects the contractor to meet the 
contract's requirements without incentives. Specifically, according to 
the former functional area chief, because the contract was issued under 
the rules established by OMB Circular A-76, the focus of the project 
was on achieving cost savings, and incentives were not included in the 
contract. As a result, he noted that the provider did not have any 
incentive to be innovative or creative.; Did the project use the 
practice?: Army TAC-SWA: No--There are no monetary incentives, but, 
according to TAC-SWA officials, the fact that the agency will prepare a 
performance evaluation report can help motivate a provider to meet 
requirements. However, the contracting officer said that incentives 
might have been useful to motivate the provider to exceed performance 
requirements.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--The contract includes incentives linked to each performance 
requirement. Each requirement has a positive, negative, and acceptable 
performance range that provides a basis for monetary incentives (as 
well as penalties).; Did the project use the practice?: Navy NMCI: Yes-
-The NMCI contract includes a one-time payment of $10 million if the 
provider successfully completes full operational capability. The 
contract also has incentives for customer satisfaction, information 
assurance, and small business and small disadvantaged business 
participation.; Did the project use the practice?: NIMA IT/IS: Yes--The 
contract includes monetary incentives in accordance with an award fee 
plan. It also includes share-in-savings provisions to encourage process 
improvements.

Practice: Use penalties to motivate provider(s) to meet performance 
requirements, such as; ; * assess penalties for failure to perform at 
required individual service as well as aggregate service levels;; * 
apply penalties in the form of credit to the client;; * increase 
penalty for recurring deficient performance;; * hold back a percentage 
of provider's pay for a particular service until performance 
requirements are met;; * refund a penalty if the provider returns to 
agreed-upon performance levels within a designated period of time; and; 
* ensure that the provider will cover costs, but not profit, when a 
particular performance requirement is not met.; Did the project use the 
practice?: Air Force C4 Services: Yes--The contract contains monetary 
penalties that are linked to each of the performance requirements. If 
imposed, they would reduce the amount of the payment owed to the 
provider.; Did the project use the practice?: Army TAC-SWA: Limited--
The contract does not have specific monetary penalties linked to 
performance requirements. However, the contract states that 
unacceptable work must be redone at the provider's expense, and, if the 
defects and services cannot be corrected, the government may reduce the 
contract's price to reflect the reduced value of the services 
performed. TAC-SWA project officials stated that monetary penalties 
were not included in the contract because the Army was concerned that 
contractors might not bid on the contract, and administrating this type 
of contract is more difficult.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--The contract includes monetary penalties linked to 
each performance requirement. Each requirement has positive, negative, 
and acceptable performance ranges that provide a basis for monetary 
penalties as well as incentives.; Did the project use the practice?: 
Navy NMCI: Yes--The contract includes monetary penalties in the form of 
credits to the agency if the provider fails to perform to the levels 
specified in the SLAs.; Did the project use the practice?: NIMA IT/IS: 
Limited--NIMA did not include monetary penalties in its contract. 
However, if the provider does not meet certain minimum performance 
standards, only its costs would be covered.

Practice: Periodically undertake studies to assess: (1) how the 
provider's performance compares with the value being delivered to 
similar clients and (2) the extent to which the provider's performance 
is improving over time.; Did the project use the practice?: Air Force 
C4 Services: Limited--(1) This has not been done because, according to 
a project official, the Air Force was not aware of similar clients.; 
(2) This is done through periodic monitoring of performance by quality 
assurance evaluators and through monthly review meetings held by the 
functional area chief to identify and address any problems that are 
starting to occur.; Did the project use the practice?: Army TAC-SWA: 
No--TAC-SWA officials stated that they address performance issues at 
the time that they occur and that they have not performed any studies.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--The 
provider's performance is reviewed monthly by the government task 
manager and in quarterly management reviews of the provider's 
performance. According to MHS program officials, the results are 
compared with peers and reviewed for how the provider has improved over 
time.; Did the project use the practice?: Navy NMCI: Limited--(1) NMCI 
only has performance data since October 2002. The Navy intends to do 
such a study but has not established a schedule for it.; (2) The 
provider's improvements over time are being monitored monthly.; Did the 
project use the practice?: NIMA IT/IS: Limited--(1) At this time, NIMA 
has not undertaken such studies because it only recently transitioned 
functions to the provider. However, NIMA program officials stated that 
they intend to use a private-sector firm to periodically compare the 
provider's performance with those of similar organizations.; (2) The 
provider's improvements over time are being monitored monthly.

Practice: Schedule periodic working-level meetings with both the end-
user groups and the provider to review the provider's performance.; Did 
the project use the practice?: Air Force C4 Services: Yes--Monthly 
meetings are held at which the contractor briefs the functional area 
chief and other Kirtland Air Force Base representatives.; Did the 
project use the practice?: Army TAC-SWA: Yes--According to the provider 
and TAC-SWA officials, they hold daily meetings to discuss any issues.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--Periodic 
working-level meetings are held with the government task manager and 
the program management office representatives to obtain help-desk 
feedback.; Did the project use the practice?: Navy NMCI: Yes--At each 
implemented site, the provider and government managers meet on a 
frequent, as-needed basis to review performance and resolve any 
issues.; Did the project use the practice?: NIMA IT/IS: Yes--Quarterly 
performance management review meetings are held and are attended by 
NIMA and the provider. In addition, the provider hired an ombudsman to 
obtain anonymous comments from NIMA staff and management on its 
performance.

Practice: Conduct executive-level oversight meetings with the 
provider's senior management to review provider's performance.; Did the 
project use the practice?: Air Force C4 Services: Yes--Air Force 
schedules executive-level oversight meetings whenever they are 
necessary. A recent meeting included the president of the provider.; 
Did the project use the practice?: Army TAC-SWA: Yes--According to 
provider and TAC-SWA officials, quarterly in-progress reviews are held 
with the provider's corporate management and the battalion commander.; 
Did the project use the practice?: MHS/ITO Help Desk: Yes--Executive 
oversight meetings are held quarterly with the provider's senior 
management to review performance.; Did the project use the practice?: 
Navy NMCI: Yes--The NMCI Director and the provider executive meet 
weekly to review performance and discuss other NMCI implementation 
issues. In addition, the Department of the Navy recently established an 
operations advisory board consisting of Department of the Navy 
leadership and the provider. The goal of this board is to focus senior 
leadership on issues affecting NMCI in order to establish priorities 
and make decisions.; Did the project use the practice?: NIMA IT/IS: 
Yes--This is done through the quarterly review meetings.

Practice: Distribute performance data to stakeholders.; Did the project 
use the practice?: Air Force C4 Services: Yes--Performance data were 
not distributed to stakeholders at the beginning of the project, but 
began after complaints from stakeholders about the need for such 
information. Metrics, such as system reliability or "uptime," are now 
provided to stakeholders on line.; Did the project use the practice?: 
Army TAC-SWA: Yes--The contracting officer's representative 
distributes monthly performance reports to Army stakeholders for 
review. Performance issues or problems identified are discussed with 
the contracting officer's representative and the contracting officer, 
and corrective actions are planned to prevent reoccurrence.; Did the 
project use the practice?: MHS/ITO Help Desk: Yes--The help-desk 
monthly reports, including results against performance metrics, are e-
mailed to stakeholders monthly.; Did the project use the practice?: 
Navy NMCI: Yes--The NMCI Navy and Marine Corps program managers are 
responsible for providing SLA performance data to their commands.; Did 
the project use the practice?: NIMA IT/IS: Yes--Performance data are 
distributed at quarterly review meetings, which according to the 
contracting officer, are attended by stakeholders.

Practice: Reserve audit rights on performance data supplied by the 
provider.; Did the project use the practice?: Air Force C4 Services: 
Yes--Addressed in the contract.; Did the project use the practice?: 
Army TAC-SWA: Yes--Addressed in the contract.; Did the project use the 
practice?: MHS/ITO Help Desk: Yes--Addressed in the contract.; Did the 
project use the practice?: Navy NMCI: Yes--Addressed in the contract.; 
Did the project use the practice?: NIMA IT/IS: Yes--Addressed in the 
contract.

Practice: Ensure that provider measures and reports on performance.; 
Did the project use the practice?: Air Force C4 Services: Yes--As 
required by the quality control plan in the contract, the provider 
makes information available on product or service quality and any 
actions needed to correct decreases in quality. In addition, according 
to the former functional area chief, the provider provides performance 
information during monthly meetings.; Did the project use the 
practice?: Army TAC-SWA: Yes--The contracting officer's representative 
receives performance information in daily meetings with the provider's 
project manager.; Did the project use the practice?: MHS/ITO Help Desk: 
Yes--The contract requires the provider to submit monthly and quarterly 
performance reports that are then reviewed and verified.; Did the 
project use the practice?: Navy NMCI: Yes--The contract specifies that 
the provider must measure and report on whether its SLA performance 
goals are being met.; Did the project use the practice?: NIMA IT/IS: 
Yes--The contract requires the provider to submit data, including 
measurements of service, quarterly.

Practice: Work with provider to redefine service levels, as 
appropriate.; Did the project use the practice?: Air Force C4 Services: 
Yes--This is done on an as-needed basis. For example, the service 
levels were redefined in April 2002.; Did the project use the 
practice?: Army TAC-SWA: Yes--According to TAC-SWA and provider 
officials, they have worked together to redefine performance 
requirements in the contract.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--MHS and the provider have worked together to 
redefine SLAs. For example, the first call closure performance metric 
was redefined to be more realistic.; Did the project use the practice?: 
Navy NMCI: Yes--The Navy and service provider have and continue to 
refine NMCI SLAs to ensure more precise performance measures and to 
more accurately capture user satisfaction with the system. For example, 
a contract modification standardized many of the performance categories 
that dealt with service availability.; Did the project use the 
practice?: NIMA IT/IS: Yes--The redefinition of service levels is 
provided for in the contract. According to NIMA, the redefinition of 
service levels is expected to occur in the future, generally at the 
time that the annual contract option is exercised.

Practice: Sample performance data frequently enough to perform trend 
analysis and to permit extrapolation based on historical data.; Did the 
project use the practice?: Air Force C4 Services: Yes--Air Force 
quality assurance evaluators monitor performance in accordance with the 
quality assurance surveillance plan and the performance requirements 
summary. Any trends identified are addressed in monthly status 
meetings.; Did the project use the practice?: Army TAC-SWA: Yes--The 
provider's proposal and quality control plan state that it will provide 
various trend analyses to the government. According to TAC-SWA project 
officials, these data are submitted to the contracting officer's 
representative for analysis.; Did the project use the practice?: MHS/
ITO Help Desk: Yes--Performance data are sampled and reported monthly 
and quarterly. They are analyzed and verified, including any supporting 
data. In addition, the provider's knowledge management system provides 
analysis and trend data to MHS.; Did the project use the practice?: 
Navy NMCI: Yes--The NMCI Director monitors provider performance to 
identify trends by assessing provider-supplied information as well as 
information from Navy independent verification and validation testing 
and customer satisfaction survey results. In addition, NMCI program 
officials said that they have requested funding to develop a 
performance measurement data repository to support trend analyses.; Did 
the project use the practice?: NIMA IT/IS: Yes--The contract requires 
data to be provided periodically for NIMA to use for trend and other 
types of analyses.

Practice: Allow employees and possibly stakeholders to rate the 
provider on a regular basis (e.g., scorecards and quarterly report 
cards).; Did the project use the practice?: Air Force C4 Services: 
Limited--The provider conducts annual customer satisfaction surveys, 
but it is not required to submit the results to the Air Force. Kirtland 
Air Force Base also obtains ad hoc feedback from employees who, after 
they report a problem, are asked to provide information about how well 
the problem was addressed and their overall satisfaction level.; Did 
the project use the practice?: Army TAC-SWA: Yes--The contracting 
officer's representative and on-site points of contact obtain feedback 
from employees on a regular basis.; Did the project use the practice?: 
MHS/ITO Help Desk: Yes--The customer satisfaction survey is one method 
used to rate the provider. In addition, according to MHS program 
officials, stakeholder input is also obtained from program management 
reviews and meetings held by the government task manager.; Did the 
project use the practice?: Navy NMCI: Yes--User surveys are used to 
measure satisfaction with specific services such as E-mail access, help 
desk, etc., and overall user satisfaction with the service provider's 
performance.; Did the project use the practice?: NIMA IT/IS: Yes--The 
provider surveys NIMA employees on their satisfaction with the help-
desk function. The results of these surveys are reviewed by NIMA during 
quarterly meetings. In addition, stakeholders are also responsible for 
providing monthly assessments of contractor performance. Moreover, 
customer satisfaction is a factor in determining the amount of the 
semiannual award fee earned by the provider.

Source: GAO.

[End of table]:

[End of section]

Appendix VIII: Projects' Implementation of Phase VII: Ensure Services 
are Provided:

[See PDF for image]

[End of figure]

Although outsourcing focuses on the provider's ability to perform, the 
client organization is ultimately responsible for ensuring that 
services are provided and that end-user needs are met. The previous 
phases addressed the extensive preparation that must precede the 
provider's assuming responsibility for the client organization's 
services. This phase encompasses practices designed to ensure that an 
effective oversight approach is established. In addition, over the 
course of the outsourcing project, the client learns more about the 
capabilities of the provider, and market conditions may change. As a 
result, it is important to monitor service levels internally as well as 
maintain an external view of the performance of other providers in 
order to identify opportunities to improve and ensure that the 
outsourcing arrangement maintains its value to the client.

The six practices in this phase that we used to evaluate the five 
projects are as follows:

* Monitor the provider's work to anticipate issues for resolution.

* Make sure that the provider uses the standard tools and processes 
defined as part of the operational model.

* Use provider performance data to continuously improve processes.

* Pursue improvement based on customer satisfaction surveys.

* Ensure that an appropriately empowered individual from the client 
organization oversees the work.

* Set realistic time frames that are agreed to by the provider.

As illustrated by figure 12, the five projects in our review largely 
implemented the practices.

Figure 12: Project Implementation of Phase VII: Ensure Services Are 
Provided:

[See PDF for image]

Note: Not applicable--The practice was not relevant to the project's 
particular circumstances. 
No--The agency did not implement the practice. Limited--The agency 
fully implemented some but not all aspects of the practice and did not 
take alternative actions that fully satisfied the practice. Yes--The 
agency fully implemented the practice or took an alternative action 
that fully satisfied its intent.

[End of figure]

Table 10 provides details on whether and how each project implemented 
each of the six practices associated with this phase.

Table 10: Summary of Projects' Use of Phase VII Practices:

Practice: Monitor the provider's work to anticipate issues for 
resolution.; Did the project use the practice?: Air Force C4 Services: 
Yes--Quality assurance evaluators monitor the quality of the provider's 
work and identify problems or trends. The results and any problems are 
reported to the contracting officer and the functional area chief.; Did 
the project use the practice?: Army TAC-SWA: Yes--The contracting 
officer's representative reports monthly on contractor performance. In 
addition, he meets daily with provider officials to discuss performance 
results.; Did the project use the practice?: MHS/ITO Help Desk: Yes--
The provider's work is monitored through monthly reports and meetings 
as well as through on-site meetings and readiness assessments.; Did the 
project use the practice?: Navy NMCI: Yes--Navy representatives at each 
implemented site (including headquarters) monitor the provider's work 
and identify issues. According to NMCI program officials, the NMCI 
Enterprise Management System enables the government to monitor the 
performance of the network and associated service delivery.; Did the 
project use the practice?: NIMA IT/IS: Yes--NIMA reviews the provider's 
work in quarterly program management reviews and as needed in response 
to issues raised by NIMA's performance monitoring officials.

Practice: Make sure the provider uses the standard tools and processes 
defined as part of the operational model.; Did the project use the 
practice?: Air Force C4 Services: Yes--The provider uses software tools 
to help monitor system performance against performance standards to 
monitor and manage the help-desk function.; Did the project use the 
practice?: Army TAC-SWA: Yes--According to TAC-SWA project officials, 
the provider uses various standard tools outlined in its proposal.; Did 
the project use the practice?: MHS/ITO Help Desk: Yes--According to MHS 
program officials, the provider uses the standard tools outlined in the 
contract.; Did the project use the practice?: Navy NMCI: Yes--The Navy 
has monitored the service provider's use of standard tools and 
processes. Tools are being used to support legacy migration, client 
installation, and help-desk procedures.; Did the project use the 
practice?: NIMA IT/IS: Yes--The tools are documented in the quality 
management plan. For example, the provider is using a specific quality 
and process improvement methodology.

Practice: Use provider performance data to continuously improve 
processes.; Did the project use the practice?: Air Force C4 Services: 
Yes--Although Kirtland Air Force Base does not rely on data provided by 
its provider, its quality assurance evaluators continually monitor 
provider performance.; Did the project use the practice?: Army TAC-SWA: 
Yes--The contracting officer's representative monitors provider 
performance data, and any potential areas for improvement are 
incorporated in the monthly report.; Did the project use the practice?: 
MHS/ITO Help Desk: Yes--For example, MHS used provider performance data 
to improve the customer satisfaction survey process.; Did the project 
use the practice?: Navy NMCI: Yes--The Navy uses the provider's data to 
help determine ways to improve processes, such as to improve NMCI 
implementation procedures and the timeliness of help-desk problem 
resolution.; Did the project use the practice?: NIMA IT/IS: Yes--NIMA's 
performance monitors use provider data to assess progress and ways to 
improve performance.

Practice: Pursue improvement based on customer satisfaction surveys.; 
Did the project use the practice?: Air Force C4 Services: Limited--The 
Air Force does not perform, or require the provider to perform, 
customer surveys. However, the provider has initiated such surveys but 
is not required to distribute the results to the Air Force.; Did the 
project use the practice?: Army TAC-SWA: Yes--Surveys are received by 
the contracting officer's technical representatives and provided as 
input for the contracting officer's representative report.; Did the 
project use the practice?: MHS/ITO Help Desk: Yes--Negative feedback 
that is received based on the results of the customer satisfaction 
surveys is reviewed by the MHS help-desk manager.; Did the project use 
the practice?: Navy NMCI: Yes--Surveys of user satisfaction with 
specific services and overall satisfaction with provider performance 
are used to identify areas needing improvement.; Did the project use 
the practice?: NIMA IT/IS: Yes--To pursue improvement, NIMA uses both 
customer satisfaction surveys and interviews with senior-level 
customers.

Practice: Ensure that an appropriately empowered individual from the 
client organization oversees the work.; Did the project use the 
practice?: Air Force C4 Services: Yes--The functional area chief 
oversees the work, and the quality assurance evaluators support this 
oversight effort.; Did the project use the practice?: Army TAC-SWA: 
Yes--The contracting officer's representative is the empowered 
individual from NETCOM.; Did the project use the practice?: MHS/ITO 
Help Desk: Yes--The government task manager is the individual empowered 
to oversee the work of the provider.; Did the project use the 
practice?: Navy NMCI: Yes--The NMCI Director oversees the work and 
discusses performance with the NMCI provider executive during weekly 
meetings.; Did the project use the practice?: NIMA IT/IS: Yes--The 
contracting officer's representative and a staff of technical monitors 
oversee the work.

Practice: Set realistic time frames that are agreed to by the 
provider.; Did the project use the practice?: Air Force C4 Services: 
Yes--Kirtland Air Force Base and the provider agree that reasonable 
time frames for performance have now been set. Some revisions were made 
to them in April 2002.; Did the project use the practice?: Army TAC-
SWA: Yes--Time frames for new tasks are established by contract 
modifications that are agreed to by the provider.; Did the project use 
the practice?: MHS/ITO Help Desk: Yes--The monthly performance reports 
and quarterly program reviews set time frames that have been agreed to 
by the provider.; Did the project use the practice?: Navy NMCI: Yes--
According to NMCI program officials, Navy's practice is to work with 
the provider in establishing schedules.; Did the project use the 
practice?: NIMA IT/IS: Yes--NIMA and the provider have agreed on dates 
associated with specific actions.

Source: GAO:

[End of table]

[End of section]

Appendix IX: Comments from the Department of Defense:

DEPARTMENT OF DEFENSE:

6000 DEFENSE PENTAGON WASHINGTON, DC 20301-6000:

April 8, 2003:

CHIEF INFORMATION OFFICER:

Mr. Randolph C. Hite Director:

Information Technology Architecture and Systems Issues United States 
General Accounting Office:

Washington, DC 20548:

Dear Mr. Hite:

This is the Department of Defense (DoD) response to GAO draft Report 
No. GAO-03-371, "INFORMATION TECHNOLOGY: DoD Needs to Leverage Lessons 
Learned from its Outsourcing Projects," dated March 11, 2002 (GAO Code 
310239).

The Department partially concurs with the report. The DoD agrees that 
capturing lessons learned in the development and implementation of its 
IT outsourcing initiatives is important to continually improving the 
methods used and the results achieved. However, specifying a particular 
method of accession is premature at this time. The DoD currently has 
several processes and communities of interest that collect and 
disseminate lessons learned in other areas; one noteworthy example is 
"Share A-76." The processes used by these groups would be a logical 
starting point to determine the best path forward. Before the 
Department commits to a specific means of provision, we intend to 
explore a variety of mechanisms by which we can exploit lessons learned 
in IT outsourcing initiatives.

One minor correction to the matrix on page 57 is requested. The 
National Imagery and Mapping Agency (NIMA) has authorized its 
contractor to procure supplies, but not products or services under the 
Federal Supply Schedule. Therefore, under the NIMA IT/IS column, in the 
third box (at the bottom), please change the sentence from:

"Yes-NIMA has authorized the provider to use government sources, such 
as the Federal Supply Schedule, in procuring products, services, and 
supplies related to this contract.":

to:

"Yes-NIMA has authorized the provider to use government sources, such 
as the Federal Supply Schedule, in procuring supplies related to this 
contract.":

If you require further information, please contact the action officer, 
Leo Milanowski, at (703) 602-2720 x142 or leo.milanowskiAosd.mil.

Sincerely,

Margaret E. Myers,
Principal Director
Deputy Assistant Secretary of Defense (Deputy CIO):

Signed by Margaret E. Myers:

Attachment:

Response to Recommendations:

GAO DRAFT REPORT DATED MARCH 11, 2003 GAO-03-371 (GAO CODE 310239):

"INFORMATION TECHNOLOGY: DoD Needs to Leverage Lessons Learned from Its 
Outsourcing Projects:

DEPARTMENT OF DEFENSE COMMENTS TO THE GAO RECOMMENDATIONS:

RECOMMENDATION 1: The GAO recommended that the Undersecretary of 
Defense for Acquisition, Technology, and Logistics, working in 
conjunction with the Assistant Secretary of Defense for Command, 
Control, Communications, and Intelligence, provide senior management 
support and adequate resources to develop and implement an electronic 
tool to capture and disseminate examples and lessons learned from 
actual IT outsourcing projects.

DOD RESPONSE:

Partially concur; DoD agrees that capturing lessons learned in the 
development and implementation of its IT outsourcing initiatives is 
important to continually improving the methods used and the results 
achieved. However, specifying a particular method of accession is 
premature at this time. DoD currently has several processes and 
communities of interest that collect and disseminate lessons learned. 
Before the Department commits to a specific means of provision, we 
intend to explore a variety of mechanisms by which we can exploit 
lessons learned in IT outsourcing initiatives.

RECOMMENDATION 2: The GAO recommended that the DoD ensure that the 
method used to gather information for this electronic tool incorporate 
the main element of a lessons learned process, namely, collection, 
verification, storage, and dissemination.

DOD RESPONSE:

Partially concur; see above:

The following are GAO's comments on the Department of Defense's letter 
dated April 8, 2003.

GAO Comments:

1. Addressed in the Agency Comments and Our Evaluation section of this 
report.

2. On January 29, 2003, NIMA granted the provider authorization to use 
government sources in performing the IT/IS contract that was limited to 
"products, services, and supplies that support the performance of the 
specific functional areas and miscellaneous items required under this 
contract." Accordingly, we did not modify this report.

3. Addressed in the Agency Comments and Our Evaluation section of this 
report.

[End of section]

Appendix X: GAO Contact and Staff Acknowledgments:

GAO Contact:

Linda J. Lambert, (202) 512-9556:

:

Staff Acknowledgments:

Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and 
Thomas Wright made key contributions to this report.

(310239):

FOOTNOTES

[1] This figure is from the Federal Procurement Data System, which 
contains detailed information on contract actions over $25,000. 

[2] IT services outsourcing is a type of acquisition in which a client 
organization transfers responsibility for performance of one or more IT 
services to one or more external providers.

[3] U.S. General Accounting Office, Information Technology: Leading 
Commercial Practices for Outsourcing of Services, GAO-02-214 
(Washington, D.C.: Nov. 30, 2001).

[4] We asked the military services and other DOD components to identify 
candidate projects because DOD does not maintain a central list of IT 
services outsourcing projects. We chose each project on the basis of 
the following criteria: (1) no more than one project from each military 
service and two agencies, (2) illustrative example of DOD IT 
outsourcing, (3) dollar value greater than $10 million, and (4) enough 
time elapsed for services to have been delivered and performance 
measured.

[5] We identified 70 practices in our November 2001 report on leading 
commercial practices that (1) are typically applied at the project 
level and (2) were verifiable through documentation and interviews. 

[6] This calculation does not include practices that were not 
applicable to a particular project. 

[7] This consensus view was based on interviews with managers in 
leading commercial organizations, discussions with academic and 
professional authorities, and extensive research on IT acquisition 
practices. 

[8] This figure is from the Federal Procurement Data System, which 
contains detailed information on contract actions over $25,000. 

[9] GAO-02-214. 

[10] Giga Information Group, Inc., IT Trends 2003: IT Services (Dec. 
19, 2002). We did not independently verify these data.

[11] INPUT, The Federal IT Outsourcing Market View (December 2002). We 
did not independently verify these data.

[12] GAO-02-214. 

[13] This consensus view was based on interviews with managers in 
leading commercial organizations, discussions with academic and 
professional authorities, and extensive research on IT acquisition 
practices. 

[14] Office of Management and Budget, Circular A-76, Performance of 
Commercial Activities (Washington, D.C.: June 14, 1999). In November 
2002, OMB issued proposed revisions that would substantially change 
this circular. As of April 8, 2003, these revisions have not yet been 
made final.

[15] Final report of the Commercial Activities Panel, Improving the 
Sourcing Decisions of the Government (Washington, D.C.: Apr. 30, 2002).

[16] Section 5 of P.L. 105-270, codified at 31 U.S.C. 501 note (1998), 
defines an inherently governmental function as a "function that is so 
intimately related to the public interest as to require performance by 
Federal Government employees." 

[17] U.S. General Accounting Office, Proposed Revisions to OMB Circular 
A-76, GAO-03-391R (Washington, D.C.: Jan. 16, 2003). 

[18] U.S. General Accounting Office, Desktop Outsourcing: Positive 
Results Reported, but Analyses Could Be Strengthened, GAO-02-329 
(Washington, D.C.: Mar. 29, 2002). 

[19] The Industrial College of the Armed Forces, Case Study: Complex 
Business Management for Competitive Sourcing (2001). 

[20] PricewaterhouseCoopers Endowment for the Business of Government, 
Contracting for the 21ST Century: A Partnership Model (January 2002). 

[21] GAO-02-329. 

[22] U.S. General Accounting Office, Contract Management: Taking a 
Strategic Approach to Improving Service Acquisitions, GAO-02-499T 
(Washington, D.C.: Mar. 7, 2002).

[23] This calculation does not include practices that were not 
applicable to a particular project.

[24] Giga Information Group, Inc., Optimizing IT Sourcing Strategy: Key 
Stages and Phases of the IT Sourcing Process (Jan. 31, 2003). 

[25] This calculation does not include practices that were not 
applicable to a particular project.

[26] Gartner, Inc., Strategic Analysis Report, How to Build a Sourcing 
Strategy, Research Note R-18-1099 (Sept. 23, 2002). 

[27] The Department of the Interior's GovWorks Program is a federal 
fee-for-service acquisition center that helps other federal agencies 
acquire supplies and services for their programs on a project-by-
project basis.

[28] Gartner, Inc., defines benchmarking as a method to compare the 
cost or price of an IT environment to peer groups with the same 
workload characteristics. 

[29] Giga Information Group, Inc., Payment and Incentives for 
Outsourcing Management (July 27, 2000) and Gartner, Inc., Benchmarking 
Helps Outsourcing Deals Stay Competitive, Research Note COM-16-8055 
(June 14, 2002).

[30] This calculation does not include practices that were not 
applicable to a particular project.

[31] This calculation does not include practices that were not 
applicable to a particular project.

[32] Seat management generally refers to service provision arrangements 
in which contractor-owned desktop and other computing hardware, 
software, and related services are bundled and provided to a client 
organization at a fixed price per unit (or seat). 

[33] This calculation does not include practices that were not 
applicable to a particular project.

[34] Gartner, Inc., Retain Enough Resources to Manage Outsourcing 
Deals, Research Note COM-16-8425 (June 17, 2002).

[35] An Interagency-Industry Partnership in Performance, Seven Steps to 
Performance-Based Services Acquisition, Benchmark Version (January 
2002).

[36] U.S. General Accounting Office, NASA: Better Mechanisms Needed for 
Sharing Lessons Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002). 

[37] U.S. General Accounting Office, Information Technology Investment 
Management: A Framework for Assessing and Improving Process Maturity, 
GAO/AIMD-10-1.23, Exposure Draft (Washington, D.C.: May 2000).

[38] U.S. General Accounting Office, DOD Competitive Sourcing: Lessons 
Learned System Could Enhance A-76 Study Process, GAO/NSIAD-99-152 
(Washington, D.C.: July 21, 1999). 

[39] GAO-02-329.

[40] GAO-02-195.

[41] GAO-02-195. 

[42] U.S. General Accounting Office, Information Technology: Leading 
Commercial Practices for Outsourcing of Services, GAO-02-214 
(Washington, D.C.: Nov. 30, 2001).

[43] GAO-02-214. 

[44] U.S. General Accounting Office, NASA: Better Mechanisms Needed for 
Sharing Lessons Learned, GAO-02-195 (Washington, D.C.: Jan. 30, 2002). 

[45] Gartner, Inc., Five Tough Questions About Skill Sourcing, Research 
Note SPA-13-2537 (Mar. 28, 2001). 

[46] SLAs define the agency's expectations and are used to track and 
measure a contractor's performance. 

[47] An Interagency-Industry Partnership in Performance, Seven Steps to 
Performance-Based Services Acquisition, Benchmark Version (January 
2002).

[48] Gartner, Inc., Retain Enough Resources to Manage Outsourcing 
Deals, Research Note COM-16-8425 (June 17, 2002).

[49] Gartner, Inc., Successful Outsourcing Means Retaining Some Staff, 
Research Note COM-18-9692 (Dec. 18, 2002). 

[50] An Interagency-Industry Partnership in Performance, Seven Steps to 
Performance-Based Services Acquisition, Benchmark Version (January 
2002).

GAO's Mission:

The General Accounting Office, the investigative arm of Congress, 
exists to support Congress in meeting its constitutional 
responsibilities and to help improve the performance and accountability 
of the federal government for the American people. GAO examines the use 
of public funds; evaluates federal programs and policies; and provides 
analyses, recommendations, and other assistance to help Congress make 
informed oversight, policy, and funding decisions. GAO's commitment to 
good government is reflected in its core values of accountability, 
integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony:

The fastest and easiest way to obtain copies of GAO documents at no 
cost is through the Internet. GAO's Web site ( www.gao.gov ) contains 
abstracts and full-text files of current reports and testimony and an 
expanding archive of older products. The Web site features a search 
engine to help you locate documents using key words and phrases. You 
can print these documents in their entirety, including charts and other 
graphics.

Each day, GAO issues a list of newly released reports, testimony, and 
correspondence. GAO posts this list, known as "Today's Reports," on its 
Web site daily. The list contains links to the full-text document 
files. To have GAO e-mail this list to you every afternoon, go to 
www.gao.gov and select "Subscribe to daily E-mail alert for newly 
released products" under the GAO Reports heading.

Order by Mail or Phone:

The first copy of each printed report is free. Additional copies are $2 
each. A check or money order should be made out to the Superintendent 
of Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 
more copies mailed to a single address are discounted 25 percent. 
Orders should be sent to:

U.S. General Accounting Office

441 G Street NW,

Room LM Washington,

D.C. 20548:

To order by Phone: 	

	Voice: (202) 512-6000:

	TDD: (202) 512-2537:

	Fax: (202) 512-6061:

To Report Fraud, Waste, and Abuse in Federal Programs:

Contact:

Web site: www.gao.gov/fraudnet/fraudnet.htm E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470:

Public Affairs:

Jeff Nelligan, managing director, NelliganJ@gao.gov (202) 512-4800 U.S.

General Accounting Office, 441 G Street NW, Room 7149 Washington, D.C.

20548: