Information Technology: DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects
Published: 2003-04-25T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Given the magnitude of its reported spending on information technology (IT) services--more than $6.2 billion in fiscal year 2001--it is critical that the Department of Defense (DOD) adopt effective practices for acquiring IT services. GAO researched leading commercial practices for the outsourcing of IT services, and, in November 2001, published a framework consisting of seven phases that span the full range of activities that are performed during the outsourcing of those services (this is an acquisition in which a client organization transfers responsibility for performing services to an external provider). GAO was asked to determine (1) the extent to which selected DOD projects for outsourcing IT services use leading commercial practices as specified in GAO's framework and (2) whether DOD is sharing lessons learned from its IT outsourcing projects across the department.
section_title: What GAO Found
paragraphs: The projects in GAO's review substantially used leading commercial practices as specified in GAO's framework for outsourcing IT services. Specifically, the agencies fully implemented 88 percent of the practices (not including practices not applicable to a particular project). This framework consists of practices organized into seven phases: (I) determine sourcing strategy, (II) define operational model, (III) develop the contract, (IV) select the provider(s), (V) transition to provider(s), (VI) manage the performance of the provider(s), and (VII) ensure services are provided. Although DOD has acted on gathering and disseminating lessons learned and commercial leading practices related to general acquisition issues, its actions have generally not been focused on outsourcing or on sharing the lessons learned from IT services outsourcing across the department. By not systematically capturing and disseminating such information across the department, DOD is losing the opportunity to leverage the knowledge gained on IT services projects like those in GAO's review. Lessons learned that are pragmatic and easily accessible would give DOD managers a more informed understanding of important issues to be addressed when making outsourcing decisions, as well as the factors to be considered to help ensure the success of these endeavors.

section_title: Letter
paragraphs: The Department of Defense (DOD) is the government’s largest purchaser of  information technology (IT) services, such as desktop support, network  operations, and software development services. In fiscal year 2001, DOD  reportedly obligated more than $6.2 billion on IT services, and this amount  is expected to grow substantially. Given the magnitude of DOD’s spending  on such services, it is critical that the department adopt effective practices  for acquiring IT services.
Since 1996, we have conducted a series of studies for the Senate  Committee on Armed Services concerning how DOD can improve its  acquisition processes by adopting proven practices of leading commercial  organizations. In this vein, in November 2001, we issued a guide that  organized leading commercial practices for the outsourcing of IT services  into a framework of seven phases that span the full range of activities that  are performed during IT services outsourcing.
This report responds to your request that we determine (1) the extent to  which selected DOD IT services outsourcing projects use leading  commercial practices as specified in our framework and  (2) whether DOD is sharing lessons learned from its IT outsourcing  projects across the department. To address the first objective, we selected  five projects from a group of projects to outsource IT services that were identified by the military services and other DOD components. We then  asked the component responsible for each project to perform a self- assessment against selected practices in our framework for outsourcing IT  services. Next, we obtained and reviewed agencies’ supporting  documentation and interviewed the appropriate agency and provider  project officials to independently determine whether a practice was met.  To address the second objective, we reviewed applicable DOD approaches  for capturing and disseminating lessons learned from IT services  outsourcing projects and interviewed the applicable acquisition and IT  officials. Details of our objectives, scope, and methodology are discussed  in appendix I.
subsections: section_title: Background
paragraphs: To protect the security of the United States, DOD relies on a complex array  of computer-dependent and mutually supportive organizational  components, including the military services, Commanders in Chief, and  Defense agencies. As such, it invests tens of billions of dollars each year in  a broad array of computer systems, which include weapon systems,  command and control systems, satellite systems, inventory management  systems, transportation management systems, health systems, financial  systems, personnel systems, and payment systems. In addition, DOD  spends billions of dollars annually on IT services, which include database  management, help-desk operations, software maintenance, and network  services. In fiscal year 2001, DOD reportedly obligated more than $6.2  billion on IT services alone.
Decisions regarding the purchasing of services are critical to ensuring the  effectiveness of DOD’s operations as well as those of the government as a  whole. Our November 2001 report recognizes the importance of such  sourcing decisions and provides a framework that spans the full range of  activities that are performed during IT services outsourcing. At the same  time, governmentwide policies, initiatives, and challenges exist that  significantly influence the government’s sourcing decisions.
subsections: section_title: GAO’s Framework for Outsourcing IT Services
paragraphs: Outsourcing of IT services has become increasingly popular in both the  public and private sector. For example, according to the Giga Information  Group, Inc., a leading research firm, such outsourcing is expected to grow  an average of 5 to 6 percent in 2003. The federal sector’s outsourcing is  predicted to rise at an even greater rate. For example, INPUT, an IT market  research firm, forecasts that defense IT outsourcing will increase about 143  percent between fiscal years 2002 and 2007.
IT outsourcing involves the activities associated with acquiring services  from one or more external providers. During outsourcing, a client  organization transfers responsibilities for performing one or more IT  services to one or more external providers. This responsibility is executed  through control and management of the processes, people, and technology  associated with these services.
Figure 2 depicts the roles of the client and provider organizations in an  outsourcing relationship.
Our November 2001 guide on leading commercial practices for outsourcing  IT services provides a generic framework of practices from leading  commercial organizations that can improve purchasing decisions and  manage the resulting government/provider relationship. The framework is  represented in figure 3 as a hierarchy of phases, practices, and critical  success factors.
Table 1 provides a definition of each phase of the framework. Each of the  phases has specific practices associated with it. Implementing these  practices does not guarantee the success of an outsourcing project.  However, our November 2001 study reflected a consensus view that these  practices were the most critical to success when IT services are being  acquired. Restated, application of these practices increases the  probability of a successful outsourcing project.
The organizations that we studied also identified certain capabilities  (identified as critical success factors) that were essential for implementing  the practices identified in our framework. First, executive leadership  strengthens the interaction between executive management and the  employees of the client organization. Second, partner alignment  strengthens the interaction between the client and provider organization at  the executive level, which ensures that the goals and objectives of these  organizations support each other. Third, relationship management  strengthens the interaction between the client and provider organization at  the operational level.
section_title: Influences on Government Sourcing Decisions
paragraphs: The federal government is one of the world’s largest users of services.  Because of the large dollar value and the number of private- and public- sector jobs involved, deciding whether the public or private sector would  be the most appropriate provider of the services the government needs (IT  or otherwise) is an important, and often highly charged, question. Among  the factors that agencies must consider as they determine how best to meet  their missions is whether the public or private sector would be the most  appropriate provider of the services the government needs. Phase I of our  framework, determine the sourcing strategy, addresses the client’s  assessment of whether expertise from within or outside of the organization  can more effectively meet the client’s needs.
The National Defense Authorization Act for Fiscal Year 2001 required the  Comptroller General of the United States to convene a panel of experts to  study the current process used by the government to make sourcing  decisions. The resulting Commercial Activities Panel conducted a year-long  study and heard repeatedly about the importance of competition and its  central role in fostering economy, efficiency, and continuous performance  improvement. In particular, the panel reviewed the government’s  implementation of the Office of Management and Budget’s (OMB) Circular  A-76, which sets forth federal policy for determining whether federal  employees or private contractors will perform commercial activities for the  government. Circular A-76 (1) outlines conditions under which agencies  are permitted to perform a commercial activity with government  employees or by contract and (2) provides guidance for whether, and if so,  how, agencies should conduct a cost comparison when they are  considering transferring the performance of commercial activities from the  public to the private sector (or vice versa). The panel reported that there  were positive elements to Circular A-76 but that both federal employees  and private firms complained that it does not meet the standard of a clear,  transparent, and consistently applied process. For example, both federal  employees and private firms criticized the Circular A-76 process as unequal  and therefore unfair.
The Commercial Activities Panel strongly supported continued emphasis  on competition and concluded that whenever the government is  considering converting work from one sector to another, public/private  competitions should be the norm. In addition, the panel made four  recommendations, including that all sourcing decisions be consistent with  the principles adopted unanimously by the panel, such as the principle that  federal policy provide for accountability in connection with all sourcing  decisions.
As part of the administration’s efforts to implement the recommendations  of the Commercial Activities Panel, OMB has published proposed changes  to Circular A-76. Key highlights of the proposed changes include presuming  that all functions are commercial in nature unless they are justified as  inherently governmental; limiting the length of time for competitions; and  emphasizing awarding contracts on the basis of best value, not just lowest  cost. Best value allows the contracting official to consider technical  superiority, quality, innovation, and past performance as well as price.
However, we reported that there are several areas in which the proposed  revisions to the circular are not consistent with the principles or  recommendations of the Commercial Activities Panel. Specifically, the  proposed revision does not include a link between sourcing policy and  agency missions, has unnecessarily complicated source selection  procedures, contains certain unrealistic time frames, and includes  insufficient guidance on calculating savings.
Beyond the Commercial Activities Panel, other bodies have identified  challenges that the federal government faces in reaching and executing  effective sourcing decisions. For example, members of the Coalition for  Government Procurement, the Professional Services Council, and the  Information Technology Association of America told us that organizational  culture is one of the biggest differences between the commercial sector  and the federal government and one of the greatest barriers to the  government’s use of commercial practices. Also, as we have previously  reported, moving to outsourcing solutions can involve a cultural change for  government organizations because it may require a change to an agency’s  operating model, such as using a contractor to provide IT services  previously performed by government staff or using a performance-based  contract. This view was echoed by a 2001 study of DOD competitive sourcing that found cultural, process, execution, and training barriers.  The study stated that these barriers need to be understood and mitigated  before the benefits of outsourcing can be fully realized. Barriers such as  these can be overcome by strong executive leadership, which is a critical  success factor in our framework.
Another challenge is creating a productive agency/provider relationship— another critical success factor in our framework. According to a report  sponsored by the PricewaterhouseCoopers Endowment for the Business of  Government, such public/private partnerships are based on trust,  commitment to problem or conflict resolution, and the recognition that  flexibility is necessary and that the relationship will evolve and change over  time. If deadlines are not met, or public agency goals change with  differing political climates, the partners need to discuss the basis of the  partnership and construct a different relationship. Our prior report on  desktop outsourcing found that developing a productive agency/contractor  relationship is not always easy. Both sides must recognize and understand  each other’s underlying motives and strive to achieve established  expectations.
Finally, human capital issues are another challenge facing federal agencies  that affect their ability to implement outsourcing. Our framework  recognizes the importance of having the right skills in place to support the  outsourcing relationship. However, as we have previously reported,  procurement reforms and technological changes have placed  unprecedented demands on the acquisition workforce. Contracting  personnel are now expected to have a much greater knowledge of market  conditions, industry trends, and the technical details of the commodities  and services they procure. The Commercial Activities Panel report stated  that developing and maintaining a skilled acquisition workforce is the  critical first step in managing this more complex procurement  environment. The panel also reported that DOD bore the brunt of a 22  percent downsizing of the federal acquisition workforce in the last decade,  going from 96,000 staff in 1991 to about 68,000 in fiscal year 2001.
Addressing human capital issues is not just a matter of the size of the  workforce; it is also a knowledge and skills issue. According to the  Commercial Activities Panel, it is critically important that federal agencies  adequately address human capital needs in meeting the current and  emerging needs of government and its citizens in the most effective,  efficient, and economical manner possible. This will require increased  emphasis on training and development, particularly in the area of  technology.
section_title: Description of Five Projects Reviewed
paragraphs: The five projects in our study varied in how they approached outsourcing  IT services, such as in using various solicitation methods, including holding  a public/private competition under the policies outlined in OMB Circular A- 76 or carrying out a negotiated competitive procurement. In addition, the  types of services being outsourced differed: services ranged from the  narrowly focused (e.g., help-desk services) to the very broad (e.g.,  enterprisewide end-to-end information services); contract terms ranged  from 5 to 15 years (assuming all option years are exercised); and estimated  contract values ranged from $23 million to $8.8 billion. Table 2 provides  information on the variety of IT services and outsourcing approaches taken  by the projects.
section_title: Projects Substantially Used Leading Commercial Practices
paragraphs: As illustrated in figure 4, the five IT services projects substantially used  leading commercial practices. Specifically, each project used at least 76  percent of the practices. Reasons for projects implementing different  percentages of the practices include differences in their individual  circumstances and objectives. For example, the Army’s Total Army  Communications—Southwest Asia (TAC-SWA) project, which used the  fewest number of practices and had the largest number of practices that  were not applicable, was largely a continuation of an existing approach  that already relied on the private sector but with fewer providers. In  contrast, the National Imagery and Mapping Agency’s (NIMA) Information  Technology/ Information Services (IT/IS) project, which fully or partially  implemented all of the applicable practices, involved a significant  operational shift (e.g., functions previously performed by NIMA staff are  now performed by a contractor) and was intended to result in substantial  process improvements. In addition, the three projects that implemented the  largest percentage of practices also used third-party assistance—including  employing a contractor with sourcing expertise—to help formulate their  sourcing strategy, which could account for the extent of their compliance.
In addition, figure 5 illustrates that project compliance extended to each of  the phases of our framework. Collectively, the projects fully implemented  from 70 to 97 percent of the practices in each phase. Phase I, determine  sourcing strategy, had the lowest percentage of practices implemented by  the projects (70 percent). This result is not inconsistent with a recent Giga  Information Group, Inc., survey, which found that only half of the respondent organizations had documented an IT sourcing strategy. This  approach carries risk since phase I sets the tone for the outsourcing  initiative within the client organization.
In addition, collectively the projects fully implemented 88 percent of the  practices (see table 3).
The following provides additional information on the projects’  implementation of each phase of our framework.
Phase I: Determine sourcing strategy. In the first phase of our  outsourcing framework, the client organization determines whether  internal capability or external expertise can more effectively meet its IT  needs. The purpose of a sourcing strategy is to achieve the optimal  balance between internal and external capabilities, activities, processes,  and services to ensure the achievement of strategic business objectives  at the lowest risk. The five projects’ implementation of this phase was  uneven. In particular, two of the six practices in this phase were fully  implemented by all five projects, but the other four practices were not.  Among the practices that were implemented by all of the projects was  determining the business reasons for outsourcing. In addition, the three  projects that implemented the largest percentage of practices in our  framework—the Military Health System/Information Technology  Organization (MHS/ITO) Help Desk, the Department of the Navy’s Navy  and Marine Corps Intranet (NMCI), and NIMA IT/IS projects—used the  third-party assistance practice in this phase to help formulate their  sourcing strategy, which could account for the extent of their  compliance. For example, the MHS/ITO Help Desk project, which  implemented the largest percentage of practices, worked with the  Department of the Interior’s GovWorks Program, the Defense  Acquisition University staff, and a private-sector contractor to obtain  expertise on sourcing strategies.
The practice in this phase that was the most unevenly implemented was  the benchmarking and baselining of existing internal services. Of the  five projects in our review, (1) one fully benchmarked and baselined the  productivity of the activity being outsourced before making the final  sourcing decision, (2) two partially baselined their existing activities,  and (3) two did not perform benchmark and baseline analyses at all.  The agencies’ reasons for not fully implementing this practice included  that an executive decision had been made to conduct a public/private  competition following the OMB Circular A-76 policy, so such an  analysis would not have affected the sourcing decision, or that  available documentation to perform such an analysis was limited.  Leading research firms suggest benchmarking and baselining the  entity’s current processes before outsourcing because only then would  it be able to determine whether the arrangement has been successful.  In addition, the risk of not fully baselining the existing environment is  illustrated by the NMCI project. Specifically, the NMCI project’s  baseline of its existing environment was limited because it did not  include an assessment of its legacy applications since project officials  decided to rely on a preexisting inventory developed to address the  Year 2000 challenge. The Navy subsequently found that it had  substantially underestimated its number of legacy applications, that,  according to program officials, later contributed to the transition  period slipping from 2-½ years to 3-½ years. Appendix II provides  additional information on projects’ implementation of the practices  comprising this phase.
Phase II: Define operational model. The operational model is an  important mechanism for an organization to compare its plans with the  expectations that were set when the decision to outsource was made  and to ascertain whether these plans will enable the organization to  meet expectations. The five projects had largely implemented the 13  practices contained in this phase. Specifically, about 90 percent of the  practices were implemented. For example, all projects implemented the  practice that executive leadership be established to facilitate the  outsourcing effort. NIMA, for instance, formed a strategic sourcing  office to oversee the IT/IS project. Another practice—training the  provider on the organization’s business environment and goals—was  fully implemented by one project (in two cases, the practice was not  applicable). One project that did not fully implement this practice was  the Air Force’s Kirtland Air Force Base’s Command, Control,  Communications, and Computer (C4) Services project. Although some  training was provided (e.g., Kirtland held an orientation session for  potential bidders), provider officials stated that they did not receive  adequate training, which made the transition period more difficult.  Appendix III provides additional information on projects’  implementation of the practices comprising this phase.
Phase III: Develop the contract. A well-written contract is necessary for  the outsourcing organization to meet its requirements while allowing  the service provider to make a fair profit. It sets the expectations for  service levels, delivery of essential services, and continuous  improvement and should protect the interests of all parties. The five  projects largely implemented the practices in this phase. Specifically,  about 89 percent of the practices were implemented in this phase, and  two projects (the MHS/ITO Help Desk and NIMA IT/IS projects)  implemented all of the practices. Several practices in this phase address  performance requirements. For example, all five projects implemented  the practices that called for basing performance requirements on  business requirements and reviewing and updating them periodically.  One practice that was not fully implemented by two projects was  including performance measures that address both technical and end- user satisfaction aspects of performance. For example, the Army TAC- SWA project included technical performance measures in its contract  but not measures related to end-user satisfaction, even though the  contract included help-desk services. According to the project official  that developed the performance work statement in the contract, the  command did not include customer satisfaction measures because it did  not think that it was necessary to have a performance standard for the  help-desk service. However, without such measures, the agency does  not have a contractual standard with which to judge the provider’s  performance. Appendix IV provides additional information on projects’  implementation of the practices comprising this phase.
Phase IV: Select the provider(s). Critical to the success of any  outsourcing project for IT services is identifying potential providers and  ultimately selecting a provider(s) that will best meet the needs of the  agency. The five projects had largely implemented the seven practices  contained in this phase. About 89 percent of the practices were  implemented in this phase and two projects (the MHS/ITO Help Desk  and Navy NMCI projects) used each applicable practice. For example,  the five projects implemented the practice related to conducting due  diligence activities to verify provider capabilities before signing the  contract. In the case of the Army TAC-SWA project, the Network  Enterprise Technology Command (NETCOM) evaluated the provider’s  financial and past performance information. The Department of the  Navy’s NMCI project also evaluated bidders’ past performance and  performed reference checks. Moreover, as part of its due diligence  activities, the Navy required all bidders to demonstrate that they had  experience in implementing large seat management contracts. A  practice that was not implemented by two projects was using third-party  assistance when selecting the provider. Projects that did not implement  this practice believed that they had adequate in-house expertise with  outsourcing, making third-party assistance unnecessary. However,  because third-party assistance provides an independent resource that  can suggest options or processes that the client organization may not be  aware of, these projects may have missed an opportunity to implement  their outsourcing projects more effectively. Appendix V provides  additional information on projects’ implementation of the practices  comprising this phase.
Phase V: Transition to provider(s). This phase focuses on the client  organization’s transfer of the IT function to one or more providers. As  part of this transition, the clear definition of responsibilities and the  careful consideration of employees’ needs matched against the client  organization’s needs enable both the client and provider to focus on  execution and give staff confidence in their future employment. The five  projects largely implemented the 11 practices associated with this  phase. Specifically, about 90 percent of the practices were  implemented in this phase, and two projects implemented all of the  practices (the MHS/ITO Help Desk and NIMA IT/IS projects). Several  practices in this phase address dealing with employees affected by the  outsourcing projects. For example, in the four projects in which federal  employees were affected, the projects provided assistance to those who  did not want to transfer to the provider, including helping to place them  in other positions and helping with résumé writing. A related practice is  to clearly communicate to all employees what is going to happen and  when it is going to happen. Two projects did not fully implement this  practice. For example, the Navy used its normal chain of command to  communicate transition information, but found that implementation of  this practice was uneven. As a result, some staff did not know current  information about how NMCI would affect them until the provider was  ready to contact them regarding their possible transition to the  contractor. However, according to the NMCI’s Director’s office, this  problem was somewhat mitigated by the provider’s Web site that  provides transition information to all NMCI customers/users. Appendix  VI provides additional information on projects’ implementation of the  practices comprising this phase.
Phase VI: Manage provider(s) performance. The effectiveness with  which the provider(s) performance is managed is critical to the  successful implementation of an outsourcing project. Indeed, according  to Gartner, Inc., a leading research firm, an outsourcing project can be  thwarted by poorly designed, funded, and delivered processes for  managing the delivery of services. The five projects generally  implemented the 11 practices contained in this phase, with about 84  percent of the practices being implemented in this phase. For example,  the practices related to obtaining feedback on provider performance  were largely implemented. This is important because different levels of  an agency can have different perceptions about the value of the  outsourcing project. For example, an outsourcing project may be  considered successful by the agency’s executive management if it is  focused on controlling costs, but be considered inadequate by business  managers and users who may be expecting higher levels of service. Each  of these viewpoints is valid and should be taken into account when the  provider’s performance is evaluated. Two other practices—including  incentives and penalties in contracts—were fully implemented by two  projects. Incentives and penalties are important because they can help  motivate the provider to exceed or meet performance requirements.  Nonetheless, two projects did not include monetary incentives and two  projects did not include monetary penalties in their contracts. For  example, the Army TAC-SWA project did not include monetary  incentives, although the contracting officer stated that incentives might  have been useful to motivate the provider to exceed performance  requirements. Incentives can also help control risks. According to a  guide on performance-based services acquisition, if the incentives in the  contract are right and if the provider and agency share the same goals,  risk is largely controlled and effective performance is “almost the  inevitable outcome.” Appendix VII provides additional information on  projects’ implementation of the practices comprising this phase.
Phase VII: Ensure services are provided. Although outsourcing  transfers responsibility for performing the service to the provider(s), the  client organization is ultimately responsible for ensuring that services  are provided and that end-user needs are met. Accordingly, it is critical  that the agency ensure that services are provided. The projects had  implemented 97 percent of these practices, and four projects  implemented all of them. For example, every project monitored the  providers’ work. In the case of the Air Force’s C4 Services project,  quality assurance evaluators monitored the provider’s work to identify  problems or trends in accordance with the project’s quality assurance  surveillance plan. The results were reported to the contracting officer  and to the functional area chief for resolution. Another practice, using  customer satisfaction surveys, was fully implemented by four of the five  projects. However, the Air Force project did not conduct, or require its  contractor to conduct, customer satisfaction surveys. Although the  provider surveys staff annually, the Air Force is nevertheless relying on  the provider to voluntarily implement an important practice for  determining how customers view the services being delivered and  whether changes need to be made. Appendix VIII provides additional  information on projects’ implementation of the practices comprising  this phase.
section_title: Leveraging Lessons Learned DOD-wide Could Assist Other DOD Projects
paragraphs: We have previously reported on the importance of collecting and  disseminating lessons learned. For example, a critical activity in IT  investment management is establishing a process for developing and  capturing lessons learned in a written product or knowledge base and  disseminating them to decision-makers. In addition, one of the practices  in our framework for outsourcing IT services addresses incorporating  lessons learned from peers who have engaged in similar sourcing  decisions. Use of lessons learned is a principal component of an  organizational culture committed to continuous improvement. Sharing  such information serves to communicate acquired knowledge more  effectively and to ensure that beneficial information is factored into  planning, work processes, and activities. Lessons learned can be based on  positive experiences or on negative experiences that result in undesirable  outcomes.
Although DOD has taken action to gather and disseminate lessons learned  and best practices on general acquisition issues, these efforts generally do  not focus on outsourcing or include sharing the lessons learned from IT  outsourcing projects across the department. Specifically, a number of DOD  Web sites provide guidance, lessons learned, and best practices related to  general acquisition issues. However, using these sites to locate specific  information on IT outsourcing best practices and lessons learned can be  time-consuming and difficult because so many topics and information  sources are provided. Specifically, MHS/ITO Help Desk project officials  said that searching numerous Web sites to get relevant information to  address questions and concerns about outsourcing IT services can  consume hours. For example, when we entered the keywords “IT  outsourcing” and “best practices” into the search feature on the Office of  the Undersecretary for Defense of Acquisition, Technology, and Logistics  site, ACQWeb (www.acq.osd.mil) in early March, it provided us with links  to 1,251 documents. Likewise, lessons learned covers so many topics that it  is difficult to search for an applicable IT lesson. For example, when we  used the phrase “lessons learned in IT outsourcing” no documents were  identified on the ACQWeb, but when we inserted “lessons learned” and “IT  outsourcing” links to more than 1,700 documents were produced.
One DOD Web site, Share A-76!, was established to address one of our  previous recommendations, that is, to establish a framework for  identifying and analyzing best practices and lessons learned from  competitive sourcing studies and disseminating them DOD-wide. Share A- 76! promotes the sharing of best practices and lessons learned related to  one form of outsourcing that was conducted under the OMB Circular A-76  competitive sourcing process. Among other things, the site contains  guidance, links to other relevant sites, sample documents, and a best  practices library that communicates field staff experiences and advice  about the Circular A-76 process. A NIMA project official said that NIMA’s  staff routinely accesses the Web site because it contains a wealth of  information on policies, procedures, lessons learned, and links to other  outsourcing sites. The DOD analyst responsible for Share A-76! estimated  that the site receives about 12,000 visits per month and said that on the  basis of E-mails and anecdotes, user satisfaction is favorable. However, this  Web site is specific to the OMB Circular A-76 process, which may not apply  to other types of outsourcing. For example, Circular A-76’s policy pertains  to public/private competitions and requires that the final evaluation  between the government and the private sector be based exclusively on  cost.
DOD acquisition and IT officials acknowledged that there is no mechanism  in DOD to easily share and leverage lessons learned relating to outsourcing  IT services. However, these officials agreed that a departmentwide effort to  identify, capture, and disseminate lessons learned and leading practices of  projects with experience in carrying out IT outsourcing could offer  valuable insights and new ideas that would benefit others. Moreover,  officials from three of the projects in our review told us that there is value  in collecting and disseminating the knowledge acquired from IT  outsourcing projects in a systematic manner across the department.
Each of the projects in our review identified knowledge and experience  gained from their approaches to outsourcing IT services that could offer  insights and practices for other ongoing and future projects to consider.  For example:  MHS developed specific guidance and lessons learned for implementing  a performance-based incentive contract for help-desk operations.
The Department of the Navy’s NMCI project has developed a series of  lessons learned related to transitioning to the provider that is being  shared within the NMCI community; one example was that all personnel  should be available during scheduled testing and deployment.
NIMA has experience in contracting techniques emphasizing a  partnering approach with providers to refine requirements and establish  a common understanding of costs.
In addition, a departmental IT outsourcing knowledge-sharing approach  could include links to information about other government agencies’ IT  outsourcing projects. For example, our 2002 report on desktop outsourcing  includes an extensive discussion of lessons learned by agencies that have  implemented this type of IT services outsourcing.
Developing an effective lessons learned activity is not easy. For example,  NMCI officials said that for a lessons learned initiative to be effective, a  process must exist that is clearly understood by everyone and allows  capturing and sharing of knowledge to occur with minimum effort. Other  challenges in developing an effective lessons learned process were outlined  by the Share A-76! analyst. The analyst stated that only a small number of  site users have contributed lessons learned to the Share A-76! Web site,  which she attributed, in part, to the amount of time and effort needed to  document and obtain agreement by all levels of the organization on the  lessons learned. In addition, the analyst stated that there is reluctance to  share negative lessons, and often the review and approval process sanitizes  best practice information so that it becomes too general to be most helpful  to users. Such challenges can be overcome by executive-level support.  Indeed, DOD acquisition and IT officials stated that for lessons learning  activities to be effective, senior management must devote support and  resources to the effort. This is consistent with our prior work, which  showed that knowledge can be effectively shared only when employees are  given adequate time as well as established places where they can actually  transfer knowledge.
Last year, we outlined a generic lessons learned process that could be used  to guide the development of such a process for outsourcing IT services.  Although the mechanism or processes used to collect, share, and  disseminate lessons learned may vary, in general such a process comprises  four main elements: collection, verification, storage, and dissemination.  The collection process involves the capture of information through  structured and unstructured processes. Verification serves to verify the  correctness and applicability of lessons submitted. The storage aspect of  lessons learned usually involves incorporating the lessons into an  electronic database for the dissemination and sharing of information,  including the ability to conduct information searches. The final element,  and the most important, is the dissemination of lessons learned, since  lessons are of little benefit unless they are distributed and used by people  who will benefit from them. Lessons can be “pushed,” or automatically  delivered to a user, or “pulled” in situations where a user must manually  search for them. Lessons can also be disseminated with an assigned  priority descriptor, which denotes the risk, immediacy, and urgency of the  lessons learned content.
section_title: Conclusions
paragraphs: The projects in our review substantially implemented leading commercial  practices for outsourcing IT services, which has increased each project’s  probability of success. Capturing how these projects operationalized  leading commercial practices could help other IT services outsourcing  projects succeed. Although currently there is no such DOD-wide  mechanism, such as an electronic tool, to easily share and leverage lessons  learned, DOD IT and acquisition officials agreed that a departmentwide  effort to identify, capture, and disseminate lessons learned could offer  valuable insights and new ideas that would benefit others. Lessons learned  that are pragmatic and easily accessible could give DOD managers a more  informed understanding of the important issues to be addressed when  making outsourcing decisions, as well as the factors to be considered to  help ensure the success of these endeavors. DOD managers can also  benefit from lessons learned on the basis of negative experiences. The  projects in our review were well into implementation, and therefore, at this  late stage, we see little advantage for them to revisit practices that were not  implemented. Nevertheless, an electronic tool for capturing and  disseminating lessons learned would allow the rest of DOD to benefit from  the negative consequences and increased risks associated with those  practices that the projects did not implement.
Developing a lessons learned mechanism is not easy; thus, senior  management support and resources are keys to success. Without such  support driving the capture and dissemination of lessons learned, DOD is  losing an opportunity for wider application of leading practices and thus  better ensuring that its IT outsourcing efforts are successful.
section_title: Recommendations
paragraphs: To assist DOD organizations in planning and implementing outsourcing  projects for IT services, we recommend that the Secretary of Defense  direct the Undersecretary of Defense for Acquisition, Technology, and  Logistics, working in conjunction with the Assistant Secretary of Defense  for Command, Control, Communications, and Intelligence (C3I), to provide  senior management support and adequate resources to develop and  implement an electronic tool to capture and disseminate examples and  lessons learned from actual IT outsourcing projects. These examples and  lessons learned, at a minimum, should include the results of our review of  the five projects discussed in this report.
We also recommend that the Secretary of Defense direct the  Undersecretary of Defense for Acquisition, Technology, and Logistics,  working in conjunction with the Assistant Secretary of Defense for  Command, Control, Communications, and Intelligence (C3I), to ensure that  the method used to gather information for this electronic tool incorporate  the main elements of a lessons learned process—namely, collection,  verification, storage, and dissemination.
section_title: Agency Comments and Our Evaluation
paragraphs: In written comments on a draft of our report, signed by DOD’s Principal  Director, Deputy Assistant Secretary of Defense (Deputy Chief Information  Officer), the department partially concurred with our recommendations.  Specifically, DOD agreed that capturing lessons learned in the development  and implementation of its IT outsourcing initiatives is important to  continually improve its outsourcing methods and results achieved. The  department also stated that before deciding on a specific method to  achieve this aim, it intends to explore a variety of mechanisms that could  be used. In particular, the department stated that it currently has several  processes and communities of interest that collect and disseminate lessons  learned in other areas, which are logical starting points for determining the  best path forward. DOD’s written comments are reproduced in appendix  IX.
We agree that it is prudent to explore various alternatives to leveraging  lessons learned from DOD’s IT services outsourcing experiences. Our  recommendations are not prescriptive as to the electronic method to be  used to capture and disseminate lessons learned. Therefore, the  department’s plan to explore various alternatives is consistent with our  recommendations.
We are sending copies of this report to the appropriate congressional  committees, the Secretary of Defense, and the Director of the Office of  Management and Budget. We will also provide copies to others upon  request. In addition, the report will be available at no charge on the GAO  Web site at http://www.gao.gov.
If you have any questions on matters discussed in this report, please  contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director, at  (202) 512-9556. We can also be reached by E-mail at hiter@gao.gov and  lambertl@gao.gov, respectively. Other contacts and key contributors to this  report are listed in appendix X.
section_title: Objectives, Scope, and Methodology
paragraphs: Our objectives were to determine (1) the extent to which selected  Department of Defense (DOD) information technology (IT) services  outsourcing projects use leading commercial practices as specified in our  framework and (2) whether DOD is sharing lessons learned from its IT  outsourcing projects across the department.
To determine the extent to which selected DOD outsourcing projects for IT  services use leading commercial practices, we identified the practices in  our November 2001 report on leading commercial practices that (1) are  typically applied at the project level and (2) were verifiable through  documentation and interviews. Because DOD did not centrally maintain a  list of outsourcing projects for IT services, we asked the department to  identify candidate projects for our evaluation. From this list, we selected  the following five projects for our review: (1) Air Force Kirtland Air Force  Base’s Command, Control, Communications, and Computer (C4) Services  project; (2) Army Network Enterprise Technology Command’s (NETCOM)  Total Army Communications – Southwest Asia (TAC-SWA) project; (3)  Military Health System/Information Technology Organization (MHS/ITO)  Help Desk project; (4) Department of the Navy’s Navy and Marine Corps  Intranet (NMCI) project; and (5) National Imagery and Mapping Agency  (NIMA) Information Technology/Information Services (IT/IS) project. We  chose each project on the basis of the following criteria: (1) no more than  one project from each military service and two agencies, (2) illustrative  example of DOD IT outsourcing, (3) dollar value greater than $10 million,  and (4) enough time elapsed for services to have been delivered and  performance measured.
At our request, each project completed a self-assessment on whether and  how it implemented leading commercial practices. We reviewed the agency  self-assessments and accompanying documentation and interviewed the  appropriate agency project officials to verify whether the practices were  followed. In addition, we interviewed representatives from each of the  providers associated with these projects.
We also researched additional information on commercial practices in our  November 2001 guide and the challenges the federal government faces in  implementing them. Specifically, we performed a literature search, which  included reviewing reports issued by leading research firms, such as  Gartner, Inc., and Giga Information Group, Inc. In addition, we interviewed  representatives from industry organizations that have an interest in  outsourcing IT services, including the Coalition for Government  Procurement; the Information Technology Association of America; the  Professional Services Council; and Acquisition Solutions, Inc.
To determine whether DOD is sharing lessons learned from its IT  outsourcing projects across the department, we identified and reviewed  various approaches that DOD currently uses to capture and disseminate  such information. This included identifying and reviewing various Web  sites and performing key word searches on these sites to identify lessons  learned for outsourcing IT services. We also interviewed applicable DOD  acquisition and IT officials. Finally, we identified and reviewed a generic  lessons learned process contained in our January 2002 report on the  National Aeronautics and Space Administration’s lessons learned  mechanisms.
We performed our work at the Army’s NETCOM in Ft. Huachuca, Arizona;  the MHS/ITO Help Desk project office in Falls Church, Virginia; the NMCI  Director’s office in Crystal City, Virginia; and NIMA’s headquarters in  Bethesda, Maryland. We conducted our review between May 2002 and early  March 2003 in accordance with generally accepted government auditing  standards.
section_title: Projects’ Implementation of Phase I: Determine Sourcing Strategy
paragraphs: In the first phase of our outsourcing framework, the client organization  determines whether internal capability or external expertise can more  effectively meet its IT needs. The purpose of this sourcing strategy is to  achieve the optimal balance between internal and external capabilities,  activities, processes, and services to ensure that strategic business  objectives are achieved at the lowest risk. Among the factors that an  organization should evaluate in crafting this strategy are technology,  business, financial, and personnel requirements and whether it has skilled  business and IT managers. In addition, according to Gartner, Inc., sound  sourcing decisions depend on whether IT organizations (1) know and  understand their business priorities, (2) are prepared to invest in some  skills and divest others, and (3) identify and assess trade-offs.
The six practices in this phase that we used to evaluate the five projects are  as follows:  Use third-party assistance with experience in a variety of sourcing  arrangements when formulating a sourcing strategy.
Incorporate lessons learned from peers who have engaged in similar  sourcing decisions.
Estimate impact of sourcing decision on internal organization.
Benchmark and baseline productivity of internal services before making  the final sourcing decision.
Determine the business reasons for outsourcing IT.
Determine reasons for outsourcing IT that can improve the  organization’s ability to use and manage technology.
Figure 6 shows that the implementation of the practices by the five projects  in our review was uneven.
Table 4 provides detailed information on whether and how each project  implemented each of the six practices in this phase.
section_title: Project’s Implementation of Phase II: Define Operational Model
paragraphs: Critical to the successful outsourcing relationship is an operational model  for guiding the structure of the contract and the plans for transition. In  defining the operational model, client organizations formalize executive  leadership, team composition, client responsibilities, and operating  relationships between the client and provider. The operational model helps  the organization to compare its plans with the expectations that were set as  the initial decision to outsource was made and to ascertain whether these  plans will enable the organization to meet those objectives. An important  aspect of the operational model is an explicit understanding of how the  client organization plans to communicate its needs and provide feedback to  the provider. In addition, communication between the business and IT  offices within the client organization is always critical. This is particularly  true in the case of outsourcing because the IT service provider is outside  the client organization and disconnects are more likely to occur. Therefore,  organizational processes to facilitate good communication are critical.
The 13 practices in this phase that we used to evaluate the five projects are  as follows:  Establish executive leadership for IT to facilitate the outsourcing  initiative.
Continually communicate/clarify outsourcing objectives, while  correcting misinformation that affects the organization.
Establish a core group of people who will be involved in all phases of  outsourcing.
Select a person involved in the negotiation of the contract to manage the  outsourcing relationship.
Create and define a contract management structure with operational  points of contact and managers.
Define the role of internal IT managers and business leaders.
Ensure that the right skills are in place to support the outsourcing  relationship.
Establish a point of contact high in the provider management structure  for elevating provider performance concerns.
Have provider establish an on-site support team to serve as liaison  between client and provider.
Train provider on client business environment and goals.
Select or develop standard tools for managing the relationship.
Use third-party assistance to take advantage of expertise from a variety  of outsourcing arrangements in defining the operational model.
Ensure that the provider management team has prior experience in the  client’s field of business.
Figure 7 illustrates that the five projects in our review largely implemented  the practices.
Table 5 provides detailed information on whether and how each project  implemented each of the 13 practices in this phase.
section_title: Project’s Implementation of Phase III: Develop the Contract
paragraphs: Phase III focuses on the development of the contract, which defines the  legal terms of the relationship between client and provider. While other  phases in the outsourcing process describe the need for mutual trust and a  close, flexible working relationship, this phase focuses on the development  of the contract, which is the foundation on which a working relationship  will be built. A well-written contract helps the outsourcing organization  meet its requirements while allowing the service provider to make a fair  profit. It sets the expectations for service levels, delivery of essential  services, and continuous improvement and is intended to protect the  interests of all parties.
The 16 practices in this phase that we used to evaluate the five projects are  as follows:  Base performance requirements on business outcomes.
Include measures that reflect end-user satisfaction as well as technical  IT performance.
Review and update performance requirements periodically.
Require the provider to meet the minimum performance in each  category of service.
Require the provider to achieve escalating performance standards at  agreed-upon intervals.
Incorporate sufficient flexibility so that minimum acceptable  performance can be adjusted as conditions change, as the provider  becomes more adept at satisfying customer demands, and as  improvement goals are achieved.
Use service-level agreements (SLA) to clearly articulate all aspects of  performance, including management, processes, and requirements.
Client and provider work together to define the appropriate number of  SLAs and appropriate structure for each.
Specify circumstances under which the provider is excused from  performance levels mandated by master service agreements.
Client and provider work together to identify SLAs for which  compensation is based, while additional SLAs may be defined to manage  performance.
The contract should include clauses for (1) determining pricing  structures; (2) performing customer satisfaction surveys and using the  results to redefine performance levels; (3) terminating the contract; (4)  resolving disputes in a timely manner; (5) taking work away, without  penalty, from provider for nonperformance; (6) declaring a significant  event that can lead to a change in the contract; (7) defining performance  requirements; and (8) conducting regularly scheduled meetings.
Consider setting up a master services agreement under which all  arrangements between client and provider operate.
Include the appropriate representation from each major organizational  unit on the contract negotiation team.
Specify the use of volume purchases to obtain optimal discounts.
Use third-party assistance in negotiating and developing the contract.
Sign the contract after contract negotiations and final vendor selections.
As figure 8 shows, the five projects largely implemented the practices.
Table 6 provides detailed information on whether and how each project  implemented each of the 16 practices in this phase.
section_title: Projects’ Implementation of Phase IV: Select the Provider(s)
paragraphs: Critical to the success of any project to outsource IT services is the  identification of potential providers and the ultimate selection of a  provider(s) that will best meet the needs of the organization. Developing a  strategy that will lead to the selection of the “right contractor” is especially  important in a performance-based acquisition. The overall success of the  outsourcing project requires the contractor to understand the  performance-based approach, know or develop an understanding of the  organization’s requirement, have a history of performing exceptionally in  the field, and have the processes and resources in place to support the  mission.
The seven practices in this phase that we used to evaluate the five projects  are as follows:  Conduct research on the state of the market, vendors, and technology  before defining vendor selection criteria.
Identify and evaluate various sourcing solutions (e.g., single vendor,  multivendor, and alliance).
Define a process for selecting vendors to be providers.
Define vendor selection and evaluation (acceptance) criteria at the  outset.
When issuing an RFP, identify services with expected performance  levels and define client and provider roles and responsibilities.
Use third-party assistance with expertise in a variety of outsourcing  arrangements when selecting provider(s), including developing the RFP.
Conduct due diligence activities to verify vendor capabilities before  signing the contract.
As figure 9 illustrates, all five projects largely implemented the practices.
Table 7 provides detailed information on whether and how the projects in  our review implemented each of the seven practices in this phase.
section_title: Projects’ Implementation of Phase V: Transition to Provider(s)
paragraphs: In this phase, client organizations transfer responsibility of IT functions to  one or more providers. A clear definition of responsibilities among the  various parties and the careful consideration of employees’ needs matched  against the organization’s needs enable both the client and provider to  focus on execution and give staff confidence in their future employment. If  the contractor is assuming responsibility for functions previously  performed by federal employees, it is especially important that the  organization communicate a clear transition process. Without such  communication, an outsourcing project can be negatively affected if  misinformation and mistrust ensues.
The 11 practices in this phase that we used to evaluate the five projects are  as follows:  Communicate a clear transition process to all key players from both  client and provider organizations.
Clearly communicate to employees what is going to happen and when it  is going to happen.
Establish a client transition team with representatives from across the  organization to facilitate the transition.
Place the transition under a single program manager.
Create client/provider transition teams to address short-term transition  tasks as required.
Recognize that it takes time to effect transition and plan accordingly.
Encourage the transition of staff to the provider, where appropriate,  using bonuses, stock options, and other appropriate methods.
Develop employee-retention programs and offer bonuses to keep key  people, where appropriate.
When consistent with organizational objectives, assist employees who  do not want to transfer in finding other jobs, either within an  organization or at another organization.
Document key information to preserve organizational knowledge in the  event that one or more providers change.
Use change management strategies to help client employees deal with  the transition.
Figure 10 demonstrates that the five projects in our review were largely  using the practices.
Table 8 provides information on whether and how the projects  implemented each of the 11 practices in this phase.
section_title: Projects’ Implementation of Phase VI: Manage Provider(s) Performance
paragraphs: The effectiveness with which the performance of the provider(s) is  managed—the focus of this phase—is critical to the successful  implementation of an outsourcing project. Indeed, according to Gartner,  Inc., an outsourcing project can be thwarted by poorly designed, funded,  and delivered processes for managing the delivery of services. This firm  also points out that an enterprise needs to retain the resources to oversee  the planning and implementation of the IT services being delivered by the  provider to ensure that the contractor meets the client’s business needs  throughout the life of the agreement. Moreover, frequent and clear  communication between the client and provider ensures that potential  problems are resolved before they cause disruptions. In addition,  performance reviews should take place regularly to keep the project on  course, measure performance levels, and make adjustments as necessary.
The 11 practices in this phase that we used to evaluate the five projects are  as follows:  Consider incentives to motivate provider(s) to exceed performance  requirements.
Use penalties to motivate provider(s) to meet performance  requirements.
Periodically undertake studies to assess (1) how the provider’s  performance compares with the value being delivered to similar clients  and (2) the extent to which the provider’s performance is improving  over time.
Schedule periodic working-level meetings with both the end-user groups  and the provider to review the provider’s performance.
Conduct executive-level oversight meetings with the provider’s senior  management to review provider’s performance.
Distribute performance data to stakeholders.
Reserve audit rights on performance data supplied by the provider.
Ensure that the provider measures and reports on performance.
Work with the provider to redefine service levels, as appropriate.
Sample performance data frequently enough to perform trend analysis  and to permit extrapolation based on historical data.
Allow employees and possibly stakeholders to rate the provider on a  regular basis.
As shown by figure 11, the five projects in our review generally  implemented the practices.
Table 9 depicts whether and how the five projects in our review  implemented each of the 11 practices in this phase.
section_title: Projects’ Implementation of Phase VII: Ensure Services are Provided
paragraphs: Although outsourcing focuses on the provider’s ability to perform, the  client organization is ultimately responsible for ensuring that services are  provided and that end-user needs are met. The previous phases addressed  the extensive preparation that must precede the provider’s assuming  responsibility for the client organization’s services. This phase  encompasses practices designed to ensure that an effective oversight  approach is established. In addition, over the course of the outsourcing  project, the client learns more about the capabilities of the provider, and  market conditions may change. As a result, it is important to monitor  service levels internally as well as maintain an external view of the  performance of other providers in order to identify opportunities to  improve and ensure that the outsourcing arrangement maintains its value  to the client.
The six practices in this phase that we used to evaluate the five projects are  as follows:  Monitor the provider’s work to anticipate issues for resolution.
Make sure that the provider uses the standard tools and processes  defined as part of the operational model.
Use provider performance data to continuously improve processes.
Pursue improvement based on customer satisfaction surveys.
Ensure that an appropriately empowered individual from the client  organization oversees the work.
Set realistic time frames that are agreed to by the provider.
As illustrated by figure 12, the five projects in our review largely  implemented the practices.
Table 10 provides details on whether and how each project implemented  each of the six practices associated with this phase.
section_title: Comments from the Department of Defense
paragraphs: The following are GAO’s comments on the Department of Defense’s letter  dated April 8, 2003.
subsections: section_title: GAO Comments
paragraphs: 1. Addressed in the Agency Comments and Our Evaluation section of this  report. 2. On January 29, 2003, NIMA granted the provider authorization to use  government sources in performing the IT/IS contract that was limited  to “products, services, and supplies that support the performance of the  specific functional areas and miscellaneous items required under this  contract.” Accordingly, we did not modify this report.  3. Addressed in the Agency Comments and Our Evaluation section of this  report.
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and  Thomas Wright made key contributions to this report.
section_title: GAO’s Mission
paragraphs: The General Accounting Office, the audit, evaluation and investigative arm of  Congress, exists to support Congress in meeting its constitutional responsibilities  and to help improve the performance and accountability of the federal government  for the American people. GAO examines the use of public funds; evaluates federal  programs and policies; and provides analyses, recommendations, and other  assistance to help Congress make informed oversight, policy, and funding  decisions. GAO’s commitment to good government is reflected in its core values of  accountability, integrity, and reliability.
section_title: Obtaining Copies of GAO Reports and Testimony
paragraphs: The fastest and easiest way to obtain copies of GAO documents at no cost is  through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older  products. The Web site features a search engine to help you locate documents  using key words and phrases. You can print these documents in their entirety,  including charts and other graphics.
Each day, GAO issues a list of newly released reports, testimony, and  correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site  daily. The list contains links to the full-text document files. To have GAO e-mail this  list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO  Mailing Lists” under “Order GAO Products” heading.
subsections: section_title: Order by Mail or Phone
section_title: To Report Fraud, Waste, and Abuse in Federal Programs
section_title: Public Affairs