Information Technology: DOD Needs to Leverage Lessons Learned from Its Outsourcing Projects
Released: 2003-04-25T00:00:00

Letter

Background

To protect the security of the United States, DOD relies on a complex array of computer-dependent and mutually supportive organizational components, including the military services, Commanders in Chief, and Defense agencies. As such, it invests tens of billions of dollars each year in a broad array of computer systems, which include weapon systems, command and control systems, satellite systems, inventory management systems, transportation management systems, health systems, financial systems, personnel systems, and payment systems. In addition, DOD spends billions of dollars annually on IT services, which include database management, help-desk operations, software maintenance, and network services. In fiscal year 2001, DOD reportedly obligated more than $6.2 billion on IT services alone.

Decisions regarding the purchasing of services are critical to ensuring the effectiveness of DOD’s operations as well as those of the government as a whole. Our November 2001 report recognizes the importance of such sourcing decisions and provides a framework that spans the full range of activities that are performed during IT services outsourcing. At the same time, governmentwide policies, initiatives, and challenges exist that significantly influence the government’s sourcing decisions.

GAO’s Framework for Outsourcing IT Services

Outsourcing of IT services has become increasingly popular in both the public and private sector. For example, according to the Giga Information Group, Inc., a leading research firm, such outsourcing is expected to grow an average of 5 to 6 percent in 2003. The federal sector’s outsourcing is predicted to rise at an even greater rate. For example, INPUT, an IT market research firm, forecasts that defense IT outsourcing will increase about 143 percent between fiscal years 2002 and 2007.

IT outsourcing involves the activities associated with acquiring services from one or more external providers. During outsourcing, a client organization transfers responsibilities for performing one or more IT services to one or more external providers. This responsibility is executed through control and management of the processes, people, and technology associated with these services.

Figure 2 depicts the roles of the client and provider organizations in an outsourcing relationship.

Our November 2001 guide on leading commercial practices for outsourcing IT services provides a generic framework of practices from leading commercial organizations that can improve purchasing decisions and manage the resulting government/provider relationship. The framework is represented in figure 3 as a hierarchy of phases, practices, and critical success factors.

Table 1 provides a definition of each phase of the framework. Each of the phases has specific practices associated with it. Implementing these practices does not guarantee the success of an outsourcing project. However, our November 2001 study reflected a consensus view that these practices were the most critical to success when IT services are being acquired. Restated, application of these practices increases the probability of a successful outsourcing project.

The organizations that we studied also identified certain capabilities (identified as critical success factors) that were essential for implementing the practices identified in our framework. First, executive leadership strengthens the interaction between executive management and the employees of the client organization. Second, partner alignment strengthens the interaction between the client and provider organization at the executive level, which ensures that the goals and objectives of these organizations support each other. Third, relationship management strengthens the interaction between the client and provider organization at the operational level.

Influences on Government Sourcing Decisions

The federal government is one of the world’s largest users of services. Because of the large dollar value and the number of private- and public- sector jobs involved, deciding whether the public or private sector would be the most appropriate provider of the services the government needs (IT or otherwise) is an important, and often highly charged, question. Among the factors that agencies must consider as they determine how best to meet their missions is whether the public or private sector would be the most appropriate provider of the services the government needs. Phase I of our framework, determine the sourcing strategy, addresses the client’s assessment of whether expertise from within or outside of the organization can more effectively meet the client’s needs.

The National Defense Authorization Act for Fiscal Year 2001 required the Comptroller General of the United States to convene a panel of experts to study the current process used by the government to make sourcing decisions. The resulting Commercial Activities Panel conducted a year-long study and heard repeatedly about the importance of competition and its central role in fostering economy, efficiency, and continuous performance improvement. In particular, the panel reviewed the government’s implementation of the Office of Management and Budget’s (OMB) Circular A-76, which sets forth federal policy for determining whether federal employees or private contractors will perform commercial activities for the government. Circular A-76 (1) outlines conditions under which agencies are permitted to perform a commercial activity with government employees or by contract and (2) provides guidance for whether, and if so, how, agencies should conduct a cost comparison when they are considering transferring the performance of commercial activities from the public to the private sector (or vice versa). The panel reported that there were positive elements to Circular A-76 but that both federal employees and private firms complained that it does not meet the standard of a clear, transparent, and consistently applied process. For example, both federal employees and private firms criticized the Circular A-76 process as unequal and therefore unfair.

The Commercial Activities Panel strongly supported continued emphasis on competition and concluded that whenever the government is considering converting work from one sector to another, public/private competitions should be the norm. In addition, the panel made four recommendations, including that all sourcing decisions be consistent with the principles adopted unanimously by the panel, such as the principle that federal policy provide for accountability in connection with all sourcing decisions.

As part of the administration’s efforts to implement the recommendations of the Commercial Activities Panel, OMB has published proposed changes to Circular A-76. Key highlights of the proposed changes include presuming that all functions are commercial in nature unless they are justified as inherently governmental; limiting the length of time for competitions; and emphasizing awarding contracts on the basis of best value, not just lowest cost. Best value allows the contracting official to consider technical superiority, quality, innovation, and past performance as well as price.

However, we reported that there are several areas in which the proposed revisions to the circular are not consistent with the principles or recommendations of the Commercial Activities Panel. Specifically, the proposed revision does not include a link between sourcing policy and agency missions, has unnecessarily complicated source selection procedures, contains certain unrealistic time frames, and includes insufficient guidance on calculating savings.

Beyond the Commercial Activities Panel, other bodies have identified challenges that the federal government faces in reaching and executing effective sourcing decisions. For example, members of the Coalition for Government Procurement, the Professional Services Council, and the Information Technology Association of America told us that organizational culture is one of the biggest differences between the commercial sector and the federal government and one of the greatest barriers to the government’s use of commercial practices. Also, as we have previously reported, moving to outsourcing solutions can involve a cultural change for government organizations because it may require a change to an agency’s operating model, such as using a contractor to provide IT services previously performed by government staff or using a performance-based contract. This view was echoed by a 2001 study of DOD competitive sourcing that found cultural, process, execution, and training barriers. The study stated that these barriers need to be understood and mitigated before the benefits of outsourcing can be fully realized. Barriers such as these can be overcome by strong executive leadership, which is a critical success factor in our framework.

Another challenge is creating a productive agency/provider relationship— another critical success factor in our framework. According to a report sponsored by the PricewaterhouseCoopers Endowment for the Business of Government, such public/private partnerships are based on trust, commitment to problem or conflict resolution, and the recognition that flexibility is necessary and that the relationship will evolve and change over time. If deadlines are not met, or public agency goals change with differing political climates, the partners need to discuss the basis of the partnership and construct a different relationship. Our prior report on desktop outsourcing found that developing a productive agency/contractor relationship is not always easy. Both sides must recognize and understand each other’s underlying motives and strive to achieve established expectations.

Finally, human capital issues are another challenge facing federal agencies that affect their ability to implement outsourcing. Our framework recognizes the importance of having the right skills in place to support the outsourcing relationship. However, as we have previously reported, procurement reforms and technological changes have placed unprecedented demands on the acquisition workforce. Contracting personnel are now expected to have a much greater knowledge of market conditions, industry trends, and the technical details of the commodities and services they procure. The Commercial Activities Panel report stated that developing and maintaining a skilled acquisition workforce is the critical first step in managing this more complex procurement environment. The panel also reported that DOD bore the brunt of a 22 percent downsizing of the federal acquisition workforce in the last decade, going from 96,000 staff in 1991 to about 68,000 in fiscal year 2001.

Addressing human capital issues is not just a matter of the size of the workforce; it is also a knowledge and skills issue. According to the Commercial Activities Panel, it is critically important that federal agencies adequately address human capital needs in meeting the current and emerging needs of government and its citizens in the most effective, efficient, and economical manner possible. This will require increased emphasis on training and development, particularly in the area of technology.

Description of Five Projects Reviewed

The five projects in our study varied in how they approached outsourcing IT services, such as in using various solicitation methods, including holding a public/private competition under the policies outlined in OMB Circular A- 76 or carrying out a negotiated competitive procurement. In addition, the types of services being outsourced differed: services ranged from the narrowly focused (e.g., help-desk services) to the very broad (e.g., enterprisewide end-to-end information services); contract terms ranged from 5 to 15 years (assuming all option years are exercised); and estimated contract values ranged from $23 million to $8.8 billion. Table 2 provides information on the variety of IT services and outsourcing approaches taken by the projects.

Projects Substantially Used Leading Commercial Practices

As illustrated in figure 4, the five IT services projects substantially used leading commercial practices. Specifically, each project used at least 76 percent of the practices. Reasons for projects implementing different percentages of the practices include differences in their individual circumstances and objectives. For example, the Army’s Total Army Communications—Southwest Asia (TAC-SWA) project, which used the fewest number of practices and had the largest number of practices that were not applicable, was largely a continuation of an existing approach that already relied on the private sector but with fewer providers. In contrast, the National Imagery and Mapping Agency’s (NIMA) Information Technology/ Information Services (IT/IS) project, which fully or partially implemented all of the applicable practices, involved a significant operational shift (e.g., functions previously performed by NIMA staff are now performed by a contractor) and was intended to result in substantial process improvements. In addition, the three projects that implemented the largest percentage of practices also used third-party assistance—including employing a contractor with sourcing expertise—to help formulate their sourcing strategy, which could account for the extent of their compliance.

In addition, figure 5 illustrates that project compliance extended to each of the phases of our framework. Collectively, the projects fully implemented from 70 to 97 percent of the practices in each phase. Phase I, determine sourcing strategy, had the lowest percentage of practices implemented by the projects (70 percent). This result is not inconsistent with a recent Giga Information Group, Inc., survey, which found that only half of the respondent organizations had documented an IT sourcing strategy. This approach carries risk since phase I sets the tone for the outsourcing initiative within the client organization.

In addition, collectively the projects fully implemented 88 percent of the practices (see table 3).

The following provides additional information on the projects’ implementation of each phase of our framework.

Phase I: Determine sourcing strategy. In the first phase of our outsourcing framework, the client organization determines whether internal capability or external expertise can more effectively meet its IT needs. The purpose of a sourcing strategy is to achieve the optimal balance between internal and external capabilities, activities, processes, and services to ensure the achievement of strategic business objectives at the lowest risk. The five projects’ implementation of this phase was uneven. In particular, two of the six practices in this phase were fully implemented by all five projects, but the other four practices were not. Among the practices that were implemented by all of the projects was determining the business reasons for outsourcing. In addition, the three projects that implemented the largest percentage of practices in our framework—the Military Health System/Information Technology Organization (MHS/ITO) Help Desk, the Department of the Navy’s Navy and Marine Corps Intranet (NMCI), and NIMA IT/IS projects—used the third-party assistance practice in this phase to help formulate their sourcing strategy, which could account for the extent of their compliance. For example, the MHS/ITO Help Desk project, which implemented the largest percentage of practices, worked with the Department of the Interior’s GovWorks Program, the Defense Acquisition University staff, and a private-sector contractor to obtain expertise on sourcing strategies.

The practice in this phase that was the most unevenly implemented was the benchmarking and baselining of existing internal services. Of the five projects in our review, (1) one fully benchmarked and baselined the productivity of the activity being outsourced before making the final sourcing decision, (2) two partially baselined their existing activities, and (3) two did not perform benchmark and baseline analyses at all. The agencies’ reasons for not fully implementing this practice included that an executive decision had been made to conduct a public/private competition following the OMB Circular A-76 policy, so such an analysis would not have affected the sourcing decision, or that available documentation to perform such an analysis was limited. Leading research firms suggest benchmarking and baselining the entity’s current processes before outsourcing because only then would it be able to determine whether the arrangement has been successful. In addition, the risk of not fully baselining the existing environment is illustrated by the NMCI project. Specifically, the NMCI project’s baseline of its existing environment was limited because it did not include an assessment of its legacy applications since project officials decided to rely on a preexisting inventory developed to address the Year 2000 challenge. The Navy subsequently found that it had substantially underestimated its number of legacy applications, that, according to program officials, later contributed to the transition period slipping from 2-½ years to 3-½ years. Appendix II provides additional information on projects’ implementation of the practices comprising this phase.

Phase II: Define operational model. The operational model is an important mechanism for an organization to compare its plans with the expectations that were set when the decision to outsource was made and to ascertain whether these plans will enable the organization to meet expectations. The five projects had largely implemented the 13 practices contained in this phase. Specifically, about 90 percent of the practices were implemented. For example, all projects implemented the practice that executive leadership be established to facilitate the outsourcing effort. NIMA, for instance, formed a strategic sourcing office to oversee the IT/IS project. Another practice—training the provider on the organization’s business environment and goals—was fully implemented by one project (in two cases, the practice was not applicable). One project that did not fully implement this practice was the Air Force’s Kirtland Air Force Base’s Command, Control, Communications, and Computer (C4) Services project. Although some training was provided (e.g., Kirtland held an orientation session for potential bidders), provider officials stated that they did not receive adequate training, which made the transition period more difficult. Appendix III provides additional information on projects’ implementation of the practices comprising this phase.

Phase III: Develop the contract. A well-written contract is necessary for the outsourcing organization to meet its requirements while allowing the service provider to make a fair profit. It sets the expectations for service levels, delivery of essential services, and continuous improvement and should protect the interests of all parties. The five projects largely implemented the practices in this phase. Specifically, about 89 percent of the practices were implemented in this phase, and two projects (the MHS/ITO Help Desk and NIMA IT/IS projects) implemented all of the practices. Several practices in this phase address performance requirements. For example, all five projects implemented the practices that called for basing performance requirements on business requirements and reviewing and updating them periodically. One practice that was not fully implemented by two projects was including performance measures that address both technical and end- user satisfaction aspects of performance. For example, the Army TAC- SWA project included technical performance measures in its contract but not measures related to end-user satisfaction, even though the contract included help-desk services. According to the project official that developed the performance work statement in the contract, the command did not include customer satisfaction measures because it did not think that it was necessary to have a performance standard for the help-desk service. However, without such measures, the agency does not have a contractual standard with which to judge the provider’s performance. Appendix IV provides additional information on projects’ implementation of the practices comprising this phase.

Phase IV: Select the provider(s). Critical to the success of any outsourcing project for IT services is identifying potential providers and ultimately selecting a provider(s) that will best meet the needs of the agency. The five projects had largely implemented the seven practices contained in this phase. About 89 percent of the practices were implemented in this phase and two projects (the MHS/ITO Help Desk and Navy NMCI projects) used each applicable practice. For example, the five projects implemented the practice related to conducting due diligence activities to verify provider capabilities before signing the contract. In the case of the Army TAC-SWA project, the Network Enterprise Technology Command (NETCOM) evaluated the provider’s financial and past performance information. The Department of the Navy’s NMCI project also evaluated bidders’ past performance and performed reference checks. Moreover, as part of its due diligence activities, the Navy required all bidders to demonstrate that they had experience in implementing large seat management contracts. A practice that was not implemented by two projects was using third-party assistance when selecting the provider. Projects that did not implement this practice believed that they had adequate in-house expertise with outsourcing, making third-party assistance unnecessary. However, because third-party assistance provides an independent resource that can suggest options or processes that the client organization may not be aware of, these projects may have missed an opportunity to implement their outsourcing projects more effectively. Appendix V provides additional information on projects’ implementation of the practices comprising this phase.

Phase V: Transition to provider(s). This phase focuses on the client organization’s transfer of the IT function to one or more providers. As part of this transition, the clear definition of responsibilities and the careful consideration of employees’ needs matched against the client organization’s needs enable both the client and provider to focus on execution and give staff confidence in their future employment. The five projects largely implemented the 11 practices associated with this phase. Specifically, about 90 percent of the practices were implemented in this phase, and two projects implemented all of the practices (the MHS/ITO Help Desk and NIMA IT/IS projects). Several practices in this phase address dealing with employees affected by the outsourcing projects. For example, in the four projects in which federal employees were affected, the projects provided assistance to those who did not want to transfer to the provider, including helping to place them in other positions and helping with résumé writing. A related practice is to clearly communicate to all employees what is going to happen and when it is going to happen. Two projects did not fully implement this practice. For example, the Navy used its normal chain of command to communicate transition information, but found that implementation of this practice was uneven. As a result, some staff did not know current information about how NMCI would affect them until the provider was ready to contact them regarding their possible transition to the contractor. However, according to the NMCI’s Director’s office, this problem was somewhat mitigated by the provider’s Web site that provides transition information to all NMCI customers/users. Appendix VI provides additional information on projects’ implementation of the practices comprising this phase.

Phase VI: Manage provider(s) performance. The effectiveness with which the provider(s) performance is managed is critical to the successful implementation of an outsourcing project. Indeed, according to Gartner, Inc., a leading research firm, an outsourcing project can be thwarted by poorly designed, funded, and delivered processes for managing the delivery of services. The five projects generally implemented the 11 practices contained in this phase, with about 84 percent of the practices being implemented in this phase. For example, the practices related to obtaining feedback on provider performance were largely implemented. This is important because different levels of an agency can have different perceptions about the value of the outsourcing project. For example, an outsourcing project may be considered successful by the agency’s executive management if it is focused on controlling costs, but be considered inadequate by business managers and users who may be expecting higher levels of service. Each of these viewpoints is valid and should be taken into account when the provider’s performance is evaluated. Two other practices—including incentives and penalties in contracts—were fully implemented by two projects. Incentives and penalties are important because they can help motivate the provider to exceed or meet performance requirements. Nonetheless, two projects did not include monetary incentives and two projects did not include monetary penalties in their contracts. For example, the Army TAC-SWA project did not include monetary incentives, although the contracting officer stated that incentives might have been useful to motivate the provider to exceed performance requirements. Incentives can also help control risks. According to a guide on performance-based services acquisition, if the incentives in the contract are right and if the provider and agency share the same goals, risk is largely controlled and effective performance is “almost the inevitable outcome.” Appendix VII provides additional information on projects’ implementation of the practices comprising this phase.

Phase VII: Ensure services are provided. Although outsourcing transfers responsibility for performing the service to the provider(s), the client organization is ultimately responsible for ensuring that services are provided and that end-user needs are met. Accordingly, it is critical that the agency ensure that services are provided. The projects had implemented 97 percent of these practices, and four projects implemented all of them. For example, every project monitored the providers’ work. In the case of the Air Force’s C4 Services project, quality assurance evaluators monitored the provider’s work to identify problems or trends in accordance with the project’s quality assurance surveillance plan. The results were reported to the contracting officer and to the functional area chief for resolution. Another practice, using customer satisfaction surveys, was fully implemented by four of the five projects. However, the Air Force project did not conduct, or require its contractor to conduct, customer satisfaction surveys. Although the provider surveys staff annually, the Air Force is nevertheless relying on the provider to voluntarily implement an important practice for determining how customers view the services being delivered and whether changes need to be made. Appendix VIII provides additional information on projects’ implementation of the practices comprising this phase.

Leveraging Lessons Learned DOD-wide Could Assist Other DOD Projects

We have previously reported on the importance of collecting and disseminating lessons learned. For example, a critical activity in IT investment management is establishing a process for developing and capturing lessons learned in a written product or knowledge base and disseminating them to decision-makers. In addition, one of the practices in our framework for outsourcing IT services addresses incorporating lessons learned from peers who have engaged in similar sourcing decisions. Use of lessons learned is a principal component of an organizational culture committed to continuous improvement. Sharing such information serves to communicate acquired knowledge more effectively and to ensure that beneficial information is factored into planning, work processes, and activities. Lessons learned can be based on positive experiences or on negative experiences that result in undesirable outcomes.

Although DOD has taken action to gather and disseminate lessons learned and best practices on general acquisition issues, these efforts generally do not focus on outsourcing or include sharing the lessons learned from IT outsourcing projects across the department. Specifically, a number of DOD Web sites provide guidance, lessons learned, and best practices related to general acquisition issues. However, using these sites to locate specific information on IT outsourcing best practices and lessons learned can be time-consuming and difficult because so many topics and information sources are provided. Specifically, MHS/ITO Help Desk project officials said that searching numerous Web sites to get relevant information to address questions and concerns about outsourcing IT services can consume hours. For example, when we entered the keywords “IT outsourcing” and “best practices” into the search feature on the Office of the Undersecretary for Defense of Acquisition, Technology, and Logistics site, ACQWeb (www.acq.osd.mil) in early March, it provided us with links to 1,251 documents. Likewise, lessons learned covers so many topics that it is difficult to search for an applicable IT lesson. For example, when we used the phrase “lessons learned in IT outsourcing” no documents were identified on the ACQWeb, but when we inserted “lessons learned” and “IT outsourcing” links to more than 1,700 documents were produced.

One DOD Web site, Share A-76!, was established to address one of our previous recommendations, that is, to establish a framework for identifying and analyzing best practices and lessons learned from competitive sourcing studies and disseminating them DOD-wide. Share A- 76! promotes the sharing of best practices and lessons learned related to one form of outsourcing that was conducted under the OMB Circular A-76 competitive sourcing process. Among other things, the site contains guidance, links to other relevant sites, sample documents, and a best practices library that communicates field staff experiences and advice about the Circular A-76 process. A NIMA project official said that NIMA’s staff routinely accesses the Web site because it contains a wealth of information on policies, procedures, lessons learned, and links to other outsourcing sites. The DOD analyst responsible for Share A-76! estimated that the site receives about 12,000 visits per month and said that on the basis of E-mails and anecdotes, user satisfaction is favorable. However, this Web site is specific to the OMB Circular A-76 process, which may not apply to other types of outsourcing. For example, Circular A-76’s policy pertains to public/private competitions and requires that the final evaluation between the government and the private sector be based exclusively on cost.

DOD acquisition and IT officials acknowledged that there is no mechanism in DOD to easily share and leverage lessons learned relating to outsourcing IT services. However, these officials agreed that a departmentwide effort to identify, capture, and disseminate lessons learned and leading practices of projects with experience in carrying out IT outsourcing could offer valuable insights and new ideas that would benefit others. Moreover, officials from three of the projects in our review told us that there is value in collecting and disseminating the knowledge acquired from IT outsourcing projects in a systematic manner across the department.

Each of the projects in our review identified knowledge and experience gained from their approaches to outsourcing IT services that could offer insights and practices for other ongoing and future projects to consider. For example: MHS developed specific guidance and lessons learned for implementing a performance-based incentive contract for help-desk operations.

The Department of the Navy’s NMCI project has developed a series of lessons learned related to transitioning to the provider that is being shared within the NMCI community; one example was that all personnel should be available during scheduled testing and deployment.

NIMA has experience in contracting techniques emphasizing a partnering approach with providers to refine requirements and establish a common understanding of costs.

In addition, a departmental IT outsourcing knowledge-sharing approach could include links to information about other government agencies’ IT outsourcing projects. For example, our 2002 report on desktop outsourcing includes an extensive discussion of lessons learned by agencies that have implemented this type of IT services outsourcing.

Developing an effective lessons learned activity is not easy. For example, NMCI officials said that for a lessons learned initiative to be effective, a process must exist that is clearly understood by everyone and allows capturing and sharing of knowledge to occur with minimum effort. Other challenges in developing an effective lessons learned process were outlined by the Share A-76! analyst. The analyst stated that only a small number of site users have contributed lessons learned to the Share A-76! Web site, which she attributed, in part, to the amount of time and effort needed to document and obtain agreement by all levels of the organization on the lessons learned. In addition, the analyst stated that there is reluctance to share negative lessons, and often the review and approval process sanitizes best practice information so that it becomes too general to be most helpful to users. Such challenges can be overcome by executive-level support. Indeed, DOD acquisition and IT officials stated that for lessons learning activities to be effective, senior management must devote support and resources to the effort. This is consistent with our prior work, which showed that knowledge can be effectively shared only when employees are given adequate time as well as established places where they can actually transfer knowledge.

Last year, we outlined a generic lessons learned process that could be used to guide the development of such a process for outsourcing IT services. Although the mechanism or processes used to collect, share, and disseminate lessons learned may vary, in general such a process comprises four main elements: collection, verification, storage, and dissemination. The collection process involves the capture of information through structured and unstructured processes. Verification serves to verify the correctness and applicability of lessons submitted. The storage aspect of lessons learned usually involves incorporating the lessons into an electronic database for the dissemination and sharing of information, including the ability to conduct information searches. The final element, and the most important, is the dissemination of lessons learned, since lessons are of little benefit unless they are distributed and used by people who will benefit from them. Lessons can be “pushed,” or automatically delivered to a user, or “pulled” in situations where a user must manually search for them. Lessons can also be disseminated with an assigned priority descriptor, which denotes the risk, immediacy, and urgency of the lessons learned content.

Conclusions

The projects in our review substantially implemented leading commercial practices for outsourcing IT services, which has increased each project’s probability of success. Capturing how these projects operationalized leading commercial practices could help other IT services outsourcing projects succeed. Although currently there is no such DOD-wide mechanism, such as an electronic tool, to easily share and leverage lessons learned, DOD IT and acquisition officials agreed that a departmentwide effort to identify, capture, and disseminate lessons learned could offer valuable insights and new ideas that would benefit others. Lessons learned that are pragmatic and easily accessible could give DOD managers a more informed understanding of the important issues to be addressed when making outsourcing decisions, as well as the factors to be considered to help ensure the success of these endeavors. DOD managers can also benefit from lessons learned on the basis of negative experiences. The projects in our review were well into implementation, and therefore, at this late stage, we see little advantage for them to revisit practices that were not implemented. Nevertheless, an electronic tool for capturing and disseminating lessons learned would allow the rest of DOD to benefit from the negative consequences and increased risks associated with those practices that the projects did not implement.

Developing a lessons learned mechanism is not easy; thus, senior management support and resources are keys to success. Without such support driving the capture and dissemination of lessons learned, DOD is losing an opportunity for wider application of leading practices and thus better ensuring that its IT outsourcing efforts are successful.

Recommendations

To assist DOD organizations in planning and implementing outsourcing projects for IT services, we recommend that the Secretary of Defense direct the Undersecretary of Defense for Acquisition, Technology, and Logistics, working in conjunction with the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I), to provide senior management support and adequate resources to develop and implement an electronic tool to capture and disseminate examples and lessons learned from actual IT outsourcing projects. These examples and lessons learned, at a minimum, should include the results of our review of the five projects discussed in this report.

We also recommend that the Secretary of Defense direct the Undersecretary of Defense for Acquisition, Technology, and Logistics, working in conjunction with the Assistant Secretary of Defense for Command, Control, Communications, and Intelligence (C3I), to ensure that the method used to gather information for this electronic tool incorporate the main elements of a lessons learned process—namely, collection, verification, storage, and dissemination.

Agency Comments and Our Evaluation

In written comments on a draft of our report, signed by DOD’s Principal Director, Deputy Assistant Secretary of Defense (Deputy Chief Information Officer), the department partially concurred with our recommendations. Specifically, DOD agreed that capturing lessons learned in the development and implementation of its IT outsourcing initiatives is important to continually improve its outsourcing methods and results achieved. The department also stated that before deciding on a specific method to achieve this aim, it intends to explore a variety of mechanisms that could be used. In particular, the department stated that it currently has several processes and communities of interest that collect and disseminate lessons learned in other areas, which are logical starting points for determining the best path forward. DOD’s written comments are reproduced in appendix IX.

We agree that it is prudent to explore various alternatives to leveraging lessons learned from DOD’s IT services outsourcing experiences. Our recommendations are not prescriptive as to the electronic method to be used to capture and disseminate lessons learned. Therefore, the department’s plan to explore various alternatives is consistent with our recommendations.

We are sending copies of this report to the appropriate congressional committees, the Secretary of Defense, and the Director of the Office of Management and Budget. We will also provide copies to others upon request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions on matters discussed in this report, please contact me at (202) 512-3439 or Linda J. Lambert, Assistant Director, at (202) 512-9556. We can also be reached by E-mail at hiter@gao.gov and lambertl@gao.gov, respectively. Other contacts and key contributors to this report are listed in appendix X.

Objectives, Scope, and Methodology

Our objectives were to determine (1) the extent to which selected Department of Defense (DOD) information technology (IT) services outsourcing projects use leading commercial practices as specified in our framework and (2) whether DOD is sharing lessons learned from its IT outsourcing projects across the department.

To determine the extent to which selected DOD outsourcing projects for IT services use leading commercial practices, we identified the practices in our November 2001 report on leading commercial practices that (1) are typically applied at the project level and (2) were verifiable through documentation and interviews. Because DOD did not centrally maintain a list of outsourcing projects for IT services, we asked the department to identify candidate projects for our evaluation. From this list, we selected the following five projects for our review: (1) Air Force Kirtland Air Force Base’s Command, Control, Communications, and Computer (C4) Services project; (2) Army Network Enterprise Technology Command’s (NETCOM) Total Army Communications – Southwest Asia (TAC-SWA) project; (3) Military Health System/Information Technology Organization (MHS/ITO) Help Desk project; (4) Department of the Navy’s Navy and Marine Corps Intranet (NMCI) project; and (5) National Imagery and Mapping Agency (NIMA) Information Technology/Information Services (IT/IS) project. We chose each project on the basis of the following criteria: (1) no more than one project from each military service and two agencies, (2) illustrative example of DOD IT outsourcing, (3) dollar value greater than $10 million, and (4) enough time elapsed for services to have been delivered and performance measured.

At our request, each project completed a self-assessment on whether and how it implemented leading commercial practices. We reviewed the agency self-assessments and accompanying documentation and interviewed the appropriate agency project officials to verify whether the practices were followed. In addition, we interviewed representatives from each of the providers associated with these projects.

We also researched additional information on commercial practices in our November 2001 guide and the challenges the federal government faces in implementing them. Specifically, we performed a literature search, which included reviewing reports issued by leading research firms, such as Gartner, Inc., and Giga Information Group, Inc. In addition, we interviewed representatives from industry organizations that have an interest in outsourcing IT services, including the Coalition for Government Procurement; the Information Technology Association of America; the Professional Services Council; and Acquisition Solutions, Inc.

To determine whether DOD is sharing lessons learned from its IT outsourcing projects across the department, we identified and reviewed various approaches that DOD currently uses to capture and disseminate such information. This included identifying and reviewing various Web sites and performing key word searches on these sites to identify lessons learned for outsourcing IT services. We also interviewed applicable DOD acquisition and IT officials. Finally, we identified and reviewed a generic lessons learned process contained in our January 2002 report on the National Aeronautics and Space Administration’s lessons learned mechanisms.

We performed our work at the Army’s NETCOM in Ft. Huachuca, Arizona; the MHS/ITO Help Desk project office in Falls Church, Virginia; the NMCI Director’s office in Crystal City, Virginia; and NIMA’s headquarters in Bethesda, Maryland. We conducted our review between May 2002 and early March 2003 in accordance with generally accepted government auditing standards.

Projects’ Implementation of Phase I: Determine Sourcing Strategy

In the first phase of our outsourcing framework, the client organization determines whether internal capability or external expertise can more effectively meet its IT needs. The purpose of this sourcing strategy is to achieve the optimal balance between internal and external capabilities, activities, processes, and services to ensure that strategic business objectives are achieved at the lowest risk. Among the factors that an organization should evaluate in crafting this strategy are technology, business, financial, and personnel requirements and whether it has skilled business and IT managers. In addition, according to Gartner, Inc., sound sourcing decisions depend on whether IT organizations (1) know and understand their business priorities, (2) are prepared to invest in some skills and divest others, and (3) identify and assess trade-offs.

The six practices in this phase that we used to evaluate the five projects are as follows: Use third-party assistance with experience in a variety of sourcing arrangements when formulating a sourcing strategy.

Incorporate lessons learned from peers who have engaged in similar sourcing decisions.

Estimate impact of sourcing decision on internal organization.

Benchmark and baseline productivity of internal services before making the final sourcing decision.

Determine the business reasons for outsourcing IT.

Determine reasons for outsourcing IT that can improve the organization’s ability to use and manage technology.

Figure 6 shows that the implementation of the practices by the five projects in our review was uneven.

Table 4 provides detailed information on whether and how each project implemented each of the six practices in this phase.

Project’s Implementation of Phase II: Define Operational Model

Critical to the successful outsourcing relationship is an operational model for guiding the structure of the contract and the plans for transition. In defining the operational model, client organizations formalize executive leadership, team composition, client responsibilities, and operating relationships between the client and provider. The operational model helps the organization to compare its plans with the expectations that were set as the initial decision to outsource was made and to ascertain whether these plans will enable the organization to meet those objectives. An important aspect of the operational model is an explicit understanding of how the client organization plans to communicate its needs and provide feedback to the provider. In addition, communication between the business and IT offices within the client organization is always critical. This is particularly true in the case of outsourcing because the IT service provider is outside the client organization and disconnects are more likely to occur. Therefore, organizational processes to facilitate good communication are critical.

The 13 practices in this phase that we used to evaluate the five projects are as follows: Establish executive leadership for IT to facilitate the outsourcing initiative.

Continually communicate/clarify outsourcing objectives, while correcting misinformation that affects the organization.

Establish a core group of people who will be involved in all phases of outsourcing.

Select a person involved in the negotiation of the contract to manage the outsourcing relationship.

Create and define a contract management structure with operational points of contact and managers.

Define the role of internal IT managers and business leaders.

Ensure that the right skills are in place to support the outsourcing relationship.

Establish a point of contact high in the provider management structure for elevating provider performance concerns.

Have provider establish an on-site support team to serve as liaison between client and provider.

Train provider on client business environment and goals.

Select or develop standard tools for managing the relationship.

Use third-party assistance to take advantage of expertise from a variety of outsourcing arrangements in defining the operational model.

Ensure that the provider management team has prior experience in the client’s field of business.

Figure 7 illustrates that the five projects in our review largely implemented the practices.

Table 5 provides detailed information on whether and how each project implemented each of the 13 practices in this phase.

Project’s Implementation of Phase III: Develop the Contract

Phase III focuses on the development of the contract, which defines the legal terms of the relationship between client and provider. While other phases in the outsourcing process describe the need for mutual trust and a close, flexible working relationship, this phase focuses on the development of the contract, which is the foundation on which a working relationship will be built. A well-written contract helps the outsourcing organization meet its requirements while allowing the service provider to make a fair profit. It sets the expectations for service levels, delivery of essential services, and continuous improvement and is intended to protect the interests of all parties.

The 16 practices in this phase that we used to evaluate the five projects are as follows: Base performance requirements on business outcomes.

Include measures that reflect end-user satisfaction as well as technical IT performance.

Review and update performance requirements periodically.

Require the provider to meet the minimum performance in each category of service.

Require the provider to achieve escalating performance standards at agreed-upon intervals.

Incorporate sufficient flexibility so that minimum acceptable performance can be adjusted as conditions change, as the provider becomes more adept at satisfying customer demands, and as improvement goals are achieved.

Use service-level agreements (SLA) to clearly articulate all aspects of performance, including management, processes, and requirements.

Client and provider work together to define the appropriate number of SLAs and appropriate structure for each.

Specify circumstances under which the provider is excused from performance levels mandated by master service agreements.

Client and provider work together to identify SLAs for which compensation is based, while additional SLAs may be defined to manage performance.

The contract should include clauses for (1) determining pricing structures; (2) performing customer satisfaction surveys and using the results to redefine performance levels; (3) terminating the contract; (4) resolving disputes in a timely manner; (5) taking work away, without penalty, from provider for nonperformance; (6) declaring a significant event that can lead to a change in the contract; (7) defining performance requirements; and (8) conducting regularly scheduled meetings.

Consider setting up a master services agreement under which all arrangements between client and provider operate.

Include the appropriate representation from each major organizational unit on the contract negotiation team.

Specify the use of volume purchases to obtain optimal discounts.

Use third-party assistance in negotiating and developing the contract.

Sign the contract after contract negotiations and final vendor selections.

As figure 8 shows, the five projects largely implemented the practices.

Table 6 provides detailed information on whether and how each project implemented each of the 16 practices in this phase.

Projects’ Implementation of Phase IV: Select the Provider(s)

Critical to the success of any project to outsource IT services is the identification of potential providers and the ultimate selection of a provider(s) that will best meet the needs of the organization. Developing a strategy that will lead to the selection of the “right contractor” is especially important in a performance-based acquisition. The overall success of the outsourcing project requires the contractor to understand the performance-based approach, know or develop an understanding of the organization’s requirement, have a history of performing exceptionally in the field, and have the processes and resources in place to support the mission.

The seven practices in this phase that we used to evaluate the five projects are as follows: Conduct research on the state of the market, vendors, and technology before defining vendor selection criteria.

Identify and evaluate various sourcing solutions (e.g., single vendor, multivendor, and alliance).

Define a process for selecting vendors to be providers.

Define vendor selection and evaluation (acceptance) criteria at the outset.

When issuing an RFP, identify services with expected performance levels and define client and provider roles and responsibilities.

Use third-party assistance with expertise in a variety of outsourcing arrangements when selecting provider(s), including developing the RFP.

Conduct due diligence activities to verify vendor capabilities before signing the contract.

As figure 9 illustrates, all five projects largely implemented the practices.

Table 7 provides detailed information on whether and how the projects in our review implemented each of the seven practices in this phase.

Projects’ Implementation of Phase V: Transition to Provider(s)

In this phase, client organizations transfer responsibility of IT functions to one or more providers. A clear definition of responsibilities among the various parties and the careful consideration of employees’ needs matched against the organization’s needs enable both the client and provider to focus on execution and give staff confidence in their future employment. If the contractor is assuming responsibility for functions previously performed by federal employees, it is especially important that the organization communicate a clear transition process. Without such communication, an outsourcing project can be negatively affected if misinformation and mistrust ensues.

The 11 practices in this phase that we used to evaluate the five projects are as follows: Communicate a clear transition process to all key players from both client and provider organizations.

Clearly communicate to employees what is going to happen and when it is going to happen.

Establish a client transition team with representatives from across the organization to facilitate the transition.

Place the transition under a single program manager.

Create client/provider transition teams to address short-term transition tasks as required.

Recognize that it takes time to effect transition and plan accordingly.

Encourage the transition of staff to the provider, where appropriate, using bonuses, stock options, and other appropriate methods.

Develop employee-retention programs and offer bonuses to keep key people, where appropriate.

When consistent with organizational objectives, assist employees who do not want to transfer in finding other jobs, either within an organization or at another organization.

Document key information to preserve organizational knowledge in the event that one or more providers change.

Use change management strategies to help client employees deal with the transition.

Figure 10 demonstrates that the five projects in our review were largely using the practices.

Table 8 provides information on whether and how the projects implemented each of the 11 practices in this phase.

Projects’ Implementation of Phase VI: Manage Provider(s) Performance

The effectiveness with which the performance of the provider(s) is managed—the focus of this phase—is critical to the successful implementation of an outsourcing project. Indeed, according to Gartner, Inc., an outsourcing project can be thwarted by poorly designed, funded, and delivered processes for managing the delivery of services. This firm also points out that an enterprise needs to retain the resources to oversee the planning and implementation of the IT services being delivered by the provider to ensure that the contractor meets the client’s business needs throughout the life of the agreement. Moreover, frequent and clear communication between the client and provider ensures that potential problems are resolved before they cause disruptions. In addition, performance reviews should take place regularly to keep the project on course, measure performance levels, and make adjustments as necessary.

The 11 practices in this phase that we used to evaluate the five projects are as follows: Consider incentives to motivate provider(s) to exceed performance requirements.

Use penalties to motivate provider(s) to meet performance requirements.

Periodically undertake studies to assess (1) how the provider’s performance compares with the value being delivered to similar clients and (2) the extent to which the provider’s performance is improving over time.

Schedule periodic working-level meetings with both the end-user groups and the provider to review the provider’s performance.

Conduct executive-level oversight meetings with the provider’s senior management to review provider’s performance.

Distribute performance data to stakeholders.

Reserve audit rights on performance data supplied by the provider.

Ensure that the provider measures and reports on performance.

Work with the provider to redefine service levels, as appropriate.

Sample performance data frequently enough to perform trend analysis and to permit extrapolation based on historical data.

Allow employees and possibly stakeholders to rate the provider on a regular basis.

As shown by figure 11, the five projects in our review generally implemented the practices.

Table 9 depicts whether and how the five projects in our review implemented each of the 11 practices in this phase.

Projects’ Implementation of Phase VII: Ensure Services are Provided

Although outsourcing focuses on the provider’s ability to perform, the client organization is ultimately responsible for ensuring that services are provided and that end-user needs are met. The previous phases addressed the extensive preparation that must precede the provider’s assuming responsibility for the client organization’s services. This phase encompasses practices designed to ensure that an effective oversight approach is established. In addition, over the course of the outsourcing project, the client learns more about the capabilities of the provider, and market conditions may change. As a result, it is important to monitor service levels internally as well as maintain an external view of the performance of other providers in order to identify opportunities to improve and ensure that the outsourcing arrangement maintains its value to the client.

The six practices in this phase that we used to evaluate the five projects are as follows: Monitor the provider’s work to anticipate issues for resolution.

Make sure that the provider uses the standard tools and processes defined as part of the operational model.

Use provider performance data to continuously improve processes.

Pursue improvement based on customer satisfaction surveys.

Ensure that an appropriately empowered individual from the client organization oversees the work.

Set realistic time frames that are agreed to by the provider.

As illustrated by figure 12, the five projects in our review largely implemented the practices.

Table 10 provides details on whether and how each project implemented each of the six practices associated with this phase.

Comments from the Department of Defense

The following are GAO’s comments on the Department of Defense’s letter dated April 8, 2003.

GAO Comments

1. Addressed in the Agency Comments and Our Evaluation section of this report. 2. On January 29, 2003, NIMA granted the provider authorization to use government sources in performing the IT/IS contract that was limited to “products, services, and supplies that support the performance of the specific functional areas and miscellaneous items required under this contract.” Accordingly, we did not modify this report. 3. Addressed in the Agency Comments and Our Evaluation section of this report.

GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

Season Dietrich, James Houtz, Anjalique Lawrence, Patricia Slocum, and Thomas Wright made key contributions to this report.

GAO’s Mission

The General Accounting Office, the audit, evaluation and investigative arm of Congress, exists to support Congress in meeting its constitutional responsibilities and to help improve the performance and accountability of the federal government for the American people. GAO examines the use of public funds; evaluates federal programs and policies; and provides analyses, recommendations, and other assistance to help Congress make informed oversight, policy, and funding decisions. GAO’s commitment to good government is reflected in its core values of accountability, integrity, and reliability.

Obtaining Copies of GAO Reports and Testimony

The fastest and easiest way to obtain copies of GAO documents at no cost is through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and full- text files of current reports and testimony and an expanding archive of older products. The Web site features a search engine to help you locate documents using key words and phrases. You can print these documents in their entirety, including charts and other graphics.

Each day, GAO issues a list of newly released reports, testimony, and correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site daily. The list contains links to the full-text document files. To have GAO e-mail this list to you every afternoon, go to www.gao.gov and select “Subscribe to GAO Mailing Lists” under “Order GAO Products” heading.

Order by Mail or Phone

To Report Fraud, Waste, and Abuse in Federal Programs

Public Affairs