Information Technology: Critical Factors Underlying Successful Major Acquisitions
Released: 2011-11-21T00:00:00

Letter

Background

Investments in IT can enrich people’s lives and improve organizational performance. For example, during the last two decades the Internet has matured from being a means for academics and scientists to communicate with each other to a national resource where citizens can interact with their government in many ways, such as by receiving services, supplying and obtaining information, asking questions, and providing comments on proposed rules.

While these investments have the potential to improve lives and organizations, federally funded IT projects can—and have—become risky, costly, unproductive mistakes. As we have described in numerous reports and testimonies, although a variety of best practice documentation exists to guide their successful acquisition, federal IT projects too frequently incur cost overruns and schedule slippages while contributing little to mission-related outcomes.

IT Acquisition Best Practices Have Been Identified by Industry and Government and Promoted by Legislation

IT acquisition best practices have been developed by both industry and the federal government. For example, the Software Engineering Institute (SEI) has developed highly regarded and widely used guidance on best practices such as requirements development and management, risk management, configuration management, validation and verification, and project monitoring and control. In the federal government, GAO’s own research in IT management best practices led to the development of the Information Technology Investment Management (ITIM) Framework, which describes essential and complementary IT investment management disciplines, such as oversight of system development and acquisition management, and organizes them into a set of critical processes for successful investments.

Congress has also enacted legislation that reflects IT management best practices. For example, the Clinger-Cohen Act of 1996, which was informed by GAO best practice recommendations, requires federal agencies to focus more on the results they have achieved through IT investments, while concurrently improving their IT acquisition processes. Specifically, the act requires agency heads to implement a process to maximize the value of the agency’s IT investments and assess, manage, and evaluate the risks of its IT acquisitions. Further, the act establishes chief information officers (CIO) to advise and assist agency heads in carrying out these responsibilities. The act also requires OMB to encourage agencies to develop and use best practices in IT acquisition.

Additionally, the E-Government Act of 2002 established a CIO Council, which is led by the Federal CIO, to be the principal interagency forum for improving agency practices related to the development, acquisition, and management of information resources, including sharing best practices. Consistent with this mandate, the CIO Council established a Management Best Practices Committee in order to serve as a focal point for promoting IT best practices within the federal government.

Prior GAO Work Has Identified IT Acquisition Management Weaknesses, Cost Increases, and Schedule Delays on Troubled Investments

We have often reported on a range of acquisition management weaknesses facing federal IT investments—including problems relating to senior leadership, requirements management, and testing. For example, for the investments described below, we have identified acquisition weaknesses, and have reported on significant cost increases and schedule delays. Additionally, each of these investments was ultimately cancelled or significantly restructured as a result of agency reviews conducted in response to acquisition weaknesses, cost increases, and schedule delays.

In June 2009, we reported that an executive committee for the National Polar-orbiting Operational Environmental Satellite System (NPOESS)—a program jointly managed by the Department of Commerce’s National Oceanic and Atmospheric Administration, the Department of Defense, and the National Aeronautics and Space Administration—lacked the membership and leadership needed to effectively and efficiently oversee and direct the program. Specifically, the Defense committee member with acquisition authority did not attend committee meetings and sometimes contradicted the committee’s decisions. Further, the committee did not track its action items to closure, and many of the committee’s decisions did not achieve desired outcomes. To address these issues, we recommended that the Secretary of Defense direct the key committee member to attend and participate in committee meetings. Additionally, we recommended that the heads of the agencies that participate in the committee direct the committee members to track action items to closure, and identify the desired outcomes associated with each of the committee’s actions.

Further, we reported that the launch date for an NPOESS demonstration satellite had been delayed by over 5 years and the cost estimate for the program had more than doubled—from $6.5 billion to about $15 billion. In February 2010, a presidential task force decided to disband NPOESS and, instead, have the agencies undertake separate acquisitions.  Since 2007, we have reported on a range of acquisition weaknesses facing the Department of Homeland Security’s (DHS) Secure Border Initiative Network—also known as SBInet. For example, in January 2010, we reported that DHS had not effectively managed key aspects of the SBInet testing program such as defining test plans and procedures in accordance with important elements of relevant guidance. In light of these weaknesses, we made recommendations to DHS related to the content, review, and approval of test planning documentation.

In May 2010, we reported that the final acceptance of the first two SBInet deployments had slipped from November 2009 and March 2010 to September 2010 and November 2010, respectively, and that the cost-effectiveness of the system had not been justified. We concluded that DHS had not yet demonstrated that the considerable time and money being invested to acquire and develop SBInet was a wise and prudent use of limited resources. The Secretary of Homeland Security ordered a departmentwide assessment of the SBInet program; the Secretary’s decision was motivated in part by continuing delays in the development and deployment of SBInet capabilities and concerns that the SBInet system had not been adequately justified by a quantitative assessment of cost and benefits. Based on the results of the assessment, in January 2011, the DHS Secretary decided to end SBInet as originally conceived.

In May 2010, we reported that after spending $127 million over 9 years on an outpatient scheduling system, the Department of Veterans Affairs (VA) has not implemented any of the planned system’s capabilities and is essentially starting over. After determining that the system could not be deployed, the department terminated the contract and ended the program in September 2009. We concluded that the department’s efforts to successfully complete the system had been hindered by weaknesses in several key project management disciplines and a lack of effective oversight that, if not addressed, could undermine the department’s second effort to replace the scheduling system. We recommended that the department take action to improve key processes, including acquisition management, requirements management, system testing, implementation of earned value management, risk management, and program oversight.

In June 2011, we reported that end users were not sufficiently involved in defining requirements for the Federal Emergency Management Agency’s (FEMA) National Flood Insurance Program’s insurance policy and claims management system. After conducting an assessment of the program prompted by problems identified in end user testing, FEMA leadership cancelled the system because it failed to meet end user expectations. This decision forced the agency to continue to rely on an outdated system that is neither effective nor efficient. In order to avoid the root causes of this program’s failure, we recommended that for future related modernization attempts, DHS should ensure that key stakeholders are adequately involved in requirements development and management.

Additionally, we have previously reported on investments in need of management attention across the federal government. For example, in April 2011, we reported on the visibility into federal IT investments provided by the IT Dashboard—a publicly available website that displays detailed information on federal agencies’ major IT investments, including assessments of actual performance against cost and schedule targets (referred to as ratings) for approximately 800 major federal IT investments. Specifically, we reported that, as of March 2011, the Dashboard provided visibility into over 300 IT investments, totaling almost $20 billion, in need of management attention. We noted that  272 investments with costs totaling $17.7 billion had ratings that indicated the need for attention, and  39 investments with costs totaling $2.0 billion had ratings that indicated significant concerns.

OMB Has Several Initiatives Under Way to Improve the Oversight and Management of IT Investments

OMB plays a key role in helping federal agencies manage their investments by working with them to better plan, justify, and determine how much they need to spend on projects and how to manage approved projects.

In June 2009, OMB established the IT Dashboard to improve the transparency into and oversight of agencies’ IT investments. According to OMB officials, agency CIOs are required to update each major investment in the IT Dashboard with a rating based on the CIO’s evaluation of certain aspects of the investment, such as risk management, requirements management, contractor oversight, and human capital. According to OMB, these data are intended to provide a near real-time perspective of the performance of these investments, as well as a historical perspective. Further, the public display of these data is intended to allow OMB, congressional and other oversight bodies, and the general public to hold government agencies accountable for results and progress.

In January 2010, the Federal CIO began leading TechStat sessions— reviews of selected IT investments between OMB and agency leadership—to increase accountability and transparency and improve performance. OMB has identified factors that may result in an investment being selected for a TechStat session, such as—but not limited to— evidence of (1) poor performance; (2) duplication with other systems or projects; (3) unmitigated risks; and (4) misalignment with policies and best practices.

OMB officials stated that as of June 30, 2011, 63 TechStat sessions had been held with federal agencies. According to OMB, these sessions enabled the government to improve or terminate IT investments that were experiencing performance problems. For example, in June 2010, the Federal CIO led a TechStat on the National Archives and Records Administration’s (NARA) Electronic Records Archives investment that resulted in six corrective actions, including halting fiscal year 2012 development funding pending the completion of a strategic plan. Similarly, in January 2011, we reported that NARA had not been positioned to identify potential cost and schedule problems early, and had not been able to take timely actions to correct problems, delays, and cost increases on this system acquisition program. Moreover, we estimated that the program would likely overrun costs by between $205 and $405 million if the agency completed the program as originally designed. We made multiple recommendations to the Archivist of the United States, including establishing a comprehensive plan for all remaining work, improving the accuracy of key performance reports, and engaging executive leadership in correcting negative performance trends.

Drawing on the visibility into federal IT investments provided by the IT Dashboard and TechStat sessions, in December 2010, OMB issued a plan to reform IT management throughout the federal government over an 18-month time frame. The plan contains two high-level objectives:  achieving operational efficiency, and  effectively managing large-scale IT programs.

To achieve these high-level objectives, the plan outlines 25 action items. According to OMB officials, they have taken several actions pursuant to this plan. For example, pursuant to Action Item Number 10—development of an IT best practices collaboration platform—in April 2011 the CIO Council launched an IT best practices collaboration website. According to OMB, this portal provides federal program managers with access to a searchable database of program management best practices in order to promote interagency collaboration and real-time problem solving related to IT programs. The portal contains links to case studies by federal agencies demonstrating the use of best practices in managing large-scale IT systems. For example, a recent case study posted by the Social Security Administration outlined efforts to develop a cadre of highly skilled, trained, and qualified program managers to promote the success of its investments.

Seven IT Investments Were Reported as Being Successfully Acquired

According to federal department officials, the following seven investments best achieved their respective cost, schedule, scope, and performance goals.

The estimated total life-cycle cost of the seven investments is about $5 billion. Six of the seven investments are currently operational. The following provides descriptions of each of the seven investments.

Commerce Decennial Response Integration System

The U.S. Census Bureau is the primary source of basic statistics about the population and economy of the nation and is best known for the decennial census of population and housing. The most recent decennial census was conducted in 2010. Between March and August 2010, the Census Bureau provided assistance to respondents and captured their response data via paper and telephone agent to allow sufficient time for post-capture processing, review, and tabulation. The Decennial Response Integration System (DRIS) provided a system for collecting and integrating census responses from forms and telephone interviews. Specifically, DRIS integrated the following three primary functions:  Paper data capture: Processed paper census questionnaires sent by mail from respondents. The system sorted the questionnaires and captured data from them, which were turned into electronic data.  Telephone questionnaire assistance: Provided respondents with assistance in understanding their questionnaire, and captured responses for forms completed over the phone. This function utilized interactive voice response as the initial contact mechanism with an option to speak with call center representatives if needed.  Coverage follow up: Contacted a sample of respondents by telephone to determine if changes should be made to their household roster as reported on their initial census return with the goal of ensuring that every person in the United States is counted once and in the right place.

To help carry out the 2010 Decennial Census, the government engaged a contractor to design, build, test, deploy, implement, operate, and maintain the systems, infrastructure, staffing, procedures, and facilities needed for DRIS. The DRIS contract was divided into three primary phases. Phase 1 included the development, testing, deployment, implementation, and support of the DRIS components needed for a 2008 Census Dress Rehearsal. Phase 2 included the nationwide deployment of the DRIS components and full-scale production operations of the paper data capture, telephone questionnaire assistance, and coverage follow up functions for the 2010 Census. Phase 3 is to address post-2010 Census DRIS component disposition and data archiving. Phase 3 was to be completed in September 2011. For purposes of our report, we focused only on the first two phases of DRIS because the DRIS system was being acquired during these phases.

In October 2009, we reported that DRIS fully implemented the key practices necessary for a sound implementation of earned value management—a project management approach that, if implemented appropriately, provides objective reports of project status, produces early warning signs of impending schedule delays and cost overruns, and provides unbiased estimates of anticipated costs at completion. Additionally, we reported that, as of May 2009, the DRIS contractor was experiencing a cumulative cost underrun and was ahead of schedule; however, the life-cycle cost estimate for DRIS had increased from $574 million to $946 million. This cost increase was mostly due to increases in both paper and telephone workloads. For example, the paper workload increased due to an April 2008 redesign of the 2010 Census that reverted planned automated operations to paper-based processes and required DRIS to process an additional estimated 40 million paper forms.

Defense Global Combat Support System-Joint Increment 7

Number of users: 20-30 joint warfighter logistician users; 13,000 single-sign-on users Operations start date: March 2009 (deployment of initial operational capability for Increment 7)

The Global Combat Support System-Joint (GCSS-J) Increment 7 is a system that supports military logistics operations that provide military personnel with the supplies and information they need to accomplish their missions. GCSS- J combines data, such as the location and quantity of a particular resource, from multiple authoritative data sources (e.g., Asset Visibility, Joint Operation Planning and Execution System, and Global Decision Support System) and analyzes the data to provide information needed by logistics decision makers. The end users of the system are the logisticians at the various Combatant Commands, which are made up of representatives from multiple branches, each having a geographical or functional responsibility. According to Defense Information Systems Agency (DISA) officials, the analyses generated by the system enable the commanders of the Combatant Commands to rapidly make critical decisions, and to plan, execute, and control logistics operations. Additionally, the system provides other end users with single sign-on access to the individual data sources. The diverse end user group, combined with a wide spectrum of data, provides a unified supply chain for the Army, Navy, Air Force, and Marine forces in a given area, which is to help eliminate inefficiencies and provide a more useful view into the supply chain.

DISA started GCSS-J in 1997 as a prototype. The system is being developed incrementally using Agile software development— specifically, the Scrum methodology. DISA is currently developing and deploying major releases for Increment 7. A total of five major releases were planned within Increment 7; Releases 7.0 and 7.1, which were implemented in March 2009 and December 2009 respectively, were the subject of our review.

To date, according to DISA, Increment 7 releases have improved performance and provided new capabilities and enhancements to existing capabilities. For example, the system provides real-time information about road conditions, construction, incidents, and weather to facilitate rapid deployment of military assets.

Energy Manufacturing Operations Management Project

The Manufacturing Operations Management (MOMentum) Project aims to replace a suite of aging mission-essential shop floor, manufacturing control systems at the Y-12 National Security Complex that support the National Nuclear Security Administration’s (NNSA) Stockpile Stewardship and Management Program. The shop floor at the Y-12 complex is responsible for the construction, restoration, and dismantling of nuclear weapon components. The core software currently used in the shop floor manufacturing control systems was deployed in the mid-1980s and will no longer be supported by the vendor on its current hardware platform beginning in 2012.

The MOMentum Project has two phases. Phase 1, which was the subject of our review, was implemented in September 2010, and is a deployment of the Production Planning module of SAP for manufacturing planning and scheduling. Phase 2 is to include the deployment of the Manufacturing Execution module of SAP software and support the execution of production schedules on the shop floor. Phase 2 is scheduled to be completed in September 2013. The implementation of the system is expected to save $6 million annually, reduce cycle times for manufacturing, remove dependencies on obsolete technology and unsupported software, and reduce administrative errors and product deviations, among other things.

Homeland Security Western Hemisphere Travel Initiative

Operations start date: September 2008 (initial operating capability), June 2009 (full operating capability)

To facilitate inspections at the nation’s 330 air, sea, and land ports of entry, the Western Hemisphere Travel Initiative (WHTI) requires all citizens of the United States and citizens of Canada, Mexico, and Bermuda traveling to the United States as nonimmigrant visitors to have a passport or other accepted document that establishes the bearer’s identity and citizenship to enter the country from within the Western Hemisphere. In order to implement WHTI at the land border while limiting its impact on the public, U.S. Customs and Border Protection (CBP) engaged a contractor to procure and deploy technology—including Radio Frequency Identification, License Plate Reader, and Vehicle Primary Client technologies. These technologies help to provide CBP officers with law enforcement and border crossing history information for each traveler and vehicle. Initial operating capability was achieved in September 2008 when these technologies were deployed to two ports of entry. Full operating capability was achieved in June 2009 when the WHTI technology was deployed to 37 additional ports of entry. The 39 total ports of entry are high-volume land ports with 354 traffic lanes supporting 95 percent of land border traffic. After reaching full operating capability, the program’s scope was expanded to include deployment of technology and processes to outbound operations, inbound pedestrian processing, and border patrol checkpoint processing. For purposes of our report, we focused on the program’s efforts to achieve full operating capability at 39 land ports of entry.

In October 2009, we reported that WHTI fully met 6 of the 11 key practices for implementing earned value management and partially met the remaining 5 practices. Practices not fully met included, for example, a master schedule with activities that were out of sequence or lacked dependencies. Nevertheless, we reported that according to program officials, the WHTI contract was completed on time and on budget. We recommended that the department modify its earned value management policies to be consistent with best practices, implement earned value management practices that address identified weaknesses, and manage negative earned value trends.

Additionally, in June 2010, we reported that program officials anticipated total funding shortfalls for the second phase of the program (which is outside of the scope of our review) for fiscal years 2011 through 2015. Further, we reported that schedule delays for a CBP effort to upgrade local and wide area network bandwidth capacity at ports of entry could jeopardize program performance, particularly in terms of response times. Nonetheless, we noted that actual response times exceeded the expected performance levels from June 2009 to June 2010. We did not make any new recommendations at that time.

Transportation Integrated Terminal Weather System

Operations start date: April 2003 (initial operating capability), August 2010 (full operating capability)

Initially operational since April 2003, the Federal Aviation Administration’s (FAA) Integrated Terminal Weather System (ITWS) provides weather information to air traffic controllers and flight support personnel. ITWS receives observation and forecast data from the National Weather Service and combines them with data from FAA terminal sensors and sensors on nearby aircraft to integrate weather hazard information for air traffic controllers, air traffic managers, and airlines. This information is presented to end users in one integrated display. According to FAA, a prototype ITWS solution was deployed to four airports beginning in 1994. Based on those successful prototypes, FAA engaged a contractor in 1997 to design, develop, test, and deploy the ITWS system. The system was deployed to its first site in 2003; deployments to other sites continued through August 2010.

According to FAA officials, one main advantage of ITWS is that it can provide a 60-minute forecast that can anticipate short-term weather changes (such as tornadoes, thunderstorms, hail, and severe icing) that could result in airplane delays or diversions to other airports, which affect the capacities at the airports. The pre-ITWS system did not have the capability to do this. According to FAA, the implementation of ITWS increases terminal airspace capacity by 25 percent in certain weather conditions and serves to maintain capacity when it would otherwise be lost.

Treasury Customer Account Data Engine 2

The Internal Revenue Service’s (IRS) Business Systems Modernization program, which began in 1999, is a multibillion-dollar, high-risk, highly complex effort that involves the development and delivery of a number of modernized tax administration and internal management systems, as well as core infrastructure projects. These systems are intended to replace the agency’s aging business and tax processing systems, and provide improved and expanded service to taxpayers and internal business efficiencies for IRS. One of the cornerstone projects since the inception of the Business Systems Modernization program has been the Customer Account Data Engine (CADE), which was slated to modernize taxpayer account processing through replacement of the legacy Individual Master File, a 40-year old sequential, flat-file master file processing system for individual taxpayers. In August 2008, IRS began defining a new strategy—referred to as CADE 2—which would build on the progress that the current CADE processing platform had created and leverage lessons learned to date.

IRS plans to deliver CADE 2 functionality incrementally through three phases: (1) Transition State 1, (2) Transition State 2, and (3) Target State.

Operations start date: January 2012 (estimated completion date for Transition State 1)

Transition State 1 consists of the following two projects:  Daily processing: This project is to enable IRS to process and post all eligible individual taxpayer returns filed and other transactions by updating and settling individual taxpayer accounts in 24 to 48 hours with current, complete, and authoritative data, and provide employees with timely access.  Database implementation: This project is to establish the CADE 2 database, a relational database that will house data on individual taxpayers and their accounts; develop a capability to transfer data from the Individual Master File to the database; and provide for the access of data from the database to downstream IRS financial, customer service, and compliance systems.

In April 2011, IRS completed the Transition State 1 detailed design phase, which includes activities such as documenting the physical design of the solution. For purposes of this report, we focused only on the IRS’s efforts on Transition State 1 through the completion of the detailed design phase.

In March 2011, we reported that although IRS had taken some positive steps on defining benefits, estimating costs, and managing risks for CADE 2, it did not fully identify and disclose the CADE 2 costs and benefits. Specifically, we reported that  although IRS had identified benefits for the first phase of CADE 2, it had yet to set quantitative targets for 5 of the 20 identified benefits, and had yet to finalize the benefits expected in Transition State 2 or define related quantitative targets;  although IRS’s process for developing preliminary life-cycle cost estimates was generally consistent with best practices, the agency did not perform all practices associated with credible cost estimates; the schedule for delivering the initial phase of CADE 2 was ambitious; and IRS’s process for managing the risks associated with CADE 2 was generally consistent with best practices.

Our recommendations included (1) identifying all of the benefits associated with CADE 2, setting the related targets, and identifying how systems and business processes might be affected, and (2) improving the credibility of revised cost estimates.

Veterans Affairs Occupational Health Record-keeping System

During the development of the National Flu Plan, which was released in 2006, the White House Homeland Security Council directed VA to develop an employee health tracking and management system. According to VA officials, the need for this system became urgent due to the threat of pandemic influenza in 2007. As a result, the Veterans Health Administration (VHA), working with VA’s Office of Information and Technology, developed the Occupational Health Record-keeping System (OHRS). According to VA officials, OHRS was divided into two increments. The first increment consisted of a minimum feature set which represented the functionality that would provide the agency with the largest return on investment. The first increment became operational in September 2009. The second increment was intended to add functionality to the minimum feature set and to address any remaining requirements. For purposes of our report we focused on the first increment—VA’s efforts to acquire the minimum feature set. OHRS was developed using Agile software development—specifically, the Scrum methodology.

OHRS serves as the electronic health record system specifically for VA employees. OHRS provides the end users (i.e., VHA employees who work in occupational health offices at VHA healthcare facilities) the ability to collect and monitor clinical data on its employees (e.g., specific immunizations and medical training) and generate reports. Additionally, a VA official stated that OHRS allows physicians to document a number of health issues related to the workforce, including training and infectious disease management. Among other things, the information in this system is used to allocate staff to appropriate patient care assignments. For example, the system can identify whether a provider has received a vaccine for a certain illness and is therefore able to treat a patient with that illness.

Nine Factors Were Commonly Identified as Critical to the Success of Major IT Investments

Nine factors were identified as critical to the success of three or more of the seven IT investments. The factors most commonly identified include active engagement of stakeholders, program staff with the necessary knowledge and skills, and senior department and agency executive support for the program. These nine critical success factors are consistent with leading industry practices for IT acquisitions. Table 2 shows the nine factors, and examples of how agencies implemented them are discussed below.

Officials from all seven selected investments cited active engagement with program stakeholders—individuals or groups (including, in some cases, end users) with an interest in the success of the acquisition—as a critical factor to the success of those investments. Agency officials stated that stakeholders, among other things, reviewed contractor proposals during the procurement process, regularly attended program management office sponsored meetings, were working members of integrated project teams, and were notified of problems and concerns as soon as possible. For example:  Census officials stated that the DRIS stakeholders were members of the integrated project team. Their responsibilities as members of the team included involvement in requirements development, participation in peer reviews of contractual deliverables, and review of contractor proposals.

IRS officials told us that consistent and open communication with internal and external stakeholders has been critical to the success of CADE 2. For example, IRS officials told us that they regularly report progress made on CADE 2, as well as risk information on the program to oversight bodies, IRS executives, and IRS internal stakeholders.

In addition, officials from two investments noted that actively engaging with stakeholders created transparency and trust, and increased the support from the stakeholders. For example, NNSA officials noted that notifying MOMentum stakeholders of potential issues as soon as they were identified helped to foster transparency and trust; this included getting stakeholders’ approval to use a cost- and schedule-tracking approach that was not the agency’s policy, but which ultimately saved the program money and time. Additionally, CBP officials noted that communication with the WHTI stakeholders was greatly enhanced by the use of a consistent message that described, for example, the goals of the program, deployment plans, privacy implications of the Radio Frequency Identification infrastructure, and impact of the program on select groups crossing the border, including U.S. and Canadian children and Native Americans. CBP officials stated that this standardization created a consistent, unified vision and ensured that the message stayed on course.

Consistent with this factor, relevant guidance calls for programs to coordinate and collaborate with stakeholders in order to address their concerns and ensure that they fulfill their commitments. Active engagement with stakeholders increases the likelihood that the program will not encounter problems resulting from unresolved stakeholder issues.

Program Staff Had the Necessary Knowledge and Skills

Officials from six of the seven selected investments indicated that the knowledge and skills of the program staff were critical to the success of the program. This included knowledge of acquisitions and procurement processes, monitoring of contracts, large-scale organizational transformation, Agile software development concepts, and areas of program management such as earned value management and technical monitoring. For example: IRS officials stated that the Treasury Secretary utilized his critical position pay authority to hire executives for CADE 2 who had demonstrated success in managing large-scale transformation efforts in accordance with best practices. Specifically, IRS officials stated that the CADE 2 program manager was previously responsible for the design, development, and implementation of several major global information technology solutions for a major corporation.  CBP officials explained that a factor critical to the success of the acquisition was that almost every member of the team working on WHTI had a good understanding of acquisitions—some even held acquisition certifications—in addition to their understanding of program management. According to those officials, these skills contributed to effective program oversight of the WHTI contractors through all phases of the acquisition, not just during contract award.

Additionally, officials from three of the seven investments also cited the use of subject matter experts’ knowledge in their cognizant areas as a contributing factor to their programs’ successes. For example, VA officials stated that the OHRS program relied extensively on the subject matter experts’ occupational health experience—treating them as part of the development team and including them in decision making. Two investments in our sample even went one step further—by selecting the program manager from the end user organization as opposed to an individual with an IT background. For example, NNSA officials stated that they used a project manager from the end user organization as opposed to an individual from the department’s information technology office. This individual had decades of experience managing shop floor control systems. As a result, he was well aware of how the work on the shop floor is done and focused on safely delivering the necessary functional requirements to the end user.

Leading guidance also recognizes that programs should ensure that program staffs acquire the knowledge and skills needed to perform the project. Individuals who have developed the knowledge and skills needed for the programs are more likely to perform their roles effectively and efficiently.

Senior Department and Agency Executives Supported the Program

Officials from six of the seven selected investments identified support from senior department and agency executives as critical to the success of their programs. According to those officials, these senior leaders supported the success of these programs in various ways, such as by procuring funding, providing necessary information at critical times, intervening when there were difficulties working with another department, defining a vision for the program, and ensuring that end users participated in the development of the system. For example:  The WHTI program manager told us that the former DHS Deputy Secretary reached out to another department in order to finalize a memorandum of understanding that would be used to share information on passports and passcards needed for WHTI. According to the WHTI program manager, prior to the Deputy Secretary’s involvement, the other department’s efforts to collaborate on this issue were not meeting the schedule requirements of the WHTI program. That official told us that after receiving the necessary support from the other department, CBP was able to more rapidly query that department’s data.

IRS officials explained that endorsement for CADE 2 has come from the highest levels of the organization. In particular, those officials told us that the IRS Commissioner has made CADE 2 one of his top priorities. IRS officials told us that the Commissioner, through, for example, his keynote speech at a CADE 2 town hall meeting for IRS employees, has provided a clear and unwavering message about CADE 2. This speech and other activities have unified IRS employees, driven change, and removed barriers that can often impede programs of this magnitude.

In our experience, strong leadership support can result in benefits to a program, including providing the program manager with the resources necessary to make knowledge-based, disciplined decisions that increase the likelihood of their program’s success.

End Users and Stakeholders Were Involved in the Development of Requirements

Officials from five of seven selected investments identified the involvement of stakeholders—including end users—in the requirements development process as a factor that was critical to the success of their programs. For example:  Census officials told us that the DRIS program management office collaborated extensively with the stakeholders and the contractor to develop requirements. For example, program management office personnel, contractor staff, and the stakeholders all worked together to analyze the requirements in order to ensure they were understood, unique, and verifiable.  VA officials told us that an OHRS end user identified a set of requirements for an occupational health system 3 years prior to the initiation of OHRS development efforts. Those officials told us that the developers worked closely with the OHRS end user representative to ensure that those requirements were still valid once the program was initiated, given the length of time since the requirements were initially identified.

Relevant industry guidance recognizes the importance of eliciting end user needs and involving stakeholders in requirements development. When stakeholders and end users communicate their requirements throughout the project life cycle, the resulting system is more likely to perform as intended in the end user’s environment.

End Users Participated in Testing of System Functionality Prior to Formal End User Acceptance Testing

Officials from five of the seven selected investments identified having the end users test and validate the system components prior to formal end user acceptance testing for deployment as critical to the success of their program. For example:  DISA officials told us they used a virtual site to connect developers and end users in online testing of evolving software repeatedly during the development of GCSS-J. Using the tool, the developers were able to record the sessions, which was helpful in addressing defects identified during testing.  CBP created a fully functional test lab facility for the WHTI program at a mock port of entry test facility constructed at an old private airport in Virginia. Using this facility, they were able to test the software that was being developed and the hardware that was being proposed. Additionally, a core end user group was established and brought to the facility multiple times a year during the acquisition to test the forthcoming technology.

Similar to this factor, leading guidance recommends testing selected products and product components throughout the program life cycle. Testing of functionality by end users prior to acceptance demonstrates, earlier rather than later in the program life cycle, that the functionality will fulfill its intended use. If problems are found during this testing, programs are typically positioned to make changes that are less costly and disruptive than ones made later in the life cycle would be.

Government and Contractor Staff Were Consistent and Stable

Officials from four of the seven selected investments stated that government and contractor organizations’ personnel were consistent and stable. For example:  DISA officials indicated that the longevity of the program management office and contractor staffs has been a contributing factor to GCSS-J’s success. For example, the longevity of the staff contributed to them becoming subject matter experts in their areas of responsibility.  CBP officials explained that key program management office staff remained consistent throughout the WHTI program. In addition, according to a CBP official, the staffs genuinely liked to work with one another and were able to collaborate effectively.

This factor is consistent with relevant guidance that espouses the importance of having adequate and skilled resources. In particular, having consistent and stable staff can allow teams to keep pace with their workload, make decisions, and have the necessary accountability.

Program Staff Prioritized Requirements

Officials from four of the seven selected investments cited the prioritization of requirements as enabling the efficient and effective development of system functionality. For example:  FAA officials told us that ITWS end users presented the development team with a “wish list” of requirements that would help them significantly. Those officials told us that end users and developers prioritized those requirements by balancing importance to the end users with the maturity of the technology. FAA officials stated that prototypes of these new requirements were developed and evaluated by end users in the field and were ultimately implemented in the initial operating capability for ITWS.  DISA officials explained that during development, GCSS-J end user representatives met with the GCSS-J program office and the GCSS-J developer twice a week for between a full and a half day in order to identify and prioritize requirements. Those officials explained that this frequent interaction was necessary because of the short development iterations (4 to 5 weeks), at the end of which useable functionality was presented to the end users for review. The frequent prioritization ensured that the functionality most critical to the end user representative was developed, and could be deployed sooner than functionality of less importance.

Consistent with leading guidance, having prioritized requirements guides the programs in determining the system’s scope and ensures that the functionality and quality requirements most critical to the end users are deployed before less-desired requirements.

Program Officials Maintained Regular Communication with the Prime Contractor

Officials from four of the seven selected investments indicated that regular communication between the program management office and the prime contractor was critical to the success of the program. This communication was proactive in that there were regularly scheduled meetings between the program management office and the prime contractor, with an expectation of full and honest disclosure of problems. For example:  Census officials stated that the DRIS program management office took a proactive, “no surprises” approach to communicating with the contractor. For example, on a monthly basis, the program management office formally documented the technical performance of the contractor based on the relevant elements of the work breakdown structure and the award fee plan. These reports were provided to the contractor, who in turn used the feedback to improve its technical performance. In addition, DRIS program managers and their contractor counterparts met weekly to discuss significant issues. DRIS officials emphasized that the expectation of open communication and trust from senior leadership fostered an environment where issues could be freely discussed with the contractor.  CBP officials stated that during the deployment of the WHTI technology to the ports of entry, the program management office held daily conference calls with the contractor to ensure proper coordination and the rapid resolution of problems. For example, during deployment to one port of entry it was determined that the electric system that provided power to the lanes was not adequate. This problem was quickly identified, responsibility for resolving it was assigned, and the issue was quickly resolved.

Additionally, Census and VA officials stated that ensuring a positive, non- adversarial relationship between the prime contractor and the program management office was critical to the success of the investment. Census officials noted that both the government and the contractor staff recognized that the only way for the program to succeed was for both parties to succeed.

Consistent with this factor, leading guidance recognizes the importance of communication between program officials and the contractor organizations. Implementation of this critical success factor enables programs to ensure that requirements are understood and risks and issues are identified and addressed earlier rather than later in the process, thereby increasing the likelihood that the delivered system will meet its intended purpose and resulting in less costly and less disruptive changes and work efforts.

Programs Received Sufficient Funding

Officials from three of the seven selected investments explained that sufficient funding for the programs contributed to the success of those investments. Officials from two of the investments attributed funding to strong congressional support; in a third case, officials cited strong leadership from senior agency and program officials as being a factor. For example:  The WHTI program manager stated that the WHTI program received the requested funding from Congress for the 2 years leading up to the June 1, 2009, mandated implementation date. Additionally, that official told us that Congress provided 2-year money, that is, money that could be obligated over a period of 2 years. Officials told us that the 2- year money gave the program great flexibility to accommodate the inherent complexities and expenditures incurred in a multiyear deployment, and to adapt to inevitable modifications in deployment requirements (that is, additional sites, lanes, and functionality).

IRS officials told us that the IRS Commissioner helped the CADE 2 program obtain funding. For example, those officials told us that the IRS Commissioner spoke with congressional representatives frequently in order to sustain interest and support for CADE 2.

Relevant guidance recognizes the importance of sufficiently funding IT investments. Investments that receive funding commensurate with their requirements are better positioned to ensure the availability of needed resources, and therefore, deliver the investment within established goals.

The nine commonly identified critical success factors are consistent with OMB’s 25-point plan to improve IT management and oversight. In particular, one high-level objective of the plan—effectively managing large-scale IT programs—aims to improve areas that impact the success rates of large IT programs across the federal government. As part of this high-level objective, the plan addresses the importance of ensuring that program management professionals have extensive experience and training, defining requirements by engaging with stakeholders, and providing senior executives with visibility into the health of their IT programs. These principles of effective IT management are reflected in the commonly identified critical success factors. For example, as previously mentioned, six of the seven agencies identified the knowledge and skills of program staff and five of seven agencies cited the involvement of end users and stakeholders in the development of requirements as critical to the success of their IT investments.

While our analysis of critical success factors identified by agencies resulted in nine commonly identified factors, agencies also identified additional factors as contributing to the success of their investments. For example:  Agile software development: DISA officials stated that the use of Agile software development was critical to the success of the program. Among other things, Agile enhanced the participation of the end users in the development process and provided for capabilities to be deployed in shorter periods of time.  Streamlined and targeted governance: IRS officials told us that in comparison to other IRS business systems modernization projects, the governance model for CADE 2 has been streamlined. For example, those officials stated that the CADE 2 governance structure includes an executive steering committee that, in contrast to other programs at IRS that utilize an executive steering committee, is dedicated solely to the CADE 2 program. IRS officials told us that this gives an added measure of accountability and responsibility for the successful outcome of the program.  Continuous risk management: VA officials stated that the risk management strategy that the program used was critical to its success. According to the VA officials, risks were identified at daily team meetings and mitigation strategies were developed. Furthermore an official explained that risk management is built in the Agile software development process by, for example, involving the end user early and often to ensure that the requirements were as thoroughly vetted as possible.

Several of these factors are also consistent with best practices, such as the critical factors relating to risk management and governance. The full list of critical success factors and how agencies implemented them are presented in appendix II.

Concluding Observations

Although the critical success factors identified by the seven agencies were cited as practices that contributed to the success of their acquisitions, implementation of these factors will not necessarily ensure that federal agencies will successfully acquire IT systems because many different factors contribute to successful acquisitions. Nevertheless, the examples of how agencies implemented the critical success factors may help federal agencies address the well-documented acquisition challenges they face. Moreover, the critical success factors in this report also support OMB’s objective of improving the management of large- scale IT acquisitions across the federal government, and wide dissemination of these factors and how agencies implemented them could complement these efforts.

Agency Comments and Our Evaluation

We received written, e-mail, or verbal responses on a draft of this report from all seven departments in our review as well as OMB. These responses are summarized below.  The Acting Secretary for the Department of Commerce provided written comments. The department stated that the report provides a good overview and assessment of governmentwide critical factors and elements that led to the successful acquisition of IT investments. The department also provided technical comments, which we incorporated as appropriate.  An acquisition analyst from the Department of Defense CIO Acquisition Directorate, writing on behalf of the department, provided an e-mail, which stated that the department had no comments on the draft report.  The Director of the NNSA’s Office of Internal Controls, responding on behalf of the Department of Energy, provided an e-mail stating that they agreed with the report and had no further comments. They also noted that the department is committed to supporting OMB’s objective of improving the management of large-scale IT acquisitions, and that wide dissemination of the factors in our report could complement OMB’s efforts.  The Director of DHS’s Departmental GAO/Office of Inspector General Liaison Office provided written comments. In its comments, the department noted that it remains committed to continuing its work with OMB to improve the oversight and management of IT investments to help ensure that systems are acquired on time and within budget, and that they deliver the expected benefits and functionality. The department further stated that it will use this report to enhance and improve the factors critical to the successful acquisition of the department’s investments, such as creating a structured training program to assist in obtaining certification in the program management career field, and conducting reviews to provide insight into the cost, schedule, and performance of IT investments. The department also provided technical comments, which we incorporated as appropriate.  The Deputy Director of Audit Relations within the Department of Transportation’s Office of the Secretary provided an e-mail with technical comments, which we incorporated as appropriate.  A program analyst within the Office of the Chief Information Officer for the Department of the Treasury, writing on behalf of the department, provided an e-mail, which stated that the department had no comments on the draft report.  The Department of Veterans Affairs Chief of Staff provided written technical comments, which we incorporated as appropriate.  A policy analyst from OMB’s Office of E-Government and Information Technology, speaking on behalf of OMB, provided verbal technical comments, which we incorporated as appropriate.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution of it until 30 days from the date of this letter. At that time, we will send copies of this report to interested congressional committees; the Director of OMB; the secretaries and agency heads of the departments and agencies addressed in this report; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff members have any questions on the matters discussed in this report, please contact me at (202) 512-9286 or pownerd@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to (1) identify federal information technology (IT) investments that were or are being successfully acquired and (2) identify the critical factors that led to the successful acquisition of these investments.

To address our first objective, we selected 10 departments with the largest planned IT budgets as reported in the Office of Management and Budget’s (OMB) fiscal year 2011 Exhibit 53. Collectively, these departments accounted for 88 percent of the federal government’s requested total IT budget for fiscal year 2011. We then asked the chief information officers (CIO) and other acquisition and procurement officials from the departments to select one major, mission-critical IT investment that was, preferably, operational and that best achieved its cost, schedule, scope, and performance goals. Seven departments—the Departments of Defense, Commerce, Energy, Homeland Security, Transportation, the Treasury, and Veterans Affairs—identified successful IT investments. Collectively, these departments accounted for 73 percent of the planned IT spending for fiscal year 2011.

To address our second objective, we interviewed officials responsible for each investment, asking them to identify and describe the critical factors that led to their success, and to provide examples where possible. We validated our understanding of the factors and examples collected during the interviews by providing written summaries to agency officials to ensure that their information was accurately portrayed. Because of the open-ended nature of our discussions with officials, we conducted a content analysis of the information we received in order to identify common critical success factors. We then totaled the number of times each factor was mentioned by department and agency officials, choos to report on the critical success factors that were identified by three or more investments. This resulted in our list of nine commonly iden tified critical succes s factors. We then compared these nine critical success factors to leading industry practices on IT acquisitions, such as the Software Engineering Institute’s (SEI) Capability Maturity Model® Integration (CMMI®) for Acquisition, the Project Management Institute’s A ’s Guide to the Project Management Body of Knowledge, and GAO Information Technology Investment Management: A Framework for Assessing and Improving Process Maturity. Finally, we compared the nine commonly identified critical success factors to OMB’s 25 Point Implementation Plan to Reform Federal Information Technology Management in order to determine whether those critical succes are related to the high-level objectives found in the plan.

Appendix II: Critical Success Factors

The following seven tables provide a description of critical success factors identified by officials with each of the investments in our sample.

Appendix III: GAO Contact and Staff Acknowledgments

G

Staff Acknowledgments

dition to the contact named above, Deborah A. Davis (Assistant tor), Kaelin P. Kuhn, Lee McCracken, Thomas E. Murphy, Jamelyn n, and Andrew Stavisky made key contributions to this report.