Information Technology: Critical Factors Underlying Successful Major Acquisitions
Published: 2011-10-21T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Planned federal information technology (IT) spending has now risen to at least $81 billion for fiscal year 2012. As GAO has previously reported, although a variety of best practices exists to guide their successful acquisition, federal IT projects too frequently incur cost overruns and schedule slippages while contributing little to mission-related outcomes. Recognizing these problems, the Office of Management and Budget (OMB) has launched several initiatives to improve the oversight and management of IT investments. GAO was asked to identify (1) federal IT investments that were or are being successfully acquired and (2) the critical factors that led to the successful acquisition of these investments. To do this, GAO interviewed agency officials from selected federal departments responsible for each investment. In commenting on a draft of GAO's report, three departments generally agreed with the report. OMB and the other departments either provided minor technical comments, or stated that they had no comments at all.
section_title: What GAO Found
paragraphs: According to federal department officials, the following seven investments were successfully acquired in that they best achieved their respective cost, schedule, scope, and performance goals: (1) Department of Commerce's Decennial Response Integration System; (2) Department of Defense's Global Combat Support System-Joint, Increment 7; (3) Department of Energy's Manufacturing Operations Management (MOMentum) Project; (4) Department of Homeland Security's Western Hemisphere Travel Initiative; (5) Department of Transportation's Integrated Terminal Weather System; (6) Department of the Treasury's Customer Account Data Engine 2 (CADE 2); and (7) Department of Veterans Affairs' Occupational Health Record-keeping System. Department officials identified nine common factors that were critical to the success of three or more of the seven investments: (1) Program officials were actively engaged with stakeholders; (2) Program staff had the necessary knowledge and skills; (3) Senior department and agency executives supported the programs; (4) End users and stakeholders were involved in the development of requirements; (5) End users participated in testing of system functionality prior to formal end user acceptance testing; (6) Government and contractor staff were stable and consistent; (7) Program staff prioritized requirements; (8) Program officials maintained regular communication with the prime contractor; and (9) Programs received sufficient funding. Officials from all seven investments cited active engagement with program stakeholders as a critical factor to the success of those investments. Agency officials stated that stakeholders regularly attended program management office sponsored meetings; were working members of integrated project teams; and were notified of problems and concerns as soon as possible. Implementation of these critical factors will not necessarily ensure that federal agencies will successfully acquire IT systems because many different factors contribute to successful acquisitions. Nonetheless, these critical factors support OMB's objective of improving the management of large-scale IT acquisitions across the federal government, and wide dissemination of these factors could complement OMB's efforts.

section_title: Letter
paragraphs: Planned federal information technology (IT) spending has now risen to at  least $81 billion for fiscal year 2012. As we have previously reported,  federal IT projects too frequently incur cost overruns and schedule  slippages while contributing little to mission-related outcomes. Given the  size of these investments and the criticality of many of these systems to  the health, economy, and security of the nation, it is important that federal  agencies successfully acquire these systems—that is, ensure that the  systems are acquired on time and within budget and that they deliver the  expected benefits and functionality.
This report responds to your request that we:  1.  Identify federal IT investments that were or are being successfully  acquired.  2.  Identify the critical factors that led to the successful acquisition of  these investments.
To address our first objective, we interviewed the chief information  officers (CIO) and other acquisition and procurement officials from  selected departments in order to identify one mission-critical, major IT  investment that was, preferably, operational, and that best achieved its  cost, schedule, scope, and performance goals.
To address our second objective, we interviewed officials responsible for  each investment and asked them what critical factors led to the  investment’s success. We then categorized the critical success factors  and totaled the number of times each factor was mentioned by the  department and agency officials. In order to identify common critical  success factors, we generalized critical success factors that were  mentioned by three or more investments. We also compared the critical  success factors to the Office of Management and Budget’s (OMB) 25  Point Implementation Plan to Reform Federal Information Technology  Management in order to determine whether those critical success factors  support OMB’s efforts.
We conducted our work from December 2010 through October 2011 in  accordance with all sections of GAO’s Quality Assurance Framework that  are relevant to our objectives. The framework requires that we plan and  perform the engagement to obtain sufficient and appropriate evidence to  meet our stated objectives and to discuss any limitations in our work. We  believe that the information and data obtained, and the analysis  conducted, provide a reasonable basis for any findings and conclusions in  this product. Further details of our objectives, scope, and methodology  are in appendix I.
subsections: section_title: Background
paragraphs: Investments in IT can enrich people’s lives and improve organizational  performance. For example, during the last two decades the Internet has  matured from being a means for academics and scientists to  communicate with each other to a national resource where citizens can  interact with their government in many ways, such as by receiving  services, supplying and obtaining information, asking questions, and  providing comments on proposed rules.
While these investments have the potential to improve lives and  organizations, federally funded IT projects can—and have—become  risky, costly, unproductive mistakes. As we have described in numerous  reports and testimonies, although a variety of best practice  documentation exists to guide their successful acquisition, federal IT  projects too frequently incur cost overruns and schedule slippages while  contributing little to mission-related outcomes.
subsections: section_title: IT Acquisition Best Practices Have Been Identified by Industry and Government and Promoted by Legislation
paragraphs: IT acquisition best practices have been developed by both industry and  the federal government. For example, the Software Engineering Institute  (SEI) has developed highly regarded and widely used guidance on best  practices such as requirements development and management, risk  management, configuration management, validation and verification, and  project monitoring and control. In the federal government, GAO’s own  research in IT management best practices led to the development of the  Information Technology Investment Management (ITIM) Framework,  which describes essential and complementary IT investment  management disciplines, such as oversight of system development and  acquisition management, and organizes them into a set of critical  processes for successful investments.
Congress has also enacted legislation that reflects IT management best  practices. For example, the Clinger-Cohen Act of 1996, which was  informed by GAO best practice recommendations, requires federal  agencies to focus more on the results they have achieved through IT  investments, while concurrently improving their IT acquisition processes.  Specifically, the act requires agency heads to implement a process to  maximize the value of the agency’s IT investments and assess, manage,  and evaluate the risks of its IT acquisitions. Further, the act establishes  chief information officers (CIO) to advise and assist agency heads in  carrying out these responsibilities. The act also requires OMB to  encourage agencies to develop and use best practices in IT acquisition.
Additionally, the E-Government Act of 2002 established a CIO Council,  which is led by the Federal CIO, to be the principal interagency forum for  improving agency practices related to the development, acquisition, and  management of information resources, including sharing best practices.  Consistent with this mandate, the CIO Council established a Management  Best Practices Committee in order to serve as a focal point for promoting  IT best practices within the federal government.
section_title: Prior GAO Work Has Identified IT Acquisition Management Weaknesses, Cost Increases, and Schedule Delays on Troubled Investments
paragraphs: We have often reported on a range of acquisition management  weaknesses facing federal IT investments—including problems relating to  senior leadership, requirements management, and testing. For example,  for the investments described below, we have identified acquisition  weaknesses, and have reported on significant cost increases and  schedule delays. Additionally, each of these investments was ultimately  cancelled or significantly restructured as a result of agency reviews  conducted in response to acquisition weaknesses, cost increases, and  schedule delays.
In June 2009, we reported that an executive committee for the  National Polar-orbiting Operational Environmental Satellite System  (NPOESS)—a program jointly managed by the Department of  Commerce’s National Oceanic and Atmospheric Administration, the  Department of Defense, and the National Aeronautics and Space  Administration—lacked the membership and leadership needed to  effectively and efficiently oversee and direct the program.  Specifically, the Defense committee member with acquisition authority  did not attend committee meetings and sometimes contradicted the  committee’s decisions. Further, the committee did not track its action  items to closure, and many of the committee’s decisions did not  achieve desired outcomes. To address these issues, we  recommended that the Secretary of Defense direct the key committee  member to attend and participate in committee meetings. Additionally,  we recommended that the heads of the agencies that participate in  the committee direct the committee members to track action items to  closure, and identify the desired outcomes associated with each of the  committee’s actions.
Further, we reported that the launch date for an NPOESS  demonstration satellite had been delayed by over 5 years and the cost  estimate for the program had more than doubled—from $6.5 billion to  about $15 billion. In February 2010, a presidential task force  decided to disband NPOESS and, instead, have the agencies  undertake separate acquisitions.    Since 2007, we have reported on a range of acquisition weaknesses  facing the Department of Homeland Security’s (DHS) Secure Border  Initiative Network—also known as SBInet. For example, in January  2010, we reported that DHS had not effectively managed key aspects  of the SBInet testing program such as defining test plans and  procedures in accordance with important elements of relevant  guidance. In light of these weaknesses, we made recommendations  to DHS related to the content, review, and approval of test planning  documentation.
In May 2010, we reported that the final acceptance of the first two  SBInet deployments had slipped from November 2009 and March  2010 to September 2010 and November 2010, respectively, and that  the cost-effectiveness of the system had not been justified. We  concluded that DHS had not yet demonstrated that the considerable  time and money being invested to acquire and develop SBInet was a  wise and prudent use of limited resources. The Secretary of  Homeland Security ordered a departmentwide assessment of the  SBInet program; the Secretary’s decision was motivated in part by  continuing delays in the development and deployment of SBInet  capabilities and concerns that the SBInet system had not been  adequately justified by a quantitative assessment of cost and benefits.  Based on the results of the assessment, in January 2011, the DHS  Secretary decided to end SBInet as originally conceived.
In May 2010, we reported that after spending $127 million over 9  years on an outpatient scheduling system, the Department of  Veterans Affairs (VA) has not implemented any of the planned  system’s capabilities and is essentially starting over. After  determining that the system could not be deployed, the department  terminated the contract and ended the program in September 2009.  We concluded that the department’s efforts to successfully complete  the system had been hindered by weaknesses in several key project  management disciplines and a lack of effective oversight that, if not  addressed, could undermine the department’s second effort to replace  the scheduling system. We recommended that the department take  action to improve key processes, including acquisition management,  requirements management, system testing, implementation of earned  value management, risk management, and program oversight.
In June 2011, we reported that end users were not sufficiently  involved in defining requirements for the Federal Emergency  Management Agency’s (FEMA) National Flood Insurance Program’s  insurance policy and claims management system. After conducting  an assessment of the program prompted by problems identified in end  user testing, FEMA leadership cancelled the system because it failed  to meet end user expectations. This decision forced the agency to  continue to rely on an outdated system that is neither effective nor  efficient. In order to avoid the root causes of this program’s failure, we  recommended that for future related modernization attempts, DHS  should ensure that key stakeholders are adequately involved in  requirements development and management.
Additionally, we have previously reported on investments in need of  management attention across the federal government. For example, in  April 2011, we reported on the visibility into federal IT investments  provided by the IT Dashboard—a publicly available website that displays  detailed information on federal agencies’ major IT investments, including  assessments of actual performance against cost and schedule targets  (referred to as ratings) for approximately 800 major federal IT  investments. Specifically, we reported that, as of March 2011, the  Dashboard provided visibility into over 300 IT investments, totaling almost  $20 billion, in need of management attention. We noted that     272 investments with costs totaling $17.7 billion had ratings that  indicated the need for attention, and    39 investments with costs totaling $2.0 billion had ratings that  indicated significant concerns.
section_title: OMB Has Several Initiatives Under Way to Improve the Oversight and Management of IT Investments
paragraphs: OMB plays a key role in helping federal agencies manage their  investments by working with them to better plan, justify, and determine  how much they need to spend on projects and how to manage approved  projects.
In June 2009, OMB established the IT Dashboard to improve the  transparency into and oversight of agencies’ IT investments. According to  OMB officials, agency CIOs are required to update each major investment  in the IT Dashboard with a rating based on the CIO’s evaluation of certain  aspects of the investment, such as risk management, requirements  management, contractor oversight, and human capital. According to  OMB, these data are intended to provide a near real-time perspective of  the performance of these investments, as well as a historical perspective.  Further, the public display of these data is intended to allow OMB,  congressional and other oversight bodies, and the general public to hold  government agencies accountable for results and progress.
In January 2010, the Federal CIO began leading TechStat sessions— reviews of selected IT investments between OMB and agency  leadership—to increase accountability and transparency and improve  performance. OMB has identified factors that may result in an investment  being selected for a TechStat session, such as—but not limited to— evidence of (1) poor performance; (2) duplication with other systems or  projects; (3) unmitigated risks; and (4) misalignment with policies and  best practices.
OMB officials stated that as of June 30, 2011, 63 TechStat sessions had  been held with federal agencies. According to OMB, these sessions  enabled the government to improve or terminate IT investments that were  experiencing performance problems. For example, in June 2010, the  Federal CIO led a TechStat on the National Archives and Records  Administration’s (NARA) Electronic Records Archives investment that  resulted in six corrective actions, including halting fiscal year 2012  development funding pending the completion of a strategic plan. Similarly,  in January 2011, we reported that NARA had not been positioned to  identify potential cost and schedule problems early, and had not been  able to take timely actions to correct problems, delays, and cost increases  on this system acquisition program. Moreover, we estimated that the  program would likely overrun costs by between $205 and $405 million if  the agency completed the program as originally designed. We made  multiple recommendations to the Archivist of the United States, including  establishing a comprehensive plan for all remaining work, improving the  accuracy of key performance reports, and engaging executive leadership  in correcting negative performance trends.
Drawing on the visibility into federal IT investments provided by the IT  Dashboard and TechStat sessions, in December 2010, OMB issued a  plan to reform IT management throughout the federal government over an  18-month time frame. The plan contains two high-level objectives:    achieving operational efficiency, and    effectively managing large-scale IT programs.
To achieve these high-level objectives, the plan outlines 25 action items.  According to OMB officials, they have taken several actions pursuant to  this plan. For example, pursuant to Action Item Number 10—development  of an IT best practices collaboration platform—in April 2011 the CIO  Council launched an IT best practices collaboration website. According  to OMB, this portal provides federal program managers with access to a  searchable database of program management best practices in order to  promote interagency collaboration and real-time problem solving related  to IT programs. The portal contains links to case studies by federal  agencies demonstrating the use of best practices in managing large-scale  IT systems. For example, a recent case study posted by the Social  Security Administration outlined efforts to develop a cadre of highly  skilled, trained, and qualified program managers to promote the success  of its investments.
section_title: Seven IT Investments Were Reported as Being Successfully Acquired
paragraphs: According to federal department officials, the following seven investments  best achieved their respective cost, schedule, scope, and performance  goals.
The estimated total life-cycle cost of the seven investments is about $5  billion. Six of the seven investments are currently operational. The  following provides descriptions of each of the seven investments.
subsections: section_title: Commerce Decennial Response Integration System
paragraphs: The U.S. Census Bureau is the primary source of basic statistics about  the population and economy of the nation and is best known for the  decennial census of population and housing. The most recent decennial  census was conducted in 2010. Between March and August 2010, the  Census Bureau provided assistance to respondents and captured their  response data via paper and telephone agent to allow sufficient time for  post-capture processing, review, and tabulation. The Decennial  Response Integration System (DRIS) provided a system for collecting and  integrating census responses from forms and telephone interviews.  Specifically, DRIS integrated the following three primary functions:    Paper data capture: Processed paper census questionnaires sent by  mail from respondents. The system sorted the questionnaires and  captured data from them, which were turned into electronic data.    Telephone questionnaire assistance: Provided respondents with  assistance in understanding their questionnaire, and captured  responses for forms completed over the phone. This function utilized  interactive voice response as the initial contact mechanism with an  option to speak with call center representatives if needed.    Coverage follow up: Contacted a sample of respondents by telephone to  determine if changes should be made to their household roster as  reported on their initial census return with the goal of ensuring that every  person in the United States is counted once and in the right place.
To help carry out the 2010 Decennial Census, the government engaged a  contractor to design, build, test, deploy, implement, operate, and maintain  the systems, infrastructure, staffing, procedures, and facilities needed for  DRIS. The DRIS contract was divided into three primary phases. Phase 1  included the development, testing, deployment, implementation, and  support of the DRIS components needed for a 2008 Census Dress  Rehearsal. Phase 2 included the nationwide deployment of the DRIS  components and full-scale production operations of the paper data  capture, telephone questionnaire assistance, and coverage follow up  functions for the 2010 Census. Phase 3 is to address post-2010 Census  DRIS component disposition and data archiving. Phase 3 was to be  completed in September 2011. For purposes of our report, we focused  only on the first two phases of DRIS because the DRIS system was being  acquired during these phases.
In October 2009, we reported that DRIS fully implemented the key  practices necessary for a sound implementation of earned value  management—a project management approach that, if implemented  appropriately, provides objective reports of project status, produces early  warning signs of impending schedule delays and cost overruns, and  provides unbiased estimates of anticipated costs at completion.  Additionally, we reported that, as of May 2009, the DRIS contractor was  experiencing a cumulative cost underrun and was ahead of schedule;  however, the life-cycle cost estimate for DRIS had increased from $574  million to $946 million. This cost increase was mostly due to increases in  both paper and telephone workloads. For example, the paper workload  increased due to an April 2008 redesign of the 2010 Census that reverted  planned automated operations to paper-based processes and required  DRIS to process an additional estimated 40 million paper forms.
section_title: Defense Global Combat Support System-Joint Increment 7
paragraphs: Number of users: 20-30 joint warfighter logistician users; 13,000  single-sign-on users Operations start date:  March 2009 (deployment of initial operational  capability for Increment 7)
The Global Combat Support System-Joint (GCSS-J) Increment 7 is a system  that supports military logistics operations that provide military personnel with  the supplies and information they need to accomplish their missions. GCSS- J combines data, such as the location and quantity of a particular resource,  from multiple authoritative data sources (e.g., Asset Visibility, Joint Operation  Planning and Execution System, and Global Decision Support System) and  analyzes the data to provide information needed by logistics decision  makers. The end users of the system are the logisticians at the various  Combatant Commands, which are made up of representatives from multiple  branches, each having a geographical or functional responsibility.  According to Defense Information Systems Agency (DISA) officials, the  analyses generated by the system enable the commanders of the  Combatant Commands to rapidly make critical decisions, and to plan,  execute, and control logistics operations. Additionally, the system provides  other end users with single sign-on access to the individual data sources.  The diverse end user group, combined with a wide spectrum of data,  provides a unified supply chain for the Army, Navy, Air Force, and Marine  forces in a given area, which is to help eliminate inefficiencies and provide a  more useful view into the supply chain.
DISA started GCSS-J in 1997 as a prototype. The system is being  developed incrementally using Agile software development— specifically, the Scrum methodology. DISA is currently developing and  deploying major releases for Increment 7. A total of five major releases  were planned within Increment 7; Releases 7.0 and 7.1, which were  implemented in March 2009 and December 2009 respectively, were the  subject of our review.
To date, according to DISA, Increment 7 releases have improved  performance and provided new capabilities and enhancements to existing  capabilities. For example, the system provides real-time information about  road conditions, construction, incidents, and weather to facilitate rapid  deployment of military assets.
section_title: Energy Manufacturing Operations Management Project
paragraphs: The Manufacturing Operations Management (MOMentum) Project aims to  replace a suite of aging mission-essential shop floor, manufacturing  control systems at the Y-12 National Security Complex that support the  National Nuclear Security Administration’s (NNSA) Stockpile Stewardship  and Management Program. The shop floor at the Y-12 complex is  responsible for the construction, restoration, and dismantling of nuclear  weapon components. The core software currently used in the shop floor  manufacturing control systems was deployed in the mid-1980s and will no  longer be supported by the vendor on its current hardware platform  beginning in 2012.
The MOMentum Project has two phases. Phase 1, which was the subject  of our review, was implemented in September 2010, and is a deployment  of the Production Planning module of SAP for manufacturing planning  and scheduling. Phase 2 is to include the deployment of the  Manufacturing Execution module of SAP software and support the  execution of production schedules on the shop floor. Phase 2 is  scheduled to be completed in September 2013. The implementation of  the system is expected to save $6 million annually, reduce cycle times for  manufacturing, remove dependencies on obsolete technology and  unsupported software, and reduce administrative errors and product  deviations, among other things.
section_title: Homeland Security Western Hemisphere Travel Initiative
paragraphs: Operations start date:  September 2008 (initial operating capability),  June 2009 (full operating capability)
To facilitate inspections at the nation’s 330 air, sea, and land ports of  entry, the Western Hemisphere Travel Initiative (WHTI) requires all  citizens of the United States and citizens of Canada, Mexico, and  Bermuda traveling to the United States as nonimmigrant visitors to have a  passport or other accepted document that establishes the bearer’s  identity and citizenship to enter the country from within the Western  Hemisphere. In order to implement WHTI at the land border while  limiting its impact on the public, U.S. Customs and Border Protection  (CBP) engaged a contractor to procure and deploy technology—including  Radio Frequency Identification, License Plate Reader, and Vehicle  Primary Client technologies. These technologies help to provide CBP  officers with law enforcement and border crossing history information for  each traveler and vehicle. Initial operating capability was achieved in  September 2008 when these technologies were deployed to two ports of  entry. Full operating capability was achieved in June 2009 when the  WHTI technology was deployed to 37 additional ports of entry. The 39  total ports of entry are high-volume land ports with 354 traffic lanes  supporting 95 percent of land border traffic. After reaching full operating  capability, the program’s scope was expanded to include deployment of  technology and processes to outbound operations, inbound pedestrian  processing, and border patrol checkpoint processing. For purposes of  our report, we focused on the program’s efforts to achieve full operating  capability at 39 land ports of entry.
In October 2009, we reported that WHTI fully met 6 of the 11 key  practices for implementing earned value management and partially met  the remaining 5 practices. Practices not fully met included, for example,  a master schedule with activities that were out of sequence or lacked  dependencies. Nevertheless, we reported that according to program  officials, the WHTI contract was completed on time and on budget. We  recommended that the department modify its earned value management  policies to be consistent with best practices, implement earned value  management practices that address identified weaknesses, and manage  negative earned value trends.
Additionally, in June 2010, we reported that program officials anticipated  total funding shortfalls for the second phase of the program (which is  outside of the scope of our review) for fiscal years 2011 through 2015.  Further, we reported that schedule delays for a CBP effort to upgrade  local and wide area network bandwidth capacity at ports of entry could  jeopardize program performance, particularly in terms of response times.  Nonetheless, we noted that actual response times exceeded the  expected performance levels from June 2009 to June 2010. We did not  make any new recommendations at that time.
section_title: Transportation Integrated Terminal Weather System
paragraphs: Operations start date:  April 2003 (initial operating capability), August  2010 (full operating capability)
Initially operational since April 2003, the Federal Aviation Administration’s  (FAA) Integrated Terminal Weather System (ITWS) provides weather  information to air traffic controllers and flight support personnel. ITWS  receives observation and forecast data from the National Weather  Service and combines them with data from FAA terminal sensors and  sensors on nearby aircraft to integrate weather hazard information for air  traffic controllers, air traffic managers, and airlines. This information is  presented to end users in one integrated display. According to FAA, a  prototype ITWS solution was deployed to four airports beginning in 1994.  Based on those successful prototypes, FAA engaged a contractor in 1997  to design, develop, test, and deploy the ITWS system. The system was  deployed to its first site in 2003; deployments to other sites continued  through August 2010.
According to FAA officials, one main advantage of ITWS is that it can  provide a 60-minute forecast that can anticipate short-term weather  changes (such as tornadoes, thunderstorms, hail, and severe icing) that  could result in airplane delays or diversions to other airports, which affect  the capacities at the airports. The pre-ITWS system did not have the  capability to do this. According to FAA, the implementation of ITWS  increases terminal airspace capacity by 25 percent in certain weather  conditions and serves to maintain capacity when it would otherwise be lost.
section_title: Treasury Customer Account Data Engine 2
paragraphs: The Internal Revenue Service’s (IRS) Business Systems Modernization  program, which began in 1999, is a multibillion-dollar, high-risk, highly  complex effort that involves the development and delivery of a number of  modernized tax administration and internal management systems, as well  as core infrastructure projects. These systems are intended to replace the  agency’s aging business and tax processing systems, and provide  improved and expanded service to taxpayers and internal business  efficiencies for IRS. One of the cornerstone projects since the inception of  the Business Systems Modernization program has been the Customer  Account Data Engine (CADE), which was slated to modernize taxpayer  account processing through replacement of the legacy Individual Master  File, a 40-year old sequential, flat-file master file processing system for  individual taxpayers. In August 2008, IRS began defining a new  strategy—referred to as CADE 2—which would build on the progress that  the current CADE processing platform had created and leverage lessons  learned to date.
IRS plans to deliver CADE 2 functionality incrementally through three  phases: (1) Transition State 1, (2) Transition State 2, and (3) Target  State.
Operations start date:  January 2012 (estimated completion date for  Transition State 1)
Transition State 1 consists of the following two projects:    Daily processing: This project is to enable IRS to process and post all  eligible individual taxpayer returns filed and other transactions by  updating and settling individual taxpayer accounts in 24 to 48 hours  with current, complete, and authoritative data, and provide employees  with timely access.    Database implementation: This project is to establish the CADE 2  database, a relational database that will house data on individual  taxpayers and their accounts; develop a capability to transfer data  from the Individual Master File to the database; and provide for the  access of data from the database to downstream IRS financial,  customer service, and compliance systems.
In April 2011, IRS completed the Transition State 1 detailed design  phase, which includes activities such as documenting the physical design  of the solution. For purposes of this report, we focused only on the IRS’s  efforts on Transition State 1 through the completion of the detailed design  phase.
In March 2011, we reported that although IRS had taken some positive  steps on defining benefits, estimating costs, and managing risks for  CADE 2, it did not fully identify and disclose the CADE 2 costs and  benefits. Specifically, we reported that    although IRS had identified benefits for the first phase of CADE 2, it  had yet to set quantitative targets for 5 of the 20 identified benefits,  and had yet to finalize the benefits expected in Transition State 2 or  define related quantitative targets;    although IRS’s process for developing preliminary life-cycle cost  estimates was generally consistent with best practices, the agency did  not perform all practices associated with credible cost estimates;  the schedule for delivering the initial phase of CADE 2 was ambitious;  and  IRS’s process for managing the risks associated with CADE 2 was  generally consistent with best practices.
Our recommendations included (1) identifying all of the benefits  associated with CADE 2, setting the related targets, and identifying how  systems and business processes might be affected, and (2) improving the  credibility of revised cost estimates.
section_title: Veterans Affairs Occupational Health Record-keeping System
paragraphs: During the development of the National Flu Plan, which was released in  2006, the White House Homeland Security Council directed VA to  develop an employee health tracking and management system.  According to VA officials, the need for this system became urgent due to  the threat of pandemic influenza in 2007. As a result, the Veterans Health  Administration (VHA), working with VA’s Office of Information and  Technology, developed the Occupational Health Record-keeping System  (OHRS). According to VA officials, OHRS was divided into two  increments. The first increment consisted of a minimum feature set which  represented the functionality that would provide the agency with the  largest return on investment. The first increment became operational in  September 2009. The second increment was intended to add functionality  to the minimum feature set and to address any remaining requirements.  For purposes of our report we focused on the first increment—VA’s efforts  to acquire the minimum feature set. OHRS was developed using Agile  software development—specifically, the Scrum methodology.
OHRS serves as the electronic health record system specifically for VA  employees. OHRS provides the end users (i.e., VHA employees who  work in occupational health offices at VHA healthcare facilities) the ability  to collect and monitor clinical data on its employees (e.g., specific  immunizations and medical training) and generate reports. Additionally, a  VA official stated that OHRS allows physicians to document a number of  health issues related to the workforce, including training and infectious  disease management. Among other things, the information in this system  is used to allocate staff to appropriate patient care assignments. For  example, the system can identify whether a provider has received a  vaccine for a certain illness and is therefore able to treat a patient with  that illness.
section_title: Nine Factors Were Commonly Identified as Critical to the Success of Major IT Investments
paragraphs: Nine factors were identified as critical to the success of three or more of  the seven IT investments. The factors most commonly identified include  active engagement of stakeholders, program staff with the necessary  knowledge and skills, and senior department and agency executive  support for the program. These nine critical success factors are  consistent with leading industry practices for IT acquisitions. Table 2  shows the nine factors, and examples of how agencies implemented  them are discussed below.
Officials from all seven selected investments cited active engagement  with program stakeholders—individuals or groups (including, in some  cases, end users) with an interest in the success of the acquisition—as a  critical factor to the success of those investments. Agency officials stated  that stakeholders, among other things, reviewed contractor proposals  during the procurement process, regularly attended program  management office sponsored meetings, were working members of  integrated project teams, and were notified of problems and concerns  as soon as possible. For example:    Census officials stated that the DRIS stakeholders were members of  the integrated project team. Their responsibilities as members of the  team included involvement in requirements development, participation  in peer reviews of contractual deliverables, and review of contractor  proposals.
IRS officials told us that consistent and open communication with  internal and external stakeholders has been critical to the success of  CADE 2. For example, IRS officials told us that they regularly report  progress made on CADE 2, as well as risk information on the program  to oversight bodies, IRS executives, and IRS internal stakeholders.
In addition, officials from two investments noted that actively engaging  with stakeholders created transparency and trust, and increased the  support from the stakeholders. For example, NNSA officials noted that  notifying MOMentum stakeholders of potential issues as soon as they  were identified helped to foster transparency and trust; this included  getting stakeholders’ approval to use a cost- and schedule-tracking  approach that was not the agency’s policy, but which ultimately saved the  program money and time. Additionally, CBP officials noted that  communication with the WHTI stakeholders was greatly enhanced by the  use of a consistent message that described, for example, the goals of the  program, deployment plans, privacy implications of the Radio Frequency  Identification infrastructure, and impact of the program on select groups  crossing the border, including U.S. and Canadian children and Native  Americans. CBP officials stated that this standardization created a  consistent, unified vision and ensured that the message stayed on  course.
Consistent with this factor, relevant guidance calls for programs to  coordinate and collaborate with stakeholders in order to address their  concerns and ensure that they fulfill their commitments. Active  engagement with stakeholders increases the likelihood that the program  will not encounter problems resulting from unresolved stakeholder issues.
subsections: section_title: Program Staff Had the Necessary Knowledge and Skills
paragraphs: Officials from six of the seven selected investments indicated that the  knowledge and skills of the program staff were critical to the success of  the program. This included knowledge of acquisitions and procurement  processes, monitoring of contracts, large-scale organizational  transformation, Agile software development concepts, and areas of  program management such as earned value management and technical  monitoring. For example:  IRS officials stated that the Treasury Secretary utilized his critical  position pay authority to hire executives for CADE 2 who had  demonstrated success in managing large-scale transformation efforts  in accordance with best practices. Specifically, IRS officials stated that  the CADE 2 program manager was previously responsible for the  design, development, and implementation of several major global  information technology solutions for a major corporation.    CBP officials explained that a factor critical to the success of the  acquisition was that almost every member of the team working on  WHTI had a good understanding of acquisitions—some even held  acquisition certifications—in addition to their understanding of  program management. According to those officials, these skills  contributed to effective program oversight of the WHTI contractors  through all phases of the acquisition, not just during contract award.
Additionally, officials from three of the seven investments also cited the  use of subject matter experts’ knowledge in their cognizant areas as a  contributing factor to their programs’ successes. For example, VA officials  stated that the OHRS program relied extensively on the subject matter  experts’ occupational health experience—treating them as part of the  development team and including them in decision making. Two  investments in our sample even went one step further—by selecting the  program manager from the end user organization as opposed to an  individual with an IT background. For example, NNSA officials stated that  they used a project manager from the end user organization as opposed  to an individual from the department’s information technology office. This  individual had decades of experience managing shop floor control  systems. As a result, he was well aware of how the work on the shop floor  is done and focused on safely delivering the necessary functional  requirements to the end user.
Leading guidance also recognizes that programs should ensure that  program staffs acquire the knowledge and skills needed to perform the  project. Individuals who have developed the knowledge and skills  needed for the programs are more likely to perform their roles effectively  and efficiently.
section_title: Senior Department and Agency Executives Supported the Program
paragraphs: Officials from six of the seven selected investments identified support  from senior department and agency executives as critical to the success  of their programs. According to those officials, these senior leaders  supported the success of these programs in various ways, such as by  procuring funding, providing necessary information at critical times,  intervening when there were difficulties working with another department,  defining a vision for the program, and ensuring that end users participated  in the development of the system. For example:    The WHTI program manager told us that the former DHS Deputy  Secretary reached out to another department in order to finalize a  memorandum of understanding that would be used to share  information on passports and passcards needed for WHTI. According  to the WHTI program manager, prior to the Deputy Secretary’s  involvement, the other department’s efforts to collaborate on this issue  were not meeting the schedule requirements of the WHTI program.  That official told us that after receiving the necessary support from the  other department, CBP was able to more rapidly query that  department’s data.
IRS officials explained that endorsement for CADE 2 has come from  the highest levels of the organization. In particular, those officials told  us that the IRS Commissioner has made CADE 2 one of his top  priorities. IRS officials told us that the Commissioner, through, for  example, his keynote speech at a CADE 2 town hall meeting for IRS  employees, has provided a clear and unwavering message about  CADE 2. This speech and other activities have unified IRS  employees, driven change, and removed barriers that can often  impede programs of this magnitude.
In our experience, strong leadership support can result in benefits to a  program, including providing the program manager with the resources  necessary to make knowledge-based, disciplined decisions that increase  the likelihood of their program’s success.
section_title: End Users and Stakeholders Were Involved in the Development of Requirements
paragraphs: Officials from five of seven selected investments identified the  involvement of stakeholders—including end users—in the requirements  development process as a factor that was critical to the success of their  programs. For example:    Census officials told us that the DRIS program management office  collaborated extensively with the stakeholders and the contractor to  develop requirements. For example, program management office  personnel, contractor staff, and the stakeholders all worked together  to analyze the requirements in order to ensure they were understood,  unique, and verifiable.    VA officials told us that an OHRS end user identified a set of  requirements for an occupational health system 3 years prior to the  initiation of OHRS development efforts. Those officials told us that the  developers worked closely with the OHRS end user representative to  ensure that those requirements were still valid once the program was  initiated, given the length of time since the requirements were initially  identified.
Relevant industry guidance recognizes the importance of eliciting end  user needs and involving stakeholders in requirements development.  When stakeholders and end users communicate their requirements  throughout the project life cycle, the resulting system is more likely to  perform as intended in the end user’s environment.
section_title: End Users Participated in Testing of System Functionality Prior to Formal End User Acceptance Testing
paragraphs: Officials from five of the seven selected investments identified having the  end users test and validate the system components prior to formal end  user acceptance testing for deployment as critical to the success of their  program. For example:    DISA officials told us they used a virtual site to connect developers  and end users in online testing of evolving software repeatedly during  the development of GCSS-J. Using the tool, the developers were able  to record the sessions, which was helpful in addressing defects  identified during testing.    CBP created a fully functional test lab facility for the WHTI program at  a mock port of entry test facility constructed at an old private airport in  Virginia. Using this facility, they were able to test the software that  was being developed and the hardware that was being proposed.  Additionally, a core end user group was established and brought to  the facility multiple times a year during the acquisition to test the  forthcoming technology.
Similar to this factor, leading guidance recommends testing selected  products and product components throughout the program life cycle.  Testing of functionality by end users prior to acceptance demonstrates,  earlier rather than later in the program life cycle, that the functionality will  fulfill its intended use. If problems are found during this testing, programs  are typically positioned to make changes that are less costly and  disruptive than ones made later in the life cycle would be.
section_title: Government and Contractor Staff Were Consistent and Stable
paragraphs: Officials from four of the seven selected investments stated that  government and contractor organizations’ personnel were consistent and  stable. For example:    DISA officials indicated that the longevity of the program management  office and contractor staffs has been a contributing factor to GCSS-J’s  success. For example, the longevity of the staff contributed to them  becoming subject matter experts in their areas of responsibility.    CBP officials explained that key program management office staff  remained consistent throughout the WHTI program. In addition,  according to a CBP official, the staffs genuinely liked to work with one  another and were able to collaborate effectively.
This factor is consistent with relevant guidance that espouses the  importance of having adequate and skilled resources. In particular,  having consistent and stable staff can allow teams to keep pace with their  workload, make decisions, and have the necessary accountability.
section_title: Program Staff Prioritized Requirements
paragraphs: Officials from four of the seven selected investments cited the  prioritization of requirements as enabling the efficient and effective  development of system functionality. For example:    FAA officials told us that ITWS end users presented the development  team with a “wish list” of requirements that would help them  significantly. Those officials told us that end users and developers  prioritized those requirements by balancing importance to the end  users with the maturity of the technology. FAA officials stated that  prototypes of these new requirements were developed and evaluated  by end users in the field and were ultimately implemented in the initial  operating capability for ITWS.    DISA officials explained that during development, GCSS-J end user  representatives met with the GCSS-J program office and the GCSS-J  developer twice a week for between a full and a half day in order to  identify and prioritize requirements. Those officials explained that this  frequent interaction was necessary because of the short development  iterations (4 to 5 weeks), at the end of which useable functionality was  presented to the end users for review. The frequent prioritization  ensured that the functionality most critical to the end user  representative was developed, and could be deployed sooner than  functionality of less importance.
Consistent with leading guidance, having prioritized requirements guides  the programs in determining the system’s scope and ensures that the  functionality and quality requirements most critical to the end users are  deployed before less-desired requirements.
section_title: Program Officials Maintained Regular Communication with the Prime Contractor
paragraphs: Officials from four of the seven selected investments indicated that  regular communication between the program management office and the  prime contractor was critical to the success of the program. This  communication was proactive in that there were regularly scheduled  meetings between the program management office and the prime  contractor, with an expectation of full and honest disclosure of problems.  For example:    Census officials stated that the DRIS program management office  took a proactive, “no surprises” approach to communicating with the  contractor. For example, on a monthly basis, the program  management office formally documented the technical performance of  the contractor based on the relevant elements of the work breakdown  structure and the award fee plan. These reports were provided to  the contractor, who in turn used the feedback to improve its technical  performance. In addition, DRIS program managers and their  contractor counterparts met weekly to discuss significant issues. DRIS  officials emphasized that the expectation of open communication and  trust from senior leadership fostered an environment where issues  could be freely discussed with the contractor.    CBP officials stated that during the deployment of the WHTI  technology to the ports of entry, the program management office held  daily conference calls with the contractor to ensure proper  coordination and the rapid resolution of problems. For example,  during deployment to one port of entry it was determined that the  electric system that provided power to the lanes was not adequate.  This problem was quickly identified, responsibility for resolving it was  assigned, and the issue was quickly resolved.
Additionally, Census and VA officials stated that ensuring a positive, non- adversarial relationship between the prime contractor and the program  management office was critical to the success of the investment. Census  officials noted that both the government and the contractor staff  recognized that the only way for the program to succeed was for both  parties to succeed.
Consistent with this factor, leading guidance recognizes the importance of  communication between program officials and the contractor  organizations. Implementation of this critical success factor enables  programs to ensure that requirements are understood and risks and  issues are identified and addressed earlier rather than later in the  process, thereby increasing the likelihood that the delivered system will  meet its intended purpose and resulting in less costly and less disruptive  changes and work efforts.
section_title: Programs Received Sufficient Funding
paragraphs: Officials from three of the seven selected investments explained that  sufficient funding for the programs contributed to the success of those  investments. Officials from two of the investments attributed funding to  strong congressional support; in a third case, officials cited strong  leadership from senior agency and program officials as being a factor. For  example:    The WHTI program manager stated that the WHTI program received  the requested funding from Congress for the 2 years leading up to the  June 1, 2009, mandated implementation date. Additionally, that official  told us that Congress provided 2-year money, that is, money that  could be obligated over a period of 2 years. Officials told us that the 2- year money gave the program great flexibility to accommodate the  inherent complexities and expenditures incurred in a multiyear  deployment, and to adapt to inevitable modifications in deployment  requirements (that is, additional sites, lanes, and functionality).
IRS officials told us that the IRS Commissioner helped the CADE 2  program obtain funding. For example, those officials told us that the  IRS Commissioner spoke with congressional representatives  frequently in order to sustain interest and support for CADE 2.
Relevant guidance recognizes the importance of sufficiently funding IT  investments. Investments that receive funding commensurate with their  requirements are better positioned to ensure the availability of needed  resources, and therefore, deliver the investment within established goals.
The nine commonly identified critical success factors are consistent with  OMB’s 25-point plan to improve IT management and oversight. In  particular, one high-level objective of the plan—effectively managing  large-scale IT programs—aims to improve areas that impact the success  rates of large IT programs across the federal government. As part of this  high-level objective, the plan addresses the importance of ensuring that  program management professionals have extensive experience and  training, defining requirements by engaging with stakeholders, and  providing senior executives with visibility into the health of their IT  programs. These principles of effective IT management are reflected in  the commonly identified critical success factors. For example, as  previously mentioned, six of the seven agencies identified the knowledge  and skills of program staff and five of seven agencies cited the  involvement of end users and stakeholders in the development of  requirements as critical to the success of their IT investments.
While our analysis of critical success factors identified by agencies  resulted in nine commonly identified factors, agencies also identified  additional factors as contributing to the success of their investments. For  example:    Agile software development: DISA officials stated that the use of Agile  software development was critical to the success of the program.  Among other things, Agile enhanced the participation of the end users  in the development process and provided for capabilities to be  deployed in shorter periods of time.    Streamlined and targeted governance: IRS officials told us that in  comparison to other IRS business systems modernization projects,  the governance model for CADE 2 has been streamlined. For  example, those officials stated that the CADE 2 governance structure  includes an executive steering committee that, in contrast to other  programs at IRS that utilize an executive steering committee, is  dedicated solely to the CADE 2 program. IRS officials told us that this  gives an added measure of accountability and responsibility for the  successful outcome of the program.    Continuous risk management: VA officials stated that the risk  management strategy that the program used was critical to its  success. According to the VA officials, risks were identified at daily  team meetings and mitigation strategies were developed. Furthermore  an official explained that risk management is built in the Agile software  development process by, for example, involving the end user early  and often to ensure that the requirements were as thoroughly vetted  as possible.
Several of these factors are also consistent with best practices, such as  the critical factors relating to risk management and governance. The full  list of critical success factors and how agencies implemented them are  presented in appendix II.
section_title: Concluding Observations
paragraphs: Although the critical success factors identified by the seven agencies  were cited as practices that contributed to the success of their  acquisitions, implementation of these factors will not necessarily ensure  that federal agencies will successfully acquire IT systems because many  different factors contribute to successful acquisitions. Nevertheless, the  examples of how agencies implemented the critical success factors may  help federal agencies address the well-documented acquisition  challenges they face. Moreover, the critical success factors in this report  also support OMB’s objective of improving the management of large- scale IT acquisitions across the federal government, and wide  dissemination of these factors and how agencies implemented them  could complement these efforts.
section_title: Agency Comments and Our Evaluation
paragraphs: We received written, e-mail, or verbal responses on a draft of this report  from all seven departments in our review as well as OMB. These  responses are summarized below.    The Acting Secretary for the Department of Commerce provided  written comments. The department stated that the report provides a  good overview and assessment of governmentwide critical factors and  elements that led to the successful acquisition of IT investments. The  department also provided technical comments, which we incorporated  as appropriate.    An acquisition analyst from the Department of Defense CIO  Acquisition Directorate, writing on behalf of the department, provided  an e-mail, which stated that the department had no comments on the  draft report.    The Director of the NNSA’s Office of Internal Controls, responding on  behalf of the Department of Energy, provided an e-mail stating that  they agreed with the report and had no further comments. They also  noted that the department is committed to supporting OMB’s objective  of improving the management of large-scale IT acquisitions, and that  wide dissemination of the factors in our report could complement  OMB’s efforts.    The Director of DHS’s Departmental GAO/Office of Inspector General  Liaison Office provided written comments. In its comments, the  department noted that it remains committed to continuing its work with  OMB to improve the oversight and management of IT investments to  help ensure that systems are acquired on time and within budget, and  that they deliver the expected benefits and functionality. The  department further stated that it will use this report to enhance and  improve the factors critical to the successful acquisition of the  department’s investments, such as creating a structured training  program to assist in obtaining certification in the program  management career field, and conducting reviews to provide insight  into the cost, schedule, and performance of IT investments. The  department also provided technical comments, which we incorporated  as appropriate.    The Deputy Director of Audit Relations within the Department of  Transportation’s Office of the Secretary provided an e-mail with  technical comments, which we incorporated as appropriate.    A program analyst within the Office of the Chief Information Officer for  the Department of the Treasury, writing on behalf of the department,  provided an e-mail, which stated that the department had no  comments on the draft report.    The Department of Veterans Affairs Chief of Staff provided written  technical comments, which we incorporated as appropriate.    A policy analyst from OMB’s Office of E-Government and Information  Technology, speaking on behalf of OMB, provided verbal technical  comments, which we incorporated as appropriate.
As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution of it until 30 days from  the date of this letter. At that time, we will send copies of this report to  interested congressional committees; the Director of OMB; the  secretaries and agency heads of the departments and agencies  addressed in this report; and other interested parties. In addition, the  report will be available at no charge on the GAO website at  http://www.gao.gov.
If you or your staff members have any questions on the matters discussed  in this report, please contact me at (202) 512-9286 or pownerd@gao.gov.  Contact points for our Offices of Congressional Relations and Public Affairs  may be found on the last page of this report. GAO staff who made key  contributions to this report are listed in appendix III.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to (1) identify federal information technology (IT)  investments that were or are being successfully acquired and (2) identify  the critical factors that led to the successful acquisition of these  investments.
To address our first objective, we selected 10 departments with the  largest planned IT budgets as reported in the Office of Management and  Budget’s (OMB) fiscal year 2011 Exhibit 53. Collectively, these  departments accounted for 88 percent of the federal government’s  requested total IT budget for fiscal year 2011. We then asked the chief  information officers (CIO) and other acquisition and procurement officials  from the departments to select one major, mission-critical IT investment  that was, preferably, operational and that best achieved its cost,  schedule, scope, and performance goals. Seven departments—the  Departments of Defense, Commerce, Energy, Homeland Security,  Transportation, the Treasury, and Veterans Affairs—identified successful  IT investments. Collectively, these departments accounted for 73  percent of the planned IT spending for fiscal year 2011.
To address our second objective, we interviewed officials responsible for  each investment, asking them to identify and describe the critical factors  that led to their success, and to provide examples where possible. We  validated our understanding of the factors and examples collected during  the interviews by providing written summaries to agency officials to  ensure that their information was accurately portrayed. Because of the  open-ended nature of our discussions with officials, we conducted a  content analysis of the information we received in order to identify  common critical success factors. We then totaled the number of times  each factor was mentioned by department and agency officials, choos to report on the critical success factors that were identified by three or  more investments. This resulted in our list of nine commonly iden tified  critical succes s factors. We then compared these nine critical success  factors to leading industry practices on IT acquisitions, such as the  Software Engineering Institute’s (SEI) Capability Maturity Model®  Integration (CMMI®) for Acquisition, the Project Management Institute’s A ’s  Guide to the Project Management Body of Knowledge, and GAO Information Technology Investment Management: A Framework for  Assessing and Improving Process Maturity. Finally, we compared the  nine commonly identified critical success factors to OMB’s 25 Point  Implementation Plan to Reform Federal Information Technology  Management in order to determine whether those critical succes are related to the high-level objectives found in the plan.
section_title: Appendix II: Critical Success Factors
paragraphs: The following seven tables provide a description of critical success factors identified by officials with each of the investments in our sample.
section_title: Appendix III: GAO Contact and Staff Acknowledgments
subsections: section_title: G
section_title: Staff Acknowledgments
paragraphs: dition to the contact named above, Deborah A. Davis (Assistant  tor), Kaelin P. Kuhn, Lee McCracken, Thomas E. Murphy, Jamelyn  n, and Andrew Stavisky made key contributions to this report.