Defense Acquisitions: Further Management and Oversight Changes Needed for Efforts to Modernize Cheyenne Mountain Attack Warning Systems
Published: 2006-07-06T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Cheyenne Mountain Operations Center houses numerous complex computer systems for tracking air, missile, and space events that could threaten homeland security or undermine military operations in theater. To ensure this mission can be met, the systems require ongoing upgrades. The most recent upgrade program--the Combatant Commanders' Integrated Command and Control System (CCIC2S)--was initiated in 2000. Given the critical missions supported by Cheyenne Mountain systems, GAO initiated a review to (1) determine the status of the CCIC2S program in terms of meeting its cost, schedule, and performance goals; (2) gauge the extent to which DOD has followed best practices in managing program requirements; and (3) assess DOD's control and oversight mechanisms for CCIC2S.
section_title: What GAO Found
paragraphs: Like its predecessor, the Department of Defense's (DOD) CCIC2S program is over cost, behind schedule, and some capabilities have been deferred indefinitely which could pose risks to performing some future operations. The Air Force, which has overall responsibility for the program, currently estimates program costs will total about $707 million through fiscal year 2006--about a 51 percent increase over initial estimates. Schedules have also expanded significantly, and most critical mission capabilities will not be delivered in fiscal year 2006, as initially planned. The deferral of capabilities and performance shortfalls has significant implications for future missions--especially if program dollars are needed to maintain legacy systems longer than expected. The tracking of space objects could be particularly affected, given that none of the work on CCIC2S's critical space mission capabilities has been completed, and estimated completion dates for this work have yet to be determined. Unstable program requirements and the failure to match requirements to available resources have contributed to the program's cost and schedule overruns. Since the program began in 2000, the Air Force has added, deleted, and modified requirements without adequately determining the effect of these changes on resources. To control cost growth, the Air Force has frequently deferred work to later years and has yet to determine when and at what cost development is to be completed. Several key controls needed to mitigate the CCIC2S program's cost and schedule problems are not in place. First, DOD did not designate the CCIC2S program as a major automated information system acquisition--a designation that would have required high-level oversight other than that provided by the Air Force, which has been ineffective. In addition, the Air Force's contracting approach has limited the program's ability to thoroughly assess the reliability of the contractor's cost and schedule performance information and the impact of defining, prioritizing, and adding capabilities. Despite this risky approach, the Defense Contract Management Agency did not independently monitor contractor performance. According to DOD officials, actions are being taken to implement better controls and to determine whether the CCIC2S program should be categorized as a major automated information system acquisition.

section_title: Letter
paragraphs: The Department of Defense’s (DOD) Cheyenne Mountain Operations  Center, built in the early 1960s, currently houses numerous complex  computer systems intended to help monitor, process, and interpret air,  missile, and space events that could threaten North America or have  operational impacts on U.S. forces or capabilities. New threats have  emerged over time that have necessitated improved capabilities from  Cheyenne Mountain systems. Examples of these evolving threats include  events such as the September 11, 2001, terrorist attacks on the U.S. and  attacks on space assets that could negatively impact military operations as  well as the economy.
These systems—which are housed in three major centers: Air Warning,  Missile Correlation, and Space Control—require ongoing upgrades in order  to incorporate new mission capabilities and technologies and reduce the  cost of maintaining older systems. However, our reviews over the past   2 decades have found that DOD’s efforts to modernize and integrate  Cheyenne Mountain systems have been fraught with cost increases,  schedule delays, and performance shortfalls due, in large part, to poor  program management and oversight. For example, in September 1994, we  reported that upgrade efforts were 8 years behind schedule and   $792 million over budget, due to development and integration problems  stemming from management shortfalls.
While DOD declared the most recent Cheyenne Mountain upgrades to be  operational in 1998, that same year, it determined that some of the  systems’ components were not well integrated, were becoming  unsupportable because they were no longer produced, and would be  unresponsive to future mission needs. Subsequently, DOD initiated a  program in 2000 to modernize and integrate Cheyenne Mountain systems  under a program called the Combatant Commanders’ Integrated Command  and Control System, or CCIC2S, and assigned overall responsibility for the  program to the Air Force.
We have recently testified that DOD’s costly current and planned  acquisitions are running head-on into the nation’s unsustainable fiscal  path. DOD starts more programs than it can afford and sustain, and in the  past 5 years, it has doubled its planned investments in programs.   However, these programs continue to experience recurring problems with  cost overruns, missed deadlines, and performance shortfalls. Our work has  shown that DOD will continue to experience such problems until it is able  to make oversight and management control improvements, including  making sure programs are executable, locking in requirements before  programs are initiated, and making clear who is responsible for what and  holding people accountable when these responsibilities are not fulfilled.  Additionally, our work has shown that once a program is initiated,  inevitable changes to the requirements baseline need to be controlled in  terms of assessing the cost, schedule, and performance implications of the  changes. As part of these assessments, the risks associated with these  factors need to be identified and mitigated.
Within this context, we initiated this engagement under the authority of  the Comptroller General of the United States to (1) determine the status of  the CCIC2S program in terms of meeting its cost, schedule, and  performance goals; (2) gauge the extent to which DOD has followed best  practices with regard to managing program requirements, including  matching requirements to available resources; and (3) assess DOD’s  oversight and control mechanisms for Cheyenne Mountain systems  modernization and integration efforts under way and planned. We are  addressing this report to you because of your committees’ and  subcommittees’ jurisdictions.
In conducting our work, we spoke with officials from appropriate DOD,  Air Force, and contractor offices. We also reviewed DOD and Air Force  acquisition policies; planning documents; and program requirements, cost,  and schedule data. Additional information on our scope and methodology  is in appendix I. We conducted our work from July 2005 to April 2006 in  accordance with generally accepted government auditing standards.
subsections: section_title: Background
paragraphs: DOD built the Cheyenne Mountain Operations Center in the early 1960s at  Cheyenne Mountain Air Force Station, Colorado. Throughout its history,  its mission has continually evolved to adapt to changing threats—which  have ranged from a perceived Soviet nuclear threat after the 1957 launch  of Sputnik, the first man-made satellite successfully placed in orbit, to  theater ballistic missiles in the Persian Gulf during Operation Desert Storm  in the early 1990s. Currently, Cheyenne Mountain’s mission is to help  monitor, process, and interpret air, missile, and space events that could  threaten North America or have operational impacts on U.S. forces or  capabilities, using air, ground, and space-based sensors that link to the  complex’s computer systems located more than 2,000 feet under ground.  The President, Secretary of Defense, combatant commanders, and the  Prime Minister of Canada rely on Cheyenne Mountain systems to provide  accurate, timely, and unambiguous information essential for determining  courses of actions warfighters are to carry out. Air Force Space Command  is responsible for operating, funding, and setting requirements for the  systems, and Air Force Materiel Command’s Electronic Systems Center is  responsible for maintaining the systems and for acquiring new systems  and capabilities.
Table 1 provides a summary of Cheyenne Mountain’s three mission  centers, operations, and systems.
subsections: section_title: Previous Cheyenne Mountain Upgrade Efforts
paragraphs: In 1981, the Air Force began efforts to modernize its Cheyenne Mountain  systems under five separate programs scheduled to be completed in 1987  at a total estimated cost of $968 million. Subsequently, the Air Force  consolidated the five separate Cheyenne Mountain upgrade programs into  a single Cheyenne Mountain Upgrade (CMU) program. Beginning in the  late 1980s, we issued a dozen reports on the program’s cost, schedule, and  performance problems. For example, we reported that the Air Force did  not adequately define program requirements which resulted in an  underestimation of the upgrade efforts’ cost and schedule goals. In its  attempts to maintain short-term cost and schedule goals, the Air Force  continually deferred requirements and problem resolution to the future, as  well as relaxed performance requirements. (See Related GAO Products at  the end of this report.)
Despite not meeting original performance expectations for the program, in  1998, the Air Force declared the Cheyenne Mountain upgrades to be  operational—almost $1 billion over cost and 11 years late. That same year,  DOD determined that some of the systems were not well integrated, would  be unresponsive to future mission needs, and some components were  becoming unsupportable. As a result, in 2000, the Air Force initiated the  CCIC2S program to modernize and integrate Cheyenne Mountain systems.  Figure 1 depicts the timeline of Cheyenne Mountain upgrades.
On September 19, 2000, the Air Force awarded a cost-plus award fee  contract to Lockheed Martin Mission Systems to begin work on the  CCIC2S program. The initial period of performance was for 6 years, with  annual options for operations and maintenance activities for an additional  9 years.
section_title: CCIC2S Program Has Experienced Cost and Schedule Overruns and Performance Shortfalls
paragraphs: Like its predecessor, the CMU program, the CCIC2S program is over cost,  behind schedule, and some capabilities have been deferred indefinitely.  This deferral could pose risks to performing some future operations. The  Air Force’s current program cost estimates through fiscal year 2006  represent about a 51 percent increase over its initial estimate. Additionally,  while the Air Force initially estimated CCIC2S upgrades to be completed  in fiscal year 2006, most critical mission capabilities will not be delivered  by this time, and for some of these capabilities, the Air Force has yet to  estimate a completion date. The deferrals of capabilities and performance  shortfalls could have implications for future missions.
subsections: section_title: The CCIC2S Program Is Over Cost and Behind Schedule
paragraphs: As shown in table 2, the Air Force initially estimated the CCIC2S program  to complete upgrades in fiscal year 2006 at an estimated cost of about   $467 million. However, program cost estimates through the same year  have increased about 51 percent to nearly $707 million. Air Force officials  said they never estimated the life-cycle cost of delivering all CCIC2S  validated requirements.
Along with the increases in estimated program costs, schedules have  expanded significantly. The Air Force initially estimated development  activities to be completed in fiscal year 2006, but only a fraction of the  expected capabilities have been delivered (see fig. 2). Of the three mission  areas—air, missile, and space—the Air Force has delivered only initial air  and some initial missile mission critical capabilities. DOD considers these  critical capabilities to be the most essential for the program—failure to  provide these capabilities can be cause for a program to be reassessed or  terminated. The remaining missile warning critical capabilities are now  expected to be delivered in fiscal year 2007—over 3 years later than  initially estimated. The Air Force has postponed the delivery of all critical  space mission capabilities to some undetermined point in the future.  According to Air Force officials, the space mission capability deliveries  will not be scheduled until future investment decisions are made.
According to the CCIC2S program office and other DOD officials, one of  the reasons the program’s initial cost and schedule estimates differ from  current estimates is that work that was not included in the original  estimates was added to the program. Examples of work added to the  program include replacing an aging missile warning processing and display  system as well as developing a computer operational status monitoring  system. We did not assess the cost and schedule impacts of these added  capabilities. Also, as a result of the September 11, 2001, attacks on the  U.S., the Air Force diverted a portion of CCIC2S funding and personnel to  make improvements to other Cheyenne Mountain systems so that aircraft  traffic within the U.S. could be monitored by the air warning center.  However, according to DOD officials, the impacts of the September 11,  2001, terrorist attacks on the CCIC2S program were not significant  because monitoring aircraft traffic within the U.S. is not a required  upgrade under the CCIC2S program. Prior to these improvements, the  focus of the air warning system was to monitor aircraft traffic outside the  North American border.
section_title: Deferral of Capabilities and Performance Shortfalls Have Implications for Future Missions
paragraphs: The deferral of capabilities and performance shortfalls that have resulted  from cost and schedule overruns in the CCIC2S program could  significantly impact future missions—especially if program dollars are  needed to maintain legacy systems longer than expected.  USSTRATCOM—responsible for protecting U.S. space assets—could be  particularly affected, given that none of the work on CCIC2S’s critical  space mission capabilities has been completed and that estimated  completion dates for this work have yet to be determined.
DOD has been counting on CCIC2S to replace the Space Defense  Operations Center (SPADOC)—Cheyenne Mountain’s current space object  tracking system—and provide a system capable of processing larger  volumes of data. Some DOD officials have stated that CCIC2S is to be the  foundation for carrying out future space awareness functions. SPADOC’s  capabilities are currently overtaxed. It currently handles about 400,000  space object observations per day from sensors—about 167 percent more  than it was designed to handle. The Air Force has implemented an  inefficient workaround to address SPADOC’s limited processing  capabilities. Specifically, the Space Control Center is using CaveNet, an  off-line tool, to assist with analyzing and processing space object  observations. However, to analyze and process the information in  CaveNet, Space Control Center operators must first download information  from SPADOC and then manually type the processed information back  into SPADOC. Air Force officials stated that manually typing CaveNet  information back into SPADOC has recently become significant enough to  request an automated interface.
According to Air Force officials, efforts are on-going to assess the mission  impact of SPADOC performing beyond design specifications. With other  programs to improve data collection sensors that feed data into CCIC2S— such as the Space Fence and Space Based Space Surveillance system— the number of observations and processing requirements will increase  significantly in the future, taxing the current system further. In a 2005  study, the CCIC2S contractor determined that SPADOC’s processing  capacity will not be able to handle the increased number of observations  predicted beginning in 2012. For the near term, according to Air Force  officials, the CCIC2S contractor has identified operations and maintenance  changes for SPADOC to handle increases in the number of observations.  However, DOD has not made a decision on when to pursue upgrades for  SPADOC and cost estimates for upgrading SPADOC have not been  finalized. With the delay of CCIC2S space capabilities and no decision on  when SPADOC will be replaced, USSTRATCOM will have to continue  relying on pushing SPADOC to its limits.
Delays in the delivery of expected CCIC2S capabilities also affect  programs that are not directly part of the program but are relying on  CCIC2S for their implementation. For example, the Air Force’s Rapid  Attack Identification Detection and Reporting System (RAIDRS)—a  defensive counterspace system—is expected to use information provided  by Cheyenne Mountain systems to aid in the detection, reporting,  identification, location, and classification of attacks against valuable space  assets. However, according to a DOD official, the CCIC2S space capability  delays have presented a serious acquisition dilemma for the RAIDRS  program because it is dependent on the CCIC2S space segment. Funding  has not been made available for the CCIC2S space segment that is to  support RAIDRS, according to Air Force Space Command. Furthermore,  other programs will experience the same acquisition dilemma because Air  Force Space Command has instructed all of its sensor, satellite, and  weapon programs to be able to integrate with CCIC2S.
section_title: Cost and Schedule Overruns Caused by Not Adhering to Best Practices for Managing Program Requirements
paragraphs: The CCIC2S program’s failure to match requirements and resources before  its start has contributed to the cost and schedule overruns that have  forced the program to defer delivery of CCIC2S capabilities. For each year  since the program’s inception, the Air Force has rebaselined CCIC2S’s cost  and schedule goals. The unreliable cost and schedule estimates that  resulted have forced the program to frequently defer work to later years to  control cost growth. The Air Force has yet to determine when and at what  cost development is to be completed.
subsections: section_title: Match Not Made between Program Requirements and Available Resources
paragraphs: Our body of work on best practices has shown that matching program  requirements to resources—that is, time, funding, technology, and  personnel—is key to successful outcomes. We have also reported that the  requirements-setting phase is one of the most important for software- intensive acquisitions. Missing, vague, or changing requirements tend to  be a major cause of poor outcomes in these programs. For example,  according to Air Force officials, the major causes for the delay in the Air  Mission Evolution system—a CCIC2S system that was delivered in  January 2004, over 2 years behind schedule—included problems with  software development and integration. Additionally, the contractor stated  that changing priorities within the program as well as added requirements  contributed to the delay.
Without a stable set of requirements, the Air Force could not appropriately  estimate program costs and in hindsight, the resources needed to satisfy  program requirements were severely underestimated. To better ensure  programs are affordable and fully funded before they are initiated, DOD  guidance calls for every acquisition program to stabilize requirements and  determine the program’s total costs before program start. Despite best  practices and DOD policy, the Air Force initiated the CCIC2S program  without establishing a stable set of program-specific requirements and did  not do so until January 2004—more than 3 years after program start. In  that time, the Air Force made numerous additions, deletions, and  modifications to CCIC2S’s requirements. For example, the Air Force  deleted, added back, and then modified a requirement for multiple security  levels. According to an Air Force review of program requirements, some  deleted requirements were duplicative or conflicting.
Assessments of available resources were also inadequate. According to  DOD guidance, every acquisition program should rigorously address issues  such as the program’s life-cycle costs and whether it is affordable and fully  funded. By not conducting these rigorous assessments, DOD and the Air  Force did not have the information needed to commit to the program over  its life cycle. Although the CCIC2S program acquisition plan refers to the  development of a life-cycle cost estimate before the program was initiated,  it did not include the operations and maintenance costs of sustaining  existing systems through the program’s entire life cycle.
Additionally, software development capabilities of the contractor should  also be assessed. The contract for CCIC2S development specifies that the  contractor be rated at the highest level (level 5) using the Capability  Maturity Model® for Software developed by Carnegie Mellon University’s  Systems Engineering Institute. At this most mature level, quantitative  feedback about performance and innovative ideas and technologies  contribute to continuous process improvements. A senior program official  stated that the CCIC2S contractor was rated at a level 5 for software  development. According to DOD documentation, Capability Maturity  Model® assessments are typically performed at an organization wide level  and may not necessarily apply to a project team associated with any given  acquisition program. We did not examine the program’s software  development efforts and the capabilities of the contractor. For more  information on this model, see appendix III.
Furthermore, DOD has not funded the program at requested levels since  the program began due to other priorities. We have previously reported  that program funding instability often occurs because DOD starts too  many programs, creating severe budget constraints. Another factor we  have reported that contributes to funding instability is that DOD’s  programs are funded annually, requiring competition among other  programs for limited available funding. The CCIC2S program has not  been an exception to these factors. For example, prior to initiating the  CCIC2S program in 2000, DOD reduced the program’s initial estimated  resource requirements by a total of $75 million. In 2002, DOD decided not  to increase requested program funding. According to DOD, funding was  limited to program baseline levels because the program was a lesser  priority. Furthermore, in 2005, DOD and Congress reduced CCIC2S  program funding estimates over the next 6 years by about $135 million,  citing higher priority funding needs and poor program performance.
section_title: The Air Force Has Implemented the CCIC2S Program without Solid Expectations of What Capabilities Are To Be Delivered, When, and at What Cost
paragraphs: According to the acquisition strategy for the CCIC2S program, annual  rebaselines were to be conducted on the program to review and approve  program implementation plans, including cost and schedule baselines.  However, because the Air Force failed to make a match between  requirements and resources, including not effectively assessing the  impacts of adding unanticipated work to the program, the result of  CCIC2S’s annual rebaselines has been to defer significant amounts of  work to the future in order to address cost increases. As a result, the Air  Force has implemented the program as a “level of effort”—that is, without  solid expectations of what capabilities are to be delivered, when, and at  what cost. Currently, the CCIC2S program is undergoing its fifth  rebaseline. According to program officials, after this fifth rebaseline,  CCIC2S will no longer plan for annual rebaselinings, but would only  rebaseline the program when necessary. Figure 3 depicts the effects of  each of the rebaselines on the program.
The fifth and current rebaseline was prompted by efforts to complete  development of the program’s critical missile warning mission capabilities.  To free up resources for the missile warning development efforts, the Air  Force stopped work on many other aspects of the CCIC2S program in late  2004 and subsequently began rebaselining the program in early 2005. Air  Force officials stated the rebaseline was to be finalized by August 2005;  however, it has yet to be approved as of June 2006. According to Air Force  officials, pending changes in program oversight as well as recent DOD and  congressional funding estimate reductions are reasons for the delay in  approving the rebaseline. As currently structured, the rebaseline would  again significantly defer the amount of capability scheduled to be  delivered. Specifically, missile warning critical requirements would not be  delivered until fiscal year 2007, and the schedule for the work on the space  portion is undetermined.
section_title: CCIC2S Program Oversight and Controls Have Been Ineffective
paragraphs: The oversight and control mechanisms of the CCIC2S program have been  inadequate to prevent or mitigate many of the problems it has  experienced. Additionally, the Air Force’s contract management approach  limited the program office’s ability to thoroughly assess the reliability of  the contractor’s cost and schedule performance information and the  impact of defining, prioritizing, and adding capabilities. Furthermore, the  Defense Contract Management Agency (DCMA) did not independently  monitor contractor performance. According to DOD officials, actions are  being taken to implement better controls.
subsections: section_title: DOD Did Not Designate CCIC2S as a Major Acquisition
paragraphs: Placement into a specific acquisition category (ACAT) determines the  accountability level and analysis requirements of a program. DOD policy  requires major automated information system acquisitions to have high- level oversight and comprehensive and independent assessments of life- cycle cost estimates and benefits to ensure they are reasonable and built  on realistic program and schedule assumptions. In determining whether  an automated information system acquisition program warrants a “major”  designation, DOD applies dollar value thresholds. If a program does not  meet these thresholds, it can be designated as a major acquisition based  on other factors such as technical complexity. (See appendix IV for more  detail on the designation criteria for acquisition programs that were in  place when the Air Force initiated the CCIC2S program.)
The CCIC2S program met the dollar threshold criteria for designation as a  major automated information system acquisition program. However, DOD  did not designate the program as a major automated information system  acquisition. According to a senior Air Force Space Command official, the  program was initially considered to be a weapon system acquisition and  therefore below the threshold criteria for designating the program as a  major defense acquisition program. At this lower acquisition category,  oversight for the program was assigned to Air Force headquarters.
Initial cost estimates for CCIC2S indicated that it should have been  designated as a major automated information system acquisition.  Specifically, at program initiation, CCIC2S’s program cost estimate of  about $463 million well exceeded DOD’s threshold of $360 million (fiscal  year 1996 constant dollars)—DOD’s then-threshold for total life-cycle  costs—for designation as a major automated information system program  (see table 3).
Additionally, the CCIC2S program is a technically challenging and  complex acquisition. For example, one of the primary goals of the program  is to integrate 40 stovepiped systems into a single common architecture.  The complexity of the CCIC2S effort is further illustrated by our previous  work which showed that the CMU program had experienced long- standing, serious integration problems, as well as software development  problems with incorporating new technologies.
DOD and Office of Management and Budget have issued guidance on  conducting analyses that can help decision making on major automated  information system acquisitions. These analyses include    an affordability assessment to demonstrate that the program’s  resources (including projected funding and manpower requirements)  are realistic and achievable in the context of the DOD component’s  overall long-range modernization plan to ensure full funding and  funding stability for the program;    an economic analysis to determine the best program acquisition  alternative by assessing the net life-cycle costs and benefits of the  proposed program; and    an independent estimate of program life-cycle costs.
Taken together, such reviews can give DOD the knowledge it needs to  prevent or mitigate many of the types of problems the CCIC2S program  has experienced.
We have consistently reported that the decision to invest in any system  should be based on reliable analyses of estimated system costs and  expected benefits over the life of the program. Without designation as a  major acquisition, the CCIC2S program lacked the high-level oversight and  independent analyses and review that could have identified key program  weaknesses, including    program cost, schedule, and performance problems;  the mismatch between requirements and available resources; and  the inability to prioritize and commit to the program within the context  of other acquisition programs and long-range investment plans.
section_title: CCIC2S Milestone Decision Authority Did Not Provide Effective Oversight
paragraphs: DOD acquisition policy states that overall program responsibility rests  with the milestone decision authority. In this role, the milestone decision  authority is responsible for ensuring accountability and maximizing  credibility in program cost, schedule, and performance reporting by  reviewing the program throughout its acquisition life cycle, including   (1) whenever the program reaches a milestone decision point;   (2) whenever costs, schedule, or performance goals are baselined or must  be changed; and (3) periodically through review of program status reports.
While the Assistant Secretary of the Air Force for Acquisition—the  designated milestone decision authority for the CCIC2S program—played  an active oversight role from 2000 to 2001, its program oversight thereafter  was minimal, including during the numerous program rebaselines and  attempts to address cost, schedule, and performance shortfalls. The  Assistant Secretary’s oversight appears to have been limited to brief  program status updates provided by the program executive officer as part  of a weekly reporting on all programs under the program executive  officer’s purview, and, according to DOD officials, annual and semi-annual  program execution and budget reviews. According to Air Force officials,  the Assistant Secretary delegated its milestone decision authority  responsibilities for the program to the program executive officer level.  However, officials from the Office of the Assistant Secretary of the Air  Force for Acquisition and the Office of the Program Executive Officer  were unable to provide us documentation of a formal delegation of these  responsibilities, or explain why this delegation took place.
section_title: Program Lacked Sufficient Management Controls over the Contractor
paragraphs: In addition to matching requirements to resources, our best practices  work has shown that to better ensure programs deliver capabilities within  expected costs and schedules, program offices must have in place certain  controls, including: (1) assessments of the effects of any changes in  requirements on cost, schedule, and performance goals and (2) insights  into contractor activities and progress to ensure commitments are  implemented. The contract to develop CCIC2S was awarded under a  Total System Performance Responsibility (TSPR) approach, which  transfers certain government management responsibilities from the  program office to the contractor.
In the mid-1990s, DOD began making sweeping changes to its acquisition  policy and procedures to streamline its acquisition infrastructure. One of  these reforms was the introduction of the TSPR concept, which gave a  contractor total responsibility for the integration of an entire weapon  system and for meeting DOD’s requirements. According to DOD officials,  the Air Force no longer uses TSPR for its acquisition programs due to the  shortcomings resulting from using this management concept. We have  previously reported that TSPR contracts often result in the government’s  lacking accurate cost, schedule, and performance information needed to  adequately manage programs because it had to rely on unverified  contractor data.
For CCIC2S, the risks inherent in a TSPR approach were exacerbated by  the program office’s lack of sufficient staffing to oversee the definition and  prioritization of needed capabilities. Because the Air Force reduced the  number of personnel in the program office from 200 to between 15 and   30 in an effort to achieve efficiencies, it lacked the personnel to assess the  cost and schedule impacts of defining and prioritizing requirements on the  program. Without such information, the program office was not in a  position to make trade-off decisions between different types of needed  capabilities. According to the program office, as a result of its inability to  manage the requirements definition and prioritization process, the  contractor ended up communicating directly with the users to help define  and prioritize requirements, sometimes without the knowledge of the  program office.
Program officials also noted that they lacked sufficient staff with the  technical knowledge to determine whether contractor cost, schedule, and  performance data were reliable. For example, according to the program  office and the CCIC2S contractor, the program initially called for a  contractor outside of the CCIC2S program to develop air mission software  and provide it to the CCIC2S contractor for integration into the program.  However, shortly after the Air Force initiated the CCIC2S program, it  terminated the air mission software development contract due to  unsatisfactory performance. Subsequently, the users recommended, and  the program office added, the unfinished work to the CCIC2S program as  part of the re-baseline process in January 2001. However, according to the  program office, it was not able to effectively assess the cost and schedule  impacts of this change on the program.
TSPR does not relieve the government of its oversight responsibilities.  Also, according to DCMA officials, the government should conduct  contract surveillance and gauge contractor performance using specific  performance metrics. However, the CCIC2S program office was not  always aware of the contractor’s decisions or performance. For example,  program office officials acknowledged that they were not aware of some  re-prioritization of work done by the contractor to meet the users’ needs.
section_title: Program Lacked Independent Contractor Performance Assessment
paragraphs: Acquisition programs commonly use DCMA to monitor higher risk  contracts. Typically, after DOD awards a contract, DCMA can monitor  contractors’ information systems to ensure that cost, performance, and  delivery schedules are in compliance with the terms and conditions of the  contract. According to DCMA officials, however, their involvement in the  CCIC2S program from inception through 2004 was minimal because  DCMA lacked resources and the CCIC2S program office did not request  DCMA assistance until that time. This is despite the fact that in January  2003, the Air Force and DCMA signed a memorandum of agreement calling  for DCMA assistance on the contract for the CCIC2S program.
The agreement called for DCMA to analyze the contractor’s earned value  management (EVM) systems to verify that monthly cost performance  reports and related documents present a valid picture of contract cost and  schedule progress. This type of analysis could have identified problems  with the EVM system and led to corrective actions sooner. Since 2004,  DCMA has been working with the program office and Air Force systems  engineering officials to improve the quality of metrics to assess cost,  technical performance, and risk, as well as to make improvements to the  contractor’s EVM system.
section_title: Needed Management Controls and Redesignation of CCIC2S Program Are Being Considered
paragraphs: DOD is reconsidering CCIC2S’s current designation as a non-major  automated information system program. In September 2005, the Assistant  Secretary of Defense for Networks and Information Integration notified  the Assistant Secretary of the Air Force for Acquisition that it intended to  recategorize the CCIC2S program as a major automated information  system acquisition because of increased funding levels, national interest in  the program, and our review of the program. Designating the CCIC2S  program as a major automated information system acquisition would put  in place the high-level oversight and rigorous analyses needed to help  ensure (1) the program has top-level accountability and support for the  program, (2) DOD has the knowledge necessary for making trade-offs in  program requirements so that they match available resources, and   (3) DOD has the ability to prioritize and commit to the program within the  context of its other acquisition programs and long-range investment plans.  As of June 2006, however, the office of the Assistant Secretary of Defense  for Networks and Information Integration has yet to formally make a  decision on whether to designate the program as a major automated  information system acquisition.
According to Air Force and DCMA officials, the program office has taken  several steps, which we did not assess, to address some management  control problems, including the following actions:    Prohibited direct contact between the contractor and users—which  they acknowledge had a direct impact on program cost and schedule  performance—to keep the contractor from reprioritizing work to meet  the immediate needs of the users, in 2004. Additionally, the program  office and the contractor implemented an engineering review board  process to evaluate any proposed change in terms of its validity in  relation to validated requirements as well as its impact to program cost  and schedule estimates.  Initiated a User Feedback Control Board process to identify where  deliverables are not meeting users’ needs and assess identified  solutions in terms of their relationship to validated requirements as  well as program cost and schedule implications, beginning in the spring  of 2005.
Adopted a planning approach that is to more realistically account for  program risks through a consideration of limiting factors such as test  resources and the complexities surrounding installing and testing  hardware and software in Cheyenne Mountain’s operational  environment.
Took steps to add technical staff to assess the effects of changes in  requirements on cost, schedule, and performance goals and provide  additional insight into contractor activities.  Implemented a formal system-level technical review process consisting  of milestones, or gates, each of which have entrance and exit criteria.  For example, deficiencies identified under the formal review must be  addressed prior to proceeding into the next development phase.
section_title: Conclusions
paragraphs: Cheyenne Mountain systems—which are to warn the U.S. and Canada of  air, missile, and space attacks—are critical to national security. However,  since the 1980s, DOD’s efforts to modernize these systems have been  fraught with cost increases, schedule delays, and performance shortfalls.  Under the current development effort, the Air Force continues to defer the  completion of key critical space requirements, placing the DOD’s ability to  perform air and missile defense and space situational awareness missions  at risk. Without matching requirements to available resources, the CCIC2S  program will continue to flounder. Failing to conduct this match at the  onset of this complex effort led DOD to develop unrealistic cost and  schedule estimates as well as to over-promise capability. Specifically, the  Air Force did not ensure that the program had stable requirements and a  sound process for clarifying and controlling them, and thus the resources  to satisfy the requirements were severely underestimated. The minimal  top-level and external oversight and accountability of the program  contributed to the problems. Failure to make course corrections now  without correcting the shortfalls will put the program at risk of failing to  achieve promised capability.
section_title: Recommendations for Executive Action
paragraphs: We recommend that the Secretary of Defense direct the Secretary of the  Air Force to maintain Cheyenne Mountain’s essential operation and  maintenance activities and to limit future development activities to those  deemed critical to national security needs until the Department takes the  following five actions:    Designate the program as a major automated information system  acquisition.
Conduct an affordability assessment to demonstrate that the program’s  resource estimates are realistic and achievable in terms of DOD’s  overall long-range modernization priorities and investment plans for  Cheyenne Mountain. Such an assessment would require ensuring the  requirements baseline is verified and validated and making a match  between these requirements and available resources.
Conduct an economic analysis to assess the life-cycle costs and  benefits of the program.
Conduct an independent estimate of program life-cycle costs, to  provide a basis for relying on the cost estimates.
Determine and clearly delineate oversight responsibilities and  accountability for the successful implementation of CCIC2S.
Additionally, the Secretary of Defense should direct the Secretary of the  Air Force to take the following three actions:    Develop and implement an approach that requires a rigorous analysis  of resource impacts of any change in requirements for continuously  making trade-offs between requirements and resources to ensure a  match is maintained.
Ensure management controls are in place so that changes to  requirements are decided on the basis of costs, benefits, risks, and  affordability.
Develop and implement an approach for ensuring program cost and  performance data are reliable.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report.   DOD agreed that the CCIC2S program be designated as a major automated  information system with the Assistant Secretary of Defense for Networks  and Information Integration as the milestone decision authority with  oversight responsibility and accountability for its successful  implementation. In addition, DOD agreed with conducting an affordability  assessment, economic analysis, and independent estimate of life-cycle  costs on future development activities but not with the current activities  due to potential mission impacts. Specifically, DOD stated these  assessments should be conducted on future delivery blocks of the CCIC2S  program, but not on the current block. DOD asserted that its recent   $127 million CCIC2S funding reduction across the Future Years Defense  Program (FYDP) pared down CCIC2S development activities to those  essential to national security needs.
We do not disagree with this approach to the extent that continuing these  development activities is critical to national security. However, continuing  CCIC2S development without establishing a match between requirements  and available resources may also hinder DOD’s ability to satisfy national  security needs because of the increased risk that the program fails to  achieve its promised capability.  While DOD raises the issue of the cost of  delaying the program, it does not address the cost of proceeding as  planned. Our work has shown that in numerous programs, continuing to  spend money on development programs without sufficient knowledge or  clear direction results in cost overruns and schedule delays. Further, given  recent congressional concerns over DOD’s management of CCIC2S,  demonstrating on a timely basis how requirements and available resources  are to be matched while protecting national security is especially  important.  Since an assessment of CCIC2S’s specific operational risks to  national security was beyond the scope of our review, we clarified our  recommendation to read that the Secretary of Defense direct the Secretary  of the Air Force to limit future development activities to those deemed  critical to national security needs until the Department takes the actions  discussed above.
DOD agreed to take actions relating to establishing and implementing  management controls and approaches to analyze the resource impacts of  changes in requirements and to ensure reliable program data. DOD stated  that these changes had already been implemented or are being planned.  During our review, program and DCMA officials told us that some  changes—such as prohibiting the contractor from reprioritizing work  without Air Force approval and implementing an engineering review board  to assess requirements changes—had been made. However, they also  noted that other measures to manage requirements and institute better  management controls—such as increasing program office management  staffing and capabilities, modifying the work breakdown structure and  earned value management system, and implementing a new approach and  metrics to track program progress—were under way or planned. Given  DOD has not fully demonstrated the effectiveness of the changes it has  made or is making, and that it is planning to make other changes, we are  retaining our recommendation.
We are sending copies of this report to interested congressional  committees and the Secretaries of Defense and the Air Force. We will also  provide copies to others on request. In addition, this report will be  available at no charge on the GAO Web site at http://www.gao.gov.
If you have any questions about this report or need additional information,  please call me at (202) 512-4841 (shamesl@gao.gov). Contact points for  our Offices of Congressional Relations and Public Affairs may be found on  the last page of this report. GAO staff who made major contributions to  this report are listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the status of the Combatant Commanders’ Integrated  Command and Control System, or CCIC2S, in terms of meeting its cost,  schedule, and performance goals, we reviewed our past work to determine  whether the CCIC2S program has continued to experience the problems  previous Cheyenne Mountain upgrade efforts had experienced. We also  reviewed program status reports and budget data and spoke with  Department of Defense (DOD) and contractor officials to assess program  cost, schedule, and performance progress to date. Additionally, we spoke  with CCIC2S users to determine their level of satisfaction with CCIC2S  program performance. Specifically, we reviewed documentation from and  interviewed officials in the Office of the Assistant Secretary of Defense for  Networks and Information Integration; National Security Space Office;  Office of the Secretary of Defense, Cost Analysis Improvement Group;  Office of the Secretary of Defense, Program Analysis and Evaluation;  Office of the Director, Operational Test and Evaluation; Office of the Joint  Chiefs of Staff; Defense Contract Management Agency’s CCIC2S Program  Support Team; Office of the Assistant Secretary of the Air Force for  Acquisition; Air Force Space Command; Air Force Air Combat Command;  Office of the Air Force Program Executive Officer for Command and  Control and Combat Support; Air Force Electronic Systems Center; Air  Force Operational Test and Evaluation Center; United States Strategic  Command; United States Northern Command; North American Aerospace  Defense Command; and Lockheed Martin Integrated Systems & Solutions.
To gauge the extent to which DOD has followed best practices with regard  to managing program requirements, including matching requirements to  available resources, we reviewed DOD, Office of Management and Budget,  and Air Force acquisition guidance, as well as our previous best practices  work, to determine criteria for managing requirements. We also reviewed  program requirements documentation and spoke with DOD officials on  how DOD developed requirements for the CCIC2S program. Furthermore,  we reviewed requirements management documentation and spoke with  DOD and contractor officials to assess how the Air Force managed its  requirements, including the process it used for defining and prioritizing the  requirements. Specifically, we reviewed documentation from and  conducted interviews with officials in the Office of the Secretary of  Defense, Cost Analysis Improvement Group; Office of the Secretary of  Defense, Program Analysis and Evaluation; Office of the Director,  Operational Test and Evaluation; Office of the Joint Chiefs of Staff;  Defense Contract Management Agency’s CCIC2S Program Support Team;  Office of the Assistant Secretary of the Air Force for Acquisition; Air Force  Space Command; Office of the Air Force Program Executive Officer for  Command and Control and Combat Support; Air Force Electronic Systems  Center; Air Force Operational Test and Evaluation Center; United States  Strategic Command; United States Northern Command; North American  Aerospace Defense Command; and Lockheed Martin Integrated Systems &  Solutions.
To assess DOD’s oversight and control mechanisms for Cheyenne  Mountain systems modernization and integration efforts underway and  planned, we reviewed DOD and Air Force acquisition guidance and  interviewed officials in the Office of the Assistant Secretary of Defense for  Networks and Information Integration; Office of the Secretary of Defense,  Cost Analysis Improvement Group; Office of the Secretary of Defense,  Program Analysis and Evaluation; Office of the Director, Operational Test  and Evaluation; Defense Contract Management Agency’s CCIC2S Program  Support Team; Office of the Assistant Secretary of the Air Force for  Acquisition; Air Force Space Command; Office of the Air Force Program  Executive Officer for Command and Control and Combat Support; Air  Force Electronic Systems Center; Air Force Operational Test and  Evaluation Center; United States Strategic Command; United States  Northern Command; North American Aerospace Defense Command; and  Lockheed Martin Integrated Systems & Solutions.
We conducted our work from July 2005 through April 2006 in accordance  with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Defense
section_title: Appendix III: Software Development Capability Maturity Model
paragraphs: Carnegie Mellon University’s Software Engineering Institute, a federally  funded research and development center, has identified specific processes  and practices that have proven successful in fostering quality software  development. The Software Capability Maturity Model®a rates maturity  according to the following five levels of maturity. See table 4 for a  description of these maturity levels.
section_title: Appendix IV: DOD Acquisition Categories, Thresholds, and Oversight
paragraphs: Placement into a specific acquisition category (ACAT) determines the  accountability level and analysis requirements of a program. As shown in  table 5, for a major automated information system acquisition, the  milestone decision authority is the Assistant Secretary of Defense for  Networks and Information Integration, who is also the DOD Chief  Information Officer. The Assistant Secretary of Defense for Networks and  Information Integration may delegate the authority to the DOD component  acquisition executive (such as the Assistant Secretary of the Air Force for  Acquisition). For non-major automated information system acquisitions,  the milestone decision authority is the component acquisition executive or  his or her designee.
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: Acknowledgments
paragraphs: In addition to the contact named above, Art Gallegos, Peter Grana,  Randolph C. Hite, Arturo Holguin, Rich Horiuchi, Gary Mountjoy,   Karl Seifert, Karen Sloan, and Rona B. Stillman made key contributions to  this report.
section_title: Related GAO Products
paragraphs: Attack Warning: NORAD’s Communications System Segment  Replacement Program Should Be Reassessed. GAO/IMTEC-89-1.  Washington, D.C.: November 30, 1988.
Space Defense: Management and Technical Problems Delay Operations  Center Acquisition. GAO/IMTEC-89-18. Washington, D.C.: April 20, 1989.
Attack Warning: Better Management Required to Resolve NORAD  Integration Deficiencies. GAO/IMTEC-89-26. Washington, D.C.: July 7,  1989.
Attack Warning: Defense Acquisition Board Should Address NORAD’s  Computer Deficiencies. GAO/IMTEC-89-74. Washington, D.C.: September  13, 1989.
Defense Acquisition: Air Force Prematurely Recommends ADP  Acquisitions. GAO/IMTEC-90-7. Washington, D.C.: March 29, 1990.
Attack Warning: Costs to Modernize NORAD’s Computer System  Significantly Understated. GAO/IMTEC-91-23. Washington, D.C.: April 10,  1991.
Computer Technology: Air Attack Warning System Cannot Process All  Radar Track Data. GAO/IMTEC-91-15. Washington, D.C.: May 13, 1991.
Attack Warning: Lack of System Architecture Contributes to Major  Development Problems. GAO/IMTEC-92-52. Washington, D.C.: June 11,  1992.
Attack Warning: Status of the Survivable Communications Integration  System. GAO/IMTEC-92-61BR. Washington, D.C.: July 9, 1992.
Granite Sentry. GAO/IMTEC-92-84R. Washington, D.C.: September 21,  1992.
Attack Warning: Status of the Cheyenne Mountain Upgrade Program.  GAO/AIMD-94-175. Washington, D.C.: September 1, 1994.
Defense Acquisitions: Status of Strategic C4 System Modernization and  Plans To Integrate Additional Mission Capabilities.   GAO/NSIAD-00-212R. Washington, D.C.: August 25, 2000.