Defense Acquisitions: Further Management and Oversight Changes Needed for Efforts to Modernize Cheyenne Mountain Attack Warning Systems
Released: 2006-07-06T00:00:00

Letter

Background

DOD built the Cheyenne Mountain Operations Center in the early 1960s at Cheyenne Mountain Air Force Station, Colorado. Throughout its history, its mission has continually evolved to adapt to changing threats—which have ranged from a perceived Soviet nuclear threat after the 1957 launch of Sputnik, the first man-made satellite successfully placed in orbit, to theater ballistic missiles in the Persian Gulf during Operation Desert Storm in the early 1990s. Currently, Cheyenne Mountain’s mission is to help monitor, process, and interpret air, missile, and space events that could threaten North America or have operational impacts on U.S. forces or capabilities, using air, ground, and space-based sensors that link to the complex’s computer systems located more than 2,000 feet under ground. The President, Secretary of Defense, combatant commanders, and the Prime Minister of Canada rely on Cheyenne Mountain systems to provide accurate, timely, and unambiguous information essential for determining courses of actions warfighters are to carry out. Air Force Space Command is responsible for operating, funding, and setting requirements for the systems, and Air Force Materiel Command’s Electronic Systems Center is responsible for maintaining the systems and for acquiring new systems and capabilities.

Table 1 provides a summary of Cheyenne Mountain’s three mission centers, operations, and systems.

Previous Cheyenne Mountain Upgrade Efforts

In 1981, the Air Force began efforts to modernize its Cheyenne Mountain systems under five separate programs scheduled to be completed in 1987 at a total estimated cost of $968 million. Subsequently, the Air Force consolidated the five separate Cheyenne Mountain upgrade programs into a single Cheyenne Mountain Upgrade (CMU) program. Beginning in the late 1980s, we issued a dozen reports on the program’s cost, schedule, and performance problems. For example, we reported that the Air Force did not adequately define program requirements which resulted in an underestimation of the upgrade efforts’ cost and schedule goals. In its attempts to maintain short-term cost and schedule goals, the Air Force continually deferred requirements and problem resolution to the future, as well as relaxed performance requirements. (See Related GAO Products at the end of this report.)

Despite not meeting original performance expectations for the program, in 1998, the Air Force declared the Cheyenne Mountain upgrades to be operational—almost $1 billion over cost and 11 years late. That same year, DOD determined that some of the systems were not well integrated, would be unresponsive to future mission needs, and some components were becoming unsupportable. As a result, in 2000, the Air Force initiated the CCIC2S program to modernize and integrate Cheyenne Mountain systems. Figure 1 depicts the timeline of Cheyenne Mountain upgrades.

On September 19, 2000, the Air Force awarded a cost-plus award fee contract to Lockheed Martin Mission Systems to begin work on the CCIC2S program. The initial period of performance was for 6 years, with annual options for operations and maintenance activities for an additional 9 years.

CCIC2S Program Has Experienced Cost and Schedule Overruns and Performance Shortfalls

Like its predecessor, the CMU program, the CCIC2S program is over cost, behind schedule, and some capabilities have been deferred indefinitely. This deferral could pose risks to performing some future operations. The Air Force’s current program cost estimates through fiscal year 2006 represent about a 51 percent increase over its initial estimate. Additionally, while the Air Force initially estimated CCIC2S upgrades to be completed in fiscal year 2006, most critical mission capabilities will not be delivered by this time, and for some of these capabilities, the Air Force has yet to estimate a completion date. The deferrals of capabilities and performance shortfalls could have implications for future missions.

The CCIC2S Program Is Over Cost and Behind Schedule

As shown in table 2, the Air Force initially estimated the CCIC2S program to complete upgrades in fiscal year 2006 at an estimated cost of about $467 million. However, program cost estimates through the same year have increased about 51 percent to nearly $707 million. Air Force officials said they never estimated the life-cycle cost of delivering all CCIC2S validated requirements.

Along with the increases in estimated program costs, schedules have expanded significantly. The Air Force initially estimated development activities to be completed in fiscal year 2006, but only a fraction of the expected capabilities have been delivered (see fig. 2). Of the three mission areas—air, missile, and space—the Air Force has delivered only initial air and some initial missile mission critical capabilities. DOD considers these critical capabilities to be the most essential for the program—failure to provide these capabilities can be cause for a program to be reassessed or terminated. The remaining missile warning critical capabilities are now expected to be delivered in fiscal year 2007—over 3 years later than initially estimated. The Air Force has postponed the delivery of all critical space mission capabilities to some undetermined point in the future. According to Air Force officials, the space mission capability deliveries will not be scheduled until future investment decisions are made.

According to the CCIC2S program office and other DOD officials, one of the reasons the program’s initial cost and schedule estimates differ from current estimates is that work that was not included in the original estimates was added to the program. Examples of work added to the program include replacing an aging missile warning processing and display system as well as developing a computer operational status monitoring system. We did not assess the cost and schedule impacts of these added capabilities. Also, as a result of the September 11, 2001, attacks on the U.S., the Air Force diverted a portion of CCIC2S funding and personnel to make improvements to other Cheyenne Mountain systems so that aircraft traffic within the U.S. could be monitored by the air warning center. However, according to DOD officials, the impacts of the September 11, 2001, terrorist attacks on the CCIC2S program were not significant because monitoring aircraft traffic within the U.S. is not a required upgrade under the CCIC2S program. Prior to these improvements, the focus of the air warning system was to monitor aircraft traffic outside the North American border.

Deferral of Capabilities and Performance Shortfalls Have Implications for Future Missions

The deferral of capabilities and performance shortfalls that have resulted from cost and schedule overruns in the CCIC2S program could significantly impact future missions—especially if program dollars are needed to maintain legacy systems longer than expected. USSTRATCOM—responsible for protecting U.S. space assets—could be particularly affected, given that none of the work on CCIC2S’s critical space mission capabilities has been completed and that estimated completion dates for this work have yet to be determined.

DOD has been counting on CCIC2S to replace the Space Defense Operations Center (SPADOC)—Cheyenne Mountain’s current space object tracking system—and provide a system capable of processing larger volumes of data. Some DOD officials have stated that CCIC2S is to be the foundation for carrying out future space awareness functions. SPADOC’s capabilities are currently overtaxed. It currently handles about 400,000 space object observations per day from sensors—about 167 percent more than it was designed to handle. The Air Force has implemented an inefficient workaround to address SPADOC’s limited processing capabilities. Specifically, the Space Control Center is using CaveNet, an off-line tool, to assist with analyzing and processing space object observations. However, to analyze and process the information in CaveNet, Space Control Center operators must first download information from SPADOC and then manually type the processed information back into SPADOC. Air Force officials stated that manually typing CaveNet information back into SPADOC has recently become significant enough to request an automated interface.

According to Air Force officials, efforts are on-going to assess the mission impact of SPADOC performing beyond design specifications. With other programs to improve data collection sensors that feed data into CCIC2S— such as the Space Fence and Space Based Space Surveillance system— the number of observations and processing requirements will increase significantly in the future, taxing the current system further. In a 2005 study, the CCIC2S contractor determined that SPADOC’s processing capacity will not be able to handle the increased number of observations predicted beginning in 2012. For the near term, according to Air Force officials, the CCIC2S contractor has identified operations and maintenance changes for SPADOC to handle increases in the number of observations. However, DOD has not made a decision on when to pursue upgrades for SPADOC and cost estimates for upgrading SPADOC have not been finalized. With the delay of CCIC2S space capabilities and no decision on when SPADOC will be replaced, USSTRATCOM will have to continue relying on pushing SPADOC to its limits.

Delays in the delivery of expected CCIC2S capabilities also affect programs that are not directly part of the program but are relying on CCIC2S for their implementation. For example, the Air Force’s Rapid Attack Identification Detection and Reporting System (RAIDRS)—a defensive counterspace system—is expected to use information provided by Cheyenne Mountain systems to aid in the detection, reporting, identification, location, and classification of attacks against valuable space assets. However, according to a DOD official, the CCIC2S space capability delays have presented a serious acquisition dilemma for the RAIDRS program because it is dependent on the CCIC2S space segment. Funding has not been made available for the CCIC2S space segment that is to support RAIDRS, according to Air Force Space Command. Furthermore, other programs will experience the same acquisition dilemma because Air Force Space Command has instructed all of its sensor, satellite, and weapon programs to be able to integrate with CCIC2S.

Cost and Schedule Overruns Caused by Not Adhering to Best Practices for Managing Program Requirements

The CCIC2S program’s failure to match requirements and resources before its start has contributed to the cost and schedule overruns that have forced the program to defer delivery of CCIC2S capabilities. For each year since the program’s inception, the Air Force has rebaselined CCIC2S’s cost and schedule goals. The unreliable cost and schedule estimates that resulted have forced the program to frequently defer work to later years to control cost growth. The Air Force has yet to determine when and at what cost development is to be completed.

Match Not Made between Program Requirements and Available Resources

Our body of work on best practices has shown that matching program requirements to resources—that is, time, funding, technology, and personnel—is key to successful outcomes. We have also reported that the requirements-setting phase is one of the most important for software- intensive acquisitions. Missing, vague, or changing requirements tend to be a major cause of poor outcomes in these programs. For example, according to Air Force officials, the major causes for the delay in the Air Mission Evolution system—a CCIC2S system that was delivered in January 2004, over 2 years behind schedule—included problems with software development and integration. Additionally, the contractor stated that changing priorities within the program as well as added requirements contributed to the delay.

Without a stable set of requirements, the Air Force could not appropriately estimate program costs and in hindsight, the resources needed to satisfy program requirements were severely underestimated. To better ensure programs are affordable and fully funded before they are initiated, DOD guidance calls for every acquisition program to stabilize requirements and determine the program’s total costs before program start. Despite best practices and DOD policy, the Air Force initiated the CCIC2S program without establishing a stable set of program-specific requirements and did not do so until January 2004—more than 3 years after program start. In that time, the Air Force made numerous additions, deletions, and modifications to CCIC2S’s requirements. For example, the Air Force deleted, added back, and then modified a requirement for multiple security levels. According to an Air Force review of program requirements, some deleted requirements were duplicative or conflicting.

Assessments of available resources were also inadequate. According to DOD guidance, every acquisition program should rigorously address issues such as the program’s life-cycle costs and whether it is affordable and fully funded. By not conducting these rigorous assessments, DOD and the Air Force did not have the information needed to commit to the program over its life cycle. Although the CCIC2S program acquisition plan refers to the development of a life-cycle cost estimate before the program was initiated, it did not include the operations and maintenance costs of sustaining existing systems through the program’s entire life cycle.

Additionally, software development capabilities of the contractor should also be assessed. The contract for CCIC2S development specifies that the contractor be rated at the highest level (level 5) using the Capability Maturity Model® for Software developed by Carnegie Mellon University’s Systems Engineering Institute. At this most mature level, quantitative feedback about performance and innovative ideas and technologies contribute to continuous process improvements. A senior program official stated that the CCIC2S contractor was rated at a level 5 for software development. According to DOD documentation, Capability Maturity Model® assessments are typically performed at an organization wide level and may not necessarily apply to a project team associated with any given acquisition program. We did not examine the program’s software development efforts and the capabilities of the contractor. For more information on this model, see appendix III.

Furthermore, DOD has not funded the program at requested levels since the program began due to other priorities. We have previously reported that program funding instability often occurs because DOD starts too many programs, creating severe budget constraints. Another factor we have reported that contributes to funding instability is that DOD’s programs are funded annually, requiring competition among other programs for limited available funding. The CCIC2S program has not been an exception to these factors. For example, prior to initiating the CCIC2S program in 2000, DOD reduced the program’s initial estimated resource requirements by a total of $75 million. In 2002, DOD decided not to increase requested program funding. According to DOD, funding was limited to program baseline levels because the program was a lesser priority. Furthermore, in 2005, DOD and Congress reduced CCIC2S program funding estimates over the next 6 years by about $135 million, citing higher priority funding needs and poor program performance.

The Air Force Has Implemented the CCIC2S Program without Solid Expectations of What Capabilities Are To Be Delivered, When, and at What Cost

According to the acquisition strategy for the CCIC2S program, annual rebaselines were to be conducted on the program to review and approve program implementation plans, including cost and schedule baselines. However, because the Air Force failed to make a match between requirements and resources, including not effectively assessing the impacts of adding unanticipated work to the program, the result of CCIC2S’s annual rebaselines has been to defer significant amounts of work to the future in order to address cost increases. As a result, the Air Force has implemented the program as a “level of effort”—that is, without solid expectations of what capabilities are to be delivered, when, and at what cost. Currently, the CCIC2S program is undergoing its fifth rebaseline. According to program officials, after this fifth rebaseline, CCIC2S will no longer plan for annual rebaselinings, but would only rebaseline the program when necessary. Figure 3 depicts the effects of each of the rebaselines on the program.

The fifth and current rebaseline was prompted by efforts to complete development of the program’s critical missile warning mission capabilities. To free up resources for the missile warning development efforts, the Air Force stopped work on many other aspects of the CCIC2S program in late 2004 and subsequently began rebaselining the program in early 2005. Air Force officials stated the rebaseline was to be finalized by August 2005; however, it has yet to be approved as of June 2006. According to Air Force officials, pending changes in program oversight as well as recent DOD and congressional funding estimate reductions are reasons for the delay in approving the rebaseline. As currently structured, the rebaseline would again significantly defer the amount of capability scheduled to be delivered. Specifically, missile warning critical requirements would not be delivered until fiscal year 2007, and the schedule for the work on the space portion is undetermined.

CCIC2S Program Oversight and Controls Have Been Ineffective

The oversight and control mechanisms of the CCIC2S program have been inadequate to prevent or mitigate many of the problems it has experienced. Additionally, the Air Force’s contract management approach limited the program office’s ability to thoroughly assess the reliability of the contractor’s cost and schedule performance information and the impact of defining, prioritizing, and adding capabilities. Furthermore, the Defense Contract Management Agency (DCMA) did not independently monitor contractor performance. According to DOD officials, actions are being taken to implement better controls.

DOD Did Not Designate CCIC2S as a Major Acquisition

Placement into a specific acquisition category (ACAT) determines the accountability level and analysis requirements of a program. DOD policy requires major automated information system acquisitions to have high- level oversight and comprehensive and independent assessments of life- cycle cost estimates and benefits to ensure they are reasonable and built on realistic program and schedule assumptions. In determining whether an automated information system acquisition program warrants a “major” designation, DOD applies dollar value thresholds. If a program does not meet these thresholds, it can be designated as a major acquisition based on other factors such as technical complexity. (See appendix IV for more detail on the designation criteria for acquisition programs that were in place when the Air Force initiated the CCIC2S program.)

The CCIC2S program met the dollar threshold criteria for designation as a major automated information system acquisition program. However, DOD did not designate the program as a major automated information system acquisition. According to a senior Air Force Space Command official, the program was initially considered to be a weapon system acquisition and therefore below the threshold criteria for designating the program as a major defense acquisition program. At this lower acquisition category, oversight for the program was assigned to Air Force headquarters.

Initial cost estimates for CCIC2S indicated that it should have been designated as a major automated information system acquisition. Specifically, at program initiation, CCIC2S’s program cost estimate of about $463 million well exceeded DOD’s threshold of $360 million (fiscal year 1996 constant dollars)—DOD’s then-threshold for total life-cycle costs—for designation as a major automated information system program (see table 3).

Additionally, the CCIC2S program is a technically challenging and complex acquisition. For example, one of the primary goals of the program is to integrate 40 stovepiped systems into a single common architecture. The complexity of the CCIC2S effort is further illustrated by our previous work which showed that the CMU program had experienced long- standing, serious integration problems, as well as software development problems with incorporating new technologies.

DOD and Office of Management and Budget have issued guidance on conducting analyses that can help decision making on major automated information system acquisitions. These analyses include an affordability assessment to demonstrate that the program’s resources (including projected funding and manpower requirements) are realistic and achievable in the context of the DOD component’s overall long-range modernization plan to ensure full funding and funding stability for the program; an economic analysis to determine the best program acquisition alternative by assessing the net life-cycle costs and benefits of the proposed program; and an independent estimate of program life-cycle costs.

Taken together, such reviews can give DOD the knowledge it needs to prevent or mitigate many of the types of problems the CCIC2S program has experienced.

We have consistently reported that the decision to invest in any system should be based on reliable analyses of estimated system costs and expected benefits over the life of the program. Without designation as a major acquisition, the CCIC2S program lacked the high-level oversight and independent analyses and review that could have identified key program weaknesses, including program cost, schedule, and performance problems; the mismatch between requirements and available resources; and the inability to prioritize and commit to the program within the context of other acquisition programs and long-range investment plans.

CCIC2S Milestone Decision Authority Did Not Provide Effective Oversight

DOD acquisition policy states that overall program responsibility rests with the milestone decision authority. In this role, the milestone decision authority is responsible for ensuring accountability and maximizing credibility in program cost, schedule, and performance reporting by reviewing the program throughout its acquisition life cycle, including (1) whenever the program reaches a milestone decision point; (2) whenever costs, schedule, or performance goals are baselined or must be changed; and (3) periodically through review of program status reports.

While the Assistant Secretary of the Air Force for Acquisition—the designated milestone decision authority for the CCIC2S program—played an active oversight role from 2000 to 2001, its program oversight thereafter was minimal, including during the numerous program rebaselines and attempts to address cost, schedule, and performance shortfalls. The Assistant Secretary’s oversight appears to have been limited to brief program status updates provided by the program executive officer as part of a weekly reporting on all programs under the program executive officer’s purview, and, according to DOD officials, annual and semi-annual program execution and budget reviews. According to Air Force officials, the Assistant Secretary delegated its milestone decision authority responsibilities for the program to the program executive officer level. However, officials from the Office of the Assistant Secretary of the Air Force for Acquisition and the Office of the Program Executive Officer were unable to provide us documentation of a formal delegation of these responsibilities, or explain why this delegation took place.

Program Lacked Sufficient Management Controls over the Contractor

In addition to matching requirements to resources, our best practices work has shown that to better ensure programs deliver capabilities within expected costs and schedules, program offices must have in place certain controls, including: (1) assessments of the effects of any changes in requirements on cost, schedule, and performance goals and (2) insights into contractor activities and progress to ensure commitments are implemented. The contract to develop CCIC2S was awarded under a Total System Performance Responsibility (TSPR) approach, which transfers certain government management responsibilities from the program office to the contractor.

In the mid-1990s, DOD began making sweeping changes to its acquisition policy and procedures to streamline its acquisition infrastructure. One of these reforms was the introduction of the TSPR concept, which gave a contractor total responsibility for the integration of an entire weapon system and for meeting DOD’s requirements. According to DOD officials, the Air Force no longer uses TSPR for its acquisition programs due to the shortcomings resulting from using this management concept. We have previously reported that TSPR contracts often result in the government’s lacking accurate cost, schedule, and performance information needed to adequately manage programs because it had to rely on unverified contractor data.

For CCIC2S, the risks inherent in a TSPR approach were exacerbated by the program office’s lack of sufficient staffing to oversee the definition and prioritization of needed capabilities. Because the Air Force reduced the number of personnel in the program office from 200 to between 15 and 30 in an effort to achieve efficiencies, it lacked the personnel to assess the cost and schedule impacts of defining and prioritizing requirements on the program. Without such information, the program office was not in a position to make trade-off decisions between different types of needed capabilities. According to the program office, as a result of its inability to manage the requirements definition and prioritization process, the contractor ended up communicating directly with the users to help define and prioritize requirements, sometimes without the knowledge of the program office.

Program officials also noted that they lacked sufficient staff with the technical knowledge to determine whether contractor cost, schedule, and performance data were reliable. For example, according to the program office and the CCIC2S contractor, the program initially called for a contractor outside of the CCIC2S program to develop air mission software and provide it to the CCIC2S contractor for integration into the program. However, shortly after the Air Force initiated the CCIC2S program, it terminated the air mission software development contract due to unsatisfactory performance. Subsequently, the users recommended, and the program office added, the unfinished work to the CCIC2S program as part of the re-baseline process in January 2001. However, according to the program office, it was not able to effectively assess the cost and schedule impacts of this change on the program.

TSPR does not relieve the government of its oversight responsibilities. Also, according to DCMA officials, the government should conduct contract surveillance and gauge contractor performance using specific performance metrics. However, the CCIC2S program office was not always aware of the contractor’s decisions or performance. For example, program office officials acknowledged that they were not aware of some re-prioritization of work done by the contractor to meet the users’ needs.

Program Lacked Independent Contractor Performance Assessment

Acquisition programs commonly use DCMA to monitor higher risk contracts. Typically, after DOD awards a contract, DCMA can monitor contractors’ information systems to ensure that cost, performance, and delivery schedules are in compliance with the terms and conditions of the contract. According to DCMA officials, however, their involvement in the CCIC2S program from inception through 2004 was minimal because DCMA lacked resources and the CCIC2S program office did not request DCMA assistance until that time. This is despite the fact that in January 2003, the Air Force and DCMA signed a memorandum of agreement calling for DCMA assistance on the contract for the CCIC2S program.

The agreement called for DCMA to analyze the contractor’s earned value management (EVM) systems to verify that monthly cost performance reports and related documents present a valid picture of contract cost and schedule progress. This type of analysis could have identified problems with the EVM system and led to corrective actions sooner. Since 2004, DCMA has been working with the program office and Air Force systems engineering officials to improve the quality of metrics to assess cost, technical performance, and risk, as well as to make improvements to the contractor’s EVM system.

Needed Management Controls and Redesignation of CCIC2S Program Are Being Considered

DOD is reconsidering CCIC2S’s current designation as a non-major automated information system program. In September 2005, the Assistant Secretary of Defense for Networks and Information Integration notified the Assistant Secretary of the Air Force for Acquisition that it intended to recategorize the CCIC2S program as a major automated information system acquisition because of increased funding levels, national interest in the program, and our review of the program. Designating the CCIC2S program as a major automated information system acquisition would put in place the high-level oversight and rigorous analyses needed to help ensure (1) the program has top-level accountability and support for the program, (2) DOD has the knowledge necessary for making trade-offs in program requirements so that they match available resources, and (3) DOD has the ability to prioritize and commit to the program within the context of its other acquisition programs and long-range investment plans. As of June 2006, however, the office of the Assistant Secretary of Defense for Networks and Information Integration has yet to formally make a decision on whether to designate the program as a major automated information system acquisition.

According to Air Force and DCMA officials, the program office has taken several steps, which we did not assess, to address some management control problems, including the following actions: Prohibited direct contact between the contractor and users—which they acknowledge had a direct impact on program cost and schedule performance—to keep the contractor from reprioritizing work to meet the immediate needs of the users, in 2004. Additionally, the program office and the contractor implemented an engineering review board process to evaluate any proposed change in terms of its validity in relation to validated requirements as well as its impact to program cost and schedule estimates. Initiated a User Feedback Control Board process to identify where deliverables are not meeting users’ needs and assess identified solutions in terms of their relationship to validated requirements as well as program cost and schedule implications, beginning in the spring of 2005.

Adopted a planning approach that is to more realistically account for program risks through a consideration of limiting factors such as test resources and the complexities surrounding installing and testing hardware and software in Cheyenne Mountain’s operational environment.

Took steps to add technical staff to assess the effects of changes in requirements on cost, schedule, and performance goals and provide additional insight into contractor activities. Implemented a formal system-level technical review process consisting of milestones, or gates, each of which have entrance and exit criteria. For example, deficiencies identified under the formal review must be addressed prior to proceeding into the next development phase.

Conclusions

Cheyenne Mountain systems—which are to warn the U.S. and Canada of air, missile, and space attacks—are critical to national security. However, since the 1980s, DOD’s efforts to modernize these systems have been fraught with cost increases, schedule delays, and performance shortfalls. Under the current development effort, the Air Force continues to defer the completion of key critical space requirements, placing the DOD’s ability to perform air and missile defense and space situational awareness missions at risk. Without matching requirements to available resources, the CCIC2S program will continue to flounder. Failing to conduct this match at the onset of this complex effort led DOD to develop unrealistic cost and schedule estimates as well as to over-promise capability. Specifically, the Air Force did not ensure that the program had stable requirements and a sound process for clarifying and controlling them, and thus the resources to satisfy the requirements were severely underestimated. The minimal top-level and external oversight and accountability of the program contributed to the problems. Failure to make course corrections now without correcting the shortfalls will put the program at risk of failing to achieve promised capability.

Recommendations for Executive Action

We recommend that the Secretary of Defense direct the Secretary of the Air Force to maintain Cheyenne Mountain’s essential operation and maintenance activities and to limit future development activities to those deemed critical to national security needs until the Department takes the following five actions: Designate the program as a major automated information system acquisition.

Conduct an affordability assessment to demonstrate that the program’s resource estimates are realistic and achievable in terms of DOD’s overall long-range modernization priorities and investment plans for Cheyenne Mountain. Such an assessment would require ensuring the requirements baseline is verified and validated and making a match between these requirements and available resources.

Conduct an economic analysis to assess the life-cycle costs and benefits of the program.

Conduct an independent estimate of program life-cycle costs, to provide a basis for relying on the cost estimates.

Determine and clearly delineate oversight responsibilities and accountability for the successful implementation of CCIC2S.

Additionally, the Secretary of Defense should direct the Secretary of the Air Force to take the following three actions: Develop and implement an approach that requires a rigorous analysis of resource impacts of any change in requirements for continuously making trade-offs between requirements and resources to ensure a match is maintained.

Ensure management controls are in place so that changes to requirements are decided on the basis of costs, benefits, risks, and affordability.

Develop and implement an approach for ensuring program cost and performance data are reliable.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. DOD agreed that the CCIC2S program be designated as a major automated information system with the Assistant Secretary of Defense for Networks and Information Integration as the milestone decision authority with oversight responsibility and accountability for its successful implementation. In addition, DOD agreed with conducting an affordability assessment, economic analysis, and independent estimate of life-cycle costs on future development activities but not with the current activities due to potential mission impacts. Specifically, DOD stated these assessments should be conducted on future delivery blocks of the CCIC2S program, but not on the current block. DOD asserted that its recent $127 million CCIC2S funding reduction across the Future Years Defense Program (FYDP) pared down CCIC2S development activities to those essential to national security needs.

We do not disagree with this approach to the extent that continuing these development activities is critical to national security. However, continuing CCIC2S development without establishing a match between requirements and available resources may also hinder DOD’s ability to satisfy national security needs because of the increased risk that the program fails to achieve its promised capability. While DOD raises the issue of the cost of delaying the program, it does not address the cost of proceeding as planned. Our work has shown that in numerous programs, continuing to spend money on development programs without sufficient knowledge or clear direction results in cost overruns and schedule delays. Further, given recent congressional concerns over DOD’s management of CCIC2S, demonstrating on a timely basis how requirements and available resources are to be matched while protecting national security is especially important. Since an assessment of CCIC2S’s specific operational risks to national security was beyond the scope of our review, we clarified our recommendation to read that the Secretary of Defense direct the Secretary of the Air Force to limit future development activities to those deemed critical to national security needs until the Department takes the actions discussed above.

DOD agreed to take actions relating to establishing and implementing management controls and approaches to analyze the resource impacts of changes in requirements and to ensure reliable program data. DOD stated that these changes had already been implemented or are being planned. During our review, program and DCMA officials told us that some changes—such as prohibiting the contractor from reprioritizing work without Air Force approval and implementing an engineering review board to assess requirements changes—had been made. However, they also noted that other measures to manage requirements and institute better management controls—such as increasing program office management staffing and capabilities, modifying the work breakdown structure and earned value management system, and implementing a new approach and metrics to track program progress—were under way or planned. Given DOD has not fully demonstrated the effectiveness of the changes it has made or is making, and that it is planning to make other changes, we are retaining our recommendation.

We are sending copies of this report to interested congressional committees and the Secretaries of Defense and the Air Force. We will also provide copies to others on request. In addition, this report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you have any questions about this report or need additional information, please call me at (202) 512-4841 (shamesl@gao.gov). Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

To determine the status of the Combatant Commanders’ Integrated Command and Control System, or CCIC2S, in terms of meeting its cost, schedule, and performance goals, we reviewed our past work to determine whether the CCIC2S program has continued to experience the problems previous Cheyenne Mountain upgrade efforts had experienced. We also reviewed program status reports and budget data and spoke with Department of Defense (DOD) and contractor officials to assess program cost, schedule, and performance progress to date. Additionally, we spoke with CCIC2S users to determine their level of satisfaction with CCIC2S program performance. Specifically, we reviewed documentation from and interviewed officials in the Office of the Assistant Secretary of Defense for Networks and Information Integration; National Security Space Office; Office of the Secretary of Defense, Cost Analysis Improvement Group; Office of the Secretary of Defense, Program Analysis and Evaluation; Office of the Director, Operational Test and Evaluation; Office of the Joint Chiefs of Staff; Defense Contract Management Agency’s CCIC2S Program Support Team; Office of the Assistant Secretary of the Air Force for Acquisition; Air Force Space Command; Air Force Air Combat Command; Office of the Air Force Program Executive Officer for Command and Control and Combat Support; Air Force Electronic Systems Center; Air Force Operational Test and Evaluation Center; United States Strategic Command; United States Northern Command; North American Aerospace Defense Command; and Lockheed Martin Integrated Systems & Solutions.

To gauge the extent to which DOD has followed best practices with regard to managing program requirements, including matching requirements to available resources, we reviewed DOD, Office of Management and Budget, and Air Force acquisition guidance, as well as our previous best practices work, to determine criteria for managing requirements. We also reviewed program requirements documentation and spoke with DOD officials on how DOD developed requirements for the CCIC2S program. Furthermore, we reviewed requirements management documentation and spoke with DOD and contractor officials to assess how the Air Force managed its requirements, including the process it used for defining and prioritizing the requirements. Specifically, we reviewed documentation from and conducted interviews with officials in the Office of the Secretary of Defense, Cost Analysis Improvement Group; Office of the Secretary of Defense, Program Analysis and Evaluation; Office of the Director, Operational Test and Evaluation; Office of the Joint Chiefs of Staff; Defense Contract Management Agency’s CCIC2S Program Support Team; Office of the Assistant Secretary of the Air Force for Acquisition; Air Force Space Command; Office of the Air Force Program Executive Officer for Command and Control and Combat Support; Air Force Electronic Systems Center; Air Force Operational Test and Evaluation Center; United States Strategic Command; United States Northern Command; North American Aerospace Defense Command; and Lockheed Martin Integrated Systems & Solutions.

To assess DOD’s oversight and control mechanisms for Cheyenne Mountain systems modernization and integration efforts underway and planned, we reviewed DOD and Air Force acquisition guidance and interviewed officials in the Office of the Assistant Secretary of Defense for Networks and Information Integration; Office of the Secretary of Defense, Cost Analysis Improvement Group; Office of the Secretary of Defense, Program Analysis and Evaluation; Office of the Director, Operational Test and Evaluation; Defense Contract Management Agency’s CCIC2S Program Support Team; Office of the Assistant Secretary of the Air Force for Acquisition; Air Force Space Command; Office of the Air Force Program Executive Officer for Command and Control and Combat Support; Air Force Electronic Systems Center; Air Force Operational Test and Evaluation Center; United States Strategic Command; United States Northern Command; North American Aerospace Defense Command; and Lockheed Martin Integrated Systems & Solutions.

We conducted our work from July 2005 through April 2006 in accordance with generally accepted government auditing standards.

Appendix II: Comments from the Department of Defense

Appendix III: Software Development Capability Maturity Model

Carnegie Mellon University’s Software Engineering Institute, a federally funded research and development center, has identified specific processes and practices that have proven successful in fostering quality software development. The Software Capability Maturity Model®a rates maturity according to the following five levels of maturity. See table 4 for a description of these maturity levels.

Appendix IV: DOD Acquisition Categories, Thresholds, and Oversight

Placement into a specific acquisition category (ACAT) determines the accountability level and analysis requirements of a program. As shown in table 5, for a major automated information system acquisition, the milestone decision authority is the Assistant Secretary of Defense for Networks and Information Integration, who is also the DOD Chief Information Officer. The Assistant Secretary of Defense for Networks and Information Integration may delegate the authority to the DOD component acquisition executive (such as the Assistant Secretary of the Air Force for Acquisition). For non-major automated information system acquisitions, the milestone decision authority is the component acquisition executive or his or her designee.

Appendix V: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, Art Gallegos, Peter Grana, Randolph C. Hite, Arturo Holguin, Rich Horiuchi, Gary Mountjoy, Karl Seifert, Karen Sloan, and Rona B. Stillman made key contributions to this report.

Related GAO Products

Attack Warning: NORAD’s Communications System Segment Replacement Program Should Be Reassessed. GAO/IMTEC-89-1. Washington, D.C.: November 30, 1988.

Space Defense: Management and Technical Problems Delay Operations Center Acquisition. GAO/IMTEC-89-18. Washington, D.C.: April 20, 1989.

Attack Warning: Better Management Required to Resolve NORAD Integration Deficiencies. GAO/IMTEC-89-26. Washington, D.C.: July 7, 1989.

Attack Warning: Defense Acquisition Board Should Address NORAD’s Computer Deficiencies. GAO/IMTEC-89-74. Washington, D.C.: September 13, 1989.

Defense Acquisition: Air Force Prematurely Recommends ADP Acquisitions. GAO/IMTEC-90-7. Washington, D.C.: March 29, 1990.

Attack Warning: Costs to Modernize NORAD’s Computer System Significantly Understated. GAO/IMTEC-91-23. Washington, D.C.: April 10, 1991.

Computer Technology: Air Attack Warning System Cannot Process All Radar Track Data. GAO/IMTEC-91-15. Washington, D.C.: May 13, 1991.

Attack Warning: Lack of System Architecture Contributes to Major Development Problems. GAO/IMTEC-92-52. Washington, D.C.: June 11, 1992.

Attack Warning: Status of the Survivable Communications Integration System. GAO/IMTEC-92-61BR. Washington, D.C.: July 9, 1992.

Granite Sentry. GAO/IMTEC-92-84R. Washington, D.C.: September 21, 1992.

Attack Warning: Status of the Cheyenne Mountain Upgrade Program. GAO/AIMD-94-175. Washington, D.C.: September 1, 1994.

Defense Acquisitions: Status of Strategic C4 System Modernization and Plans To Integrate Additional Mission Capabilities. GAO/NSIAD-00-212R. Washington, D.C.: August 25, 2000.