DOD Major Automated Information Systems: Improvements Can Be Made in Reporting Critical Changes and Clarifying Leadership Responsibility
Published: 2016-03-30T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The National Defense Authorization Act for Fiscal Year 2012 includes a provision for GAO to select, assess, and report on DOD MAIS programs annually through March 2018. MAIS programs are intended to help the department sustain its key operations. This report: (1) evaluates DOD's implementation of statutory reporting requirements for MAIS programs experiencing a critical change; (2) describes the extent to which selected MAIS programs have changed their planned cost and schedule estimates, and met performance targets; (3) assesses the extent to which selected MAIS programs have used key IT acquisition best practices, including requirements and risk management; and (4) determines the extent to which MAIS programs are represented on the Dashboard. GAO compared information on programs with a critical change to the reporting requirements. GAO selected three programs based on factors, such as representation from each military service (Air Force, Army, and Navy), identified changes to cost, schedule, and performance, and assessed them against selected best practices. GAO traced the programs to the Dashboard and reviewed relevant processes.
section_title: What GAO Found
paragraphs: All 18 major automated information system (MAIS) programs that experienced a critical change to program cost, schedule, or system performance targets submitted complete reports to Congress that contained all four statutory elements, but 16 programs did not meet the requirement to report to Congress within 60 days of the program manager's submission to the senior Department of Defense (DOD) official that led to the critical change determination. Of the 16 critical change reports that exceeded the 60-days to report, 10 of the programs took over 100 days. Officials said that 60 days is too short to perform a program evaluation. Since the reports were not always timely, Congress may not have the necessary information when it is needed to make decisions. Finally, the DOD did not demonstrate that it had an internal control to ensure that MAIS programs not in compliance with reporting requirements were restricted from obligating funds on major contracts as required by law.
All three MAIS programs GAO selected to review experienced changes in their cost and schedule estimates, and one program did not fully meet its technical performance targets (see table).
Source: GAO analysis of data provided by DOD officials. | GAO-16-336
a Delay was attributed to a major change in project scope and restructuring of the program.
The three selected programs implemented all seven IT acquisition best practices for risk management, and most of the best practices were implemented for requirements management: the Army and Navy implemented three of five best practices and the Air Force implemented four of five best practices. For example, the Army program did not adequately manage requirements changes and ensure that deliverables were in alignment with requirements. Until the programs fully implement best practices for requirements management, management of development efforts will likely be impaired.
As of October 2015, all appropriate programs were represented on the Federal IT Dashboard (Dashboard) as required by the Office of Management and Budget (OMB); however, the organization responsible for performance of MAIS programs was not provided. Specifically, DOD's Chief Information Officer is shown as the responsible party because OMB requires this, but the Under Secretary of Defense for Acquisition, Technology, and Logistics (AT&L) has overall responsibility for the MAIS programs. Therefore, users of the Dashboard are unaware that AT&L is the responsible organization and, thus, public accountability of the MAIS programs is decreased.
section_title: What GAO Recommends
paragraphs: GAO recommends, among others, that DOD examine the critical change reporting process and implement corrections for the reports' timeliness, and address weaknesses with requirements management, and add AT&L as a responsible organization for MAIS programs to the Dashboard. DOD concurred with all recommendations. OMB did not concur but GAO continues to believe that improved transparency is needed.

section_title: Letter
paragraphs: The Department of Defense (DOD) is one of the largest and most  complex organizations in the world. To meet its mission to protect the  security of our nation and to deter war, it relies heavily on the use of  information technology (IT) to support our warfighters. In this regard,  according to DOD’s IT investment portfolio for fiscal year 2015, the  department spent approximately $30 billion for IT investments. Of this  amount, approximately $3.7 billion was spent on major automated  information system (MAIS) programs, which are intended to help the  department sustain its key operations. These include communications,  business, and command and control systems that provide department  and component officials with access to information to organize, plan,  direct, and monitor mission operations.
DOD IT investments that fall within one of the following categories are  designated as a MAIS program when: (1) program costs in any single  year exceed $40 million, (2) total program acquisition costs exceed $165  million, or (3) total life-cycle costs exceed $520 million. The Secretary of  Defense can also use discretion to designate a program as a MAIS if it  does not meet these cost thresholds. In addition, MAIS programs must  comply with certain annual and quarterly reporting requirements identified  in statute.
The National Defense Authorization Act for Fiscal Year 2012 includes a  provision that we select, assess, and report on DOD MAIS programs  annually through March 2018. In addition, Senate Report 113-176  accompanying S. 2410 includes a provision that we evaluate DOD’s  implementation of statutory reporting requirements for MAIS programs  experiencing a critical change. A critical change must be declared if the  program has experienced, among other things, a schedule delay of 1 year  or more, a full life-cycle cost increase of 25 percent or more over the  original estimate, or a change that will undermine the system’s ability to  perform as intended. The Carl Levin and Howard P. “Buck” McKeon  National Defense Authorization Act for Fiscal Year 2015 mandated that  agencies, including DOD, enhance transparency and risk management of  their major IT investments through the reporting of performance  information on the Federal IT Dashboard, a website that allows  stakeholders and the public to view details on the effectiveness of  government IT programs.
Our objectives for this review were to: (1) evaluate DOD’s implementation  of statutory reporting requirements for MAIS programs experiencing a  critical change; (2) describe the extent to which selected MAIS programs  have changed their planned cost and schedule estimates, and met  performance targets; (3) assess the extent to which selected MAIS  programs have used key IT acquisition best practices, including  requirements and risk management; and (4) determine the extent to  which MAIS programs are represented on the Federal IT Dashboard.
To accomplish the first objective, we identified 18 of 39 MAIS programs  that experienced a critical change and compared their status reports  submitted to Congress to statutory reporting requirements to determine  whether gaps exist. We also interviewed DOD officials responsible for the  quality of the data and assessed their procedures for maintaining its  accuracy and completeness for MAIS programs that experienced a critical  change between December 2008 and June 2014, as well as factors  impacting the timelines in delivering these critical change reports to  Congress and the processes used to ensure that appropriated funds were  not being obligated on major contracts if prohibited under law.
To address the second and third objectives, we selected 3 programs by  identifying them from the MAIS population of 39 programs that met  several criteria, such as programs with a baseline that could be used as a  reference point for evaluating cost and schedule characteristics. We  selected the Army’s Tactical Mission Command (TMC), the Navy’s  Common Aviation Command and Control System (CAC2S), and the Air  Force’s Defense Enterprise Accounting and Management System  (DEAMS).
To determine the extent that selected program estimates changed, we  compared their best or objective cost and schedule estimates established  in the first acquisition baseline estimate (where available) to the latest  total life-cycle estimates. To determine whether technical performance  targets were met, we compared each program’s system performance  targets against actual performance data, and reviewed the results of  assessments conducted on the systems. We then aggregated and  summarized the results of our analyses across the programs.
To determine the extent that best practices were used, we identified key  risk management and requirements management practices from the  Software Engineering Institute’s Capability Maturity Model® Integration  for Acquisition and assessed each program against these and other  criteria. We analyzed documents, such as risk register logs and risk  management plans, and compared them to each program’s processes  and practices. We interviewed program officials to obtain additional  information on the processes and practices used.
To address the fourth objective, we compared all 39 MAIS programs to  the IT Dashboard website to identify reporting gaps. We also evaluated  the process the department used to determine and submit status and  performance information to the website. We interviewed officials from the  DOD Office of the Chief Information Officer (CIO), Office of the Under  Secretary of Defense for Acquisition, Technology, and Logistics (AT&L),  and the Office of Management and Budget (OMB) to obtain their  perspectives on the quality of reporting of DOD MAIS programs.
We conducted this performance audit from April 2015 to March 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives. See appendix I for a more  detailed discussion of our objectives, scope, and methodology.
subsections: section_title: Background
paragraphs: DOD’s organizational structure includes the Office of the Secretary of  Defense, the Joint Chiefs of Staff, the military departments, numerous  defense agencies and field activities, and various unified combatant  commands that contribute to the oversight of DOD’s acquisition  programs. Figure 1 provides a simplified depiction of DOD’s  organizational structure.
The Under Secretary of Defense for AT&L serves as the Defense  Acquisition Executive and the Under Secretary has responsibility for  oversight of MAIS acquisition programs. AT&L has policy and procedural  authority for the defense acquisition system, which establishes the steps  that DOD programs generally take as DOD plans, acquires, deploys,  operates, and maintains its IT systems (discussed in more detail following  this section). Additionally, AT&L is the principal acquisition official of the  department and is the acquisition advisor to the Secretary of Defense.  AT&L’s authority includes directing the military services and defense  agencies on acquisition matters and making milestone decisions for MAIS  programs. AT&L can delegate decision authority for MAIS programs to a  component head who may further delegate the authority to the  component acquisition executive.
DOD’s CIO is the Principal Staff Assistant and senior IT advisor to the  Secretary of Defense. This role includes overseeing many national  security and defense business systems and managing information  resources. The CIO coordinates with AT&L to develop and maintain a  process for assessing and managing the risks related to the department’s  IT acquisitions, including MAIS programs.
subsections: section_title: DOD’s Acquisition Guidance and Framework for Managing MAIS Programs
paragraphs: The Department of Defense Instruction 5000.02 establishes policy for  the management of all DOD acquisition programs. In January 2015, DOD  updated these guidelines which outline the framework for MAIS  programs. This framework consists of six models for acquiring and  deploying a program, including two hybrid models that each describe how  a program may be structured based on the type of product being acquired  (e.g., software-intensive programs and hardware-intensive programs). A  generic acquisition model that shows all of the program life-cycle phases  and key decision points is shown in figure 2 and described following the  figure.
Materiel solution analysis: Refine the initial system solution  (concept) and create a strategy for acquiring the solution. A  decision—referred to as milestone A—is made at the end of this  phase to authorize entry into the technology maturation and risk  reduction phase.
Technology maturation and risk: Determine the preferred  technology solution and validate that it is affordable, satisfies program  requirements, and has acceptable technical risk. A decision—referred  to as milestone B—is made at the end of this phase to authorize entry  of the program into the engineering and manufacturing development  phase and award development contracts. An acquisition program  baseline is first established at the milestone B decision point or at  program initiation, whichever occurs later. A program’s first acquisition  program baseline contains the original life-cycle cost estimate (which  includes acquisition and operations and maintenance costs), the  schedule estimate (which consists of major milestones and decision  points), and performance parameters that were approved for that  program by the milestone decision authority. The first acquisition  program baseline is established after the program has refined user  requirements and identified the most appropriate technology solution  that demonstrates that it can meet users’ needs.
Engineering and manufacturing development: Develop a system  and demonstrate through testing that the system meets all program  requirements. A decision—referred to as milestone C—is made during  this phase to authorize entry of the system into the production and  deployment phase or into limited deployment in support of operational  testing.
Production and deployment: Achieve an operational capability that  meets program requirements, as verified through independent  operational tests and evaluation, and implement the system at all  applicable locations.
Operations and support: Operationally sustain the system in the  most cost-effective manner over its life cycle.
section_title: Statutory Reporting Requirements for MAIS Programs
paragraphs: MAIS programs enable DOD to organize, plan, direct and monitor  important mission operations. As previously mentioned, MAIS programs  must comply with certain annual and quarterly reporting requirements  identified in statute. Each calendar year, DOD must submit to Congress  budget justification documents on each MAIS program, including  information on cost, schedule, and performance. Specifically, these  programs must report, among other things, on the development and  implementation schedules and total acquisition and full life-cycle cost  estimates and provide a summary of the key performance parameters for  each program. DOD must also provide a summary of any significant  changes to information previously provided for each program.
Moreover, on a quarterly basis, the program manager for each MAIS  program is required to provide the senior DOD official responsible for the  program a written report that identifies any variance in the program’s cost,  schedule, or performance. Depending on the determination after  reviewing the variances identified in the quarterly report, the senior DOD  official responsible for the program must notify the congressional defense  committees of any programs that have experienced either a significant or  critical change. During our review, MAIS programs were required to  comply with the following reporting requirements:
Significant change. A significant change must be declared if a  program experienced a schedule delay of more than 6 months but  less than a year; estimated total acquisition or full life-cycle cost for  the program has increased by at least 15 percent but less than 25  percent; or there has been a significant adverse change in the  expected performance of the system. If such an event occurs, the  senior DOD official responsible for the program must notify the  congressional defense committees in writing no later than 45 days  after receiving the quarterly report from the program manager.
Critical change. A critical change must be declared if a program  failed to achieve a full deployment decision within 5 years after the  milestone A decision or, if there was no milestone A decision, the date  when the preferred alternative was selected for the program;  experienced a schedule delay of 1 year or more; experienced an  estimated total acquisition or full life-cycle cost increase of 25 percent  or more over the original estimate; or experienced a change in the  expected performance of the system that will undermine the ability of  the system to perform as intended. If such an event occurs, the senior  DOD official responsible for the program must carry out an evaluation  and submit a critical change report to the congressional defense  committees no later than 60 days after receiving the quarterly report.
Since the December 19, 2014, enactment of the Carl Levin and Howard  P. “Buck” McKeon National Defense Authorization Act for Fiscal Year  2015, MAIS programs are now required to declare a significant change— instead of a critical change—if they fail to achieve a full deployment  decision within 5 years after the milestone A decision, the date when the  preferred alternative was selected for the program (excluding any time  during which program activity is delayed as a result of a bid protest).
More recently, the National Defense Authorization Act for Fiscal Year  2016 directed the Secretary of Defense to issue guidance for MAIS  programs to establish an acquisition baseline within 2 years after program  initiation. This statute provides a response to a recommendation we made  in our last annual report on MAIS programs. In particular, we found that  these programs spent, on average, more than 5 years and $450 million  prior to establishing baselines. We noted that programs that have not  established baselines were subject to less oversight and could not be  measured against cost, schedule, and performance targets. Also, the  propensity to carry out MAIS programs for multiple years prior to  committing to baselines is inconsistent with incremental and rapid  development as called for in federal law and GAO’s IT management best  practices. Accordingly, we recommended that these programs be  baselined within 2 years; for which DOD partially concurred. We  maintained that establishing baselines within 2 years would improve  outcomes and increase accountability.
section_title: OMB’s IT Dashboard Codified to Provide Transparency into the Performance of Agency IT Investments
paragraphs: DOD’s CIO, along with other agencies, must report on the progress of its  IT investments, including MAIS programs, on a public website known as  the IT Dashboard. OMB established this website in June 2009 to  improve the transparency and oversight of agencies’ investments. The  Dashboard visually displays federal agencies’ cost, schedule, and  performance data for over 700 major federal investments at 26 federal  agencies. It also includes a risk rating that is to be performed by agency  CIOs. According to OMB, these data are intended to provide a near-real- time perspective on the performance of these investments.
The public display of agency data is intended to allow OMB; other  oversight bodies, including Congress; and the general public to hold  federal agencies accountable for their progress and results. In August  2011, OMB issued guidance that stated, among other things, that agency  CIO’s shall be held accountable for the performance of IT investments.  The Dashboard presents performance ratings for individual investments  using metrics that OMB has defined—cost, schedule, and CIO evaluation.  If OMB or the agency CIO determine the reported data is not timely or  reliable, the CIO must notify OMB and establish within 30 days of this  determination an improvement program and the progress the agency is  making. According to OMB, the addition of CIO names and photos on the  website is intended to highlight this accountability and link the  Dashboard’s reporting on investment performance.
In order to enhance transparency and improve risk management of  federal IT acquisitions, Congress codified the Dashboard reporting  process through key provisions, known as the Federal Information  Technology Acquisition Reform provisions in the Carl Levin and Howard  P. “Buck” McKeon National Defense Authorization Act for Fiscal Year  2015.
section_title: Best Practices for Managing IT Acquisition Programs
paragraphs: Entities such as the Project Management Institute, the Software  Engineering Institute at Carnegie Mellon University, and GAO have  developed and identified best practices to help guide organizations to  effectively plan and manage their acquisitions of major IT systems, such  as MAIS programs. Our prior reviews have shown that properly applying  such practices can significantly increase the likelihood of delivering  promised system capabilities on time and within budget. These practices  include, but are not limited to:
Requirements management: Requirements establish what the  system is to do, how well it is to do it, and how it is to interact with  other systems. Appropriate requirements development involves  eliciting and developing customer and stakeholder requirements, and  analyzing them to ensure that they will meet users’ needs and  expectations. It also consists of validating requirements as the system  is being developed to ensure that the final systems to be deployed will  perform as intended in an operational environment.
Risk management: A process for anticipating problems and taking  appropriate steps to mitigate risks and minimize their impact on  program commitments. It involves identifying and documenting risks,  categorizing them based on their estimated impact, prioritizing them,  developing risk mitigation strategies, and tracking progress in  executing the strategies.
section_title: All Programs with a Critical Change Reported Required Elements, but Most Missed Deadlines and Obligations Were Not Monitored
paragraphs: According to statute, for programs that declare a critical change, the  report that is submitted to Congress must include a written certification  stating that:  the automated information system or IT investment to be acquired is  essential to the national security or to the efficient management of the  DOD;  there is no alternative to the system or IT investment which will  provide equal or greater capability at less cost;  the new estimates of the costs, schedule, and performance  parameters have been determined, with the concurrence of the  Director of Cost Assessment and Program Evaluation, to be  reasonable; and  the management structure for the program is adequate to manage  and control program costs.
All 18 MAIS critical change reports in our review contained the required  elements.
In addition, this report must be prepared and submitted to Congress no  later than 60 days after the senior DOD official responsible for the  program receives the quarterly report from the program manager that  leads to the determination that a critical change event has occurred.  Programs that do not submit the report to Congress within the 60-day  period are statutorily prohibited from obligating appropriated funds for any  major contract until the date that Congress receives the report. Further, if  a MAIS program violates the statutory prohibition against obligations, it  will also violate the Antideficiency Act. This act prohibits an officer or  employee of the United States government from making or authorizing an  expenditure or obligation in excess of or in advance of available  appropriations. The Antideficiency Act also requires that an appropriation  must be available for an agency to incur an obligation. Thus, if DOD  incurs an obligation against an appropriation that is not legally available,  the department has violated the act. Violating the Antideficiency Act  would require the Secretary of Defense to immediately report to the  President and Congress all relevant facts and a statement of actions  taken.
Of the 18 MAIS programs experiencing a critical change, most exceeded  the 60-day reporting requirement, several by a substantial amount.  Specifically, 16 exceeded the 60-day reporting requirement and 10 of  those programs took over 100 days to report. Of the 10 programs, 5 of  the programs took over 200 days to report. Two programs—Teleport Gen  I/II and General Fund Enterprise Business Systems—delivered their  reports to Congress within the 60-day requirement. Figure 3 shows the  extent that programs met or exceeded the 60-day reporting requirement.
Officials from several programs provided various reasons for why they  delayed submitting their critical change reports. For example:
Mission Planning Systems #2 submitted a report to the Office of the  Secretary of Defense on time but it took 29 days to transmit the  package to Congress and it was 3 days late.
CAC2S program was late because of the need to conduct an  independent assessment that was directed by DOD.
The Expeditionary Combat Support System program had delays due  to changes that were made to the size and complexity of its originally  scoped effort and contracting process that, in turn, required additional  updates. The update triggered a process to re-evaluate the revised  strategy.
According to a DOD AT&L official, 60 days is too short to perform a  program evaluation and achieve all the coordination necessary for an  important communication with Congress. The official also said the  addition of the Office of Cost Assessment and Program Evaluation  requirement to review and approve the reports, as mandated by the  Weapon Systems Acquisition Reform Act of 2009, consumes much of  the 60-day allotment time. A Cost Assessment and Program Evaluation  official noted that reviews often exceed the 60-day period because, most  notably, the significant amount of time needed to collect and develop  comprehensive information used to determine a program’s cost. The  official added that DOD is working to strengthen the data collection efforts  to improve the ability of the Office of Cost Assessment and Program  Evaluation to complete its evaluation such as reviewing the basis for the  revised cost and schedule estimates. However, the official noted that  there has been no overall evaluation or study on the cause for delays.
While it may be possible that 60 days is too short of a time frame for  submitting reports, without understanding the cause for the delays DOD is  not in a position to state what time frame would be feasible. Further, the  fact that two programs were able to submit reports in a timely manner  suggests that 60 days is achievable. Until DOD ascertains the cause for  the delays and implements corrective actions, the reports may continue to  be delivered in a manner which may impact the timeliness of information  considered by Congress in making oversight and funding decisions for  MAIS programs. This may also affect budget and other strategic  decisions on how and what programs to prioritize.
Further, DOD does not have a mechanism to monitor and ensure that  MAIS programs with late reports were restricted as required by law from  obligating funds on major contracts prior to Congress receiving the report.  This mechanism is especially important because of the potential for  violating the Antideficiency Act. Although DOD states in its guidance that  program managers should not obligate any funds during the entire period  in which the report is being prepared, DOD does not currently have a way  to monitor this.
According to a DOD AT&L official, DOD is not required by statute,  regulation, or guidance to collect the information for monitoring purposes.  However, our guidance on internal controls for federal agencies states  that agency management should establish a baseline to monitor the  current state of a control system. Once established, management should  monitor the agencies’ internal control system through ongoing monitoring  and separate evaluations  DOD does not have a management internal  control to monitor the system and evaluate whether programs are  complying with the DOD guidance. Instead, DOD relies on the programs  to act in accordance with the law. This official said the need to obligate  funds on major contracts should be a driver for programs to submit their  reports to Congress as expeditiously as possible. However, with so many  programs submitting the critical change reports well after the 60-day  period, there is a risk that programs could potentially violate the  prohibition on obligations, and thus, in addition, the Antideficiency Act.
section_title: Selected MAIS Programs Had Varying Cost and Schedule Estimate Changes, One Did Not Meet Its Technical Performance Targets
paragraphs: The extent to which the three selected MAIS programs in our study  experienced changes in their cost and schedule estimates and met  performance targets varied. Specifically, the Army and Air Force  programs experienced slight changes in their cost and schedule  estimates, while the Navy program experienced more significant changes.  In addition, only one program, Air Force’s DEAMS, did not fully meet its  technical performance targets. Table 1 provides a status of the cost,  schedule changes, and the results of technical performance targets for  the programs. See appendix II for the detailed profiles of each program.
subsections: section_title: Army’s Program Had a Slight Cost Increase, but Was Within Schedule and Met All Its Performance Targets
paragraphs: As of January 2016, the latest life-cycle cost estimate for the Army’s TMC  program had increased about 19 percent from the program’s February  2008 acquisition program baseline estimate (from approximately $1.97  billion up to $2.34 billion). Program officials attributed the cost increase to  a breach in the research and development testing and evaluation cost  estimation that was reported to Congress. The program’s estimated  program development cost increased by 45 percent over the original  acquisition program baseline due to program scope changes derived from  the realignment of certain missions, such as the endorsement of the  Command Post of the Future as a foundation for mission command.
As of January 2016, TMC program experienced a 3-month slippage in its  full deployment date, currently scheduled for December 2018. The  slippage was within the program’s pre-established threshold allowance to  account for minor changes in schedule and, program officials stated that  this slippage was considered to be a low risk that the program has  accepted. Program officials stated that, although the Command Post of  the Future product is 95 percent fielded and is on schedule to reach full  deployment by December 2018, continued support of the Command Post  Computing Environment is needed beyond fiscal year 2019.
As of January 2016, the TMC program met all three of its key  performance targets, which include supporting net-centric military  operations, disseminating orders with future Army and Joint Command  and Control Systems, and displaying unified information on subject  matters.
section_title: The Navy Program Experienced Significant Cost and Schedule Increases, but Met All Performance Targets
paragraphs: As of October 2015, the latest life-cycle cost estimate for Navy’s CAC2S  Increment 1 program had increased about 477 percent from its first  acquisition program baseline estimate (from approximately $347 million  up to $2 billion). As previously reported, factors attributed to the  program’s early developmental challenges contributed to an increase in  its cost estimate, which include program scope growth and restructuring.  According to program documentation, despite the program’s initial  challenges in its cost estimates due to an increase in its operations and  support expenditures, the CAC2S program has demonstrated gradual  improvement as it reported a cost avoidance of $54.4 million for  implementing the DOD Better Buying Power initiatives that befitted from  competitive market forces that drove down cost. As of October 2015, the  program’s latest life-cycle estimate relative to its November 2010  production acquisition program baseline cost estimate had decreased by  about 19 percent (from approximately $2.46 billion down to $2 billion).
As of October 2015, compared to its first acquisition program baseline  schedule, the program experienced a 13 year and 9 month slippage in its  full deployment date—currently scheduled for March 2022. As previously  reported, factors that attributed to the prior schedule slippage included  the addition of new requirements and program restructuring. However,  program officials stated that the program has been executing in  accordance within its approved schedule. As of October 2015, the  estimated milestone C phase 2 was delayed by 6 months from the  program’s production acquisition program baseline but achieved it  milestone approved within the program’s pre-established schedule  threshold. Program officials attributed this delay, in part, to administrative  factors, which included the review and approval process. CAC2S  successfully achieved milestone C phase 2 approval in February 2015 but  the acquisition decision memorandum had not been signed until March  2015.
As of October 2015, program officials reported that, during performance  testing, it was meeting both of its key performance targets related to net- ready and data fusion.
section_title: Air Force’s Program Had Slight Cost and Schedule Increases and Did Not Meet Performance Targets
paragraphs: As of October 2015, the latest life-cycle cost estimate for the Air Force’s  DEAMS program had increased about 9 percent from its first February  2012 acquisition program baseline estimate (from approximately $1.43  billion up to $1.56 billion). Program officials attributed the cost increase, in  part, to program scope growth and the addition of software upgrade  enhancements. Specifically, as of October 2015, the program’s life-cycle  cost estimate incorporated additional infrastructure maintenance costs  throughout the life-cycle that added performance monitoring and  additional deployment support. Also, according to program officials, the  program brought forward increment 2 requirements and a second Oracle  software upgrade in year 2021.
DEAMS experienced a 6 month slip in its milestone C but was within its  threshold, a predefined point where programs that exceed it are at  increased risk. However, it did experience a 1 year slip in its full  deployment decision date—currently scheduled for February 2016.  Program officials attributed this slippage due to findings identified  DEAMS’s initial operational test and evaluation report. While the program  had been established since August 2003, a full deployment date had not  been determined. As of September 2015, program officials expect full  deployment to be reached by October 2016.
As of October 2015, DEAMS program officials reported that the program  did not meet all of its nine key performance targets. Specifically, DEAMS  did not meet five performance targets: Balance with Treasury, Accurate  Balance of Available Funds, Timely Reporting, Period-End Processing,  and Net-Ready. For example, the two operational assessments that were  conducted from 2012 to 2014 identified significant weaknesses in  three measures of effectiveness and suitability. Further, the Initial  Operational Test and Evaluation report identified system performance  issues within the DEAMS program, which included change management  issues, transaction backlogs, and ineffective reporting tools.
Subsequently, the Air Force Operational Test and Evaluation Center  provided 29 recommendations for the Air Force to implement to support  the successful fielding of DEAMS Increment 1, 17 of which were  documented as being completed, while corrective action for the remaining  12 are still underway. However, according to program documentation,  DEAMS must demonstrate measureable improvement by the full  deployment decision date of February 2016, in order to avert future  schedule delays in its fielding deployment.
section_title: Selected Programs Use of IT Acquisition Best Practices Were Mostly Applied for Requirements Management and Fully Applied for Risk Management
paragraphs: According to the Software Engineering Institute’s Capability Maturity  Model Integration for Acquisition, an appropriate requirements  management involves establishing an agreed-upon set of requirements,  ensuring traceability between requirements and work products, and  managing any changes to the requirements in collaboration with  stakeholders. Likewise, an effective risk management process identifies  potential problems before they occur, so that risk-handling activities may  be planned and invoked, as needed, across the life of the project in order  to mitigate the potential for adverse impacts. Table 2 provides key  practices used to comprehensively manage requirements and risk.
All three selected programs implemented IT acquisition best practices for  risk management, but requirements management best practices were not  consistently implemented by the programs. Table 3 provides a summary  of the extent to which requirements and risk management best practices  were implemented by each program.
subsections: section_title: The Army Program Implemented Risk Management Practices, but Did Not Apply Several Requirements Management Practices
paragraphs: The Army implemented all risk management best practices for the TMC  program. For example, the risk management plan, dated May 2014,  identified risk sources to include, among other things, unclear system  requirements, immature technology, and an unstable organizational  environment. In addition, program officials analyzed, categorized, and  controlled risks using a probability and impact model that considered the  risks (from very low to very high) and the potential consequences. The  program also used a risk radar tool to track and monitor risks. These are  reviewed weekly during staff meetings and updated monthly. Further,  TMC’s risk management plan indicated that contingency plans are  invoked whenever adjustments to cost, schedule, or performance are  required. In taking these and other actions, the TMC program had  established and utilized the key risk management practices. Doing so  should better position the program to mitigate adverse impacts from  potential problems before they occur.
The Army had implemented three requirements management best  practices for the program, but did not fully implement two: the practice of  managing requirements changes and ensuring that work products are in  alignment with requirements. For example, one key practice that the  program implemented included the maintenance of the bidirectional  traceability tool among requirement. Specifically, program officials utilized  a traceability tool used to generate a requirements matrix to track all of its  program elements to the requirements. Regarding managing  requirements changes, while program officials tracked requirements  changes in a database, requirements changes were not always available  at the stakeholder level to evaluate the impact and determine the status  of requirements changes for all elements of the program. Even though  TMC was in the production phase, it relied solely on the functions within  its requirements database rather than a requirements management  document. This left the program without a formal mechanism to track and  ensure project plans, activities, and work products are consistent with  defined requirements. Without such a document, the program does not  have a formal mechanism to track and ensure that project plans,  activities, and work products are consistent with defined requirements.
section_title: The Navy Program Implemented Risk Management Practices, but Did Not Apply All Requirements Management Practices
paragraphs: The Navy implemented all risk management best practices for the CAC2S  program. For example, the risk management plan assessed risks in terms  of their probability and consequence of occurrence. The program also  identified and documented risks and had the supporting documentation  that included risk and issue register logs, detailed reports, the integrated  master schedules, and risk assessments. In addition, the risk  management plan established the strategy that included the processes to  guide risk mitigation efforts at the lowest, appropriate level. The  program’s risk registers, which included risk mitigation steps, were  provided by program officials to demonstrate that risk mitigation plans had  been implemented for each risk. In taking these and other actions, the  program had established and utilized effective risk management  practices.
The Navy implemented three requirements management best practices  for the program but did not implement two. For example, one key practice  implemented was managing requirements changes. Specifically, the  program office demonstrated the ability to effectively manage changes to  requirements as they evolved during the project. Program officials did this  by documenting the alignment of requirements to its respective  requirements changes, maintaining a history of requirements changes  with rationale explaining the change request within its configuration  documentation, and publishing requirements data using a database tool.  However, the program did not fully implement the practice of maintaining  traceability among requirements and work products to ensure that work  products were in alignment.
Further, with regard to the program’s bidirectional traceability tool,  according to the July 2014 traceability tool, 25 specifications and 13  capabilities did not map to its respective work products. According to  program officials, as of November 2015, 11 of the 25 specifications had  been mapped, but due to an oversight, mapping of these specifications  were not associated, while the remaining 14 specifications did not map to  its respective capability production documentation to demonstrate  completeness. According to program officials, 4 capabilities had been  recently mapped but the remaining 9 capabilities listed in the capability  production documentation did not map the traceability to the respective  requirements. According to program officials, the mapping discrepancy of  7 capabilities was attributed to unfunded, obsolete, and programmatic  requirements. Furthermore, after notifying the program of the gaps we  identified, officials stated that they would take action to ensure mapping of  14 specifications and 2 capabilities to their respective requirements work  products would be addressed.
Regarding the alignment of requirements, the program’s requirements  management plan had not been updated since May 2009 and software  specifications and capabilities were not consistently maintained.  According to program officials, the current requirements management  plan had been previously reviewed and was determined to be suitable for  the purpose of implementing requirements management best practices.  According to the Capability Maturity Model® Integration for Acquisition,  without a clear linkage between requirements all the lower-level  requirements and capabilities, the program may not be effectively  managing development efforts in accordance with the most recent  requirements. Until work products are updated, the program cannot  provide assurance that its requirements are aligned with the most  updated work products and is at-risk of potential cost and schedule  consequences.
section_title: The Air Force Program Implemented Risk Management Practices, but Did Not Apply One Requirements Management Practice
paragraphs: The Air Force implemented all seven risk management best practices for  the DEAMS program. To the program’s credit, DEAMS demonstrated  great strides in improving its risk management best practices. Since our  prior MAIS review, the program has made improvements such as  monitoring the status of each risk periodically, and ensuring that risk  reports were up to date, which included the status of actions to mitigate  risks. Other key practices include defining parameters to analyze and  categorize risks, documenting risk, and developing risk mitigation plans in  accordance with the risk management strategy, among other areas. In  taking these and other actions, the DEAMS program had established and  utilized effective risk management practices.
The Air Force had implemented four requirements management best  practices, but did not fully implement one requirements management  practice, the practice of developing an understanding with providers on  the meaning of requirements. For example, the Air Force ensured that  project plans and work products were aligned with the most recent  requirements. Specifically, the program maintained consistent  documentation and oversight of work products, which included an up-to- date requirements management plan, system specifications, and  capabilities documentation. However, while the program had established  an adjudication process by which requirements were reviewed and  approved, and implemented a test methodology to validate requirements  prior to production installation, the function to ensure accountability was  not working properly.
As such, program officials did not determine whether key requirements  were validated during system integration testing prior to deploying  software into production, which was released with unresolved issues.  Program officials subsequently resolved the issues without any negative  impacts. The program office attributed the issue of not fully validating  requirements to environmental issues that were considered to be  acceptable risks and, subsequently, scheduled production installation in  November 2014. Nevertheless, according to the Software Engineering  Institute’s Capability Maturity Model® Integration for Acquisition,  requirements should be analyzed to ensure that established criteria are  met so that proper control functions are in place.
section_title: All Appropriate MAIS Programs Were Represented on the Federal IT Dashboard, but the Responsible Organization Was Not Provided
paragraphs: Pursuant to its statutory responsibility to analyze, track, and evaluate  risks, OMB requires agency CIOs to provide cost, schedule, and risk  information for all major IT investments on the IT Dashboard. In  addition, the IT Dashboard shows CIO names and photographs who are  responsible for investments to increase accountability for IT acquisitions.
As of October 2015, 27 of the 39 DOD MAIS programs were listed on the  IT Dashboard. According to DOD officials, and in accordance with OMB  policy, 8 MAIS programs that have not been funded in the President’s  budget submission are not reported to the Dashboard, as appropriate. In  addition, 4 MAIS programs have been designated by DOD as containing  national security-sensitive information and were therefore classified and  not subject to being reported on the Dashboard. According to DOD CIO  and AT&L officials, the 8 unfunded MAIS programs will be reported to the  Dashboard after the President’s 2017 budget submission has been  finalized.
However, the organization responsible for supervising MAIS acquisition  programs—AT&L—is not represented on the Dashboard. Instead, the  Dashboard publicly shows DOD’s CIO as the responsible party, pursuant  to OMB’s direction, but is not accurate. AT&L has oversight responsibility  for the acquisition performance of MAIS programs. In this regard, not only  does AT&L supervise department acquisitions and establish its  acquisition policies, but as the milestone decision authority for MAIS  programs, the Under Secretary or his designee, has overall responsibility  for each program. By contrast, the CIO is not involved in managing the  performance of the MAIS programs but is responsible for submitting the  rating to the Dashboard.
Officials from DOD’s Office of the CIO and OMB’s Office of E- Government and Information Technology told us that they were aware of  this inconsistency on the Dashboard but did not think it was a significant  issue. Further, the DOD officials stated that since the CIO is involved in  the rating process the representation of their office on the Dashboard is  sufficient. Nonetheless, since only the DOD CIO is represented on the  Dashboard, the public and other users may be unaware that AT&L has  overall oversight for the acquisition performance of MAIS programs,  minimizing the intended accountability the Dashboard is to provide.
section_title: Conclusions
paragraphs: Since MAIS programs account for billions of dollars of DOD’s IT budget, it  is important that the required critical change reports are timely so  Congress has the necessary information to make budgetary and  oversight decisions. While the reports contained the required elements,  many were not submitted in a timely manner, potentially hampering  Congress’ ability to make informed decisions.
Further, all three selected programs implemented IT acquisition best  practices for risk management and implemented most practices for  requirements management. While this is a significant achievement,  improvements can be made in managing requirements. Among other  things, programs were operating without a current requirements  management plan that was considered to be acceptable risks. Managing  requirements effectively is especially necessary since MAIS programs are  intended to help the department sustain its key operations.
Finally, there is a lack of accountability for AT&L on the IT Dashboard.  While OMB intended for accountability by requiring that major  investments show agency CIOs as responsible, it did not consider that  DOD’s AT&L is the responsible party for oversight of the acquisition  performance of MAIS programs. Since the Dashboard does not reflect  that AT&L has such responsibility, there is decreased public  accountability.
section_title: Recommendations for Executive Action
paragraphs: To help improve the management of MAIS programs, we are making six  recommendations that:
The Secretary of Defense examine the MAIS critical change reporting  process to identify root causes for delays and implement corrective  actions for the timely delivery of critical change reports.
The Secretary of Defense develop a mechanism for monitoring  whether MAIS programs with late reports are restricted from obligating  funds and in turn ensuring compliance with the Antideficiency Act.
The Secretary of the Army direct the TMC program manager to  develop a requirements management plan to document and manage  its requirements process.
The Secretary of the Navy direct the CAC2S program manager to  identify weaknesses in the requirements traceability process and take  corrective actions to manage the traceability of requirements to the  respective lower-level requirements, and periodically evaluate work  products, including the requirements management plan, and update  them in accordance with the requirements guidance.
The Secretary of the Air Force direct the DEAMS program manager to  address weaknesses in its controls for ensuring that all software  requirements are tested and validated before deployment of new  software releases.
Director of OMB instruct the Federal CIO to add the Under Secretary  of Defense for AT&L as a responsible party to DOD’s MAIS entries on  the Federal IT Dashboard website, alongside the CIO, to publicly  disclose the responsible party for the acquisition performance  management of MAIS programs.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD and OMB. We received written  comments from DOD’s Acting Principal Deputy Assistant Secretary of  Defense for Acquisition, which are reprinted in appendix III. In its  comments, the department concurred with all five recommendations to  improve oversight, IT acquisition practices, and tools used to manage  MAIS programs.
In e-mail comments, an official from OMB’s audit liaison group stated that  OMB’s Office of E-Government and Information Technology does not  agree with the recommendation to add AT&L to the IT Dashboard as a  responsible party for MAIS programs but would work with DOD to  address it. The official did not provide a rationale for this position or  explain how OMB would work with DOD.
Nonetheless, we continue to believe there is a lack of transparency and  accountability for AT&L on the IT Dashboard. The IT Dashboard publicly  shows DOD’s CIO as the responsible party, pursuant to OMB’s direction.  However, AT&L has oversight responsibility for the acquisition  performance of MAIS programs. In this regard, not only does AT&L  supervise department acquisitions and establish its acquisition policies,  but as the milestone decision authority for MAIS programs, the Under  Secretary or his designee, has overall responsibility for each program.  The DOD CIO is not involved in managing the performance of the  programs and is only responsible for submitting the rating to the  Dashboard. We believe that adding AT&L to the IT Dashboard would  increase public accountability and leadership transparency for the  acquisition management of MAIS programs.
We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; the Secretary of the Air Force, the  Secretary of the Army, the Secretary of the Navy, the Office of  Management and Budget, and other interested parties. This report also is  available at no charge on the GAO website at http://www.gao.gov.
Should you or your staffs have any questions on information discussed in  this report, please contact me at (202) 512-4456 or ChaC@gao.gov.  Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. GAO staff who made  major contributions to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The National Defense Authorization Act for Fiscal Year 2012 includes a  provision that we select, assess, and report on selected Department of  Defense (DOD) major automated information system (MAIS) programs  annually through March 2018. In addition, Senate Report 113-176  accompanying S. 2410 includes a provision that we evaluate DOD’s  implementation of statutory reporting requirements for MAIS programs  experiencing a critical change. Our objectives for this report were to (1)  evaluate DOD’s implementation of statutory reporting requirements for  MAIS programs experiencing a critical change; (2) describe the extent to  which selected MAIS programs have changed their planned cost and  schedule estimates, and met performance targets; (3) assess the extent  to which selected MAIS programs have used key IT acquisition best  practices, including risk management; and (4) determine the extent to  which MAIS programs are accurately represented on the Federal IT  Dashboard.
To evaluate DOD’s implementation of statutory reporting requirements for  MAIS programs experiencing a critical change—a schedule delay of 1  year or more, a full life-cycle cost increase of 25 percent or more over the  original estimate, or a change that will undermine the system’s ability to  perform as intended—we collected and analyzed information about  critical changes from December 2008 to June 2014 and their  corresponding reports. We assessed whether DOD had met reporting  requirements by reviewing MAIS critical change reports and supporting  documentation to determine if they included required written certifications  stating that:  the automated information system or IT investment to be acquired is  essential to the national security or to the efficient management of the  DOD;  there is no alternative to the system or IT investment which will  provide equal or greater capability at less cost;  the new estimates of the costs, schedule, and performance  parameters have been determined, with the concurrence of the  Director of Cost Assessment and Program Evaluation, to be  reasonable; and  the management structure for the program is adequate to manage  and control program costs.
We did not look at the quality of the assessments and estimate done. We  reviewed the reports to determine if the particular elements were  included. We then summarized the number of elements addressed by  program.
In order to determine if the critical change reports met or exceeded the  60-day requirement, we collected all MAIS reports and compared the  dates within them. We subtracted the date in which the reports were  delivered to Congress via letter from the date in which the program  manager provided the quarterly report to the senior DOD official  responsible for the program in order to determine the number of days that  had lapsed. Additionally, we rearranged the programs by number of days  (from greatest to least) it took to deliver the critical change reports. We  noted how many programs took more than 100 days, and 200 days to  deliver the reports. We also interviewed DOD officials in order to ask their  opinion on the length of time it takes to deliver a critical change report to  Congress.
We interviewed DOD officials responsible for the quality of the data and  assessed their procedures for maintaining its accuracy and  completeness. In addition, we examined the data for outliers or others  extraordinary items. Based on these procedures, we have concluded that  these data are sufficiently reliable for our purposes.
Additionally, in order to determine whether DOD is tracking the obligation  of funds during a program’s critical change, we interviewed DOD officials  in order to determine what processes and tools they had in place to  ensure that funds were not being obligated.
To address the second and third objectives, we used DOD’s official list of  39 MAIS programs, as of February 25, 2015, to establish the basis for  selecting the MAIS programs that were used to assess objectives two and  three. We used the criteria below to select three MAIS programs.
Any programs that were used in the prior two reviews should be  excluded.
Any programs that are fully deployed or cancelled should be  eliminated from consideration.
The program should not be new to the MAIS list; otherwise, there may  not be sufficient acquisition activity and documentation to evaluate.
The program must have a baseline in order to have a reference point  for evaluating cost and schedule performance characteristics.
The program cannot be a National Security Agency program.
We included one program from each military department—Army, Air  Force, and Navy in order to diversify the portfolio. Thus, we excluded  any DOD-wide programs.
We selected programs that had the lowest relative ratings on the  Federal IT Dashboard as of April 2015.
We preferred that the program be complex (e.g., integration across  domains, global, critical to battle operations, etc.), rather than an  upgrade.
We preferred to select programs with funding profiles that are  significant when compared to the rest of the portfolio.
We considered issues identified from credible sources of information,  such as Defense Acquisition Management Information Retrieval  online resources, IT Dashboard ratings, etc.
We filtered the original list of MAIS programs using the criteria above.  Based on this filtering, we chose the following systems:  the Air Force’s Defense Enterprise Accounting and Management  System-Increment 1 (DEAMS Increment 1),  the Army’s Tactical Mission Command (TMC), and  the Navy’s Common Aviation Command and Control System  Increment 1 (CAC2S Increment 1).
To address the second objective, we analyzed and compared each  selected program’s first acquisition program baseline cost estimate to the  latest life-cycle estimate to determine the extent to which planned  program costs had changed. Similarly, to determine the extent to which  these programs changed their planned schedule estimates, we compared  each program’s first acquisition program baseline schedule to the latest  schedule. We relied on the thresholds established by statute to describe  the amount of any deviation (i.e., significant or critical) that each  program’s latest life-cycle cost and schedule estimates experienced from  the first acquisition program baseline.
To determine whether the selected programs met their performance  targets, we compared program and system performance targets against  actual performance data in test reports and program management  briefings. We reviewed the results of operational assessments and  program evaluations conducted on the systems. We also reviewed  additional information on each program’s cost, schedule, and  performance, including program documentation, such as DOD’s MAIS  annual and quarterly reports; information from the Office of Management  and Budget’s (OMB) IT Dashboard; acquisition program baselines;  monthly status briefings; system test reports; and our prior reports. We  also interviewed program officials from each of the selected MAIS  programs to obtain additional information on cost, schedule, and  performance. We provided our assessments to the program management  offices of each selected program for comment. We aggregated and  summarized the results of these analyses across the programs, as well  as developed individual profiles for each program (see appendix II).
To address the third objective, we analyzed each selected program’s IT  acquisition documentation and compared it to key requirements  management and risk management best practices—including Software  Engineering Institute’s Capability Maturity Model® Integration for  Acquisition (CMMI- ACQ) practices—to determine the extent to which  the programs were implementing these practices. In particular, the key  requirements management best practices we reviewed were: develop an understanding with the requirements providers on the  meaning of the requirements, obtain commitment to requirements from project participants, manage changes to requirements as they evolve during the project, maintain bidirectional traceability among requirements and work, and ensure that project plans and work products remain aligned with  requirements.
Specifically, we analyzed program requirements documentation, including  requirements management plans, requirements traceability matrices,  requirements change forms, technical performance assessments, and  requirements board meeting minutes. Additionally, we interviewed  program officials to obtain additional information about their requirements  management practices.
Additionally, we reviewed the following key risk management best  practices: determine risk sources and categories; define parameters used to analyze and categorize risks and to control  the risk management effort; establish and maintain the strategy to be used for risk management; identify and document risks; evaluate and categorize each identified risk using defined risk  categories and parameters, and determine its relative priority; develop a risk mitigation plan in accordance with the risk management monitor the status of each risk periodically and implement the risk  mitigation plan as appropriate.
Specifically, we analyzed program risk documentation, including monthly  risk logs and reports, risk-level assignments, risk management plans, risk  mitigation plans, and risk board meeting minutes. Additionally, we  interviewed program officials to obtain additional information about their  risks and risk management practices.
To address the fourth objective, we used DOD’s official list of MAIS  programs, as of February 25, 2015. These programs were used as basis  to determine whether programs were reported to the Federal IT  Dashboard. To do so, we:
Exported the IT portfolio program data reported to the Federal IT  Dashboard by DOD in fiscal year 2015, and compared it to the 39  programs on DOD’s official list of MAIS programs.
For those programs that were found to not be reported on the  Dashboard, we met with agency officials from the Office of the Chief  Information Officer (OCIO) and the Under Secretary of Defense  (Acquisition, Technology, and Logistics) (AT&L) to determine the  reasons for not reporting.
We also interviewed OCIO officials to obtain information on the processes  used by the OCIO when reviewing programs for IT Dashboard updates.  Further, we interviewed officials at OMB to obtain their views on  representing the CIO as the sole party responsible for programs reported  to the IT Dashboard was accurate. Specifically, we discussed DOD’s  unique structure that AT&L is responsible for the acquisition performance  of MAIS programs but is a separate organization from the OCIO.
We conducted this performance audit from April 2015 to March 2016 in  accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Profiles of Selected MAIS Programs
paragraphs: This section contains profiles of the three selected major automated  information system (MAIS) programs for which we determined whether  they had changed their planned cost and schedule estimates and met  performance measures. Each profile presents data on the program’s  purpose and status, its latest cost and schedule estimates compared to  the first acquisition program baseline (where established), as well as  system performance data.
The first page of each two-page profile contains a description of the  program’s purpose and a figure that provides a comparison of the  program’s first acquisition program baseline to the program’s latest  schedule. The years depicted on the figure represent calendar years, and  the milestones represent the program’s best estimates of dates for those  milestones. The program’s start represents the date that program officials  reported that they first started work on the program.
The first page also provides (1) essential program details, such as the  name of the prime contractor, the total number of active contractors— which includes the prime contractor—and any other contractors (and in  some cases subcontractors) supporting the program; (2) program costs  (in then-year dollars), comparing the program’s latest life-cycle cost  estimate (separated into acquisition and operations and maintenance  costs) to its first acquisition program baseline (subsequent acquisition  program baselines that may have been established are not identified),  (3) locations to which the system will be deployed; and (4) a summary of  the cost, schedule, and performance of each program, which is further  discussed on the second page of the profile. The symbols, denoted by  arrows or filled circles, included in the summary box on the first page of  each profile and in the headings on the second page represent whether a  program’s cost estimate had increased (^), decreased (V), or stayed  within (i.e., not to exceed threshold) planned cost estimate (i.e., () and  whether the program’s schedule estimate had slipped (>),been  accelerated to meet milestones earlier than planned (<), or stayed within  (i.e., not to exceed threshold) planned schedule estimate () of meeting  milestones.
The second page of each profile provides detailed information on each  program’s status, costs, schedule, and performance.
In the status section, we discuss recent and upcoming milestones and  events for each program.  In the cost section, we identify the extent to which the program’s life- cycle cost estimate has changed from its first acquisition program  baseline, as well as the causes for any changes identified.  In the schedule section, we discuss the extent to which the program’s  schedule has changed from its first acquisition program baseline, and  the causes for any schedule changes identified.  In the performance section, we identify the extent to which each  program has met its established measures, as well as discuss the  results of system performance tests. These performance ratings  represent a point-in-time assessment as reported by the program.  System performance targets were rated as “met” when (1) system  tests were passed with no deficiencies or limitations, (2) the program  fully met all of its key performance parameters, or (3) a program had  addressed all deficiencies or limitations that were identified during  system tests. System performance was rated as “not fully met” when  a program either (1) did not fully pass system testing and was still in  the process of addressing the deficiencies or limitations identified  during system testing or (2) did not pass system testing and  subsequently removed the problematic functionality from the system  in order to pass subsequent system tests, instead of fixing the  problematic functionality and keeping it in the planned release of the  system.
Army Tactical Mission Command (TMC)
TMC is a suite of products—comprised of hardware and software  equipment and elements—that are intended to provide the Army  commanders and their staff with mission command capabilities, such as  real-time situational awareness and a user-defined common operational  picture. TMC products are fielded worldwide and are intended to support  decision-making, planning, rehearsal, and execution management. TMC  is now engaged in transitioning to the web-based Command Post  Computing Environment. One key element of this new environment — known as Tactical Applications—is aimed to minimize administrative  burdens on the user, and simplify the overall Mission Command  collaborative experience.
All of the products included in TMC are post-development and in  production. The program continues to field equipment and perform a  technical refresh of the hardware and software in the field. The program is  also working to resolve sustainment metric issues and updating their life- cycle sustainment plan.
Exceeded Planned Cost Estimate (^)
TMC’s planned total life-cycle cost estimate has increased by 19 percent  from the program’s first acquisition program baseline estimate of  approximately $1.97 billion. Specifically, as of January 2016, the life-cycle  cost estimate was approximately $2.34 billion. Program officials reported  that the increased costs were attributed to research and development  testing and evaluation cost estimate breach that was reported to  Congress. The Army’s TMC program estimated program development  cost increased by 45 percent over the original estimate due to program  scope changes derived from the realignment of Command Post of the  Future as a foundation for Mission Command Collapse, the integration of  Personalized Assistant that Learns, and the incorporation of future force  requirements.
Stayed within Planned Schedule Estimate ()
As of January 2016, the program had experienced a 3 month slippage in  its full deployment date compared to its first acquisition program baseline  of September 2018. The slippage was within the pre-established  threshold allowance to account for minor shifts in program schedule.  Program officials stated that, although the Command Post of the Future  product is 95 percent fielded and is on schedule to reach full deployment  by December 2018, continued support of the Command Post Computing  Environment is needed beyond fiscal year 2019. Program officials  considered the slippage to be of low risk and will continue to operate  within the planned schedule estimate.
As of January 2016, the TMC program met all three of its key  performance parameters, including net-centric military operations,  disseminate orders with future Army and Joint C2 systems, and  displaying unified information on subject matters, such as friendly and  enemy forces.
Navy Common Aviation Command and Control System (CAC2S)  Increment 1  CAC2S is an integrated and coordinated modernization effort for the  equipment of the Marine Air Command and Control System and is  intended to provide enhanced capability for three defense centers to  support aviation employment in joint, combined, and coalition operations.  CAC2S provides the tactical situational display, information management,  sensor and data link interface, and operational facilities for planning and  execution of Marine Aviation missions within the Marine Air Ground Task  Force. It is intended to replace existing aviation command and control  equipment from 12 legacy systems. CAC2S Increment 1 will eliminate the  Air Command and Control systems and will capability for aviation combat  direction and air defense functions by providing a single networked  system.
CAC2S Increment 1, which comprises of two phases, is currently in post- milestone C where the phase 2 system is now in development. CAC2S’s  current work consists of developmental testing and the production of the  limited deployment units in preparation for the March 2016 Initial  Operational Test and Evaluation. To help achieve its goals of a  successful Initial Operation Test and Evaluation, a phase 2 milestone C  decision was authorized in February 2015 for CAC2S to procure four  limited deployment units. As of October 2015, CAC2S had delivered all  four limited units to support current developmental testing. Production of  the limited deployments units are on schedule and planning activities are  underway. Regarding the developmental testing, program officials  anticipate favorable test results after subsequent software enhancements  had been made to address software concerns identified in prior  developmental testing.
Exceeded Planned Cost Estimate (^)
As of October 2015, CAC2S’s life cycle cost estimate was $2 billion,  which was about a 477 percent increase from its first acquisition program  baseline estimate of $347 million established in August 2000. As  previously reported, factors that attributed to the cost increase were early  challenges in estimating costs due to program scope growth and  restructuring. According to program documentation, operations and  support expenditures of approximately $1.6 billion for the production of  milestone C had been carried over into the program’s total life-cycle cost  estimate. However, since our previous report, program documentation  indicated that improvements to the cost position are being made.  Specifically, the milestone C service cost position, dated February 2015,  produced a cost avoidance of $54.4 million compared to its 2010 cost  assessment. As of October 2015, the program’s latest life-cycle estimate  relative to its November 2010 production acquisition program baseline  cost estimate had decreased about 19 percent. Program officials  attributed this decrease due to the program embracing the DOD Better  Buying Power initiatives that benefitted from competitive market forces  that drove down cost.
Exceeded Planned Schedule Estimate (>)
As of October 2015, CAC2S Increment 1’s estimated full deployment date  was March 2022, which represented a 13 year and 9 month schedule slip  from the program’s first acquisition program baseline schedule estimate.  As previously reported, factors that attributed to the schedule delay  included the addition of new requirements and program restructure.  Program officials stated that subsequent to our prior report, the program  has been executing in accordance within its approved schedule. As of  October 2015, the estimated milestone C phase 2 was delayed by 6  months from the program’s production acquisition program baseline  schedule but, as stated above, achieved its milestone. Program officials  attributed this delay, in part, to administrative factors, which included the  review and approval process of getting signature approval. CAC2S  successfully achieved milestone C phase 2 approval in February 2015 but  the acquisition decision memorandum had not been signed until March  2015.
As of October 2015, CAC2S program documentation reported that it was  meeting both of its key performance parameters related to net-ready and  data fusion. Program officials stated that, during testing of the program’s  key performance parameters, its net-ready and data fusion performance  targets were both met, while many attributes for the data fusion key  performance parameter were consistently above the threshold for being  met.
Air Force Defense Enterprise Accounting and Management System  (DEAMS) Increment 1  The DEAMS Increment 1 program is intended to provide the Air Force  with the entire spectrum of financial management capabilities, including  collections; commitments and obligations; cost accounting; general  ledger; funds control; receipts and acceptance; accounts payable and  disbursement; billing; and financial reporting DEAMS is also intended to  be a key component of DOD’s solution for achieving fully-auditable  financial statements by September 30, 2017, as required by the National  Defense Authorization Act for Fiscal Year 2010.
As of November 2015, the DEAMS program is working to achieve full  deployment decision by February 2016. In August 2015, the initial  operational test and evaluation report, conducted by the Air Force  Operational Test and Evaluation Center, indicated a number of findings  requiring remediation prior to the February 2016 full deployment decision.  Nevertheless, DEAMS was granted a limited deployment decision, but the  program experienced a significant change as a result of breaching the full  deployment decision threshold for a timing issue only. DEAMS current  work efforts consist of deployment to new users and remaining 35 sites,  capability development for deployment, training new users, and resolving  initial operational test and evaluation findings.
Exceeded Planned Cost Estimate (^)
As of October 2015, DEAMS’s latest life-cycle cost estimate was about  $1.56 billion, which was about a 9 percent increase from its first  acquisition program baseline estimate of approximately $1.43 billion— established in February 2012. Program officials attributed this increase, in  part, to program scope growth due to addition of requirements from  increment 2 and the addition of a second Oracle software upgrade  projected for 2021.
Exceeded Planned Schedule Estimate (>)
DEAMS experienced a 6 month slippage in its milestone C but  successfully attained milestone C approval within the established  threshold. Program officials did not provide a rationale for factors that  attributed to this delay but maintained that the program operated within  the threshold requirements. DEAMS also experienced a 1 year slippage  in its full deployment decision date—currently scheduled for February  2016. Program officials attributed this slippage due to findings identified  DEAMS’s initial operational test and evaluation report.
Did Not Fully Meet System Performance Targets  As of October 2015, DEAMS program officials reported that it did not  meet all of its nine key performance parameters. Specifically, DEAMS did  not meet five key performance parameters: Balance with Treasury,  Accurate Balance of Available Funds, Timely Report, Period-End  Processing, and Net-Ready. For example, an initial operational test and  evaluation report, listed above, identified system performance issues,  which included unstable change management issues, transaction  backlogs, and ineffective reporting tools. Subsequently, the Air Force  Operational Test and Evaluation Center provided 29 recommendations  for the Air Force to implement to support the successful fielding of  DEAMS Increment 1, 17 of which were documented as being completed,  while corrective action for the remaining 12 are still underway. The  program is expected to demonstrate improvement before it will be  authorized to be deployed to all users.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact name above, the following staff also made key  contributions to this report: Eric Winter, Assistant Director; Ronalynn  (Lynn) Espedido; Corey Evans; Rebecca Eyler; Franklin Jackson; Kate  Nielsen; John Ortiz; Kathleen Sharkey; and Jeanne Sung.