DOD Major Automated Information Systems: Adherence to Best Practices Is Needed to Better Manage and Oversee Business Programs
Released: 2018-05-24T00:00:00

Letter

Background

DOD’s organizational structure includes the Office of the Secretary of Defense, the Joint Chiefs of Staff, the military departments, numerous defense agencies and field activities, and various unified combatant commands that contribute to the oversight of DOD’s acquisition programs. Prior to February 2018, the former Under Secretary of Defense for Acquisition, Technology, and Logistics also served as the principal acquisition official of the department and was the acquisition advisor to the Secretary of Defense.

The former Under Secretary also served as the Defense Acquisition Executive and was the official responsible for supervising the acquisition of MAIS programs. The former Under Secretary’s authority included directing the military services and defense agencies on acquisition matters and making milestone decisions for MAIS and other programs. This official also had policy and procedural authority for the defense acquisition system, which establishes the steps that DOD programs generally take to plan, design, acquire, deploy, operate, and maintain the department’s information systems.

However, as of February 2018, the department changed the way it conducts business and operations with the statutory elimination of the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics. The statute contains a provision that required DOD to establish a new Office of the Under Secretary of Defense for Research and Engineering to be responsible for driving innovation and acceleration of the advancement of warfighting capability. In addition, a new Office of the Under Secretary of Defense for Acquisition and Sustainment was created to focus on delivering proven technology more quickly. The creation of these offices within the department is intended to shift the principal focus of the Office of the Secretary of Defense from a role of program oversight to that of directing major department investments. Further, the statutory creation of a Chief Management Officer to replace the former Deputy Chief Management Officer is intended to improve the quality and productivity of the department’s business operations.

DOD’s Acquisition Guidance and Framework for Managing MAIS Acquisitions

In January 2015, DOD updated its guidelines that outline the framework for MAIS programs. This framework consists of six models for acquiring and deploying a program, including two hybrid models that each describe how a program may be structured based on the type of product being acquired (e.g., software-intensive programs and hardware-intensive programs). A generic acquisition model that shows all of the program life- cycle phases and key decision points is depicted in figure 1 and described below.

Materiel solution analysis: Refine the initial system solution (concept) and create a strategy for acquiring the solution. A decision—referred to as Milestone A—is made at the end of this phase to authorize entry into the technology maturation and risk reduction phase.

Technology maturation and risk: Determine the preferred technology solution and validate that it is affordable, satisfies program requirements, and has acceptable technical risk. A decision—referred to as Milestone B—is made at the end of this phase to authorize entry of the program into the engineering and manufacturing development phase and award development contracts. An acquisition program baseline is first established at the Milestone B decision point. A program’s first acquisition program baseline contains the original life-cycle cost estimate (which includes acquisition and operations and maintenance costs), the schedule estimate (which consists of major milestones and decision points), and performance parameters that were approved for that program by the milestone decision authority. The first baseline is established after the program has refined user requirements and identified the most appropriate technology solution that demonstrates that it can meet users’ needs.

Engineering and manufacturing development: Develop a system and demonstrate through testing that the system meets all program requirements. A decision—referred to as Milestone C—is made during this phase to authorize entry of the system into the production and deployment phase or into limited deployment in support of operational testing.

Production and deployment: Achieve an operational capability that meets program requirements, as verified through independent operational tests and evaluation, and implement the system at all applicable locations.

Operations and support: Operationally sustain the system in the most cost-effective manner over its life cycle.

Leading Practices for Managing IT Investments and Acquisition Programs

We have developed and identified leading practices for governing IT investments to help guide organizations to better manage and oversee their projects. GAO’s Information Technology Investment Management guide states that good performance data and stakeholder oversight are elements that can lead to positive outcomes, such as helping to ensure a project is keeping to its initial cost, schedule, and performance goals. The guide also states that projects should be reviewed at regular intervals to monitor performance so that stakeholders can be aware of and review any differences between actual outcomes and goals.

In addition, we and other entities, such as the Software Engineering Institute at Carnegie Mellon University, have identified leading practices to help guide organizations to effectively plan and manage their acquisitions of major IT systems. Our prior reviews have shown that proper implementation of such practices can significantly increase the likelihood of delivering promised system capabilities on time and within budget. These practices include, but are not limited to: Requirements management: Requirements establish what the system is to do, how well it is to do it, and how it is to interact with other systems. Appropriate requirements management involves eliciting and developing customer and stakeholder requirements, and analyzing them to ensure that they will meet users’ needs and expectations. It also consists of validating requirements as the system is being developed to ensure that the final systems to be deployed will perform as intended in an operational environment.

Risk management: Risk management is a process for anticipating problems and developing plans to take appropriate steps to mitigate risks and minimize their impact on program commitments. It involves identifying and documenting risks, categorizing them based on their estimated impact, prioritizing them, developing risk mitigation strategies, and tracking progress in executing the strategies.

DOD’s Policies for Managing MAIS Programs Do Not Always Adhere to Leading IT Management Practices

According to GAO’s Information Technology Investment Management guide, leading practices for managing IT projects include: instituting the investment board, which is the process for creating and defining the membership, guiding policies, operations, roles, responsibilities, and authorities within the organization; identifying decision authorities for making important acquisition decisions; providing oversight whereby the organization monitors each project on its performance progress (e.g., establishing and tracking baseline estimates on cost and schedule goals, and thresholds to identify high risk on cost and schedule); and capturing and providing performance information about a particular investment (project) to decision makers at regular intervals (e.g., quarterly and annually).

To align MAIS programs with the functions they perform, DOD recently made changes in how it characterizes its MAIS programs and, as a result, different programs must follow different management policies. Specifically, in April 2017, DOD identified 10 of 34 total MAIS programs as business programs and the Director, Acquisition Resources and Analysis announced that these programs would adhere to DOD’s Instruction 5000.75 policy for management and oversight. Further, in November 2017, the former Under Secretary of Defense for Acquisition, Technology, and Logistics announced that non-business MAIS programs would adhere to DOD’s Instruction 5000.02 policy for management and oversight.

However, the policies used for MAIS business programs are not consistent in their adherence to leading IT management practices. For example, while the policy for non-business MAIS programs is consistent in its adherence to all four of the leading IT management practices, the policy for MAIS business programs is consistent in its adherence to only two of the four practices. Table 1 shows our analysis of DOD’s policies for non-business MAIS programs and MAIS business program and their adherence to the leading IT management practices.

As shown in the table, DOD’s policy for non-business MAIS programs adheres to all four leading IT management practices. For example, the policy requires non-business MAIS programs to report the status of each program’s cost, schedule, and technical performance information quarterly and annually. The policy also designates specific decision makers who are responsible for monitoring and overseeing the progress of non-business system MAIS programs. Further, the policy requires each program to establish and report their initial baseline estimates and current estimates on cost and schedule so their performance can be tracked and monitored. In addition, to identify when programs may be at risk of significant cost or schedule increases, the policy requires programs to predetermine cost and schedule threshold estimates as an early warning indicator on when programs reach the point where they are at increased risk.

In contrast, DOD’s policy for MAIS business programs policy only adheres to two of the four practices. Specifically, the policy adheres to the practice of instituting an investment board with processes for creating and defining the membership, policies, operations, roles, responsibilities, and authorities within the organization. In addition, the policy identifies decision authorities for making important executive-level acquisition decisions. However, the policy does not specify the establishment of initial and current baseline estimates on cost and schedule, and does not specify the reporting of threshold cost and schedule estimates to identify the point when programs may be at high risk. In addition, the policy does not adhere to leading practices requiring the periodic (quarterly and annual) reporting of performance information to stakeholders.

To help address the need for improved guidance, the former Under Secretary of Defense for Acquisition, Technology and Logistics established a cross-functional team that is to examine the future of non- business MAIS programs and MAIS business programs from a policy, organization, management, and reporting perspective. The team was expected to provide its recommendations to the Under Secretary of Defense for Acquisition and Sustainment by March 15, 2018. However, because no final decisions had been made by the Under Secretary as of that date, it is unclear what specific actions the department will take regarding its policy recommendations, among other recommendations, to improve the management of non-business MAIS programs and MAIS business programs.

Until DOD updates its policy for MAIS business programs to adhere to leading practices on the establishment of baseline estimates on cost and schedule to include threshold estimates on cost and schedule to identify when programs may be at high risk, stakeholders may not have the information they need to manage and oversee MAIS business programs. Further, unless the department updates its policy for MAIS business programs to adhere to the leading practice for periodically (quarterly and annually) reporting essential performance information, stakeholders may not have the information they need to make informed decisions for managing and overseeing MAIS business programs.

All Selected MAIS Programs Had Changes in Cost and Schedule Estimates, and Most Programs Had Met Performance Targets

All of the 15 selected MAIS programs had either increased or decreased their planned cost estimates, and 10 of them had delays in their planned schedule estimates when comparing the first acquisition program baseline to the most recent acquisition program baseline estimates. The changes in the cost estimates ranged from a decrease of $1.6 billion (-41 percent) to an increase of $1.5 billion (163 percent), and slippages in the schedule estimates ranged from a delay of 5 years to a delay of 5 months.

Further, 9 of the 15 selected programs had conducted testing in which we could report on the number of performance targets met for each program. Of those 9, 6 programs reported that they had met all of their performance targets. The remaining 3 programs reported that they met several but not all performance targets. The following table shows the extent of changes in planned cost and schedule estimates for the selected MAIS programs since the first baseline estimate, as well as the number of performance targets met.

All Selected MAIS Programs Had Either Increases or Decreases in Their Planned Cost

All 15 selected MAIS programs had experienced increases or decreases in their planned cost estimates when comparing the initial, or first, baseline estimate to the current estimate. Specifically, 10 programs had decreases in their cost estimates that ranged from $1.2 million (less than -1 percent) for the Defense Agencies Initiative, Increment 2 program to $1.6 billion (-41 percent) for the Air Force’s Base Information Transport Infrastructure Wired program.

Program officials reported that reductions in planned cost estimates were due to changes in program scope. Specifically, the reasons for reduction in cost include:

Program scope changes. Officials for the Air Force’s Joint Space Operations Center Mission System Increment 2 program reported that its 12 percent cost decrease was due to a reduction in its estimate for operations and support that was changed from 20 years to 10 years. Officials for the Defense Information Systems Agency’s Global Combat Support System–Joint Increment 8 program reported that its 20 percent cost decrease was due to a reduction in the program’s scope for the number of development hours required to meet the logistics and operational needs. In addition, officials for the Defense Information Systems Agency’s Teleport Generation 3 program reported that its 22 percent cost decrease was due to a revised scope in terms of what is needed at the Milestone C decision point for low rate production.

Design reconfiguration. Officials for the Air Force’s Base Information Transport Infrastructure Wired program reported that its 41 percent cost decrease was due to a reduction in the program’s scope when they changed from a base network system to a critical core configuration.

In addition, 5 of the programs had experienced cost increases. These cost increases ranged from $2.9 million (less than 1 percent) for the Army’s Logistics Modernization Program Increment 2 to $1.5 billion (163 percent) for the Army’s Tactical Mission Command program. Program officials reported a variety of reasons for the increases in planned cost estimates. These reasons included the following:

Underestimating schedule. Officials for the Air Force’s Defense Enterprise Accounting and Management System Increment 1 program attributed its 60 percent cost increase to underestimating the level of effort that was needed to develop the system within the estimated schedule. For example, the program did not account for software upgrades and, when this effort was added to the schedule to account for the work, the cost increased.

Contractor issues. Officials from the National Security Agency’s Key Management Infrastructure Increment 2 program attributed its cost increase of 14 percent to schedule delays caused by the contractor and, as a result, increased funding at the Milestone C decision point.

Underestimating development and test efforts. Officials from the Army’s Tactical Mission Command program attributed the cost increase of 163 percent to higher than expected costs to conduct research and developmental tests.

Ten Selected MAIS Programs Had Delays in Their Planned Schedule Estimates

Ten of the 15 selected MAIS programs had experienced changes in their planned schedule estimates, and 5 programs had no changes to their schedule estimates. The changes consisted of schedule slippages that ranged from 5 months for both the Army’s Logistics Modernization Program Increment 2 and the Defense Health Agency’s Department of Defense Healthcare Management System program, to 5 years for the Defense Enterprise Accounting and Management System Increment 1 program.

Program officials reported that delays in the planned schedule estimates were due to unplanned budget reductions or unrealistic expectations regarding project milestones. Specifically, the reasons for these schedule slippages included:

Aggressive schedule, funding reduction, and contract issues. Officials for the Air Force’s Joint Space Operations Center Mission System Increment 2 program attributed its schedule slippage of 2 years and 11 months to funding reductions of $18.9 million in fiscal years 2013 and 2014. In addition, the officials noted that an aggressive schedule for a Milestone B decision, contracting issues in the earlier acquisition phase, and longer than expected time to obtain personnel had contributed to the slippage.

Longer than expected time to reach deployment. Officials for the Air Force’s Defense Enterprise Accounting and Management System Increment 1 program reported that its schedule slippage of 5 years occurred because of a change in the approach to deliver the system in multiple increments, thereby increasing the amount of time it would take to reach the deployment decision milestone. Also, officials for the Defense Information Systems Agency’s Teleport Generation 3 program reported a 3-year and 2-month slip. This schedule delay was due to the program’s inability to develop the mobile user and system interface capability by the estimated deployment milestone. Further, program officials for the Navy’s Consolidated Afloat Networks and Enterprise Services program attributed its schedule slip of 2 years and 6 months to a longer than expected maintenance period for the test platform and to a lengthy budget approval process, resulting in a slippage in the deployment date.

Unplanned procurement fund reduction. Officials for the Army’s Global Combat Support System-Army program reported that its schedule delay of 11 months was due, in part, to a $16 million dollar decrease to the fiscal year 2016 budget. This unplanned reduction in procurement fund affected their ability to field the system as originally planned.

Contractor staffing issues. Officials for the National Security Agency’s Key Management Infrastructure Increment 2 program reported significant schedule delays due to the contractor’s inability to staff the program with software developers that had the required security clearances. As a result, a critical change was reported in January 2012 that led to a new independent cost estimate, which extended program development by 10 months. The new estimate included additional time to improve the governance structure, such as increasing discipline across the oversight process, adding more stakeholder interaction, and improving the use of metrics.

Six of Nine MAIS Programs Had Met All Performance Targets

Among other information, DOD uses key performance parameters as a metric to report on programs’ progress toward meeting system performance targets. This information includes a description of the performance characteristics, the objective and threshold value for each target and, importantly, whether the target has been met in demonstrating performance.

Of the nine programs we evaluated, six programs reported that they met all of their performance targets. For example, the Navy’s Common Aviation Command and Control System, Increment 1 program reported in May 2017 that both of its technical performance targets had been met. According to the program, these targets were related to the readiness of the system to fully support all operational activities and satisfy all technical requirements for military operations and the fusion of all kinds of data onto any workstation.

In another example, the Army’s Logistics Modernization Program Increment 2 program reported in June 2017 that all seven of its performance targets had been met. According to the program, these targets were related to the system’s ability to support military operations, exchange information in the network, provide system and information assurance in a disaster recovery scenario, and be operationally available.

Further, three programs reported that they met several, but not all, of their performance targets. For example, the Navy’s Consolidated Afloat Networks and Enterprise Services program reported that it met eight of nine performance targets. According to program officials, the remaining target (i.e., network shall fully support joint critical operational activities) had not been met because the program lacked an operational platform that was required to demonstrate its performance.

The Defense Information Systems Agency’s Teleport Generation 3 program reported that it met 8 of 12 performance targets. According to programs officials, the remaining 4 targets (i.e., coverage to allow warfighter communications, capacity to provide 100 percent of the required services, and interoperability with military and commercial frequencies and wave forms) had not been met because the program needed to field multiple systems and perform solution testing, which they expect to be completed in fiscal year 2018. Further, the Air Force’s Defense Enterprise Accounting and Management System Increment 1 program officials reported that it met 3 of 4 targets (i.e., compliance with requirements, network ready, and sustainment to ensure materiel availability). The officials reported that the program did not meet the remaining target because it was waiting for an evaluation of cyber test results before proceeding.

Selected Programs Fully Implemented Most, but Not All, of the Leading Practices for Managing Requirements and Risk

According to the Software Engineering Institute’s Capability Maturity Model Integration® for Acquisition (CMMI®-ACQ), an appropriate requirements management process involves establishing an agreed-upon set of requirements, ensuring traceability between requirements and work products, and managing any changes to the requirements in collaboration with stakeholders. Likewise, an effective risk management process identifies potential problems before they occur, so that risk-handling activities may be planned and invoked, as needed, across the life of the project in order to mitigate the potential for adverse impacts.

Leading requirements management practices help organizations to better manage the design, development, and delivery of systems within established cost and schedule time frames. These practices include developing an understanding with the requirements providers of the meaning of the requirements, obtaining commitment to requirements from project participants, managing changes to requirements as they evolve during the project, maintaining bidirectional traceability among requirements and work, ensuring that project plans and work products remain aligned with requirements.

An effective risk management process includes the following leading practices determining risk sources and categories; defining parameters used to analyze and categorize risks and to control the risk management effort; establishing and maintaining the strategy to be used for risk identifying and documenting risks; evaluating and categorizing each identified risk using defined risk categories and parameters, and determining its relative priority; developing a risk mitigation plan in accordance with the risk monitoring the status of each risk periodically and implementing the risk mitigation plan as appropriate.

The three selected MAIS programs that we evaluated had fully implemented most, but not all, of the five leading practices for managing requirements and the seven leading practices for managing risks. Specifically, two of three programs implemented all of the requirements management practices, while one program implemented most, but not all, of the practices.

Further, one of three programs implemented all of the risk management practices, while two programs implemented most, but not all of the practices. Table 3 shows the extent to which practices were implemented by the three selected programs.

Two Programs Fully and One Program Partially Implemented Leading Practices for Managing Requirements

Two of the three programs had fully implemented the requirements management practices. The other program had partially implemented two practices and fully implemented three practices.

Navy — Navy Consolidated Afloat Networks and Enterprise Services The Navy had fully implemented the five requirements management practices for the Consolidated Afloat Networks and Enterprise Services program. For example, the program developed an understanding with requirements providers of the meaning of the requirements. Specifically, there was a plan for documenting, managing, and controlling changes to requirements throughout the system lifecycle. This plan served as the primary guidance for integrating the management of all specified and derived requirements for the Consolidated Afloat Networks and Enterprise Services system program.

In addition, the program had established criteria for determining requirements providers. Specifically, roles and responsibilities for requirements management had been identified. Further, the program managed changes to requirements as they evolved during the project. For example, the program provided evidence that it maintains a requirements change history, including the rationale for changes.

Defense Logistics Agency — Defense Agencies Initiative, Increment 2 The Defense Logistics Agency had fully implemented the five requirements management practices for the Defense Agencies Initiative, Increment 2. For example, the program had established objective criteria for the evaluation and acceptance of requirements. Specifically, there was a process in place to develop and finalize deliverables in support of the business requirements identified by the stakeholders, ensure that requirements management activities were performed in a timely manner throughout the life of the project, and review and approve requirements deliverables. Further, throughout the process, the requirements manager tracked requirements changes and maintained traceability of end user needs to the system performance specification.

The Defense Health Agency had fully implemented three and partially implemented two of the five requirements management practices for the Defense Healthcare Management System Modernization program. For example, the program had established objective criteria for the evaluation and acceptance of requirements. Specifically, any new or updated requirements were presented to a Configuration Steering Board for review and approval prior to any changes being made. Further, throughout the process, the requirements manager tracked requirements changes and maintained traceability to ensure they were documented.

The program has not developed an understanding with the requirements providers on the specific meaning of the requirements. For example, although the program had developed a requirements management plan which provided guidance in this area, according to program officials, the plan was not signed and approved based on the recent shift of the program from a non-business MAIS program to a MAIS business program operating under DOD Instruction 5000.75. Program officials stated that the requirements management plan is not expected to be complete until final guidance is provided by the Office of the Secretary.

Regardless of this recent shift, the program should have already had an approved requirements management plan in place since program initiation. In the absence of an approved plan, the program lacks assurance that it can effectively communicate and manage requirements practices.

Further, the program had not demonstrated that it identified any changes that should be made to plans and work products resulting from changes to the requirements baseline. Programs officials stated that efforts to review modifications to the plan due to requirements changes had not been conducted, but they expected the review and approval to be done at some future date. However, they could not provide a specific time frame. According to CMMI®-ACQ, until project plans and work products are updated to coincide with changes in requirements, the program will not be able to effectively identify inconsistencies between requirement changes and project plans and work products, and initiate corrective actions to resolve them.

One Program Fully and Two Programs Partially Implemented Applicable Risk Management Leading Practices

One program had fully implemented the risk management practices, while two had fully implemented all but one practice.

Navy — Navy Consolidated Afloat Networks and Enterprise Services The Navy had fully implemented six and partially implemented one risk management practice for the Consolidated Afloat Networks and Enterprise Services system program. For example, the program’s risks defined consistent criteria for evaluating and quantifying risk likelihood and severity levels. Specifically, the program provided a risk exposure (e.g., a risk source used to examine and oversee changes that impact the project), which is the value that is given to a risk event, a product, or the overall program based on the analysis of the probability and consequences of the event. Further, the program’s Risk Management Guide outlined risk performance, cost, and schedule criteria. In addition, the program demonstrated that it included the cost and benefits of implementing risk mitigation plans. Specifically, a risk’s description provided the cost impacts associated with the risk, which in turn provided evidence that cost and benefits were considered during risk evaluation.

However, the Navy partially implemented one practice. Specifically, although the program provided its failover/recovery plan that is intended to return the program to a state of readiness after a failure, the plan did not explicitly identify environmental elements. A program official stated that environmental factors, such as risks that could negatively affect their work, is understood, but these factors had not been documented in the plan. Further, the official stated that the program should update the plan accordingly, but did not provide a time frame to complete this effort. Until all potential issues, hazards, threats, and vulnerabilities that could negatively affect work efforts have been identified in the plan, successful risk management cannot be ensured.

Defense Logistics Agency—Defense Agencies Initiative, Increment 2 The Defense Logistics Agency had fully implemented all seven risk management practices for the Defense Agencies Initiative, Increment 2. For example, the program identified program risks, including risk sources, categories, and stakeholders. In addition, Defense Agencies Initiative, Increment 2 risks followed consistent criteria for evaluating and quantifying risk likelihood and severity levels. Specifically, risk level was based on a combination of factors to include both likelihood and consequence. In all instances, consensus on the risk levels was required between the risk owner and the customer counterpart. Further, the program’s contingency plan provided guidance when outages fell into one of three disaster categories including natural disasters, man-made disasters, and technological disasters.

Defense Health Agency—Defense Healthcare Management System Modernization The Defense Health Agency had fully implemented six and partially implemented one of the seven risk management practices for the Defense Healthcare Management System Modernization program. For example, the program’s risks followed consistent criteria for evaluating and quantifying risk likelihood and severity levels. Specifically, the program’s Risk and Issue Management Plan described how to assess the impact level in each risk area (performance, project and program schedules, and cost). Further, the program prioritized risks for mitigation. For example, risks were categorized and charted as low, medium, or high, and grouped accordingly in the program’s risk register. Further, the program’s Disaster Recovery Plan provides processes that allowed rapid support recovery for critical operations during a disaster, including environmental disasters such as tornadoes.

Regarding the partially implemented practice, the program provided an example of a risk mitigation plan. However, the program indicated that costs and benefits were not quantified within the program-level risk mitigation plans. According to CMMI®-ACQ, risk mitigation activities should be examined for benefits they provide versus resources they will expend. Just like any other design activity, alternative plans may need to be developed and costs and benefits of each alternative assessed.

However, the program does not require that costs and benefits be included as part of its risk mitigation planning efforts. As a result, the information for making an informed decision on cost and benefits of risk mitigation solutions is limited. Program officials did not indicate whether they have plans to implement this practice, and did not provide an explanation as to why they are unable to provide this information. Until the program quantifies costs and benefits, it will not be able to effectively select the most appropriate risk mitigation plan to address each risk.

Conclusions

While DOD’s policy for non-business MAIS programs adheres to all four leading IT management practices, the department’s policy for MAIS business programs does not adhere to two leading practices on establishing initial and current baseline estimates on cost and schedule and predetermining threshold estimates, as well as reporting periodically on performance information to stakeholders. Until DOD adheres to these practices in its policies that govern MAIS business programs, it cannot ensure that stakeholders will have the information they need to manage and oversee their investments.

Following leading IT acquisition practices on requirements and risk management is essential to help programs effectively plan and direct their development and acquisition efforts. All of the leading IT acquisition practices for requirements and risk management had been fully or partially implemented by three programs that we reviewed. However, the Defense Health Agency’s Defense Healthcare Management System Modernization has not finalized its requirements management plan nor has it identified changes that should be made to plans and work products resulting from changes to the requirements baseline. Until the program addresses these practices, it will lack a comprehensive plan for managing its requirements and it may not be able to effectively identify inconsistencies and initiate corrective actions.

Further, the Navy Consolidated Afloat Networks and Enterprise Services program did not fully identify and document risks that could negatively affect work efforts. In addition, the Defense Health Agency’s Defense Healthcare Management System Modernization did not quantify costs and benefits of risk mitigation within its program-level risk mitigation plans. As a result, successful risk management for avoiding, reducing, and controlling the probability of risk occurrence cannot be ensured.

Recommendations for Executive Action

We are making the following three recommendations to the Secretary of Defense to direct:

The Under Secretary of Defense for Acquisition and Sustainment to update the policy or guidance for MAIS business programs. Specifically, the update should include the following elements: establishment of initial and current baseline cost and schedule predetermined threshold cost and schedule estimates to identify the point when programs may be at high risk, and quarterly and annual reports on the performance of programs to stakeholders. (Recommendation 1)

The Director of the Defense Health Agency to direct the program manager for the Defense Healthcare Management System Modernization program to: finalize and approve its requirements management plan, identify and document changes that should be made to plans and work products resulting from changes to the requirements baseline, and quantify costs and benefits of risk mitigation within its program- level risk mitigation plans. (Recommendation 2)

The Secretary of the Navy to direct the program manager for the Navy Consolidated Afloat Networks and Enterprise Services program to: identify and document, in the failover/recovery plan, all potential external environmental issues, such as hazards, threats, and vulnerabilities that could negatively affect work efforts. (Recommendation 3)

Agency Comments and Our Evaluation

DOD provided written comments on a draft of this report, which are reproduced in appendix II. In its comments, the department partially concurred with our first recommendation and concurred with the two other recommendations.

DOD partially concurred with the first recommendation on updating the policy or guidance for MAIS business programs. Specifically, the Under Secretary of Defense for Acquisition and Sustainment stated that regarding establishing baselines, the DOD Instruction 5000.75 requires establishment of cost, schedule, and performance parameters for each release before development or delivery. The 5000.75 also requires consideration of program progress against baselined cost, schedule, and performance as a criterion at the limited deployment and full deployment decision points. A baseline requirement thus exists in DOD Instruction 5000.75 but it is not described as an acquisition program baseline, which may be familiar to readers of DOD Instruction 5000.02. However, the Under Secretary added that the Army’s implementation guidance includes guidance that states each increment must have an acquisition program baseline with its own set of threshold and objective values set by the user.

While we agree that the existing policy requires such parameters to be captured and included in the department’s decision making process, we found the policy to be vague in its discussion of these parameters and to not clearly define what a baseline is, or which baselines are to be used or reported for comparison purposes. For example, the policy does not make a distinction between the initial acquisition program baseline, current baseline, and baseline deviations. Yet, such information is important because it provides a basis for decision makers to identify the extent to which a program may have deviated from its initial cost, schedule, or technical performance baseline. By making these distinctions in the policy, the department’s policy for its MAIS business programs will be more consistent with its other policy for non-business MAIS programs with regard to the way an acquisition program baseline is defined and the elements that should be captured and reported to its decision makers. In turn, the program managers who prepare these reports and the decision makers who rely on them will have information that is consistently and succinctly prepared for making credible decisions.

Regarding adding provisions in its policy for the establishment of predetermined thresholds, the Under Secretary stated that the 5000.75 states that the milestone decision authority is responsible for delivery within cost, schedule, and performance parameters, and the milestone decision authority is to do this by establishing oversight controls for programs, including procedures to report and address variances. The Under Secretary added that the 5000.75 does not suggest the practice of establishing a predetermined threshold for the variance, and DOD will consider the addition of this feature to the 5000.75 update.

Finally, regarding providing periodic annual and quarterly reports to the department’s leadership, the Under Secretary stated that such a periodic report would add value only if there had been no recent communication of program status from the program office to the leadership or stakeholders communities. While such communication is expected to occur frequently, its regularity is not specified in current policy or guidance. The Under Secretary stated that DOD will consider adding a provision for a report to leadership and functional stakeholder if such communication has not occurred within the past 3 or 4 months.

DOD concurred with the second and third recommendations related to the department’s implementation of selected IT management practices. Regarding the second recommendation, the Under Secretary of Defense for Acquisition and Sustainment agreed to direct the Defense Healthcare Management System Modernization program manager to update and approve the requirements management plan, identify and document changes to the requirements baseline, and quantify the costs and benefits in the risk mitigation plans. Further, regarding the third recommendation, the Secretary of the Navy agreed to direct the program manager to identify and document all potential external environmental issues that could negatively affect work efforts for the Navy’s Consolidated Afloat Networks and Enterprise Services program. By taking these steps, these programs should be better positioned to effectively identify inconsistencies in managing changes to their requirements, and be more responsive to the potential for environmental issues.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force; the Under Secretary of Defense for Acquisition and Sustainment; the Director of the Defense Health Agency; and other interested parties. This report also is available at no charge on the GAO website at http://www.gao.gov.

Should you or your staffs have any questions on information discussed in this report, please contact me at (202) 512-4456 or harriscc@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made major contributions to this report are listed in appendix III.

Appendix I: Objectives, Scope, and Methodology

The National Defense Authorization Act for Fiscal Year 2012 mandated that we select, assess, and report on selected major automated information systems (MAIS) programs annually through March 2018. GAO satisfied the statutory mandate by submitting a draft of this report to the congressional committees on March 29, 2018. This final version of the report is the sixth and last report in the series of annual mandated assessments.

Our objectives were to: (1) assess the Department of Defense’s (DOD) policy for the management and oversight of MAIS programs; (2) describe the extent to which selected MAIS programs have changed their planned cost and schedule estimates and met performance targets; and (3) assess the extent to which selected MAIS programs have used leading information technology (IT) acquisition practices, including requirements and risk management.

To address the first objective, we identified four leading IT management practices in GAO’s Information Technology Investment Management guide and compared DOD’s policy adherence to those practices. These leading practices are: instituting the investment board, which is the process for creating and defining the membership, guiding policies, operations, roles, responsibilities, and authorities within the organization; identifying decision authorities for making important acquisition decisions; providing oversight whereby the organization monitors each project on its performance progress (e.g., establishing and tracking baseline estimates on cost and schedule goals, and thresholds to identify high risk on cost and schedule); and capturing and providing performance information about a particular investment (project) to decision makers at regular intervals (e.g., quarterly and annually).

We then compared DOD’s policies used to manage and oversee the department’s non-business MAIS programs and MAIS business programs against these leading IT management practices. The department’s policy documents for managing and overseeing non-business MAIS programs and MAIS business programs include the:

Memorandum by the Under Secretary of Defense for Acquisition, Technology, and Logistics, dated November 17, 2017, regarding the regulatory response to the repeal of title 10, United States Code, Chapter 144A, Major Automated Information System Programs.

Memorandum by the Under Secretary of Defense for Acquisition, Technology, and Logistics, dated April 24, 2017, regarding the transition of programs to business system categories.

DOD Instruction 5000.75, Business Systems Requirements and Acquisition, effective February 2, 2017.

DOD Instruction 5000.02, Operation of the Defense Acquisition System, effective February 2, 2017.

We also interviewed an official from the former Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, who was responsible for the development of plans and policy for the administration regarding the management and monitoring of non-business MAIS programs and MAIS business programs To address the second objective, we used DOD’s official list of 34 business and non-business MAIS programs, as of April 18, 2017, to establish a basis for selecting programs. Of the 34 programs, we selected the 15 business and non-business MAIS programs that met our criteria: programs must be unclassified and have an initial acquisition program baseline that could be used as a reference point for evaluating cost, schedule, and technical performance characteristics.

We then collected and analyzed key documents, reports, and artifacts for each program and summarized the information on estimated cost, schedule, and technical performance goals, including their latest program status in meeting those estimated goals. Next, we analyzed and compared each selected program’s first acquisition program baseline cost estimate to the latest estimate to determine the extent to which planned program costs had changed. Specifically, we used the total life-cycle cost estimate and analyzed and compared them to the latest estimate to determine the extent to which planned program costs had changed. Similarly, to determine the extent to which these programs changed their planned schedule estimates, we compared each program’s first acquisition program baseline schedule to the latest schedule.

To determine whether the selected programs met their performance targets, we analyzed each program’s self-identified system performance targets and compared them against actual system performance metrics and latest test reports. We also reviewed additional information on each program’s cost, schedule, and performance, including program documentation, such as DOD’s MAIS annual and quarterly reports, acquisition program baselines, system test reports, and our prior reports. We then aggregated and summarized the results of these analyses across the programs.

To address the third objective, we started with the list of the 15 programs from the second objective as a basis for selecting three MAIS programs as case studies. We used a combination of the following criteria to select the MAIS programs to review.

Programs used in a most recent MAIS review were eliminated from consideration.

The program was not designated as classified.

The program had a baseline.

Based on these criteria, we chose the following systems:

Navy Consolidated Afloat Networks and Enterprise Services;

Defense Logistics Agency’s Defense Agencies Initiative, Increment 2;

Defense Health Agency’s Defense Healthcare Management System Modernization.

We then analyzed each selected program’s IT acquisition documentation and compared it to key requirements and risk management and leading practices—including Software Engineering Institute’s Capability Maturity Model® Integration for Acquisition (CMMI- ACQ) practices—to determine the extent to which the programs were implementing these practices. In particular, the requirements management practices we reviewed were: develop an understanding with the requirements providers on the meaning of the requirements, obtain commitment to requirements from project participants, manage changes to requirements as they evolve during the project, maintain bidirectional traceability among requirements and work, and ensure that project plans and work products remain aligned with requirements.

Specifically, we analyzed program requirements documentation, including requirements management plans, requirements traceability matrices, requirements change forms, technical performance assessments, and requirements board meeting minutes. Additionally, we interviewed program officials to obtain additional information about their requirements management practices. The conclusions reached for this objective are not generalizable to the larger population of 34 business and non-business MAIS programs.

We also reviewed the following risk management practices: determine risk sources and categories; define parameters used to analyze and categorize risks and to control the risk management effort; establish and maintain the strategy to be used for risk management; identify and document risks; evaluate and categorize each identified risk using defined risk categories and parameters, and determine its relative priority; develop a risk mitigation plan in accordance with the risk management strategy; and monitor the status of each risk periodically and implement the risk mitigation plan as appropriate.

Specifically, we analyzed program risk documentation, including risk reports, risk-level assignments, risk management plans, risk mitigation plans, and risk board meeting minutes. Additionally, we interviewed program officials to obtain additional information about their risks and risk management practices.

To assess the reliability of the data of these programs we used to support the findings in this report, we corroborated program office responses with relevant program documentation and interviews with agency officials. We found no data reliability issues and determined that the data used in this report were sufficiently reliable for our reporting purposes. We have also made appropriate attribution indicating the sources of the data.

We conducted this performance audit from April 2017 to May 2018 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact name above, the following staff also made key contributions to this report: Eric Winter (Assistant Director), John Ortiz (Analyst in Charge), Alex Bennett, Neha Bhatt, Chris Businsky, and Rebecca Eyler.