Defense Acquisitions: Better Approach Needed to Account for Number, Cost, and Performance of Non-Major Programs
Released: 2015-03-02T00:00:00

Letter

Background

DOD acquisition policy defines an acquisition program as a directed, funded effort that provides a new, improved, or continuing materiel, weapon, or information system, or a service capability in response to an As shown in table 1, defense acquisition programs are approved need.classified into acquisition categories that depend on the value and type of acquisition. The Army, Navy, Air Force, and SOCOM also have supplemental acquisition policies that address certain aspects of acquisition program categorization and management.

ACAT II and III programs encompass a wide range of efforts and program sizes. Programs may range from an ACAT II program with a total acquisition cost of more than $3 billion to an ACAT III program with an acquisition cost in the millions of dollars or lower. DOD’s acquisition policy does not establish a minimum cost for ACAT III programs.

The level of oversight for acquisition programs varies based on the assigned ACAT level. DOD and component acquisition policies specify the organizational level of the milestone decision authority—the designated individual with overall responsibility for a program—for each ACAT level. The organizational level at which program requirements and requirements changes are approved may vary by ACAT level as well. The organizational level of the milestone decision authority for Air Force ACAT I-III programs is shown in figure 1 as an example.

All acquisition programs are required by statute or DOD guidance to provide program information at milestones and other decision points, although these requirements differ by ACAT level. MDAP and MAIS programs, also known as ACAT I and IA programs, require more documentation and analysis to support program decisions and have to regularly report to Congress on their cost, schedule, and technical performance. These programs are required to enter and maintain program cost, schedule, and performance data and create APBs within DOD’s Defense Acquisition Management Information Retrieval (DAMIR) system, a web-based data system intended to provide data transparency of acquisition management information across DOD. Components may use DAMIR for other programs, but it is not required. Appendix II provides additional detail on acquisition documentation requirements and congressional reporting requirements by ACAT level.

DOD Does Not Have Reliable Data on ACAT II and III Programs and Component Actions to Improve Data Do Not Fully Address Limitations

DOD components could not provide sufficiently reliable data for us to accurately determine the number, total cost, or performance of DOD’s current ACAT II and III programs. We found that the accuracy, completeness, and consistency of DOD’s data was undermined by (1) widespread data entry issues and missing data, and (2) inconsistent identification of current ACAT II and III programs across and within components. DOD components have taken some steps to improve ACAT II and III program data, but their efforts do not fully address the causes of the problems we identified. In addition to data reliability problems, DOD lacks consistent cost and schedule metrics across components to assess ACAT II and III program performance. Further, the lack of baseline cost and schedule data and comparable schedule milestones prevents DOD from consistently measuring the performance of ACAT II and III programs. Taken together, these issues limit the utility of DOD’s data on ACAT II and III programs for oversight, decision-making, and reporting purposes.

DOD Components Cannot Provide Accurate, Complete, or Consistent Data on Current ACAT II and III Programs

We identified data reliability issues related to accuracy, completeness, or consistency with data for about 60 percent of ACAT II and III programs reported to us by DOD components. These issues prevented us from accurately determining the number, total cost, or performance of DOD’s current ACAT II and III programs. According to DOD acquisition policy, complete and current program information is essential to the acquisition process. Internal control standards for federal executive branch agencies also emphasize that agencies should have relevant, reliable, and timely information for decision-making and external reporting purposes. We found obvious accuracy and completeness issues in program cost data for ACAT II and III programs reported to us by DOD components. We also observed consistency issues in program data across components and within some components that affected the comparability of the data.

Widespread Data Entry Issues and Missing Data

Inaccurate data were evident across all of the components; issues we observed included reported dollar values outside the range of ACAT II and III programs and basic math errors. Inaccuracies like these suggest overall data quality problems. Further, when we reviewed a sample of programs and compared reported cost estimates to source documents, we found that cost estimate data was often misreported. Components incorrectly reported data or data was missing for 64 out of 95 programs for which we had complete source documents in our non-generalizable sample. We also observed missing data elements to varying degrees at all of the components except CBDP. For example, 333 out of 836 programs reported by the components were missing one or more cost estimate elements or basic information such as the ACAT level. Lastly, in numerous instances components did not follow the instructions of the data collection instrument, which also affected our ability to use the data. See table 2 for examples of the reliability issues we identified.

In some instances, the accuracy and completeness issues we identified were consistent with known limitations of information in component systems. For example, Army acquisition officials told us that in the past there has been conflicting guidance about whether completed programs should be deleted from the Army’s acquisition information system, and some completed programs were never removed from the system. Officials at all components, except CBDP, further told us that accuracy of data in their systems relies primarily on the quality of information submitted by the Program Executive Officer (PEO) or program offices.Army, Navy, and Air Force acquisition officials told us that they work with PEOs to address data quality problems, such as by conducting ad hoc checks to flag obvious errors or missing data elements and following up with PEOs as necessary. SOCOM acquisition officials told us the Acquisition Executive emphasizes the importance of maintaining up-to- date data to program managers. Based on the data provided to us by DOD components, these existing data quality practices are insufficient to ensure the accuracy and completeness of data on DOD’s ACAT II and III programs as required by DOD policy and federal internal control standards.

Inconsistent Identification of Current ACAT II and III Programs Across and Within Components

Data provided in response to our request for information on current ACAT II and III programs (1) included acquisitions that were not “current” ACAT II and III programs in accordance with our definition, (2) likely reflected inconsistent reporting of acquisitions at lower dollar levels across components, and (3) likely excluded certain acquisitions considered current ACAT II and III programs in accordance with certain component acquisition policies. For the purposes of our report, we define a current program as one that has been formally initiated in the acquisition process, but has not yet delivered 90 percent of its items or made 90 percent of planned expenditures. This definition is consistent with statutory reporting thresholds used by Congress in its reporting requirements for current MDAP programs.

Inconsistent interpretations of what constitutes a current program and the inability of some components to reliably identify these programs contributed to the inclusion of programs that were not current in the data that components reported to us. DOD acquisition policy and component guidance generally do not define which ACAT II or III programs are considered to be current for management and reporting purposes. For example, DOD acquisition policy defines when a program is formally initiated and its operations and support phase begins, but it does not identify when a program should be considered current. Further, some components told us PEOs may have different interpretations of what constitutes a current program. Components also told us that they could not consistently use the information in their data systems to readily identify current programs. Of the 836 programs initially reported by the five components, we identified 199 programs across the Army, Navy, Air Force, and SOCOM that should not have been included because they did not meet our criteria for current programs. For example, according to an Army PEO, 90 of 140 programs originally reported to us as current ACAT II or III programs by the Army were not current based on our definition because these programs had delivered more than 90 percent of planned items or expended more than 90 percent of planned funds. Without the consistent identification of current ACAT II and III programs, DOD and component officials do not know the accurate number of these programs and may miss opportunities to identify programs that may need more or less oversight depending on whether or not most of the anticipated acquisition funding has been spent.

Additionally, component guidance for defining acquisition programs varies and likely resulted in inconsistent reporting across components of acquisitions at lower dollar values. DOD acquisition policy establishes a cost ceiling and cost floor for ACAT II programs and a cost ceiling, but no cost floor, for ACAT III programs. However, some components have supplemental guidance that establishes additional acquisition categories or exclusions. For example, SOCOM and Navy have guidance that provides for certain acquisitions with less than $10 million in total RDT&E contracts, less than $25 million per year in annual procurement funding, and less than $50 million total in procurement contracts to be categorized as non-ACAT programs. Specifically, SOCOM designates these low cost, schedule, and technical risk efforts to field special operations-peculiar capabilities as abbreviated acquisition projects. Similarly, the Navy designates lower dollar value programs that do not require operational testing and evaluation as abbreviated acquisition programs.Air Force policies do not provide for lower dollar threshold categories Army and beneath the ACAT III level. According to officials from the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, acquisition programs at these dollar levels should be reported as ACAT III programs. However, Navy and SOCOM did not include their non-ACAT programs in the ACAT II and III program data reported to us.

There was also variation within components as to how acquisition programs were categorized. Army and Air Force acquisition officials told us that some programs that should have been considered ACAT II or III programs in accordance with component acquisition policy were not. These officials also told us that PEOs may have counted and handled programs differently in the absence of a clear definition of what should be considered a program of record. For example, Army officials told us that categorizing information technology programs was sometimes challenging, and they have worked with PEOs to review the categorization of certain information technology programs. Army and Air Force officials told us that as a result of confusion among PEOs about whether or not certain programs should be considered ACAT II or III programs, they have needed to add and remove numerous ACAT II and III programs from component information systems over the past year.

The types of issues we identified may have also contributed to components reporting varying numbers of ACAT II and III programs in response to different requests for information during the same time frame. Specifically, concurrent with reporting 755 current ACAT II and III programs to us, the Army, Navy, and Air Force reported 1,360 ACAT II and III programs in a presentation to the DOD Business Senior Integration Group, which is chaired by the Under Secretary of Defense for Acquisition, Technology, and Logistics and oversees DOD’s Better Buying Power initiatives. Acquisition officials from these components told us they were unable to fully explain the reasons for the difference between the numbers of programs reported.

Components Have Taken Steps to Improve ACAT II and III Program Data, but Have Not Fully Addressed the Causes of the Reliability Problems

The Army, Navy, Air Force, and SOCOM have established information systems to track cost and schedule data for ACAT II and III programs and taken some steps to address issues related to the completeness and accuracy of information tracked in these systems. CBDP officials told us they recognize the value of establishing a system to track data on ACAT II and III programs and are determining the capabilities that would be needed in such a system. Specifics of component efforts follow:

According to Navy officials, data in the Navy’s Research, Development & Acquisition Information System has potentially been incomplete because ACAT II and III program data has not been consistently entered into the system. The Navy issued an updated policy in August 2014 that requires input of programmatic information into the system for all ACAT programs.

All Air Force ACAT II and III programs have been required to enter cost and schedule data into the System Metric and Reporting Tool since 2012, but Air Force officials told us that not all programs had complied with the requirement. They told us in June 2014 that they had an ongoing effort to review ACAT II and III programs in the system, including assessing whether cost and schedule data has been populated. Further, the Air Force has established an investment master list that will capture all programs receiving RDT&E and procurement funding.

SOCOM requires that all ACAT II and III programs enter program and cost data into its centralized acquisition portal data system, however SOCOM officials told us some program managers have been more diligent than others in populating the system. These officials told us that data from the portal is now used to conduct monthly program reviews, rather than having the program prepare briefing slides, as a way to encourage program managers to populate and regularly update the system.

Components have also taken steps to improve the consistency with which they identify current ACAT II and III programs. For example, officials at four of the five components told us they are exploring ways to identify the acquisition phase—technology maturation and risk reduction, engineering and manufacturing development, production and deployment, operations and support—for programs in component information systems, which could improve their ability to reliably identify current programs. The Air Force and Army have also made efforts to address concerns they had previously identified related to the consistent identification of ACAT II and III programs within their components. The Air Force issued guidance in January 2014, detailing which programs are and are not considered to be acquisition programs and acquisition policy officials told us they have been meeting with individual PEOs to clarify any misunderstandings about how acquisition programs should be categorized. Officials from the Office of the Assistant Secretary of the Army for Acquisition, Logistics, and Technology also told us they are in the process of revising the Army’s acquisition guidance, with input from PEOs, to more precisely define the types of programs that should and should not be considered to be acquisition programs.

However, the components’ efforts do not fully address the accuracy, completeness, and consistency issues we identified with ACAT II and III program data. For example, the components have not established systematic processes to perform data quality tests on PEO-submitted data and assess the results to help identify problems, such as basic math errors or missing data, for further review. These types of tests and assessments can be an important step in determining whether data can be used for its intended purposes. Additionally, the components have not developed plans that detail how they will implement or sustain data improvement efforts. For example, the components have not developed implementation steps for assessing data reliability on an ongoing basis or metrics to assess the success of data cleanup efforts. Developing such Without establishing this plans is a key project management practice.planning foundation, the components will not be in a sound position to effectively monitor and evaluate the implementation of efforts to improve component data. Finally, efforts to consistently identify current ACAT II and III programs have been focused within individual components. Without a consistent understanding about which programs should be considered to be current ACAT II or III programs across components, similar programs will continue to be reported on differently, thereby limiting the consistency and comparability of ACAT II and III program data across DOD.

DOD Lacks Consistent Cost and Schedule Performance Metrics and Faces Additional Challenges Measuring Trends

DOD components lack consistent cost and schedule performance metrics to assess performance trends across ACAT II and III programs. As part of the department’s Better Buying Power initiatives, the Under Secretary of Defense for Acquisition, Technology, and Logistics instructed DOD components to determine how best to measure performance trends for non-ACAT I programs. Federal internal control standards also emphasize the importance of comparing actual performance to planned or expected results throughout the organization to help ensure effective results are achieved and actions are taken to address risks. The Army, Navy, and Air Force briefed DOD’s Business Senior Integration Group in November 2013 on their current efforts and plans regarding assessing ACAT II and III cost and schedule performance, but no specific follow-on actions or action plans have been developed.

Unlike MDAPs and MAIS programs, ACAT II and III programs are not required to report cost and schedule data in a consistent fashion, despite the potential benefits of such reporting. MDAP and MAIS programs are required to report key cost and schedule metrics to Congress in a standardized format through Selected Acquisition Reports and MAIS Annual Reports, respectively. Cost and schedule data for these reports are pulled from DOD’s web-based DAMIR system. MDAP cost and schedule data are used by DOD for its annual assessment of the performance of the defense acquisition system, which the department uses to improve acquisition program performance and inform policy and programmatic decisions. ACAT II and III programs are not required to produce similar cost and schedule reporting as larger programs and do not have to provide program data in DAMIR. According to officials from the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics, although minor adjustments may be needed for reporting purposes, there is nothing that prevents components from using DAMIR to capture data on ACAT II and III programs, and acquisition officials from the Army have considered using it. Additionally, CBDP officials told us they would consider DAMIR when exploring potential systems to track ACAT II and III program data.

DOD component officials told us they are not yet sure how best to measure cost and schedule performance across ACAT II and III programs. For example, Army officials told us that analysis of component- wide ACAT II and III performance trends may not make sense given the differences across programs. Navy, SOCOM, Air Force, and CBDP officials told us they are interested in tracking cost and schedule performance trends across ACAT II or III programs, but are still working to define performance metrics and address limitations in existing data or reporting capabilities. For example, the Air Force has attempted to assess cost and schedule performance for a subset of ACAT II and III programs, but acquisition officials noted that the process was very resource- intensive, and they had concerns about the reliability of the cost and schedule information used in their analysis given the lack of variability in program performance over time. While the components have developed oversight mechanisms to review individual ACAT II and III program performance, such as through periodic program status reviews at the PEO level and program or portfolio reviews by senior component acquisition officials, without assessing performance trends across ACAT II and III programs, DOD and its components may be missing opportunities to identify and analyze differences between actual and expected performance and develop strategies to address related risks throughout the department.

Data Limitations for Measuring Cost and Schedule Performance Trends

When we analyzed information available on cost and schedule performance, we determined that we could not assess cost performance for 139 programs out of a non-generalizable sample of 170 programs and schedule performance for 105 of the 170 programs. In addition to missing or misreported cost data, we identified two challenges to measuring cost and schedule performance trends for ACAT II and III programs: (1) programs without available APBs and (2) a lack of consistent and comparable key schedule milestones across programs. See figure 2 for a summary of our assessment of the data available to measure cost and schedule performance and appendix III for additional details.

For 75 of the 170 programs that we examined in detail, we could not assess cost or schedule performance because DOD components had not developed, or did not provide, an original APB, a current one, or both. The components were unable to provide APBs for various reasons, such as because they could not locate the original or an APB was not developed at program start or to this point in the life of the program. APBs are critical management tools that establish how systems will perform, when they will be delivered, and what they will cost. According to DOD acquisition policy, APBs are required of all acquisition programs, and Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics officials told us they generally expect all acquisition programs to have one.prior to entry into system development (Milestone B), or at program initiation, whichever occurs later. APBs may be revised at the time of significant program decisions, such as milestones, or as a result of major program changes or breaches to cost, schedule, or performance parameters. Table 3 shows the number of missing APBs by component in our sample.

ACAT II and III Program Performance Most Frequently Affected by Requirements Changes and Testing Issues

Thirteen of the 15 ACAT II or III programs we reviewed in-depth had exceeded the cost or schedule targets in their original APBs. These programs cited changing requirements, testing issues, quantity changes, and flaws in original cost estimates, among other factors, as the reasons for cost and schedule growth. The programs we reviewed cited other factors, such as a reliance on mature technology—including commercial or government off-the-shelf or other non-developmental items—and early involvement of stakeholders or users as contributing to reduced risk of cost or schedule growth. We have previously reported that similar factors affect the performance of DOD’s MDAPs. Appendix IV provides additional details about the programs we reviewed.

Programs Most Frequently Attributed Cost and Schedule Growth to Requirements Changes

Thirteen of the 15 ACAT II or III programs we reviewed in-depth had exceeded the cost or schedule targets in their original APBs.attempt to quantify the extent to which these programs had exceeded cost or schedule targets due to overall concerns about the reliability of ACAT II and III cost and schedule data and because not all of these programs had developed APBs at program start. These programs most frequently attributed cost growth or schedule delays to changing requirements. Testing issues, quantity changes, and flaws in original cost estimates were also cited by at least 5 of the 13 programs as contributing We did not to cost growth or schedule delays. All but 1 of the 13 programs cited multiple causes for cost growth or schedule delays, including factors beyond those listed in table 4.

Requirements changes were associated with cost growth or schedule delays by at least one program at each of the five components in our review. According to program officials, programs added or increased requirements due to situations such as: adding capability to a new platform that had not been planned for when the original requirements were approved; creating additional variants to meet requirements that emerged after the original requirements were approved; or making improvements or refinements to a system in development or production as a result of changes in the operational environment, including new threats. For example, officials from the Army’s Synthetic Environment Core program, which is providing the Army a common virtual environment that links virtual simulators and simulations into an integrated and interoperable training environment, told us that increasing terrain database requirements to meet additional training needs have contributed to program cost increases significant enough to require the program to be recategorized from an ACAT III to an ACAT II program. Program officials stated that in some cases, the additional requirements have been unrealistic from either a cost or technological perspective, but that historically there had not been an effective process to prioritize requirements or enforce capability tradeoffs. Table 5 provides additional examples from our case studies of factors cited by program offices as contributing to cost growth or schedule delays.

We have previously reported that similar factors have negatively affected the cost and schedule performance of MDAPs. DOD’s weapons system programs often enter the acquisition process without a full understanding of requirements, and we have reported numerous times that requirements changes or changes to designs to meet requirements are factors in poor cost and schedule outcomes. Additionally, in part due to high levels of uncertainty about requirements, program cost estimates and their related funding needs are often flawed. For example, in 2008 we assessed cost estimates for 20 MDAPs and found that the estimates were too low in most cases and that in some programs, cost estimates were off by billions of dollars.knowledge and detail to develop sound cost estimates, which effectively set programs up for cost growth and schedule delays.

Programs Most Frequently Cited the Use of Mature Technologies and Early Stakeholder Involvement as Factors That Helped to Reduce the Risk of Cost or Schedule Growth

Program officials for the ACAT II and III programs we reviewed most frequently cited the reliance on mature technology—including commercial or government off-the-shelf or other non-developmental items—and early involvement of stakeholders or users as factors that helped to reduce the risk of cost or schedule growth. Both of these factors were cited by 5 or more of the 15 ACAT II or III program offices we reviewed. In some cases, these factors were cited by programs that experienced cost growth or schedule delays, for example, because one of these factors may have helped a program partially recover from a cost or schedule breach or keep initial program costs lower or schedules shorter than otherwise would be expected.

Reliance on existing mature technologies was a relevant factor for the two programs we reviewed that did not report cost growth or schedule delays, and the most frequently cited factor contributing to reduced risk of cost or schedule growth among all of the programs we reviewed. The two programs we reviewed that appeared to be on track to meet original cost and schedule targets—the Army’s 5.56 millimeter Enhanced Performance Round program and SOCOM’s Nonstandard Aviation program—relied on modified commercial off-the-shelf equipment or modified existing military service equipment or assets. The Army’s 5.56 millimeter Enhanced Performance Round was an incremental engineering change to replace the Army’s general purpose 5.56 millimeter bullet with a new bullet design, which features a copper slug and exposed hardened steel penetrator. SOCOM’s Nonstandard Aviation program acquires, modifies, fields, and sustains commercial aircraft to transport special operations forces. The use of mature technologies was also cited as contributing to reduced risk of cost or schedule growth by 6 of the 13 other programs we reviewed. For example, according to program documentation for the Air Force’s F-15E Radar Modernization Program, the program planned to leverage existing commercial and government off-the-shelf technology from other fighter aircraft radar systems and the maturity of these technologies significantly lowered program development risk and costs.

Early stakeholder or user involvement was cited by 5 of the 15 programs we reviewed as contributing to reduced risk of cost or schedule growth, including 1 of the 2 programs that did not experience cost growth or schedule delays. For example, officials with the Army’s 5.56 millimeter Enhanced Performance Round program noted that constant communication with all stakeholders, engineers, testers, and contractors was essential and a key success factor for the program. Similarly, program officials for CBDP’s Dismounted Reconnaissance Sets, Kits, and Outfits program—which provides protective equipment for chemical, biological, radiological, or nuclear hazards—told us that the participation of all of the military services at the beginning of the program helped to the keep program cost and schedule on track. According to program officials, they integrated user input from the outset, including in developing the concept of operations, which reduced the number of later requirements changes. At the time of our review, the program was on track to meet its original schedule targets. The program’s unit cost also decreased between the start of development and production.

We have previously reported that similar factors appear to positively affect the cost and schedule performance of MDAPs. For example, in 2010, we reported on MDAPs that appeared to be stable and on track to meet their original cost and schedule targets. We found that the stable programs we reviewed leveraged mature technologies that had been demonstrated to work in relevant or realistic environments, and either did not consider immature technologies or deferred immature technologies to later program increments. We also reported in 2012 that early stakeholder involvement in pre-system development reviews helped facilitate trade-offs among cost, schedule, and technical performance requirements. For example, by involving both the requirements and acquisition communities in these reviews, the Army was able to identify trade-offs that reduced the projected unit costs for the Joint Light Tactical Vehicle without impairing operational needs.

At Least Five Current Programs Are Near or Over the ACAT I Cost Threshold, but Data Limitations Hinder DOD’s Ability to Identify Additional Programs

Data provided by DOD components indicated that at least five current ACAT II programs were approaching or had exceeded ACAT I cost thresholds as of November 2013, though DOD component officials told us that most were not expected to become MDAPs.identify with certainty the number of programs likely to become MDAPs because of data reliability issues related to identifying the population of ACAT II and III programs and their estimated cost. Using the 836 programs initially reported by DOD components as our starting point, we identified two current ACAT II programs that exceeded the ACAT I threshold for RDT&E—$480 million in fiscal year 2014 constant dollars— and three current ACAT II programs that were within 10 percent of the ACAT I RDT&E or procurement threshold—$2.79 billion in fiscal year 2014 constant dollars—as of November 2013. Of these five programs, DOD component officials told us that four would not become MDAPs because, for example, they did not expect further program cost growth or were considering restructuring the program, and that component-level discussions were underway with regard to the status of the remaining program (see table 6).

Conclusions

DOD weapon system acquisition represents one of the largest areas of the government’s discretionary spending, but much of this spending is still not well understood. DOD’s primary focus has been on overseeing and assessing the performance of its large ACAT I major defense acquisition programs, but the annual funding spent on ACAT II and III acquisition programs may be just as significant. Yet data provided by DOD components were so unreliable that we were unable to accurately identify even a minimum number or total cost of DOD’s ACAT II and III programs. While tailoring documentation and reporting requirements for “smaller” programs can be a reasonable approach to help prioritize limited oversight resources, if DOD and its components are to effectively manage their investment dollars, they must be able to account for how they are spending their money and how well they are spending it on the full range of acquisition programs. Having timely and reliable data on smaller acquisition programs is also critical for providing effective oversight and bringing the right oversight resources to bear, when needed, to make sure troubled smaller programs do not grow into major ones.

The Under Secretary of Defense for Acquisition, Technology, and Logistics has recognized the value of having good data on DOD’s acquisition programs—including its ACAT II and III programs—to assess the performance of the defense acquisition system and identify the factors that affect program performance. But work remains to make sure information on the complete range of DOD acquisition programs is consistently available. DOD components have taken and continue to take steps to improve the reliability of ACAT II and III program data, but they do not fully address the limitations we identified—missing data, widespread data entry issues and inconsistent reporting—or the causes of these issues, including: the lack of a common definition of a current acquisition program; insufficient data reliability testing; and inconsistent compliance with requirements for acquisition program baselines and reporting on ACAT II and III programs that may become major programs due to cost growth. Components also lacked plans to ensure their intended actions are implemented and improvements to data collection and analysis are sustained over the long term. Until these limitations are addressed, DOD components will be unable to generate reliable information to effectively manage and oversee their ACAT II and III programs.

Recommendations for Executive Action

We are making four recommendations to improve DOD’s ability to collect and maintain reliable data on its acquisitions. Specifically, we recommend that the Secretary of Defense direct the Under Secretary of Defense for Acquisition, Technology, and Logistics, in consultation with DOD components, to take the following actions: establish guidelines on what constitutes a “current” ACAT II or III program for reporting purposes; the types of programs, if any, that do not require ACAT designations; and whether the rules for identifying current MDAPs would be appropriate for ACAT II and III programs; and determine what metrics should be used and what data should be collected on ACAT II and III programs to measure cost and schedule performance; and whether the use of DAMIR and the MDAP selected acquisition report format may be appropriate for collecting data on ACAT II and III programs.

We also recommend that the Secretary of Defense direct the Secretaries of the Air Force, Army, and Navy and the Commander of SOCOM to take the following actions: assess the reliability of data collected on ACAT II and III programs and work with PEOs to develop a strategy to improve procedures for the entry and maintenance of data; and develop implementation plans to coordinate and execute component initiatives to improve data on ACAT II and III programs.

We are also making two recommendations to help ensure compliance with relevant provisions of DOD acquisition policy with the purpose of improving DOD’s ability to provide oversight for ACAT II and III programs, including those programs that may become MDAPs.

We recommend that the Secretary of Defense direct the Secretary of the Air Force and Commander of SOCOM to establish a mechanism to ensure compliance with APB requirements in DOD policy.

We recommend that the Secretary of Defense direct the Secretaries of the Air Force, Army, and Navy to improve component procedures for notifying the Defense Acquisition Executive of programs with a cost estimate within 10 percent of ACAT I cost thresholds.

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for review and comment. In its written comments, which are reprinted in full in appendix V, DOD partially concurred with all six of our recommendations. However, as discussed below, it is unclear whether the actions that DOD plans to take will fully address the issues we raised in this report.

DOD partially concurred with our first recommendation to establish guidelines on what constitutes a “current” ACAT II or III program for reporting purposes; the types of programs, if any, that do not require ACAT designations; and whether the rules for identifying current MDAPs would be appropriate for ACAT II and III programs. DOD also partially concurred with our second recommendation related to determining what metrics should be used and what data should be collected on ACAT II and III programs to measure cost and schedule performance. In its response, DOD stated that the Under Secretary of Defense for Acquisition, Technology, and Logistics would review the existing policy direction for ACAT II and III programs to determine whether it needs to be altered or supplemented to facilitate data collection or reporting. However, as our review found, the question is not whether policy needs to be revised, but how it needs to be revised. We found that the existing policy direction was not adequate to ensure consistent data collection and reporting on ACAT II and III programs or their cost and schedule performance and our recommendations were designed to address those issues. We continue to believe that additional guidelines for components regarding which programs should be considered current ACAT II and III programs for reporting purposes and consistent metrics to measure performance trends, among other actions, are needed to correct the issues we found.

DOD partially concurred with our third and fourth recommendations to assess the reliability of data collected on ACAT II and III programs and work with PEOs to develop a strategy to improve procedures for the entry and maintenance of data; and develop implementation plans to coordinate and execute component initiatives to improve data on ACAT II and III programs, respectively. In its response, DOD stated the Under Secretary of Defense for Acquisition, Technology, and Logistics will direct the DOD components to evaluate the data collected on ACAT II and III programs and report back to him on their assessment of the reliability of that data and the status of the plans to improve the availability and quality of the data. DOD’s response represents a good first step towards assessing the reliability of its ACAT II and III program data, but the response does not fully address our recommendations. DOD’s response does not address whether components would be required to develop strategies with PEOs to improve the entry and maintenance of data, as we recommended. We continue to believe that developing these strategies with those responsible for entering and maintaining program data on a day-to-day basis, including PEOs, is important to make sure the causes of DOD’s data quality problems are fully understood and addressed in a manner that can be implemented. Further, DOD’s response does not directly address our recommendation to develop implementation plans for component efforts. We believe that fully implementing this recommendation is essential for ensuring that DOD and its components can effectively monitor and evaluate the implementation of component initiatives to improve ACAT II and III data.

DOD partially concurred with our fifth recommendation to direct the Secretary of the Air Force and Commander of SOCOM to establish a mechanism to ensure compliance with APB requirements in DOD policy. DOD also partially concurred with our sixth recommendation to direct the Secretaries of the Air Force, Army, and Navy to improve component procedures for notifying the Defense Acquisition Executive of programs with a cost estimate within 10 percent of ACAT I cost thresholds. In its response, DOD stated that the Under Secretary of Defense for Acquisition, Technology, and Logistics will issue guidance to DOD components reiterating the APB requirements for ACAT II and III programs and directing that the Defense Acquisition Executive be notified when an increase or estimated increase in program cost is within 10 percent of the ACAT I cost thresholds. Reiterating existing departmental policy on these issues may help raise awareness at the component level, but without additional enforcement mechanisms it may not address the causes of the deficiencies we discuss in this report. For example, the Air Force has issued component-level guidance directing the development of APBs. However, we found that programs were not in compliance with the guidance, which demonstrates the need to improve enforcement mechanisms, such as ensuring milestone decision authorities do not approve programs to proceed through acquisition milestones without APBs. Similarly, with regard to our recommendation on notification requirements for programs approaching the ACAT I threshold, we found that component officials cited reasons other than a lack of awareness of the policy for not notifying the Defense Acquisition Executive of these programs’ cost growth. As a result, we continue to believe that DOD should fully implement our recommendation by directing components to improve their notification procedures.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense; the Under Secretary of Defense for Acquisition, Technology, and Logistics; the Secretaries of the Army, Navy, and Air Force; the Commander of U.S. Special Operations Command; the Assistant Secretary of Defense for Nuclear, Chemical, and Biological Defense Programs; and other interested parties. This report will also be available at no charge on GAO’s website at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or by e-mail at sullivanm@gao.gov.

Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to this report are listed in appendix VI.

Appendix I: Objectives, Scope, and Methodology

Our objectives were to assess (1) the extent to which information is available on the number of the Department of Defense’s (DOD) current acquisition category (ACAT) II and III programs, their total estimated acquisition cost, and cost and schedule performance; (2) the factors affecting the cost and schedule performance of selected ACAT II and III programs; and (3) the number of current ACAT II and III programs that are likely to become major defense acquisition programs (MDAP).

To address our first objective, we used a data collection instrument (DCI) to identify and collect data on the number and cost of current ACAT II and III programs from five DOD components that accounted for approximately 88 percent of DOD’s requested research, development, test, and evaluation (RDT&E) and procurement funding in the President’s Fiscal Year 2014 budget request: Army, Air Force, Navy, U.S. Special Operations Command (SOCOM), and DOD’s Chemical and Biological Defense Program (CBDP). We used a DCI to obtain ACAT II and III program data based on preliminary discussions with DOD and component officials that a DCI would be the best way to collect the information of interest. We requested that each component identify all of its current ACAT II and III programs and provide cost data and descriptive information for each program. For the purposes of this report, we defined a current program as one that has been formally initiated in the acquisition process but has not yet delivered 90 percent of its planned units or expended 90 percent of its planned expenditures. For cost data, we requested components provide baseline and current program estimates in millions of base year dollars, to include estimates for RDT&E, procurement, acquisition operation and maintenance, and military construction, as well as the program’s total acquisition cost estimate and the base year associated with the estimate. We also collected pertinent information for each program including program name, ACAT level, type of acquisition (automated information system or non- automated information system), milestone decision authority, lead DOD component, and program executive office.

To obtain additional information on schedule performance, we collected and analyzed acquisition program baseline (APB) documents, which contain program schedule and cost parameters, for a non-generalizable random sample of 170 non-automated information system ACAT II and III programs. To select the programs, we used the initial data provided to us by DOD components that included 836 reported ACAT II or III programs as a starting point. We adjusted our selection as appropriate to account for known errors in the data at the time of selection in May 2014, such as programs that were known to not be current ACAT II or III programs. Our intention was to select a sample that would be generalizable to the population of current ACAT II and III programs. However, after selecting our sample we determined through our data reliability assessment that the population of current ACAT II and III programs could not be reliably determined and that our sample would therefore be non-generalizable.As such, results of this analysis cannot be used to make inferences about all current ACAT II and III programs. When APB documents were available for programs in our sample, we reviewed them to determine whether they contained comparable program start and initial operational capability milestones to allow us to measure program schedule performance. We also used the APBs collected from this sample of programs as part of our reliability assessment of ACAT II and III cost data provided by DOD components.

Our observations on DOD’s ACAT II and III program data are based on the original data submitted by the components. We did not assess the reliability of any underlying data systems that may have been used to generate this information. We analyzed the original data provided by the components because it reflects the information DOD would have had on ACAT II and III programs at the time we collected data. From January through July 2014, we worked with components to attempt to correct problems we identified in the data. However, we continued to identify additional errors. As a result, we determined that the data provided by DOD components in response to our DCI were not sufficiently reliable to identify the number of current ACAT II and III programs, their estimated acquisition cost, or the cost performance of DOD’s ACAT II and III programs. Appendix III contains a more detailed discussion of our data reliability assessment. We also determined that we could not assess schedule performance for ACAT II and III programs because more than half of programs we reviewed in our sample of 170 programs were missing source documents or lacked comparable schedule milestones.

To address our second objective, we selected a non-generalizable sample of 15 programs from the data provided by DOD in response to our DCI. We selected 3 programs from each component included in our review. For each component, these programs were selected to include the largest current non-automated information system ACAT II and III program based on total acquisition cost as of the President’s Fiscal Year 2014 budget submission and one additional program based on factors such as significant cost growth or whether the program was part of a family of systems, which we defined as a related group of programs consisting of multiple increments or fielding similar capabilities for multiple To select the programs, we used the initial data provided to us platforms.by DOD components that included 836 reported ACAT II or III programs as a starting point. Programs that lacked data for current acquisition cost, commodity type, or ACAT level were excluded from selection. We also adjusted our selection as appropriate to account for known errors in the data at the time of selection, such as incorrectly-reported cost estimates, or programs that were known to not be current ACAT II or III programs. However, after our selection we identified additional concerns with the data reported by DOD that would likely have changed the results of our selection of the largest ACAT II or III programs at certain components. We did not make any subsequent adjustments to our original selection because we determined that the data provided by DOD was not sufficiently reliable to enable us to determine the largest ACAT II or III program at each component.

For each program, we analyzed key program documents, such as APBs, program status reports, acquisition strategies, acquisition decision memoranda, and requirements documentation, to assess cost and schedule performance and identify factors affecting that performance. We also conducted semi-structured interviews with program officials to discuss the information identified through reviews of program documentation and obtain additional insights into factors that affected program cost or schedule performance. Additionally, we analyzed prior GAO reports to determine the extent to which the factors we identified as affecting cost and schedule performance for selected ACAT II and III programs were similar to factors that we have identified in prior work as affecting performance of MDAPs.

To address our third objective, we reviewed DOD acquisition policy related to the reclassification of ACAT II or III programs to ACAT I programs and analyzed program cost data provided by DOD components. Based on the requirement in DOD acquisition policy for components to notify the Defense Acquisition Executive of ACAT II or III programs within 10 percent of the next ACAT level, we analyzed data provided by DOD through our DCI to identify programs that appeared to be within 10 percent of or have exceeded either the ACAT I RDT&E or procurement threshold. We were unable to identify an actual number of programs likely to become MDAPs because of reliability issues related to identifying the population of ACAT II and III programs. However, we determined the initial data provided to us by DOD that included 836 reported ACAT II or III programs were sufficiently reliable to serve as a starting point to identify the minimum number of programs likely to become MDAPs because we were able to confirm data with relevant program offices for those programs that appear to be within 10 percent of or have exceeded either ACAT I threshold. We excluded certain programs from further review that were known at the time that we initially identified programs to have incorrectly-reported cost estimates.

For programs that appeared to meet our criteria for current ACAT II or III programs likely to become MDAPs, we collected additional information using a structured set of questions to determine whether the relevant DOD component had notified the Defense Acquisition Executive that the program was approaching or had exceeded the ACAT I threshold and whether the program had been or was expected to be reclassified as an ACAT I program. We also requested and reviewed supporting documentation when available, including documentation of notification to the Defense Acquisition Executive that the program was within 10 percent of the ACAT I threshold. After we received the information from the components, we identified additional programs that had incorrectly reported cost estimates or were no longer current ACAT II or III programs and we removed these programs from our analysis as appropriate.

We conducted this performance audit from October 2013 to March 2015 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected Acquisition Program Milestone Documentation Requirements and Congressional Reporting Requirements by Acquisition Category

Summarizes program cost, schedule, and performance parameters Program cost estimate completed outside of the supervision of the entity responsible for the acquisition program Documents capability requirements to which the program responds Describes program’s overall technical approach and details timing and criteria for technical reviews Technology Readiness Assessment Assessment of the maturity of critical Primary planning and management tool for integrated test program 10 U.S.C. § 2366a and b.

Provides notification of unit cost breaches above a certain threshold Provides notification of cost or schedule changes above a certain threshold 10 U.S.C § 2433 and 2433a. Notification must be provided to Congress when the program acquisition unit cost or average procurement unit cost increases by at least 15 percent over the current baseline estimate or 30 percent over the original baseline estimate. 10 U.S.C § 2445c. Notification must be provided to Congress when there is a schedule change that will cause a delay of more than 6 months; an increase in the expected development cost or full life- cycle cost for the program by at least 15 percent; or a significant, adverse change in the expected performance of the major automated information system to be acquired.

Appendix III: Observations on Reliability of Department of Defense Components’ Acquisition Category II and III Program Data

We conducted an analysis to determine whether data provided by Department of Defense (DOD) components were sufficiently reliable for the purpose of determining the number, total acquisition cost, and cost performance of DOD’s current acquisition category (ACAT) II and III programs. For our analysis, we conducted electronic and manual testing on data for all programs reported by components in response to our request for completion of a data collection instrument (DCI) and compared cost data for a non-generalizable sample of programs to source documents when available. We also reviewed relevant DOD and component acquisition policy, and interviewed knowledgeable officials. We identified reliability issues with the data for about 60 percent of the programs components initially reported to us. As a result, we determined that the data provided by DOD components were not sufficiently reliable to identify the number of current ACAT II and III programs, their estimated total acquisition cost, or the cost performance of DOD’s ACAT II and III programs.

Methodology

To assess the accuracy and completeness of the ACAT II and III program data reported by DOD components, we electronically tested the data for: values outside the designated range of values for ACAT II and III programs, defined per DOD acquisition policy; obvious calculation or data entry errors (for example, individual cost elements do not sum to total reported); missing data in baseline or current cost estimate data elements, including estimates for research, development, test, and evaluation; procurement; military construction; and acquisition operation and maintenance, as well as the total acquisition cost estimate, and base year; and missing data in program descriptive data elements, such as ACAT level, milestone decision authority, or commodity type.

Additionally, we compared cost data for our sample of 170 programs to source documents when available. Specifically, for each program in our sample, we first requested and reviewed original and current acquisition program baselines (APB) to determine whether or not they reflected the actual baseline from program start and the current APB based on the approval date of the APB and relevant schedule milestones that trigger the development of an APB or an APB revision in accordance with DOD acquisition policy. When APBs were not provided or did not appear to reflect the actual baseline and/or current APB, we followed up with DOD components to obtain the correct documents when possible. When we were able to obtain both original and current APBs, we took the following steps to assess the accuracy of the information reported in the DCI:

Compared baseline cost data from the program’s original APB to the baseline cost data reported in the DCI.

Compared cost data in the current APB to cost data reported in the DCI to identify obvious errors in the cost data reported in the DCI, such as current cost data in the DCI that was significantly less than the amount reported in the APB without explanation.

To assess the consistency of ACAT II and III program data, we manually reviewed the data provided in response to the DCI and subsequent requests to identify programs that did not appear to meet our criteria for current ACAT II and III programs. For the purposes of this report, we defined a current program as one that has been formally initiated in the acquisition process but has not yet delivered 90 percent of its planned units or expended 90 percent of its planned expenditures. For each program that did not appear to be a current ACAT II or III program, we analyzed whether the program was pre-program start, in sustainment, completed, or not a separate ACAT II or III program (for example, was a subprogram of another ACAT II or III program reported to us).

Results of Analysis

The results of our data reliability analysis capture accuracy, completeness, and consistency issues with the data provided by DOD components. Accuracy, completeness, and consistency are key characteristics of reliable data and refer to (1) the extent that recorded data reflect the actual underlying information; (2) data elements for each program are populated appropriately; and (3) the need to obtain and use data that are clear and well defined enough to yield similar results in similar analyses, respectively.

Accuracy and Completeness

We identified numerous types of accuracy and completeness issues with the data provided by DOD components, including cost estimate values outside of the ACAT II and III program range, basic math errors, and missing data. For example, 333 out of 836 ACAT II and III programs reported by the components were missing a baseline or current cost estimate element or descriptive program data. Table 9 provides detail on accuracy and completeness issues by component.

We identified additional issues with the accuracy of the ACAT II and III program cost information when we compared reported cost estimates to available source documents for a non-generalizable sample of ACAT II and III programs. Specifically, of the 81 programs in our sample that reported complete cost estimates and provided source documents, 50 reported incorrect cost data. For example, for 37 of these 50 programs, we determined that baseline cost data was inaccurate because either the baseline cost estimate or the base year reported for this estimate did not match the source documents. Details of the accuracy and completeness issues we identified when assessing the cost data reported for our sample are provided by component in table 10.

We identified 226 of the 836 programs reported by DOD components that did not meet our criteria for current ACAT II or III programs because, for example, they were not current or they were not stand-alone acquisition programs. Additionally, because information on program phase was not available for all programs reported by DOD components, the number of programs we identified as not a current ACAT II or III program reflects a minimum number of such programs. Table 11 provides details on consistency issues we identified by component.

Appendix IV: Description of Selected Department of Defense (DOD) Acquisition Category (ACAT) II and III Programs

Appendix V: Comments from the Department of Defense

Appendix VI: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Ron Schwenn, Assistant Director; Leslie Ashton, Jenny Chanley, Teakoe Coleman, Dani Greene, John Krump, Jesse Lamarre-Vincent, Anne McDonough-Hughes, Carol Petersen, and Oziel Trevino made key contributions to this report.