Defense Acquisitions: Better Approach Needed to Account for Number, Cost, and Performance of Non-Major Programs
Published: 2015-03-02T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: DOD requested $168 billion in fiscal year 2014 to develop, test, and acquire weapon systems and other products and equipment. About 40 percent of that total is for major defense acquisition programs or ACAT I programs. DOD also invests in other, non-major ACAT II and III programs that are generally less costly at the individual program level. These programs typically have fewer reporting requirements and are overseen at lower organizational levels than ACAT I programs, although they may have annual funding needs that are just as significant.
GAO was asked to examine ACAT II and III programs. This report addresses, among other issues, (1) the extent to which information is available on the number, cost, and performance of ACAT II and III programs and (2) factors that affected the performance of selected ACAT II and III programs. GAO collected program and cost data on current ACAT II and III programs from five DOD components. GAO also selected a non-generalizable sample of 15 programs based on program cost and other criteria and reviewed documentation and interviewed officials about program performance.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) could not provide sufficiently reliable data for GAO to determine the number, total cost, or performance of DOD's current acquisition category (ACAT) II and III programs. These non-major programs range from a multibillion dollar aircraft radar modernization program to soldier clothing and protective equipment programs in the tens of millions of dollars. GAO found that the accuracy, completeness, and consistency of DOD's data on these programs were undermined by widespread data entry issues, missing data, and inconsistent identification of current ACAT II and III programs. See the figure below for selected data reliability issues GAO identified.
DOD components are taking steps to improve ACAT II and III data, but these steps do not fully address the problems GAO identified. For example, the components have not established systematic processes to perform data quality tests and assess the results to help identify problems for further review. These types of tests and assessments can be an important step in determining whether data can be used for its intended purposes. Additionally, DOD lacks metrics to assess ACAT II and III cost and schedule performance trends across programs and in some cases was missing baseline cost and schedule data to measure performance. Having timely and reliable cost, schedule, and performance data on smaller acquisition programs is critical to ensuring that DOD and its components can account for how they are spending their money and how well they are spending it. Reliable data are also essential for effective oversight and bringing the right oversight resources to bear when programs approach the cost threshold to become a major defense acquisition program due to cost growth.
Thirteen of the 15 ACAT II or III programs GAO reviewed in-depth had exceeded their original cost or schedule targets. Program officials from ACAT II and III programs GAO reviewed cited changing performance requirements, testing issues, quantity changes, and flaws in original cost estimates, among other factors, as the reasons for cost and schedule growth. GAO has previously found that similar factors affect the performance of major acquisition programs.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD establish guidelines on what constitutes a current ACAT II and III program, take steps to improve data reliability, and determine how to measure cost and schedule performance. DOD partially concurred with the recommendations and described actions it plans to take. However, as discussed in the report, DOD's planned actions may not fully address the issues that GAO identified.

section_title: Letter
paragraphs: Department of Defense (DOD) weapon system acquisition represents one  of the largest areas of the federal government’s discretionary spending. In  fiscal year 2014, DOD requested $168 billion to develop, test, and acquire  weapon systems and other products and equipment. About 40 percent of  that total was for major defense acquisition programs (MDAP) or  acquisition category (ACAT) I programs. The remaining approximately  60 percent of the budget request included, among other investments,  funding for DOD’s non-major ACAT II and III programs. These programs,  which include everything from a multibillion dollar aircraft radar  modernization program to soldier clothing and protective equipment  programs in the tens of millions of dollars, are generally less costly than  MDAPs at the individual program level. Due to the lower level of  investment involved on a program-by-program basis, ACAT II and III  programs typically have fewer reporting and documentation requirements  and are overseen at lower organizational levels than MDAP and major  automated information system (MAIS) programs. Accordingly,  Congress’s and DOD’s insight into the performance of these programs is  more limited.
You asked us to examine the number, acquisition cost, and performance  of DOD’s ACAT II and III programs. This report assesses (1) the extent to  which information is available on the number of current ACAT II and III  programs, their total estimated acquisition cost, and their cost and  schedule performance; (2) the factors affecting the cost and schedule  performance of selected ACAT II and III programs; and (3) the number of  DOD’s current ACAT II and III programs that are likely to become  MDAPs.
To determine the extent to which information was available on the  number of current ACAT II and III programs and their estimated  acquisition costs, we used a data collection instrument to collect data on  the number and cost of current ACAT II and III programs from five DOD  components: Army, Air Force, Navy, U.S. Special Operations Command  (SOCOM), and the DOD Chemical and Biological Defense Program  (CBDP). Our observations on DOD’s ACAT II and III program data are  based on the original data submitted by the components. To assess  information available on cost and schedule performance, we collected  and analyzed acquisition program baseline (APB) documents for a non- generalizable random sample of 170 non-automated information system  ACAT II and III programs. To assess the reliability of the data, we  reviewed the data for missing values and obvious errors, and compared  the cost data for our sample of 170 programs to source documents when  available. We found the data were unreliable as further discussed in the  report.
To identify the factors that affected the cost and schedule performance of  selected ACAT II and III programs, we analyzed factors cited in program  documentation and by DOD program officials. We reviewed a non- generalizable sample of 15 programs including each component’s largest  ACAT II and III program based on data reported by DOD components and  one additional program per component based on factors such as cost  growth or being part of a family of related systems. For each program,  we analyzed APBs, reviewed program documents, and interviewed  program officials to assess and identify factors that affected cost or  schedule performance.
To determine the number of current ACAT II and III programs that were  likely to become MDAPs, we analyzed data provided by DOD  components through our data collection instrument to identify programs  that appeared to be within 10 percent of or to have exceeded the ACAT I  threshold for research, development, test, and evaluation (RDT&E) or  procurement. We then collected information about these programs from  DOD components using a set of structured questions. We determined the  data were sufficiently reliable to serve as a starting point to identify the  minimum number of programs likely to become MDAPs because we were  able to confirm data with relevant program offices. Appendix I provides  additional details on our scope and methodology.
We conducted this performance audit from October 2013 to March 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOD acquisition policy defines an acquisition program as a directed,  funded effort that provides a new, improved, or continuing materiel,  weapon, or information system, or a service capability in response to an   As shown in table 1, defense acquisition programs are  approved need.classified into acquisition categories that depend on the value and type of  acquisition. The Army, Navy, Air Force, and SOCOM also have  supplemental acquisition policies that address certain aspects of  acquisition program categorization and management.
ACAT II and III programs encompass a wide range of efforts and program  sizes. Programs may range from an ACAT II program with a total  acquisition cost of more than $3 billion to an ACAT III program with an  acquisition cost in the millions of dollars or lower. DOD’s acquisition policy  does not establish a minimum cost for ACAT III programs.
The level of oversight for acquisition programs varies based on the  assigned ACAT level. DOD and component acquisition policies specify  the organizational level of the milestone decision authority—the  designated individual with overall responsibility for a program—for each  ACAT level. The organizational level at which program requirements and  requirements changes are approved may vary by ACAT level as well. The  organizational level of the milestone decision authority for Air Force ACAT  I-III programs is shown in figure 1 as an example.
All acquisition programs are required by statute or DOD guidance to  provide program information at milestones and other decision points,  although these requirements differ by ACAT level. MDAP and MAIS  programs, also known as ACAT I and IA programs, require more  documentation and analysis to support program decisions and have to  regularly report to Congress on their cost, schedule, and technical  performance. These programs are required to enter and maintain  program cost, schedule, and performance data and create APBs within  DOD’s Defense Acquisition Management Information Retrieval (DAMIR)  system, a web-based data system intended to provide data transparency  of acquisition management information across DOD. Components may  use DAMIR for other programs, but it is not required. Appendix II provides  additional detail on acquisition documentation requirements and  congressional reporting requirements by ACAT level.
section_title: DOD Does Not Have Reliable Data on ACAT II and III Programs and Component Actions to Improve Data Do Not Fully Address Limitations
paragraphs: DOD components could not provide sufficiently reliable data for us to  accurately determine the number, total cost, or performance of DOD’s  current ACAT II and III programs. We found that the accuracy,  completeness, and consistency of DOD’s data was undermined by (1)  widespread data entry issues and missing data, and (2) inconsistent  identification of current ACAT II and III programs across and within  components. DOD components have taken some steps to improve ACAT  II and III program data, but their efforts do not fully address the causes of  the problems we identified. In addition to data reliability problems, DOD  lacks consistent cost and schedule metrics across components to assess  ACAT II and III program performance. Further, the lack of baseline cost  and schedule data and comparable schedule milestones prevents DOD  from consistently measuring the performance of ACAT II and III  programs. Taken together, these issues limit the utility of DOD’s data on  ACAT II and III programs for oversight, decision-making, and reporting  purposes.
subsections: section_title: DOD Components Cannot Provide Accurate, Complete, or Consistent Data on Current ACAT II and III Programs
paragraphs: We identified data reliability issues related to accuracy, completeness, or  consistency with data for about 60 percent of ACAT II and III programs  reported to us by DOD components. These issues prevented us from  accurately determining the number, total cost, or performance of DOD’s  current ACAT II and III programs. According to DOD acquisition policy,  complete and current program information is essential to the acquisition  process. Internal control standards for federal executive branch  agencies also emphasize that agencies should have relevant, reliable,  and timely information for decision-making and external reporting  purposes. We found obvious accuracy and completeness issues in  program cost data for ACAT II and III programs reported to us by DOD  components. We also observed consistency issues in program data  across components and within some components that affected the  comparability of the data.
subsections: section_title: Widespread Data Entry Issues and Missing Data
paragraphs: Inaccurate data were evident across all of the components; issues we  observed included reported dollar values outside the range of ACAT II  and III programs and basic math errors. Inaccuracies like these suggest  overall data quality problems. Further, when we reviewed a sample of  programs and compared reported cost estimates to source documents,  we found that cost estimate data was often misreported. Components  incorrectly reported data or data was missing for 64 out of 95 programs  for which we had complete source documents in our non-generalizable  sample. We also observed missing data elements to varying degrees at all of the components except CBDP. For example, 333 out of 836  programs reported by the components were missing one or more cost  estimate elements or basic information such as the ACAT level. Lastly, in  numerous instances components did not follow the instructions of the  data collection instrument, which also affected our ability to use the data.  See table 2 for examples of the reliability issues we identified.
In some instances, the accuracy and completeness issues we identified  were consistent with known limitations of information in component  systems. For example, Army acquisition officials told us that in the past  there has been conflicting guidance about whether completed programs  should be deleted from the Army’s acquisition information system, and  some completed programs were never removed from the system.  Officials at all components, except CBDP, further told us that accuracy of  data in their systems relies primarily on the quality of information  submitted by the Program Executive Officer (PEO) or program offices.Army, Navy, and Air Force acquisition officials told us that they work with  PEOs to address data quality problems, such as by conducting ad hoc  checks to flag obvious errors or missing data elements and following up  with PEOs as necessary. SOCOM acquisition officials told us the  Acquisition Executive emphasizes the importance of maintaining up-to- date data to program managers. Based on the data provided to us by  DOD components, these existing data quality practices are insufficient to  ensure the accuracy and completeness of data on DOD’s ACAT II and III  programs as required by DOD policy and federal internal control  standards.
section_title: Inconsistent Identification of Current ACAT II and III Programs Across and Within Components
paragraphs: Data provided in response to our request for information on current ACAT  II and III programs (1) included acquisitions that were not “current” ACAT  II and III programs in accordance with our definition, (2) likely reflected  inconsistent reporting of acquisitions at lower dollar levels across  components, and (3) likely excluded certain acquisitions considered  current ACAT II and III programs in accordance with certain component  acquisition policies. For the purposes of our report, we define a current  program as one that has been formally initiated in the acquisition process,  but has not yet delivered 90 percent of its items or made 90 percent of  planned expenditures. This definition is consistent with statutory reporting  thresholds used by Congress in its reporting requirements for current  MDAP programs.
Inconsistent interpretations of what constitutes a current program and the  inability of some components to reliably identify these programs  contributed to the inclusion of programs that were not current in the data  that components reported to us. DOD acquisition policy and component  guidance generally do not define which ACAT II or III programs are  considered to be current for management and reporting purposes. For  example, DOD acquisition policy defines when a program is formally  initiated and its operations and support phase begins, but it does not  identify when a program should be considered current. Further, some  components told us PEOs may have different interpretations of what  constitutes a current program. Components also told us that they could  not consistently use the information in their data systems to readily  identify current programs. Of the 836 programs initially reported by the  five components, we identified 199 programs across the Army, Navy, Air  Force, and SOCOM that should not have been included because they did  not meet our criteria for current programs. For example, according to an  Army PEO, 90 of 140 programs originally reported to us as current ACAT  II or III programs by the Army were not current based on our definition  because these programs had delivered more than 90 percent of planned  items or expended more than 90 percent of planned funds. Without the  consistent identification of current ACAT II and III programs, DOD and  component officials do not know the accurate number of these programs  and may miss opportunities to identify programs that may need more or  less oversight depending on whether or not most of the anticipated  acquisition funding has been spent.
Additionally, component guidance for defining acquisition programs varies  and likely resulted in inconsistent reporting across components of  acquisitions at lower dollar values. DOD acquisition policy establishes a  cost ceiling and cost floor for ACAT II programs and a cost ceiling, but no  cost floor, for ACAT III programs. However, some components have  supplemental guidance that establishes additional acquisition categories  or exclusions. For example, SOCOM and Navy have guidance that  provides for certain acquisitions with less than $10 million in total RDT&E  contracts, less than $25 million per year in annual procurement funding,  and less than $50 million total in procurement contracts to be categorized  as non-ACAT programs. Specifically, SOCOM designates these low cost,  schedule, and technical risk efforts to field special operations-peculiar  capabilities as abbreviated acquisition projects. Similarly, the Navy  designates lower dollar value programs that do not require operational  testing and evaluation as abbreviated acquisition programs.Air Force policies do not provide for lower dollar threshold categories   Army and  beneath the ACAT III level. According to officials from the Office of the  Under Secretary of Defense for Acquisition, Technology, and Logistics,  acquisition programs at these dollar levels should be reported as ACAT III  programs. However, Navy and SOCOM did not include their non-ACAT  programs in the ACAT II and III program data reported to us.
There was also variation within components as to how acquisition  programs were categorized. Army and Air Force acquisition officials told  us that some programs that should have been considered ACAT II or III  programs in accordance with component acquisition policy were not.  These officials also told us that PEOs may have counted and handled  programs differently in the absence of a clear definition of what should be  considered a program of record. For example, Army officials told us that  categorizing information technology programs was sometimes  challenging, and they have worked with PEOs to review the  categorization of certain information technology programs. Army and Air  Force officials told us that as a result of confusion among PEOs about  whether or not certain programs should be considered ACAT II or III  programs, they have needed to add and remove numerous ACAT II and  III programs from component information systems over the past year.
The types of issues we identified may have also contributed to  components reporting varying numbers of ACAT II and III programs in  response to different requests for information during the same time frame.  Specifically, concurrent with reporting 755 current ACAT II and III  programs to us, the Army, Navy, and Air Force reported 1,360 ACAT II  and III programs in a presentation to the DOD Business Senior  Integration Group, which is chaired by the Under Secretary of Defense for  Acquisition, Technology, and Logistics and oversees DOD’s Better  Buying Power initiatives. Acquisition officials from these components  told us they were unable to fully explain the reasons for the difference  between the numbers of programs reported.
section_title: Components Have Taken Steps to Improve ACAT II and III Program Data, but Have Not Fully Addressed the Causes of the Reliability Problems
paragraphs: The Army, Navy, Air Force, and SOCOM have established information  systems to track cost and schedule data for ACAT II and III programs and  taken some steps to address issues related to the completeness and  accuracy of information tracked in these systems. CBDP officials told us  they recognize the value of establishing a system to track data on ACAT  II and III programs and are determining the capabilities that would be  needed in such a system. Specifics of component efforts follow:
According to Navy officials, data in the Navy’s Research,  Development & Acquisition Information System has potentially been  incomplete because ACAT II and III program data has not been  consistently entered into the system. The Navy issued an updated  policy in August 2014 that requires input of programmatic information  into the system for all ACAT programs.
All Air Force ACAT II and III programs have been required to enter  cost and schedule data into the System Metric and Reporting Tool  since 2012, but Air Force officials told us that not all programs had  complied with the requirement. They told us in June 2014 that they  had an ongoing effort to review ACAT II and III programs in the  system, including assessing whether cost and schedule data has  been populated. Further, the Air Force has established an investment  master list that will capture all programs receiving RDT&E and  procurement funding.
SOCOM requires that all ACAT II and III programs enter program and  cost data into its centralized acquisition portal data system, however  SOCOM officials told us some program managers have been more  diligent than others in populating the system. These officials told us  that data from the portal is now used to conduct monthly program  reviews, rather than having the program prepare briefing slides, as a  way to encourage program managers to populate and regularly  update the system.
Components have also taken steps to improve the consistency with which  they identify current ACAT II and III programs. For example, officials at  four of the five components told us they are exploring ways to identify the  acquisition phase—technology maturation and risk reduction, engineering  and manufacturing development, production and deployment, operations  and support—for programs in component information systems, which  could improve their ability to reliably identify current programs. The Air  Force and Army have also made efforts to address concerns they had  previously identified related to the consistent identification of ACAT II and  III programs within their components. The Air Force issued guidance in  January 2014, detailing which programs are and are not considered to be  acquisition programs and acquisition policy officials told us they have  been meeting with individual PEOs to clarify any misunderstandings  about how acquisition programs should be categorized. Officials from the  Office of the Assistant Secretary of the Army for Acquisition, Logistics,  and Technology also told us they are in the process of revising the Army’s  acquisition guidance, with input from PEOs, to more precisely define the  types of programs that should and should not be considered to be  acquisition programs.
However, the components’ efforts do not fully address the accuracy,  completeness, and consistency issues we identified with ACAT II and III  program data. For example, the components have not established  systematic processes to perform data quality tests on PEO-submitted  data and assess the results to help identify problems, such as basic math  errors or missing data, for further review. These types of tests and  assessments can be an important step in determining whether data can  be used for its intended purposes. Additionally, the components have  not developed plans that detail how they will implement or sustain data  improvement efforts. For example, the components have not developed  implementation steps for assessing data reliability on an ongoing basis or  metrics to assess the success of data cleanup efforts. Developing such   Without establishing this  plans is a key project management practice.planning foundation, the components will not be in a sound position to  effectively monitor and evaluate the implementation of efforts to improve  component data. Finally, efforts to consistently identify current ACAT II  and III programs have been focused within individual components.  Without a consistent understanding about which programs should be  considered to be current ACAT II or III programs across components,  similar programs will continue to be reported on differently, thereby  limiting the consistency and comparability of ACAT II and III program data  across DOD.
section_title: DOD Lacks Consistent Cost and Schedule Performance Metrics and Faces Additional Challenges Measuring Trends
paragraphs: DOD components lack consistent cost and schedule performance metrics  to assess performance trends across ACAT II and III programs. As part of  the department’s Better Buying Power initiatives, the Under Secretary of  Defense for Acquisition, Technology, and Logistics instructed DOD  components to determine how best to measure performance trends for  non-ACAT I programs. Federal internal control standards also emphasize  the importance of comparing actual performance to planned or expected  results throughout the organization to help ensure effective results are  achieved and actions are taken to address risks. The Army, Navy, and Air  Force briefed DOD’s Business Senior Integration Group in November  2013 on their current efforts and plans regarding assessing ACAT II and  III cost and schedule performance, but no specific follow-on actions or  action plans have been developed.
Unlike MDAPs and MAIS programs, ACAT II and III programs are not  required to report cost and schedule data in a consistent fashion, despite  the potential benefits of such reporting. MDAP and MAIS programs are  required to report key cost and schedule metrics to Congress in a  standardized format through Selected Acquisition Reports and MAIS  Annual Reports, respectively. Cost and schedule data for these reports  are pulled from DOD’s web-based DAMIR system. MDAP cost and  schedule data are used by DOD for its annual assessment of the  performance of the defense acquisition system, which the department  uses to improve acquisition program performance and inform policy and  programmatic decisions. ACAT II and III programs are not required to  produce similar cost and schedule reporting as larger programs and do  not have to provide program data in DAMIR. According to officials from  the Office of the Under Secretary of Defense for Acquisition, Technology,  and Logistics, although minor adjustments may be needed for reporting  purposes, there is nothing that prevents components from using DAMIR  to capture data on ACAT II and III programs, and acquisition officials from  the Army have considered using it. Additionally, CBDP officials told us  they would consider DAMIR when exploring potential systems to track  ACAT II and III program data.
DOD component officials told us they are not yet sure how best to  measure cost and schedule performance across ACAT II and III  programs. For example, Army officials told us that analysis of component- wide ACAT II and III performance trends may not make sense given the  differences across programs. Navy, SOCOM, Air Force, and CBDP  officials told us they are interested in tracking cost and schedule  performance trends across ACAT II or III programs, but are still working to  define performance metrics and address limitations in existing data or  reporting capabilities. For example, the Air Force has attempted to assess  cost and schedule performance for a subset of ACAT II and III programs,  but acquisition officials noted that the process was very resource- intensive, and they had concerns about the reliability of the cost and  schedule information used in their analysis given the lack of variability in  program performance over time. While the components have developed  oversight mechanisms to review individual ACAT II and III program  performance, such as through periodic program status reviews at the  PEO level and program or portfolio reviews by senior component  acquisition officials, without assessing performance trends across ACAT  II and III programs, DOD and its components may be missing  opportunities to identify and analyze differences between actual and  expected performance and develop strategies to address related risks  throughout the department.
subsections: section_title: Data Limitations for Measuring Cost and Schedule Performance Trends
paragraphs: When we analyzed information available on cost and schedule  performance, we determined that we could not assess cost performance  for 139 programs out of a non-generalizable sample of 170 programs and  schedule performance for 105 of the 170 programs. In addition to  missing or misreported cost data, we identified two challenges to  measuring cost and schedule performance trends for ACAT II and III  programs: (1) programs without available APBs and (2) a lack of  consistent and comparable key schedule milestones across programs.  See figure 2 for a summary of our assessment of the data available to  measure cost and schedule performance and appendix III for additional  details.
For 75 of the 170 programs that we examined in detail, we could not  assess cost or schedule performance because DOD components had not  developed, or did not provide, an original APB, a current one, or both.  The components were unable to provide APBs for various reasons, such  as because they could not locate the original or an APB was not  developed at program start or to this point in the life of the program. APBs  are critical management tools that establish how systems will perform,  when they will be delivered, and what they will cost. According to DOD  acquisition policy, APBs are required of all acquisition programs, and  Office of the Under Secretary of Defense for Acquisition, Technology, and  Logistics officials told us they generally expect all acquisition programs to  have one.prior to entry into system development (Milestone B), or at program  initiation, whichever occurs later. APBs may be revised at the time of  significant program decisions, such as milestones, or as a result of major  program changes or breaches to cost, schedule, or performance  parameters. Table 3 shows the number of missing APBs by component in  our sample.
section_title: ACAT II and III Program Performance Most Frequently Affected by Requirements Changes and Testing Issues
paragraphs: Thirteen of the 15 ACAT II or III programs we reviewed in-depth had  exceeded the cost or schedule targets in their original APBs. These  programs cited changing requirements, testing issues, quantity changes,  and flaws in original cost estimates, among other factors, as the reasons  for cost and schedule growth. The programs we reviewed cited other  factors, such as a reliance on mature technology—including commercial  or government off-the-shelf or other non-developmental items—and early  involvement of stakeholders or users as contributing to reduced risk of  cost or schedule growth. We have previously reported that similar factors  affect the performance of DOD’s MDAPs. Appendix IV provides additional  details about the programs we reviewed.
subsections: section_title: Programs Most Frequently Attributed Cost and Schedule Growth to Requirements Changes
paragraphs: Thirteen of the 15 ACAT II or III programs we reviewed in-depth had  exceeded the cost or schedule targets in their original APBs.attempt to quantify the extent to which these programs had exceeded  cost or schedule targets due to overall concerns about the reliability of  ACAT II and III cost and schedule data and because not all of these  programs had developed APBs at program start. These programs most  frequently attributed cost growth or schedule delays to changing  requirements. Testing issues, quantity changes, and flaws in original cost  estimates were also cited by at least 5 of the 13 programs as contributing   We did not  to cost growth or schedule delays. All but 1 of the 13 programs cited  multiple causes for cost growth or schedule delays, including factors  beyond those listed in table 4.
Requirements changes were associated with cost growth or schedule  delays by at least one program at each of the five components in our  review. According to program officials, programs added or increased  requirements due to situations such as: adding capability to a new  platform that had not been planned for when the original requirements  were approved; creating additional variants to meet requirements that  emerged after the original requirements were approved; or making  improvements or refinements to a system in development or production  as a result of changes in the operational environment, including new  threats. For example, officials from the Army’s Synthetic Environment  Core program, which is providing the Army a common virtual environment  that links virtual simulators and simulations into an integrated and  interoperable training environment, told us that increasing terrain  database requirements to meet additional training needs have contributed  to program cost increases significant enough to require the program to be  recategorized from an ACAT III to an ACAT II program. Program officials  stated that in some cases, the additional requirements have been  unrealistic from either a cost or technological perspective, but that  historically there had not been an effective process to prioritize  requirements or enforce capability tradeoffs. Table 5 provides additional  examples from our case studies of factors cited by program offices as  contributing to cost growth or schedule delays.
We have previously reported that similar factors have negatively affected  the cost and schedule performance of MDAPs. DOD’s weapons system  programs often enter the acquisition process without a full understanding  of requirements, and we have reported numerous times that requirements  changes or changes to designs to meet requirements are factors in poor  cost and schedule outcomes. Additionally, in part due to high levels of  uncertainty about requirements, program cost estimates and their related  funding needs are often flawed. For example, in 2008 we assessed cost  estimates for 20 MDAPs and found that the estimates were too low in  most cases and that in some programs, cost estimates were off by billions  of dollars.knowledge and detail to develop sound cost estimates, which effectively  set programs up for cost growth and schedule delays.
section_title: Programs Most Frequently Cited the Use of Mature Technologies and Early Stakeholder Involvement as Factors That Helped to Reduce the Risk of Cost or Schedule Growth
paragraphs: Program officials for the ACAT II and III programs we reviewed most  frequently cited the reliance on mature technology—including commercial  or government off-the-shelf or other non-developmental items—and early  involvement of stakeholders or users as factors that helped to reduce the  risk of cost or schedule growth. Both of these factors were cited by 5 or  more of the 15 ACAT II or III program offices we reviewed. In some  cases, these factors were cited by programs that experienced cost growth  or schedule delays, for example, because one of these factors may have  helped a program partially recover from a cost or schedule breach or  keep initial program costs lower or schedules shorter than otherwise  would be expected.
Reliance on existing mature technologies was a relevant factor for the two  programs we reviewed that did not report cost growth or schedule delays,  and the most frequently cited factor contributing to reduced risk of cost or  schedule growth among all of the programs we reviewed. The two  programs we reviewed that appeared to be on track to meet original cost  and schedule targets—the Army’s 5.56 millimeter Enhanced Performance  Round program and SOCOM’s Nonstandard Aviation program—relied on  modified commercial off-the-shelf equipment or modified existing military  service equipment or assets. The Army’s 5.56 millimeter Enhanced  Performance Round was an incremental engineering change to replace  the Army’s general purpose 5.56 millimeter bullet with a new bullet  design, which features a copper slug and exposed hardened steel  penetrator. SOCOM’s Nonstandard Aviation program acquires, modifies,  fields, and sustains commercial aircraft to transport special operations  forces. The use of mature technologies was also cited as contributing to  reduced risk of cost or schedule growth by 6 of the 13 other programs we  reviewed. For example, according to program documentation for the Air  Force’s F-15E Radar Modernization Program, the program planned to  leverage existing commercial and government off-the-shelf technology  from other fighter aircraft radar systems and the maturity of these  technologies significantly lowered program development risk and costs.
Early stakeholder or user involvement was cited by 5 of the 15 programs  we reviewed as contributing to reduced risk of cost or schedule growth,  including 1 of the 2 programs that did not experience cost growth or  schedule delays. For example, officials with the Army’s 5.56 millimeter  Enhanced Performance Round program noted that constant  communication with all stakeholders, engineers, testers, and contractors  was essential and a key success factor for the program. Similarly,  program officials for CBDP’s Dismounted Reconnaissance Sets, Kits, and  Outfits program—which provides protective equipment for chemical,  biological, radiological, or nuclear hazards—told us that the participation  of all of the military services at the beginning of the program helped to the  keep program cost and schedule on track. According to program officials,  they integrated user input from the outset, including in developing the  concept of operations, which reduced the number of later requirements  changes. At the time of our review, the program was on track to meet its  original schedule targets. The program’s unit cost also decreased  between the start of development and production.
We have previously reported that similar factors appear to positively  affect the cost and schedule performance of MDAPs. For example, in  2010, we reported on MDAPs that appeared to be stable and on track to  meet their original cost and schedule targets. We found that the stable  programs we reviewed leveraged mature technologies that had been  demonstrated to work in relevant or realistic environments, and either did  not consider immature technologies or deferred immature technologies to  later program increments. We also reported in 2012 that early  stakeholder involvement in pre-system development reviews helped  facilitate trade-offs among cost, schedule, and technical performance  requirements. For example, by involving both the requirements and  acquisition communities in these reviews, the Army was able to identify  trade-offs that reduced the projected unit costs for the Joint Light Tactical  Vehicle without impairing operational needs.
section_title: At Least Five Current Programs Are Near or Over the ACAT I Cost Threshold, but Data Limitations Hinder DOD’s Ability to Identify Additional Programs
paragraphs: Data provided by DOD components indicated that at least five current  ACAT II programs were approaching or had exceeded ACAT I cost  thresholds as of November 2013, though DOD component officials told us  that most were not expected to become MDAPs.identify with certainty the number of programs likely to become MDAPs  because of data reliability issues related to identifying the population of  ACAT II and III programs and their estimated cost. Using the 836  programs initially reported by DOD components as our starting point, we  identified two current ACAT II programs that exceeded the ACAT I  threshold for RDT&E—$480 million in fiscal year 2014 constant dollars— and three current ACAT II programs that were within 10 percent of the  ACAT I RDT&E or procurement threshold—$2.79 billion in fiscal year  2014 constant dollars—as of November 2013. Of these five programs,  DOD component officials told us that four would not become MDAPs  because, for example, they did not expect further program cost growth or  were considering restructuring the program, and that component-level  discussions were underway with regard to the status of the remaining  program (see table 6).
section_title: Conclusions
paragraphs: DOD weapon system acquisition represents one of the largest areas of  the government’s discretionary spending, but much of this spending is still  not well understood. DOD’s primary focus has been on overseeing and  assessing the performance of its large ACAT I major defense acquisition  programs, but the annual funding spent on ACAT II and III acquisition  programs may be just as significant. Yet data provided by DOD  components were so unreliable that we were unable to accurately identify  even a minimum number or total cost of DOD’s ACAT II and III programs.  While tailoring documentation and reporting requirements for “smaller”  programs can be a reasonable approach to help prioritize limited  oversight resources, if DOD and its components are to effectively manage  their investment dollars, they must be able to account for how they are  spending their money and how well they are spending it on the full range  of acquisition programs. Having timely and reliable data on smaller  acquisition programs is also critical for providing effective oversight and  bringing the right oversight resources to bear, when needed, to make  sure troubled smaller programs do not grow into major ones.
The Under Secretary of Defense for Acquisition, Technology, and  Logistics has recognized the value of having good data on DOD’s  acquisition programs—including its ACAT II and III programs—to assess  the performance of the defense acquisition system and identify the factors  that affect program performance. But work remains to make sure  information on the complete range of DOD acquisition programs is  consistently available. DOD components have taken and continue to take  steps to improve the reliability of ACAT II and III program data, but they  do not fully address the limitations we identified—missing data,  widespread data entry issues and inconsistent reporting—or the causes  of these issues, including: the lack of a common definition of a current  acquisition program; insufficient data reliability testing; and inconsistent  compliance with requirements for acquisition program baselines and  reporting on ACAT II and III programs that may become major programs  due to cost growth. Components also lacked plans to ensure their  intended actions are implemented and improvements to data collection  and analysis are sustained over the long term. Until these limitations are  addressed, DOD components will be unable to generate reliable  information to effectively manage and oversee their ACAT II and III  programs.
section_title: Recommendations for Executive Action
paragraphs: We are making four recommendations to improve DOD’s ability to collect  and maintain reliable data on its acquisitions. Specifically, we recommend  that the Secretary of Defense direct the Under Secretary of Defense for  Acquisition, Technology, and Logistics, in consultation with DOD  components, to take the following actions: establish guidelines on what constitutes a “current” ACAT II or III  program for reporting purposes; the types of programs, if any, that do  not require ACAT designations; and whether the rules for identifying  current MDAPs would be appropriate for ACAT II and III programs;  and determine what metrics should be used and what data should be  collected on ACAT II and III programs to measure cost and schedule  performance; and whether the use of DAMIR and the MDAP selected  acquisition report format may be appropriate for collecting data on  ACAT II and III programs.
We also recommend that the Secretary of Defense direct the Secretaries  of the Air Force, Army, and Navy and the Commander of SOCOM to take  the following actions: assess the reliability of data collected on ACAT II and III programs  and work with PEOs to develop a strategy to improve procedures for  the entry and maintenance of data; and develop implementation plans to coordinate and execute component  initiatives to improve data on ACAT II and III programs.
We are also making two recommendations to help ensure compliance  with relevant provisions of DOD acquisition policy with the purpose of  improving DOD’s ability to provide oversight for ACAT II and III programs,  including those programs that may become MDAPs.
We recommend that the Secretary of Defense direct the Secretary of  the Air Force and Commander of SOCOM to establish a mechanism  to ensure compliance with APB requirements in DOD policy.
We recommend that the Secretary of Defense direct the Secretaries  of the Air Force, Army, and Navy to improve component procedures  for notifying the Defense Acquisition Executive of programs with a  cost estimate within 10 percent of ACAT I cost thresholds.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to DOD for review and comment. In its  written comments, which are reprinted in full in appendix V, DOD partially  concurred with all six of our recommendations. However, as discussed  below, it is unclear whether the actions that DOD plans to take will fully  address the issues we raised in this report.
DOD partially concurred with our first recommendation to establish  guidelines on what constitutes a “current” ACAT II or III program for  reporting purposes; the types of programs, if any, that do not require  ACAT designations; and whether the rules for identifying current MDAPs  would be appropriate for ACAT II and III programs. DOD also partially  concurred with our second recommendation related to determining what  metrics should be used and what data should be collected on ACAT II  and III programs to measure cost and schedule performance. In its  response, DOD stated that the Under Secretary of Defense for  Acquisition, Technology, and Logistics would review the existing policy  direction for ACAT II and III programs to determine whether it needs to be  altered or supplemented to facilitate data collection or reporting. However,  as our review found, the question is not whether policy needs to be  revised, but how it needs to be revised. We found that the existing policy  direction was not adequate to ensure consistent data collection and  reporting on ACAT II and III programs or their cost and schedule  performance and our recommendations were designed to address those  issues. We continue to believe that additional guidelines for components  regarding which programs should be considered current ACAT II and III  programs for reporting purposes and consistent metrics to measure  performance trends, among other actions, are needed to correct the  issues we found.
DOD partially concurred with our third and fourth recommendations to  assess the reliability of data collected on ACAT II and III programs and  work with PEOs to develop a strategy to improve procedures for the entry  and maintenance of data; and develop implementation plans to  coordinate and execute component initiatives to improve data on ACAT II  and III programs, respectively. In its response, DOD stated the Under  Secretary of Defense for Acquisition, Technology, and Logistics will direct  the DOD components to evaluate the data collected on ACAT II and III  programs and report back to him on their assessment of the reliability of  that data and the status of the plans to improve the availability and quality  of the data. DOD’s response represents a good first step towards  assessing the reliability of its ACAT II and III program data, but the  response does not fully address our recommendations. DOD’s response  does not address whether components would be required to develop  strategies with PEOs to improve the entry and maintenance of data, as  we recommended. We continue to believe that developing these  strategies with those responsible for entering and maintaining program  data on a day-to-day basis, including PEOs, is important to make sure the  causes of DOD’s data quality problems are fully understood and  addressed in a manner that can be implemented. Further, DOD’s  response does not directly address our recommendation to develop  implementation plans for component efforts. We believe that fully  implementing this recommendation is essential for ensuring that DOD and  its components can effectively monitor and evaluate the implementation  of component initiatives to improve ACAT II and III data.
DOD partially concurred with our fifth recommendation to direct the  Secretary of the Air Force and Commander of SOCOM to establish a  mechanism to ensure compliance with APB requirements in DOD policy.  DOD also partially concurred with our sixth recommendation to direct the  Secretaries of the Air Force, Army, and Navy to improve component  procedures for notifying the Defense Acquisition Executive of programs  with a cost estimate within 10 percent of ACAT I cost thresholds. In its  response, DOD stated that the Under Secretary of Defense for  Acquisition, Technology, and Logistics will issue guidance to DOD  components reiterating the APB requirements for ACAT II and III  programs and directing that the Defense Acquisition Executive be notified  when an increase or estimated increase in program cost is within 10  percent of the ACAT I cost thresholds. Reiterating existing departmental  policy on these issues may help raise awareness at the component level,  but without additional enforcement mechanisms it may not address the  causes of the deficiencies we discuss in this report. For example, the Air  Force has issued component-level guidance directing the development of  APBs. However, we found that programs were not in compliance with the  guidance, which demonstrates the need to improve enforcement  mechanisms, such as ensuring milestone decision authorities do not  approve programs to proceed through acquisition milestones without  APBs. Similarly, with regard to our recommendation on notification  requirements for programs approaching the ACAT I threshold, we found  that component officials cited reasons other than a lack of awareness of  the policy for not notifying the Defense Acquisition Executive of these  programs’ cost growth. As a result, we continue to believe that DOD  should fully implement our recommendation by directing components to  improve their notification procedures.
We are sending copies of this report to the appropriate congressional  committees; the Secretary of Defense; the Under Secretary of Defense  for Acquisition, Technology, and Logistics; the Secretaries of the Army,  Navy, and Air Force; the Commander of U.S. Special Operations  Command; the Assistant Secretary of Defense for Nuclear, Chemical, and  Biological Defense Programs; and other interested parties. This report will  also be available at no charge on GAO’s website at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841 or by e-mail at sullivanm@gao.gov.
Contact points for our Offices of Congressional Relations and Public  Affairs may be found on the last page of this report. Key contributors to  this report are listed in appendix VI.
section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: Our objectives were to assess (1) the extent to which information is  available on the number of the Department of Defense’s (DOD) current  acquisition category (ACAT) II and III programs, their total estimated  acquisition cost, and cost and schedule performance; (2) the factors  affecting the cost and schedule performance of selected ACAT II and III  programs; and (3) the number of current ACAT II and III programs that  are likely to become major defense acquisition programs (MDAP).
To address our first objective, we used a data collection instrument (DCI)  to identify and collect data on the number and cost of current ACAT II and  III programs from five DOD components that accounted for approximately  88 percent of DOD’s requested research, development, test, and  evaluation (RDT&E) and procurement funding in the President’s Fiscal  Year 2014 budget request: Army, Air Force, Navy, U.S. Special  Operations Command (SOCOM), and DOD’s Chemical and Biological  Defense Program (CBDP). We used a DCI to obtain ACAT II and III  program data based on preliminary discussions with DOD and component  officials that a DCI would be the best way to collect the information of  interest. We requested that each component identify all of its current  ACAT II and III programs and provide cost data and descriptive  information for each program. For the purposes of this report, we defined  a current program as one that has been formally initiated in the  acquisition process but has not yet delivered 90 percent of its planned  units or expended 90 percent of its planned expenditures. For cost data,  we requested components provide baseline and current program  estimates in millions of base year dollars, to include estimates for  RDT&E, procurement, acquisition operation and maintenance, and  military construction, as well as the program’s total acquisition cost  estimate and the base year associated with the estimate. We also  collected pertinent information for each program including program name,  ACAT level, type of acquisition (automated information system or non- automated information system), milestone decision authority, lead DOD  component, and program executive office.
To obtain additional information on schedule performance, we collected  and analyzed acquisition program baseline (APB) documents, which  contain program schedule and cost parameters, for a non-generalizable  random sample of 170 non-automated information system ACAT II and III  programs. To select the programs, we used the initial data provided to us  by DOD components that included 836 reported ACAT II or III programs  as a starting point. We adjusted our selection as appropriate to account  for known errors in the data at the time of selection in May 2014, such as  programs that were known to not be current ACAT II or III programs. Our  intention was to select a sample that would be generalizable to the  population of current ACAT II and III programs. However, after selecting  our sample we determined through our data reliability assessment that  the population of current ACAT II and III programs could not be reliably  determined and that our sample would therefore be non-generalizable.As such, results of this analysis cannot be used to make inferences about  all current ACAT II and III programs. When APB documents were  available for programs in our sample, we reviewed them to determine  whether they contained comparable program start and initial operational  capability milestones to allow us to measure program schedule  performance. We also used the APBs collected from this sample of  programs as part of our reliability assessment of ACAT II and III cost data  provided by DOD components.
Our observations on DOD’s ACAT II and III program data are based on  the original data submitted by the components. We did not assess the  reliability of any underlying data systems that may have been used to  generate this information. We analyzed the original data provided by the  components because it reflects the information DOD would have had on  ACAT II and III programs at the time we collected data. From January  through July 2014, we worked with components to attempt to correct  problems we identified in the data. However, we continued to identify  additional errors. As a result, we determined that the data provided by  DOD components in response to our DCI were not sufficiently reliable to  identify the number of current ACAT II and III programs, their estimated  acquisition cost, or the cost performance of DOD’s ACAT II and III  programs. Appendix III contains a more detailed discussion of our data  reliability assessment. We also determined that we could not assess  schedule performance for ACAT II and III programs because more than  half of programs we reviewed in our sample of 170 programs were  missing source documents or lacked comparable schedule milestones.
To address our second objective, we selected a non-generalizable  sample of 15 programs from the data provided by DOD in response to our  DCI. We selected 3 programs from each component included in our  review. For each component, these programs were selected to include  the largest current non-automated information system ACAT II and III  program based on total acquisition cost as of the President’s Fiscal Year  2014 budget submission and one additional program based on factors  such as significant cost growth or whether the program was part of a  family of systems, which we defined as a related group of programs  consisting of multiple increments or fielding similar capabilities for multiple   To select the programs, we used the initial data provided to us  platforms.by DOD components that included 836 reported ACAT II or III programs  as a starting point. Programs that lacked data for current acquisition cost,  commodity type, or ACAT level were excluded from selection. We also  adjusted our selection as appropriate to account for known errors in the  data at the time of selection, such as incorrectly-reported cost estimates,  or programs that were known to not be current ACAT II or III programs.  However, after our selection we identified additional concerns with the  data reported by DOD that would likely have changed the results of our  selection of the largest ACAT II or III programs at certain components.  We did not make any subsequent adjustments to our original selection  because we determined that the data provided by DOD was not  sufficiently reliable to enable us to determine the largest ACAT II or III  program at each component.
For each program, we analyzed key program documents, such as APBs,  program status reports, acquisition strategies, acquisition decision  memoranda, and requirements documentation, to assess cost and  schedule performance and identify factors affecting that performance. We  also conducted semi-structured interviews with program officials to  discuss the information identified through reviews of program  documentation and obtain additional insights into factors that affected  program cost or schedule performance. Additionally, we analyzed prior  GAO reports to determine the extent to which the factors we identified as  affecting cost and schedule performance for selected ACAT II and III  programs were similar to factors that we have identified in prior work as  affecting performance of MDAPs.
To address our third objective, we reviewed DOD acquisition policy  related to the reclassification of ACAT II or III programs to ACAT I  programs and analyzed program cost data provided by DOD  components. Based on the requirement in DOD acquisition policy for  components to notify the Defense Acquisition Executive of ACAT II or III  programs within 10 percent of the next ACAT level, we analyzed data  provided by DOD through our DCI to identify programs that appeared to  be within 10 percent of or have exceeded either the ACAT I RDT&E or  procurement threshold. We were unable to identify an actual number of  programs likely to become MDAPs because of reliability issues related to  identifying the population of ACAT II and III programs. However, we  determined the initial data provided to us by DOD that included 836  reported ACAT II or III programs were sufficiently reliable to serve as a  starting point to identify the minimum number of programs likely to  become MDAPs because we were able to confirm data with relevant  program offices for those programs that appear to be within 10 percent of  or have exceeded either ACAT I threshold. We excluded certain  programs from further review that were known at the time that we initially  identified programs to have incorrectly-reported cost estimates.
For programs that appeared to meet our criteria for current ACAT II or III  programs likely to become MDAPs, we collected additional information  using a structured set of questions to determine whether the relevant  DOD component had notified the Defense Acquisition Executive that the  program was approaching or had exceeded the ACAT I threshold and  whether the program had been or was expected to be reclassified as an  ACAT I program. We also requested and reviewed supporting  documentation when available, including documentation of notification to  the Defense Acquisition Executive that the program was within 10 percent  of the ACAT I threshold. After we received the information from the  components, we identified additional programs that had incorrectly  reported cost estimates or were no longer current ACAT II or III programs  and we removed these programs from our analysis as appropriate.
We conducted this performance audit from October 2013 to March 2015  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Selected Acquisition Program Milestone Documentation Requirements and Congressional Reporting Requirements by Acquisition Category
paragraphs: Summarizes program cost, schedule,  and performance parameters  Program cost estimate completed  outside of the supervision of the entity  responsible for the acquisition program  Documents capability requirements to  which the program responds  Describes program’s overall technical  approach and details timing and criteria  for technical reviews  Technology Readiness Assessment  Assessment of the maturity of critical  Primary planning and management tool  for integrated test program  10 U.S.C. § 2366a and b.
Provides notification of unit cost  breaches above a certain threshold Provides notification of cost or schedule  changes above a certain threshold 10 U.S.C § 2433 and 2433a. Notification must be provided to Congress when the program  acquisition unit cost or average procurement unit cost increases by at least 15 percent over the  current baseline estimate or 30 percent over the original baseline estimate.  10 U.S.C § 2445c. Notification must be provided to Congress when there is a schedule change that  will cause a delay of more than 6 months; an increase in the expected development cost or full life- cycle cost for the program by at least 15 percent; or a significant, adverse change in the expected  performance of the major automated information system to be acquired.
section_title: Appendix III: Observations on Reliability of Department of Defense Components’ Acquisition Category II and III Program Data
paragraphs: We conducted an analysis to determine whether data provided by  Department of Defense (DOD) components were sufficiently reliable for  the purpose of determining the number, total acquisition cost, and cost  performance of DOD’s current acquisition category (ACAT) II and III  programs. For our analysis, we conducted electronic and manual testing  on data for all programs reported by components in response to our  request for completion of a data collection instrument (DCI) and  compared cost data for a non-generalizable sample of programs to  source documents when available. We also reviewed relevant DOD and  component acquisition policy, and interviewed knowledgeable officials.  We identified reliability issues with the data for about 60 percent of the  programs components initially reported to us. As a result, we determined  that the data provided by DOD components were not sufficiently reliable  to identify the number of current ACAT II and III programs, their estimated  total acquisition cost, or the cost performance of DOD’s ACAT II and III  programs.
subsections: section_title: Methodology
paragraphs: To assess the accuracy and completeness of the ACAT II and III program  data reported by DOD components, we electronically tested the data for:  values outside the designated range of values for ACAT II and III  programs, defined per DOD acquisition policy; obvious calculation or data entry errors (for example, individual cost  elements do not sum to total reported); missing data in baseline or current cost estimate data elements,  including estimates for research, development, test, and evaluation;  procurement; military construction; and acquisition operation and  maintenance, as well as the total acquisition cost estimate, and base  year; and missing data in program descriptive data elements, such as ACAT  level, milestone decision authority, or commodity type.
Additionally, we compared cost data for our sample of 170 programs to  source documents when available. Specifically, for each program in our  sample, we first requested and reviewed original and current acquisition  program baselines (APB) to determine whether or not they reflected the  actual baseline from program start and the current APB based on the  approval date of the APB and relevant schedule milestones that trigger  the development of an APB or an APB revision in accordance with DOD  acquisition policy. When APBs were not provided or did not appear to  reflect the actual baseline and/or current APB, we followed up with DOD  components to obtain the correct documents when possible. When we  were able to obtain both original and current APBs, we took the following  steps to assess the accuracy of the information reported in the DCI:
Compared baseline cost data from the program’s original APB to the  baseline cost data reported in the DCI.
Compared cost data in the current APB to cost data reported in the  DCI to identify obvious errors in the cost data reported in the DCI,  such as current cost data in the DCI that was significantly less than  the amount reported in the APB without explanation.
To assess the consistency of ACAT II and III program data, we manually  reviewed the data provided in response to the DCI and subsequent  requests to identify programs that did not appear to meet our criteria for  current ACAT II and III programs. For the purposes of this report, we  defined a current program as one that has been formally initiated in the  acquisition process but has not yet delivered 90 percent of its planned  units or expended 90 percent of its planned expenditures. For each  program that did not appear to be a current ACAT II or III program, we  analyzed whether the program was pre-program start, in sustainment,  completed, or not a separate ACAT II or III program (for example, was a  subprogram of another ACAT II or III program reported to us).
subsections: section_title: Results of Analysis
paragraphs: The results of our data reliability analysis capture accuracy,  completeness, and consistency issues with the data provided by DOD  components. Accuracy, completeness, and consistency are key  characteristics of reliable data and refer to (1) the extent that recorded  data reflect the actual underlying information; (2) data elements for each  program are populated appropriately; and (3) the need to obtain and use  data that are clear and well defined enough to yield similar results in  similar analyses, respectively.
subsections: section_title: Accuracy and Completeness
paragraphs: We identified numerous types of accuracy and completeness issues with  the data provided by DOD components, including cost estimate values  outside of the ACAT II and III program range, basic math errors, and  missing data. For example, 333 out of 836 ACAT II and III programs  reported by the components were missing a baseline or current cost  estimate element or descriptive program data. Table 9 provides detail on  accuracy and completeness issues by component.
We identified additional issues with the accuracy of the ACAT II and III  program cost information when we compared reported cost estimates to  available source documents for a non-generalizable sample of ACAT II  and III programs. Specifically, of the 81 programs in our sample that  reported complete cost estimates and provided source documents, 50  reported incorrect cost data. For example, for 37 of these 50 programs,  we determined that baseline cost data was inaccurate because either the  baseline cost estimate or the base year reported for this estimate did not  match the source documents. Details of the accuracy and completeness  issues we identified when assessing the cost data reported for our  sample are provided by component in table 10.
We identified 226 of the 836 programs reported by DOD components that  did not meet our criteria for current ACAT II or III programs because, for  example, they were not current or they were not stand-alone acquisition  programs. Additionally, because information on program phase was not  available for all programs reported by DOD components, the number of  programs we identified as not a current ACAT II or III program reflects a  minimum number of such programs. Table 11 provides details on  consistency issues we identified by component.
section_title: Appendix IV: Description of Selected Department of Defense (DOD) Acquisition Category (ACAT) II and III Programs
section_title: Appendix V: Comments from the Department of Defense
section_title: Appendix VI: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Ron Schwenn, Assistant Director;  Leslie Ashton, Jenny Chanley, Teakoe Coleman, Dani Greene, John  Krump, Jesse Lamarre-Vincent, Anne McDonough-Hughes, Carol  Petersen, and Oziel Trevino made key contributions to this report.