Using Data to Improve Defense Acquisitions: Background, Analysis, and Questions for Congress
Released: 2016-01-05T00:00:00

Introduction

In FY2014, the Department of Defense (DOD) obligated more than $280 billion for federal contracts, more than all other federal agencies combined. DOD's obligations were equal to 8% of federal spending. Many analysts and government officials have argued that by more effectively using data to support acquisition decisionmaking, DOD could save billions of dollars, allocate resources more efficiently and effectively, and improve the effectiveness of military operations.

Some have argued that better use of data may be one of the keys to reforming defense acquisitions. In a letter to the House and Senate Armed Services Committees, the National Defense Industrial Association listed data as the first critical step toward improving defense acquisitions.

In recent years Congress has pursued a variety of approaches to improving DOD's efficiency, such as requiring the department to be auditable, including provisions on acquisition reform in National Defense Authorization Acts, and holding numerous hearings on agency operations and acquisition reform. To the extent that improved data analysis could enable more effective decisionmaking, Congress may choose to enhance oversight in this area and explore ways to enable DOD to conduct more effective data analysis.

This report examines (1) the extent to which DOD effectively uses data to inform decisionmaking, (2) some of the critical elements needed for DOD to use data more effectively, (3) recent efforts to improve DOD's use of data, and (4) potential questions for Congress. This report focuses primarily (but not exclusively) on defense acquisitions as a case study examining how effectively DOD uses data to support its decisionmaking.

Related CRS reports include CRS Report R43566, Defense Acquisition Reform: Background, Analysis, and Issues for Congress , by [author name scrubbed], and CRS Report R43074, Department of Defense's Use of Contractors to Support Military Operations: Background, Analysis, and Issues for Congress , by [author name scrubbed].

Background

In light of advances in information technology that have made it possible to track and analyze large amounts of data, many analysts now believe that data analysis has become a critical element in making well-informed policy decisions and managing government programs. When decisionmakers have access to sufficient data from which to draw reasonable conclusions, they are in a better position to measure or assess the effectiveness of government programs, inform policy decisions, and provide transparency into government operations. In some circumstances, a lack of data can lead analysts and decisionmakers to draw incorrect or misleading conclusions. The result may be policies that squander resources, waste taxpayer dollars, and/or undermine the effectiveness of government programs or military operations. As two analysts wrote, "data-driven decisions are better decisions—it's as simple as that." Consider:

A 2015 GAO report found that officials responsible for acquisitions and developing requirements lacked access to data and the analytical tools necessary to conduct effective reviews. A 2015 Inspector General report found that DOD did not properly review or track data on spare parts, contributing to excess inventories exceeding $1 billion. A 2014 Inspector General report found that the Army's audit readiness was at risk "because of unreliable data in the appropriations status report." A 2013 GAO report found that due to a lack of data to support better decisions, DOD did not effectively manage its supply chain in Afghanistan. This mismanagement had the effect of "hampering the distribution of supplies and equipment to the warfighter" and affecting operations in Afghanistan. According to a 2011 report, after almost 10 years of war, DOD possessed neither a consolidated theater-wide database capable of tracking the amount of money spent on reconstruction nor a database tracking the amount of money spent on contracting in Afghanistan.

Despite the importance of data, general consensus exists among government officials and analysts that DOD does not sufficiently incorporate data into decisionmaking. Many major policy decisions—from economic development programs in Afghanistan to choosing the weapon systems of the future—are made without the benefit of substantive data. These policies are sometimes based on assumptions, and some have argued that program reviews do not always sufficiently incorporate relevant data against which to measure success. As retired Air Force General Norman Schwartz noted in a discussion on what he learned about the private sector after leaving the military,

People make decisions in the private sector largely based on data – some of it is instinct, but data driven decisions is a big thing and I have learned that and certainly that applies in a government setting.

In recent years, Congress has sought to highlight the importance of data in improving government operations. In 2014, Congress enacted and the President signed into law the DATA (Digital Accountability and Transparency) Act, which was intended, among other things, to

enable taxpayers and policy makers to track Federal spending more effectively; (2) establish Government-wide data standards for financial data and provide consistent, reliable, and searchable Government-wide spending data that is displayed accurately for taxpayers and policy makers ... [and] improve the quality of data submitted to USASpending.gov by holding Federal agencies accountable for the completeness and accuracy of the data submitted.

In a recent hearing on management of DOD, Senate Armed Services Committee Chairman John McCain stated

it's hard to address management problems when you lack basic data that are essential to understanding and diagnosing those problems. And, yet, that is the case with the Department of Defense.

Here is how former Secretary of Defense Robert Gates described the dilemma. He said, quote, "My staff and I have learned that it was nearly impossible to get accurate information and answers to questions, such as, 'How much money did you spend?' and, 'How many people do you have?'"

The result is not just greater inefficiency and wasted resources. It also harms the effectiveness of the Department of Defense, and, thus, our national security. The result of these shortfalls in information, as Secretary Gates has explained, is that department leaders and their overseers in Congress cannot measure the results of our national security policies, or make judgments about priorities for our military, or accurately assess the tradeoffs involved in different courses of action.

Challenges in Using Data to Support Decisions

A number of analysts and government officials have argued that some of the critical elements required for DOD to use data more effectively include:

1. having the information systems to gather and manage data; 2. ensuring that data is sufficiently comprehensive and accurate; and 3. using data to inform decisionmaking.

Having Information Systems

Having the right information systems presupposes that a system exists. In some cases, however, decisionmakers lack even a simple database to maintain and manage information. According to GAO, the Joint Staff lacked a database for gathering the information necessary to manage weapon system portfolios. The consequence of not having sound systems is that managers of multibillion dollar programs might manage and oversee activities using manual, paper-based processes. As an article in Defense AT&L magazine argued, such a manual process is labor-intensive and time-consuming, and more likely to introduce unintended errors during the process. Such a process can also prevent leaders from fully exploiting "valuable troves of program information because the process creates innumerable separate and often conflicting data sources, rather than authoritative and searchable information sources."

If a system does exist, it needs to be capable of effectively performing the required functions. A number of DOD systems have been found to be fundamentally flawed. GAO recently testified

the department's ability to improve its budgetary accounting has historically been hindered by its reliance on fundamentally flawed financial management systems and processes and transaction control weaknesses....

While DOD is working to improve its financial management systems, according to GAO, a number of these efforts are not expected to be complete until 2017.

Another aspect of having the right data systems includes having a single, agreed source for analyzing specific data sets. In some cases, DOD has multiple information technology (IT) systems and databases containing the same or similar information. There may be sound reasons to have the same data stored in more than one system. However, the multiplicity of systems can become a concern when the data in the various systems conflict—and it is not always clear which, if any, system is the agreed source for given data.

All of these concerns have led to calls to improve the architecture of the data systems used for collecting, managing, and analyzing DOD acquisition data.

Ensuring Data Is Sufficiently Reliable and Comprehensive

In some instances, existing systems do not have reliable or complete information. Instances of insufficient or nonexistent data are known to exist in a number of DOD data sets, including those related to contract obligations, workforce, contractor past performance, and operations and support costs. The costs of both the Federal Procurement Data System—Next Generation (FPDS), and Operations and Support illustrate the challenges with data reliability and sufficiency, as well as with government efforts to improve it.

FPDS is a central database of U.S. gove rnment-wide procurement. Congress, legislative and executive branch agencies, analysts, and the public all rely on FPDS as a primary source of information for understanding how and where the federal government spends contracting dollars. Congress and the executive branch rely on the information to help make and oversee informed policy and spending decisions. Analysts and the public rely on the data to conduct analysis and gain visibility into government operations.

The FPDS database plays an important role in decisionmaking, and a number of observers have raised concerns over the accuracy, limitations, and reliability of the data contained therein. According to GAO, FPDS often contains data with limited "utility, accuracy, and completeness." The Office of Management and Budget has also released guidance requiring executive branch agencies to implement GAO recommendations seeking to improve FPDS data quality. Continued concerns raised over the reliability of FPDS data have prompted many analysts to rely on it only for identifying broad trends and making rough estimations.

According to the General Services Administration (GSA), a number of data systems, including FPDS, are undergoing a significant overhaul. This overhaul is a multiyear process that is expected to improve the reliability and usefulness of the information contained in the data systems. Part of the effort includes focus groups with stakeholders, including agency decisionmakers and congressional staff, to solicit feedback on how to improve the reliability, usability, and relevance of the data stored in the systems being updated. While no date has been set for completing this effort, officials believe that the upgrades will be rolled out sometime in 2017 or 2018.

Tracking Operations and Support (O&S) costs is another case in point. In 2012, GAO found that DOD reports to Congress on weapon system O&S costs were "often inconsistent and sometimes unreliable, limiting the visibility needed for effective oversight of these costs." As a result of having insufficiently comprehensive and reliable data, GAO also found that DOD officials did not have the information necessary to analyze operations and support cost growth for major systems, identify cost drivers, or develop plans for managing and controlling such costs. Not only were O&S data incomplete, sometimes the data were inconsistently reported. According to DOD, "nothing prevented a program office from reporting unit O&S cost estimates on a per item basis (e.g., per airplane) in one year and on a per usage basis (e.g., per flying hour) in the next year." Nor have DOD components consistently tracked O&S costs against final estimates after the systems were fielded.

The office of the Under Secretary of Defense for Acquisition, Technology, and Logistics (AT&L) is now requiring more consistent use of units for calculating O&S, but it will take years to build a bank of data for conducting analysis. According to DOD, other phases of the acquisition process experience similar data access, quality, and availability problems.

Using Data to Inform Decisions

Even when reliable and complete data do exist, they do not necessarily reach the appropriate decisionmakers or get effectively incorporated into the decisionmaking process. As a result, some policies and acquisition decisions are driven by perceptions or anecdotal evidence, without sufficient data analysis.

In some instances, analysts and decisionmakers may not know of certain data's existence or may not be able to get access to the data they seek. There are times when the need to protect information argues against sharing data. However, in some circumstances, there are barriers to sharing data that are not grounded in such security concerns. According to a recent RAND report

there are institutional and cultural barriers to sharing. For example, interviewees pointed to a stovepiped structure of DoD and how it limits visibility and the sharing of information. Institutional structure and bureaucratic incentives to restrict data access are exacerbated by policy and guidance to protect information. The result is a strong conservative bias in labeling and a reluctance to share.

Another challenge is getting decisionmakers to use existing data effectively. Some observers argue that DOD does not have sufficient numbers of people capable of conducting complex data analyses and providing the results of the analyses to decisionmakers. Some analysts and government personnel believe that there are senior leaders within DOD who do not understand how to use data effectively to inform decisions.

Part of using data effectively is ensuring that the data are accurate and relevant. But as one senior official stated, "Often times, unchecked statistics and slick presentations are used to make decisions without substantive or realistic validation of underlying assumptions." Some government officials suggested that in other cases, decisionmakers do not use data because the data may not support the decision they intend to make.

Understanding the Limits of Data41

Recognizing the importance of data, a number of DOD officials and analysts have argued that in pursuing a data-driven approach to decisionmaking, it is important to understand the limitations of data.

One concern raised by a number of observers is the need to avoid gathering data for data's sake. Another common trap cited by some observers is the tendency to just start collecting data and then asking what the data can do for you. Without a well-thought-out understanding of what data is needed—and why— the quest for data could end up being an end in itself, with very limited usefulness. Some DOD officials believe that too much time is already spent responding to data calls and following reporting requirements that are not connected to a useful analytical purpose, diverting the attention of acquisition personnel from managing their programs.

Another caution raised by some observers is the need to ensure that the benefit of data analyses outweighs the cost of collecting and analyzing data (in terms of dollars, time, and other resources). For example, spending two years and $100 million dollars to build an IT system to inform a decision of whether to invest $75 million dollars in a one-time project is likely to be considered by most people as a poor use of resources.

Some observers have argued that striving for more and better data can sometimes be an elusive goal that can actually get in the way of effective data analysis. These observers note that data do not need to be perfect to be useful. Rather, decisionmakers should focus on whether sufficient data exist from which to draw reasonable conclusions. One official in DOD with experience in the insurance industry suggested that when working with data, critical questions to ask include:

1. What useful analysis can be developed now? 2. How can we model correctly? 3. What are the uncertainty bounds of the data (as an effort to understand the usefulness of the data)?

Just as imperfect data can be useful, even perfect data is unlikely to yield perfect analysis. As one senior official stated (paraphrasing famed statistician George Box), all models are wrong, but some are useful. Cost estimating is a case in point. Even under the best of circumstances, cost estimates for large programs are imprecise. But a good cost estimate can aid in understanding the potential probability and consequence of various risks, which in turn can help decisionmakers craft budgets, allocate resources, and develop test plans.

Case Study: Lack of Reliable Data Hampered Decisionmaking in Afghanistan

From FY2007 to FY2013, DOD obligated more than $80 billion on contracts in Afghanistan. Yet during that time period, DOD and the U.S. government were unable to accurately or sufficiently track data with which to make strategic contracting decisions in Afghanistan. As one study found, "due to a lack of reliable information, neither the Afghan government nor the international community can determine the amount of money spent in Afghanistan over the past 10 years."

DOD Did Not Have the Right Information Systems or Data Architecture

In some instances, DOD did not possess the IT systems to track data—or existing systems were not sufficiently customized to track relevant contract data. For example, no consolidated theater-wide database capable of tracking the amount of money spent on reconstruction existed. Nor was there a database tracking the amount of money spent on contracting in Afghanistan. DOD lacked the necessary data to effectively manage its supply chain in Afghanistan. GAO concluded that the inability to track and manage the supply chain "hindered the distribution of supplies and equipment to the warfighter and will likely continue to affect operations in Afghanistan and limit DOD's visibility and oversight of the supply chain."

In other instances, different systems possessed conflicting information. In 2009, Greg Gardner, then-Deputy Chief Information Officer (DCIO) for the Director of National Intelligence, stated that the U.S. government had at that time 23 different network information technology systems in Afghanistan, many of which were duplicative and/or not interoperable. According to Gardner, this multiplicity of IT systems and the lack of interoperability resulted in wasteful spending and poor data sharing between and within agencies.

Existing Data Was Not Always Sufficiently Reliable

Even when IT systems existed and information was tracked, the reliability of the data remained questionable. According to database managers, data was often kept at the local level and did not flow up to the central database. When data was tracked and sent to the central database, it was often incomplete and unreliable. For example, there were five required data elements for reconstruction projects. According to the managers of the theatre-wide database, in one instance, of approximately 59,000 records submitted to the database, only about 8,000 (14%) contained all five required elements.

Some areas of data were more accurately tracked than others, such as such as IED attack situation reports, which were entered fairly consistently and accurately. Commanders Emergency Response Program data was not accurately tracked until 2009, when an emphasis was placed on accurately tracking such data. Data managers estimated that overall, only about 10% of all data records were eventually put into the central database and that the error rate of data in the system was approximately 50%.

Existing Data Was Not Always Used by Decisionmakers

Because people were often assigned to perform tasks in areas in which they had little or no experience, officials who should have had certain information were not always aware that the data existed. For example, a number of military and civilian personnel involved in developing counterinsurgency (COIN) contracting strategies and policies were not aware of the existence of the Central Command quarterly contractor census that was released by the office of the Deputy Assistant Secretary of Defense (Program Support). Many of these officials were also unfamiliar with the contracting data contained in FPDS. These sources included information directly related to contracting in Afghanistan.

DOD officials acknowledged data shortcomings and worked to improve the reliability and appropriateness of the data gathered. Since then, DOD has made significant progress in identifying the types of data needed to make better contract decisions, identify sources of data, and gather the identified data. DOD may be able to leverage the data systems built to support operations in Afghanistan to prepare for and execute future operations. Ensuring that such data systems are available early in future operations could provide commanders and policymakers with timely access to critical information.

Can DOD Improve the Use of Data in Acquisitions?

Analysts and Senior DOD officials acknowledge that the department does not sufficiently use data to inform decisionmaking. Significant gaps continue in what data is available and the reliability of some of the data that exists. DOD has numerous efforts underway aimed at improving IT systems, and has embarked on a number of wide-ranging efforts to gather and analyze data to inform policy decisions, often at the behest of Congress. Many analysts argue that to succeed in these efforts, DOD culture must not only value using data to drive decisions, but also integrate data gathering and analysis into the fabric of the organization, making it part of standard routines and operating procedures. Some have argued that a key to changing the culture in DOD is leadership. A report by IBM stated

For analytics to become an integral part of agency activities, leaders must live by example, using data for decisions in an open and transparent manner.... Leadership support is vital for a successful analytics program.

In recent years, a number of senior acquisition officials in DOD have emphasized the need to transition to a more data-informed decisionmaking process. This focus has become manifest in numerous ways, from the sign hanging in Under Secretary Kendall's office which states "In God We Trust. All Others Must Bring Data" to the 2013 release of the first annual report Performance of the Defense Acquisitions System — a 110-page report that relies extensively on data gathered over a 30-year period to analyze and measure the effectiveness of acquisitions. To date three annual reports have appeared, which are among the most comprehensive, data-driven analyses on defense acquisitions issued by DOD in recent years.

A number of analysts point out that senior leadership is promoting a more data-driven process for supply chain management. While viewed by many observers as a positive step, some argue that statements by senior leadership are not enough.

DOD Efforts

DOD is investing billions of dollars trying to build information technology systems and analytical tools to support a more data-based decisionmaking process. DOD and the federal government are also working to improve the quality of data. According to GSA, a number of data systems, including FPDS, are undergoing a significant overhaul in an effort to improve the reliability and usefulness of the information contained in the data systems. Moreover, DOD is working to improve the quality and consistency of operations and support data.

In 2013, Under Secretary of Defense Frank Kendall established an analytics center charged with gathering and analyzing data to measure the effectiveness of acquisitions and the use of data to help craft policy. This center is responsible for developing the annual reports on the performance of the defense acquisition system. The Cost Analysis and Program Evaluation office (CAPE) also recently launched the cost assessment data enterprise (CADE) project aimed at consolidating historical acquisition data (much of which currently resides in PDF form) into a single on-line system. The intent is to provide decisionmakers and cost analysts with timely access to a trove of historical data in a format that can easily be manipulated and used to conduct analysis.

The military services also have a number of initiatives aimed at better integrating data into decisionmaking. Navy officials have stated that they are taking a fresh look at IT systems (including enterprise resource planning systems) to determine precisely what data they need, why it is needed, how the data will be used, and the extent to which a business case justifies the necessary investment to gather and analyze the data. According to a senior official, the Air Force recently stood up the Air Force IT Business Analytics Office; to date the office has identified more than $500 million in potential cost savings. The Air Force is also developing a baseline of past acquisition cost performance across the service's different types of acquisitions.

Congressional Efforts

Many analysts and senior DOD officials have credited Congress with helping DOD focus on and improve the use of data in decisionmaking. Examples of congressional action cited as contributing to improving DOD's use of data to drive decisionmaking include

The Weapon System Acquisition Reform Act . The act focused on using data to inform decisions in the early stages of the acquisition process. Analysts and officials pointed to the establishment of the CAPE (which strengthened the use of independent cost estimates in formulating budgets), developmental test and evaluation, and systems engineering. According to a senior acquisition official, the act had a significant effect in improving the realism of cost estimates. The act also required DOD to conduct a root cause analysis of the cost, schedule, and performance of major defense acquisition programs that experience cost growth that surpasses the thresholds set forth in the Nunn-McCurdy Act. Requiring DOD t o Become Audit Ready. The FY2010 NDAA requires DOD to ensure that its financial statements "are validated as ready for audit" and to "improve the accuracy and reliability of management information…." According to a DOD presentation, auditability will bring greater awareness and control of business operations, which in turn will make DOD more efficient and better at deploying resources.

Despite current efforts to improve data-based decisionmaking, success is not guaranteed. Many past efforts to improve the efficiency and management of DOD have not succeeded, and those that have met with success sometimes fell short of initial expectations. Along the way, there will likely be successes (a number of officials and outside analysts have cited the Navy's Air System Command as an example of success) as well as failures. (In 2012, after spending more than $1 billion, the Air Force cancelled its effort to develop an enterprise resource planning system to manage logistics.) Under the best of circumstances, it will take years for DOD to implement and improve data systems and foster a culture that routinely uses data to support its decisions. As a senior DOD official stated

our ability to affect decisions with data may take years to accomplish. Because historic data is often sparse, it's difficult to make informed assessments of whether contemporary observations are endemic or transitory. We need to build a robust historical database to truly understand how our acquisition system is performing. Otherwise, there are likely going to be knee‐jerk reactions to short term events without a firm understanding of long term trends.

In this vein, it may be useful to recall the title of a report written by former Under Secretary of Defense (Comptroller) Robert Hale: "Promoting Efficiency in the Department of Defense: Keep Trying, But Be Realistic."

Questions for Congress

The extent to which DOD is successful in promoting the use of data to inform decisionmaking will depend, in part, on congressional action. Congress could choose to allocate resources to support building the systems and workforce necessary to gather and analyze data. Congress could also pass legislation to influence the decisionmaking process within DOD, as it did when it passed the Weapons System Acquisition Reform Act of 2009. Answering the following questions could help Congress determine further actions to take in its efforts to improve defense acquisitions.

To What Extent Does DOD Have the Systems and Qualified People in Place to Conduct Robust Data Analysis?

A prerequisite for conducting effective data analysis is having the appropriate information systems and the qualified personnel needed to conduct effective and sophisticated analysis. Congress may want to consider whether or not to commit resources to build analytical capability, such as investing in IT systems, or increasing the capacity of the offices within DOD, including the Cost Analysis and Program Evaluation or the Service Acquisition Executives within each service, or other relevant offices. In considering how to allocate resources, Congress may wish to examine:

To what extent does DOD have a clear strategy for understanding what data it needs, and how to deploy the right technology architecture and capabilities? What steps are required to execute the strategy and put in place the systems necessary for gathering and analyzing data? To what extent does DOD have the in-house capability and qualified personnel to conduct robust data analysis? To what extent does DOD have a strategy for hiring qualified personnel to conduct data analysis? To what extent can such capability be effectively contracted out?

To What Extent Does DOD Have the Right Culture and Processes in Place to Foster the use of Data to Inform Decisionmaking?

Many analysts argue that a prerequisite for effectively using data throughout an organization is having a culture that not only values using data to drive decisions, but also integrates data gathering and analysis into the very fabric of an organization, making it part of standard routines and operating procedures. Congress could consider legislation as a path to promote the use of data to inform decisions. In considering legislation, Congress may wish to examine:

To what extent does DOD have the right culture to foster the use of data in decisionmaking? To what extent do the incentives within DOD promote, or inhibit, the use of data? What steps can be taken to incorporate data gathering and analysis into the routine processes of DOD? To what extent does DOD have a clear strategy for how to use data analytics to improve acquisitions and business operations?

How Can DOD Balance the Benefits of Sharing Data with the Need to Safeguard Information Security Concerns?

A recent RAND report that looked at access to acquisition data concluded

Our findings show that access to data and information is inefficient at best. Many government personnel supporting the acquisition process often do not get their first choice of data, and what they do get may not be delivered in a timely fashion.

As beneficial as data-sharing can be, there may be instances where information security concerns suggest that data-sharing is not appropriate. Congress might ask: how can DOD balance the competing interests of sharing data to improve efficiency with protecting information?