Defense Logistics: Improved Performance Measures and Information Needed for Assessing Asset Visibility Initiatives
Published: 2017-03-16T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: GAO designated DOD's supply chain management as a high-risk area in 1990 and in February 2011 reported that limitations in asset visibility make it difficult to obtain timely and accurate information on assets that are present in a theater of operations. DOD defines asset visibility as the ability to provide timely and accurate information on the location, quantity, condition, movement, and status of items in its inventory. In 2015, GAO found that DOD had demonstrated leadership commitment and made considerable progress in addressing weaknesses in its supply chain management.
This report addresses the extent to which DOD has (1) identified performance measures that allow it to monitor the progress of selected asset visibility initiatives identified in its Strategies ; and (2) addressed the five criteria—leadership commitment, capacity, corrective action plan, monitoring, and demonstrated progress—for removing asset visibility from the High Risk List. GAO reviewed documents associated with selected initiatives, surveyed DOD officials, and observed demonstrations.
section_title: What GAO Found
paragraphs: The Department of Defense (DOD) has identified performance measures for the eight selected asset visibility initiatives GAO reviewed, but these performance measures generally cannot be used to monitor progress. Specifically, GAO found that the measures for the eight initiatives reviewed did not generally include key attributes of successful performance measures. For example, for six initiatives there were no baseline and trend data associated with the measures. While DOD's 2014 and 2015 Strategy for Improving DOD Asset Visibility ( Strategies ) called for performance measures to be identified for the initiatives, the Strategies lacked complete direction on how to develop performance measures that would allow DOD to assess the progress of the initiatives toward their intended outcomes. GAO also found that after-action reports for the initiatives did not always include key information needed to determine the success of the initiatives in achieving the goals described in the Strategies . Without improved performance measures and information to support that progress has been made, DOD may not be able to monitor and show progress in improving asset visibility.
DOD has made progress and meets the criteria related to capacity and its corrective action plan but needs to take additional actions to monitor implementation and demonstrate progress to meet GAO's two remaining criteria for removal from the High Risk List, as shown in the figure. For the capacity criterion, in its draft update to the 2015 Strategy , DOD provides guidance on how to document cases where the funding for the initiatives is embedded within the overall program funding. Also, for the action plan criterion, DOD included matrixes in its 2015 Strategy to link ongoing initiatives to the Strategy 's goals and objectives. DOD has also taken steps to monitor the status of initiatives. However, the performance measures for the selected initiatives that GAO reviewed generally cannot be used to track progress and are not consistently incorporated into reports to demonstrate results. Until these criteria are met, DOD will have limited ability to demonstrate sustained progress in improving asset visibility.
section_title: What GAO Recommends
paragraphs: GAO recommends that DOD use key attributes of successful performance measures in refining measures in updates to the Strategy and incorporate information related to performance measures into after-action reports for the asset visibility initiatives. DOD partially concurred with both recommendations. The actions DOD proposed are positive steps, but GAO believes the recommendations should be fully implemented, as discussed in the report.

section_title: Letter
paragraphs: One of the most complex and vital tasks facing the Department of  Defense (DOD) is managing its supply chain to effectively and efficiently  provide spare parts, food, fuel, and other critical supplies in support of  U.S. military forces. DOD’s goal and challenge are to deliver the right  items in the right quantities to the right place at the right time—and at the  right cost. Supply chain management encompasses the processes and  systems for accomplishing this goal and includes inventory management,  materiel distribution, and asset visibility. According to DOD, to achieve a  seamless and effective supply chain, DOD needs to have end-to-end  visibility of its assets from acquisition to disposal. DOD defines asset  visibility as the ability to provide timely and accurate information on the  location, quantity, condition, movement, and status of items in its  inventory, including assets in transit. Maintaining visibility of these assets  is critical to ensure that DOD provides support to the warfighter. Because  of long-standing weaknesses in DOD’s supply chain management, in  1990 we designated it as a high-risk area. In 2005, we added asset  visibility to the supply chain management high-risk area. In 2011, we  reported that limitations in asset visibility made it difficult for DOD to  obtain timely and accurate information on the assets that are present in  the theater of operations.
In our February 2015 biennial update to the High-Risk Series, we  reported that DOD had made progress addressing weaknesses in its  supply chain management, which is comprised of three issue areas:  inventory management, materiel distribution, and asset visibility. With  respect to asset visibility, we reported that the department had met one of  our five criteria—leadership commitment—for the removal of asset  visibility from the high risk list, and partially met the other four criteria— capacity, corrective action plan, monitoring, and demonstrated progress.  We reported that DOD had made considerable progress from 2013 to  2015 in addressing the four remaining criteria through actions such as  developing the department’s framework for improving asset visibility,  known as its 2014 Strategy for Improving DOD Asset Visibility (2014  Strategy). The 2014 Strategy outlined initiatives intended to improve  DOD’s asset visibility. Furthermore, we reported that to fully address the  remaining four high-risk criteria DOD needed to take a number of actions,  such as linking the goals and objectives in the 2014 Strategy with the  initiatives intended to implement the Strategy. We also reported that DOD  needed to assess and refine, as appropriate, existing performance  measures to ensure that these measures assessed the implementation of  individual initiatives as well as progress toward achieving the Strategy’s  goals and objectives.
In December 2015, we reported that DOD had updated its 2014 Strategy  in October 2015 (2015 Strategy), and had taken numerous actions to  address weaknesses in its 2014 Strategy. For example, the 2015  Strategy addressed steps DOD was taking to facilitate collaboration with  industry to capture best practices with respect to asset visibility. Like its  predecessor, the updated 2015 Strategy states that it is intended to  create a framework whereby the DOD components can work  collaboratively to enhance asset visibility in a manner that provides  accurate, reliable, and timely data to track assets throughout their life  cycles. According to the 2015 Strategy and Office of the Secretary of  Defense (OSD) officials, the Asset Visibility Working Group—comprised of representatives from the components and other government agencies,  as needed—facilitates collaboration across the components, identifies  opportunities for improvement, and monitors the implementation of the  initiatives.
We initiated this review under the authority of the Comptroller General to  address issues of broad interest to Congress. In this report, we address  the extent to which DOD has (1) identified performance measures that  allow it to monitor the progress of selected asset visibility initiatives  identified in its Strategies and (2) addressed our five criteria—leadership  commitment, capacity, corrective action plan, monitoring, and  demonstrated progress—for removing asset visibility from our High Risk  List.
To determine the extent to which DOD has identified performance  measures that allow it to monitor the progress of selected asset visibility  initiatives identified in the 2014 and 2015 Strategies (Strategies), we  reviewed documents such as the Strategies; minutes from the Asset  Visibility Working Group meetings; and documents showing the status of  the implementation, including charts that track the development and  closure of the asset visibility initiatives. Thirty initiatives have been  included in the Strategies, but 3 of these were halted, for a variety of  reasons. To develop a non-generalizable sample from the remaining 27  initiatives, we selected 8—at least one from each of the components—to  review and assess, including analyzing the performance measures  associated with each initiative. The results from this sample cannot be  generalized to the other 19 initiatives. In our selection of initiatives to  review, we used several sampling criteria to ensure that we had a range  of initiatives—from some just beginning to some that had already been  completed. We surveyed program managers and other cognizant officials  (hereafter referred to as component officials) responsible for the asset  visibility initiatives we selected. We included questions in our survey  related to the development and closure of the initiatives and took several  steps to ensure the validity and reliability of the survey instrument. We  also reviewed the Strategies to identify performance measures necessary  to monitor progress of the 8 initiatives we selected. We assessed whether  (1) DOD had followed the guidance set forth in the Strategies and (2) the  measures for the initiatives included selected key attributes of successful  performance measures (for example, are the measures clear, quantifiable  (i.e., have measurable targets and baseline and trend data), objective,  and reliable).
Additionally, we reviewed the after-action reports for all of the initiatives  that had been closed—20 of 27 initiatives, which included 5 of the 8  initiatives we reviewed in detail—by the Asset Visibility Working Group as  of October 31, 2016. We assessed these after-action reports to determine  whether they were completed for the initiative, documented whether  measures were obtained, and identified challenges and lessons learned.  Furthermore, we interviewed component officials and officials at the  Office of the Deputy Assistant Secretary of Defense for Supply Chain  Integration (hereafter referred to as OSD) to clarify survey responses and  to discuss plans to develop the initiatives, including any efforts to monitor  progress and demonstrate results.
To determine whether DOD had addressed the five criteria—leadership  commitment, capacity, corrective action plan, monitoring, and  demonstrated progress—that would have to be met for us to remove  asset visibility from the High Risk List, we reviewed documents such as  the 2014 and 2015 Strategies and charts that track the implementation  and closure of asset visibility initiatives. We included questions in our  survey to collect additional information from officials on their efforts to  address the high-risk criteria. For example, we asked how the component  monitored the implementation of the initiative and whether there had been  any demonstrated progress in addressing the opportunity, deficiency, or  gap in asset visibility capability that the initiative was designed to address.  We evaluated DOD’s actions to improve asset visibility against each of  our five high-risk criteria for removing this dimension from the high-risk  list. Additionally, we interviewed component officials and OSD officials  to clarify their survey responses and to discuss their plans to continue to  make progress in improving asset visibility. We provide additional  information about our scope and methodology in appendix I.
We conducted this performance audit from February 2016 to March 2017  in accordance with generally accepted government auditing standards.  Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: DOD’s supply chain is a global network that provides materiel, services,  and equipment to the joint force. In February 2015, we reported that DOD  had been experiencing weaknesses in the management of its supply  chain, particularly in the following areas: inventory management, materiel  distribution, and asset visibility. Regarding asset visibility, DOD has had  weaknesses in maintaining visibility of supplies, such as problems with  inadequate radio-frequency identification information to track all cargo  movements. Additionally, in February 2015, we reported on progress  DOD had made in addressing weaknesses in its asset visibility, including  developing its 2014 Strategy.
subsections: section_title: Overview of Asset Visibility at DOD
paragraphs: DOD has focused on improving asset visibility since the 1990s, and its  efforts have evolved over time, as shown in figure 1. The 2015 Strategy  states that the department introduced automatic identification technology  capabilities to improve its ability to track assets. Since we added asset  visibility to the high risk list in 2005, we have reported that DOD has made  a great deal of progress in improving asset visibility. The 2014 Strategy  notes that for more than 25 years, the department has been using  technologies, starting with linear bar codes and progressing to a variety of  more advanced technologies, with the goal of improving asset visibility.  Specifically, the Strategies state that, based on lessons learned from  years of war in Iraq and Afghanistan, the department introduced  technology capabilities to improve its ability to track assets as they  progress from posts, camps, and stations. Additionally, the 2015 Strategy  states that DOD has made significant progress toward improving asset  visibility, but opportunities for greater DOD-wide integration still exist.
section_title: DOD Strategies for Improving Asset Visibility
paragraphs: DOD has issued two strategies to guide its efforts in improving asset  visibility:
2014 Strategy: In January 2014, the department issued its Strategy  for Improving DOD Asset Visibility. The 2014 Strategy creates a  framework whereby the components work collaboratively to identify  improvement opportunities and capability gaps and to leverage  technology capabilities, such as radio frequency identification. These  capabilities aid in providing timely, accurate, and actionable  information about the location, quantity, and status of assets. The  2014 Strategy identified 22 initiatives developed by the components  that were intended to improve asset visibility. OSD officials stated that  an initiative is conducted in accordance with component-level policy  and procedures and can either be for a single component or for  potential improvement throughout DOD. According to OSD officials,  DOD components develop asset visibility initiatives, and these  initiatives may be identified by the Asset Visibility Working Group or  by components for inclusion in the Strategies.
2015 Strategy: In October 2015, DOD issued its update to the 2014  Strategy. The 2015 Strategy outlined an additional 8 initiatives  developed by the components to improve asset visibility. According to  OSD officials, they plan to issue an update to the 2015 Strategy, but  the release date for this update has not been determined. These  officials stated that the update to the 2015 Strategy will outline about  10 new initiatives.
section_title: Overview of DOD’s Efforts to Guide and Monitor Asset Visibility Initiatives
paragraphs: As we reported in January 2015, DOD has taken steps to monitor the  asset visibility initiatives. Specifically, DOD has established a structure for  overseeing and coordinating efforts to improve asset visibility. This  structure includes the Asset Visibility Working Group, which according to  the Strategies is responsible for monitoring the execution of the initiatives.  Additionally, the components are designated as the offices of primary  responsibility to ensure the successful execution of their initiatives,  including developing cost estimates and collecting performance data.  Working Group members include representatives from OSD and the  components—Joint Staff, the Defense Logistics Agency, U.S.  Transportation Command, and each of the military services. The  components submit quarterly status reports to the Working Group about  their initiatives—including progress made on implementation milestones,  return on investment, and resources and funding. Additionally, as  documented in the minutes from its May 2016 Asset Visibility Working  Group meeting, DOD uses an electronic repository that includes  information about the initiatives.
The 2015 Strategy describes a process in which the Asset Visibility  Working Group, among other things, reviews and concurs that an  initiative has met its performance objectives. The Asset Visibility  Working Group files an after-action report, which is added to the status  report, for completed initiatives; this after-action report is to include  performance measures used to assess the success of the initiative,  challenges associated with implementing the initiative, and any lessons  learned from the initiative. For example, an after-action report for the U.S.  Transportation Command (U.S. TRANSCOM) active radio frequency  identification (RFID) migration initiative stated that U.S. TRANSCOM had  successfully tracked the use of old and new active RFID tags on military  assets and updated an active RFID infrastructure to accommodate the  new tags.
section_title: Components Have Identified Performance Measures for the Eight Initiatives We Reviewed, but Limitations in the Measures and Related Status Reports Affect DOD’s Ability to Assess the Success of the Initiatives
paragraphs: DOD components have identified performance measures for the 8  initiatives we reviewed, but the measures do not generally include the key  attributes of successful performance measures (i.e., the measures were  not generally clear, quantifiable, objective, and reliable). We also found  that after-action reports for some initiatives did not always include  information on the performance measures and therefore prevent DOD  from effectively evaluating the success of the initiatives in achieving the  goals and objectives described in the Strategies.
subsections: section_title: DOD Components Have Identified Performance Measures for Selected Initiatives
paragraphs: DOD components have identified at least one performance measure for  each of the 8 initiatives we examined. These initiatives are described in  table 1. (For more details on each of the 8 initiatives, see appendix II.)  DOD’s Strategies direct that expected outcomes or key performance  indicators (which we refer to as performance measures) be identified for  assessing the implementation of each initiative. The 2015 Strategy  notes that these performance measures enable groups, such as the  Asset Visibility Working Group and the Supply Chain Executive Steering  Committee—senior-level officials responsible for overseeing asset  visibility improvement efforts—to monitor progress toward the  implementation of an initiative and to monitor the extent to which the  initiative has improved asset visibility in support of the Strategy’s goals  and objectives. For example, one of the performance measures for a U.S.  TRANSCOM initiative on the migration to a new active radio frequency  identification (RFID) tag is to track the use of old and new active RFID  tags on military assets. Additionally, one of the performance measures for  the Defense Logistics Agency’s (DLA) initiative on passive RFID  technology for clothing and textiles is to track the time it takes to issue  new uniforms to military personnel. The 2015 Strategy also notes that  the performance measures are reviewed before an initiative is closed by  the Asset Visibility Working Group.
section_title: Performance Measures for the Selected Initiatives Did Not Generally Include the Key Attributes of Successful Performance Measures
paragraphs: Our prior work on performance measurement has identified several  important attributes that performance measures should include if they are  to be effective in monitoring progress and determining how well programs  are achieving their goals. (See table 2 for a list of selected key  attributes.) Additionally, Standards for Internal Control in the Federal  Government emphasizes using performance measures to assess  performance over time. We have previously reported that by tracking  and developing a performance baseline for all performance measures,  agencies can better evaluate whether they are making progress and their  goals are being achieved. Based on an analysis of the 8 initiatives we  reviewed, we found that these performance measures did not generally  include the key attributes of successful performance measures.  Moreover, DOD’s Strategies lack sufficient direction on how components  are to develop measures for these initiatives that would ensure that the  performance measures developed include the key attributes for  successful measures. This hinders DOD’s ability to ensure that effective  measures are developed which will allow it to monitor the performance of the individual initiatives and whether the initiatives are likely to achieve  the goals and objectives of the Strategies.
We found that some of the performance measures for the 8 initiatives we  reviewed included the key attributes of successful performance  measures, such as linkage to goals and objectives in the Strategies.  However, the measures for most of the initiatives did not have many of  the key attributes of successful performance measures. As shown in table  3, for three initiatives there were no clearly identified performance  measures; for five there were no measurable targets to allow for easier  comparison with the initiatives’ actual performance; for five the measures  were not objective; for five the measures were not reliable; for six there  were no baseline and trend data associated with the measures; and for  three the performance measures were not linked to the goals and  objectives of the Strategies.
A detailed discussion of our assessment of the performance measures for  each key attribute follows:
Measures for 5 of the 8 initiatives partially included the key  attribute of “clarity.” For example, a performance measure for a  Defense Logistics Agency initiative was to reduce the time  required to issue uniforms by improving cycle times and reducing  customer wait time. We identified “to reduce the time required to  issue uniforms” as the name of the measure. However, the  definition we identified for this measure, which is to improve cycle  times and reduce customer wait time, did not include the  methodology for computing the measure. Therefore, for the clarity  attribute, we could not determine if the definition of this measure  was consistent with the methodology used to calculate it. We  reported in September 2015 that if the name and definition of the  performance measure are not consistent with the methodology  used to calculate it, data may be confusing and misleading to the  component.
For 3 of the 8 initiatives the performance measures were not  clearly stated. For example, a performance measure for an Army  initiative was to expand current capabilities by accessing data  through a defense casualty system and integrate reporting and  tracking into one application. We found that there was an overall  description of the initiative, but it did not include a name or  definition for the measure or a methodology for calculating it.
2.  Measurable Target:
Measures for 3 of the 8 initiatives fully included the key attribute of  measurable targets. For example, a performance measure for a  Joint Staff initiative is to have 100 percent visibility of condition  codes for non-munitions inventory.
Measures for 5 of the 8 initiatives did not identify a measurable  target. For example, a performance measure for a Marine Corps  initiative is to increase non-nodal visibility and the delivery status  of materiel in transit within an area of responsibility, but the  component did not provide a quantifiable goal or other measure  that permits expected performance to be compared with actual  results so that actual progress can be assessed.
Measures for 3 of the 8 initiatives partially included the key  attribute of objectivity. For example, the performance measures  for a Navy initiative indicated what is to be observed (timeliness,  accuracy, and completeness), but the measures did not specify  what population and time frames were to be observed.
Measures for 5 of the 8 initiatives did not include the key attribute  of objectivity. For example, the performance measures for an  Army initiative did not indicate what is to be observed, in which  population, and in what time frame.
Measures for 3 of the 8 initiatives partially included the key  attribute of reliability. For example, some of the performance  measures for a Navy initiative included data quality control  processes to verify or validate information such as automated or  manual reviews and the frequency of reviews. However, the Navy  did not specify how often it would perform these reviews.
Measures for 5 of 8 initiatives did not include the key attribute of  reliability. For example, the performance measures for an Army  initiative did not include a name for the measures, definitions for  these measures, or methodologies for calculating them.  Therefore, we could not determine whether the measures would  produce the same results under similar conditions.
5.  Baseline and Trend data:
Measures for 2 of 8 initiatives partially included the key attribute of  baseline and trend data. For example, a Joint Staff initiative  included a baseline (e.g., improve the visibility of condition codes  of non-munitions assets in the Global Combat Support System –  Joint (GCSS-J) from 48 percent to 100 percent), but it did not  include trend data.
Measures for 6 of 8 initiatives did not include the key attribute of  baseline and trend data. For example, the performance measures  for a U.S. TRANSCOM initiative for implementing transportation  tracking numbers did not include baseline and trend data to  identify, monitor, and report changes in performance.
Measures for 5 of 8 initiatives fully included the key attribute of  linkage. For example, the performance measures for the Joint  Staff initiative, intended to maximize the visibility of the condition  codes of non-munitions assets in GCSS-J to support joint logistics  planning, are linked to the 2015 Strategy’s goals of:  improving visibility into customer materiel requirements  and available resources;  o  enhancing visibility of assets in transit, in storage, in  process, and in theater; and  o  enabling an integrated accessible authoritative data set.
Measures for 3 of the 8 initiatives did not include the key attribute  of linkage because they were not aligned with agency-wide goals  and mission and were not clearly communicated throughout the  organization. These initiatives were identified in the 2014 Strategy  and the descriptions of the initiatives did not specify which of the  goals and objectives they were intended to support. We reported  in January 2015 that the 2014 Strategy did not direct that the  performance measures developed for the initiatives link to the  goals or objectives in the 2014 Strategy, and we found that it was  not clear whether the measures linked to the Strategy’s goals and  objectives. Therefore, we recommended that DOD ensure that  the linkage between the performance measures for the individual  initiatives and the goals and objectives outlined in the 2014  Strategy be clear. DOD concurred with our recommendation and  in its 2015 Strategy linked each initiative to the goals and  objectives.
The deficiencies that we identified in the performance measures can be  linked to the fact that the Strategies have not included complete direction  on the key attributes of successful performance measures. The 2014  Strategy provided direction on the types of expected outcomes and key  performance indicators. For example, an expected outcome is to increase  supply chain performance and the key performance indicator is to  improve customer wait time. However, when OSD updated the 2014  Strategy it did not include in the 2015 Strategy an example of the types of  expected outcomes and key performance indicators for components to  use when developing performance measures. The lack of direction on  successful performance measures may have resulted in measures that  lacked key attributes, such as clarity, measurable target, objectivity,  reliability, baseline and trend data, and linkage, as we previously  discussed. While OSD officials stated that they believed the performance  measures for the selected initiatives were sufficient to report on the status  of the initiatives, our review of these measures determined that they could  not be used to effectively assess the performance of the initiatives to  improve asset visibility. Without sufficient direction in subsequent updates  to the Strategy on developing successful performance measures, DOD  has limited assurance that the components are developing measures that  can be used to determine how the department is progressing toward  achieving its goals and objectives related to improving asset visibility.
section_title: After-Action Reports for Initiatives Have Not Always Included Performance Measures to Assess the Initiatives’ Success
paragraphs: As described in the 2015 Strategy, the Asset Visibility Working Group and  the component review the performance of the initiatives during  implementation. As we reported in January 2015, the components report  quarterly to the Asset Visibility Working Group on the status of their  initiatives—including progress made on implementation milestones, return  on investment, and resources and funding. We found that DOD  components had included performance measures in their quarterly status  reports for the 8 initiatives we reviewed.
However, DOD components have not always included performance  measures to assess the success of their initiatives in after-action reports,  which are added to the status report for completed initiatives. To close an  initiative, the components responsible for the initiative request closure  and the Asset Visibility Working Group files an after-action report, which  serves as a closure document and permanent record of an initiative’s  accomplishments. According to the 2015 Strategy, the after-action report  should include information on the objectives met, problems or gaps  resolved, challenges associated with implementing the initiative, any  lessons learned from the initiative, and measures of success obtained.  The Asset Visibility Working Group approves the closure of initiatives  when the components have completed or canceled the initiatives and  updated the status report section called the after-action report. Once an  initiative is closed, according to DOD officials, the Working Group no  longer monitors the initiative, but the components may continue to  monitor it. According to these DOD officials, DOD components may  update information provided to the Asset Visibility Working Group or the  Working Group may request additional information after the initiative is  closed, especially when implementation affects multiple components.
We found that the after-action reports did not always include all of the  information necessary. According to our review of after-action reports, as  of October 2016, the Asset Visibility Working Group had closed 5 of the 8  asset visibility initiatives that we examined. Our review of the after- action reports for the 5 closed initiatives found the following:
Two reports included information on whether the performance  measures—also referred to as measures of success—for the initiative  had been achieved.
Three reports did not follow the format identified in the 2015 Strategy,  and we could not determine whether the intent and outcomes based  on performance measures for the initiative had been achieved.
We also reviewed after-action reports for the remaining 15 initiatives that  were closed and found the following:
Seven reports included information on whether the performance  measures for the initiative had been achieved.
Five reports did not include information on performance measures,  because these measures were not a factor in measuring the success  of the initiative.
One report was not completed by the component.
Two reports did not follow the format identified in the 2015 Strategy,  and we could not determine whether the intent and outcomes based  on performance measures for the initiative had been achieved.
Based on our analysis, it appears that while the Asset Visibility Working  Group closed 20 initiatives, it generally did not have information related to  performance measures to assess the progress of these initiatives when  evaluating and closing them. Specifically, the after-action reports for 11 of  20 initiatives did not include performance measures that showed whether  the initiative had met its intended outcomes in support of the department’s  Strategies. Officials from the Asset Visibility Working Group stated that  they generally relied on the opinion of the component’s subject matter  experts, who are familiar with each initiative’s day-to-day performance, to  assess the progress of these initiatives. While including the input of the  component’s subject matter experts for the initiative in the decision to  close the initiative is important, without incorporating after-action reports  information relating to performance measures into the information  considered by the Asset Visibility Working Group, DOD does not have  assurance that closed initiatives have been fully assessed and whether  they have resulted in achieving the goals and objectives of the Strategies.
section_title: DOD Has Met the Criteria for Leadership Commitment, Capacity, and an Action Plan, but Additional Actions Are Needed to Address Monitoring and Demonstrated Progress
paragraphs: DOD has fully met three of our criteria for removal from the High Risk List  by improving leadership commitment, capacity, and its corrective action  plan, and it has partially met the criteria to monitor the implementation of  the initiatives and demonstrate progress in improving asset visibility.  Table 4 includes a description of the criteria and our assessment of  DOD’s progress in addressing each of them.
subsections: section_title: DOD Has Fully Met the Leadership Commitment, Capacity, and Corrective Action Plan Criteria for Removal from Our High Risk List
paragraphs: DOD Continues to Fully Meet Our High-Risk Criterion for Leadership  Commitment  Our high-risk criterion for leadership commitment calls for leadership  oversight and involvement. DOD has taken steps to address asset  visibility challenges, and we found—as we had in our February 2015 high- risk report—that DOD has fully met this criterion. Senior leaders at the  department have continued to demonstrate commitment to addressing  the department’s asset visibility challenges, as evidenced by the issuance  of DOD’s 2014 and 2015 Strategies. The Office of the Deputy Assistant  Secretary of Defense for Supply Chain Integration provides department- wide oversight for development, coordination, approval, and  implementation of the Strategies and reviews the implementation of the  initiatives. Also, senior leadership commitment is evident in its  involvement in asset visibility improvement efforts, including groups such  as the Supply Chain Executive Steering Committee—a group of senior- level officials responsible for overseeing asset visibility improvement  efforts—and the Asset Visibility Working Group—a group of officials that  includes representatives from the components and other government  agencies, as needed. The Asset Visibility Working Group identifies  opportunities for improvement and monitors the implementation of  initiatives. Sustained leadership commitment will be critical moving  forward, as the department continues to implement its Strategies to  improve asset visibility and associated asset visibility initiatives.
DOD Has Fully Met Our High-Risk Criterion for Capacity  Our high-risk criterion for capacity calls for agencies to demonstrate that  they have the people and other resources needed to resolve risks in the  high-risk area. In our October 2014 management letter to a senior OSD  official and our January 2015 and February 2015 reports, we noted that  resources and investments should be discussed in a comprehensive  strategic plan, to include the costs to execute the plan and the sources  and types of resources and investments—including skills, human capital,  technology, information and other resources—required to meet  established goals and objectives. DOD has demonstrated that it has the  capacity—personnel and resources—to improve asset visibility. For  example, as we previously noted, the department had established the  Asset Visibility Working Group that is responsible for identifying  opportunities for improvement and monitoring the implementation of  initiatives. The Working Group includes representatives from OSD and  the components—Joint Staff, the Defense Logistics Agency, U.S.  Transportation Command, and each of the military services. Furthermore,  DOD’s 2015 Strategy called for the components to consider items such  as manpower, materiel, and sustainment costs when documenting cost  estimates for the initiatives in the Strategy, as we recommended in our  January 2015 and February 2015 reports. For example, DOD identified  and broke down estimated costs of $10 million for implementing an  initiative to track Air Force aircraft and other assets from fiscal years 2015  through 2018 by specifying that $1.2 million was for manpower, $7.4  million for sustainment, and $1.4 million for one-time costs associated  with the consolidation of a server for the initiative. Additionally, DOD  broke down estimated costs of $465,000 for implementing an initiative to  track Marine Corps assets from fiscal years 2013 through 2015 by  specifying $400,000 for manpower and $65,000 for materials.
However, in December 2015 we found that the 2015 Strategy included  three initiatives that did not include cost estimates. To address this issue,  in December 2016, a DOD official provided an abstract from the draft  update to the 2015 Strategy that provides additional direction on how to  explain and document cases where the funding for the initiatives is  embedded within overall program funding. The draft update notes that  there may be instances where asset visibility improvements are  embedded within a larger program, making it impossible or cost  prohibitive to isolate the cost associated with specific asset visibility  improvements. In these cases, the document outlining the initiatives will  indicate that cost information is not available and why. However, if at  some point during implementation some or all costs are identified,  information about the initiative will be updated. According to OSD officials,  DOD plans to issue the update to the 2015 Strategy, but a release date  has not been determined.
DOD Has Fully Met Our High-Risk Criterion for a Corrective Action  Plan  Our high-risk criterion for a corrective action plan calls for agencies to  define the root causes of problems and related solutions and to include  steps necessary to implement the solutions. The Fiscal Year 2014  National Defense Authorization Act (NDAA) required DOD to submit to  Congress a comprehensive strategy and implementation plans for  improving asset tracking and in-transit visibility. The Fiscal Year 2014  NDAA, among other things, called for DOD to include in its strategy and  plans elements such as goals and objectives for implementing the  strategy. The Fiscal Year NDAA also included a provision that we assess  the extent to which DOD’s strategy and accompanying implementation  plans include the statutory elements. In January 2014, DOD issued its  Strategy for Improving DOD Asset Visibility and accompanying  implementation plans that outline initiatives intended to improve asset  visibility. DOD updated its 2014 Strategy and plans in October 2015. The  2014 and 2015 Strategies define the root causes of problems associated  with asset visibility and related solutions (i.e., the initiatives). In our  October 2014 management letter to a senior OSD official and our January  and February 2015 reports, we found that while the 2014 Strategy and  accompanying plans serve as a corrective action plan, there was not a  clear link between the initiatives and the Strategy’s goals and objectives.  We recommended that DOD clearly specify the linkage between the goals  and objectives in the Strategy and the initiatives intended to implement  the Strategy. DOD implemented our recommendation in its 2015  Strategy, which includes matrixes that link each of DOD’s ongoing  initiatives intended to implement the Strategy to the Strategy’s  overarching goals and objectives. DOD also added 8 initiatives to its 2015  Strategy and linked each of them to the Strategy’s overarching goals and  objectives.
section_title: DOD Has Partially Met the Monitoring and Demonstrated Progress Criteria for Removal from the High Risk List
paragraphs: DOD Has Taken Steps to Monitor the Status of Initiatives, but Its  Performance Measures Could Not Always Be Used to Track  Progress  Our high-risk criterion on monitoring calls for agencies to institute a  program to monitor and independently validate the effectiveness and  sustainability of corrective measures, for example, through performance  measures. DOD has taken steps to monitor the status of asset visibility  initiatives, but we found that it has only partially met our high-risk criterion  for monitoring. In our February 2015 High-Risk update, we referred to a  2013 report in which we had found that DOD lacked a formal, central  mechanism to monitor the status of improvements or fully track the  resources allocated to them. We also reported that while DOD’s draft  2014 Strategy included overarching goals and objectives that addressed  the overall results desired from implementation of the Strategy, it only  partially included performance measures, which are necessary to enable  monitoring of progress.
Since February 2015, DOD has taken some steps to improve its  monitoring of its improvement efforts. As noted in the 2015 Strategy, DOD  has described and implemented a process that tasks the Asset Visibility  Working Group to review the performance of the component’s initiatives  during implementation on a quarterly basis, among other things. The  Working Group uses status reports from the DOD components that  include information on resources, funding, and progress made toward  implementation milestones. DOD also identified performance measures  for its asset visibility initiatives. However, as previously discussed, the  measures for the 8 initiatives we reviewed were not generally clear,  quantifiable (i.e., lacked measurable targets and baseline and trend data),  objective, and reliable. Measures that are clear, quantifiable, objective,  and reliable can help managers better monitor progress, including  determining how well they are achieving their goals and identifying areas  for improvement, if needed. In December 2016, a DOD official provided  an abstract from the draft update to the 2015 Strategy that noted that  detailed metrics data will be collected and reviewed at the level  appropriate for the initiative. High-level summary metrics information will  be provided to the Working Group in updates to the plan outlining the  initiatives. The extent to which this planned change will affect the  development of clear, quantifiable, objective, and reliable performance  measures remains to be determined.
Additionally, as discussed previously, while the Asset Visibility Working  Group has closed 20 initiatives, it generally did not have information  related to performance measures to assess the progress of these  initiatives. Specifically, after-action reports from 11 of 20 initiatives— which are added to the status reports for completed initiatives—did not  include performance measures that showed whether the initiative had met  its intended outcomes in support of the department’s Strategies. Without  improved performance measures and information to support that progress  has been made, DOD may not be able to monitor asset visibility  initiatives.
DOD Has Demonstrated Some Progress but Cannot Demonstrate  that Its Initiatives Have Resulted in Measurable Outcomes and  Improvements for Asset Visibility  Our high-risk criterion for demonstrated progress calls for agencies to  demonstrate progress in implementing corrective measures and resolving  the high-risk area. DOD has made progress by developing and  implementing its Strategies for improving asset visibility. In our October  2014 management letter to a senior OSD official and our January and  February 2015 reports, we noted that in order to demonstrate progress in  having implemented corrective measures, DOD should continue the  implementation of the initiatives identified in the Strategy, refining them  over time as appropriate. DOD reports that it has closed or will no  longer monitor the status of 20 of the 27 initiatives and continues to  monitor the remaining 7 initiatives. Additionally, in October 2016, DOD  officials stated that they plan to add about 10 new initiatives in the update  to the 2015 Strategy. For example, the U.S. Transportation Command’s  new initiative, Military Service Air Manifesting Capability, is expected to  promote timely, accurate, and complete in-transit visibility and effective  knowledge sharing to enhance understanding of the operational  environment. OSD officials have not yet determined a date for the release  of the update to the 2015 Strategy.
As discussed previously, we found that DOD cannot use the performance  measures associated with the initiatives to demonstrate progress,  because the measures are not generally clear, quantifiable (i.e., lack  measurable targets and baseline and trend data), objective, and reliable.  Additionally, we found that DOD has not taken steps to consistently  incorporate information on an initiative’s performance measures in  closure reports, such as after-action reports, in order to demonstrate the  extent to which progress has been made toward achieving the intended  outcomes of the individual initiatives and the overall goals and objectives  of the Strategies. Without clear, quantifiable, objective, and reliable  performance measures and information to support that progress has been  made, DOD may not be able to demonstrate that implementation of these  initiatives has resulted in measurable outcomes and progress toward  achieving the goals and objectives in the Strategies. Also, DOD will be  limited in its ability to demonstrate sustained progress in implementing  corrective actions and resolving the high-risk area.
section_title: Conclusions
paragraphs: DOD has taken some positive steps to address weaknesses in asset  visibility. Long-standing management weaknesses related to DOD’s asset  visibility functions hinder the department’s ability to provide spare parts,  food, fuel, and other critical supplies in support of U.S. military forces. We  previously reported on several actions that we believe DOD should take  in order to mitigate or resolve long-standing weaknesses in asset visibility  and meet the criteria for removing asset visibility from the High Risk List.  We believe that DOD has taken the actions necessary to meet the  capacity and action plan criteria by providing additional direction to the  components on formulating cost estimates for the asset visibility  initiatives. Additionally, DOD linked the 2015 Strategy’s goals and  objectives with the specific initiatives intended to implement the Strategy.  However, DOD’s efforts to monitor initiatives show that the performance  measures DOD components currently use to assess these initiatives lack  some of the key attributes of successful performance measures that we  have identified. To the extent that these measures lack the key attributes  of successful performance measures, they limit DOD’s ability to  effectively monitor the implementation of the initiatives and assess the  effect of the initiatives on the overall objectives and goals of the  Strategies. Developing clear, quantifiable, objective, and reliable  performance measures can help DOD better assess department-wide  progress against the Strategies’ goals and clarify what additional steps  need to be taken to enable decision makers to exercise effective  oversight.
An important step in determining what effect, if any, the asset visibility  initiatives are having on the achievement of the Strategies’ goals and  objectives will be to develop sound performance measures and  incorporate information about these measures into the after-action reports  when evaluating and closing initiatives. Until DOD components  demonstrate that implementation of the initiatives will result in measurable  outcomes and progress toward achieving the goals and objectives of the  Strategies, DOD may be limited in its ability to demonstrate progress in  implementing corrective actions and resolving the high-risk area. Once  these actions are taken, DOD will be better positioned to demonstrate the  sustainable progress needed in its approach to meeting the criteria for  removing asset visibility from our High Risk List.
section_title: Recommendations for Executive Action
paragraphs: We are making two recommendations to help improve DOD’s asset  visibility.
We recommend that the Secretary of Defense direct the Assistant  Secretary of Defense for Logistics and Materiel Readiness, in  collaboration with the Director, Defense Logistics Agency; the Secretaries  of the Army, Navy, and Air Force; the Commandant of the Marine Corps;  the Commander of the United States Transportation Command; and the  Chairman of the Joint Chiefs of Staff, to: use the key attributes of successful performance measures—including  clarity, measurable target, objectivity, reliability, baseline and trend  data, and linkage—in refining the performance measures in  subsequent updates to the Strategy to improve DOD’s efforts to  monitor asset visibility initiatives; and  incorporate into after-action reports information relating to  performance measures for the asset visibility initiatives when  evaluating and closing these initiatives to ensure that implemented  initiatives will achieve the goals and objectives in the Strategies.
section_title: Agency Comments and Our Evaluation
paragraphs: In its written comments on a draft of this report, DOD partially concurred  with our two recommendations. DOD’s comments are summarized below  and reprinted in appendix IV.
DOD partially concurred with our first recommendation that it use the key  attributes of successful performance measures—including clarity,  measurable target, objectivity, reliability, baseline and trend data, and  linkage—in refining the performance measures in subsequent updates to  the Strategy to improve DOD’s efforts to monitor asset visibility initiatives.  DOD stated that it recognizes the need for performance measures to  ensure the success of an asset visibility improvement effort but noted that  the level of complexity and granularity of the metrics we suggest may not  be suitable for all initiatives. DOD also stated that the purpose of the  Strategy is to create a framework whereby the components can work  collaboratively to coordinate and integrate department-wide efforts to  improve asset visibility, not to provide complete direction on how to  define, implement, and oversee these initiatives. Additionally, DOD stated  that the next edition of the Strategy will encourage the adoption of our six  key attributes for asset visibility initiatives to the extent appropriate, but  will not mandate their use. As discussed in our report, use of the key  attributes in measuring the performance of asset visibility initiatives would  help DOD to better assess department-wide progress against the goals in  its Strategy and clarify what additional steps need to be taken to enable  decision makers to exercise effective oversight. Encouraging adoption of  the key attributes, as DOD plans to do, is a positive step, but we continue  to believe that DOD needs to use these key attributes to refine its  performance measures to monitor the initiatives in the future.
DOD partially concurred with our second recommendation that it  incorporate into after-action reports information relating to performance  measures for the asset visibility initiatives when evaluating and closing  these initiatives to ensure that implemented initiatives will achieve the  goals and objectives in the Strategies. DOD stated that it is important to  capture and review performance data prior to a component closing an  asset visibility initiative, but that the Strategy after-action report is not  intended to be used to evaluate the success of an asset visibility initiative  or to determine if an initiative has met its intended objectives. According  to DOD, documentation and information to support the evaluation of  initiatives is defined by and executed in accordance with component-level  policy and procedures. DOD agreed to update its Strategy to clarify the  purpose and use of the after-action reports and to ensure that the  Strategy specifies roles and responsibilities for evaluating and closing  initiatives. DOD’s response, however, did not state whether and how  these updates to the Strategy would result in more consistent  incorporation of information relating to performance measures when  closing initiatives in the future. As we noted previously in this report,  according to the 2015 Strategy, the after-action report for closed  initiatives should include information on the objectives met, problems or  gaps resolved, and measures of success obtained. We believe our  recommendation is consistent with this guidance. Without incorporating  this information, DOD does not have assurance that closed initiatives  have been fully assessed and have resulted in achieving the goals and  objectives of the Strategies. Therefore, we continue to believe that full  implementation of our recommendation is needed.
We are sending copies of this report to appropriate congressional  committees; the Secretary of Defense; the Secretaries of the Army, Navy,  and Air Force, and the Commandant of the Marine Corps; the Director of  Defense Logistics Agency; the Chairman of the Joint Chiefs of Staff; the  Commander of the United States Transportation Command; and other  interested parties. In addition, the report is available at no charge on the  GAO website at http://www.gao.gov.
If you or your staff have any questions about this report, please contact  me at (202) 512-5257 or merrittz@gao.gov. Contact points for our Offices  of Congressional Relations and Public Affairs may be found on the last  page of this report. GAO staff who made contributions to this report are  listed in appendix V.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the extent to which DOD identified performance measures  that allow it to monitor the progress of selected asset visibility initiatives  identified in DOD’s 2014 and 2015 Strategy for Improving DOD Asset  Visibility (Strategies), we reviewed documents such as the 2014 Strategy  and its subsequent update in October 2015 (2015 Strategy); minutes from  the Asset Visibility Working Group meetings; and documents showing the  status of the implementation, including charts that track the development  and closure of the asset visibility initiatives. Thirty initiatives have been  included in the 2014 and 2015 Strategies, but 3 of these were halted, for  a variety of reasons. From the remaining 27 initiatives, we selected a  non-generalizable sample of 8 initiatives. We selected at least one from  each of the components to review and assess, including analyzing the  performance measures associated with each initiative. In our selection of  8 initiatives to review, we also considered the stage of implementation of  the initiative, to ensure that our review encompassed initiatives at  different stages, from some that were just beginning to some that had  already been completed. Specifically, we made selections based on the  status of the initiatives as of December 2015 to include the earliest  completion dates by component. In order to cover a range of initiatives— from some just beginning to some already completed—we selected for  review 3 initiatives from the 2014 Strategy that had been closed, 2  ongoing initiatives that had been included in both Strategies, and 3 new  initiatives that were included for the first time in the 2015 Strategy. The  results from this sample cannot be generalized to the other 19 initiatives.  We did not assess the initiatives to determine if they (1) met milestones,  (2) lacked resources, or (3) had performance issues. Instead we  assessed the initiatives to determine what progress DOD had made  toward meeting the criteria for removing an area from our High Risk list.
We surveyed program managers and other cognizant officials (hereafter  referred to as component officials) responsible for the respective asset  visibility initiatives we selected. We included questions in our survey  related to the development and closure of the initiatives and took several  steps to ensure the validity and reliability of the survey instrument. We  also reviewed the Strategies to identify performance measures necessary  to monitor the progress of the 8 initiatives we had selected. Two analysts  independently assessed whether (1) DOD had followed the guidance set  forth in the Strategies and (2) the measures for the initiatives included  selected key attributes of successful performance measures (for example,  are the measures clear, quantifiable —i.e., have measurable targets and  baseline and trend data—objective, and reliable); any initial  disagreements in assessments were resolved through discussion. We  assessed these measures against 6 of 10 selected key attributes for  successful performance measures—clarity, measurable target, objectivity,  reliability, baseline and trend data, and linkage—identified in our prior  work that we identified as relevant to the sample of initiatives we were  examining. The remaining 4 attributes—government-wide priorities, core  program activities, limited overlap, and balance—are used to assess  agency-wide performance and are not applicable to our analysis, because  we did not assess agency-wide initiatives. Because we had selected a  subset of the component-level initiatives to review, these attributes did not  apply. If all of the performance measures for an initiative met the  definition of the relevant key attribute, we rated the initiative as having  “fully included” the attribute. On the other hand, if none of the measures  met the definition of the relevant key attribute, we rated the initiative as  having “not included” the attribute. If some, but not all, of the measure  met the definition of the relevant key attribute, then we rated the initiative  as having “partially included” the attribute.
We also selected sites to observe demonstrations of initiatives that were  intended to show how they have achieved progress in improving asset  visibility. We selected these demonstrations based on the location of the  initiative, the responsible component, and the scope of the initiative.  Additionally, we reviewed the after-action reports for all of the initiatives  that had been closed—20 of 27 initiatives, including 5 of the 8 initiatives  we reviewed in detail—by the Asset Visibility Working Group, as of  October 31, 2016. We performed a content analysis in which we reviewed  each of these after-action reports to determine whether it was completed  for the initiative, documented whether measures were obtained, and  identified challenges and lessons learned. One analyst conducted this  analysis, coding the information and entering it into a spreadsheet; a  second analyst checked the first analyst’s analysis for accuracy. Any  initial disagreements in the coding were discussed and reconciled by the  analysts. The analysts then tallied the responses to determine the extent  to which the information was identified in the after-action reports. We also  interviewed component officials and officials at the Office of the Deputy  Assistant Secretary of Defense for Supply Chain Integration (hereafter  referred to as OSD) to clarify survey responses and to discuss plans to  develop the initiatives, including any efforts to monitor progress and  demonstrate results.
To determine whether DOD had addressed the five criteria—leadership  commitment, capacity, corrective action plan, monitoring, and  demonstrated progress—that would have to be met for us to remove  asset visibility from our High Risk List, we reviewed documents such as  DOD’s 2014 and 2015 Strategies and charts that track the  implementation and closure of asset visibility initiatives. We included  questions in our survey to collect additional information from officials on  their efforts to address the high-risk criteria. For example, we asked how  the component monitors the implementation of the initiative and whether  there has been any demonstrated progress in addressing the opportunity,  deficiency, or gap in asset visibility capability that the initiative was  designed to address. One analyst evaluated DOD’s actions to improve  asset visibility against each of our five criteria for removing an area from  the High Risk list. A different analyst checked the first analyst’s analysis  for accuracy. Any initial disagreements were discussed and reconciled  by the analysts. We assessed DOD’s effort to meet each of the high-risk  criteria as “not met” (i.e., none of the aspects of the criterion were  addressed), “partially met” (i.e., some, but not all, aspects of the criterion  were addressed), or “fully met” (i.e., all parts of the criterion were fully  addressed). We shared with DOD officials our preliminary assessment of  asset visibility relative to each of the criteria. To help ensure that our  evaluation of improvements made relative to the high-risk criteria were  consistent with our prior evaluations of Supply Chain Management and  other issue areas, we reviewed our prior High Risk reports to gain insight  into what actions agencies had taken to address the issues identified in  these past reports. Additionally, we interviewed component officials and  OSD officials to clarify their survey responses and to discuss plans to  continue to make progress in improving asset visibility.
We met with officials from the following DOD components during our  review:
Office of the Secretary of Defense
Department of the Army
United States Marine Corps
Department of the Air Force  We surveyed component officials responsible for the asset visibility  initiatives we reviewed. We included questions in our survey related to  our high-risk criteria. As part of the survey development, we conducted an  expert review and pre-tested the draft survey. We submitted the  questionnaire for review by an independent GAO survey specialist and an  asset visibility subject matter expert from OSD. The expert review phase  was intended to ensure that content necessary to understand the  questions was included and that technical information included in the survey was correct. To minimize errors that might occur from respondents  interpreting our questions differently than we intended, we pre-tested our  questionnaire with component officials and other cognizant officials for 4  of the initiatives. During the pre-tests, conducted by telephone, we asked  the DOD officials to read the instructions and each question aloud and to  tell us how they interpreted the question. We then discussed the  instructions and questions with them to identify any problems and  potential solutions by determining whether (1) the instructions and  questions were clear and unambiguous, (2) the terms we used were  accurate, (3) the questionnaire was unbiased, and (4) the questionnaire  did not place an undue burden on the officials completing it. We noted  any potential problems and modified the questionnaire based on  feedback from the expert reviewers and the pre-tests, as appropriate.
We sent an email to each selected program office beginning on June 16,  2016, notifying them of the topics of our survey and when we expected to  send the survey. We then sent the self-administered questionnaire and a  cover email to the asset visibility program officials on June 20, 2016, and  asked them to fill in the questionnaire and email it back to us by July 6,  2016. We received 8 completed questionnaires, for an overall response  rate of 100 percent. We also collected data—such as the number of RFID  tags and number of inventory amounts for clothing and textiles—from a  sample of initiatives. The practical difficulties of conducting any survey  may introduce errors, commonly referred to as non-sampling errors. For  example, differences in how a particular question is interpreted, the  sources of information available to respondents, how the responses are  processed and analyzed, or the types of people who do not respond can  influence the accuracy of the survey results. We took steps, as described  above, in the development of the survey, the data collection, and the data  analysis to minimize these non-sampling errors and help ensure the  accuracy of the answers that we obtained. Data were electronically  extracted from the questionnaires into a comma-delimited file that was  then imported into a statistical program for analysis. We examined the  survey results and performed computer analyses to identify  inconsistencies and other indications of error, and we addressed such  issues as necessary. Our survey specialist conducted quantitative data  analyses using statistical software, and our staff conducted a review of  open-ended responses with subject matter expertise. A data analyst  conducted an independent check of the statistical computer programs for  accuracy.
We conducted this performance audit from February 2016 to March 2017  in accordance with generally accepted government auditing standards.
Those standards require that we plan and perform the audit to obtain  sufficient, appropriate evidence to provide a reasonable basis for our  findings and conclusions based on our audit objectives. We believe that  the evidence obtained provides a reasonable basis for our findings and  conclusions based on our audit objectives.
section_title: Appendix II: Selected DOD Initiatives Identified in the 2014 and 2015 Strategies
paragraphs: This appendix provides an overview of the non-generalizable sample of  initiatives that we reviewed. These initiatives are intended to improve  asset visibility as part of the Department of Defense’s (DOD) 2014  Strategy for Improving DOD Asset Visibility (2014 Strategy) and its  subsequent update in October 2015 (2015 Strategy). The process by  which we selected these initiatives for this review is described in appendix  I. The initiatives are shown in table 5.
section_title: Appendix III: Criteria for Removal from Our High Risk List and GAO’s Prior Work on Asset Visibility
paragraphs: In 1990, we began a program to report on government operations that we  identified as “high risk,” and we added the Department of Defense’s  (DOD) supply chain management area to our High Risk List. Our high-risk  program has served to identify and help resolve serious weaknesses in  areas that involve substantial resources and provide critical services to  the public. Our experience with the high-risk series over the past two  decades has shown that the key elements needed to make progress in  high-risk areas are congressional action, high-level administrative  initiatives, and agencies’ efforts grounded in the five criteria we  established for removing an area from the high-risk list. These five  criteria form a road map for efforts to improve and ultimately address  high-risk issues. Addressing some of the criteria leads to progress, while  satisfying all of the criteria is central to removing an area from the list.  These criteria call for agencies to show the following:  1.  Leadership Commitment—a strong commitment and top leadership  support.  2.  Capacity—the capacity (i.e., the people and other resources) to  resolve the risk(s).  3.  Corrective Action Plan—a plan that defines the root causes and  solutions and provides for substantially completing corrective  measures, including steps necessary to implement the solutions we  recommended.  4.  Monitoring—a program instituted to monitor and independently  validate the effectiveness and sustainability of corrective measures.  5.  Demonstrated Progress—the ability to demonstrate progress in  implementing corrective measures and resolving the high-risk area.
We have reported on various aspects of DOD’s supply chain, including  asset visibility, and noted that DOD has taken several actions to improve  asset visibility. We also noted a number of recommendations, actions,  and outcomes needed to improve asset visibility, as shown in table 6.  Specifically, in an October 2014 management letter to a senior Office of  the Secretary of Defense (OSD) official, we reported on 7 actions and  outcomes across the 5 criteria that we believed DOD should take to  address long-standing weaknesses in asset visibility. Most recently, in our  January 2015 report and February 2015 High Risk update, we reported  on progress that DOD has made in addressing weaknesses in its asset  visibility, including developing its 2014 Strategy for Improving DOD Asset  Visibility, and we made a number of recommendations.
section_title: Appendix IV: Comments from the Department of Defense
section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Carleen C. Bennett, Assistant  Director; Mary Jo LaCasse; Joanne Landesman; Amie Lesser; Felicia  Lopez; Mike Silver; John E. Trubey; Angela Watson; and John Yee made  key contributions to this report.
section_title: Related GAO Products
paragraphs: High-Risk Series: Progress on Many High-Risk Areas, While Substantial  Efforts Needed on Others. GAO-17-317. Washington, D.C.: February 15,  2017.
High-Risk Series: Key Actions to Make Progress Addressing High-Risk  Issues. GAO-16-480R. Washington, D.C.: April 25, 2016.
Defense Logistics: DOD Has Addressed Most Reporting Requirements  and Continues to Refine its Asset Visibility Strategy. GAO-16-88.  Washington, D.C.: December 22, 2015.
High-Risk Series: An Update. GAO-15-290. Washington, D.C.: February  11, 2015.
Defense Logistics: DOD Has a Strategy and Has Taken Steps to Improve  its Asset Visibility, But Further Actions are Needed. GAO-15-148.  Washington, D.C.: January, 27, 2015.
Defense Logistics: A Completed Comprehensive Strategy is Needed to  Guide DOD’s In-Transit Visibility Efforts. GAO-13-201. Washington, D.C.:  February 28, 2013.
High-Risk Series: An Update: GAO-13-283. Washington, D.C.: February  14, 2013.
Defense Logistics: Improvements Needed to Enhance DOD’s  Management Approach and Implementation of Item Unique Identification  Technology. GAO-12-482. Washington, D.C.: May 3, 2012.
Defense Logistics: DOD Needs to Take Additional Actions to Address  Challenges in Supply Chain Management. GAO-11-569. Washington,  D.C.: July 28, 2011.
High-Risk Series: An Update. GAO-11-278. Washington, D.C.: February  16, 2011.
DOD’s High-Risk Areas: Observations on DOD’s Progress and  Challenges in Strategic Planning for Supply Chain Management.  GAO-10-929T. Washington, D.C.: July 27, 2010.