Defense Logistics: Improved Performance Measures and Information Needed for Assessing Asset Visibility Initiatives
Released: 2017-03-16T00:00:00

Letter

Background

DOD’s supply chain is a global network that provides materiel, services, and equipment to the joint force. In February 2015, we reported that DOD had been experiencing weaknesses in the management of its supply chain, particularly in the following areas: inventory management, materiel distribution, and asset visibility. Regarding asset visibility, DOD has had weaknesses in maintaining visibility of supplies, such as problems with inadequate radio-frequency identification information to track all cargo movements. Additionally, in February 2015, we reported on progress DOD had made in addressing weaknesses in its asset visibility, including developing its 2014 Strategy.

Overview of Asset Visibility at DOD

DOD has focused on improving asset visibility since the 1990s, and its efforts have evolved over time, as shown in figure 1. The 2015 Strategy states that the department introduced automatic identification technology capabilities to improve its ability to track assets. Since we added asset visibility to the high risk list in 2005, we have reported that DOD has made a great deal of progress in improving asset visibility. The 2014 Strategy notes that for more than 25 years, the department has been using technologies, starting with linear bar codes and progressing to a variety of more advanced technologies, with the goal of improving asset visibility. Specifically, the Strategies state that, based on lessons learned from years of war in Iraq and Afghanistan, the department introduced technology capabilities to improve its ability to track assets as they progress from posts, camps, and stations. Additionally, the 2015 Strategy states that DOD has made significant progress toward improving asset visibility, but opportunities for greater DOD-wide integration still exist.

DOD Strategies for Improving Asset Visibility

DOD has issued two strategies to guide its efforts in improving asset visibility:

2014 Strategy: In January 2014, the department issued its Strategy for Improving DOD Asset Visibility. The 2014 Strategy creates a framework whereby the components work collaboratively to identify improvement opportunities and capability gaps and to leverage technology capabilities, such as radio frequency identification. These capabilities aid in providing timely, accurate, and actionable information about the location, quantity, and status of assets. The 2014 Strategy identified 22 initiatives developed by the components that were intended to improve asset visibility. OSD officials stated that an initiative is conducted in accordance with component-level policy and procedures and can either be for a single component or for potential improvement throughout DOD. According to OSD officials, DOD components develop asset visibility initiatives, and these initiatives may be identified by the Asset Visibility Working Group or by components for inclusion in the Strategies.

2015 Strategy: In October 2015, DOD issued its update to the 2014 Strategy. The 2015 Strategy outlined an additional 8 initiatives developed by the components to improve asset visibility. According to OSD officials, they plan to issue an update to the 2015 Strategy, but the release date for this update has not been determined. These officials stated that the update to the 2015 Strategy will outline about 10 new initiatives.

Overview of DOD’s Efforts to Guide and Monitor Asset Visibility Initiatives

As we reported in January 2015, DOD has taken steps to monitor the asset visibility initiatives. Specifically, DOD has established a structure for overseeing and coordinating efforts to improve asset visibility. This structure includes the Asset Visibility Working Group, which according to the Strategies is responsible for monitoring the execution of the initiatives. Additionally, the components are designated as the offices of primary responsibility to ensure the successful execution of their initiatives, including developing cost estimates and collecting performance data. Working Group members include representatives from OSD and the components—Joint Staff, the Defense Logistics Agency, U.S. Transportation Command, and each of the military services. The components submit quarterly status reports to the Working Group about their initiatives—including progress made on implementation milestones, return on investment, and resources and funding. Additionally, as documented in the minutes from its May 2016 Asset Visibility Working Group meeting, DOD uses an electronic repository that includes information about the initiatives.

The 2015 Strategy describes a process in which the Asset Visibility Working Group, among other things, reviews and concurs that an initiative has met its performance objectives. The Asset Visibility Working Group files an after-action report, which is added to the status report, for completed initiatives; this after-action report is to include performance measures used to assess the success of the initiative, challenges associated with implementing the initiative, and any lessons learned from the initiative. For example, an after-action report for the U.S. Transportation Command (U.S. TRANSCOM) active radio frequency identification (RFID) migration initiative stated that U.S. TRANSCOM had successfully tracked the use of old and new active RFID tags on military assets and updated an active RFID infrastructure to accommodate the new tags.

Components Have Identified Performance Measures for the Eight Initiatives We Reviewed, but Limitations in the Measures and Related Status Reports Affect DOD’s Ability to Assess the Success of the Initiatives

DOD components have identified performance measures for the 8 initiatives we reviewed, but the measures do not generally include the key attributes of successful performance measures (i.e., the measures were not generally clear, quantifiable, objective, and reliable). We also found that after-action reports for some initiatives did not always include information on the performance measures and therefore prevent DOD from effectively evaluating the success of the initiatives in achieving the goals and objectives described in the Strategies.

DOD Components Have Identified Performance Measures for Selected Initiatives

DOD components have identified at least one performance measure for each of the 8 initiatives we examined. These initiatives are described in table 1. (For more details on each of the 8 initiatives, see appendix II.) DOD’s Strategies direct that expected outcomes or key performance indicators (which we refer to as performance measures) be identified for assessing the implementation of each initiative. The 2015 Strategy notes that these performance measures enable groups, such as the Asset Visibility Working Group and the Supply Chain Executive Steering Committee—senior-level officials responsible for overseeing asset visibility improvement efforts—to monitor progress toward the implementation of an initiative and to monitor the extent to which the initiative has improved asset visibility in support of the Strategy’s goals and objectives. For example, one of the performance measures for a U.S. TRANSCOM initiative on the migration to a new active radio frequency identification (RFID) tag is to track the use of old and new active RFID tags on military assets. Additionally, one of the performance measures for the Defense Logistics Agency’s (DLA) initiative on passive RFID technology for clothing and textiles is to track the time it takes to issue new uniforms to military personnel. The 2015 Strategy also notes that the performance measures are reviewed before an initiative is closed by the Asset Visibility Working Group.

Performance Measures for the Selected Initiatives Did Not Generally Include the Key Attributes of Successful Performance Measures

Our prior work on performance measurement has identified several important attributes that performance measures should include if they are to be effective in monitoring progress and determining how well programs are achieving their goals. (See table 2 for a list of selected key attributes.) Additionally, Standards for Internal Control in the Federal Government emphasizes using performance measures to assess performance over time. We have previously reported that by tracking and developing a performance baseline for all performance measures, agencies can better evaluate whether they are making progress and their goals are being achieved. Based on an analysis of the 8 initiatives we reviewed, we found that these performance measures did not generally include the key attributes of successful performance measures. Moreover, DOD’s Strategies lack sufficient direction on how components are to develop measures for these initiatives that would ensure that the performance measures developed include the key attributes for successful measures. This hinders DOD’s ability to ensure that effective measures are developed which will allow it to monitor the performance of the individual initiatives and whether the initiatives are likely to achieve the goals and objectives of the Strategies.

We found that some of the performance measures for the 8 initiatives we reviewed included the key attributes of successful performance measures, such as linkage to goals and objectives in the Strategies. However, the measures for most of the initiatives did not have many of the key attributes of successful performance measures. As shown in table 3, for three initiatives there were no clearly identified performance measures; for five there were no measurable targets to allow for easier comparison with the initiatives’ actual performance; for five the measures were not objective; for five the measures were not reliable; for six there were no baseline and trend data associated with the measures; and for three the performance measures were not linked to the goals and objectives of the Strategies.

A detailed discussion of our assessment of the performance measures for each key attribute follows:

Measures for 5 of the 8 initiatives partially included the key attribute of “clarity.” For example, a performance measure for a Defense Logistics Agency initiative was to reduce the time required to issue uniforms by improving cycle times and reducing customer wait time. We identified “to reduce the time required to issue uniforms” as the name of the measure. However, the definition we identified for this measure, which is to improve cycle times and reduce customer wait time, did not include the methodology for computing the measure. Therefore, for the clarity attribute, we could not determine if the definition of this measure was consistent with the methodology used to calculate it. We reported in September 2015 that if the name and definition of the performance measure are not consistent with the methodology used to calculate it, data may be confusing and misleading to the component.

For 3 of the 8 initiatives the performance measures were not clearly stated. For example, a performance measure for an Army initiative was to expand current capabilities by accessing data through a defense casualty system and integrate reporting and tracking into one application. We found that there was an overall description of the initiative, but it did not include a name or definition for the measure or a methodology for calculating it.

2. Measurable Target:

Measures for 3 of the 8 initiatives fully included the key attribute of measurable targets. For example, a performance measure for a Joint Staff initiative is to have 100 percent visibility of condition codes for non-munitions inventory.

Measures for 5 of the 8 initiatives did not identify a measurable target. For example, a performance measure for a Marine Corps initiative is to increase non-nodal visibility and the delivery status of materiel in transit within an area of responsibility, but the component did not provide a quantifiable goal or other measure that permits expected performance to be compared with actual results so that actual progress can be assessed.

Measures for 3 of the 8 initiatives partially included the key attribute of objectivity. For example, the performance measures for a Navy initiative indicated what is to be observed (timeliness, accuracy, and completeness), but the measures did not specify what population and time frames were to be observed.

Measures for 5 of the 8 initiatives did not include the key attribute of objectivity. For example, the performance measures for an Army initiative did not indicate what is to be observed, in which population, and in what time frame.

Measures for 3 of the 8 initiatives partially included the key attribute of reliability. For example, some of the performance measures for a Navy initiative included data quality control processes to verify or validate information such as automated or manual reviews and the frequency of reviews. However, the Navy did not specify how often it would perform these reviews.

Measures for 5 of 8 initiatives did not include the key attribute of reliability. For example, the performance measures for an Army initiative did not include a name for the measures, definitions for these measures, or methodologies for calculating them. Therefore, we could not determine whether the measures would produce the same results under similar conditions.

5. Baseline and Trend data:

Measures for 2 of 8 initiatives partially included the key attribute of baseline and trend data. For example, a Joint Staff initiative included a baseline (e.g., improve the visibility of condition codes of non-munitions assets in the Global Combat Support System – Joint (GCSS-J) from 48 percent to 100 percent), but it did not include trend data.

Measures for 6 of 8 initiatives did not include the key attribute of baseline and trend data. For example, the performance measures for a U.S. TRANSCOM initiative for implementing transportation tracking numbers did not include baseline and trend data to identify, monitor, and report changes in performance.

Measures for 5 of 8 initiatives fully included the key attribute of linkage. For example, the performance measures for the Joint Staff initiative, intended to maximize the visibility of the condition codes of non-munitions assets in GCSS-J to support joint logistics planning, are linked to the 2015 Strategy’s goals of: improving visibility into customer materiel requirements and available resources; o enhancing visibility of assets in transit, in storage, in process, and in theater; and o enabling an integrated accessible authoritative data set.

Measures for 3 of the 8 initiatives did not include the key attribute of linkage because they were not aligned with agency-wide goals and mission and were not clearly communicated throughout the organization. These initiatives were identified in the 2014 Strategy and the descriptions of the initiatives did not specify which of the goals and objectives they were intended to support. We reported in January 2015 that the 2014 Strategy did not direct that the performance measures developed for the initiatives link to the goals or objectives in the 2014 Strategy, and we found that it was not clear whether the measures linked to the Strategy’s goals and objectives. Therefore, we recommended that DOD ensure that the linkage between the performance measures for the individual initiatives and the goals and objectives outlined in the 2014 Strategy be clear. DOD concurred with our recommendation and in its 2015 Strategy linked each initiative to the goals and objectives.

The deficiencies that we identified in the performance measures can be linked to the fact that the Strategies have not included complete direction on the key attributes of successful performance measures. The 2014 Strategy provided direction on the types of expected outcomes and key performance indicators. For example, an expected outcome is to increase supply chain performance and the key performance indicator is to improve customer wait time. However, when OSD updated the 2014 Strategy it did not include in the 2015 Strategy an example of the types of expected outcomes and key performance indicators for components to use when developing performance measures. The lack of direction on successful performance measures may have resulted in measures that lacked key attributes, such as clarity, measurable target, objectivity, reliability, baseline and trend data, and linkage, as we previously discussed. While OSD officials stated that they believed the performance measures for the selected initiatives were sufficient to report on the status of the initiatives, our review of these measures determined that they could not be used to effectively assess the performance of the initiatives to improve asset visibility. Without sufficient direction in subsequent updates to the Strategy on developing successful performance measures, DOD has limited assurance that the components are developing measures that can be used to determine how the department is progressing toward achieving its goals and objectives related to improving asset visibility.

After-Action Reports for Initiatives Have Not Always Included Performance Measures to Assess the Initiatives’ Success

As described in the 2015 Strategy, the Asset Visibility Working Group and the component review the performance of the initiatives during implementation. As we reported in January 2015, the components report quarterly to the Asset Visibility Working Group on the status of their initiatives—including progress made on implementation milestones, return on investment, and resources and funding. We found that DOD components had included performance measures in their quarterly status reports for the 8 initiatives we reviewed.

However, DOD components have not always included performance measures to assess the success of their initiatives in after-action reports, which are added to the status report for completed initiatives. To close an initiative, the components responsible for the initiative request closure and the Asset Visibility Working Group files an after-action report, which serves as a closure document and permanent record of an initiative’s accomplishments. According to the 2015 Strategy, the after-action report should include information on the objectives met, problems or gaps resolved, challenges associated with implementing the initiative, any lessons learned from the initiative, and measures of success obtained. The Asset Visibility Working Group approves the closure of initiatives when the components have completed or canceled the initiatives and updated the status report section called the after-action report. Once an initiative is closed, according to DOD officials, the Working Group no longer monitors the initiative, but the components may continue to monitor it. According to these DOD officials, DOD components may update information provided to the Asset Visibility Working Group or the Working Group may request additional information after the initiative is closed, especially when implementation affects multiple components.

We found that the after-action reports did not always include all of the information necessary. According to our review of after-action reports, as of October 2016, the Asset Visibility Working Group had closed 5 of the 8 asset visibility initiatives that we examined. Our review of the after- action reports for the 5 closed initiatives found the following:

Two reports included information on whether the performance measures—also referred to as measures of success—for the initiative had been achieved.

Three reports did not follow the format identified in the 2015 Strategy, and we could not determine whether the intent and outcomes based on performance measures for the initiative had been achieved.

We also reviewed after-action reports for the remaining 15 initiatives that were closed and found the following:

Seven reports included information on whether the performance measures for the initiative had been achieved.

Five reports did not include information on performance measures, because these measures were not a factor in measuring the success of the initiative.

One report was not completed by the component.

Two reports did not follow the format identified in the 2015 Strategy, and we could not determine whether the intent and outcomes based on performance measures for the initiative had been achieved.

Based on our analysis, it appears that while the Asset Visibility Working Group closed 20 initiatives, it generally did not have information related to performance measures to assess the progress of these initiatives when evaluating and closing them. Specifically, the after-action reports for 11 of 20 initiatives did not include performance measures that showed whether the initiative had met its intended outcomes in support of the department’s Strategies. Officials from the Asset Visibility Working Group stated that they generally relied on the opinion of the component’s subject matter experts, who are familiar with each initiative’s day-to-day performance, to assess the progress of these initiatives. While including the input of the component’s subject matter experts for the initiative in the decision to close the initiative is important, without incorporating after-action reports information relating to performance measures into the information considered by the Asset Visibility Working Group, DOD does not have assurance that closed initiatives have been fully assessed and whether they have resulted in achieving the goals and objectives of the Strategies.

DOD Has Met the Criteria for Leadership Commitment, Capacity, and an Action Plan, but Additional Actions Are Needed to Address Monitoring and Demonstrated Progress

DOD has fully met three of our criteria for removal from the High Risk List by improving leadership commitment, capacity, and its corrective action plan, and it has partially met the criteria to monitor the implementation of the initiatives and demonstrate progress in improving asset visibility. Table 4 includes a description of the criteria and our assessment of DOD’s progress in addressing each of them.

DOD Has Fully Met the Leadership Commitment, Capacity, and Corrective Action Plan Criteria for Removal from Our High Risk List

DOD Continues to Fully Meet Our High-Risk Criterion for Leadership Commitment Our high-risk criterion for leadership commitment calls for leadership oversight and involvement. DOD has taken steps to address asset visibility challenges, and we found—as we had in our February 2015 high- risk report—that DOD has fully met this criterion. Senior leaders at the department have continued to demonstrate commitment to addressing the department’s asset visibility challenges, as evidenced by the issuance of DOD’s 2014 and 2015 Strategies. The Office of the Deputy Assistant Secretary of Defense for Supply Chain Integration provides department- wide oversight for development, coordination, approval, and implementation of the Strategies and reviews the implementation of the initiatives. Also, senior leadership commitment is evident in its involvement in asset visibility improvement efforts, including groups such as the Supply Chain Executive Steering Committee—a group of senior- level officials responsible for overseeing asset visibility improvement efforts—and the Asset Visibility Working Group—a group of officials that includes representatives from the components and other government agencies, as needed. The Asset Visibility Working Group identifies opportunities for improvement and monitors the implementation of initiatives. Sustained leadership commitment will be critical moving forward, as the department continues to implement its Strategies to improve asset visibility and associated asset visibility initiatives.

DOD Has Fully Met Our High-Risk Criterion for Capacity Our high-risk criterion for capacity calls for agencies to demonstrate that they have the people and other resources needed to resolve risks in the high-risk area. In our October 2014 management letter to a senior OSD official and our January 2015 and February 2015 reports, we noted that resources and investments should be discussed in a comprehensive strategic plan, to include the costs to execute the plan and the sources and types of resources and investments—including skills, human capital, technology, information and other resources—required to meet established goals and objectives. DOD has demonstrated that it has the capacity—personnel and resources—to improve asset visibility. For example, as we previously noted, the department had established the Asset Visibility Working Group that is responsible for identifying opportunities for improvement and monitoring the implementation of initiatives. The Working Group includes representatives from OSD and the components—Joint Staff, the Defense Logistics Agency, U.S. Transportation Command, and each of the military services. Furthermore, DOD’s 2015 Strategy called for the components to consider items such as manpower, materiel, and sustainment costs when documenting cost estimates for the initiatives in the Strategy, as we recommended in our January 2015 and February 2015 reports. For example, DOD identified and broke down estimated costs of $10 million for implementing an initiative to track Air Force aircraft and other assets from fiscal years 2015 through 2018 by specifying that $1.2 million was for manpower, $7.4 million for sustainment, and $1.4 million for one-time costs associated with the consolidation of a server for the initiative. Additionally, DOD broke down estimated costs of $465,000 for implementing an initiative to track Marine Corps assets from fiscal years 2013 through 2015 by specifying $400,000 for manpower and $65,000 for materials.

However, in December 2015 we found that the 2015 Strategy included three initiatives that did not include cost estimates. To address this issue, in December 2016, a DOD official provided an abstract from the draft update to the 2015 Strategy that provides additional direction on how to explain and document cases where the funding for the initiatives is embedded within overall program funding. The draft update notes that there may be instances where asset visibility improvements are embedded within a larger program, making it impossible or cost prohibitive to isolate the cost associated with specific asset visibility improvements. In these cases, the document outlining the initiatives will indicate that cost information is not available and why. However, if at some point during implementation some or all costs are identified, information about the initiative will be updated. According to OSD officials, DOD plans to issue the update to the 2015 Strategy, but a release date has not been determined.

DOD Has Fully Met Our High-Risk Criterion for a Corrective Action Plan Our high-risk criterion for a corrective action plan calls for agencies to define the root causes of problems and related solutions and to include steps necessary to implement the solutions. The Fiscal Year 2014 National Defense Authorization Act (NDAA) required DOD to submit to Congress a comprehensive strategy and implementation plans for improving asset tracking and in-transit visibility. The Fiscal Year 2014 NDAA, among other things, called for DOD to include in its strategy and plans elements such as goals and objectives for implementing the strategy. The Fiscal Year NDAA also included a provision that we assess the extent to which DOD’s strategy and accompanying implementation plans include the statutory elements. In January 2014, DOD issued its Strategy for Improving DOD Asset Visibility and accompanying implementation plans that outline initiatives intended to improve asset visibility. DOD updated its 2014 Strategy and plans in October 2015. The 2014 and 2015 Strategies define the root causes of problems associated with asset visibility and related solutions (i.e., the initiatives). In our October 2014 management letter to a senior OSD official and our January and February 2015 reports, we found that while the 2014 Strategy and accompanying plans serve as a corrective action plan, there was not a clear link between the initiatives and the Strategy’s goals and objectives. We recommended that DOD clearly specify the linkage between the goals and objectives in the Strategy and the initiatives intended to implement the Strategy. DOD implemented our recommendation in its 2015 Strategy, which includes matrixes that link each of DOD’s ongoing initiatives intended to implement the Strategy to the Strategy’s overarching goals and objectives. DOD also added 8 initiatives to its 2015 Strategy and linked each of them to the Strategy’s overarching goals and objectives.

DOD Has Partially Met the Monitoring and Demonstrated Progress Criteria for Removal from the High Risk List

DOD Has Taken Steps to Monitor the Status of Initiatives, but Its Performance Measures Could Not Always Be Used to Track Progress Our high-risk criterion on monitoring calls for agencies to institute a program to monitor and independently validate the effectiveness and sustainability of corrective measures, for example, through performance measures. DOD has taken steps to monitor the status of asset visibility initiatives, but we found that it has only partially met our high-risk criterion for monitoring. In our February 2015 High-Risk update, we referred to a 2013 report in which we had found that DOD lacked a formal, central mechanism to monitor the status of improvements or fully track the resources allocated to them. We also reported that while DOD’s draft 2014 Strategy included overarching goals and objectives that addressed the overall results desired from implementation of the Strategy, it only partially included performance measures, which are necessary to enable monitoring of progress.

Since February 2015, DOD has taken some steps to improve its monitoring of its improvement efforts. As noted in the 2015 Strategy, DOD has described and implemented a process that tasks the Asset Visibility Working Group to review the performance of the component’s initiatives during implementation on a quarterly basis, among other things. The Working Group uses status reports from the DOD components that include information on resources, funding, and progress made toward implementation milestones. DOD also identified performance measures for its asset visibility initiatives. However, as previously discussed, the measures for the 8 initiatives we reviewed were not generally clear, quantifiable (i.e., lacked measurable targets and baseline and trend data), objective, and reliable. Measures that are clear, quantifiable, objective, and reliable can help managers better monitor progress, including determining how well they are achieving their goals and identifying areas for improvement, if needed. In December 2016, a DOD official provided an abstract from the draft update to the 2015 Strategy that noted that detailed metrics data will be collected and reviewed at the level appropriate for the initiative. High-level summary metrics information will be provided to the Working Group in updates to the plan outlining the initiatives. The extent to which this planned change will affect the development of clear, quantifiable, objective, and reliable performance measures remains to be determined.

Additionally, as discussed previously, while the Asset Visibility Working Group has closed 20 initiatives, it generally did not have information related to performance measures to assess the progress of these initiatives. Specifically, after-action reports from 11 of 20 initiatives— which are added to the status reports for completed initiatives—did not include performance measures that showed whether the initiative had met its intended outcomes in support of the department’s Strategies. Without improved performance measures and information to support that progress has been made, DOD may not be able to monitor asset visibility initiatives.

DOD Has Demonstrated Some Progress but Cannot Demonstrate that Its Initiatives Have Resulted in Measurable Outcomes and Improvements for Asset Visibility Our high-risk criterion for demonstrated progress calls for agencies to demonstrate progress in implementing corrective measures and resolving the high-risk area. DOD has made progress by developing and implementing its Strategies for improving asset visibility. In our October 2014 management letter to a senior OSD official and our January and February 2015 reports, we noted that in order to demonstrate progress in having implemented corrective measures, DOD should continue the implementation of the initiatives identified in the Strategy, refining them over time as appropriate. DOD reports that it has closed or will no longer monitor the status of 20 of the 27 initiatives and continues to monitor the remaining 7 initiatives. Additionally, in October 2016, DOD officials stated that they plan to add about 10 new initiatives in the update to the 2015 Strategy. For example, the U.S. Transportation Command’s new initiative, Military Service Air Manifesting Capability, is expected to promote timely, accurate, and complete in-transit visibility and effective knowledge sharing to enhance understanding of the operational environment. OSD officials have not yet determined a date for the release of the update to the 2015 Strategy.

As discussed previously, we found that DOD cannot use the performance measures associated with the initiatives to demonstrate progress, because the measures are not generally clear, quantifiable (i.e., lack measurable targets and baseline and trend data), objective, and reliable. Additionally, we found that DOD has not taken steps to consistently incorporate information on an initiative’s performance measures in closure reports, such as after-action reports, in order to demonstrate the extent to which progress has been made toward achieving the intended outcomes of the individual initiatives and the overall goals and objectives of the Strategies. Without clear, quantifiable, objective, and reliable performance measures and information to support that progress has been made, DOD may not be able to demonstrate that implementation of these initiatives has resulted in measurable outcomes and progress toward achieving the goals and objectives in the Strategies. Also, DOD will be limited in its ability to demonstrate sustained progress in implementing corrective actions and resolving the high-risk area.

Conclusions

DOD has taken some positive steps to address weaknesses in asset visibility. Long-standing management weaknesses related to DOD’s asset visibility functions hinder the department’s ability to provide spare parts, food, fuel, and other critical supplies in support of U.S. military forces. We previously reported on several actions that we believe DOD should take in order to mitigate or resolve long-standing weaknesses in asset visibility and meet the criteria for removing asset visibility from the High Risk List. We believe that DOD has taken the actions necessary to meet the capacity and action plan criteria by providing additional direction to the components on formulating cost estimates for the asset visibility initiatives. Additionally, DOD linked the 2015 Strategy’s goals and objectives with the specific initiatives intended to implement the Strategy. However, DOD’s efforts to monitor initiatives show that the performance measures DOD components currently use to assess these initiatives lack some of the key attributes of successful performance measures that we have identified. To the extent that these measures lack the key attributes of successful performance measures, they limit DOD’s ability to effectively monitor the implementation of the initiatives and assess the effect of the initiatives on the overall objectives and goals of the Strategies. Developing clear, quantifiable, objective, and reliable performance measures can help DOD better assess department-wide progress against the Strategies’ goals and clarify what additional steps need to be taken to enable decision makers to exercise effective oversight.

An important step in determining what effect, if any, the asset visibility initiatives are having on the achievement of the Strategies’ goals and objectives will be to develop sound performance measures and incorporate information about these measures into the after-action reports when evaluating and closing initiatives. Until DOD components demonstrate that implementation of the initiatives will result in measurable outcomes and progress toward achieving the goals and objectives of the Strategies, DOD may be limited in its ability to demonstrate progress in implementing corrective actions and resolving the high-risk area. Once these actions are taken, DOD will be better positioned to demonstrate the sustainable progress needed in its approach to meeting the criteria for removing asset visibility from our High Risk List.

Recommendations for Executive Action

We are making two recommendations to help improve DOD’s asset visibility.

We recommend that the Secretary of Defense direct the Assistant Secretary of Defense for Logistics and Materiel Readiness, in collaboration with the Director, Defense Logistics Agency; the Secretaries of the Army, Navy, and Air Force; the Commandant of the Marine Corps; the Commander of the United States Transportation Command; and the Chairman of the Joint Chiefs of Staff, to: use the key attributes of successful performance measures—including clarity, measurable target, objectivity, reliability, baseline and trend data, and linkage—in refining the performance measures in subsequent updates to the Strategy to improve DOD’s efforts to monitor asset visibility initiatives; and incorporate into after-action reports information relating to performance measures for the asset visibility initiatives when evaluating and closing these initiatives to ensure that implemented initiatives will achieve the goals and objectives in the Strategies.

Agency Comments and Our Evaluation

In its written comments on a draft of this report, DOD partially concurred with our two recommendations. DOD’s comments are summarized below and reprinted in appendix IV.

DOD partially concurred with our first recommendation that it use the key attributes of successful performance measures—including clarity, measurable target, objectivity, reliability, baseline and trend data, and linkage—in refining the performance measures in subsequent updates to the Strategy to improve DOD’s efforts to monitor asset visibility initiatives. DOD stated that it recognizes the need for performance measures to ensure the success of an asset visibility improvement effort but noted that the level of complexity and granularity of the metrics we suggest may not be suitable for all initiatives. DOD also stated that the purpose of the Strategy is to create a framework whereby the components can work collaboratively to coordinate and integrate department-wide efforts to improve asset visibility, not to provide complete direction on how to define, implement, and oversee these initiatives. Additionally, DOD stated that the next edition of the Strategy will encourage the adoption of our six key attributes for asset visibility initiatives to the extent appropriate, but will not mandate their use. As discussed in our report, use of the key attributes in measuring the performance of asset visibility initiatives would help DOD to better assess department-wide progress against the goals in its Strategy and clarify what additional steps need to be taken to enable decision makers to exercise effective oversight. Encouraging adoption of the key attributes, as DOD plans to do, is a positive step, but we continue to believe that DOD needs to use these key attributes to refine its performance measures to monitor the initiatives in the future.

DOD partially concurred with our second recommendation that it incorporate into after-action reports information relating to performance measures for the asset visibility initiatives when evaluating and closing these initiatives to ensure that implemented initiatives will achieve the goals and objectives in the Strategies. DOD stated that it is important to capture and review performance data prior to a component closing an asset visibility initiative, but that the Strategy after-action report is not intended to be used to evaluate the success of an asset visibility initiative or to determine if an initiative has met its intended objectives. According to DOD, documentation and information to support the evaluation of initiatives is defined by and executed in accordance with component-level policy and procedures. DOD agreed to update its Strategy to clarify the purpose and use of the after-action reports and to ensure that the Strategy specifies roles and responsibilities for evaluating and closing initiatives. DOD’s response, however, did not state whether and how these updates to the Strategy would result in more consistent incorporation of information relating to performance measures when closing initiatives in the future. As we noted previously in this report, according to the 2015 Strategy, the after-action report for closed initiatives should include information on the objectives met, problems or gaps resolved, and measures of success obtained. We believe our recommendation is consistent with this guidance. Without incorporating this information, DOD does not have assurance that closed initiatives have been fully assessed and have resulted in achieving the goals and objectives of the Strategies. Therefore, we continue to believe that full implementation of our recommendation is needed.

We are sending copies of this report to appropriate congressional committees; the Secretary of Defense; the Secretaries of the Army, Navy, and Air Force, and the Commandant of the Marine Corps; the Director of Defense Logistics Agency; the Chairman of the Joint Chiefs of Staff; the Commander of the United States Transportation Command; and other interested parties. In addition, the report is available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-5257 or merrittz@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made contributions to this report are listed in appendix V.

Appendix I: Scope and Methodology

To determine the extent to which DOD identified performance measures that allow it to monitor the progress of selected asset visibility initiatives identified in DOD’s 2014 and 2015 Strategy for Improving DOD Asset Visibility (Strategies), we reviewed documents such as the 2014 Strategy and its subsequent update in October 2015 (2015 Strategy); minutes from the Asset Visibility Working Group meetings; and documents showing the status of the implementation, including charts that track the development and closure of the asset visibility initiatives. Thirty initiatives have been included in the 2014 and 2015 Strategies, but 3 of these were halted, for a variety of reasons. From the remaining 27 initiatives, we selected a non-generalizable sample of 8 initiatives. We selected at least one from each of the components to review and assess, including analyzing the performance measures associated with each initiative. In our selection of 8 initiatives to review, we also considered the stage of implementation of the initiative, to ensure that our review encompassed initiatives at different stages, from some that were just beginning to some that had already been completed. Specifically, we made selections based on the status of the initiatives as of December 2015 to include the earliest completion dates by component. In order to cover a range of initiatives— from some just beginning to some already completed—we selected for review 3 initiatives from the 2014 Strategy that had been closed, 2 ongoing initiatives that had been included in both Strategies, and 3 new initiatives that were included for the first time in the 2015 Strategy. The results from this sample cannot be generalized to the other 19 initiatives. We did not assess the initiatives to determine if they (1) met milestones, (2) lacked resources, or (3) had performance issues. Instead we assessed the initiatives to determine what progress DOD had made toward meeting the criteria for removing an area from our High Risk list.

We surveyed program managers and other cognizant officials (hereafter referred to as component officials) responsible for the respective asset visibility initiatives we selected. We included questions in our survey related to the development and closure of the initiatives and took several steps to ensure the validity and reliability of the survey instrument. We also reviewed the Strategies to identify performance measures necessary to monitor the progress of the 8 initiatives we had selected. Two analysts independently assessed whether (1) DOD had followed the guidance set forth in the Strategies and (2) the measures for the initiatives included selected key attributes of successful performance measures (for example, are the measures clear, quantifiable —i.e., have measurable targets and baseline and trend data—objective, and reliable); any initial disagreements in assessments were resolved through discussion. We assessed these measures against 6 of 10 selected key attributes for successful performance measures—clarity, measurable target, objectivity, reliability, baseline and trend data, and linkage—identified in our prior work that we identified as relevant to the sample of initiatives we were examining. The remaining 4 attributes—government-wide priorities, core program activities, limited overlap, and balance—are used to assess agency-wide performance and are not applicable to our analysis, because we did not assess agency-wide initiatives. Because we had selected a subset of the component-level initiatives to review, these attributes did not apply. If all of the performance measures for an initiative met the definition of the relevant key attribute, we rated the initiative as having “fully included” the attribute. On the other hand, if none of the measures met the definition of the relevant key attribute, we rated the initiative as having “not included” the attribute. If some, but not all, of the measure met the definition of the relevant key attribute, then we rated the initiative as having “partially included” the attribute.

We also selected sites to observe demonstrations of initiatives that were intended to show how they have achieved progress in improving asset visibility. We selected these demonstrations based on the location of the initiative, the responsible component, and the scope of the initiative. Additionally, we reviewed the after-action reports for all of the initiatives that had been closed—20 of 27 initiatives, including 5 of the 8 initiatives we reviewed in detail—by the Asset Visibility Working Group, as of October 31, 2016. We performed a content analysis in which we reviewed each of these after-action reports to determine whether it was completed for the initiative, documented whether measures were obtained, and identified challenges and lessons learned. One analyst conducted this analysis, coding the information and entering it into a spreadsheet; a second analyst checked the first analyst’s analysis for accuracy. Any initial disagreements in the coding were discussed and reconciled by the analysts. The analysts then tallied the responses to determine the extent to which the information was identified in the after-action reports. We also interviewed component officials and officials at the Office of the Deputy Assistant Secretary of Defense for Supply Chain Integration (hereafter referred to as OSD) to clarify survey responses and to discuss plans to develop the initiatives, including any efforts to monitor progress and demonstrate results.

To determine whether DOD had addressed the five criteria—leadership commitment, capacity, corrective action plan, monitoring, and demonstrated progress—that would have to be met for us to remove asset visibility from our High Risk List, we reviewed documents such as DOD’s 2014 and 2015 Strategies and charts that track the implementation and closure of asset visibility initiatives. We included questions in our survey to collect additional information from officials on their efforts to address the high-risk criteria. For example, we asked how the component monitors the implementation of the initiative and whether there has been any demonstrated progress in addressing the opportunity, deficiency, or gap in asset visibility capability that the initiative was designed to address. One analyst evaluated DOD’s actions to improve asset visibility against each of our five criteria for removing an area from the High Risk list. A different analyst checked the first analyst’s analysis for accuracy. Any initial disagreements were discussed and reconciled by the analysts. We assessed DOD’s effort to meet each of the high-risk criteria as “not met” (i.e., none of the aspects of the criterion were addressed), “partially met” (i.e., some, but not all, aspects of the criterion were addressed), or “fully met” (i.e., all parts of the criterion were fully addressed). We shared with DOD officials our preliminary assessment of asset visibility relative to each of the criteria. To help ensure that our evaluation of improvements made relative to the high-risk criteria were consistent with our prior evaluations of Supply Chain Management and other issue areas, we reviewed our prior High Risk reports to gain insight into what actions agencies had taken to address the issues identified in these past reports. Additionally, we interviewed component officials and OSD officials to clarify their survey responses and to discuss plans to continue to make progress in improving asset visibility.

We met with officials from the following DOD components during our review:

Office of the Secretary of Defense

Department of the Army

United States Marine Corps

Department of the Air Force We surveyed component officials responsible for the asset visibility initiatives we reviewed. We included questions in our survey related to our high-risk criteria. As part of the survey development, we conducted an expert review and pre-tested the draft survey. We submitted the questionnaire for review by an independent GAO survey specialist and an asset visibility subject matter expert from OSD. The expert review phase was intended to ensure that content necessary to understand the questions was included and that technical information included in the survey was correct. To minimize errors that might occur from respondents interpreting our questions differently than we intended, we pre-tested our questionnaire with component officials and other cognizant officials for 4 of the initiatives. During the pre-tests, conducted by telephone, we asked the DOD officials to read the instructions and each question aloud and to tell us how they interpreted the question. We then discussed the instructions and questions with them to identify any problems and potential solutions by determining whether (1) the instructions and questions were clear and unambiguous, (2) the terms we used were accurate, (3) the questionnaire was unbiased, and (4) the questionnaire did not place an undue burden on the officials completing it. We noted any potential problems and modified the questionnaire based on feedback from the expert reviewers and the pre-tests, as appropriate.

We sent an email to each selected program office beginning on June 16, 2016, notifying them of the topics of our survey and when we expected to send the survey. We then sent the self-administered questionnaire and a cover email to the asset visibility program officials on June 20, 2016, and asked them to fill in the questionnaire and email it back to us by July 6, 2016. We received 8 completed questionnaires, for an overall response rate of 100 percent. We also collected data—such as the number of RFID tags and number of inventory amounts for clothing and textiles—from a sample of initiatives. The practical difficulties of conducting any survey may introduce errors, commonly referred to as non-sampling errors. For example, differences in how a particular question is interpreted, the sources of information available to respondents, how the responses are processed and analyzed, or the types of people who do not respond can influence the accuracy of the survey results. We took steps, as described above, in the development of the survey, the data collection, and the data analysis to minimize these non-sampling errors and help ensure the accuracy of the answers that we obtained. Data were electronically extracted from the questionnaires into a comma-delimited file that was then imported into a statistical program for analysis. We examined the survey results and performed computer analyses to identify inconsistencies and other indications of error, and we addressed such issues as necessary. Our survey specialist conducted quantitative data analyses using statistical software, and our staff conducted a review of open-ended responses with subject matter expertise. A data analyst conducted an independent check of the statistical computer programs for accuracy.

We conducted this performance audit from February 2016 to March 2017 in accordance with generally accepted government auditing standards.

Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Selected DOD Initiatives Identified in the 2014 and 2015 Strategies

This appendix provides an overview of the non-generalizable sample of initiatives that we reviewed. These initiatives are intended to improve asset visibility as part of the Department of Defense’s (DOD) 2014 Strategy for Improving DOD Asset Visibility (2014 Strategy) and its subsequent update in October 2015 (2015 Strategy). The process by which we selected these initiatives for this review is described in appendix I. The initiatives are shown in table 5.

Appendix III: Criteria for Removal from Our High Risk List and GAO’s Prior Work on Asset Visibility

In 1990, we began a program to report on government operations that we identified as “high risk,” and we added the Department of Defense’s (DOD) supply chain management area to our High Risk List. Our high-risk program has served to identify and help resolve serious weaknesses in areas that involve substantial resources and provide critical services to the public. Our experience with the high-risk series over the past two decades has shown that the key elements needed to make progress in high-risk areas are congressional action, high-level administrative initiatives, and agencies’ efforts grounded in the five criteria we established for removing an area from the high-risk list. These five criteria form a road map for efforts to improve and ultimately address high-risk issues. Addressing some of the criteria leads to progress, while satisfying all of the criteria is central to removing an area from the list. These criteria call for agencies to show the following: 1. Leadership Commitment—a strong commitment and top leadership support. 2. Capacity—the capacity (i.e., the people and other resources) to resolve the risk(s). 3. Corrective Action Plan—a plan that defines the root causes and solutions and provides for substantially completing corrective measures, including steps necessary to implement the solutions we recommended. 4. Monitoring—a program instituted to monitor and independently validate the effectiveness and sustainability of corrective measures. 5. Demonstrated Progress—the ability to demonstrate progress in implementing corrective measures and resolving the high-risk area.

We have reported on various aspects of DOD’s supply chain, including asset visibility, and noted that DOD has taken several actions to improve asset visibility. We also noted a number of recommendations, actions, and outcomes needed to improve asset visibility, as shown in table 6. Specifically, in an October 2014 management letter to a senior Office of the Secretary of Defense (OSD) official, we reported on 7 actions and outcomes across the 5 criteria that we believed DOD should take to address long-standing weaknesses in asset visibility. Most recently, in our January 2015 report and February 2015 High Risk update, we reported on progress that DOD has made in addressing weaknesses in its asset visibility, including developing its 2014 Strategy for Improving DOD Asset Visibility, and we made a number of recommendations.

Appendix IV: Comments from the Department of Defense

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Carleen C. Bennett, Assistant Director; Mary Jo LaCasse; Joanne Landesman; Amie Lesser; Felicia Lopez; Mike Silver; John E. Trubey; Angela Watson; and John Yee made key contributions to this report.

Related GAO Products

High-Risk Series: Progress on Many High-Risk Areas, While Substantial Efforts Needed on Others. GAO-17-317. Washington, D.C.: February 15, 2017.

High-Risk Series: Key Actions to Make Progress Addressing High-Risk Issues. GAO-16-480R. Washington, D.C.: April 25, 2016.

Defense Logistics: DOD Has Addressed Most Reporting Requirements and Continues to Refine its Asset Visibility Strategy. GAO-16-88. Washington, D.C.: December 22, 2015.

High-Risk Series: An Update. GAO-15-290. Washington, D.C.: February 11, 2015.

Defense Logistics: DOD Has a Strategy and Has Taken Steps to Improve its Asset Visibility, But Further Actions are Needed. GAO-15-148. Washington, D.C.: January, 27, 2015.

Defense Logistics: A Completed Comprehensive Strategy is Needed to Guide DOD’s In-Transit Visibility Efforts. GAO-13-201. Washington, D.C.: February 28, 2013.

High-Risk Series: An Update: GAO-13-283. Washington, D.C.: February 14, 2013.

Defense Logistics: Improvements Needed to Enhance DOD’s Management Approach and Implementation of Item Unique Identification Technology. GAO-12-482. Washington, D.C.: May 3, 2012.

Defense Logistics: DOD Needs to Take Additional Actions to Address Challenges in Supply Chain Management. GAO-11-569. Washington, D.C.: July 28, 2011.

High-Risk Series: An Update. GAO-11-278. Washington, D.C.: February 16, 2011.

DOD’s High-Risk Areas: Observations on DOD’s Progress and Challenges in Strategic Planning for Supply Chain Management. GAO-10-929T. Washington, D.C.: July 27, 2010.