Movement of Household Goods: DOD Should Take Additional Steps to Assess Progress toward Achieving Program Goals
Released: 2020-04-06T00:00:00

Letter

Background

The Defense Personal Property Program

DOD currently relies on more than 900 commercial industry transportation service providers (TSPs) to move and store servicemembers’ and their families’ household goods. About 40 percent of DOD’s annual household goods moves occur during the 14-week annual peak moving season, which runs from May 15 through August 31. As figure 1 shows, servicemember survey satisfaction scores declined in 2018 during the peak moving season. According to TRANSCOM officials, this decline in satisfaction scores during the peak moving season has persisted for years.

Under the current DP3, the military services own, operate, and staff most of the infrastructure involved in managing and overseeing the movement and storage-in-transit of DOD’s household goods, including the personal property processing offices and personal property shipping offices. The military services’ processing offices, among other things, ensure that the servicemembers’ reassignment orders and paperwork authorizing a move are in order, and that their application is accurate and complete. The processing offices also provide counseling, including advising servicemembers on the DP3 process, entitlements, and restrictions so they can make informed decisions about their moves. The military services’ shipping offices work with TSPs to schedule moves, monitor TSP performance, and take or recommend punitive action against poor performing TSPs.

Recent DOD Efforts to Address Long-Standing DP3 Performance Issues

DOD has conducted and sponsored several studies to help address persistent DP3 performance issues, including those related to servicemember satisfaction and challenges with meeting capacity demands during the peak summer months. For example, LMI and the Institute for Defense Analyses (IDA) produced reports on behalf of DOD in 2012 and 2018, respectively, to analyze DP3 performance issues and make recommendations to improve DP3. In its report, LMI outlined the impact of a range of options for contracting out some or all of the management of DOD’s household goods movement and storage activities. IDA recommended that DOD create consistent performance metrics, unify its DP3 operating structure, and improve servicemember support.

Concurrent with pursuing its Global Household Goods Contract, TRANSCOM has several ongoing initiatives intended to improve servicemember satisfaction in general, and household goods loss and damage specifically. According to survey data, loss of and damage to household goods is the number one reason servicemembers cite for dissatisfaction. Ongoing TRANSCOM initiatives to improve servicemembers’ moving experiences that are independent of the Global Household Goods Contract include the following: Increasing the use of shipping containers in domestic household goods moves to reduce the number of servicemember loss and damage claims. According to TRANSCOM officials, in the past TRANSCOM has primarily used shipping containers only for international moves.

Replacing the internet-based system used to manage DOD household goods moves, the Defense Personal Property System, with a new, mobile-friendly information technology operating system platform called MILMOVE. MILMOVE is intended to provide servicemembers and their families with more reliable information about their planned and ongoing household goods moves through access to all phases of the move process via their personal smartphones or tablets.

Tracking whether quality inspections occur in person or via telephone, and requiring that at least 50 percent of all shipments have an in- person quality inspection.

Increasing the liability limits for damaged or lost household goods shipments to $6 per pound and the total loss cap to $75,000 per shipment, when claims are filed within a specified time.

TRANSCOM Efforts to Develop the Global Household Goods Contract

In November 2018, TRANSCOM, in coordination with OSD and the military services, began planning and developing requirements for the Global Household Goods Contract. According to TRANSCOM officials, the goal was to have the contract in place in time for the 2021 peak moving season. TRANSCOM announced that in April 2020 it would award a Global Household Goods Contract to a single commercial move manager to oversee DP3 activities that relate to the movement and storage-in-transit of household goods. The proposed contract includes an initial 9-month transition period that will commence in May 2020, a 3-year base period, three 1-year option periods, two 1-year award terms, and an option to extend the contract 6 months. Including all option periods and award terms, the contract is expected to be completed at the end of January 2029 or, if the option to extend services is exercised, by the end of July 2029. The 9-month transition period is intended to give the Global Household Goods contractor sufficient time to, among other things, integrate its information technology systems with existing DOD information technology systems. During the initial years of the contract, the contractor’s volume of workload will incrementally increase, including the volume of household goods moves that it will handle.

The Global Household Goods Contract is not intended to replace DP3 in its entirety. Activities that are planned to be a part of the contract include the movement and storage-in-transit of household goods, direct procurement method shipments, and some level of servicemember counseling. DP3 activities that are not to be a part of the Global Household Goods Contract include the long-term storage of household goods (referred to as non-temporary storage) and the movement and storage of servicemembers’ privately owned vehicles. Additionally, while the contract is expected to include counseling to servicemembers, the military services will also perform some counseling. According to TRANSCOM and military service officials, they are working to determine the amount of counseling that will be retained by the military services.

Key Differences in How Household Goods Are Moved and Stored in DP3 Currently and under the Planned Global Household Goods Contract

The movement and storage-in-transit of household goods in the current DP3 differs in key ways from the approach under the planned Global Household Goods Contract, as shown in table 1.

TRANSCOM Has Taken Steps to Determine the Cost Implications Associated with Moving to a DP3 That Incorporates the Global Household Goods Contract, but Plans Not to Determine Some Cost Estimates for Years

TRANSCOM has developed preliminary and refined cost estimates for determining the cost implications associated with moving to a DP3 that incorporates the Global Household Goods Contract. According to TRANSCOM officials, they began developing preliminary cost estimates in December 2018 in order to (1) create a baseline cost estimate for the movement and storage of household goods under the current DP3, (2) create a cost estimate for those DP3 activities that will be a part of the Global Household Goods Contract, and (3) serve as a point of comparison between the two estimates to determine the cost implications of moving to the Global Household Goods Contract.

TRANSCOM’s preliminary cost estimates for the current DP3 and for the activities that will be a part of the Global Household Goods Contract included costs associated with the movement and storage of household goods and government personnel costs associated with the military services’ personal property processing and shipping offices. TRANSCOM developed these preliminary cost estimates in part by requesting and collecting information from the military services about their current costs under DP3. Some DP3 costs were not included in the preliminary cost estimates, such as infrastructure and vehicle costs because these assets are used to support multiple programs and could not be easily isolated.

However, our assessment of the preliminary DP3 cost estimates associated with moving to the Global Household Goods Contract found that TRANSCOM may not have accurately calculated some costs because of unanswered questions about how certain activities will be performed. TRANSCOM’s cost estimates associated with the number of government personnel that (1) will counsel servicemembers and (2) oversee contractor performance under the contract had weaknesses. Specifically, these estimates relied on assumptions that may have resulted in over- or underestimating some costs.

First, TRANSCOM’s preliminary DP3 cost estimates associated with moving to the Global Household Goods Contract assume that DOD’s costs related to government personnel will be equivalent to those under the current DP3. However, TRANSCOM officials acknowledge that this assumption is based on discussions with military service officials that occurred before the Global Household Goods Contract draft request for proposals was issued in April 2019. Furthermore, military service officials told us they have not decided how much of the counseling function their service has provided will be moved to the planned contract, and the services’ approaches will likely differ. TRANSCOM officials pointed out that while the Global Household Goods contractor will perform some level of servicemember counseling, the contract will allow the individual military services to decide how much of the counseling responsibility to retain.

Second, TRANSCOM’s cost estimates do not account for the number of DOD contracting officer’s representatives and quality assurance evaluators required to oversee contractor performance under the contract. According to TRANSCOM officials, the command’s preliminary cost estimates assume that government personnel who have been relieved of servicemember counseling responsibility will perform contract oversight and quality assurance responsibilities under the planned contract. For example, TRANSCOM officials told us that quality assurance inspectors under the current DP3 will transition to serve as quality assurance evaluators once the contract is in place. However, TRANSCOM officials acknowledge that it is possible that workload (and personnel costs) for the military services under the current DP3 will change under the planned contract. Moreover, if government personnel do switch roles and responsibilities, TRANSCOM did not take into account the cost to transition and train personnel to execute their contracting officer’s representative and quality assurance evaluator responsibilities under the contract.

TRANSCOM officials told us that they were unable to determine with any precision the number and associated costs of government personnel required to counsel servicemembers and oversee the contract because of the fractured nature of the current DP3. Nonetheless, in September 2019, TRANSCOM tasked LMI with developing a BCA with refined cost estimates. LMI finalized the BCA and we received it on January 17, 2020. When we assessed the BCA, we found that like us LMI determined that it did not have complete information to fully calculate the cost of a DP3 that incorporates the Global Household Goods contract. Specifically, LMI acknowledged some of the same limitations in its cost estimates that we identified in TRANSCOM’s preliminary cost estimates, such as uncertainty about the number of government personnel required to oversee the Global Household Goods Contract. For example, LMI states in its BCA that the Global Household Goods Contract may reduce DOD staffing requirements for functions such as counseling; however, roles such as quality assurance evaluators and contracting officer’s representatives to oversee contractor performance may increase DOD staffing requirements.

When we raised concerns about these unanswered questions and their potential cost implications with TRANSCOM officials, they told us that the move to the Global Household Goods Contract is less about saving money than it is about representing the best value to DOD when both cost and program performance are considered. They also told us they have developed a plan of action for the phase-in of the Global Household Goods Contract, and that plan includes conducting a manpower study during the third year of the contract. As described earlier in this report, the initial 3 years of the contract involve a gradual phase-in of household goods move volume for the contractor, and in the third year the contractor will be responsible for all of DOD’s household goods move volume. According to these officials, by waiting until year 3 of the contract to conduct a manpower study, DOD will have data to more precisely determine the number (and cost) of government personnel required to counsel servicemembers and oversee the contract, such as contracting officer’s representatives and quality assurance evaluators.

However, we have determined that TRANSCOM does not have a process in place to track data that would inform its manpower study in year 3, such as how many personnel within each military service are needed to perform contract oversight duties and the costs associated with these personnel. We have reported that organizations should determine their personnel requirements as part of a systematic requirements- determination process that includes (1) identifying an organization’s mission, functions, and tasks and (2) determining the minimum number and type of personnel—military, civilian, and contractor—needed to fulfill those missions, functions, and tasks by conducting a workforce analysis. Without a way to track key data, DOD risks conducting a manpower study that does not allow it to fully understand the personnel and cost implications of its move to a DP3 that incorporates a Global Household Goods Contract. Notably, in its BCA LMI recommended that DOD reexamine the BCA when additional relevant information becomes available, such as when DOD completes its manpower study. TRANSCOM officials told us they are in agreement with this recommendation. However, if TRANSCOM does not have a process in place to collect key pieces of data during the first 3 years of the contract, a reexamination of the BCA will be less fruitful than it otherwise might be.

TRANSCOM Has Developed Performance Metrics to Assess Some but Not All DP3 Activities, and Not All of TRANSCOM’s Overarching Program Goals Have Clearly Associated Performance Metrics

First, TRANSCOM has developed performance metrics—referred to as performance indicators—for its Global Household Goods Contract, and has developed performance metrics for some, but not all, activities that fall outside of the contract.

TRANSCOM’s draft quality assurance surveillance plan for the Global Household Goods Contract, which was developed in coordination with the military services, outlines how the contractor’s performance will be assessed against performance indicators. These performance indicators set measurable standards for, among other things, information technology systems’ availability, claims settlement timeliness for lost and damaged goods, and the timeliness of household goods pick-up and deliveries. Examples of performance indicators include: settling 90 percent of all loss and damage claims valued at less than $1,000 within 30 days and settling 95 percent of all claims, regardless of value, within 60 days; delivery of household goods on the scheduled date 95 percent of the time per month; and overall customer satisfaction rating of satisfactory at least 95 percent per month.

Based on a DP3 briefing TRANSCOM provided us, the command has also developed metrics for assessing other contracted DP3 activities, such as the transport of privately owned vehicles and non-temporary storage, which TRANSCOM captures under contracts separate from the Global Household Goods Contract. For example, TRANSCOM tracks the number and cost of non-temporary storage lots maintained annually to support its storage requirements.

However, TRANSCOM officials acknowledge that the command has not developed metrics for other activities that government personnel will continue to perform in DP3, once a Global Household Goods Contract is in place, such as servicemember counseling and claims resolution. Although TRANSCOM’s draft quality assurance surveillance plan includes a performance indicator to assess the contractor’s performance with respect to servicemember counseling, TRANSCOM officials told us the command has not developed commensurate metrics to assess the military services’ performance in providing servicemember counseling. TRANSCOM officials also noted that while the military services’ claims offices will be responsible for handling unresolved loss and damage claims between the servicemember and the contractor, the command has not developed metrics associated with this DP3 activity.

Second, we found that, while TRANSCOM has articulated overarching DP3 goals, it has not clearly articulated how its performance metrics align with each of these goals. According to TRANSCOM officials, they decided to move certain DP3 activities to the Global Household Goods Contract because doing so could positively impact the program’s five goals regarding cost, quality, capacity, accountability, and responsibility, terms described in LMI’s BCA as shown below:

Cost considers the potential financial impact, including either opportunities or risks to the future-state costs and any savings the program is likely to achieve.

Quality encompasses the value of a move, typically measured by customer (i.e., servicemember) satisfaction. Quality may improve with on-time performance and minimized loss or damage.

Capacity is the availability of industry providers to meet the program’s demands at a given time. This can be during peak or non-peak season.

Accountability refers to the government’s ability to affix responsibility to the contractor and their supplier network for performance.

Responsibility encompasses the authority of the government and how each government stakeholder will be held responsible for accomplishing their assigned tasks.

Planned contract performance indicators and metrics are intended to assess contractor performance, but they may also provide information that TRANSCOM could use to more broadly assess the extent to which DP3 is meeting its overarching program goals. However, TRANSCOM has not clearly articulated which of the performance metrics it has established under the Global Household Goods contract align with which program goals. For example, one performance indicator for the Global Household Goods Contract is for contractor delivery of household goods to be on-time at least 95 percent of the time. While it appears that this performance indicator could relate to the quality and capacity goals, TRANSCOM has not established linkage between this performance indicator and either of these goals. Determining how it will assess improved capacity is particularly important. When we spoke with them about the move to the Global Household Goods contract, some military service officials, representatives of the moving and storage industry, and members of TRANSCOM’s Personal Property Relocation Advisory Panel told us that they doubt whether the Global Household Goods Contract will, indeed, improve capacity.

Moreover, none of the existing performance indicators appear to relate to TRANSCOM’s responsibility goal. It is not surprising that the Global Household Goods Contract does not include performance indicators on how government stakeholders are to be held responsible for various program activities, because indicators under the contract are intended to evaluate contractor performance. However, we would expect to see performance metrics outside of the contract to measure government stakeholders’ performance.

When we spoke with TRANSCOM officials about the gaps we identified in their performance assessment approach, they agreed that clarifying the linkage between performance metrics and the overarching program goals and ensuring there are performance metrics for assessing each of the goals would improve the command’s ability to assess overall program performance. While the performance indicators and metrics related to the various contracts, including the Global Household Goods Contract, are intended to assess contractor performance, they could also provide information that TRANSCOM could use to more broadly assess the extent to which DP3 is meeting its overarching program goals. DOD Instruction 4500.57, Transportation and Traffic Management, calls for TRANSCOM, in coordination with DOD components, to conduct annual program reviews to ensure the overall effectiveness of the DP3. The instruction calls for these reviews to include a metrics-based evaluation of the program, and assessments of TSP and service-provider cost and performance, information technology systems and contracts that support DP3, and external factors that impact the program such as industry capability and changes to servicemember shipping entitlements. Moreover, the Standards for Internal Control in the Federal Government call for management to define objectives (or metrics) in terms of what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement so that performance toward achieving those objectives (or metrics) can be assessed. We have also previously reported on the importance of linking lower-order performance metrics and higher-order strategic goals to achieve desired outcomes.

Without performance metrics that account for those DP3 activities that fall outside of the Global Household Goods Contract and other DP3 contracts and without clearly articulating the linkage between performance metrics and program goals, TRANSCOM’s ability to assess progress toward, and take actions to help achieve, its overarching program goals will be hindered. The need to take these steps is particularly important, given some of the skepticism that TRANSCOM faces with its move to a DP3 that incorporates the Global Household Goods Contract. Further, because TRANSCOM officials have determined that the move to the Global Household Goods contract is about delivering better value, rather than just saving money, it is imperative that they have a robust performance assessment approach in place.

Conclusions

DOD has experienced long-standing quality issues moving and storing the household goods of servicemembers and their families, despite numerous reform efforts. To address persistent quality-of-service issues, such as late pick-up and deliveries and high claims costs for lost and damaged goods, TRANSCOM intends to award a multi-year Global Household Goods Contract, under which a single commercial move manager would oversee the movement and storage of household goods shipments. DOD has been working to award its Global Household Goods Contract in time to meet the fiscal year 2021 peak season demand, and will award the contract without precise information on the number and cost of government personnel required to counsel servicemembers and oversee the contract. Given this determination, it is particularly important that DOD put in place a process to track key data during the first 3 years of the contract to inform its planned manpower study, so that it can fully determine the cost implications of this shift in DP3. Moreover, because DOD has stated that the purpose of the move to the Global Household Goods contract is to provide better value for the customers, DOD should ensure that it has in place performance metrics to assess all DP3 activities, including those that fall outside of the Global Household Goods contract, and clearly articulate the linkage between performance metrics and overarching program goals. Without doing so, TRANSCOM will be hindered in its ability to assess whether a DP3 that incorporates the Global Household Goods Contract is an improved program, particularly as it relates to the moving experiences of servicemembers and their families.

Recommendations for Executive Action

We are making the following three recommendations to the Secretary of Defense: The Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, should ensure that the TRANSCOM Commander, in coordination with the military services and the Coast Guard, develop a process for tracking data during the first 3 years of the Global Household Goods Contract to inform its planned manpower study during the third year of the contract to more precisely determine DP3 manpower needs and associated costs. (Recommendation 1)

The Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, should ensure that the TRANSCOM Commander develop performance metrics for those DP3 activities that will not be a part of the Global Household Goods Contract, such as servicemember counseling and claims resolution that will, at least in part, continue to be performed by the military services. (Recommendation 2)

The Secretary of Defense, in coordination with the Chairman of the Joint Chiefs of Staff, should ensure that the TRANSCOM Commander articulate the linkage, where appropriate, between DP3 performance metrics, including Global Household Goods Contract performance indicators, and overarching program goals. (Recommendation 3)

Agency Comments and Our Evaluation

We provided a draft of this report to DOD for comment. In its comments, which are reproduced in Appendix III, DOD concurred with all of our recommendations and described ongoing and planned actions to address them.

We are sending copies of this report to the appropriate congressional committees; the Secretary of Defense, the Chairman of the Joint Chiefs of Staff, and the Secretaries of the Army, the Navy, and the Air Force; and the Commandant of the Marine Corps. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staff have any questions about this report, please contact me at (202) 512-2775 or fielde1@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

Our review focused on household goods movement and storage-in-transit for servicemembers in the Army, the Navy, the Air Force, the Marine Corps, and the Coast Guard. We did not evaluate Department of Defense (DOD) Defense Personal Property Program (DP3) activities that will not be a part of the Global Household Goods Contract, which include the long-term storage of household goods and the movement and storage of servicemembers’ privately owned vehicles. We did, however review metrics being tracked for these activities to determine the U.S. Transportation Command’s (TRANSCOM) ability to assess performance for the broader DP3.

During our review, we coordinated with officials from the DOD Office of the Inspector General to gain an appreciation for the objectives, scope, and methodology of their ongoing audit on the timeliness of household goods deliveries and claims resolution under the current DP3. The resulting audit report contained several recommendations, including that TRANSCOM issue warnings or letters of suspension to transportation service providers (TSPs) within 14 days of missing the agreed-upon delivery date from storage, and that TRANSCOM help servicemembers and their families file inconvenience claims with TSPs within 14 days of a missed delivery date.

To inform both of our objectives, we met with officials from the Office of the Secretary of Defense (OSD), TRANSCOM, the military services, the Coast Guard, and the DOD Office of the Inspector General. We also met with external stakeholders, including associations representing the moving and storage industry, the Small Business Administration, the American Federation of Government Employees, and members of TRANSCOM’s Personal Property Relocation Advisory Panel.

For objective one, we assessed preliminary cost estimates TRANSCOM developed in connection with the move to a DP3 approach that incorporates the planned Global Household Goods Contract against best practices in the GAO Cost Estimating and Assessment Guide. The guide states that valid and useful historical data are important in developing sound cost estimates, and that risk and uncertainty and sensitivity analyses should be performed to mitigate the effects of changing assumptions.

We analyzed TRANSCOM’s preliminary cost estimates, and discussed with TRANSCOM officials assumptions that were used to develop the estimates and techniques that were applied to account for variability in the assumptions. Given that TRANSCOM’s cost estimates were preliminary and likely to change based on the Logistics Management Institute’s (LMI) ongoing business case analysis (BCA), we did not conduct a reliability assessment of TRANSCOM’s cost estimates. We instead focused on TRANSCOM’s treatment of evolving assumptions in its preliminary cost estimates. We met with representatives from LMI to discuss their process for updating TRANSCOM’s preliminary cost estimates as part of a BCA that it was preparing for the command. We evaluated these cost estimates, including the cost estimates in the BCA that was issued in January 2020, using criteria from GAO’s Assessment Methodology for Economic Analysis that outlines five key elements for an economic analysis. Our assessment of LMI’s BCA is included in appendix II.

For objective two, we reviewed relevant DOD guidance, including DOD Instruction 4500.57, Transportation and Traffic Management. The instruction requires TRANSCOM, in coordination with the DOD components, to annually evaluate the effectiveness of the DP3. We also reviewed the Global Household Goods request for proposals and TRANSCOM’s draft quality assurance surveillance plan. The draft quality assurance surveillance plan outlines how DOD will oversee the move manager’s performance under the Global Household Goods Contract and describes the performance indicators that will be used to assess contractor performance. Using the Standards for Internal Control in the Federal Government, which calls for management to define objectives (or metrics) in terms of what is to be achieved, who is to achieve it, how it will be achieved, and the time frames for achievement so that performance toward achieving those objectives can be assessed, we also assessed DOD’s DP3 performance metrics. Additionally, we discussed with TRANSCOM officials their approach for overseeing the Global Household Goods Contract, including the linkage between performance measures for those DP3 activities that will and will not be a part of the contract and DP3’s overarching goals.

We conducted this performance audit from May 2019 to April 2020 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO’s Assessment of the Logistics Management Institute’s (LMI) Business Case Analysis (BCA)

Appendix II: GAO’s Assessment of the Logistics Management Institute’s (LMI) Business Case Analysis (BCA)

We assessed the cost and benefit information in LMI’s BCA against our Assessment Methodology for Economic Analysis. Our assessment methodology identifies the following five key components of an economic analysis: We found that the BCA informs decision-makers and stakeholders, with caveats, about the economic effects of the proposed Global Household Goods Contract. Further, we found that the BCA fully met four and partially met one of the five key components of an economic analysis. A summary of our rationale for these assessments is outlined in table 2.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

GAO Contact:

Elizabeth A. Field, (202) 512-2775 or fielde1@gao.gov.

Staff Acknowledgments:

In addition to the contact named above, GAO staff who made key contributions to this report include Marc Schwartz, Assistant Director; Pedro Almoguera, John Bumgarner, William Cordrey (retired), Tim DiNapoli, Jennifer Echard, Christopher Gezon, Mae Jones, Jason Lee, Ned Malone, Tara Porter, Oliver Richard, Mike Shaughnessy, Susan Tindall, Nate Tranquilli, John Van Schaik, and Mary Weiland.