Page 1 GAO-05-715R Defense Transportation 

June 9, 2005 

The Honorable John Warner 

Chairman 

The Honorable Carl Levin 

Ranking Minority Member 

Committee on Armed Services 

United States Senate 

The Honorable Duncan L. Hunter 

Chairman 

The Honorable Ike Skelton 

Ranking Minority Member 

Committee on Armed Services 

House of Representatives 

Subject: Defense Transportation: DOD Has Adequately Addressed 

Congressional Concerns Regarding the Cost of Implementing the New 

Personal Property Program Initiatives

Military personnel and their families can expect to relocate many times 

during a servicemember’s career. As the moving industry’s single largest 

customer, the Department of Defense (DOD) spends more than $1.7 billion 

annually for its personal property program, which provides household 

goods transportation and storage services for military personnel and their 

families when they relocate. The program manages more than 600,000 

personal property shipments each year. 

For more than 10 years, DOD has been pursuing various initiatives for 

improving the quality of its personal property program. In June 2002, the 

U.S. Transportation Command completed an extensive study that 

compared the features of the current personal property program with 

three pilot programs that tested alternative approaches for improving the 

current program. In November 2002, DOD issued a report to Congress that 

included three recommended program improvement initiatives resulting 

from this study and estimated that an additional 13 percent increase over 

United States Government Accountability Office

Washington, DC 20548 
Page 2 GAO-05-715R Defense Transportation 

current program costs would be required to implement two of these 

initiatives.1

In April 2003, we reported on the pilot program evaluation and stated that 

the recommendations contained in DOD’s November 2002 report offered 

solutions to long-standing problems in the personal property program and 

should be implemented within budget constraints.2

 However, we raised 

concerns about whether the two recommendations related to the claims 

and contracting processes could be implemented within the projected 13 

percent cost estimate. Our concerns were partially based on DOD not 

adequately substantiating the expected economies of scale that supported 

its assumption of a 5 percent reduction in average prices under the pilot 

program when a full-scale program is implemented. As a result, we 

recommended that DOD quantify the risk associated with the cost 

estimate before DOD’s recommended initiatives are implemented. 

In its May 2004 report on the National Defense Authorization Act for Fiscal 

Year 2005,3

 the House Armed Services Committee directed the Secretary of 

Defense to reevaluate DOD’s proposed cost estimate, quantify the risk or 

likelihood of achieving its goals within the 13 percent projected cost 

increase, and develop a range of possible cost increases associated with 

the risk. The committee also directed GAO to review and report on 

whether DOD adequately performed these tasks. 

To conduct our work, we reviewed prior DOD and GAO reports on the 

personal property program and interviewed DOD officials and officials 

from DOD’s contractor who were involved in the current cost reevaluation 

study. We assessed the methodologies used by DOD to reevaluate the cost 

estimate and quantify the risk associated with implementing the proposed 

initiatives within the cost estimate, including establishing ranges of 

possible cost increases. We also assessed the reasonableness of the 

methodologies used by DOD to perform these tasks. We performed our 

review from November 2004 through May 2005 in accordance with 

generally accepted government auditing standards. The scope and 

 1

 Department of Defense, U.S. Transportation Command Personal Property Pilot 

Programs Evaluation Report (Washington, D.C.: November 2002). 

2

 GAO, Defense Transportation: Monitoring Costs and Benefits Needed While 

Implementing a New Program for Moving Household Goods, GAO-03-367 (Washington, 

D.C.: Apr. 18, 2003). 

3

 H.R. Rep. No. 108-491, at 298 (2004). 
Page 3 GAO-05-715R Defense Transportation 

methodology we used in our review are described in further detail in 

enclosure I. 

The methodology used by DOD to address congressional concerns—-

reevaluate its cost estimate, quantify the risk associated with achieving the 

13 percent cost increase, and establish a range of possible cost increases—

was reasonable for conducting such assessments. In its reevaluation of the 

cost estimate, DOD validated the current mix of household goods 

shipments to ensure that the mix used in the original assessment was 

appropriate. DOD also provided additional support for its assessment of 

the anticipated savings that would result from expanding the pilot to all of 

DOD. Furthermore, the results of DOD’s risk-based simulation analysis, 

which included establishing ranges of possible cost increases, indicated 

that about 80 percent of the time, DOD could implement the recommended 

initiatives to improve the current program with an increase of 15 percent 

or less above current program costs. 

DOD concurred with the content of this letter. 

DOD’s personal property program is managed centrally by the Military 

Surface Deployment and Distribution Command, formerly known as the 

Military Traffic Management Command.4

 DOD has experienced longstanding problems with its current personal property program, including 

excessive loss of or damage to property, high claims costs incurred by the 

government, and poor quality of service from moving companies. 

Moreover, the program’s data management system does not provide 

reliable information on the status of individual shipments or on the types 

of shipments and their costs. 

In its November 12, 2002, report to Congress, DOD made three 

recommendations aimed at improving its current personal property 

program.5

 The three recommendations were to (1) reengineer the liability 

and claims process by adopting commercial practices of minimum 

 4

 The Military Surface Deployment and Distribution Command, a component of the U.S. 

Transportation Command, is the DOD executive agent responsible for managing the 

relocation process for servicemembers and their families. 

5

 Department of Defense, U.S. Transportation Command Personal Property Pilot 

Programs Evaluation Report.

Results in Brief 

Background 
Page 4 GAO-05-715R Defense Transportation 

valuation, simplifying the filing of claims, and providing direct settlement 

with the carrier; (2) change the acquisition process to implement 

performance-based service contracts; and (3) implement information 

technology improvements, which could interface functions across such 

areas as personnel, transportation, financial, and claims. DOD reported 

that the estimated cost of implementing its information technology 

improvements recommendation would be from $4 million to $6 million, 

and estimated that the cost of implementing the claims process and 

performance-based service contract recommendations would require an 

additional 13 percent over the current personal property program’s costs. 

In our April 2003 report, we stated that the three recommendations 

contained in DOD’s report were supported by the Transportation 

Command’s evaluation of the pilot programs’ findings and offered 

solutions to the long-standing problems that had plagued the current 

program for many years.6

 Our review showed that the soundness of the 

methodologies DOD used to develop cost estimates for implementing the 

three recommendations varied. We found that the estimates DOD reported 

to Congress might understate the total initial cost for implementing the 

information technology improvements recommendation and contained a 

questionable adjustment for costs associated with the claims and 

contracting process recommendations. 

In its original cost assessment for the recommendations related to the 

claims and contracting processes, DOD made three adjustments to the 

average costs for the pilot programs: (1) reducing the average weight of 

shipments, (2) reducing costs to adjust for a mix of small and large 

businesses, and (3) reducing the pilot programs’ costs to reflect 

anticipated savings based on “economies of scale.” In our April 2003 

report, we agreed that the first two adjustments were reasonable; 

however, we questioned the extent to which the third adjustment could be 

achieved. Part of our concern was based on DOD not adequately 

substantiating the expected economies of scale that supported its 

assumption of a 5 percent reduction in the average prices under the pilot 

programs when a full-scale program is implemented. Since DOD had not 

quantified the risk associated with its projected implementation cost 

estimate of a 13 percent increase over the current program’s cost, the 

military services and Congress lacked information needed to develop and 

review future budget requests for the program. Therefore, we questioned 

 6 GAO-03-367. 
Page 5 GAO-05-715R Defense Transportation 

the extent to which the recommendations could be implemented within 

DOD’s projected cost estimate. 

As a result of our findings, we recommended that DOD take the following 

actions to improve its personal property program: (1) initiate actions that 

will implement the recommendations contained in DOD’s report to 

Congress within budget constraints, (2) quantify the risk associated with 

achieving implementation of the recommended initiatives within the 

projected 13 percent cost estimate, (3) monitor costs during the 

implementation phase to ensure that the proposed changes are being 

achieved within an acceptable and a predefined range, and (4) assess the 

personal property program after the recommendations have been 

implemented to determine whether anticipated improvements are being 

achieved at a reasonable cost. 

DOD partially concurred with our second recommendation—-to provide 

the military services and Congress with additional information to quantify 

the risk associated with achieving implementation of the recommended 

initiatives within the projected 13 percent cost increase. DOD believed it 

could incorporate the two recommendations into a new program within its 

proposed 13 percent cost estimate due to the conservative approach it 

took in developing the estimate. However, in May 2004, the House Armed 

Services Committee, in its report on the National Defense Authorization 

Act for Fiscal Year 2005, directed DOD to reevaluate its proposed cost 

estimate, quantify the risk or likelihood of achieving its goals within the 13 

percent cost projection, and develop a range of possible cost increases 

associated with that risk. DOD issued a report addressing these 

congressional concerns on March 29, 2005.7

In addressing congressional concerns, DOD used a reasonable 

methodological approach to adequately reevaluate its cost estimate and 

quantify the risk associated with implementing the proposed personal 

property program initiatives within the 13 percent cost increase, including 

establishing a range of potential cost increases. DOD’s analysis included 

three key components: (1) an assessment of any changes in the 

distribution of shipments among the services and the mix of continental 

and overseas household goods shipments, (2) new support for its 

 7

 Department of Defense, Reevaluation of Cost Estimate for DOD Families First Program 

(Washington, D.C.: March 2005). 

DOD Adequately 

Addressed 

Congressional 

Concerns 
Page 6 GAO-05-715R Defense Transportation 

assessment of anticipated savings through “full-program efficiencies” 

rather than “economies of scale,” and (3) a simulation based on differing 

estimates of “full-program efficiencies” to determine the likelihood that 

DOD could implement its initiatives within its cost estimate. We believe 

that each of these components was necessary to reassess the 

reasonableness of DOD’s previous cost estimate. 

DOD first validated the current mix of household goods shipments to 

ensure that the mix used in the original assessment was appropriate. 

DOD’s reevaluation assessed changes, if any, in the distribution of 

shipments among the services and the mix of continental and overseas 

U.S. household goods shipments. Because the original cost estimate was 

based on household goods shipments picked up and delivered during the 

last half of fiscal year 2001, DOD needed to determine if the mix of 

household goods shipments had changed. Using data from its management 

information system, the Transportation Operational Personal Property 

Standard System, DOD found some minor differences in shipments among 

the services from the baseline year (fiscal year 2001) to fiscal year 2004, 

but concluded that the differences were insignificant and would have no 

effect on the calculations developed based upon the fiscal year 2001 data. 

This was a reasonable methodological approach to validate the mix of 

shipments. 

DOD then used a reasonable methodology to provide additional support 

for its assessment of the anticipated savings that would result from 

expanding the pilot to all of DOD. In its reassessment of the cost estimate, 

DOD reported that the anticipated savings could be better described as 

“full-program efficiencies” instead of “economies of scale.” For example, 

DOD’s reevaluation explained that prospective participants in the pilot 

programs faced limitations inherent in the pilot programs in a competitive 

market because they could not anticipate the longevity of the pilot 

program nor accurately predict the volume of potential shipments. 

Potential participants in the pilot programs also faced additional 

uncertainties that included not knowing their daily or monthly volume for 

this new program, having to project rates into option years without 

economic price adjustment, and facing possible penalties for withdrawing 

from the program. Consequently, the rates submitted by prospective 

participants reflected these uncertainties and other risk factors. It could 

reasonably be expected that as experience under the new program is 

gained and a steady-state program is reached, these risks would be 

reduced and would be reflected in lower rates. We believe that with this 

detailed description of the full-program efficiencies, DOD has provided 
Page 7 GAO-05-715R Defense Transportation 

reasonable additional support for the anticipated savings of the program, 

thereby reaffirming the 5 percent efficiency gains it previously reported. 

Finally, DOD used a risk-based simulation approach to provide a range of 

possible cost increases within statistical confidence intervals around the 

cost estimate, which is a reasonable methodology. This simulation 

quantified the risk associated with implementing the proposed initiatives 

within the 13 percent cost estimate. The results of this risk analysis 

indicated that about 80 percent of the time, DOD could implement the 

recommended initiatives to improve the current program within an 

increase of 15 percent or less above current program costs. All simulations 

used by DOD involved 10,000 iterations of six different cases of variation 

in the full-program efficiencies estimate to assess how different levels of 

full-program efficiencies affected the cost estimates.8

 Table 1 summarizes 

DOD’s simulation results for each scenario. As the table shows, only case 

two would result in 80 percent of all estimates being 12.5 percent or less—

an amount lower than the original cost estimate. The remaining five 

simulations of the random effect of the full-program efficiencies contain 

the original 13.4 percent estimate within 90 percent confidence intervals. 

Table 1: Simulation Results Showing the Effect of Varying Program Efficiency Distributions on Achieving the Projected Cost 

Estimate 

Program efficiency estimate 

90% confidence interval for cost 

estimate 

Case Low value High value Most frequent value 80th percentile Low value High value

1 5% 5% 5% 14.4% 11% 15%

2 0% 20% 5% 12.5% 4% 15%

3 0% 15% 5% 13.8% 7% 16%

4 0% 20% 2.5% 13.7% 4% 16%

5 0% 15% 2.5% 14.9% 8% 17%

6 0% 10% 5% 15.1% 10% 17%

 Source: GAO analysis of DOD data. 

We reviewed the results of the simulation to assess its rigor and how well 

the simulation reflects the household goods shipment costs. Based on our 

 8

 In its report, DOD noted that the advantage of the probability-based simulation technique 

used to conduct its risk assessment is that it can account for multiple input costs that have 

variability in risk in relationship to how they affect the overall cost estimate. 
Page 8 GAO-05-715R Defense Transportation 

review, a briefing of the methodology described in the report, and our 

knowledge of the analytical approaches for conducting such an 

assessment, we believe that DOD used a reasonable methodology that is 

consistent with professional standards to quantify the risk associated with 

implementing the program improvement initiatives within its cost 

estimate, because DOD ran a simulation that is a reasonable approach for 

assessing risk, using a commercially available program. See enclosure II 

for a detailed summary of the results of DOD’s risk assessment. 

DOD concurred with the content of this letter. DOD’s letter is included in 

enclosure III. 

We are sending copies of this report to the appropriate congressional 

committees; the Secretary of Defense; the Commander, U.S. 

Transportation Command; and the Director, Office of Management and 

Budget. We will also make copies available to others upon request. In 

addition, the report will be made available at no charge on the GAO Web 

site at http://www.gao.gov. 

Please contact me at (202) 512-8365 or solisw@gao.gov if you or your staff 

have any questions concerning this report. Contact points for our Offices 

of Congressional Relations and Public Affairs may be found on the last 

page of this report. Key contributors to this report were Jacqueline S. 

McColl, Arthur L. James, Jr., Charles W. Perdue, Karen N. Harms, Renee S. 

Brown, and Ann Borseth. 

William M. Solis 

Director, Defense Capabilities and 

 Management 

Enclosures 

Agency Comments 
Page 9 GAO-05-715R Defense Transportation 

To assess the methods used by the Department of Defense (DOD) to 

reevaluate the cost estimate and quantify the risk associated with 

implementing the program improvement initiatives within the stated cost 

projection, including establishing a range of possible cost increases, we 

reviewed the cost projection methodology used by DOD in its report on 

the reevaluation of the projected cost estimate. We also met with DOD 

officials and its contractor officials to discuss the methodology used to 

perform the cost reevaluation and risk assessment. 

To assess the reasonableness of the methodology used by DOD to perform 

these tasks, we compared DOD’s reevaluation methodology to analytical 

approaches applicable for this type of evaluation, assessed adjustments 

made in the evaluation strategy to address issues that could affect the 

validity of the results, and reviewed the evaluation techniques used to 

analyze data. We reviewed the results of the simulation DOD used to 

conduct its risk assessment to assess its rigor and how well the simulation 

reflects the household goods shipment costs. We also reviewed the 

contents of the DOD contractor’s briefings on the planned approach and 

evaluation techniques used to reevaluate the projected 13 percent cost 

estimate. 

We did not assess whether the anticipated benefits to be derived from 

implementing the three recommendations would warrant the additional 

costs that DOD estimates will be required to fund these improvements. 

Furthermore, we did not independently test the reliability of data DOD 

extracted from its data system to develop costs. 

During this review of DOD’s evaluation efforts, we met with officials and 

obtained documents from the Office of the Assistant Deputy Under 

Secretary of Defense (Transportation Policy), Washington, D.C.; the U.S. 

Transportation Command, Scott Air Force Base, Illinois; the Surface 

Deployment and Distribution Command, Alexandria, Virginia; and LMI 

(Surface Deployment and Distribution Command contractor), McLean, 

Virginia. In addition to these agency meetings and documents, we drew 

upon information contained in a previous GAO report resulting from our 

prior review of this program. 

Our work for this review was performed from November 2004 through 

May 2005 in accordance with generally accepted government auditing 

standards. 

Enclosure I: Scope and Methodology 
Enclosure II: Summary of DOD’s Risk 

Assessment 

Page 10 GAO-05-715R Defense Transportation 

DOD’s methodology to quantify the risk associated with the cost estimate 

included using a commercially available program to run a Monte Carlo 

simulation,1

 which is a reasonable methodological approach for assessing 

risk. The simulation contained six cases of variation in the full-program 

efficiencies estimate to assess how these full-program efficiencies affected 

the cost estimates. Case one was the baseline case involving the original 

13.4 percent increase resulting from an assumption of a fixed 5 percent 

reduction for full-program efficiencies. This case represents the original 

estimates provided by DOD in its November 2002 report to Congress. The 

remaining five cases randomly varied the assumption for the full-program 

efficiencies over five different ranges, from a very compact range between 

zero and 10 percent to a very diverse range from zero to 20 percent. 

Two cases in particular illustrated the effects of the simulation. Case six, 

the very compact case, allowed the frequency of the full-program 

efficiencies estimate to increase from zero smoothly to 5 percent and 

decline smoothly to 10 percent. The result of this simulation was very 

similar to the base case (case one, in which the original estimate of fullprogram efficiencies was held equal to 5 percent). The results indicate that 

90 percent of the time the cost estimate would fall from 10 to 17 percent, 

with 80 percent of all estimates less than 15.1 percent. 

The most “generous” case, case two, was based upon the assumption of 

the frequency for full-program efficiencies increasing smoothly to 5 

percent and declining smoothly to 20 percent. This case is most generous 

because it assumes that large cost savings (up to 20 percent) are 

achievable even though the most likely cost savings would be 5 percent. 

The results indicate that 90 percent of the time the cost estimate would fall 

from 4 to 15 percent, with 80 percent of all estimates less than 12.5 

percent. This was the one case where the original estimate was outside the 

80th percentile and thus a lower cost estimate was determined to be most 

likely. However, the case was most generous in its assumption of the 

frequency with which “large” full-program efficiencies would occur and 

 1

 A Monte Carlo simulation is a method of evaluating hypotheses by developing a computer 

model of a process, defining the parameters of the process to reflect the “real world” 

situation, calculating multiple results of varying parameters through some range of values, 

and evaluating the distribution of results obtained from these samples. The Monte Carlo 

method assigns a value to the model’s parameters from a sequence of random numbers, 

runs the model multiple times through randomly differing parameter values, captures the 

outcome of each iteration of the model, and assesses the distribution of outcomes using 

standard statistical methods. 

Enclosure II: Summary of DOD’s Risk 

Assessment 
Enclosure II: Summary of DOD’s Risk 

Assessment 

Page 11 GAO-05-715R Defense Transportation 

could reasonably be expected to indicate a smaller increase in estimated 

costs over the baseline program. 

The other three cases, between the two extremes, varied the frequency of 

full-program efficiencies over different ranges and the simulation results 

contained the original 13.4 percent estimate within the 90 percent 

confidence intervals. Based on our review of DOD’s analysis, we believe 

that DOD used a reasonable approach that was consistent with 

professional standards to quantify the risk associated with implementing 

the program improvement initiatives within its cost estimate. 
Enclosure III: Comments from the Department of Defense 

Page 12 GAO-05-715R Defense Transportation 

Enclosure III: Comments from the 

Department of Defense 
Related GAO Products 

Page 13 GAO-05-715R Defense Transportation 

Defense Transportation: Monitoring Costs and Benefits Needed While 

Implementing a New Program for Moving Household Goods. GAO-03-367. 

Washington, D.C.: April 18, 2003. 

Defense Transportation: Final Evaluation Plan Is Needed to Assess 

Alternatives to the Current Personal Property Program. GAO/NSIAD-00-

217R. Washington, D.C.: September 27, 2000. 

Defense Transportation: The Army’s Hunter Pilot Project Is Inconclusive 

but Provides Lessons Learned. GAO/NSIAD-99-129. Washington, D.C.: 

June 23, 1999. 

Defense Transportation: Plan Needed for Evaluating the Navy Personal 

Property Pilot. GAO/NSIAD-99-138. Washington, D.C.: June 23, 1999. 

Defense Transportation: Efforts to Improve DOD’s Personal Property 

Program. GAO/T-NSIAD-99-106. Washington, D.C.: March 18, 1999. 

Defense Transportation: The Army’s Hunter Pilot Project to Outsource 

Relocation Services. GAO/NSIAD-98-149. Washington, D.C.: June 10, 1998. 

Defense Transportation: Reengineering the DOD Personal Property 

Program. GAO/NSIAD-97-49. Washington, D.C.: November 27, 1996. 

Related GAO Products 

(350619) 
This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
The Government Accountability Office, the audit, evaluation and 

investigative arm of Congress, exists to support Congress in meeting its 

constitutional responsibilities and to help improve the performance and 

accountability of the federal government for the American people. GAO 

examines the use of public funds; evaluates federal programs and policies; 

and provides analyses, recommendations, and other assistance to help 

Congress make informed oversight, policy, and funding decisions. GAO’s 

commitment to good government is reflected in its core values of 

accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost 

is through GAO’s Web site (www.gao.gov). Each weekday, GAO posts 

newly released reports, testimony, and correspondence on its Web site. To 

have GAO e-mail you a list of newly posted products every afternoon, go 

to www.gao.gov and select “Subscribe to Updates.” 

The first copy of each printed report is free. Additional copies are $2 each. 

A check or money order should be made out to the Superintendent of 

Documents. GAO also accepts VISA and Mastercard. Orders for 100 or 

more copies mailed to a single address are discounted 25 percent. Orders 

should be sent to: 

U.S. Government Accountability Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Gloria Jarmon, Managing Director, JarmonG@gao.gov (202) 512-4400 

U.S. Government Accountability Office, 441 G Street NW, Room 7125 

Washington, D.C. 20548 

Paul Anderson, Managing Director, AndersonP1@gao.gov (202) 512-4800 

U.S. Government Accountability Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Congressional 

Relations 

Public Affairs 

PRINTED ON RECYCLED PAPER