GAO

United States General Accounting Office

Report to Congressional Committees

June 1999 DEFENSE 

TRANSPORTATION

The Army’s Hunter 

Pilot Project Is 

Inconclusive but 

Provides Lessons 

Learned

GAO/NSIAD-99-129

Page 1 GAO/NSIAD-99-129 Defense Transportation

United States General Accounting Office

Washington, D.C. 20548 Leter

National Security and

International Affairs Division

B-282406 Letter

June 23, 1999

Congressional Committees 

The Department of the Army began a pilot project in July 1997 to test an 

alternative approach for providing relocation services for its military 

personnel stationed at Hunter Army Airfield, Savannah, Georgia. The Army 

undertook this effort to address long-standing concerns and problems 

associated with the current personal property program. The Department of 

Defense (DOD) has two other pilot programs underway to test different 

approaches to improving its personal property program and is proposing a 

fourth pilot. To determine which pilot, or portions thereof, could provide 

better long-term results, the Deputy Under Secretary of Defense (Logistics) 

tasked the U.S. Transportation Command (USTRANSCOM) to oversee all 

personal property pilot tests and recommend the follow-on course of 

action.

The statement of managers in the conference report on the 1997 DOD 

Appropriations Act directed us to validate the results and savings achieved 

before DOD expands any of its personal property pilot projects.1 In this 

regard, we testified in March 1999 on the status of the Army’s pilot and 

DOD’s efforts to improve its personal property program.2 This report 

supplements our testimony and principally addresses the results of our 

review of the 12-month Hunter pilot test and lessons for evaluating other 

pilots. Our objectives were to (1) assess the Army’s evaluation 

methodology of the Hunter pilot, including the validity of data and reported 

results and (2) determine the status of all ongoing and planned pilot 

projects and the adequacy of DOD’s plans to evaluate the pilot projects.

Results in Brief Although the Army reported that the Hunter pilot was a success, we found 

that most of the results of its evaluation were inconclusive. While the 

Hunter pilot provides services and benefits that were not previously 

available during the moving process, we were unable to validate all 

1House Report 104-863 (Sept. 28, 1996) p. 865.

2Defense Transportation: Efforts to Improve DOD’s Personal Property Program (GAO/T-NSIAD-99-106, 

Mar. 18, 1999).

Letr
B-282406

Page 2 GAO/NSIAD-99-129 Defense Transportation

reported results of the Army’s evaluation of this pilot because of 

weaknesses in the evaluation methodology and data. Specifically, because 

of the methods used in conducting the customer surveys, we could not 

confirm that customer satisfaction improved. Also, due to weaknesses in 

the Army’s methodology and data reliability, we could not validate the 

extent to which pilot costs exceeded baseline costs. However, we were 

able to confirm that 33 percent of the pilot shipments were awarded to 

small business carriers and agents. Further, lessons learned by the Army in 

developing an evaluation plan, conducting the pilot test, and evaluating 

results can provide useful information to DOD as it conducts and assesses 

other pilot efforts.

The Navy and the Military Traffic Management Command (MTMC) each 

have a personal property pilot project underway, and DOD is proposing a 

fourth pilot to test different approaches to improve its personal property 

program. As a result, DOD will be running multiple pilots concurrently, 

with different goals, objectives, and expected outcomes. USTRANSCOM3 is 

tasked with evaluating the results of the pilots and using that information 

to recommend a redesigned Department-wide relocation program. 

However, DOD has not yet determined how many approaches will 

ultimately be tested and the milestones for completing the pilots’ 

evaluation and implementing an improved process, nor has it assured itself 

that a methodologically sound evaluation process is in place to execute this 

process.

Improving DOD’s personal property program has been a slow, complex 

process. DOD and the services have spent a large amount of time and effort 

to dramatically change the quality of services military personnel receive. 

We support pilots as a tool to test different approaches. However, before 

the Department can make any credible decisions on changing the current 

program, we recommend that the Secretary of Defense direct 

USTRANSCOM to develop a comprehensive strategy for evaluating each of 

the pilots’ attributes in a comparable manner. Further, on the basis of the 

Army’s lessons learned in conducting the Hunter pilot and of our evaluation 

of the pilot, we recommend that the Department seek expert 

methodological advice to enhance the quality of its assessment.

3The mission of USTRANSCOM, which is DOD’s single manager of all Defense transportation services, 

is to provide global air, land, and sea transportation to meet national security needs. USTRANSCOM 

executes its mission through three component commands: MTMC for land transportation and port 

operations, the Military Sealift Command for sea transportation, and the Air Mobility Command for air 

transportation.

Letr
B-282406

Page 3 GAO/NSIAD-99-129 Defense Transportation

Background DOD has long been concerned about the quality of service it provides 

military personnel and their families when they relocate. DOD spends 

approximately $3 billion annually to transport, store, and manage the 

household goods and unaccompanied baggage of its servicemembers and 

families. Past problems included poor service from movers, excessive 

incidence of loss or damage to servicemembers’ property, and high claims 

costs to the government. All of these problems contributed to poor quality 

of service for people using the system.

Most Results of the 

Army Hunter Pilot Are 

Inconclusive

Because of weaknesses in the Army’s evaluation methodology and data, we 

were unable to validate the reported results of the quality of life and cost 

factors of the Hunter pilot program. Moreover, the Army’s evaluation plan 

methodology was not an effective tool for collecting and analyzing the pilot 

results. However, lessons learned by the Army in conducting the 12-month 

pilot test do provide useful information to DOD as it conducts and assesses 

its pilot efforts. Also, the Hunter pilot is providing services and benefits 

that were not previously available, including point-to-point move 

management, personal move counseling and coordination, direct claims 

settlement, assistance in buying/selling a residence, and visibility of the 

shipment throughout the move. 

The Army’s Assessment and 

Evaluation Approach

The Army reported that through its first 12 months of operations, the 

Hunter pilot successfully demonstrated that commercial practices could be 

applied to the military relocation process. Specifically, the Army was able 

to contract with Cendant Mobility, a move management company, to 

provide services similar to those available in the private sector. In January 

1997, the Army began developing a test evaluation plan that defined roles 

and responsibilities, test factors, and processes for capturing data related 

to each test factor and stated how test data would be compared to that 

from the existing program (i.e., the baseline). The plan stated that the Army 

Audit Agency (AAA) would be responsible for validating the baseline and 

test data and for providing the Army a report on the test results. We 

reported in June 1998 that the Army needed to further define the various 

factors and measurements to be included in its evaluation plan.4 The Army 

clarified how it would measure pilot success and revised its methodology 

4Defense Transportation: The Army’s Hunter Pilot Project to Outsource Relocation Services

(GAO/NSIAD-98-149, June 10, 1998).
B-282406

Page 4 GAO/NSIAD-99-129 Defense Transportation

in the evaluation plan, dated August 12, 1998. This represented the third 

such revision, which was not completed until after the 12-month test 

period. 

The modified plan indicated that the pilot results would be reviewed on the 

basis of three factors—quality of life, cost, and impact on small business—

and would use a 5-point scoring system. The Army assigned the greatest 

weight to the quality of life factor, which consisted of one point each for 

three sub-factors—customer satisfaction, average claims settlement time, 

and percentage of direct deliveries. Along with the 3 points for quality of 

life, the Army assigned 1 point each for two other factors, total cost and 

impact on small business, for a total of 5 points. Appendix I provides 

information on the scale and scores used by the Army to evaluate the 

results of each factor and sub-factor.

In its October 1998 evaluation report and February and March 1999 

supplements, the Army reported to us that Cendant Mobility managed 

1,349 shipments involving household goods, unaccompanied baggage, 

do-it-yourself (DITY) moves, and mobile homes during the 12-month test. 

(See appendix II for the number of shipments in each category, including 

domestic and international volumes.) The Army also reported that the pilot 

project earned a passing score on each of the test factors that the Army 

measured and deemed the Hunter pilot a success because the factor points 

exceeded the minimum threshold of 3 based on a 5-point scale. According 

to the Army’s analysis, the Hunter pilot test earned 3.75 points as follows: 

• customer satisfaction improved by over 11 percent (0.25 point);

• the initial offer to settle claims averaged 9 days (1 point);

• 100 percent of eligible shipments were delivered directly to the 

servicemember without storage in transit (1 point);

• pilot project costs exceeded baseline costs by 18.6 percent (0.50 point); 

and

• 33 percent of the shipments were awarded to small businesses (1 point).

The Army stated that it also considered other factors generally related to 

transportation process improvements and did not rely on test scores alone 

to determine the pilot a success. These factors included simplifying the 

price structure and claims process, providing pre-audit services, and 

minimizing government-unique requirements. Each servicemember worked 

with one personal move coordinator who integrated the member’s move, 

providing point-to-point move management, coordination, and prompt 

resolution of all problems. This was in contrast to the current process, in 
B-282406

Page 5 GAO/NSIAD-99-129 Defense Transportation

which a servicemember may need to contact four or five different offices 

over the course of a single move. Neither AAA nor we reviewed these cited 

benefits because they were not included as measurable factors in the 

evaluation plan.

Evaluation Weaknesses Led 

to Inconclusive Results

We identified a number of shortfalls in the Army’s evaluation methodology 

and data. Specifically, the methodology for surveying customer satisfaction 

and calculating claims settlement time, percentage of direct deliveries 

made, and overhead costs did not allow the Army to obtain and analyze the 

data needed to accurately assess pilot results. Furthermore, in several 

instances the data collected to support the measurements did not 

demonstrate achievement of the Army’s reported results. Most importantly, 

we could not confirm that customer satisfaction improved by 11.5 percent. 

Further, we could not confirm that all eligible shipments were delivered 

directly without incurring in-transit storage costs, offer of claims 

settlement time averaged 9 days, and the pilot program cost 

18.6 percent more than the baseline cost. On the other hand, we were able 

to confirm that 33 percent of the shipments were awarded by Cendant 

Mobility to small business carriers and agents—10 percent over the Army’s 

minimum threshold.

Inconclusive Results for 

Customer Satisfaction

The Army’s methodology for evaluating customer satisfaction results was 

flawed as it allowed servicemembers to be surveyed multiple times—by 

Cendant Mobility, two survey contractors, and carriers. In addition to the 

use of different populations and respondent rates, the various survey 

instruments also contained different questions used to measure the level of 

customer satisfaction. Therefore, we could not validate that customer 

satisfaction improved by 11.5 percent, as reported by the Army, on the basis 

of the conclusions reached by one of the four survey instruments. 

Notwithstanding these concerns, other data gathered in these surveys but 

not scored by the Army in accordance with its evaluation plan, provided 

indicators of customer satisfaction as well as lessons learned that may be 

useful to DOD for conducting other pilots.

To measure customer satisfaction, the Army directed its contractor 

(Battelle’s Pacific Northwest National Laboratory) to survey members at 

Hunter and those making similar moves at Fort Stewart, Georgia (the 

baseline). Responses from servicemembers at both locations would be 

compared. Subsequently, the Army decided to use USTRANSCOM’s survey, 

conducted by Electronic Data Systems (EDS), to evaluate and score the 

pilot project.
B-282406

Page 6 GAO/NSIAD-99-129 Defense Transportation

Nonetheless, servicemembers were surveyed multiple times, possibly by as 

many as four different entities. EDS and Battelle (the survey contractors), 

Cendant Mobility (which was contractually required to survey a sample of 

customers), and the carrier (for its own quality control purposes) each may 

have contacted the same servicemembers to query them on their move 

experiences. As a result, the customer satisfaction results were 

compromised and therefore inconclusive. EDS reported that some 

servicemembers refused to respond to the survey and that an 

undetermined number who did respond gave perfunctory answers. In a 

caveat in its report, Battelle stated that it was aware that some individuals 

were interviewed multiple times.

The Army informed us that it did not use Battelle’s results because the 

Army found the draft report difficult to interpret and inconclusive. The 

Army also had concerns about a disclaimer on the survey results.5

Moreover, the contractors’ survey results varied significantly. While EDS 

reported that 11.5 percent more of the Hunter respondents were satisfied 

with their current relocation experience than the Fort Stewart 

respondents, Battelle reported only a 3-percent difference between the two 

groups of respondents.6

 We cannot explain all the reasons for these 

differences. We do know that the questions asked by each contractor to 

determine customer satisfaction were not identical and that the population 

and number of respondents in the two surveys varied significantly. EDS 

also had a much lower response rate (53 percent) than Battelle 

(89 percent). Further, EDS used a 75-percent significance level, while 

Battelle based its results using a 95-percent significance level. The latter is 

the standard used in social science research.7

Other data obtained by the surveys but not scored by the Army, in 

accordance with its evaluation plan, provides indicators of customer 

5The Army later learned that the Department of Energy requires such a disclaimer in all studies 

performed for government entities and that the disclaimer had no relevance to the results reported by 

Battelle.

6Battelle also reported that the 3-percent figure was “statistically insignificant,” meaning that the 

variance in responses from the two groups (personnel at Hunter and at Fort Stewart) was not great 

enough to conclude that the move experiences differed significantly.

7Significance level is the likelihood that a true population value may be rejected by a statistical test. In 

the case of the two surveys, Battelle used a 95-percent and EDS used a 75-percent probability that the 

differences in relocation satisfaction levels were due to something other than random occurrence. 

Battelle found no significant difference in satisfaction levels, with a 5-percent chance of reaching an 

incorrect conclusion. EDS did find a significant difference but with a 25-percent chance of reaching an 

incorrect conclusion.
B-282406

Page 7 GAO/NSIAD-99-129 Defense Transportation

satisfaction and highlight areas that could be addressed in future 

evaluations. For example, the Battelle survey found that 61 percent of the 

Hunter respondents were more satisfied with their most recent (pilot) 

moving experience than with their prior move. Eighty-seven percent of 

these respondents were satisfied or very satisfied with the carrier’s 

responsiveness. Both surveys also indicated that servicemembers 

relocating within the continental United States were generally more 

satisfied with the pilot process than were personnel moving overseas.

The surveys also showed that servicemembers liked some unique features 

the pilot offered such as one-on-one counseling and the simplified claims 

process. However, few servicemembers took advantage of many of the 

additional services that Cendant Mobility offered, particularly assistance in 

house hunting. Cendant Mobility reported to us that its referrals to real 

estate companies resulted in the sale of one residence in the Hunter area 

and the purchase of five residences at new duty stations. Moreover, 

officials of The Military Coalition, an association that represents 5 million 

active duty, retired, reserve, and National Guard personnel and their 

families, told us that more could be done to tailor relocation services to 

meet unique military needs. Such services include assistance in finding 

rental property, because military personnel tend to rent rather than 

purchase, and assistance for spouses seeking employment.

One of the lessons learned is related to access to survey data by outside 

reviewers. AAA was unable to compare respondent results across surveys 

because the survey contractors, citing privacy act restrictions, did not grant 

access to identifier documentation. The Army stated that in future pilot 

projects it would ensure that servicemembers are not subjected to multiple 

surveys and would establish a better survey method for determining 

customer satisfaction.

Limited Data on Claims 

Settlement Time

Due to weaknesses in the Army’s data collection and analysis, we could not 

validate that the initial offer to settle claims took an average of 9 days after 

a completed claim form was received by Cendant Mobility. Furthermore, 

the Army’s methodology was limited to examining the time between filing a 

completed claim form and making the initial settlement offer. This measure 

does not capture the entire claims process and therefore does not measure 

the extent to which it represents an improvement over the existing system. 

The pilot process, however, does offer benefits to the servicemember in 

terms of full replacement value protection. Household effects are insured 

for their full replacement value (up to $75,000) rather than their 

depreciated value (which is capped at $40,000 under the existing system). 
B-282406

Page 8 GAO/NSIAD-99-129 Defense Transportation

The pilot process also requires less work from the servicemember because 

the contractor provides estimates on lost and damaged household goods 

and pays the servicemember directly.

The claims settlement time measurement is based on the contractual 

requirement that Cendant Mobility offer settlement to a member within 

30 days after receiving a claim for damaged or lost property. Consequently, 

the Army developed a threshold for claims settlement of 30 days. Under the 

Hunter pilot, servicemembers could appeal to the contractor for 

reconsideration by providing additional information or file a claim with the 

Army if resolution with the contractor could not be achieved—a process 

that could take more than 2 years to complete. While we found that some 

claims remained unpaid and unsettled because the claims were put “on 

hold” by the servicemember, the Army reported that no one in the Hunter 

pilot filed a claim with the Army claims office as of January 1999.

The Army reported that Cendant Mobility took an average of 9 days to offer 

settlement for lost or damaged property, but neither AAA nor we could 

confirm this because the Army had not collected the necessary data. AAA 

determined, and we concur, that Cendant Mobility may have taken as long 

as an average of 28 days to offer settlement. This figure was based on the 

time between the date the claim was signed and the date the member was 

offered settlement. The difference between the two estimates was 

primarily due to a 19-day period between the date the claim was signed and 

the date the contractor reported entering the claim into its database. 

Although Cendant Mobility officials stated that standard company practice 

is to enter a claim into the database the same day it is received, they said 

delays could be caused by (1) the form being mailed some time after it is 

dated, (2) mail delays, and (3) contractor delays in entering the claim into 

the database. Army officials stated that, due to the difficulties in measuring 

claims settlement time, they would change their data collection and 

analysis of this sub-factor in future pilot programs.

Direct Delivery Results May Be 

Overstated

We could not confirm that all eligible direct deliveries were made without 

in-transit storage because the necessary data was not collected. The Army’s 

methodology for this sub-factor also did not allow an assessment of 

whether the results represented an improvement over the current process. 

Direct delivery is defined by the Army as delivery of a shipment to a 

servicemember’s residence without storage in transit. A shipment is eligible 

for direct delivery only if it has a destination address before the shipment is 

offered for delivery. Direct delivery is a contract performance 
B-282406

Page 9 GAO/NSIAD-99-129 Defense Transportation

measurement, and the contractor is paid an incentive for maintaining a 

direct delivery rate of over 60 percent. Consequently, the Army threshold 

for this measurement is 60 percent.

Neither AAA nor we could confirm that shipments delivered directly to 

servicemembers were the only ones eligible for direct delivery because the 

necessary information on all shipments was not collected. That is, other 

shipments may have been eligible under the Army criteria; however, AAA 

and we could not determine which shipments had addresses prior to being 

offered for delivery, and were thus eligible, but ended up requiring 

in-transit storage. The Army acknowledged the difficulty in validating 

eligible direct deliveries and stated that, in the future, it would consider 

using storage-in-transit rates as a test factor. In the February 5, 1999, 

supplemental information provided to us, the Army stated that 74 percent 

of Army-wide shipments required in-transit storage, compared with 

66 percent requiring temporary storage during the Army pilot.

Servicemembers benefit from having household goods delivered directly to 

their home because the practice limits additional handling of their property, 

reducing opportunities for loss and damage. Also, the government avoids 

temporary in-transit storage, which is costly and hard for the Army to 

control. Household goods shipments require temporary storage when 

servicemembers do not know their new address at the time of shipment, 

and/or they move to their new duty station after completing vacations or 

military requirements en route.

Some Overhead Cost Data Was 

Inaccurate and Incomplete

Due to weaknesses in the Army’s methodology for collecting overhead cost 

data and the questionable reliability of some of the overhead cost data that 

was collected, we could not validate that the additional cost of the pilot 

project was 18.6 percent higher than the baseline cost. As we reported in 

June 1998, developing overhead costs has historically been difficult for the 

government, including DOD, because such data is often unreliable and 

unavailable. (See appendix III for a detailed breakdown of pilot and 

baseline costs as reported by the Army and validated by AAA).

Neither AAA nor we could validate some overhead costs. Specifically, we 

could not confirm the accuracy of the overhead costs attributed to MTMC 

because the Army used data from fiscal year 1994. MTMC has changed 

significantly since then due to downsizing, but AAA could not obtain 

updated costs from MTMC representing the current organization to 

determine the reasonableness of this overhead cost. AAA and we also agree 
B-282406

Page 10 GAO/NSIAD-99-129 Defense Transportation

that a drop in document processing (for claims, invoices, and inbound 

shipments) would not result in cost savings to the government unless the 

activities that perform these functions, such as the Defense Finance and 

Accounting Service, take corresponding actions to reduce their costs 

(for example, by reducing personnel to reflect the workload reduction).

AAA and we also identified other costs not included in the Army’s analysis 

such as those related to awarding the Hunter contract and the contractor’s 

use of foreign-flag vessels.8 The Army could not provide an estimated 

dollar value associated with the cost of developing the contract 

solicitation, reviewing offers, and resolving bid protests. While we 

recognize that some of these estimated costs may be infrequent or one-time 

only, they should be considered. Also, the Army did not include the 

contractor’s use of foreign-flag vessels into the pilot project’s 

transportation costs. While individually, costs such as these are probably of 

low-dollar value in relation to other costs in the analysis, collectively, they 

could materially affect the difference between the baseline and pilot costs.

According to the Army, some of the reasons for the 18.6 percent higher cost 

of the pilot project can be attributed to (1) difficulty in calculating accurate 

baseline costs, which the Army believes were understated; (2) low volume 

of moves (1,349), which did not provide enough leverage to negotiate better 

rates and discounts; (3) higher than expected costs of overseas shipments; 

(4) relatively high unaccompanied baggage shipment rates, which could 

have been lower using negotiated rates; (5) packing allowance for DITY 

moves, which was calculated using commercial rates during part of the 

12-month test period, resulting in a higher rate; and (6) quality moving 

services, which cost more. In regard to the high cost of overseas shipments, 

the Army provided us with additional details of pilot project costs that 

showed a 44-percent increase for overseas household goods shipments and 

a 2-percent increase for domestic shipments over the baseline cost.

Small Business Usage Exceeded 

Army Goals

Both AAA and we validated that Cendant Mobility awarded 33 percent of 

the shipments to small businesses—10 percent to carriers and 23 percent to 

agents. The Army measured the impact of the pilot on these two types of 

providers and established a threshold of 23 percent to demonstrate 

successful participation by small businesses.9 The Army based this factor 

on section 15(g) of the Small Business Act, which establishes a 

8Foreign-flag vessels are ships registered in foreign countries.

9In the motor freight and transportation industry, firms with annual gross revenues of $18.5 million or 

less are classified as small businesses. See Federal Acquisition Regulation 19.102.
B-282406

Page 11 GAO/NSIAD-99-129 Defense Transportation

governmentwide goal for participation by small business concerns at not 

less than 23 percent of the value of all prime contracts. The Army reported 

that 261 (33 percent) of the 790 Hunter shipments billed during the 

12-month pilot were subcontracted by Cendant Mobility to small carriers 

and agents.

As we reported in June 1998, Cendant Mobility encountered problems in 

attracting small carriers and agents to participate in the Hunter pilot. When 

the pilot was implemented, several local agents and the companies they 

represented elected not to sign agreements to participate. Cendant Mobility 

reported that historically, over 40 percent of the contracts it awards go to 

small and small and disadvantaged businesses. However, as of 

December 31, 1997, only 14 percent of the shipments booked by Cendant 

Mobility during the first 6 months had been awarded to small businesses. 

By June 30, 1998, a number of national van lines and their local agents had 

reversed their decision and had signed contracts with Cendant Mobility.

No Comprehensive 

Strategy for Evaluating 

Multiple Pilots

DOD will be running multiple pilots concurrently, each with different goals 

and objectives and expected outcomes. MTMC reengineering efforts began 

in 1994 followed by the Army’s in 1996 and the Navy’s in 1997. DOD is 

proposing a fourth pilot with no specific start date. While USTRANSCOM is 

tasked with evaluating the results of these pilots and using the information 

to recommend a redesigned Department-wide relocation program, it does 

not have a comprehensive strategy for doing so. Specifically, it has not yet 

determined how many approaches it will ultimately test, it has not set the 

milestones for completing the pilots’ evaluations and implementing an 

improved process, and it has not assured itself that a methodologically 

sound evaluation process is in place to execute this process.

MTMC’s Reengineering 

Efforts Began in 1994

DOD first proposed reengineering its personal property program in 1994. 

Specifically, on June 21, 1994, USTRANSCOM directed MTMC (the Army 

component of USTRANSCOM and the program manager for DOD’s 

Personal Property Shipment and Storage Program) to reengineer the 

personal property program. On June 15, 1995, the House Committee on 

National Security,10 concurring that DOD must pursue a higher level of 

service, directed that DOD undertake a pilot program to implement 

commercial business practices and standards of service. MTMC planned to 

10Now called the House Committee on Armed Services.
B-282406

Page 12 GAO/NSIAD-99-129 Defense Transportation

award contracts for the new program pursuant to the competitive 

acquisition system provisions (10 U.S.C. ch. 137 and its implementing 

regulation in the Federal Acquisition Regulation). Expressing 

congressional concerns about the impact the competition system and any 

nonstandard commercial business practice requirements might have on 

small businesses, the statement of managers in the conference report 

accompanying the 1997 DOD Authorization Act directed that DOD report 

on the impact of the pilot program on small businesses. After reviewing the 

reports, the House Committee on National Security remained concerned 

that MTMC’s pilot program did not satisfactorily address issues raised by 

small moving companies and directed that the Secretary of Defense 

establish a working group of military and industry representatives to 

develop an alternative pilot program.

Although the working group reached consensus on many issues, including 

a set of program goals, it could not reach agreement on the approach to 

take for the pilot test. Consequently, the two sides presented separate 

proposals. In November 1996, we reported that in our assessment, MTMC’s 

proposal met the goals of reengineering the personal property program to a 

greater extent than the industry’s plan.11

The MTMC pilot program, delayed by numerous bid protests, finally started 

in January 1999.12 The pilot includes 50 percent of the moves originating in 

North Carolina, South Carolina, and Florida. The pilot will run concurrently 

with the existing MTMC-managed program at installations representing all 

military services and the Coast Guard in the three states and will involve 

approximately 18,500 annual moves. The key features of the pilot program 

include selecting carriers based on servicemember satisfaction and past 

performance rather than simply price; achieving stronger carrier 

commitment with long-term contracts; and offering full replacement value 

protection and direct claims settlement to users. MTMC’s pilot will run for 

a 3-year test period (1 year and two 1-year option periods) which will end in 

December 2002.

11Defense Transportation: Reengineering the DOD Personal Property Program (GAO/NSIAD-97-49, 

Nov. 27, 1996).

12Solicitation disputes in the following decisions: Aalco Forwarding, Inc., et al, B-277241.8, B-277241.9, 

October 21, 1997; Aalco Forwarding, Inc. et al., B-277241.12, B-277241.13, December 29,1997; 

Aalco Forwarding, Inc. et al., B-277241.15, March 11, 1998; Aalco Forwarding, Inc. et al., B-277241.16, 

March 11, 1998; Aalco Forwarding, Inc., et al., B-277241, 277241.14, June 8, 1998; and 

Aalco Forwarding, Inc., et al., B-277241, B-277241.20, B-277241.21, July 1, 1998.
B-282406

Page 13 GAO/NSIAD-99-129 Defense Transportation

Hunter Pilot Effort Began in 

1996

Separately from the MTMC pilot program, the Army decided to determine 

whether the commercial business practice of outsourcing a non-core 

function could alleviate known problems in its personal property program 

while increasing quality of life. The pilot began in February 1996 as a 

quality-of-life effort to improve the relocation process and test commercial 

business practices in a military environment at Hunter Army Airfield, 

Savannah, Georgia. On January 31, 1997, the Army selected PHH 

Relocation, a relocation/move management company (now Cendant 

Mobility, Bethesda, Maryland), as the contractor for the pilot. In July 1997, 

Cendant Mobility began relocation operations at Hunter, offering to 

distribute and manage shipments to transportation providers. Additionally, 

the contractor provided point-to-point move management, personal move 

counseling and coordination, assistance in buying/selling a residence, 

24-hour in-transit visibility of shipments, direct claims settlement with 

servicemembers, full replacement value, and a single point of contact for 

servicemembers. These services and benefits are not available under the 

existing DOD system. The current contract with Cendant Mobility will end 

on September 30, 1999.

A Cendant Mobility official stated that the management fees paid by the 

Army to the contractor pay for these services and entitlements. Cendant 

Mobility also obtained revenues from two other sources, including 

(1) commissions from carriers, agents, and freight forwarders and 

(2) referral fees from real estate companies on the purchase and sale of six 

residences. The commissions paid for services provided to carriers, agents, 

and freight forwarders, including contract training, contract 

administration, invoice auditing, and quality control services.

Navy Pilot Effort Began in 

1997

The Navy initiated its plans for a separate pilot program in 1997 to test the 

option of allowing servicemembers to select their carrier, giving them more 

control over the relocation process to meet their specific needs. The pilot, 

which began in January 1998, is intended to provide an alternative to, 

rather than a replacement of, the current program. The pilot affects a very 

small number of military personnel. Presently, the option is being offered 

only for shipments originating in the areas of Puget Sound, Washington; 

San Diego, California; Norfolk, Virginia; and Groton, Connecticut. 

Participation is limited to Navy military personnel with permanent change 

of station orders from the participating sites. Shipments must exceed 

3,000 pounds and cost between $2,500 and $25,000 (the pilot excludes 

shipments from non-temporary storage or warehouses). Servicemembers 
B-282406

Page 14 GAO/NSIAD-99-129 Defense Transportation

can select a mover from a list of carriers that are self-certified as small 

businesses and approved by MTMC. As of March 26, 1999, 223 Navy 

servicemembers had selected this option. This pilot does not have a 

specific end date.

DOD Pilot Effort Proposed 

in 1999

On February 12, 1999, DOD stated that it intended to begin a fourth pilot 

program, called the Full Service Moving Project, with certain modifications 

based on lessons learned from the Army Hunter pilot. The new pilot will 

include the National Capital Region (17,000 annual moves), Georgia 

(26,000 annual moves), and North Dakota (2,000 annual moves), and will 

involve all military services and the Coast Guard. Like the Army pilot, it is 

intended to outsource a non-core function and alleviate known problems in 

the DOD personal property program while improving the quality of life 

associated with the relocation of servicemembers and their families. While 

it is projected to begin in calendar year 1999, presently, this pilot does not 

have an official start date.

Unresolved Issues in 

Evaluating Pilot Programs

USTRANSCOM is in the process of finalizing its evaluation plan for the 

ongoing and planned pilots. It is proposing to evaluate the pilots on the 

same three factors that the Army used in its evaluation of the Hunter pilot, 

except that the factors would be defined differently and would include a 

more expansive point scale. Unresolved issues, as of February 1999, 

included the development of a method that would recognize the unique 

characteristics and/or process improvements of each pilot program and the 

validation of the baseline indirect costs that would be used for each pilot 

program. The evaluation plan does not currently include an evaluation of 

the Hunter pilot, but USTRANSCOM used lessons learned in the Hunter 

pilot to develop the plan. We provided comments to USTRANSCOM in this 

process and, among other things, encouraged the command to seek expert 

methodological advice before finalizing the evaluation plan to enhance the 

quality of its assessment.

Conclusions We support the use of pilots as a means to test new concepts. Although this 

process has been ongoing since the early 1990s, DOD is not yet in a position 

to determine what changes it needs to make to the process. Also, thus far, 

DOD has not decided how many approaches it will ultimately test and has 

not developed evaluation plans that will yield accurate and credible data. 

Lessons learned from the Hunter pilot do provide useful information for 
B-282406

Page 15 GAO/NSIAD-99-129 Defense Transportation

evaluating the three other pilots ongoing and planned. While 

USTRANSCOM is in the process of developing a strategy to conduct this 

evaluation, it has not yet resolved a number of issues, including timelines 

for completing the analysis and implementing an improved process and a 

methodology for the evaluation.

Recommendations We recommend that the Secretary of Defense direct the Commander in 

Chief, USTRANSCOM, to develop a comprehensive strategy for evaluating 

each of the pilot approaches that includes (1) tests of a finite number of 

approaches; (2) timelines for completing the analysis and for implementing 

a new personal property process; and (3) milestones for finalizing a 

methodologically sound evaluation plan, including specific criteria for 

assessing each pilot’s attributes and validating the baseline indirect costs to 

be used for each pilot program. To facilitate this process and to enhance 

the quality of the assessment, we also recommend that DOD seek expert 

methodological advice before finalizing the evaluation plan.

Agency Comments and 

Our Evaluation

In written comments on a draft of this report, DOD stated that it generally 

concurred with the report and its recommendations. DOD tasked 

USTRANSCOM to evaluate the results of all the pilots and provide a 

recommendation for a DOD-wide program to implement for the long term. 

DOD stated that USTRANSCOM is in the process of retaining a contractor 

to independently validate success criteria for a reengineered DOD personal 

property program, develop a comprehensive strategy for evaluating each of 

the pilot approaches, and support the development of recommendations 

and alternative courses of action. USTRANSCOM anticipates having the 

contractor commence its efforts by June 1999.

DOD’s comments are presented in appendix IV. DOD also provided 

technical comments and we revised our report to reflect them where 

appropriate.

Scope and 

Methodology

To validate the results and savings of the Hunter pilot, we reviewed the 

Army’s plans to determine how it would evaluate the pilot’s results. We also 

reviewed the Army’s October 1998 report and the February and March 1999 

supplements detailing the 12-month test results. We visited both Hunter 

Army Airfield and the transportation division at Fort Stewart, Georgia. We 

contacted Hunter contractor officials and representatives from the two 
B-282406

Page 16 GAO/NSIAD-99-129 Defense Transportation

survey firms hired by the Army and USTRANSCOM to measure customer 

satisfaction. We also examined the documents and data compiled by the 

AAA to validate the accuracy of the Hunter results.

We met with Army, Navy, MTMC, and USTRANSCOM officials and 

reviewed documents to determine the status and goals of other DOD 

personal property pilot programs. We also obtained statistics on actual use 

by servicemembers of the Navy’s pilot as of March 1999. We did not request 

statistics on other pilots because the MTMC pilots began in January 1999 

and the DOD-proposed Full Service Moving Project is still in the planning 

stage. We also met with USTRANSCOM officials and reviewed evaluation 

plans to determine how DOD proposes to make changes to its current 

personal property program using lessons learned from the pilots.

We met and discussed matters related to DOD pilots and efforts to improve 

the existing personal property program with officials from the Office of the 

Deputy Under Secretary of Defense (Logistics-Transportation Policy), 

Washington, D.C.; USTRANSCOM, Scott Air Force Base, Illinois; the Office 

of the Army Deputy Chief of Staff for Logistics, Washington, D.C.; MTMC, 

Falls Church, Virginia; the Naval Supply Systems Command, 

Mechanicsburg, Pennsylvania; Hunter Army Airfield, Savannah, Georgia; 

and Fort Stewart, Georgia.

In addition, we discussed the survey results with officials from Battelle’s 

Pacific Northwest National Laboratory and from Electronic Data Systems.

We conducted our review between August 1998 and April 1999 in 

accordance with generally accepted government auditing standards.

We are sending copies of this report to the Honorable William S. Cohen, 

Secretary of Defense; the Honorable Louis Caldera, Secretary of the Army; 

General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM; 

Lieutenant General John G. Coburn, Deputy Chief of Staff, Logistics, 

Department of the Army; Major General Mario F. Montero, Jr., Commander, 

MTMC; and the Honorable Jacob J. Lew, Director, Office of Management 

and Budget. We are also sending copies to other congressional members 

who expressed interest to us in DOD’s efforts to improve its personal 

property program, including the Honorable Fred Thompson, United States 

Senate, and the Honorable Neil Abercrombie, Duncan Hunter, Martin T. 

Meehan, and James M. Talent, House of Representatives. We will also make 

copies available to others upon request.
B-282406

Page 17 GAO/NSIAD-99-129 Defense Transportation

Please contact me on (202) 512-8412 if you or your staff have questions 

concerning this report. Major contributors to this report are listed in 

appendix V.

David R. Warren, Director

Defense Management Issues
B-282406

Page 18 GAO/NSIAD-99-129 Defense Transportation

List of Congressional Committees

The Honorable John W. Warner

Chairman

The Honorable Carl Levin

Ranking Minority Member

Committee on Armed Services

United States Senate

The Honorable Ted Stevens

Chairman

The Honorable Daniel K. Inouye

Ranking Minority Member

Subcommittee on Defense

Committee on Appropriations

United States Senate

The Honorable Floyd D. Spence

Chairman

The Honorable Ike Skelton

Ranking Minority Member

Committee on Armed Services

House of Representatives

The Honorable Herbert H. Bateman

Chairman

The Honorable Solomon P. Ortiz

Ranking Minority Member

Subcommittee on Military Readiness

Committee on Armed Services

House of Representatives

The Honorable Jerry Lewis 

Chairman

The Honorable John P. Murtha

Ranking Minority Member

Subcommittee on Defense

Committee on Appropriations

House of Representatives
Page 19 GAO/NSIAD-99-129 Defense Transportation
Page 20 GAO/NSIAD-99-129 Defense Transportation

Contents

Letter 1

Appendix I

Thresholds and Scores 

Used by the Army for 

Hunter Pilot 

Evaluation Factors

22

Appendix II

Types and Number of 

Shipments Made 

During the Hunter Pilot 

Test

23

Appendix III

Hunter Pilot and 

Baseline Costs 

Calculated by the 

Department of the 

Army (Office of the 

Deputy Chief of Staff 

for Logistics)

24

Appendix IV

Comments From the 

Department of Defense

25

Appendix V

Major Contributors to 

This Report

28
Contents

Page 21 GAO/NSIAD-99-129 Defense Transportation

Related GAO Products 32

Abbreviations

AAA Army Audit Agency

DITY do-it-yourself

DOD Department of Defense

EDS Electronic Data Systems

MTMC Military Traffic Management Command

USTRANSCOM U.S. Transportation Command
Page 22 GAO/NSIAD-99-129 Defense Transportation

Appendix I

Thresholds and Scores Used by the Army for 

Hunter Pilot Evaluation Factors ApenIdxi

Factors/definitions Thresholds Scores

Quality of life/customer satisfaction rating: percent of Hunter respondents who were 

more satisfied than Fort Stewart respondents (control group).

25% and above 1.00 point

20-24% 0.75 points

15-19% 0.50 points

10-14% 0.25 points

Below 10% Unsuccessful

Quality of life/claims settlement time: time period between when a claim was received 

and when the contractor offered settlement.

0-10 days 1.00 point

11-15 days 0.75 points

16-20 days 0.50 points

20-30 days 0.25 points

Above 30 days Unsuccessful

Quality of life/percentage of eligible deliveries: percent of shipments delivered to 

residence without storage in-transit.

91-100% 1.00 point

81-90% 0.75 points

71-80% 0.50 points

61-70% 0.25 points

60% and below Unsuccessful

Total cost: extent total contract cost exceeded total baseline cost. 0% 1.00 point

1-10% 0.75 points

11-20% 0.50 points

21-30% 0.25 points

More than 30 % Unsuccessful

Impact on small business: percent of small agents and carriers used. 20% or more 1.00 point

27-28% 0.75 points

25-26% 0.50 points

23-24% 0.25 points

Less than 23% Unsuccessful
Page 23 GAO/NSIAD 99-129 Defense Transportation

Appendix II

Types and Number of Shipments Made During 

the Hunter Pilot Test ApendxIi

Shipment type Shipments initiated and recorded as completed

Household goods (intrastate – Georgia 51

Household goods (interstate – continental U.S., including Alaska) 418

Household goods (overseas – Hawaii) 29

Household goods (overseas – Germany) 79

Household goods (overseas – Korea) 50

Household goods (overseas – other) 47

Unaccompanied baggage (overseas – all locations) 378

Do-it-yourself (DITY) moves 292

Mobile homes 5

Total 1,349
Page 24 GAO/NSIAD-99-129 Defense Transportation

Appendix III

Hunter Pilot and Baseline Costs Calculated by 

the Department of the Army (Office of the 

Deputy Chief of Staff for Logistics) ApenIdxi

aArmy Audit Agency validated all costs except those under $5,000, concluding that such amounts were 

immaterial.

bMilitary Traffic Management Command (MTMC).

Source: Office of the Deputy Chief of Staff for Logistics, Department of Army.

Cost elementsa Estimated pilot costs Estimated baseline costs

Transportation costs

Transportation $3,005,229 $2,102,384

Accessorials 25,767 30,655

Storage 210,706 211,655

Total transportation costs $3,241,702 $2,344,694

Claims paid by Army

Claims paid by Army $0 $66,268

Overhead costs

Personnel $79,802 $221,721

Management price 132,700 0

MTMC headquartersb 0 63,510

Automation 0 52,612

Voucher processing 4,614 90,658

Inbound processing 0 46,834

Claims processing 0 11,510

Pay for performance 14,756 0

Building overhead 28,321 43,286

Telephone and copier 1,281 10,106

Consumables 3,760 5,189

Total overhead costs $265,234 $545,426

Total costs $3,506,936 $2,956,388

 Pilot

 costs

 Baseline

 costs Difference

 Percent

 difference

Percent pilot costs 

exceeded baseline costs $3,506,936 $2,956,388 $550,548 18.6
Page 25 GAO/NSIAD-99-129 Defense Transportation

Appendix IV

Comments From the Department of Defense ApeIVndxi

Note: GAO comment 

supplementing those in the 

report text appear at the end 

of this appendix.

See comment 1.
Appendix IV

Comments From the Department of Defense

Page 26 GAO/NSIAD-99-129 Defense Transportation
Appendix IV

Comments From the Department of Defense

Page 27 GAO/NSIAD-99-129 Defense Transportation

The following are GAO’s comments on the Department of Defense’s letter 

dated May 11, 1999.

GAO Comment 1. As stated in the report, we could not confirm the customer satisfaction 

survey results. Specifically, we could not confirm that all survey results 

indicated higher levels of customer satisfaction because of weaknesses in 

the survey methodology. Methodological weaknesses that compromised 

survey results included using different populations and respondent rates, 

using various survey instruments with different questions, and surveying 

servicemembers multiple times. In addition, one of the two survey 

contractors found no statistically significant difference between responses 

from Hunter and Fort Stewart.

Our assessment of customer satisfaction results, as indicated in the report, 

was based on the Army’s criteria set forth in its evaluation plan. Because 

the Army used the results of one question to score the pilot a success, we 

also used it as the basis for our assessment. We agree, however, that other 

data obtained by the surveys provided indicators of customer satisfaction 

and highlighted areas that could be addressed in future evaluations.

We agree that the bulk of cost elements could be validated and revised our 

report to show that some overhead cost data was inaccurate and 

incomplete.

Finally, regarding our assessment of the claims process, we did not 

acknowledge a significant improvement in the process because the Army 

measured only one aspect of it, as indicated in the report. In addition, on 

the basis of the Army’s criteria set forth in its evaluation plan for assessing 

this factor, we could not confirm the time it took to offer claims settlement 

to servicemembers. We also agree that the claims process offered benefits 

not previously available to the servicemembers.
Page 28 GAO/NSIAD-99-129 Defense Transportation

Appendix V

Major Contributors to This Report ApenxVdi

National Security and 

International Affairs 

Division, Washington, 

D.C.

Charles I. Patton, Jr., Associate Director

Nomi R. Taslitt, Assistant Director

Robert L. Self, Evaluator-in-Charge

Marc J. Schwartz, Senior Evaluator

Jacqueline E. Snead, Senior Evaluator

Richard R. Irving, Evaluator

Arthur L. James, Jr., Mathematical Statistician

Office of the General 

Counsel, Washington, 

D.C.

John G. Brosnan, Assistant General Counsel

Norfolk Field Office Daniel A. Omahen, Senior Evaluator

John R. Beauchamp, Senior Evaluator
Page 29 GAO/NSIAD-99-129 Defense Transportation
Page 30 GAO/NSIAD-99-129 Defense Transportation
Page 31 GAO/NSIAD-99-129 Defense Transportation
Page 32 GAO/NSIAD-99-129 Defense Transportation

Related GAO Products

Defense Transportation: Efforts to Improve DOD’s Personal Property

Program (GAO/T-NSIAD-99-106, Mar. 18, 1999).

Defense Transportation: The Army’s Hunter Pilot Project to Outsource

Relocation Services (GAO/NSIAD-98-149, June 10, 1998).

Defense Outsourcing: Better Data Needed to Support Overhead Rates for

A-76 Studies (GAO/NSIAD-98-62, Feb. 27, 1998).

Defense Transportation: Reengineering the DOD Personal Property

Program (GAO/NSIAD-97-49, Nov. 27, 1996).

Household Goods: Administrative Changes Would Improve DOD’s Do-ItYourself Moving Program (GAO/NSIAD-94-226, Sept. 27, 1994).

DOD Commercial Transportation: Savings Possible Through Better Audit

and Negotiation of Rates (GAO/NSIAD-92-61, Dec. 27, 1991).

Household Goods: Competition Among Commercial Movers Serving DOD

Can Be Improved (GAO/NSIAD-90-50, Feb. 12, 1990).

(709365) Letr
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the 

following address, accompanied by a check or money order made 

out to the Superintendent of Documents, when necessary, VISA and 

MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 37050

Washington, DC 20013

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000

or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and 

testimony. To receive facsimile copies of the daily list or any list 

from the past 30 days, please call (202) 512-6000 using a touchtone 

phone. A recorded menu will provide information on how to obtain 

these lists.

For information on how to access GAO reports on the INTERNET, 

send an e-mail message with “info” in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at: 

http://www.gao.gov
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. GI00