Defense Transportation: Plan Needed for Evaluating the Navy Personal Property Pilot
Published: 1999-06-23T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: Pursuant to a legislative requirement, GAO provided information on the Navy Personal Property Pilot.
section_title: What GAO Found
paragraphs: GAO noted that: (1) between January 1998 and March 1999, 223 servicemembers have used the pilot program to arrange their own move rather than use the current program; (2) participation has been relatively low compared to the other three pilots under way or planned, which involve substantially more shipments--approximately 1,400 to 45,000 shipments annually; (3) survey data indicate that participants are satisfied with the pilot and would use the option again; (4) because the pilot offers servicemembers a choice between the current program and arranging their own move, implementing the option increases the workload for local personal property officials; (5) the member arranged move option is not featured in any of the other pilots, which are broader in scope and are intended to replace the current program; (6) while the U.S. Transportation Command is responsible for evaluating all of the pilots to determine which one could provide better long-term results, its plan for doing so has not been finalized; and (7) in addition, the U.S. Transportation Command's draft evaluation plan proposes to collect information on the extent the Navy pilot works as an option within the current program at a few Navy sites, which may not provide adequate data to assess the Navy pilot's feasibility or its compatibility with the other pilots' results.

section_title: Letter
paragraphs: Since 1994, the Department of Defense (DOD) has been engaged in  initiatives to reengineer the personal property program to simplify current  processes, control program costs, ensure quality of service by adopting  commercial business processes characteristic of world-class businesses,  and relieve carriers of DOD-unique terms and conditions.  We recently  testified before the Military Readiness Subcommittee, House Armed  Services Committee, on the results of the Army’s pilot and DOD’s plans to  evaluate the other three pilots that are planned or under way.  The  statement of managers in the conference report on the 1997 DOD  Appropriations Act conference report directed us to validate the results  and savings achieved from any personal property pilot program before  DOD proposes further expansion of such programs.  In January 1998, the  Navy began a servicemember arranged move pilot that allows  servicemembers to select a carrier and arrange their own moves. This  report provides status information on the Navy pilot and our assessment of  plans for evaluating the pilot’s results.
subsections: section_title: Background
paragraphs: The Navy pilot allows servicemembers the option of either participating in  the current program or of selecting their own carrier from a list of  approved small business carriers.  Key features of the servicemember  arranged move include shipment location information, direct claims  settlement with the carrier, full replacement value for lost or damaged  household goods, and payment via the government charge card.  Initiated  in January 1998, the pilot included only shipments originating at Puget  Sound, Washington, headed to San Diego, California; Norfolk, Virginia; New  London, Connecticut; Pensacola, Florida; and Jacksonville, Florida.  To  increase participation, the pilot’s origin site was expanded from Puget  Sound (including Whidbey Island and Everett, Washington) to include San  Diego, Norfolk, and New London in July 1998.  However, due to potential  competition with another pilot that will also serve the Florida area,  Pensacola and Jacksonville remain as destination sites only.  The pilot does  not have a specific end date; however, DOD established a target in the  November 1997 Defense Reform Initiative Report to offer the Navy option  to every servicemember by January 1, 2000.
The Navy pilot is one of four quality-of-life initiatives DOD is pursuing to  change the way it is currently doing business, adopt commercial business  practices, and achieve quality moving services for military families.  The  other three pilots are described below.
The Army’s pilot at Hunter Army Airfield, Savannah, Georgia, which was  initiated in February 1996, is a quality-of-life effort to improve the  relocation process and to test commercial practices in a military  environment.  Services provided by the contractor include point-to-point  move management with a single point of contact for the member,  assistance in buying/selling a residence, full replacement value for lost  or damaged household goods, direct claims settlement with the  servicemember, and visibility of the shipment for approximately 1,400  annual moves.  The contract will end on September 30, 1999.
In January 1999, the Military Traffic Management Command (MTMC),  which manages the current personal property program, began a pilot  involving 50 percent of the moves in North Carolina, South Carolina, and  Florida, which will total approximately 18,500 moves annually.  Key  features of the pilot include the selection of carriers based on  servicemember satisfaction and past performance rather than on price  alone; achieving stronger carrier commitment through long-term  contracts; offering full-value replacement protection and direct claims  settlement to users.  The pilot is planned to run for 3 years, ending  September 2002.
DOD announced on February 12, 1999, that it intended to begin the Full  Service Moving Project as a fourth test.  This pilot would be similar to  the pilot at Hunter Army Airfield, with modifications based on the  Army’s lessons learned, and it would involve a larger number of moves  (approximately 45,000 annually).  The pilot would be tested in the  National Capital (Washington, D.C.) Region, Georgia, and North Dakota.   Like the Army pilot, it is intended to replace the existing program by  using a contractor or contractors to provide both transportation and  move management services.  No official start date has been set for this  pilot program.
Status of the Navy Pilot As of March 1999, 223 servicemembers have participated in the pilot.  In its  second year of operation, the pilot has been expanded from one to four  shipment sites.  Program results thus far show (1) participation has been  relatively low compared to the other pilots that are under way or planned,  (2) servicemembers are satisfied arranging their own move, according to  limited survey data, and (3) workload would increase for personal property  officials because this option adds an alternative to the current program.
subsections: section_title: Servicemember Participation Is Low
paragraphs: As of March 26, 1999, a total of 223 members have participated in the pilot  and 132 moves have been completed.  Participation has been low, in part,  because of the pilot’s short duration and eligibility restrictions.  Although  the Puget Sound office began to offer this option in January 1998, the other  three sites did not offer it until late July 1998, and then only for Navy  military members moving to one of six destinations.  The pilot’s eligibility  restrictions exclude civilians and members of the other services.  Also, only  certain types of domestic household goods shipments are eligible. The pilot  excludes shipments of boats and mobile homes as well as shipments from  non-temporary storage or from a mini-warehouse.  Shipments must weigh  at least 3,000 pounds and are expected to cost between $2,500 and $25,000.
As of the end of calendar year 1998, the participating personal property  offices reported interviewing 1,083 Navy members that were moving to or  from the six pilot sites to determine whether their shipments met pilot  eligibility criteria.  Some 573, or 53 percent, of them had eligible shipments  and 133, or 23 percent, selected the option.  Over 90 percent of those  determined as ineligible had household goods not weighing at least 3,000  pounds.  Site officials reported that about 70 percent of eligible members  cited the effort involved in selecting a carrier as the primary reason that  they did not use the option (see table 1 for participant information by site).
Navy officials have recently modified eligibility requirements to increase  participation in the pilot.  The major changes would increase eligibility by  (1) reducing the weight minimum from 3,000 pounds to 1,000 pounds, (2)  allowing boats, and (3) eliminating current restrictions to allow shipments  to any domestic destination.
section_title: Limited Customer Survey Results Indicate Satisfaction With Pilot
paragraphs: To measure customer satisfaction with the pilot, the Navy uses a nine-question customer survey that servicemembers are asked to fill out  and return after the move is completed.  The survey includes questions  pertaining to pickup and delivery time, loss, damage, and overall  satisfaction.  The survey asks, among other things, if the move was of a  better quality than the servicemembers’ prior move and if the customer  would (1) choose the pilot again, (2) recommend it to someone else, and  (3) use the same carrier again.
As of February 1999, the Navy had received 30 customer satisfaction  surveys.  These surveys are predominantly from the Puget Sound personal  property office because Puget Sound was the first site to offer the pilot.  Of  the 30 surveys, 23 indicate that the pilot was a better quality move than  other military moves.  Eighteen of the 30 shipments (60 percent) had no  claims, and 12 had a claim.  Of the 12 claims made, only 2 exceeded $500.   One of these claims was for $1,900 for broken china.  Despite this damage,  the customer responded that this was a better quality move than prior ones  and stated that they would use the same carrier and the pilot again (see  table 2 for selected survey responses).
section_title: Increased Workload at Personal Property Offices
paragraphs: While the pilot offers advantages to the servicemember, it would add to the  workload of the personal property officials who are responsible for the  bulk of day-to-day program management.  Among their additional duties, or  duties that were previously handled by MTMC, are negotiating agreements  with participating small business carriers, providing individual counseling  to potential participants, maintaining up-to-date carrier information and  performance data, and tracking customer satisfaction survey results.   These are additional duties not previously handled by the personal  property office.
Local contracting officers at each participating installation enter into  agreements with companies that offer acceptable discounts off of  commercial rates.  The agreements are entered into with companies that  are self-certified as small businesses and on MTMC’s approved list.  These  agreements provide for the use of a government charge card for simplified  payment and for direct claims settlement with the carrier at full repair or  replacement value at no additional cost to the government.  In addition,  carriers are required to provide information on a shipment’s location  through the use of a toll-free help line and a pager for direct delivery  notification, which are designed to improve service and reduce storage  costs.
The personal property office provides servicemembers with the names of  participating movers that have been determined to have reasonable prices.   The property offices also maintain carrier quality books that contain a  carrier’s performance history, the returned customer satisfaction surveys,  and the carrier’s marketing materials.  Participating servicemembers are  required to contact at least three moving companies and document the  basis for their preference of one of the carriers.  The contracting officer can  then make the selection considering both price and the servicemember’s  recommendation concerning non-price factors such as quality.  The award  is made to the firm selected under the simplified acquisition procedures  contained in Federal Acquisition Regulation part 13.
section_title: Plan for Evaluating the Pilot Is Incomplete
paragraphs: The Navy does not plan to conduct an evaluation of the pilot program  separate from its own evaluation.  The U.S. Transportation Command  (USTRANSCOM) is currently in the process of developing and coordinating  an evaluation plan with the services to evaluate the personal property pilot  tests, and it will make a recommendation as to the follow-on course of  action.  In this regard, the Navy has modified its customer survey questions  so that the survey questions match the USTRANSCOM survey questions.  Navy officials are also accumulating pilot project transportation costs and  developing a methodology to compare these costs with those that the  government would have otherwise paid.
Presently, the pilot has only been tested as an option to the current  program at a few Navy sites and not as an option at the other pilot sites.   The other pilots are designed to test an approach to replace the current  program.  However, the USTRANSCOM draft evaluation plan does not  address the feasibility or potential benefits of incorporating the pilot option  into the pilots that may replace the current program.  Further, the  evaluation plan does not directly assess the unique aspects of the pilot,  which include limiting carrier selection to small business carriers and using  the government charge card for payment.  Consequently, DOD may not  have the information it needs to craft a DOD-wide personal property  program.
Navy officials are concerned about several aspects of the USTRANSCOM  evaluation plan.  In March 1999, they stated that the data elements to be  collected should be better defined, that a consistent evaluation time period  should be established, and that expert advice should be sought.  The  officials also believe that cost comparisons of overhead will be difficult  because the same personnel who administer the current program are  implementing the Navy pilot.  In our recent testimony and report on the  Army’s Hunter pilot and in our comments on several draft evaluation plans,  we have stated similar concerns about the current evaluation plan.  These  concerns include the need to (1) develop a comprehensive strategy for  testing the unique characteristics and/or processes of each pilot and (2) use  expert advice in finalizing a methodologically-sound evaluation plan.
section_title: Conclusions
paragraphs: The Navy pilot program differs from other ongoing or proposed pilot  programs because it adds an option to the current program; it is not  intended to replace the current program.  The pilot participation levels and  results thus far provide general information about the program’s potential  benefits and customer satisfaction compared to the current program— which DOD is proposing to replace.  Since the Navy pilot option is not to be  integrated and tested with the other pilots, information on the viability of  providing this option will only be available in comparison to the current  program.  Further, the draft evaluation plan does not identify a specific  method for assessing the pilot’s unique features.
section_title: Recommendation
paragraphs: Consistent with the recommendation in our report on the Army’s Hunter  pilot that the Secretary of Defense develop a comprehensive strategy for  testing each of the approaches, we recommend that the Secretary of  Defense, in developing this strategy, consider testing the Navy pilot and/or  its unique features in conjunction with one or more of the other pilots.   Doing so would test the Navy pilot in an environment that is more  consistent with the changes being considered and likely to be  implemented.
section_title: Agency Comments
paragraphs: DOD stated that it concurred with the report and its recommendations.   Specifically, DOD stated that, as part of its plan to develop a  comprehensive strategy for evaluating each of the pilots, it would  determine—in concert with the services—how best to incorporate the  features of the Navy pilot into the other pilots.  DOD’s comments are  reprinted in appendix I.
section_title: Scope and Methodology
paragraphs: To determine how the Navy plans to implement and evaluate the pilot  program, we reviewed available program documents and met with program  management officials at the Naval Supply Systems Command,  Mechanicsburg, Pennsylvania.  We discussed how the pilot is being  implemented and managed, the contract terms and conditions, and the key  characteristics of the pilot. Further, we discussed and reviewed the  customer survey results and other data that will be provided to  USTRANSCOM for its evaluation of the pilot.  We also discussed with Navy  Supply Systems Command and Naval Audit Service officials their concerns  with the USTRANSCOM evaluation plan.
We visited and interviewed officials at the Fleet and Industrial Supply  Center, Puget Sound, Bremerton, Washington, to determine how the pilot  was implemented at its first location.  We discussed the process used to  establish and manage the pilot at Puget Sound while continuing to operate  the current system.  Additionally, we discussed their experiences and  observations with the pilot.  We also visited and interviewed program,  contracting, and government charge card officials at the Fleet and  Industrial Supply Center, Norfolk, Virginia, to determine how  implementation was progressing and to compare the pilot operations at  Puget Sound to those of Norfolk.
To provide information on the pilot’s progress, we visited and interviewed  Naval Supply Systems Command officials to determine how sites were  selected for expansion after the pilot was established at Puget Sound.   Further, we attended a “lessons learned” conference that included personal  property officials from all of the pilot sites as well as from the Navy  Transportation Support Center, Bureau of Naval Personnel, and the  Defense Finance and Accounting Service to understand the problems these  organizations encountered with the pilot and possible solutions.
To obtain information on pilot participation, we reviewed weekly reports  summarizing the number of military members that were eligible and those  that were selected to participate.  We also discussed the reasons for their  participation with site officials.
To provide information on available cost data, we reviewed the cost  comparisons developed by site officials that compare actual pilot  transportation costs to estimated/constructed costs for the same move  under the current system.  We did not independently verify any data  reported by pilot sites or Naval Supply Systems Command headquarters.
Our review was conducted between October 1998 and April 1999 in  accordance with generally accepted government auditing standards.
We are sending copies of this report to the Honorable William S. Cohen,  Secretary of Defense; the Honorable Richard Danzig, Secretary of the Navy;  General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM;  Rear Admiral Donald E. Hickman, Commander, Naval Supply Systems  Command; Major General Mario F. Montero, Jr., Commander, MTMC; and  the Honorable Jacob J. Lew, Director, Office of Management and Budget.   We will also make copies available to others upon request.
Please contact me at (202) 512-8412 if you or your staff have any questions  concerning this report. Major contributors to this report are listed in  appendix II.
section_title: Comments From the Department of Defense
section_title: Major Contributors to This Report
subsections: section_title: National Security and International Affairs Division, Washington, D.C.
section_title: Office of the General Counsel, Washington, D.C.
section_title: Norfolk Field Office
paragraphs: Daniel A. Omahen, Evaluator-in-Charge John R. Beauchamp, Senior Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each.  Orders should be sent to the  following address, accompanied by a check or money order made  out to the Superintendent of Documents, when necessary, VISA and  MasterCard credit cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are  discounted 25 percent.
U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.
Each day, GAO issues a list of newly available reports and  testimony.  To receive facsimile copies of the daily list or any list  from the past 30 days, please call (202) 512-6000 using a touchtone  phone.  A recorded menu will provide information on how to obtain  these lists.