Defense Transportation: Plan Needed for Evaluating the Navy Personal Property Pilot
Released: 1999-06-23T00:00:00

Letter

Background

The Navy pilot allows servicemembers the option of either participating in the current program or of selecting their own carrier from a list of approved small business carriers. Key features of the servicemember arranged move include shipment location information, direct claims settlement with the carrier, full replacement value for lost or damaged household goods, and payment via the government charge card. Initiated in January 1998, the pilot included only shipments originating at Puget Sound, Washington, headed to San Diego, California; Norfolk, Virginia; New London, Connecticut; Pensacola, Florida; and Jacksonville, Florida. To increase participation, the pilot’s origin site was expanded from Puget Sound (including Whidbey Island and Everett, Washington) to include San Diego, Norfolk, and New London in July 1998. However, due to potential competition with another pilot that will also serve the Florida area, Pensacola and Jacksonville remain as destination sites only. The pilot does not have a specific end date; however, DOD established a target in the November 1997 Defense Reform Initiative Report to offer the Navy option to every servicemember by January 1, 2000.

The Navy pilot is one of four quality-of-life initiatives DOD is pursuing to change the way it is currently doing business, adopt commercial business practices, and achieve quality moving services for military families. The other three pilots are described below.

The Army’s pilot at Hunter Army Airfield, Savannah, Georgia, which was initiated in February 1996, is a quality-of-life effort to improve the relocation process and to test commercial practices in a military environment. Services provided by the contractor include point-to-point move management with a single point of contact for the member, assistance in buying/selling a residence, full replacement value for lost or damaged household goods, direct claims settlement with the servicemember, and visibility of the shipment for approximately 1,400 annual moves. The contract will end on September 30, 1999.

In January 1999, the Military Traffic Management Command (MTMC), which manages the current personal property program, began a pilot involving 50 percent of the moves in North Carolina, South Carolina, and Florida, which will total approximately 18,500 moves annually. Key features of the pilot include the selection of carriers based on servicemember satisfaction and past performance rather than on price alone; achieving stronger carrier commitment through long-term contracts; offering full-value replacement protection and direct claims settlement to users. The pilot is planned to run for 3 years, ending September 2002.

DOD announced on February 12, 1999, that it intended to begin the Full Service Moving Project as a fourth test. This pilot would be similar to the pilot at Hunter Army Airfield, with modifications based on the Army’s lessons learned, and it would involve a larger number of moves (approximately 45,000 annually). The pilot would be tested in the National Capital (Washington, D.C.) Region, Georgia, and North Dakota. Like the Army pilot, it is intended to replace the existing program by using a contractor or contractors to provide both transportation and move management services. No official start date has been set for this pilot program.

Status of the Navy Pilot As of March 1999, 223 servicemembers have participated in the pilot. In its second year of operation, the pilot has been expanded from one to four shipment sites. Program results thus far show (1) participation has been relatively low compared to the other pilots that are under way or planned, (2) servicemembers are satisfied arranging their own move, according to limited survey data, and (3) workload would increase for personal property officials because this option adds an alternative to the current program.

Servicemember Participation Is Low

As of March 26, 1999, a total of 223 members have participated in the pilot and 132 moves have been completed. Participation has been low, in part, because of the pilot’s short duration and eligibility restrictions. Although the Puget Sound office began to offer this option in January 1998, the other three sites did not offer it until late July 1998, and then only for Navy military members moving to one of six destinations. The pilot’s eligibility restrictions exclude civilians and members of the other services. Also, only certain types of domestic household goods shipments are eligible. The pilot excludes shipments of boats and mobile homes as well as shipments from non-temporary storage or from a mini-warehouse. Shipments must weigh at least 3,000 pounds and are expected to cost between $2,500 and $25,000.

As of the end of calendar year 1998, the participating personal property offices reported interviewing 1,083 Navy members that were moving to or from the six pilot sites to determine whether their shipments met pilot eligibility criteria. Some 573, or 53 percent, of them had eligible shipments and 133, or 23 percent, selected the option. Over 90 percent of those determined as ineligible had household goods not weighing at least 3,000 pounds. Site officials reported that about 70 percent of eligible members cited the effort involved in selecting a carrier as the primary reason that they did not use the option (see table 1 for participant information by site).

Navy officials have recently modified eligibility requirements to increase participation in the pilot. The major changes would increase eligibility by (1) reducing the weight minimum from 3,000 pounds to 1,000 pounds, (2) allowing boats, and (3) eliminating current restrictions to allow shipments to any domestic destination.

Limited Customer Survey Results Indicate Satisfaction With Pilot

To measure customer satisfaction with the pilot, the Navy uses a nine-question customer survey that servicemembers are asked to fill out and return after the move is completed. The survey includes questions pertaining to pickup and delivery time, loss, damage, and overall satisfaction. The survey asks, among other things, if the move was of a better quality than the servicemembers’ prior move and if the customer would (1) choose the pilot again, (2) recommend it to someone else, and (3) use the same carrier again.

As of February 1999, the Navy had received 30 customer satisfaction surveys. These surveys are predominantly from the Puget Sound personal property office because Puget Sound was the first site to offer the pilot. Of the 30 surveys, 23 indicate that the pilot was a better quality move than other military moves. Eighteen of the 30 shipments (60 percent) had no claims, and 12 had a claim. Of the 12 claims made, only 2 exceeded $500. One of these claims was for $1,900 for broken china. Despite this damage, the customer responded that this was a better quality move than prior ones and stated that they would use the same carrier and the pilot again (see table 2 for selected survey responses).

Increased Workload at Personal Property Offices

While the pilot offers advantages to the servicemember, it would add to the workload of the personal property officials who are responsible for the bulk of day-to-day program management. Among their additional duties, or duties that were previously handled by MTMC, are negotiating agreements with participating small business carriers, providing individual counseling to potential participants, maintaining up-to-date carrier information and performance data, and tracking customer satisfaction survey results. These are additional duties not previously handled by the personal property office.

Local contracting officers at each participating installation enter into agreements with companies that offer acceptable discounts off of commercial rates. The agreements are entered into with companies that are self-certified as small businesses and on MTMC’s approved list. These agreements provide for the use of a government charge card for simplified payment and for direct claims settlement with the carrier at full repair or replacement value at no additional cost to the government. In addition, carriers are required to provide information on a shipment’s location through the use of a toll-free help line and a pager for direct delivery notification, which are designed to improve service and reduce storage costs.

The personal property office provides servicemembers with the names of participating movers that have been determined to have reasonable prices. The property offices also maintain carrier quality books that contain a carrier’s performance history, the returned customer satisfaction surveys, and the carrier’s marketing materials. Participating servicemembers are required to contact at least three moving companies and document the basis for their preference of one of the carriers. The contracting officer can then make the selection considering both price and the servicemember’s recommendation concerning non-price factors such as quality. The award is made to the firm selected under the simplified acquisition procedures contained in Federal Acquisition Regulation part 13.

Plan for Evaluating the Pilot Is Incomplete

The Navy does not plan to conduct an evaluation of the pilot program separate from its own evaluation. The U.S. Transportation Command (USTRANSCOM) is currently in the process of developing and coordinating an evaluation plan with the services to evaluate the personal property pilot tests, and it will make a recommendation as to the follow-on course of action. In this regard, the Navy has modified its customer survey questions so that the survey questions match the USTRANSCOM survey questions. Navy officials are also accumulating pilot project transportation costs and developing a methodology to compare these costs with those that the government would have otherwise paid.

Presently, the pilot has only been tested as an option to the current program at a few Navy sites and not as an option at the other pilot sites. The other pilots are designed to test an approach to replace the current program. However, the USTRANSCOM draft evaluation plan does not address the feasibility or potential benefits of incorporating the pilot option into the pilots that may replace the current program. Further, the evaluation plan does not directly assess the unique aspects of the pilot, which include limiting carrier selection to small business carriers and using the government charge card for payment. Consequently, DOD may not have the information it needs to craft a DOD-wide personal property program.

Navy officials are concerned about several aspects of the USTRANSCOM evaluation plan. In March 1999, they stated that the data elements to be collected should be better defined, that a consistent evaluation time period should be established, and that expert advice should be sought. The officials also believe that cost comparisons of overhead will be difficult because the same personnel who administer the current program are implementing the Navy pilot. In our recent testimony and report on the Army’s Hunter pilot and in our comments on several draft evaluation plans, we have stated similar concerns about the current evaluation plan. These concerns include the need to (1) develop a comprehensive strategy for testing the unique characteristics and/or processes of each pilot and (2) use expert advice in finalizing a methodologically-sound evaluation plan.

Conclusions

The Navy pilot program differs from other ongoing or proposed pilot programs because it adds an option to the current program; it is not intended to replace the current program. The pilot participation levels and results thus far provide general information about the program’s potential benefits and customer satisfaction compared to the current program— which DOD is proposing to replace. Since the Navy pilot option is not to be integrated and tested with the other pilots, information on the viability of providing this option will only be available in comparison to the current program. Further, the draft evaluation plan does not identify a specific method for assessing the pilot’s unique features.

Recommendation

Consistent with the recommendation in our report on the Army’s Hunter pilot that the Secretary of Defense develop a comprehensive strategy for testing each of the approaches, we recommend that the Secretary of Defense, in developing this strategy, consider testing the Navy pilot and/or its unique features in conjunction with one or more of the other pilots. Doing so would test the Navy pilot in an environment that is more consistent with the changes being considered and likely to be implemented.

Agency Comments

DOD stated that it concurred with the report and its recommendations. Specifically, DOD stated that, as part of its plan to develop a comprehensive strategy for evaluating each of the pilots, it would determine—in concert with the services—how best to incorporate the features of the Navy pilot into the other pilots. DOD’s comments are reprinted in appendix I.

Scope and Methodology

To determine how the Navy plans to implement and evaluate the pilot program, we reviewed available program documents and met with program management officials at the Naval Supply Systems Command, Mechanicsburg, Pennsylvania. We discussed how the pilot is being implemented and managed, the contract terms and conditions, and the key characteristics of the pilot. Further, we discussed and reviewed the customer survey results and other data that will be provided to USTRANSCOM for its evaluation of the pilot. We also discussed with Navy Supply Systems Command and Naval Audit Service officials their concerns with the USTRANSCOM evaluation plan.

We visited and interviewed officials at the Fleet and Industrial Supply Center, Puget Sound, Bremerton, Washington, to determine how the pilot was implemented at its first location. We discussed the process used to establish and manage the pilot at Puget Sound while continuing to operate the current system. Additionally, we discussed their experiences and observations with the pilot. We also visited and interviewed program, contracting, and government charge card officials at the Fleet and Industrial Supply Center, Norfolk, Virginia, to determine how implementation was progressing and to compare the pilot operations at Puget Sound to those of Norfolk.

To provide information on the pilot’s progress, we visited and interviewed Naval Supply Systems Command officials to determine how sites were selected for expansion after the pilot was established at Puget Sound. Further, we attended a “lessons learned” conference that included personal property officials from all of the pilot sites as well as from the Navy Transportation Support Center, Bureau of Naval Personnel, and the Defense Finance and Accounting Service to understand the problems these organizations encountered with the pilot and possible solutions.

To obtain information on pilot participation, we reviewed weekly reports summarizing the number of military members that were eligible and those that were selected to participate. We also discussed the reasons for their participation with site officials.

To provide information on available cost data, we reviewed the cost comparisons developed by site officials that compare actual pilot transportation costs to estimated/constructed costs for the same move under the current system. We did not independently verify any data reported by pilot sites or Naval Supply Systems Command headquarters.

Our review was conducted between October 1998 and April 1999 in accordance with generally accepted government auditing standards.

We are sending copies of this report to the Honorable William S. Cohen, Secretary of Defense; the Honorable Richard Danzig, Secretary of the Navy; General Charles T. Robertson, Jr., Commander in Chief, USTRANSCOM; Rear Admiral Donald E. Hickman, Commander, Naval Supply Systems Command; Major General Mario F. Montero, Jr., Commander, MTMC; and the Honorable Jacob J. Lew, Director, Office of Management and Budget. We will also make copies available to others upon request.

Please contact me at (202) 512-8412 if you or your staff have any questions concerning this report. Major contributors to this report are listed in appendix II.

Comments From the Department of Defense

Major Contributors to This Report

National Security and International Affairs Division, Washington, D.C.

Office of the General Counsel, Washington, D.C.

Norfolk Field Office

Daniel A. Omahen, Evaluator-in-Charge John R. Beauchamp, Senior Evaluator The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary, VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.