Defense Management: DOD Needs to Demonstrate That Performance-Based Logistics Contracts Are Achieving Expected Benefits
Published: 2005-09-09T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Department of Defense (DOD) contracts with private sector companies to perform depot maintenance of weapon systems using performance-based logistics--that is, purchasing a defined level of performance over a defined time period at a fixed cost to the government. After implementing such contracts, program offices are to validate their efficacy using cost and performance data; DOD cannot otherwise ensure cost savings and improved performance are being achieved through the use of performance-based logistics. GAO was asked to review the implementation of performance-based logistics to determine whether DOD could demonstrate cost savings and improved responsiveness from these arrangements. In conducting its review, GAO analyzed the implementation of performance-based logistics arrangements for 15 weapon system programs.
section_title: What GAO Found
paragraphs: DOD program offices could not demonstrate that they have achieved cost savings or performance improvements through the use of performance-based logistics arrangements. Although DOD guidance on implementing these arrangements states program offices should update their business case analysis based on actual cost and performance data, only 1 of the 15 program offices included in GAO's review had performed such an update consistent with DOD guidance. In the single case where the program office had updated its business case analysis, it determined that the performance-based logistics contract did not result in expected cost savings and the weapon system did not meet established performance requirements. In general, program offices had not updated their business case analysis after entering into a performance-based logistics contract because they assumed that the costs for weapon system maintenance incurred under a fixed-price performance-based logistics contract would always be lower than costs under a more traditional contracting approach and because they lacked reliable cost and performance data needed to validate assumptions used. Furthermore, the Office of the Secretary of Defense has not established procedures to monitor program offices to ensure they follow guidance and update the business case analysis. Additionally, program officials said because of limitations in their own information systems, they typically relied on cost and performance data generated by the contractors' information systems to monitor performance-based logistics contracts. The program offices, however, had not determined whether contractor-provided data were sufficiently reliable to update their business case analysis. Although the Defense Contract Management Agency and the Defense Contract Audit Agency are most commonly used to monitor higher risk contracts, such as cost plus contracts, they are potential resources available to assist program offices in monitoring fixed-price performance-based contracts. In doing so, these DOD agencies have the capability to verify the reliability of contractors' information systems and collect cost and performance data needed to update their business case analysis. Until program offices follow DOD's guidance and update their business case analysis based on reliable cost and performance data, DOD cannot evaluate the extent to which performance-based logistics arrangements are achieving expected benefits and being effectively implemented within DOD.

section_title: Letter
paragraphs: The Department of Defense (DOD) reported that it spent $23.6 billion in  fiscal year 2004 for depot maintenance of military weapon systems.   Almost half of those expenditures were paid to private sector companies to  perform depot maintenance activities for selected weapon systems.  DOD’s  preferred approach for providing long-term total system support for  weapon systems is a concept it calls performance-based logistics.   Performance-based logistics is a variation of other contractor logistics  support strategies and involves defining a level of performance that the  weapon system is to achieve over a period of time at a fixed cost to the  government.  Specific performance measures are established in each  contract.  DOD believes this contracting strategy for supporting weapon  systems and components will achieve higher levels of performance while  minimizing costs.  For example, DOD has claimed millions of dollars in cost  savings and significant performance improvements in such areas as  material availability and logistics response time through the use of  performance-based logistics.  As a result of aggressively pursuing this  contracting strategy, DOD expected an increasing number of weapon  systems and components to be maintained for the military services by  private sector companies under long-term, fixed-price, performance-based  contracts.  To justify the use of a performance-based logistics strategy,  according to DOD guidance, program offices should conduct a business  case analysis prior to adopting this approach.  Following implementation of  a performance-based logistics contract, program offices should update  their business case analysis to validate the assumptions used based on  actual cost and performance data and to support future weapon systems  plans.
As requested, we reviewed DOD’s implementation of performance-based  logistics arrangements.  Our objective was to determine whether DOD  could demonstrate cost savings or improved responsiveness through the  use of performance-based logistics arrangements.  This is the second of  two reports addressing DOD’s implementation of performance-based  logistics.  In an August 2004 report, we identified the types of contractor  logistics support arrangements the private sector used when outsourcing  the maintenance of complex and costly equipment with life-cycle  management issues similar to those of military systems.  We also identified  several lessons learned from comparisons between private sector  contractor support practices and the performance-based logistics approach  that DOD was urging the services to use.  That report recommended that  DOD incorporate into guidance the private sector’s practice of using  performance-based contracts to achieve economies at the component  level, rather than as a preferred practice at the platform level.  DOD issued  policy memoranda and revised its guidebook to implement our  recommendation.
As part of our current review, we collected and analyzed data on 15 weapon  system programs that were among those identified by the Office of the  Secretary of Defense and the military services as programs that have  successfully used performance-based logistics arrangements.  We selected  weapon system programs that were sufficiently mature to provide lessons  learned on the use of performance-based logistics.  We reviewed contracts  and related files and interviewed program officials to determine how these  arrangements were structured and managed.  We examined business case  analysis documents prepared by the program offices to justify the use of a  performance-based logistics strategy.  We discussed with program officials  the systems they used to monitor contractor cost and performance.  We  also reviewed DOD policies, procedures, and guidance related to the use of  performance-based logistics for weapon system support.  We determined  that the data used were sufficiently reliable for our purposes.  We  performed our work from August 2004 through June 2005 in accordance  with generally accepted government auditing standards.  The Scope and  Methodology section contains more detailed information about the work  we performed.
subsections: section_title: Background
paragraphs: DOD has increasingly relied on contractors to provide logistics support for  weapon system maintenance.  These logistics support arrangements have  taken various forms.  In fiscal year 1998, DOD directed the armed services  to pursue logistics support “reengineering” efforts with contractors to  achieve cost savings and improve efficiency.  A 1999 DOD study identified  30 pilot programs to test logistics support concepts that placed greater  reliance on the private sector.  Some of the pilot programs involved  performance-type arrangements that were subsequently converted to, or  designated as, performance-based logistics contracts.  DOD’s Quadrennial  Defense Review Report advocated the implementation of performance- based logistics, with appropriate metrics, to compress the supply chain by  removing steps in the warehousing, distribution, and order fulfillment  processes; reducing inventories; and reducing overhead costs while  improving the readiness of major weapon systems and commodities.
Over the last few years, DOD has issued guidance on the implementation of  performance-based logistics.  In November 2001, the Office of the Deputy  Under Secretary of Defense issued guidance recommending that program  managers conduct a sound business case analysis to decide whether they  should implement performance-based logistics for new systems and major  acquisitions for already fielded systems.  In an August 2003 memorandum  to the military departments, the Under Secretary of Defense (Acquisition,  Technology and Logistics) stated that DOD should continue to increase its  use of performance-based logistics acquisitions.  On February 4, 2004, the  Deputy Secretary of Defense (1) directed the Under Secretary of Defense  (Acquisition, Technology and Logistics), in conjunction with the Under  Secretary of Defense (Comptroller), to issue clear guidance on purchasing  logistics support using performance criteria and (2) directed each service  to provide a plan to aggressively implement performance-based logistics  for current and planned weapon system platforms.  Then, based on  recommendations in our August 2004 report, the Under Secretary of  Defense (Acquisition, Technology and Logistics) issued a memorandum  reemphasizing that the use of this type of support strategy was intended to  optimize weapon system availability while minimizing costs and the  logistics footprint and may be applied to weapon systems, subsystems,  and components.  The memorandum also provided specific definitions of  performance metrics to be used.
DOD describes performance-based logistics as the process of (1) identifying a level of performance required by the warfighter and  (2) negotiating a performance-based arrangement between the government  and a contractor or government facility to provide long-term total system  support for a weapon system at a fixed level of annual funding.  Instead of  buying spare parts, repairs, tools, and data in individual transactions, DOD  program offices that use a performance-based logistics arrangement buy a  predetermined level of performance that meets the warfighter’s objectives.   Although established performance measures should be tailored to reflect  the unique circumstances of each performance-based logistics  arrangement, the measures are expected to support five general objectives:  (1) percentage of time that a weapon system is available for a mission  (operational availability); (2) percentage of mission objectives met  (operational reliability); (3) operating costs divided by a specified unit of  measure (cost per unit usage); (4) size or presence of support required to  deploy, sustain, or move a weapon system (logistics footprint); and  (5) period of time that is acceptable between the demand or request for  support and the satisfactory fulfillment of that request (logistics response  time).  Currently, a DOD task force is refining these objectives into DOD  standard performance definitions to be used by program offices in every  service when preparing performance-based logistics arrangements.
DOD guidance recommends that program offices prepare a business case  analysis prior to adopting a performance-based logistics approach to  support a weapon system.  The aim of the business case analysis is to  justify the decision to enter into a performance-based logistics contract.   The business case analysis is to include cost savings that are projected as a  result of using a performance-based logistics approach and the  assumptions used in developing the business case analysis.  Furthermore,  DOD guidance states that program offices should update their business  case analyses at appropriate decision points when sufficient cost and  performance data have been collected to validate the assumptions used in  developing the business case analyses, including the costs of alternative  approaches, projected cost savings, and expected performance levels.   Further, GAO Internal Control Standards state that it is necessary to  periodically review and validate the propriety and integrity of program  performance measures and indicators.  Also, actual performance data  should be continually compared against expected or planned goals, and  any difference should be analyzed.  Additionally, management should have  a monitoring strategy that emphasizes to program managers their  responsibility for internal controls (i.e., to review and validate performance  measures and indicators) and that includes a plan for periodic evaluation of  control activities.
section_title: DOD Program Offices Could Not Demonstrate Cost Savings and Performance Improvement Resulting from the Use of Performance-Based Logistics
paragraphs: DOD program offices could not demonstrate that their use of performance- based logistics arrangements had achieved cost savings and performance  improvements because they had not updated their business case analysis  as suggested by DOD guidance.  Specifically, of the 15 DOD program  offices, only 1 updated its business case analysis to validate assumptions  concerning cost and performance.  Other DOD program offices had not  updated their business case analysis in part because they lacked reliable  contractor cost and performance data.  The program offices typically relied  on cost and performance data generated by contractors’ information  systems without verifying that the data were sufficiently reliable to update  the business case analysis.  Two DOD agencies, DCMA and DCAA, have the  capability to assist program offices in monitoring fixed-price performance- based contracts, verifying the reliability of contractors’ information  systems, and collecting cost and performance data.
subsections: section_title: Projected Cost Savings and Performance Improvements Were Not Validated
paragraphs: None of the 15 program offices included in our review could demonstrate  that use of a performance-based logistics arrangement had achieved cost  savings and performance improvements.  Although an updated business  case analysis based on actual cost and performance data might show that  cost savings and performance improvements were being achieved, only  1 of the 15 program offices had updated its business case analysis  consistent with DOD guidance.  Of the 15 program offices, 11 had  developed a business case analysis prior to entering into a performance- based logistics arrangement. In their analysis, these program offices  projected that they would achieve significant cost savings.  For example, an  Army program office projected total cost savings of $508.5 million, and a  Navy program office projected cost savings of $29.7 million.  However, only  the Navy’s T-45 program office had subsequently updated its business case  analysis consistent with DOD guidance to determine whether cost savings  were being achieved.  Realizing that the contractor was not meeting the  aircraft availability performance measure, the program office reassessed  its business case assumptions and found that costs per flying hour were  higher than estimated because the aircraft was flying fewer hours than  forecasted.  As a result, the program office negotiated separate contracts  for the airframes and engines, which resulted in estimated cost savings of  $144 million over 5 years.
Performance indicators tracked by the program offices showed that the  contractors met or exceeded performance requirements.  Of the 15  programs, 10 reported that performance levels exceeded contract  requirements, and 5 reported that performance levels were meeting  contract requirements.  For example, an Army program office reported a  weapon system availability rate of 99 percent, which is 7 percent higher  than what was projected in the business case analysis.  Similarly, a Navy  program office reported a weapon system availability rate of 97 percent,  which is 7 percent higher than projected.
Despite the reported performance improvements, the program offices had  not analyzed the performance data to validate the improvements and  determine whether these improvements could be attributed directly to  their use of performance-based logistics arrangements to support the  weapon systems.  In addition, we noted that program offices in the past  reported they had also met or exceeded required levels of performance  using other contractual arrangements for weapon system maintenance.   Moreover, the DOD program offices reporting that performance levels were  exceeding contract requirements under performance-based logistics  arrangements had not determined the incremental costs associated with  achieving these higher levels of performance.  As a result, they had no way  of knowing whether incremental costs outweighed the benefits derived  from achieving performance levels in excess of requirements.
Program officials did not follow DOD guidance to update and validate their  business case analyses because they assumed that costs incurred under  fixed-price performance-based logistics arrangements would always be  lower than costs incurred under more traditional contracting  arrangements, and several program officials cited a lack of reliable data  needed to validate expected costs savings and improved performance.   However, the experience of the T-45 program showed that it is possible for  program offices to validate the assumptions in the business case analysis  and to determine whether expected cost savings and performance  improvements were achieved.  There are also other benefits derived from  validating the assumptions used in the business case analysis.  Validation  can provide a better understanding of costs associated with the repair and  maintenance of weapon systems, ensure that proper performance metrics  are in place to satisfy logistical demand, isolate incremental costs  associated with achieving higher levels of performance, and make cost and  performance data available for contract renegotiations in order to obtain  the best value for the government.  Furthermore, we did not find evidence  that the Office of the Secretary of Defense had established procedures to  monitor whether program offices were following its guidance to update  their business case analyses.  The results of these updates could be used by  DOD to assess the implementation of performance-based logistics  arrangements and evaluate the extent to which performance-based  logistics arrangements are achieving expected benefits.
section_title: DOD Program Offices Relied on Contractors’ Data Without Verifying the Reliability of the Data
paragraphs: DOD program offices included in our review stated that because of  limitations in their own information systems, they typically relied on cost  and performance data generated by the contractors’ information systems to  monitor performance-based logistics contracts.  Program offices  acknowledged limitations in their own information systems in providing  reliable data to closely monitor contractor cost and performance.  Existing  systems are capable of collecting some cost and performance information  on performance-based logistics contracts; however, according to program  officials, the systems are not capturing sufficiently detailed cost and  performance information for monitoring performance-based logistics contracts.  Program officials told us they had more confidence in the  accuracy and completeness of contractor systems than in their legacy  systems.  The program offices, however, had not determined whether the  contractor-provided data were sufficiently reliable to update their business  case analyses.  As a result, the program offices did not have the reliable  data they needed to validate the assumptions used in the business case  analysis and to determine whether their performance-based logistics  arrangements were achieving expected cost savings and improved  performance.
As we noted in a prior report on DOD’s management of depot maintenance  contracting, to reduce personnel and save costs, DOD decided to rely  more on contractors to manage and oversee fixed-price contracts because  these contracts are considered low risk.  The contractor assumes most of  the risks for fixed-priced contracts, with the government taking a more  limited role in monitoring these contracts.  In our prior work on defense  contract management, we discussed the importance of monitoring  contractors’ systems to ensure the accuracy and completeness of  information generated by these systems.  In addition, during our review of  the private sector’s use of performance-based logistics, we noted that  private-sector companies that use performance-based logistics contracts,  whether fixed price or cost-plus, closely monitor cost and performance  information to effectively manage their contracts.  These companies said  they rely on their own systems and personnel to verify the cost and quality  of work performed by the contractor.  The private sector takes this  approach (1) to ensure that expected costs under the contracts are  accurate and meet the company’s reliability standards, (2) to validate the  business case decision used to justify a performance-based logistics  arrangement, and (3) to obtain the data necessary to renegotiate the  contract.
section_title: DOD Program Offices Made Limited Use of Resources Available to Assist in Monitoring Performance- Based Logistics Contracts
paragraphs: DCMA and DCAA have the capability to monitor contractor cost and  performance, verify the reliability of contractor-provided data, and collect  detailed cost and performance data.  However, most of the DOD program  offices we reviewed made limited use of these agencies’ resources because  they viewed fixed-price performance-based logistics contracts to be low  risk compared with other types of contracts.  Before a contract is awarded,  DCMA can provide advice and service to help construct effective  solicitations, identify potential risk, select the most capable contractors,  and write contracts that meet the needs of DOD customers.  After the  contract is awarded, DCMA can monitor contractors’ information systems  to ensure that cost, performance, and delivery schedules are in compliance  with the terms and conditions of the contracts.  DCAA performs contract  audits for DOD components and provides accounting and financial  advisory services during contract negotiation and administration of  contracts.
DCMA and DCAA officials said that they have a greater role in monitoring  cost information for cost-plus contracts because such contracts are  considered high risk.  According to DCMA and DCAA officials, their level of  oversight is significantly less for fixed-priced contracts, including  performance-based logistics arrangements, because DOD considers these  contracts to be low risk, thereby diminishing the need for monitoring  contractor performance.  Without a request from program offices or  specific contract clauses, DCMA and DCAA generally would not conduct  periodic reviews or audits of fixed-price contracts to verify cost and  performance information.  DCMA and DCAA officials also said that in the  past, monitoring fixed-price contracts was included in their workload, but  because of a reduction in staff and streamlining of operations, they focused  their efforts on contract areas that have the highest risk for cost growth.   DCMA and DCAA officials said they would support increasing their role in  monitoring fixed-price performance-based contracts depending on the  availability of their resources.
section_title: Conclusions
paragraphs: DOD is expanding its use of performance-based logistics as its preferred  support strategy in support of weapon systems but has not yet  demonstrated that this long-term support strategy is being effectively  implemented DOD-wide.  DOD guidance states that program offices, after  entering into performance-based logistics arrangements, should update  their original business case analysis using actual cost and performance  data to validate their assumptions, but most of the program offices we  reviewed had not followed this guidance, and the Office of the Secretary of  Defense was not monitoring whether program offices were following the  guidance.  The program offices therefore could not substantiate that cost  savings and performance improvements for weapon system support were  being achieved through the use of performance-based logistics  arrangements.  Program offices also have lacked reliable cost and  performance data needed to validate the results of performance-based  logistics arrangements.  Reliable data could be collected and analyzed by  increasing oversight of these contracts with the assistance of DCMA and  DCAA.
section_title: Recommendations for Executive Action
paragraphs: To demonstrate that performance-based logistics arrangements are  resulting in reduced costs and increased performance, and to improve  oversight of performance-based logistics contracts, we recommend that  the Secretary of Defense direct the Under Secretary of Defense  (Acquisition, Technology and Logistics) to take the following two actions:   1. Reaffirm DOD guidance that program offices update their business  case analyses following implementation of a performance-based  logistics arrangement and develop procedures, in conjunction with the  military services, to track whether program offices that enter into these  arrangements validate their business case decisions consistent with  DOD guidance.   2. Direct program offices to improve their monitoring of performance- based logistics arrangements by verifying the reliability of contractor  cost and performance data.  The program offices may wish to increase  the role of DCMA and DCAA in overseeing performance-based logistics  contracts.
section_title: Agency Comments and Our Evaluation
paragraphs: In commenting on a draft of this report, DOD concurred with our  recommendations regarding the validation of business case decisions for  performance-based logistics arrangements and verification of reliability of  contractor data.  While DOD was generally responsive to our  recommendations, specific details on how DOD planned to validate and  verify contractor data were not provided.
Regarding our recommendation to reaffirm guidance and develop  procedures to track whether program offices validate their business case  decisions, DOD stated that the department will reaffirm DOD guidance on  updating the business case analysis after implementing performance-based  logistics arrangements and will work with the military services to develop  procedures to track whether program offices validate their business case  decisions consistent with DOD guidance.
With regard to our second recommendation to direct program offices to  verify the reliability of contractor cost and performance data, DOD stated  that it will issue guidance on verifying the reliability of contractor cost and  performance data. DOD did not provide specific information on what the  guidance would include nor did it indicate whether it would increase the  use of DCMA or DCAA to verify the reliability of contractor cost and  performance data.
DOD also provided technical comments, which we have incorporated as  appropriate.
section_title: Scope and Methodology
paragraphs: To determine whether DOD could demonstrate that cost savings and  improved performance were being achieved through the use of  performance-based logistics arrangements, we collected and analyzed data  on 15 weapon system programs identified by the Office of the Secretary of  Defense and the military services as programs that have successfully used  performance-based logistics arrangements.  The 15 programs are listed in  table 1.
We reviewed DOD and service policies, procedures, and guidance related  to the use of performance-based logistics and met with program officials to  discuss how their performance-based logistics contracts were structured  and managed and how these contracts were validated to ensure that cost  savings and improved performance were being achieved as a result of using  performance-based logistics.  We also obtained and analyzed available  documentation, including business case analyses, contracts, and related  files.  We did not assess the methodology program offices used to prepare  their business case analyses or the quality of these analyses.  We discussed  with program officials the systems they used to monitor contractor cost  and performance.  We also interviewed officials from the Office of the  Secretary of Defense and military department headquarters to discuss  implementation of performance-based logistics, lessons learned, and the  benefits derived from using performance-based logistics approaches and  practices.  To determine how private-sector companies ensure that cost  and performance levels under a performance-based contract are as  expected, we reviewed the information provided by seven companies  identified in our prior report that used complex and costly equipment that  had life-cycle management issues similar to military weapon systems, and  outsourced some portion of their maintenance work under performance- based contracts.  These seven companies consisted of six airline  companies and one mining company.  We contacted officials at DCMA and  DCAA to determine those agencies’ roles in monitoring the costs and  performance of fixed-priced contracts, including performance-based  logistics contracts, how audits are requested or initiated, and the  procedures for reporting the results of the audits.
We are sending this report to the Chairman and Ranking Minority Member,  Senate Subcommittee on Readiness and Management Support, Committee  on Armed Services.  We will also send copies to the Under Secretary of  Defense (Acquisition, Technology and Logistics).  Copies of this report will  be made available to others upon request.  In addition, the report will be  available at no charge on GAO’s Web site at http://www.gao.gov.  If you or  your staff have any questions on the matters discussed in this report, please  contact me at (202) 512-8412 or solisw@gao.gov.  Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report.  GAO staff who made major contributions to this  report are listed in appendix II.
section_title: Comments from the Department of Defense
section_title: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact
paragraphs: William M. Solis (202) 512-8412 or solisw@gao.gov.
section_title: Acknowledgments
paragraphs: In addition to the contact named above, Thomas Gosling, Assistant  Director; Thom Barger; Judith Collins; Pamela Valentine; and Cheryl  Weissman were major contributors to this report.