Defense Logistics: Improved Analysis and Cost Data Needed to Evaluate the Cost-effectiveness of Performance Based Logistics
Released: 2008-12-19T00:00:00

Letter

Background

PBL is a method of providing support for weapon systems by designating what system performance is required, such as a given level of system availability, and placing the responsibility for how it is accomplished on the support provider, which manages resources to achieve performance objectives. Logistics support for almost all of DOD’s weapon systems, such as materiel management, maintenance, and engineering, is provided by a combination of government and private-sector sources. In the past, under traditional support arrangements, the government generally managed the provision of weapon system support, using a combination of support providers from the government and the private sector. PBL support arrangements often use a private-sector support integrator to manage support providers from both the public and private sectors to meet specified performance requirements.

PBL evolved from performance-based service contracting, which has been used in both the public and private sectors. The Federal Acquisition Regulation defines performance-based contracting as structuring all aspects of an acquisition around the purpose of the work to be performed. The Federal Acquisition Regulation further defines the statement of work for a performance-based acquisition as describing the required results in clear, specific, and objective terms with measurable outcomes. Performance-based service contracting has been referenced in regulation, guidance, and policy for more than two decades, and federal agencies have used it to varying degrees for acquiring a range of services. In 1991 the Office of Management and Budget issued a policy letter establishing the use of a performance-based approach for service contracting, and in 1994 it initiated a governmentwide pilot project to encourage the use of performance-based service contracts in federal agencies, including DOD. In October 1997, the Federal Acquisition Regulation was amended to incorporate the Office of Management and Budget’s 1991 policy. The Federal Acquisition Regulation currently establishes a policy that agencies use performance-based contracting methods to the maximum extent practicable for the acquisition of services, with certain exceptions.

Using performance-based service contracts is intended to offer a number of potential benefits, such as encouraging contractors to be innovative and to find cost-effective ways of delivering services for a fixed level of funding. By shifting the focus from process to results, these contracts can potentially produce better outcomes and reduced costs. The Office of Management and Budget reported that the agencies participating in the pilot reduced contract prices and improved customer satisfaction with contractor work after introducing performance-based contracting.

PBL Implemented as a Strategy to Reduce or Minimize Costs and Improve Performance

As an approach for supporting military weapon systems, PBL emerged from a 1999 DOD study to test logistics reengineering concepts that placed greater reliance on the private sector for providing weapon system support to reduce support costs and improve weapon system performance. The goal was for the military departments to reduce costs and improve efficiency by pursuing logistics support “reengineering” efforts using contractors. The fiscal years 2001-2005 Defense Planning Guidance advanced this cost reduction effort by establishing a goal that directed each military department to reduce the operation and support costs of its fielded systems by 20 percent by the year 2005. During this time, the Under Secretary of Defense (Acquisition and Technology) directed the services to use an existing pilot program containing 30 weapon systems to demonstrate the type of cost savings depicted in the fiscal years 2001-2005 Defense Planning Guidance. The areas identified for potential cost savings were reducing demand on the supply chain by improving the reliability and maintainability of the equipment, reducing supply chain response time, and increasing competitive sourcing of product support. Some of the 30 pilot programs involved performance- type arrangements that the services subsequently converted to, or designated as, PBL arrangements.

This emphasis on reducing costs through PBL implementation was reiterated in DOD’s 2001 Quadrennial Defense Review Report, which advocated the implementation of PBL to compress the supply chain by removing steps in the warehousing, distribution, and order fulfillment processes; to reduce inventories; and to decrease overhead costs while improving the readiness of major weapon systems and commodities. In November 2001, DOD identified PBL as the preferred weapons system support strategy. In May 2003, DOD further strengthened this emphasis on PBL by stating in a DOD policy directive that acquisition managers shall use performance-based strategies for sustaining products and services whenever feasible and PBL strategies shall optimize total system availability while minimizing cost and the logistics footprint.

In concept, a properly structured PBL arrangement is supposed to provide a level of performance and also reduce costs over time. According to the DOD/Defense Acquisition University PBL guide, a key aspect of PBL is the inclusion of an incentive for the support provider to reduce costs through increased reliability. Further, PBL arrangements can inherently motivate support providers to improve component and system reliability, since such improvements can provide the foundation for increased profit over the long term. In other words, the support provider should have the incentive to make reliability improvements to ensure that performance metrics are met and also to increase profit by earning a performance incentive tied to the metrics (an award fee or award term) and by reducing costs while still being paid the agreed-upon, fixed price for the remainder of the contract. The DOD/Defense Acquisition University PBL guide also states that a critical element of the PBL arrangement that facilitates this incentive and motivation is contract length. Further, long-term contracts provide the support provider with confidence in continuing cash flows and provide sufficient time for receiving an adequate return on any investments made to improve reliability.

DOD Recommended That Analyses Be Conducted to Guide Decisions regarding Weapon System Support Options

In 1995, before DOD identified PBL as the preferred weapons system support strategy, DOD’s economic analysis instruction recommended using an economic analysis for evaluating options for weapon system support. The instruction stressed the importance of considering in the analysis both qualitative and quantitative factors. With respect to quantitative factors, the instruction recommended that costs and benefits be expressed in terms of net present value to account for the time value of money. These were also to be expressed as life cycle costs and benefits that were to be calculated and compared for each feasible alternative for meeting a given weapon system support objective.

Specifically, the economic analysis instruction identified and characterized the following seven elements that should be present in an economic analysis: objectives—to clearly identify the function to be accomplished and not to assume a specific means of achieving a desired result; assumptions—to incorporate both actual data and future alternatives—to comprise a comprehensive list of the feasible and infeasible options followed by a discussion of the infeasible options and comparisons of only the feasible options; costs and benefits—to compare the quantitative (expressed in terms of net present value) and qualitative factors for each option; sensitivity and uncertainty (risk) analyses—to determine the effect of uncertainties on the results of the analysis and to provide a range of costs and benefits; summary of the results of the analysis; and summary of the analysis’s recommendations.

In DOD’s 2001 PBL guide, in which DOD identified PBL as the preferred weapon system support strategy, the department recommended that for all new systems and fielded acquisition category I and II systems, program offices use an analytical tool called a business case analysis to support the decision to use PBL arrangements for weapon system support.

In 2004 and 2005, DOD guidance on conducting business case analyses described this tool in less specific terms than those used to describe the criteria laid out in DOD’s economic analysis instruction. However, there are some common themes in the guidance and instruction. For example, a January 2004 Under Secretary of Defense (Acquisition, Technology and Logistics) memorandum on PBL business case analysis calls for an assessment of “best value,” or the greatest overall benefit. The DOD/Defense Acquisition University PBL guide repeats the characterization of the business case analysis as a tool with the goal of determining a best-value solution and suggests that at a minimum a business case analysis should include an introduction outlining the purpose and objectives of the program, an explanation of the methods and assumptions used in the analysis, calculations of the relative costs and benefits of each weapon system support option, the financial and nonfinancial impacts of each, a risk assessment, and a section with conclusions and recommendations. Finally, both DOD’s economic analysis instruction and the DOD/Defense Acquisition University PBL guide recommend documenting the results of the analysis, including all calculations and sources of data, down to the most basic inputs, to provide an auditable and stand-alone document. According to the guidance, a business case analysis must stand on its own and be able to withstand rigorous analysis and review by independent agencies.

DOD’s 2004 and 2005 guidance on conducting business case analyses also recommended that program offices update their business case analyses at key decision points both to validate the approach taken and to support future plans and use certain criteria, such as the capability of a PBL arrangement to reduce the cost per operational unit of performance (i.e., cost per flight hour), to assess all acquisition I and II programs without plans for a PBL arrangement for the potential application of a PBL strategy at the system, subsystem, or major assembly level. If the assessment showed potential for a PBL arrangement, a business case analysis should be conducted and completed by the September 30, 2006, deadline required by DOD’s Strategic Planning Guidance.

PBL Arrangements Included in Our Review

In our review, we looked at PBL arrangements initiated as early as 1996 (when performance-based contracting was encouraged governmentwide) and as recently as 2007 (by which time, at the DOD level, PBL arrangements had moved from being encouraged to being required whenever feasible). These PBL arrangements represent contract values totaling approximately $12.1 billion and range from a low of $10.5 million to a high of $4.9 billion. Table 1 lists, by service, the weapon system programs supported by the 29 PBL arrangements we reviewed.

Business Case Analyses Have Not Been Used Consistently or Effectively to Influence Decision Making regarding Use of PBL

DOD has generally not used business case analyses consistently or effectively to influence decision making regarding the use of PBL. Although DOD guidance recommended that these analyses be used to guide decisions on the cost-effectiveness of weapon system support arrangements, about half of the programs we reviewed either did not conduct such an analysis or did not retain adequate supporting documentation. Further, most of the remaining programs in our sample used analyses that were not comprehensive. For example, some analyses did not evaluate alternative support options and most did not contain all of the elements recommended in DOD’s economic analysis instruction. Additionally, analyses were often not updated to support decision making after PBL implementation in accordance with service policies and guidance. The key reasons for DOD’s ineffective use of business case analyses to support PBL decision making are that (1) DOD has not required such analyses or provided specific criteria for conducting them and (2) the services’ internal controls have been inadequate to ensure that the analyses are performed and updated. As a result, DOD cannot ensure that decisions regarding weapon system support options are guided by comprehensive, consistent, and sound analytical tools. Further, the department cannot be sure that the support arrangements being implemented will result in the most cost-effective support program.

About Half of Programs Reviewed Either Did Not Use Business Case Analyses to Guide Decisions to Enter into PBL Arrangements or Did Not Retain Details of the Analysis

For 9 of the 29 PBL arrangements we reviewed, a business case analysis had not been completed. Additionally, for 6 others, program officials could not provide supporting details of the business case analysis they told us that they had conducted. When program offices did not conduct business case analyses as part of their PBL decision making, as we found for many of the programs we reviewed, the department cannot be sure that the support arrangements implemented will result in the most cost-effective support program.

Programs That Did Not Conduct a Business Case Analysis Before Entering into PBL Arrangements

Table 2 provides the number of PBL arrangements we reviewed by service that were initiated with and without the use of a business case analysis.

Although both of the Marine Corps programs we reviewed conducted a business case analysis, about 50 percent of Air Force programs, 22 percent of Army programs, and 25 percent of Navy programs did not. In general, the Air Force programs only developed a source-of-repair analysis, which evaluated only the repair element of weapon system support. For the two PBL arrangements for which the Army did not conduct an analysis—the Apache sensor and airframe arrangements—the Deputy Assistant Secretary of the Army (Integrated Logistics Support) approved the Apache program office’s request for a waiver from Army business case analysis policy based on prior analyses and program decisions. However, the U.S. Army Audit Agency reported that the prior analyses did not consider all components included in the two PBL arrangements, other support strategies, performance metrics, and relative costs. According to an Army official, a business case analysis for the airframe components is being conducted and is expected to be completed in December 2008, and efforts to develop a business case analysis for the program’s sensors are expected to begin in November 2008.

The F-22A Raptor and KC-130J are examples of programs where the Air Force and Navy did not conduct a business case analysis as part of the PBL decision-making process. When DOD recommended in 2001 that program offices fielding new systems base PBL arrangement decisions on business case analyses, the F-22 was beginning low-rate initial production. In 2002, the Assistant Secretary of the Air Force (Acquisition) and the Air Force Deputy Chief of Staff (Logistics, Installations and Mission Support) directed the F-22 program office to develop a long-term support strategy and manage life cycle product support through a PBL arrangement that includes government-contractor partnerships as necessary. From 2003 through 2007, the program office acquired support as part of the aircraft’s production contract, and in 2008 the office signed a separate PBL support contract with Lockheed Martin, one of the original equipment manufacturers, to support the aircraft from 2008 to 2012. In March 2008, program officials said that they did not conduct a business case analysis before awarding the 2008 contract because current program data, such as material usage rates and costs, are immature. Officials planned to conduct an analysis in 2010 or 2011 when it could be completed using more meaningful data. However, program officials subsequently decided that the available data were sufficient and in July 2008 awarded a contract to develop a business case analysis. Completion of the analysis is expected in late 2009.

In 2002, the Navy contracted for a PBL arrangement to support the Marine Corps’ KC-130J engines without first preparing a business case analysis. Program officials explained that the decision was made not to develop an analysis because the technical data needed to repair the engines were not available and the Marine Corps did not have the infrastructure necessary to support the system. Officials also said that a market analysis was conducted prior to implementing the PBL arrangement, but they could not provide a copy of the analysis. Nonetheless, a business case analysis that documented the results of the market analysis and the program’s negotiated costs versus expected flight hours and anticipated repairs and maintenance could have been used to monitor the actual results and cost- effectiveness of the performance-based approach, especially since support for the engines is obtained under a commercial contract and the contractor does not provide detailed cost data to the government.

Supporting Documentation Not Fully Available for Some Programs That Conducted Business Case Analyses

Program officials from 20 of the PBL arrangements we reviewed told us that they had conducted business case analyses before implementing the arrangements; however, officials for 6 programs could not provide all or some of the data to support the analyses. According to DOD’s economic analysis instruction, the results of the analysis, including all calculations and sources of data, must be documented down to the most basic inputs to provide an auditable and stand-alone document. Table 3 lists by service the number of PBL arrangements for which all of the business case analysis documentation was retained and those for which it was not.

In general, program officials for six programs said that they were unable to locate all the details of their analyses; however, the amount of data that was missing varied. For example: Although officials for the Army’s Common Ground Station said that an analysis was performed in 2002, they were unable to provide any details of the analysis or the results.

While program officials for the Army’s Shadow Tactical Unmanned Aircraft System were able to provide the results of their 2002 analysis, they did not retain the details regarding the assumptions, data sources, or calculations used to develop the analysis. However, program officials said that the analysis was developed early in the life cycle of the program and was not based on historical cost and maintenance data, and therefore they did not consider it to be very accurate based on actual program results that have occurred since the analysis was developed.

For the Army’s Javelin PBL arrangement, the DOD Office of the Inspector General reported in 2005 that it was unable to validate the program office’s 2001 analysis because the program office was not able to provide adequate documentation. The documentation has not been located, and program officials were only able to provide a summary of the results of the analysis.

Most Remaining Programs Used Business Case Analyses That Were Not Comprehensive

Service program officials could provide documentation of the business case analyses conducted for 14 of the PBL arrangements we reviewed, but all but 1 of the 14 analyses were missing one or more of the elements recommended in DOD’s economic analysis instruction. As a result, decisions regarding weapon system support options for many of the programs we reviewed were not guided by comprehensive, consistent, and sound economic analysis. Further, the department cannot be sure that the support arrangements implemented will result in the most cost-effective support programs. Figure 1 shows which elements were missing from the 14 business case analyses.

For three PBL arrangements, the business case analyses did not compare alternative support options and either evaluated only a single option for weapon system support or evaluated contracting strategies instead of alternative support arrangements. For example, the 2007 business case analysis for the Joint Surveillance and Target Attack Radar System did not analyze the costs and benefits of alternative support strategies for the program. The business case analysis was developed in response to a 2006 recommendation from the DOD Office of the Inspector General after a review found that the program office had not evaluated alternative support strategies prior to implementing a PBL arrangement in 2000. However, the 2007 analysis covered only the potential impacts of changing the type of contract used to obtain support for the program from cost plus award fee to firm fixed price. The business case analysis for the B-2 also did not analyze alternative support strategies but focused on potential efficiencies available to the current program through funding consolidation, funding stability, and long-term contracting. According to program officials, the only assumption in the analysis that actually occurred to some extent after PBL implementation was funding consolidation. Finally, although the C-17 program office developed a business case analysis in 2003, the DOD Office of the Inspector General reported in 2006 that the analysis focused only on one support option and did not evaluate the costs and benefits of multiple support options. In 2007, a contract was awarded for development of another business case analysis planned for completion prior to awarding the next C-17 support contract.

Two other important elements missing from some analyses were an evaluation of costs over the remaining life cycle of the program and the calculation of net present value to account for the time value of money. For example, the analysis for the Patriot PBL arrangement only evaluated the costs and benefits over a 3-year period. On the other hand, while the business case analysis for the Assault Breacher Vehicle evaluated costs over the 20-year life cycle of the program, the net present value of the alternatives was not calculated. Four other business case analyses—those prepared for the F/A-18 E/F, AV-8B Harrier, Close-In Weapon System, and Consolidated Automated Support System—did not include these two elements and several others, such as sensitivity or risk analysis. These analyses were prepared in a similar format by the Naval Inventory Control Point, an organization that provides supply support to the Navy, Marine Corps, and others. We conducted a net present value analysis on the amounts contained in the Naval Inventory Control Point’s business case analysis for the F/A-18 E/F PBL arrangement and found that the PBL option it chose was about $1.7 million more expensive than the alternative option. Its analysis, which did not use net present value, found that the PBL option was about $277,000 less expensive. The Naval Inventory Control Point’s philosophy is that if the costs of PBL are equal to or less than the costs for government-provided support, a PBL arrangement will be used. Therefore, if Naval Inventory Control Point officials had conducted a net present value analysis, based on this decision criterion, they would not have implemented the PBL arrangement. According to Naval Inventory Control Point officials, there is confusion in the Navy regarding the purpose of the analyses they prepare. Officials said that the analyses were conducted for internal decision making and were not intended to satisfy Navy PBL policy, which places responsibility for development of a life cycle business case analysis on the weapon system program office. However, program officials for the Close-In Weapon System, Harrier, and Consolidated Automated Support System did not develop business case analyses that evaluated PBL implementation over the remaining life cycle of the system.

Other Factors Influencing the Quality of Business Case Analyses

Several other factors affected the quality of the business case analyses we reviewed. Most of the analyses we reviewed did not identify and quantify the benefits that could be expected from contractor incentives to increase reliability or improve processes to reduce support costs. The only business case analysis that specifically identified cost savings based on contractor incentives to reduce costs was the 2005 life cycle business case analysis for the F/A-18 E/F. The life cycle analysis included an estimate of future cost savings associated with investments the contractor was willing to make under the arrangement to reduce supply chain costs. In addition, most of the analyses did not recognize or quantify the costs associated with the transfer of risk that occurs under a performance-based support arrangement. According to the DOD/Defense Acquisition University PBL guide, PBL arrangements transfer responsibility for making support decisions—and corresponding risk—to the support provider, and risk is one of the major cost drivers for contractors. Therefore, the use of performance metrics could introduce a large element of risk for the contractor that may be built into the costs of such an arrangement. In general, many of the business case analyses we reviewed simply estimated the costs for contract logistics support and government-provided support. One exception was the business case analysis for the Marine Corps’ Assault Breacher Vehicle, which attempted to address the costs of risk transfer and the benefits of contractor incentive to reduce costs by estimating the costs for a traditional contractor logistics support arrangement and a performance-based contractor logistics support arrangement, in addition to estimates for government-provided support and performance-based, government-managed support.

Another business case analysis was based on questionable assumptions. The 2002 business case analysis for the Sentinel program estimated that the cost for the government depot to overhaul the system was 50 percent of the total cost of the system. The business case analysis estimated that for the alternative option—a partnership between the government and the contractor with the government depot performing enough work to meet the system’s core requirements—the cost for an overhaul was only 25 percent of the system’s cost. The analysis assumed that under the partnership option the overhaul cost less because, instead of complete disassembly and parts replacement, the system would be inspected and repaired only as necessary. However, according to an official at the Army depot that would perform the overhaul, the depot also could have used the inspect and repair concept as the basis for its maintenance work. Therefore, this assumption in the business case analysis may have overstated the costs for the government depot to perform the work.

Lastly, the Naval Air Systems Command’s 2005 life cycle business case analysis for the F/A-18 E/F estimated that over the 28-year life cycle of the program, PBL support costs were $76 million more expensive than costs for government-provided support. However, the business case analysis estimated that Naval Inventory Control Point surcharges would be $325 million less for the PBL arrangement. The Naval Inventory Control Point adds a surcharge to the cost of goods sold to its customers (including weapon system program offices) to recoup its expenses, but according to officials, they do not adjust their resources based on PBL implementation and would still need to recoup their expenses through surcharges to other customers. Therefore, while the F/A-18 program office may realize a $325 million benefit from the reduced surcharge, the overall costs to the Navy may remain the same. Including this reduced amount in the analysis is inconsistent with DOD’s economic analysis instruction, which states that all measurable costs and benefits to the federal government should be included in the analysis. In addition, in the 2006 business case analysis prepared by the Naval Inventory Control Point, which estimated supply chain management costs for both PBL and government-provided support for a 5-year period from 2006 through 2011, the Naval Inventory Control Point’s costs were estimated to remain the same under either option. If the Naval Inventory Control Point’s costs had been the same in the life cycle business case analysis prepared by the Naval Air Systems Command for the F/A-18 E/F, the PBL arrangement would be $401 million more expensive than government support for the 28-year period.

Business Case Analyses Were Often Not Updated for Decision Making After PBL Implementation

In 2004, DOD guidance recommended that business case analyses continue throughout the weapons system’s life cycle and be updated at key decision points both to validate the approach taken and to support future plans. The services, with the exception of the Air Force, have also issued policies on conducting such updates. However, even when business cases were prepared, we found that program offices often did not update them in accordance with DOD’s guidance and service policies, nor did the program offices validate them against actual support costs for decision making after PBL implementation. Neither DOD nor the services has issued guidance that specifies what should occur when updating or validating a business case analysis.

Army policy states that PBL business case analyses shall be updated prior to the exercise of an option period when there are significant changes to the performance period/terms of the contract or evaluation period, or updated whenever there are major programmatic changes or at least every 5 years. Program offices for four of the Army PBL arrangements we reviewed had not updated their business case analyses as called for by Army policy. For example, the Tube-launched Optically-tracked Wire- guided missile – Improved Target Acquisition System program office developed a business case analysis in 1998 before awarding the original PBL contract and has not updated or validated the original analysis. Further, officials negotiated a follow-on PBL contract in 2007 after the terms of the original contract were complete. Although program officials for the Shadow Tactical Unmanned Aircraft System had planned to complete an update to their 2002 business case analysis by the end of 2007, the effort was delayed and the update is expected to be completed before the end of 2008. Although the Javelin program office implemented a PBL arrangement in January 2004, the business case analysis was developed in 2001, and program officials do not have plans to update the analysis. Additionally, although program office officials for the Army’s Sentinel weapon system had updated their 2002 business case analysis, Army policy calls for submission of the business case analysis to both the Office of the Deputy Assistant Secretary of the Army for Integrated Logistics Support and Army Materiel Command Headquarters for review and concurrence and then to the Program Executive Office for approval. In February 2008, before the new analysis had been reviewed and formally approved, another PBL contract was awarded.

The Navy’s 2007 business case analysis guidance calls for updates every 3 to 5 years or when significant programmatic changes occur. Based on this policy, the T-45 program office should complete an update to its business case analysis for the PBL arrangement for support of the aircraft’s engines by 2008. Although program officials updated their 2003 analysis in 2006 with actual support costs and flying hours, they did not expand the analysis to account for the remaining life cycle of the engines. The analysis projected costs only through 2008, the original contract period. Program officials did not plan to further update the business case analysis or prepare another one because they believed that it was not required.

Neither DOD’s nor the services’ policies clearly specify what should occur when a program office updates or validates a business case analysis. Although some programs are conducting another business case analysis, as mentioned earlier, program officials for the T-45 did not plan to conduct another analysis because they had updated their analysis with actual data. Program officials for the V-22 engine updated their 1998 analysis in 2004. The update focused on assessing if several of the ground rules, assumptions, and factors used in the original study were still valid and providing a preliminary recommendation on pursuing a follow-on PBL contract, from a cost standpoint. However, the entire analysis was not updated.

Lack of Specific Criteria and Inadequate Internal Controls Led to Business Case Weaknesses

Business case analyses were inconsistently used for PBL decision making because DOD did not require that the analyses be conducted and updated or provide specific criteria to guide their development. Further, with the exception of the Army, the services’ have not established effective internal controls to ensure that the analyses are prepared in accordance with service policies and guidance. As a result, DOD cannot ensure that decisions regarding weapon system support options are consistently guided by comprehensive and sound analytical tools.

DOD Guidance Has Not Provided Specific Criteria or Requirements for Conducting Sound Business Case Analyses

DOD guidance has not provided specific criteria for conducting and updating business case analyses for PBL decision making. Despite DOD’s preexisting economic analysis instruction recommending the analysis of both quantitative and qualitative factors, in 2001 DOD recommended the development of a business case analysis prior to implementing a PBL arrangement but provided little criteria for conducting such an analysis. In 2003, a Defense Business Board study recommended that DOD issue standard guidance for the services to take a more consistent approach to PBL decision making and also require the use of business case analyses. In response, the January 2004 Under Secretary of Defense (Acquisition, Technology and Logistics) memorandum was issued containing “guiding principles” for business case analyses. The memorandum stated that business case analyses “will evaluate all services or activities needed to meet warfighter performance requirements using ‘best value’ assessments.” This memorandum also listed several quantitative and qualitative factors for consideration when developing a business case analysis; however, it did not indicate how these factors were to be evaluated or their relative importance in the decision-making process. The memorandum also recommended that business case analyses be updated or repeated to validate the approach taken or to support future plans, but did not provide specific guidance as to when such updates should occur. According to the memorandum, a DOD PBL business case analysis handbook was supposed to be forthcoming. Later that year, the DOD/Defense Acquisition University PBL guide was published. It had two pages dedicated to the business case analysis concept—providing additional criteria and also incorporating the guiding business case analysis principles. However, a handbook specifically for PBL business case analyses was never issued.

In 2003, when DOD incorporated PBL implementation into DOD Directive 5000.1, which provides mandatory policies for all acquisition programs, a requirement to conduct and update a business case analysis was not included. Specifically, the directive only stated that acquisition managers shall use performance-based strategies for sustaining products and services whenever feasible and that such PBL strategies shall optimize total system availability while minimizing cost and logistics footprint. Also, despite the Defense Business Board’s recommendation later that same year to require the use of business case analyses, DOD subsequently neither required program managers to prepare the analyses prior to PBL implementation nor required them to update the analyses after implementation. In fact, although most of the services have issued some guidance and requirements for business case analyses, the current Defense Acquisition Guidebook no longer specifically refers to a business case analysis, but rather recommends the development of a “support strategy analysis” as part of the PBL implementation process. According to the guidebook, the support strategy analysis can be a business case analysis, economic analysis, decision-tree analysis, or other best-value-type assessment.

Service Policies and Guidance for Conducting Business Case Analyses Were Slow to Develop, Inconsistent, and Generally Not Enforced Because of Lack of Effective Internal Controls

Another reason for the inconsistent use of business case analyses is that the services’ policies and guidance for conducting the analyses were slow to develop and were generally not enforced because of a lack of effective internal controls. Moreover, we found inconsistencies among the services’ policies and guidance. In response to DOD’s recommendation that program offices conduct a business case analysis prior to implementing a PBL arrangement for weapon system support, the services issued their own policies and guidance. The time frames for these are summarized in table 4.

Although DOD recommended the use of business case analyses in 2001, the services’ business case analysis policies and guidance have evolved over time. In some cases, guidance was not issued until years later. For example, Marine Corps policy did not call for PBL business case analyses until 2007. Further, although the Air Force included business case analyses among mandatory procedures in 2004, these procedures were not specific. The Air Force’s instruction states only that “the program manager is responsible for construction of a business case analysis to determine the best strategy for meeting the PBL goals.” Final Air Force guidance for business case analyses, including PBL business case analyses, was not issued until 2008. As another example, the Army’s early business case analysis guidance was general in nature, and more specific policy memorandums were issued in 2005 and 2006. In 2007, these policies were included in an Army regulation.

Currently, service policies and guidance vary with respect to which programs should implement PBL arrangements, which of those programs shall conduct business case analyses, and how often program managers should update their business case analyses. Until 2007 the services’ policies and guidance varied significantly. However, the issuance of Navy guidance and Marine Corps policy in 2007 resulted in more consistency. Table 5 summarizes the services’ business case analysis policies and guidance.

With the exception of the Army, the services have not established the internal controls, including a review and approval process, necessary to ensure that business case analyses are conducted prior to PBL implementation and updated after implementation. For example, the Navy’s 2003 guidance assigns responsibility for reviewing individual business cases analyses to the system commands’ cost departments. However, the review only occurs when requested. Although a recently issued Air Force instruction calls for a formal review of all business case analyses, including those conducted for PBL arrangements, that meet certain criteria, it is unclear how many PBL business case analyses will meet any of the criteria and be subject to this review. The 2008 Air Force instruction calls for a review of all business case analyses that will be (1) forwarded outside of the Air Force; (2) forwarded to senior Air Force officials, such as the Secretary of the Air Force; and (3) provided for weapon systems that require Defense Acquisition Board or Air Force Acquisition Board approval.

In contrast, Army policy states that program managers shall report semiannually on the status of PBL implementation and that business case analyses for acquisition category I and II programs should be submitted for review and verification to multiple offices—including Army headquarters, the Army Materiel Command, and the Office of the Deputy Assistant Secretary of the Army (Cost and Economics)—and for approval to the Army Acquisition Executive. In addition, business case analyses for lower-level programs should be reviewed and approved but will not verified by the Office of the Deputy Assistant Secretary of the Army (Cost and Economics), and the approval authority is the program executive officer or commander of the related life cycle management command. While the Army’s policy first provided for these internal controls in 2005, Army officials said that no programs have yet passed the review and approval process completely.

Impact of PBL Arrangements on Reducing Weapon System Support Costs Is Unclear

The extent to which PBL arrangements are reducing costs for weapon system support is unclear and generally remains undocumented even after several years of PBL implementation. A major difficulty in assessing the cost impact of PBL arrangements is the lack of detailed and standardized cost data maintained by the program offices. Various other factors, such as the lack of systems that are supported by both PBL and non-PBL support arrangements, the lack of sound program baseline information, and changing operational and materiel conditions, also limited our ability to assess the impact of PBL implementation on support costs. While the overall cost impact was unclear because of a lack of data and these other factors, the limited evidence on cost impact that was available showed mixed results. We did find some evidence that a few PBL arrangements have reduced costs. However, we also found that characteristics of DOD’s PBL support arrangements, such as short-term contracts and unstable program requirements and funding, may limit their potential to reduce costs. Further, DOD has not sufficiently emphasized the potential to reduce costs as a goal for PBL programs. As a result, DOD cannot be assured that PBL arrangements will reduce support costs and provide cost-effective support for DOD systems.

Program Managers Often Lacked Detailed and Standardized Cost Data

In 2004, a memorandum from the Under Secretary of Defense (Acquisition, Technology and Logistics) recognized the importance of cost data for contract management and future cost estimating and price analysis and stated that PBL contracts shall include cost reporting requirements. However, for the PBL arrangements we reviewed, program offices often did not have detailed cost data that would provide insights regarding what the program office was spending for various aspects of the support program—such as the cost of depot maintenance by subsystem and major component or the cost of engineering support, supply support, and transportation. When cost data were available, the level of detail and format of cost data varied considerably. This condition significantly affected our ability to determine the impact of the implementation of PBL on the costs of supporting the systems in our sample, as many factors influence support costs.

For PBL arrangements using fixed-price contracts or fixed-price contract line items—DOD’s “ideal” type of PBL arrangement—we found that program offices generally did not receive detailed cost data for the program and only knew the overall amounts paid for support. Only two program offices in our sample obtained contractor support cost data for their fixed-price PBL arrangements, and the format and contents of the reports were very different. For example, the F/A-18 E/F program office obtained Boeing’s report on fiscal year 2006 costs, including general/administrative costs and profit, in a detailed reporting structure approved by the Office of the Secretary of Defense (OSD), Cost Analysis Improvement Group. According to program officials, an annual cost reporting requirement was included in the 2005 fixed-price PBL contract to provide cost visibility into the program and was added at no additional cost to the government. In contrast, the B-2 program office receives a monthly funds utilization report that allocates the amount the Air Force pays the contractor into seven high-level categories, such as planned depot maintenance.

Although the PBL arrangements that used cost-reimbursable contracts generally obtained more detailed cost data than those with fixed-price contracts, the format and level of detail also varied. For example, under the 1998 PBL contract, the F-117 program office did not receive cost data in a format that was detailed enough to report in OSD’s standard support cost structure. The program office subsequently required more detailed cost data reporting from Lockheed Martin in the 2006 PBL contract. As another example, the 2003 C-17 PBL contract has both fixed-price and cost-reimbursable elements. According to program officials, Boeing did not report support cost data at the level of detail necessary to report in OSD’s support cost structure under the contract. According to an Air Force Cost Analysis Agency official, a cost-reporting requirement was included in the contract’s option years and more detailed cost reporting will begin in fiscal year 2009.

Limited Available Evidence on Cost Impact Indicates Mixed Results

Although cost data were generally lacking, the limited available evidence on cost impact showed mixed results. Data we reviewed for the two systems that were managed by both PBL and non-PBL arrangements indicate that the PBL arrangements were more costly, but based on other assessments of available data, there are some indications that PBL arrangements can reduce costs. However, in seven out of eight programs we reviewed where follow-on, fixed-price PBL contracts had been negotiated, expected cost reductions either did not materialize or could not be determined. Finally, we noted that officials reported performance levels for some programs that were significantly higher than required under the PBL arrangement, but it is unknown whether the required levels could be achieved at a lower cost.

Two Comparisons of PBL Arrangements to Traditional Government Support Show That PBL Arrangements Cost More

Of the 29 programs we reviewed, only the F100-PW-220 engine and the KC- 130J/C-130J airframes are maintained by both PBL arrangements and traditional government support strategies. We found that the Air Force’s traditional support arrangement for the F100-PW-220 engine costs slightly less than the Navy’s PBL arrangement for the same engine. The Navy uses the F100-PW-220 engines in its F-16A/B aircraft and sustains the engines through a PBL contract with Pratt & Whitney. The Air Force uses the same engines in its F-16 and F-15 aircraft and supports the engines at the Oklahoma City Air Logistics Center. The Air Force maintains an engine total ownership cost estimate that includes all costs incurred (depot-level repairables, general services division (expendable repair parts), depot programmed equipment maintenance, organizational-level maintenance, intermediate-level maintenance, and continuous improvement program). To compare the Navy’s PBL costs with the Air Force’s engine total ownership costs, we removed the costs associated with organizational- level maintenance from the Air Force’s costs. As shown in figure 2, converted to costs per flight hour, the Navy’s PBL costs were slightly higher than the Air Force’s costs in constant fiscal year 2008 dollars.

Although the cost difference appears to be decreasing, the Navy’s 5-year contract ended in 2008 and a new PBL contract has not yet been negotiated. The engines are currently being supported under a 6-month extension to the original contract, and the fixed price paid per engine cycle is significantly higher than that paid during the previous 5 years. According to Navy officials, the decision to contract with Pratt & Whitney for the support of the Navy’s engines was not solely based on costs but was also based on other factors, such as turnaround time for engine repairs. However, program officials could not provide the data on which they based their decision.

Elements of the Air Force’s PBL arrangement to support the C-130J airframe are more expensive than the support for the KC-130J airframe provided by the Navy. According to Navy officials, an analysis was prepared in 2005 to compare costs for alternative repair arrangements to determine whether to continue using the Navy’s repair capability or to transition to contractor-provided repair in 2006. The Navy’s analysis concluded that the support provided by the Naval Surface Warfare Center, Crane Division, would cost 43 percent less than the support provided by the contractor. The analysis was based on anticipated 2006 flight hours, actual government support costs from 2005, and the costs to exercise an option for repair under a preexisting contract. Additionally, we independently compared overall costs for inventory management and repair of repairable components and found that the Air Force’s PBL costs on a per flight hour basis for these elements were significantly higher than the Navy’s costs—approximately 131 percent higher in 2006 and 164 percent higher in 2007. However, according to officials, several factors account for some of the difference. For example, the Air Force’s PBL arrangement includes 36 percent more consumable and repair parts than the Navy’s support arrangement, maintenance of support equipment, and support for six locations, while the Navy’s arrangement includes support for only three locations.

A Few Indicators Showed That PBL Arrangements Reduced Support Costs for Some Programs

Only a few of programs we reviewed were able to provide some indicators of reduced weapon system support costs that could be attributed to the use of a PBL arrangement. As mentioned earlier, some programs did not have a business case analysis demonstrating how current support costs compared to other support approaches. Of the nine PBL arrangements that had been implemented and had a business case analysis that looked at alternative support options, only four could be compared with PBL contract costs. Based on this comparison, three of these four PBL arrangements indicate potential savings from PBL implementation, while the fourth is more expensive than estimated in the business case analysis. The remaining analyses could not be compared to actual program costs after PBL implementation because of programmatic changes that occurred after the analyses were conducted.

The 2005 business case analysis for the Army’s Patriot estimated a 3-year cost savings of $1.6 million from using a PBL arrangement to provide 107 high-demand parts. According to a program official, the contract is in its final year and total obligations are expected to be about $1 million less than estimated in the business case analysis. Additionally, two business case analyses prepared by the Naval Inventory Control Point estimated that supply chain management support costs were reduced by implementing a PBL arrangement. The business case analyses projected cost savings of about $2.2 million for the 5-year Close-In Weapon System PBL arrangement awarded in 2006 and $1.3 million for the 5-year Harrier PBL arrangement awarded in 2007. Based on actual contract costs—and if the contracts are not modified in the future—the total savings for these programs are projected to be $5.2 million and $5.8 million, respectively. Although the F/A-18 E/F business case analysis estimated a 5-year supply chain management savings of approximately $1.4 million, the actual contract cost is about $1.6 million more than the estimated amount in the analysis. Given the difference, the PBL arrangement has not reduced support costs for the program. As previously noted, two of the PBL arrangements having evidence of reduced support costs are managed by the Naval Inventory Control Point. This activity has used PBL arrangements since fiscal year 2000 and has reported achieving cost reductions as a result, using the Navy working capital fund to issue longer- term, multiyear contracts that can extend up to 5 years in length to support aircraft or ship subsystems or components. According to agency officials, these longer-term agreements have enabled the Naval Inventory Control Point to guarantee the contractors a more stable business base, which provides contractors an incentive to make investments to reduce costs. Overall, as a result of using PBL arrangements, Naval Inventory Control Point officials estimate that they have reduced support costs by approximately $26.7 million and $63.8 million—or 2.8 and 5.8 percent—in fiscal years 2006 and 2007, respectively.

Although the V-22 program conducted a business case analysis in 1998 to estimate alternative costs for supporting the engines and projected savings of $249.5 million over the 53-year life cycle of the program, the analysis did not take into account the time value of money and calculate savings based on net present value. For this and other reasons, we cannot validate that the savings are reasonable. In addition to DOD’s economic analysis instruction, guidance from the Office of Management and Budget also states that net present value is the standard criterion for deciding whether a government program can be justified on economic principles. In 2004, another analysis was prepared for the V-22 engine program to determine (1) if several assumptions used in the 1998 analysis were still valid and (2) the impact of any changes to those assumptions on the cost savings estimate for the PBL arrangement. The later analysis concluded that differences in three of the original assumptions increased the projected PBL cost savings to $305.9 million—an increase of $56.4 million. The updated savings again were not calculated using net present value. Moreover, although limited actual data were available, the calculations generally made adjustments using assumptions that generated the maximum potential savings for the PBL alternative. For example, when adjusting the 1998 analysis to account for differences in the costs experienced for excluded repairs (repairs that were not covered by the PBL arrangement), the total potential PBL cost savings were increased by $48 million because the average excluded repair cost was lower than previously estimated. However, even though data showed that excluded repairs occurred at a higher frequency than projected in the original analysis, the later analysis did not adjust for the higher frequency of excluded repairs. Thus, the savings calculation is questionable, because the analysis noted that the frequency of these repairs could eliminate all of the estimated cost savings. Finally, the 10-year-old analysis has not been completely updated to estimate costs based on actual data.

The remainder of the analyses could not be compared to current PBL arrangement costs because of programmatic changes that occurred after the analyses were conducted. For example: According to an Air Force C-130J program official involved in the development of the 2004 business case analysis, the analysis was conducted while the aircraft was supported by a commercial contract; therefore, the program office did not have detailed cost data on which to base the estimate. The estimate was developed, in part, using cost data from other legacy programs and other assumptions that program officials said did not turn out to be accurate. Thus, though the business case analysis helped program officials develop the program’s support strategy, the cost estimates contained within are not useful for monitoring current program costs.

The 2002 business case analysis for the Army’s Sentinel PBL arrangement estimated costs for a 26-year period beginning in 2003 using a fleet size ranging from 126 to 198 radars. According to program officials, since 2003 the fleet size has ranged from 140 to 143 radars and additional radars are not anticipated. Although a new business case analysis was prepared, it had not completed the Army’s review and approval process at the time of our review.

PBL Cost Reductions Either Did Not Materialize or Were Not Documented for Most Follow-on Contracts

Few of the remaining programs in our sample could document cost reductions attributable to the use of a PBL arrangement after negotiating a follow-on fixed-price contract. The PBL concept envisions that support providers are incentivized to improve reliability to ensure that performance metrics are met and reduce their costs to provide support to increase profits—especially under fixed-price arrangements. To the extent practicable, we examined how contract costs changed for eight programs in our sample that negotiated follow-on contracts or priced previously unpriced contract options after completing fixed-price PBL contracts. According to officials, a variety of factors affected the support costs negotiated in the PBL contracts that caused both costs increases and decreases. Only one program had decreasing support costs that program officials attributed to the use of a PBL arrangement. One additional program supported under a cost-plus-award-fee contract also reduced support costs by changing the metrics included in the contract. However, these two programs did not have updated business case analyses that analyzed alternative support options over the remaining life cycle of the program. Finally, only one program office had developed a methodology for tracking and verifying reliability improvements made under the PBL arrangement, although this is necessary for quantifying the related cost savings.

Support costs for the Navy’s Consolidated Automated Support System have decreased over the 8-year PBL arrangement that began in April 2000. Program officials attribute the cost reductions the program has experienced to the implementation of a PBL arrangement. Depending on the level of support chosen, the fixed price charged for the annual support of a test station decreased from 53 to 20 percent (constant 2008 dollars) from 2000 through 2008. Program officials said that they closely monitored maintenance data and failure rates in order to negotiate lower fixed prices where possible. In addition, officials said that they were able to increase the number of repair and consumable parts covered under the arrangement over the years. According to officials, prior to the implementation of the PBL strategy support costs for the program were even higher, but officials were unable to locate the contracts.

Although support costs for a few of the other seven programs decreased, officials said that there were a number of other factors that influenced costs, such as changes in the scope of work or planned usage of the systems. For example, according to Tube-launched Optically-tracked Wire- guided missile – Improved Target Acquisition System program officials, a variety of factors affected the costs negotiated in the 2007 contract, and increased fleet size was one factor that allowed them to negotiate lower rates per system. In addition, when the first fixed-price PBL arrangement was implemented in 2001 the program was fairly new with very few systems, so the program office did not have an extensive amount of historical program data with which to negotiate. Since 2001, the program office has collected actual data that it used to negotiate lower rates in the latest contract. However, according to program officials, the contractor only recently started making changes to the system to improve reliability. These improvements were not included in negotiations for the 2007-2011 contract but have begun to improve failure rates and are expected to reduce costs in future contracts.

Although the Army’s Shadow Tactical Unmanned Aircraft System is not supported by a firm-fixed-price PBL contract, program officials for the system said that they were able to reduce support costs by changing the performance metrics used in the PBL arrangement. The maximum amounts authorized in the annual cost-reimbursable PBL contract for the support of this system were reduced by 28 percent from fiscal years 2006 through 2007. According to program officials, a program office review of PBL processes in early fiscal year 2006 concluded that while the PBL arrangement was effective in terms of meeting the performance levels specified in the contract, it was not cost efficient and costs associated with the vehicle’s high accident rate were an area of particular concern. In response, the program office changed the performance metrics in the contract to encourage the contractor to improve reliability and reduce the accident rate, and also to improve depot maintenance efficiency. As the accident rate improved, the program office was able to negotiate for lower support costs in the 2007 PBL contract.

Finally, while the 2005 life cycle business case analysis for the F/A-18 E/F program office estimated that support provided under a PBL arrangement would be more expensive than government-provided support, program officials for the Navy’s F/A-18 E/F PBL arrangement have developed a process to track and document support cost reductions attributed to contractor investments to improve reliability and reduce costs. Program officials said that both the Navy and Boeing have funded initiatives to improve F/A-18E/F component reliability, maintainability, and supportability as part of the Supportability Cost Reduction Initiatives program. Under the current fixed-price PBL arrangement, Boeing has invested approximately $11.39 million to fund initiatives that officials currently estimate will generate cost reductions of approximately $279 million over the remaining life cycle of the system. According to program officials, Naval Air Systems Command cost analysts have validated baseline estimates and will annually track the results of the initiatives in order to quantify actual support cost reductions attributed to the investments in the future.

Costs to Achieve Higher-Than- Required Performance Levels Are Unknown

According to program officials, eight of the PBL arrangements within our sample of 29 systems generally achieved a level of performance that significantly exceeded what is required under the contracts. According to the DOD/Defense Acquisition University PBL guide, PBL arrangements should be structured to meet the needs of the warfighter. Therefore, if actual performance exceeds what is called for in the PBL arrangement, it also exceeds the level of performance that is needed. According to program officials, for eight of the PBL arrangements we reviewed, the contractors significantly exceeded some of the contractual performance requirements. For example: Since 2002, Army officials said that the average annual operational readiness for the Tube-launched, Optically-tracked, Wire-guided missile – Improved Target Acquisition System has not been below 99 percent, and the system’s operational readiness has averaged 100 percent since 2004. According to a program official, the Army’s readiness standard for this system is 90 percent. Despite the Army’s standard, it continued to include a performance incentive that encouraged higher levels of performance when negotiating a follow-on PBL contract in 2007. The performance incentive includes payment of an award fee that encourages operational readiness rates from 91 to 100 percent, with the highest award fee paid for 100 percent average operational readiness.

According to officials, since early 2005, monthly readiness rates for the Army’s Javelin have generally been measured above 98 percent.

However, the PBL contract for support of this system only requires 90 percent mission readiness.

Although the contractual requirement for parts availability for the Navy’s V-22 engine PBL arrangement has been 90 percent since 1998, according to program officials, actual parts availability has consistently averaged 98 to 100 percent.

Similarly, with availability averaging 98 percent since 2004, Air Force program officials for the LITENING Advanced Airborne Targeting and Navigation Pod said that the contractor has consistently exceeded the contract requirement for 92 percent system availability.

For programs where performance significantly exceeded contractual requirements, it is unclear how much extra was paid to get the added performance. Since the government is paying for this excess performance, then the arrangement, as structured, may not provide the best value to the government, particularly since there are other DOD programs that are not funded at levels that would be required to achieve their desired level of readiness.

Characteristics of DOD’s PBL Arrangements May Limit Their Potential to Reduce Support Costs

Several characteristics of DOD’s PBL arrangements may limit their potential to reduce costs. First, DOD’s PBL contracts are limited to relatively short time periods, while proponents of the PBL concept believe that longer-term PBL arrangements are necessary to encourage support providers to make investments to improve reliability. Second, in DOD— where changing requirements and priorities can result in fluctuations in the funding for support of DOD’s weapon systems—creating a stable level of funding is challenging. Third, many PBL arrangements only transfer responsibility for inventory management to the contractor and do not transfer inventory ownership, which reduces incentives for ensuring a correctly sized inventory level. Finally, many of DOD’s PBL arrangements do not contain cost metrics or offer specific incentives to encourage cost reduction initiatives.

Short-term Contracts

According to program officials, DOD support contracts, including PBL contracts, that are funded by operation and maintenance funds are generally limited to 1 year, and working-capital-funded contracts are generally limited to 5 years, with subsequent option years allowed up to a total of 10 years. However, according to the DOD/Defense Acquisition University PBL guide, longer-term PBL contracts are preferred because a key aspect of PBL is the provision of incentives for contractors to reduce costs over time through increased reliability while still making a profit. Further, contract length should be sufficient to allow for an adequate return on any investments made to improve reliability.

Officials from several PBL arrangements cited instances in which reliability improvements were needed but contractors were hesitant to make investments while under annual support contracts. For example, Joint Primary Air Training System program officials said that during the original 10-year PBL arrangement that began in 1996, the contractor did not make any investments to improve unreliable components. Although officials were expecting the fixed-price performance contract to motivate the contractor to invest in improvements to increase reliability and maximize profit, they found that the contractor minimized its own costs during the contract period and passed on the costs to improve the reliability of components with high failures to the government when the contract was renegotiated. Our prior work found that the private sector sometimes used PBL contracts of 10 to 12 years.

Unstable Requirements and Funding

Stable requirements and funding, like longer-term contracts, could enable contractors to make reliability improvements and other business decisions, such as long-term supplier arrangements, that could improve performance and reduce future support costs because they have reasonable assurance of future needs. For example, officials representing one of the PBL arrangements we reviewed credited stable funding for much of the program’s cost savings. The F-117 program office estimated that its arrangement would have cost over $80 million more if the Air Force had not agreed to stabilize the program’s support budget and provide the necessary support funding each year of the contract. However, DOD’s requirements and priorities, and related funding, for weapon system support are not always stable. For example, according to Army officials, the Tactical Airspace Integration System’s PBL arrangement was affected by a significant reduction of the program’s support budget. The Army subsequently requested that the Defense Acquisition University study the implications of funding on PBL arrangements and prepare a case study based on this example. In addition, for the last several years some of the Army’s PBL arrangements we reviewed did not receive all of their support funds at the beginning of the fiscal year but rather in increments throughout the year. Program officials for one Army system said that at one point during fiscal year 2005, they almost had to discontinue some of the support provided under the contract because they did not have adequate support funds. Additional funding was eventually made available after the program office notified its major command of the situation. Army program officials said that this funding instability further exacerbates the impact of having short-term contracts, since all of the funds are not available to the contractor to make business arrangements or investments for reliability improvements.

DOD’s Ownership of Inventory

Many of the PBL arrangements we reviewed only transferred responsibility for inventory management, not ownership, to the contractor. An analysis by Sang-Hyun Kim, Morris A. Cohen, and Serguei Netessine of the Wharton School, University of Pennsylvania, suggests that the efficiency of a PBL arrangement depends heavily on the asset ownership structure: with higher ownership responsibility, the supplier is more likely to spend more to increase reliability and less for maintaining inventory levels. According to this study, under an arrangement in which the contractor owns and manages the inventory, reliability improvements and inventory costs will both be evaluated in terms of their ability to meet performance metrics and minimize costs. If the PBL arrangement only includes inventory management, higher inventory levels may be used, instead of investments to improve reliability, to meet performance metrics—particularly those that measure availability—since inventory holding costs are not incurred by the contractor. Consequently, under DOD’s PBL arrangements, contractors may choose to make fewer reliability improvements.

Availability of Cost Metrics and Effective Incentives

Finally, many of DOD’s PBL arrangements do not contain cost metrics or offer specific incentives to encourage reduced costs. According to an August 2004 memorandum from the Under Secretary of Defense (Acquisition, Technology and Logistics) regarding performance-based criteria, PBL should be constructed to purchase performance, which is defined in terms of operational availability, operational reliability, cost per unit usage, logistics footprint, and logistics response time. The guidance recommended that PBL metrics be tailored to reflect the unique circumstance of the arrangement, but still support desired outcomes in terms of the five performance criteria. A subsequent 2005 memorandum from the Under Secretary of Defense (Acquisition, Technology and Logistics) directed the use of these metrics as the standard set of metrics for evaluating overall total life cycle systems management.

Some of the aviation PBL arrangements we reviewed negotiated their support on a cost per flight hour basis. For those that did not, cost per flight hour was generally not included as part of the contract performance plan, with the exception of the F/A-18 E/F PBL arrangement. For example, the C-17 program office did not negotiate its contract on a per flight hour basis and does not monitor cost per flight hour as part of its PBL arrangement. None of the nonaviation PBL arrangements we reviewed included cost metrics as part of the PBL arrangement. In addition, only four of the PBL arrangements we reviewed contained incentives for reducing or controlling costs. For example, the F-117 and Shadow Tactical Unmanned Aircraft System PBL arrangements each included a cost- sharing provision where the government and the contractor would share annual savings if actual costs were below negotiated costs. Further, officials said that the award plan for the F-22 PBL arrangement also will consider how actual costs compare to negotiated costs when calculating the amount of award fee the contractor earns at the end of the year.

DOD Did Not Emphasize the Potential for PBL Arrangements to Reduce Costs or Require the Collection and Reporting of Detailed Contractor Support Cost Data

Although PBL arrangements were included in a DOD pilot program intended to demonstrate the ability of various initiatives to reduce support costs, DOD did not emphasize this goal in its guidance or requirements as it established the concept as the department’s preferred weapon system support strategy. In general, improved performance was given greater emphasis, and we found only a few references to cost reduction in DOD’s guidance on implementing PBL arrangements. With respect to requirements for cost reporting, DOD and the services do not require that programs using PBL arrangements, or other contractor logistics support arrangements, collect and report detailed cost data in a consistent, standardized format.

Since 2001 DOD’s guidance regarding PBL has emphasized higher levels of readiness and stressed rapid implementation. For example, in 2001, when DOD cited PBL as the preferred weapon system support strategy, PBL was described as a strategy for achieving a higher level of system readiness through efficient management and direct accountability. In a 2002 Under Secretary of Defense (Acquisition, Technology and Logistics) memorandum, the services were instructed to prepare PBL implementation plans that aggressively pursue the earliest feasible program implementation end dates. A January 2004 Under Secretary of Defense (Acquisition, Technology and Logistics) memorandum stated that PBL was the department’s near-term strategy to increase weapon system readiness through integrated logistics chains and public/private partnerships. The memorandum contained guidance to implement PBL where economically feasible and provided guiding principles for a best- value assessment. The following month a Deputy Secretary of Defense memorandum again directed the services to provide plans for aggressively implementing PBL arrangements.

In contrast to DOD’s clearly stated goal to reduce support costs in the late 1990s, we found few references to the potential for PBL to reduce support costs since 2001. DOD guidance generally only indirectly refers to potential PBL cost reductions to “compress the supply chain” and “reduce non-value added steps.” In May 2003, DOD Directive 5000.1, The Defense Acquisition System, was updated to emphasize that program managers shall implement PBL strategies “that optimize total system availability while minimizing cost and logistics footprint.” In March 2004, an Under Secretary of Defense (Acquisition, Technology and Logistics) memorandum reiterated that PBL was the preferred strategy and provided criteria on which to assess potential for PBL application. One of the criteria stated that the cost per operational unit of performance (such as a flying hour) should be capable of being reduced through PBL implementation. Finally, in 2005, the DOD/Defense Acquisition University PBL guide contained several references to the potential for PBL to improve reliability and reduce costs.

Program offices often lacked detailed and standardized weapon system support cost data because DOD has not required them to obtain and report cost data from the contractors that provide such support, including those involved in PBL arrangements. According to the OSD Office of Program Analysis and Evaluation, historical operating and support costs, organized in a standard format, are necessary for preparation of life cycle cost estimates for new systems, budget formulation, analysis of working capital funds, development of business case analyses, and future contract negotiations. Until 2007, DOD’s guidance for structure of support cost estimates, which is also suggested as a defined presentation format for historical operating and support costs, included all contractor support— labor, materials, overhead, and other assets—in one category, while government-provided support was reported in greater detail among multiple categories and lower-level subcategories. Therefore, amounts paid for contractor support were generally reported in the aggregate. In October 2007, DOD changed its guidance to include a more detailed presentation of contractor support costs in the various categories, similar to the reporting of government support costs. However, neither DOD nor the services have required program offices to obtain or report contractor support costs, including PBL arrangements with contractors, in this format.

OSD and service officials are beginning to recognize the need for further visibility of the costs of support provided by contractors. In late 2006, OSD’s Office of Program Analysis and Evaluation began a study regarding the collection of contractor support costs because the department acknowledged that visibility into these costs in DOD’s systems was generally limited. Many of the programs studied were PBL arrangements also included in our sample. OSD’s study also found that program offices often did not have detailed cost data and, if cost data were provided, the data often did not conform to, or could not be converted to, the standard support cost structure. Based on the study results, OSD is considering requiring contractors to report their actual costs for providing logistics support, including profit and general and administrative expenses, in DOD’s standard cost structure. However, the details of the requirement and which programs will be subject to such reporting have not been finalized.

Similarly, Air Force officials have also recognized the limitations on visibility into contractor support costs for weapon systems. The Air Force is currently considering expanding visibility by requiring that all contractor-supported programs report actual obligations for contractor labor and materials (including PBL arrangements) in each of DOD’s cost structure categories for each aircraft mission design series. According to Air Force Cost Analysis Agency officials, this requirement is different from the one being considered by OSD in that the Air Force will have visibility over the Air Force’s costs for contractor support but not the contractor’s actual costs.

Use and Characteristics of Availability Contracts by the United Kingdom’s Ministry of Defence

The United Kingdom’s Ministry of Defence also uses performance-based arrangements to support its weapon systems. Ministry of Defence officials refer to this initiative as contracting for availability. Similar to DOD, when using availability contracts the Ministry of Defence pays industry for aircraft, engines, or components to be available for military operations, rather than paying for specific repairs, spares, and technical support. According to officials, the use of contracting for availability also started as an approach for reducing costs for weapon system support. Ministry of Defence officials said that their current contracts for availability generally provide support for aviation systems, such as helicopters and combat aircraft. Although there are maritime availability contracts, they said that most of the ministry’s maritime availability contracts support specific types of equipment rather than entire ships. In general, the availability contracts used by the ministry are significantly longer than those used by DOD, and the ministry uses an “open book accounting” arrangement to gain visibility into the contractors’ costs to provide support. According to officials, the annual budget for the Defence Equipment and Support organization is approximately £13 billion, including funds for conflict operations.

Ministry of Defence Budget Pressures Drove Need for Support Cost Reductions

In 1999, the United Kingdom’s Defence Logistics Operation, one of two entities that merged into the current Defence Equipment and Support organization, established a goal to reduce costs 20 percent by 2005/2006. According to Ministry of Defence officials, contracting for availability began during this period as a way to maintain or improve performance while assisting in achieving cost reductions. They believe that if industry is paid for a given level of availability, there are incentives to reduce support chain costs and make the weapon system more reliable and processes more efficient. The cost reduction goal was a key driver in the transformation of the maintenance, repair, and overhaul activity for Harrier and Tornado fast jet aircraft. A member of the Tornado Integrated Project Team stated that a number of factors drove the support strategy change for the Tornado aircraft, but the primary factor was the need to reduce costs to match budget reductions; the team identified availability contracting as an effective way to reduce costs and maintain performance. Officials also stated that the support strategies for all of the ministry’s helicopters were changed because of increased budget pressures.

In 2007, the United Kingdom’s National Audit Office reported that the Ministry of Defence has experienced significant reductions in the costs to support its fast jets; the Tornado and Harrier costs have been reduced from a total of £711 million in 2001through 2002 to £328 million in 2006 through 2007, providing a cumulative saving of some £1.4 billion over the 6-year period. The National Audit Office reported that the savings were achieved by working with industry to reform traditional contracts into availability contracts. However, the report also stated that the ministry did not have sufficient data to assess the impact of changes in the pattern of frontline operations and productivity increases from the use of lean techniques on total costs. National Audit Office officials with whom we met confirmed that while they could validate overall cost reductions, they could not attribute the entire savings solely to the use of availability contracts. Other related initiatives, such as the reorganization and reduction of locations for aircraft repair and upgrade, the use of lean techniques, and the use of reliability-centered maintenance, also contributed to the support cost reductions.

Analytical Tools Used for Decision Making

Ministry of Defence officials said that they do not require the use of availability contracts or promote their use as the preferred strategy. According to officials, the support strategy can and should vary from system to system depending on the circumstances; in some cases, it may be appropriate for government activities to support some systems in the traditional manner and for others to use contracting for availability. To assist with the decision-making process, the Defence Equipment and Support organization developed a “support options matrix” for use in reviewing current and future support arrangements. Officials said that the matrix was developed to assist with analyzing components of support for cost and performance drivers, illustrating a range of support options differentiated by the gradual transfer of cost and performance drivers into industry management and presenting a clear rationale for each support chain design in terms of the benefit to be derived from the transfer of specific cost and performance drivers into industry management. In addition to the matrix, a contractor capability assessment is also completed to determine the ability of industry to assume greater management responsibility. Finally, according to officials, before they enter into a contract for availability, two additional analyses are conducted. The first is an investment appraisal, or an “internal value benchmark,” which calculates the lowest cost at which the service could be provided by the government. The second is a business case analysis, which discusses the different proposals and justifies the selection of the proposed approach. Officials noted that the proposed approach does not have to be the lowest-cost option, but is usually the option that offers the best value solution overall.

In its 2007 report, the National Audit Office indicated that internal value benchmarks were not developed consistently and recommended development of improved guidance and consistent application of a common methodology for benchmarks against which to assess the value of proposed availability contracts. National Audit Office officials said that they found variance in the quality of these cost estimates and a shortage of qualified people for cost modeling. They also pointed out that as less and less support is provided by the government, accurate cost modeling for use when renegotiating contracts will become more important, and the Ministry of Defence needs to maintain or improve visibility of support costs for its weapon systems.

Long-term Contracts (19- 25 Years) Are Critical to Success

Defence Equipment and Support officials said that they have found the long-term nature of availability contracts a key factor in reducing costs and that annual contracts cannot achieve the same benefits as the longer- term contracts do. According to officials, the long-term contracts for Tornado aircraft and helicopter fleets reduced costs because the contractors were able to stabilize their supply chain and obtain better prices from the supplier base. The Ministry of Defence also found that industry preferred long-term contracts. In a discussion of contracting for availability, the “Defence Industrial Strategy,” a white paper dated December 2005, stated that companies are generally interested in using availability contracts because it provides the commercial firms with greater returns over a longer period.

Ministry of Defence officials provided us with the following examples of their long-term availability contracts: The Ministry of Defence has a 10-year contract with AgustaWestland to support the Sea King helicopter until it is projected to be removed from service. The Ministry of Defence has priced the contract for the first 5 years, and thereafter it will establish the price in 5-year increments.

The Ministry of Defence has a 23-year contract with VT Group to support two survey ships owned by the ministry. The contract has price renegotiation points at the 7-, 15-, and 20-year points.

The Ministry of Defence has a 19-year contract with BAE to support the fleet of Tornado aircraft. The ministry awarded the contract in December 2006 and priced it for the first 10 years.

The Ministry of Defence has a 25-year contract with AgustaWestland to support the Merlin helicopter until it is projected to be removed from service. The price for the initial 5-year period of the contract is fixed, and the ministry is currently negotiating prices for the next 5-year period of performance that begins in 2011.

Other Characteristics of Ministry of Defence Availability Contracts

Ministry of Defence officials said that other factors, such as inventory ownership, contract incentives, and cost visibility, were also important when contracting for availability. Officials told us that they preferred to transfer not only management of inventory but also inventory ownership under such arrangements. They noted that under some of their current availability contracts this had not been possible for a variety of reasons. Nonetheless, in the future they intend to pursue transfer of inventory ownership as much as possible. Examples of Ministry of Defence availability contracts where officials said that inventory is owned by industry, also known as spares inclusive, include a contract for support of two survey ships.

In addition, according to ministry officials, several of the availability contracts—including those supporting the Sea King and Merlin helicopters and Tornado fast jets—had incentives referred to as gain share or pain share. In these types of arrangements, the contractor and government share cost savings or cost overruns in prenegotiated proportions. According to officials, they found that these types of metrics are useful to influence contractor cost control measures and provide an incentive for industry to develop changes and modifications that reduce support costs. Officials familiar with the Tornado fast jet availability contract explained that their arrangement included gain sharing and pain sharing on both the variable and fixed-price portions of the contract.

Finally, officials explained that in many of the Ministry of Defence’s availability contracts, the concept of open book accounting is employed. Open book accounting is not a defined term but is more of a general expression describing a level of access to accounting data that would not normally be available under a conventional contract. In availability contracts, open book accounting allows government program officials to review the accounting records of the contractor. This access is not without limits. Officials said that the level of access must be agreed to in advance on a case-by-case basis and reflects the circumstances of the arrangement and the need for access to certain data to monitor performance or benefits arising from the arrangement. For example, one contract may only provide for man-hour data because that is all that needs to be shared given the circumstances. However, another contract may allow access to direct cost, direct labor hours, and other rates and factors that are relevant for the work involved. According to officials, the Ministry of Defence has an open book accounting agreement with AgustaWestland for the Merlin contract and the government has full visibility of the accounts pertaining to Merlin, including overhead costs. The contract must explicitly address the data access arrangements and not rely on vague and undefined phrases that could be open to misinterpretation.

The Ministry of Defence’s Long-term Availability Contracts May Affect Budget Flexibility

According to the 2007 National Audit Office report, long-term availability contracts may limit flexibility to respond to changes in resources. In the past, integrated project team leaders in the Ministry of Defence had some ability to move funding between resource lines to overcome short-term funding issues. However, this flexibility is diminishing because of the transition to availability contracts, as larger portions of the budget are pre- allocated to fund these contracts. The Mine Warfare Patrol and Hydrographic Team also raised concerns about loss of budget flexibility. This team is responsible for providing support for 2 hydrographic ships, 1 patrol ship (HMS Clyde), 3 River class ships, 16 mine hunters, and 38 smaller ships. The budget for providing support to these ships is approximately £40 million, with £18 million devoted to the long-term availability contracts for the 2 survey ships, 1 patrol ship, and 3 River class patrol ships. According to Ministry of Defence officials, these arrangements have for the most part been beneficial. However, as they are structured, these programs do not allow for any flexibility. When the Mine Warfare Patrol and Hydrographic Team recently had to absorb a 20 percent budget cut, officials said that the mine hunter ships bore the brunt of the cut because they had the majority of the remaining support budget not earmarked for an availability contract. The team views the 20 percent cut to its budget to be, effectively, a 40 percent cut to the mine hunter ship budget. Defence Equipment and Support organization officials said that they are looking to add more flexibility to future availability contracts.

The Ministry of Defence has already incorporated some flexibility in a few availability contracts. Officials said that the Tornado contract contains both fixed-price elements for management team, training, logistics, and information systems and a variable price element for flying hours. Given this, the contract is fairly flexible and payment is based on certain flying hour availability bands—with the bands ranging from 70 to 110 percent availability in 10 percent increments that are agreed to annually. As another example, officials explained that the Merlin contract provides flexibility in that the prenegotiated price is linked to banded flying hours with fixed and variable elements. Under traditional contracting, they estimate that only 20 percent of the cost would vary with flying hours. Also, within the basic contract parameters there is a provision for surge delivery for the Merlin helicopter. Finally, according to officials, the Sea King helicopter support contract has a similar flexibility because there are a number of flying hour bands and each band has its own price. In this manner, the Ministry of Defence can increase or decrease flying hours without renegotiating the contract. Officials pointed out that one drawback is that the price charged per flying hour at the lower bands is higher because the contractor must be able to cover fixed costs with fewer flight hours to charge. However, they said that the cost per flying hour is still far less than it would have been under a more flexible traditional arrangement.

Conclusions

While the concept of using PBL support arrangements was intended to be a cost reduction strategy as well as one that would result in improved performance, DOD’s emphasis has been more focused on performance and less focused on cost. DOD no longer emphasizes reducing costs as a goal for PBL programs, and DOD’s implementation of PBL, in its current form, does not ensure that its PBL arrangements are cost effective. DOD’s emphasis on the implementation of PBL as the preferred weapon system support strategy has deemphasized the importance of the development of consistent, comprehensive, and sound business case analyses to influence decisions regarding the use of a PBL arrangement. Although DOD’s guidance recommends using business case analyses to guide decisions about using PBL arrangements for weapon system support, the DOD guidance does not require these analyses and almost half of the programs we reviewed either did not perform a business case analysis or did not retain documentation of their analysis. Further, the quality of the analyses of those programs that had performed a business case analysis varied considerably since many were missing elements of what DOD guidance recommends for sound economic analyses. Additionally, most of those analyses that should have been updated had not been. Thus, DOD lacks a sound approach for analyzing whether proposed PBL arrangements are the most cost-effective strategy for supporting weapon systems. Without instituting a more consistent, comprehensive, and sound process on which to base decisions regarding the type of arrangement to be used in supporting DOD systems, it is unlikely that the department will be successful in achieving the sizable savings that were envisioned when the PBL concept was adopted.

Assessing the cost-effectiveness of PBL programs also requires the availability of better cost data at a level of detail that would support the improved management of ongoing PBL programs, including awarding contract fees, assessing performance versus the cost to achieve it, evaluating historical costs to determine whether the status quo should be maintained over time, and making support decisions about future follow- on programs. Such data are usually not available for PBL programs, limiting the ability of program offices to make program adjustments or take restructuring actions when appropriate. Nonetheless, a few program offices have acquired data at this level and indicate that they obtained them in a cost-effective manner. Improved access to detailed cost data is another essential element in improving the quality of data available to DOD decision makers regarding the cost-effectiveness of PBL arrangements.

Recommendations for Executive Action

To ensure that PBL arrangements are the most cost-effective option for weapon system support, we recommend that the Secretary of Defense direct the Under Secretary of Defense (Acquisition, Technology and Logistics) to take the following five actions: revise DOD’s Acquisition Directive to require development of a business case analysis to support the decision-making process regarding weapon system support alternatives, including PBL; revise PBL business case analysis guidance to more clearly define what should be included in a business case analysis and to establish specific criteria and methods for evaluating PBL support arrangements, including evaluation at the subsystem and component levels; revise PBL business case analysis guidance to more clearly define when business case analyses should be updated during the weapon system life cycle; require that each service revise guidance to implement internal controls to ensure that program offices prepare and update business case analyses that are comprehensive and sound; and require program offices to collect and report cost data for PBL arrangements in a consistent, standardized format with sufficient detail to support traditional cost analysis and effective program management.

Agency Comments and Our Evaluation

In written comments to a draft of this report (see app. II), DOD generally concurred with our five recommendations, noting that the department is committed to evaluating PBL strategies using business case analyses as part of the overall supportability assessment made during the development stages of weapon system acquisition programs. Specifically, the department fully concurred with three recommendations and partially concurred with two.

DOD fully concurred with our first recommendation to revise DOD’s acquisition directive to require the development of a business case analysis to support the decision-making process regarding weapon system support alternatives, including PBL. DOD stated that the department will take steps to address this issue in the next iteration of the DOD Directive 5000.1 and DOD Instruction 5000.2 acquisition regulations. According to DOD’s response, this new policy will require that the use of a business case analysis be mandatory and that this analysis serve as a sound basis for the selected supportability strategy.

In response to our second recommendation to revise PBL business case analysis guidance to clearly define what should be included in a business case analysis and to establish specific criteria and methods for evaluating PBL support arrangements, DOD partially concurred, stating that it established a Life Cycle Product Support Assessment Team in September 2008 to study product support policy, guidance, past performance, and results. As part of the study, existing business case analysis policy is being reviewed, and the department will evaluate the team’s recommendations on providing specific criteria and methods for evaluating support arrangements and determine how best to incorporate these recommendations into mandatory policy. The team’s initial recommendations are expected in April 2009.

DOD fully concurred with our third recommendation to revise PBL business case analysis guidance to more clearly define when, during the weapon system life cycle, business case analyses should be updated. According to DOD’s response, the department’s Life Cycle Product Support Assessment Team will evaluate the appropriate timing of initial business case analyses and follow-on updates to validate the life cycle support approach for weapon systems, and the team’s recommendations will be evaluated for inclusion into mandatory policy.

DOD fully concurred with our fourth recommendation to require that each service revise guidance to implement internal controls to ensure that program offices prepare and update business case analyses that are comprehensive and sound. As we noted in our report, the Army has already implemented a PBL business case analysis review and approval process. DOD stated that the Army’s internal controls will be reviewed by the Life Cycle Product Support Assessment Team, which will make recommendations for expansion for DOD-wide governance policy as part of the team’s overall recommendations expected in April 2009.

DOD partially concurred with our fifth recommendation to require program offices to collect and report support cost data for PBL arrangements in a consistent, standardized format with sufficient detail to support traditional cost analysis and effective program management. DOD stated that a provision for tailored cost reporting for major acquisition programs designed to facilitate future cost estimating and price analysis has been included in the draft DOD Instruction 5000.2, which is expected to be approved in the next 30 days. Additionally, the Life Cycle Product Support Assessment Team is reviewing support cost reporting and cost analysis as a part of its ongoing study. According to DOD’s response, the ultimate goal is standardized support cost reporting for all life cycle product support efforts, to include support provided by government activities.

While concurring with our recommendations, DOD’s response noted that the department disagrees with the assertion that the goal of PBL arrangements is to reduce costs. Rather, the primary goal of PBL arrangements is to increase readiness and availability while reducing overall sustainment costs in the long run. Our report recognized that the current DOD Directive 5000.1 provides that PBL arrangements shall optimize total system availability. However, our report notes that this directive also provides that PBL arrangements shall minimize costs and the logistics footprint. Moreover, our report stated that PBL emerged from a 1999 DOD study to test logistics reengineering concepts that placed greater reliance on the private sector for providing weapon system support to both reduce support costs and improve weapon system performance. Thus, reducing costs was a central focus of the adoption of PBL as DOD’s preferred support strategy. Based on our analysis in this report, we continue to believe that the PBL support arrangement concept was intended to be a cost reduction strategy as well as a strategy that would result in improved performance.

DOD’s response also noted that 22 of the 29 programs we reviewed produced business case analyses that enabled sound support strategy determinations. DOD further stated that for 28 of the 29 programs, the PBL strategies produced performance benefits, readiness benefits, or both, and 15 of the programs reflect cost-neutral or savings benefits resulting from the application of the PBL strategies. However, based on our analysis in this report, we continue to believe that only 20, rather than 22, of the programs had business case analyses that evaluated PBL strategies. Further, as we stated in our report, 6 of these did not retain some or all of the documentation and 13 were missing elements of DOD’s criteria for economic analyses. For example, we found that for one analysis the less costly option would have changed if the department had calculated the net present value of the two options considered. Additionally, because the department did not document all the potential support options in the business case analyses, it is not possible to determine if the most cost- effective options were chosen. Thus we continue to question the extent to which these analyses enabled sound support strategy determination. Finally, while we recognize that the PBL arrangements may have produced performance benefits, readiness benefits, or both, deficiencies in updated business case analyses and detailed cost data did not support an assessment of support costs. Therefore, it is unclear how many of the programs may have actually had cost-neutral or savings benefits resulting from PBL strategies. We continue to believe that improvements in collection and reporting of support cost data and the updating of business case analyses are essential if DOD is to determine the cost-effectiveness of its PBL arrangements.

We are sending copies of this report to interested congressional committees and the Secretary of Defense. The report is also available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions regarding this report, please contact me at (202) 512-8365 or solisw@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Key contributors to the report are listed in appendix III.

Appendix I: Scope and Methodology

To evaluate (1) the extent to which the Department of Defense (DOD) used business case analyses to guide decisions regarding performance based logistics (PBL) arrangements and (2) the impact PBL arrangements have had on weapon system support costs, we selected a nonprobability sample of 29 PBL arrangements for weapon system support initiated from 1996 through 2007. The 29 PBL arrangements were selected from lists of weapon systems supported by PBL arrangements provided by service officials. With the exception of the Navy’s, we found that the lists provided by the services either were not current or contained inaccuracies, and the content of the lists changed significantly during the course of our review, which affected our sample selection. We chose system-, subsystem-, and component-level PBL arrangements from each of the services based on length of time since implementation, location of program office, dollar value, and prior audit findings. The 29 PBL arrangements we selected constitute a nonprobability sample and the results are not generalizable to the population of PBL arrangements.

To evaluate the extent to which DOD used business case analyses to guide decisions regarding PBL arrangements, we interviewed officials regarding DOD and service requirements, policies, and guidance for business case analyses since 2001 and reviewed applicable documents. We also reviewed DOD’s 1995 economic analysis instruction, which states that analytical studies that evaluate the cost and effectiveness of weapon system support are considered to be “economic analyses,” and determined that the guidance is consistent with Office of Management and Budget guidance for benefit-cost analyses of federal programs. We interviewed program officials to discuss any business case analyses prepared to evaluate the 29 PBL arrangements before or after PBL implementation and examined the analyses using the criteria contained in DOD’s economic analysis guidance.

To evaluate the impact that PBL arrangements have had on weapon system support costs, we interviewed program officials to discuss the characteristics of the PBL arrangements, including contract length, contract type, scope of work, performance measures, performance incentives or disincentives, and cost data availability. In addition, we asked program officials to identify support cost reductions that occurred as a result of PBL implementation. If a program had renewed a fixed-price PBL arrangement or had finalized contract options that were not priced, we analyzed the contracts for trends in PBL support costs. We also compared PBL contract costs to estimated PBL support costs in business case analyses, where available, to determine how closely the estimates matched the actual PBL arrangement costs. We also relied on previously issued GAO reports on DOD’s implementation of PBL.

To analyze the use of availability contracts for weapon system support by the United Kingdom’s Ministry of Defence, we interviewed officials from the Defence Equipment and Support organization regarding policies or requirements for availability contracts and trends regarding the use of these arrangements. We also interviewed officials from programs identified by the Ministry of Defence as using availability contracts for weapon system support to identify the characteristics of the specific arrangements and the impact that the use of these contracts had on support costs. In addition, we interviewed National Audit Office officials who reviewed the cost and performance of two availability contracts for support of fast jets. Finally, we reviewed audit reports and other documents from the Ministry of Defence and National Audit Office. We obtained these data for informational purposes only and did not independently verify the statements or data provided by Ministry of Defence and National Audit Office officials.

Specifically, in performing our work we interviewed officials and obtained documents related to PBL at the Office of the Under Secretary of Defense (Acquisition, Technology and Logistics), the Office of the Assistant Secretary of the Navy (Research, Development and Acquisition), the Office of the Assistant Secretary of the Air Force (Acquisition), the Office of the Assistant Secretary of the Air Force (Installations, Environment and Logistics), the Office of the Assistant Secretary of the Army (Acquisition, Logistics and Technology), the Marine Corps Headquarters, the U.S. Army Materiel Command, the U.S. Army Aviation and Missile Command, the U.S. Army Communications and Electronics Command, the Naval Sea Systems Command, the Naval Air Systems Command, the Naval Center for Cost Analysis, the Air Force Cost Analysis Agency, the Air Force Directorate of Economics and Business Management, the Air Force Materiel Command, the Air Force Aeronautical Systems Center, the Oklahoma City Air Logistics Center, the Warner Robins Air Logistics Center, the Ogden Air Logistics Center, the United Kingdom Ministry of Defence, and the United Kingdom National Audit Office. We conducted this performance audit from February 2007 through December 2008 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

GAO Contact

Acknowledgments

In addition to the contact named above, Julia Denman, Assistant Director; Harold Brumm; Matt Dove; Jennifer Echard; Chaneé Gaskin; Tom Gosling; Jennifer Jebo; Mae Jones; Kevin Keith; Charles Perdue; Janine Prybyla; and Karen Thornton made major contributions to this report.