Background

PBL is a method of providing support for weapon systems by designating  what system performance is required, such as a given level of system  availability, and placing the responsibility for how it is accomplished on  the support provider, which manages resources to achieve performance  objectives. Logistics support for almost all of DOD’s weapon systems, such  as materiel management, maintenance, and engineering, is provided by a  combination of government and private-sector sources. In the past, under  traditional support arrangements, the government generally managed the  provision of weapon system support, using a combination of support  providers from the government and the private sector. PBL support  arrangements often use a private-sector support integrator to manage  support providers from both the public and private sectors to meet  specified performance requirements.
PBL evolved from performance-based service contracting, which has been  used in both the public and private sectors. The Federal Acquisition  Regulation defines performance-based contracting as structuring all  aspects of an acquisition around the purpose of the work to be performed.  The Federal Acquisition Regulation further defines the statement of work  for a performance-based acquisition as describing the required results in  clear, specific, and objective terms with measurable outcomes.  Performance-based service contracting has been referenced in regulation,  guidance, and policy for more than two decades, and federal agencies have  used it to varying degrees for acquiring a range of services. In 1991 the  Office of Management and Budget issued a policy letter establishing the  use of a performance-based approach for service contracting, and in 1994  it initiated a governmentwide pilot project to encourage the use of  performance-based service contracts in federal agencies, including DOD.  In October 1997, the Federal Acquisition Regulation was amended to  incorporate the Office of Management and Budget’s 1991 policy. The  Federal Acquisition Regulation currently establishes a policy that  agencies use performance-based contracting methods to the maximum  extent practicable for the acquisition of services, with certain exceptions.
Using performance-based service contracts is intended to offer a number  of potential benefits, such as encouraging contractors to be innovative and  to find cost-effective ways of delivering services for a fixed level of  funding. By shifting the focus from process to results, these contracts can  potentially produce better outcomes and reduced costs. The Office of  Management and Budget reported that the agencies participating in the  pilot reduced contract prices and improved customer satisfaction with  contractor work after introducing performance-based contracting.

		PBL Implemented as a Strategy to Reduce or Minimize Costs and Improve Performance

As an approach for supporting military weapon systems, PBL emerged  from a 1999 DOD study to test logistics reengineering concepts that placed  greater reliance on the private sector for providing weapon system  support to reduce support costs and improve weapon system  performance. The goal was for the military departments to reduce costs  and improve efficiency by pursuing logistics support “reengineering”  efforts using contractors. The fiscal years 2001-2005 Defense Planning  Guidance advanced this cost reduction effort by establishing a goal that  directed each military department to reduce the operation and support  costs of its fielded systems by 20 percent by the year 2005. During this  time, the Under Secretary of Defense (Acquisition and Technology)  directed the services to use an existing pilot program containing 30  weapon systems to demonstrate the type of cost savings depicted in the  fiscal years 2001-2005 Defense Planning Guidance. The areas identified for  potential cost savings were reducing demand on the supply chain by  improving the reliability and maintainability of the equipment, reducing  supply chain response time, and increasing competitive sourcing of  product support. Some of the 30 pilot programs involved performance- type arrangements that the services subsequently converted to, or  designated as, PBL arrangements.
This emphasis on reducing costs through PBL implementation was  reiterated in DOD’s 2001 Quadrennial Defense Review Report, which  advocated the implementation of PBL to compress the supply chain by  removing steps in the warehousing, distribution, and order fulfillment  processes; to reduce inventories; and to decrease overhead costs while  improving the readiness of major weapon systems and commodities. In  November 2001, DOD identified PBL as the preferred weapons system  support strategy. In May 2003, DOD further strengthened this emphasis  on PBL by stating in a DOD policy directive that acquisition managers shall  use performance-based strategies for sustaining products and services  whenever feasible and PBL strategies shall optimize total system  availability while minimizing cost and the logistics footprint.
In concept, a properly structured PBL arrangement is supposed to provide  a level of performance and also reduce costs over time. According to the  DOD/Defense Acquisition University PBL guide, a key aspect of PBL is  the inclusion of an incentive for the support provider to reduce costs  through increased reliability. Further, PBL arrangements can inherently  motivate support providers to improve component and system reliability,  since such improvements can provide the foundation for increased profit  over the long term. In other words, the support provider should have the  incentive to make reliability improvements to ensure that performance  metrics are met and also to increase profit by earning a performance  incentive tied to the metrics (an award fee or award term) and by reducing  costs while still being paid the agreed-upon, fixed price for the remainder  of the contract. The DOD/Defense Acquisition University PBL guide also  states that a critical element of the PBL arrangement that facilitates this  incentive and motivation is contract length. Further, long-term contracts  provide the support provider with confidence in continuing cash flows and  provide sufficient time for receiving an adequate return on any  investments made to improve reliability.

		DOD Recommended That Analyses Be Conducted to Guide Decisions regarding Weapon System Support Options

In 1995, before DOD identified PBL as the preferred weapons system  support strategy, DOD’s economic analysis instruction recommended  using an economic analysis for evaluating options for weapon system  support. The instruction stressed the importance of considering in the  analysis both qualitative and quantitative factors. With respect to  quantitative factors, the instruction recommended that costs and benefits  be expressed in terms of net present value to account for the time value of  money. These were also to be expressed as life cycle costs and benefits  that were to be calculated and compared for each feasible alternative for  meeting a given weapon system support objective.
Specifically, the economic analysis instruction identified and  characterized the following seven elements that should be present in an  economic analysis:  objectives—to clearly identify the function to be accomplished and not  to assume a specific means of achieving a desired result;    assumptions—to incorporate both actual data and future    alternatives—to comprise a comprehensive list of the feasible and  infeasible options followed by a discussion of the infeasible options  and comparisons of only the feasible options;  costs and benefits—to compare the quantitative (expressed in terms of  net present value) and qualitative factors for each option;  sensitivity and uncertainty (risk) analyses—to determine the effect  of uncertainties on the results of the analysis and to provide a range of  costs and benefits;  summary of the results of the analysis; and  summary of the analysis’s recommendations.
In DOD’s 2001 PBL guide, in which DOD identified PBL as the preferred  weapon system support strategy, the department recommended that for all  new systems and fielded acquisition category I and II systems, program  offices use an analytical tool called a business case analysis to support the  decision to use PBL arrangements for weapon system support.
In 2004 and 2005, DOD guidance on conducting business case analyses  described this tool in less specific terms than those used to describe the  criteria laid out in DOD’s economic analysis instruction. However, there  are some common themes in the guidance and instruction. For example, a  January 2004 Under Secretary of Defense (Acquisition, Technology and  Logistics) memorandum on PBL business case analysis calls for an  assessment of “best value,” or the greatest overall benefit. The  DOD/Defense Acquisition University PBL guide repeats the  characterization of the business case analysis as a tool with the goal of  determining a best-value solution and suggests that at a minimum a  business case analysis should include an introduction outlining the  purpose and objectives of the program, an explanation of the methods and  assumptions used in the analysis, calculations of the relative costs and  benefits of each weapon system support option, the financial and  nonfinancial impacts of each, a risk assessment, and a section with  conclusions and recommendations. Finally, both DOD’s economic analysis  instruction and the DOD/Defense Acquisition University PBL guide  recommend documenting the results of the analysis, including all  calculations and sources of data, down to the most basic inputs, to provide  an auditable and stand-alone document. According to the guidance, a  business case analysis must stand on its own and be able to withstand  rigorous analysis and review by independent agencies.
DOD’s 2004 and 2005 guidance on conducting business case analyses also  recommended that program offices    update their business case analyses at key decision points both to  validate the approach taken and to support future plans and    use certain criteria, such as the capability of a PBL arrangement to  reduce the cost per operational unit of performance (i.e., cost per  flight hour), to assess all acquisition I and II programs without plans  for a PBL arrangement for the potential application of a PBL strategy  at the system, subsystem, or major assembly level. If the assessment  showed potential for a PBL arrangement, a business case analysis  should be conducted and completed by the September 30, 2006,  deadline required by DOD’s Strategic Planning Guidance.

		PBL Arrangements Included in Our Review

In our review, we looked at PBL arrangements initiated as early as 1996  (when performance-based contracting was encouraged governmentwide)  and as recently as 2007 (by which time, at the DOD level, PBL  arrangements had moved from being encouraged to being required  whenever feasible). These PBL arrangements represent contract values  totaling approximately $12.1 billion and range from a low of $10.5 million  to a high of $4.9 billion. Table 1 lists, by service, the weapon system  programs supported by the 29 PBL arrangements we reviewed.

	Business Case Analyses Have Not Been Used Consistently or Effectively to Influence Decision Making regarding Use of PBL

DOD has generally not used business case analyses consistently or  effectively to influence decision making regarding the use of PBL.  Although DOD guidance recommended that these analyses be used to  guide decisions on the cost-effectiveness of weapon system support  arrangements, about half of the programs we reviewed either did not  conduct such an analysis or did not retain adequate supporting  documentation. Further, most of the remaining programs in our sample  used analyses that were not comprehensive. For example, some analyses  did not evaluate alternative support options and most did not contain all of  the elements recommended in DOD’s economic analysis instruction.  Additionally, analyses were often not updated to support decision making  after PBL implementation in accordance with service policies and  guidance. The key reasons for DOD’s ineffective use of business case  analyses to support PBL decision making are that (1) DOD has not  required such analyses or provided specific criteria for conducting them  and (2) the services’ internal controls have been inadequate to ensure that  the analyses are performed and updated. As a result, DOD cannot ensure  that decisions regarding weapon system support options are guided by  comprehensive, consistent, and sound analytical tools. Further, the  department cannot be sure that the support arrangements being  implemented will result in the most cost-effective support program.

		About Half of Programs Reviewed Either Did Not Use Business Case Analyses to Guide Decisions to Enter into PBL Arrangements or Did Not Retain Details of the Analysis

For 9 of the 29 PBL arrangements we reviewed, a business case analysis  had not been completed. Additionally, for 6 others, program officials could  not provide supporting details of the business case analysis they told us  that they had conducted. When program offices did not conduct business  case analyses as part of their PBL decision making, as we found for many  of the programs we reviewed, the department cannot be sure that the  support arrangements implemented will result in the most cost-effective  support program.

			Programs That Did Not Conduct a Business Case Analysis Before Entering into PBL Arrangements

Table 2 provides the number of PBL arrangements we reviewed by service  that were initiated with and without the use of a business case analysis.
Although both of the Marine Corps programs we reviewed conducted a  business case analysis, about 50 percent of Air Force programs, 22 percent  of Army programs, and 25 percent of Navy programs did not. In general,  the Air Force programs only developed a source-of-repair analysis, which  evaluated only the repair element of weapon system support. For the two  PBL arrangements for which the Army did not conduct an analysis—the  Apache sensor and airframe arrangements—the Deputy Assistant  Secretary of the Army (Integrated Logistics Support) approved the Apache  program office’s request for a waiver from Army business case analysis  policy based on prior analyses and program decisions. However, the U.S.  Army Audit Agency reported that the prior analyses did not consider all  components included in the two PBL arrangements, other support  strategies, performance metrics, and relative costs. According to an Army  official, a business case analysis for the airframe components is being  conducted and is expected to be completed in December 2008, and efforts  to develop a business case analysis for the program’s sensors are expected  to begin in November 2008.
The F-22A Raptor and KC-130J are examples of programs where the Air  Force and Navy did not conduct a business case analysis as part of the  PBL decision-making process. When DOD recommended in 2001 that  program offices fielding new systems base PBL arrangement decisions on  business case analyses, the F-22 was beginning low-rate initial production.  In 2002, the Assistant Secretary of the Air Force (Acquisition) and the Air  Force Deputy Chief of Staff (Logistics, Installations and Mission Support)  directed the F-22 program office to develop a long-term support strategy  and manage life cycle product support through a PBL arrangement that  includes government-contractor partnerships as necessary. From 2003  through 2007, the program office acquired support as part of the aircraft’s  production contract, and in 2008 the office signed a separate PBL support  contract with Lockheed Martin, one of the original equipment  manufacturers, to support the aircraft from 2008 to 2012. In March 2008,  program officials said that they did not conduct a business case analysis  before awarding the 2008 contract because current program data, such as  material usage rates and costs, are immature. Officials planned to conduct  an analysis in 2010 or 2011 when it could be completed using more  meaningful data. However, program officials subsequently decided that the  available data were sufficient and in July 2008 awarded a contract to  develop a business case analysis. Completion of the analysis is expected in  late 2009.
In 2002, the Navy contracted for a PBL arrangement to support the Marine  Corps’ KC-130J engines without first preparing a business case analysis.  Program officials explained that the decision was made not to develop an  analysis because the technical data needed to repair the engines were not  available and the Marine Corps did not have the infrastructure necessary  to support the system. Officials also said that a market analysis was  conducted prior to implementing the PBL arrangement, but they could not  provide a copy of the analysis. Nonetheless, a business case analysis that  documented the results of the market analysis and the program’s  negotiated costs versus expected flight hours and anticipated repairs and  maintenance could have been used to monitor the actual results and cost- effectiveness of the performance-based approach, especially since support  for the engines is obtained under a commercial contract and the  contractor does not provide detailed cost data to the government.

	Supporting Documentation Not Fully Available for Some Programs That Conducted Business Case Analyses

Program officials from 20 of the PBL arrangements we reviewed told us  that they had conducted business case analyses before implementing the  arrangements; however, officials for 6 programs could not provide all or  some of the data to support the analyses. According to DOD’s economic  analysis instruction, the results of the analysis, including all calculations  and sources of data, must be documented down to the most basic inputs to  provide an auditable and stand-alone document. Table 3 lists by service  the number of PBL arrangements for which all of the business case  analysis documentation was retained and those for which it was not.
In general, program officials for six programs said that they were unable to  locate all the details of their analyses; however, the amount of data that  was missing varied. For example:    Although officials for the Army’s Common Ground Station said that an  analysis was performed in 2002, they were unable to provide any  details of the analysis or the results.
While program officials for the Army’s Shadow Tactical Unmanned  Aircraft System were able to provide the results of their 2002 analysis,  they did not retain the details regarding the assumptions, data sources,  or calculations used to develop the analysis. However, program  officials said that the analysis was developed early in the life cycle of  the program and was not based on historical cost and maintenance  data, and therefore they did not consider it to be very accurate based  on actual program results that have occurred since the analysis was  developed.
For the Army’s Javelin PBL arrangement, the DOD Office of the  Inspector General reported in 2005 that it was unable to validate the  program office’s 2001 analysis because the program office was not able  to provide adequate documentation. The documentation has not been  located, and program officials were only able to provide a summary of  the results of the analysis.

	Most Remaining Programs Used Business Case Analyses That Were Not Comprehensive

Service program officials could provide documentation of the business  case analyses conducted for 14 of the PBL arrangements we reviewed, but  all but 1 of the 14 analyses were missing one or more of the elements  recommended in DOD’s economic analysis instruction. As a result,  decisions regarding weapon system support options for many of the  programs we reviewed were not guided by comprehensive, consistent, and  sound economic analysis. Further, the department cannot be sure that the  support arrangements implemented will result in the most cost-effective  support programs. Figure 1 shows which elements were missing from the  14 business case analyses.
For three PBL arrangements, the business case analyses did not compare  alternative support options and either evaluated only a single option for  weapon system support or evaluated contracting strategies instead of  alternative support arrangements. For example, the 2007 business case  analysis for the Joint Surveillance and Target Attack Radar System did not  analyze the costs and benefits of alternative support strategies for the  program. The business case analysis was developed in response to a 2006  recommendation from the DOD Office of the Inspector General after a  review found that the program office had not evaluated alternative support  strategies prior to implementing a PBL arrangement in 2000. However,  the 2007 analysis covered only the potential impacts of changing the type  of contract used to obtain support for the program from cost plus award  fee to firm fixed price. The business case analysis for the B-2 also did not  analyze alternative support strategies but focused on potential efficiencies  available to the current program through funding consolidation, funding  stability, and long-term contracting. According to program officials, the  only assumption in the analysis that actually occurred to some extent after  PBL implementation was funding consolidation. Finally, although the C-17  program office developed a business case analysis in 2003, the DOD Office  of the Inspector General reported in 2006 that the analysis focused only on  one support option and did not evaluate the costs and benefits of multiple  support options. In 2007, a contract was awarded for development of  another business case analysis planned for completion prior to awarding  the next C-17 support contract.
Two other important elements missing from some analyses were an  evaluation of costs over the remaining life cycle of the program and the  calculation of net present value to account for the time value of money.  For example, the analysis for the Patriot PBL arrangement only evaluated  the costs and benefits over a 3-year period. On the other hand, while the  business case analysis for the Assault Breacher Vehicle evaluated costs  over the 20-year life cycle of the program, the net present value of the  alternatives was not calculated. Four other business case analyses—those  prepared for the F/A-18 E/F, AV-8B Harrier, Close-In Weapon System, and  Consolidated Automated Support System—did not include these two  elements and several others, such as sensitivity or risk analysis. These  analyses were prepared in a similar format by the Naval Inventory Control  Point, an organization that provides supply support to the Navy, Marine  Corps, and others. We conducted a net present value analysis on the  amounts contained in the Naval Inventory Control Point’s business case  analysis for the F/A-18 E/F PBL arrangement and found that the PBL  option it chose was about $1.7 million more expensive than the alternative  option. Its analysis, which did not use net present value, found that the  PBL option was about $277,000 less expensive. The Naval Inventory  Control Point’s philosophy is that if the costs of PBL are equal to or less  than the costs for government-provided support, a PBL arrangement will  be used. Therefore, if Naval Inventory Control Point officials had  conducted a net present value analysis, based on this decision criterion,  they would not have implemented the PBL arrangement. According to  Naval Inventory Control Point officials, there is confusion in the Navy  regarding the purpose of the analyses they prepare. Officials said that the  analyses were conducted for internal decision making and were not  intended to satisfy Navy PBL policy, which places responsibility for  development of a life cycle business case analysis on the weapon system  program office. However, program officials for the Close-In Weapon  System, Harrier, and Consolidated Automated Support System did not  develop business case analyses that evaluated PBL implementation over  the remaining life cycle of the system.

	Other Factors Influencing the Quality of Business Case Analyses

Several other factors affected the quality of the business case analyses we  reviewed. Most of the analyses we reviewed did not identify and quantify  the benefits that could be expected from contractor incentives to increase  reliability or improve processes to reduce support costs. The only business  case analysis that specifically identified cost savings based on contractor  incentives to reduce costs was the 2005 life cycle business case analysis  for the F/A-18 E/F. The life cycle analysis included an estimate of future  cost savings associated with investments the contractor was willing to  make under the arrangement to reduce supply chain costs. In addition,  most of the analyses did not recognize or quantify the costs associated  with the transfer of risk that occurs under a performance-based support  arrangement. According to the DOD/Defense Acquisition University PBL  guide, PBL arrangements transfer responsibility for making support  decisions—and corresponding risk—to the support provider, and risk is  one of the major cost drivers for contractors. Therefore, the use of  performance metrics could introduce a large element of risk for the  contractor that may be built into the costs of such an arrangement. In  general, many of the business case analyses we reviewed simply estimated  the costs for contract logistics support and government-provided support.  One exception was the business case analysis for the Marine Corps’  Assault Breacher Vehicle, which attempted to address the costs of risk  transfer and the benefits of contractor incentive to reduce costs by  estimating the costs for a traditional contractor logistics support  arrangement and a performance-based contractor logistics support  arrangement, in addition to estimates for government-provided support  and performance-based, government-managed support.
Another business case analysis was based on questionable assumptions.  The 2002 business case analysis for the Sentinel program estimated that  the cost for the government depot to overhaul the system was 50 percent  of the total cost of the system. The business case analysis estimated that  for the alternative option—a partnership between the government and the  contractor with the government depot performing enough work to meet  the system’s core requirements—the cost for an overhaul was only 25  percent of the system’s cost. The analysis assumed that under the  partnership option the overhaul cost less because, instead of complete  disassembly and parts replacement, the system would be inspected and  repaired only as necessary. However, according to an official at the Army  depot that would perform the overhaul, the depot also could have used the  inspect and repair concept as the basis for its maintenance work.  Therefore, this assumption in the business case analysis may have  overstated the costs for the government depot to perform the work.
Lastly, the Naval Air Systems Command’s 2005 life cycle business case  analysis for the F/A-18 E/F estimated that over the 28-year life cycle of the  program, PBL support costs were $76 million more expensive than costs  for government-provided support. However, the business case analysis  estimated that Naval Inventory Control Point surcharges would be   $325 million less for the PBL arrangement. The Naval Inventory Control  Point adds a surcharge to the cost of goods sold to its customers  (including weapon system program offices) to recoup its expenses, but  according to officials, they do not adjust their resources based on PBL  implementation and would still need to recoup their expenses through  surcharges to other customers. Therefore, while the F/A-18 program office  may realize a $325 million benefit from the reduced surcharge, the overall  costs to the Navy may remain the same. Including this reduced amount in  the analysis is inconsistent with DOD’s economic analysis instruction,  which states that all measurable costs and benefits to the federal  government should be included in the analysis. In addition, in the 2006  business case analysis prepared by the Naval Inventory Control Point,  which estimated supply chain management costs for both PBL and  government-provided support for a 5-year period from 2006 through 2011,  the Naval Inventory Control Point’s costs were estimated to remain the  same under either option. If the Naval Inventory Control Point’s costs had  been the same in the life cycle business case analysis prepared by the  Naval Air Systems Command for the F/A-18 E/F, the PBL arrangement  would be $401 million more expensive than government support for the  28-year period.

		Business Case Analyses Were Often Not Updated for Decision Making After PBL Implementation

In 2004, DOD guidance recommended that business case analyses  continue throughout the weapons system’s life cycle and be updated at  key decision points both to validate the approach taken and to support  future plans. The services, with the exception of the Air Force, have also  issued policies on conducting such updates. However, even when business  cases were prepared, we found that program offices often did not update  them in accordance with DOD’s guidance and service policies, nor did the  program offices validate them against actual support costs for decision  making after PBL implementation. Neither DOD nor the services has  issued guidance that specifies what should occur when updating or  validating a business case analysis.
Army policy states that PBL business case analyses shall be updated prior  to the exercise of an option period when there are significant changes to  the performance period/terms of the contract or evaluation period, or  updated whenever there are major programmatic changes or at least every  5 years. Program offices for four of the Army PBL arrangements we  reviewed had not updated their business case analyses as called for by  Army policy. For example, the Tube-launched Optically-tracked Wire- guided missile – Improved Target Acquisition System program office  developed a business case analysis in 1998 before awarding the original  PBL contract and has not updated or validated the original analysis.  Further, officials negotiated a follow-on PBL contract in 2007 after the  terms of the original contract were complete. Although program officials  for the Shadow Tactical Unmanned Aircraft System had planned to  complete an update to their 2002 business case analysis by the end of 2007,  the effort was delayed and the update is expected to be completed before  the end of 2008. Although the Javelin program office implemented a PBL  arrangement in January 2004, the business case analysis was developed in  2001, and program officials do not have plans to update the analysis.  Additionally, although program office officials for the Army’s Sentinel  weapon system had updated their 2002 business case analysis, Army  policy calls for submission of the business case analysis to both the Office  of the Deputy Assistant Secretary of the Army for Integrated Logistics  Support and Army Materiel Command Headquarters for review and  concurrence and then to the Program Executive Office for approval. In  February 2008, before the new analysis had been reviewed and formally  approved, another PBL contract was awarded.
The Navy’s 2007 business case analysis guidance calls for updates every 3  to 5 years or when significant programmatic changes occur. Based on this  policy, the T-45 program office should complete an update to its business  case analysis for the PBL arrangement for support of the aircraft’s engines  by 2008. Although program officials updated their 2003 analysis in 2006  with actual support costs and flying hours, they did not expand the  analysis to account for the remaining life cycle of the engines. The analysis  projected costs only through 2008, the original contract period. Program  officials did not plan to further update the business case analysis or  prepare another one because they believed that it was not required.
Neither DOD’s nor the services’ policies clearly specify what should occur  when a program office updates or validates a business case analysis.  Although some programs are conducting another business case analysis,  as mentioned earlier, program officials for the T-45 did not plan to conduct  another analysis because they had updated their analysis with actual data.  Program officials for the V-22 engine updated their 1998 analysis in 2004.  The update focused on assessing if several of the ground rules,  assumptions, and factors used in the original study were still valid and  providing a preliminary recommendation on pursuing a follow-on PBL  contract, from a cost standpoint. However, the entire analysis was not  updated.

		Lack of Specific Criteria and Inadequate Internal Controls Led to Business Case Weaknesses

Business case analyses were inconsistently used for PBL decision making  because DOD did not require that the analyses be conducted and updated  or provide specific criteria to guide their development. Further, with the  exception of the Army, the services’ have not established effective internal  controls to ensure that the analyses are prepared in accordance with  service policies and guidance. As a result, DOD cannot ensure that  decisions regarding weapon system support options are consistently  guided by comprehensive and sound analytical tools.

			DOD Guidance Has Not Provided Specific Criteria or Requirements for Conducting Sound Business Case Analyses

DOD guidance has not provided specific criteria for conducting and  updating business case analyses for PBL decision making. Despite DOD’s  preexisting economic analysis instruction recommending the analysis of  both quantitative and qualitative factors, in 2001 DOD recommended the  development of a business case analysis prior to implementing a PBL  arrangement but provided little criteria for conducting such an analysis. In  2003, a Defense Business Board study recommended that DOD issue  standard guidance for the services to take a more consistent approach to  PBL decision making and also require the use of business case analyses.  In response, the January 2004 Under Secretary of Defense (Acquisition,  Technology and Logistics) memorandum was issued containing “guiding  principles” for business case analyses. The memorandum stated that  business case analyses “will evaluate all services or activities needed to  meet warfighter performance requirements using ‘best value’  assessments.” This memorandum also listed several quantitative and  qualitative factors for consideration when developing a business case  analysis; however, it did not indicate how these factors were to be  evaluated or their relative importance in the decision-making process. The  memorandum also recommended that business case analyses be updated  or repeated to validate the approach taken or to support future plans, but  did not provide specific guidance as to when such updates should occur.  According to the memorandum, a DOD PBL business case analysis  handbook was supposed to be forthcoming. Later that year, the  DOD/Defense Acquisition University PBL guide was published. It had two  pages dedicated to the business case analysis concept—providing  additional criteria and also incorporating the guiding business case  analysis principles. However, a handbook specifically for PBL business  case analyses was never issued.
In 2003, when DOD incorporated PBL implementation into DOD Directive  5000.1, which provides mandatory policies for all acquisition programs, a  requirement to conduct and update a business case analysis was not  included. Specifically, the directive only stated that acquisition managers  shall use performance-based strategies for sustaining products and  services whenever feasible and that such PBL strategies shall optimize  total system availability while minimizing cost and logistics footprint. Also,  despite the Defense Business Board’s recommendation later that same  year to require the use of business case analyses, DOD subsequently  neither required program managers to prepare the analyses prior to PBL  implementation nor required them to update the analyses after  implementation. In fact, although most of the services have issued some  guidance and requirements for business case analyses, the current Defense  Acquisition Guidebook no longer specifically refers to a business case  analysis, but rather recommends the development of a “support strategy  analysis” as part of the PBL implementation process. According to the  guidebook, the support strategy analysis can be a business case analysis,  economic analysis, decision-tree analysis, or other best-value-type  assessment.

			Service Policies and Guidance for Conducting Business Case Analyses Were Slow to Develop, Inconsistent, and Generally Not Enforced Because of Lack of Effective Internal Controls

Another reason for the inconsistent use of business case analyses is that  the services’ policies and guidance for conducting the analyses were slow  to develop and were generally not enforced because of a lack of effective  internal controls. Moreover, we found inconsistencies among the services’  policies and guidance. In response to DOD’s recommendation that  program offices conduct a business case analysis prior to implementing a  PBL arrangement for weapon system support, the services issued their  own policies and guidance. The time frames for these are summarized in  table 4.
Although DOD recommended the use of business case analyses in 2001,  the services’ business case analysis policies and guidance have evolved  over time. In some cases, guidance was not issued until years later. For  example, Marine Corps policy did not call for PBL business case analyses  until 2007. Further, although the Air Force included business case analyses  among mandatory procedures in 2004, these procedures were not specific.  The Air Force’s instruction states only that “the program manager is  responsible for construction of a business case analysis to determine the  best strategy for meeting the PBL goals.” Final Air Force guidance for  business case analyses, including PBL business case analyses, was not  issued until 2008. As another example, the Army’s early business case  analysis guidance was general in nature, and more specific policy  memorandums were issued in 2005 and 2006. In 2007, these policies were  included in an Army regulation.
Currently, service policies and guidance vary with respect to which  programs should implement PBL arrangements, which of those programs  shall conduct business case analyses, and how often program managers  should update their business case analyses. Until 2007 the services’  policies and guidance varied significantly. However, the issuance of Navy  guidance and Marine Corps policy in 2007 resulted in more consistency.  Table 5 summarizes the services’ business case analysis policies and  guidance.
With the exception of the Army, the services have not established the  internal controls, including a review and approval process, necessary to  ensure that business case analyses are conducted prior to PBL  implementation and updated after implementation. For example, the  Navy’s 2003 guidance assigns responsibility for reviewing individual  business cases analyses to the system commands’ cost departments.  However, the review only occurs when requested. Although a recently  issued Air Force instruction calls for a formal review of all business case  analyses, including those conducted for PBL arrangements, that meet  certain criteria, it is unclear how many PBL business case analyses will  meet any of the criteria and be subject to this review. The 2008 Air Force  instruction calls for a review of all business case analyses that will be   (1) forwarded outside of the Air Force; (2) forwarded to senior Air Force  officials, such as the Secretary of the Air Force; and (3) provided for  weapon systems that require Defense Acquisition Board or Air Force  Acquisition Board approval.
In contrast, Army policy states that program managers shall report  semiannually on the status of PBL implementation and that business case  analyses for acquisition category I and II programs should be submitted  for review and verification to multiple offices—including Army  headquarters, the Army Materiel Command, and the Office of the Deputy  Assistant Secretary of the Army (Cost and Economics)—and for approval  to the Army Acquisition Executive. In addition, business case analyses for  lower-level programs should be reviewed and approved but will not  verified by the Office of the Deputy Assistant Secretary of the Army (Cost  and Economics), and the approval authority is the program executive  officer or commander of the related life cycle management command.  While the Army’s policy first provided for these internal controls in 2005,  Army officials said that no programs have yet passed the review and  approval process completely.

	Impact of PBL Arrangements on Reducing Weapon System Support Costs Is Unclear

The extent to which PBL arrangements are reducing costs for weapon  system support is unclear and generally remains undocumented even after  several years of PBL implementation. A major difficulty in assessing the  cost impact of PBL arrangements is the lack of detailed and standardized  cost data maintained by the program offices. Various other factors, such as  the lack of systems that are supported by both PBL and non-PBL support  arrangements, the lack of sound program baseline information, and  changing operational and materiel conditions, also limited our ability to  assess the impact of PBL implementation on support costs. While the  overall cost impact was unclear because of a lack of data and these other  factors, the limited evidence on cost impact that was available showed  mixed results. We did find some evidence that a few PBL arrangements  have reduced costs. However, we also found that characteristics of DOD’s  PBL support arrangements, such as short-term contracts and unstable  program requirements and funding, may limit their potential to reduce  costs. Further, DOD has not sufficiently emphasized the potential to  reduce costs as a goal for PBL programs. As a result, DOD cannot be  assured that PBL arrangements will reduce support costs and provide  cost-effective support for DOD systems.

		Program Managers Often Lacked Detailed and Standardized Cost Data

In 2004, a memorandum from the Under Secretary of Defense (Acquisition,  Technology and Logistics) recognized the importance of cost data for  contract management and future cost estimating and price analysis and  stated that PBL contracts shall include cost reporting requirements.  However, for the PBL arrangements we reviewed, program offices often  did not have detailed cost data that would provide insights regarding what  the program office was spending for various aspects of the support  program—such as the cost of depot maintenance by subsystem and major  component or the cost of engineering support, supply support, and  transportation. When cost data were available, the level of detail and  format of cost data varied considerably. This condition significantly  affected our ability to determine the impact of the implementation of PBL  on the costs of supporting the systems in our sample, as many factors  influence support costs.
For PBL arrangements using fixed-price contracts or fixed-price contract  line items—DOD’s “ideal” type of PBL arrangement—we found that  program offices generally did not receive detailed cost data for the  program and only knew the overall amounts paid for support. Only two  program offices in our sample obtained contractor support cost data for  their fixed-price PBL arrangements, and the format and contents of the  reports were very different. For example, the F/A-18 E/F program office  obtained Boeing’s report on fiscal year 2006 costs, including  general/administrative costs and profit, in a detailed reporting structure  approved by the Office of the Secretary of Defense (OSD), Cost Analysis  Improvement Group. According to program officials, an annual cost  reporting requirement was included in the 2005 fixed-price PBL contract  to provide cost visibility into the program and was added at no additional  cost to the government. In contrast, the B-2 program office receives a  monthly funds utilization report that allocates the amount the Air Force  pays the contractor into seven high-level categories, such as planned depot  maintenance.
Although the PBL arrangements that used cost-reimbursable contracts  generally obtained more detailed cost data than those with fixed-price  contracts, the format and level of detail also varied. For example, under  the 1998 PBL contract, the F-117 program office did not receive cost data  in a format that was detailed enough to report in OSD’s standard support  cost structure. The program office subsequently required more detailed  cost data reporting from Lockheed Martin in the 2006 PBL contract. As  another example, the 2003 C-17 PBL contract has both fixed-price and  cost-reimbursable elements. According to program officials, Boeing did  not report support cost data at the level of detail necessary to report in  OSD’s support cost structure under the contract. According to an Air  Force Cost Analysis Agency official, a cost-reporting requirement was  included in the contract’s option years and more detailed cost reporting  will begin in fiscal year 2009.

		Limited Available Evidence on Cost Impact Indicates Mixed Results

Although cost data were generally lacking, the limited available evidence  on cost impact showed mixed results. Data we reviewed for the two  systems that were managed by both PBL and non-PBL arrangements  indicate that the PBL arrangements were more costly, but based on other  assessments of available data, there are some indications that PBL  arrangements can reduce costs. However, in seven out of eight programs  we reviewed where follow-on, fixed-price PBL contracts had been  negotiated, expected cost reductions either did not materialize or could  not be determined. Finally, we noted that officials reported performance  levels for some programs that were significantly higher than required  under the PBL arrangement, but it is unknown whether the required levels  could be achieved at a lower cost.

			Two Comparisons of PBL Arrangements to Traditional Government Support Show That PBL Arrangements Cost More

Of the 29 programs we reviewed, only the F100-PW-220 engine and the KC- 130J/C-130J airframes are maintained by both PBL arrangements and  traditional government support strategies. We found that the Air Force’s  traditional support arrangement for the F100-PW-220 engine costs slightly  less than the Navy’s PBL arrangement for the same engine. The Navy uses  the F100-PW-220 engines in its F-16A/B aircraft and sustains the engines  through a PBL contract with Pratt & Whitney. The Air Force uses the same  engines in its F-16 and F-15 aircraft and supports the engines at the  Oklahoma City Air Logistics Center. The Air Force maintains an engine  total ownership cost estimate that includes all costs incurred (depot-level  repairables, general services division (expendable repair parts), depot  programmed equipment maintenance, organizational-level maintenance,  intermediate-level maintenance, and continuous improvement program).  To compare the Navy’s PBL costs with the Air Force’s engine total  ownership costs, we removed the costs associated with organizational- level maintenance from the Air Force’s costs. As shown in figure 2,  converted to costs per flight hour, the Navy’s PBL costs were slightly  higher than the Air Force’s costs in constant fiscal year 2008 dollars.
Although the cost difference appears to be decreasing, the Navy’s 5-year  contract ended in 2008 and a new PBL contract has not yet been  negotiated. The engines are currently being supported under a 6-month  extension to the original contract, and the fixed price paid per engine  cycle is significantly higher than that paid during the previous 5 years.  According to Navy officials, the decision to contract with Pratt & Whitney  for the support of the Navy’s engines was not solely based on costs but  was also based on other factors, such as turnaround time for engine  repairs. However, program officials could not provide the data on which  they based their decision.
Elements of the Air Force’s PBL arrangement to support the C-130J  airframe are more expensive than the support for the KC-130J airframe  provided by the Navy. According to Navy officials, an analysis was  prepared in 2005 to compare costs for alternative repair arrangements to  determine whether to continue using the Navy’s repair capability or to  transition to contractor-provided repair in 2006. The Navy’s analysis  concluded that the support provided by the Naval Surface Warfare Center,  Crane Division, would cost 43 percent less than the support provided by  the contractor. The analysis was based on anticipated 2006 flight hours,  actual government support costs from 2005, and the costs to exercise an  option for repair under a preexisting contract. Additionally, we  independently compared overall costs for inventory management and  repair of repairable components and found that the Air Force’s PBL costs  on a per flight hour basis for these elements were significantly higher than  the Navy’s costs—approximately 131 percent higher in 2006 and 164  percent higher in 2007. However, according to officials, several factors  account for some of the difference. For example, the Air Force’s PBL  arrangement includes 36 percent more consumable and repair parts than  the Navy’s support arrangement, maintenance of support equipment, and  support for six locations, while the Navy’s arrangement includes support  for only three locations.

			A Few Indicators Showed That PBL Arrangements Reduced Support Costs for Some Programs

Only a few of programs we reviewed were able to provide some indicators  of reduced weapon system support costs that could be attributed to the  use of a PBL arrangement. As mentioned earlier, some programs did not  have a business case analysis demonstrating how current support costs  compared to other support approaches. Of the nine PBL arrangements that  had been implemented and had a business case analysis that looked at  alternative support options, only four could be compared with PBL  contract costs. Based on this comparison, three of these four PBL  arrangements indicate potential savings from PBL implementation, while  the fourth is more expensive than estimated in the business case analysis.  The remaining analyses could not be compared to actual program costs  after PBL implementation because of programmatic changes that occurred  after the analyses were conducted.
The 2005 business case analysis for the Army’s Patriot estimated a 3-year  cost savings of $1.6 million from using a PBL arrangement to provide 107  high-demand parts. According to a program official, the contract is in its  final year and total obligations are expected to be about $1 million less  than estimated in the business case analysis. Additionally, two business  case analyses prepared by the Naval Inventory Control Point estimated  that supply chain management support costs were reduced by  implementing a PBL arrangement. The business case analyses projected  cost savings of about $2.2 million for the 5-year Close-In Weapon System  PBL arrangement awarded in 2006 and $1.3 million for the 5-year Harrier  PBL arrangement awarded in 2007. Based on actual contract costs—and  if the contracts are not modified in the future—the total savings for these  programs are projected to be $5.2 million and $5.8 million, respectively.  Although the F/A-18 E/F business case analysis estimated a 5-year supply  chain management savings of approximately $1.4 million, the actual  contract cost is about $1.6 million more than the estimated amount in the  analysis. Given the difference, the PBL arrangement has not reduced  support costs for the program. As previously noted, two of the PBL  arrangements having evidence of reduced support costs are managed by  the Naval Inventory Control Point. This activity has used PBL  arrangements since fiscal year 2000 and has reported achieving cost  reductions as a result, using the Navy working capital fund to issue longer- term, multiyear contracts that can extend up to 5 years in length to  support aircraft or ship subsystems or components. According to agency  officials, these longer-term agreements have enabled the Naval Inventory  Control Point to guarantee the contractors a more stable business base,  which provides contractors an incentive to make investments to reduce  costs. Overall, as a result of using PBL arrangements, Naval Inventory  Control Point officials estimate that they have reduced support costs by  approximately $26.7 million and $63.8 million—or 2.8 and 5.8 percent—in  fiscal years 2006 and 2007, respectively.
Although the V-22 program conducted a business case analysis in 1998 to  estimate alternative costs for supporting the engines and projected savings  of $249.5 million over the 53-year life cycle of the program, the analysis  did not take into account the time value of money and calculate savings  based on net present value. For this and other reasons, we cannot validate  that the savings are reasonable. In addition to DOD’s economic analysis  instruction, guidance from the Office of Management and Budget also  states that net present value is the standard criterion for deciding whether  a government program can be justified on economic principles. In 2004,  another analysis was prepared for the V-22 engine program to determine  (1) if several assumptions used in the 1998 analysis were still valid and   (2) the impact of any changes to those assumptions on the cost savings  estimate for the PBL arrangement. The later analysis concluded that  differences in three of the original assumptions increased the projected  PBL cost savings to $305.9 million—an increase of $56.4 million. The  updated savings again were not calculated using net present value.  Moreover, although limited actual data were available, the calculations  generally made adjustments using assumptions that generated the  maximum potential savings for the PBL alternative. For example, when  adjusting the 1998 analysis to account for differences in the costs  experienced for excluded repairs (repairs that were not covered by the  PBL arrangement), the total potential PBL cost savings were increased by  $48 million because the average excluded repair cost was lower than  previously estimated. However, even though data showed that excluded  repairs occurred at a higher frequency than projected in the original  analysis, the later analysis did not adjust for the higher frequency of  excluded repairs. Thus, the savings calculation is questionable, because  the analysis noted that the frequency of these repairs could eliminate all of  the estimated cost savings. Finally, the 10-year-old analysis has not been  completely updated to estimate costs based on actual data.
The remainder of the analyses could not be compared to current PBL  arrangement costs because of programmatic changes that occurred after  the analyses were conducted. For example:    According to an Air Force C-130J program official involved in the  development of the 2004 business case analysis, the analysis was  conducted while the aircraft was supported by a commercial contract;  therefore, the program office did not have detailed cost data on which  to base the estimate. The estimate was developed, in part, using cost  data from other legacy programs and other assumptions that program  officials said did not turn out to be accurate. Thus, though the business  case analysis helped program officials develop the program’s support  strategy, the cost estimates contained within are not useful for  monitoring current program costs.
The 2002 business case analysis for the Army’s Sentinel PBL  arrangement estimated costs for a 26-year period beginning in 2003  using a fleet size ranging from 126 to 198 radars. According to program  officials, since 2003 the fleet size has ranged from 140 to 143 radars  and additional radars are not anticipated. Although a new business  case analysis was prepared, it had not completed the Army’s review  and approval process at the time of our review.

			PBL Cost Reductions Either Did Not Materialize or Were Not Documented for Most Follow-on Contracts

Few of the remaining programs in our sample could document cost  reductions attributable to the use of a PBL arrangement after negotiating a  follow-on fixed-price contract. The PBL concept envisions that support  providers are incentivized to improve reliability to ensure that  performance metrics are met and reduce their costs to provide support to  increase profits—especially under fixed-price arrangements. To the extent  practicable, we examined how contract costs changed for eight programs  in our sample that negotiated follow-on contracts or priced previously  unpriced contract options after completing fixed-price PBL contracts.  According to officials, a variety of factors affected the support costs  negotiated in the PBL contracts that caused both costs increases and  decreases. Only one program had decreasing support costs that program  officials attributed to the use of a PBL arrangement. One additional  program supported under a cost-plus-award-fee contract also reduced  support costs by changing the metrics included in the contract. However,  these two programs did not have updated business case analyses that  analyzed alternative support options over the remaining life cycle of the  program. Finally, only one program office had developed a methodology  for tracking and verifying reliability improvements made under the PBL  arrangement, although this is necessary for quantifying the related cost  savings.
Support costs for the Navy’s Consolidated Automated Support System  have decreased over the 8-year PBL arrangement that began in April 2000.  Program officials attribute the cost reductions the program has  experienced to the implementation of a PBL arrangement. Depending on  the level of support chosen, the fixed price charged for the annual support  of a test station decreased from 53 to 20 percent (constant 2008 dollars)  from 2000 through 2008. Program officials said that they closely monitored  maintenance data and failure rates in order to negotiate lower fixed prices  where possible. In addition, officials said that they were able to increase  the number of repair and consumable parts covered under the  arrangement over the years. According to officials, prior to the  implementation of the PBL strategy support costs for the program were  even higher, but officials were unable to locate the contracts.
Although support costs for a few of the other seven programs decreased,  officials said that there were a number of other factors that influenced  costs, such as changes in the scope of work or planned usage of the  systems. For example, according to Tube-launched Optically-tracked Wire- guided missile – Improved Target Acquisition System program officials, a  variety of factors affected the costs negotiated in the 2007 contract, and  increased fleet size was one factor that allowed them to negotiate lower  rates per system. In addition, when the first fixed-price PBL arrangement  was implemented in 2001 the program was fairly new with very few  systems, so the program office did not have an extensive amount of  historical program data with which to negotiate. Since 2001, the program  office has collected actual data that it used to negotiate lower rates in the  latest contract. However, according to program officials, the contractor  only recently started making changes to the system to improve reliability.  These improvements were not included in negotiations for the 2007-2011  contract but have begun to improve failure rates and are expected to  reduce costs in future contracts.
Although the Army’s Shadow Tactical Unmanned Aircraft System is not  supported by a firm-fixed-price PBL contract, program officials for the  system said that they were able to reduce support costs by changing the  performance metrics used in the PBL arrangement. The maximum  amounts authorized in the annual cost-reimbursable PBL contract for the  support of this system were reduced by 28 percent from fiscal years 2006  through 2007. According to program officials, a program office review of  PBL processes in early fiscal year 2006 concluded that while the PBL  arrangement was effective in terms of meeting the performance levels  specified in the contract, it was not cost efficient and costs associated  with the vehicle’s high accident rate were an area of particular concern. In  response, the program office changed the performance metrics in the  contract to encourage the contractor to improve reliability and reduce the  accident rate, and also to improve depot maintenance efficiency. As the  accident rate improved, the program office was able to negotiate for lower  support costs in the 2007 PBL contract.
Finally, while the 2005 life cycle business case analysis for the F/A-18 E/F  program office estimated that support provided under a PBL arrangement  would be more expensive than government-provided support, program  officials for the Navy’s F/A-18 E/F PBL arrangement have developed a  process to track and document support cost reductions attributed to  contractor investments to improve reliability and reduce costs. Program  officials said that both the Navy and Boeing have funded initiatives to  improve F/A-18E/F component reliability, maintainability, and  supportability as part of the Supportability Cost Reduction Initiatives  program. Under the current fixed-price PBL arrangement, Boeing has  invested approximately $11.39 million to fund initiatives that officials  currently estimate will generate cost reductions of approximately   $279 million over the remaining life cycle of the system. According to  program officials, Naval Air Systems Command cost analysts have  validated baseline estimates and will annually track the results of the  initiatives in order to quantify actual support cost reductions attributed to  the investments in the future.

			Costs to Achieve Higher-Than- Required Performance Levels Are Unknown

According to program officials, eight of the PBL arrangements within our  sample of 29 systems generally achieved a level of performance that  significantly exceeded what is required under the contracts. According to  the DOD/Defense Acquisition University PBL guide, PBL arrangements  should be structured to meet the needs of the warfighter. Therefore, if  actual performance exceeds what is called for in the PBL arrangement, it  also exceeds the level of performance that is needed. According to  program officials, for eight of the PBL arrangements we reviewed, the  contractors significantly exceeded some of the contractual performance  requirements. For example:    Since 2002, Army officials said that the average annual operational  readiness for the Tube-launched, Optically-tracked, Wire-guided  missile – Improved Target Acquisition System has not been below 99  percent, and the system’s operational readiness has averaged 100  percent since 2004. According to a program official, the Army’s  readiness standard for this system is 90 percent. Despite the Army’s  standard, it continued to include a performance incentive that  encouraged higher levels of performance when negotiating a follow-on  PBL contract in 2007. The performance incentive includes payment of  an award fee that encourages operational readiness rates from 91 to  100 percent, with the highest award fee paid for 100 percent average  operational readiness.
According to officials, since early 2005, monthly readiness rates for the  Army’s Javelin have generally been measured above 98 percent.
However, the PBL contract for support of this system only requires 90  percent mission readiness.
Although the contractual requirement for parts availability for the  Navy’s V-22 engine PBL arrangement has been 90 percent since 1998,  according to program officials, actual parts availability has  consistently averaged 98 to 100 percent.
Similarly, with availability averaging 98 percent since 2004, Air Force  program officials for the LITENING Advanced Airborne Targeting and  Navigation Pod said that the contractor has consistently exceeded the  contract requirement for 92 percent system availability.
For programs where performance significantly exceeded contractual  requirements, it is unclear how much extra was paid to get the added  performance. Since the government is paying for this excess performance,  then the arrangement, as structured, may not provide the best value to the  government, particularly since there are other DOD programs that are not  funded at levels that would be required to achieve their desired level of  readiness.

		Characteristics of DOD’s PBL Arrangements May Limit Their Potential to Reduce Support Costs

Several characteristics of DOD’s PBL arrangements may limit their  potential to reduce costs. First, DOD’s PBL contracts are limited to  relatively short time periods, while proponents of the PBL concept believe  that longer-term PBL arrangements are necessary to encourage support  providers to make investments to improve reliability. Second, in DOD— where changing requirements and priorities can result in fluctuations in  the funding for support of DOD’s weapon systems—creating a stable level  of funding is challenging. Third, many PBL arrangements only transfer  responsibility for inventory management to the contractor and do not  transfer inventory ownership, which reduces incentives for ensuring a  correctly sized inventory level. Finally, many of DOD’s PBL arrangements  do not contain cost metrics or offer specific incentives to encourage cost  reduction initiatives.

			Short-term Contracts

According to program officials, DOD support contracts, including PBL  contracts, that are funded by operation and maintenance funds are  generally limited to 1 year, and working-capital-funded contracts are  generally limited to 5 years, with subsequent option years allowed up to a  total of 10 years. However, according to the DOD/Defense Acquisition  University PBL guide, longer-term PBL contracts are preferred because a  key aspect of PBL is the provision of incentives for contractors to reduce  costs over time through increased reliability while still making a profit.  Further, contract length should be sufficient to allow for an adequate  return on any investments made to improve reliability.
Officials from several PBL arrangements cited instances in which  reliability improvements were needed but contractors were hesitant to  make investments while under annual support contracts. For example,  Joint Primary Air Training System program officials said that during the  original 10-year PBL arrangement that began in 1996, the contractor did  not make any investments to improve unreliable components. Although  officials were expecting the fixed-price performance contract to motivate  the contractor to invest in improvements to increase reliability and  maximize profit, they found that the contractor minimized its own costs  during the contract period and passed on the costs to improve the  reliability of components with high failures to the government when the  contract was renegotiated. Our prior work found that the private sector  sometimes used PBL contracts of 10 to 12 years.

			Unstable Requirements and Funding

Stable requirements and funding, like longer-term contracts, could enable  contractors to make reliability improvements and other business  decisions, such as long-term supplier arrangements, that could improve  performance and reduce future support costs because they have  reasonable assurance of future needs. For example, officials representing  one of the PBL arrangements we reviewed credited stable funding for  much of the program’s cost savings. The F-117 program office estimated  that its arrangement would have cost over $80 million more if the Air  Force had not agreed to stabilize the program’s support budget and  provide the necessary support funding each year of the contract. However,  DOD’s requirements and priorities, and related funding, for weapon  system support are not always stable. For example, according to Army  officials, the Tactical Airspace Integration System’s PBL arrangement was  affected by a significant reduction of the program’s support budget. The  Army subsequently requested that the Defense Acquisition University  study the implications of funding on PBL arrangements and prepare a case  study based on this example. In addition, for the last several years some of  the Army’s PBL arrangements we reviewed did not receive all of their  support funds at the beginning of the fiscal year but rather in increments  throughout the year. Program officials for one Army system said that at  one point during fiscal year 2005, they almost had to discontinue some of  the support provided under the contract because they did not have  adequate support funds. Additional funding was eventually made available  after the program office notified its major command of the situation. Army  program officials said that this funding instability further exacerbates the  impact of having short-term contracts, since all of the funds are not  available to the contractor to make business arrangements or investments  for reliability improvements.

			DOD’s Ownership of Inventory

Many of the PBL arrangements we reviewed only transferred  responsibility for inventory management, not ownership, to the contractor.  An analysis by Sang-Hyun Kim, Morris A. Cohen, and Serguei Netessine of  the Wharton School, University of Pennsylvania, suggests that the  efficiency of a PBL arrangement depends heavily on the asset ownership  structure: with higher ownership responsibility, the supplier is more likely  to spend more to increase reliability and less for maintaining inventory  levels. According to this study, under an arrangement in which the  contractor owns and manages the inventory, reliability improvements and  inventory costs will both be evaluated in terms of their ability to meet  performance metrics and minimize costs. If the PBL arrangement only  includes inventory management, higher inventory levels may be used,  instead of investments to improve reliability, to meet performance  metrics—particularly those that measure availability—since inventory  holding costs are not incurred by the contractor. Consequently, under  DOD’s PBL arrangements, contractors may choose to make fewer  reliability improvements.

			Availability of Cost Metrics and Effective Incentives

Finally, many of DOD’s PBL arrangements do not contain cost metrics or  offer specific incentives to encourage reduced costs. According to an  August 2004 memorandum from the Under Secretary of Defense  (Acquisition, Technology and Logistics) regarding performance-based  criteria, PBL should be constructed to purchase performance, which is  defined in terms of operational availability, operational reliability, cost per  unit usage, logistics footprint, and logistics response time. The guidance  recommended that PBL metrics be tailored to reflect the unique  circumstance of the arrangement, but still support desired outcomes in  terms of the five performance criteria. A subsequent 2005 memorandum  from the Under Secretary of Defense (Acquisition, Technology and  Logistics) directed the use of these metrics as the standard set of metrics  for evaluating overall total life cycle systems management.
Some of the aviation PBL arrangements we reviewed negotiated their  support on a cost per flight hour basis. For those that did not, cost per  flight hour was generally not included as part of the contract performance  plan, with the exception of the F/A-18 E/F PBL arrangement. For example,  the C-17 program office did not negotiate its contract on a per flight hour  basis and does not monitor cost per flight hour as part of its PBL  arrangement. None of the nonaviation PBL arrangements we reviewed  included cost metrics as part of the PBL arrangement. In addition, only  four of the PBL arrangements we reviewed contained incentives for  reducing or controlling costs. For example, the F-117 and Shadow Tactical  Unmanned Aircraft System PBL arrangements each included a cost- sharing provision where the government and the contractor would share  annual savings if actual costs were below negotiated costs. Further,  officials said that the award plan for the F-22 PBL arrangement also will  consider how actual costs compare to negotiated costs when calculating  the amount of award fee the contractor earns at the end of the year.

		DOD Did Not Emphasize the Potential for PBL Arrangements to Reduce Costs or Require the Collection and Reporting of Detailed Contractor Support Cost Data

Although PBL arrangements were included in a DOD pilot program  intended to demonstrate the ability of various initiatives to reduce support  costs, DOD did not emphasize this goal in its guidance or requirements as  it established the concept as the department’s preferred weapon system  support strategy. In general, improved performance was given greater  emphasis, and we found only a few references to cost reduction in DOD’s  guidance on implementing PBL arrangements. With respect to  requirements for cost reporting, DOD and the services do not require that  programs using PBL arrangements, or other contractor logistics support  arrangements, collect and report detailed cost data in a consistent,  standardized format.
Since 2001 DOD’s guidance regarding PBL has emphasized higher levels of  readiness and stressed rapid implementation. For example, in 2001, when  DOD cited PBL as the preferred weapon system support strategy, PBL was  described as a strategy for achieving a higher level of system readiness  through efficient management and direct accountability. In a 2002 Under  Secretary of Defense (Acquisition, Technology and Logistics)  memorandum, the services were instructed to prepare PBL  implementation plans that aggressively pursue the earliest feasible  program implementation end dates. A January 2004 Under Secretary of  Defense (Acquisition, Technology and Logistics) memorandum stated that  PBL was the department’s near-term strategy to increase weapon system  readiness through integrated logistics chains and public/private  partnerships. The memorandum contained guidance to implement PBL  where economically feasible and provided guiding principles for a best- value assessment. The following month a Deputy Secretary of Defense  memorandum again directed the services to provide plans for aggressively  implementing PBL arrangements.
In contrast to DOD’s clearly stated goal to reduce support costs in the late  1990s, we found few references to the potential for PBL to reduce support  costs since 2001. DOD guidance generally only indirectly refers to  potential PBL cost reductions to “compress the supply chain” and “reduce  non-value added steps.” In May 2003, DOD Directive 5000.1, The Defense  Acquisition System, was updated to emphasize that program managers  shall implement PBL strategies “that optimize total system availability  while minimizing cost and logistics footprint.” In March 2004, an Under  Secretary of Defense (Acquisition, Technology and Logistics)  memorandum reiterated that PBL was the preferred strategy and provided  criteria on which to assess potential for PBL application. One of the  criteria stated that the cost per operational unit of performance (such as a  flying hour) should be capable of being reduced through PBL  implementation. Finally, in 2005, the DOD/Defense Acquisition University  PBL guide contained several references to the potential for PBL to  improve reliability and reduce costs.
Program offices often lacked detailed and standardized weapon system  support cost data because DOD has not required them to obtain and report  cost data from the contractors that provide such support, including those  involved in PBL arrangements. According to the OSD Office of Program  Analysis and Evaluation, historical operating and support costs, organized  in a standard format, are necessary for preparation of life cycle cost  estimates for new systems, budget formulation, analysis of working capital  funds, development of business case analyses, and future contract  negotiations. Until 2007, DOD’s guidance for structure of support cost  estimates, which is also suggested as a defined presentation format for  historical operating and support costs, included all contractor support— labor, materials, overhead, and other assets—in one category, while  government-provided support was reported in greater detail among  multiple categories and lower-level subcategories. Therefore, amounts  paid for contractor support were generally reported in the aggregate. In  October 2007, DOD changed its guidance to include a more detailed  presentation of contractor support costs in the various categories, similar  to the reporting of government support costs. However, neither DOD nor  the services have required program offices to obtain or report contractor  support costs, including PBL arrangements with contractors, in this  format.
OSD and service officials are beginning to recognize the need for further  visibility of the costs of support provided by contractors. In late 2006,  OSD’s Office of Program Analysis and Evaluation began a study regarding  the collection of contractor support costs because the department  acknowledged that visibility into these costs in DOD’s systems was  generally limited. Many of the programs studied were PBL arrangements  also included in our sample. OSD’s study also found that program offices  often did not have detailed cost data and, if cost data were provided, the  data often did not conform to, or could not be converted to, the standard  support cost structure. Based on the study results, OSD is considering  requiring contractors to report their actual costs for providing logistics  support, including profit and general and administrative expenses, in  DOD’s standard cost structure. However, the details of the requirement  and which programs will be subject to such reporting have not been  finalized.
Similarly, Air Force officials have also recognized the limitations on  visibility into contractor support costs for weapon systems. The Air Force  is currently considering expanding visibility by requiring that all  contractor-supported programs report actual obligations for contractor  labor and materials (including PBL arrangements) in each of DOD’s cost  structure categories for each aircraft mission design series. According to  Air Force Cost Analysis Agency officials, this requirement is different from  the one being considered by OSD in that the Air Force will have visibility  over the Air Force’s costs for contractor support but not the contractor’s  actual costs.

	Use and Characteristics of Availability Contracts by the United Kingdom’s Ministry of Defence

The United Kingdom’s Ministry of Defence also uses performance-based  arrangements to support its weapon systems. Ministry of Defence officials  refer to this initiative as contracting for availability. Similar to DOD, when  using availability contracts the Ministry of Defence pays industry for  aircraft, engines, or components to be available for military operations,  rather than paying for specific repairs, spares, and technical support.  According to officials, the use of contracting for availability also started as  an approach for reducing costs for weapon system support. Ministry of  Defence officials said that their current contracts for availability generally  provide support for aviation systems, such as helicopters and combat  aircraft. Although there are maritime availability contracts, they said that  most of the ministry’s maritime availability contracts support specific  types of equipment rather than entire ships. In general, the availability  contracts used by the ministry are significantly longer than those used by  DOD, and the ministry uses an “open book accounting” arrangement to  gain visibility into the contractors’ costs to provide support. According to  officials, the annual budget for the Defence Equipment and Support  organization is approximately £13 billion, including funds for conflict  operations.

		Ministry of Defence Budget Pressures Drove Need for Support Cost Reductions

In 1999, the United Kingdom’s Defence Logistics Operation, one of two  entities that merged into the current Defence Equipment and Support  organization, established a goal to reduce costs 20 percent by 2005/2006.  According to Ministry of Defence officials, contracting for availability  began during this period as a way to maintain or improve performance  while assisting in achieving cost reductions. They believe that if industry is  paid for a given level of availability, there are incentives to reduce support  chain costs and make the weapon system more reliable and processes  more efficient. The cost reduction goal was a key driver in the  transformation of the maintenance, repair, and overhaul activity for  Harrier and Tornado fast jet aircraft. A member of the Tornado Integrated  Project Team stated that a number of factors drove the support strategy  change for the Tornado aircraft, but the primary factor was the need to  reduce costs to match budget reductions; the team identified availability  contracting as an effective way to reduce costs and maintain performance.  Officials also stated that the support strategies for all of the ministry’s  helicopters were changed because of increased budget pressures.
In 2007, the United Kingdom’s National Audit Office reported that the  Ministry of Defence has experienced significant reductions in the costs to  support its fast jets; the Tornado and Harrier costs have been reduced  from a total of £711 million in 2001through 2002 to £328 million in 2006  through 2007, providing a cumulative saving of some £1.4 billion over the  6-year period. The National Audit Office reported that the savings were  achieved by working with industry to reform traditional contracts into  availability contracts. However, the report also stated that the ministry did  not have sufficient data to assess the impact of changes in the pattern of  frontline operations and productivity increases from the use of lean  techniques on total costs. National Audit Office officials with whom we  met confirmed that while they could validate overall cost reductions, they  could not attribute the entire savings solely to the use of availability  contracts. Other related initiatives, such as the reorganization and  reduction of locations for aircraft repair and upgrade, the use of lean  techniques, and the use of reliability-centered maintenance, also  contributed to the support cost reductions.

		Analytical Tools Used for Decision Making

Ministry of Defence officials said that they do not require the use of  availability contracts or promote their use as the preferred strategy.  According to officials, the support strategy can and should vary from  system to system depending on the circumstances; in some cases, it may  be appropriate for government activities to support some systems in the  traditional manner and for others to use contracting for availability. To  assist with the decision-making process, the Defence Equipment and  Support organization developed a “support options matrix” for use in  reviewing current and future support arrangements. Officials said that the  matrix was developed to assist with analyzing components of support for  cost and performance drivers, illustrating a range of support options  differentiated by the gradual transfer of cost and performance drivers into  industry management and presenting a clear rationale for each support  chain design in terms of the benefit to be derived from the transfer of  specific cost and performance drivers into industry management. In  addition to the matrix, a contractor capability assessment is also  completed to determine the ability of industry to assume greater  management responsibility. Finally, according to officials, before they  enter into a contract for availability, two additional analyses are  conducted. The first is an investment appraisal, or an “internal value  benchmark,” which calculates the lowest cost at which the service could  be provided by the government. The second is a business case analysis,  which discusses the different proposals and justifies the selection of the  proposed approach. Officials noted that the proposed approach does not  have to be the lowest-cost option, but is usually the option that offers the  best value solution overall.
In its 2007 report, the National Audit Office indicated that internal value  benchmarks were not developed consistently and recommended  development of improved guidance and consistent application of a  common methodology for benchmarks against which to assess the value  of proposed availability contracts. National Audit Office officials said that  they found variance in the quality of these cost estimates and a shortage of  qualified people for cost modeling. They also pointed out that as less and  less support is provided by the government, accurate cost modeling for  use when renegotiating contracts will become more important, and the  Ministry of Defence needs to maintain or improve visibility of support  costs for its weapon systems.

		Long-term Contracts (19- 25 Years) Are Critical to Success

Defence Equipment and Support officials said that they have found the  long-term nature of availability contracts a key factor in reducing costs  and that annual contracts cannot achieve the same benefits as the longer- term contracts do. According to officials, the long-term contracts for  Tornado aircraft and helicopter fleets reduced costs because the  contractors were able to stabilize their supply chain and obtain better  prices from the supplier base. The Ministry of Defence also found that  industry preferred long-term contracts. In a discussion of contracting for  availability, the “Defence Industrial Strategy,” a white paper dated  December 2005, stated that companies are generally interested in using  availability contracts because it provides the commercial firms with  greater returns over a longer period.
Ministry of Defence officials provided us with the following examples of  their long-term availability contracts:    The Ministry of Defence has a 10-year contract with AgustaWestland to  support the Sea King helicopter until it is projected to be removed  from service. The Ministry of Defence has priced the contract for the  first 5 years, and thereafter it will establish the price in 5-year  increments.
The Ministry of Defence has a 23-year contract with VT Group to  support two survey ships owned by the ministry. The contract has  price renegotiation points at the 7-, 15-, and 20-year points.
The Ministry of Defence has a 19-year contract with BAE to support  the fleet of Tornado aircraft. The ministry awarded the contract in  December 2006 and priced it for the first 10 years.
The Ministry of Defence has a 25-year contract with AgustaWestland to  support the Merlin helicopter until it is projected to be removed from  service. The price for the initial 5-year period of the contract is fixed,  and the ministry is currently negotiating prices for the next 5-year  period of performance that begins in 2011.

		Other Characteristics of Ministry of Defence Availability Contracts

Ministry of Defence officials said that other factors, such as inventory  ownership, contract incentives, and cost visibility, were also important  when contracting for availability. Officials told us that they preferred to  transfer not only management of inventory but also inventory ownership  under such arrangements. They noted that under some of their current  availability contracts this had not been possible for a variety of reasons.  Nonetheless, in the future they intend to pursue transfer of inventory  ownership as much as possible. Examples of Ministry of Defence  availability contracts where officials said that inventory is owned by  industry, also known as spares inclusive, include a contract for support of  two survey ships.
In addition, according to ministry officials, several of the availability  contracts—including those supporting the Sea King and Merlin helicopters  and Tornado fast jets—had incentives referred to as gain share or pain  share. In these types of arrangements, the contractor and government  share cost savings or cost overruns in prenegotiated proportions.  According to officials, they found that these types of metrics are useful to  influence contractor cost control measures and provide an incentive for  industry to develop changes and modifications that reduce support costs.  Officials familiar with the Tornado fast jet availability contract explained  that their arrangement included gain sharing and pain sharing on both the  variable and fixed-price portions of the contract.
Finally, officials explained that in many of the Ministry of Defence’s  availability contracts, the concept of open book accounting is employed.  Open book accounting is not a defined term but is more of a general  expression describing a level of access to accounting data that would not  normally be available under a conventional contract. In availability  contracts, open book accounting allows government program officials to  review the accounting records of the contractor. This access is not without  limits. Officials said that the level of access must be agreed to in advance  on a case-by-case basis and reflects the circumstances of the arrangement  and the need for access to certain data to monitor performance or benefits  arising from the arrangement. For example, one contract may only provide  for man-hour data because that is all that needs to be shared given the  circumstances. However, another contract may allow access to direct cost,  direct labor hours, and other rates and factors that are relevant for the  work involved. According to officials, the Ministry of Defence has an open  book accounting agreement with AgustaWestland for the Merlin contract  and the government has full visibility of the accounts pertaining to Merlin,  including overhead costs. The contract must explicitly address the data  access arrangements and not rely on vague and undefined phrases that  could be open to misinterpretation.

		The Ministry of Defence’s Long-term Availability Contracts May Affect Budget Flexibility

According to the 2007 National Audit Office report, long-term availability  contracts may limit flexibility to respond to changes in resources. In the  past, integrated project team leaders in the Ministry of Defence had some  ability to move funding between resource lines to overcome short-term  funding issues. However, this flexibility is diminishing because of the  transition to availability contracts, as larger portions of the budget are pre- allocated to fund these contracts. The Mine Warfare Patrol and  Hydrographic Team also raised concerns about loss of budget flexibility.  This team is responsible for providing support for 2 hydrographic ships, 1  patrol ship (HMS Clyde), 3 River class ships, 16 mine hunters, and 38  smaller ships. The budget for providing support to these ships is  approximately £40 million, with £18 million devoted to the long-term  availability contracts for the 2 survey ships, 1 patrol ship, and 3 River class  patrol ships. According to Ministry of Defence officials, these  arrangements have for the most part been beneficial. However, as they are  structured, these programs do not allow for any flexibility. When the Mine  Warfare Patrol and Hydrographic Team recently had to absorb a 20  percent budget cut, officials said that the mine hunter ships bore the brunt  of the cut because they had the majority of the remaining support budget  not earmarked for an availability contract. The team views the 20 percent  cut to its budget to be, effectively, a 40 percent cut to the mine hunter ship  budget. Defence Equipment and Support organization officials said that  they are looking to add more flexibility to future availability contracts.
The Ministry of Defence has already incorporated some flexibility in a few  availability contracts. Officials said that the Tornado contract contains  both fixed-price elements for management team, training, logistics, and  information systems and a variable price element for flying hours. Given  this, the contract is fairly flexible and payment is based on certain flying  hour availability bands—with the bands ranging from 70 to 110 percent  availability in 10 percent increments that are agreed to annually. As  another example, officials explained that the Merlin contract provides  flexibility in that the prenegotiated price is linked to banded flying hours  with fixed and variable elements. Under traditional contracting, they  estimate that only 20 percent of the cost would vary with flying hours.  Also, within the basic contract parameters there is a provision for surge  delivery for the Merlin helicopter. Finally, according to officials, the Sea  King helicopter support contract has a similar flexibility because there are  a number of flying hour bands and each band has its own price. In this  manner, the Ministry of Defence can increase or decrease flying hours  without renegotiating the contract. Officials pointed out that one  drawback is that the price charged per flying hour at the lower bands is  higher because the contractor must be able to cover fixed costs with fewer  flight hours to charge. However, they said that the cost per flying hour is  still far less than it would have been under a more flexible traditional  arrangement.

	Conclusions

While the concept of using PBL support arrangements was intended to be  a cost reduction strategy as well as one that would result in improved  performance, DOD’s emphasis has been more focused on performance and  less focused on cost. DOD no longer emphasizes reducing costs as a goal  for PBL programs, and DOD’s implementation of PBL, in its current form,  does not ensure that its PBL arrangements are cost effective. DOD’s  emphasis on the implementation of PBL as the preferred weapon system  support strategy has deemphasized the importance of the development of  consistent, comprehensive, and sound business case analyses to influence  decisions regarding the use of a PBL arrangement. Although DOD’s  guidance recommends using business case analyses to guide decisions  about using PBL arrangements for weapon system support, the DOD  guidance does not require these analyses and almost half of the programs  we reviewed either did not perform a business case analysis or did not  retain documentation of their analysis. Further, the quality of the analyses  of those programs that had performed a business case analysis varied  considerably since many were missing elements of what DOD guidance  recommends for sound economic analyses. Additionally, most of those  analyses that should have been updated had not been. Thus, DOD lacks a  sound approach for analyzing whether proposed PBL arrangements are  the most cost-effective strategy for supporting weapon systems. Without  instituting a more consistent, comprehensive, and sound process on which  to base decisions regarding the type of arrangement to be used in  supporting DOD systems, it is unlikely that the department will be  successful in achieving the sizable savings that were envisioned when the  PBL concept was adopted.
Assessing the cost-effectiveness of PBL programs also requires the  availability of better cost data at a level of detail that would support the  improved management of ongoing PBL programs, including awarding  contract fees, assessing performance versus the cost to achieve it,  evaluating historical costs to determine whether the status quo should be  maintained over time, and making support decisions about future follow- on programs. Such data are usually not available for PBL programs,  limiting the ability of program offices to make program adjustments or  take restructuring actions when appropriate. Nonetheless, a few program  offices have acquired data at this level and indicate that they obtained  them in a cost-effective manner. Improved access to detailed cost data is  another essential element in improving the quality of data available to  DOD decision makers regarding the cost-effectiveness of PBL  arrangements.

	Recommendations for Executive Action

To ensure that PBL arrangements are the most cost-effective option for  weapon system support, we recommend that the Secretary of Defense  direct the Under Secretary of Defense (Acquisition, Technology and  Logistics) to take the following five actions:  revise DOD’s Acquisition Directive to require development of a  business case analysis to support the decision-making process  regarding weapon system support alternatives, including PBL;  revise PBL business case analysis guidance to more clearly define  what should be included in a business case analysis and to establish  specific criteria and methods for evaluating PBL support  arrangements, including evaluation at the subsystem and component  levels;  revise PBL business case analysis guidance to more clearly define  when business case analyses should be updated during the weapon  system life cycle;  require that each service revise guidance to implement internal  controls to ensure that program offices prepare and update business  case analyses that are comprehensive and sound; and  require program offices to collect and report cost data for PBL  arrangements in a consistent, standardized format with sufficient  detail to support traditional cost analysis and effective program  management.

	Agency Comments and Our Evaluation

In written comments to a draft of this report (see app. II), DOD generally  concurred with our five recommendations, noting that the department is  committed to evaluating PBL strategies using business case analyses as  part of the overall supportability assessment made during the development  stages of weapon system acquisition programs. Specifically, the  department fully concurred with three recommendations and partially  concurred with two.
DOD fully concurred with our first recommendation to revise DOD’s  acquisition directive to require the development of a business case  analysis to support the decision-making process regarding weapon system  support alternatives, including PBL. DOD stated that the department will  take steps to address this issue in the next iteration of the DOD Directive  5000.1 and DOD Instruction 5000.2 acquisition regulations. According to  DOD’s response, this new policy will require that the use of a business  case analysis be mandatory and that this analysis serve as a sound basis  for the selected supportability strategy.
In response to our second recommendation to revise PBL business case  analysis guidance to clearly define what should be included in a business  case analysis and to establish specific criteria and methods for evaluating  PBL support arrangements, DOD partially concurred, stating that it  established a Life Cycle Product Support Assessment Team in September  2008 to study product support policy, guidance, past performance, and  results. As part of the study, existing business case analysis policy is being  reviewed, and the department will evaluate the team’s recommendations  on providing specific criteria and methods for evaluating support  arrangements and determine how best to incorporate these  recommendations into mandatory policy. The team’s initial  recommendations are expected in April 2009.
DOD fully concurred with our third recommendation to revise PBL  business case analysis guidance to more clearly define when, during the  weapon system life cycle, business case analyses should be updated.  According to DOD’s response, the department’s Life Cycle Product  Support Assessment Team will evaluate the appropriate timing of initial  business case analyses and follow-on updates to validate the life cycle  support approach for weapon systems, and the team’s recommendations  will be evaluated for inclusion into mandatory policy.
DOD fully concurred with our fourth recommendation to require that each  service revise guidance to implement internal controls to ensure that  program offices prepare and update business case analyses that are  comprehensive and sound. As we noted in our report, the Army has  already implemented a PBL business case analysis review and approval  process. DOD stated that the Army’s internal controls will be reviewed by  the Life Cycle Product Support Assessment Team, which will make  recommendations for expansion for DOD-wide governance policy as part  of the team’s overall recommendations expected in April 2009.
DOD partially concurred with our fifth recommendation to require  program offices to collect and report support cost data for PBL  arrangements in a consistent, standardized format with sufficient detail to  support traditional cost analysis and effective program management. DOD  stated that a provision for tailored cost reporting for major acquisition  programs designed to facilitate future cost estimating and price analysis  has been included in the draft DOD Instruction 5000.2, which is expected  to be approved in the next 30 days. Additionally, the Life Cycle Product  Support Assessment Team is reviewing support cost reporting and cost  analysis as a part of its ongoing study. According to DOD’s response, the  ultimate goal is standardized support cost reporting for all life cycle  product support efforts, to include support provided by government  activities.
While concurring with our recommendations, DOD’s response noted that  the department disagrees with the assertion that the goal of PBL  arrangements is to reduce costs. Rather, the primary goal of PBL  arrangements is to increase readiness and availability while reducing  overall sustainment costs in the long run. Our report recognized that the  current DOD Directive 5000.1 provides that PBL arrangements shall  optimize total system availability. However, our report notes that this  directive also provides that PBL arrangements shall minimize costs and  the logistics footprint. Moreover, our report stated that PBL emerged from  a 1999 DOD study to test logistics reengineering concepts that placed  greater reliance on the private sector for providing weapon system  support to both reduce support costs and improve weapon system  performance. Thus, reducing costs was a central focus of the adoption of  PBL as DOD’s preferred support strategy. Based on our analysis in this  report, we continue to believe that the PBL support arrangement concept  was intended to be a cost reduction strategy as well as a strategy that  would result in improved performance.
DOD’s response also noted that 22 of the 29 programs we reviewed  produced business case analyses that enabled sound support strategy  determinations. DOD further stated that for 28 of the 29 programs, the PBL  strategies produced performance benefits, readiness benefits, or both, and  15 of the programs reflect cost-neutral or savings benefits resulting from  the application of the PBL strategies. However, based on our analysis in  this report, we continue to believe that only 20, rather than 22, of the  programs had business case analyses that evaluated PBL strategies.  Further, as we stated in our report, 6 of these did not retain some or all of  the documentation and 13 were missing elements of DOD’s criteria for  economic analyses. For example, we found that for one analysis the less  costly option would have changed if the department had calculated the net  present value of the two options considered. Additionally, because the  department did not document all the potential support options in the  business case analyses, it is not possible to determine if the most cost- effective options were chosen. Thus we continue to question the extent to  which these analyses enabled sound support strategy determination.  Finally, while we recognize that the PBL arrangements may have produced  performance benefits, readiness benefits, or both, deficiencies in updated  business case analyses and detailed cost data did not support an  assessment of support costs. Therefore, it is unclear how many of the  programs may have actually had cost-neutral or savings benefits resulting  from PBL strategies. We continue to believe that improvements in  collection and reporting of support cost data and the updating of business  case analyses are essential if DOD is to determine the cost-effectiveness of  its PBL arrangements.
We are sending copies of this report to interested congressional  committees and the Secretary of Defense. The report is also available at no  charge on the GAO Web site at http://www.gao.gov.
If you or your staff have any questions regarding this report, please  contact me at (202) 512-8365 or solisw@gao.gov. Contact points for our  Offices of Congressional Relations and Public Affairs may be found on the  last page of this report. Key contributors to the report are listed in  appendix III.

Appendix I: Scope and Methodology

To evaluate (1) the extent to which the Department of Defense (DOD)  used business case analyses to guide decisions regarding performance  based logistics (PBL) arrangements and (2) the impact PBL arrangements  have had on weapon system support costs, we selected a nonprobability  sample of 29 PBL arrangements for weapon system support initiated from  1996 through 2007. The 29 PBL arrangements were selected from lists of  weapon systems supported by PBL arrangements provided by service  officials. With the exception of the Navy’s, we found that the lists provided  by the services either were not current or contained inaccuracies, and the  content of the lists changed significantly during the course of our review,  which affected our sample selection. We chose system-, subsystem-, and  component-level PBL arrangements from each of the services based on  length of time since implementation, location of program office, dollar  value, and prior audit findings. The 29 PBL arrangements we selected  constitute a nonprobability sample and the results are not generalizable to  the population of PBL arrangements.
To evaluate the extent to which DOD used business case analyses to guide  decisions regarding PBL arrangements, we interviewed officials regarding  DOD and service requirements, policies, and guidance for business case  analyses since 2001 and reviewed applicable documents. We also reviewed  DOD’s 1995 economic analysis instruction, which states that analytical  studies that evaluate the cost and effectiveness of weapon system support  are considered to be “economic analyses,” and determined that the  guidance is consistent with Office of Management and Budget guidance for  benefit-cost analyses of federal programs. We interviewed program  officials to discuss any business case analyses prepared to evaluate the 29  PBL arrangements before or after PBL implementation and examined the  analyses using the criteria contained in DOD’s economic analysis  guidance.
To evaluate the impact that PBL arrangements have had on weapon  system support costs, we interviewed program officials to discuss the  characteristics of the PBL arrangements, including contract length,  contract type, scope of work, performance measures, performance  incentives or disincentives, and cost data availability. In addition, we  asked program officials to identify support cost reductions that occurred  as a result of PBL implementation. If a program had renewed a fixed-price  PBL arrangement or had finalized contract options that were not priced,  we analyzed the contracts for trends in PBL support costs. We also  compared PBL contract costs to estimated PBL support costs in business  case analyses, where available, to determine how closely the estimates  matched the actual PBL arrangement costs. We also relied on previously  issued GAO reports on DOD’s implementation of PBL.
To analyze the use of availability contracts for weapon system support by  the United Kingdom’s Ministry of Defence, we interviewed officials from  the Defence Equipment and Support organization regarding policies or  requirements for availability contracts and trends regarding the use of  these arrangements. We also interviewed officials from programs  identified by the Ministry of Defence as using availability contracts for  weapon system support to identify the characteristics of the specific  arrangements and the impact that the use of these contracts had on  support costs. In addition, we interviewed National Audit Office officials  who reviewed the cost and performance of two availability contracts for  support of fast jets. Finally, we reviewed audit reports and other  documents from the Ministry of Defence and National Audit Office. We  obtained these data for informational purposes only and did not  independently verify the statements or data provided by Ministry of  Defence and National Audit Office officials.
Specifically, in performing our work we interviewed officials and obtained  documents related to PBL at the Office of the Under Secretary of Defense  (Acquisition, Technology and Logistics), the Office of the Assistant  Secretary of the Navy (Research, Development and Acquisition), the Office  of the Assistant Secretary of the Air Force (Acquisition), the Office of the  Assistant Secretary of the Air Force (Installations, Environment and  Logistics), the Office of the Assistant Secretary of the Army (Acquisition,  Logistics and Technology), the Marine Corps Headquarters, the U.S. Army  Materiel Command, the U.S. Army Aviation and Missile Command, the U.S.  Army Communications and Electronics Command, the Naval Sea Systems  Command, the Naval Air Systems Command, the Naval Center for Cost  Analysis, the Air Force Cost Analysis Agency, the Air Force Directorate of  Economics and Business Management, the Air Force Materiel Command,  the Air Force Aeronautical Systems Center, the Oklahoma City Air  Logistics Center, the Warner Robins Air Logistics Center, the Ogden Air  Logistics Center, the United Kingdom Ministry of Defence, and the United  Kingdom National Audit Office. We conducted this performance audit  from February 2007 through December 2008 in accordance with generally  accepted government auditing standards. Those standards require that we  plan and perform the audit to obtain sufficient, appropriate evidence to  provide a reasonable basis for our findings and conclusions based on our  audit objectives. We believe that the evidence obtained provides a  reasonable basis for our findings and conclusions based on our audit  objectives.

Appendix II: Comments from the Department of Defense

Appendix III: GAO Contact and Staff Acknowledgments

	GAO Contact

	Acknowledgments

In addition to the contact named above, Julia Denman, Assistant Director;  Harold Brumm; Matt Dove; Jennifer Echard; Chaneé Gaskin; Tom Gosling;  Jennifer Jebo; Mae Jones; Kevin Keith; Charles Perdue; Janine Prybyla;  and Karen Thornton made major contributions to this report.

Summary:
Why GAO Did This Study

In 2001, the Department of Defense (DOD) identified performance based logistics (PBL) as the preferred weapon system support strategy. Within DOD, PBL is the purchase of performance outcomes, such as system availability, rather than the purchase of individual elements of logistics support--such as parts, repairs, and engineering support. Although PBL initially arose from efforts to reduce support costs, questions have arisen about whether PBL has reduced support costs as originally intended. GAO was asked to evaluate the extent to which DOD has used business case analyses to guide decisions related to PBL arrangements and the impact PBL arrangements have had on weapon system support costs. In conducting the review, GAO analyzed the implementation of PBL arrangements for 29 weapon system programs. GAO also looked at the use and characteristics of performance-based contracting in the United Kingdom's Ministry of Defence.

What GAO Found

Although DOD's guidance recommends that business case analyses be used to guide decision making regarding the implementation of PBL to provide weapon system support, the services are not consistent in their use of such analyses. About half of the DOD program offices responsible for the 29 PBL arrangements GAO reviewed either did not use a business case analysis or could not provide documentation for significant parts of their analyses. Almost all of the remaining analyses were missing one or more of the recommended elements in DOD's instruction for economic analysis. Finally, business case analyses were often not updated in accordance with service policies and guidance. Program office use of these analyses is inconsistent because DOD only recommends, but does not require, that they be prepared and because DOD's guidance on preparing a business case analysis is not comprehensive and does not adequately specify the criteria to be included. Also, most of the services have not established effective internal controls to ensure that the analyses are prepared or that they provide a consistent and comprehensive assessment. As a result, DOD has implemented PBL arrangements without the benefit of sound analyses that ensure that the chosen approach will provide the most cost-effective support option. While one of DOD's goals in moving toward the use of PBL arrangements was to reduce weapon system support costs, the ability of these arrangements to reduce costs remains unclear 7 years after DOD first identified PBL as the preferred weapon system support strategy. Many DOD program offices that implemented PBL arrangements have limited cost data, and various other factors--such as the lack of business case analyses--further limit an evaluation of the costs of this support strategy. Available data from the programs GAO reviewed indicated mixed results. Although a few programs in GAO's sample provided evidence of some cost reductions, GAO's analysis of the only two systems in its sample that are managed using both a PBL arrangement and a more traditional, non-PBL arrangement indicated that in both cases the PBL arrangement had higher costs. Also, GAO found that certain characteristics of DOD's PBL arrangements--contract length, funding stability, ownership of inventory, and the lack of cost metrics and effective incentives--could limit the ability of and incentive for contractors to reduce support costs. Neither DOD nor the services require detailed cost reporting for PBL arrangements and the lack of detailed cost data hinders DOD's ability to determine whether PBL has reduced support costs as intended. GAO describes the use of performance-based arrangements for weapon system support in the United Kingdom's Ministry of Defence, which the ministry refers to as contracting for availability. The Ministry of Defence began awarding availability contracts as an approach to reduce weapon system support costs, and officials believe that support cost reductions have been achieved as a result of using availability contracts. In general, the availability contracts used are significantly longer than those used by DOD, and the ministry uses an "open book accounting" arrangement to gain visibility into the contractors' costs to provide support.