Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks
Published: 2008-03-11T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Joint Strike Fighter (JSF) program seeks to produce and field three aircraft variants for the Air Force, Navy, Marine Corps, and eight international partners. The estimated total investment for JSF now approaches $1 trillion to acquire and maintain 2,458 aircraft. Under congressional mandate, GAO has annually reviewed the JSF program since 2005. GAO's prior reviews have identified a number of issues and recommended actions for reducing risks and improving the program's outcomes. This report, the fourth under the mandate, focuses on the program's progress in meeting cost, schedule, and performance goals; plans and risks in development and test activities; the program's cost-estimating methods; and future challenges facing the program. To conduct its work, GAO identified changes in cost and schedule from prior years and their causes, evaluated development progress and plans, assessed cost-estimating methodologies against best practices, and analyzed future budget requirements.
section_title: What GAO Found
paragraphs: Since last year's report, the JSF program office estimates that total acquisition costs increased by more than $23 billion, primarily because of higher estimated procurement costs. The JSF development cost estimate stayed about the same. Development costs were held constant by reducing requirements, eliminating the alternate engine program, and spending management reserve faster than budgeted. Facing a probable contract cost overrun, DOD implemented a Mid-Course Risk Reduction Plan to replenish management reserves from about $400 million to about $1 billion by reducing test resources. Progress has been reported in several important areas, including partner agreements, first flights of a JSF prototype and test bed, and a more realistic procurement schedule. The midcourse plan carries the risk of design and performance problems not being discovered until late in the operational testing and production phases, when it is significantly more costly to address such problems. The plan also fails to address the production and schedule concerns that depleted management reserves. Cost and schedule pressures are mounting. Two-thirds of budgeted funding for JSF development has been spent, but only about one-half of the work has been completed. The contractor is on its third, soon to be fourth, manufacturing schedule, but test aircraft in manufacturing are still behind, the continuing impacts of late designs, delayed delivery of parts, and manufacturing inefficiencies. We believe that JSF costs will likely be much higher than reported. The estimates do not include all costs, including about $6.8 billion for the alternate engine program. In addition, some assumptions are overly optimistic and not well documented. Three independent defense offices separately concluded that program cost estimates are understated by as much as $38 billion and that the development schedule is likely to slip from 12 to 27 months. Discrepancies in cost estimates add to program risks and hinder congressional oversight. Even so, DOD does not plan for another fully documented, independent total program life-cycle cost estimate until 2013. As JSF finalizes the three designs, matures manufacturing processes, conducts flight tests, and ramps up production, it faces significant challenges. JSF's goal--to develop and field an affordable, highly common family of strike aircraft--is threatened by rising unit procurement prices and lower commonality than expected. The program also makes unprecedented funding demands--an average of $11 billion annually for two decades--and must compete with other defense and nondefense priorities for the shrinking federal discretionary dollar. Further, expected cost per flight hour now exceeds that of the F-16 legacy fighter, one of the aircraft it is intended to replace. With almost 90 percent (in terms of dollars) of the acquisition program still ahead, it is important to address these challenges, effectively manage future risks, and move forward with a successful program that meets our and our allies' needs.

section_title: Letter
paragraphs: The Joint Strike Fighter (JSF) is Department of Defense’s (DOD) most  complex and ambitious aircraft acquisition, seeking to simultaneously  produce and field three aircraft variants for the Air Force, Navy, Marine  Corps, and eight international partners. For the United States, the JSF will  need a joint, long-term commitment to very large annual funding  requirements and a total investment now approaching $1 trillion dollars— $300 billion to acquire 2,458 aircraft and $650 billion in life-cycle operation  and support costs, according to official program estimates. The JSF is  critical to our nation’s plans for recapitalizing tactical aircraft and just as  important to our allies.
The Ronald W. Reagan National Defense Authorization Act for Fiscal Year  2005 requires GAO to review the JSF program annually for five years.  Previous reports identified opportunities for the program to reduce risks  and improve the chance for more successful outcomes. We have  expressed concern about the substantial overlap of development, test, and  production activities and recommended a more evolutionary and  knowledge-based acquisition strategy with limited investment in  production aircraft until each variant demonstrates required capabilities in  flight testing. This is the fourth report under the mandate in which we (1)  determine the JSF program’s progress in meeting cost, schedule, and  performance goals; (2) assess plans and risks in development and test  activities; (3) evaluate program office cost-estimating methodology; and  (4) identify future challenges facing the program.
The act also requires us to certify whether we had access to sufficient  information to make informed judgments on the matters contained in our  report. While we were provided sufficient information to make the  assessments contained in this report, a continuing concern has been the  currency of cost and schedule data. The data we are reporting here are  from the JSF Selected Acquisition Report (SAR) dated December 2006 that  was released to Congress and us in April 2007. That SAR reflects the  program position at the time of the submission of the fiscal year 2008  President’s Budget. We were not able to review the program office’s  updated estimates associated with the fiscal year 2009 budget, which will  be reported in the new December 2007 SAR to be released in April 2008.  The program office declined to provide updated costs, stating that those  figures are sensitive because the new budget request had not been  finalized at the time of our review. Every year, this timing disconnect  results in us reporting soon-to-be-outdated cost and schedule data. For  example, shortly after our last report was issued on March 15, 2007, DOD  released new cost estimates that disclosed an increase of more than $23  billion in JSF program costs. On the basis of the evidence we do have and  our analysis, we fully expect future cost estimates to be substantially  higher than the program estimates in this report.
To conduct this work, we tracked and compared current cost and  schedule estimates with those of prior years, identified major changes, and  determined causes.  We obtained earned value data, contractor workload  statistics, performance indicators, and manufacturing results.  We  assessed the program office’s cost estimating methodologies against best  practices prescribed in GAO’s Cost Assessment Guide.  We discussed  results to date, plans, and future challenges with DOD and contractor  officials.  We conducted this performance audit from June 2007 to March  2008 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.
subsections: section_title: Background
paragraphs: The JSF program goals are to develop and field an affordable, highly  common family of stealthy, next-generation strike fighter aircraft for the  Navy, Air Force, Marine Corps, and U.S. allies. The JSF family consists of  three variants. The conventional takeoff and landing (CTOL) variant will  primarily be an air-to-ground replacement for the Air Force’s F-16 Falcon  and the A-10 Warthog aircraft, and will complement the F-22A Raptor. The  short takeoff and vertical landing (STOVL) variant will be a multirole  strike fighter to replace the Marine Corps’ F/A-18C/D and AV-8B Harrier  aircraft. The carrier-suitable (CV) variant will provide the Navy a multi- role, stealthy strike aircraft to complement the F/A-18E/F Super Hornet.  DOD is planning to buy a total of 2,458 JSFs. The F-35 JSF was christened  Lightning II in July 2006.
Because of the program’s sheer size and the numbers of aircraft it will  replace, the JSF is the linchpin of DOD’s long-term plan to modernize  tactical air forces. It is DOD’s largest acquisition program, with total cost  currently estimated at $300 billion; the longest in planned duration, with  procurement projected through 2034; and the largest cooperative  international development program. Our international partners are  providing about $4.8 billion toward development, and foreign firms are  part of the industrial base producing aircraft. They are expecting to  procure a minimum of 646 CTOL and STOVL JSFs. DOD’s funding  requirements for the JSF assume the benefits in reduced unit costs from  these purchases.
Figure 1 shows the JSF’s current procurement profile for U.S. and  international partners. Partner purchases begin in 2009 and reach a  maximum of 95 per year in fiscal year 2016. Total expected procurement in  that peak year, including U.S. quantities, is 225 aircraft.
The JSF is a single-seat, single-engine aircraft, designed to rapidly  transition between air-to-ground and air-to-air missions while still  airborne. To achieve its mission, JSF will incorporate low-observable  technologies, defensive avionics, advanced onboard and offboard sensor  fusion, internal and external weapons, and advanced prognostic  maintenance capability. According to DOD, these technologies represent a  quantum leap over legacy tactical aircraft capabilities. In several ways, JSF  development is also more complex and challenging than the F-22A Raptor  and F/A-18E/F Super Hornet programs, the other two contemporary  aircraft that DOD is acquiring with JSF to recapitalize tactical air forces.  The JSF program is simultaneously developing several airframes and  engines for multiple customers and is projected to have significantly more  lines of operational flight plan software code than the other aircraft.
The JSF program began in November 1996 with a 5-year competition  between Lockheed Martin and Boeing to determine the most capable and  affordable preliminary aircraft design. Lockheed Martin won the  competition. The program entered system development and demonstration  in October 2001. At that time, officials planned on a 10½ years  development period costing about $34 billion (amount includes costs of  about $4 billion incurred before system development start).  By 2003,  system integration efforts and a preliminary design review revealed  significant airframe weight problems that affected the aircraft’s ability to  meet key performance requirements. Weight reduction efforts were  ultimately successful but added substantially to program cost and  schedule estimates. In March 2004, DOD rebaselined the program (2004  Replan), extending development by 18 months and adding $7.5 billion to  development costs. Program officials also delayed the critical design  reviews, first flights of development aircraft, and the low-rate initial  production decision to allow more time to mitigate risks and mature  designs.
section_title: Progress Measured against Cost, Schedule, and Performance Goals Was Mixed over the Last Year
paragraphs: The total program acquisition cost estimate by the JSF program office has  increased since our report last year, primarily due to higher projected  procurement unit prices. The reported schedule for major events showed  mostly minor slips. Engineering analyses continue to show performance  requirements are met, but flight and ground tests planned through 2013  will be necessary to confirm these assessments. DOD and the contractor  reported progress in several areas, including international partner  agreements, first flights of a JSF prototype and test bed, and a more  realistic procurement schedule.
subsections: section_title: The Program Cost Estimate Increased, while Schedule and Performance Estimates Remained about the Same
paragraphs: JSF costs increased since last year.  Table 1 shows the evolution of cost,  quantity, and delivery estimates from the initiation of system development,  through the 2004 Replan, to the latest data available. It demonstrates the  impacts of higher procurement costs on unit costs and schedule delays on  the delivery of promised capabilities to the warfighters.
The current estimate for procurement costs, dated December 2006, shows  an increase of $23.4 billion (plus 10 percent) from the estimate of a year  earlier and a total of $55.3 billion more (plus 28 percent) since 2004.
Procurement cost increases were primarily due to (1) extending the  procurement period seven years at lower annual rates, (2) increased future  price estimates based on contractor proposals for the first production lot,  (3) airframe material cost increases, and (4) increases resulting from  design maturation. Offsetting a portion of the procurement cost increases  were lower estimates for labor rates and subcontractor costs.
The official development cost estimate has remained relatively constant  since the 2004 Replan. However, there were significant changes in scope  and planned use of funds in order to maintain that estimate as officials  reduced requirements, did not include full funding for the alternate engine  program despite congressional interest in the program, and spent  management reserves much faster than budgeted. Management reserves  are a pool of money set aside—in this case about 10 percent of the  development contract value remaining—to handle unanticipated changes  and other risks encountered as a development program proceeds. Weight  growth early in development and subsequent problems resulting from late  aircraft design changes and subsequent manufacturing inefficiencies  depleted reserve funds to an untenable level by 2007. The program faced a  probable contract overrun. DOD officials opted not to request additional  funding and time to complete development and instead adopted a  controversial plan that reduced budgeted funds for development test  aircraft and flight plans in order to replenish management reserves from  $400 million to about $1 billion, an amount deemed prudent to complete  the development phase on time. This plan, known as the Mid-Course Risk  Reduction plan, is discussed in more detail later in this report.
Reported schedule slips for key events since last year’s report were minor  for the most part, but schedules could worsen considerably if the delays in  maturing the aircraft and engine designs and manufacturing test aircraft  continue to push work effort into later years. This would further compress  the time available to complete development and test efforts, affecting the  scheduled start of initial operational test and evaluation and the full-rate  production decision, and increasing the risk of further delivery delays. The  CV’s critical design review, the last of three design reviews for the  program, occurred in June 2007, seven months later than had been  expected. The initial operational capability date for this variant was  pushed out two years to March 2015, to provide more time to mature  design and test this variant in the demanding carrier environment. The  carrier variant is the least developed of the three, incorporates larger  wings, is heavier, and has different speed and range performance  requirements than the other two variants.
On the basis of engineering analyses and computer modeling, the JSF  program projects that the aircraft design will meet seven of the eight key  performance parameters by the end of development. The aircraft is  currently not meeting the interoperability parameter, but this depends on  capabilities being developed outside the JSF program. Key performance  parameters will be verified during ground and flight testing from 2010 to  2013.
section_title: Progress Was Made This Year in Several Important Areas
paragraphs: DOD and the contractor made solid progress this year in several areas that  could establish a foundation to spur future successes. With almost 90  percent (in terms of dollars) of the acquisition program still ahead, these  and other improvements could be leveraged to help better meet cost,  schedule and performance goals.
In February 2007, the United States and eight international partners  signed the Production, Sustainment, and Follow-on Development  Memorandum of Understanding, committing to purchase aircraft and  continuing joint development activities.
DOD reduced near-term procurement quantities and the rate of ramp-  up to full rate production. These actions somewhat lessened the  concurrency of development and production we have previously cited  and make for a more achievable schedule.
The prime contractor and major subcontractors continued to  implement advanced design and development techniques and utilize  extensive computer modeling and simulation in innovative ways for  design, test, and integration activities.
DOD and contractor officials also made good progress toward refining  system capabilities, including establishing mission software  requirements, with the goal of improving future program executability  while still meeting warfighter requirements.
First flights of the prototype test aircraft and a flying test bed occurred  in fiscal year 2007. Both are viewed as important risk reducers in the  test program and initial flights provided valuable and useful  information, according to program and contractor officials.
All test aircraft were in manufacturing during 2007. Low-rate initial  production of the first two production aircraft and advance buys for  the second production lot also got under way.
section_title: Development Program Faces Increased Risks of Further Cost Increases and More Schedule Delays
paragraphs: Late in 2007, DOD officials approved a risky and controversial plan that  replenishes management reserves by reducing development test aircraft  and test flights in order to stay within current cost and schedule estimates.  Difficulties in stabilizing aircraft designs and inefficient production of test  aircraft resulted in spending management reserves faster than anticipated.  The flight test program has barely begun, but faces substantial risks with  reduced assets as delays in design and manufacturing continue to further  compress the time available to complete development work prior to  operational testing and to support the full-rate production decision. The  JSF program is halfway to its planned completion, but is behind schedule  and over cost. On the basis of evidence we have gathered, development  costs can be expected to increase substantially from the current reported  program estimate, and the time needed to complete development testing  and subsequent initial operational testing will likely need to be extended,  delaying the full-rate production decision now planned for October 2013.
subsections: section_title: Plan to Address Management Reserve Depletion Adds Risk to the JSF Development Effort
paragraphs: The Office of the Secretary of Defense (OSD) approved the Mid-Course  Risk Reduction plan in September 2007. The plan reduces development  test aircraft and test flights, and accelerates the reduction of the  contractor’s development workforce in order to restore management  reserves to the level considered prudent to complete the development  contract as planned and within the current cost estimate. The test  community and others within DOD believe the plan puts the development  flight program at considerable risk and trades known cost risk today for  unknown cost and schedule risk in the future.
Management reserves are budgeted funds set aside for unanticipated  development challenges and increase a program’s capacity to deal with  unknowns. At development start, JSF budgeted reserves at 10 percent of  contract value and expected to draw on them at about the same rate as  contract execution. However, the program has had to use these funds  much faster than expected to pay for persistent development cost  increases and schedule delays. A combination of factors contributed to  this problem, such as late release of engineering drawings, production  taking longer than planned, and late delivery of parts from suppliers. In  turn, these contributed to continuing cost and schedule impacts in the  manufacture of development test aircraft, including extensive and  inefficient out-of-station work and delays in proving out the production  schedule. Figure 2 shows how management reserves totaling almost $1.4  billion have been depleted since the 2004 Replan.
By mid-2007, the development program had completed one-half of the  amount of work scheduled, but had expended two-thirds of the budget.  Management reserves had shrunk to about $400 million, less than one-half  the amount officials believed necessary to complete the final 6 years of  development. At the same time, the program faced significant  manufacturing and software integration challenges, costly flight testing,  and $950 million in other known cost risks. This presented the program  with a likely untenable contract overrun sometime in 2008 if no action was  taken. JSF program management identified a continuing persistent cost  variance of $250 million to $300 million in the aircraft development  contract and the associated shortfall in reserves that required near-term  action beyond “belt tightening.”
An overarching integrated product team considered several alternative  actions, including doing nothing and adding funds from procurement, but  the team chair concluded that replenishment of the management reserve  was essential to position the JSF program to successfully address its  anticipated future development challenges. This option, dubbed the Mid- Course Risk Reduction Plan, removed two development aircraft (one  CTOL and one CV), eliminated approximately 850 test flights from the  current test plan, revised the verification strategy, increased the use of  ground test labs and the flying test bed, and maximized the number of test  points to be accomplished during test flights. The plan also accelerated  reductions in contractor staff and took other actions. In total, these  planned actions are expected to add between $470 million and $650  million into the reserve to recapitalize it to about $1 billion, an amount  officials believe will be needed to complete development. Officials intend  to use reserves to recover cost and schedule losses in manufacturing and  to cover additional future needs.
This plan was subsequently approved by OSD, although serious risks were  acknowledged and the team was divided on whether the added risks  outweighed the intended benefit. Those in favor of the plan believed that  actions were urgently needed to fix the funding imbalance and avoid a  contract overrun. In this view, the plan would serve as a stopgap measure  to delay another program restructure until more program knowledge and a  clearer understanding of future cost requirements were gained.
Officials from several defense offices thought the risks to testing were too  great and that the plan did not address the underlying design and  manufacturing problems. The Director, Operational Test and Evaluation,  identified specific risks associated with the revised test verification  strategy and recommended against deleting the aircraft, citing inadequate  capacity to handle the pace of mission testing, and for ship suitability,  signature testing, and suitability evaluations. This increased the likelihood  of not finding and resolving critical design deficiencies until operational  testing, when it is more costly and disruptive to do so. OSD’s Systems and  Software Engineering office concurred, expressed concerns that the plan  did not treat the root causes of ongoing production problems, and doubted  that the contractor schedule was achievable.  The Cost Analysis  Improvement Group and others agreed that there was too much risk in  reducing test assets at this time since no production representative variant  had started flight tests and no analysis of the management reserve  depletion had been completed. In summary, the plan trades known cost  risk today for unknown cost and schedule risk in the future.
section_title: Manufacturing Inefficiencies Continue to Increase Costs and Delay the Production of Development Aircraft
paragraphs: According to our analysis of available evidence, manufacturing test  aircraft continue to run behind schedule. The prime contractor has revised  the test aircraft manufacturing schedule three times, resulting in slips of  up to 16 months in first flight dates of test aircraft. To date, about 3  months of progress has been made for every 4 months of effort. As  officials for now have decided not to extend the development period and  delay operational tests and full-rate production, this inefficiency increases  risk and further compresses time and assets available to complete test  activities.
Repercussions from the late release of engineering drawings, design  changes, and parts shortages continue to cause delays and force inefficient  production line workarounds where unfinished work is completed out of  station. Production data provided by the Defense Contract Management  Agency (DCMA) show continuing critical part shortages, high change  traffic, out-of-station work, quality issues, and planning rework. These  conditions have also delayed efforts to mature and demonstrate the  production process even as work begins on the first production lot. The  contractor has not yet proven it can efficiently build the JSF, and test  aircraft are being built differently from the process expected for the  production aircraft.
The first test aircraft, a non-production-representative conventional  landing prototype completed in 2006, required 65,000 more labor hours  (about 35 percent more) to build than planned. It encountered most of its  inefficiencies in the wing and final assembly phases. The second test  aircraft, a STOVL model, left the production line in December 2007, and its  first flight is expected in May 2008, 8 months later than originally  scheduled. It cost about 25-30 percent more to build than planned.  Contractor data show that the wings were only three-fifths complete when  moved to final assembly. As a result, over 25,000 more labor hours had to  be performed out of station to complete the wing assembly for this  aircraft.
Table 2 shows work performance on the first seven test aircraft to enter  manufacturing. (This does not include the original prototype completed in  December 2006.) These data show that nearly all aircraft are persistently  behind schedule in completing work on these three critical components at  the Fort Worth, Texas, facility. In terms of cost, the data show overall  good performance in constructing the forward fuselage, but poor results  for the wing and final assembly.
Because of production inefficiencies and delays, the contractor has had to  lengthen the manufacturing schedule three times to provide more time to  complete work.  Production line problems have resulted in slips of  between 11 and 16 months to first flight dates for each variant.  At the time  of our review, a fourth schedule was being prepared that would add  another 1 to 4 months to schedules. Officials are reporting some  improvements in parts shortages, assembly, and product quality, but  expect the cascading effects from the design delays and manufacturing  inefficiencies to linger for another couple of years.
subsections: section_title: Development Flight Test Efforts Are Beginning with Fewer Assets and Revised Verification Strategy
paragraphs: The flight test program has just begun, with only about 25 flights  completed as of January 2008. The program had originally planned to  conduct development flight tests using 15 aircraft. The recent decision to  reduce test aircraft to 13 (including the prototype), cut back the number of  flights, and change how some capabilities are tested will stress resources,  compress time to complete testing, and increase the number of  development test efforts that will overlap the planned start of operational  testing in October 2012. Test officials are concerned that capacity will be  too constrained to meet schedules and to adequately test and demonstrate  aircraft in time to support operational testing and the full-rate production  decision in October 2013. The full extent of changes and impacts from a  revised test verification strategy are still evolving. Program officials  reported that if test assets become too constrained, production aircraft  may eventually be used to complete development testing.
The number of development flight tests had already been reduced twice  before the Mid-Course Risk Reduction plan, as shown in figure 3. Test  flights have now been reduced by more than 1,800 flights (26 percent) over  the last 2 years.
Other test issues and events included the following:    Flight tests started with the initial development test aircraft, which is  not considered to be a production-representative aircraft. According to  program officials, initial flights of this aircraft yielded very useful  information on flight characteristics. However, three incidents—an  electrical flight control actuator malfunction in-flight and an engine  blade failure during a ground test—delayed further testing from May to  December 2007. Another blade failure occurred in February 2008.
Initial flights of the Cooperative Airborne Test Bed aircraft in 2007  verified its airworthiness, and it was then modified to integrate some  JSF systems hardware and software. In December 2007, it began some  limited mission flight tests, but is not yet fully configured. The Mid- Course Risk Reduction plan revised the development test verification  strategy to increase reliance on this specially configured aircraft to test  capabilities that were going to be demonstrated on JSF aircraft.
An operational assessment by testers from the Navy, Air Force, and the  United Kingdom’s Royal Air Force was accomplished from March 2004  to December 2005 to assess development progress and current JSF  mission capability. The February 2006 report concluded that the  baseline flight test schedule provided little capability to deal with  unforeseen problems and still meet the scheduled start of operational  test and evaluation in October 2012. Testing officials said the JSF flight  test program was following the historical pattern of legacy programs in  making overoptimistic plans and using assumptions not supported by  historical data. In legacy aircraft, these practices resulted in capacity  constraints, program slips, and reduced testing tasks. We note that  these concerns about the JSF were expressed at a time when the test  program was expected to have the full complement of 15 test aircraft,  not the 13 now planned.
section_title: Development Challenges Have Been Exacerbated by Inattention to Best Acquisition Practices
paragraphs: A program as complex and technically challenging as the JSF would be  expected to have some setbacks, but we believe that the cause of many  cost and schedule problems can be traced to an acquisition strategy and  decisions at key junctures that did not adequately follow the best practices  we have documented in successful commercial and government  programs. The JSF started system development before requisite  technologies were ready, started manufacturing test aircraft before  designs were stable, and moved to production before flight tests have  adequately demonstrated that the aircraft design meets performance and  operational suitability requirements. We previously reported that the JSF  acquisition strategy incorporated excessive overlap in development and  production, posing substantial risks for cost overruns, schedule slips, and  late delivery of promised capabilities to the warfighter.
Six years after system development start, only two of the JSF’s eight  critical technologies are mature by best practice standards, three are  approaching maturity, and three are immature. Maturing critical  technologies during system development led to cost growth. For example,  development costs for the electric-hydraulic actuation and power thermal  management systems have increased by 195 and 93 percent respectively  since 2003.
All three variants fell significantly short of meeting the best practices  standard of 90 percent of drawings released at the times of their respective  critical design reviews: 46 percent for the STOVL, 43 percent for the CV,  and 3 percent for the CTOL. Design delays and changes to designs were  cited by the Mid-Course Risk Reduction team as the precipitating cause  leading to the depletion of management reserves. The late release of  drawings resulted in a cascading of problems in establishing suppliers and  manufacturing process, which led to late parts deliveries, delayed the  program schedule, and forced inefficient manufacturing processes to  compensate for the delay.
Also, the program began initial low-rate production in 2007 before  delivering an aircraft that fully represents the expected design. Efforts to  mature production are constrained because the designs are not fully  proven and tested, and manufacturing processes are not demonstrated. A  fully integrated, capable production aircraft is not expected to enter flight  testing until fiscal year 2012, increasing risks that problems found may  require redesign, production line changes, and retrofit expenses for  aircraft already built.
section_title: Program Costs Expected to Increase and Schedule Worsen
paragraphs: On the basis of the evidence, we expect JSF program costs to increase and  the schedule worsen to the point where the development period will likely  need to be extended and Initial Operational Test and Evaluation (IOT&E)  and full-rate production delayed. A major program restructure seems  inevitable, unless significant elements of the program can be safely  eliminated or deferred. The Mid-Course Risk Reduction plan does not  directly address design and manufacturing inefficiencies that created the  problem in the first place. If the root causes are not identified and fixed,  the rapid depletion of management reserves can be expected to continue,  and more funding will be needed to complete development.
There is no reason to believe that these problems can be easily and quickly  fixed. While there have been some assembly line improvements, program  officials expect the manufacturing problems to persist for about 2 more years.  Officials hope this plan will give them a period of time to better and more fully  assess all the issues and reevaluate development costs and schedule  requirements. They are depending on the revised test verification plans to  maintain the pace and efficacy of development testing, but the test community  is dubious. What seem more likely are additional costs and time to overcome  inadequate capacity and the elimination or deferral of more test activities.  Eliminating development test activities and deferring additional tasks to be  completed during operational testing increase the likelihood that design and  performance problems will not be identified and resolved until late in the  program, when it is more costly and disruptive and could delay the delivery of  capabilities to the warfighter.
There are also abundant other indicators that acquisition costs will  substantially increase from what is now being reported to Congress.  Specifically:    DOD has identified billions of dollars in unfunded requirements that  are not in the program office estimate, including additional tooling and  procurement price hikes.
A new manufacturing schedule in the works indicates continued  degradation in the schedule and further extends times for first flights.
Both the aircraft and engine development contracts have persistent,  substantial cost variances that cost analysts believe are too large and  too late in the program to resolve without adding to budget.
The prime contractor and program office are readying a new estimate  at completion, which is expected to be much larger than what is now  budgeted.
Three defense organizations independent of the JSF program office  have all concluded that the program office’s cost estimate is  significantly understated and the current schedule unlikely to be  achieved.
For these and other reasons, we believe that the current JSF cost and  schedule reported to Congress are not reliable for decision making, as  discussed next.
section_title: JSF Program Cost Estimate Is Not Reliable
paragraphs: The $299.8 billion acquisition cost estimate for the JSF program is not  reliable because it is not sufficiently comprehensive, accurate,  documented, or credible. GAO’s Cost Assessment Guide outlines best  practices used throughout the federal government and industry for  producing reliable and valid cost estimates. We assessed the cost- estimating methodologies used by the JSF program office against these  best practices and determined that certain key costs were excluded,  assumptions used were overly optimistic, documentation was inadequate,  and no analysis had been done to state the confidence and certainty the  program office had in its cost estimate. As a result of these weaknesses,  the JSF program acquisition cost estimate is not reliable for decision  making. Appendix II contains a more detailed discussion of the specific  shortcomings we and the other DOD organizations have found in the  program office cost-estimating methodologies and their potential impacts.
subsections: section_title: The JSF Cost Estimate Is Not Comprehensive
paragraphs: Estimates are comprehensive when they contain a level of detail that  ensures that all pertinent costs are included and no costs are double- counted. It is important to ensure the completeness, consistency, and  realism of the information contained in the cost estimate. Our review of  the JSF development cost estimate showed that there are several cost  categories totaling more than $10 billion that are excluded or  underreported in the program office estimate. These items are summarized  in table 3 below.
The current acquisition cost estimate includes only near-term  development funding for the alternate engine program, excluding  procurement-related and other development costs of about $6.8 billion.
The military services have not firmly established basing needs for the  entire planned JSF force, but an earlier top-line estimate for military  construction was at least $2 billion. The current total cost estimate  includes only near-term budgeted costs of $533 million.
The JSF program recently increased its estimate of tooling costs by  $2.1 billion due to the inclusion of additional tooling requirements and  estimating methodology changes.
Cost and performance trade-offs during development deferred some  requirements from the current program that may later require  additional funding. The program office has not quantified these  deferrals, but Naval Air Systems Command (NAVAIR) officials told us  that the amount could be in the billions of dollars.
section_title: The JSF Cost Estimate is Not Accurate
paragraphs: Estimates are accurate when they are based on an assessment of the costs  most likely to be incurred. Therefore, when costs change, best practices  require that the estimate be updated to reflect changes in technical or  program assumptions and new phases or milestones. DOD’s Cost Analysis  Improvement Group (CAIG) found that the assumptions the JSF program  office used for weight growth, staffing head counts, commonality savings  for similar parts, and outsourced labor rate savings were overly optimistic  and not supported by historical data. For example, the program office had  used a 3 percent factor for weight growth whereas the CAIG used a 6  percent factor more in line with historical data from other programs. With  three variants, a joint program with international participation, three  different engines (cruise, second engine, and lift) in development, and  more than double the amount of operational flight software lines of code  than the F-22A and more than four times that of the F/A-18E/F, the JSF  program is substantially more complex than the F-22 or F/A-18E/F, and  therefore may not merit assumptions that are even as optimistic as the  historical data for those programs.
The program cost estimate is also considered inaccurate because it relies  on data and reports found to be deficient. JSF program office used  Lockheed Martin earned value management (EVM) data in estimating  development costs. However, DCMA determined that the data as being of  very poor quality and issued a report in November 2007 stating that it is  deficient to the point where the government is not obtaining useful  program performance data to manage risks. Among other problem areas,  DCMA found that the contractor was using management reserve funds to  alter its own and subcontractor performance levels and cost overruns.  DCMA officials who conducted the review told us that the poor quality of  the data invalidated key performance metrics regarding cost and schedule,  as well as the contractor’s estimate of the cost to complete the contract. At  the time of our review, corrective actions and plans were in process.
section_title: The JSF Cost Estimate Is Not Well Documented
paragraphs: Cost estimates are well documented when they can be easily repeated or  updated and can be traced to original sources through auditing. Rigorous  documentation increases the credibility of an estimate and helps support  an organization’s decision-making process. The documentation should  explicitly identify the primary methods, calculations, results, rationales,  assumptions, and sources of the data used to generate each cost element.  All the steps involved in developing the estimate should be documented so  that a cost analyst unfamiliar with the program can recreate the estimate  with the same result.
We found that the JSF cost model is highly complex and the level of  documentation is not sufficient for someone unfamiliar with the program  to easily recreate it. Specifically, we found that the program office does  not have formal documentation for the development, production, and  operation and support cost models and could not provide detailed  documentation such as quantitative analysis to support its assumptions.  For the development cost estimate, the JSF program officials said they did  not have a cost model that was continually updated with actual costs.
Instead the program office relies heavily on earned value management  data and contractor analysis to update its development cost estimate.
section_title: The JSF Cost Estimate Is Not Credible
paragraphs: Estimates are credible when they have been cross-checked with an  independent cost estimate and when a level of uncertainty associated with  the estimate has been identified. An independent cost estimate provides  the estimator with an unbiased test of the reasonableness of the estimate  and reduces the cost risk associated with the project by demonstrating  that alternative methods generate similar results.
Several independent organizations have reviewed the JSF program and are  predicting much higher costs than the program office. Table 4 below  provides a summary of these assessments.
CAIG estimates were prepared using different and more realistic  assumptions and schedule projections than the program office estimate.  NAVAIR, which provides resources to the JSF program office cost-  estimating function, derived much higher cost estimates and a longer  development period based on historical cost performance and removing  what it considered to be artificial and unachievable schedule constraints.  Officials were also concerned about the amount and future impact of  requirements potentially traded or pushed off into the procurement phase,  which could be even more costly. DCMA projected higher development  costs for the aircraft contract based on adjusted cost and schedule  performance to date and assuming additional slips. Officials continue to  examine the contractor’s deficient earned value management system and  its misreporting of cost and schedule data.
section_title: DOD Intends to Wait to Make a New and Independent Cost Estimate
paragraphs: The JSF program has not conducted a fully documented independent cost  estimate since system development start in 2001. Despite reliability  concerns and all the significant events and changes in cost, schedule, and  quantity since then—those reported by the program office as well as those  identified by other defense organizations and us—DOD does not intend to  accomplish another one until required to support the full-rate production  decision in 2013. If so, this will mean that the program—DOD’s largest  acquisition and vitally important to our allies—will have a 12-year gap  between official validated cost estimates. The program may complete  development and be 6 years into production before an accurate, up-to- date, and reliable official cost estimate is done.
Despite widely held views that costs will likely be higher and the schedule  longer than reported, the JSF program continues to be funded to the level  of the program office estimate. DOD acquisition policy requires fully  documented total program life-cycle cost estimates, with validation by the  CAIG, at certain major decision points and when mandated by the  milestone decision authority. DOD officials decided not to do such an  estimate at the start of low-rate initial production in 2007, which typically  coincides with a major milestone.
section_title: Future Challenges as Program Moves Forward
paragraphs: The JSF is entering its most challenging phase as it finalizes three designs,  matures manufacturing processes, conducts flight tests, and ramps up  production. The first and foremost challenge is maintaining affordability in  three dimensions—reasonable procurement prices, stable annual funding,  and economical life-cycle operating and support costs. If affordability is  not maintained during the acquisition program, quantities bought by the  United States and allies may either decrease or else consume more of the  available defense budgets. Over the life cycle of a system, higher costs for  maintaining readiness and maintainability drive up annual operating  expenses and may limit funds for new investments. Other program  challenges could affect future quantities and the mix of aircraft procured  by the United States and our allies.
subsections: section_title: Affordability Concerns Have Major Repercussions
paragraphs: From its outset, the JSF goal has been to develop and field an affordable,  highly common family of strike aircraft.  Rising unit procurement prices,  and somewhat lower commonality than expected, raise concerns that the  United States and its allies may not be able to buy as many aircraft as  currently planned. Average unit procurement costs are up 27 percent since  the 2004 Replan and 51 percent since the start of system development (see  table 1).  Rising prices erode buying power, likely resulting in reduced  quantities and delays in delivering promised capabilities to the warfighter.
The program also places an unprecedented demand for funding on the  defense budget—an annual average of about $11 billion for the next two  decades--with attendant funding risk should political, economic or military  conditions change. The JSF will have to annually compete with other  defense and nondefense priorities for the shrinking discretionary federal  dollar.  To complete the acquisition program as currently planned, JSF will  require about $269 billion from 2008 through 2034.  Annual funding  requirements for procurement increase rapidly as production ramps up to  the full-rate production decision expected in October 2013.  During the  peak years of production, JSF procurement funding requirements are  expected to average about $12.5 billion per year for the 12-year period  spanning fiscal years 2012-2023.
Figure 4 illustrates the annual funding requirements as of December 2006  and contrasts these with plans from prior years.  The December 2003 line  shows the funding profile resulting from the 2004 Replan and the 2005 line  shows the jump in funding needed to accommodate program cost  increases in the period following the Replan.  The 2006 data reflect the  impact on annual funding requirements from extending procurement 7  years.  The extension reduced annual budget amounts, but requires  continued funding through 2034 to procure deferred quantities. DOD  calculated that the extension added $11.2 billion to total procurement  cost.
A third aspect of affordability is the life-cycle cost of ownership. DOD is  recapitalizing its tactical air forces by replacing aging legacy systems with  new, more capable systems, like the JSF, that incorporates reliability and  maintainability features designed to reduce future operating costs.   Recently, DOD sharply increased its projection of JSF operating and  support costs compared to previous estimates. The December 2006 SAR  projected life-cycle operating and support costs for all three variants at  $650.3 billion, almost double the $346.7 billion amount shown in the  December 2005 SAR and similar earlier estimates.  The operating cost per  flying hour for the JSF CTOL is now estimated to be greater than current  flying hour cost for the F-16, one of the legacy aircraft to be replaced.
Officials explained that the amounts reported in 2005 and before were  early estimates based on very little data, whereas the new estimate is of  higher fidelity, informed by more information as JSF development  progresses and more knowledge is obtained.  Factors responsible for the  increased cost estimate included a revised fielding and basing plan,  changes in repair plans, revised costs for depot maintenance, increased  fuel costs, increased fuel consumption, revised estimates for manpower  and mission personnel, and a new estimate of the cost of the JSF’s  autonomic logistics system.
Overall,  the  cost  of  ownership  represents  a  very  large  and  continuing  requirement for the life of fielded aircraft. According to the new estimate,  we calculate that DOD will incur about $24 billion per year to operate and  support JSF units, assuming the quantities now planned and an 8,000-hour  service life for each JSF aircraft fielded over time.
section_title: Commonality Is Less than Expected
paragraphs: From the inception of the program, DOD has anticipated major cost  savings from developing and fielding JSF variants that share many  common components and subsystems. While a degree of commonality has  been achieved, expectations are now lower than they were at program  start.  Substantial commonality has been maintained for the mission  systems among all three variants and for the propulsion system of the  conventional and carrier variants.  However, commonality among  airframes and vehicle systems has declined overall since the start of  system development.  Figure 5 shows the decline in airframe commonality,  the most costly of the four major categories. For example, in October 2001  DOD anticipated that the CTOL airframe would be more than 60 percent  common with the other variants.  Commonality had declined to about 40  percent by December 2006.  Lesser commonality will likely increase  acquisition and future support costs.
section_title: Navy and Marine Corps Requirements and Mix Are Unsettled
paragraphs: The current JSF program shows a total quantity of 680 aircraft to be  procured by the Department of the Navy, but the allocation between the  CV and STOVL variants has not been officially established. We observe  that the Navy and Marine Corps have somewhat divergent views on the  quantities, intended employment, and basing of JSF aircraft. The Navy  wants the Marine Corps to buy some CV variants and continue to man  some of its carrier-based squadrons. The Marine Corps, however, wants to  have a future strike force composed solely of the STOVL variant and has  established a requirement for 420 aircraft. During conflicts, the Marines  plan to forward deploy JSFs to accompany and support the expeditionary  ground forces.
Navy officials told us that they have some time to make decisions because  they will be buying a mix of both CVs and STOVLs in the early years of  production and that funding requirements are not significantly affected  since unit prices for both variants are about the same. However, we  believe the continuing disagreements on basing, employment, and force  mix will have increasingly stronger impacts on JSF plans, costs, and  international partner relations. Decreased quantities of STOVLs bought by  the Department of the Navy would likely result in higher unit prices paid  by the Marine Corps and two allies buying STOVLs. Fundamental  decisions on the mix of naval aircraft also affect future operating and  support costs, military construction, and carrier requirements.
Officials also have some reservations whether they can afford the  quantities now planned at peak production rates. Navy and Marine Corps  officials told us last year that buying the JSF at the current planned rate— requiring a ramp-up to 50 CV and STOVL aircraft by fiscal year 2015—will  be difficult to achieve and to afford, particularly if costs increase and  schedules slip. Officials told us that a maximum of 35 per year was  probably affordable, given budget plans at that time.
section_title: Containing Future Weight Growth
paragraphs: Weight growth was the most significant challenge faced by the JSF  program early in development. Redesign efforts to address weight growth  was the single largest factor causing the $10 billion cost increase and 18- month extension in the development schedule since the start of system  development.
While the weight increase has been addressed for now, projections are  that the aircraft weight will continue to increase during the balance of the  development period, consistent with weight increases seen on legacy  aircraft programs. According to an OSD official with knowledge of legacy  aircraft development efforts, half of all weight growth during the  development effort can be typically expected after first flight but prior to   initial operational capability, and that additional small but persistent  weight increases can be expected during the aircraft’s service life. First  flight of a production-representative JSF has not yet occurred, and weight  is running very close to the limits as evaluated by engineering analyses and  trend extrapolation. As designs continue to mature and flight testing  intensifies, maintaining weight within limits to meet warfighter capability  requirements will be a continuing challenge and pose a major risk to  meeting cost, schedule, and performance goals.
section_title: Conclusions
paragraphs: The clear implication from performance to date and the Mid-Course Risk  Reduction plan is that additional costs and time will be needed to  complete JSF development. The plan to recapitalize management reserve  at the expense of test assets is risky with potential major impacts down  the road on costs, performance requirements, and fielding schedules. The  remaining development effort will be less robust than originally planned  and depends on a revised test verification strategy that is still evolving. As  a result, the development effort has an increased risk of not fully  measuring JSF capabilities and deficiencies prior to operational testing  and could result, in the words of one DOD official, in the future  operational test period being one of discovery rather than validation of the  aircraft’s capabilities and deficiencies. Finding and fixing deficiencies  during operational testing and after production has ramped up is costly,  disruptive, and delays getting new capabilities to the warfighter.
Because the program cost estimate is not reliable when judged against  best standards, the decision making and oversight by Congress, top  military leaders, and our allies are diminished. The picture they do have is  one where costs continue to rise and schedules slip. The situation will be  considerably worsened if the cost estimates of defense offices outside the  program are more accurate than the conservative, official in-house  estimates. Waiting 12 years between fully documented and validated total  program cost estimates is contrary to policy and good management, given  all the changes in cost, quantity, schedules, and other events that have  occurred since the 2001 estimate. The size of the JSF acquisition, its  impact on our and allied tactical air forces, and the unreliability of the  current estimate argue for an immediate new and independent cost  estimate and uncertainty analysis. This is critical information needed by  DOD management to make sound trade-off decisions against competing  demands and by Congress to perform oversight and hold DOD  accountable.
Program problems and setbacks must be put into perspective: The JSF is  DOD’s largest and most complex aircraft acquisition and an integral  component of the future force. Problems happen in such an environment.  Progress has been made and some significant challenges overcome, but  more await as program moves into flight testing and low-rate production.  Maintaining affordability so the United States military and our allies can  buy, field, and support the numbers needed by the warfighter remains the  overarching challenge.
section_title: Recommendations for Executive Action
paragraphs: Because of the elevated risks and the valid objections raised by the test  community and other DOD offices, we recommend that the Secretary of  Defense direct elements of the department to revisit and, if appropriate,  revise the Mid-Course Risk Reduction plan recently approved. This should  be supported by an intensive analysis that includes causes of management  reserve depletion, an evaluation of progress against the baseline  manufacturing schedule, and the progress made in correcting deficiencies  in the contractor’s earned value management system. It should also  include an in-depth examination of alternatives to the current plan and  address the specific concerns raised by officials regarding testing capacity,  the integration of ground and flight tests, and backup plans should  capacity become overloaded.
So that DOD may have an accurate picture of JSF cost and schedule  requirements, and that Congress may have an accurate understanding of  future funding requirements, we recommend that the Secretary of Defense  direct that  1.  The JSF program office update its cost estimate using best practices,  so that the estimate is comprehensive, accurate, well documented, and  credible. Specifically, the JSF program office should  include costs that were inappropriately omitted from the estimate;  identify performance requirements that have been traded off in  development;  fully document assumptions, data sources and methodologies in the  cost model; and    perform a risk and uncertainty analysis to focus on key cost drivers and  reduce the risk of cost overruns.  2.  The program conduct a full Schedule Risk Analysis to ensure that its  schedules are fully understood, manageable, and executable;  3.  DOD conduct a full, independent cost estimate should be conducted  according to the highest standards of any DOD cost estimating  organization, based on a comprehensive review of program data; that  this cost estimate be reviewed by an independent third party such as  the CAIG; and that the results of these estimates be briefed to all  interested parties in DOD and Congress.
section_title: Agency Comments and Our Response
paragraphs: DOD provided us with written comments on a draft of this report.  The  comments appear in appendix III. DOD also provided several technical  comments, which we incorporated in this report.
DOD substantially agreed with our recommendation to revisit the Mid- Course Risk Reduction plan. DOD stated that the plan is a cost-effective  approach with a manageable level of risk that will be monitored and  revised if necessary.  We believe the plan’s reduction of test resources will  hamper development testing and that the Department will eventually have  to make programmatic adjustments, adding cost and time.
DOD also substantially agreed with our three recommendations on cost  estimating.  DOD indicated that it will implement all elements except the  risk and uncertainty analysis, which is unwarranted in its view. We believe  that risk and uncertainty analysis is an important tool that establishes a  confidence interval for a range of possible costs—as opposed to a single- point estimate—and facilitates  good management decisions and oversight.   Such analysis is a best practice in our Cost Assessment Guide and we note  that OSD’s Cost Analysis Improvement Group supports and uses this cost- estimating tool.
We are sending copies of this report to the Secretary of Defense; the  Secretaries of the Air Force, Army, and Navy; and the Director of the  Office of Management and Budget. We will also provide copies to others  on request. In addition, the report will be made available at no charge on  the GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Staff members making key contributions to this report are  listed in appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the Joint Strike Fighter (JSF) program’s progress in meeting  cost, schedule, and performance goals, we received briefings by program  and contractor officials and reviewed financial management reports,  budget documents, annual Selected Acquisition Reports, monthly status  reports, performance indicators, and other data.  We compared reported  progress with prior years’ data, identified changes in cost and schedule,  and obtained officials’ reasons for these changes.  We interviewed  Department of Defense (DOD), JSF program, and contractor officials to  obtain their views on progress, ongoing concerns and actions taken to  address them, and future plans to complete JSF development and ramp up  procurement.
To assess plans and risks in development, manufacturing, and test  activities, we examined program documents and interviewed DOD and  contractor officials about changes to the test plan and actions taken to  modify these plans to address funding and schedule challenges. This  included reviewing and interviewing program and Office of the Secretary  of Defense (OSD) officials about changes to development testing that  evolved in response to a projected shortfall in management reserves and a  goal to stay on schedule toward a full-rate production decision in October  2013. We reviewed information compiled by program officials to document  options they considered viable, changes to the test plan and test resources  that would occur under a proposed risk reduction option, and  challenges/risks to taking this course of action and possible fallback plans.  We also reviewed stakeholder views of options and the benefits and  challenges of going forward with the changes made to the development  test plan.  We collected manufacturing cost and work performance data to  assess progress against plans, determined reasons for manufacturing  delays, discussed program and contractor plans to improve, and expected  impacts on development and operational tests.
In assessing program cost estimates, we also evaluated the JSF joint  program office estimating methodologies, assumptions, and results to  determine whether the official cost estimates were comprehensive,  accurate, well documented, and credible. We used our draft guide on  estimating program schedules and costs, which is based on extensive  research of best practices. Our Cost Assessment Guide considers an  estimate to be accurate if it is not overly conservative, is based on an  assessment of the most likely costs, and is adjusted properly for inflation;  comprehensive if its level of detail ensures that all pertinent costs are  included and no costs are double-counted; well documented if the  estimate can be easily repeated or updated and can be traced to original  sources through auditing; and credible if the estimate has been cross- checked with an independent cost estimate and a level of uncertainty  associated with the estimate has been identified. We also interviewed the  JSF program office’s cost estimating team to obtain a detailed  understanding of the cost model and met with the Department of Defense  Cost Analysis Improvement Group (CAIG) to understand their  methodology, data and approach in developing their Joint Strike Fighter  independent cost estimate. We analyzed earned value management (EVM)  reports and met with the Naval Air Systems Command and the Defense  Contract Management Agency (DCMA) to discuss the EVM data and to  obtain their independent cost estimates for JSF development efforts.  To  assess the validity and reliability of prime contractors’ earned value  management systems and reports, we analyzed the EVM reports and  reviewed audit reports prepared by the DCMA.
To identify future challenges, we continued discussions with DOD and  contractor officials on forward-looking plans and areas of emphasis.  We  analyzed budget requirements from successive plans and tracked  contributing factors to changes in budget.  We collected information on  commonality assessments among the three variants and trends.  With Navy  and Marine Corps officials, we discussed future plans on the employment  and quantity mix of aircraft and identified differences in plans and  perspectives.  We discussed past and present weight growth issues with  engineers and plans for controlling future growth.
In performing our work, we obtained information and interviewed officials  from the JSF Joint Program Office, Arlington, Virginia; Aeronautical  Systems Center, Wright-Patterson Air Force Base, Ohio; Naval Air Systems  Command, Patuxent River, Maryland; Defense Contract Management  Agency, Fort Worth, Texas; and Lockheed Martin Aeronautics, Fort Worth,  Texas. We also met and obtained data from the following OSD offices in  Washington, D.C.: Director, Operational Test and Evaluation; Program  Analysis and Evaluation; Cost Analysis Improvement Group; Portfolio  Systems Acquisition (Air Warfare); and Systems and Software Engineering.  We conducted this performance audit from June 2007 to February 2008 in  accordance with generally accepted government auditing standards. Those  standards require that we plan and perform the audit to obtain sufficient,  appropriate evidence to provide a reasonable basis for our findings and  conclusions based on our audit objectives. We believe that the evidence  obtained provides a reasonable basis for our findings and conclusions  based on our audit objectives.
section_title: Appendix II: GAO Assessment of JSF Program Cost Estimate
subsections: section_title: The Current JSF Program Office Acquisition Cost Estimate Is Not Reliable
paragraphs: The $299.8 billion acquisition cost estimate for JSF is not reliable because it is  not sufficiently comprehensive, accurate, documented, or credible. Cost- estimating organizations throughout the federal government and industry use  certain key practices to produce sound cost estimates that are comprehensive  and accurate and that can be easily traced, replicated, and updated.  GAO’s Cost  Assessment Guide outlines practices that, if followed correctly, should result in  high-quality, reliable, and valid cost estimates that management can use for  making informed decisions.  We assessed the methodology used by the JSF  program office to determine its development cost estimate against four best  practices characteristics, which are that an estimate should be comprehensive,  accurate, well documented, and credible.  We found that the JSF program  office has not followed best practices for developing a reliable and valid life  cycle cost estimate because it did not include certain key costs, assumptions  used to develop the estimate are overly optimistic, the estimate is not well  documented, and no analysis has been done to state the confidence and  certainty it has in its estimate.  As a result of these weaknesses, the JSF  program acquisition cost estimate is not reliable for decision making.
subsections: section_title: The JSF Cost Estimate Is Not Comprehensive
paragraphs: Estimates are comprehensive when they contain a level of detail that ensures  that all pertinent costs are included and no costs are double-counted.  It is  important to ensure the completeness, consistency, and realism of the  information contained in the cost estimate.  Our review of the JSF development  cost estimate showed that there are several cost categories totaling more than  $10 billion that are excluded or underreported in the program office estimate.   These items are summarized in table 5.
Alternate engine program.  Congress has been interested in DOD  developing a second source for the JSF engine to induce competition  and to reduce operational risks in the future should the sole engine  develop problems requiring the grounding of all JSFs.  DOD has not  wanted to pursue this second engine source and twice removed  funding from the JSF program line.  In 2005, DOD deleted a total of  about $7.2 billion from the JSF’s development and procurement  accounts for the alternate engine.  In 2006, it reinserted $340 million for  the program, reflecting only development funding in the future years  defense program.  This omits about $6.8 billion left out of the JSF cost  estimate for this program.
Military construction.  In prior years, the JSF cost estimate included  $2 billion for military construction costs.  Since the services had not yet  fully established basing plans, this amount was a top-level parametric  estimate not based on discrete estimates for specific sites.  The current  December 2006 cost estimate reported military construction costs of  $533 million, reflecting only the amount budgeted in the fiscal year  2008 future years defense program.  This means that about $1.5 billion  in military construction—and possibly more—will eventually be  required for specific basing needs of the JSF fleets. DOD will update  military construction estimates as the services identify specific site  requirements.
Tooling.  The JSF program recently increased its estimate of tooling  costs due to the inclusion of additional tooling requirements and  estimating methodology changes. This change is ongoing, and has not  yet been included in official program estimates.  According to a recent  press report, a Lockheed Martin official stated that the full requirement  to support procurement by our allies was not adequately factored into  prior tooling estimates. The program estimates the additional cost  through 2015 at about $2.1 billion.
Deferred capabilities.  Cost and performance trade-offs during  development have resulted in some requirements being deleted from  the program cost estimate and deferred until later years.  This includes  a number of planned capabilities dropped from the final block of  development software. The program office has not quantified the cost  of these deferred capabilities, and the costs are not reflected anywhere  in the program office’s life-cycle cost estimate. Naval Air Systems  Command (NAVAIR) officials told us that the total deferred amount  could be in the billions of dollars.  We note that prior acquisitions such  as the Global Hawk and F-22A programs also deferred requirements  that would later need additional funding.  For example, we reported in  2005 that the Global Hawk program costs did not include $400.6 million  in known additional procurement costs for sensors, ground station  enhancements, and other items required to achieve the system’s initial  full-up capability.  These costs had been in the program baseline but  were later deferred and reclassified because of cost pressures and  schedule changes. Similarly, the Air Force’s $5.9 billion modernization  and reliability improvement program includes capabilities deferred  from the acquisition program and reliability enhancements needed to  correct deficiencies and achieve the level of reliability that was  supposed to be accomplished during acquisition.
section_title: The JSF Cost Estimate Is Not Accurate
paragraphs: Estimates are accurate when they are based on an assessment of the costs  most likely to be incurred.  Therefore, when costs change, best practices  require that the estimate be updated to reflect changes in technical or  program assumptions and new phases or milestones. DOD’s Cost Analysis  Improvement Group found that the assumptions the JSF program office  used for weight growth, staffing head counts, commonality savings for  cousin (similar) parts, and outsourced labor rate savings could be too  optimistic, given the program’s complexity.  With three variants and three  engines (cruise, alternate, and lift) in development, multiple customers,  and more than double the amount of operational flight software than the  F-22A and four times that of the F/A-18E/F, the JSF acquisition program is  substantially more complex than those contemporary systems, and  therefore may not merit assumptions that are even as optimistic as the  historical data for those programs. The following table shows some major  differences in assumptions used by the program office and the CAIG in  estimating JSF costs.
JSF program officials told us that they use Lockheed Martin earned value  management data in creating their estimate of JSF development costs.  However, DCMA, which reviews contracts and industrial performance for  DOD, identified this data as being of very poor quality, calling into  question the accuracy of any estimate based on these data. In November  2007, DCMA issued a report saying that Lockheed Martin’s tracking of cost  and schedule information at its aerospace unit in Fort Worth, Texas— where the JSF program is managed—is deficient to the point where the  government is not obtaining useful program performance data to manage  risks.
DCMA said that Lockheed’s earned value data at the Fort Worth facility  are not sufficient to manage complex, multibillion-dollar weapon systems  acquisition programs. Among other problem areas, DCMA found that  Lockheed had not clearly defined roles and responsibilities, and was using  management reserve funds to alter its own and subcontractor  performance levels and cost overruns. These issues hurt DOD’s ability to  use the Lockheed data to determine product delivery dates and develop  accurate estimates of program costs. DCMA officials who conducted the  review at Lockheed Martin told us that the poor quality of the data  invalidated key performance metrics regarding cost and schedule, as well  as the contractor’s estimate of the cost to complete the contract. NAVAIR  had also raised concerns about Lockheed Martin’s earned value system as  early as June 2005, and these officials told us they were in agreement with  the findings in the November 2007 DCMA report. NAVAIR officials also  said that most deficiencies identified by the DCMA report have the effect  of underreporting costs, and that the official program cost estimates will  increase if the deficiencies are corrected.
Also in 2007, the prime contractor alerted DOD to a billing error involving  duplicate charges for the portion of the earned award fee paid to  subcontractors. This resulted in $266 million in overcharges. Government  officials became concerned that such a large discrepancy could occur  without the government’s knowledge and questioned the adequacy of the  contractor’s billing system and accounting procedures. DCMA and the  Defense Contract Audit Agency were tasked to conduct an investigation.   Their investigation found that the overbilling resulted from an accounting  system error in the internal handling of award fees on the JSF contract.  According to the investigation report, the error that created the overbilling  has been corrected, and the government has recouped the overbilled  principal and interest.
section_title: The JSF Cost Estimate Is Not Well Documented
paragraphs: Cost estimates are well documented when they can be easily repeated or  updated and can be traced to original sources through auditing.  Rigorous  documentation increases the credibility of an estimate and helps support  an organization’s decision-making process.  The documentation should  explicitly identify the primary methods, calculations, results, rationales or  assumptions, and sources of the data used to generate each cost element.   All the steps involved in developing the estimate should be documented so  that a cost analyst unfamiliar with the program can recreate the estimate  with the same result.
We found that the JSF cost model is highly complex and the level of  documentation is not sufficient for someone unfamiliar with the program  to easily recreate it. Specifically, we found that the program office does  not have formal documentation for the development, production, and  operating support cost models. Instead, it relies on briefing slides that  describe the methodology and the data sources used, but did not provide  detailed documentation such as quantitative analysis to support the  assumptions that were involved in producing the life-cycle cost estimate.   For the development cost estimate, the JSF program office admitted it did  not have a cost model that continually updates with actual costs.  Instead  the program office relies heavily on earned value management data and  analysis from Lockheed Martin to update its development cost estimate,  but provided us no documentation to back up this claim.
section_title: The JSF Cost Estimate Is Not Credible
paragraphs: Estimates are credible when they have been cross-checked with an  independent cost estimate and when a level of uncertainty associated with  the estimate has been identified.  An independent cost estimate provides  the estimator with an unbiased test of the reasonableness of the estimate  and reduces the cost risk associated with the project by demonstrating  that alternative methods generate similar results.
Several independent organizations have reviewed the JSF program and are  predicting much higher costs than the program office.  Table 7 below  provides a summary of these assessments.
In 2005, the CAIG performed an independent estimate of JSF program  development costs, which include the cost of the Lockheed Martin  contract and fees as well as the government’s in-house costs. The CAIG  expected that the development phase would cost $5.1 billion more than  expected by the program office, measured against the program office’s  most recent data available at that time, from the 2004 Selected Acquisition  Report (SAR). The CAIG official in charge of the estimate told us that  while it has not formally presented an updated estimate to the program  office, the order of magnitude of difference between the CAIG and  program office estimates remains roughly the same as at the time of the  2005 independent estimate.
The variance between the CAIG and program office estimates grew  significantly as the program encountered problems. The CAIG explained  that some of the $15.2 billion growth in its development cost estimate from  Milestone B in 2001 to the December 2004 SAR was due to initial  assumptions that 5,000 engineers would be available to work on the three  JSF variants.  This assumption turned out to be too optimistic since only  about 3,000 engineers have been working on the program.  Because of  fewer people available to support the JSF design and development, the  CAIG shifted the program schedule to the right, increasing the costs.  The  program office, on the other hand, assumed it could get the same effort  done with fewer people.  In addition, the CAIG used historical data from  the F-22A program, including the costs to design the aircraft, test it, and  redesign any fixes, and adjusted these data to account for differences in  the JSF program, including the three variants.  The program office relies  mostly on contractor data.
When it was awarded the development contract in 2001, Lockheed Martin  agreed to develop the JSF aircraft for $16.5 billion, excluding fee.  In April  2005, the development program was rebaselined, adding more than $6  billion to reflect funds added to the program due to weight growth issues  in 2003.  This raised the JSF baseline development contract cost estimate  to $23.2 billion, excluding fee.
Despite the additional funding to cover preexisting cost and schedule  overruns, Lockheed Martin’s JSF development cost and schedule  performance has continued to decline over time.  As shown in figure 6,  cost and schedule variances continued on a downward trend despite the  April 2005 rebaseline.  As of September 2007, Lockheed Martin was  reporting cumulative cost overruns of $305.7 million and was behind  schedule to an extent valued at $251.3 million.
Key drivers of cost overruns to date have included unfunded requirements  for design changes, loss of commonality savings, critical part shortages,  high change traffic, inefficient productivity due to performing work out of  sequence, constant rework, suppliers’ performance, late release of  engineering requirements, a greater than planned effort for designs of the  short takeoff and landing and the conventional takeoff and landing  variants, and additional radar testing.  Some of this cost variance is due to  optimistic assumptions at the beginning of the program.  For example,  cost estimates assumed that only one design iteration would be needed,  whereas in reality it takes numerous design iterations before the final  designs are determined.
Despite its poor performance since the rebaseline, Lockheed Martin was  predicting only a $113 million cost overrun at contract completion.  This is  unrealistic given the persistence and size of the $305.7 million overrun  reported in September 2007, at which point the contract was 67 percent  complete.  In order to achieve a $113 million overrun at completion,  Lockheed Martin would have to not only incur no further cost variances  from now until completion of the contract, but it would also have to  significantly improve its performance.  This is unlikely given that studies  of more than 700 defense programs have shown limited opportunity for  getting a program back on track once it is more than 15 percent to 20  percent complete. The true cost to complete the contract may be  significantly greater, as DCMA has expressed its concern to DOD over  Lockheed Martin’s failure to regularly update its estimate of the costs to  complete the JSF contract, stating that Lockheed’s infrequent updates are  insufficient to provide the government with information bearing on  potential cost growth and funding needs.
Like the CAIG, both DCMA and NAVAIR believe that Lockheed Martin’s  estimate at completion is too optimistic and that the program office will  most likely require significantly more funding to complete the  development program.  NAVAIR provides resources to the JSF program  office cost-estimating function, and it estimated in 2006 that JSF  development costs could be almost $8 billion to $13 billion higher than  estimated by the program office, or else cost billions more in procurement  due to requirements pushed off from development. NAVAIR officials told  us they believe that the 2006 estimate continues to be accurate today, but  explained that since the JSF program is a joint program they do not  control JSF cost-estimating procedures, although their estimates are  briefed to JSF program management. The estimate removed what NAVAIR  views as artificial constraints on the JSF schedule and projected forward,  resulting in an estimate that the schedule would likely slip 19 to 27  months, and combined this with trends in cost performance.  NAVAIR  officials said that their confidence in the achievability of the JSF program  schedule is low, as the master schedule comprises more than 600  individual schedules, making it difficult to accurately assess the  achievability of the overall schedule.
DCMA estimates that JSF development could cost as much as $4.9 billion  more than program office estimates, accounting for poor cost and  schedule performance to date and assuming further schedule slips of up to  12 months. DCMA confirmed that a schedule risk analysis, which uses  statistical techniques to obtain a measure of confidence in completing a  program, has never been performed on the JSF program. Since historically  state-of-the-art development programs have taken longer than planned, a  schedule risk analysis should be conducted to determine the level of  uncertainty in the schedule. Despite these outside organizations’  predictions of significantly higher costs to complete the JSF contract and  the lack of realism in the contractor’s own estimate, the JSF program  office continues to use the contractor’s estimate as its own.
In addition to expected cost overruns for JSF development, the CAIG is  predicting significantly higher costs for JSF for the military services to  purchase the aircraft. Using different assumptions about weight growth,  labor rates, avionics and propulsion costs, and contractor fees, the CAIG  calculated significantly higher unit costs for the aircraft variants (see table  6 earlier in this report for comparison of CAIG and program office  assumptions affecting both development and procurement costs  estimates). Multiplying these higher unit costs by the expected  procurement quantities leads to a more than $33 billion (in constant year  2002 dollars) difference from official program office estimates for  procurement costs. The CAIG estimates were briefed in 2006 to the DOD  working group that oversees the JSF program, and top OSD officials were  aware of the discrepancy between the CAIG and JSF program office  estimates.
The program office has not conducted an uncertainty analysis on its cost  estimates despite the complexity of the program and associated risk and  uncertainty. As shown in table 10, the JSF program is significantly more  complicated than comparable aircraft development programs.
This complexity makes it all the more necessary to fully account for the  effect various risks can have on the overall cost estimate.  An uncertainty  analysis assesses the extent to which the variability of an outcome  variable is caused by uncertainty in the input parameters.  It should be  performed for every cost estimate in order to inform decision makers  about the likelihood of success.  In performing uncertainty analysis, an  organization varies the effects of multiple elements on costs, and as a  result, can express a level of confidence in the point estimate.
We found that the JSF program has not conducted an uncertainty analysis.  Such analysis would provide a range of possible values to program  management and an estimate of the likelihood of the various possibilities.  Instead, the program office only offers a single point estimate—one dollar  figure, with no associated range—and no technical analysis of the  likelihood that this estimate is credible. The lead cost estimator for the  program office acknowledged that such a single point estimate is virtually  certain to be wrong, but also stated that the analysis used to develop a  range of values is easily manipulated and therefore not valuable. It is  GAO’s view that a point estimate should be accompanied by an estimated  confidence level to quantify the uncertainty surrounding the estimate in  order for management to make good decisions. Because the JSF program  office has not conducted an uncertainty analysis, it is unable to provide  Congress with any confidence level for its point estimate of approximately  $300 billion for JSF acquisition.
section_title: Appendix III: Comments from the Department of Defense
section_title: Appendix IV: GAO Contact and Staff Acknowledgments
subsections: section_title: Staff Acknowledgments
section_title: Related GAO Products
paragraphs: Joint Strike Fighter: Impact of Recent Decisions on Program Risks.  GAO-08-569T. Washington, D.C.: March 11, 2008  Tactical Aircraft: DOD Needs a Joint and Integrated Strategy.    GAO-07-415. Washington, D.C.: April 2, 2007.
Defense Acquisitions: Assessments of Selected Major Weapon Programs.  GAO-07-406SP. Washington D.C.: March 30, 2007.
Best Practices: An Integrated Portfolio Management Approach to Weapon  System Investments Could Improve DOD’s Acquisition Outcomes.   GAO-07-388. Washington, D.C.: March 30, 2007.
Defense Acquisitions: Analysis of Costs for the Joint Strike Fighter  Engine Program. GAO-07-656T.  Washington, D.C.: March 22, 2007.
Joint Strike Fighter: Progress Made and Challenges Remain. GAO-07-360.   Washington, D.C.: March 15, 2007.
Systems Acquisition: Major Weapon Systems Continue to Experience  Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368.  Washington, D.C.: April 13, 2006.
Defense Acquisitions: Actions Needed to Get Better Results on Weapon  Systems Investments. GAO-06-585T. Washington, D.C.: April 5, 2006.
Tactical Aircraft: Recapitalization Goals Are Not Supported by  Knowledge-Based F-22A and JSF Business Cases. GAO-06-487T.  Washington, D.C.: March 16, 2006.
Joint Strike Fighter: DOD Plans to Enter Production before Testing  Demonstrates Acceptable Performance. GAO-06-356. Washington, D.C.:  March 15, 2006.
Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications  for Tactical Aircraft Modernization. GAO-05-519T. Washington, D.C.:  April 6, 2005.
Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike  Fighter Program with Different Acquisition Strategy. GAO-05-271.  Washington, D.C.: March 15, 2005.