Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance
Published: 2006-03-15T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: The Joint Strike Fighter (JSF) is DOD's most expensive aircraft program. The program represents 90 percent of the remaining planned investment for recapitalizing DOD's aging tactical aircraft fleet. GAO is required by law to review the program annually for 5 years, beginning in fiscal year 2005. This is our second report and GAO assessed the program's acquisition approach--in terms of capturing knowledge for key investment decisions--and identified an alternative to improve outcomes.
section_title: What GAO Found
paragraphs: DOD is investing heavily in procuring JSF aircraft before flight testing proves it will perform as expected. For example, the JSF program plans to produce 424 low-rate initial production aircraft, at a total estimated cost of more than $49 billion, by 2013--the same time at which the program plans to complete initial operational testing. Producing aircraft before testing demonstrates the design is mature increases the likelihood of design changes that will lead to cost growth, schedule delays, and performance problems. Because the program will lack key design and testing knowledge, DOD plans to use cost reimbursement contracts to procure early production aircraft. This type of contract places a substantially greater cost risk on DOD and the taxpayers. Confidence that investment decisions will deliver expected capability within cost and schedule goals increases as testing proves the JSF will work as expected. At the same time, the JSF program has not adopted an evolutionary approach to acquiring the aircraft--despite DOD policy that prefers such an approach. Instead, the JSF program has contracted to develop and deliver the aircraft's full capability in a single-step, 12-year development program--a daunting task given the need to incorporate the technological advances that, according to DOD, represent a quantum leap in capability. DOD's buying power has already been reduced. Since initial estimates, program acquisition unit costs have increased by 28 percent, or $23 million. Development costs have increased 84 percent, planned purchases have decreased by 535 aircraft, and the completion of development has slipped 5 years, delaying delivery of capabilities to the warfighter. With more than 90 percent of the JSF investment remaining, DOD officials have the opportunity to adopt a knowledge-based and evolutionary acquisition strategy that would maximize DOD's return on its investment. The acquisition approach used for the F-16 fighter, the Air Force's JSF predecessor, could provide a model for delivering JSF capabilities to the warfighter sooner and recapitalizing tactical aircraft forces more quickly while lowering risk. The F-16 program successfully evolved capabilities over the span of about 30 years, with an initial capability delivered to the warfighter about 4 years after development started.

section_title: Letter
paragraphs: Since the Joint Strike Fighter (JSF) program began in 1996, Congress has  appropriated nearly $25 billion for its development. It is the Department of  Defense’s (DOD) most expensive aircraft acquisition program. As the  program is currently planned, DOD estimates it will spend $257 billion to  develop and procure about 2,443 aircraft and related support equipment by  2027 and an additional $347 billion to operate and support these aircraft  once they have been fielded.
JSF program goals are to develop and field a family of stealthy, strike  fighter aircraft for the Navy, Air Force, and Marine Corps, and United  States allies. Design plans call for three variants of the aircraft, but the  program aims to provide maximum commonality to minimize life cycle  costs. JSF is a central part of DOD’s overall recapitalization strategy for its  tactical aircraft fleet, representing 90 percent of the remaining planned  investment for its major tactical aircraft programs. How DOD manages the  JSF program in the future will be critical for getting the most out of this  remaining large investment and for success in replacing the aging fleet of  tactical aircraft.
The Ronald W. Reagan National Defense Authorization Act for Fiscal Year  2005 (P.L. 108-375) requires GAO to review the JSF program annually for   5 years. In March 2005, we reported the JSF’s original business case was  unexecutable and recommended that DOD establish an executable  program consistent with best practices and DOD policy regarding  evolutionary acquisitions. We also recommended that this new business  case be accompanied by a knowledge-based acquisition approach—an  approach that ensures attainment and use of demonstrated product  knowledge before making future investments. In commenting on our  report, DOD stated that JSF’s restructured acquisition plan would  incorporate a knowledge-based, evolutionary approach consistent with  DOD policy. This is our second report. Specifically, we (1) determined  whether the JSF program acquisition strategy captures critical knowledge  in time to make production investment decisions and (2) identified an  alternative to the current acquisition strategy to improve JSF program  outcomes.
The act also requires us to certify whether we had access to sufficient  information to make informed judgments on the matters contained in our  report. We were provided sufficient information to assess the plans to  capture technology, design, and manufacturing knowledge for the three  JSF variants, the evolutionary nature of the acquisition strategy, and the  opportunities to improve outcomes. At the time of our review, the Office  of the Secretary of Defense (OSD) was still preparing its fiscal year 2007  budget request as well as conducting its independent assessment of the  program cost estimate, which is not expected to be completed until  sometime in 2006. Additionally, the Quadrennial Defense Review was  ongoing during our review, and we did not have insights into the potential  outcomes or how the results of the study might affect the JSF program. We  performed our work from June 2005 through March 2006 in accordance  with generally accepted government auditing standards. For more on our  scope and methodology, see appendix I.
subsections: section_title: Background
paragraphs: JSF is a joint, multinational acquisition program for the Air Force, Navy,  Marine Corps, and eight cooperative international partners. The program  began in November 1996 with a 5-year competition between Lockheed  Martin and Boeing to determine the most capable and affordable  preliminary aircraft design. Lockheed Martin won the competition, and the  program entered system development and demonstration in October 2001.
DOD has set major expectations for the program. The program’s objective  is to develop and deploy a technically superior and affordable fleet of  aircraft that support the warfighter in performing a wide range of missions  in a variety of theaters. The single-seat, single-engine aircraft is being  designed to be self-sufficient or part of a multisystem and multiservice  operation, and to rapidly transition between air-to-surface and air-to-air  missions while still airborne. To achieve its mission, JSF will incorporate  low observable technologies, defensive avionics, advanced onboard and  offboard sensor fusion, internal and external weapons, and advanced  prognostic maintenance capability. According to DOD, these technologies  represent a quantum leap over legacy tactical aircraft capabilities. At the  same time, the JSF aircraft design includes three variants: a conventional  takeoff and landing variant for the Air Force; an aircraft carrier-suitable  variant for the Navy; and a short takeoff and vertical landing variant for  the Marine Corps, the Air Force, and the United Kingdom. JSF is intended  to replace a substantial number of aging fighter and attack aircraft in  DOD’s current inventory (see table 1).
In recent years, DOD has revised its acquisition policy to embrace an  evolutionary, or incremental, approach to improve program outcomes. The  acquisition policy states evolutionary acquisition is the preferred DOD  strategy for rapid acquisition of mature technology for the user by  delivering capability in increments, recognizing up front the need for  future capability improvements. The objective is to balance needs with  resources in order to put capability into the hands of the warfighter more  quickly. The policy states that a product enters system development and  demonstration when an affordable increment of militarily useful capability  has been identified, technology has been demonstrated in a relevant  environment, and a system can be developed for production in a short  time frame (normally less than 5 years). Each increment of an evolutionary  acquisition will have its own decision milestones and baseline—cost,  schedule, and performance requirements. In commenting on our March  2005 JSF report, DOD stated that JSF’s restructured acquisition plan  would balance technical, cost, and schedule risk and incorporate a  knowledge-based, evolutionary approach to system acquisition consistent  with DOD policy.
section_title: JSF Acquisition Strategy Will Begin Procurement before Testing Demonstrates the Aircraft’s Performance
paragraphs: DOD’s acquisition strategy for JSF has not substantially changed as a  result of the program’s restructuring last year. In May 2005, DOD  leadership approved the program’s plan, which intends to start production  in 2007—more than 2 years before all three JSF variants have completed  some flight testing of the aircraft’s basic design, 4 years before a fully  configured and integrated aircraft is expected to be flight tested, and   6 years before development and initial operational testing are scheduled to  be completed. Before development has ended, DOD will have potentially  signed procurement contracts for 424 JSF aircraft at an estimated cost of   $49 billion. Starting production before ensuring the design is mature  through flight testing significantly increases the risk of costly design  changes that will push the program over budget and behind schedule.  Evidence of the risk associated with concurrently developing, testing, and  producing the JSF aircraft can be seen in the program office strategy to  place initial production orders on a cost reimbursement contract, placing  an unusually high risk burden on the government during the early  production phase.
subsections: section_title: Key Testing Events Will Not Be Completed before Significant Procurement Investments Are Made
paragraphs: The JSF program plans to begin low-rate initial production of the aircraft  before many of JSF’s technology advances and basic flying qualities are  flight-tested and to produce substantial quantities of aircraft before initial  operational testing is completed. According to JSF program plans, DOD’s  low-rate initial production quantities will increase from 5 aircraft a year in  2007 to 133 a year, before development and initial operational testing are  completed in 2013. This production rate will require DOD to invest  significantly in tooling, facilities, and personnel. Initial contractor cost  estimates indicate that close to $1 billion of new tooling will be needed to  support low-rate initial production rates. Early contractor data also show  that the program will need to double its manufacturing workforce by 2008  and will need six times the number of manufacturing personnel before  low-rate initial production ends. Total monthly spending for production  activities in 2013 is expected to approach $1 billion, a significant increase  from $100 million a month when production is scheduled to begin in 2007.
The cost of discovering design problems during production could be  significant if testing shows that large, structural components of the aircraft  require modifications. Design changes needed in one variant could also  ripple through the other two variants, reducing efficiencies necessary to  lower production and operational costs with common parts and  manufacturing processes for the three variants. Some industry officials  have indicated that the cost of design changes such as these could be 10 to  1,000 times greater, depending on how far the product has progressed into  production.
When the JSF program is expected to begin low-rate initial production in  2007, the program will have completed less than 1 percent of the flight  test program and none of the three JSF variants will have a production  representative prototype built and in flight testing. Features critical to  JSF’s operational success, such as the low-observable and highly common  airframe and the advanced mission systems, will not have been  demonstrated in a flight test environment. The program plans to proceed  into low-rate initial production without demonstrating that (1) the  aircraft’s flying qualities function within the parameters of the flight  envelope—that is, the set limits for altitude, speed, and angles of attack;  (2) the aircraft design is reliable; or (3) a fully integrated and capable  aircraft system can perform as intended. These are key indicators of a  program’s readiness for entering production and making significant  investments in tooling, facilities, and materials. When the first fully  integrated and capable development JSF is expected to fly, in 2011, DOD  will have committed to buy 190 aircraft at an estimated cost of $26 billion  under current plans. By the time testing is planned to be completed, in  2013, DOD will have procured more than double that amount—424 aircraft  at an estimated cost of about $49 billion. Figure 1 shows the significant  overlap in development and testing and the major investments in  production capability and aircraft that inject additional cost and delivery  risks that could delay delivery of a proven capability to the warfighter.
Under the current schedule, the JSF program plans to manufacture and  deliver 15 flight test aircraft and 7 ground test (nonflying) articles in   5 years—an aggressive schedule when compared with schedules of other  programs with fewer variables. Current JSF schedules are already showing  delivery of early test aircraft could be later than the planned delivery date.
Unplanned manufacturing and technical problems can delay the  completion of a flight test program, increase the number of flight test  hours needed to verify that the system will work as intended, and affect  when key knowledge will be available for making investment decisions.  For example, when the B-2 program began flight testing in July 1989, it  estimated that the flight test program would last approximately 4.5 years  and require about 3,600 flight test hours. When the test program ended in  1997, the flight test hours had grown to 5,000 hours, or by 40 percent, over  an 8-year period. Program officials cited several causes, including  difficulties in manufacturing test aircraft and correcting deficiencies from  unanticipated problems discovered during testing. The F-22A encountered  similar delays, increasing a planned 4-year flight test program to about 8  years and affecting the program’s ability to conduct operational testing  and move into production on schedule.
subsections: section_title: Initial Procurement Decisions Will Be Made before the Basic Aircraft Design Is Demonstrated
paragraphs: While each JSF variant is similar—all are being designed to have stealth  airframes, fly at supersonic speeds, shoot air-to-air missiles, and drop  bombs on a target—there are subtle airframe design differences to support  the services’ different operational concepts and environments. Test  officials acknowledge that each airframe variant will require flight testing  to demonstrate that each will fly as intended. Yet at the time the JSF  program expects to begin low-rate initial production, only 1 of 15 flight  test aircraft is expected to have started testing. Further, the aircraft’s test  flights are not scheduled to begin until August 2006, less than 6 months  before the planned decision to begin low-rate initial production, and will  not assess the aircraft in more demanding operational environments, such  as high angles of attack. Moreover, the first flight test aircraft will not  include a large number of design changes that resulted from an effort to  significantly reduce aircraft weight over the past 2 years and, therefore,  will not represent the planned production configuration. The first  demonstration of a production representative airframe that includes the  latest design changes is scheduled for late 2007—after production has  been initiated—with first flight of the short takeoff and vertical landing  variant.
All three variants will not be in flight testing until 2009, when the carrier  variant is expected to have its first flight. Several key test events identified  by the contractor to demonstrate the flight characteristics of these aircraft  in the intended operational environment are not scheduled until 2009 or  later. These include shipboard operations for the carrier and short takeoff  and vertical landing variants. According to the contractor, the full-flight  envelope for all three variants will not be fully demonstrated until 2011 or  2012—4 to 5 years after low-rate initial production is scheduled to begin.
According to DOD, modeling and simulation will be used to evaluate the  performance of the aircraft’s flying qualities and to support decisions to  invest in production. However, the Rand Corporation recently reported in  a study on testing and evaluation that modeling is not a substitute for flight  testing. Rand found that even in performance areas that are well  understood, it is not unusual for flight testing to uncover problems that  were not apparent in simulations. Examples include flight effects on the  wing of the F/A-18 EF and buffeting of stores externally carried on various  aircraft when flown in certain conditions. Additionally, OSD testing  officials indicated that flight testing of each variant is necessary to  demonstrate designed capabilities. Our past work has found that flying  quality problems were identified during actual flight testing on programs  like the F-22A, B-2A, and V-22.
Finally, two major ground tests of the airframe’s structural integrity— fatigue and static testing—will be in their very early stages or not have  started when production begins. Fatigue testing, which measures the  aircraft’s durability over its expected life, is slated to begin in 2008 with  testing to show structural fatigue over one lifetime of flying to be  completed in 2009 and a second lifetime completed by 2010. Fatigue  testing uses actual JSF airframe structures that are subjected to the long- term stresses expected over the aircraft’s life. Standards require this  testing to be done to represent the stresses expected over two lifetimes.  Static testing, which measures the aircraft’s ability to withstand the  stresses expected to be encountered throughout the aircraft’s flight  regime, is slated to begin in 2007 and last through 2009. These tests are  important to reduce the risk of structural problems emerging during  production or after aircraft are fielded.
section_title: Mission Capabilities Will Not Be Tested until 4 Years after Procurement Has Begun
paragraphs: In addition to late design testing, many of the mission systems planned for  JSF will not be available for initial flight testing either. Although  laboratory tests are under way, DOD does not plan to flight-test several of  the new technologies needed for the JSF to perform its intended missions  until 2009 at the earliest. Defense Operational Test and Evaluation officials  have stated that flight-testing capabilities in a production representative  test aircraft and in the operational environment planned for the new  system are important to reducing risk. This actual environment differs  from what can be demonstrated in a laboratory and has historically  identified unexpected problems. For example, the F-22A software worked  as expected in the laboratory, but when tested in the aircraft, significant  problems were identified. These problems delayed testing and the delivery  of a proven capability. The different levels of mission capability will be  tested in JSF aircraft as follows.
Block 1, an initial air interdiction capability, is scheduled to begin  testing in 2009, with initial operational testing scheduled for 2011—4  years after DOD plans to begin production.
Block 2, an improved air interdiction and close air support capability, is  scheduled to begin testing in 2010, with initial operational testing  scheduled for 2012. This block will include several critical technologies  that are not fully mature, such as the advanced missions systems and  prognostics and health maintenance, but are critical to meeting  requirements like sortie generation and mission capabilities.
Block 3, the fully integrated and capable JSF, is scheduled to begin  testing in 2011. At this time less than 50 percent of the planned mission  capability testing will have been completed. This is close to the same  point that the F-22A and other past programs experienced difficulties  integrating all the complex software and hardware components into  the aircraft. Flight testing to evaluate the effectiveness and suitability  of the fully integrated system is expected to continue until the full-rate  decision in late 2013.
section_title: Aggressive Software Development Schedule May Cause Delays
paragraphs: JSF’s expected performance is largely dependent on demonstrating  software that supports vehicle, mission system, and other capabilities. The  program plans to develop over 19 million lines of code—substantially  more than the lines of code needed for the F-22A. The software is planned  to be developed in five blocks. The first block is scheduled for completion  in 2006, and the last block is scheduled for completion in 2011. At the time  the program enters low-rate initial production, the program will have  completed less than 35 percent of the software needed for the system’s  functionality. Past programs have encountered difficulties in developing  software, delaying flight test schedules. Data provided by the program  office indicate that the program is already showing early signs of falling  behind its software delivery schedule. JSF program officials recognize the  risk associated with this large software effort.
section_title: The Current Acquisition Strategy Requires Prolonged Reliance on Cost Reimbursement Contracts That Place an Increased Risk on DOD
paragraphs: According to program officials, the uncertainties inherent in concurrently  developing, testing, and producing the JSF aircraft prevent the pricing of  initial production orders on a fixed price basis. Consequently, the program  office plans to place initial production orders on a cost reimbursement  basis. Cost reimbursement contracts provide for payment of allowable  incurred costs, to the extent prescribed in the contract. Such contracts are  generally used only when the uncertainties involved in contract  performance do not permit costs to be estimated with sufficient accuracy  to use any type of fixed price contract. Cost reimbursement contracts  place substantial risk on the buyer—in this case DOD—because the  contractor’s responsibility for performance costs is minimized or reduced.  In contrast, a fixed price contract provides for a pre-established price, and  places more risk and responsibility for costs and resulting profit or loss on  the contractor and provides more incentive for efficient and economical  performance.
The program plans to transition to fixed-price-type contracts once the air  vehicle has a mature design, has been demonstrated in flight test, and is  producible at established cost targets. According to program officials, this  transition will occur sometime before full-rate production begins in 2013.  The program office believes the combination of the early concept  development work, the block development approach, and the relatively  small numbers of aircraft in the initial production buys allow decisions to  be made earlier than normal with an acceptable level of risk.
section_title: Evolutionary Acquisition Strategy Provides a Less Risky Alternative for the JSF Program
paragraphs: The JSF acquisition strategy currently plans a single-step approach to  deliver a quantum leap in tactical fighter capability by 2013 and has  already felt the negative cost and schedule impacts from the executing this  approach. The length and scope of the remaining effort in the JSF program  make it even more difficult to accurately estimate cost and delivery  schedules. The JSF funding profile—which requires an average of   $11 billion annually for the next two decades—is also at risk to increase if  costs continue to grow or schedules are further delayed to develop the  ultimate JSF capabilities. An alternative acquisition strategy, such as used  by the F-16 program, that sequences capabilities over time based on  proven technologies and design would reduce risk and deliver aircraft  sooner. This evolutionary approach is actually the preferred approach in  DOD’s acquisition policy for acquiring new systems for more rapid  delivery of incremental capabilities to the warfighter.
subsections: section_title: JSF Program Is Designed to Deliver Full Required Capability in a Single-Step Development Program
paragraphs: Instead of establishing time-phased requirements for aircraft to be  delivered in sequence that could first meet DOD’s need to recapitalize its  aging fleet of aircraft and then evolve the aircraft to eventually achieve  improved capabilities in future system development increments, DOD  chose a single-step development approach to deliver the full required  capability by the end of system development in 2013. That approach is  now planned to last 17 years (5 years in concept development and 12 years  in system development)—including the development of immature,  undemonstrated technologies and will take at least two to three times  longer than the development time of leading commercial firms or that  suggested by defense acquisition policy. The JSF acquisition strategy is  shown in figure 2.
While JSF’s acquisition strategy calls for delivering a small number of  aircraft with limited capabilities, the program has committed to deliver the  full required capability by the end of system development and  demonstration in 2013. For JSF, this includes a set of objectives that  exceeds those of aircraft development programs of the past. JSF will use  cutting-edge technology to fuse data from the aircraft itself and other  sources, from all aspects of the aircraft, and display the information to the  pilot. The aircraft must be able to rapidly transition from ground attack to  air-to-air missions while airborne. JSF also expects advances in  technologies for mission systems, prognostics, and autonomic logistics  support requiring hardware development as well as extensive and complex  software development and integration to have lower cost of ownership  than the legacy aircraft while being able to deploy rapidly, sustain high  mission reliability, and maintain a high sortie generation rate.
Past single-step development programs have been unsuccessful in  predicting acquisition costs and delivery schedules. For example,  development costs for the F-22A fighter, B-2 bomber, Crusader artillery  vehicle, and Comanche helicopter skyrocketed, and production quantities  were either substantially reduced or the program was canceled. Such  outcomes are a poor return on investment and a failure to recapitalize  aging legacy systems. So far the JSF program is experiencing similar  results. Since the program’s start, development cost has increased 84  percent, the development schedule has increased by almost 5 years, and  planned delivery of capabilities to the warfighter has been delayed. DOD  now plans to buy 535 fewer aircraft than originally planned. As a result,  DOD’s buying power has been significantly reduced—program acquisition  unit costs have increased by 28 percent, or $23 million, since first  estimates. See appendix II for more details on JSF outcomes.
section_title: Funding Needs Could Increase Given the Level of Risk to Complete the JSF Program
paragraphs: Despite a lengthy restructuring effort that added over $7 billion and   18 months to the development program, JSF officials have stated that the  restructured program has little or no flexibility for future changes or  unanticipated risks. Furthermore, the length and scope of the remaining  effort make it even more difficult to accurately estimate cost and  completion schedules. While it has been over 9 years since the program  started, the first flight test aircraft still has not been delivered. The  program has planned almost 8 years to complete the remaining activities  of the system development and demonstration phase. These remaining  activities include  fully maturing 7 of the 8 critical technologies;    completing the designs and releasing the engineering drawings for all  three variants;    manufacturing and delivering 15 flight test aircraft and 7 ground test    developing 19 million lines of software code; and   completing a 7-year, over 12,000-hour flight test program.
The JSF program’s latest planned funding profile for development and  procurement, produced in December 2004, expects annual funding  requirements to hover close to $13 billion in between 2012 and 2022,  peaking at $13.8 billion in 2013. If the program fails to achieve its current  estimated costs, funding challenges could be even greater than they are  today. Thus, even a modest cost increase would have dramatic impacts on  funding. For example, a 10 percent increase in production costs would  amount to over $21 billion (see fig. 3).
The current cost estimate reflects the position of the JSF program office.  The Office of Secretary of Defense Cost Analysis Improvement Group was  to update its formal independent cost estimate in the spring of 2005, and  the Navy and the Air Force were expected to fully fund the program  consistent with the estimate. The group now does not expect to formally  complete its estimate until the spring of 2006, but its preliminary estimate  was substantially higher than the program office’s. According to Cost  Analysis Improvement Group officials, an assessment of the software and  mission systems requirements based on more recent information could  further increase the estimate of JSF costs.
section_title: An Incremental Development Approach Provides an Alternative Model for Reducing JSF Risks
paragraphs: An incremental development approach consistent with DOD’s policy on  evolutionary acquisition and best practices has potential to deliver  warfighter capabilities in planned product increments that would increase  JSF capabilities over time. The F-16 fighter program, the Air Force’s JSF  predecessor, successfully evolved capabilities over the span of about 30  years, delivering increases of capabilities quickly and often, as  technologies became available. That program may provide a model for a  possible alternative acquisition strategy for the JSF program. Structuring  the program into separate and manageable increments based on what is  achievable now and in the future would allow more predictable cost and  delivery estimates.
Over the past three decades, the Air Force successfully procured more  than 2,200 F-16s. The F-16 acquisition approach allowed the timely and  affordable delivery of aircraft and capability to meet the warfighter’s  needs, including the recapitalization of aging aircraft. By using an  evolutionary approach to develop the aircraft, the program was able to  quickly deliver new and improved capabilities to the warfighter and  increase the aircraft’s capability as new technologies were matured and  added to the aircraft. The first increment, developed during the 1970s,  provided a “day fighter” aircraft with basic air-to-air and air-to-ground  capabilities. This allowed the developer to deliver new and useful military  capability to the warfighter in less than 4 years. With each subsequent  increment, new technology was used to improve the engine, radar,  structure, avionics, and other systems that allow the aircraft today to  perform close air support, ground attack, air defense, and suppression of  enemy defense missions (see fig. 4). The evolutionary approach also  enriched the industrial base capabilities by extending the life of the  production over the length of this incremental approach.
In contrast, JSF’s fully configured design represents a quantum leap in  capability that far exceeds the capability of legacy systems that JSF is  intended to replace. While the program is using a block structure—where  each block adds capabilities over the proceeding block—the blocks are  part of a single development effort, and DOD is on contract with the  developer to deliver the warfighter the full capability (see fig. 5). The  program’s block structure provides for an escalating capability, but DOD  already plans to buy 95 percent of JSF aircraft with the ultimate capability  (block 3). Unlike the approach used with the F-16, this risky approach will  likely be like past programs that have encountered significant increases in  cost and time, not allowing DOD to quickly recapitalize the aging legacy  aircraft.
An evolutionary approach would entail delivering a first product  increment with at least as much capability as currently operating legacy  aircraft with sufficient quantities to allow DOD to retire older planes  sooner and reduce operating inefficiencies. DOD has repeatedly raised  concerns about the age of its fighter aircraft fleet, which was bought  largely in the 1970s and 1980s and will need to be replaced around 2010.  Delays in fielding JSF aircraft may increase costs to maintain legacy  aircraft to meet force structure requirements. Limiting development to 5- year increments or less, as suggested in the DOD acquisition policy, would  force smaller, more manageable commitments in capabilities and make  costs and schedules more predictable. Some of the more challenging JSF  capabilities, such as the advanced mission systems or prognostics  technologies, would be deferred, kept in the technology base (off the  critical path of the JSF program), and added to follow-on increments once  they are demonstrated in the more conducive technology development  environment. In addition to considering available resources—including  technology and design knowledge, budget, and time—each increment  should be based on the warfighter’s most immediate needs and the  number of aircraft needed to maintain a viable fleet. Figure 6 shows what  an incremental approach might look like for JSF.
Because an incremental approach would reduce the amount of risk in each  development phase, it would make the program manager and contractor  accountable for a lower-risk system development phase with precise cost  and schedule targets. A shorter system development phase also makes it  more feasible to align a program manager’s tenure to the completion of the  phase, holding the program manager accountable for decisions made. It  also would enable the use of fixed-price-type contracts for production that  contains a pricing structure that reduces government risk.
section_title: Conclusions
paragraphs: While the JSF program plays a critical role in DOD plans to recapitalize the  services’ aging tactical aircraft fleet, DOD’s current acquisition approach  for the JSF program continues to be risky and could further jeopardize  meeting this important objective. The JSF program has already  encountered increases to estimated development costs, delays to planned  deliveries, and reductions in the planned number of JSF to be procured  that have eroded DOD’s buying power. We believe the current acquisition  strategy to begin production in 2007 is too risky. By committing to procure  large quantities of the aircraft before completing testing that demonstrates  that the design is mature and reliable, DOD has significantly increased the  risk of further compromising its return on investment—as well as adding  more cost and delaying the delivery of critical capabilities to the  warfighter. Also, making sizable investments in tooling and other  manufacturing capability needed to produce JSFs at higher rates before a  fully integrated aircraft is demonstrated in testing places the program at  risk for expensive design changes as testing uncovers problems. Deferring  production decisions until all three variants’ performance has been  demonstrated throughout their flight envelopes and with the full  integrated capability would allow additional time to capture knowledge  and help to significantly minimize these risks.
To continue with the current plan to deliver the ultimate JSF capability  will require (1) execution of a 7-year test program without further  schedule delays, (2) development of 5 major blocks of software (19 million  lines of code), and (3) new and yet undemonstrated technologies for  advanced mission systems that must collect, analyze, and synthesize  information from other platforms or sensors not all a part of the JSF  aircraft. These are necessary to ensure the JSF has increased situational  awareness to destroy enemy air defenses and perform deep strike  missions with advanced weapons. The JSF program will also need,  annually, an average of $11 billion of development and procurement  funding for the next two decades.
All of these factors add to the challenges faced by DOD in trying to  manage this highly concurrent program, increasing the risk it will have the  same poor outcomes experienced by similar major acquisition programs  with significant cost and schedule growth and delayed modernization and  recapitalization of aging systems. DOD has an alternative. With 90 percent  of DOD’s remaining planned investment in JSF, it can adopt a knowledge- driven and evolutionary acquisition approach to reduce JSF program risks,  recapitalize its aging tactical air force sooner, and deliver needed  capabilities to the warfighter more quickly. The experiences of the F-16  program—evolving and improving capabilities over time while providing  the warfighter combat capability—provide a precedent for this.  Requirements for the first increment of JSF under this evolutionary  approach would match a level supported by current knowledge of  technologies and design. This would allow the testing of those reduced  requirements to support a knowledge-based low-rate production decision  sooner and allow delivery of a useful product and in sufficient quantities  to start replacing the aging legacy fighter force. Capabilities that demand  as yet undemonstrated technologies, for example, advanced mission  systems and software, would be included as requirements in subsequent  future JSF aircraft increments—managed as separate development  programs—as technology is demonstrated in the technology base and  becomes available.
section_title: Matters for Congressional Consideration
paragraphs: DOD does not plan to change its business case or acquisition plan for  developing and buying the JSF. Without changes, the acquisition plan will  put at risk $50 billion for procuring JSF aircraft at the same time the  program develops and tests the aircraft’s expected performance  capabilities over a 7-year, 12,000 hour flight test program. The JSF  program has continually missed its cost and schedule targets over the 5  years it has been in development. If DOD were to make smaller, more  manageable commitments in capabilities, it would make cost and schedule  more predictable and deliver needed capabilities to the war fighter sooner.  For these reasons, Congress may want to consider delaying authorizations  and appropriations for procuring JSF aircraft until   1.  DOD develops a knowledge-based business case that matches  requirements with proven technologies and design knowledge and  available funding. Capabilities that demand technological advances not  yet demonstrated should be part of future increments that are funded  and managed separately once demonstrated.   2.  DOD demonstrates the aircraft design qualities and integrated mission  capabilities of the fully configured and integrated JSF variants work as  designed based on actual flight testing.
section_title: Recommendations for Executive Action
paragraphs: We are making the following recommendations to reduce program risk  and increase the likelihood of successful program outcomes by delivering  capabilities to the warfighter when needed and within available resources.  Specifically, we recommend that the Secretary of Defense:  1.  Delay the investment in production aircraft until sufficient testing has  at least demonstrated the basic airframe design of each JSF variant in  important parts of the flight envelope, including combinations of  critical altitudes, speeds, and angles of attack.  2.  Once the basic airframe design has been demonstrated, limit  production quantities to the number of aircraft needed for operational  testing and that can be produced using the current production  capability (based on existing tooling, materials, and personnel) until a  fully integrated aircraft (with advanced mission systems and predictive  maintenance systems) demonstrates through flight testing the required  capabilities.  3.  Plan an incremental acquisition strategy that follows the intent of DOD  evolutionary acquisition policy and delivers a first increment that  limits new content to proven technologies and design. The plan should  be completed and reported to the Congress by July 2006, and include  adjustments in associated programs as necessary to maintain needed  capabilities.
section_title: Agency Comments and Our Evaluation
paragraphs: DOD provided us with written comments on a draft of this report. The  comments appear in appendix III.
DOD partially concurred with our recommendation that the Secretary of  Defense delay the investment in production aircraft until sufficient testing  has at least demonstrated the basic airframe design of each JSF variant in  important parts of the flight envelope, including combinations of critical  altitudes, speeds, and angles of attack. DOD agreed that a knowledge- based approach is critical to making prudent decisions and that this type  of approach is being used by JSF. However, DOD’s response did not  provide a plan of action to show how it will ensure flight testing  demonstrates acceptable performance before significant production  investments are made. In the past, Congress has raised concerns about the  costly outcomes of highly concurrent development and production efforts  that are not “flying before buying.” DOD points out that the JSF program is  in its fifth year of a 12-year development, but starting production with  nearly 7 years of development and testing remaining leaves a high  probability that, because it is not flying before buying, it will incur costly  design changes and delay getting capability to the warfighter.  Until there  has been testing to demonstrate that all three variants will work as  intended, we believe entering production in 2007 places financial  investments at significant risk. The risk increases as spending for  production goes from about $100 million a month in 2007 to over $500  million a month just 2 years later.
DOD partially concurred with our recommendation to the Secretary of  Defense to limit production quantities to the number of aircraft needed for  operational testing and that can be produced using the current production  capability until a fully integrated aircraft demonstrates the required  capabilities through flight testing. DOD stated it is limiting production  until aircraft capabilities are fully demonstrated and that further limits are  not necessary. It stated that various program reviews to assess  performance, including test objectives, are conducted to ensure associated  program risks are understood and mitigated. We believe DOD’s plan to  invest almost $50 billion to buy over 400 aircraft concurrent with testing is  very risky. Significant efforts remain in the JSF program to demonstrate  the aircraft will perform as expected. A number of major DOD acquisition  programs have employed highly concurrent acquisition strategies in the  past and experienced significant cost increases and schedule delays. DOD  stated its use of modeling and simulation and laboratory testing reduces  risk. However, DOD operational test officials have stated that  demonstrations need to occur in the actual aircraft in an operational  environment to verify that the system works as intended. For example, the  F-22A avionics software performed successfully in the laboratory but  experienced significant problems that delayed the program and increased  costs once it entered actual flight testing in the F-22A aircraft. JSF  software and advanced electronics are more complex than the F-22A’s.  While DOD believes it can manage program risk by holding regular  program reviews, DOD’s own experience has shown this approach is not  effective. Accepting and managing risk instead of capturing technology,  design, and manufacturing knowledge, as suggested in DOD policy, has  made it difficult for DOD to make informed decisions at key points.
DOD partially concurred with our recommendation to the Secretary of  Defense to plan an incremental acquisition strategy that follows the intent  of DOD evolutionary acquisition policy and delivers a first increment that  limits new content to proven technologies and design and to report this  plan to Congress by July 2006. It stated the JSF Acquisition Strategy fully  complies with policy and is a knowledge-based, incremental approach that  includes three blocks of increasing degrees of warfighter capability. We  believe DOD’s strategy does not provide the benefits of an evolutionary  approach, as suggested by DOD’s policy and best practices. In commenting  on our report, DOD did not address the salient points concerning the true  significance of adopting an incremental acquisition approach—reducing  program risk, delivering needed capabilities to the warfighter quicker, and  recapitalizing the aging tactical aircraft fleet sooner. The JSF strategy  resembles other past major acquisition programs that have attempted to  achieve the ultimate capability in a single development increment,  producing nearly all of the fleet with the fully required capability from the  outset. DOD has allowed technology development to spill over into  product development, weakening the foundation for program cost or  schedule estimates. This has led to disastrous outcomes for other major  acquisition programs in the past. We continue to believe the successful F- 16 program can serve as a model for the JSF program.  The F-16 program  evolved capabilities over a 30-year period, buying substantial quantities of  each increment in order to recapitalize aging tactical aircraft and provide  new capabilities to the warfighter more quickly.  If DOD were to make  smaller, more manageable commitments in capabilities, it would make  cost and schedule more predictable and deliver needed capabilities to the  warfighter sooner.
We are sending copies of this report to the Secretary of Defense; the  Secretaries of the Air Force, Army, and Navy; and the Director of the  Office of Management and Budget. We will also provide copies to others  on request. In addition, the report will be available at no charge on the  GAO Web site at http://www.gao.gov.
If you or your staff have any questions concerning this report, please  contact me at (202) 512-4841. Contact points for our offices of  Congressional Relations and Public Affairs may be found on the last page  of this report. Other staff making key contributions to this report were  Michael Hazard, Assistant Director; Lily Chin; Matthew Drerup; Matthew  Lea; Gary Middleton; Karen Sloan; and Adam Vodraska.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine if the Joint Strike Fighter (JSF) program will capture critical  product knowledge before making production investments, we compared  the program’s plans and results to date against best practice standards for  applying knowledge to support major program investment decisions. The  best practice standards are based on a GAO body of work that  encompasses nearly 10 years and visits to over 25 major commercial  companies. Our work has shown that valuable lessons can be learned from  the commercial sector, as well as successful Department of Defense  (DOD) cases, and can be applied to the development of weapon systems.  We reviewed JSF management reports, acquisition plans, test plans, risk  assessments, cost reports, independent program assessments, and  program status briefings. We identified gaps in product knowledge at the  production decision points, reasons for these gaps, and the risks to  program cost and schedule associated with moving forward. We reviewed  DOD’s acquisition policy to determine whether JSF’s approach met its  framework and intent. We interviewed officials from the DOD acquisition  program management office and prime contractor to gain their  perspectives on program risks and their approaches to managing risks.
To evaluate whether the current acquisition plan follows an evolutionary  or incremental approach, a key best practice for meeting business case  goals, we examined the JSF program’s acquisition framework and the  actions taken by DOD to address the recommendations made in our 2005  report. We compared the JSF program’s approach with the approaches  used by leading companies and successful DOD programs to evolve  products to their ultimate capabilities. We also reviewed DOD guidance on  structuring evolutionary acquisition programs. To determine the JSF  program outcomes to date, we used the program estimates that justified  the program when it started in 1996. This was the point JSF transitioned  from a technology development environment to an acquisition program  environment, with the commitment to deliver a family of strike aircraft  that meets the Air Force, Navy, and Marine Corps needs. At that time, total  production, acquisition, and ownership costs had not been estimated. The  total production, acquisition, and ownership cost estimates were first  established to support the decision to enter the system development and  demonstration phase in 2001. We used these estimates as the baseline for  these costs. We also obtained information on past DOD programs from  Selected Acquisition Reports and prior work conducted by GAO over the  past 25 years.
To identify opportunities to improve JSF program outcomes, we assessed  the potential of applying an incremental development approach to the JSF  acquisition program based on the commercial model and DOD guidance.
We also examined the program history of the F-16 fighter, the JSF  predecessor, which applied an incremental approach over its 30-year  acquisition. The F-16 acquisition has been touted by DOD and others as a  successful program and a model for others to follow. To examine the  program history of the F-16, we met with F-16 program officials and  analyzed acquisition plans, management reports, and program outcomes.
In performing our work, we obtained information and interviewed officials  from the JSF Joint Program Office, Arlington, Virginia; Lockheed Martin  Aeronautical Systems, Fort Worth, Texas; Defense Contract Management  Agency, Fort Worth, Texas; and offices of the Director, Operational Test  and Evaluation, and Acquisition, Technology and Logistics, Program  Analysis and Evaluation-Cost Analysis Improvement Group, which are part  of the Office of Secretary of Defense in Washington, D.C. We also obtained  information from the F-16 program office and Rand Corporation, Santa  Monica, California.
section_title: Appendix II: Changes in JSF Program Purchase Quantities and Costs
paragraphs: Appendix II: Changes in JSF Program  Purchase Quantities and Costs November 1996  (program start) development start) (18) (18)
Cost estimates (then year dollars in billions) (90)
Unit cost estimates (then year dollars in millions)
Unit recurring flyaway costs (fiscal year 2002 dollars in millions)
section_title: Appendix III: Comments from the Department of Defense