Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance
Released: 2006-03-15T00:00:00

Letter

Background

JSF is a joint, multinational acquisition program for the Air Force, Navy, Marine Corps, and eight cooperative international partners. The program began in November 1996 with a 5-year competition between Lockheed Martin and Boeing to determine the most capable and affordable preliminary aircraft design. Lockheed Martin won the competition, and the program entered system development and demonstration in October 2001.

DOD has set major expectations for the program. The program’s objective is to develop and deploy a technically superior and affordable fleet of aircraft that support the warfighter in performing a wide range of missions in a variety of theaters. The single-seat, single-engine aircraft is being designed to be self-sufficient or part of a multisystem and multiservice operation, and to rapidly transition between air-to-surface and air-to-air missions while still airborne. To achieve its mission, JSF will incorporate low observable technologies, defensive avionics, advanced onboard and offboard sensor fusion, internal and external weapons, and advanced prognostic maintenance capability. According to DOD, these technologies represent a quantum leap over legacy tactical aircraft capabilities. At the same time, the JSF aircraft design includes three variants: a conventional takeoff and landing variant for the Air Force; an aircraft carrier-suitable variant for the Navy; and a short takeoff and vertical landing variant for the Marine Corps, the Air Force, and the United Kingdom. JSF is intended to replace a substantial number of aging fighter and attack aircraft in DOD’s current inventory (see table 1).

In recent years, DOD has revised its acquisition policy to embrace an evolutionary, or incremental, approach to improve program outcomes. The acquisition policy states evolutionary acquisition is the preferred DOD strategy for rapid acquisition of mature technology for the user by delivering capability in increments, recognizing up front the need for future capability improvements. The objective is to balance needs with resources in order to put capability into the hands of the warfighter more quickly. The policy states that a product enters system development and demonstration when an affordable increment of militarily useful capability has been identified, technology has been demonstrated in a relevant environment, and a system can be developed for production in a short time frame (normally less than 5 years). Each increment of an evolutionary acquisition will have its own decision milestones and baseline—cost, schedule, and performance requirements. In commenting on our March 2005 JSF report, DOD stated that JSF’s restructured acquisition plan would balance technical, cost, and schedule risk and incorporate a knowledge-based, evolutionary approach to system acquisition consistent with DOD policy.

JSF Acquisition Strategy Will Begin Procurement before Testing Demonstrates the Aircraft’s Performance

DOD’s acquisition strategy for JSF has not substantially changed as a result of the program’s restructuring last year. In May 2005, DOD leadership approved the program’s plan, which intends to start production in 2007—more than 2 years before all three JSF variants have completed some flight testing of the aircraft’s basic design, 4 years before a fully configured and integrated aircraft is expected to be flight tested, and 6 years before development and initial operational testing are scheduled to be completed. Before development has ended, DOD will have potentially signed procurement contracts for 424 JSF aircraft at an estimated cost of $49 billion. Starting production before ensuring the design is mature through flight testing significantly increases the risk of costly design changes that will push the program over budget and behind schedule. Evidence of the risk associated with concurrently developing, testing, and producing the JSF aircraft can be seen in the program office strategy to place initial production orders on a cost reimbursement contract, placing an unusually high risk burden on the government during the early production phase.

Key Testing Events Will Not Be Completed before Significant Procurement Investments Are Made

The JSF program plans to begin low-rate initial production of the aircraft before many of JSF’s technology advances and basic flying qualities are flight-tested and to produce substantial quantities of aircraft before initial operational testing is completed. According to JSF program plans, DOD’s low-rate initial production quantities will increase from 5 aircraft a year in 2007 to 133 a year, before development and initial operational testing are completed in 2013. This production rate will require DOD to invest significantly in tooling, facilities, and personnel. Initial contractor cost estimates indicate that close to $1 billion of new tooling will be needed to support low-rate initial production rates. Early contractor data also show that the program will need to double its manufacturing workforce by 2008 and will need six times the number of manufacturing personnel before low-rate initial production ends. Total monthly spending for production activities in 2013 is expected to approach $1 billion, a significant increase from $100 million a month when production is scheduled to begin in 2007.

The cost of discovering design problems during production could be significant if testing shows that large, structural components of the aircraft require modifications. Design changes needed in one variant could also ripple through the other two variants, reducing efficiencies necessary to lower production and operational costs with common parts and manufacturing processes for the three variants. Some industry officials have indicated that the cost of design changes such as these could be 10 to 1,000 times greater, depending on how far the product has progressed into production.

When the JSF program is expected to begin low-rate initial production in 2007, the program will have completed less than 1 percent of the flight test program and none of the three JSF variants will have a production representative prototype built and in flight testing. Features critical to JSF’s operational success, such as the low-observable and highly common airframe and the advanced mission systems, will not have been demonstrated in a flight test environment. The program plans to proceed into low-rate initial production without demonstrating that (1) the aircraft’s flying qualities function within the parameters of the flight envelope—that is, the set limits for altitude, speed, and angles of attack; (2) the aircraft design is reliable; or (3) a fully integrated and capable aircraft system can perform as intended. These are key indicators of a program’s readiness for entering production and making significant investments in tooling, facilities, and materials. When the first fully integrated and capable development JSF is expected to fly, in 2011, DOD will have committed to buy 190 aircraft at an estimated cost of $26 billion under current plans. By the time testing is planned to be completed, in 2013, DOD will have procured more than double that amount—424 aircraft at an estimated cost of about $49 billion. Figure 1 shows the significant overlap in development and testing and the major investments in production capability and aircraft that inject additional cost and delivery risks that could delay delivery of a proven capability to the warfighter.

Under the current schedule, the JSF program plans to manufacture and deliver 15 flight test aircraft and 7 ground test (nonflying) articles in 5 years—an aggressive schedule when compared with schedules of other programs with fewer variables. Current JSF schedules are already showing delivery of early test aircraft could be later than the planned delivery date.

Unplanned manufacturing and technical problems can delay the completion of a flight test program, increase the number of flight test hours needed to verify that the system will work as intended, and affect when key knowledge will be available for making investment decisions. For example, when the B-2 program began flight testing in July 1989, it estimated that the flight test program would last approximately 4.5 years and require about 3,600 flight test hours. When the test program ended in 1997, the flight test hours had grown to 5,000 hours, or by 40 percent, over an 8-year period. Program officials cited several causes, including difficulties in manufacturing test aircraft and correcting deficiencies from unanticipated problems discovered during testing. The F-22A encountered similar delays, increasing a planned 4-year flight test program to about 8 years and affecting the program’s ability to conduct operational testing and move into production on schedule.

Initial Procurement Decisions Will Be Made before the Basic Aircraft Design Is Demonstrated

While each JSF variant is similar—all are being designed to have stealth airframes, fly at supersonic speeds, shoot air-to-air missiles, and drop bombs on a target—there are subtle airframe design differences to support the services’ different operational concepts and environments. Test officials acknowledge that each airframe variant will require flight testing to demonstrate that each will fly as intended. Yet at the time the JSF program expects to begin low-rate initial production, only 1 of 15 flight test aircraft is expected to have started testing. Further, the aircraft’s test flights are not scheduled to begin until August 2006, less than 6 months before the planned decision to begin low-rate initial production, and will not assess the aircraft in more demanding operational environments, such as high angles of attack. Moreover, the first flight test aircraft will not include a large number of design changes that resulted from an effort to significantly reduce aircraft weight over the past 2 years and, therefore, will not represent the planned production configuration. The first demonstration of a production representative airframe that includes the latest design changes is scheduled for late 2007—after production has been initiated—with first flight of the short takeoff and vertical landing variant.

All three variants will not be in flight testing until 2009, when the carrier variant is expected to have its first flight. Several key test events identified by the contractor to demonstrate the flight characteristics of these aircraft in the intended operational environment are not scheduled until 2009 or later. These include shipboard operations for the carrier and short takeoff and vertical landing variants. According to the contractor, the full-flight envelope for all three variants will not be fully demonstrated until 2011 or 2012—4 to 5 years after low-rate initial production is scheduled to begin.

According to DOD, modeling and simulation will be used to evaluate the performance of the aircraft’s flying qualities and to support decisions to invest in production. However, the Rand Corporation recently reported in a study on testing and evaluation that modeling is not a substitute for flight testing. Rand found that even in performance areas that are well understood, it is not unusual for flight testing to uncover problems that were not apparent in simulations. Examples include flight effects on the wing of the F/A-18 EF and buffeting of stores externally carried on various aircraft when flown in certain conditions. Additionally, OSD testing officials indicated that flight testing of each variant is necessary to demonstrate designed capabilities. Our past work has found that flying quality problems were identified during actual flight testing on programs like the F-22A, B-2A, and V-22.

Finally, two major ground tests of the airframe’s structural integrity— fatigue and static testing—will be in their very early stages or not have started when production begins. Fatigue testing, which measures the aircraft’s durability over its expected life, is slated to begin in 2008 with testing to show structural fatigue over one lifetime of flying to be completed in 2009 and a second lifetime completed by 2010. Fatigue testing uses actual JSF airframe structures that are subjected to the long- term stresses expected over the aircraft’s life. Standards require this testing to be done to represent the stresses expected over two lifetimes. Static testing, which measures the aircraft’s ability to withstand the stresses expected to be encountered throughout the aircraft’s flight regime, is slated to begin in 2007 and last through 2009. These tests are important to reduce the risk of structural problems emerging during production or after aircraft are fielded.

Mission Capabilities Will Not Be Tested until 4 Years after Procurement Has Begun

In addition to late design testing, many of the mission systems planned for JSF will not be available for initial flight testing either. Although laboratory tests are under way, DOD does not plan to flight-test several of the new technologies needed for the JSF to perform its intended missions until 2009 at the earliest. Defense Operational Test and Evaluation officials have stated that flight-testing capabilities in a production representative test aircraft and in the operational environment planned for the new system are important to reducing risk. This actual environment differs from what can be demonstrated in a laboratory and has historically identified unexpected problems. For example, the F-22A software worked as expected in the laboratory, but when tested in the aircraft, significant problems were identified. These problems delayed testing and the delivery of a proven capability. The different levels of mission capability will be tested in JSF aircraft as follows.

Block 1, an initial air interdiction capability, is scheduled to begin testing in 2009, with initial operational testing scheduled for 2011—4 years after DOD plans to begin production.

Block 2, an improved air interdiction and close air support capability, is scheduled to begin testing in 2010, with initial operational testing scheduled for 2012. This block will include several critical technologies that are not fully mature, such as the advanced missions systems and prognostics and health maintenance, but are critical to meeting requirements like sortie generation and mission capabilities.

Block 3, the fully integrated and capable JSF, is scheduled to begin testing in 2011. At this time less than 50 percent of the planned mission capability testing will have been completed. This is close to the same point that the F-22A and other past programs experienced difficulties integrating all the complex software and hardware components into the aircraft. Flight testing to evaluate the effectiveness and suitability of the fully integrated system is expected to continue until the full-rate decision in late 2013.

Aggressive Software Development Schedule May Cause Delays

JSF’s expected performance is largely dependent on demonstrating software that supports vehicle, mission system, and other capabilities. The program plans to develop over 19 million lines of code—substantially more than the lines of code needed for the F-22A. The software is planned to be developed in five blocks. The first block is scheduled for completion in 2006, and the last block is scheduled for completion in 2011. At the time the program enters low-rate initial production, the program will have completed less than 35 percent of the software needed for the system’s functionality. Past programs have encountered difficulties in developing software, delaying flight test schedules. Data provided by the program office indicate that the program is already showing early signs of falling behind its software delivery schedule. JSF program officials recognize the risk associated with this large software effort.

The Current Acquisition Strategy Requires Prolonged Reliance on Cost Reimbursement Contracts That Place an Increased Risk on DOD

According to program officials, the uncertainties inherent in concurrently developing, testing, and producing the JSF aircraft prevent the pricing of initial production orders on a fixed price basis. Consequently, the program office plans to place initial production orders on a cost reimbursement basis. Cost reimbursement contracts provide for payment of allowable incurred costs, to the extent prescribed in the contract. Such contracts are generally used only when the uncertainties involved in contract performance do not permit costs to be estimated with sufficient accuracy to use any type of fixed price contract. Cost reimbursement contracts place substantial risk on the buyer—in this case DOD—because the contractor’s responsibility for performance costs is minimized or reduced. In contrast, a fixed price contract provides for a pre-established price, and places more risk and responsibility for costs and resulting profit or loss on the contractor and provides more incentive for efficient and economical performance.

The program plans to transition to fixed-price-type contracts once the air vehicle has a mature design, has been demonstrated in flight test, and is producible at established cost targets. According to program officials, this transition will occur sometime before full-rate production begins in 2013. The program office believes the combination of the early concept development work, the block development approach, and the relatively small numbers of aircraft in the initial production buys allow decisions to be made earlier than normal with an acceptable level of risk.

Evolutionary Acquisition Strategy Provides a Less Risky Alternative for the JSF Program

The JSF acquisition strategy currently plans a single-step approach to deliver a quantum leap in tactical fighter capability by 2013 and has already felt the negative cost and schedule impacts from the executing this approach. The length and scope of the remaining effort in the JSF program make it even more difficult to accurately estimate cost and delivery schedules. The JSF funding profile—which requires an average of $11 billion annually for the next two decades—is also at risk to increase if costs continue to grow or schedules are further delayed to develop the ultimate JSF capabilities. An alternative acquisition strategy, such as used by the F-16 program, that sequences capabilities over time based on proven technologies and design would reduce risk and deliver aircraft sooner. This evolutionary approach is actually the preferred approach in DOD’s acquisition policy for acquiring new systems for more rapid delivery of incremental capabilities to the warfighter.

JSF Program Is Designed to Deliver Full Required Capability in a Single-Step Development Program

Instead of establishing time-phased requirements for aircraft to be delivered in sequence that could first meet DOD’s need to recapitalize its aging fleet of aircraft and then evolve the aircraft to eventually achieve improved capabilities in future system development increments, DOD chose a single-step development approach to deliver the full required capability by the end of system development in 2013. That approach is now planned to last 17 years (5 years in concept development and 12 years in system development)—including the development of immature, undemonstrated technologies and will take at least two to three times longer than the development time of leading commercial firms or that suggested by defense acquisition policy. The JSF acquisition strategy is shown in figure 2.

While JSF’s acquisition strategy calls for delivering a small number of aircraft with limited capabilities, the program has committed to deliver the full required capability by the end of system development and demonstration in 2013. For JSF, this includes a set of objectives that exceeds those of aircraft development programs of the past. JSF will use cutting-edge technology to fuse data from the aircraft itself and other sources, from all aspects of the aircraft, and display the information to the pilot. The aircraft must be able to rapidly transition from ground attack to air-to-air missions while airborne. JSF also expects advances in technologies for mission systems, prognostics, and autonomic logistics support requiring hardware development as well as extensive and complex software development and integration to have lower cost of ownership than the legacy aircraft while being able to deploy rapidly, sustain high mission reliability, and maintain a high sortie generation rate.

Past single-step development programs have been unsuccessful in predicting acquisition costs and delivery schedules. For example, development costs for the F-22A fighter, B-2 bomber, Crusader artillery vehicle, and Comanche helicopter skyrocketed, and production quantities were either substantially reduced or the program was canceled. Such outcomes are a poor return on investment and a failure to recapitalize aging legacy systems. So far the JSF program is experiencing similar results. Since the program’s start, development cost has increased 84 percent, the development schedule has increased by almost 5 years, and planned delivery of capabilities to the warfighter has been delayed. DOD now plans to buy 535 fewer aircraft than originally planned. As a result, DOD’s buying power has been significantly reduced—program acquisition unit costs have increased by 28 percent, or $23 million, since first estimates. See appendix II for more details on JSF outcomes.

Funding Needs Could Increase Given the Level of Risk to Complete the JSF Program

Despite a lengthy restructuring effort that added over $7 billion and 18 months to the development program, JSF officials have stated that the restructured program has little or no flexibility for future changes or unanticipated risks. Furthermore, the length and scope of the remaining effort make it even more difficult to accurately estimate cost and completion schedules. While it has been over 9 years since the program started, the first flight test aircraft still has not been delivered. The program has planned almost 8 years to complete the remaining activities of the system development and demonstration phase. These remaining activities include fully maturing 7 of the 8 critical technologies; completing the designs and releasing the engineering drawings for all three variants; manufacturing and delivering 15 flight test aircraft and 7 ground test developing 19 million lines of software code; and completing a 7-year, over 12,000-hour flight test program.

The JSF program’s latest planned funding profile for development and procurement, produced in December 2004, expects annual funding requirements to hover close to $13 billion in between 2012 and 2022, peaking at $13.8 billion in 2013. If the program fails to achieve its current estimated costs, funding challenges could be even greater than they are today. Thus, even a modest cost increase would have dramatic impacts on funding. For example, a 10 percent increase in production costs would amount to over $21 billion (see fig. 3).

The current cost estimate reflects the position of the JSF program office. The Office of Secretary of Defense Cost Analysis Improvement Group was to update its formal independent cost estimate in the spring of 2005, and the Navy and the Air Force were expected to fully fund the program consistent with the estimate. The group now does not expect to formally complete its estimate until the spring of 2006, but its preliminary estimate was substantially higher than the program office’s. According to Cost Analysis Improvement Group officials, an assessment of the software and mission systems requirements based on more recent information could further increase the estimate of JSF costs.

An Incremental Development Approach Provides an Alternative Model for Reducing JSF Risks

An incremental development approach consistent with DOD’s policy on evolutionary acquisition and best practices has potential to deliver warfighter capabilities in planned product increments that would increase JSF capabilities over time. The F-16 fighter program, the Air Force’s JSF predecessor, successfully evolved capabilities over the span of about 30 years, delivering increases of capabilities quickly and often, as technologies became available. That program may provide a model for a possible alternative acquisition strategy for the JSF program. Structuring the program into separate and manageable increments based on what is achievable now and in the future would allow more predictable cost and delivery estimates.

Over the past three decades, the Air Force successfully procured more than 2,200 F-16s. The F-16 acquisition approach allowed the timely and affordable delivery of aircraft and capability to meet the warfighter’s needs, including the recapitalization of aging aircraft. By using an evolutionary approach to develop the aircraft, the program was able to quickly deliver new and improved capabilities to the warfighter and increase the aircraft’s capability as new technologies were matured and added to the aircraft. The first increment, developed during the 1970s, provided a “day fighter” aircraft with basic air-to-air and air-to-ground capabilities. This allowed the developer to deliver new and useful military capability to the warfighter in less than 4 years. With each subsequent increment, new technology was used to improve the engine, radar, structure, avionics, and other systems that allow the aircraft today to perform close air support, ground attack, air defense, and suppression of enemy defense missions (see fig. 4). The evolutionary approach also enriched the industrial base capabilities by extending the life of the production over the length of this incremental approach.

In contrast, JSF’s fully configured design represents a quantum leap in capability that far exceeds the capability of legacy systems that JSF is intended to replace. While the program is using a block structure—where each block adds capabilities over the proceeding block—the blocks are part of a single development effort, and DOD is on contract with the developer to deliver the warfighter the full capability (see fig. 5). The program’s block structure provides for an escalating capability, but DOD already plans to buy 95 percent of JSF aircraft with the ultimate capability (block 3). Unlike the approach used with the F-16, this risky approach will likely be like past programs that have encountered significant increases in cost and time, not allowing DOD to quickly recapitalize the aging legacy aircraft.

An evolutionary approach would entail delivering a first product increment with at least as much capability as currently operating legacy aircraft with sufficient quantities to allow DOD to retire older planes sooner and reduce operating inefficiencies. DOD has repeatedly raised concerns about the age of its fighter aircraft fleet, which was bought largely in the 1970s and 1980s and will need to be replaced around 2010. Delays in fielding JSF aircraft may increase costs to maintain legacy aircraft to meet force structure requirements. Limiting development to 5- year increments or less, as suggested in the DOD acquisition policy, would force smaller, more manageable commitments in capabilities and make costs and schedules more predictable. Some of the more challenging JSF capabilities, such as the advanced mission systems or prognostics technologies, would be deferred, kept in the technology base (off the critical path of the JSF program), and added to follow-on increments once they are demonstrated in the more conducive technology development environment. In addition to considering available resources—including technology and design knowledge, budget, and time—each increment should be based on the warfighter’s most immediate needs and the number of aircraft needed to maintain a viable fleet. Figure 6 shows what an incremental approach might look like for JSF.

Because an incremental approach would reduce the amount of risk in each development phase, it would make the program manager and contractor accountable for a lower-risk system development phase with precise cost and schedule targets. A shorter system development phase also makes it more feasible to align a program manager’s tenure to the completion of the phase, holding the program manager accountable for decisions made. It also would enable the use of fixed-price-type contracts for production that contains a pricing structure that reduces government risk.

Conclusions

While the JSF program plays a critical role in DOD plans to recapitalize the services’ aging tactical aircraft fleet, DOD’s current acquisition approach for the JSF program continues to be risky and could further jeopardize meeting this important objective. The JSF program has already encountered increases to estimated development costs, delays to planned deliveries, and reductions in the planned number of JSF to be procured that have eroded DOD’s buying power. We believe the current acquisition strategy to begin production in 2007 is too risky. By committing to procure large quantities of the aircraft before completing testing that demonstrates that the design is mature and reliable, DOD has significantly increased the risk of further compromising its return on investment—as well as adding more cost and delaying the delivery of critical capabilities to the warfighter. Also, making sizable investments in tooling and other manufacturing capability needed to produce JSFs at higher rates before a fully integrated aircraft is demonstrated in testing places the program at risk for expensive design changes as testing uncovers problems. Deferring production decisions until all three variants’ performance has been demonstrated throughout their flight envelopes and with the full integrated capability would allow additional time to capture knowledge and help to significantly minimize these risks.

To continue with the current plan to deliver the ultimate JSF capability will require (1) execution of a 7-year test program without further schedule delays, (2) development of 5 major blocks of software (19 million lines of code), and (3) new and yet undemonstrated technologies for advanced mission systems that must collect, analyze, and synthesize information from other platforms or sensors not all a part of the JSF aircraft. These are necessary to ensure the JSF has increased situational awareness to destroy enemy air defenses and perform deep strike missions with advanced weapons. The JSF program will also need, annually, an average of $11 billion of development and procurement funding for the next two decades.

All of these factors add to the challenges faced by DOD in trying to manage this highly concurrent program, increasing the risk it will have the same poor outcomes experienced by similar major acquisition programs with significant cost and schedule growth and delayed modernization and recapitalization of aging systems. DOD has an alternative. With 90 percent of DOD’s remaining planned investment in JSF, it can adopt a knowledge- driven and evolutionary acquisition approach to reduce JSF program risks, recapitalize its aging tactical air force sooner, and deliver needed capabilities to the warfighter more quickly. The experiences of the F-16 program—evolving and improving capabilities over time while providing the warfighter combat capability—provide a precedent for this. Requirements for the first increment of JSF under this evolutionary approach would match a level supported by current knowledge of technologies and design. This would allow the testing of those reduced requirements to support a knowledge-based low-rate production decision sooner and allow delivery of a useful product and in sufficient quantities to start replacing the aging legacy fighter force. Capabilities that demand as yet undemonstrated technologies, for example, advanced mission systems and software, would be included as requirements in subsequent future JSF aircraft increments—managed as separate development programs—as technology is demonstrated in the technology base and becomes available.

Matters for Congressional Consideration

DOD does not plan to change its business case or acquisition plan for developing and buying the JSF. Without changes, the acquisition plan will put at risk $50 billion for procuring JSF aircraft at the same time the program develops and tests the aircraft’s expected performance capabilities over a 7-year, 12,000 hour flight test program. The JSF program has continually missed its cost and schedule targets over the 5 years it has been in development. If DOD were to make smaller, more manageable commitments in capabilities, it would make cost and schedule more predictable and deliver needed capabilities to the war fighter sooner. For these reasons, Congress may want to consider delaying authorizations and appropriations for procuring JSF aircraft until 1. DOD develops a knowledge-based business case that matches requirements with proven technologies and design knowledge and available funding. Capabilities that demand technological advances not yet demonstrated should be part of future increments that are funded and managed separately once demonstrated. 2. DOD demonstrates the aircraft design qualities and integrated mission capabilities of the fully configured and integrated JSF variants work as designed based on actual flight testing.

Recommendations for Executive Action

We are making the following recommendations to reduce program risk and increase the likelihood of successful program outcomes by delivering capabilities to the warfighter when needed and within available resources. Specifically, we recommend that the Secretary of Defense: 1. Delay the investment in production aircraft until sufficient testing has at least demonstrated the basic airframe design of each JSF variant in important parts of the flight envelope, including combinations of critical altitudes, speeds, and angles of attack. 2. Once the basic airframe design has been demonstrated, limit production quantities to the number of aircraft needed for operational testing and that can be produced using the current production capability (based on existing tooling, materials, and personnel) until a fully integrated aircraft (with advanced mission systems and predictive maintenance systems) demonstrates through flight testing the required capabilities. 3. Plan an incremental acquisition strategy that follows the intent of DOD evolutionary acquisition policy and delivers a first increment that limits new content to proven technologies and design. The plan should be completed and reported to the Congress by July 2006, and include adjustments in associated programs as necessary to maintain needed capabilities.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. The comments appear in appendix III.

DOD partially concurred with our recommendation that the Secretary of Defense delay the investment in production aircraft until sufficient testing has at least demonstrated the basic airframe design of each JSF variant in important parts of the flight envelope, including combinations of critical altitudes, speeds, and angles of attack. DOD agreed that a knowledge- based approach is critical to making prudent decisions and that this type of approach is being used by JSF. However, DOD’s response did not provide a plan of action to show how it will ensure flight testing demonstrates acceptable performance before significant production investments are made. In the past, Congress has raised concerns about the costly outcomes of highly concurrent development and production efforts that are not “flying before buying.” DOD points out that the JSF program is in its fifth year of a 12-year development, but starting production with nearly 7 years of development and testing remaining leaves a high probability that, because it is not flying before buying, it will incur costly design changes and delay getting capability to the warfighter. Until there has been testing to demonstrate that all three variants will work as intended, we believe entering production in 2007 places financial investments at significant risk. The risk increases as spending for production goes from about $100 million a month in 2007 to over $500 million a month just 2 years later.

DOD partially concurred with our recommendation to the Secretary of Defense to limit production quantities to the number of aircraft needed for operational testing and that can be produced using the current production capability until a fully integrated aircraft demonstrates the required capabilities through flight testing. DOD stated it is limiting production until aircraft capabilities are fully demonstrated and that further limits are not necessary. It stated that various program reviews to assess performance, including test objectives, are conducted to ensure associated program risks are understood and mitigated. We believe DOD’s plan to invest almost $50 billion to buy over 400 aircraft concurrent with testing is very risky. Significant efforts remain in the JSF program to demonstrate the aircraft will perform as expected. A number of major DOD acquisition programs have employed highly concurrent acquisition strategies in the past and experienced significant cost increases and schedule delays. DOD stated its use of modeling and simulation and laboratory testing reduces risk. However, DOD operational test officials have stated that demonstrations need to occur in the actual aircraft in an operational environment to verify that the system works as intended. For example, the F-22A avionics software performed successfully in the laboratory but experienced significant problems that delayed the program and increased costs once it entered actual flight testing in the F-22A aircraft. JSF software and advanced electronics are more complex than the F-22A’s. While DOD believes it can manage program risk by holding regular program reviews, DOD’s own experience has shown this approach is not effective. Accepting and managing risk instead of capturing technology, design, and manufacturing knowledge, as suggested in DOD policy, has made it difficult for DOD to make informed decisions at key points.

DOD partially concurred with our recommendation to the Secretary of Defense to plan an incremental acquisition strategy that follows the intent of DOD evolutionary acquisition policy and delivers a first increment that limits new content to proven technologies and design and to report this plan to Congress by July 2006. It stated the JSF Acquisition Strategy fully complies with policy and is a knowledge-based, incremental approach that includes three blocks of increasing degrees of warfighter capability. We believe DOD’s strategy does not provide the benefits of an evolutionary approach, as suggested by DOD’s policy and best practices. In commenting on our report, DOD did not address the salient points concerning the true significance of adopting an incremental acquisition approach—reducing program risk, delivering needed capabilities to the warfighter quicker, and recapitalizing the aging tactical aircraft fleet sooner. The JSF strategy resembles other past major acquisition programs that have attempted to achieve the ultimate capability in a single development increment, producing nearly all of the fleet with the fully required capability from the outset. DOD has allowed technology development to spill over into product development, weakening the foundation for program cost or schedule estimates. This has led to disastrous outcomes for other major acquisition programs in the past. We continue to believe the successful F- 16 program can serve as a model for the JSF program. The F-16 program evolved capabilities over a 30-year period, buying substantial quantities of each increment in order to recapitalize aging tactical aircraft and provide new capabilities to the warfighter more quickly. If DOD were to make smaller, more manageable commitments in capabilities, it would make cost and schedule more predictable and deliver needed capabilities to the warfighter sooner.

We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. We will also provide copies to others on request. In addition, the report will be available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841. Contact points for our offices of Congressional Relations and Public Affairs may be found on the last page of this report. Other staff making key contributions to this report were Michael Hazard, Assistant Director; Lily Chin; Matthew Drerup; Matthew Lea; Gary Middleton; Karen Sloan; and Adam Vodraska.

Appendix I: Scope and Methodology

To determine if the Joint Strike Fighter (JSF) program will capture critical product knowledge before making production investments, we compared the program’s plans and results to date against best practice standards for applying knowledge to support major program investment decisions. The best practice standards are based on a GAO body of work that encompasses nearly 10 years and visits to over 25 major commercial companies. Our work has shown that valuable lessons can be learned from the commercial sector, as well as successful Department of Defense (DOD) cases, and can be applied to the development of weapon systems. We reviewed JSF management reports, acquisition plans, test plans, risk assessments, cost reports, independent program assessments, and program status briefings. We identified gaps in product knowledge at the production decision points, reasons for these gaps, and the risks to program cost and schedule associated with moving forward. We reviewed DOD’s acquisition policy to determine whether JSF’s approach met its framework and intent. We interviewed officials from the DOD acquisition program management office and prime contractor to gain their perspectives on program risks and their approaches to managing risks.

To evaluate whether the current acquisition plan follows an evolutionary or incremental approach, a key best practice for meeting business case goals, we examined the JSF program’s acquisition framework and the actions taken by DOD to address the recommendations made in our 2005 report. We compared the JSF program’s approach with the approaches used by leading companies and successful DOD programs to evolve products to their ultimate capabilities. We also reviewed DOD guidance on structuring evolutionary acquisition programs. To determine the JSF program outcomes to date, we used the program estimates that justified the program when it started in 1996. This was the point JSF transitioned from a technology development environment to an acquisition program environment, with the commitment to deliver a family of strike aircraft that meets the Air Force, Navy, and Marine Corps needs. At that time, total production, acquisition, and ownership costs had not been estimated. The total production, acquisition, and ownership cost estimates were first established to support the decision to enter the system development and demonstration phase in 2001. We used these estimates as the baseline for these costs. We also obtained information on past DOD programs from Selected Acquisition Reports and prior work conducted by GAO over the past 25 years.

To identify opportunities to improve JSF program outcomes, we assessed the potential of applying an incremental development approach to the JSF acquisition program based on the commercial model and DOD guidance.

We also examined the program history of the F-16 fighter, the JSF predecessor, which applied an incremental approach over its 30-year acquisition. The F-16 acquisition has been touted by DOD and others as a successful program and a model for others to follow. To examine the program history of the F-16, we met with F-16 program officials and analyzed acquisition plans, management reports, and program outcomes.

In performing our work, we obtained information and interviewed officials from the JSF Joint Program Office, Arlington, Virginia; Lockheed Martin Aeronautical Systems, Fort Worth, Texas; Defense Contract Management Agency, Fort Worth, Texas; and offices of the Director, Operational Test and Evaluation, and Acquisition, Technology and Logistics, Program Analysis and Evaluation-Cost Analysis Improvement Group, which are part of the Office of Secretary of Defense in Washington, D.C. We also obtained information from the F-16 program office and Rand Corporation, Santa Monica, California.

Appendix II: Changes in JSF Program Purchase Quantities and Costs

Appendix II: Changes in JSF Program Purchase Quantities and Costs November 1996 (program start) development start) (18) (18)

Cost estimates (then year dollars in billions) (90)

Unit cost estimates (then year dollars in millions)

Unit recurring flyaway costs (fiscal year 2002 dollars in millions)

Appendix III: Comments from the Department of Defense