Joint Strike Fighter: Accelerating Procurement before Completing Development Increases the Government's Financial Risk
Released: 2009-03-12T00:00:00

Letter

Background

The JSF is a joint, multinational acquisition program for the Air Force, Navy, and Marine Corps and eight international partners. The program began in November 1996 with a 5-year competition between Lockheed Martin and Boeing to determine the most capable and affordable preliminary aircraft design. Lockheed Martin won the competition, and the program entered system development and demonstration in October 2001. Program goals are to develop and field an affordable, highly common family of stealthy, next-generation strike fighter aircraft for the Navy, Air Force, Marine Corps, and U.S. allies.

The JSF is a single-seat, single-engine aircraft, designed to rapidly transition between air-to-ground and air-to-air missions while still airborne. To achieve its mission, the JSF will incorporate low-observable (stealth) technologies, defensive avionics, advanced onboard and offboard sensor fusion, internal and external weapons, and advanced prognostic maintenance capability. The JSF family consists of three variants. The conventional takeoff and landing (CTOL) variant will primarily be an air- to-ground replacement for the Air Force’s F-16 Falcon and the A-10 Warthog aircraft, and will complement the F-22A Raptor. The short takeoff and vertical landing (STOVL) variant will be a multi-role strike fighter to replace the Marine Corps’ F/A-18C/D Hornet and AV-8B Harrier aircraft. The carrier-suitable variant will provide the Navy a multi-role, stealthy strike aircraft to complement the F/A-18E/F Super Hornet. DOD is planning to buy a total of 2,456 JSFs and allies are expected to procure a minimum of 730 CTOL and STOVL aircraft.

Because of the program’s sheer size and the numbers of aircraft it will replace, the JSF is the linchpin of DOD’s long-term plan to modernize tactical air forces. It is DOD’s largest acquisition program, with total cost currently estimated at $300 billion, and the longest in planned duration, with procurement projected through 2034. In addition, the JSF remains DOD’s largest cooperative program. Our international partners are providing about $4.8 billion toward development, and foreign firms are part of the industrial base producing aircraft. DOD’s funding requirements for the JSF assume economic benefits from these foreign purchases in reducing unit costs for U.S aircraft.

Table 1 shows the evolution of DOD’s official estimated cost, quantity, and deliveries from the initiation of system development in October 2001 to the current official program of record dated December 2007 and submitted to the Congress in April 2008. It depicts quantities reduced in the last major program restructure in 2004, the impacts of increased costs on unit prices, and the slip in delivering initial operating capability to the warfighter.

In our March 2008 report, we stated that JSF costs would likely grow much higher than reported because the program of record at that time did not include all acquisition costs (including the alternate engine program directed by the Congress), made overoptimistic assumptions, and did not fully reflect the mounting cost and schedule pressures from manufacturing inefficiencies and compressed time frames for completing development. We questioned the Mid-Course Risk Reduction Plan adopted by DOD in September 2007 that cut two development test aircraft, reduced test flights, and accelerated the reduction in the prime contractor’s development workforce in order to replenish management reserves depleted by design changes and manufacturing problems. We recommended that DOD accomplish a full and comprehensive estimate of the total program verified by an independent third party and revisit the Mid-Course Risk Reduction Plan with an intensive analysis of the causes of management reserve depletion, progress against the baseline manufacturing schedule, and correction of deficiencies in the contractor’s earned value management system. DOD agreed to make a comprehensive independent cost estimate, but decided to go ahead as planned with the Mid-Course Risk Reduction Plan, stating that it would monitor and evaluate progress and revise the plan later if it failed to achieve expectations.

More Money and Time Will Be Needed to Complete JSF Development, While DOD Plans to Accelerate Procurement

Two recent estimates indicate that JSF development will cost more and take longer to complete than reported to the Congress in April 2008, primarily because of contract cost overruns and extended time to complete flight testing. DOD also plans to accelerate aircraft procurement over the next 6 years—buying more aircraft sooner than planned last year. This new plan will require significantly more procurement funding sooner, but officials did not assess its net effect on total program costs through completion of JSF acquisition.

New Estimates Project Rising Costs and Further Delays to Complete JSF Development

Development costs are projected to increase between $2.4 billion and $7.4 billion and the schedule for completing system development extended from 1 to 3 years, according to recent estimates—one by the JSF Program Office and one by a joint team of Office of the Secretary of Defense (OSD), Air Force, and Navy officials. Cost overruns on both the aircraft and engine contracts, delays in manufacturing test aircraft, and a need for a longer, more robust flight test program are the primary cost drivers. The joint team’s estimate is higher than the program office’s because it included costs for the alternate engine program directed by the Congress and used more conservative assumptions based on current and legacy aircraft experiences. Program officials contend that funding the program to the higher cost estimate is premature and believe processes are in place to substantially improve on the test experiences of past programs. Regardless, both estimates agree that cost and time to complete development have increased from the official program of record at the time of our review. (See table 2.)

The program office’s revised development cost estimate projects an additional $2.4 billion and a 1-year extension in the schedule compared to the official program of record reported to the Congress. This would increase the system development portion of the acquisition program to $46.8 billion and delay its completion to October 2014. The cost increases primarily resulted from the following factors. $1.2 billion for aircraft development. Program officials declared a cost overrun on the prime air system contract because of increased labor hours, higher prices, and supply shortages. Included in this figure is $200 million to be added to the contractor’s management reserve. Last year, we reported that mounting cost and schedule pressures depleted reserves much faster than anticipated. By summer 2007, the program had spent about two-thirds of budgeted funds but had accomplished only half the work required. Since then, DOD’s efforts to restore reserves and fix manufacturing inefficiencies have not fully achieved intended results, requiring another cash infusion. $800 million for engine development. According to officials from the Defense Contract Management Agency, the engine contractor continued to face development problems, which resulted in a contract cost overrun. Higher costs for labor and materials, supplier problems, and the rework needed to correct deficiencies with the engine blade design discovered during ground testing were major contributing factors. $300 million for flight test extension. The program extended system development by 1 year to provide more time for development and operational flight testing. In April 2008, an operational test review team recommended extending the development contract by 1 year. The review team considered but dismissed several other options to address the schedule problem, including deferring requirements.

On the other hand, the joint estimating team estimates that it will cost $14.8 billion to complete JSF development, $7.4 billion more than the official program of record at the time of our review. This would increase the total development costs to $51.8 billion from the $44.4 billion reported to the Congress last year—a 17 percent increase. The joint team also projected a 3-year program extension beyond the program of record in order to complete system development, 2 years more than the new program office estimate.

The joint team’s estimate was $5 billion more than the new program office estimate. Several factors account for the difference between the two estimates.

Alternate engine. The joint estimating team included $1.4 billion to complete development of an alternate engine for the JSF; the program office estimate did not include alternate engine costs. The Congress has directed DOD to develop a second source for the JSF engine to induce competition and to ensure that one engine’s failures would not ground all JSFs, thereby reducing operational risks in the future. DOD has not wanted to pursue this second engine source and twice removed funding from the JSF program line.

Engineering staffing. The joint estimating team projected a need for the contractor to retain considerably more engineering staff and for longer periods of time than the program office estimate to complete development, evaluate test results, and correct deficiencies. Releasing engineering staff prematurely risks not discovering problems until late in development or during fielding, when they would be more expensive to address.

Software development. The joint estimating team believes that the software productivity rate will be less than the program’s calculation and anticipates much more growth in software requirements. The JSF aircraft is expected to require 7.5 million lines of computer code—the most by far of any aircraft. By comparison, the F/A-18E/F has only 1.1 million and the F-22A 2.2 million. Experiences on past acquisitions have shown 30 to 100 percent growth in software requirements over time, while the JSF Program Office estimate assumed no growth.

Flight testing. The joint estimating team projects that flight testing will require more time and effort than the program office has built into its current schedule. Continuing delays in delivering test aircraft are expected to hamper and further compress test schedules. In particular, the joint team projects that the two aircraft dedicated to carrier suitability tests will be late off the production line, thereby delaying test activities. It also projected that the JSF will require about 2,700 hours of flight testing for mission systems, significantly more than the 1,700 hours that the program office currently estimates.

Manufacturing production hours. The joint estimating team projects that production span times for the JSF will be longer than the program office estimates based on the program’s performance to date and experience of recent programs, such as the F/A-18 E/F and the F-22A. The span time is an indicator of how long the manufacturing effort takes and when flight testing can begin. The program office assumes that span times will decrease over the course of the development contract. We note that span times typically increase during development, as was the case for both the B-2 bomber and F-22A programs.

Program officials believe that their estimate is more accurate and that providing extra funding to address future risks is premature and does not provide incentives for contractors to control costs. The program office attributes its lower cost estimate to several factors. First, the quantity, quality, and flexibility of the JSF laboratories should enable the program to reduce more risks in a laboratory environment, rather than through flight testing, which is considerably more expensive. In addition, the program’s efforts to develop the final software system infrastructure early should reduce significant software problems later in the program, according to the program office. The program office also believes that costs will be lower because progress in several key development areas is either matching plans or is ahead of where legacy programs were at similar points of time in their development. For example, the program is currently reducing engineering staff as planned. The program is also producing software at a rate significantly higher than that of the F-22A program and is at least 18 months ahead of where the F-22A program was at a similar point in developing mission systems, according to program officials. Officials told us that they intend to fund the fiscal year 2010 development budget based on the joint team’s higher estimate. However, it is not clear at this stage which estimate will serve as the basis for future budget submissions.

Much Higher Annual Procurement Funding Required to Accelerate JSF Procurement

The program office and joint estimating team also projected procurement funding requirements for the 6-year period fiscal years 2010-2015 based on DOD plans to accelerate procurement of operational aircraft. Through this effort, DOD wants to recapitalize tactical air forces sooner and mitigate projected fighter shortfalls in the future. Compared to last year, this accelerated plan would procure an additional 169 aircraft during these 6 years, moving aircraft that had been scheduled for procurement beyond 2015 to earlier years. According to the two estimates, this plan would require from $21.8 billion to $33.4 billion more funding than the official program of record, as shown in table 3.

The joint team’s estimate is higher than the program office’s, primarily for these reasons: The joint team projected slower gains in production efficiency than the program office. Typically, production efficiency is improved and unit costs are lowered over time as a workforce becomes more experienced building a new product and manufacturing processes are honed.

The joint team also assumed fewer savings from commonality.

Commonality—a key selling point for the JSF program—refers to the use of the same or similar parts, structures, and subsystems shared by the three variants. Greater commonality can save money by decreasing development times and facilitating economic order quantities.

The team projected higher labor and material costs and longer production span times, based on JSF performance to date in manufacturing development test aircraft.

Table 4 shows the additional aircraft and funding requirements for DOD’s accelerated plan compared to the official program of record. These quantities are for the United States only; during this same period, the international partners are expected to buy 273 aircraft.

Procurement of Operational Aircraft under Cost Reimbursement Contracts to Continue; Increases the Government’s Exposure to Risks

The JSF program is procuring a substantial number of aircraft on cost reimbursement contracts. Cost contracts place most of the risk on the buyer—DOD in this case—who is liable to pay more than budgeted should labor, material, or other incurred costs be more than expected when the contract was signed. JSF officials plan to procure at least the first four low-rate production lots under cost reimbursement contracts and to transition to fixed-price instruments when appropriate, possibly between lots 5 and 7 (fiscal years 2011 to 2013). It is unclear exactly how and when this will happen, but the expectation is to transition to fixed pricing once the air vehicle has a mature design, has been demonstrated in flight tests, and is producible at established cost targets. To date, DOD has procured the first three lots for a total of 30 aircraft and $7.4 billion on cost reimbursement terms. Under the accelerated procurement plan, DOD could procure as many as 360 aircraft costing about $57 billion through fiscal year 2013 on cost reimbursement contracts, as illustrated in figure 1.

Cost reimbursement contracts provide for payment of allowable incurred costs, to the extent prescribed in the contract. According to the Federal Acquisition Regulation, cost reimbursement contracts are suitable for use only when uncertainties involved in contract performance do not permit costs to be estimated with sufficient accuracy to use any type of fixed- price contract. Cost reimbursement contracts for weapon production are considered appropriate when the program lacks sufficient knowledge about system design, manufacturing processes, and testing results to establish firm prices and delivery dates. In contrast, a fixed-price contract provides for a pre-established price, places more of the risk and responsibility for costs on the contractor, and provides more incentive for efficient and economical performance.

Procuring up to 360 production aircraft on cost reimbursement contracts—-nearly 15 percent of the total DOD program—seems to be a tacit acknowledgment by DOD and the contractor that knowledge on JSF design, production processes, and costs for labor and material is not yet sufficiently mature and that pricing information is not exact enough for the contractor to assume the risk under a fixed-price contract. It also seems to be a consequence of the substantial concurrency of development, test, and production built into the JSF schedule. Significant overlap of these activities means that DOD is procuring considerable quantities of operational aircraft while development test aircraft are still on the production line and much in advance of testing to prove aircraft performance and suitability. Establishing a clear and accountable path to ensure that the contractor assumes more of the risk is prudent. We note that the significant ramp up in JSF production under the accelerated profile starts with lot 5, the fiscal year 2011 procurement of 70 aircraft.

Continued Manufacturing Inefficiencies Will Make It Difficult for the Program to Meet Its Production Schedule

Manufacturing of development test aircraft is taking more time, money, and effort than planned, but officials believe that they can work through these problems and deliver the 11 remaining test aircraft by early 2010. However, by that time, DOD may have procured as many as 62 production aircraft, accumulating a backlog of aircraft to be produced. Manufacturing inefficiencies and parts shortages continue to delay the completion and delivery of development test aircraft needed for flight testing. The contractor has not yet demonstrated mature manufacturing processes, or an ability to produce at currently planned annual rates. It has taken steps to improve manufacturing, the supplier base, and schedule management. However, given the manufacturing challenges, we believe that DOD’s plan to accelerate production in the near term adds considerable risk and will be difficult to achieve in a cost-effective manner.

Time and Money Needed for Manufacturing Development Test Aircraft Continue to Increase

The prime contractor has restructured the JSF manufacturing schedule three times, each time lengthening the time to deliver aircraft to the test program. Delays and manufacturing inefficiencies are prime causes of contract cost overruns. The contractor has produced two development test aircraft—an original non–production representative model and the first STOVL aircraft. It now projects delivering the remaining 11 aircraft in 2009 and early 2010. Problems and delays are largely the residual effects from difficulties early in development. The effects of the late release of engineering drawings, design changes, delays in establishing a supplier base, and parts shortages continue to cause delays and force inefficient production line work-arounds where unfinished work is completed out of station. Data provided by the Defense Contract Management Agency and the JSF Program Office show continuing critical parts shortages, out-of- station work, and quality issues. The total projected labor hours to manufacture test aircraft increased by 40 percent just in the past year, as illustrated in figure 2.

An efficient production line establishes an orderly flow of work as a product moves from workstation to workstation and on to final assembly. Out-of-station work, sometimes referred to as traveled work, refers to completing unfinished work on major components, for example, the wings, after they have left the wing workstation and moved down the production line to another station, such as mate and final assembly. decline in labor hours, indicating lesser gains in worker efficiency. As of June 2008, the planned hours for these two major stations increased by about 90 percent over the June 2007 schedule, which itself had shown an increase from the 2006 schedule.

The overlap in the work schedule between manufacturing the wing and mating (connecting) it to the aircraft fuselage has been a major concern for several years because it causes inefficient out-of-station work. The contractor continues to address this concern, but the new schedule indicates that this problem will continue at least through 2009. One indicator of its persistence is the projected hours for building the last test aircraft. As figure 4 shows, estimated labor hours increased more than 80 percent from the June 2007 to June 2008 schedules, and the planned hours for wing assembly and for the mate and delivery phases more than tripled.

Prime Contractor Actions to Improve Schedule Management, Manufacturing Efficiency, and Supplier Base

Our evaluation determined that the prime contractor now has good tools and integrated processes in place that should improve its schedule management activities and is also implementing actions to improve manufacturing efficiency, the delivery of parts, and proactive oversight of subcontractors. The effects from these recent actions are not yet fully apparent, and the contractor has not yet accomplished its own schedule risk assessment that could provide more insight into impacts from areas of risk and uncertainty. The coming year will be critical for implementing management improvements in order to accomplish a firm and effective transition from manufacturing a few test aircraft to producing operational aircraft at high annual rates.

Prime Contractor’s Scheduling Management Processes Meet Many Best Practices, but Program Risks Are Not Entirely Visible

The prime contractor demonstrated to us that its schedule management processes largely meet established best practices criteria. With improvements implemented in 2008, the contractor’s management systems meet or partially meet eight of the nine established criteria. For example, the master schedule can identify and track activities associated with over 600 projects. It also establishes the critical path between activities, allowing the program to examine the impacts of schedule delays and determine schedule flexibility. Appendix II discusses our examination of the prime contractor’s schedule management process against best practices criteria in more detail.

The one area not meeting best practices was related to performing a schedule risk analysis, and as a result, the contractor has limited insight into areas of risk and uncertainty in the schedule. The prime contractor has not conducted its own risk assessment that would (1) determine the level of confidence it has in meeting completion dates and (2) identify and apportion reserve funds for contingencies. A thorough risk analysis could improve management insight and subsequent corrective actions on two recurring problem areas in particular: schedule slippage and inadequate management reserve levels. Naval Air Systems Command officials did accomplish an independent schedule risk analysis that indicated that the program could slip more than 2 years based on the productivity risks associated with software development and assembly of the various airframes as well as the time needed to complete all flight testing. Both the contractor and the JSF Program Office disputed the findings of the Naval Air Systems Command schedule risk analysis primarily because the analysis was done without direct involvement of program officials.

Improvements in Manufacturing and Supplier Base Are Ongoing

The prime contractor is implementing changes designed to address the manufacturing inefficiencies and parts shortages discussed earlier. These include (1) increasing oversight of key subcontractors that are having problems, (2) securing long-term raw material purchase price agreements for both the prime and key subcontractors, and (3) implementing better manufacturing line processes. On this latter point, according to program officials, the prime contractor has taken specific steps to improve wing manufacturing performance—one of the most troublesome workstations. Defense Contract Management Agency officials noted that the contractor produced the second STOVL aircraft variant with less work performed out of station than for the first STOVL aircraft. Also, program office and contractor officials report some alleviation of parts shortages and improvements in quality, but also believe that the effects from previous design delays, parts shortages, and labor inefficiencies will continue to persist over the near term.

The lag time between taking action and demonstrating improvement may partly explain why some manufacturing performance metrics are not demonstrating a clear continued rate of quality improvement, as would be desirable and expected for a program ramping up annual production rates. This lag time may be evident in two important metrics—scrap, rework, and repair rates and manufacturing defect rates—both of which have increased somewhat since 2006. Program and contracting officials point out, however, that while this performance is not desirable, these and other metrics compare very favorably with those of prior acquisitions at similar stages of development, including the F-16 and F-22A.

Supplier costs are expected to make up an even more substantial share of total expenses as the program moves further into production. According to contractor officials, efforts are focused on maturing the supply base and working more closely and directly with key suppliers to reduce costs, alleviate parts shortages, and support higher production rates. Key suppliers have struggled to develop critical and complex parts while others have had problems with limited production capacity. For example, the supplier responsible for the advanced electro-hydraulic actuation system had delivered parts with missing subcomponents and parts that were not built to specifications. The major “team mate” supplier responsible for fuselage and tail assembly has experienced delays caused by its limited machining capacity. Given these supplier issues, manufacturing inefficiencies, and accumulating backlog in production, we believe that the program’s plans to accelerate production in the near term adds considerable risk and will be difficult to achieve in a cost-effective manner.

The JSF’s Test Plan Is Improved but Faces Numerous Challenges to Complete Development on Time and on Budget

DOD will make significant investments—in both dollars and the number of aircraft procured—before completing JSF flight testing. DOD’s proposal to accelerate procurement further increases financial risks in a very challenging test environment. DOD’s new test plan adds an extra year to the schedule and better aligns resources, but is still aggressive with little room for error, presenting a formidable challenge to complete system development, support initial operational testing, and, eventually, a full rate production decision. DOD decisions to reduce development test aircraft and flight tests add to the risks, while any additional delays in manufacturing test aircraft will further compress the schedule. The JSF has just begun development flight testing with two test aircraft and has already experienced some setbacks—normal in any program, but of special concern when assets are minimal. Some in DOD forecast that another 2 or more years beyond the 1-year extension just approved will eventually be needed to successfully prove aircraft performance and complete system development. The department has stated that the contractor’s state-of-the-art ground test labs and a flying test bed will mitigate risks in the flight regimen and their use will effectively substitute for flight testing. This approach is promising, but not yet proven.

Significant Investments Made before Development Flight Tests Are Completed

DOD is investing heavily in procuring JSF aircraft before flight testing proves that they will perform as expected. Procuring aircraft before testing successfully demonstrates that the design is mature and that the weapon system will work as intended increases the likelihood of design and requirements changes resulting in subsequent cost growth, schedule delays, and performance limitations. Also, systems already built and fielded may later require substantial modifications, further adding to costs. The uncertain environment as testing progresses is one reason why the prime contractor and DOD are using cost reimbursable contracts until rather late in procurement. Table 5 depicts planned investments—in both dollars and aircraft—prior to the completion of development flight testing. Under the accelerated production plan and using the lower procurement cost estimate prepared by the program office, DOD may procure 360 aircraft at a total estimated cost of $57 billion before development flight testing is completed. This overlap will be further exacerbated should the joint estimating team’s predictions of higher cost and lengthier schedule prove accurate.

Flight Testing Is Still in Its Infancy and Has Fallen Behind Schedule

The JSF program had completed about 2 percent of its development flight testing as of November 2008. Figure 5 shows the expected ramp up in flight testing with most effort occurring in fiscal years 2010 through 2012. Past programs have shown that many problems are not discovered until flight testing. As such, the program is likely to experience considerable cost growth in the future as it steps up its flight testing, discovers new problems, and makes the necessary technical and design corrections.

While the program has been able to demonstrate basic aircraft flying capabilities, it has recently experienced testing delays and has fallen behind the flight test plan established in 2007. At the time of our review, the program had flown about half of its planned 155 flight tests for 2008. The test program currently has two development test aircraft and an integrated airborne test bed, with the following experiences to date: Sixty-five test flights on the original, non–production representative prototype contributed to discoveries in, among other things, landing gear door fitting, aerial refueling operations, and weapons bay functions. The prototype experienced a 3-month delay because of engine bay nacelle vent fan malfunctions that were subsequently resolved.

Initial testing of the first of 12 production representative prototypes began in June 2008—a STOVL variant flown in conventional mode. By the time of our review it accumulated 14 test flights demonstrating important handling qualities and reducing risks associated with, among other things, the landing gear, fuel system performance, and STOVL doors operation. Engine problems delayed full STOVL testing by 6 months.

Thirty-seven flights on the cooperative avionics test bed tested mission system software and demonstrated communication and navigation capabilities.

Looking ahead, the program expects to take delivery of the remaining development test aircraft during 2009 and early in fiscal year 2010. In the same time frame, it also plans to begin flight testing 6 of its 12 production representative prototype test aircraft (CTOL and STOVL aircraft), including the first 2 aircraft dedicated to mission system testing. The first carrier variant development test aircraft is expected to begin full flight testing—including ship suitability testing—in 2010. A fully integrated, mission capable aircraft is not expected to enter flight testing until 2012 by which time DOD plans to have already purchased 241 aircraft for about $43 billion under cost reimbursement contracts.

Program’s Test Plan Extends Development and Relies Heavily on Ground Testing and Simulations to Verify Aircraft Performance

The JSF Program Office developed a new and improved test plan in the spring of 2008 that extended the development period by 1 year, better aligned test resources and availability dates, and lessened the overlap between development and operational testing. The new plan is still aggressive, however, and has little room for error discovery, rework, and recovery from downtime should test assets be grounded or otherwise unavailable. The sheer complexity of the JSF program—with 7.5 million lines of software code, three variants, and multi-mission development— suggests that the aircraft will encounter many unforeseen problems during flight testing requiring additional schedule to rework. Our past work has shown that programs that do not allow sufficient time to address the inevitable problems discovered in testing, run a greater risk of significant cost increases and schedule delays when problems do arise. The joint estimating team noted that the program’s flight test plan assumed a higher productivity than has been seen on recent flight test programs. As such, the joint team believes that flight testing will require an additional 2 years to complete beyond the 1 year already added to development and suggests that more flight test hours will be necessary to test mission systems and the carrier variant in particular.

The Mid-Course Risk Recovery Plan, approved in late 2007, cut two development test aircraft, reduced test flights, and relies more on ground laboratories and simulations to verify performance, adding substantial risk to the program. Our 2008 report discussed the objections from several prominent DOD offices to the Mid-Course Risk Recovery Plan. The Director of Operational Test and Evaluation, for example, identified risks in the revised verification strategy and cited inadequate capacity to handle the pace of mission testing and ship suitability and signature testing. This increases the likelihood of not finding and resolving critical design deficiencies until operational testing, when it is more costly and disruptive to do so.

The test plan relies heavily on a series of advanced and robust simulation labs and a flying test bed to verify aircraft and subsystem performance. Figure 6 shows that 83 percent of the aircraft’s capabilities are to be verified through labs, the flying test bed, and subject matter analysis, while only 17 percent of test points are to be verified through flight testing. The JSF program spent $5 billion on its system of simulation labs and models. Program officials argue that their heavy investment in simulation labs will allow early risk reduction, thereby reducing the need for additional flight testing, controlling costs, and meeting the key milestones of the program’s aggressive test plan.

The JSF program’s simulation labs appear more prolific, integrated, and capable than the labs used in past aircraft programs. The program utilizes 18 labs for development whereas only 9 were used for the F-22A, 7 for the F-18, and 5 for the F-16. According to program officials, the greater number of labs allows engineers to work simultaneously on different development blocks, reducing bottlenecks that may occur in testing. In contrast, engineers of the F-18 and F-22A programs had to interrupt or shut down development on one development block while they were making corrections to another. Also in contrast to past programs, key JSF simulation labs are colocated at a Lockheed Martin plant in Fort Worth, Texas. The F-22A program utilized three locations and two different companies. According to the program office, the colocation of the key testing labs facilitates more seamless integration of key aircraft components. Program officials also noted that JSF labs use the actual aircraft components to a greater extent than labs did in past programs and also have greater software processing capacity. This allows for more realistic data, which should reduce the need for additional flight testing. Further, the JSF utilizes the first fully integrated airborne test bed for mission system testing. According to program officials, the test bed’s design is geospatially proportionate to an actual F-35 aircraft, enhancing its ability to accurately verify aircraft performance.

While the labs appear more prolific, integrated, and complex than those used in legacy programs, concerns about their extensive use in verifying aircraft performance remain. The extent of the JSF program’s planned lab use is unprecedented, but the ability to substitute for flight testing has not yet been demonstrated. In addition, the labs have yet to be fully accredited. Accreditation is required to ensure the adequate capability of labs and models to perform verification tasks. It is critical that the models behave like aircraft to ensure that the system’s performance requirements are being verified accurately. The program office said that it is on track to complete the accreditation of the labs in time to begin verifying system performance. However, the Director of Operational Test and Evaluation reports that the progress of the accreditation support packages is behind schedule and suggests that more flight testing may be needed as the accreditation process reveals the limitations of the models. Some DOD officials are also concerned that the labs will be understaffed. The Director of Operational Test and Evaluation and DOD’s joint estimating team both reported that the program’s current resource plans reduce engineering staff too rapidly. Engineering and test personnel are critical to analyzing the data generated from the labs. Without adequate staff, there is a greater risk that the labs will not be sufficiently utilized, which could, in turn, result in schedule delays or cost increases.

Early Operational Assessment Identifies Several Challenges That May Have Operational Impact If Not Addressed

While the program is projecting that it will meet all key performance parameters, most will not be verified through ground and flight testing until fiscal years 2010 through 2013. In addition, a 2008 operational assessment by the Air Force Operational Test and Evaluation Center pointed out several technical challenges that it believes are likely to have a severe operational impact if not adequately addressed. While some of the report’s concerns are not specific requirements, some DOD officials believe that the shortfalls may adversely affect the JSF’s ability to meet warfighter needs. For example: The current F-35 power system may cause excessive damage to runway surfaces which could limit its ability to operate in certain locations. The program is still evaluating the problem and plans to gather data and conduct further studies when full-scale models or actual aircraft are available. According to a program official, changes to the aircraft’s design or to the current concept of operations may be needed. The program has alerted the services and believes it will have a better understanding of the problem sometime in 2009.

Thermal management challenges hamper the ability to conduct missions in hot and cold environments. The Director of Operational Test and Evaluation reported that an alternative main engine fuel pump to remedy this problem is under development but will not be available before the low-rate initial production Lot 3, which is likely to affect operational testing. The test team aborted a test sortie because of high fuel temperatures in June.

The JSF’s advanced integrated support system aims to improve and streamline aircraft logistics and maintenance functions in order to reduce life cycle costs. The current integrated support system for the JSF prohibits operating two detachments from one squadron simultaneously. This limitation will severely affect current operating practices, which include dividing one squadron of aircraft into subgroups to deploy and operate at different locations.

Conclusions

The JSF is DOD’s largest and most complex acquisition program and the linchpin of efforts to recapitalize our tactical air forces. It is now in its most challenging phase, at a crossroads of a sort. Challenges are many— continuing cost and schedule pressures; complex, extensive, and unproven software requirements; and a nascent, very aggressive test program with diminished flight test assets. Looking forward, the contractor plans to complete work expeditiously to deliver the test assets, significantly step up flight testing, begin verifying mission system capabilities, mature manufacturing processes, and quickly ramp up production of operational aircraft. As such, the credibility of the program’s test plans, manufacturing capacity, and subsequent cost and schedule estimates should become more apparent.

The program must move forward, but given all these challenges, accelerating procurement in a cost-reimbursement contract environment—where uncertainties in contract performance do not permit costs to be estimated with sufficient accuracy to use any type of fixed- price contract—places very significant financial risk on the government. Accelerating plans also does not equate to an ability to deliver to those plans. Because the program’s manufacturing processes are still maturing and flight testing is still in its infancy, incorporating an accelerated production schedule introduces even more risk and uncertainty to the program. Our past work has shown that programs that make production decisions prior to fully proving a system’s design through testing and demonstration of mature manufacturing processes have an increased risk of design and production changes and retrofits of completed aircraft. Until the contractor demonstrates that it can produce aircraft in a timely and efficient manner, DOD cannot fully grasp future funding requirements. DOD needs tangible assurance from the prime contractor that it can meet expected development and production expectations. By accelerating low-rate production quantities before manufacturing and testing processes are mature, DOD accepts most of the contractors’ production and manufacturing inefficiencies. At minimum, the contractor needs to develop a detailed plan demonstrating how it can successfully meet program development and production goals in the near future within cost and schedule parameters. With an improved contracting framework and a more reasoned look to the future, the JSF program can more effectively meet DOD and warfighter needs in a constrained budget environment.

Recommendations for Executive Action

Given the program’s ongoing manufacturing problems and nascent flight test program, we believe that moving forward with an accelerated procurement plan is very risky. This risk is reflected by the extended use of cost reimbursement contracts for low-rate production quantities—a contract mechanism that places most of the cost risk on the government.

As such, to enhance congressional oversight, increase the likelihood of more successful program outcomes, and maintain confidence that the program is on track to meet planned cost, schedule, and performance goals, we recommend that the Secretary of Defense take the following two actions: 1. Direct the Under Secretary of Defense for Acquisition, Technology and Logistics to report to the congressional defense committees by October 1, 2009. This report should include, at minimum, an explanation of the cost and other risks associated with a cost- reimbursable contract as compared to a fixed-price contract for JSF’s future low-rate production quantities, the program’s strategy for managing and mitigating risks associated with the use of cost contracts, and plans for transitioning to fixed-price contracts for production to include time frames and criteria. 2. Direct the JSF Program Office to ensure that the prime contractor performs periodic schedule risk analyses for the JSF program to provide better insight into management reserve, production efficiencies, and schedule completion to allow for corrections as early as possible.

Agency Comments and Our Evaluation

DOD provided us with written comments on a draft of this report. The comments are reprinted in appendix III.

DOD substantively agreed with our first recommendation regarding a report to the Congress on contracting strategy, but believed that the Under Secretary of Defense for Acquisition, Technology and Logistics should be responsible for the report, not the JSF Program Office as stated in our draft. As the milestone decision authority, the Under Secretary is responsible for approving the contracting strategy, contract awards, and the transition to full rate production. We agree with DOD and revised the recommendation accordingly.

DOD also agreed with the second recommendation and will direct that the prime contractor perform periodic schedule risk analysis. In coordination with the JSF Program Office, the department intends to determine an optimum schedule for the contractor that will provide insight into JSF cost and schedule to influence key milestones and decision making.

We are sending copies of this report to the Secretary of Defense; the Secretaries of the Air Force, Army, and Navy; and the Director of the Office of Management and Budget. The report also is available at no charge on the GAO Web site at http://www.gao.gov.

If you or your staff have any questions concerning this report, please contact me at (202) 512-4841 or sullivanm@gao.gov. Contact points for our Offices of Congressional Relations and Public Affairs may be found on the last page of this report. Staff members making key contributions to this report are listed in appendix IV.

Appendix I: Scope and Methodology

To determine the Joint Strike Fighter (JSF) program’s progress in meeting cost, schedule, and performance goals, we received briefings by program and contractor officials and reviewed financial management reports, budget documents, annual selected acquisition reports, monthly status reports, performance indicators, and other data. We compared reported progress with prior years’ data, identified changes in cost and schedule, and obtained officials’ reasons for these changes. We interviewed Department of Defense (DOD), JSF Program Office, and contractor officials to obtain their views on progress, ongoing concerns and actions taken to address them, and future plans to complete JSF development and accelerate procurement. This review was the fifth and final effort under the mandate of the Ronald W. Reagan National Defense Authorization Act for Fiscal Year 2005. We were provided sufficient information to make the assessments contained in this report.

In assessing program cost estimates, we compared the official program cost estimate in the December 31, 2007, selected acquisition report to estimates developed by the JSF program and an independent joint estimating team for fiscal years 2010 through 2015. Because the fiscal year 2010 budget had not been submitted to the Congress at the time of the draft report, some of the report’s findings are based on preliminary cost projections that existed at the time of our review rather than the official program of record. As such, the cost projections in this report may be different than the final fiscal year 2010 program of record. We interviewed members of the joint estimating team to obtain a detailed understanding of the methodology, data, and approach used in developing their cost estimate and schedule risk analysis of the JSF program. We also interviewed JSF program officials to understand the program’s cost estimate methodology, assumptions, and results and to obtain their response to the joint estimating team’s analysis. Based on this analysis, we were able to identify significant differences in the cost estimating methodologies and assumptions used by the joint estimating team and the program office and to determine major risk drivers in the program.

To assess the program’s plans and risks in manufacturing and its capacity to accelerate production from fiscal years 2010 through 2015, we analyzed manufacturing cost and work performance data to assess progress against plans. We compared budgeted program labor hours to actual labor hours, identified growth trends, and noted differences between future labor requirements and current plans to release engineering staff. We reviewed data and briefings provided by the program and the Office of the Secretary of Defense (OSD) to assess supplier performance and ability to support accelerated production from fiscal years 2010 through 2015. We also determined reasons for manufacturing delays, discussed program and contractor plans to improve, and projected expected impacts on development and operational tests. We also reviewed the prime contractor’s schedule estimates and compared them with relevant best practices to determine the extent to which they reflect key estimating practices that are fundamental to having a reliable schedule. In doing so, we interviewed cognizant program officials to discuss their use of best practices in creating the program’s current schedule and interviewed officials from Naval Air Systems Command to understand their approach and to obtain results of their independent schedule risk analysis.

To assess plans, progress, and risks in test activities, we examined program documents and interviewed DOD, program office, and contractor officials about current test plans and progress. To assess progress toward test plans, we compared the number of flight tests conducted as of October 2008 to those in the original test plan established in 2007. We also reviewed documents and interviewed prime contractor officials regarding flight testing, the integrated airborne test bed, and ground testing. To further assess the ground labs and test bed, we interviewed DOD and program officials and toured the testing labs and aircraft at the Lockheed Martin plant in Fort Worth, Texas.

In performing our work, we obtained information and interviewed officials from the JSF Joint Program Office, Arlington, Virginia; Lockheed Martin Aeronautics, Fort Worth, Texas; Defense Contract Management Agency, Fort Worth, Texas; Defense Contract Management Agency, East Hartford, Connecticut; Naval Air Systems Command, Patuxent River, Maryland; Air Force Operational Test and Evaluation Center, Kirtland Air Force Base, New Mexico; Air Force Cost Analysis Agency, Arlington, Virginia; and OSD offices of the Under Secretary of Defense for Acquisition, Technology and Logistics, the Director of Program Analysis and Evaluation and its Cost Analysis Improvement Group, and the Director of Operational Test and Evaluation in Washington, D.C.

We conducted this performance audit from June 2008 to March 2009 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: GAO Assessment of Prime Contractor Schedule Management Processes

Prime Contractor’s Scheduling Management Processes Meet Many Best Practices, but Program Risks Are Not Entirely Visible

The success of any program depends in part on having a reliable schedule of when the program’s set of work activities will occur, how long they will take, and how they are related to one another. As such, the schedule not only provides a road map for systematic execution of a program, but also provides the means by which to gauge progress, identify and address potential problems, and promote accountability.

In general, best practices and related federal guidance call for a program schedule to identify, sequence, integrate, and resource all key activities to be performed, and to understand and proactively address activities that pose critical risks. More specifically, our research has identified nine practices associated with effective schedule estimating. These practices are (1) capturing all activities, (2) sequencing all activities, (3) assigning resources to all activities, (4) establishing the duration of all activities, (5) integrating schedule activities horizontally and vertically, (6) establishing the critical path for all activities, (7) identifying float between activities, (8) conducting a schedule risk analysis, and (9) updating the schedule using logic and durations to determine dates.

Of these nine practices, the JSF program either met or partially met eight with only one not being met. The area that did not meet the practices related to performing a schedule risk analysis. Specifically, the JSF program has not conducted its own schedule risk analysis that would determine the level of confidence it has in meeting completion dates. Further, an assessment is also critical to identifying and apportioning reserves for contingencies. Since the JSF program has not conducted its own schedule risk analysis, it has limited insight into areas of risk and uncertainty in the schedule. Naval Air Systems Command officials did accomplish an independent schedule risk analysis that indicated that the program could slip more than 2 years based on the productivity risks associated with software development and assembly of the various airframes as well as the time needed to complete all flight testing.

In addition to a schedule risk analysis not being performed, we found several other schedule management concerns that further reduce the visibility of manufacturing risks. First, the use of best scheduling practices at the subcontractor level is still being developed, potentially affecting the integration of subcontractor schedules into the integrated master schedule. Integrating prime and subcontractor schedules is critical to meeting program schedules and cost expectations. The prime contractor is working with subcontractors to increase their level of schedule maturity. Another area of concern is that out-of-station work made it difficult to identify specific span times for individual manufacturing tasks. As a result, the detailed information related to the manufacturing work was not visible in the master schedule. Furthermore, because of the program’s enormous size and complexity, the schedule has been difficult to maintain, requiring manual validation processes to ensure its integrity and validity. Ongoing JSF schedule validity will be an area that needs careful attention as it represents a potential weak point in the overall implementation of the integrated master schedule.

Despite this shortcoming, it is also important to recognize the significant progress that the JSF program team has made in the area of schedule management. Since the previous Defense Contract Management Agency schedule review, both the schedule and the processes to manage it have greatly improved. For example, the schedule can track and verify activities associated with over 600 projects. It also successfully captures and sequences key activities and establishes the critical path between key activities, allowing the program to examine the impacts of schedule delays and determine schedule flexibility.

The Sheer Size and Complexity of the JSF Schedule Have Created Major Challenges to Ensuring Schedule Integrity and Validity

The JSF schedule is maintained in Microsoft Project and consists of over 600 individual projects. Because the size and complexity of the schedule is so immense, it has been difficult to maintain. As such, a number of manual validation processes are required to ensure its integrity and validity. To its credit, the contractor has developed custom processes and tools to help manage the program schedule. However, because of its enormous size and complexity, the JSF’s ongoing schedule validity will be an area that needs careful attention as it represents a potential weak point in the overall implementation of the integrated master schedule.

Because the schedule was so large, we reviewed a subset of it, focusing on the delivery of one airframe for each variant of the F-35 being produced (i.e., BF4, AF1, and AF3). This subset schedule covered a time span from August 2006 through September 2014, and we analyzed it against our best practices for effective schedule estimating. See table 6 for the results of our analyses relative to each of the nine practices.

Appendix III: Comments from the Department of Defense

Appendix IV: GAO Contact and Staff Acknowledgments

Acknowledgments

In addition to the contact named above, the following staff members made key contributions to this report: Bruce Fairbairn, Assistant Director; Ridge Bowman; Charlie Shivers; Georgeann Higgins; Matt Lea; Karen Richey; Tim Boatwright; and Greg Campbell.

Related GAO Products

Defense Acquisitions: Better Weapon Program Outcomes Require Discipline, Accountability, and Fundamental Changes in the Acquisition Environment. GAO-08-782T. Washington, D.C.: June 3, 2008.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-08-467SP. Washington, D.C.: March 31, 2008.

Joint Strike Fighter: Impact of Recent Decisions on Program Risks. GAO-08-569T. Washington, D.C.: March 11, 2008.

Joint Strike Fighter: Recent Decisions by DOD Add to Program Risks. GAO-08-388. Washington, D.C.: March 11, 2008.

Tactical Aircraft: DOD Needs a Joint and Integrated Investment Strategy. GAO-07-415. Washington, D.C.: April 2, 2007.

Defense Acquisitions: Assessments of Selected Weapon Programs. GAO-07-406SP. Washington, D.C.: March 30, 2007.

Defense Acquisitions: Analysis of Costs for the Joint Strike Fighter Engine Program. GAO-07-656T. Washington, D.C.: March 22, 2007.

Joint Strike Fighter: Progress Made and Challenges Remain. GAO-07-360. Washington, D.C.: March 15, 2007.

Tactical Aircraft: DOD’s Cancellation of the Joint Strike Fighter Alternate Engine Program Was Not Based on a Comprehensive Analysis. GAO-06-717R. Washington, D.C.: May 22, 2006.

Defense Acquisitions: Major Weapon Systems Continue to Experience Cost and Schedule Problems under DOD’s Revised Policy. GAO-06-368. Washington, D.C.: April 13, 2006.

Defense Acquisitions: Actions Needed to Get Better Results on Weapons Systems Investments. GAO-06-585T. Washington, D.C.: April 5, 2006.

Tactical Aircraft: Recapitalization Goals Are Not Supported by Knowledge-Based F-22A and JSF Business Cases. GAO-06-487T. Washington, D.C.: March 16, 2006.

Joint Strike Fighter: DOD Plans to Enter Production before Testing Demonstrates Acceptable Performance. GAO-06-356. Washington, D.C.: March 15, 2006.

Joint Strike Fighter: Management of the Technology Transfer Process. GAO-06-364. Washington, D.C.: March 14, 2006.

Tactical Aircraft: F/A-22 and JSF Acquisition Plans and Implications for Tactical Aircraft Modernization. GAO-05-519T. Washington, D.C: April 6, 2005.

Tactical Aircraft: Opportunity to Reduce Risks in the Joint Strike Fighter Program with Different Acquisition Strategy. GAO-05-271. Washington, D.C.: March 15, 2005.